How to Crawl Interactive Media via Conifer/Webrecorder

Not everything on the web sits still long enough to be captured. Some pages load content dynamically, some depend on user interaction, and others break entirely when a traditional crawler tries to scan them. If you’ve ever tried to archive a site with embedded maps, comment widgets, or canvas-based games, you’ve likely run into blank snapshots or half-loaded pages.

This is where Conifer - formerly known as Webrecorder.io (https://webrecorder.io) - comes in. Built specifically to capture interactive media, Conifer gives you the tools to archive what most crawlers can’t: pages that respond to scrolling, clicking, or real-time scripts. It doesn’t just request a page and log the response - it records a session, like a user browsing the site in real time.

Whether you’re preserving a digital artwork, investigating a web-based tool, or archiving a platform-dependent page for research, Conifer fills a gap that archive.org or Archive.today can’t always reach. Let’s walk through how it works and why it matters.

Why Traditional Archiving Misses the Mark

Most crawler-based archiving systems, including the Wayback Machine, rely on automated bots that visit a URL, follow the links, and save what they can. That works well for basic HTML pages, especially when you're filtering for clean statuscode:200 responses, as outlined in our guide on fetching only valid CDX results.

But once you introduce elements like JavaScript-driven menus, infinite scrolls, embedded social feeds, or visual novels that respond to mouse movement, things start to fall apart. Crawlers don’t wait for you to interact. They don’t click. They don’t type into fields. So they miss things - sometimes critical things.

Conifer flips this approach on its head. It watches what you do and saves that. It doesn’t guess. It observes.

How Conifer Works Differently

Instead of crawling with a bot, Conifer opens a remote browser session in the cloud. When you log in and start a recording, you're essentially browsing through their infrastructure. As you move through a page - scrolling, clicking, entering data - Conifer captures not just the HTML but also all triggered network requests, dynamic changes, and embedded media.

This means that even complex, JavaScript-heavy sites can be archived in full fidelity. You can capture web-based games, visual essays, product demos, interactive timelines, and more.

It also preserves the context - what you clicked, how the page responded, what was shown and when. You’re not just saving a file. You’re saving an experience.

Getting Started with Conifer

Conifer is web-based and free to use with an account. Just head to https://conifer.rhizome.org/ and sign up. Once inside, you can create a new collection and start recording directly from your browser. It uses a virtual browser environment that handles all the tricky parts in the background.

As you interact with the site, every loaded asset - images, fonts, AJAX requests, API calls - is captured in real time. When you're finished, the recording is saved in WARC format, just like the Wayback Machine uses, but with additional session metadata that reflects your actual navigation.

This is especially useful for preserving non-linear web experiences, such as interactive storytelling pieces or platform UIs that require input before content is displayed.

Where This Fits in Your Archiving Strategy

Conifer isn’t a replacement for large-scale archiving. It’s more like a scalpel - a precision tool for preserving pages that can’t be saved any other way. You wouldn’t use it to crawl 50,000 blog posts, but you might use it to preserve one that matters because of its layout, its embedded commentary, or how it feels in motion.

It pairs beautifully with other archiving methods. You can use Smartial tools to extract static content, filter large CDX result sets, or rebuild dead pages. Then, for the few captures that truly don’t work through traditional means, you bring in Conifer to round out the picture.

And because it saves in WARC, you can always import the results into other systems later - ArchiveBox, local playback environments, or even forensic workflows.

Limitations and Considerations

Because Conifer records from live interaction, it doesn’t scale well. You have to visit each page manually and drive the session. That’s not a bug - it’s the price of capturing the complex stuff. But it does mean you need to be selective.

Also, some heavily authenticated sites or apps using DRM protections won’t record properly. Content behind paywalls or user-specific dashboards may still pose a challenge. But for most public, dynamic media, Conifer works better than anything else out there.

If you’re working on a collection that includes political campaign sites, activist content, indie web projects, or tools likely to vanish without warning, this kind of targeted recording can be the difference between a useful archive and a blank page.

Capture What You Can’t See Twice

Some pages aren’t meant to last. And others were never meant to be captured at all - not because of intent, but because of how they’re built.

That’s where tools like Conifer/Webrecorder matter. They let you save the web not just as data, but as something lived and layered. You’re not just grabbing the surface but walking through the page and letting the archive follow behind.

And sometimes, that’s the only way to save something real. Because once it’s gone, you’ll wish you’d clicked that button.