Export formats

Scrape envelopes

A scrape’s JSON envelope carries the page and what happened to it:

{
  "url": "https://example.com",
  "final_url": "https://example.com",
  "title": "Example Domain",
  "markdown": "This domain is for use in documentation examples without…",
  "structured_data": null
}

structured_data holds the extracted record when a schema or vertical ran. With XHR capture on (browser render), an xhr array carries each matching response — URL, status, MIME, body — capped at 50 entries of 64 KiB. The page formats change what markdown holds: llm is the token-lean rendering, text plain text, html the cleaned markup, raw the bytes as served. Screenshots are PNG bytes, written to --out on the CLI or page.png from the Viewport tab.

magpie batch writes one envelope per URL as jsonl, with an ok/error field per record instead of failing the batch.

Crawl records

On disk, a crawl writes per-run JSONL — <runId>.jsonl, plus zero-padded .part files appended by resumed legs, in leg order. Two shapes, one per mode:

• Corpus — {"url": …, "title": …, "depth": …, "markdown": …}, one record per page, cleaned main-content text, zero LLM.
• Extracted — {"url": …, "extracted": {…}}, the schema’s record per page.

Crawl exports

The GUI converts at export time (the CLI fixes the format at run time):

Format	Contents
jsonl	The records verbatim, one per line — the safe choice for large crawls
json	The same records as one array
csv	One row per page: the extracted fields flattened (nested objects and arrays as JSON strings), url included
sqlite	A database file with the run’s records

Corpus runs export jsonl or json only — the same rule the CLI enforces, so downstream tools see one contract. A CSV’s columns come from the schema; the same schema across runs yields the same columns.

Research and diffs

A research run exports:

• Markdown — the report with its footnotes.
• Sources CSV — one row per footnote: footnote, url, quote, checked_at, content_hash. The hash pins the exact snapshot the quote was checked against.
• BibTeX — one entry per cited snapshot.

The Compare screen and a watch’s diff both produce word-level diffs — - prefixed deletions, + prefixed additions, one per line — copyable as text or written through the save dialog. The CLI’s magpie diff prints the same shape to stdout.

Where all of this lives on disk, and how to back it up, is the next page: Data and cache.