CLI reference

Install the CLI

The CLI is the open-source core — AGPL-3.0 or commercial, pure Go, zero CGO, no runtime dependencies. Build it from source:

git clone https://github.com/motherlodelab/magpie.git
cd magpie && go build -o magpie ./cmd/magpie

Cross-compile static binaries with CGO_ENABLED=0. magpie [command] --help is the authoritative flag list; every command prints to stdout unless --out says otherwise.

Commands

scrape — fetch, clean, extract one URL

magpie scrape https://example.com/article --render static
magpie scrape https://example.com/product --schema price.yaml --provider openai
magpie scrape https://example.com/docs --include '.main,.content' --exclude 'nav,footer'

Key flags: --schema (extract structured data), --render auto|static|browser, --page-format markdown|llm|text|json|html|raw|screenshot, --vertical auto|<name>, --browser <fingerprint>, --header-profile, --cookies, --include/--exclude (CSS), --action/--actions (browser DSL: click, type, scroll, wait, wait-for, eval-js — forces browser render), --capture-xhr (Go regexps, browser only, caps 50 × 64 KiB), --cdp-url (drive a running browser), --lang, --provider/--model, --no-cache, --format json|jsonl, --out.

extract — no fetch, stdin or file

cat page.md | magpie extract --schema price.yaml --content-type markdown
echo "$HTML" | magpie extract --prompt "list every price mentioned"

--schema or --prompt, never both; --content-type html|markdown.

batch — up to 100 URLs, zero LLM

magpie batch --file urls.txt --concurrency 4 --format jsonl --out pages.jsonl

One ok/error record per URL; a bad URL never fails the batch. Markdown only.

map — sitemap URLs, zero LLM

magpie map https://example.com --format json   # {site, urls, truncated}

BFS to depth 5, gzip- and entity-aware; partial results on dead children or a 25-second budget, flagged truncated.

summarize / diff / brand

magpie summarize https://example.com/article --max-sentences 1
magpie scrape https://example.com --out snapshot.md && \
  magpie diff https://example.com --against snapshot.md
magpie brand https://example.com --out brand.json

summarize caps at 1–20 sentences, enforced even when the model rambles. diff prints -old +new word pairs; identical input prints nothing and exits 0. brand returns colors, fonts, logo, favicon.

vertical — zero-LLM typed extraction

magpie vertical --list                    # all 27 extractors
magpie vertical https://arxiv.org/abs/1706.03762
magpie vertical https://example.com/any --name og

auto dispatch is strict — the permissive extractors (og, article, event, job_posting, local_business, rss, woocommerce_product) only fire with an explicit --name.

crawl — BFS crawl + extract a site

magpie crawl https://example.com/docs --schema page.yaml --path-prefix /docs \
  --max-pages 50 --max-depth 3 --out pages.jsonl
magpie crawl https://example.com --corpus --max-pages 500
magpie crawl --status <run_id>
magpie crawl https://example.com --schema page.yaml --resume <run_id>

Key flags: --schema or --corpus (mutually exclusive), --format jsonl|json|csv|sqlite, --max-pages (100), --max-depth (3), --concurrency (8), --rate (1/s per host), --same-host (default true), --allow-subdomains, --path-prefix, --include/--exclude (URL globs, exclude wins), --no-sitemap, --sitemap-only, --auto-throttle (delay ×2 on 429/5xx, cap 60 s), --ignore-robots (prints a warning), --exporter-cmd (pipe each record to a program), --status, --resume. Binary assets never enqueue.

search — SERP, optional scrape-through

magpie search "vector databases"                    # duckduckgo, zero key
magpie search "launch" --scrape-top 3 --out hits.jsonl

Engines: brave, serper, serpapi, searxng, exa, duckduckgo (default). --scrape-top N scrapes the first N hits through the normal pipeline. Keys are per-engine env vars; MAGPIE_API_KEY never reaches a search engine. Missing keys exit 7 with the engine’s own hint.

research — deep research with verified citations

magpie research --yes --effort quick --max-cost 0.10 "What changed in Go 1.26?" > report.md

Efforts: quick/standard/deep = 1/3/5 sub-researchers and 10/45/100 tool calls. --allow-domain, --deny-domain, --prefer-domain, --search-provider, --from/--to, --judge-provider, --json (the whole ledger). The cap defaults to $1.00 and research stops at 85% of it. Ctrl-C once writes from what is verified; twice stops with facts kept. The app’s Research screen is this, with a GUI (Research).

serve and init — the agent surface

magpie serve                                   # stdio (Claude Desktop)
magpie serve --transport http --addr 127.0.0.1:8089
magpie init --client claude-code               # write the MCP stanza for you

Thirteen tools over MCP, stateless over Streamable HTTP — the full list and setup are on MCP setup.

watch — monitor a page

magpie watch https://example.com/pricing --every 1h        # loop
magpie watch https://example.com/pricing --once            # cron mode
magpie watch https://example.com/pricing --every 5m --webhook https://hooks.local/magpie

Every check stores a snapshot; on change the word-diff prints and --webhook receives one POST {url, changed, old_hash, new_hash, diff} — unsigned, with webhook-id and webhook-timestamp headers. The app’s Watch screen is the always-on version (Monitor).

Global configuration

Variable	Action
MAGPIE_<PROVIDER>_API_KEY	Per-provider key (also engines: MAGPIE_BRAVE_API_KEY, …)
MAGPIE_API_KEY	Fallback key, LLM providers only
MAGPIE_EXTRACT_PROVIDER / MAGPIE_MODEL	Default provider and model
MAGPIE_MAX_COST	Default USD cost ceiling
MAGPIE_CACHE_DB	SQLite store path
MAGPIE_PROXY / MAGPIE_PROXY_FILE / MAGPIE_PROXY_STRATEGY	Proxy pool (round-robin or sticky-host)
MAGPIE_SEARXNG_URL	Self-hosted SearXNG instance
MAGPIE_CDP_URL	Remote browser endpoint
MAGPIE_BASE_URL	Provider base override (e.g. remote Ollama)
MAGPIE_ALLOW_FILE=1	Allow file:// URLs

Key resolution: --api-key flag > MAGPIE_<PROVIDER>_API_KEY > MAGPIE_API_KEY (LLM only) > OS keyring > config file. Standard HTTP_PROXY/HTTPS_PROXY/NO_PROXY are honored unless the MAGPIE proxy settings override.

Exit codes

0 success · 1 generic failure (research stop, watch changed is still 0) · 2 bad flags or malformed input · 3 sitemap-only crawl with zero in-scope URLs · 4 unknown --status run id · 6 the write itself would pass the cost cap · 7 missing key. A crawl’s per-page failures never change the exit code — the counters and the feed carry them.