An AI-powered, self-healing web scraper for e-commerce price & stock monitoring. When a site changes its layout, Crawlr repairs its own selectors — so your monitoring never silently dies.
$ crawlr watch "https://store.example/product/123" --target 25 Watching site #1 (trigger=price_below, target $25, every 60m) $ crawlr watchlist ID Product Current Was Change Stock Status 1 Wireless Headphones $219 $249 ▼ 12% yes price dropped
If you need to know when something on the web changes, Crawlr is for you.
Track rivals' prices across many products and react before they undercut you.
Get pinged the moment a product hits your target price.
Never miss a restock on hard-to-get items.
Bundled presets track listings and headlines — or add your own vertical.
Built for continuous, hands-off competitor monitoring.
Detects breakage after a redesign and regenerates selectors automatically — no maintenance.
Watch any product's price and availability, with full history and change detection.
Alert on any change, price drops, a target price, or back-in-stock — your choice, per watch.
Describe what happens in different circumstances in one editable crawlr.rules.yaml.
Ships with e-commerce schemas; add new verticals (jobs, real estate) as simple YAML.
Webhook, Slack, and email notifications when something you care about changes.
Works with zero API keys via the heuristic engine. Add an OpenAI key only if you want.
Proxy rotation, robots.txt compliance, request jitter, and JS rendering when needed.
Concurrent monitoring, a built-in scheduler daemon, and SQLite or Postgres storage.
Crawlr uses the model only to generate & repair selectors — then extracts deterministically.
| Pipe every page to an LLM | Crawlr | |
|---|---|---|
| Cost per page | LLM call every time | ✓ once per site, then free |
| Speed | Slow (model latency) | ✓ deterministic & fast |
| Consistent output | Non-deterministic | ✓ stable selectors |
| Survives redesigns | Re-prompts, re-pays | ✓ self-heals automatically |
| Runs offline | Needs an API key | ✓ free heuristic mode |
A layered pipeline where every stage cross-checks the next — for accuracy and stability.
Static → auto JS, proxy rotation, robots.txt, request jitter.
JSON-LD, microdata & OpenGraph — the reliable, redesign-proof source.
Cached CSS selectors with self-healing regeneration on breakage.
Merge sources, resolve disagreements, and score per-field confidence.
Types, required fields, and a plausibility guard against bad data.
Only real, meaningful changes reach you — via your chosen trigger.
Up and running in under a minute — no API key required.
Install in one line with pipx (or pip) — no clone needed.
Point Crawlr at a product URL and set an optional target price.
Run the scheduler and get alerted when the price or stock changes.
# 1 — install (one line, no clone needed) pipx install crawlr # 2 — watch a product's price and stock crawlr watch "https://store.example/product/123" --target 25 # 3 — see your watchlist and keep monitoring crawlr watchlist crawlr monitor --daemon
Run the built-in JSON API and drive Crawlr from any app or language.
# start the API (protect it with a key) CRAWLR_API_KEY=secret crawlr serve --host 0.0.0.0 --port 8000 # watch a product from anywhere curl -X POST http://localhost:8000/api/watch \ -H "X-API-Key: secret" \ -d '{"url":"https://store.example/p/1","trigger":"price_below","target_price":25}'
Built-in schemas: productproduct_list jobsreal_estate news+ custom YAML
The quick answers. Full details in the docs.
No. Crawlr runs fully offline using a free heuristic selector engine. An OpenAI or Anthropic key is optional and only improves accuracy on unusual layouts — and even then the model runs once per site, so cost is tiny.
Crawlr detects when its cached selectors stop matching and automatically regenerates them — so your monitoring keeps working instead of silently breaking.
Choose a trigger per watch — any change, price drop, a target price, or back-in-stock —
and connect a webhook, Slack, or email. For richer logic, edit a single
crawlr.rules.yaml.
Yes. The engine is general-purpose; add new verticals (jobs, real estate, leads) as plain YAML schemas — no code.
Crawlr honors robots.txt by default. You're responsible for respecting each site's terms of service — point it at sites you're allowed to monitor.
Free, offline, and self-healing. Install it and watch your first product in under a minute.