Documentation

Overview

Crawlr is a command-line, AI-powered self-healing web scraper for e-commerce price and stock monitoring. Most "AI scrapers" send every page to an LLM — slow, expensive, and non-deterministic. Crawlr instead uses an LLM (or a free offline heuristic) only to generate and repair CSS selectors, then extracts pages deterministically with those cached selectors. When a site's markup changes and the selectors break, Crawlr detects it and regenerates them automatically.

Install

Requires Python 3.10+.

pipx install crawlr        # or: pip install crawlr

Optional extras:

pip install "crawlr[js]" && playwright install chromium   # JS-heavy sites
pip install "crawlr[postgres]"                            # Postgres backend

Quickstart

crawlr watch "https://store.example/product/123" --target 25
crawlr watchlist
crawlr monitor --daemon
crawlr doctor          # verify your setup

How it works

Every scrape flows through a layered pipeline where each stage cross-checks the next:

1. Fetch — static HTTP with retries; auto-escalates to a headless browser for JS-heavy pages. Proxy rotation, robots.txt compliance, and request jitter are applied.
2. Structured data — reads schema.org JSON-LD, microdata, and OpenGraph tags. This is the most reliable, redesign-proof source.
3. Selectors — cached CSS selectors (generated once per site) extract deterministically; they self-heal when they break.
4. Consensus — structured data and selector output are merged; disagreements defer to structured data for numeric fields, and each field gets a confidence score.
5. Validate — types and required fields are checked, plus a plausibility guard that rejects absurd values (e.g. a 100× price jump).
6. Alert — only real, meaningful changes reach you, via the trigger you chose.

Self-healing

The LLM/heuristic runs once per site to produce reusable selectors, which are cached (by host + schema, and by page-content hash). On later runs Crawlr extracts with the cached selectors — fast and free. If the required fields come back empty (the site changed its markup), Crawlr regenerates the selectors and retries automatically. That's why your monitoring doesn't silently die after a redesign.

Confidence & validation

On single-product pages, Crawlr cross-checks structured data against selector output and assigns a per-field confidence (agreement = high, one source = medium, disagreement = low). The overall run confidence is surfaced in the CLI and dashboard. A plausibility guard suppresses implausible price moves so extraction glitches don't fire false alerts. You can require a minimum field confidence before alerting with CRAWLR_MIN_FIELD_CONFIDENCE.

Command reference

Watch & watchlist

crawlr watch "https://store/p/1"              # track price + stock
crawlr watch "https://store/p/1" --target 25  # alert at/below $25
crawlr watch "https://store/p/1" --restock    # alert when back in stock
crawlr watch "https://store/p/1" --interval 30 # check every 30 min

crawlr watchlist shows current price, previous price, % change, stock, target, and status for every watch.

Alert triggers

Rules template

For richer logic across many situations, crawlr init writes crawlr.rules.yaml. When present, it overrides per-watch triggers.

default_action: ignore
rules:
  - when: price_drops_below
    amount: 25
    action: alert
  - when: back_in_stock
    action: alert
  - when: price_increases
    action: ignore

Schemas & verticals

A schema describes what to extract. Built-in: product, product_list. Bundled presets: jobs, real_estate, news. Add your own by dropping a YAML file in your schema dir (precedence: user > presets > built-in):

name: jobs
item_selector: ".job-card"
fields:
  - name: title
    description: the job title
    type: text
    required: true
  - name: salary
    description: annual salary
    type: number

Then crawlr scrape <url> --schema jobs. Validate with crawlr validate-schema file.yaml.

Alerts & digests

Configure any sinks via env vars; verify with crawlr test-alert.

export CRAWLR_ALERT_WEBHOOK="https://webhook.site/your-id"
export CRAWLR_ALERT_SLACK="https://hooks.slack.com/services/..."
export CRAWLR_ALERT_EMAIL_TO="you@example.com"

Prefer a daily roll-up over one alert per change? Use the digest: crawlr digest --hours 24 --send, or run the daemon with crawlr monitor --daemon --digest 24.

Hosted API

Run the JSON API and drive Crawlr from any language. Set CRAWLR_API_KEY to require an X-API-Key (or Authorization: Bearer) header.

CRAWLR_API_KEY=secret crawlr serve --host 0.0.0.0 --port 8000

# endpoints
POST /api/scrape     {"url": "...", "schema_name": "product"}
POST /api/watch      {"url": "...", "trigger": "price_below", "target_price": 25}
GET  /api/watchlist  |  /api/stats  |  /api/digest?hours=24  |  /api/changes

Configuration

Storage

Crawlr defaults to a local SQLite database (zero config). For scale, set CRAWLR_DATABASE_URL to a Postgres URL — the same portable SQL runs on both. Sites, runs, time-series records, and the change log are all persisted.

Reliability

Crawlr archives a compressed snapshot of each fetched page, so you can crawlr replay <url> to re-extract offline and debug selectors without re-fetching. Anti-bot blocks (HTTP 403/429/503, Cloudflare/CAPTCHA) are detected and a browser escalation is attempted. A golden-fixture eval harness (crawlr eval) guards extraction accuracy as a regression gate.

Deploy

Crawlr is a CLI — "deploy" means running the scheduler where it can check on a timer:

• Cron — schedule crawlr monitor.
• Daemon — crawlr monitor --daemon on any always-on machine.
• Docker — docker compose up runs the scheduler with Postgres.

Legal & ethics

Crawlr honors robots.txt by default and applies polite rate limiting. You are responsible for complying with each site's terms of service and applicable law. Point it at sites you're permitted to monitor (your own, competitors where allowed, or sites that permit scraping). Don't use it to overload servers or collect personal data.

← Back to home · GitHub · PyPI