Overview

Collecting public data from the web for aggregation, price comparison, or machine learning pipelines often devolves into aggressive, chaotic crawler implementations that overwhelm target infrastructure, trigger IP blocks, and ignore crawling guidelines.

async-web-collector is an asynchronous data collection framework designed with architectural responsibility and predictability at its core: strict concurrency limits, per-host rate limiting, compliant crawler policies, and resilient retry behaviors.

---

Polite and Compliant by Default

Unlike typical unconstrained scrapers that fire hundreds of concurrent requests, async-web-collector enforces polite crawling semantics:

• Robots.txt Adherence: Crawl rules and directives from robots.txt are parsed, cached, and respected before any endpoint is requested.
• Strict Per-Host Rate Limiting: Token-bucket rate limiting enforces maximum requests-per-second per destination host, ensuring upstream servers are never degraded.
• Bounded Concurrency via Semaphores: Resource utilization is strictly constrained via asyncio.Semaphore, preventing socket exhaustion and local memory starvation.
• Respect for Retry-After: In the event of HTTP 429 (Too Many Requests) or 503 responses, the client honors the upstream Retry-After header with exponential backoff and jitter.

---

High-Performance HTML Parsing

The framework supports multiple parsing backends tailored for throughput and expressiveness:

• selectolax Backend: Ultra-fast Modest/Lexbor C-engine parsing capable of evaluating complex CSS selectors tens of times faster than standard Python libraries.
• lxml Backend: Feature-complete XPath and XML parsing for complex document hierarchies.

---

Declarative YAML Pipelines and CLI

Jobs can be configured declaratively through human-readable YAML specifications or invoked directly through a comprehensive Typer CLI:

version: 1
target:
  domain: "example.org"
  rate_limit: 5.0 # max 5 requests per second
  concurrency: 3

schema:
  scope: "article.post-card"
  fields:
    title:
      selector: "h2.title"
      extract: "text"
    url:
      selector: "a.read-more"
      extract: "attr:href"
    category:
      selector: "span.badge"
      extract: "text"

output:
  format: "jsonl"
  destination: "output/articles.jsonl"

Execution via CLI:

uv run async-web-collector run --config config.yaml