Web data crawler and API
Architecture, backend, operations
The problem
My websites needed fresh product data every day, and a paid scraping API was getting expensive while telling me little about why requests failed.
What I built
- Vendor-compatible request and response format, so switching was a URL change
- Per-route pacing, cooldowns, retries, and a total time budget below the CDN timeout
- Content-based block detection, since some block pages return HTTP 200
- Pass-rate tracking per route and page type, and merging of duplicate in-flight requests