Building a Self-Hosted Web Scraping API: Lessons From 14,000 Requests a Day

My content sites depend on fresh product data: names, ratings, review counts, specifications. For a while that data came from a paid scraping API. It worked, but the bill grew with every new site, and I had no visibility into why a request failed. So I built my own crawling service. This post covers the design, the numbers, and the mistakes.

The requirements

• Drop-in compatible. My WordPress plugins and pipelines already spoke the vendor’s request format. The new service had to accept the same parameters and return the same JSON field names, so switching was a URL change, not a rewrite.
• Structured output. Not raw HTML, but parsed product details and search results.
• Honest health data. Every request logged with its route, status, size, and outcome.
• Low footprint. It shares a server with production websites and must never slow them down.

The result is a small FastAPI service with SQLite, running in its own container with memory and CPU limits.

Architecture in one paragraph

A request comes in with a token, a target URL, and a scraper type. The service checks a short cache, then picks an exit route: one of several outbound connections, each with its own pacing and cooldown. It fetches the page, decides whether the response is real content or a block page, parses it into a fixed schema, logs everything, and returns JSON. If a route fails, it tries another, up to a fixed number of attempts and a total time budget.

Lesson 1: HTTP 200 does not mean success

The most expensive bug I had was a block page that returned HTTP 200. It was a tiny 2 KB challenge page, and my code treated it as a valid response. Downstream, the keyword tool read it as “this search has no products”.

By the time I noticed, 780 keyword clusters had been marked as empty. They were reset and re-checked, and now the service classifies responses by content, not by status code: expected markers present, sensible size, no challenge markers. A 200 that fails those tests counts as blocked.

If you build anything that scrapes, decide what “success” means in terms of the content you need, and measure that.

Lesson 2: measure pass rate per route and per page type

Different routes behave differently, and so do different page types. Product pages and search pages do not get the same treatment from the sites that serve them. So the service tracks pass rates separately per page type and switches to the configuration that is currently working, probing the others periodically in case conditions change.

That one change turned a vague feeling (“it seems flaky today”) into a number I can act on.

Lesson 3: collapse duplicate work

Several pipelines can ask for the same URL at the same moment, for example two sites researching overlapping keywords. Without care, that means paying for the same page twice. The service now merges identical in-flight requests: the second caller waits for the first fetch and gets the same result.

Lesson 4: respect the timeouts around you

My sites sit behind Cloudflare, which cuts a request at 100 seconds. A crawler call that takes 110 seconds does not fail gracefully. It just disappears. So the WordPress side uses a 95-second timeout, and the service keeps a total fetch budget of 75 seconds with up to 8 attempts on rotating routes. Every layer has a deadline shorter than the one above it.

The numbers

Here is one real measurement window, from about 21:00 on October 9 to 16:24 on October 10, 2026 (Vietnam time), on the main proxy route:

Outcome	Requests
Succeeded	12,511
Blocked	823
Errors	856
Total	14,190

That is about 88% success. Data use for the window was roughly 3 GB, which works out to about US$0.24 per 1,000 successful pages at the proxy price I was paying. Page weight matters a lot here: product pages came in around 142 KB to 283 KB depending on the client profile, and search pages around 260 KB to 390 KB.

That cost profile is why I could later offer the same engine as the DatPilot Data API, priced well below what I used to pay.

Lesson 5: crawl politely, crawl less

The cheapest request is the one you do not make. Two changes cut volume without losing quality:

• Caching recent results, so repeated research within a short window does not refetch.
• Lean research: reading rating counts directly from search results and skipping listings with fewer than 80 ratings before fetching their product pages. Re-running 926 past research jobs showed this removes about 29% of product page fetches, while about 95% of the products that were finally selected still made the list.

Each route also has its own pacing of a few seconds between requests, so traffic stays modest and evenly spread.

What did not work

I also tried compression tricks, mobile pages, cutting downloads short, and rotating IPv6 addresses. None of them helped, and one made things six times worse. Those experiments get their own post: five optimizations that failed.

Takeaways

1. Define success by content, not status code.
2. Log every request with enough detail to answer “why” a week later.
3. Track health per route and per page type, and let the numbers choose.
4. Give every layer a deadline shorter than the layer above.
5. Treat bandwidth as the real cost, and cut requests before you optimize them.

Building this took effort, but I now own the most important input to my business, and I can see exactly what it costs.