Measure Before You Optimize: 5 Experiments That Did Not Work

Most engineering blogs publish the wins. This post is about the ideas that looked clever, got tested properly, and failed. Each one would have been easy to ship on intuition. Measuring first saved me from all five.

The context: my self-hosted crawler pays for residential proxy bandwidth by the gigabyte. Every kilobyte per page multiplies across thousands of pages a day, so reducing transfer size is the most direct way to cut cost.

Experiment 1: ask for Brotli or Zstandard instead of gzip

The idea. Brotli and Zstandard usually compress HTML better than gzip. Advertise support for them, get smaller responses, pay for less bandwidth.

What happened. The target server only serves gzip. When the request asked for Brotli, it did not fall back to gzip. It sent the HTML uncompressed, about 6 times heavier than the gzip version.

The lesson. Content negotiation is only as good as the server on the other end. Test what you actually receive, not what the standard says you should receive.

Experiment 2: stop the download early

The idea. The data I need sits in the first part of the page. Parse the stream as it arrives and close the connection as soon as the important fields have been found.

What happened. Transfer barely changed: 264 KB with early cut-off versus 263 KB without. By the time the code decided it had enough, the rest of the response was already in flight and counted against the bandwidth.

The lesson. On fast connections, data arrives in big bursts. The point at which you “stop reading” is not the point at which the network stops sending.

Experiment 3: fetch the mobile version

The idea. Mobile pages are lighter. Request those instead.

What happened. They were lighter, but they showed fewer search results and dropped the review counts, which my product selection depends on. A cheaper page that lacks the data is not cheaper. It just fails quietly.

The lesson. Define the data you need before optimizing the page you fetch. Size is only one column of the comparison.

Experiment 4: rotate IPv6 addresses inside one /64

The idea. My server has a whole IPv6 /64 block, which is an enormous number of addresses. Rotate through them so each request comes from a fresh address, and skip paying for proxies.

What happened. About 140 requests went through, then the challenge pages started. When I added three new addresses from the same block, they were challenged almost immediately: 2 out of 15 requests passed.

The lesson. Reputation is tracked per network block, not per address. A /64 is treated as one customer, because that is usually what it is. I recommended against renting extra IPv6 servers for this, and saved the money.

Experiment 5: sort by popularity and research only the top results

The idea. Products with the most ratings are the likely winners. Sort search results by rating count, research only the top 8 to 10, and skip the rest.

What happened. It missed too many good products. Accessories with huge rating counts, like replacement parts and add-ons, crowded the top of the list and pushed real candidates out.

What worked instead was a gentler filter: skip listings with fewer than 80 ratings among the first results, and keep the three strongest low-rating listings just in case. Re-running 926 past research jobs with that rule cut product page fetches by about 29%, while about 95% of the products that were finally selected still made it through.

The lesson. A good filter removes obvious noise. An aggressive filter replaces your judgment with a proxy metric that does not mean what you think.

The pattern behind all five

Each failed idea had the same shape: a reasonable assumption about how a system behaves, applied without checking. The fixes were not clever. They were measurements:

1. Record the real number before the change: bytes, pass rate, or products found.
2. Change one thing.
3. Record again under the same conditions.
4. Compare the outcome you care about, not just the metric you optimized.

Experiment 3 is the clearest example. The metric I optimized, page size, improved. The outcome I cared about, correct product data, got worse.

I keep a short note for every failed experiment, with the numbers, so I do not repeat it six months later. Writing them up in public is the next step. If one of these saves you a week, it was worth publishing.