The AI Bill That Was Paying for HTML

  • 94% fewer tokens per request
  • AI bill: from $11,400 to $1,180 a month
  • $122,000 saved a year
  • 94% Fewer input tokens per request
  • $11.4k to $1.2k Monthly AI bill, before and after
  • $122k Saved per year

The situation

An online retailer used a large language model to write product descriptions, tag new listings and draft replies to customer emails. It worked, and that was the problem: it worked well enough that nobody looked inside. In four months the monthly API invoice went from $2,900 to $11,400, while the number of products and emails grew by a few percent. Finance flagged it, the team assumed prices had gone up or that the model had become chattier, and the first fix on the table was to switch to a cheaper model and accept worse descriptions. Before anyone downgraded anything, they asked us to find out where the money was actually going.

The real problem

The model was not the expensive part. The input was. Every request in an AI pipeline is billed for the tokens you send as well as the tokens you get back, and on this pipeline the tokens sent had quietly exploded. We pulled a day of request logs and measured the prompt size of every call. A product description should need the product name, its attributes and a short brief, a few hundred tokens. The average request was carrying more than sixteen times that.

The cause sat one step before the model, in preprocessing. The pipeline was meant to take the supplier’s product page, strip it down to clean text and pass that text on. After a change to the scraper months earlier, the cleaning step had started failing silently on most pages and falling back to the raw page: navigation menus, footer links, inline CSS, tracking scripts and, on some pages, images embedded as base64 text. The model still wrote decent descriptions because the real product details were in there somewhere, buried in markup it was paying to read.

A second fault multiplied the first. When the model’s answer for one item in a batch failed to parse, the retry logic resent the entire batch, not the one failed item, and it retried up to five times. A single bad item could send the same oversized batch six times. Nothing in the system counted tokens per job, so there was no alarm, only a bill that arrived at the end of the month.

The results

Fewer input tokens per request
94%
Monthly AI bill, before and after
$11.4k to $1.2k
Saved per year
$122k
Faster responses
3×
Whole-batch retries left
0
Alert on a token regression
Same day

The approach

Measure before changing anything

We logged the token count of every prompt and response for a full day and grouped them by job type. Within an hour the picture was clear: description jobs averaged far more input tokens than their content justified, and a small share of batches accounted for a large part of the spend because they had been retried several times. The cheaper-model plan would have cut quality without touching either cause.

Fix the preprocessing at its root

We rebuilt the cleaning step so it extracts only the product content: title, attributes, specifications and the supplier’s description, deduplicated and trimmed. When extraction fails it now stops that item and reports it, instead of falling back to the raw page. One function owns what “clean input” means, and every job type uses it.

Retry the item, not the batch

Failed items are now retried on their own, with backoff, at most twice, and a parse failure returns a clear error for that one item. Identical prompts are cached, so re-running a job on unchanged products costs nothing.

A token budget with an alarm

Every job now records its token use, and each job type has a budget per item. A request that goes over it is logged and flagged the same day, so a silent regression shows up as an alert instead of an invoice.

94% Fewer input tokens per request

Get in touch

Tell us what you want built. We tell you where to start and what it takes.

What people ask us first

Why not just switch to a cheaper model?

Because the model was not the problem. A cheaper model would have kept reading the same raw HTML and the same repeated batches, at a lower price per token but with worse descriptions. Fixing the input brought the bill down further than any model switch could have, and quality stayed where it was.

How do you find where AI spend goes?

Log the tokens of every prompt and response, grouped by job type, for at least a day of real traffic. Compare the input size with what the task actually needs. Oversized inputs, repeated requests and missing caches show up quickly once you can see them.

Could this happen again after the fix?

The same failure would now be caught the day it starts: extraction failures stop the item instead of falling back to raw pages, and every job type has a token budget with an alert. A regression shows up as a flag, not a surprise at the end of the month.

ByFounder and Imaginator

Published