Updated 2 Sep 2026 • 3 mins read
Repo Rightsizing: Audit Every Model Call You Shipped
Repo rightsizing is auditing every model call in a codebase you already shipped to find oversized models, unbounded outputs, and bloated prompts. This guide shows how to find every call, what to check on each, and the rightsizing moves that cut AI cost without changing what the product does.
Here is how the waste gets in. During the sprint to ship an AI feature, you reach for the most capable model so nothing fails in the demo, hardcode it, set no output limit because you are moving fast, and stuff the whole document into the prompt because it is easier than chunking. It works, you ship, and you never go back. Multiply that by every model call in the repo, and the first big invoice is a bill for decisions you made under deadline and forgot about.
Repo rightsizing is the fix: going back into a codebase you already shipped and auditing every model call, one by one, for the cost decisions baked into it. It is not a rewrite and it rarely changes what the product does. It is a targeted pass that finds the frontier model doing a small model's job, the unbounded response, and the bloated prompt, and corrects them. This guide is how to run that pass.
Key Takeaways: Repo rightsizing is auditing every model call in a shipped repo for cost, without changing product behavior. Shipped code is full of oversized calls because models are chosen under deadline, with no cost signal at the moment of writing them. Find every call by searching the codebase for provider SDKs and endpoints, and by cross-checking against gateway or provider logs. Audit each call on model choice, output cap, context size, caching, retries, and batch eligibility. The biggest single lever is model choice: downgrading a frontier model to a smaller one on routine calls often cuts that call's cost 10 to 40x. Make it durable with a gateway, per-call cost visibility, and CI checks, so new oversized calls are caught before they ship.
Why Shipped Repos Are Full of Oversized Calls
The root cause is that cost is invisible at the moment you write a model call. An engineer choosing a model in an editor gets no feedback that this line will cost twenty times that line; both are one function call. Under deadline, the safe choice is always the most capable model and the most generous limits, because the goal is to make it work, not to make it cheap. So repos accumulate frontier-model defaults, missing output caps, and whole-document prompts, none of which is a bug, all of which is spend. This is the same gap between list price and real cost we unpack in the true cost of tokens.
Step 1: Find Every Model Call
You cannot audit what you cannot locate, so start by building a complete inventory. Two methods together get you full coverage. Search the codebase for the fingerprints of model calls: provider SDK imports and client names (openai, anthropic, and others), endpoint strings, and the method names that send completions or messages. That finds every call site in source. Then cross-check against runtime data, gateway or provider logs, to catch calls made through wrappers, dynamic model names, or dependencies that a code search misses. The source search tells you where the calls are; the logs tell you which ones actually run and how often, so you can rank the audit by real spend.
Step 2: Audit Each Call
For each call site, run the same checklist. Most waste falls into six patterns
| What to check | The waste it catches | The fix |
|---|---|---|
| Model choice | A frontier model on a routine task | Downgrade to the smallest capable model |
| Output cap (max_tokens) | Unbounded, verbose responses | Set max_tokens to the real need |
| Context size | Whole documents stuffed into every prompt | Trim, chunk, or retrieve only what is needed |
| Caching | Fixed system prompts and context re-sent each call | Enable prompt caching on the stable prefix |
| Retries | Silent retry loops multiplying calls | Bound retries and handle errors explicitly |
| Batch eligibility | Synchronous calls for non-urgent work | Move to the batch API for around half price |
Step 3: The Rightsizing Moves
Each pattern has a proven, low-risk correction:
- Downgrade the model. The single biggest lever: routing a routine call from a frontier model to a smaller one often cuts that call's cost 10 to 40x. Test output quality on a sample before rolling out, but most classification, extraction, and formatting tasks do not need the flagship.
- Cap the output. Set max_tokens to the length the response actually needs; output tokens cost several times input, so unbounded generation is a silent multiplier.
- Trim the context. Send only the tokens the task requires, chunk large inputs, and retrieve relevant passages instead of pasting whole documents.
- Cache the fixed prefix. A stable system prompt or reference document re-sent on every call should be served from cache, often cutting input cost by around 75 percent.
- Batch the non-urgent. Move offline and asynchronous jobs to the batch API for roughly half the price.
Step 4: Keep It From Coming Back
A one-time audit decays as new code ships with the same deadline defaults. To make repo rightsizing durable, close the feedback loop that was missing in the first place. Route calls through a gateway so every one is logged with its cost and can be re-audited. Give engineers per-call cost visibility so the expensive line is obvious at code time. And add lightweight CI checks that flag a new call using a frontier model without an output cap, or a hardcoded model name, before it merges. This turns rightsizing from a cleanup into a standard, the ongoing discipline in our AI cost optimization guide and FinOps for AI guides.
The Bottom Line
The reason repo rightsizing works so well is that most AI waste is not a strategy failure; it is a set of small, reasonable decisions made under deadline and never revisited. The frontier-model default that de-risked the demo, the missing output cap that saved five minutes, the whole-document prompt that was easier to write, each made sense in the moment and none was ever meant to be permanent. The audit is simply the moment you go back and make the decisions you would make now, with the bill in front of you.
So treat every shipped AI repo as auditable, not finished. Inventory the calls, check each against the same short list, fix the biggest spenders first, and put a gateway and a CI check in place so the next sprint's shortcuts get caught early. Done once, it often cuts an AI bill by a third or more without touching what users see. Done as a habit, it keeps the code you ship fast from quietly becoming the code that costs the most.
FAQs
Repo rightsizing is auditing every model call in a codebase you already shipped to find and fix cost inefficiencies, oversized models, unbounded outputs, bloated prompts, missing caching, without changing what the product does. It is a targeted cost pass, not a rewrite.
Because cost is invisible when you write a model call. Under deadline, engineers default to the most capable model and generous limits to make the feature work, and never revisit those choices. The result is frontier-model defaults, missing output caps, and whole-document prompts throughout the code.
Combine two methods: search the codebase for provider SDK imports, client names, endpoint strings, and completion or message methods to find call sites in source; then cross-check against gateway or provider logs to catch calls through wrappers or dynamic model names and to rank them by real spend.
Model choice (is a frontier model doing a routine task?), output cap (is max_tokens set?), context size (are whole documents being stuffed in?), caching (is a fixed prefix re-sent each call?), retries (are there silent retry loops?), and batch eligibility (could non-urgent work move to batch?).
Model choice. Routing a routine call from a frontier model to a smaller one often cuts that call's cost 10 to 40x, and most classification, extraction, and formatting tasks do not need the flagship. Verify quality on a sample, then roll out.
Route calls through a gateway so each is logged and re-auditable, give engineers per-call cost visibility so expensive lines are obvious at code time, and add CI checks that flag frontier-model calls without output caps or hardcoded model names before they merge.