Never optimize the number: seven principles for faster LLM agents

I spent the last few weeks making a production LLM agent faster. Median latency dropped from minutes to under twenty seconds on replayed real traffic, a bit more than an order of magnitude. The model never changed. What follows is what I've learned so far: seven principles, and at the bottom the prompt they became, the one I now hand to any agent or engineer doing optimization work with me. Steal it.

A note on method, since the rules below demand one. The headline came from replaying a fixed set of real production conversations, not demo prompts. Baseline and candidate ran the same model, same harness, same serving path, same delivery rubric, both measured from persisted traces. I tracked median, tail, failed deliveries, and fallback reasons. I'm deliberately omitting exact traces, customer context, and internal eval IDs: where I give numbers they're rounded, and where the detail would identify the workload I describe the failure class instead.

Principle 1: the model decides slowly

Everyone blames typing speed first. Big artifacts, maybe 70 characters a second, so the artifact must be the cost. Then you profile a real failure and find the model spent five times longer deciding than writing. The expensive part is deliberation: which source, which semantics, whether to re-check, what the ask even means. Thinking scales with open decisions, not output length.

Guidance alone doesn't fix this. I've measured instruction-only improvements at roughly 1x on cost. Better prose improves what the model decides; it doesn't reduce how much deciding there is. A three-token reference to a concept the system already defines deletes the decision entirely. That difference is the whole game.

Principle 2: if the agent can't express the answer, it will stall

Sometimes the agent isn't slow. Its action space cannot represent what the user asked for. When that happens, the model may not fail loudly. It may deliberate until the budget is gone and deliver nothing. Inexpressible asks look like latency. I've watched a single ask burn minutes of reasoning and tens of thousands of thinking tokens, twice in a row, because the ask wanted a collection and the grammar could only emit one item at a time.

No prompt fixes that and no thinking budget survives it. The fix was raising the rank of the action space to match the rank of the ask, plus one standing rule: if an ask exceeds what the fast path can express, decompose it and deliver partial output with a note. Never stall. Never drop work silently. Stalls hide inside latency dashboards; a partial delivery with reasons shows up somewhere you can fix it.

Principle 3: every token is wall-clock time

An output token is around 15 milliseconds at typical serving speeds. Input tokens don't count the same: they're read in parallel and cost a couple orders of magnitude less. Reading is cheap, writing is slow, and that asymmetry is why handing the model facts beats making it derive them. A 3,000-token attempt that ends in an error burns 45 seconds before you pay the round trip to notice it and a full turn to fix it. Duplicate calls, re-fetched context, regenerated artifacts: all seconds. Even when tokens are free for you, time never is.

So spend cheap deterministic work to make the expensive probabilistic work land on the first try. Scan the data first. Verify the schema. Resolve what the ask refers to. These models write very well when the context is right, and slowly and wrongly when they have to guess.

Principle 4: exhaust deterministic levers before touching the model

Caching, merged round-trips, parallel tool calls, code generators, validation gates, and tool contracts are repeatable wins. Prompt wording, model choice, and thinking budgets are rented: fragile across model versions and hard to measure in isolation.

Order matters more than people think. Touch the model layer first and every measurement after it is confounded; you can never again say which change did what. Exhaust the deterministic levers, write an audit showing the remaining time is genuinely model-bound, then open the prompt. "The model is slow" means nothing before that audit exists.

Principle 5: compile what repeats

Most of what an agent generates per request is identical across requests. Definitions, conventions, boilerplate, output contracts. That's compile-time work being paid at runtime, by a probabilistic process, on every single request.

Move it into code. The model emits a small typed intent, about 100 tokens, and a deterministic generator expands it into the 10,000-token artifact, correct by construction. Log every request the generator can't express, and promote shapes that keep recurring. The generator grows by measured demand and nobody has to guess what to build next.

Principle 6: harness is not foundation

Most agent problems I've debugged were model-understanding problems wearing an agent costume. Before adding more harness, learn where the model spends its time, what it re-decides on every request, which mistakes it makes reliably.

A demo is easy to make fast. A product is fast across the real distribution, tail included. The gap between those two is where the actual work lives, and it's why so many agent demos never ship.

Principle 7: benchmarks lie unless you make lying hard

The learnings that cost me the most:

• Freeze a blind holdout before you iterate: real cases picked by ID, never read during development, scored once on the final build. If you tuned to the cases you saw, this is where it shows.
• Blind is not the same as representative. Check who generated your eval data and when. A quiet week can be entirely internal traffic and scheduled jobs.
• Metrics misgrade honest work. My scoreboard once graded a correct delivery as a failure because the answer arrived in a format the metric didn't count. Delivery is what the user asked for, not one artifact type, and an honest "that doesn't exist, here's why" is a delivery too.
• Budgets and caps keyed to names the model writes get dodged the moment it renames something. A per-item retry cap of mine was dodged by an innocent rename; the durable fix was an outer per-run budget on an identity the model can't touch.
• Test the serving path production actually runs. Green tests on a sibling path are testing the sibling.
• Verify reviews too. I've received a review with confident, line-cited findings about code that wasn't the code we ran; it had audited a stale checkout. Pin the exact artifact under review, and spot-check two claims before acting on any of them.
• A failed pre-locked gate is the best fuel you'll get. Mine failed by one case, and the fix only counted after a fresh blind set passed. Diagnose the miss to a class, fix the structure, never re-score the burned set.

Other people keep arriving at the same ideas

I read these properly instead of quoting their headlines. Each is a version of "spend fewer tokens, decide less at runtime", and each carries a caveat the headline drops.

• TOON does to LLM input what typed intents do to output: declare structure once, then send pure data. Field headers appear once per table instead of keys repeated on every row, plus a length marker the model can validate against. Around 40 percent fewer tokens than JSON on uniform arrays, 59 percent on time-series data. The caveats are honest and live in the repo itself: CSV is still smaller for flat tables, deeply nested data can come out worse, and some local deployments process JSON faster despite the higher token count. Their advice is to benchmark your own setup, which is the measure-first rule doing its job.
• A generation-side benchmark of TOON found something more useful than the marketing: for simple structures, plain JSON had the best accuracy and constrained decoding the lowest token count, because teaching the model a new format costs prompt tokens before it saves any. Format savings only win past a complexity threshold. Every optimization has a frame.
• Don't Break the Cache measured prompt caching across OpenAI, Anthropic and Google on 500+ real agent sessions: 41 to 80 percent cost reduction, 13 to 31 percent faster first token. It also found naive full-context caching can make latency worse. Cache boundaries are an invariant: static content first, dynamic tool results kept out of the cached block, enforced in your prompt-assembly code rather than left to judgment.
• The LLM-Tool Compiler fuses similar tool calls into batched operations at runtime, the way hardware compilers fuse multiply-adds: up to 4x more parallelism, around 40 percent lower token cost, around 12 percent lower latency. Merged round-trips, formalized.
• LLMLingua is the probabilistic cousin: a small trained model deletes non-essential tokens from your context, up to 20x compression, and its long-context variant improved RAG accuracy by 21 percent while using a quarter of the tokens. Powerful, and worth knowing it's a rented lever: a model deciding what another model gets to see.
• 12-factor agents is the closest thing I've found to this post in spirit: numbered engineering principles for production agents. Own your prompts, own your control flow, tools are just structured outputs. Those factors are about building agents. These rules are about making them fast without lying to yourself. They compose.
• DSPy is the structure-over-steering argument taken to its conclusion: programs and optimizers instead of hand-patched prompts.

The anti-patterns, and how to catch them

Every one of these looks like progress from the inside. The third column is the giveaway:

The prompt

Here's the generalized version. Nothing in it cares what your agent does; the rules are the same whether it writes code, queries, documents, or plans. It's written as instructions to an agent because that's how I use it, pasted into the context of whatever model is doing optimization work. It reads fine as instructions to a human too.

# Optimization rules
 
Never optimize the number. Optimize the thing the number measures.
If the metric improved and the user's experience didn't, stop and say so.
A fast build that delivers nothing is 0x, not infinity.
 
1. Measure before believing. Measure after changing. Same harness, same
   environment, same counting rules. Medians over repeats, ranges
   reported, outliers kept in the record. No claim without a number,
   no number without a method.
 
2. Real data or it didn't happen. Replay real production traffic: the
   worst 10% and a stratified representative slice. Freeze a blind
   holdout before you iterate, picked by ID, never read during
   development, scored exactly once on the final build. Then audit its
   provenance: "nobody read it" is not "it's representative". Check who
   generated it and when. Optimizing demo prompts produces demo
   software; a product is fast across the real distribution, tail
   included.
 
3. Find the real bottleneck before touching anything:
   wall_time ~= model tokens x per-token cost + turns x round-trip
   + tool/DB time + fixed overhead.
   For agents, deliberation dominates: the model decides slowly, it
   doesn't type slowly. Count thinking tokens before blaming the
   database. Wrong outputs are a multiplier, not a footnote.
   Every output token is ~15ms of wall clock: duplicate calls and
   long generations that die in errors are waste even when the
   token budget isn't.
 
4. Deterministic levers first: data layer, caching, merged round-trips,
   parallelism, generators, validation gates. These wins are owned.
   Prompt and model tweaks are rented, and they confound every
   measurement made after them. Open the model layer only after a
   written audit shows the remaining time is model-bound.
 
5. Architecture over prompts. In order of strength: make the wrong
   thing impossible, make the right thing automatic, make it easy,
   guide, and only then steer with prose. Anything that must always
   hold lives in code, keyed to identities the model cannot rewrite.
   If you patched the prompt after each observed failure, you are
   fitting the test set. Ground the model before it writes: scan
   the data, verify the schema, resolve references first. One
   guided attempt beats three blind retries.
 
6. Move repeated reasoning to compile time. The model emits a small
   typed intent; a compiler produces the artifact, correct by
   construction. Grow the compiler from fallback telemetry, never
   from speculation.
 
7. If an ask exceeds what the fast path can express, decompose it and
   deliver partial output with a note. Never stall. Never drop work
   silently.
 
8. Coverage failures are enumerable. Log and classify every fallback
   from real traffic; fix by workload weight, not by the last failure
   you happened to see. Every fix must generalize past the case that
   prompted it, or it's a patch, not a capability.
 
9. Verify adversarially, on the real serving path. Unit invariants
   fuzzed with hostile inputs, live end-to-end with the real model,
   and a fresh-context reviewer told to assume you cheated. Pin the
   exact artifact under review, and spot-check the review itself.
 
10. One change per iteration. Never weaken a test to make a change
    pass. A failed gate is fuel: fail publicly, diagnose to a class,
    fix structurally, prove on new blind data. Never re-score the
    burned set.
 
11. Report like an adversary is reading. Lead with the worst number.
    Every number carries its method (measured / estimated / hoped).
    Keep an honesty ledger of every intentional behavior change.
 
12. Ship behind a kill switch. Flag off = old behavior, byte-identical,
    CI-verified. Staged rollout by blast radius, telemetry watching
    delivery and fallback rates. Nothing ships until the blind holdout
    has spoken.
 
# Before claiming any optimization win
 
1. Measured before AND after, identically, on real data?
   Median + range + method?
2. Delivery and correctness gates passed 100%, and would a blind
   case pass too?
3. Same accuracy proven (parity check or a second-path oracle)?
4. Any gate, test, or rubric weakened along the way? Disclosed
   and justified?
5. Adversarially reviewed by fresh eyes with authority to block?
6. Tested on the real serving path, not a sibling?
7. Does the fix generalize past the cases that motivated it?
8. Honest about the residual that did NOT improve?
9. Kill switch + staged rollout + telemetry for what you missed?
10. Is the blind data actually blind AND actually representative?
11. Does every ask the fast path can't express degrade to
    partial-with-a-note, never to silence?
12. Would every claim survive the person who wrote the baseline
    reading your methodology?
 
If any answer is no, the work isn't done. Say so plainly and keep going.

If you run the prompt against a real system, or you've measured something that contradicts a principle here, I want to know: me at abhishek.it.