Articles

How to calculate ROI on an LLM project without fooling yourself

Published 20 September 2026 · 11 minute read

Almost every LLM business case starts the same way: multiply the number of tasks by the minutes each one takes, multiply by an hourly rate, and present the result as savings. It produces a large number, the project gets funded, and eighteen months later nobody can find the money in the budget.

The calculation isn’t wrong so much as radically incomplete. It models the work the machine does and ignores all the work created around it. Below is a model that holds up, the three inputs that actually decide the answer, and a calculator you can put your own numbers into.

Why the naive number is always too high

Three reasons, in order of how much damage they do.

1. Automation is partial, and the remainder is the expensive part. A model that handles 70% of cases cleanly doesn’t deliver 70% of the savings, because someone still has to work out which 70% those were. If every output gets reviewed, you haven’t removed the labour at all. You’ve converted “doing” into “checking”, and checking a plausible wrong answer can take longer than doing the task from scratch.

2. Errors have a price, and it is not the price of the task. A misrouted ticket costs a few minutes. A wrong number in a customer quote costs you the margin on the deal, plus the relationship. What an error costs is set by where it lands, not by the work that produced it, and the errors that escape review are always the expensive ones.

3. Saved minutes are not money. This is the one that quietly destroys most business cases. Saving four minutes on a task performed by thirty people doesn’t produce cash. It produces slightly less busy people. The saving is only real if you remove a cost — headcount you do not backfill, contractors you stop using, a queue you stop outsourcing, or if you redeploy the capacity to something with measurable value. If neither is true, the honest entry in the model is zero, and it is better to find that out in a spreadsheet than in a post-implementation review.

Before modelling anything: write down what specific cost line goes down, or what revenue line goes up. If you cannot name one, the project may still be worth doing for other reasons, but it does not have an ROI.

The costs that belong in the model

Note where the weight sits. Teams spend weeks optimising token cost, which is often under 2% of the total — while the review labour line, ten times larger, goes unmodelled.

The three numbers that decide the answer

Everything else is detail. These three move the result by an order of magnitude:

• The automation rate (α) — the share of tasks that complete end to end with no human touching them. Not the share where output “looks reasonable”; the share you are willing to let through unreviewed. Pilots routinely report 85% and settle at 50–60% once real traffic arrives, because real traffic contains the messy inputs the pilot set did not.
• Review time on the remainder — minutes a human spends on tasks the model did not fully handle. If reviewing a draft takes nearly as long as writing one, the economics collapse regardless of how good the model is.
• The cost of an undetected error — what it costs when a mistake escapes review, multiplied by how often that happens. This term decides whether a high-volume, low-value task is worth automating at all.

Verification asymmetry: the best predictor there is

If you want one heuristic for whether an LLM project will pay, use this: how much cheaper is checking the answer than producing it?

Where checking is far cheaper — does this code compile and pass tests, does this extracted total match the invoice, is this classification right — automation compounds, because verification is cheap and errors are caught early. Where checking costs as much as doing — is this legal analysis sound, is this strategy memo correct, is this medical summary complete — you get a tool that makes a knowledgeable person slightly faster, which is a real benefit but a much smaller one, and it should be modelled as a percentage improvement rather than as automation.

Tasks with an objective, cheap correctness check are where the returns are. It’s the same property that makes a task easy to put in an evaluation harness, which is no coincidence.

The formula

Use loaded cost, not salary: employer taxes, benefits, tooling and management overhead put the real figure 25–40% above base pay. And remember the earlier warning — baseline_monthly only counts as a saving to the extent you actually remove or redeploy that cost.

Try it with your numbers

The defaults describe a mid-sized support triage workload: 2,000 tickets a month, fifteen minutes each, partially automated. Change anything and the result updates.

Reading the result honestly

The defaults produce a positive case with payback inside four months. Now do the thing that separates a business case from a brochure: run it again pessimistically.

• Drop the automation rate by 15 points. This is the single most over-estimated input, and a pilot-to-production drop of that size is normal rather than unlucky.
• Raise review time by half. Reviewers are slower than the people who designed the workflow assume.
• Double the build cost. Integration, security review and change management are where estimates go to die.
• Raise the error cost to the worst plausible case rather than the typical one.

If the pessimistic case still pays back inside a year, you’ve got a solid project. If it only works on optimistic inputs, what you’ve got is a hypothesis. Structure the work as a time-boxed pilot that measures the real automation rate before you commit to the full build: that measurement is the actual deliverable of phase one.

The revenue side

Cost reduction is the easiest case to model and often the smaller opportunity. The same arithmetic works on the revenue side, and the inputs are usually better evidenced than people expect:

• Conversion — faster or better responses to inbound leads, measured against a held-back control group rather than against last quarter.
• Retention — reduced churn from faster resolution, valued at the margin on retained revenue.
• Capacity — serving more volume without adding headcount, which is a real saving when the alternative hire was actually going to happen.
• Cycle time — shipping or quoting faster, where speed demonstrably wins work.

Hold the same standard here: a control group or a before-and-after with a plausible counterfactual. “Revenue went up after launch” is not evidence when three other things changed in the same quarter.

When the answer is “do not build this”

• The saving is diffuse minutes across many people, with no cost line that goes down.
• Checking the output costs about as much as producing it.
• Undetected errors are expensive and the volume is high, so N × α × e × C_error swamps the saving.
• The process changes faster than you can maintain the automation.
• The real problem is upstream (a bad form, a missing field, an unclear policy) and a model is being used to paper over it. Fix the upstream cause; it is cheaper and it stays fixed.
• Volume is too low for the build cost to amortise. Below a few hundred tasks a month, almost nothing custom pays back.

A clear “no” here is worth as much as a “yes”. It’s a few days of analysis against a build that would have eaten a quarter.

Measuring it after launch

The model is a forecast. Instrument the real thing so you can replace estimates with measurements within a month or two:

• Automation rate — share completed with no human intervention, tracked weekly. Expect it to fall before it rises.
• Review time — actual minutes, sampled properly rather than guessed at.
• Escaped error rate — found through sampling and complaints. If you are not sampling, you are not measuring this.
• Cost per completed task — inference plus review, as one number, trended over time.
• The counterfactual — what the manual process costs now, not what it cost at kick-off.

Recompute the model quarterly with measured inputs. Projects that looked marginal often improve sharply once the automation rate is tuned; projects that looked spectacular sometimes turn out to have moved cost from one team to another. Both are worth knowing, and both are invisible without instrumentation.