AI agents in production

Context

Agents show up in four of our products, and none of them is a chatbot. MagicalCalls places a real phone call and holds a conversation with the person who answers. Dispatch agents assign rides on a taxi platform serving more than a million users. SurgeRank measures and defends how brands are cited inside AI search. And a multi-agent dev team writes, reviews and ships code under human review.

They were built over a period where the model layer changed roughly every quarter, which shaped every architectural decision that mattered.

Constraint

Each of these systems acts in the real world. A dispatch agent assigns a car to a person waiting on a street. A voice agent occupies a stranger's phone line. There is no acceptable version of these where a bad output is just a bad paragraph on a screen.

Cost per action was the second constraint. Anything priced per token that runs on every ride, every call or every page is a variable cost that scales exactly with success, which is the worst shape a cost can have.

Decision

We rejected letting agents act on anything irreversible without a check. The line we hold: an agent may act freely on reversible actions, must be verified on costly ones, and never touches destructive ones. That is why the dev team agents ship under human review and dispatch agents handle the routine cases while humans keep the exceptions.

We rejected sending everything to the largest model. On foiparcursonline that meant OCR first and the model second, purely on cost per scan. The same reasoning runs through the rest: cheap deterministic work does the heavy lifting, the model handles interpretation, and both failure modes stay separately debuggable.

We rejected building around one provider's API shape. Models were swapped repeatedly across these products; the ones where the provider was an implementation detail behind our own interface survived those swaps in an afternoon, and the ones where it was not took a week each.

We rejected demos as evidence. An agent that works in a recorded run and fails on the long tail is worse than no agent, because it has already been trusted. Every one of these systems was validated against real traffic before it was allowed to act unattended.

What shipped

MagicalCalls: an AI that places a real phone call, no app required by the recipient, sold at a one-time price rather than a subscription. Dispatch agents that assign rides without an operator for routine cases on a platform with more than a million users. SurgeRank: measuring citations inside AI search instead of classic rankings, and publishing content to move them rather than only reporting on them. And an internal multi-agent dev team that plans, writes, reviews and ships under human sign-off.

Numbers

Four agent systems running in production. Routine dispatch handled with no operator in the loop on a platform serving more than a million users. Model providers swapped multiple times without a rewrite in the products built behind our own interface.

What I'd do differently

I would build the evaluation set before the agent, every time. We wrote evals after shipping on two of these, and until they existed every improvement was an argument rather than a measurement.

I would also put a hard per-action cost budget in the design document from the start. Cost got treated as an optimisation for later, and later it turned into rework on prompts and routing that a number on day one would have prevented.