The state of AI in production

We catalogue eleven production deployments where retrieval-augmented agents degraded after launch, and group the failures by cause. In eight of eleven cases the retrieval layer was working exactly as specified; the failure was in what the specification assumed about the corpus. Documents drifted, formats changed upstream, and the assumptions baked into chunking and ranking quietly stopped being true. We propose a small set of monitoring probes that would have caught ten of the eleven failures before users did, and report the cost of running them.

A practical method for scoring agent output when no labelled dataset exists. We describe a rubric-based approach used across six client engagements, in which domain experts write graded criteria once and a judge model applies them continuously. We report where rubric scores agreed with later human review, where they diverged, and the two failure patterns that produced most of the divergence. The method is cheap enough to run on every output in production.

Twelve months of operating cost data from four production agent deployments, broken down by inference, retrieval, monitoring, and human review. Human review was the largest line item in three of four cases, and grew as a share of cost in all four. Inference, the line most teams optimise first, was never the largest. We report the numbers, the drivers behind them, and where reduction effort actually paid off.

We argue that designing for handoff from the first week produces materially different architecture than retrofitting documentation at the end, and show two systems built each way against the same brief. The handoff-first build made different choices about configuration, observability, and failure behaviour, and its receiving team made their first unassisted change in four days rather than five weeks. Handoff is a design constraint, not a delivery phase.

An analysis of 31 pilots that did not ship, based on post-mortems conducted with the teams that ran them. Technical failure was the primary cause in only six cases. Procurement, unclear ownership, and the absence of a named production budget appear more often than any property of the model or the system. We group the causes, order them by frequency, and note which were visible before the pilot started.

Treating model output schemas as versioned API contracts rather than prompt artefacts. We describe the migration path used on two client systems, in which every schema change ships as a version with a compatibility test against downstream consumers. On both systems the practice prevented a class of silent failure that had previously reached production, and made model swaps a routine operation rather than a project.

A short position paper on why we cap client count, with delivery timing data from our first two years. Projects staffed by three or fewer people shipped faster at every scope band we measured, and the gap widened with scope rather than narrowing. We argue the mechanism is coordination cost, not talent, and describe the operating choices that follow from taking the data seriously.