Evaluation without ground truth

We catalogue eleven production deployments where retrieval-augmented agents degraded after launch, and group the failures by cause. In eight of eleven cases the retrieval layer was working exactly as specified; the failure was in what the specification assumed about the corpus. Documents drifted, formats changed upstream, and the assumptions baked into chunking and ranking quietly stopped being true. We propose a small set of monitoring probes that would have caught ten of the eleven failures before users did, and report the cost of running them.

Our annual report on language models in live systems, drawn from a survey of 240 teams running them in production. Covers deployment patterns, evaluation practice, cost structure, and the gap between pilot and production. This year the gap widened: more teams ran pilots than last year, and a smaller share of those pilots shipped. The teams that did ship look different in consistent, reportable ways.

Twelve months of operating cost data from four production agent deployments, broken down by inference, retrieval, monitoring, and human review. Human review was the largest line item in three of four cases, and grew as a share of cost in all four. Inference, the line most teams optimise first, was never the largest. We report the numbers, the drivers behind them, and where reduction effort actually paid off.

We argue that designing for handoff from the first week produces materially different architecture than retrofitting documentation at the end, and show two systems built each way against the same brief. The handoff-first build made different choices about configuration, observability, and failure behaviour, and its receiving team made their first unassisted change in four days rather than five weeks. Handoff is a design constraint, not a delivery phase.

An analysis of 31 pilots that did not ship, based on post-mortems conducted with the teams that ran them. Technical failure was the primary cause in only six cases. Procurement, unclear ownership, and the absence of a named production budget appear more often than any property of the model or the system. We group the causes, order them by frequency, and note which were visible before the pilot started.

Treating model output schemas as versioned API contracts rather than prompt artefacts. We describe the migration path used on two client systems, in which every schema change ships as a version with a compatibility test against downstream consumers. On both systems the practice prevented a class of silent failure that had previously reached production, and made model swaps a routine operation rather than a project.

A short position paper on why we cap client count, with delivery timing data from our first two years. Projects staffed by three or fewer people shipped faster at every scope band we measured, and the gap widened with scope rather than narrowing. We argue the mechanism is coordination cost, not talent, and describe the operating choices that follow from taking the data seriously.