AI pilot: design the evidence first

On August 5, 2026, I abandoned a before-and-after protocol: the baseline was lost, and usage had already changed. I proposed a side-by-side comparison. A September 10 review confirmed the status of the material: a protocol and historical readings, with no controlled comparison carried out. No time saving or business ROI has been demonstrated at this stage.

An AI enablement pilot must be able to produce an answer that goes against the tool. Otherwise, you are running a demonstration. Prepare the evidence before the first enthusiastic reports: comparable work, expected quality, full human cost and the limits of the comparison.

A reference that answers your question

First, write down what you are comparing. A coding agent against work without AI answers a different question from two AI environments compared with each other. In the latter case, a relative advantage proves no gain over working without AI.

Choose a task family representative of real work, bounded to a few hours. Define shared requirements: expected tests, documentation, review criteria and delivery scope. The team rules for coding agents provide a useful framework for keeping obligations identical on both sides.

If you have a reliable earlier baseline, preserve its context. If it is missing, an old bill or a memory of working faster cannot reconstruct it. The proposed replacement here: the same person, in the same period, performs two comparable tasks under two conditions. Describe their complexity, familiarity and dependencies before execution.

Repeating the exact same task gives the second attempt knowledge acquired during the first. Assign conditions to tasks before starting, randomly where possible, then counterbalance the order across subsequent pairs. Some start with A, others with B. This reduces biases; learning, fatigue and differences between tasks remain possible.

Measure through acceptance

Stop the clock at the agreed delivery criterion. A successful first generation can still require corrections and a lengthy review. Record active preparation, execution, correction and review time separately, then add these human-time categories without counting any interval twice.

Also keep elapsed time and interruptions: waiting for a machine can delay delivery without occupying the person. Conversely, reviewing during that wait consumes human time. When another reviewer participates, add their time to the total. Rework following review remains attached to the original task.

Here is a record to complete for each execution, with no results filled in:

Field	What to record
Reference	Task pair, condition, order, person, date, tool and model version
Comparability	Estimated complexity, familiarity, dependencies and shared requirements
Time	Preparation, active execution, correction, review, waiting and interruptions separately
Quality	Criteria met, defects found, required rework and acceptance decision
Costs and friction	Attributable expense, allocation convention, blockers and abandoned attempts

Apply the same quality rubric to both deliverables. Where possible, have them reviewed without revealing the condition used. Keep failures and abandoned attempts in the report: excluding tasks where the tool gets stuck artificially improves its record.

Enthusiasm remains subjective data

In its study published on July 10, 2025, METR randomly assigned 246 tasks to conditions with and without AI, involving 16 experienced developers working on their own repositories. The tools were primarily Cursor with Claude 3.5/3.7 Sonnet. Observed completion time increased by 19%, while participants still estimated after the experiment that they had been 20% faster.

This historical result concerns that setting and those tools. METR now states that it no longer reflects the current impact of models. It therefore cannot support a conclusion about 2026 models. The documented gap between perception and duration is a reason to record the two separately in your pilot.

Volunteering also selects participants. Describe their initial experience and the profiles missing from the pilot; documenting friction does not remove this bias. The original protocol proposed four to six weeks before rotation to allow room for learning. That duration is a practical protocol choice, not a universal scientific threshold. Separate discovery from more settled usage, and keep tool or model changes visible.

API value stops short of business ROI

Consumption valued at API rates describes an equivalent amount of usage. It measures neither an actual paid bill nor the value of delivered work. A quota percentage likewise remains a consumption indicator, with no automatic conversion into euros.

For the assessment, separate attributable expenses, setup, training, maintenance and human time per accepted task. Any allocation of a shared cost must state its convention. Time potentially freed up becomes a business benefit only if its use and valuation are defensible; a few tasks do not prove additional annual capacity.

In the final report, present differences by pair, their spread, quality and unfavorable cases. A short, heterogeneous comparison can remain inconclusive. The agentic workflow principles help bound the work; this protocol measures its human impact, without covering automated CI evaluations or rollout at scale. You can stop, extend or narrow the pilot. The result has yet to be obtained.