The ledger

Every model switch Margin has tested on its own and open-source evals: the model in use, the cheaper one tried in its place, how often the cheaper one passed, and whether the parity gate let it through. A switch is promoted only when the cheaper model is proven to pass within 3% of the one it replaces.

Captured 2026-10-13 from production

The short version

  • 15 switches tested, 81 distinct results. A result re-recorded on the same evals counts once.
  • 5 promoted. 6 of the 15 switches failed the gate on their latest run.
  • Customer results are never on this page. These are Margin's own runs.

Every switch, newest first

“Passed” is the cheaper model's pass rate divided by the current model's on the latest run, so above 100% means it passed more. “Matched, not proven” means that run passed at least 97% as often but the evals were too few, or the two models disagreed on too many, to prove it. “Costs less by” is the price gap on the same evals. It is a saving only where the result is Promoted: a refused switch saves nothing, because it was never made.

TaskFromToPassedEvalsCosts less byResultMeasured
bug-fixgemini-3.8-flashclaude-haiku-5.5101%8997%Promoted2026-10-13
code-editgemini-3.8-flashclaude-haiku-5.5114%9096%Promoted2026-10-13
algorithmgemini-3.8-flashclaude-haiku-5.5105%9196%Promoted2026-10-13
bug-fixgemini-3.8-flashgemini-3.5-flash-lite99%8994%Matched, not proven2026-10-13
code-editgemini-3.8-flashgemini-3.5-flash-lite109%9092%Promoted2026-10-13
algorithmgemini-3.8-flashgemini-3.5-flash-lite100%9191%Matched, not proven2026-10-13
structured-extractiongemini-2.5-flashgemini-2.5-flash-lite100%1470%Matched, not proven2026-10-06
bug-fixgemini-2.5-flashgemini-3.1-flash-lite100%8944%Matched, not proven2026-09-30
algorithmgemini-2.5-flashgemini-3.1-flash-lite105%9167%Promoted2026-09-30
document-extractiongemini-2.5-flashgemini-2.5-flash-lite90%1070%Refused2026-09-18
multi-hop-reasoninggemini-2.5-flashgemini-2.5-flash-lite0%167%Refused2026-09-18
signal-checkgemini-2.5-flashgemini-2.5-flash-lite86%767%Refused2026-09-18
sidekick-taggingclaude-sonnet-4-6claude-haiku-4-588%867%Refused2026-09-18
bug-fixgemini-2.5-flashgemini-2.5-flash-lite95%8970%Refused2026-09-16
algorithmgemini-2.5-flashgemini-2.5-flash-lite94%9198%Refused2026-09-08

How to read it

A cheaper model that passes as often is not enough. On a few dozen evals, a model that matches by luck looks the same as one that matches for real, so Margin requires enough evals to rule luck out before it promotes anything. The same gate runs on a customer's own evals in the console; see case study zero for one agent taken through it end to end.