The agents and their evals
The agents call models through OpenRouter, and their own eval suite grades 100 cases per run. They run on one person’s private life, and the repository is private, so this page names no agent, no suite and no file.
Margin switches to a cheaper model on its own only if it keeps at least 97% of the current pass rate and the result is strong enough to rule out luck. A model that costs more is never switched to automatically: that trade belongs to a person.
Every face-off
Baseline run #151 on deepseek/deepseek-v4.1-flash passed 88 of 100, at $0.000080 per passed case. Every face-off against it is below, including the ones that failed before they measured anything.
| Run | Candidate | Passed | Per task | Result |
|---|---|---|---|---|
| #152 | deepseek-v4-flash-0731 | 85 of 100 | $0.000037 | Refused |
| #153 | gemma-4-26b-a4b-it | 86 of 100 | $0.000025 | Refused |
| #154 | mistral-large-4-0 | none | none | Failed: the worker stopped mid-run; its spend could not be read |
| #155 | deepseek-v4-flash-0731 | none | none | Failed: the two runs graded different units |
| #156 | deepseek-v4-flash | 78 of 100 | $0.000002 | Refused |
| #157 | gemma-4-26b-a4b-it | 88 of 100 | $0.000058 | Refused |
| #158 | mistral-large-4-0 | none | none | Failed: the worker stopped mid-run; its spend could not be read |
| #159 | mistral-large-4-0 | none | none | Failed: the worker stopped mid-run; its spend could not be read |
| #160 | mistral-large-4-0 | 96 of 100 | $0.000581 | Stronger, and dearer |
| #161 | ministral-3b-2512 | 74 of 100 | $0.000043 | Refused |
mistral-large-4-0 passed 96 of 100 at $0.000581 per task, against $0.000070 for deepseek-v4.1-flash. Margin opened pull request #6 for it as a draft, and our founder merged it on 2026-10-16.
Every run after the merge
| Run | Outcome | Passed | Per passed case |
|---|---|---|---|
| #166 | measured mistral-large-4-0, mistral-large-4-0-20261006 | 94 of 100 | not quoted |
| #209 | measured mistral-large-4-0-20261006 | 91 of 100 | $0.000399 |
Before the switch, run #151 cost $0.000080 per passed case, against the same 100 cases.
Run #166’s recorded bill ($0.019834) is far below what its own metered calls cost ($0.058828): the key’s usage counter read low, a fault since fixed. Its cost is not quoted, rather than quoting the low figure.
That is lower than the face-off’s 8.2× per task. One visible difference is caching: run #209 read 31,488 prompt tokens from the provider’s cache, the face-off 1,968. Pass counts on 100 cases move by a few points from run to run, so read 94 and 91 against 88 as a direction, not a precise gain.
Being tested now
Face-offs against the code on mistral-large-4-0, the switch back included. None of these has been switched to.
| Run | Candidate | Passed | Per task | Result |
|---|---|---|---|---|
| #167 | deepseek-v4.1-flash | 88 of 100 | $0.000087 | Refused |
| #168 | claude-haiku-5.5 | none | none | Failed: the candidate left 6 of the 55 cases the current model graded without a verdict, or graded fewer times |
| #169 | gpt-6-luna-pro | none | none | Failed: the worker stopped mid-run; its spend could not be read |
| #210 | decision-model | none | none | Failed: the harnesses ran but none gave a pass or fail |
| #211 | claude-haiku-5.5 | none | none | Failed: the candidate left 8 of the 55 cases the current model graded without a verdict, or graded fewer times |
| #212 | gpt-6-luna-pro | none | none | Failed: the candidate left 3 of the 55 cases the current model graded without a verdict, or graded fewer times |
Switching back to deepseek-v4.1-flash passed 88 of 100, below the bar, so Margin did not recommend undoing the change.
One set of agents and one merged switch show the loop working on code outside Margin’s own repository. They do not show what a larger workload would save or spend.
The first case study, on our own agent, is here. Want the same measurement on your agents? Start here.