Switch models only when your evals say it holds.Switch to a cheaper model only when your evals say it holds.
Margin tests cheaper models on your own evals, opens a pull request for the ones that hold, and a revert if quality slips.
SIMULATED sample data below, not a customer
Cheaper only where quality holds.
less per passed outcome on our own agent, 120 of 120 evals still passing.
Both models passed every case, so the evals could not tell them apart and a person chose.
- Beforegpt-4o-mini$9.00per million passed · 120/120 · run #26
- After 2 pull requestsministral-8b-2512$6.92per million passed · 120/120 · run #88
It turns down a saving when quality falls.
On the same agent, a cheaper model failed the evals. Margin refused it and opened no pull request.
More results, each on its own record: the merged switches on our founder’s agents, each re-measured after its merge and the paired evals behind the gate.
67% cheaper per task, and it passed 72 of 120, 0.60 of the incumbent’s quality. No pull request was opened.
The black tick is the gate’s floor: 0.97 of the incumbent’s quality.
The loop, in four steps.
How each step works- MeasureMargin meters every call, and your own evals are the outcome count.
- TestIt tests cheaper models on those evals.
- SwitchOnly a cheaper model that holds quality ships.
- RevertIf quality slips, it goes back to the model that held.
Margin never sees your live prompts.
A meter, not a proxy: we cannot read a request, and an ingest outage cannot stop your agents. Margin reads your repo and your eval results, never sits in your request path, and changes nothing in your code: it opens a pull request you merge, and if quality slips it opens the revert as a second pull request. The exception is a console run of your own evals on our server. What is on the wire.
sends: a pull request you merge
The right model keeps changing. Margin re-tests on its own.
- Value first.
Price an outcome and Margin weighs what a switch saves against the passes it loses. With no price, it reports cost per outcome and says so. The number we measure.
- It keeps checking.
When a model's price drops or a new one ships, Margin tests it on your evals without being asked, and asks an old answer again once it goes stale.
- No model to sell.
Margin builds, hosts and resells no model. Its flat fee is the same whichever model wins on your evals.
How Margin compares to routers, dashboards and model builders
Margin checks before it swaps, and says no when the work doesn’t hold.
Or paste this into Claude Code, Codex or Cursor:
Follow https://trymargin.io/mcp-prompt.md to connect Margin to this agent.