Model cost comparison
Pick 2–6 models and set how many tokens one task burns. Effective cost weights the per-attempt price by the SWE-bench Verified solve rate: a cheaper model with a lower score statistically needs more attempts to get there.
| Model | SWE-bench | Cost/attempt | P(solve) | Expected attempts | Effective cost | vs winner |
|---|
Methodology & caveats. Naive model: SWE-bench Verified is read as the probability that one attempt solves the task; expected attempts are then 1/p (geometric distribution) and effective cost = cost per attempt ÷ p. Tokens split a fixed 70/30 input/output (typical agentic loop). SWE-bench measures real GitHub issues — your task may behave differently, scores depend heavily on the agent scaffold, and failed attempts often burn fewer tokens than successful ones. Treat this as an order-of-magnitude comparison, not a quote. Models without a published SWE score or without an API price are excluded — see the full benchmark.