I have a toolset designed to help me understand codebases and application design. I wanted to understand which model was best suited to the job, and which gave the best value for money. I tested two dozen language models. The cheapest decent model got me 80% of the way for half a cent. The last stretch didn’t come from a bigger model. It came from the harness around it.

What I Built

Learn Codebase is a small tool I built to get up to speed on someone else’s code. You point it at a GitHub or GitLab repository, or a folder on your machine, and a language model reads the code. It then writes a set of flashcards and a multiple-choice test about how the project works. It runs as a simple Jekyll site, and talks to any OpenAI-compatible endpoint: LM Studio locally, or OpenRouter for hosted models. There is also a Claude Code version (a /learn-codebase skill) that runs on my Claude subscription.

Which raised the question: which model should I pick to do the work?

Why Test?

I had four questions:

  • Local or cloud? I’ve written on my personal blog about developing with local LLMs and about cloud inference endpoints. This was a chance to put numbers on that trade-off for a real task.
  • Is my go-to model good enough? GLM-5.3-Flash has been my everyday model, because it is fast, reasonably capable and very cheap. But it has lost coherence on me a couple of times, once deleting its own skill files, which had to be recovered from backup. Does it have challengers at its price?
  • How big is the gap to the frontier, and what does closing it cost?
  • What does an agent harness add? Claude Code runs Claude Opus 5.5. How much of its result comes from the model, and how much from Claude Code itself?

How I Tested

  • Three codebases: Learn Codebase itself (JavaScript), Pallets’ itsdangerous (Python) and chi (Go), from 44 to 110 files.
  • The same job every time: 20 flashcards and 10 multiple-choice questions with four options each.
  • Blind judging against the code: headless Claude Code read each codebase and scored every item for correctness and insight. For questions, it also checked whether the answer key was right and whether the wrong options were plausible. It never knew which model wrote what, and every verdict had to cite a file and line.
  • Coverage: before judging, Claude Code listed the 20 key concepts a newcomer needs for each codebase. Each set was scored on how many of them it taught correctly.
  • One number: each result is a percentage of the score the Claude Code skill achieved on the same codebase.
  • Repeats: the important comparisons were run up to nine times, because the judge alone can move a score by about 10 points.

Local models ran in LM Studio on my Apple M2 Max with 64GB RAM. Everything hosted went through OpenRouter, with my zero-data-retention guardrail switched on. The whole exercise cost about $25 on OpenRouter, plus a bit of my Claude Pro subscription time for the judging.

A caveat up front: the judge is Claude inside the Claude Code harness. The sets were anonymised and every verdict needed evidence, but if anything the method favours Claude. Treat differences under 10 points as noise.

The Results

Every tier against Claude Code

Tier Model vs Claude Code Cost per set Time per set
Agent Claude Code skill (Opus 5.5) 100% subscription (about $0.80 API-equivalent) 2 min
Frontier Claude Opus 5.5 via the API 89% $0.55 2 min
Frontier Gemini 4 Argon 88% $0.41 6.5 min
Frontier GPT-6 Astra 85% $1.04 3.7 min
Mid-price GLM-5.3 85% $0.04 2 min
Budget GLM-5.3-Flash 80% $0.006 1.6 min
Budget MiMo-V2.6-Pro / Claude Haiku 5.5 77-79% $0.02 2-4 min
Frontier Grok 4.7 77% $0.38 7 min
Local gemma-4-31b-qat about 70% free 25 min
Local qwen3.6-35b-a3b about 55% free 3-4 min

What each point of quality costs

What I Learned

1. The harness matters more than the model.

This was the most interesting result. Called through the API at the same effort, Claude Opus 5.5 scored 89% of what it achieves inside Claude Code. It was every bit as accurate and insightful; the whole gap was coverage. Claude Code’s sets taught 87% of the key concepts, the API’s about 70%.

My first thought was that the API version simply saw less of the code, so I sent it two to four times as much. It barely moved: 90%, at twice the cost. On the Go codebase it read all 108 files and scored lower. The API runs had the code in front of them. What they lacked was a way to tell which parts mattered. Claude Code explores like an engineer: it looks at the structure, picks what matters, follows references across files and checks its own draft. That is worth more than a bigger model or a bigger context window.

I may investigate harnesses further with smaller models to see just what you can get out of them, especially when combining with targetted post training.

2. Price buys reliability, not much more quality.

The curve is steep. Half a cent buys 80% (GLM-5.3-Flash), four cents buys 85% (GLM-5.3), and 55 cents buys 89% (Opus 5.5). What the frontier tier really bought was reliability: every frontier model got every answer key right, and missed fewer important ideas. The budget models were 93-97% accurate, but occasionally keyed the wrong answer, or left two defensible answers to a question.

3. Max effort is mostly overthinking.

I started the frontier models at the settings they’re rated at on the Artificial Analysis leaderboard: “max” or “xhigh” effort. That was a mistake. GPT-6 Astra at max cost $3.50 a set and took 20 minutes. At high effort it scored the same for $1.04 in under four minutes. Around 90% of its max-effort output was reasoning that never reached the answer. For a well-defined task with limited context, effort is a cost lever in its own right, and probably the next thing worth testing in the other direction: a frontier model at low effort.

4. Reading strategy is the biggest lever for smaller models.

The app has three reading modes. Quick sends what fits in one request. Thorough summarises the codebase in parts first. Progressive reads in rounds, writing as it goes. Moving from Quick to Thorough lifted almost every model, typically by 10 to 25 points, mostly through coverage. For hosted models, Thorough was the best mode; it is now the default.

5. Some models can’t stop thinking.

The app originally asked every model not to reason, which keeps local models fast. GLM-5.3-Flash ignored that, thought anyway, ran out of budget and returned nothing: 8 out of 9 sets failed. With reasoning on, it was faultless. I can’t prove it, but a model that keeps reasoning past its budget and then returns nothing could well explain the odd incoherent moment in longer agentic sessions.

6. A data policy is part of the model choice.

Muse Wonder never ran. Its only provider on OpenRouter is Meta, with no zero-data-retention endpoint, so my guardrail refused every request before anything was sent. IMHO that speaks for itself. Check where your code goes, not just what it costs.

7. Local models are accurate, but narrow and slow.

The best local result, gemma-4-31b-qat, reached about 70% but took 25 minutes a set. qwen3.6-35b-a3b managed about 55% in three or four minutes. A humbling footnote to my January post on local LLMs: qwen3-coder-30b, which I recommended there for autocomplete, scored only 33% here. And devstral-small-2 wouldn’t load at all with KV cache quantisation enabled in LM Studio, the very setting that post suggested switching on. The right model depends on the task, not the leaderboard.

8. Benchmarking your own tool finds bugs.

The test found four genuine defects in the app. Weaker models repeated their first batch word for word. Some replied in a JSON form the parser couldn’t read. There was no way to turn reasoning on. And there was no fallback for models that can’t turn it off. All are now fixed, and several models went from half a set to a full one.

My Take: What Each Tier Buys You

My Iron Triangle post on jimm.uk says you can pick two of fast, cheap and good. Language models bend that a little, but it still holds. Here is how I’d now choose for this kind of task.

Local (free, private, slow). You gain privacy and zero marginal cost; you pay in time and completeness.

  • Use the biggest model your memory allows (gemma-4-31b-qat on 64GB), not a small or code-specialised one.
  • Use Progressive reading, and give it at least 16k tokens of context.
  • Let it run unattended. It’s the right choice when the code genuinely can’t leave the machine.

Budget hosted (GLM-5.3-Flash, Claude Haiku 5.5, MiMo-V2.6-Pro). You gain 80% of the quality for pocket change, in a couple of minutes. For personal learning, this is good enough most of the time.

  • Keep reasoning on, and use Thorough reading.
  • Glance at the answer keys before trusting them.
  • If mistakes matter more than pennies, Haiku 5.5 and MiMo-V2.6-Pro made the fewest at this price. GLM-5.3-Flash stays my default, and nothing at its price beat it.

Mid-price (GLM-5.3). This is the sweet spot. Frontier-level results (it matched GPT-6 Astra) for 4 cents a set.

  • If I’m handing a study set to someone else, this is where I’d start.

Frontier via the API (Claude Opus 5.5, Gemini 4 Argon). You gain reliability: every answer right, and the fewest gaps.

  • Opus 5.5 was also the most consistent and the fastest.
  • Use high effort, not max. Max is overthinking you pay for by the token.
  • It’s worth it when the output will be relied on, say onboarding a team or assessing understanding. Otherwise it’s a lot of money for the last few points.

Agent (Claude Code). You gain completeness, because it reads a codebase the way an engineer would. Nothing else came close on coverage, and it is included in a Claude subscription.

  • For a large or unfamiliar codebase, this is the one to use.
  • Point it at the repository, let it run, and review the result.

In short: with models, you can now have fast, cheap and good enough for most jobs. What you pay more for is reliability and completeness, and the cheapest way to get completeness is a better harness, not a bigger model.

What’s Next

  • Frontier models at low or medium effort. If Opus 5.5 at low effort keeps its accuracy, it could give near-frontier quality at close to budget prices.
  • Agent-like reading in the app. Let the model ask for specific files, or search for symbols, before it writes. That’s the gap the harness comparison exposed.
  • A long-running agentic test of GLM-5.3-Flash against its challengers, to see whether the coherence problems show up under controlled conditions.

References

Summary

I tested local, budget, mid-price and frontier models, plus Claude Code, on writing study material about three codebases, with every item judged blind against the code. Budget models got 80% of the way for half a cent, and the mid-price GLM-5.3 matched the frontier models for 4 cents. Max reasoning effort mostly wasted money. Most of all, the same frontier model scored noticeably better inside an agent harness than through a raw API, and feeding it more code didn’t close the gap. How a model explores the code mattered more than which model it was.

This is a fast-moving field. The models, prices and providers here are as tested in October 2026, and some will have changed by the time you read this.