Nobody Has Measured Which Copilot Model Is Best

Search for the best GitHub Copilot model for your language and you will find confident rankings. Model X for Python. Model Y for TypeScript. Ordered lists, sometimes with numbers beside them.

Almost none of them are measurements.

They are somebody's impression after a week of use, or a summary of a benchmark that tested the underlying model in a completely different harness — different system prompt, different tool access, and without the repository context Copilot supplies.

That last part matters more than it sounds.

Copilot is not a thin wrapper around a model

Copilot does not hand a model your prompt and nothing else. It assembles context: open files, related files, instruction files, sometimes an index of the repository. The assembly is Copilot's, not the model's.

So a benchmark of the raw model is measuring a different system from the one you are using. The score might correlate with what you experience. Nobody has published the work that would tell you whether it does.

What is actually knowable

Three things, and they are enough to decide with.

1. GitHub groups models by task shape. The picker is long, but the documentation sorts models into a small number of groupings — broadly, models tuned for speed, for general-purpose work, for reasoning, and for visual input. That grouping is published, maintained by the people who built the routing, and it changes when the line-up changes.

That is a far better basis for a decision than a ranking assembled by feel, because it is a statement about what each model was optimised for rather than a claim about which is better.

2. Models cost different amounts of your allowance. Requests against different models consume different amounts of your premium request allowance. This is published, it is a number, and it is the part most people ignore until the month they run out. Using a heavyweight reasoning model for "rename this variable" is not just slow — it is expensive in a currency you have a fixed amount of.

3. Preview is not generally available. A model in preview can change behaviour, change availability, or disappear. If you have written a model name into a config file or a team convention, check its status before relying on it.

The rule that survives the next release

Pick by task shape, not by model name. That rule will still be the rule after the next six models ship, which is not true of any ranking.

"But what about Python specifically?"

Nothing language-specific, because the grouping decision dominates it. A reasoning model debugging unfamiliar Python beats a fast model debugging unfamiliar Python by a much larger margin than any two reasoning models differ from each other on Python.

If your language is one where the assistant is visibly weaker — and there are some — the fix is almost never a different model. It is an instructions file stating the conventions it keeps getting wrong. A rule like "money is Decimal, never float, and rounds half-up at two places" changes output in a way no model swap does, because it supplies information the model did not have rather than asking for more effort.

Before switching models to fix a quality problem, spend ten minutes writing three non-obvious rules about your codebase into .github/copilot-instructions.md. Higher leverage, roughly every time, and it applies to every model.

I did not publish a ranking either — here is the apparatus instead

It would be easy for me to write the listicle. I have the traffic incentive to. What I have published instead is the harness that would measure it, with no results in it:

• Three tasks, each a small repository with a failing test suite. Pass/fail comes from the tests, not from a rubric — there is more than one correct answer and scoring resemblance to mine would measure conformity.
• Two properties enforced before a task counts: the tests must fail on the starting repository (otherwise every agent scores a point for doing nothing), and pass with the reference solution (otherwise the task is broken, not hard).
• allowed_paths never includes tests/. An agent that edits the test suite to make it pass is recorded as having touched a disallowed path.
• The runner refuses to write a record without an agent version, and marks any run outside the documented isolation as unpublishable.
• The intervention count is typed in by a human, because no script can detect that somebody nudged an agent. When nobody answers it is null, and the scorer refuses to sum null as zero — "not measured" and "zero" are different claims and only one of them is flattering.

results/ is empty. Running it needs paid accounts for several products I do not have, and publishing plausible-looking figures from an unrun benchmark is the specific failure the whole design exists to prevent.

If you have the accounts, clone it and run it. If you publish numbers that disagree with whatever I eventually publish, that disagreement is the most useful thing either of us could produce.

The harness and the method · The longer version of this argument

Cross-posted from The Copilot Stack. If you just want the decision rather than the argument, the model-choice lesson and the Model Selector apply the grouping to the current line-up.