How to pick an AI model per task: why capability is "jagged"

An AI model is not universal but sharply strong only where its training data is thick. In 2026, models exceed human baselines in verifiable domains like math and code where data exploded, yet produce plausible but wrong output where data is thin. Dwarkesh Patel called this "the data black hole at the center of AI." ASAP organizes model selection per task around this jaggedness.

Why capability is jagged

Capability is jagged by domain because the training data differs by domain, as Dwarkesh Patel argues. Models spike in capability in verifiable environments like math and code through reinforcement learning, and fray at the edges where such data is thin. The map of a model's strengths is the map of where data piled up.

Where benchmarks betray real work

This is where practitioners get confused. Top leaderboard scores mostly come from verifiable domains, because tasks whose answers can be graded by machine, like math and code, are the ones reinforcement learning drills hardest. So the benchmark leader often underperforms on the copy in your brand's voice or a delicate support reply. A score measures the sharpest peak of capability, but real work tends to break in the lowest valley of capability. The data-black-hole lens is useful precisely because it reframes a benchmark not as the average of ability but as a biased sample of it.

How to choose by task

Model selection is matching a data-thick model to the nature of the task. The criteria are as follows.

Task	Model to prefer	Why
Reasoning, math, code	Top reasoning model	Verifiable data is thick
Bulk, simple, repetitive	Small fast cheap model	Cost and speed first
Long documents, codebases	Large context model	More at once
Writing, tone, summary	Large general model	Range of expression

The more verifiable and precise the task, the more a top model makes sense; the more simple and bulk, the more a cheap fast model does.

The most expensive model is not always the answer

Using the most expensive model for every task is pure waste, since a small model can cost 10 times less. Bulk simple work like classification, extraction, and summarization runs fine on a small model, and the cost differs several to dozens of times. Conversely, using a cheap model for a task that needs one precise inference only gets you a wrong answer faster.

Porting this to non-English teams

For teams working in a language other than English, the friction is that thinner-data languages behave like the "data-thin" edge of a model. That makes the verifiability test even more valuable. Summary or tone work in such a language can look plausible while being hard to verify, so rather than reflexively bolting on the top model, it is safer to keep a step where a human eyeballs a sample. A practical approach is to split a task into verifiable pieces (extraction, classification) and unverifiable ones (expression, judgment), routing the former to cheap models with automatic grading and the latter to large models with human review.

Trace it back to verifiability

After choosing a model, ask "can I verify the answer to this task." Verifiable tasks are where models are strong and mistakes get caught. Tasks that are hard to verify are risky on any model, so keep human judgment in the loop, all the more where data is thin.

Source: ASAP analysis grounded in Dwarkesh Patel, "The data black hole at the center of AI" (2026; jagged capability concentrated in verifiable domains).