ModelsGround Truth

One million tokens fixes long-document recall

DeepSeek-V4 reports MRCR 1M MMR 83.5 and CorpusQA 1M ACC 62.0 at Pro Max. Strong on paper. Not a buyer corpus guarantee.

4 min readFrontier Surveylong-context, evaluation, deepseek-v4

Capacity is not recall. DeepSeek-V4's paper posts MRCR 1M MMR 83.5 and CorpusQA 1M ACC 62.0 at Pro Max. Non-Think is 44.7 and 35.6. Claude Opus 4.6 still leads both benches in that table.

Quoted claim

The claim is that frontier 1M-token context windows, marketed after DeepSeek-V4, Claude Opus 4.6, and Gemini 3.1 Pro, let teams retire retrieval and trust the model to locate procedural clauses, policy exceptions, numeric thresholds, and cross-references wherever they sit in a packed prompt. Sales decks treat a single needle-in-a-haystack demo or a vendor leaderboard row as proof that production workflows can stuff full PDF corpora, ticket threads, or diligence rooms into context and rely on mid-document recall at scale.

Where made

The claim appears in long-context marketing from model providers, in RAG pilot decks proposing "just put everything in the prompt," and in procurement checklists that accept vendor benchmark tables as evidence. Counter-evidence: DeepSeek-AI, "DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence" (arXiv:2606.19348, posted April 2026). The HuggingFace model card at deepseek-ai/DeepSeek-V4-Pro publishes the same long-context numbers as the paper. DeepSeek selected OpenAI MRCR (OpenAI, 2024e) and CorpusQA (Lu et al., 2026; arXiv:2601.14952) as the 1M-token evaluation suite and re-evaluated Claude Opus 4.6 and Gemini 3.1 Pro under standardized configuration.

Supporting number / method

Architecture and context claim. DeepSeek-V4-Pro ships with 1.6T total parameters, 49B activated, and a 1M-token context window. The paper's hybrid attention design, Compressed Sparse Attention plus Heavily Compressed Attention, targets long-context efficiency: at 1M tokens, DeepSeek-V4-Pro requires roughly 27% of single-token inference FLOPs and 10% of KV cache versus DeepSeek-V3.2. That is an engineering achievement. It is not the same as flat recall across buyer documents.

MRCR 1M (MMR). MRCR measures in-context retrieval, finding inserted needles across long synthetic contexts. Table 6 in arXiv:2606.19348 reports DeepSeek-V4-Pro Max at MRCR 1M MMR 83.5. Claude Opus 4.6 Max scores 92.9. Gemini 3.1 Pro High scores 76.3. The paper states retrieval performance "remains highly stable within a 128K context window" and that "a performance degradation becomes visible beyond the 128K mark," while 1M-token capability "remains remarkably strong compared to both proprietary and open-source counterparts." Figure 9 plots MRCR performance by context length. The HuggingFace blog on DeepSeek-V4 notes 8-needle accuracy stays above 0.82 through 256K and holds at 0.59 at 1M. Strong relative to prior models. Not flat.

CorpusQA 1M (ACC). CorpusQA is a 10-million-token benchmark for corpus-level analysis and reasoning. DeepSeek evaluates at the 1M setting. Table 6 lists CorpusQA 1M ACC 62.0 for DeepSeek-V4-Pro Max versus Claude Opus 4.6 Max at 71.7 and Gemini 3.1 Pro High at 53.8. The paper notes CorpusQA "is similar to real scenarios" and that DeepSeek-V4-Pro beats Gemini-3.1-Pro. 62.0% accuracy on a benchmark corpus is not 100% recall on a buyer's contract annexes. The leading proprietary model still outscores DeepSeek by nearly ten points.

Reasoning mode sensitivity. Table 7 in the same paper shows mode matters. DeepSeek-V4-Pro Non-Think scores MRCR 1M MMR 44.7 and CorpusQA 1M ACC 35.6. Think High reaches 83.3/56.5. Think Max reaches 83.5/62.0. Buyers who deploy default fast mode without Max reasoning get roughly half the long-context retrieval the headline table advertises. DeepSeek-V4-Flash Non-Think is worse: 37.5 MMR and 15.5 ACC at 1M.

What the benchmarks do not cover. MRCR uses controlled needle placement. CorpusQA uses benchmark corpora with defined tasks. Neither substitutes for placement sweeps on buyer PDFs with the buyer's tokenizer, system prompt, tool outputs, and chat history interleaved, the production prompt shape Liu et al. showed creates U-shaped recall even before 1M tokens. DeepSeek did not evaluate GPT-5.4 on these tasks because "its API failed to respond to a large portion of our queries." Leaderboard comparisons are incomplete.

How to reproduce on your checkpoint. Pull DeepSeek-V4-Pro weights or API access. Run MRCR and CorpusQA at 1M with the same reasoning mode you will ship. Non-Think versus Max changes scores by nearly 40 points on MRCR. Then run a buyer placement suite: fixed gold clauses at start, middle, and end of real contract PDFs with distractor documents, scoring whether the model cites the correct clause without retrieval. Report best-, middle-, and worst-case placement. The paper's Figure 9 curve is the minimum bar. Flat performance on buyer corpora is UNKNOWN until tested.

What would have to be true

Mid-document procedural clauses would need stable recall at 1M tokens regardless of position in a production prompt mixing system instructions, tool results, chat history, and attachments, not only in DeepSeek's standardized MRCR/CorpusQA harness. Default deployment mode would need to match the Max-mode scores in vendor tables. A 62% CorpusQA accuracy would need to meet the same error bar as human review on action-changing facts. Extended context length alone would need to predict flat performance on buyer corpora. The paper shows degradation beyond 128K even on synthetic retrieval.

Verdict (supported | overstated | unsupported | too_early)

overstated for action-changing constraints on buyer corpora without retrieval or buyer-run placement tests.

Language we will use instead

"We treat 1M context as a capacity ceiling, not a recall guarantee. Vendor MRCR and CorpusQA scores describe benchmark corpora at specific reasoning modes. We report our own placement suite on the shipped checkpoint and tokenizer before skipping retrieval. For mid-document rules that change actions (approvals, liability caps, data-handling exceptions) we require retrieve-then-cite unless best-, middle-, and worst-case placement passes on buyer documents."

Whether a buyer's document mix flatlines recall at 1M on the checkpoint they deploy is UNKNOWN without buyer-run placement tests.

FAQ

Does a 1M-token window replace retrieval for policy PDFs?

DeepSeek-V4 reports strong MRCR/CorpusQA scores on paper benchmarks. Buyer corpora still need placement tests; CorpusQA ACC 62.0 on Pro Max is not flat recall on your annexes.