Atlas — Findings Ledger (human view)

Distribution

03By axis: A=10, B=8, C=9, D=10, E=8, F=4, G=8, H=30

04By evidence class: benchmark=48, theory=17, manifesto=7, case-study=6, production=6, anecdote=3

05By status vs prior knowledge: refines=46, new=15, contradicts=5, known=5, unclassified=16 (the sixteen are the BLD-* records in axis H, which carry no status field at all)

06Surprise findings: 11 — DR1-011, DR2-003, DR2-006, DR2-015, BLD-003, BLD-007, BLD-023, R2C-002, R2C-003, R2C-007, R2C-008

Axis A — Language shape (10)

• 07DR1-001 [CONTRADICTS] — MoonBit flattened design for transformer KV-cache efficiency manifesto · low · contradicts:H5 — MoonBit (AI-native general language, IDEA/Shenzhen) makes top-level vs local definitions explicit with mandatory toplevel type signatures and structural interface implementation, so interfaces and impls can be generated linearly without nested back-and-forth. Claim: non-nested code is more KV-cache friendly and more readable. Secondary axis E via its sampler.
• DR1-005 — JPL Power of Ten: restrictions chosen for static decidability production · high · new — Holzmann's 10 rules (2006) target restrictions that most increase analyzability per unit of lost expressiveness: simple control flow (no goto/recursion/longjmp), fixed loop bounds provable by a checker, no dynamic allocation after init, functions that fit one page. Used on Mars Science Lab (~3M LoC). The decidability-first selection criterion is the key transferable idea.
• DR1-007 — C++ Core Guidelines profiles + 'break-the-glass' opt-out manifesto · med · refines:H2 — A 'profile' is a deterministically, portably enforceable subset of rules designed to guarantee one property (type, bounds, lifetime safety). P3081 proposes a default 'one big red button' strict mode requiring explicit per-site opt-out. Chisnall's cited insight: it is far easier to prevent code that doesn't compile than code that merely raises analyzer warnings. Secondary axis E.
• DR1-008 — Rust unsafe in practice: sparse, encapsulated, the model escape hatch benchmark · high · refines:H2 — Large-corpus study: only 0.6% of 7.4M functions dereference a raw pointer; unsafe appears in ~24-29% of crates but is mostly hidden behind safe abstractions ('Rust hypothesis'). Confirms a forbidden-by-default-with-recorded-machinery boundary can survive real-world economics — the template for every discipline escape hatch.
• DR1-016 — EsoLang-Bench: frontier models collapse on out-of-distribution languages benchmark · high · refines:H1 — The same 80 problems solved at 100% in Python/JS by top frontier models (GPT-5.4, Gemini 3.1 Pro, Qwen3-235B, etc.) score only 0-11% in esoteric languages; 'peak esoteric accuracy is only 11.2% (GPT-5.4 xhigh, self-scaffolding, Befunge-98) — an 89-point collapse on identical algorithmic content', with esoteric languages having '340x to over 60,000x fewer public GitHub repositories than Python'. The decisive H5 datapoint: distribution, not legibility, dominates. Secondary axis G.
• DR2-014 — Search/replace edit format aligns with pretraining (diff) distribution benchmark · med · refines:H5 — SRI (Search-and-Replace Instruction) infilling outperforms NL-framed FIM; authors attribute it to structural alignment with VCS diff patterns prevalent in pretraining ('Context to Edit Pattern'). NL conversational framing introduces distributional shift. Aider similarly finds weaker models do better with whole-file or specific diff formats matched to capability.
• DR2-024 — Code pretraining/structure improves reasoning; canonical form is in-distribution benchmark · low · refines:H6 — Surveys + studies: code data improves LLM reasoning/planning via structured patterns; structure- and execution-aware modeling (GraphCodeBERT data-flow edges) improves understanding. Code-induced reasoning work shows small structural corruptions significantly reduce reasoning — reinforcing reliance on familiar canonical forms.
• R2C-002 * — EsoLang-Bench: OOD collapse confirmed; ICL gives ~0 benefit; compile errors dominate failures benchmark · high · refines:H5 — Mechanism detail behind the 89-point collapse (Sharma & Chopra, contamination-resistant, 5 esolangs 340x-60000x scarcer than Python). Critical for us: 'in-context learning provides NO significant benefit for OOD tasks' (few-shot +0.8pp, Wilcoxon p=0.505 n.s.) - so you cannot teach an unfamiliar surface form via prompt examples; coverage must come from pretraining. Compilation errors dominate failures (59%). Whitespace 0% across all configs. Confirms H5 mechanism: distribution coverage, not legibility, sets the ceiling, and ICL cannot patch a distribution hole.
• R3-002 — Contract-first ordering: intent before body within every item theory · med · new — Autoregression makes reading order conditioning order. Signature, invariants, error contract, canonical example precede implementation so generation conditions on intent.
• R3-006 — The codebase is the few-shot prompt: uniformity is load-bearing, exceptions must be marked theory · high · refines:H6 — Models imitate visible neighbors. One way per operation; unmarked exceptions become false training signal and propagate; marked exceptions (deviates+reason) do not.

Axis B — Names & tokens (8)

• 08DR1-009 — Identifier renaming alone drops LLM code comprehension 18.6 pts benchmark · high · refines:naming-rule — 'The Code Barrier' (comments stripped, then obfuscations applied) finds variable renaming causes the largest degradation: average description accuracy drops 18.6 percentage points, some models over 30; GPT-4o most resilient (smallest decline; e.g. 58.8% on literal-encrypted code). Implies LLMs lean on lexical semantics in names rather than structure. Secondary axis G.
• DR1-012 — Token efficiency varies 2.6x across languages case-study · med · new — A RosettaCode study over ~1000 tasks across 19 languages (GPT-4 tokenizer) finds a 2.6x spread; terse ASCII array langs (J ~70 tokens/task) and Lisps (Clojure ~109) are most efficient, and typed functional langs (Haskell, F#) are nearly as efficient as dynamic ones due to type inference. More token-efficient code lets agents work longer within a context window.
• DR1-013 — Token Sugar: reversible token-efficient shorthand for code benchmark · med · new — Token Sugar (ASE 2025) mines 799 high-frequency verbose code patterns and maps each to a reversible shorthand, integrated via pretraining; up to 15.1% token reduction in source, ~11.2% at generation, with near-identical Pass@1. Stronger models desugarize with zero failures. A semantic-level complement to grammar simplification (Simpy). Secondary axis C.
• DR2-003 * — Names carry intent; obfuscation collapses summarization but not canonical code benchmark · high · refines:H5 — 'When Names Disappear' (FPT/UTD): semantics-preserving obfuscation (alpha-rename, ambiguous, cross-domain, misleading names) sharply degrades intent summarization on real-world code (regresses to line-by-line narration) but NOT on algorithmic/competitive code with sparse naming. Surprisingly, execution-prediction tasks also drop, revealing identifier-as-retrieval-cue memorization. Releases ClassEval-Obf.
• DR2-004 — Renaming alone mild; renaming + structural change compounds benchmark · med · refines:H5 — Metamorphic APR study: renaming transformations alone rarely impair repair, but amplify degradation when paired with structural rewrites (e.g., NestElseIf). Gemma-2-27B uniquely sensitive to renaming in isolation. Patterns indicate reliance on familiar syntactic forms over abstract semantics; used to diagnose Defects4J memorization.
• DR2-005 — Comments perturbation least impactful on logic; conditionals most benchmark · med · refines:H5 — Metamorphic testing of LLM APR (MT-LAPR): of nine perturbation rules, AddingComments had the LEAST impact on performance (compilation ignores comments and models barely act on them), while Conditional-Expression changes had the greatest impact (core control flow). VariableRenaming had the largest edit distance.
• DR2-021 — Misleading identifiers survive even deobfuscation; poison persists case-study · med · refines:H4 — Wang et al.: shuffled/misleading variable names degrade analysis MORE than random strings because they actively misdirect. 'Poisoned Identifiers' case study (Claude Opus): wrong-but-plausible names are actively preserved in reconstructed code, not merely degrade understanding. Cross-domain term substitution severely degrades intent-level tasks.
• R3-004 — Names are token programs: closed-vocabulary composition, one name one referent, no shadowing theory · high · refines:R-020 — Identifiers are BPE sequences; derivable patterns compress and bind attention across files; synonyms and shadowing create ambiguous bindings models resolve worse than compilers do.

Axis C — Meta-layer (9)

• 09DR1-004 — MISRA deviation regime: mandatory/required/advisory + recorded rationale production · high · refines:deviation-records — MISRA C:2012/2025 separates guidelines into mandatory (no deviation), required (formal deviation with recorded rationale, risk assessment, mitigation, scope) and advisory (document but no formal deviation). Compliance is claimed per-project, not per-org, and only within a documented process. Distinguishes Project vs Specific deviations. Secondary axis A.
• DR1-017 — Comments help LLM comprehension but indiscriminate comments add noise benchmark · high · refines:H4 — Multiple studies: comment prevalence improves LLM comprehension of legacy code and major comment inaccuracies significantly degrade it; for code translation, model-added comments changed accuracy from -90% to +435%, helping only when short, descriptive, English, inline, and added selectively to failure cases. Some ablations find removing comments improves summaries. Natural-language carriers are double-edged. Secondary axis G.
• DR1-020 — Literate programming: documentation-primary source lineage theory · med · known:A1 — Knuth's literate programming (1984) interleaves prose and code so author intent is primary and tooling extracts both a runnable program and a readable essay. The historical root of structured, machine-extractable meta-information co-located with code — but its drawback is that extending code is slow.
• DR2-001 [CONTRADICTS] — Misleading NL in code degrades reasoning ~23% (CodeCrash) benchmark · high · contradicts:H5 — CodeCrash (NeurIPS 2025) perturbed 1,279 CRUXEval/LiveCodeBench items with structural changes and misleading NL. Across 17 LLMs, misleading NL contexts cause avg 23.2% degradation in output prediction; even with CoT, 13.8% drop. Aggregated structural perturbations cause >14pp drop; textual perturbations >11pp. QwQ-32B exhibits 'reasoning collapse' generating 12,000+ redundant tokens.
• DR2-002 — Incorrect docs hurt; missing docs don't (Macke & Doyle) benchmark · high · refines:H4 — MITRE study: providing an LLM 'incorrect' documentation greatly hinders code understanding (measured via unit-test generation), while incomplete or missing documentation does not significantly affect understanding. First controlled study on documentation reliability for comprehension.
• DR2-006 * — Docstring presence has negligible effect on functional correctness benchmark · med · new — Controlled real-world class-level study across 7 models: docstrings do NOT significantly improve functional correctness for most models (only Codestral and Deepseek-V3 showed significant but negligible-magnitude benefits; avg 1.25% raw, Cliff's delta <0.15). Adequate statistical power (0.78-0.87) suggests genuinely minimal effect.
• R2C-004 — CodeCrash perturbation taxonomy: a direct catalog of banned anti-patterns benchmark · high · refines:R3-015 — Mechanism map (Lam et al., NeurIPS 2025, 17 LLMs, CRUX+LiveCodeBench, AST-normalized vanilla baseline). Two families: (I) Structural - variable RENAMING, expression REFORMATTING, GARBAGE-code injection (PSC-ALL combines all three); (II) Textual distraction - Misleading Code Comments (MCC: comments that contradict logic), Misleading Print Statements (MPS: same via prints - effect is NOT format-specific), Misleading High-level Hints (MHC: plausible-but-wrong output hints, reasoning-level). Models 'rationalize' false comments instead of filtering them; QwQ-32B shows 'Reasoning Collapse' (12k+ redundant tokens). MCC/MPS parity proves the harm is the false claim, not the comment syntax.
• R3-005 — Convention half-life is set by enforcement, not documentation quality theory · high · refines:A5 — No persistent memory: the Nth session never read the guide. Conventions are checker-enforced or delivered just-in-time; prose-only conventions decay to noise.
• R3-015 — Wrong prose is worse than no prose: machine-check or trust-label all code-adjacent text theory · high · refines:H4 — Models condition on in-repo text with high trust; a lying comment is adversarial input. Prose is doc-tested/drift-checked or labeled verified/unverified/aspirational; harm of wrong prose exceeds harm of absence.

Axis D — Context & repo (10)

• 10DR2-007 — SWE-agent ACI design moves resolve rate 10.7pp over raw shell benchmark · high · known:Star-Griesemer-boundary-objects — SWE-agent (Yang et al., NeurIPS 2024): a purpose-built agent-computer interface (search/navigation, file viewer, guarded editor with syntax-checker, context management) lifts GPT-4 Turbo to 12.47% on full SWE-bench (286/2,294) / 18.00% Lite, a 6.7-fold improvement over the 3.79% prior best non-interactive RAG (Claude 3 Opus). Ablation: SWE-agent 'solves 10.7 percentage points more issues than the baseline agent that uses just the default Linux shell' (64% relative gain, 18.00% vs 11.00% Lite). Edit guardrails (linter) materially help.
• DR2-008 — Aider PageRank repo map: structure-aware context beats flat inclusion case-study · med · refines:Hutchins-distributed-cognition — Aider builds a tree-sitter symbol graph of the whole repo, ranks symbols by PageRank over the reference graph, and sends only the top symbols within a token budget ('--map-tokens switch, which defaults to 1k tokens'). Claimed to give higher edit accuracy than naive file inclusion; runs at '15B Tokens/Week' in production. Map is symbol-level, not semantic — can miss scattered logic.
• DR2-009 — Code graphs improve issue localization and downstream resolve rate benchmark · med · new — Multiple papers: CGM (graph-integrated LLM + agentless graph-RAG) hits 43.00% on SWE-bench Lite with open Qwen2.5-72B; another code-graph method 40.67% Lite. SpIDER (graph-aware retrieval) yields consistent retrieval gains and +2.74pp downstream resolve at K=10 on SWE-bench Verified (12 additional resolved instances) with Claude Sonnet 4.5.
• DR2-010 — Long context degrades retrieval and reasoning for code benchmark · high · known:Liu-Lost-in-the-Middle — Repo-QA and long-context studies: performance degrades as context grows; even when relevant code is present, models often fail to locate it. Reasoning lags retrieval (retrieval sets an upper bound). 'Context rot' (ChromaDB, 18 models incl GPT-4.1/Claude 4/Gemini 2.5): reliability decreases with input length even on simple retrieval; distractors and hard negatives reduce accuracy.
• DR2-015 * [CONTRADICTS] — Agentless beats complex agents at fraction of cost benchmark · high · contradicts:H1 — Agentless (Xia et al., FSE 2025), a simple three-phase pipeline (localize, repair, validate) with no autonomous agent loop, achieves 32.00% (96/300) on SWE-bench Lite at $0.70/instance — highest open-source performance and lowest cost vs prior agents (early v1: 27.33% @ $0.34 vs $3.34 for some agents). Reaches 50.8% on SWE-bench Verified with Claude 3.5 Sonnet. Adopted by OpenAI/DeepSeek as their eval approach.
• DR2-022 — Repo-level context extraction beats long-context dumping for repair benchmark · med · refines:Liu-Lost-in-the-Middle — RepoBugs/RLCE: structure-aware repository context extraction (repo structure parsing + relevant context retrieval) significantly improves repository-level repair vs naive long context. RES-Q shows repo-scale editing differentiates models that saturate HumanEval (Claude 3.5 +12pp over GPT-4o on RES-Q despite 1% HumanEval gap).
• DR2-023 — Context engineering: minimal high-signal tokens, compaction, JIT retrieval manifesto · med · known:Liu-Lost-in-the-Middle — Anthropic engineering guidance: LLMs have an 'attention budget'; 'context rot' degrades recall as tokens grow. Recommends smallest set of high-signal tokens, compaction (summarize+reinit), just-in-time retrieval via lightweight identifiers, and sub-agents that return 1-2k token distilled summaries from tens-of-thousands-token explorations.
• R3-001 — Closure under paging: editable units declare their full semantic dependency set theory · high · refines:R-002/A1 — Finite context means agents see pages, not repos. Units must be mechanically assemblable into sufficient context: explicit imports, contracts, invariants, flags. Ambient coupling (globals, inheritance magic) breaks closure.
• R3-003 — Position is a resource: critical invariants at file edges, file length bounded theory · med · new — Positional attention effects degrade mid-file content. Safety-critical facts go top/bottom; prefer more small single-purpose files at equal token mass.
• R3-012 — Written for retrieval: stable anchors + uniform one-line what/why per item theory · high · refines:A1/R-030 — Context assembly is retrieval. Authored uniform one-liners in a fixed grammar plus stable URIs make the retrieval index authored, not inferred; explicit cross-references over conventions.

Axis E — Verification (8)

• 11DR1-002 — MoonBit semantic sampler: local+global constrained decoding case-study · med · refines:H2 — MoonBit pairs the language with two custom samplers: local sampling enforces syntax, global sampling checks type correctness during generation, with backtracking and a speculation buffer that informs the model of valid continuations. Reported significant compilation-rate improvement at ~3% performance penalty. A working compiler-as-oracle instance.
• DR1-006 — SPARK/Ada: graded assurance levels + SPARK_Mode escape hatch production · high · refines:H2 — SPARK is the largest Ada subset to which automatic formal verification is practical, with five graded levels (Stone/Bronze/Silver/Gold/Platinum) so projects pay only for the assurance they need, and an explicit SPARK_Mode => Off escape for code that can't be verified, with a defined way to mix proof and test evidence. Contracts are executable. Secondary axis F.
• DR2-012 — Type-constrained decoding cuts compile errors ~75%, lifts correctness benchmark · high · refines:H2 — Mündler & He et al. (PLDI 2025): type-constrained decoding (prefix automata + inhabitable-type search) reduces compilation errors by 75.3% (synthesis) and 70.2% (translation), increases functional correctness relatively 3.5-5.5%, and repairs non-compiling code relatively ~37% on average, across model families incl 30B+ open-weight. Only 3.5% of functional errors and 6% of compile errors are syntactic; 'on average 94% of compilation errors result from failing type checks' — types catch the rest.
• R2C-005 — Type-constrained decoding is PER-LANGUAGE manual work; no Rust impl exists benchmark · high · refines:H2 — Decisive scope boundary (Mündler et al. PLDI 2025, eth-sri repo). Method formalized on a simply-typed lambda calculus, instantiated only on 'a non-trivial subset of TypeScript'. Authors' repo FAQ states plainly: the constraining algorithm 'has to be implemented manually for every language' and they know of NO implementations for other languages as of June 2025. ~94% of TS compile errors are type-level (only ~6% syntactic), which is why type-awareness matters - but the prefix-automaton + type-reachability machinery is bespoke per language.
• R3-007 — Verification locality beats model capability: per-cell sub-minute checks (the swarm bet) theory · med · new — Tool loops substitute for understanding; value set by time-and-scope-to-first-signal. Per-cell fast checks beat global CI; beyond a modest threshold, locality matters more than parameters.
• R3-008 — Misuse-resistant API shape converts probable hallucinations into compile errors theory · high · refines:H2 — Make the statistically likely wrong call unrepresentable: newtypes, typestate, exhaustive matches, no boolean/positional soups, no stringly interfaces. Errors surface in the loop, not production.
• R3-009 — Invariants restated at use sites: redundancy is ground truth for paged readers theory · med · new — Small models cannot do whole-program inference. Restate the load-bearing invariant where used (debug_assert!, contract-citing comment), not only where defined.
• R3-011 — Tool output is agent food: structured, requirement-citing diagnostics theory · high · refines:R-010 — Error text is the highest-leverage prompt in the loop. Diagnostics should be stable, structured, cite the violated requirement and the fix surface; free text is wasted conditioning.

Axis F — Spec-binding (4)

• 12DR1-003 — Unison: content-addressed definitions, names as metadata production · high · refines:A2 — Each Unison definition is identified by a hash of its (alpha-normalized) syntax tree; names are just metadata pointers. Renames never break dependents, no builds, perfect incremental compile, no dependency-version conflicts, test caching. Structurally equivalent functions share a hash; reordering operands changes it. Secondary axis C.
• DR1-018 — Intentional programming: intent as the invariant, implementation evolves manifesto · med · known:A1 — Simonyi's intentional programming (1995-2017, Intentional Software, acquired by Microsoft) separates computational intent from implementation: self-describing abstractions carry how they look and what they do, stored as an intentional tree, with implementation produced by re-runnable transformations. The lineage behind 'every artifact carries an explanation chain'. Secondary axis C.
• DR1-019 — Design by Contract: machine-checkable intent that survives refactoring production · high · refines:F — DbC (Meyer/Eiffel; realized in Ada/SPARK, GContracts) externalizes pre/postconditions and invariants as annotations separate from control flow, keeping intent attached to the unit and checkable at runtime or by proof. Contracts are a bug-tracking concept, not input validation. The canonical pattern for first-class, refactor-surviving intent carriers. Secondary axis E.
• DR1-024 — 'Intent' DSL: AI-writes / human-reads, contracts + Z3 verification manifesto · low · new — A practitioner project ('Intent') designs a language for AI assistants to write that compiles to native/JS/WASM for humans, prioritizing explicit contracts, declared intent, invariants, and Z3-verified pre/postconditions over brevity. A donor showing the 'contracts-first, intent-linked-to-verification' shape — but it is a compiler/transpiler, so principle-only.

Axis G — Comprehension empirics (8)

• 13DR1-010 — 'When Names Disappear': models narrate, not reason, after obfuscation benchmark · high · refines:H6 — Name-only obfuscation collapses intent-level summaries into 'surface-level, line-by-line narration despite identical structure and behavior'; on LiveCodeBench Llama-4 Maverick falls from 80.2% to 56.4%, with double-digit drops across nearly all models including GPT-4o and Qwen3-Coder-480B. Argues benchmarks conflate memorization with structural reasoning. Bears on H5/H6.
• DR1-022 — Same syntactic paradigm in prompt and test boosts extraction up to 125% benchmark · med · refines:H6 — LLM-aided code profiling finds that keeping prompt and test examples in the same syntactic paradigm improves rule-extraction accuracy by 2.3% to 125% (2.25x) versus cross-paradigm, largest for functional/expression-oriented syntax. Quantitative support that intra-corpus syntactic uniformity is an in-context signal. Bears on H6.
• DR2-011 — Longer/more-complex functions are harder to repair benchmark · high · new — Empirical APR study across 6 datasets/4 LLMs: successfully repaired bugs are significantly shorter than failed ones; effectiveness drops sharply for functions >100 lines. RepoDebug: performance worsens monotonically as code length grows (500 to 10,000 tokens). A new LM-CC complexity metric shows strong negative correlation (up to 0.92 partial) with LLM task scores controlling for length.
• DR2-013 — SWE-bench scores partly reflect memorization, not reasoning benchmark · high · new — 'SWE-Bench Illusion' (Zhang et al.): SOTA models identify buggy file paths from issue text alone (no repo) up to 76% on SWE-bench Verified vs up to 53% on external repos (e.g. o3-mini); verbatim 5-gram overlap up to 35% on SWE-Bench Verified/Full vs up to 18% on other benchmarks. Aleithan et al.: 32.67% of Lite 'successes' have solution leakage; ~94% of issues predate training cutoffs. OpenAI audit: 59.4% of o3 failures were test flaws.
• DR2-018 — Failure taxonomies: requirement misinterpretation & incomplete fixes dominate benchmark · high · new — SWE-Compass (600 failed trajectories x3 models) six-category taxonomy: Requirement Misinterpretation, Inadequate Testing, Incomplete Solution & Side Effects, Technical Knowledge Gap, etc. SWE-Bench Pro: large models fail on multi-file semantic/algorithmic correctness; small models (Qwen 3 32B) fail more on syntax, tool use, context management. SWE-EVO: open models fail mostly on Incorrect Implementation (weak semantics, stable tools).
• DR2-019 — Code reasoning (execution prediction) is weak even on tiny functions benchmark · high · new — CRUXEval (800 functions, 5-20 LOC): GPT-4 only 67% (input-pred) / 63% (output-pred); best open models 47%/44%. Models fail on 'surprisingly simple' programs. Replacing variable names with placeholders changes performance — evidence of name reliance. REval/CodeMind extend to state/coverage prediction with similar gaps.
• DR2-020 — Bug localization is the bottleneck; symbol-level accuracy matters benchmark · med · new — 'How Far Are We' bug-fixing agent study: fault localization at code-symbol level is more critical than file/line level; incorrect bug reproduction cascades to whole-task failure. AutoFL: function-call navigation improves method-level acc@1 up to 233% over baselines. Imperfect FL causes significant accuracy drops vs perfect-FL assumption.
• R2C-006 — Rust benchmark contradiction resolved: difficulty mix, not Rust-unfriendliness benchmark · high · refines:H2 — The 58% (SWE-bench Multilingual) vs 10-17% (Multi-SWE-bench) Rust gap is fully explained. Multi-SWE-bench is harder by construction: 77.1% medium+hard vs 61.2% in Verified; methods are Python-optimized (bias); and Rust/C++ PRs are large by nature (ripgrep/simdjson patches >200 lines, 7 files). SWE-bench Multilingual (300 curated tasks) has Rust HIGHEST of 9 langs (58%) even though Rust gold patches modify MORE lines than other langs - i.e. within a fixed difficulty tier, Rust's compiler GUIDANCE outweighs its larger-edit burden. Net: Rust is compiler-advantaged per-task; it only looks bad when the Rust slice is harder.

Axis H — Weak-reader & swarm (30)

• 14BLD-001 — Fast deterministic feedback loop is why coding agents work at all theory · high · — Code uniquely offers cheap, fast, machine-readable correctness signals (compile/type/lint/test) that the agent reads as structured output and self-corrects against in seconds, without human interpretation. This tight loop is the core mechanism separating coding from domains where agents failed.
• BLD-003 * — Rust topped SWE-bench Multilingual despite larger patches benchmark · med · — On SWE-bench Multilingual (300 tasks, 9 languages, Claude 3.7 + SWE-agent), Rust had the HIGHEST resolution rate (25/43 = 58.14%) and C/C++ the lowest (28.57%), even though Rust solutions modified more lines on average. Overall multilingual rate 42.67% vs ~63% on Python-only SWE-bench Verified.
• BLD-004 — Multi-SWE-bench: Python dominant, Rust far lower — benchmarks disagree benchmark · med · — ByteDance's Multi-SWE-bench (1,632 validated instances, 7 languages, multiple models) finds models resolve Python well but generalize poorly elsewhere; Rust 'Easy' resolved rates were ~10-17% vs Python ~64-68% for Claude-3.7/o1, with Rust/C++ patches often >200 lines and 7 files.
• BLD-007 * — AGENTS.md/CLAUDE.md: human-written +4%, auto-generated -0.5/-2%, +20% cost benchmark · high · — Gloaguen, Mündler, Müller, Raychev & Vechev (arXiv:2602.11988, Feb 13 2026) built AGENTbench (138 real tasks, 12 Python repos): human-written context files improved success ~4%, LLM-generated ones reduced success (-0.5% SWE-bench Lite, -2% AgentBench) vs none, and all raised inference cost >20%. Files act as a gap-filler for missing docs, not a booster atop good docs; directory overviews don't help navigation.
• BLD-008 — Don't send an LLM to do a linter's job — enforce style deterministically anecdote · med · — Putting style rules in CLAUDE.md wastes context and degrades instruction-following; Claude Code's system prompt alone holds ~50 instructions and instruction-following decays uniformly as count rises. Better: run formatter/linter via hooks and feed errors back deterministically.
• BLD-009 — Custom lint beats written code conventions for agents (deterministic > prose) anecdote · med · — Japanese practitioner writing (Loglass/Zenn) argues code conventions are non-deterministic for agents because agent behavior is non-deterministic and rule-reading varies by model; project-specific rules should be encoded as custom lint rules (which agents can even author) and enforced via commit hooks, read only when triggered.
• BLD-010 — Strict lint helps agents (backpressure) but can also cause thrash loops anecdote · med · — Practitioners report two effects: (1) 'anything too annoying for humans is now free for agents' — turn on strict lint/types as backpressure; but (2) agents declare tasks done with project-wide lint failing, or spin in fix loops re-introducing prior errors when feedback is unbounded. Bounded retries + project-wide enforcement resolve it.
• BLD-011 — Kani can prove small AI-generated Rust edits correct automatically case-study · med · — Kani (AWS, model-checking for Rust) verifies memory safety, overflow, and panic-freedom for all inputs via #[kani::proof] harnesses. Carl Kadie demonstrated ChatGPT-generated Rust verified by Kani — even merging verified AI code into the RangeSetBlaze crate; agentic verification (VeruSAGE, Verus) extends this.
• BLD-012 — Comments help LLM comprehension — but wrong comments measurably hurt benchmark · med · — Evidence is split. One 2025 study found extensive comments raised LLM comprehension of legacy code; another found major comment inaccuracies degraded comprehension ~12% even when the model knew the code. RepoQA ablations and others found REMOVING comments improved performance for most models. Net: accurate comments help, stale/wrong ones are actively dangerous.
• BLD-014 — AI-generated/modified code carries a substantial vulnerability base rate benchmark · high · — Pearce et al. ('Asleep at the Keyboard?', IEEE S&P 2022) produced 1,689 Copilot programs across 89 scenarios and found ~40% vulnerable (39.33% of top suggestions). A 2024 study found ~30% of Copilot snippets had weaknesses across 43 CWEs; benchmarks report 12-65% of generated snippets non-compliant with secure coding; developers using AI assistants wrote less secure code while feeling more confident.
• BLD-018 — Cody context engine: RAG + code-graph + BM25 reranking over enterprise repos production · med · — Sourcegraph Cody retrieves context via embeddings + adapted BM25 + precise code-intelligence (definitions/references) reranking, across up to 10 repos with up to ~1M-token windows, plus a layered cache to preserve prompt-cache hits. A RecSys'24 paper documents the retrieval/eval lessons.
• BLD-019 — Monorepo enables atomic cross-cutting agent edits; polyrepo fragments context case-study · med · — With a monorepo an agent can change schema, backend, and frontend in one atomic commit and learn consistent cross-stack patterns; polyrepos cause 'contextual fragmentation'. Nx reports a Claude Code cross-project change ~4x faster in an Nx monorepo; Airbnb migrated ~3.5K React test files from Enzyme to RTL in 6 weeks (vs an estimated 1.5 years by hand) at a 97% automated rate with six engineers.
• BLD-020 — Cross-file dependency is the hard part: CrossCodeEval/RepoBench benchmark · med · — CrossCodeEval (Python/Java/TS/C#, permissively licensed repos) shows models are far worse without cross-file context and never reach a ceiling even with it; RepoBench formalizes retrieve+complete. Static-analysis-pinpointed cross-file uses are where agents fail most.
• BLD-021 — llms.txt is a convention with grassroots adoption but no proven efficacy manifesto · med · — Jeremy Howard's llms.txt (Sept 2024) proposes a root markdown file linking LLM-friendly content, for inference-time consumption. Adopted by Anthropic docs and some libraries, but it is NOT a formal standard; neutral sources find no canonical evidence it improves answers/visibility, and Google reps liken it to the dead keywords meta tag.
• BLD-022 — Compiler/LSP feedback can be turned into RL/process reward for agents benchmark · med · — Beyond pass/fail, fine-grained signals (diagnostics, rename-safety checks, apply-conflicts, type errors) can be used as transition-level rewards (Lanser-CLI/RL from compiler feedback) and as validation-feedback loops (typia in TypeScript getting an 8B model to reliably compose 289-function tool calls).
• BLD-023 * — Training on Rust data gave the biggest cross-lingual reasoning gain benchmark · low · — SWE-Mirror found a 7B model trained on non-Python data still resolved Python tasks; the Rust-trained variant produced the largest gain, attributed to Rust's complexity and rich type system fostering more robust reasoning. Suggestive that Rust's constraints shape better problem-solving traces.
• DR1-011 * [CONTRADICTS] — Inverse scaling: larger models fail Python identifier swaps harder benchmark · med · contradicts:H3 — Miceli Barone et al. ('The Larger they are, the Harder they Fail', Findings of ACL 2023) show that when default Python function names are swapped, LLMs generate wrong code and some 'become more confident in their incorrect predictions as model size increases' — inverse scaling. Larger OPT/GPT-3 models more strongly prefer the incorrect output; classification accuracy can hit zero.
• DR1-015 — Constrained decoding helps weak models most; can hurt strong ones benchmark · med · refines:H3 — Across structured-output benchmarks, guided decoding acts as a safety net for small models: Qwen2.5-Coder-7B went from 0% to 75% accuracy under constraints, while Hermes-4-405B dropped 92.5%->35.0% (over-constraint). Lazy-k: 'added value of constrained decoding increases as the model gets smaller.' The core mechanism for lowering minimum maintainer capability.
• DR1-021 — SLM-default / LLM-fallback agentic architecture manifesto · med · refines:H3 — Survey argues small models (1-12B, up to ~20B) are sufficient and often superior for agentic workloads where the objective is schema/API-constrained accuracy, not open-ended generation, using guided decoding (XGrammar, Outlines), strict schemas, and validator-first execution, with uncertainty-aware routing to larger models. Metrics: cost-per-successful-task, schema-validity rate. The deployment model for the weak-reader swarm.
• DR1-023 — Identifier renaming differentially benefits LARGER models in translation benchmark · med · refines:H3 — A long-code-translation study finds identifier replacement helps larger models (GPT) more than smaller ones (Mixtral, CodeLlama), interpreted as weaker abstraction and greater reliance on explicit semantic cues in smaller models. Nuance for the weak-reader question: small models lean harder on surface lexical cues, so degrade more when those cues are degraded.
• DR2-016 — Small models gain 30-140% on code tasks with agentic scaffolding benchmark · med · refines:H3 — 'David vs Goliath' (CVDP Verilog): agentic decomposition + I/O validation + iterative refinement gives SLMs (1.7B-7B) 30-140% relative improvement; DeepSeek-R1 7B (21.25%→51.25%) and Granite-4 3B (20.51%→48.75%) more than double single-shot rates and even surpass GPT-o4-mini (44.74%) on a linting category. Gains are category-specific; some categories show degradation.
• DR2-017 — SWE-Gym: training + verifier lifts 32B agent 19.7%→26.3% benchmark · high · refines:H3 — SWE-Gym training yields up to 19% absolute resolve-rate gains; 7B improves 7.0%→10.0% over training iterations, 32B 19.0%→19.7%. Inference-time verifier scaling lifts 7B 10%→13.3% and 32B 19.7%→26.3% on SWE-bench Lite. Scaffold choice alone (Moatless vs OpenHands) swings zero-shot from 1-3% to 7-19%. R2E-Gym: 32B + hybrid verifier 34.4%→51%.
• R2C-001 — AGENTbench full-text: harm is unbounded exploration, not bad instructions benchmark · high · refines:R3-014 — Mechanism behind the AGENTS.md result (ETH, n=138, 12 niche Python repos, dev-committed context files; CC-BY-4.0). Agents DO follow context-file instructions (uv-usage rose to ~1.6 calls/instance vs ~0 without). The cost is behavioral: context files trigger more grep (+22%), file reads (+18%), ls/find (+12%), more testing and more thinking, inflating cost >20%. Human files +4%, LLM-generated -3%. Authors' own conclusion: 'unnecessary requirements make tasks harder; human-written context files should describe only minimal requirements.'
• R2C-003 * — Agentic metaprogramming: strong tool-equipped agents partly recover OOD via interpreter loops benchmark · med · refines:R3-007 — Follow-up to EsoLang (arXiv:2606.10933, ~2 days old) builds an agentic pipeline over the same esolangs and runs a capability ladder: Claude Opus/Sonnet/Haiku 4.x, GPT-5.4 xhigh/mini, Kimi K2.5, common tool-use protocol. Finding: frontier agents use METAPROGRAMMING (writing helper code / generators in a familiar language to emit the unfamiliar target) plus iterative interpreter feedback to adapt within a session - partially recovering OOD performance that single-shot generation cannot. Directly probes our RQ4 harness fork: tools move the OOD ceiling, but capability still stratifies the ladder.
• R2C-007 * [CONTRADICTS] — DATING CORRECTION: EsoLang collapse is a 3-month-old static-prompt artifact, recovered by Jun-2026 agents benchmark · high · contradicts:H5 — Decisive cross-paper dating result. EsoLang-Bench (arXiv 2603, 10 Mar 2026) tested GPT-5.2/Gemini-3-Pro/Qwen3-235B/Kimi-K2 under STATIC prompting: best 3.8% mean, 0-11% range, ICL useless. The follow-up (arXiv 2606, 9 Jun 2026) ran the SAME 80-problem tasks as AGENTS with file editing + local interpreter + current models: Opus 4.6 = 86.9% mean, GPT-5.4 xhigh = 99.7% mean. Same tasks, +3 months of model generation + a tool loop = ~4% -> ~90-100%. The pessimistic OOD reading is an artifact of (a) one-shot prompting and (b) a superseded model generation.
• R2C-008 * — The real mechanism: executable scaffolds transfer capability; written strategy does not benchmark · high · refines:R3-010 — The follow-up's causal core, directly relevant to the spec-vs-code debate. THREE conditions tested on weak agents (Sonnet 4.6, GPT-5.4 mini, Haiku 4.5): (Base) no help; (+Text) a written distillation of the strong agent's strategy; (+Lib) a small EXECUTABLE host-language helper library (cell allocator, BCD helpers, Befunge simulator) - no solved problems, no answers. Result: +Text barely moves anything (Sonnet 12->12 on Brainfuck); +Lib is transformative (Sonnet 12->64, GPT-5.4-mini 5->53). Mechanism stated by authors: mid-tier agents 'do not lack the high-level idea; they struggle to build the reusable code to carry it out.'
• R2C-009 — AGENTbench reframed (owner correction): the lesson is 'no irrelevant requirements', not 'no specs' benchmark · high · refines:R2C-001 — Re-reading AGENTbench under a less pessimistic lens. The authors' OWN conclusion is 'unnecessary requirements from context files make tasks harder, and human-written context files should describe only minimal requirements' - and human files DID help (+4%). The mechanism is that BLOATED overviews trigger unbounded exploration. This is NOT 'spec-driven development is useless'; it is 'specs must carry only what's not better expressed in code, and must not bloat.' The authors also flag the niche-language caveat: dynamics 'might shift dramatically' where the model lacks parametric knowledge (i.e. exactly our spec-anchored, less-public-corpus setting).
• R3-010 — Explanation capital: authoring-time rationale outperforms maintenance-time model strength per dollar theory · med · refines:A2/A1 — Strong author runs once, weak readers run thousands of times. Materialize every nontrivial decision as queryable content-addressed rationale; unrecorded cleverness is theft from the reader pool.
• R3-013 — Swarm contention: ownership aligns with file boundaries; deterministic toolchain kills phantom diffs theory · med · refines:R-002/R-030 — Parallel agents meet only at merge. One cell one file-set one registration point; shared facts in append-only ledgers; deterministic generators keep diffs semantic.
• R3-014 — Active-rule-set minimality: rule-paging beats rule-dumping at equal rule mass theory · med · refines:lazy-push — Weak models degrade with simultaneously active instructions. Rulebook may be large; delivered per-edit rule-set must be small, prioritized, activation-matched.