Informed-vs-Naive Surprise: Construct Testable Information Differences to Challenge Wrong Consensus

Pattern: cross-domain structural transfer (transferring SP from mechanism design/crowdsourcing to LLM sampling). A sentence reviewers might remember is: "SP fails on LLMs because the samples lack an information gap, rather than because SP is wrong; that gap can be created and tested." If accompanied by reproducible diagnostics—measured r values across domains, where monotonicity holds and fails, and how it changes with model scale—this would be an analytical conclusion with a clear perspective, rather than merely another voting rule. Conversely, if the result is only that "tool chains are more accurate," that memorable contribution disappears. It therefore requires a clear win over tool-only and weighted voting, or a clear negative result.

Surprisingly-Popular (SP) aggregation requires respondents to have different private information, a condition not met by i.i.d. reasoning chains from the same model. Consequently, agreement-based selection without labels can fail when the majority answer is "confident but wrong." This work proposes artificially creating information asymmetry: for each problem, sample n0 long text-only chains (class N) and n1 chains that can call a Python sandbox (class I, for enumeration, substitution checks, small-scale verification, and test execution). Instead of verbalized meta-predictions, we use the measured class-N answer distribution q0 as the "expectation" and calculate, for each candidate answer, the 20% lower confidence bound (LCB) of log q1 − log q0 under a Dirichlet posterior. The combined majority vote is overridden only when the candidate has LCB>0 and c1≥2. Hyperparameters (alpha, quantile, and thresholds) are fixed once on MATH-train. We also provide a monotonic-informativeness condition, r(a*)>r(a_wrong), that can be checked beforehand on a labeled development set. Planned equal-token comparisons on MATH-500 hard, AIME, the numerical subset of OlympiadBench, and LiveCodeBench include combined tool-vote, tool-only, weighted voting, CoT/PoT selection, verbalized SP/ISP, DeepConf, and cross-model jury baselines. Expected gains are limited (+2–4 percentage points); the principal value lies in characterizing when the condition holds. The project will terminate if Phase 1 (10 days) fails its go/no-go criteria.

Majority voting can entrench incorrect answers on hard problems: when samples share the same misconception, the incorrect answer becomes a common attractor, and i.i.d. samples from the same model cannot provide a reference population with different errors. Cross-model juries can decorrelate errors, but require a second model family and are costly; "LLMs and the Madness of Crowds" also notes that errors across different LLMs are themselves correlated. Directly transferring SP/BTS to LLMs through verbalized meta-predictions has already been reported to perform worse than majority voting ("Bayesian Truth Serum for LLMs?": shared pretraining violates conditional independence). This work tests whether SP's identification logic can be restored by actually creating a population with different information, rather than asking a model to "predict the crowd." This addresses the brief's bottleneck of "confident but wrong consensus" and could provide a signal for selection without labels or a PRM, and for adaptive budget allocation.

The default expert approach is to ask the model to verbally predict the crowd. This is not self-consistent for exchangeable samples, because the meta-prediction is simply the model's view of its own distribution. Another default expectation is that "tool chains are more accurate, so they should be trusted more." The non-obvious move is both to abandon verbalized meta-predictions and to avoid treating tool chains as stronger voters, instead using the contrast between the two populations as evidence. We must be candid: this has not yet been demonstrated. The red team noted that systematic tool bugs—enumeration bounds that are too small, off-by-one errors, floating-point errors, and incorrect modeling—can produce answers with c0≈0, thereby increasing r for incorrect answers. This is precisely where the monotonicity assumption is most likely to fail. Experiments must determine whether the contrast adds value beyond simply trusting tools.

1. Sampling two classes of respondents: for each problem, sample n0 long text-only chains (class N) and n1 chains that can call sandboxed Python (class I, from the same model). Cache all chains, token counts (including interpreter output), and normalized answers; use vLLM and prefix caching.
2. Answer normalization and clustering: normalize integers/expressions by symbolic equivalence; use test-output signatures as answers for coding problems; count c0(a) and c1(a) for each candidate a.
3. Posterior and score: q_j(a) ~ Dirichlet(c_j + alpha), alpha=0.5; s(a)=E[log q1(a) − log q0(a)]. Use the 20% lower confidence bound (LCB) to handle rare answers; candidates with c0+c1<3 are ineligible to overturn the vote.
4. Selection rule: default to the majority answer in the combined vote. If a candidate a satisfies LCB s(a)>0 and c1(a)≥2, select the candidate with the largest LCB. Fix all thresholds once on the MATH-train development set and make no subsequent adjustments; no second model family is used at any point.
5. Monotonic-informativeness diagnostics: measure r(a*)/r(a_wrong) on labeled development data; report where the condition holds and fails by category (arithmetic, combinatorics, coordinate geometry, code versus proof-based problems, and problems opaque to tools), and measure the proportion of top-r incorrect answers that originate from tool-specific bugs.
6. Red-team-driven safeguards (new, requiring ablation): require c0(a)≥1 before an answer can overturn the vote, or cross-check candidates by substitution; compare both against the unconstrained version.
7. Correlated-error flagging and routing: flag a problem as having correlated errors when the combined majority answer has s≤0 and the class-I and class-N majority answers disagree. Add further I chains only for these problems; compare against a simple disagreement-based adaptive-SC allocation, and include the additional chains in the equal-token accounting.
8. Phase 2 (only after Phase 1 passes): a prefix-level version compares I-versus-N continuations of the prefix at branching nodes to obtain node values for beam pruning. It must outperform answer-level selection at equal token cost; otherwise, abandon it.

The primary model is Qwen3-32B (a single model, with no second model family); Qwen3-8B is also used for a scale comparison (at least two sizes). Each problem receives 32 N chains + 32 I chains, with sandboxed Python execution. Hyperparameters are fixed on the MATH-train development set, with no tuning on the test sets. The primary metric is accuracy at equal total token cost (including interpreter output and additional I chains introduced by routing), aggregated across datasets and seeds, with bootstrap 95% CIs.

• datasets: MATH-500 hard split (levels 4-5); AIME 2024/2025 (combined with AIME 2022-23 to increase statistical power); OlympiadBench (numerical subset); GPQA-Diamond (secondary; tool signals are weak, so it serves as an expected-failure check); LiveCodeBench (answer = test-output signature; class I runs self-generated tests); ProcessBench (Phase 2 only)
• baselines: equal-token text-only majority voting; equal-token combined tool-integrated majority voting (the key baseline for isolating the contrast effect); tool-only majority voting; combined weighted voting with one tuned weight for tool chains; CoT-versus-PAL/PoT selection and mixed self-consistency (Zhao et al. 2023, XoT; require reproduction and prior bibliographic verification); Ai et al.'s SP/ISP aggregation using verbalized meta-predictions (reimplemented); original SP with verbalized meta-predictions (ablation baseline); DeepConf / self-certainty weighted voting; a cross-model jury (a second model family at approximately the 32B scale, at equal total token cost); CodeT / MBR-exec / execution clustering (LiveCodeBench only); Qwen2.5-Math-PRM-7B and Math-Shepherd best-of-N / beam (Phase 2)
• metrics: overall accuracy at equal total token cost (primary metric); the correct answer's percentile rank among minority answers according to LCB s on plurality-wrong problems, versus random ranking (go/no-go statistic); the proportion of plurality-wrong problems flipped to correct versus the proportion of plurality-right problems flipped to incorrect; AUROC of correlated-error flagging, with comparison against disagreement-based adaptive-SC; r(a*)/r(a_wrong) for each problem category (validity of the identification condition); the proportion of top-r incorrect answers originating from tool-specific bugs; ProcessBench step AUROC/F1 (Phase 2)
• ablations: verbalized meta-prediction p in place of measured q0; combined voting versus contrast (to isolate the non-additive interaction); class I without tool access (the same prompt, without execution), to demonstrate that heterogeneity rather than prompting drives the effect; Dirichlet/LCB on versus off and minimum-support thresholds; class I with different personas/prompts versus execution signals; with versus without the c0≥1 requirement / substitution cross-check; class I from a second model (jury-style), used only for labeled comparison and not for tuning; model scale (two sizes); answer-level selection versus prefix-level node values
• expected: moderate expectations: at equal token cost, +2–4 percentage points on AIME/OlympiadBench/hard MATH above the better of combined tool-vote and text voting; recovery of 10–20% of plurality-wrong problems among those satisfying the r condition; flip-to-wrong below 2%; flagging AUROC above 0.7. No gain is expected on GPQA, which will be reported as a boundary condition. Expectations must remain candid: with stronger models, tool chains may dominate and the contrast may collapse to tool-only voting, with gains potentially shrinking as scale increases; code gains may primarily arise from established execution clustering.
• falsification criteria: Phase 1 termination criteria on day 10: with a single model and no test-set tuning, on problems where the combined plurality is wrong, the correct answer's median percentile rank among minority answers according to LCB s is no higher than 65% (random = 50%), or the bootstrap CI does not exclude 55%; or the selection rule's equal-token gain over combined tool-vote is less than 1 percentage point or its CI is not above 0. An additional falsification test proposed by the area chair is that if LCB-ratio selection flips no more problems to correct than tool-only majority voting or single-parameter weighted combined voting without increasing flip-to-wrong, the SP framework adds no value and the project terminates. Abandon Phase 2 if prefix-level node values do not outperform answer-level selection.

1. Days 1–3: using Qwen3-32B (and Qwen3-8B for inexpensive checks), sample 32 text chains and 32 tool chains per problem on MATH-500 levels 4-5, AIME 2022-25, and the numerical subset of OlympiadBench. Cache all chains, token counts (including interpreter output), and normalized answers; simultaneously fix alpha, the quantile, and thresholds on MATH-train.
2. Days 4–5: construct a labeled plurality-wrong stratum (for evaluation only); calculate r(a*) and r(a_wrong) to test the identification condition; measure the proportion of top-r incorrect answers attributable to tool bugs; calculate the correct answer's ranking by LCB s versus random ranking. This step provides the least expensive initial go/no-go check.
3. Days 6–8: run the selection rule with fixed hyperparameters and compare it against text voting, combined tool-vote, tool-only, single-parameter weighted voting, and reimplemented SP/ISP at equal token cost; add ablations for c0≥1 and substitution checks.
4. Days 9–10: bootstrap CIs, analyze flip-to-wrong, make the go/no-go decision according to the kill criterion, and write a one-page conclusion (including whether to undertake Phase 2 and whether to remove code from the principal conclusions). In parallel, complete the CoT/PoT and CodeT literature searches.

• Systematic tool bugs (enumeration bounds that are too small, off-by-one errors, floating-point errors, and incorrect modeling) produce incorrect answers with c0≈0, causing the likelihood ratio to favor those errors instead (a potentially fatal issue identified by the red team). → Measure the proportion of tool bugs among top-r errors; require c0≥1 or substitution cross-checks and conduct ablations; if they still cannot be distinguished, report the negative result faithfully.
• The gain reduces to 'tool chains are more accurate,' and cannot be attributed to the SP contrast. → Tool-only, combined tool-vote, single-parameter weighted voting, and CoT/PoT selection baselines are mandatory; use the falsification experiment in the kill criterion as the decision standard.
• Tool and text chains share the same misconception, r≈1, and there is no contrast signal. → Measure r on the development set first and stratify by problem type; claim applicability only for categories where the condition holds.
• Rare-answer noise on AIME is large, and the go/no-go statistic may fail the threshold. → Combine AIME 2022-25 with other datasets and use LCBs and bootstrap; terminate if the threshold is not met, rather than tuning parameters.
• The scope is narrow (only problems verifiable by tools) and may shrink as models improve; data contamination is also a concern (MATH, AIME 2022-24). → Use at least two model sizes; report contamination risks and prioritize the newer AIME 2025; frame the contribution as an analysis of condition diagnostics.
• Related work (CoT/PoT mixtures, CodeT, SP-for-LLM) may already cover this contrast, and others may publish within 3 months. → Complete the literature search before writing; finish Phase 1 promptly; exclude code results from the principal conclusions or include execution-clustering baselines.
• Compute accounting: omitting interpreter output and chains added by routing would make equal-token comparisons unfair. → Account uniformly for every token and compare against disagreement-based adaptive-SC allocation under the same budget.

• Q: The likelihood ratio q1/q0 is largest for errors produced only by tools: tool code has its own systematic failure modes (enumeration bounds that are too small, off-by-one errors, floating-point errors, and incorrect modeling). Text chains almost never produce these answers, so c0≈0, and just a few I chains can give an incorrect answer a very high r. A: This is a real and potentially fatal risk, and we cannot claim to have resolved it. The plan is to measure it directly: on the development set, quantify the proportion of top-r incorrect answers that originate from tool bugs; add c0≥1 and substitution cross-check constraints and ablate them. The LCB and c1≥2 thresholds offer only partial mitigation. If the measured proportion is high and the gains disappear after imposing the constraints, we will report the negative conclusion faithfully.
• Q: On LiveCodeBench, the class-I mechanism is execution clustering based on self-generated tests, belonging to the established CodeT / MBR-exec / execution-voting family. None of these is included as a baseline, so code gains could be wrongly credited to the I-versus-N contrast. A: Agreed. Either add CodeT/MBR-exec/execution-clustering baselines or remove code from the principal conclusions. Our plan is to add the baselines first; if the contrast does not outperform them, code will serve only as an illustrative result and will not enter the central claim.
• Q: If this is A+B, tool-only voting already delivers most of the gains, verbalized SP is not self-consistent, and the added value of the contrast has not been demonstrated. A: It has indeed not been demonstrated. We therefore make tool-only voting and combined voting with one tuned weight mandatory baselines to beat, and use the question of whether 'LCB-ratio selection flips more problems to correct than they do without increasing flips to incorrect' as the falsification experiment. If it loses, the paper retains only the conceptual point that 'SP requires creating an information gap' and the r diagnostics, which may be insufficient for a methods paper.
• Q: There is substantial overlap with CoT-versus-PAL/PoT selection and mixtures (Zhao et al. 2023, XoT, hybrid self-consistency), which are not cited. A: These papers are absent from the neighboring-work list I received, so I cannot verify their details here. They must be retrieved and reproduced as baselines before writing. The novelty claim can only be narrowed to 'using the measured naive distribution for a likelihood ratio and testing identifiability conditions,' and experiments must show that it outperforms those methods.
• Q: Gains of +2–4 percentage points are within seed noise at AIME's scale, and the benchmarks are contaminated. A: We will control this through aggregation across datasets, multiple seeds, and bootstrap. The primary metric is overall accuracy at equal token cost, rather than just the plurality-wrong subset. Contamination cannot be completely excluded; we will disclose it explicitly and prioritize newer data. If the CI still includes 0, we will terminate according to the kill criterion.
• Q: The advantage will disappear as models become stronger: tool chains will dominate, the contrast will collapse to tool-only voting, and the scope is narrow. A: Agreed that this may happen. Results at a minimum of two sizes will directly show the trend. We will state the domain restrictions (arithmetic, combinatorics, and code) and expected failure on GPQA faithfully. The contribution should be positioned as an analysis of 'when the condition holds,' rather than as a general method.

The reasoning tracks of NeurIPS/ICLR/ACL could be suitable, provided the paper is written to develop understanding: its core is that SP requires creating information asymmetry, together with a testable monotonic-informativeness condition and measurements across domains and model scales. If presented only as a selection rule, it will be viewed as A+B with a predictable effect. The area chair's overall assessment was borderline (significance 3, insight 3, generality 2), which I consider reasonable. A low-cost, 10-day Phase 1 is worth undertaking first; invest several months only after passing the kill criterion, especially by beating tool-only and weighted voting.