Optimizer Lock-In: Correct Self-Selected Runtime Feedback with Bounded Exploration

This is an anomaly_reframe: it reinterprets “why feedback learning fails to recover after drift” as censoring/lock-in caused by self-selected labels, and supplies a measurable quantity—the error on unexecuted alternative subplans. Reviewers may remember the “optimizer as logging policy” framework and the quantification of lock-in. Even if the correction mechanism has limited effectiveness, the measurement and analysis have independent value; the AC also considers this the most durable contribution.

Cost-based optimizers improve cardinality estimation through runtime feedback (LEO, POP, FLAIR) or residual correction (TiCard), usually treating feedback as “free supervised labels.” However, the labeled subplans are chosen by the optimizer’s own argmin policy, censoring the label set: underestimated bad subplans are executed and corrected, while overestimated good alternatives are never executed and never corrected. This closed loop can persist after drift. We propose treating the optimizer as a logging policy in off-policy evaluation: select plans randomly by softmax within a near-tie safety set defined by conformal cost bounds, record propensities, train a residual model on top of a frozen base estimator using clipped doubly robust weights, and use ACI/weighted conformal methods to provide plan-cost-level replanning triggers. Planned experiments on drifting STATS-CEB, JOB/CEB, DSB and TPC-DS streams measure the error on unexecuted alternative subplans (the lock-in metric), P-Error, total latency and P99 regressions. The red team has identified a central risk: the safety set may fail to cover “overestimated but actually good” subplans. We therefore regard measuring and characterizing lock-in as the safest contribution, and propensity-based correction as a secondary contribution requiring experimental validation.

Feedback-driven cardinality correction is now common; TiCard and FLAIR make online correction inexpensive. The real bottleneck has shifted from “the learner” to “where labels come from.” Labels come from plans chosen by the optimizer itself and are subject to selection bias, yet the literature has not systematically named and measured this problem for residual/in-context feedback learners. If this bias does cause persistent suboptimality after drift, simply adding more feedback or switching to a stronger learner cannot solve it.

The usual assumptions are “more labels are better” and “the learner is the bottleneck.” Treating the optimizer itself as the logging policy in off-policy evaluation (OPE) has no precedent in the feedback literature, and it would interact non-additively with the learner: the same residual model would perform significantly better on randomized logs. However, this is currently only a hypothesis, and the respective contributions of randomization and weighting have not yet been separated. Selection bias is already generally recognized in the motivation for exploration in Balsa/Bao.

1. Safe-set randomized logging policy: among candidate plans, those whose conformal-upper cost is ≤ (1+eps) × the best conformal-lower cost form the safety set. Sample by softmax over estimated costs, with temperature tau; otherwise choose argmin. Record propensities, enforce a lower bound pmin within the safety set, and use a global budget to limit the fraction of exploratory queries, for example <5%.
2. Subplan propensity: sum the propensities of all plans containing the subplan to obtain the sampling probability of its checkpoint label.
3. Residual learner: freeze the base estimator and train a residual model on log-error, such as GBR/kNN/TabPFN, using clipped doubly robust weights. The outcome model is the residual model itself, and weights come from the recorded pi. For subplans with pi=0, fall back to the base estimate and widen the interval.
4. Uncertainty: apply ACI to each subplan’s log-error and use known pi as the likelihood ratio for weighted conformal prediction. Handle drift through ACI’s online adjustment of alpha.
5. Replanning trigger: switch only when the alternative’s conformal-upper remaining cost < the current plan’s conformal-lower remaining cost + switching cost. Apply a union bound (alpha/k) over the k subplans in the comparison. The guarantee is an approximate guarantee at plan-cost level under covariate shift with bounded propensities; under drift, the formal result is limited to ACI’s long-run coverage.
6. [Required addition in response to the fatal flaw] Correct blind-spot reachability: construct the safety set using optimistic (lower-bound) costs, or use low-cost sampling/cardinality-only/LIMIT-bounded probes for overestimated subplans with pi=0, ensuring that blind-spot subplans receive positive propensity. Report the proportion of nontrivial near-tie sets and blind-spot coverage on every benchmark.
7. Engineering implementation: wrap plan selection through PostgreSQL’s pg_hint_plan and DuckDB optimizer hooks, and record propensity logs.

Implement optimizer hooks in PostgreSQL and DuckDB. First construct a counterfactual simulator: enumerate candidate plans for each query and execute them offline to obtain true subplan cardinalities, allowing logging policies to be replayed. Then run end-to-end online-stream experiments with repeated-template streams under data/workload drift, using NeurBench-style updates, multiple seeds and multiple query streams. Give all baselines the same tuning budget.

• datasets: STATS-CEB, including NeurBench-style data and workload drift; JOB and CEB, with data updates and unseen templates; DSB, with scaled refreshes; TPC-DS, with skewed refreshes.
• baselines: Frozen PostgreSQL/DuckDB statistics; TiCard using the same checkpoint labels, the main residual baseline; FLAIR; LEO-style exact-match feedback caching; POP/Perron reoptimization without cross-query learning; retrained NeuroCard/FactorJoin/MSCN as an upper reference; Bao; epsilon-greedy random plan selection without a safety set or propensity weighting; oracle cardinalities; [Added] Balsa/Bao-style bandit exploration + unweighted TiCard training; [Added] stronger base estimators using pessimistic bounds, such as SafeBound.
• metrics: Error on unexecuted alternative subplans, the lock-in measure; P-Error and Q-error quantiles; total latency and P99 latency; regressions relative to the native optimizer and exploration regret; plan-cost-interval coverage under drift; update time, number of labels required and probe fraction; proportion of nontrivial near-tie sets and coverage of blind-spot subplans.
• ablations: [Critical addition] Randomized exploration + unweighted training vs. randomization + IPW vs. DR, separating off-policy validity from label diversity; logging policy: argmin only, epsilon-greedy, safe-set softmax, with eps and tau sweeps; safety-set size vs. blind-spot-coverage sweeps; conformal-upper safety sets vs. optimistic-lower-bound safety sets vs. low-cost probes; IPW vs. DR vs. no weighting; clipping level; residual-learner type, including GBR, kNN and TabPFN; weighted conformal vs. ACI vs. ordinary split conformal; triggers without a union bound or fallback; drift type and strength; seen vs. unseen templates; exploratory queries needed to achieve a given error, yielding sample-complexity curves.
• expected: Unverified hypotheses: biased-label learners exhibit visible lock-in; a learner using randomization + propensity weighting reduces post-drift P-Error by at least 25% relative to TiCard/FLAIR using checkpoint labels, with exploration adding <5% total-latency overhead; latency is within 15% of that of a retrained estimator, with update cost more than 50 times lower; P99 regressions decrease by 20–40% relative to an uncalibrated trigger. H2: with eps at 5–10% and exploratory queries <5%, DR residual learning closes at least half the P-Error gap between a biased-label learner and the oracle. A substantial share of the gain may come from label diversity rather than weighting; experiments must distinguish these contributions.
• Falsification criteria: Stop or reduce the contribution to an analysis paper if any of the following occurs: (a) TiCard/FLAIR using biased checkpoint labels is within 5% of the full method in post-drift P-Error and latency, indicating no lock-in; (b) weighted randomized logs do not outperform unweighted randomized logs, indicating no propensity benefit; (c) exploration cost exceeds the total latency saved on any two of the four benchmarks. [Additional criterion] (d) blind-spot subplans retain propensity 0 at every feasible eps, and the probe approach cannot resolve this.

1. Days 1–4: construct a JOB counterfactual simulator in DuckDB/PostgreSQL. Enumerate candidate plans for each query and execute each offline once to obtain true subplan cardinalities. Also measure the size of each query’s near-tie set.
2. Days 5–6: first run a diagnostic experiment on data variants with drift: count how many “overestimated but actually good” subplans enter the safety set (propensity>0), directly testing the red team’s fatal concern. If coverage is extremely low, immediately switch to an optimistic-lower-bound safety set or low-cost probes.
3. Days 7–9: on online streams, compare TiCard (GBR) trained on argmin logs, epsilon-greedy logs and safe-set randomized logs. Measure unexecuted-subplan error and P-Error.
4. Days 10–12: add IPW and DR and the critical “randomization + no weighting” ablation. Evaluate kill criteria (a) and (b).
5. Days 13–14: inject corrections into the optimizer, measure latency and exploration regret, assess variance across multiple seeds, and summarize the go/no-go decision. Success signal: after 500 queries, the biased-label learner’s error on unexecuted alternatives is at least 30% higher than that of the random-label learner, weighting recovers at least half the gap, and exploratory queries remain <5%. The plan is described as 21 days; the first two weeks complete the work above. Use 64 CPU cores, with the A100 only for TabPFN/FLAIR.

• Fatal risk: the safety set excludes overestimated good alternatives by construction (pi=0), preventing the mechanism from reaching the blind spots that motivate it. → Construct the safety set with optimistic lower bounds; introduce sampling/cardinality-only/LIMIT-bounded probes; first diagnose coverage. If blind spots cannot receive positive propensity, reposition the work as an analysis paper.
• Near-tie sets are too small, overlap is insufficient, and lock-in is only weakly correctable. → Report the proportion of nontrivial sets on each benchmark, sweep eps, and widen the safety set if necessary while honestly reporting regret.
• DR/weighted estimates have high small-sample variance, while clipping introduces bias. → Sweep clipping levels, use self-normalized weighting and regularized priors in the residual model.
• Gains come from label diversity rather than off-policy weighting. → Require the “randomization + no weighting” ablation and Bao/Balsa-style exploration baselines. If no weighting matches the method, reposition the contribution as lock-in measurement.
• Union-bound intervals are too wide, making the trigger overly conservative. → Report trigger rates and try tighter joint approaches, such as alternatives to Bonferroni or direct calibration at plan level.
• Conformal coverage under drift is only approximate; only long-run ACI coverage is guaranteed. → State the guarantee’s scope accurately and report empirical short-window coverage.
• Randomization increases latency variance; JOB has only 113 queries, making significance difficult; lock-in depends on deliberately constructed repeated-template drift streams. → Use multiple seeds and query streams, bootstrap confidence intervals, CEB/DSB for longer streams, public drift constructions and analyses insensitive to those constructions.
• A stronger base estimator, such as SafeBound, reduces blind spots and near-tie sets, eliminating the advantage. → Include such base estimators and define the method’s scope.
• Commercial optimizers may resist randomized plan selection, limiting adoption. → Emphasize strict exploration budgets and regret bounds, and provide a deployment mode restricted to shadow or test environments.

• Q: The safety-set rule cannot reach the blind spots motivating the paper. Overestimated but actually good alternatives are far from argmin, outside the near-tie safety set, with pi=0, and are never corrected under the method’s own rules. A: This criticism is correct and identifies a fatal flaw in the current design. The safety set covers these subplans only if the overestimation falls within the eps window or conformal intervals are sufficiently wide. Our responses are: (1) construct the safety set with optimistic lower bounds; (2) use low-cost probes for pi=0 subplans; (3) first quantify propensity coverage of blind-spot subplans in the simulator. If coverage remains very low, we will not claim to correct blind spots, and the paper will be reduced to measuring and characterizing lock-in.
• Q: Query-feedback literature already proposes changing plan selection so feedback covers expressions outside the selected plan while limiting overhead. Bandit exploration in learned optimizers is also standard, weakening the novelty. A: We partly agree. The supplied related-work evidence is limited, so we cannot claim coverage of all query-feedback work and need a fuller literature audit. We do not claim exploration itself is new. The claimed novelty is limited to explicit propensity logging for cardinality labels, DR residual training and plan-cost-level conformal triggers. If these also prove to have precedents, the work will shift toward measuring and analyzing lock-in.
• Q: A critical ablation is missing: randomized exploration + unweighted training. Without it, gains cannot be attributed to off-policy validity rather than label diversity. A: Agreed; it is now a required ablation and kill criterion (b). We also add epsilon-greedy and Bao/Balsa-style exploration feeding unweighted TiCard. If weighting gives no additional benefit, we will report that and reposition the contribution.
• Q: This is mainly an A+B combination of exploration, IPW and conformal prediction, and the conformal component overlaps with Conformal Prediction for Verifiable Learned Query Optimization. A: The combination characterization is valid. Our argument is that the combination serves a new diagnosis, lock-in, and IPW is valid only on randomized logs. If experiments cannot demonstrate non-additive interaction, however, that argument fails. The extent of overlap with the conformal paper requires careful full-text reading.
• Q: The evaluation is contrived: synthetic drift and repeated-template streams may favor the method, and there are no real production traces. A: Correct. We will publish the drift constructions, use multiple drift types and strengths and unseen templates, and limit conclusions to those settings. If real traces cannot be obtained, we will state that limitation.
• Q: The systems depth is insufficient; the work reads like an ML paper, and commercial optimizers would not randomize plans. A: End-to-end implementation in PostgreSQL/DuckDB, exploration budgets and regret bounds must address this. Adoption constraints are real; we recommend initially targeting shadow execution or test environments.

VLDB, or SIGMOD, is a fit provided the core is an end-to-end PostgreSQL/DuckDB implementation, real latency metrics and drift experiments across multiple benchmarks. If blind-spot reachability cannot be resolved, a safer direction is a VLDB experimental/analysis paper that quantifies lock-in and treats propensity-aware logging as a secondary design. The current internal AC assessment is borderline, with risks concentrated in differentiation, ablations and blind-spot coverage. A 2–3-week pilot should test these points before committing several months.