[zzylol]

Stacked on #1232 (refactor/1207-pass1, PR 3). Part of the #1207 merge unit: do not merge on its own.

Why

#1207's Order, row "4. Pass 2": the reuse rules over memo groups, each readout's required accuracy, dead-candidate elimination with root causes, chain groups and the logical edges $(u,c,i)$ with $\alpha^{\log}$, in the order decided on 2026-11-07 and the accuracy design of 2026-11-08 (no θ floors: a readout input carries RequiredAccuracy, and $\alpha^{\log}$ = guarantee(summary type, θ_u, statistic) at the required δ ≤ required ε). Concepts doc §3 "Stage 1, Pass 2" and §4.

What

Pass 2 (generate_logical_asap_memo_candidates(pass1, entries, evidence)) now runs these steps:

Step	Code	What it adds or decides
1. Rollup from the finest common group	rules/rollup.rs add_rollup_candidates	Coarse aggregate group $G_c$ from finer $G_f$ of an equal definition (keys ⊆, selection ⊆ with compensation only on $G_f$'s keys by lineage, equal time window): SummaryMerge(group_by) for states, sum/min/max for finalized sums, counts, minima, maxima; one synthesized finest common group $G^*$ per cluster
2. Top-k reuse	rules/topk.rs	A smaller exact top-k reads the largest top-k of the same input
3. Reach, caps, readout requirements	requirements.rs	Caps $\min(b_{c,i}, \max_q \varepsilon_q/\lambda)$ top down through error_propagation; each sketch readout gets one candidate per requirement level (single-read query: the cap; composed query: the ε grid × δ grid not looser than the cap and per-value δ)
4. Dead-candidate elimination	dead_candidates.rs	Removes candidates no query can use, transitively; LogicalASAPMemoCandidates::pruned lists each with every root cause (one shortest path per failing edge)
5. Chain groups	rules/chains.rs	SummaryTumblingWindowChain[w](X) per width (gcd closure of $\gcd(g_q, W, o_j)$ over reaching queries with a known grid), one build per surviving θ, re-wrapped TimeShift/TimeRange groups, SummaryMerge in the state group; @ T reuses existing chains only; @ end()/@ start() none
6. Logical edges	logical_memo_candidates.rs	Every $(u,c,i)$ between surviving candidates with $\alpha^{\log}$ from producer_satisfies_logical_requirements; evaluation_grid_of_group (gcd of the reaching queries' grids)

Deleted (replaced, no shims)

Deleted	Replaced by
Pass 2 rules 2 (scan widening), 3 (summary inputs), 5 (finer aggregate), 6 (rollup), 7 (window rollup): memo/rules/{scan_labels,summary_inputs,finer_aggregate,rollup}.rs, memo/window_candidates.rs, memo.rs	the general rollup rule (step 1) and chain groups (step 5); a longer range reads a shorter one's chain through TimeRange
pass2_local_window_composition, pass2_after_local_window_composition (pub)	generate_logical_asap_memo_candidates (chains run after the reach now, so there is no "rule 1 first" stage); devtools prints one pass2 stage
OriginOfCandidateForMemoGroup::{SummaryInput, WidenedScans}	nothing adds them any more
Pass2RuleError::{InvalidGroupReference, TooManyGroups}	never produced; new Group, Candidate, LogicalProperties variants for rule bugs
asap_accuracy::{composition, propagation} (node-tree guarantee composition and propagation), budget::per_input_target, function_rules, evidence::PropagationStats, test_support and the test-support feature (and its three dev-dep uses)	per-edge LogicalRequirements::error_propagation + readout requirements (the user agreed 2026-11-06 that node-tree bind_guarantees goes in PR 4)
Test-only sizing: estimators::size_params, kll::{kll_k,size_params}, ddsketch::size_params, cms::{cms_width,cms_depth}, cardinality::kmv_k, count_sketch::{count_sketch_width,count_sketch_depth}, hll::hll_precision, univmon::{l2_shape,distinct_heap,distinct_chebyshev,size_params}, topk_capacity, saturating_ceil and the tests built on them	decision 2026-11-08 (PR 1 Q6): a per-entry guarantee test (every_listed_parameters_has_a_guarantee_at_one_percent)

The concepts doc already describes this design; it is not changed. The pending items below are not in it yet (added once approved).

Pending approval (new or changed pub items not in the doc)

P1. error_propagation takes the memo. The doc's signature is (consumer, metric); the key-only Filter contract (E1(b)) must read the input group's column lineage, which only the memo has.

impl LogicalRequirements {
    pub fn error_propagation(
        memo: &Memo,                       // new: the Filter contract reads G_{c,0}'s column_lineage
        consumer: &Operator<MemoGroupId>,
        metric: AccuracyMetric,
    ) -> Option<ErrorPropagation>;
}

Example: over the readout group R of quantile by (job)(latency) (columns [job: SourceColumn(latency.job), value: Computed]), Filter(job = "api", R) gives Some { factor_per_input: [1.0], .. } in every metric; Filter(value > 0.5, R) gives None (it compares the approximate estimate). Alternative: keep the doc signature and have the caller decide filters (two places decide contracts).

P2. producer_satisfies_logical_requirements returns the failing reason.

pub fn producer_satisfies_logical_requirements(
    memo: &Memo,
    producer: CandidateForMemoGroupId,
    consumer: CandidateForMemoGroupId,
    input_index_of_consumer: usize,
    evidence: &dyn AccuracyEvidence,
) -> Result<(), RootPruneReason>;          // was bool

Example: KLL k = 26 into the p99 readout requiring relative 0.02 gives Err(BelowRequiredAccuracy { consumer: G4#2, input: 0, required: relative 0.02 δ 1 }); LogicalEdge::satisfies_logical_requirement is .is_ok(). Pass 2's pruned list needs the reason; a bool plus a second function would decide it twice.

P3. RootPruneReason::HeapSmallerThanTopK. The doc's α^log includes "a top-k's heap holds k", but none of its three root reasons names that failure, which happens also without a requirement (accuracy not enforced).

pub enum RootPruneReason {
    BelowRequiredAccuracy { consumer, input, required },    // unchanged
    MetricRule { consumer, input, producer_metric, consumer_carries }, // unchanged
    /// The consumer is a top-k readout whose k exceeds the producer's heap size.
    HeapSmallerThanTopK { consumer: CandidateForMemoGroupId, input: usize, k: usize }, // new
    NoConsumer,                                             // unchanged
}

Example: topk(50, …) over CmsWithHeap: the 56 builds with heap 10 or 20 are pruned with HeapSmallerThanTopK { k: 50 } (test a_heap_smaller_than_k_is_pruned).

P4. asap_accuracy::readout_metric.

/// The metric of reading `statistic` from a state of `summary_type`: its first listed θ's guarantee's.
pub fn readout_metric(summary_type: &FieldDataType, statistic: &SketchStatistic) -> Option<AccuracyMetric>;

Example: (Sketch(Kll), Quantile{0.5}) → Rank; (Sketch(DDSketch), Quantile{0.5}) → Relative; (ExactAggregate(..), _) → None. Used by step 3 (the metric a readout's levels are in) and by the metric rule (a readout's producer metric).

P5. AccuracyMetric::ALL: [AccuracyMetric; 5] (ABSOLUTE, ABSOLUTE_STREAM_L1, ABSOLUTE_STREAM_L2, Relative, Rank): caps for a target that names no metric, and consumer_carries of MetricRule.

P6. SummaryCoverage::definition(), SummaryCoverage::selection(), SelectionBox::columns() are public (were test-support only): the rollup rule reads them.

Design questions (conservative choice taken; please decide)

Q1. A readout over a merge. The doc checks a readout's requirement against θ_u, but a rollup or chain merge candidate has no θ (LogicalProperties::new rejects θ on a merge). Taken: α^log(merge u → readout r) holds only when every θ the merge may carry (the builds below it, through Filter/TimeRange/TimeShift/SummaryMerge groups) meets r's requirement and the summary type's merged error is Exact or AsIfBuiltOverUnion. Safe, but a chain with weak builds blocks the tight levels. Proposal: a merge (and the window-selecting TimeRange/TimeShift over a chain) carries the θ of the states it reads — one candidate per θ, LogicalProperties { parameters } allowed on SummaryMerge — and the build → merge edge requires equal θ. Example: chain Gw over latency with KLL k ∈ {269, 547}; S's merge reads TimeRange(5m, Gw); readout at rank 0.005 (needs k ≥ 547): today the merge edge fails (269 is below); with the proposal SummaryMerge[k=547] passes and SummaryMerge[k=269] fails.

Q2. Metric rule without targets. Taken: the metric rule applies to an edge only when a query with an accuracy target reaches the consumer; with accuracy not enforced every edge passes (test without_targets_every_edge_is_admissible), matching "(K) has no rows".

Q3. Composed query. Taken: a query is composed when a candidate it reaches has two or more inputs below which a readout group lies (a ratio); readouts of one value by different summary types are alternatives, not a composition.

Q4. Levels at the edges of the grid. Taken: when no grid level is within the cap (exact target, cap 0) the readout gets one level at (cap, δ); a whole-result δ with $n_q$ unknown gives per-value δ 0 (no sketch meets it).

Q5. Finest common group construction. Taken: $G^$ is built over the rows of a member whose selection equals the union; a new Scan of the union of label sets is not built, so {job="a"} with {job="b"} gets no $G^$ (scan widening, R7, not folded in). Members grouped without keys or per entity are not rolled up. Interval constraints (time window, value ranges) must be equal within a cluster.

Q6. Chain builds. Taken: a chain group gets one build per θ of the state group's surviving builds (union over the state groups sharing it).

How

• types (stages/candidates.rs): the constant error contracts in error_propagation; RELATIVE_OPERAND_VALIDITY_CAP = 0.1 private; add_candidate treats input requirements as part of a candidate's identity (the doc: two readouts with different requirements are different candidates).
• accuracy: theoretical_guarantee evaluates KLL (rank error scaled below δ = 0.01) and KMV (bisection on ε) at the required δ, and checks every other algorithm's own failure probability against it; producer_satisfies_logical_requirements applies the readout rule (every carried θ, heap ≥ k) or the metric rule (output_metrics bottom up, cached per group).
• Pass 2: as the table above; rules add only through find_or_add_group/add_candidate, and swallow only OutputDiffersFromGroup.

Key code interfaces added / modified

crates/stage1-pass2-reuse-candidates/src/lib.rs, logical_memo_candidates.rs, rules.rs

/// Stage 1, Pass 2: steps 1–6. `entries[i]` is query i's workload entry (its evaluation grid).
pub fn generate_logical_asap_memo_candidates(
    pass1: Memo,
    entries: &[QueryWorkloadEntry],
    evidence: &dyn AccuracyEvidence,
) -> Result<LogicalASAPMemoCandidates, Pass2RuleError>;           // unchanged signature, now implemented
pub fn pass2_local_window_composition(/* … */);                   // removed
pub fn pass2_after_local_window_composition(/* … */);             // removed

/// Why Pass 2 failed: a rule bug or a cyclic memo. Stays.
pub enum Pass2RuleError {
    /// A group or candidate reads itself through other groups.
    CyclicDependencies,                                            // unchanged
    /// A rule built a group the node rules reject (a bug in the rule).
    Group(SchemaDerivationError),                                  // new
    /// A rule built a candidate the node rules reject (a bug in the rule).
    Candidate(CandidateError),                                     // new
    /// A rule gave a candidate parameters its expression cannot have (a bug).
    LogicalProperties(LogicalPropertiesError),                     // new
    InvalidGroupReference,                                         // removed
    TooManyGroups,                                                 // removed
}

crates/types/src/stages/candidates.rs, logical_memo_candidates.rs, ir/properties/accuracy.rs

impl LogicalRequirements {
    /// g_c's linear bound per metric; None = no contract (metric rule). See P1.
    pub fn error_propagation(memo: &Memo, consumer: &Operator<MemoGroupId>, metric: AccuracyMetric)
        -> Option<ErrorPropagation>;                               // was (consumer, metric), todo!()
}
/// Which rule added a candidate. Stays.
pub enum OriginOfCandidateForMemoGroup {
    Pass1,            // unchanged
    TumblingWindows,  // unchanged: a chain build, a TimeRange/TimeShift over a chain, the merge over them
    Rollup,           // unchanged: also a compensating Filter and a finest common group's builds
    TopKReuse,        // unchanged
    SummaryInput,     // removed
    WidenedScans,     // removed
}
pub enum RootPruneReason { /* P3: HeapSmallerThanTopK new; the other three unchanged */ }
impl AccuracyMetric { pub const ALL: [Self; 5]; }                 // new (P5)

crates/accuracy/src/model.rs, estimators/mod.rs

/// The error bound of θ for `statistic` at `failure_probability`; None if θ cannot fail that rarely.
pub fn theoretical_guarantee(summary_type: &FieldDataType, parameters: &SketchParams,
    statistic: &SketchStatistic, failure_probability: f64) -> Option<AccuracyGuarantee>; // was todo!()
pub fn producer_satisfies_logical_requirements(/* see P2 */) -> Result<(), RootPruneReason>; // was bool, todo!()
pub fn readout_metric(summary_type: &FieldDataType, statistic: &SketchStatistic)
    -> Option<AccuracyMetric>;                                     // new (P4)
pub mod composition; pub mod propagation; pub fn per_input_target(/* … */); // removed

Usage (end to end; the test dead_candidates_are_pruned_and_listed_before_chain_groups at bcae9e5):

// The user's query: quantile_over_time(0.99, m[5m]) / quantile_over_time(0.5, m[5m]),
// relative error 0.05, evaluated every 60 s at phase 0 (FixedIntervalAt).
// Stage 0 + Pass 1 (built by the test as they build it): G2 = TimeRange(5m, Scan m);
// per quantile an Aggregate group with its exact candidate and readouts of a KLL state
// (7 builds) and a DDSketch state (7 builds); G6 = Divide(p99, p50), the query output.
let target = AccuracyTarget::max_error(0.05).accuracy_metric(AccuracyMetric::Relative);
query(&mut memo, ratio, Some(target)); // MemoQuery { root: G6, accuracy_target, values_per_evaluation: 1 }
// Pass 2, as the planner facade calls it after Pass 1:
let logical = generate_logical_asap_memo_candidates(memo, &[entry(60_000)], &NoMeasuredGuarantees)?;
// logical.pruned: each KLL readout -> MetricRule { producer_metric: Rank, consumer_carries: [Relative] };
//   every KLL build through it; DDSketch alpha 0.1 and 0.05 -> BelowRequiredAccuracy.
// Each DDSketch readout group: 5 readout candidates requiring relative 0.02, 0.01, 0.005, 0.002, 0.001 (δ 1).
// One chain group SummaryTumblingWindowChain[DDSketch, w = 60 s](Scan m) with the 5 surviving builds,
// and SummaryMerge(TimeRange(5m, chain)) in the DDSketch state group.
// logical.edges: only between survivors; Stage 2 (PR 5) expands them.

Major algorithm changes

Rollup. Problem: three rules (scan widening, finer aggregate, rollup) each matched their own shapes by column name; no group finer than every member was ever built. Before: rule 2 widened scans, rule 5 read a finer SQL result, rule 6 merged a finer state, each pairwise. After: (1) cluster aggregate groups by coverage definition without keys (a Scan input normalized) and equal intervals; (2) per cluster, $G^*$ = union selection, keys ∪ narrowing columns, built over a member's rows; (3) every ordered pair gets Rollup(by keys)(Filter(compensation, G_f)) when keys ⊆, selection ⊆ on keys by lineage. Correctness: a compensation reads only $G_f$'s keys, whose rows each combine all raw rows of one key. Complexity: O(groups²) per cluster. Tests: a_filtered_coarse_sketch_state_merges_the_finer_states_through_a_compensating_filter, a_finest_common_group_no_query_asks_for_serves_both, a_narrowing_on_a_column_that_is_no_key_gives_no_rollup, a_coarse_exact_sum_rolls_up_the_finer_sum.

Readout requirements and dead candidates. Problem: θ floors per input (Pass 1 heap floor, Pass 2 cap floors) mixed ε and δ and could not express multi-readout allocation. After: caps top down (caps), levels per readout (query_levels), $\alpha^{\log}$ per edge, then elimination to a fixpoint and BFS traces. Correctness: a pruned candidate has no edge to any live consumer, so no plan can use it. Tests: dead_candidates_are_pruned_and_listed_before_chain_groups, a_single_readout_requires_the_query_target, a_whole_result_failure_probability_is_split_per_value, a_ratio_reads_no_rank_error.

Chain groups. Problem: tumbling layouts were a property; widths came only from queries reaching a state through Stage 0 expressions, offsets were ignored. After: range-free chain groups after the reach, widths include offsets (e3_*, e2_*, an_absolute_anchor_reuses_only_existing_chains, no_chain_without_a_merge_an_anchor_or_a_grid).

Before / After

Before (PR 3 head): Pass 2's rules and edges were todo!(); every planning run stopped there. After: Pass 2 produces LogicalASAPMemoCandidates (memo, edges, pruned, evaluation grids); planning stops at Stage 2 (todo! of PR 5).

Example (concepts doc "ratio of quantiles", test dead_candidates_are_pruned_and_listed_before_chain_groups): quantile_over_time(0.99, m[5m]) / quantile_over_time(0.5, m[5m]), relative 0.05, every 60 s at phase 0. Pass 2 prunes both KLL readouts (MetricRule { producer_metric: Rank, consumer_carries: [Relative] }), every KLL build through them, and the DDSketch builds with alpha 0.1 and 0.05 (BelowRequiredAccuracy); each DDSketch readout group has five levels 0.02 … 0.001; one 60 s DDSketch chain with five builds.

Representation, rules, enumeration

• Representation: LogicalRequirements { required_accuracy } on readout inputs; LogicalASAPMemoCandidates { memo, edges, pruned, evaluation_grid_of_group }; chain groups in the memo.
• Rules: steps 1–6 above.
• Enumeration: one readout candidate per requirement level; one chain per width; Stage 3 chooses among them.

Self-review

• No duplicated interface: one add_candidate helper in Pass 2 (swallows only OutputDiffersFromGroup), one readout_metric, contracts only in error_propagation, θ only in parameter_list (no strongest/weakest/floor tables).
• Dead code removed: see the deletion table; no #[allow(dead_code)] added (the three existing ones in hll.rs/univmon.rs now point to PR 8).
• Stale develop docs (docs/develop_docs/asap-aware-mapping-*.md, end-to-end-accuracy-guarantees.md) still describe node-tree propagation; they were already stale before this stack and are left for the PR 9 doc pass. planner-layering.md's Pass 2 rows are updated.

Verification

• cargo fmt --all, cargo clippy --workspace --all-features --all-targets -- -D warnings: clean.
• Pass 2: 21 tests (6 decided examples + 15), types memo tests +4, accuracy +2.

Test status at bcae9e5 (cargo test -p <crate> --all-features, per crate)

Crate	Passed	Failed	Ignored	Failures stop at
asap-types	204	0	0	—
asap-accuracy	24	0	0	—
asap-cost	31	1	0	#1207 PR 6: check θ, which the exported node carries: 1
asap-frontend-common	49	0	0	—
asap-sql-function-catalog	10	0	0	—
metricsql_common	3	0	0	—
asap-frontend-promql	218	0	0	—
asap-frontend-metricsql	15	0	0	—
asap-frontend-sql	175	0	0	—
asap-executor	210	63	1	#1207 PR 6: check θ, which the exported node carries: 32; #1207 PR 6: the Hydra grid's sizes are θ, which the exported node carries: 13; #1207 PR 6: each materialized node's retention_ms is its h_v, which Stage 3 sets: 8; #1207 PR 6: the weighted-frequency θ, which the exported node carries: 7; #1207 PR 6: check the weighted-frequency θ, which the exported node carries: 2; #1207 PR 5: physical expansion and physical edges: 1
asap-planner	8	18	1	#1207 PR 5: physical expansion and physical edges: 18
asap-devtools	37	56	3	#1207 PR 5: physical expansion and physical edges: 43; #1207 PR 9: a group's node from its expression: 9; OTHER: assertion failed: output.status.success(): 2; #1207 PR 9: label a group by its expression: 2
asap-integration-tests	100	50	4	#1207 PR 5: physical expansion and physical edges: 43; #1207 PR 6: each materialized node's retention_ms is its h_v, which Stage 3 sets: 3; #1207 PR 6: realize the sum-and-count plan of avg from Pass 1's memo: 3; OTHER: called Result::unwrap() on an Err value: Error("invalid type: sequence, expected string or map", line: 0, column: 0): 1
asap-stage3-plan-selection	3	0	0	—
asap-stage0-memo	15	0	0	—
asap-stage2-physical-optimization	0	0	0	—
asap-stage1-pass2-reuse-candidates	21	0	0	—
asap-stage1-pass1-per-computation-candidates	32	0	0	—
Total	1155	188	9

PR 3 (61f585a): 1167 / 188 / 9. By test name: no test that passes on PR 3 fails here; every failure stops at a later PR's todo!() (PR 5 "physical expansion and physical edges", PR 6, PR 9) or was already failing. 26 new tests pass (21 Pass 2, 3 types, 2 accuracy); 39 tests are deleted with the code they tested: asap-accuracy's budget::tests (3), composition::tests (16), function_rules::tests (1), propagation::tests (1) and the test-only sizing tests (18: cms 2, count_sketch 3, hll 2, kll 4, estimators::parameter_list_is_the_sizing_over_the_error_grid, univmon 4, sizing 1), replaced by every_listed_parameters_has_a_guarantee_at_one_percent and a_guarantee_is_evaluated_at_the_required_failure_probability.