Skip to content

refactor: one crate per stage (#1207 PR 0b) - #1229

Draft
zzylol wants to merge 4 commits into
refactor/1207-interfacesfrom
refactor/1207-crates-per-stage
Draft

zzylol wants to merge 4 commits into
refactor/1207-interfacesfrom
refactor/1207-crates-per-stage

Conversation

@zzylol

@zzylol zzylol commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Merge unit. This is PR 0b of #1207's merge unit (PRs 0–9, with 0b). It is stacked on PR 0 (#1226, refactor/1207-interfaces); PR 1 (#1227) is stacked on it. The PRs merge into main together, after PR 9. This PR only moves code between crates: no logic changes. It also deletes the tests that crossed stage crates (decided 2026-11-06, see "Deleted tests"; 79 on PR 0's head be6ea1d6a); every other test passes or fails exactly as on PR 0 (see "Verification").

Why

#1054's crate rules say:

  • stage crates never depend on each other; each reads the previous stage's output through asap-types;
  • shared code lives in asap-types, asap-accuracy or asap-cost;
  • each crate exposes only what other crates use;
  • the planner facade calls the stages in order, and the crate-graph test checks the edges.

Before this PR, Stage 0's hash-consing and both passes of Stage 1 were one crate, asap-logical-asap-optimization, so nothing stopped Pass 2 from calling into Pass 1 (only a source-scanning test, pass_layering.rs). #1207 (Order, decided 2026-11-06) makes each stage and pass its own crate before any stage logic is filled in, so PRs 1–8 land in their final crates.

What

Crates

Folder names are the crate names without the asap- prefix, like the existing crates (asap-types in crates/types).

Stage Crate (folder) Content
Stage 0 asap-stage0-memo (crates/stage0-memo) hash-consing into the Memo (memo_of_queries, whose body PR 2 fills in; PR 0 deleted the old cse.rs), from asap-logical-asap-optimization
Stage 1, Pass 1 asap-stage1-pass1-per-computation-candidates (crates/stage1-pass1-per-computation-candidates) initial candidates (pass1_memo), from asap-logical-asap-optimization/src/pass1/
Stage 1, Pass 2 asap-stage1-pass2-reuse-candidates (crates/stage1-pass2-reuse-candidates) reuse rules and logical edges (generate_logical_asap_memo_candidates), from asap-logical-asap-optimization/src/pass2/
Stage 2 asap-stage2-physical-optimization (crates/stage2-physical-optimization) renamed from asap-physical-asap-optimization; content unchanged
Stage 3 asap-stage3-plan-selection (crates/stage3-plan-selection) renamed from asap-plan-selection; content unchanged

asap-logical-asap-optimization is deleted.

File → crate map

Every file of the deleted crate, and every file of the renamed crates:

Before After
logical-asap-optimization/Cargo.toml split into stage0-memo/Cargo.toml, stage1-pass1-per-computation-candidates/Cargo.toml, stage1-pass2-reuse-candidates/Cargo.toml (each lists only its own dependencies; asap_sketchlib, serde and rmp-serde of the old crate were unused)
logical-asap-optimization/src/lib.rs deleted; each new crate root re-exports its own entry points, and its crate docs moved to them (the accuracy paragraph to Pass 1)
logical-asap-optimization/src/stage0.rs stage0-memo/src/lib.rs
logical-asap-optimization/src/stage0/cse.rs deleted in PR 0 (Stage 0 builds groups through Memo::find_or_add_group, PR 2)
logical-asap-optimization/src/pass1.rs stage1-pass1-per-computation-candidates/src/lib.rs
logical-asap-optimization/src/pass1/logical_memo_candidates.rs stage1-pass1-per-computation-candidates/src/logical_memo_candidates.rs
logical-asap-optimization/src/pass1/per_computation_candidates.rs stage1-pass1-per-computation-candidates/src/per_computation_candidates.rs
logical-asap-optimization/src/pass1/per_computation_candidates_tests.rs stage1-pass1-per-computation-candidates/src/per_computation_candidates_tests.rs
logical-asap-optimization/src/pass1/realization.rs stage1-pass1-per-computation-candidates/src/realization.rs (only Pass 1 calls it)
logical-asap-optimization/src/test_support.rs split, see below
logical-asap-optimization/src/pass2.rs stage1-pass2-reuse-candidates/src/lib.rs
logical-asap-optimization/src/pass2/logical_memo_candidates.rs stage1-pass2-reuse-candidates/src/logical_memo_candidates.rs
logical-asap-optimization/src/pass2/memo.rs stage1-pass2-reuse-candidates/src/memo.rs
logical-asap-optimization/src/pass2/memo/rules.rs stage1-pass2-reuse-candidates/src/memo/rules.rs
logical-asap-optimization/src/pass2/memo/rules/{finer_aggregate,rollup,scan_labels,summary_inputs,topk}.rs stage1-pass2-reuse-candidates/src/memo/rules/{finer_aggregate,rollup,scan_labels,summary_inputs,topk}.rs
logical-asap-optimization/src/pass2/memo/window_candidates.rs stage1-pass2-reuse-candidates/src/memo/window_candidates.rs
logical-asap-optimization/src/deployment_capabilities.rs already types/src/deployment_capabilities.rs on PR 0 (PR 0 moved it on 2026-11-07, same content); this PR takes PR 0's file
logical-asap-optimization/tests/pass_layering.rs deleted: it checked by scanning source that pass1/ and pass2/ do not import each other; the crates now cannot, and crate_graph.rs checks it
physical-asap-optimization/** (Cargo.toml, src/lib.rs, src/catalog/mod.rs) stage2-physical-optimization/**, same paths (src/catalog/realize.rs is deleted in PR 0)
plan-selection/** (Cargo.toml, src/{lib,report,select}.rs, src/milp/{mod,model}.rs, tests/cost_inputs_json.rs) stage3-plan-selection/**, same paths (src/models.rs is deleted in PR 0)

test_support.rs (the old crate's test helpers) is split by who calls each item:

Item Called by After
generate_per_computation_candidates (re-export) tests of executor, devtools, integration-tests asap_stage1_pass1_per_computation_candidates::test_support (feature test-support)
lower_promql only deleted tests (Stage 0's and Pass 1's tests that lowered PromQL) deleted; Pass 1 no longer depends on the PromQL frontend at all
share_common_sub_dags (re-export) — deleted in PR 0 with cse.rs; asap-stage0-memo has no test-support feature
population_over_selector executor tests asap_types::test_support (an IR fixture; no stage involved)
metric_scan, metric_scan_with_keys, scan, scan_from Pass 1's own tests only stage1-pass1-per-computation-candidates/src/test_fixtures.rs (#[cfg(test)], not exported)
cms_summary, weighted, estimate_over only the deleted cross-stage tests deleted
PerComputationCandidates (re-export) nobody outside the crate dropped from the re-exports
hll_classic_failure_statistic, aggregate, agg, without_agg, filter, dedup, time_range, entry, entry_with nobody (dead since PR 0) deleted

Crate graph

Before ([dependencies] edges between workspace crates; frontend helpers, devtools and integration-tests omitted):

graph LR
  planner[asap-planner] --> logical[asap-logical-asap-optimization]
  planner --> physical[asap-physical-asap-optimization]
  planner --> ps[asap-plan-selection]
  planner --> fe[frontends]
  planner --> executor[asap-executor]
  logical --> types[asap-types]
  logical --> accuracy[asap-accuracy]
  logical -. optional, test-support .-> fepromql[asap-frontend-promql]
  physical --> types
  ps --> types
  ps --> accuracy
  ps --> cost[asap-cost]
  accuracy --> types
  cost --> types
  executor --> types
  fe --> types
Loading

After:

graph LR
  planner[asap-planner] --> s0[asap-stage0-memo]
  planner --> p1[asap-stage1-pass1-per-computation-candidates]
  planner --> p2[asap-stage1-pass2-reuse-candidates]
  planner --> s2[asap-stage2-physical-optimization]
  planner --> s3[asap-stage3-plan-selection]
  planner --> fe[frontends]
  planner --> executor[asap-executor]
  s0 --> types[asap-types]
  p1 --> types
  p1 --> accuracy[asap-accuracy]
  p2 --> types
  p2 --> accuracy
  s2 --> types
  s3 --> types
  s3 --> accuracy
  s3 --> cost[asap-cost]
  accuracy --> types
  cost --> types
  executor --> types
  fe --> types
Loading

No stage crate depends on another, in [dependencies] or in [dev-dependencies], and no stage crate's tests depend on the facade (asap-planner): no test of a stage crate calls another stage crate, directly or through the facade. The stage crates here are Stage 0, Pass 1, Pass 2, Stage 2, Stage 3 and the three query-language frontends (crate_graph.rs gives the frontends the Stage role). The old optional edge Pass 1 → asap-frontend-promql is gone with lower_promql.

crate_graph.rs now also checks [dev-dependencies], with no exceptions. The rule added: a [dev-dependencies] edge from a Stage crate (the five planning-stage crates and the three frontends) to another Stage crate or to the Facade (asap-planner) is forbidden ("a stage crate's tests never call another stage crate, directly or through the facade"). Dev-dependencies on the shared crates (asap-types, asap-accuracy, asap-cost) and the frontend helpers stay allowed; a new forbidden edge fails the test.

MetricsQL → PromQL (test-only). Deleted, not kept: keeping it would have needed an exception or a new role for the frontends in crate_graph.rs. Its only users were external_corpus.rs (via PromQL's tests/support.rs, deleted for calling Pass 1) and lowering_semantics.rs's vector_negation_lowers_as_in_promql (deleted, see "Deleted tests").

Other changes

  • Facade. asap-planner depends on the five stage crates and calls them in order (run.rs: memo_of_queries → pass1_memo → generate_logical_asap_memo_candidates → Stage 2 → select_plan), as before; only the paths changed. asap_planner::{DeploymentCapabilities, SupportedSummary} re-export from asap_types::deployment_capabilities.
  • Crate-graph test (integration-tests/tests/crate_graph.rs). The three new crates get the Stage role; the two renamed crates keep it under their new names. The optional-edge exception (logical → asap-frontend-promql under test-support) is deleted with its only_if_optional field, since no optional edge is left. Dev-dependencies from a stage crate to another stage crate or to the facade are now checked, with no exceptions. Synthetic tests: optional_only_exception_rejects_the_edge_made_non_optional deleted, checker_ignores_dev_dependencies_and_external_crates → checker_ignores_external_crates_and_non_stage_dev_dependencies, checker_rejects_a_dev_dependency_between_stage_crates and checker_rejects_a_frontend_dev_dependency_on_the_facade added.
  • Cargo.toml. Every dependent crate names the new crates. Pass 2 depends on asap-accuracy: PR 0 gave its entry the deployment's &AccuracyEvidence. Dead dev-dependencies removed: asap-cost on the logical and physical crates (no test of asap-cost uses them since PR 0 deleted tests/held_memory.rs), asap-planner on the logical crate with test-support (unused); after the test deletions: Pass 1 on asap-stage0-memo and on asap-frontend-promql; Stage 0 on Pass 1; asap-frontend-promql and asap-frontend-sql on asap-stage0-memo and Pass 1; asap-frontend-metricsql on asap-stage0-memo, Pass 1 and asap-frontend-promql. Group 3: asap-frontend-promql and asap-frontend-sql on asap-planner (SQL's had no user), and asap-frontend-promql on asap-accuracy (no user, already on PR 0). The metrics_observability test target is removed from frontend-promql/Cargo.toml (its file is deleted), and docs/develop_docs/metrics-observability-corpora.md says so (its fixtures stay).
  • Docs. Crate names and paths updated in docs/develop_docs/ (architecture, contracts, extension guide, library API), docs/design_docs/architecture/, and the design proposals and decisions that name a file. Historical records are left as written (joint-candidate-workload-results.json, joint-candidate-workload-evaluation.md). The stage-document JSON keys (stage1_logical_asap_optimization, stage2_physical_asap_optimization, stage3_plan_selection) are a document format, not crate names, and are unchanged.

Not changed

  • No function body, type or signature changes, except module visibility inside the new crate roots (pub(crate) mod → mod).
  • The JSON stage-document format, the viewer and the snapshots.

How

git mv for every file, so history follows (git log --follow). Inside the moved files only paths change: crate::pass1::X → crate::X, crate::pass2::memo → crate::memo, crate::stage0::memo_of_queries → asap_stage0_memo::memo_of_queries (Pass 1 tests only). The asap_types::deployment_capabilities paths come from PR 0.

Key code interfaces added / modified

Only crate paths change. Each crate's public surface, at [b768e8d]:

stage0-memo/src/lib.rs

/// Stage 0: the memo groups of the workload's query roots (no candidates). // was asap_logical_asap_optimization::memo_of_queries
pub fn memo_of_queries(
    roots: &[QueryRoot],
    accuracy_targets: &[Option<AccuracyTarget>],
    data: &DataWorkload,
) -> Result<Memo, SchemaDerivationError>;

stage1-pass1-per-computation-candidates/src/lib.rs

/// Pass 1's entry. // was asap_logical_asap_optimization::pass1_memo
pub fn pass1_memo(
    memo: Memo,
    supported_summaries: Option<&[SupportedSummary]>,
) -> Result<Memo, Pass1Error>;
/// Why `pass1_memo` failed. // was asap_logical_asap_optimization::Pass1Error
pub enum Pass1Error { /* unchanged */ }
/// A query's DAG Pass 1 cannot plan. // was asap_logical_asap_optimization::PerComputationCandidateError
pub enum PerComputationCandidateError { /* unchanged */ }
/// Test-only (feature `test-support`). // was asap_logical_asap_optimization::test_support
pub mod test_support {
    pub use crate::per_computation_candidates::generate_per_computation_candidates;
}

stage1-pass2-reuse-candidates/src/lib.rs

// All four were at asap_logical_asap_optimization::*; signatures unchanged.
pub fn generate_logical_asap_memo_candidates(pass1: Memo, entries: &[QueryWorkloadEntry], evidence: &AccuracyEvidence) -> Result<LogicalASAPMemoCandidates, Pass2RuleError>;
pub fn pass2_local_window_composition(pass1: Memo, entries: &[QueryWorkloadEntry]) -> Result<Memo, Pass2RuleError>;
pub fn pass2_after_local_window_composition(memo: Memo, evidence: &AccuracyEvidence) -> Result<LogicalASAPMemoCandidates, Pass2RuleError>;
pub enum Pass2RuleError { /* unchanged */ }

types/src/test_support.rs

// asap_types::test_support (feature `test-support`)
/// The maintained-population sub-DAG for executor tests. // was asap_logical_asap_optimization::test_support
pub fn population_over_selector(root: &Rc<OperatorNode>, statistic: PopulationStatistic, grouping: &[&str]) -> Rc<OperatorNode>;

asap-stage2-physical-optimization and asap-stage3-plan-selection: same items as asap-physical-asap-optimization and asap-plan-selection.

Usage. The facade's flow is unchanged; only the paths it calls changed (crates/planner/src/run.rs):

// The workload's lowered query roots, one per query (Stage 0 input from the frontends).
let memo = asap_stage0_memo::memo_of_queries(&roots, &targets, data)?;
// Pass 1: the initial candidates of each group.
let pass1 = asap_stage1_pass1_per_computation_candidates::pass1_memo(memo, supported)?;
// Pass 2: reuse rules and logical edges, which Stage 2 expands.
let logical = asap_stage1_pass2_reuse_candidates::generate_logical_asap_memo_candidates(pass1, &entries, evidence)?;
// Stage 2 and Stage 3 as before, under their new crate names.

Planning still reaches PR 0's first todo!() (Stage 0's memo_of_queries, filled in by PR 2), as on PR 0.

Major algorithm changes

None.

New abstractions (pending approval)

None. One module was added: stage1-pass1-per-computation-candidates/src/test_fixtures.rs (#[cfg(test)]), holding the old test_support fixtures that only Pass 1's own tests call, so the crate does not export them.

Design notes for review

  • No stage needs another stage's code today. realization.rs (summary families per intent, summary inputs) was documented as shared by Pass 1 and Pass 2, but only Pass 1 calls it now; it stays in Pass 1. If Pass 2's rules (PR 4) need it, that is a crate-boundary question: it would move to asap-types (IR helpers) rather than Pass 2 depending on Pass 1.

Before this PR

Stage 0 and both Stage 1 passes were modules of asap-logical-asap-optimization; a source-scanning test kept the passes from importing each other. Stage 2 and Stage 3 were asap-physical-asap-optimization and asap-plan-selection. DeploymentCapabilities is already in asap-types (PR 0).

After this PR

One crate per stage and pass; the crate-graph test forbids any edge between them, including test-only edges. PRs 1–8 change code in its final crate.

Deleted tests

2026-11-09: PR 0 now deletes Pass 1's whole-workload composition and the 74 tests that compose through it (P11), so 36 of the tests listed below are deleted there, not here: a_quantile_estimate_shares_its_raw_data_samples_sign_only_inside_the_unit_interval, a_quantile_of_merged_tumbling_windows_shares_its_raw_data_samples_sign, aggregate_over_topk_candidates_compose, aggregate_replacement_preserves_sql_field_identity, avg_states_out_of_order_are_refused, benchmark_corpora_are_total_and_keep_their_coverage, binding_is_total_over_the_entire_corpus, cms_count_updates_total_ten_for_zero_positive_and_negative_samples, composed_avg_divides_sum_and_count_states, composed_candidate_replaces_chosen_target_with_summary_evaluation, composed_filters_and_having, composed_guarantees, composed_mixed_exact_and_sketch_measures_keep_output_names, composed_multi_measure_aggregate_joins_its_measures, corpus_lowers_and_binds, count_ranked_topk_fuses_to_one_global_heap_sketch, frequency_count_candidates_use_unit_weights, grouped_avg_composition_preserves_the_whole_schema, grouped_count_offers_hydra_but_selection_keeps_per_group_state, heap_weights_over_raw_data_samples_record_an_open_requirement, kubernetes_mixin_aggregates_over_topk_compose, multidimensional_ranked_item_is_a_structured_tuple, nested_sum_merges_into_one_exact_accumulator, normalized_additive_errors_compose_as_absolute_errors, own_having_subquery_is_substituted, per_series_avg_with_series_identity_composes_from_sum_and_count, per_series_rate_keeps_labels_and_retypes_its_state, subpopulation_columns_are_not_part_of_the_ranked_item_tuple, temporal_average_divides_with_a_finite_division_guard, temporal_per_entity_topk_uses_series_identity_and_sample_value, topk_keeps_known_labels_beside_series_identity, uncalibrated_frequency_evaluations_do_not_bypass_accuracy_targets, values_estimating_raw_data_samples_name_their_sign_by_the_metric, whole_expression_topk_over_a_grouped_sum_of_range_sums_reads_the_raw_data_samples, whole_expression_topk_partitions_index_the_heap_input, whole_expression_topk_replaces_the_heap_over_the_exact_sum. The tables keep the original lists for the record.

Decided 2026-11-06: tests that cross stage crates are deleted, not rewritten; PR 3 re-creates Pass 1 coverage against the new interfaces. 82 tests are deleted (63 passed, 18 failed and 1 ignored on PR 0), plus pass_layering.rs's 2. Three groups follow: (1) Stage 0 ↔ Pass 1 (54 tests), (2) frontends → Stage 0 / Pass 1, Pass 1 → PromQL frontend, and MetricsQL → PromQL (27 tests), (3) frontends → the facade (1 test).

(1) Stage 0 ↔ Pass 1

Pass 1's generate_per_computation_candidates takes Stage 0's Memo, so every Pass 1 test that built its input with memo_of_queries (directly or through a helper) goes. Their helpers with no caller left are deleted too (listed after the table).

File Tests PR 0 result What they covered
stage0-memo/src/lib.rs (was stage0.rs), mod tests repeated_results_share_groups_but_quantile_levels_do_not, equal_keyless_deterministic_results_share_one_group, scalar_and_literal_roots_keep_workload_positions (3) 3 passed memo_of_queries over PromQL lowered through Pass 1's lower_promql: equal results share a group, different quantile levels do not; G2 / G2 is one group read twice; scalar and literal roots keep their query positions. cse.rs and its 24 hash-consing tests were deleted in PR 0.
stage1-pass1-per-computation-candidates/src/per_computation_candidates_tests.rs scalar_root_producers_are_discovered_once, example_1_exact_q1_keeps_exact_accumulators_only, asap_input_and_invalid_accuracy_are_rejected, multi_measure_aggregate_has_one_target_per_measure, aggregate_without_realization_is_not_a_target, filtered_measure_keeps_candidates_and_records_its_filter (target discovery and input checks, 6); key_only_having_is_pushed_below_and_keeps_candidates, value_having_stays_above_output_and_keeps_candidates, having_on_one_consumer_of_a_shared_aggregate_stays_above, a_scalar_reference_consumer_keeps_a_key_only_having_above, only_the_adjacent_of_stacked_filters_is_having, global_aggregate_having_stays_above, scalar_reference_filter_is_not_having, promql_scalar_operand_is_not_having, having_shared_with_a_project_consumer_stays_above, without_and_per_entity_having_stays_above, memo_cached_rules_preserve_occurrence_specific_having (HAVING placement, 11); composed_candidate_replaces_chosen_target_with_summary_evaluation, composed_filters_and_having, composed_avg_divides_sum_and_count_states, composed_multi_measure_aggregate_joins_its_measures, composed_mixed_exact_and_sketch_measures_keep_output_names, composed_guarantees, own_having_subquery_is_substituted, per_series_avg_with_series_identity_composes_from_sum_and_count, per_series_rate_keeps_labels_and_retypes_its_state, grouped_avg_composition_preserves_the_whole_schema (composing a chosen candidate, 10) — 27 21 passed, 6 failed (composed_candidate_replaces_chosen_target_with_summary_evaluation, composed_filters_and_having, composed_guarantees, composed_mixed_exact_and_sketch_measures_keep_output_names, composed_multi_measure_aggregate_joins_its_measures, own_having_subquery_is_substituted) Frontend-to-Pass-1 acceptance: which computation targets a query has, where a HAVING goes, and the DAG a chosen candidate composes to. The file's 4 tests of computation_candidates_for_intent stay.
stage1-pass1-per-computation-candidates/src/per_computation_candidates.rs, mod tests whole_expression_topk_replaces_the_heap_over_the_exact_sum, whole_expression_topk_partitions_index_the_heap_input, whole_expression_topk_over_a_grouped_sum_of_range_sums_reads_the_raw_data_samples, heap_weights_over_raw_data_samples_record_an_open_requirement, grouped_range_sums_a_raw_data_samples_heap_cannot_rank_get_no_fused_heap, a_fused_computation_absorbs_what_its_absorbed_targets_merges_absorb, combinations_skip_choices_a_whole_expression_candidate_absorbs, count_ranked_topk_fuses_to_one_global_heap_sketch (fused heaps and whole-expression top-k, 8); nested_sum_merges_into_one_exact_accumulator, nested_sum_merge_needs_a_sum_of_sums, nested_sum_merge_never_groups_by_the_dropped_metric_name, nested_sum_merges_do_not_stack (nested-sum merges, 4); avg_states_out_of_order_are_refused, temporal_average_divides_with_a_finite_division_guard (2); multidimensional_ranked_item_is_a_structured_tuple, subpopulation_columns_are_not_part_of_the_ranked_item_tuple, temporal_per_entity_topk_uses_series_identity_and_sample_value, topk_keeps_known_labels_beside_series_identity (ranked items, 4); a_quantile_estimate_shares_its_raw_data_samples_sign_only_inside_the_unit_interval, a_quantile_of_merged_tumbling_windows_shares_its_raw_data_samples_sign, values_estimating_raw_data_samples_name_their_sign_by_the_metric (raw-data-samples signs, 3); guarantee_composition::normalized_additive_errors_compose_as_absolute_errors (1); stream_norms::a_norm_follows_the_update_weight, stream_norms::the_absolute_bound_names_the_weighted_norm (2) — 24 19 passed, 5 failed (the two quantile-sign tests, normalized_additive_errors_compose_as_absolute_errors, both stream_norms tests) Pass 1's fused and whole-expression candidates, merges, ranked items, and the guarantees and signs of its estimates. Of the module's other 18 tests, 14 are deleted in group (2) for lowering PromQL; 4 stay, with realization.rs's 5.

Helpers deleted with them (no caller left): in per_computation_candidates_tests.rs composed, scan, aggregate_over, aggregate, always, exact_promql, choices_of, assert_sql_count_choices, quantile, grouped_quantile, grouped_count, having_filter, equals, having_of, above_scalar, columns, without_output_aliases, summary_agg; in per_computation_candidates.rs's tests inventory, inventory_root, inventory_at, open_inventory, heap_weight_domains, composed_with, heap_states, topk_target, enumerated_choices, nested_sum_target, fused_heap, heap_update, and the emptied mod stream_norms; in test_fixtures.rs cms_summary, weighted, estimate_over. Dev-dependencies removed: Pass 1 → asap-stage0-memo, Stage 0 → asap-stage1-pass1-per-computation-candidates.

(2) Frontends → Stage 0 / Pass 1, Pass 1 → PromQL frontend, MetricsQL → PromQL

File Tests PR 0 result What they covered
frontend-promql/tests/count_planning.rs grouped_count_offers_hydra_but_selection_keeps_per_group_state, frequency_count_candidates_use_unit_weights, cms_count_updates_total_ten_for_zero_positive_and_negative_samples, logical_candidates_fails_on_a_candidate_that_does_not_compose, aggregate_over_topk_candidates_compose (5) 2 passed (logical_candidates_fails_on_a_candidate_that_does_not_compose, aggregate_over_topk_candidates_compose), 2 failed (frequency_count_candidates_use_unit_weights, cms_count_updates_total_ten_for_zero_positive_and_negative_samples), 1 ignored (grouped_count_offers_hydra_but_selection_keeps_per_group_state) Pass 1 candidates of PromQL count queries: Hydra offered for grouped counts, unit weights for frequency counts, CMS update totals, composition failures surfacing, aggregate-over-topk composing. The file's 4 tests that lower or plan through the facade stay.
frontend-promql/tests/univmon_candidates.rs (file deleted) uncalibrated_frequency_evaluations_do_not_bypass_accuracy_targets (1) failed UnivMon candidates generated by Pass 1 are rejected by selection without calibration.
frontend-promql/tests/observability/promql_corpus.rs binding_is_total_over_the_entire_corpus (1) failed Pass 1 binding (logical_plan) never errors over the Prometheus docs and testdata corpora. lowering_is_total_over_the_entire_corpus stays.
frontend-promql/tests/observability/metrics_observability.rs (file and test target deleted) benchmark_corpora_are_total_and_keep_their_coverage, kubernetes_mixin_aggregates_over_topk_compose (2) 1 failed, 1 passed Coverage ratchets of ASAP candidates per metrics-observability corpus; Kubernetes-mixin aggregates over topk compose through Pass 1.
frontend-sql/tests/frequency_l2.rs stage1_offers_exact_and_summary_l2_candidates (1) passed Pass 1 offers exact and summary candidates for the SQL L2 intent. The file's lowering tests stay.
frontend-metricsql/tests/external_corpus.rs (file deleted) corpus_lowers_and_binds, floor_line (2) 2 passed MetricsQL external corpus lowers and binds through Pass 1 (skipped without the corpus directory); floor_line tested only that test's parsing helpers.
frontend-metricsql/tests/lowering_semantics.rs vector_negation_lowers_as_in_promql (1) passed MetricsQL vector negation lowers to the same IR as PromQL's (the only MetricsQL → PromQL use left).
stage1-pass1-per-computation-candidates/src/per_computation_candidates.rs, mod tests aggregate_replacement_preserves_sql_field_identity, raw_data_samples_of_metric_ends_at_any_other_operator, classic_hll_confidence_needs_one_instance_over_a_sources_own_rows; guarantee_composition::{absolute_errors_compose_through_arithmetic, products_and_quotients_of_estimates_have_closed_or_symbolic_bounds, relative_and_rank_errors_compose_only_where_sound, clamp_and_abs_keep_the_largest_argument_error, comparisons_over_estimates_have_no_guarantee, a_scalar_subquery_inherits_its_guarantee, failure_probability_counts_once_per_distinct_estimate, join_beside_an_exact_input_on_exact_columns_keeps_the_guarantee, join_on_an_approximate_key_has_no_guarantee, case_on_exact_columns_keeps_the_branch_guarantee, projection_over_a_join_reads_each_columns_guarantee} (14) 13 passed, 1 failed (classic_hll_confidence_needs_one_instance_over_a_sources_own_rows) Pass 1's guarantee composition over PromQL-lowered estimates (arithmetic, products and quotients, metrics, clamp and abs, comparisons, scalar subqueries, δ counting, joins, CASE, projections), raw-data-samples requirements, HLL confidence, SQL field identity through aggregate replacement. The emptied mod guarantee_composition (its helpers with_bound, abs, rel, lit, arith, call, of, bound, delta, close, join, column, equals, case_over, project) is deleted.

Helpers deleted with them: lower_promql (Pass 1's test_fixtures.rs); node, node_over, scalar, approximate_and_exact (Pass 1's tests); logical_candidates, local_computation_candidates, LogicalCandidate, require_composed_operator_candidates, logical_plan (PromQL's tests/support.rs); reads_hydra (count_planning.rs); bind_tally, BindTally (promql_corpus.rs). MetricsQL's tests no longer include PromQL's support.rs.

(3) Frontends → the facade (asap-planner)

File Tests PR 0 result What they covered
frontend-promql/tests/count_planning.rs exact_counts_select_count_accumulators (1) failed (Pass 1's todo!()) count(up), count by(job)(up) and count_over_time(up[5m]) at an exact target select a count accumulator when planned through asap_planner::run_planning. The file's 2 fixture tests of lowered count aggregates stay.

Helpers deleted with it: frontend-promql/tests/planner_support.rs (run_all_stages, selected_dag; file deleted). Helpers that had no caller left: physical_asap_dag (its callers were deleted in group 2) and declared_data (no caller on PR 0 either) in PromQL's tests/support.rs. No SQL or MetricsQL test used asap-planner; SQL's dev-dependency on it was unused and is removed.

Verification

  • cargo check --workspace --all-targets, cargo clippy --workspace --all-targets -- -D warnings and cargo fmt --all -- --check: clean.
  • crate_graph.rs: passes with the new crates and edges, including the new dev-dependency checks (6 tests).
  • Rebased 2026-11-06 (three times) onto PR 0's memo changes (Result signatures, private groups, MemoGroup { expression, group_properties }, coverage only in the memo, size_for_all_readers deleted; last, InputOrTemporaryBuild, unlifted_operators: Vec<Operator<Option<MemoGroupId>>> and no node-tree wrapper of merged_window_build, which touch only asap-types and asap-accuracy and applied without conflicts): the move commit carries them to the new paths. One commit is added: the planner's tests now take asap-types' test-support feature directly; it came through asap-logical-asap-optimization's test-support, which the move deleted, so cargo test -p asap-planner did not build (the workspace build hid it by feature unification). cargo check, cargo clippy --workspace --all-targets -- -D warnings and cargo fmt --check are clean, and crate_graph passes (6).
  • Rebased 2026-11-07 onto PR 0's head facaa92f8 (chains without a window count, TimeBound, Pass 2 pruned list, Stage 2 entry (logical, data, &DeploymentCapabilities), and DeploymentCapabilities moved into asap-types by PR 0 itself with the content this PR had). The move commit takes PR 0's deployment_capabilities.rs and PR 0's new call sites (run.rs, planning_stages.rs, Stage 2's catalog/mod.rs) under the new crate paths, and no longer moves DeploymentCapabilities. Still move-only: git range-diff against the previous head shows only that and PR 0's own changes. Previous head kept as backup/1207-crates-per-stage-4445d45df. cargo clippy --workspace --all-targets --all-features -- -D warnings and cargo fmt --check are clean; crate_graph passes (6).
  • Test status at the head (058ea42, 2026-11-09 after P11; compared by test name with the previous head 0e0f75a7d: only the 36 tests PR 0 now deletes with pass1/composition.rs are gone, 35 failed and 1 passed there; no test changed result), cargo test -p <crate> --all-features --no-fail-fast. Compared by test name with the previous head b768e8dd1 (r1108c logs): the only differences are PR 0's own (its 8 deleted tests and the added target_limit_admits_only_floating_point_noise); compared with PR 0 at be6ea1d6a, only the deleted cross-stage tests and the moved test paths differ, as before. No test changed result. crate_graph: 6 passed. Every failure panics at a todo!(), except the stale fixture (PR 9):
Crate Passed Failed Ignored Fails at
asap-types 153 2 0 reject the invalid combination of section 2 (PR 1): 2
asap-accuracy 59 0 0 —
asap-cost 31 1 0 check θ, which the exported node carries (PR 6): 1
asap-frontend-common 49 0 0 —
asap-sql-function-catalog 10 0 0 —
metricsql_common 3 0 0 —
asap-frontend-promql 217 0 0 —
asap-frontend-metricsql 15 0 0 —
asap-frontend-sql 173 0 0 —
asap-executor 210 63 1 check θ, which the exported node carries (PR 6): 32; the Hydra grid's sizes are θ, which the exported node carries (PR 6): 13; each materialized node's retention_ms is its h_v, which Stage 3 sets (PR 6): 8; the weighted-frequency θ, which the exported node carries (PR 6): 7; check the weighted-frequency θ, which the exported node carries (PR 6): 2; Stage 0 over Memo::find_or_add_group, each query with its target and n_q (PR 2): 1
asap-planner 8 18 1 Stage 0 over Memo::find_or_add_group, each query with its target and n_q (PR 2): 18
asap-devtools 37 56 3 Stage 0 over Memo::find_or_add_group, each query with its target and n_q (PR 2): 54; OTHER: assertion failed: output.status.success(): 2
asap-integration-tests 83 51 4 Stage 0 over Memo::find_or_add_group, each query with its target and n_q (PR 2): 47; each materialized node's retention_ms is its h_v, which Stage 3 sets (PR 6): 3; OTHER: called Result::unwrap() on an Err value: Error("invalid type: sequence, expected string or map", line: 0, column: 0): 1
asap-stage3-plan-selection 3 0 0 —
asap-stage0-memo 0 0 0 —
asap-stage2-physical-optimization 0 0 0 —
asap-stage1-pass2-reuse-candidates 0 0 0 —
asap-stage1-pass1-per-computation-candidates 11 0 0 —
Total 1062 191 9
  • History. Before the rebases, the move was checked test by test: cargo test --workspace --no-fail-fast on PR 0 (dab3d064f) and on this PR's first commit (80b5b09b8, the move): every test has the same result, compared by name, except pass_layering.rs's 2 deleted tests. On the deletion commits (dfc78f40d, group 1; f6e2daa49, group 2; 4d5537310, group 3) cargo test was re-run for the crates they change (asap-stage0-memo, asap-stage1-pass1-per-computation-candidates, asap-stage1-pass2-reuse-candidates, the three frontends, and crate_graph.rs in asap-integration-tests); each remaining test has its PR 0 result. Every other crate's code is unchanged, and cargo clippy --workspace --all-targets builds all tests. Per crate:
Crate PR 0 (dab3d064f) passed / failed / ignored This PR passed / failed / ignored
asap-types 200 / 2 / 0 201 / 2 / 0
asap-accuracy 77 / 0 / 0 77 / 0 / 0
asap-cost 31 / 1 / 0 31 / 1 / 0
asap-frontend-common 49 / 0 / 0 49 / 0 / 0
asap-sql-function-catalog 10 / 0 / 0 10 / 0 / 0
metricsql_common 3 / 0 / 0 3 / 0 / 0
asap-frontend-promql 221 / 6 / 1 218 / 0 / 0
asap-frontend-metricsql 18 / 0 / 0 15 / 0 / 0
asap-frontend-sql 175 / 0 / 0 174 / 0 / 0
asap-logical-asap-optimization 95 / 13 / 0 —
asap-stage0-memo — 24 / 0 / 0
asap-stage1-pass1-per-computation-candidates — 12 / 1 / 0
asap-stage1-pass2-reuse-candidates — 0 / 0 / 0
asap-stage2-physical-optimization (was asap-physical-asap-optimization) 0 / 0 / 0 0 / 0 / 0
asap-stage3-plan-selection (was asap-plan-selection) 3 / 0 / 0 3 / 0 / 0
asap-executor 217 / 56 / 1 217 / 56 / 1
asap-planner 9 / 19 / 1 9 / 19 / 1
asap-devtools 49 / 48 / 3 49 / 48 / 3
asap-integration-tests 102 / 76 / 4 103 / 76 / 4
Total 1259 / 221 / 10 1195 / 203 / 9

(Measured on PR 0 as it was then, dab3d064f; the head's numbers are in the table above.)

The old crate's 95 / 13 / 0: 24 / 0 / 0 stay in Stage 0 and 12 / 1 / 0 in Pass 1; 1 / 0 / 0 moved to asap-types (deployment_capabilities.rs's default_supports_everything_and_json_fills_the_rest: 200 → 201); 56 / 12 / 0 deleted as cross-stage tests (groups 1 and 2); 2 / 0 / 0 deleted with pass_layering.rs. The frontends lost 7 / 5 / 1 (group 2) and 0 / 1 / 0 (group 3). In asap-integration-tests, crate_graph.rs has 6 tests (one deleted, two added). Every other test has the same result as on PR 0. The remaining failures are PR 0's todo!() panics.

Rebased 2026-11-07 (second time) onto PR 0 facaa92f8 (IntervalConstraintLimit replaces TimeBound; ScalarValue::Timestamp). Move-only, unchanged: git range-diff shows every commit identical. Previous head kept as backup/1207-crates-per-stage-84cf80d6f. Test table re-measured at 69fed95: 1089 / 230 / 9, the 3 more passing tests are PR 0's new summary_coverage unit tests; crate_graph 6/6.

Rebased 2026-11-08 onto PR 0 7e83db97d (interval examples left-open, right-closed; ParametersOfAnotherFamily deleted). Move-only, unchanged: git range-diff shows every commit identical. Previous head kept as backup/1207-crates-per-stage-69fed95ae. fmt and clippy -D warnings (--all-features) clean; crate_graph 6/6.

Rebased 2026-11-08 (second time) onto PR 0 7708f3668 (the 2026-11-08 interfaces: summary types, required accuracy, the plan is one PhysicalASAPDAG). Still move-only: the conflicts were resolved by taking PR 0's content and applying this PR's path changes. What moved with it:

  • PR 0's new pass1/composition.rs (test support only) → stage1-pass1-per-computation-candidates/src/composition.rs; test_support re-exports compose_logical_candidate from it.
  • PR 0 deleted Stage 2's catalog/realize.rs and Stage 3's models.rs: taken.
  • PR 0's new signatures (memo_of_queries(roots, targets, data), pass1_memo(memo, supported), Pass 2's evidence: &AccuracyEvidence) under the new crate paths; Pass 2 now depends on asap-accuracy.
  • The commit "build(planner): its tests read asap-types' test-support directly" is dropped: PR 0 now adds the same dev-dependency itself (the commit would have repeated the key).
  • The deletion commits delete the same tests by name. On PR 0 7708f3668 they are 79 cross-stage tests (17 passed, 61 failed, 1 ignored there; PR 0 renamed asap_input_and_invalid_accuracy_are_rejected to asap_input_is_rejected and itself deleted count_min_estimate_over_raw_data_samples_is_gated_on_the_metric), plus pass_layering.rs's 2. The result columns of the "Deleted tests" tables were measured on PR 0 as of 2026-11-06.

git range-diff 7e83db97d..dfd6a5bf0 7708f3668..b768e8dd1: four commits, each changed only by PR 0's own changes and the path renames above; the fifth dropped. Two doc rows (asap-aware-mapping-architecture.md: Pass 2 memo rules and plan enumeration) keep this PR's paths, not PR 0's stale ones. Test status by name, against PR 0 at 7708f3668: the only differences are the deleted tests and the moved test paths (pass1::… → crate root); no test changed result. fmt and clippy -D warnings (--all-features) clean; crate_graph 6/6. Previous head kept as backup/1207-crates-per-stage-dfd6a5bf0.

Self-review (2026-11-08). Over-abstraction: none added. Duplication: none; composition.rs stays test support (PR 3 deletes it). Dead code: none (clippy -D warnings); the dropped planner commit would have duplicated PR 0's dev-dependency.

Rebased 2026-11-09 onto PR 0 be6ea1d6a (the 2026-11-09 interfaces: guarantees at a failure probability, AccuracyEvidence trait, MemoQuery, SupportedSummary::summary_type, satisfies deleted, delivers_on(&PhysicalASAPMemoCandidates, context), depends_on_evaluation_time, evaluation_grid_of_group). Still move-only. One conflict: the DeploymentCapabilities row of library-api.md, resolved as this PR's crate path with PR 0's summary_type. git range-diff 7708f3668..b768e8dd1 be6ea1d6a..0e0f75a7d: commits 2–4 identical; commit 1 differs only in context lines (PR 0's own changes, e.g. pass.rs importing NoMeasuredGuarantees) and that doc row. fmt and clippy -D warnings (--all-features) clean; crate_graph 6/6. Previous head kept as backup/1207-crates-per-stage-b768e8dd1.

Self-review (2026-11-09). Over-abstraction, duplication, dead code: none added; the rebase only carried PR 0's changes to the new paths.

Rebased 2026-11-09 (second time) onto PR 0 2ed6ec0b8, which deletes Pass 1's whole-workload composition (P11). Still move-only: the move commit no longer moves pass1/composition.rs (it is gone), test_support no longer re-exports compose_logical_candidate, and test_fixtures.rs no longer gets cms_summary, weighted, estimate_over; the deletion commits keep their other deletions. git range-diff be6ea1d6a..0e0f75a7d 2ed6ec0b8..058ea4234: four commits, changed only there. Previous head kept as backup/1207-crates-per-stage-0e0f75a7d. fmt and clippy -D warnings (--all-features) clean; crate_graph 6/6.

Self-review (2026-11-09, second). Nothing added; the composition and its fixtures are gone from both the move and the deletion commits.

@zzylol
zzylol force-pushed the refactor/1207-crates-per-stage branch 2 times, most recently from 84cf80d to 69fed95 Compare November 7, 2026 20:51
@zzylol
zzylol force-pushed the refactor/1207-crates-per-stage branch 3 times, most recently from b768e8d to 0e0f75a Compare November 9, 2026 03:33
@zzylol
zzylol force-pushed the refactor/1207-interfaces branch from e9d9f5a to 2ed6ec0 Compare November 9, 2026 04:25
Move-only. Split asap-logical-asap-optimization into asap-stage0-memo,
asap-stage1-pass1-per-computation-candidates and
asap-stage1-pass2-reuse-candidates; rename asap-physical-asap-optimization
to asap-stage2-physical-optimization and asap-plan-selection to
asap-stage3-plan-selection. The old test_support splits by caller. The
crate-graph test checks the new crates; no stage crate depends on another.
Pass 1's tests that build Stage 0's memo and Stage 0's tests that lower
PromQL through Pass 1 are deleted, with the dev-dependencies and helpers
only they used. lower_promql moves to Pass 1's test fixtures, so Pass 1
no longer depends on the PromQL frontend outside its own tests. The
crate-graph test now also checks dev-dependencies between stage crates.
…1207 PR 0b)

Frontend tests that compose Pass 1 candidates and Pass 1 tests that
lower PromQL through the frontend are deleted, with the helpers,
dev-dependencies and test targets only they used. MetricsQL's test that
compares with PromQL's lowering goes too, so no frontend tests depend on
another frontend. The crate-graph test checks dev-dependencies between
stage crates with no exceptions.
…R 0b)

The PromQL test that planned through asap-planner is deleted with its
planner_support helpers; the PromQL and SQL frontends drop their
asap-planner dev-dependency, and PromQL its unused asap-accuracy one.
The crate-graph test now also forbids a stage crate's dev-dependency on
the facade.
@zzylol
zzylol force-pushed the refactor/1207-crates-per-stage branch from 0e0f75a to 058ea42 Compare November 9, 2026 04:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

optimizer Workload optimization and plan selection refactor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants