Repository navigation
refactor(types): properties and requirements from #1207; chains as time-indexed state (#1207 PR 1) - #1227
Draft
zzylol wants to merge 1 commit into
Draft
refactor(types): properties and requirements from #1207; chains as time-indexed state (#1207 PR 1)#1227zzylol wants to merge 1 commit into
zzylol wants to merge 1 commit into
Conversation
zzylol
force-pushed
the
refactor/1207-properties
branch
4 times, most recently
from
November 6, 2026 15:37
5ef1ef3 to
d9d64df
Compare
zzylol
force-pushed
the
refactor/1207-properties
branch
from
November 6, 2026 15:47
d9d64df to
2a62109
Compare
zzylol
force-pushed
the
refactor/1207-properties
branch
from
November 6, 2026 16:30
2a62109 to
b87b7c0
Compare
zzylol
changed the base branch from
refactor/1207-interfaces
to
refactor/1207-crates-per-stage
November 6, 2026 16:30
zzylol
force-pushed
the
refactor/1207-properties
branch
3 times, most recently
from
November 6, 2026 17:12
460e9c7 to
ba60db8
Compare
zzylol
force-pushed
the
refactor/1207-crates-per-stage
branch
from
November 6, 2026 18:56
4d55373 to
ae4843e
Compare
zzylol
force-pushed
the
refactor/1207-properties
branch
from
November 6, 2026 18:56
ba60db8 to
60a16ee
Compare
zzylol
force-pushed
the
refactor/1207-crates-per-stage
branch
from
November 6, 2026 19:10
ae4843e to
8d020b0
Compare
zzylol
force-pushed
the
refactor/1207-properties
branch
from
November 6, 2026 19:12
60a16ee to
107a77b
Compare
zzylol
force-pushed
the
refactor/1207-crates-per-stage
branch
from
November 6, 2026 21:29
8d020b0 to
d1feaad
Compare
zzylol
force-pushed
the
refactor/1207-properties
branch
from
November 6, 2026 21:31
107a77b to
c7a0bc9
Compare
zzylol
force-pushed
the
refactor/1207-crates-per-stage
branch
from
November 6, 2026 21:38
d1feaad to
9a3bb19
Compare
zzylol
force-pushed
the
refactor/1207-properties
branch
from
November 6, 2026 21:38
c7a0bc9 to
9e5ce62
Compare
zzylol
force-pushed
the
refactor/1207-crates-per-stage
branch
from
November 6, 2026 21:57
9a3bb19 to
345b5df
Compare
zzylol
force-pushed
the
refactor/1207-properties
branch
from
November 6, 2026 22:11
9e5ce62 to
96d3457
Compare
zzylol
force-pushed
the
refactor/1207-crates-per-stage
branch
from
November 6, 2026 22:46
cbd0359 to
4445d45
Compare
zzylol
force-pushed
the
refactor/1207-properties
branch
from
November 6, 2026 22:46
96d3457 to
b250852
Compare
5 tasks done
zzylol
force-pushed
the
refactor/1207-crates-per-stage
branch
from
November 7, 2026 17:47
4445d45 to
84cf80d
Compare
zzylol
force-pushed
the
refactor/1207-properties
branch
from
November 7, 2026 18:04
b250852 to
7ceb7c3
Compare
zzylol
force-pushed
the
refactor/1207-crates-per-stage
branch
from
November 7, 2026 20:51
84cf80d to
69fed95
Compare
zzylol
force-pushed
the
refactor/1207-properties
branch
from
November 7, 2026 20:55
7ceb7c3 to
12a4b96
Compare
zzylol
force-pushed
the
refactor/1207-crates-per-stage
branch
from
November 8, 2026 15:06
69fed95 to
dfd6a5b
Compare
zzylol
force-pushed
the
refactor/1207-properties
branch
from
November 8, 2026 15:10
12a4b96 to
cf93b37
Compare
zzylol
force-pushed
the
refactor/1207-crates-per-stage
branch
from
November 8, 2026 21:29
dfd6a5b to
b768e8d
Compare
zzylol
force-pushed
the
refactor/1207-properties
branch
from
November 8, 2026 21:42
cf93b37 to
1ca33f1
Compare
zzylol
force-pushed
the
refactor/1207-crates-per-stage
branch
from
November 9, 2026 03:33
b768e8d to
0e0f75a
Compare
zzylol
force-pushed
the
refactor/1207-properties
branch
from
November 9, 2026 03:45
1ca33f1 to
3c28259
Compare
zzylol
force-pushed
the
refactor/1207-crates-per-stage
branch
from
November 9, 2026 04:31
0e0f75a to
058ea42
Compare
- LogicalProperties::new(expression, parameters): a sketch build (a chain included) needs θ from parameter_list(algorithm) of its FieldDataType::Sketch(algorithm); an exact operator has none. - parameter_list(algorithm): structures only. Count-Min lists each width with depths 5, 7, 10 and 12, Count Sketch with 47, 81, 117 and 155 (its median-of-rows δ levels 1e-2 … 1e-5); heap variants each of those with the 1-2-5 heap sizes; UnivMon 40 entries; KLL, DDSketch, HLL, KMV one entry per ε level. A test keeps the lists equal to today's sizing, which is test-only. - PhysicalProperties::new rejects (IngestionTime, OneEvaluation); producer_satisfies_physical_requirements(producer, consumer_timing). - PhysicalCandidate::delivers_on(candidates, context): None in every context, OneEvaluation only in Evaluation contexts, a chain with AcrossEvaluations when its width divides the grid's period and phase (or the refresh window's width and phase), any other AcrossEvaluations result everywhere when its group does not depend on the evaluation time, else when its group's evaluation grid holds the context's times; Materialized(v) contexts stay todo!() for PR 7. - Chains: one schema (window_end at column 0, keys, state), TimeRange and TimeShift over a chain select windows (an instant TimeRange is rejected), a SummaryMerge over them finalizes and rolls up as the chain's build over that range (the two PR 0 todos). - summary_type_of_operator; QueryRecurrence::evaluation_grid_ms is None for a fixed interval without a phase.
zzylol
force-pushed
the
refactor/1207-properties
branch
from
November 9, 2026 04:33
3c28259 to
8030d3f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Every later stage reads the property checks of the concepts doc, section 2:
parameter_list), checked byLogicalProperties::new.producer_satisfies_logical_requirements, inasap-accuracy, PR 4).PhysicalProperties::new) and edges with α^phys (producer_satisfies_physical_requirements).delivers_on).PR 0 defined these signatures with
todo!()bodies. On 2026-11-08 PR 0 changed them: θ is structure only (no sized-for fields, derived equality), a summary type isFieldDataType::Sketch(SketchAlgorithm)(no grouping strategy), the accuracy a readout needs isRequiredAccuracyinstead of a parameter floor, and execution contexts are evaluation grids. This PR implements the checks against that design.What
Logical properties.
LogicalProperties::new(expression, parameters)reads the summary type of a build (summary_type_of_operator) and, forFieldDataType::Sketch(algorithm), requires θ fromparameter_list(algorithm). It rejects a sketch build, a chain included, without θ (SketchBuildWithoutParameters), θ on an operator that builds no sketch (ParametersOnExactOperator), and θ not in the list (ParametersNotInList, which also covers another algorithm's θ).Predefined θ lists (structures only).
parameter_list(algorithm):distinct_failure_probability)The widths,
ks, precisions and heap sizes are today's sizing over ε = 0.1 … 0.001 (smallest first); a test inasap-accuracykeeps them equal to it. That sizing is test-only.No parameter floors. This PR adds no
floor_for_cap,strongest,weakest,dominatesorcombine: a readout's input carries aRequiredAccuracy, checked against the producer's guarantee (PR 4). The previous version of this PR had them; they are deleted.Physical properties.
PhysicalProperties::newrejects the one invalid combination, (IngestionTime, OneEvaluation).producer_satisfies_physical_requirements(producer, consumer_timing)is the timing × lifetime table.delivers_on(&self, candidates: &PhysicalASAPMemoCandidates, context)over execution contexts (2026-11-08, PR 7 J3 and K3; the stored-result rows 2026-11-09):Evaluation { grid }IngestionPipeline(RefreshWindow)Materialized(v)period_msandphase_mswidth_msandphase_mstodo!()(PR 7: v's own schedule)depends_on_evaluation_timeevaluation_grid_of_group[group]period_msandphase_ms≡ g's phase (mod g's period)width_mstodo!()(PR 7)None)todo!()(PR 7)Example (doc, 2026-11-09): X =
sum_over_time(m[5m])read every 60 s and every 120 s has grid 60 s @ 0; a stored X delivers on 60 s @ 0, 120 s @ 0 and 120 s @ 60 s, not on 30 s @ 0 or 60 s @ 15 s.Chains as time-indexed state.
SummaryTumblingWindowChain: (window_end, keys…, state).window_endat column 0 is the time index, a computed column; the build's own time column is dropped; the unique key is (window_end, keys).TimeRangeandTimeShiftaccept a time-indexed state and keep its schema; over any other state they are rejected, and aTimeRangeover a chain must be a range (Instantis a validation error).SummaryMergeover a time-indexed state reads exactly oneTimeRangeover a chain; its schema is the chain's build over that range, so it can join the original state group.todo!()s are filled in:FinalizeExactAccumulatorand a rollup over merged windows derive as over the chain's build over that range.selected_fromandsummary_buildsee through theTimeRange/TimeShiftover a chain.Evaluation grid.
QueryRecurrence::evaluation_grid_ms(range_evaluation)(moved fromQueryWorkloadEntry) isNonefor aFixedIntervalwithout a phase. Pass 2's chain widths and Stage 3's evaluation grids come from it.Accuracy crate. The sizing functions are
#[cfg(test)];sizing::default_size_params,test_support::default_size_paramsandtest_support::cms_widthare deleted (no caller).propagation.rsreads merged windows throughsummary_build(PR 0'smerged_chainis gone).Decisions
floor_for_cap/strongest/weakest/dominates/combine;RequiredAccuracyinsteadFieldDataType::Sketch(SketchAlgorithm);summary_type_of_operator(wasfamily_of_operator);sketch_algorithm_ofdeletedproducer_satisfies_physical_requirements(producer, consumer_timing)delivers_onEvaluationcontextsdelivers_onforIngestionPipeline(RefreshWindow)/Materialized(v)Materialized(v)staystodo!()for PR 7delivers_on(&PhysicalASAPMemoCandidates, context)(P2)GroupProperties::depends_on_evaluation_time,LogicalASAPMemoCandidates::evaluation_grid_of_group(P3)delivers_onreads them hereTimeRangeover a chain rejectedSummaryMergeover a time-indexed stateTimeRangeselected_from/summary_buildsee throughTimeRange/TimeShiftevaluation_grid_msNoneforFixedIntervalwithout a phase; theQueryWorkloadEntrywrapper deletedasap-accuracyPending approval
None: P1–P3 were approved on 2026-11-09 (see "Decisions").
Deleted
strongest,weakest,dominates,combineand their tests (strongest_and_weakest_combine_one_family_per_parameter,a_listed_univmon_dominates_an_l2_and_a_distinct_floor,hydra_dominance_compares_every_size_of_one_kind)producer_satisfies_logical_requirements(properties, requirement)(θ dominance) and its test,LogicalRequirements::newandlogical_requirement_new_accepts_any_floorasap_accuracy::producer_satisfies_logical_requirements(memo, producer, consumer, input_index, evidence)(PR 0 interface, PR 4 body);LogicalRequirements { required_accuracy }a_configuration_satisfying_a_listed_floor_is_no_less_accurate(accuracy)family_of_operatorandsketch_algorithm_ofsummary_type_of_operator;FieldDataType::Sketch(algorithm)holds the algorithmdistinct_failure_probabilitydelivers_on's reading ofQueryRecurrence/RangeEvaluationfrom the contextEvaluationGrid/RefreshWindowpub fn merged_chain(PR 0's,tumbling_window_chain.rs)summary_build/selected_fromthrough the window selectionQueryWorkloadEntry::evaluation_grid_msQueryRecurrence::evaluation_grid_ms(range_evaluation)sizing::default_size_params,test_support::default_size_params,test_support::cms_widthnew_rejects_one_evaluation_at_ingestion,new_accepts_each_valid_pairnew_over_every_timing_and_lifetimeHow
Lists.
constarrays built by twoconst fns:with_depths(widths, depths, shape)crosses 7 widths with the 4 depths (CMS_DEPTHSorCOUNT_SKETCH_DEPTHS) (asCmsorCountSketch),with_heap_sizes(shapes)crosses those 28 with the 10 heap sizes;univmon_parameters()builds the 40 UnivMon shapes. Thematchinparameter_listlists everySketchAlgorithm(no_), so a new sketchlib algorithm fails to compile until it gets a list.delivers_onmatches ℓ; for AcrossEvaluations it reads the candidate's operator fromcandidates.logical_candidates.memo. A chain needs itstumbling_window_width_msw to divide the context's (period, phase): the grid's, or the refresh window's (width, phase), so w divides every time phase + k·period. Any other result delivers everywhere when its group'sdepends_on_evaluation_timeis false; else every context time must lie onevaluation_grid_of_group[group]: its period divides the context's and the phases agree modulo it.Chain schema.
chain_schemaderives the build's schema, removes its time column if it has one, inserts a computedwindow_end(the input time column's type) at 0 as the time index, and sets the unique key to (window_end, plain columns). A merge over a window selection derivessummary_agg_schema(chain input, summary type, reduction).producer_satisfies_physical_requirements:Key code interfaces
Bodies are elided as
;.crates/types/src/stages/candidates.rscrates/types/src/ir/operator/asap.rs,tumbling_window_chain.rscrates/types/src/workload/mod.rscrates/types/src/stages/physical/memo_candidates.rs,crates/types/src/physical/properties.rscrates/accuracy/src/sizing.rs,crates/accuracy/src/test_support.rsUsage
No workload reaches these functions end to end yet: Stage 0 (PR 2), Pass 1 (PR 3), Pass 2 (PR 4) and Stage 2 (PR 5) are still
todo!(). The flow below is the one those PRs run for the doc's section 5 example (Q_A =quantile_over_time(0.99, latency[1m])every 10 s at phase 0, declared withFixedIntervalAt), using the calls this PR implements; the unit tests named in "Verification" check each step at the linked commit.The planner's text output does not change in this PR: planning still stops at Stage 0's
todo!().Major algorithm changes
None in planning results: no stage that would reach these checks runs yet. Schema derivation changes for chains (time-indexed output, merges over a
TimeRangeover a chain), replacing the count-based chain PR 0 deleted.Before this PR
Every section 2 function was
todo!()(PR 0), as were finalizing and rolling up merged windows. A chain had the plain build's schema (no time index) andTimeRange/TimeShiftrejected state.After this PR
The section 2 checks run, each covered by a table test. θ comes from a fixed list of structures per algorithm.
delivers_onanswers for every execution context exceptMaterialized(v)with ℓ = AcrossEvaluations (PR 7). A chain is a time-indexed state; a merge over aTimeRange(andTimeShifts) over a chain has the schema of the build over that range.Verification
Tests added or changed:
stages::candidates::tests:logical_properties_new_over_every_operator_and_parameters(5 operators: KLL build,Sketch(HydraCms)build, exact Sum, KLL chain, readout × 5 θ: none, KLL 269, CMS, Hydra CMS, unlisted KLL 270);summary_type_of_operator_reads_the_builds_summary_type;parameter_list_holds_its_algorithm_without_repeats(one type per list, empty exactly for algorithms without an error model, Count-Min depths 5, 7, 10, 12, Count Sketch 47, 81, 117, 155, sizes 28 / 280 / 40).stages::physical::memo_candidates::tests:delivers_on_by_lifetime_without_windows(None in every context; OneEvaluation only inEvaluation);delivers_on_a_chain_when_its_width_divides_the_context_times(grids 10 s @ 0, @ 5 s, @ 2 s, 7 s @ 0; refresh windows 5 s @ 0, 10 s @ 5 s, 10 s @ 2.5 s; widths 5 s and 10 s):delivers_on_a_stored_result_on_its_group_grid(new 2026-11-09: X = the scan's last 5 minutes, grid 60 s @ 0, against 60 s @ 0, 120 s @ 0, 120 s @ 60 s, 30 s @ 0, 60 s @ 15 s, refresh 60 s @ 0, and no grid); both fixtures build the memo withfind_or_add_group, so they panic at PR 2'stodo!()here;producer_satisfies_physical_requirements_over_every_valid_timing_and_lifetime.physical::properties::tests::new_over_every_timing_and_lifetime.types/tests/tumbling_window_chain.rs:a_chain_is_a_state_with_a_time_index,an_instant_time_range_over_a_chain_is_rejected,time_range_and_time_shift_select_a_chains_windows,merged_windows_are_the_build_over_the_selected_range,a_merge_over_a_chain_reads_one_time_range.workload::tests::range_evaluation_is_optional_carried_and_validated:FixedIntervalwithout a phase has no grid;FixedIntervalAtkeeps its grids.asap-accuracy:parameter_list_is_the_sizing_over_the_error_grid: every list is today's sizing over the ε grid at δ = 0.01, in order; Count-Min with depths 5, 7, 10, 12, Count Sketch with 47, 81, 117, 155; heap variants with every heap size; UnivMon has 28 more (the combined shapes).Checks at the head:
cargo fmt --all -- --checkandcargo clippy --workspace --all-targets --all-features -- -D warnings: clean;crate_graph: 6 passed.Test status
cargo test -p <crate> --all-features --no-fail-fastat the head (8030d3f). Every failure panics at a later PR'stodo!()(or is the stale fixture, PR 9).Compared by test name with the previous head
3c2825903(2026-11-09, second round): only the 36 tests PR 0 now deletes withpass1/composition.rs(P11) are gone (35 failed, 1 passed there); no test changed result.crate_graph: 6 passed. Earlier comparison: Compared by test name with PR 0b at0e0f75a7d: no test that passed fails. The differences are this PR's tests: 12 new tests pass; PR 0's twoPhysicalProperties::newtests (which failed at itstodo!()) are replaced bynew_over_every_timing_and_lifetime;delivers_on_a_chain_when_its_width_divides_the_context_timesand the newdelivers_on_a_stored_result_on_its_group_gridfail at PR 2'stodo!()(their memo fixture), and pass once PR 2 lands. Compared with the previous head1ca33f19c: only PR 0's 2026-11-09 test changes (8 deleted,target_limit_admits_only_floating_point_noiseadded) and the newdelivers_on_a_stored_result_on_its_group_grid.crate_graph: 6 passed.Result::unwrap()on anErrvalue: Error("invalid type: sequence, expected string or map", line: 0, column: 0): 1Self-review (2026-11-09)
delivers_onreads the two new fields directly; no helper type for "a context's times" (the(period, phase)pair is local).with_depthstakes the depth array instead of a second function for Count Sketch. The chain and stored-result branches share the context-times match.SKETCH_DEPTHSreplaced byCMS_DEPTHSandCOUNT_SKETCH_DEPTHS. The accuracy list test still compares the lists with test-only sizing (PR 4 replaces it).delivers_ondoes not shift the context's times by aTimeShifton the reading path:evaluation_grid_of_groupis the grid of the times the group itself is evaluated at, so Pass 2 must fold the queries' shifts into it when it fills it.Self-review (2026-11-08)
const fns exist because the lists areconst(no allocation,&'staticslices); they replace hand-written 28- and 280-entry arrays.strongest/weakest/dominates/combine(now deleted),sketch_algorithm_of(the algorithm is in the summary type),family_of_operator(renamed), the sizing wrappers insizing.rsandtest_support.rs. The lists duplicate today's sizing as constants by design; one test keeps them equal (PR 4 replaces that test with a per-entry guarantee test).#[cfg(test)](kll_k,kmv_k,count_sketch_depth, UnivMon'sdistinct_heapand its constants).QueryRecurrence::evaluation_grid_mshas no caller in this PR; it is the doc's interface for Pass 2's chain widths and Stage 3's grids (PRs 4, 7).History. 2026-11-09 (second): plain rebase onto PR 0b
058ea4234(PR 02ed6ec0b8, P11: Pass 1's whole-workload composition and its 74 tests deleted), no conflict; previous head kept asbackup/1207-properties-3c2825903; fmt and clippy-D warningsclean. 2026-11-09: rebased onto PR 0b0e0f75a7d(PR 0be6ea1d6a, the 2026-11-09 interfaces) and applied P1–P3 in the same single commit. Previous head kept asbackup/1207-properties-1ca33f19c. 2026-11-08 (second round): squashed the previous four commits (9c2cfa05e…cf93b3707) into one and rebased onto PR 0bb768e8dd1(PR 07708f3668, the 2026-11-08 interfaces), adapting to them as listed above. Previous head kept asbackup/1207-properties-cf93b3707.