Repository navigation
Replace asap-precompute-rs with the Rust OTAP engine collector on asap_sketchlib main - #615
Merged
Merged
Conversation
The Go CollectorPlan decoder test read its fixture from the Rust crate's tests/fixtures directory. The Rust crate is being replaced by the OTAP engine collector, which has no CollectorPlan, so keep the fixture next to the Go test that uses it.
Copy asap-precompute-rs/ from ProjectASAP/ASAPCollector-public at commit cfd3dde (Show every demo in one Grafana on port 3000). The copy is a Rust collector on the otel-arrow OTAP dataflow engine: the urn:asap:processor:asap_sketches processor, the asap-collector binary, tumbling-window merge, answer and baseline nodes, benchmarks and the plugins/asap_sketches sample config. It emits full-state ASAPv1 envelopes from asap_sketchlib. The old mirror's CollectorPlan, control channel, frame identity, snapshot cache, adapter, sampling and OTAP plugin lifecycle modules are removed with it; the collector plan is control plane and out of scope for this collector. The two repositories stay independent.
The Rust CollectorPlan consumer left with the old asap-precompute-rs mirror, and its fixture now lives in asap-precompute-go/testdata. Update the three developer docs that named the Rust consumer or the old fixture path.
Depend on the asap_sketchlib main branch instead of the 60101c6 rev pin; Cargo.lock records a05b35c. The collector compiles against it unchanged: it reads no DDSketch sum/min/max, and it does not use Bloom, Space-Saving, CountSketch-with-L2, HLL-HIP or the capability table, whose wire and naming changes land in this range. DDSketch envelopes are now kind 0x05 0x01; update the wrapper's doc comment and note the branch tracking in the README.
The retained KLL store keeps one sketch per tumbling window of width W (the sliding slide interval). Rename the pane identifiers to match: PaneSeries to WindowSeries, pane_ms to window_width_ms, merge_pane to merge_window and retained_panes to retained_windows, plus the error messages, doc comments, test names, README and sample config comments. No config keys change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replace asap-precompute-rs with the Rust OTAP engine collector
Why
asap-precompute-rs/in this repository is a stale Rust mirror of the Goprecompute runtime: a CollectorPlan decoder, a control channel, frame identity
for full and delta frames, a snapshot cache and an OTAP plugin lifecycle that no
running collector uses. The Rust collector that actually runs lives in
ProjectASAP/ASAPCollector-public: an otel-arrow OTAP dataflow engine with ASAP
nodes linked in. The SDK and the sink are moving to Rust as well, so this
repository should carry the working Rust collector instead of the mirror.
What
asap-precompute-rs/is replaced by a copy of ASAPCollector-publicasap-precompute-rs/at commitcfd3dde. The two repositories stayindependent; the source commit is recorded in the crate README and in the
commit message.
asap_sketchlibmain(branch, no rev pin;Cargo.lockrecords
a05b35c) instead of the60101c6pin.asap-precompute-go/testdata/, so the GoCollectorPlan test no longer reads files from the Rust crate. Three developer
docs now name the Go consumer and the new fixture path.
asapedgeprocessor,asap-precompute-go) is unchanged.Removed with the old mirror (control plane or replaced by the engine nodes):
src/collector_plan.rs,tests/collector_plan.rs,tests/fixtures/collector_plan_v1.jsonsrc/control_channel.rssrc/frame_identity.rs,src/snapshot_cache.rssrc/adapter.rs,src/otap/plugin.rs,src/otap/lifecycle.rslocal::Processor<OtapPdata>node insrc/otap/processor.rssrc/otap/decode.rs,src/otap/encode.rs,src/otap/records.rssrc/otap/codec.rs(MetricsViewover envelopes,decode_each)src/sampling.rstests/api_surface.rs,tests/otap_codec.rs,tests/otap_lifecycle.rstests/otap_pipeline_e2e.rs,tests/kll_windows.rs,tests/runtime.rsand unit testsHow
asap-precompute-go(Go test staysself-contained).
cfd3dde.asap_sketchlibmain. The collector compiled unchanged againsta05b35c:sketchlib dropped them (DDSketch is now kind
0x05 0x01);capability table, whose wire and naming changes are in this range;
0x01 0x02); CMS, CMS-with-heap, CountSketch andKLL kinds are unchanged.
Only the DDSketch wrapper's kind comment and the README dependency note
change.
kll_windowsand its callers, tests,README and sample config comments. No config keys change.
Key code interfaces
asap-collector --config <pipeline.yaml|json>: the upstreamotel_dataflow/v1controller with these ASAP nodes registered throughlinkme:urn:asap:processor:asap_sketches(src/otap/processor.rs)urn:asap:processor:window_mergeurn:asap:processor:sketch_detectorurn:asap:processor:heavy_hitter_accuracyurn:asap:processor:{exact_branch,exact_merge,exact_detector,raw_sink}urn:asap:receiver:scenario_generatorotap::codec::decode_each(pdata, |&Observation| ..)andencode_envelopes_to_pdata(&[SketchEnvelope]): OTAP metrics to and fromobservations and envelopes.
precompute::Precompute(observe,observe_envelope,tick,drain,update_config) andSketchObserver.kll_windows::KllWindowStore:new(config, window_width, retention),merge_window,query,quantile_over_time,retained_windows; renamedfrom
merge_pane/retained_panes.Major algorithm changes
Compared with the old mirror:
no frame identity and no snapshot cache.
sketch.envelopeismerged as a sketch. A Gauge or Sum point is one scalar observation. Histogram
and ExponentialHistogram points are counted and skipped.
them (sliding output,
window_slide).Before
asap-precompute-rsmirror had a CollectorPlan, a control channel,delta frames and an OTAP plugin lifecycle that no deployed collector runs.
asap_sketchlib60101c6.After
asap-precompute-rsis the running Rust OTAP collector fromASAPCollector-public at
cfd3dde, onasap_sketchlibmaina05b35c.Not changed here:
main(071e9d26) does not depend on this crate.deploy/mvp-multinode/scripts/run_demo.shstill passesasap-precompute-rsas a build context to backend Dockerfiles that no longer exist on backendmain; that script is already stale for the same reason and is left as is.plugins/asap_sketches/sample.tomluses an older layout; the working pipeline YAMLs live in ASAPCollector-publicdeploy/telemetry-functions/and come with the end-to-end setup.Verification
In
asap-precompute-rs/:cargo fmt --check: cleancargo clippy --all-targets --all-features: no warningscargo test --all-features: 143 passed, 0 failed, 1 ignored (benchmark)kll_windows: 11otap_pipeline_e2e: 1runtime: 26In
asap-precompute-go/:go test ./...with sketchlib-go main through a localGOWORK: allpackages ok
Smoke run of
asap-collector:scenario_generator->asap_sketches->urn:otel:exporter:consolepipeline, with 2 s windows.
transmit_sketch: falseemits p50/p99 Gauges perhttp.route, withsketch.window_start_ms/sketch.window_end_msattributes.transmit_sketch: trueemits Summary points whosesketch.envelopestartsASAPv1 01 02 05 01, the new DDSketch kind.