Skip to content

Replace asap-precompute-rs with the Rust OTAP engine collector on asap_sketchlib main - #615

Merged
zzylol merged 5 commits into
mainfrom
feat/rust-collector
Nov 9, 2026
Merged

zzylol merged 5 commits into
mainfrom
feat/rust-collector

Conversation

@zzylol

@zzylol zzylol commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Replace asap-precompute-rs with the Rust OTAP engine collector

Why

asap-precompute-rs/ in this repository is a stale Rust mirror of the Go
precompute runtime: a CollectorPlan decoder, a control channel, frame identity
for full and delta frames, a snapshot cache and an OTAP plugin lifecycle that no
running collector uses. The Rust collector that actually runs lives in
ProjectASAP/ASAPCollector-public: an otel-arrow OTAP dataflow engine with ASAP
nodes linked in. The SDK and the sink are moving to Rust as well, so this
repository should carry the working Rust collector instead of the mirror.

What

  • asap-precompute-rs/ is replaced by a copy of ASAPCollector-public
    asap-precompute-rs/ at commit cfd3dde. The two repositories stay
    independent; the source commit is recorded in the crate README and in the
    commit message.
  • The crate depends on asap_sketchlib main (branch, no rev pin; Cargo.lock
    records a05b35c) instead of the 60101c6 pin.
  • KLL retention code says "tumbling window" (width W) instead of "pane".
  • The CollectorPlan fixture moves to asap-precompute-go/testdata/, so the Go
    CollectorPlan test no longer reads files from the Rust crate. Three developer
    docs now name the Go consumer and the new fixture path.
  • The Go processor (asapedgeprocessor, asap-precompute-go) is unchanged.

Removed with the old mirror (control plane or replaced by the engine nodes):

Removed Replaced by / reason
src/collector_plan.rs, tests/collector_plan.rs, tests/fixtures/collector_plan_v1.json CollectorPlan is control plane; the Go decoder and its moved fixture remain
src/control_channel.rs control plane, not in this collector
src/frame_identity.rs, src/snapshot_cache.rs the collector emits full-state envelopes only, so there are no delta frames to identify or rebase
src/adapter.rs, src/otap/plugin.rs, src/otap/lifecycle.rs the local::Processor<OtapPdata> node in src/otap/processor.rs
src/otap/decode.rs, src/otap/encode.rs, src/otap/records.rs src/otap/codec.rs (MetricsView over envelopes, decode_each)
src/sampling.rs no sampling in this collector yet
tests/api_surface.rs, tests/otap_codec.rs, tests/otap_lifecycle.rs tests/otap_pipeline_e2e.rs, tests/kll_windows.rs, tests/runtime.rs and unit tests

How

  1. Move the CollectorPlan fixture into asap-precompute-go (Go test stays
    self-contained).
  2. Replace the crate directory with the public repository's copy at cfd3dde.
  3. Update the docs that pointed at the Rust CollectorPlan consumer or fixture.
  4. Track asap_sketchlib main. The collector compiled unchanged against
    a05b35c:
    • it never read DDSketch sum/min/max, so nothing had to be removed when
      sketchlib dropped them (DDSketch is now kind 0x05 0x01);
    • it does not use Bloom, Space-Saving, CountSketch-with-L2, HLL-HIP or the
      capability table, whose wire and naming changes are in this range;
    • HLL stays Ertl-MLE (kind 0x01 0x02); CMS, CMS-with-heap, CountSketch and
      KLL kinds are unchanged.
      Only the DDSketch wrapper's kind comment and the README dependency note
      change.
  5. Rename pane to tumbling window in kll_windows and its callers, tests,
    README and sample config comments. No config keys change.

Key code interfaces

  • asap-collector --config <pipeline.yaml|json>: the upstream
    otel_dataflow/v1 controller with these ASAP nodes registered through
    linkme:
    • urn:asap:processor:asap_sketches (src/otap/processor.rs)
    • urn:asap:processor:window_merge
    • urn:asap:processor:sketch_detector
    • urn:asap:processor:heavy_hitter_accuracy
    • urn:asap:processor:{exact_branch,exact_merge,exact_detector,raw_sink}
    • urn:asap:receiver:scenario_generator
  • otap::codec::decode_each(pdata, |&Observation| ..) and
    encode_envelopes_to_pdata(&[SketchEnvelope]): OTAP metrics to and from
    observations and envelopes.
  • precompute::Precompute (observe, observe_envelope, tick, drain,
    update_config) and SketchObserver.
  • kll_windows::KllWindowStore: new(config, window_width, retention),
    merge_window, query, quantile_over_time, retained_windows; renamed
    from merge_pane / retained_panes.

Major algorithm changes

Compared with the old mirror:

  • Output is always full ASAPv1 state per tumbling window. There are no deltas,
    no frame identity and no snapshot cache.
  • Ingest is content-routed. A Summary point that carries sketch.envelope is
    merged as a sketch. A Gauge or Sum point is one scalar observation. Histogram
    and ExponentialHistogram points are counted and skipped.
  • A tumbling window closes as soon as a point's event time passes its end.
  • KLL can retain W-wide tumbling windows and answer aligned ranges by merging
    them (sliding output, window_slide).

Before

  • The asap-precompute-rs mirror had a CollectorPlan, a control channel,
    delta frames and an OTAP plugin lifecycle that no deployed collector runs.
  • It pinned asap_sketchlib 60101c6.
  • The Go CollectorPlan test read its fixture from the Rust crate.

After

  • asap-precompute-rs is the running Rust OTAP collector from
    ASAPCollector-public at cfd3dde, on asap_sketchlib main a05b35c.
  • The Go CollectorPlan test is self-contained.

Not changed here:

  • ASAPQuery-backend main (071e9d26) does not depend on this crate. deploy/mvp-multinode/scripts/run_demo.sh still passes asap-precompute-rs as a build context to backend Dockerfiles that no longer exist on backend main; that script is already stale for the same reason and is left as is.
  • No runnable pipeline file is included yet. plugins/asap_sketches/sample.toml uses an older layout; the working pipeline YAMLs live in ASAPCollector-public deploy/telemetry-functions/ and come with the end-to-end setup.

Verification

In asap-precompute-rs/:

  • cargo fmt --check: clean
  • cargo clippy --all-targets --all-features: no warnings
  • cargo test --all-features: 143 passed, 0 failed, 1 ignored (benchmark)
    • lib: 99
    • stream bench: 6
    • kll_windows: 11
    • otap_pipeline_e2e: 1
    • runtime: 26

In asap-precompute-go/:

  • go test ./... with sketchlib-go main through a local GOWORK: all
    packages ok

Smoke run of asap-collector:

  • Setup: a scenario_generator -> asap_sketches -> urn:otel:exporter:console
    pipeline, with 2 s windows.
  • KLL with transmit_sketch: false emits p50/p99 Gauges per http.route, with
    sketch.window_start_ms / sketch.window_end_ms attributes.
  • DDSketch with transmit_sketch: true emits Summary points whose
    sketch.envelope starts ASAPv1 01 02 05 01, the new DDSketch kind.

The Go CollectorPlan decoder test read its fixture from the Rust crate's
tests/fixtures directory. The Rust crate is being replaced by the OTAP
engine collector, which has no CollectorPlan, so keep the fixture next
to the Go test that uses it.
Copy asap-precompute-rs/ from ProjectASAP/ASAPCollector-public at
commit cfd3dde (Show every demo in one Grafana on port 3000). The copy
is a Rust collector on the otel-arrow OTAP dataflow engine: the
urn:asap:processor:asap_sketches processor, the asap-collector binary,
tumbling-window merge, answer and baseline nodes, benchmarks and the
plugins/asap_sketches sample config. It emits full-state ASAPv1
envelopes from asap_sketchlib.

The old mirror's CollectorPlan, control channel, frame identity,
snapshot cache, adapter, sampling and OTAP plugin lifecycle modules are
removed with it; the collector plan is control plane and out of scope
for this collector. The two repositories stay independent.
The Rust CollectorPlan consumer left with the old asap-precompute-rs
mirror, and its fixture now lives in asap-precompute-go/testdata. Update
the three developer docs that named the Rust consumer or the old fixture
path.
Depend on the asap_sketchlib main branch instead of the 60101c6 rev pin;
Cargo.lock records a05b35c. The collector compiles against it unchanged:
it reads no DDSketch sum/min/max, and it does not use Bloom, Space-Saving,
CountSketch-with-L2, HLL-HIP or the capability table, whose wire and
naming changes land in this range. DDSketch envelopes are now kind
0x05 0x01; update the wrapper's doc comment and note the branch tracking
in the README.
The retained KLL store keeps one sketch per tumbling window of width W
(the sliding slide interval). Rename the pane identifiers to match:
PaneSeries to WindowSeries, pane_ms to window_width_ms, merge_pane to
merge_window and retained_panes to retained_windows, plus the error
messages, doc comments, test names, README and sample config comments.
No config keys change.
@zzylol zzylol added the enhancement New feature or request label Nov 9, 2026
@zzylol
zzylol marked this pull request as ready for review November 9, 2026 04:38
@zzylol
zzylol merged commit fd53ec8 into main Nov 9, 2026
1 check passed
@zzylol
zzylol deleted the feat/rust-collector branch November 9, 2026 04:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant