Skip to content

About

Runs a Vector pipeline as a supervised transform stage in a DFE deployment.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

270 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

dfe-transform-vector

Runs a Vector pipeline as a supervised transform stage in a DFE deployment.

Build Status License

What it does

  • Vector does the transform, and the supervisor supplies the rest a stage needs: config, credentials, health, metrics and restarts. Reach for it when you need one of Vector's transforms beyond remap. For VRL alone, dfe-transform-vrl runs it in-process with no Vector binary.
  • One small YAML (pipeline name, brokers, topics, SASL and TLS) generates Vector's source, sink and observability config, so you write only the transforms: and enrichment_tables: blocks.
  • Kafka (bus) or a direct gRPC push with no broker (direct), chosen for each end. The transform you author is the same on both.
  • A crashed Vector is restarted with exponential backoff, and readiness follows it.
  • With reload enabled, a transform change is validated, sent to Vector with a SIGHUP, and read back from Vector's reload counters as applied, refused (the running config is put back) or unconfirmed. One that fails validation is logged and never sent.
  • Vector's vector_* metrics are merged into the supervisor's, so one endpoint and one OTLP push serve both.

The config engine and the process manager run Vector inside the pod, the bridge moves records on the direct transport, and the metrics merge serves Vector's metrics beside the supervisor's.

flowchart TB
    accTitle: L3 Components -- dfe-transform-vector
    accDescr: The previous stage sends records into the pod and the config and transform files configure it. Inside, the config engine writes the Vector config that the process manager starts Vector on, and on the direct transport the bridge intake hands records to Vector and the bridge return takes them back. Vector's metrics are merged into the supervisor's and served on the ops port. Records leave for the next stage and metrics go to the collector.

    prev["Previous stage"]
    files[("Config files")]

    subgraph pod["dfe-transform-vector"]
        direction LR
        cfg[["Config engine"]]
        proc["Process manager"]
        intake[["Bridge intake"]]
        vec[["Vector"]]
        ret[["Bridge return"]]
        merge["Metrics merge"]
        cfg --> proc
        proc --> vec
        intake --> vec
        vec --> ret
        vec --> merge
    end

    next["Next stage"]
    collector["Metrics collector"]

    prev -->|records| pod
    files -->|config| pod
    pod -->|records| next
    pod -->|metrics| collector

    classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A
    classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327
    classDef ext fill:#EEF0F2,stroke:#6B7280,stroke-width:1px,stroke-dasharray:4 3,color:#1F2937
    classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4
    class prev,next,collector ext
    class files store
    class pod group
Loading

Figure 1. L3 Components -- dfe-transform-vector. Dashed boxes are external. Up: top of dfe-transform-vector. Down: L4 Detail -- Config assembly, L4 Detail -- Direct transport bridge, L4 Detail -- Vector lifecycle.

On bus Vector reads and writes Kafka itself and the bridge does nothing. On direct the bridge takes records from the previous stage's Push client and hands Vector's output to the next stage's Push listener.

Quick start

cargo build --release --bin dfe-transform-vector
scripts/fetch-vector.sh        # caches the pinned Vector under .tmp/ and prints its path
./target/release/dfe-transform-vector --config config.example.yaml config-check

--config goes before the subcommand, and run is the default when you leave the subcommand out. To run it off the image, point vector.binary at the path scripts/fetch-vector.sh printed and vector.config_dir and vector.data_dir at directories you can write, then ./target/release/dfe-transform-vector --config config.example.yaml.

config.example.yaml is the annotated reference. templates/bus.yaml and templates/direct.yaml are the whole Vector topology for each transport, commented per field and runnable as they stand: copy one, replace the dfe_transform step, and you have a working pipeline. In a deployment the supervisor generates sources: and sinks:, and rejects any other top-level key in a transform file.

bus direct
In Kafka <source>_land a scalo Push listener on source.listen (port 6000 by default)
Out Kafka <source>_load a scalo Push client to sink.endpoint
Needs a broker yes no

One port, 9090 by default, serves the ops surface: /livez, /readyz and /metrics. /readyz answers 503 until Vector has survived its first 500 ms, and again when it is down or its sink has held or refused records for metrics.sink_stall_secs without delivering any. A bus config with no topics idles instead: Ready, with no Vector running, until the config file names a topic. Vector's exporter defaults to 127.0.0.1:9598. Keep metrics.vector_metrics_address on loopback, because nothing checks it.

Configuration

The config file (--config, else config.yaml or config.yml in the working directory) sits over the defaults, and DFE_TRANSFORM_VECTOR_* environment variables sit over the file. The working directory's .env is read into the environment first, and a variable set before it wins. The flat names take a single underscore and a comma list (DFE_TRANSFORM_VECTOR_SOURCE_BROKERS=a:9092,b:9092, _SINK_TOPIC, _TRANSFORMS_DIR, the SASL pairs) and beat the double-underscore form. That form reaches any key and takes a list in brackets (DFE_TRANSFORM_VECTOR_SOURCE__BROKERS='[a:9092,b:9092]'), and a bare comma list there fails to load. --log-level, --log-format and --metrics-addr, or LOG_LEVEL, LOG_FORMAT and METRICS_ADDR, outrank logging.* and metrics.address. docs/config-schema.yaml lists the app's keys with their types and defaults.

With reload on, the default, a transform file change applies once it validates, with no restart. Any other change to the app's keys, dfe_source, pipeline.name and scaling.pressure_threshold included, is logged as needing a pod restart, and no later transform change applies until the pod restarts or the edit is reverted. scalo's sections in the file, such as version_check, scaling.enabled and metrics.namespace, are read once at startup, so an edit there needs a restart, and the watcher logs a successful reload without applying it. In a chart deployment a config or file set change rolls the pods anyway, because the chart hashes both into the pod template. The reload path, the readiness rules, the metrics merge and the Kafka settings are in docs/internals.md and docs/librdkafka.md.

Documentation

  • docs/architecture.md: the problem, the ownership boundary and the invariants.
  • docs/internals.md: the config engine, hot reload, the Vector subprocess, health and metrics, dead letters, the Vector version pin, the chart.
  • docs/migration.md: moving a pipeline off the official Vector chart.
  • docs/librdkafka.md: the librdkafka settings Vector's Kafka source and sink are given.
  • tests/TESTING.md: the test tiers, and the e2e cases that need Docker and a Vector binary.

The suite's published docs are at https://dfe.docs.hyperi.io, and this repository's pages at https://dfe.docs.hyperi.io/dfe-transform-vector/.

Help and contributing

How to work in this repo, the commands that test a change and what tends to bite are in CONTRIBUTING.md. A question that is not a bug goes in an issue.

Open bugs: this repo | whole DFE suite

License

BUSL-1.1, see LICENSE.

About

Runs a Vector pipeline as a supervised transform stage in a DFE deployment.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages