Skip to content

feat(ingestors): add Zeek Modbus/DNP3 deep packet inspection integration - #4

Open
adamsrnmsu wants to merge 1 commit into
sandialabs:mainfrom
adamsrnmsu:feat-zeek-ics-dpi
Open

adamsrnmsu wants to merge 1 commit into
sandialabs:mainfrom
adamsrnmsu:feat-zeek-ics-dpi

Conversation

@adamsrnmsu

@adamsrnmsu adamsrnmsu commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Adds an opt-in zeek integration that pulls Modbus register values and DNP3 point values out of raw traffic (deep packet inspection), without needing Malcolm. Enable with manage-parseus install --integrations zeek. This supersedes #2 (branch renamed to the type-description convention); the hand-written CHANGELOG.md entry is dropped because the changelog is now generated by releasaurus (#3).

Design:

  • Stock Zeek 9.0.1 only (zeek/zeek:9.0.1), no ICSNPP and no zkg, so the image builds offline.
  • deploy/integrations/zeek/zeek/parseus.zeek adds two JSON logs from stock analyzer events: modbus_detailed.log (request/response with register values) and dnp3_points.log (one row per point: group, variation, index, value, flags).
  • parseus.ingestors.zeek normalizes rows to the Malcolm document shape Parseus already queries (zeek.<log>.<field>, event.dataset), with stable document ids so re-reads are idempotent, and tails the logs with inode-keyed offsets.
  • A custom Spicy analyzer (deploy/integrations/zeek/spicy/) runs next to the stock DNP3 analyzer and logs what stock Zeek cannot: g40v1-v4 (analog output status) and g2v1-v3 (binary input events). It checks every link-frame CRC, reassembles transport segments and walks object headers with a size table; it stops at an object of unknown size rather than guess, and only emits points stock does not log, so nothing is logged twice. The Dockerfile is multi-stage: a builder with g++ compiles the .hlto, and the final image carries only that file.
  • New dnp3_points dataset; stock dnp3 rows also populate zeek.dnp3_objects.function_code for the OT function-code analytic. The Registers page now reads Modbus registers and DNP3 points.

Related Issues/PRs

Type of Change

  • Bugfix (fix)
  • Feature (feat)
  • Documentation (docs)
  • Refactor (refactor)
  • Chore (CI, build, dependencies, etc.) (chore)
  • Other (please describe):

Checklist

Testing

  • .venv/bin/python -m pytest tests/test_zeek_ingestor.py: 20 passed.
  • Full suite: same 26 failures and 1 error (Elasticsearch/env) as on the base commit; 4 new tests, which failed before the fixes.
  • python3 -m pylint --fail-under=7.5 src/parseus: 8.81/10.
  • Zeek over make_test_pcap.py output via replay_pcap.sh --check: all expected Modbus and DNP3 values match, including g2v1, range qualifier 0x02 and float inf cases.
  • Cross-checked against tshark on modbusSmall.pcap and Zeek's dnp3_read.pcap; ingest into Elasticsearch 8.19.5 verified idempotent.
  • Unchanged from feat(ingestors): add Zeek Modbus/DNP3 deep packet inspection integration #2 apart from the dropped CHANGELOG.md entry (one commit, amended).
  • Spicy analyzer: replay_pcap.sh --check reports "OK: 7 modbus rows, 108 dnp3 points match" over cases for every g40/g2 variation and qualifier, a 40-point fragment split over four link frames, a bad CRC, unknown objects, a segment gap and a truncated fragment. Without the .hlto, all g40/g2 rows are missing. tshark agrees on all 59 g40/g2 points. conn.log shows dnp3_tcp,spicy_parseusdnp3 on the same connection. tests/test_zeek_ingestor.py: 22 passed.
  • Review fix: g80v1 (IIN bits) is sized as packed bits; a g40 point after an IIN object was lost before (new pcap case failed first).
  • Follow-up folded in: host-side ZEEK_IFACE check and the function_code keyword mapping, each with a test that failed first; full suite unchanged apart from the new tests.

Additional Notes

Known limits and follow-ups:

  • g40 and g2 come from the Spicy analyzer, since stock Zeek raises no value event for them. It does not walk request objects, and a truncated fragment keeps only its complete objects. Inf/NaN floats and doubles are dropped.
  • The image build needs the Debian mirrors once, for g++ in the builder stage (or a proxy, or a prebuilt image); zeek/zeek ships spicyz without a C++ compiler.
  • Double-precision analogs (g30v6, g31v8, g32v6/v8, g33v6/v8) ARE decoded, from the two 32-bit halves Zeek's DPwFlag/DPwTime/DPwoTime events carry. This was added after testing on a live range (ferrofire/bennu outstations send analog inputs as g30v6); the ground-truth pcap now covers g30v6 and g32v6.
  • ZEEK_IFACE defaults to eth0. manage-parseus checks it against the host's interfaces before bringing up zeek (the service uses host networking), and the Zeek container also exits with a message listing interfaces. On Docker Desktop for macOS the check sees the Mac's interfaces, not the Docker VM's, so it assumes a Linux host.
  • zeek.dnp3_objects.function_code is mapped as keyword in the Zeek index template so the OT function-code analytic can aggregate DNP3 requests. Indices created before this change keep a dynamic text mapping until rollover or reindex.
  • Rotated logs are pruned by the Zeek container after a day.
  • Once ci(release): adopt releasaurus with CalVer and the phenix-apps contribution workflow #3 is merged this branch should be rebased on main; the codespell check covers the whole repo and ci(release): adopt releasaurus with CalVer and the phenix-apps contribution workflow #3 fixes a typo (incase) in a file this PR touches.

🤖 Generated with Claude Code

Parseus reads Modbus register values from Malcolm-normalized documents but had
no way to produce them without Malcolm, and had no DNP3 point values at all.
This adds a `zeek` integration that extracts both from raw traffic.

- deploy/integrations/zeek: Dockerfile (zeek/zeek:9.0.1, no ICSNPP or zkg, so it
  builds offline) and parseus.zeek, which adds modbus_detailed.log and
  dnp3_points.log from stock analyzer events; JSON logs, hourly rotation.
  Includes README, verify_zeek_runtime.sh, replay_pcap.sh and make_test_pcap.py,
  a stdlib pcap generator with known values and a --check mode.
- parseus.ingestors.zeek: pure normalize() to the Malcolm document layout, plus a
  runner that tails the logs with inode-keyed offsets and stable document ids
  (restart-safe, idempotent) into arkime_sessions3-parseus-zeek-%y%m%d.
- query_api.get_dnp3_points and reg_ops.dnp3_to_regop mirror the Modbus helpers.
- Compose services/profile, build and image lists (offline bundles), integration
  map, CLI help, env defaults, and integrations README.

Known gaps: stock Zeek raises no event for DNP3 g40 (analog output status) or
the g2 binary-event flag byte; a custom Spicy analyzer (below) covers them.
Inf/NaN float analog values are dropped.

Verified: new unit tests on real Zeek output fixtures; Zeek over a generated
pcap matches every expected Modbus/DNP3 value, and agrees with tshark on
modbusSmall.pcap and dnp3_read.pcap point counts and values; ingest into
Elasticsearch 8.19.5 is idempotent and get_modbus_registers / get_dnp3_points
return the values; pylint 8.81/10.

Review fixes folded in: g2v1 no longer logs invented value 1.0/flags 255; start/stop
range qualifier 0x02 keeps its real start index; Inf/NaN float analogs are skipped;
normalize() maps stock dnp3 fc_request to zeek.dnp3_objects.function_code so the OT
function-code analytic sees DNP3; the Registers page now also reads DNP3 points;
the ingestor caps reads at 8 MiB per file per poll, skips fully-read rotated files
and drops pruned state keys; the Zeek container prunes rotated logs after a day and
exits with a clear message listing interfaces when ZEEK_IFACE does not exist.
make_test_pcap.py gained g2v1, q=0x02 and g30v5 inf cases (failed first, now match).
pytest: 4 new tests failed first; full suite has the same 26 elasticsearch/env
failures as before the change.

DNP3 double-precision analogs (g30v6, g31v8, g32v6/v8, g33v6/v8) are
decoded from the two 32-bit halves Zeek's DPwFlag/DPwTime/DPwoTime events
carry; inf/NaN are skipped like float32. Outstations such as bennu and
ferrofire send analog inputs as g30v6, so without this the live waterway
range only yielded its g30v1 points. Ground truth: make_test_pcap.py now
has g30v6/g32v6 cases; the previous script misses exactly those 5 rows,
the fixed one matches all 31 DNP3 points.

manage-parseus now checks ZEEK_IFACE on the host before bringing up the
zeek service (only when the zeek integration is enabled; the service uses
host networking) and fails with a message listing the host's interfaces.
The Zeek index template maps zeek.dnp3_objects.function_code as keyword,
so the function-code analytic's terms aggregation works instead of hitting
a dynamically mapped text field. Both have tests that failed first.

DNP3 g40v1-v4 (analog output status) and g2v1-v3 (binary input events) are
logged by a custom Spicy analyzer (deploy/integrations/zeek/spicy/) attached
to TCP/20000 next to the stock DNP3 analyzer. It parses link frames (every
CRC-16 checked, bad frames dropped), transport FIR/FIN/sequence reassembly,
the application header, then walks object headers with a size table so it
can skip objects it does not log; an object of unknown size ends the
fragment, it never guesses. It raises points only for objects stock does
not log, so nothing is logged twice; parseus.zeek writes them into
dnp3_points with the same columns (g2 value is the flag's state bit). The
zeek/zeek image has spicyz but no C++ compiler, so the Dockerfile is
multi-stage: a builder installs g++ and compiles the .hlto, and only the
.hlto is copied into the final image. The build needs the Debian mirrors
once. make_test_pcap.py has ground truth for g40v1-v4 (all qualifiers),
g2v1-v3, a multi-object response, a 40-point fragment split over four link
frames, and a second connection with a bad CRC, unknown objects, a segment
gap and a truncated fragment. replay --check: "OK: 7 modbus rows, 108 dnp3
points match"; without the .hlto all g40/g2 rows go missing; tshark agrees
on all 59 g40/g2 points; conn.log service is dnp3_tcp,spicy_parseusdnp3.
Review fix: g80v1 (IIN) is sized as packed bits like g1v1, so a g40 point
after an IIN object is reached (failed before the fix).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant