Skip to content

feat(ingestors): add Zeek Modbus/DNP3 deep packet inspection integration - #2

Closed
adamsrnmsu wants to merge 1 commit into
sandialabs:mainfrom
adamsrnmsu:feat/zeek-ics-dpi
Closed

adamsrnmsu wants to merge 1 commit into
sandialabs:mainfrom
adamsrnmsu:feat/zeek-ics-dpi

Conversation

@adamsrnmsu

Copy link
Copy Markdown
Collaborator

Summary

Adds an opt-in zeek integration that pulls Modbus register values and DNP3 point values out of raw traffic (deep packet inspection), without needing Malcolm. Enable with manage-parseus install --integrations zeek.

Design

  • Stock Zeek 9.0.1 only (zeek/zeek:9.0.1), no ICSNPP and no zkg, so the image builds offline.
  • deploy/integrations/zeek/zeek/parseus.zeek adds two JSON logs from stock analyzer events: modbus_detailed.log (request/response with register values) and dnp3_points.log (one row per point: group, variation, index, value, flags).
  • parseus.ingestors.zeek normalizes rows to the Malcolm document shape Parseus already queries (zeek.<log>.<field>, event.dataset), with stable document ids so re-reads are idempotent, and tails the logs with inode-keyed offsets.
  • New dnp3_points dataset; stock dnp3 rows also populate zeek.dnp3_objects.function_code for the OT function-code analytic. The Registers page now reads Modbus registers and DNP3 points.

Verification

  • .venv/bin/python -m pytest tests/test_zeek_ingestor.py: 20 passed.
  • Full suite: same 26 failures and 1 error (Elasticsearch/env) as on the base commit; 4 new tests, which failed before the fixes.
  • python3 -m pylint --fail-under=7.5 src/parseus: 8.81/10.
  • Zeek over make_test_pcap.py output via replay_pcap.sh --check: all expected Modbus and DNP3 values match, including g2v1, range qualifier 0x02 and float inf cases.
  • Cross-checked against tshark on modbusSmall.pcap and Zeek's dnp3_read.pcap; ingest into Elasticsearch 8.19.5 verified idempotent.

Known limits and follow-ups

  • Stock Zeek raises no event for g40 (analog output status), double-precision objects, or the g2v1 flag byte. g2v1 is not logged rather than logged wrong; Inf/NaN floats are dropped.
  • ZEEK_IFACE defaults to eth0; the Zeek container now exits with a message listing interfaces if it does not exist, but manage-parseus does not validate it on the host yet.
  • Rotated logs are pruned by the Zeek container after a day.

🤖 Generated with Claude Code

Parseus reads Modbus register values from Malcolm-normalized documents but had
no way to produce them without Malcolm, and had no DNP3 point values at all.
This adds a `zeek` integration that extracts both from raw traffic.

- deploy/integrations/zeek: Dockerfile (zeek/zeek:9.0.1, no ICSNPP or zkg, so it
  builds offline) and parseus.zeek, which adds modbus_detailed.log and
  dnp3_points.log from stock analyzer events; JSON logs, hourly rotation.
  Includes README, verify_zeek_runtime.sh, replay_pcap.sh and make_test_pcap.py,
  a stdlib pcap generator with known values and a --check mode.
- parseus.ingestors.zeek: pure normalize() to the Malcolm document layout, plus a
  runner that tails the logs with inode-keyed offsets and stable document ids
  (restart-safe, idempotent) into arkime_sessions3-parseus-zeek-%y%m%d.
- query_api.get_dnp3_points and reg_ops.dnp3_to_regop mirror the Modbus helpers.
- Compose services/profile, build and image lists (offline bundles), integration
  map, CLI help, env defaults, integrations README and CHANGELOG.

Known gaps: stock Zeek raises no event for DNP3 g40 (analog output status),
double-precision objects, or the g2v1 binary-event flag byte, so those are not
extracted (g2v1 is deliberately not logged rather than logged wrong). Inf/NaN
float analog values are dropped.

Verified: new unit tests on real Zeek output fixtures; Zeek over a generated
pcap matches every expected Modbus/DNP3 value, and agrees with tshark on
modbusSmall.pcap and dnp3_read.pcap point counts and values; ingest into
Elasticsearch 8.19.5 is idempotent and get_modbus_registers / get_dnp3_points
return the values; pylint 8.81/10.

Review fixes folded in: g2v1 no longer logs invented value 1.0/flags 255; start/stop
range qualifier 0x02 keeps its real start index; Inf/NaN float analogs are skipped;
normalize() maps stock dnp3 fc_request to zeek.dnp3_objects.function_code so the OT
function-code analytic sees DNP3; the Registers page now also reads DNP3 points;
the ingestor caps reads at 8 MiB per file per poll, skips fully-read rotated files
and drops pruned state keys; the Zeek container prunes rotated logs after a day and
exits with a clear message listing interfaces when ZEEK_IFACE does not exist.
make_test_pcap.py gained g2v1, q=0x02 and g30v5 inf cases (failed first, now match).
pytest: 4 new tests failed first; full suite has the same 26 elasticsearch/env
failures as before the change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@adamsrnmsu

Copy link
Copy Markdown
Collaborator Author

Superseded by #4: branch renamed to the type-description convention and conventions aligned with sceptre-phenix-apps.

@adamsrnmsu adamsrnmsu closed this Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant