Repository navigation
feat(ingestors): add Zeek Modbus/DNP3 deep packet inspection integration - #2
Closed
adamsrnmsu wants to merge 1 commit into
Closed
adamsrnmsu wants to merge 1 commit into
adamsrnmsu wants to merge 1 commit into
Conversation
Parseus reads Modbus register values from Malcolm-normalized documents but had no way to produce them without Malcolm, and had no DNP3 point values at all. This adds a `zeek` integration that extracts both from raw traffic. - deploy/integrations/zeek: Dockerfile (zeek/zeek:9.0.1, no ICSNPP or zkg, so it builds offline) and parseus.zeek, which adds modbus_detailed.log and dnp3_points.log from stock analyzer events; JSON logs, hourly rotation. Includes README, verify_zeek_runtime.sh, replay_pcap.sh and make_test_pcap.py, a stdlib pcap generator with known values and a --check mode. - parseus.ingestors.zeek: pure normalize() to the Malcolm document layout, plus a runner that tails the logs with inode-keyed offsets and stable document ids (restart-safe, idempotent) into arkime_sessions3-parseus-zeek-%y%m%d. - query_api.get_dnp3_points and reg_ops.dnp3_to_regop mirror the Modbus helpers. - Compose services/profile, build and image lists (offline bundles), integration map, CLI help, env defaults, integrations README and CHANGELOG. Known gaps: stock Zeek raises no event for DNP3 g40 (analog output status), double-precision objects, or the g2v1 binary-event flag byte, so those are not extracted (g2v1 is deliberately not logged rather than logged wrong). Inf/NaN float analog values are dropped. Verified: new unit tests on real Zeek output fixtures; Zeek over a generated pcap matches every expected Modbus/DNP3 value, and agrees with tshark on modbusSmall.pcap and dnp3_read.pcap point counts and values; ingest into Elasticsearch 8.19.5 is idempotent and get_modbus_registers / get_dnp3_points return the values; pylint 8.81/10. Review fixes folded in: g2v1 no longer logs invented value 1.0/flags 255; start/stop range qualifier 0x02 keeps its real start index; Inf/NaN float analogs are skipped; normalize() maps stock dnp3 fc_request to zeek.dnp3_objects.function_code so the OT function-code analytic sees DNP3; the Registers page now also reads DNP3 points; the ingestor caps reads at 8 MiB per file per poll, skips fully-read rotated files and drops pruned state keys; the Zeek container prunes rotated logs after a day and exits with a clear message listing interfaces when ZEEK_IFACE does not exist. make_test_pcap.py gained g2v1, q=0x02 and g30v5 inf cases (failed first, now match). pytest: 4 new tests failed first; full suite has the same 26 elasticsearch/env failures as before the change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 12, 2026
Merged
Collaborator
Author
|
Superseded by #4: branch renamed to the type-description convention and conventions aligned with sceptre-phenix-apps. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an opt-in
zeekintegration that pulls Modbus register values and DNP3 point values out of raw traffic (deep packet inspection), without needing Malcolm. Enable withmanage-parseus install --integrations zeek.Design
zeek/zeek:9.0.1), no ICSNPP and no zkg, so the image builds offline.deploy/integrations/zeek/zeek/parseus.zeekadds two JSON logs from stock analyzer events:modbus_detailed.log(request/response with register values) anddnp3_points.log(one row per point: group, variation, index, value, flags).parseus.ingestors.zeeknormalizes rows to the Malcolm document shape Parseus already queries (zeek.<log>.<field>,event.dataset), with stable document ids so re-reads are idempotent, and tails the logs with inode-keyed offsets.dnp3_pointsdataset; stockdnp3rows also populatezeek.dnp3_objects.function_codefor the OT function-code analytic. The Registers page now reads Modbus registers and DNP3 points.Verification
.venv/bin/python -m pytest tests/test_zeek_ingestor.py: 20 passed.python3 -m pylint --fail-under=7.5 src/parseus: 8.81/10.make_test_pcap.pyoutput viareplay_pcap.sh --check: all expected Modbus and DNP3 values match, including g2v1, range qualifier 0x02 and float inf cases.Known limits and follow-ups
ZEEK_IFACEdefaults toeth0; the Zeek container now exits with a message listing interfaces if it does not exist, butmanage-parseusdoes not validate it on the host yet.🤖 Generated with Claude Code