Repository navigation
feat(ingestors): add Zeek Modbus/DNP3 deep packet inspection integration - #4
Open
adamsrnmsu wants to merge 1 commit into
Open
adamsrnmsu wants to merge 1 commit into
adamsrnmsu wants to merge 1 commit into
Conversation
adamsrnmsu
force-pushed
the
feat-zeek-ics-dpi
branch
from
November 12, 2026 04:12
6304488 to
8783515
Compare
This was referenced Nov 12, 2026
adamsrnmsu
marked this pull request as ready for review
November 12, 2026 20:09
adamsrnmsu
force-pushed
the
feat-zeek-ics-dpi
branch
from
November 12, 2026 20:19
8783515 to
8477c4a
Compare
5 of 11 tasks
Parseus reads Modbus register values from Malcolm-normalized documents but had no way to produce them without Malcolm, and had no DNP3 point values at all. This adds a `zeek` integration that extracts both from raw traffic. - deploy/integrations/zeek: Dockerfile (zeek/zeek:9.0.1, no ICSNPP or zkg, so it builds offline) and parseus.zeek, which adds modbus_detailed.log and dnp3_points.log from stock analyzer events; JSON logs, hourly rotation. Includes README, verify_zeek_runtime.sh, replay_pcap.sh and make_test_pcap.py, a stdlib pcap generator with known values and a --check mode. - parseus.ingestors.zeek: pure normalize() to the Malcolm document layout, plus a runner that tails the logs with inode-keyed offsets and stable document ids (restart-safe, idempotent) into arkime_sessions3-parseus-zeek-%y%m%d. - query_api.get_dnp3_points and reg_ops.dnp3_to_regop mirror the Modbus helpers. - Compose services/profile, build and image lists (offline bundles), integration map, CLI help, env defaults, and integrations README. Known gaps: stock Zeek raises no event for DNP3 g40 (analog output status) or the g2 binary-event flag byte; a custom Spicy analyzer (below) covers them. Inf/NaN float analog values are dropped. Verified: new unit tests on real Zeek output fixtures; Zeek over a generated pcap matches every expected Modbus/DNP3 value, and agrees with tshark on modbusSmall.pcap and dnp3_read.pcap point counts and values; ingest into Elasticsearch 8.19.5 is idempotent and get_modbus_registers / get_dnp3_points return the values; pylint 8.81/10. Review fixes folded in: g2v1 no longer logs invented value 1.0/flags 255; start/stop range qualifier 0x02 keeps its real start index; Inf/NaN float analogs are skipped; normalize() maps stock dnp3 fc_request to zeek.dnp3_objects.function_code so the OT function-code analytic sees DNP3; the Registers page now also reads DNP3 points; the ingestor caps reads at 8 MiB per file per poll, skips fully-read rotated files and drops pruned state keys; the Zeek container prunes rotated logs after a day and exits with a clear message listing interfaces when ZEEK_IFACE does not exist. make_test_pcap.py gained g2v1, q=0x02 and g30v5 inf cases (failed first, now match). pytest: 4 new tests failed first; full suite has the same 26 elasticsearch/env failures as before the change. DNP3 double-precision analogs (g30v6, g31v8, g32v6/v8, g33v6/v8) are decoded from the two 32-bit halves Zeek's DPwFlag/DPwTime/DPwoTime events carry; inf/NaN are skipped like float32. Outstations such as bennu and ferrofire send analog inputs as g30v6, so without this the live waterway range only yielded its g30v1 points. Ground truth: make_test_pcap.py now has g30v6/g32v6 cases; the previous script misses exactly those 5 rows, the fixed one matches all 31 DNP3 points. manage-parseus now checks ZEEK_IFACE on the host before bringing up the zeek service (only when the zeek integration is enabled; the service uses host networking) and fails with a message listing the host's interfaces. The Zeek index template maps zeek.dnp3_objects.function_code as keyword, so the function-code analytic's terms aggregation works instead of hitting a dynamically mapped text field. Both have tests that failed first. DNP3 g40v1-v4 (analog output status) and g2v1-v3 (binary input events) are logged by a custom Spicy analyzer (deploy/integrations/zeek/spicy/) attached to TCP/20000 next to the stock DNP3 analyzer. It parses link frames (every CRC-16 checked, bad frames dropped), transport FIR/FIN/sequence reassembly, the application header, then walks object headers with a size table so it can skip objects it does not log; an object of unknown size ends the fragment, it never guesses. It raises points only for objects stock does not log, so nothing is logged twice; parseus.zeek writes them into dnp3_points with the same columns (g2 value is the flag's state bit). The zeek/zeek image has spicyz but no C++ compiler, so the Dockerfile is multi-stage: a builder installs g++ and compiles the .hlto, and only the .hlto is copied into the final image. The build needs the Debian mirrors once. make_test_pcap.py has ground truth for g40v1-v4 (all qualifiers), g2v1-v3, a multi-object response, a 40-point fragment split over four link frames, and a second connection with a bad CRC, unknown objects, a segment gap and a truncated fragment. replay --check: "OK: 7 modbus rows, 108 dnp3 points match"; without the .hlto all g40/g2 rows go missing; tshark agrees on all 59 g40/g2 points; conn.log service is dnp3_tcp,spicy_parseusdnp3. Review fix: g80v1 (IIN) is sized as packed bits like g1v1, so a g40 point after an IIN object is reached (failed before the fix). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
adamsrnmsu
force-pushed
the
feat-zeek-ics-dpi
branch
from
November 13, 2026 02:18
8477c4a to
1f39fb8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds an opt-in
zeekintegration that pulls Modbus register values and DNP3 point values out of raw traffic (deep packet inspection), without needing Malcolm. Enable withmanage-parseus install --integrations zeek. This supersedes #2 (branch renamed to thetype-descriptionconvention); the hand-writtenCHANGELOG.mdentry is dropped because the changelog is now generated by releasaurus (#3).Design:
zeek/zeek:9.0.1), no ICSNPP and no zkg, so the image builds offline.deploy/integrations/zeek/zeek/parseus.zeekadds two JSON logs from stock analyzer events:modbus_detailed.log(request/response with register values) anddnp3_points.log(one row per point: group, variation, index, value, flags).parseus.ingestors.zeeknormalizes rows to the Malcolm document shape Parseus already queries (zeek.<log>.<field>,event.dataset), with stable document ids so re-reads are idempotent, and tails the logs with inode-keyed offsets.deploy/integrations/zeek/spicy/) runs next to the stock DNP3 analyzer and logs what stock Zeek cannot: g40v1-v4 (analog output status) and g2v1-v3 (binary input events). It checks every link-frame CRC, reassembles transport segments and walks object headers with a size table; it stops at an object of unknown size rather than guess, and only emits points stock does not log, so nothing is logged twice. The Dockerfile is multi-stage: a builder with g++ compiles the.hlto, and the final image carries only that file.dnp3_pointsdataset; stockdnp3rows also populatezeek.dnp3_objects.function_codefor the OT function-code analytic. The Registers page now reads Modbus registers and DNP3 points.Related Issues/PRs
Type of Change
fix)feat)docs)refactor)chore)Checklist
Testing
.venv/bin/python -m pytest tests/test_zeek_ingestor.py: 20 passed.python3 -m pylint --fail-under=7.5 src/parseus: 8.81/10.make_test_pcap.pyoutput viareplay_pcap.sh --check: all expected Modbus and DNP3 values match, including g2v1, range qualifier 0x02 and float inf cases.CHANGELOG.mdentry (one commit, amended).replay_pcap.sh --checkreports "OK: 7 modbus rows, 108 dnp3 points match" over cases for every g40/g2 variation and qualifier, a 40-point fragment split over four link frames, a bad CRC, unknown objects, a segment gap and a truncated fragment. Without the.hlto, all g40/g2 rows are missing. tshark agrees on all 59 g40/g2 points.conn.logshowsdnp3_tcp,spicy_parseusdnp3on the same connection.tests/test_zeek_ingestor.py: 22 passed.ZEEK_IFACEcheck and thefunction_codekeyword mapping, each with a test that failed first; full suite unchanged apart from the new tests.Additional Notes
Known limits and follow-ups:
zeek/zeekshipsspicyzwithout a C++ compiler.ZEEK_IFACEdefaults toeth0.manage-parseuschecks it against the host's interfaces before bringing up zeek (the service uses host networking), and the Zeek container also exits with a message listing interfaces. On Docker Desktop for macOS the check sees the Mac's interfaces, not the Docker VM's, so it assumes a Linux host.zeek.dnp3_objects.function_codeis mapped askeywordin the Zeek index template so the OT function-code analytic can aggregate DNP3 requests. Indices created before this change keep a dynamic text mapping until rollover or reindex.main; the codespell check covers the whole repo and ci(release): adopt releasaurus with CalVer and the phenix-apps contribution workflow #3 fixes a typo (incase) in a file this PR touches.🤖 Generated with Claude Code