Repository navigation
Pattern-Guard 0.3: adaptive ML engine, WAF and protection gateway - #1
Merged
Merged
Conversation
Constant-memory building blocks for the adaptive engine:
- stats: exponentially decayed counters, smoothed ratios, inter-arrival
regularity, small top-k counters and a recent-id ring for enumeration
- normalize: path to route template (/product/123?x -> /product/{n}),
object-id extraction and a list of well-known scanner probe paths
Fixes problems found when testing against realistic traffic: - windows use Event::timestamp (monotonised), so access logs can be replayed; time_source = "system" keeps the old behaviour - per-actor windows keep incremental counts: O(1) per check instead of copying the whole window (was 5.6 ms per check after 20k flood events) - interval rule needs at least 5 gaps (two requests 2 s apart were blocked) - weighted_sum is a probabilistic OR, so extra evidence never lowers risk - rule thresholds are opt-in; max_actors defaults to 100 000 - config is validated at load; trusted actors; Challenge decision band; cooldown after repeated blocks - Event gains optional ip, user_agent and fingerprint
- streaming features per actor, subnet, client fingerprint and route - learned per-route baselines with asymmetric adaptation (resists poisoning); route surges only count together with client evidence - standardised logistic model stored as TOML, with trainer and mimicry augmentation helpers - Guard::assess returns decision, risk, bot score 1-99 and reasons; Guard::observe_response feeds response status back - hybrid mode: final risk is the max of model and rules
- logs: JSONL and nginx/Apache combined access-log parsing - pattern-guard replay: decision mix, top flagged actors with reasons and, for labelled logs, false positives / detection per scenario - pattern-guard train: scenario-balanced training with mimicry augmentation
- simulator: one site with legitimate background traffic (shoppers, office NAT, polling dashboard, partner API, mobile app, viral spike, login rush) and staggered attacks, plus unseen variants used only for evaluation - realworld_eval compares 0.2-style rules with the adaptive engine - built-in model trained on 3 h of simulated traffic (gen_traffic --seed 101 --hours 3)
Decodes values like a permissive server (repeated URL decoding, %uXXXX, HTML entities, JS escapes, SQL comments, Log4j lookups) and scores weighted indicators per attack class: SQLi, XSS, path traversal, command injection, SSRF, JNDI, XXE, SSTI and NoSQL injection. Includes a corpus test and a waf_eval example for public payload lists.
SHA-256 proof of work solved in the browser (pure-JS hash, works on plain HTTP), HMAC-signed challenges and an IP-bound HttpOnly clearance cookie; return paths are sanitised against open redirects and script injection.
- one decision per request; detect (shadow) and prevent modes - block page, challenge page, delay; trusted-proxy client IP resolution - per-IP connection cap, header-read timeout, body size limit - async EVE/ECS-style decision log, Prometheus metrics, API inventory - bot score, risk and request id forwarded to the upstream
Pulls the local API decision stream (community blocklist and local detections): ban decisions block, captcha decisions challenge.
Tails Suricata EVE and Wazuh alert files and turns qualifying alerts into temporary IP blocks, so network- and host-layer detections are enforced at the HTTP layer. Adds gateway.toml.example.
CI pins Rust 1.85, whose clippy rejects format! inside collect (format_collect); build the string with write! instead.
scripts/extract_crs_tests.py turns the CRS go-ftw regression tests (Apache-2.0, 5 247 tests) into JSONL; examples/crs_eval runs every test request through the WAF and reports blocked/logged rates per CRS category and paranoia level. CRS tests are not vendored.
How both engines decode, score and learn, with source paths; results of running the CRS regression tests against our WAF; the techniques we adopt and their licences.
Feature tracker gains the gaps found in CRS and open-appsec and the CRS benchmark result; roadmap milestone M2 now lists the concrete techniques and measurable acceptance criteria, including blocking PHP-code-plus- command payloads in multipart fields without CVE-specific rules.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Pattern-Guard goes from a fixed-threshold rate limiter to a self-hosted application-protection platform. One commit per feature; every commit builds and passes its tests.
Engine (
pattern-guardcrate)weighted_sumcan no longer lower risk when evidence is added.replay(JSONL or nginx combined logs) andtrainon labelled logs.Gateway (
pattern-guard-gatewaycrate, new)Docs
Measured (synthetic / public corpora — not yet real production traffic)
Breaking changes
Decision::Challengevariant.Eventfields.mode = "hybrid"andtime_source = "event".max_actorsdefaults to 100 000.All are listed in CHANGELOG.md.
Checks
GitHub Actions could not run (the job was never assigned a runner, which points to an account-level Actions/billing setting). All CI steps were run locally on Rust 1.85, the version CI pins, and on stable:
cargo fmt --all --checkcargo clippy --workspace --all-targets -- -D warningscargo test --workspace: 89 testscargo bench --workspace --no-runMerged by fast-forwarding
main, so the per-feature commits are kept unchanged.