Skip to content

Pattern-Guard 0.3: adaptive ML engine, WAF and protection gateway - #1

Merged
HeewaDev merged 21 commits into
mainfrom
claude/intelligent-fermi-u7j5gs
Nov 19, 2026
Merged

HeewaDev merged 21 commits into
mainfrom
claude/intelligent-fermi-u7j5gs

Conversation

@HeewaDev

@HeewaDev HeewaDev commented Nov 19, 2026 •

Copy link
Copy Markdown
Owner

Summary

Pattern-Guard goes from a fixed-threshold rate limiter to a self-hosted application-protection platform. One commit per feature; every commit builds and passes its tests.

Engine (pattern-guard crate)

  • Rule engine fixes:
    • Windows run on event time, so logs can be replayed.
    • O(1) bounded state: a check took 5.6 ms after 20k flood events; it now takes about 1 µs.
    • Two requests 2 s apart are no longer blocked.
    • weighted_sum can no longer lower risk when evidence is added.
    • Config is validated at load.
  • Adaptive behavioural ML:
    • Streaming features per actor, subnet, fingerprint and route.
    • Poisoning-resistant learned baselines.
    • An explainable logistic model, with bot score 1–99 and reasons.
  • CLI: replay (JSONL or nginx combined logs) and train on labelled logs.
  • Traffic simulator and evaluation: built-in model trained on simulated traffic.
  • WAF: decoding plus weighted indicators for 9 attack classes.

Gateway (pattern-guard-gateway crate, new)

  • Reverse proxy that fuses IP lists, reputation, WAF and behavioural ML into one decision. Detect and prevent modes.
  • Proof-of-work bot challenge with an IP-bound HMAC clearance cookie, verified in real headless Chromium.
  • CrowdSec bouncer, plus Suricata and Wazuh alert feeds enforced at the gateway.
  • Per-IP connection cap, slowloris timeout, body limit.
  • Decision log, Prometheus metrics, API inventory.

Docs

  • README, gateway, deployment, operations and architecture guides.
  • Feature tracker against open-appsec and Imperva.
  • Research notes on OWASP CRS and open-appsec.
  • Product plan with measurable milestones.

Measured (synthetic / public corpora — not yet real production traffic)

Test 0.2 0.3
Legit requests stopped (1 h simulated mixed traffic) 11.9 % ~0.3 %
Attack requests stopped (same traffic, incl. unseen variants) 52 % ~98.5 %
WAF: PayloadsAllTheThings / 19 215 benign values — 85.4 % blocked / 0 benign blocked
WAF: OWASP CRS regression tests (3 730 attacks) — 11.8 %, the main gap; plan in ROADMAP M2
Gateway throughput, full inspection (4 vCPU shared) — ~32k req/s, p99 ≈ 3 ms

Breaking changes

  • Decision::Challenge variant.
  • New Event fields.
  • Rule thresholds are opt-in.
  • Default mode = "hybrid" and time_source = "event".
  • max_actors defaults to 100 000.

All are listed in CHANGELOG.md.

Checks

GitHub Actions could not run (the job was never assigned a runner, which points to an account-level Actions/billing setting). All CI steps were run locally on Rust 1.85, the version CI pins, and on stable:

  • cargo fmt --all --check
  • cargo clippy --workspace --all-targets -- -D warnings
  • cargo test --workspace: 89 tests
  • cargo bench --workspace --no-run

Merged by fast-forwarding main, so the per-feature commits are kept unchanged.

Constant-memory building blocks for the adaptive engine:

- stats: exponentially decayed counters, smoothed ratios, inter-arrival
  regularity, small top-k counters and a recent-id ring for enumeration
- normalize: path to route template (/product/123?x -> /product/{n}),
  object-id extraction and a list of well-known scanner probe paths
Fixes problems found when testing against realistic traffic:

- windows use Event::timestamp (monotonised), so access logs can be
  replayed; time_source = "system" keeps the old behaviour
- per-actor windows keep incremental counts: O(1) per check instead of
  copying the whole window (was 5.6 ms per check after 20k flood events)
- interval rule needs at least 5 gaps (two requests 2 s apart were blocked)
- weighted_sum is a probabilistic OR, so extra evidence never lowers risk
- rule thresholds are opt-in; max_actors defaults to 100 000
- config is validated at load; trusted actors; Challenge decision band;
  cooldown after repeated blocks
- Event gains optional ip, user_agent and fingerprint
- streaming features per actor, subnet, client fingerprint and route
- learned per-route baselines with asymmetric adaptation (resists
  poisoning); route surges only count together with client evidence
- standardised logistic model stored as TOML, with trainer and mimicry
  augmentation helpers
- Guard::assess returns decision, risk, bot score 1-99 and reasons;
  Guard::observe_response feeds response status back
- hybrid mode: final risk is the max of model and rules
- logs: JSONL and nginx/Apache combined access-log parsing
- pattern-guard replay: decision mix, top flagged actors with reasons and,
  for labelled logs, false positives / detection per scenario
- pattern-guard train: scenario-balanced training with mimicry augmentation
- simulator: one site with legitimate background traffic (shoppers, office
  NAT, polling dashboard, partner API, mobile app, viral spike, login rush)
  and staggered attacks, plus unseen variants used only for evaluation
- realworld_eval compares 0.2-style rules with the adaptive engine
- built-in model trained on 3 h of simulated traffic
  (gen_traffic --seed 101 --hours 3)
Decodes values like a permissive server (repeated URL decoding, %uXXXX,
HTML entities, JS escapes, SQL comments, Log4j lookups) and scores weighted
indicators per attack class: SQLi, XSS, path traversal, command injection,
SSRF, JNDI, XXE, SSTI and NoSQL injection. Includes a corpus test and a
waf_eval example for public payload lists.
SHA-256 proof of work solved in the browser (pure-JS hash, works on plain
HTTP), HMAC-signed challenges and an IP-bound HttpOnly clearance cookie;
return paths are sanitised against open redirects and script injection.
- one decision per request; detect (shadow) and prevent modes
- block page, challenge page, delay; trusted-proxy client IP resolution
- per-IP connection cap, header-read timeout, body size limit
- async EVE/ECS-style decision log, Prometheus metrics, API inventory
- bot score, risk and request id forwarded to the upstream
Pulls the local API decision stream (community blocklist and local
detections): ban decisions block, captcha decisions challenge.
Tails Suricata EVE and Wazuh alert files and turns qualifying alerts into
temporary IP blocks, so network- and host-layer detections are enforced
at the HTTP layer. Adds gateway.toml.example.
CI pins Rust 1.85, whose clippy rejects format! inside collect
(format_collect); build the string with write! instead.
scripts/extract_crs_tests.py turns the CRS go-ftw regression tests
(Apache-2.0, 5 247 tests) into JSONL; examples/crs_eval runs every test
request through the WAF and reports blocked/logged rates per CRS
category and paranoia level. CRS tests are not vendored.
How both engines decode, score and learn, with source paths; results of
running the CRS regression tests against our WAF; the techniques we adopt
and their licences.
Feature tracker gains the gaps found in CRS and open-appsec and the CRS
benchmark result; roadmap milestone M2 now lists the concrete techniques
and measurable acceptance criteria, including blocking PHP-code-plus-
command payloads in multipart fields without CVE-specific rules.
@HeewaDev
HeewaDev merged commit 042b84c into main Nov 19, 2026
1 check failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant