diff --git a/.gitignore b/.gitignore index 1848d45..dd18cd2 100644 --- a/.gitignore +++ b/.gitignore @@ -45,3 +45,6 @@ out/ *.pfx *.key !tests/fixtures/tls/server.key + +# Optional local Streamlit portrait with unresolved redistribution rights +/Felix.jpg diff --git a/Felix.jpg b/Felix.jpg deleted file mode 100644 index 8c71e02..0000000 Binary files a/Felix.jpg and /dev/null differ diff --git a/NOTICE.md b/NOTICE.md index 7afbc38..7c9f9c0 100644 --- a/NOTICE.md +++ b/NOTICE.md @@ -4,7 +4,8 @@ NetGuardian project code and documentation are provided under the MIT license in LICENSE, as selected by the repository owner. Third-party dependencies and downloaded models retain their own licenses; they are not relicensed by this repository. -The legacy portrait asset `Felix.jpg` has unresolved redistribution provenance. -It is excluded from the project's MIT licensing declaration pending owner -confirmation and excluded from the API/worker Docker build context. Keeping the -original file locally does not establish permission to redistribute the portrait. +The optional legacy Streamlit portrait, `Felix.jpg`, has unresolved +redistribution provenance and is outside the project's MIT licensing scope. +It is excluded from the current source tree, Docker build context and source +bundle. Older Git commits may still contain the original file. A local copy +remains optional because the Streamlit sidebar has a text fallback. diff --git a/README.md b/README.md index 43b0ea8..ea36a38 100644 --- a/README.md +++ b/README.md @@ -1,489 +1,146 @@ -# NetGuardian V2 +# NetGuardian — Network Diagnostics & Evidence-Based Troubleshooting Platform -**Network diagnostics and monitoring for approved targets.** Run a probe, compare -results from two network locations, and review the evidence behind an incident. +**English** · [简体中文](README.zh-CN.md) -[Get started](#local-development-setup) · [Windows launcher](#windows-local-launcher) · [System design](docs/architecture/SYSTEM_DESIGN.md) · [Delivery status](docs/product/DELIVERY_STATUS.md) +NetGuardian helps IT teams investigate why a service is unreachable by checking +each layer of the connection and keeping a record of what each check found. -## What you can do +[Quick start](#quick-start) · [Architecture](#system-architecture) · [Demo evidence](#demo-evidence) · [Verification](#test--verification-status) -- **Diagnose connectivity** with DNS, TCP, TLS and HTTP HEAD probes. View individual - timing measurements and TLS certificate details. Enable bounded ICMP and - traceroute checks when needed. -- **Monitor over time** with scheduled checks, collection freshness, and incident - and recovery history. -- **Compare network locations** by pairing a server-side check with a check from an - outbound-only Windows Agent that enforces its own target policy. -- **Review incident evidence** in redacted reports with checksums. Optional Ollama - investigations cite recorded evidence and can be reopened alongside the report. +## Why I Built This -## How it fits together +When a workstation can reach one service but a front-desk device cannot, +the support ticket rarely tells you whether the failure is in name resolution, +connectivity, TLS, or the application. I built NetGuardian to make that investigation +repeatable and to give the next technician the observations behind a finding. -| Component | Role | -| --- | --- | -| [Next.js dashboard](web/) | Submit diagnostics and review monitoring, comparisons and reports. | -| [FastAPI service and worker](service/) | Authorize requests, schedule and run server-side jobs, and store results with SQLAlchemy. | -| [Probe engine](probe_core/) | Validate targets and collect protocol-specific measurements. | -| [Windows Agent](agent/) | Run approved checks from a local network location and return results over an outbound connection. | - -V2 is currently scoped to local Windows evaluation. See the -[Windows delivery record](docs/product/WINDOWS_DELIVERY.md) for verified scenarios -and the [delivery checklist](docs/product/DELIVERY_STATUS.md) for remaining -production and Linux deployment work. - -## Local development setup +NetGuardian grew from a Python and Streamlit toolkit into an independent portfolio +project informed by my work troubleshooting LAN/WLAN, TCP/IP, DNS, DHCP and +Windows endpoint connectivity. The scenarios here use controlled test systems. -Prerequisites: Python 3.11, uv 0.13.7, Node.js 24 or later, pnpm 11.25.0. -The lockfiles are `uv.lock` and `web/pnpm-lock.yaml`. +## Key Features -```sh -git clone https://github.com/TYF009/NetGuardian.git -cd NetGuardian -python -m pip install uv==0.13.7 -uv sync --locked +| Capability | What it shows | +| --- | --- | +| Layered diagnostics | DNS records, TCP connectivity, TLS handshake and certificate details, and HTTP HEAD status and timing. Opt-in ICMP and traceroute provide additional reachability and path evidence. | +| Scheduled monitoring | Pause and resume checks, review collection freshness, and follow incident and recovery history. Missing observations are shown as gaps. | +| Two-location comparison | Pair a server-side check with one from an enrolled Windows Agent. Compare the recorded observations within the limits of their endpoints and timing. | +| Incident evidence | Open saved, redacted report snapshots, verify their SHA-256 digests, and trace findings back to the underlying observations. | +| Optional investigation | Use a configured Ollama model to summarize report evidence. Validated findings and unverified hypotheses are displayed separately. | + +## System Architecture + +```mermaid +flowchart LR + Operator --> Web["Next.js dashboard"] + Web --> API["FastAPI control API"] + API --> DB[("SQLite / PostgreSQL")] + Worker["Background Worker"] --> DB + Worker --> Engine["Shared probe engine"] + Agent["Outbound Windows Agent"] -->|"Poll jobs and return observations over HTTPS"| API + Agent --> Engine + Engine --> Targets["Approved targets"] ``` -Configure the following environment in the API, worker and migration shells: - -| Variable | Meaning | -|---|---| -| `NETGUARDIAN_TOKENS` | JSON object mapping random ASCII bearer tokens (32–512 characters) to workspace IDs; required by API/worker | -| `NETGUARDIAN_ORIGINS` | JSON object mapping workspace IDs to exact approved HTTP(S) origins; defaults to no targets | -| `NETGUARDIAN_DATABASE_URL` | SQLAlchemy URL; defaults to `sqlite:///netguardian.db` for local development | -| `NETGUARDIAN_VANTAGE_ID` | Required by the worker: stable non-nil UUID identifying its network location | -| `NETGUARDIAN_ENABLE_SYSTEM_PROBES` | Set to `true` on the worker/agent to enable fixed ICMP/traceroute tools | +- [Web dashboard](web/) and [FastAPI API](service/api.py) handle authenticated requests and workspace views. +- The [Worker](service/worker.py) claims leased jobs from the database and persists observations. The [Agent](agent/) polls the API over HTTPS and applies its own local target policy. +- The [probe engine](probe_core/) runs bounded checks. [Alembic migrations](migrations/) track schema changes, and [report export](service/report_files.py) checks saved snapshot integrity. -Generate a token locally with `python -c "import secrets; print(secrets.token_urlsafe(32))"`. -Keep it out of source control and shell history. A token mapping has the shape -`{"":"local"}`. An origin mapping has the shape -`{"local":["https://:443"]}`. These are placeholders, not usable -credentials or authorization to probe third-party services. Start with `{}` origins -until you have approved a target you own or are authorized to diagnose. -Configuration files named `.env` are **not loaded automatically** by the Python CLI. +See the [system design](docs/architecture/SYSTEM_DESIGN.md) and [data model](docs/architecture/DATA_MODEL.md). -From the repository root, migrate before starting either service: - -```sh -uv run --locked alembic upgrade head -uv run --locked uvicorn service.api:app_factory --factory --host 127.0.0.1 --port 8000 -``` +## Engineering Highlights -Generate a probe-location UUID once with `python -c "import uuid; print(uuid.uuid4())"` -and set `NETGUARDIAN_VANTAGE_ID` in the worker environment. Preserve it across restarts. -Workers sharing one network location may share it; distinct locations must use distinct -IDs. This is an operator assertion, not automatic location attestation. Old records -retain the legacy shared ID and cannot distinguish historical worker deployments. +- **Probe safety.** Server-side probes deny targets by default and require exact approved origins. The executor connects to a validated numeric address while preserving the hostname for HTTP and TLS. HTTP uses HEAD with bounded headers and no redirect following. +- **Measurement semantics.** DNS, TCP, TLS and HTTP timing are kept separate. HTTP duration describes an application-layer request, while ICMP round-trip time measures an echo exchange. +- **Asynchronous delivery.** API acceptance is separate from execution. Jobs have leases and expiry, and the Agent has a local result spool to retry delivery without repeating a completed probe. +- **Reproducible evidence.** Reports retain the observations used to explain an incident. The export preserves the stored snapshot for [offline checksum verification](scripts/verify_report.py). -In another shell with the same configuration and repository working directory: +See [responsible probing and measurement details](docs/operations/LOCAL_DEVELOPMENT.md#target-policy-and-measurement-semantics) and the [threat model](docs/security/THREAT_MODEL.md). -```sh -uv run --locked python -m service.worker -``` +## Demo Evidence -In a third shell: +| Scenario | Evidence | +| --- | --- | +| Service returns HTTP 503, then recovers to HTTP 204 | A controlled loopback HTTPS integration exercised an Agent monitor. The local browser QA recorded both statuses, incident recovery and the checksum-verified report. [Browser QA record](docs/product/BROWSER_QA.md#populated-workflow-check-2026-11-05) | +| Operator exports a report | A local browser download was checked with the offline verifier. The test environment used isolated evidence and a test workspace. [Export QA record](docs/product/BROWSER_QA.md#real-model-investigation-and-report-export-2026-11-06) | +| An operator browses missing observations | A synthetic fixture of 101 failed jobs exercised pagination and the UI message for missing measurements. [Fixture QA record](docs/product/BROWSER_QA.md#retained-history-production-web-build-2026-11-05) | -```sh -cd web -pnpm install --frozen-lockfile -pnpm dev -``` +These are local QA records. The HTTPS results came from a controlled test server, and the history fixture contains synthetic job state. -Open [the local interface](http://127.0.0.1:3000) and enter the workspace token. -The Web proxy uses `http://127.0.0.1:8000` by default. Override `NETGUARDIAN_API_URL` -only with a trusted API origin. Browser users cannot select the proxy destination. -The API buffers at most 64 KiB per request before JSON parsing, with a 10-second -body-receive deadline (including chunked uploads). This does not replace deployment -connection limits or general request throttling. -Tokens stay in React memory; disconnect remounts the workspace and cancels requests. -A browser abort does not roll back an operation already committed by the API. +## Technology Stack -For a local production build, run `pnpm build` then `pnpm start` inside `web/`. -Keep the development services on loopback. Non-loopback operation needs verified -TLS and the unfinished production controls listed in the delivery checklist. +Python 3.11 · FastAPI · SQLAlchemy · Alembic · SQLite / PostgreSQL · Next.js · React · TypeScript · Node.js 24 · pnpm · pytest · Ruff · GitHub Actions -## Windows local launcher +## Quick Start -After installing the locked Python dependencies, installing Web dependencies and -running `pnpm build` in `web/`, configure the environment variables described above -(including a stable vantage ID). From the repository root: +**Windows local evaluation.** Install Python 3.11 and Node.js 24. From PowerShell: ```powershell +git clone https://github.com/TYF009/NetGuardian.git +Set-Location NetGuardian +python -m pip install uv==0.13.7 +npm install -g pnpm@11.25.0 +uv sync --locked +pnpm --dir web install --frozen-lockfile +pnpm --dir web build + +$workspaceToken = (uv run --locked python -c "import secrets; print(secrets.token_urlsafe(32))").Trim() +$env:NETGUARDIAN_TOKENS = @{ $workspaceToken = 'local' } | ConvertTo-Json -Compress +$env:NETGUARDIAN_ORIGINS = '{}' +$env:NETGUARDIAN_VANTAGE_ID = [guid]::NewGuid().ToString() +$workspaceToken uv run --locked python -m scripts.run_windows --migrate ``` -`--migrate` explicitly upgrades the configured database before startup; back up an -existing database first. Omit it on later starts. The launcher requires an existing -production Web build and starts API, worker and Web on fixed loopback ports 8000 -and 3000. It refuses occupied ports, missing configuration or outdated schemas. -It does not install dependencies, generate credentials, approve targets, start an -Agent or configure Ollama. Browser access still requires your workspace token. -The worker uses the configured target policy; use empty origins for a no-probe trial. - -`--check` validates prerequisites without starting services or migrating. For a -bounded smoke run, use `--run-seconds 5`; the timer starts after API and Web are ready. -The worker process is supervised, but this is not a worker health/throughput check. -Ctrl+C or a child-process exit stops the process trees created by this launcher. -Shutdown is forced: in-flight work relies on existing leases and retry rules. -The launcher joins an unnamed Windows Job Object before spawning children; its -termination also terminates owned descendants, including after force-kill. It -refuses startup if the host prevents this containment. This is not a Windows -Service manager or a signed installer. - -Per-run logs remain under ignored `.netguardian-local/`; review and protect them as -local operational data. Credentials are not printed or written by the launcher; -the Web child does not inherit NetGuardian backend configuration or credentials. -Rebuild Web after source changes. This launcher has been smoke-tested on Windows; -it does not change the deferred Linux support scope. - -## Target policy and measurement semantics - -Cloud policy defaults to deny. Only administrator-configured exact origins are -accepted; mixed public/private DNS answers, special addresses, credentials in URLs, -ambiguous hosts and unsupported schemes are rejected. Connections use validated -numeric addresses while retaining the original hostname for Host/SNI and certificate -verification. HTTP uses HEAD, downloads no body, caps headers at 16 KiB and does not -follow redirects. Ambient proxies are not used. The overall probe deadline is at most -10 seconds; only the first approved address is attempted. - -| Measurement | Meaning | -|---|---| -| `dns_lookup_ms` | Resolution and address-policy planning duration | -| `tcp_connect_ms` | TCP connection establishment duration | -| `tls_handshake_ms` | TLS stream setup/handshake duration | -| `http_ttfb_ms` | Time from sending HEAD until the first response byte | -| `http_total_ms` | Total probe time through complete HTTP response headers, including setup | -| `icmp_rtt_ms` | Exact ICMP echo RTT when the OS reports one; less-than bounds remain separate samples | - -HTTP duration is not ICMP RTT. A cloud result describes the worker's network, not the -browser's. TLS evidence does not prove service ownership or certificate revocation -checking. Agent observations are self-reported. - -System probes use canonical numeric addresses and fixed executable/argument lists, -never arbitrary shell commands. ICMP sends one echo; traceroute uses at most 15 hops. -The process runner caps execution at nine seconds and output at 64 KiB. Missing tools -or recognized permission failures are explicit. No automatic privilege elevation is -performed. Locale variations can cause parse failures. Production cloud policy still -rejects loopback; integration tests use an isolated test-only policy. - -## Monitoring and comparisons - -Create a monitor with an approved target, probe kind and cloud/agent vantage. -Intervals are at least 60 seconds. Schedules run independently of browser sessions; -at most one unfinished run exists per monitor. Each workspace currently permits -20 monitors, 20 pending jobs and 60 newly created jobs per rolling 60 seconds. -Manual submissions, diagnostic pairs and schedules share the rate budget, including -completed jobs. Pairs consume two slots. API rejection returns 429/Retry-After; schedules -wait without a catch-up burst. This is not a complete HTTP request-rate limiter. - -Pause prevents new scheduled jobs; queued/running jobs may finish. Resume rechecks -authorization, resets consecutive counters and starts a current cycle without replaying -missed intervals. HTTP 4xx/5xx is unhealthy; policy denial and unsupported capability -are not network outages. Missing results break consecutive counters without recovering -open incidents. Collection freshness is distinct from target health. - -New incidents include the failures that triggered their threshold and subsequent -observations through recovery. Missing measurements are listed explicitly. Links retain -at most the latest 1000 observations; older incidents are not automatically backfilled. - -Compare can queue matching cloud/agent jobs atomically, consuming two pending slots. -Execution is asynchronous. Numeric deltas require compatible target, kind, configuration, -probe version and destination, distinct vantages, successful observations and at most -30 seconds of skew. DNS results record A/AAAA addresses, observed TTL, canonical name and resolver endpoint. -DNS timing deltas remain withheld for every probe kind: an endpoint does not -establish equivalent cache state. Other comparable timings include an explicit -reminder that HTTP total duration includes resolution and connection setup. -IP literal targets skip DNS and leave `dns_lookup_ms` absent. -Differences do not establish causality. - -Monitor **View history** retrieves the latest 100 runs in chronological order via -`GET /api/monitors/{id}/runs`. Choose one timing metric for a scatter plot and an -accessible evidence table. Missing timings are not zero and points are not joined -across gaps. The time axis is server job creation time, not claimed agent observation -time. History is an explicit snapshot; reopen it to refresh. Timing alone does not -establish target health. The initial chart uses a small SVG component rather than a -charting dependency; richer zoom/aggregation is still future work. - -## Optional local agent - -Copy `agent/config.example.json` to an ignored local configuration and set its HTTPS -control API origin. Explicitly list allowed probe origins. Private addresses require -explicit RFC1918/IPv6 ULA `private_cidrs`; loopback, link-local and metadata addresses -remain prohibited. Server jobs cannot expand local policy. The server policy must also -authorize the origin. - -Issue a ten-minute single-use enrollment token from Agents, then run: +Open [the local dashboard](http://127.0.0.1:3000), enter the test workspace token printed +in PowerShell. The empty origin list lets you explore the UI +without enabling probes. Use a fresh checkout or back up an existing database +before migration. Keep a stable Worker location ID when retaining observations +across sessions. -```sh -uv run --locked python -m agent.client --config agent/config.local.json --enroll -uv run --locked python -m agent.client --config agent/config.local.json -``` +For separate API, Worker and Web processes, authorized targets, Agent enrollment +and configuration, see the [local development guide](docs/operations/LOCAL_DEVELOPMENT.md). +See the [operations runbook](docs/operations/RUNBOOK.md) for readiness, backups and +troubleshooting. + +## Test & Verification Status -Enter the enrollment token at the hidden prompt. The agent credential is stored in -Windows Credential Manager or Linux Secret Service/KWallet, not the JSON file. The -agent fails closed without a supported store, uses outbound verified HTTPS, and opens -no listener. Revocation stops access, pauses its monitors and terminates pending jobs -without fabricating results. Uploads must match the job, attempt, configuration digest -and lease; identical duplicate uploads are accepted. - -Control responses have a 64 KiB limit and 20-second total deadline; compressed or -malformed responses are rejected. One result is retried in memory, or in an opt-in bounded local spool; see the runbook. -Credential rotation is available through `--rotate-credential`; see the runbook. -Signed distribution and service installers remain unfinished. -Without a configured spool, restarting the agent can lose its pending result. Do not claim offline durability. - -## Reports, AI and audit - -The Reports view reopens the latest 100 saved snapshots without regenerating them. -Readable reports show incident times, recorded findings with in-page evidence citations, -missing observations, limitations and per-observation timings. Raw JSON remains available -for inspection and download. -Reports store redacted snapshots with SHA-256 checksums. Query strings and free-form -errors are removed; paths and addresses remain operational data to review before sharing. -Reads and AI submission recompute the checksum; mismatch returns HTTP 409. The checksum -is not a signature or protection against changing both content and digest. - -Optional AI requires administrator-set `NETGUARDIAN_OLLAMA_URL` and -`NETGUARDIAN_OLLAMA_MODEL`. Remote endpoints require HTTPS; loopback HTTP is allowed. -No model is downloaded automatically. The adapter uses Ollama -[structured output](https://docs.ollama.com/capabilities/structured-outputs) via -[chat](https://docs.ollama.com/api/chat), with a 30-second deadline and 64 KiB response cap. -Only the latest 30 selected evidence records are submitted, excluding URLs, IPs and -endpoint free text. Facts must match recorded values; hypotheses require citations and -remain unverified. Model tools are prohibited. Missing/rejected model output remains -explicit; deterministic reports still work. Real inference quality is not yet evaluated. - -Investigations reopens saved output and source reports without calling the model again. -Audit shows the latest 100 covered operator events. Logs identify roles, not individual -humans, and are not tamper-proof. See the [runbook](docs/operations/RUNBOOK.md) for scope. - -## Checks and operations +- **Automated checks.** GitHub Actions runs Python unit and selected runtime tests on Windows and Ubuntu, PostgreSQL integration tests, and Web tests, TypeScript checks, schema drift checks and a Next.js production build. See the [workflow](.github/workflows/ci.yml) and [main baseline run](https://github.com/TYF009/NetGuardian/actions/runs/42789779017). +- **Controlled local verification.** The [Windows delivery record](docs/product/WINDOWS_DELIVERY.md) documents loopback TLS, Agent workflows, database verification and browser checks with their scope. +- **Reproduce checks.** Run these from the repository root. The [local guide](docs/operations/LOCAL_DEVELOPMENT.md#checks-and-operations) describes optional integration prerequisites. ```sh uv run --locked ruff check . uv run --locked ruff format --check . -uv run --locked python -m compileall -q app.py diagnostics.py probe_core service agent migrations tests integration uv run --locked pytest -q -uv run --locked pytest integration/test_tls.py integration/test_process_cleanup.py -q +uv run --locked pytest integration -q -rs +pnpm --dir web test +pnpm --dir web typecheck +pnpm --dir web build ``` -The default unit suite blocks network and process execution. Integration tests are -separate: TLS uses loopback fixtures; process tests launch fixed local Python fixtures. -Windows system-tool tests require `NETGUARDIAN_TEST_SYSTEM_PROBES=true`. -PostgreSQL tests require a disposable `netguardian_test` database and -`NETGUARDIAN_TEST_POSTGRES_URL`; absent configuration produces skips, not verification. -GitHub Actions runs Python checks on Windows and Ubuntu, Web checks on Ubuntu, and PostgreSQL integration tests. See the [CI workflow](.github/workflows/ci.yml) for the test matrix. - -Back up existing databases before migration. Do not blindly stamp schemas from old -unversioned development snapshots. Docker Compose configuration is provided but remains -unverified locally. See the [operations runbook](docs/operations/RUNBOOK.md). - -## Legacy Streamlit and project documents - -The repaired legacy UI remains available with -`uv run --locked streamlit run app.py --server.address 127.0.0.1`. -Its probe entry points are intentionally disabled; use the V2 API/Web path for approved -probes. Streamlit keeps default CORS/XSRF protections. Historical HTTP timing must never -be relabeled as ICMP or component timings. - -- [Product requirements](docs/product/PRD.md) -- [System design](docs/architecture/SYSTEM_DESIGN.md) -- [Data model](docs/architecture/DATA_MODEL.md) -- [Threat model](docs/security/THREAT_MODEL.md) -- [Implementation plan and original audit](docs/product/IMPLEMENTATION_PLAN.md) -- [Current delivery gates](docs/product/DELIVERY_STATUS.md) -- [Browser QA scope](docs/product/BROWSER_QA.md) - -See [NOTICE.md](NOTICE.md) for licensing scope and the legacy asset redistribution review. - -The Web API proxy bounds POST/PATCH bodies to 8 KiB of raw UTF-8 bytes and a -10-second read deadline, including chunked requests. Encoded/compressed bodies -are rejected. Run its focused regression checks with `cd web` and `pnpm test`. - -Agent jobs have a 120-second delivery window from creation. The scheduler (or an -agent poll) marks overdue unclaimed work and exhausted retries as failed without -a measurement, records an audit event, and releases monitor queue capacity. -A live 30-second lease is allowed to finish. Missing delivery breaks consecutive -incident evidence; it never establishes target failure or recovery. - - -The opt-in local transport integration suite includes -`uv run --locked pytest integration/test_agent_workflow.py -q`. It starts an ephemeral -loopback HTTPS API and HTTP HEAD fixture, enrolls an agent, executes two real probes -(503 then 204), reloads a pending upload from disk, verifies duplicate acceptance, -and checks incident recovery and report evidence. It also executes an atomic cloud/agent -diagnostic pair through both real transports and compares their recorded measurements. Test-only DNS mapping, loopback -address authorization and a private test CA are scoped to the test; production -policies and system certificate trust are unchanged. This is not evidence of a -cross-machine agent installation, all crash boundaries, Linux keyring behavior -or PostgreSQL operation. No public target is contacted. - -Probe version 0.2.1 measures durations with Python `perf_counter`, a high-resolution -monotonic clock. This avoids the coarse Windows Python 3.11 `monotonic` timer. -Historical measurements are preserved; different probe versions are not compared. - -Diagnostic pairs now have persistent IDs and a workspace-scoped history in the -Compare page. Reloading the page does not lose pair membership. Existing jobs are -not retroactively grouped. Generic retention preserves paired evidence; automated -pair expiration is not yet implemented. Apply database migrations before startup. - -TLS certificate/hostname validation failures use `tls_verification_failed`, distinct -from generic transport errors. When available, `error.tls_verify_code` preserves the -TLS runtime verification code; its interpretation is runtime-specific. Failed -verification does not yield verified TLS certificate metadata or an HTTP status. - -Control API admission is process-local: 32 concurrent API requests, a 120-request -burst replenished at 60/second, plus a 60-mutation workspace burst replenished at -1/second. Agent poll/result traffic uses only the global budget. Health endpoints -remain accessible. These are protective defaults, not benchmarked capacity or -aggregate limits across replicas; 429 responses carry `Retry-After`. - - -Windows PostgreSQL 17.11 has now passed local schema, concurrent-worker, -concurrent-startup and tenant-constraint tests, plus the real HTTPS Agent/report/ -comparison workflow. A local dump/restore comparison matched all 12 public tables. -See `docs/operations/RUNBOOK.md` for exact scope and limitations; Docker and Linux -execution are not inferred from these results. The HTTPS integration test now runs -on SQLite and, when the explicitly named disposable PostgreSQL database is configured, -on PostgreSQL as well; otherwise its PostgreSQL case is skipped. - - -Setting `NETGUARDIAN_TEST_KEYRING=true` also opts the HTTPS workflow into a real OS -credential-store rotation check. It creates a randomly named test service, commits -a server credential rotation while retaining the old local credential and pending -rotation entry, then runs normal recovery. It verifies old-token rejection, -new-token access, revocation, and cleanup of both temporary credential entries. -This passed with Windows Credential Manager. It reproduces lost-acknowledgement -state; it does not inject an actual dropped socket or terminate an agent process. - -The model adapter requests identity encoding and rejects compressed responses -before consuming their bodies. Its 64 KiB response cap applies to raw streamed -bytes before JSON decoding. Model transport tests do not establish model quality. - - -The Agent CLI holds an OS file lock for its control origin and agent ID before -opening the keyring or processing jobs. Duplicate instances fail instead of racing -spool writes or credential rotation. Lock files live under `~/.netguardian/locks`; -they remain after exit, but ownership is released by the OS when the process dies. -Do not delete lock files while an agent is running. This coordinates processes -under the same OS user; separate users/machines must not share one enrollment. - - -### Long incident history - -Run `uv run alembic upgrade head` before starting this version. Incident links are -now stored independently of the latest-1000 summary. Authenticated control API -clients can read `/api/incidents/{id}/evidence?after=0&limit=100`, following -`next_after` until it is null. Earlier links already discarded by older versions -cannot be reconstructed. Reports remain bounded snapshots. In the Incidents tab, choose **Browse retained -history** to navigate pages of 100 links and inspect individual observations. Linked jobs are protected from generic retention; -plan storage accordingly. Downgrading populated incident history is refused. - - -### Local Ollama GPU compatibility fallback (Windows) - -The local validation machine could not load Ollama 0.40.14's CUDA runner -(`device kernel image is invalid`). CPU inference worked without driver changes. -To run a separate loopback-only CPU instance, use a PowerShell terminal: +## Current Limitations -```powershell -$env:CUDA_VISIBLE_DEVICES = "-1" -$env:OLLAMA_VULKAN = "0" -$env:OLLAMA_HOST = "127.0.0.1:11435" -& "$env:LOCALAPPDATA/Programs/Ollama/ollama.exe" serve -``` +- The documented delivery target is local Windows evaluation. Docker Compose deployment and broader Linux runtime qualification remain open. +- Loopback tests cover the job and Agent flows. Cross-machine rollout, signed Agent distribution and sustained multi-replica load remain unverified. +- Ollama is optional and experimental. A bounded local scenario demonstrates the evidence checks, with broader diagnosis quality still to evaluate. -In the API terminal, configure `NETGUARDIAN_OLLAMA_URL=http://127.0.0.1:11435` -and the installed model name in `NETGUARDIAN_OLLAMA_MODEL`. These are process-local -settings; the default Ollama tray service may still use its original GPU settings. -The official [hardware guide](https://docs.ollama.com/gpu) documents these GPU -selection controls. Stop the CPU server with Ctrl+C when no longer needed. +See the [delivery checklist](docs/product/DELIVERY_STATUS.md) for the current verification boundaries. -Real-model acceptance can be explicitly enabled using an existing verified report: +## Roadmap -```powershell -$env:NETGUARDIAN_TEST_MODEL_REPORT = "C:/path/to/exported-report.json" -$env:NETGUARDIAN_TEST_MODEL_URL = "http://127.0.0.1:11435" -$env:NETGUARDIAN_TEST_MODEL = "qwen2.5:1.5b" -uv run pytest integration/test_model_workflow.py -q -``` +- Publish a read-only demo of a controlled incident and recovery with clearly labeled data and a verifiable report. +- Verify Compose startup and migrations from a clean environment, then document the supported operating scope. +- Validate Agent enrollment, reconnect and two-location comparisons across separate test hosts. -This opt-in test calls the real model and checks persisted investigation access. -It does not certify diagnosis quality. Qwen2.5 0.5B produced unsupported causal -hypotheses in local testing and is not recommended as the default investigator. +## Author -Qwen2.5 1.5B passed the local CPU integration check against the recorded 503/204 -HTTPS workflow: both facts were correctly cited and no hypothesis was invented. -This single scenario does not establish general diagnostic reliability. +I am a Communication Engineering graduate with Huawei HCIP Datacom certification, +experience in 400G optical transport operations, and hands-on IT infrastructure +and network support experience. NetGuardian brings that troubleshooting practice +together with Python automation and a web-based evidence workflow. ## License -Project code and documentation use the [MIT License](LICENSE). See [NOTICE.md](NOTICE.md) -for third-party components and the unresolved legacy portrait asset scope. - -Current delivery target: **Windows local use**. Linux runtime support and release -validation for deployment are deferred. CI covers selected Linux test scenarios. - - -### Missing cloud-worker results - -Cloud jobs have a 120-second delivery-age limit, separate from the probe's -10-second execution deadline. Old queued jobs and expired leases become failed -without a measurement on the next job read, submission, worker claim or scheduler -tick. Active leases are not interrupted. An offline worker therefore does not -leave observed jobs queued forever, and restarting it will not execute stale work. -Expiry is evaluated when a service handles work; no clock runs while every service -is stopped. Missing measurements do not establish target failure or recovery. -The audit action is `cloud_job.delivery_expired`; each cleanup batch is bounded. - - -### Diagnostic API contract - -Task submission, history and details use generated OpenAPI types and an -`openapi-fetch` client. FastAPI validates their response models, including the -shared measurement envelope. After changing API schemas, run from the root: - -```sh -uv run python -m scripts.export_openapi -cd web -pnpm generate:api -pnpm typecheck -``` - -Commit both `web/openapi.json` and `web/lib/api-schema.d.ts` with the change. -CI checks the server export and regenerated types for drift. Schema export creates -an isolated temporary database, reads no operator environment and starts no server. -Other UI workflows still have manual TypeScript response types. Web tests use -Node.js 24's native TypeScript support; no new Python production dependency is added. - - -### Local source archive - -For a reviewed Windows source-only ZIP with a file checksum manifest, see -[packaging/README.md](packaging/README.md). This excludes dependencies, builds, -tests, test keys and local data. It is not a signed installer or a public release. - - -### Opt-in Streamlit V2 diagnostics - -The original direct probe tools remain disabled. To use the legacy page as a thin -client of the same V2 API/shared worker, start API/worker first, then in PowerShell: - -```powershell -$env:NETGUARDIAN_STREAMLIT_V2 = 'true' -$env:NETGUARDIAN_API_URL = 'http://127.0.0.1:8000' -uv run --locked streamlit run app.py --server.address 127.0.0.1 -``` - -Connect in the V2 diagnostics section with your workspace token. Select an approved -origin and probe, queue a durable job, then select its record and inspect/refresh. -Credentials remain in that Streamlit session and Disconnect V2 clears them. Do not -share browser sessions. The administrator controls the API origin; only HTTPS or -literal-loopback HTTP is accepted. Requests have a total deadline, byte cap, no -redirects and no ambient proxies. Targets are rechecked by the existing API/worker. -This adapter supplies on-demand cloud diagnostics/history, not the full monitoring, -Agent or investigation management UI; use Next.js for those. Unset the flag to -restore the default disabled legacy interface. - -Source archives r1/r2 omitted `.streamlit/config.toml`; when using either old archive, -always pass `--server.address 127.0.0.1` explicitly. The r3 source archive includes -the loopback/security configuration and this adapter. Prefer r3 or a later snapshot. +The project source and documentation are available under the [MIT License](LICENSE). +See [NOTICE.md](NOTICE.md) for third-party and legacy asset scope. diff --git a/README.zh-CN.md b/README.zh-CN.md new file mode 100644 index 0000000..ca027d8 --- /dev/null +++ b/README.zh-CN.md @@ -0,0 +1,127 @@ +# NetGuardian · 网络诊断与循证故障排查平台 + +[English](README.md) · **简体中文** + +NetGuardian 帮助 IT 支持人员排查服务为何无法访问。它逐层检查连接,并保留每次测量的结果,便于复盘故障和交接。 + +[快速开始](#快速开始) · [系统架构](#系统架构) · [演示证据](#演示证据) · [测试与验证](#测试与验证) + +## 项目缘起 + +当一台办公电脑能访问服务而前台设备却连不上时,工单通常无法说明问题出在 DNS、TCP 连接、TLS,还是应用本身。我开发 NetGuardian,是为了让这些排查步骤可以重复执行,也让接手的人能看到判断依据。 + +项目从一个 Python 和 Streamlit 工具发展而来,由我独立开发,设计受到 LAN/WLAN、TCP/IP、DNS、DHCP 和 Windows 终端支持工作的启发。本文演示使用受控测试环境。 + +## 核心能力 + +| 能力 | 可以查看的内容 | +| --- | --- | +| 分层诊断 | DNS 记录、TCP 连通性、TLS 握手和证书详情,以及 HTTP HEAD 状态和耗时。可按需启用 ICMP 和 traceroute。 | +| 定时监控 | 暂停和恢复检查,查看最近采集时间、故障及恢复记录。缺少测量时,界面会显示数据缺口。 | +| 双位置对比 | 将服务端检查与已注册的 Windows Agent 检查配对,结合测量端点和时间比较记录。 | +| 事件证据 | 打开脱敏报告快照,验证 SHA-256 摘要,并追溯报告引用的原始观测。 | +| 可选辅助分析 | 配置 Ollama 后可整理报告证据。界面分别展示经过证据校验的结论和待验证的假设。 | + +## 系统架构 + +```mermaid +flowchart LR + Operator["操作人员"] --> Web["Next.js 控制台"] + Web --> API["FastAPI 控制接口"] + API --> DB[("SQLite / PostgreSQL")] + Worker["后台 Worker"] --> DB + Worker --> Engine["共用探测引擎"] + Agent["仅主动出站的 Windows Agent"] -->|"通过 HTTPS 领取任务并上报结果"| API + Agent --> Engine + Engine --> Targets["已授权的目标"] +``` + +- [Web 控制台](web/)与 [FastAPI 接口](service/api.py)负责身份验证、作业提交和工作区数据查询。 +- [Worker](service/worker.py)领取有租约的后台作业并保存观测结果。[Windows Agent](agent/)通过 HTTPS 轮询任务,并独立执行本地目标策略。 +- [探测引擎](probe_core/)在规定的资源和时间范围内执行检查。[Alembic 迁移](migrations/)管理数据库结构,[报告导出](service/report_files.py)校验已保存快照的完整性。 + +[系统设计](docs/architecture/SYSTEM_DESIGN.md) · [数据模型](docs/architecture/DATA_MODEL.md) · [威胁模型](docs/security/THREAT_MODEL.md) + +## 工程实现 + +- **默认拒绝探测。** 服务端只允许预先批准的精确源地址。连接使用校验后的 IP,同时保留 HTTP 和 TLS 所需的主机名。HTTP 探测使用 HEAD,限制响应头大小,并禁止跟随重定向。 +- **分开记录各层耗时。** DNS、TCP、TLS 和 HTTP 保留各自的时间字段。ICMP 往返时延单独记录。 +- **异步执行与重试。** API 接收作业后,由 Worker 或 Agent 执行。作业使用租约与过期机制,Agent 在本地暂存结果,可重新投递已完成的观测。 +- **保留故障依据。** 报告记录当时引用的观测快照,导出后可用 [离线校验脚本](scripts/verify_report.py)核验 SHA-256 摘要。 + +[目标策略与测量说明](docs/operations/LOCAL_DEVELOPMENT.md#target-policy-and-measurement-semantics) + +## 演示证据 + +| 场景 | 验证记录 | +| --- | --- | +| 服务从 HTTP 503 恢复为 HTTP 204 | 受控的回环 HTTPS 服务真实返回两个状态码。本地浏览器验证了监控事件恢复及报告校验。[浏览器记录](docs/product/BROWSER_QA.md#populated-workflow-check-2026-11-05) | +| 下载事件报告 | 使用隔离测试工作区下载报告,并执行离线摘要校验。[导出验证记录](docs/product/BROWSER_QA.md#real-model-investigation-and-report-export-2026-11-06) | +| 查看缺少测量的历史记录 | 使用 101 条合成失败作业测试分页及缺失测量提示。[夹具验证记录](docs/product/BROWSER_QA.md#retained-history-production-web-build-2026-11-05) | + +## 技术栈 + +Python 3.11 · FastAPI · SQLAlchemy · Alembic · SQLite / PostgreSQL · Next.js · React · TypeScript · Node.js 24 · pnpm · pytest · Ruff · GitHub Actions + +## 快速开始 + +在 Windows 上安装 Python 3.11 和 Node.js 24,然后在 PowerShell 中执行: + +```powershell +git clone https://github.com/TYF009/NetGuardian.git +Set-Location NetGuardian +python -m pip install uv==0.13.7 +npm install -g pnpm@11.25.0 +uv sync --locked +pnpm --dir web install --frozen-lockfile +pnpm --dir web build + +$workspaceToken = (uv run --locked python -c "import secrets; print(secrets.token_urlsafe(32))").Trim() +$env:NETGUARDIAN_TOKENS = @{ $workspaceToken = 'local' } | ConvertTo-Json -Compress +$env:NETGUARDIAN_ORIGINS = '{}' +$env:NETGUARDIAN_VANTAGE_ID = [guid]::NewGuid().ToString() +$workspaceToken +uv run --locked python -m scripts.run_windows --migrate +``` + +打开[本地控制台](http://127.0.0.1:3000),输入 PowerShell 显示的测试工作区令牌。目标白名单为空时可以浏览界面。请使用全新检出目录,或在迁移前备份现有数据库。如果要跨会话保留观测并进行比较,请保持 Worker 位置标识稳定。 + +[本地开发与 Agent 注册指南](docs/operations/LOCAL_DEVELOPMENT.md) · [就绪检查、备份与故障排查手册](docs/operations/RUNBOOK.md) + +## 测试与验证 + +GitHub Actions 在 Windows 和 Ubuntu 上运行 Python 检查及选定的运行时测试,并验证 PostgreSQL 集成、Web 测试、TypeScript、OpenAPI 一致性和 Next.js 生产构建。本地 Windows 验证范围涵盖回环 TLS、Agent 作业、数据库及浏览器工作流。[CI 工作流](.github/workflows/ci.yml) · [Windows 交付记录](docs/product/WINDOWS_DELIVERY.md) + +可在仓库根目录复现: + +```sh +uv run --locked ruff check . +uv run --locked ruff format --check . +uv run --locked pytest -q +uv run --locked pytest integration -q -rs +pnpm --dir web test +pnpm --dir web typecheck +pnpm --dir web build +``` + +[集成测试所需环境](docs/operations/LOCAL_DEVELOPMENT.md#checks-and-operations) + +## 当前限制 + +- 当前交付范围为 Windows 本地评估。Docker Compose 部署和更完整的 Linux 运行验证仍待完成。 +- 回环测试覆盖了 Worker 和 Agent 工作流。跨主机部署、Agent 签名分发和持续多副本负载仍待验证。 +- Ollama 辅助分析为可选实验功能,目前只验证过有限的本地场景。 + +[交付状态与验证边界](docs/product/DELIVERY_STATUS.md) + +## 路线图 + +- 发布只读故障与恢复演示,注明数据来源并提供可校验报告。 +- 在干净环境中验证 Compose 启动和数据库迁移,记录支持范围。 +- 在独立测试主机之间验证 Agent 注册、重连和双位置测量。 + +## 作者与许可 + +我拥有通信工程背景和华为 HCIP Datacom 认证,并有 400G 光传送网络以及 IT 基础设施运维经验。NetGuardian 将这些排查经验与 Python 自动化和 Web 证据工作流结合起来。 + +项目代码和文档采用 [MIT 许可证](LICENSE)。第三方组件和历史素材的授权范围见 [NOTICE.md](NOTICE.md)。 diff --git a/docs/operations/LOCAL_DEVELOPMENT.md b/docs/operations/LOCAL_DEVELOPMENT.md new file mode 100644 index 0000000..77d3d35 --- /dev/null +++ b/docs/operations/LOCAL_DEVELOPMENT.md @@ -0,0 +1,460 @@ +# Local Development and Operator Reference + +See the [project overview](../../README.md) for architecture, verification and release scope. + +## Local development setup + +Prerequisites: Python 3.11, uv 0.13.7, Node.js 24 or later, pnpm 11.25.0. +The lockfiles are `uv.lock` and `web/pnpm-lock.yaml`. + +```sh +git clone https://github.com/TYF009/NetGuardian.git +cd NetGuardian +python -m pip install uv==0.13.7 +uv sync --locked +``` + +Configure the following environment in the API, worker and migration shells: + +| Variable | Meaning | +|---|---| +| `NETGUARDIAN_TOKENS` | JSON object mapping random ASCII bearer tokens (32–512 characters) to workspace IDs; required by API/worker | +| `NETGUARDIAN_ORIGINS` | JSON object mapping workspace IDs to exact approved HTTP(S) origins; defaults to no targets | +| `NETGUARDIAN_DATABASE_URL` | SQLAlchemy URL; defaults to `sqlite:///netguardian.db` for local development | +| `NETGUARDIAN_VANTAGE_ID` | Required by the worker: stable non-nil UUID identifying its network location | +| `NETGUARDIAN_ENABLE_SYSTEM_PROBES` | Set to `true` on the worker/agent to enable fixed ICMP/traceroute tools | + +Generate a token locally with `python -c "import secrets; print(secrets.token_urlsafe(32))"`. +Keep it out of source control and shell history. A token mapping has the shape +`{"":"local"}`. An origin mapping has the shape +`{"local":["https://:443"]}`. These are placeholders, not usable +credentials or authorization to probe third-party services. Start with `{}` origins +until you have approved a target you own or are authorized to diagnose. +Configuration files named `.env` are **not loaded automatically** by the Python CLI. + +From the repository root, migrate before starting either service: + +```sh +uv run --locked alembic upgrade head +uv run --locked uvicorn service.api:app_factory --factory --host 127.0.0.1 --port 8000 +``` + +Generate a probe-location UUID once with `python -c "import uuid; print(uuid.uuid4())"` +and set `NETGUARDIAN_VANTAGE_ID` in the worker environment. Preserve it across restarts. +Workers sharing one network location may share it; distinct locations must use distinct +IDs. This is an operator assertion, not automatic location attestation. Old records +retain the legacy shared ID and cannot distinguish historical worker deployments. + +In another shell with the same configuration and repository working directory: + +```sh +uv run --locked python -m service.worker +``` + +In a third shell: + +```sh +cd web +pnpm install --frozen-lockfile +pnpm dev +``` + +Open [the local interface](http://127.0.0.1:3000) and enter the workspace token. +The Web proxy uses `http://127.0.0.1:8000` by default. Override `NETGUARDIAN_API_URL` +only with a trusted API origin. Browser users cannot select the proxy destination. +The API buffers at most 64 KiB per request before JSON parsing, with a 10-second +body-receive deadline (including chunked uploads). This does not replace deployment +connection limits or general request throttling. +Tokens stay in React memory; disconnect remounts the workspace and cancels requests. +A browser abort does not roll back an operation already committed by the API. + +For a local production build, run `pnpm build` then `pnpm start` inside `web/`. +Keep the development services on loopback. Non-loopback operation needs verified +TLS and the unfinished production controls listed in the delivery checklist. + +## Windows local launcher + +After installing the locked Python dependencies, installing Web dependencies and +running `pnpm build` in `web/`, configure the environment variables described above +(including a stable vantage ID). From the repository root: + +```powershell +uv run --locked python -m scripts.run_windows --migrate +``` + +`--migrate` explicitly upgrades the configured database before startup; back up an +existing database first. Omit it on later starts. The launcher requires an existing +production Web build and starts API, worker and Web on fixed loopback ports 8000 +and 3000. It refuses occupied ports, missing configuration or outdated schemas. +It does not install dependencies, generate credentials, approve targets, start an +Agent or configure Ollama. Browser access still requires your workspace token. +The worker uses the configured target policy; use empty origins for a no-probe trial. + +`--check` validates prerequisites without starting services or migrating. For a +bounded smoke run, use `--run-seconds 5`; the timer starts after API and Web are ready. +The worker process is supervised, but this is not a worker health/throughput check. +Ctrl+C or a child-process exit stops the process trees created by this launcher. +Shutdown is forced: in-flight work relies on existing leases and retry rules. +The launcher joins an unnamed Windows Job Object before spawning children; its +termination also terminates owned descendants, including after force-kill. It +refuses startup if the host prevents this containment. This is not a Windows +Service manager or a signed installer. + +Per-run logs remain under ignored `.netguardian-local/`; review and protect them as +local operational data. Credentials are not printed or written by the launcher; +the Web child does not inherit NetGuardian backend configuration or credentials. +Rebuild Web after source changes. This launcher has been smoke-tested on Windows; +it does not change the deferred Linux support scope. + +## Target policy and measurement semantics + +Cloud policy defaults to deny. Only administrator-configured exact origins are +accepted; mixed public/private DNS answers, special addresses, credentials in URLs, +ambiguous hosts and unsupported schemes are rejected. Connections use validated +numeric addresses while retaining the original hostname for Host/SNI and certificate +verification. HTTP uses HEAD, downloads no body, caps headers at 16 KiB and does not +follow redirects. Ambient proxies are not used. The overall probe deadline is at most +10 seconds; only the first approved address is attempted. + +| Measurement | Meaning | +|---|---| +| `dns_lookup_ms` | Resolution and address-policy planning duration | +| `tcp_connect_ms` | TCP connection establishment duration | +| `tls_handshake_ms` | TLS stream setup/handshake duration | +| `http_ttfb_ms` | Time from sending HEAD until the first response byte | +| `http_total_ms` | Total probe time through complete HTTP response headers, including setup | +| `icmp_rtt_ms` | Exact ICMP echo RTT when the OS reports one; less-than bounds remain separate samples | + +HTTP duration is not ICMP RTT. A cloud result describes the worker's network, not the +browser's. TLS evidence does not prove service ownership or certificate revocation +checking. Agent observations are self-reported. + +System probes use canonical numeric addresses and fixed executable/argument lists, +never arbitrary shell commands. ICMP sends one echo; traceroute uses at most 15 hops. +The process runner caps execution at nine seconds and output at 64 KiB. Missing tools +or recognized permission failures are explicit. No automatic privilege elevation is +performed. Locale variations can cause parse failures. Production cloud policy still +rejects loopback; integration tests use an isolated test-only policy. + +## Monitoring and comparisons + +Create a monitor with an approved target, probe kind and cloud/agent vantage. +Intervals are at least 60 seconds. Schedules run independently of browser sessions; +at most one unfinished run exists per monitor. Each workspace currently permits +20 monitors, 20 pending jobs and 60 newly created jobs per rolling 60 seconds. +Manual submissions, diagnostic pairs and schedules share the rate budget, including +completed jobs. Pairs consume two slots. API rejection returns 429/Retry-After; schedules +wait without a catch-up burst. This is not a complete HTTP request-rate limiter. + +Pause prevents new scheduled jobs; queued/running jobs may finish. Resume rechecks +authorization, resets consecutive counters and starts a current cycle without replaying +missed intervals. HTTP 4xx/5xx is unhealthy; policy denial and unsupported capability +are not network outages. Missing results break consecutive counters without recovering +open incidents. Collection freshness is distinct from target health. + +New incidents include the failures that triggered their threshold and subsequent +observations through recovery. Missing measurements are listed explicitly. Links retain +at most the latest 1000 observations; older incidents are not automatically backfilled. + +Compare can queue matching cloud/agent jobs atomically, consuming two pending slots. +Execution is asynchronous. Numeric deltas require compatible target, kind, configuration, +probe version and destination, distinct vantages, successful observations and at most +30 seconds of skew. DNS results record A/AAAA addresses, observed TTL, canonical name and resolver endpoint. +DNS timing deltas remain withheld for every probe kind: an endpoint does not +establish equivalent cache state. Other comparable timings include an explicit +reminder that HTTP total duration includes resolution and connection setup. +IP literal targets skip DNS and leave `dns_lookup_ms` absent. +Differences do not establish causality. + +Monitor **View history** retrieves the latest 100 runs in chronological order via +`GET /api/monitors/{id}/runs`. Choose one timing metric for a scatter plot and an +accessible evidence table. Missing timings are not zero and points are not joined +across gaps. The time axis is server job creation time, not claimed agent observation +time. History is an explicit snapshot; reopen it to refresh. Timing alone does not +establish target health. The initial chart uses a small SVG component rather than a +charting dependency; richer zoom/aggregation is still future work. + +## Optional local agent + +Copy `agent/config.example.json` to an ignored local configuration and set its HTTPS +control API origin. Explicitly list allowed probe origins. Private addresses require +explicit RFC1918/IPv6 ULA `private_cidrs`; loopback, link-local and metadata addresses +remain prohibited. Server jobs cannot expand local policy. The server policy must also +authorize the origin. + +Issue a ten-minute single-use enrollment token from Agents, then run: + +```sh +uv run --locked python -m agent.client --config agent/config.local.json --enroll +uv run --locked python -m agent.client --config agent/config.local.json +``` + +Enter the enrollment token at the hidden prompt. The agent credential is stored in +Windows Credential Manager or Linux Secret Service/KWallet, not the JSON file. The +agent fails closed without a supported store, uses outbound verified HTTPS, and opens +no listener. Revocation stops access, pauses its monitors and terminates pending jobs +without fabricating results. Uploads must match the job, attempt, configuration digest +and lease; identical duplicate uploads are accepted. + +Control responses have a 64 KiB limit and 20-second total deadline; compressed or +malformed responses are rejected. One result is retried in memory, or in an opt-in bounded local spool; see the runbook. +Credential rotation is available through `--rotate-credential`; see the runbook. +Signed distribution and service installers remain unfinished. +Without a configured spool, restarting the agent can lose its pending result. Do not claim offline durability. + +## Reports, AI and audit + +The Reports view reopens the latest 100 saved snapshots without regenerating them. +Readable reports show incident times, recorded findings with in-page evidence citations, +missing observations, limitations and per-observation timings. Raw JSON remains available +for inspection and download. +Reports store redacted snapshots with SHA-256 checksums. Query strings and free-form +errors are removed; paths and addresses remain operational data to review before sharing. +Reads and AI submission recompute the checksum; mismatch returns HTTP 409. The checksum +is not a signature or protection against changing both content and digest. + +Optional AI requires administrator-set `NETGUARDIAN_OLLAMA_URL` and +`NETGUARDIAN_OLLAMA_MODEL`. Remote endpoints require HTTPS; loopback HTTP is allowed. +No model is downloaded automatically. The adapter uses Ollama +[structured output](https://docs.ollama.com/capabilities/structured-outputs) via +[chat](https://docs.ollama.com/api/chat), with a 30-second deadline and 64 KiB response cap. +Only the latest 30 selected evidence records are submitted, excluding URLs, IPs and +endpoint free text. Facts must match recorded values; hypotheses require citations and +remain unverified. Model tools are prohibited. Missing/rejected model output remains +explicit; deterministic reports still work. Real inference quality is not yet evaluated. + +Investigations reopens saved output and source reports without calling the model again. +Audit shows the latest 100 covered operator events. Logs identify roles, not individual +humans, and are not tamper-proof. See the [runbook](../operations/RUNBOOK.md) for scope. + +## Checks and operations + +```sh +uv run --locked ruff check . +uv run --locked ruff format --check . +uv run --locked python -m compileall -q app.py diagnostics.py probe_core service agent migrations tests integration +uv run --locked pytest -q +uv run --locked pytest integration/test_tls.py integration/test_process_cleanup.py -q +``` + +The default unit suite blocks network and process execution. Integration tests are +separate: TLS uses loopback fixtures; process tests launch fixed local Python fixtures. +Windows system-tool tests require `NETGUARDIAN_TEST_SYSTEM_PROBES=true`. +PostgreSQL tests require a disposable `netguardian_test` database and +`NETGUARDIAN_TEST_POSTGRES_URL`; absent configuration produces skips, not verification. +GitHub Actions runs Python checks on Windows and Ubuntu, Web checks on Ubuntu, and PostgreSQL integration tests. See the [CI workflow](../../.github/workflows/ci.yml) for the test matrix. + +Back up existing databases before migration. Do not blindly stamp schemas from old +unversioned development snapshots. Docker Compose configuration is provided but remains +unverified locally. See the [operations runbook](../operations/RUNBOOK.md). + +## Legacy Streamlit and project documents + +The repaired legacy UI remains available with +`uv run --locked streamlit run app.py --server.address 127.0.0.1`. +Its probe entry points are intentionally disabled; use the V2 API/Web path for approved +probes. Streamlit keeps default CORS/XSRF protections. Historical HTTP timing must never +be relabeled as ICMP or component timings. + +- [Product requirements](../product/PRD.md) +- [System design](../architecture/SYSTEM_DESIGN.md) +- [Data model](../architecture/DATA_MODEL.md) +- [Threat model](../security/THREAT_MODEL.md) +- [Implementation plan and original audit](../product/IMPLEMENTATION_PLAN.md) +- [Current delivery gates](../product/DELIVERY_STATUS.md) +- [Browser QA scope](../product/BROWSER_QA.md) + +See [NOTICE.md](../../NOTICE.md) for licensing scope and the optional legacy portrait. + +The Web API proxy bounds POST/PATCH bodies to 8 KiB of raw UTF-8 bytes and a +10-second read deadline, including chunked requests. Encoded/compressed bodies +are rejected. Run its focused regression checks with `cd web` and `pnpm test`. + +Agent jobs have a 120-second delivery window from creation. The scheduler (or an +agent poll) marks overdue unclaimed work and exhausted retries as failed without +a measurement, records an audit event, and releases monitor queue capacity. +A live 30-second lease is allowed to finish. Missing delivery breaks consecutive +incident evidence; it never establishes target failure or recovery. + + +The opt-in local transport integration suite includes +`uv run --locked pytest integration/test_agent_workflow.py -q`. It starts an ephemeral +loopback HTTPS API and HTTP HEAD fixture, enrolls an agent, executes two real probes +(503 then 204), reloads a pending upload from disk, verifies duplicate acceptance, +and checks incident recovery and report evidence. It also executes an atomic cloud/agent +diagnostic pair through both real transports and compares their recorded measurements. Test-only DNS mapping, loopback +address authorization and a private test CA are scoped to the test; production +policies and system certificate trust are unchanged. This is not evidence of a +cross-machine agent installation, all crash boundaries, Linux keyring behavior +or PostgreSQL operation. No public target is contacted. + +Probe version 0.2.1 measures durations with Python `perf_counter`, a high-resolution +monotonic clock. This avoids the coarse Windows Python 3.11 `monotonic` timer. +Historical measurements are preserved; different probe versions are not compared. + +Diagnostic pairs now have persistent IDs and a workspace-scoped history in the +Compare page. Reloading the page does not lose pair membership. Existing jobs are +not retroactively grouped. Generic retention preserves paired evidence; automated +pair expiration is not yet implemented. Apply database migrations before startup. + +TLS certificate/hostname validation failures use `tls_verification_failed`, distinct +from generic transport errors. When available, `error.tls_verify_code` preserves the +TLS runtime verification code; its interpretation is runtime-specific. Failed +verification does not yield verified TLS certificate metadata or an HTTP status. + +Control API admission is process-local: 32 concurrent API requests, a 120-request +burst replenished at 60/second, plus a 60-mutation workspace burst replenished at +1/second. Agent poll/result traffic uses only the global budget. Health endpoints +remain accessible. These are protective defaults, not benchmarked capacity or +aggregate limits across replicas; 429 responses carry `Retry-After`. + + +Windows PostgreSQL 17.11 has now passed local schema, concurrent-worker, +concurrent-startup and tenant-constraint tests, plus the real HTTPS Agent/report/ +comparison workflow. A local dump/restore comparison matched all 12 public tables. +See `docs/operations/RUNBOOK.md` for exact scope and limitations; Docker and Linux +execution are not inferred from these results. The HTTPS integration test now runs +on SQLite and, when the explicitly named disposable PostgreSQL database is configured, +on PostgreSQL as well; otherwise its PostgreSQL case is skipped. + + +Setting `NETGUARDIAN_TEST_KEYRING=true` also opts the HTTPS workflow into a real OS +credential-store rotation check. It creates a randomly named test service, commits +a server credential rotation while retaining the old local credential and pending +rotation entry, then runs normal recovery. It verifies old-token rejection, +new-token access, revocation, and cleanup of both temporary credential entries. +This passed with Windows Credential Manager. It reproduces lost-acknowledgement +state; it does not inject an actual dropped socket or terminate an agent process. + +The model adapter requests identity encoding and rejects compressed responses +before consuming their bodies. Its 64 KiB response cap applies to raw streamed +bytes before JSON decoding. Model transport tests do not establish model quality. + + +The Agent CLI holds an OS file lock for its control origin and agent ID before +opening the keyring or processing jobs. Duplicate instances fail instead of racing +spool writes or credential rotation. Lock files live under `~/.netguardian/locks`; +they remain after exit, but ownership is released by the OS when the process dies. +Do not delete lock files while an agent is running. This coordinates processes +under the same OS user; separate users/machines must not share one enrollment. + + +### Long incident history + +Run `uv run alembic upgrade head` before starting this version. Incident links are +now stored independently of the latest-1000 summary. Authenticated control API +clients can read `/api/incidents/{id}/evidence?after=0&limit=100`, following +`next_after` until it is null. Earlier links already discarded by older versions +cannot be reconstructed. Reports remain bounded snapshots. In the Incidents tab, choose **Browse retained +history** to navigate pages of 100 links and inspect individual observations. Linked jobs are protected from generic retention; +plan storage accordingly. Downgrading populated incident history is refused. + + +### Local Ollama GPU compatibility fallback (Windows) + +The local validation machine could not load Ollama 0.40.14's CUDA runner +(`device kernel image is invalid`). CPU inference worked without driver changes. +To run a separate loopback-only CPU instance, use a PowerShell terminal: + +```powershell +$env:CUDA_VISIBLE_DEVICES = "-1" +$env:OLLAMA_VULKAN = "0" +$env:OLLAMA_HOST = "127.0.0.1:11435" +& "$env:LOCALAPPDATA/Programs/Ollama/ollama.exe" serve +``` + +In the API terminal, configure `NETGUARDIAN_OLLAMA_URL=http://127.0.0.1:11435` +and the installed model name in `NETGUARDIAN_OLLAMA_MODEL`. These are process-local +settings; the default Ollama tray service may still use its original GPU settings. +The official [hardware guide](https://docs.ollama.com/gpu) documents these GPU +selection controls. Stop the CPU server with Ctrl+C when no longer needed. + +Real-model acceptance can be explicitly enabled using an existing verified report: + +```powershell +$env:NETGUARDIAN_TEST_MODEL_REPORT = "C:/path/to/exported-report.json" +$env:NETGUARDIAN_TEST_MODEL_URL = "http://127.0.0.1:11435" +$env:NETGUARDIAN_TEST_MODEL = "qwen2.5:1.5b" +uv run pytest integration/test_model_workflow.py -q +``` + +This opt-in test calls the real model and checks persisted investigation access. +It does not certify diagnosis quality. Qwen2.5 0.5B produced unsupported causal +hypotheses in local testing and is not recommended as the default investigator. + +Qwen2.5 1.5B passed the local CPU integration check against the recorded 503/204 +HTTPS workflow: both facts were correctly cited and no hypothesis was invented. +This single scenario does not establish general diagnostic reliability. + +## License + +Project code and documentation use the [MIT License](../../LICENSE). See [NOTICE.md](../../NOTICE.md) +for third-party components and the unresolved legacy portrait asset scope. + +Current delivery target: **Windows local use**. Linux runtime support and release +validation for deployment are deferred. CI covers selected Linux test scenarios. + + +### Missing cloud-worker results + +Cloud jobs have a 120-second delivery-age limit, separate from the probe's +10-second execution deadline. Old queued jobs and expired leases become failed +without a measurement on the next job read, submission, worker claim or scheduler +tick. Active leases are not interrupted. An offline worker therefore does not +leave observed jobs queued forever, and restarting it will not execute stale work. +Expiry is evaluated when a service handles work; no clock runs while every service +is stopped. Missing measurements do not establish target failure or recovery. +The audit action is `cloud_job.delivery_expired`; each cleanup batch is bounded. + + +### Diagnostic API contract + +Task submission, history and details use generated OpenAPI types and an +`openapi-fetch` client. FastAPI validates their response models, including the +shared measurement envelope. After changing API schemas, run from the root: + +```sh +uv run python -m scripts.export_openapi +cd web +pnpm generate:api +pnpm typecheck +``` + +Commit both `web/openapi.json` and `web/lib/api-schema.d.ts` with the change. +CI checks the server export and regenerated types for drift. Schema export creates +an isolated temporary database, reads no operator environment and starts no server. +Other UI workflows still have manual TypeScript response types. Web tests use +Node.js 24's native TypeScript support; no new Python production dependency is added. + + +### Local source archive + +For a reviewed Windows source-only ZIP with a file checksum manifest, see +[packaging/README.md](../../packaging/README.md). This excludes dependencies, builds, +tests, test keys and local data. It is not a signed installer or a public release. + + +### Opt-in Streamlit V2 diagnostics + +The original direct probe tools remain disabled. To use the legacy page as a thin +client of the same V2 API/shared worker, start API/worker first, then in PowerShell: + +```powershell +$env:NETGUARDIAN_STREAMLIT_V2 = 'true' +$env:NETGUARDIAN_API_URL = 'http://127.0.0.1:8000' +uv run --locked streamlit run app.py --server.address 127.0.0.1 +``` + +Connect in the V2 diagnostics section with your workspace token. Select an approved +origin and probe, queue a durable job, then select its record and inspect/refresh. +Credentials remain in that Streamlit session and Disconnect V2 clears them. Do not +share browser sessions. The administrator controls the API origin; only HTTPS or +literal-loopback HTTP is accepted. Requests have a total deadline, byte cap, no +redirects and no ambient proxies. Targets are rechecked by the existing API/worker. +This adapter supplies on-demand cloud diagnostics/history, not the full monitoring, +Agent or investigation management UI; use Next.js for those. Unset the flag to +restore the default disabled legacy interface. + +Source archives r1/r2 omitted `.streamlit/config.toml`; when using either old archive, +always pass `--server.address 127.0.0.1` explicitly. The r3 source archive includes +the loopback/security configuration and this adapter. Prefer r3 or a later snapshot. diff --git a/docs/product/DELIVERY_STATUS.md b/docs/product/DELIVERY_STATUS.md index bbd4204..d0ed072 100644 --- a/docs/product/DELIVERY_STATUS.md +++ b/docs/product/DELIVERY_STATUS.md @@ -9,7 +9,7 @@ The latest consolidated local regression is recorded in VERIFICATION.md. | Area | Current evidence | Remaining release gate | |---|---|---| | Core policy | Negative-input tests; pinned-transport doubles; Windows IPv4/IPv6 loopback real TLS trust/hostname/fingerprint tests | Remaining protocol fields, Linux TLS/IPv6 platform checks | -| API and jobs | Auth, persistence and tenant tests; concurrent workspace initialization tested on SQLite; Windows PostgreSQL 17.11 migration/concurrency/tenant and real HTTPS workflow tests passed; remote CI not run | Remaining audit coverage, aggregate multi-replica admission controls and broader concurrent-load verification (controlled PostgreSQL retention/scheduler overlap passed) | +| API and jobs | Auth, persistence and tenant tests; concurrent workspace initialization tested on SQLite; Windows PostgreSQL 17.11 migration/concurrency/tenant and real HTTPS workflow tests passed; Windows and Ubuntu CI passed | Remaining audit coverage, aggregate multi-replica admission controls and broader concurrent-load verification (controlled PostgreSQL retention/scheduler overlap passed) | | Monitoring | Schedule/threshold/recovery and authenticated pause/resume tests | Broader concurrent-load verification (controlled retention/scheduler overlap passed) and cross-machine agent schedule verification (real loopback HTTPS scheduling/incident recovery passed) | | Web | TypeScript/build; local browser login, navigation, empty states and disconnect (BROWSER_QA.md) | Broader responsive/accessibility QA beyond the tested 320/390 px and keyboard flows (creation, pair submission, Agent revocation, saved-pair comparison, freshness, pause/resume, history, report and disconnect inspected) | | Agent | Enrollment/replay/revoke and local policy tests; Windows real HTTPS enrollment-to-report loopback workflow with spool reload/duplicate replay passed | Linux OS-keyring checks (Windows write/read/cleanup passed), real process-crash and cross-machine recovery verification (Windows OS-keyring plus HTTPS lost-acknowledgement-state recovery and revocation passed), signed distribution | @@ -17,12 +17,28 @@ The latest consolidated local regression is recorded in VERIFICATION.md. | Reports | Immutable snapshot/redaction/citation tests | Broader export-size/archival policy (retained history pagination and actual download checksum verified) | | Comparison | Conservative evidence comparison tests; real HTTPS cloud-worker/agent paired execution and comparison passed on Windows loopback | Cross-machine paired cloud/agent runtime and resolver cache comparability (endpoint/TTL recorded; atomic enqueue tested) | | AI investigation | Real Windows CPU Ollama/Qwen2.5 1.5B evidence validation and API persistence/isolation passed; failure fallback unit tests | Broader diagnosis-quality evaluation (persisted history/source-report browser QA passed) | -| Operations | Compose and runbook written | Docker Compose and Linux/remote CI verification (portable Windows PostgreSQL startup and backup/restore content comparison passed) | +| Operations | Compose and runbook written | Docker Compose startup and Linux deployment verification (portable Windows PostgreSQL startup and backup/restore content comparison passed) | | Open source release | Repository public | MIT selected; legacy portrait excluded pending redistribution-rights confirmation | Do not mark the product complete while required gates lack evidence. Do not infer Linux, PostgreSQL, deployment or performance results from Windows/SQLite tests. +## Portfolio verification checkpoint (2026-11-10) + +- The [main baseline GitHub Actions run](https://github.com/TYF009/NetGuardian/actions/runs/42789779017) + passed on Windows and Ubuntu, including Web and disposable PostgreSQL jobs. +- Local Windows verification for this branch passed 175 Python unit tests, + 21 runtime integration cases and nine Web tests. Seven PostgreSQL cases and + one real Ollama case were skipped locally because those fixtures were absent. + Ruff lint and format, OpenAPI drift, TypeScript and the Next.js production + build passed. The Windows launcher reached readiness using a fresh temporary + SQLite database, a test token and an empty target allowlist. +- Both Web container stages now use Node.js 24, matching the package engine and + CI major version. Docker was unavailable for a local image build. Clean Compose + startup, Linux deployment and cross-machine Agent operation remain open. +- The optional legacy portrait has been removed from the current source tree + pending redistribution approval. See [NOTICE.md](../../NOTICE.md). + ## Agent single-instance checkpoint The Agent now holds an OS-owned file lock for its server/identity while running. @@ -182,7 +198,7 @@ Web tests, three relevant backend tests, schema check and production build passe Compile-only checks reject arbitrary probe kinds and omitted required parameters. The generated contract exposed optional timing fields; missing values are skipped, not displayed as zero. Other page response types remain manual. Node.js 24 is now -the documented/tested Web toolchain and CI version. Remote CI has not run. +the documented/tested Web toolchain and CI version. See the portfolio checkpoint above for the current CI result. ## Agent delivery loop restart branches diff --git a/packaging/source-files.txt b/packaging/source-files.txt index 520ed22..4acfd67 100644 --- a/packaging/source-files.txt +++ b/packaging/source-files.txt @@ -6,6 +6,7 @@ Dockerfile LICENSE NOTICE.md README.md +README.zh-CN.md agent/__init__.py agent/client.py agent/config.example.json @@ -18,6 +19,7 @@ compose.yaml diagnostics.py docs/architecture/DATA_MODEL.md docs/architecture/SYSTEM_DESIGN.md +docs/operations/LOCAL_DEVELOPMENT.md docs/operations/RUNBOOK.md docs/product/BROWSER_QA.md docs/product/DELIVERY_STATUS.md diff --git a/web/Dockerfile b/web/Dockerfile index 42b2f05..54b2a09 100644 --- a/web/Dockerfile +++ b/web/Dockerfile @@ -1,4 +1,4 @@ -FROM node:22-slim AS build +FROM node:24-slim AS build ENV NEXT_TELEMETRY_DISABLED=1 WORKDIR /app RUN npm install -g pnpm@11.25.0 @@ -6,7 +6,7 @@ COPY package.json pnpm-lock.yaml pnpm-workspace.yaml ./ RUN pnpm install --frozen-lockfile COPY . . RUN pnpm build -FROM node:22-slim +FROM node:24-slim ENV NODE_ENV=production NEXT_TELEMETRY_DISABLED=1 WORKDIR /app COPY --from=build --chown=node:node /app ./