[andrewthomaslee]

Summary

New opt-in module hostSpec.system.crashDiagnostics. It makes a machine that crashes, resets or is killed without a shutdown leave evidence. It was prompted by kamrui-h1 resetting itself today: the journal just stopped, nothing landed in pstore, and the only clue was the AMD firmware's software wrote 0x6 to reset control register 0xCF9 at the next boot.

 flake-parts/nixosModules/
+├── crash-diagnostics.nix      # options + kernel/journald/watchdog/systemd wiring
+└── _crash-diagnostics.sh      # crash-report (boot report, CLI) and crash-sampler
 vm-tests/
+└── crash-diagnostics.nix      # clean shutdown vs hard reset (machine.crash())
 documentation/docs/
+└── crash-diagnostics.md       # settings table, reading a report and a verdict
 tests/fixtures/nixos.nix       # module enabled for eval-all-enabled
 documentation/flake-module.nix # nav entry

What enabling it does (all kernel/journald values are mkDefault):

Option	Default	Effect
dmesgOnReboot	on	printk.always_kmsg_dump=1: kernel log tail to pstore on every kernel reboot/power-off, archived by systemd-pstore
panicOnHang	on	nmi_watchdog=panic softlockup_panic=1
panicRebootSeconds	10	kernel.panic (default 0 = frozen forever)
panicOnOops	off	kernel.panic_on_oops
watchdogTimeout	60s	systemd.watchdog.runtimeTime (hardware watchdog)
journalSyncInterval	10s	journald SyncIntervalSec (default 5 min)
sampler	on, 30 s	one journal line: load, PSI, free mem, swap, hottest sensor, biggest process
keepReports	50	reports kept in /var/log/crash-diagnostics

boot
  systemd-pstore            → /var/lib/systemd/pstore (previous kernel log tail)
  crash-diagnostics-report  → crash-report --boot
    verdict: clean if logind SD_MESSAGE_SHUTDOWN or PID 1 shutdown target logged
    reset reason (AMD "Previous system reset reason")
    kernel messages by class, last samples, last 30 lines, pstore tail
    → /var/log/crash-diagnostics/<time>-<boot id>.txt + one journal line (warning if unclean)

crash-report [-N] reads any boot on demand. kdump (boot.crashDump) is left out on purpose: it drops into a rescue shell.

Evidence

• Before: diagnosing today's kamrui-h1 reset took a dozen manual journalctl/last queries. pstore was empty, panic=0, the watchdog was off, and the last ~40 s of logs were lost.
  After: the report script run read-only on kamrui-h1's real journal:

  == boot -1 (372f9806…)
  last log  2026-11-13 15:17:27 CST
  next boot 2026-11-13 15:27:48 CST (621 s later)
  verdict   UNCLEAN: the journal stops without a requested shutdown (crash, reset, power loss or a forced reboot)
  reset     x86/amd: Previous system reset reason [0x00080800]: software wrote 0x6 to reset control register 0xCF9
  == kernel messages by class (boot -1)
  filesystem   51  last: … fs-verity (nvme0n1p4, inode 5932797): FILE CORRUPTED! …
  power or ACPI 2  last: … ACPI Error: Aborting method \_SB.PCI0.SBRG.EC0._REG …
  

  and for the boot before (clean): verdict clean: System is powering down. / reset … ACPI power state transition occurred.
• vm-test crash-diagnostics: passed (5 subtests). They cover kernel/journald/watchdog settings (i6300esb), no report on the first boot, sampler lines, a clean shutdown → clean, and machine.crash() → UNCLEAN with sampler lines in the report, plus crash-report -1/-2/--list. The first run failed because a oneshot without RemainAfterExit goes inactive and wait_for_unit can't see it; that's fixed.
• fmt: no changes. nix flake check: all checks passed (incl. eval-all-enabled, treefmt/shellcheck, docs, vm-tests-eval).
• Not tested: a real EFI pstore write on reboot or panic (QEMU in the test has no EFI pstore), and a real watchdog reset.

Merge Danger

Door: two-way

Off until a machine sets hostSpec.system.crashDiagnostics.enable. Disabling it reverts every setting. Reports and pstore archives are plain files.

Blast Radius: opt-in

Once enabled on a machine: a kernel hang now panics and reboots after 10 s instead of freezing (intended). The hardware watchdog resets the machine if systemd is stuck for 60 s. Each kernel reboot writes a few EFI variables (deleted again by systemd-pstore at boot). The sampler adds about 2,900 short journal lines a day. No consumer changes: home enables it per machine in its own PR after this merges and nix flake update modules.

🤖 Generated with Claude Code