diff --git a/documentation/docs/crash-diagnostics.md b/documentation/docs/crash-diagnostics.md new file mode 100644 index 0000000..89104fb --- /dev/null +++ b/documentation/docs/crash-diagnostics.md @@ -0,0 +1,69 @@ +# Crash diagnostics + +`hostSpec.system.crashDiagnostics` makes a machine that crashes, resets +or is killed without a shutdown leave enough evidence to say why. It +was written after a desktop reset itself with a journal that just +stopped: no shutdown, no panic, nothing in pstore, only the AMD +firmware's `software wrote 0x6 to reset control register 0xCF9` at the +next boot. + +```nix +hostSpec.system.crashDiagnostics.enable = true; +``` + +## What it changes + +| Setting | Default | Effect | +|---|---|---| +| `dmesgOnReboot` | `true` | `printk.always_kmsg_dump=1`: the kernel log tail goes to pstore (EFI variables) on every reboot and power-off the kernel performs, not only on a panic. systemd-pstore moves it to `/var/lib/systemd/pstore` at the next boot. | +| `panicOnHang` | `true` | `nmi_watchdog=panic softlockup_panic=1`: a hung kernel panics, so its log reaches pstore. | +| `panicRebootSeconds` | `10` | `kernel.panic`: a panic reboots after 10 s instead of freezing. | +| `panicOnOops` | `false` | `kernel.panic_on_oops`. | +| `watchdogTimeout` | `"60s"` | systemd feeds the hardware watchdog; if it stops for this long, the watchdog resets the machine (AMD firmware then reports a watchdog reset). `null` leaves the watchdog alone. | +| `journalSyncInterval` | `"10s"` | journald `SyncIntervalSec` (5 min by default): less of the journal is lost in a hard reset. | +| `sampler.enable`, `sampler.interval` | `true`, `"30s"` | `crash-sampler` writes load, pressure (PSI), free memory, swap, the hottest sensor and the biggest process to the journal. | +| `keepReports` | `50` | Boot reports kept in `/var/log/crash-diagnostics`. | + +All kernel and journald values are `mkDefault`, so a machine can still +set them itself. + +## Reading the evidence + +At every boot `crash-diagnostics-report.service` writes how the previous +boot ended to `/var/log/crash-diagnostics/