infra

Declarative infrastructure for Patchwork Labs: every Patchwork host, the NixOS modules for Patchwork services, the Ansible for the Windows hosts and AD, and their secrets.

Layout

Path	Contents
`flake.nix`	Hosts, exported modules, deploy-rs nodes, checks.
`hosts/<name>/`	One directory per lab host.
`hosts/proxmox-vm.nix`	Hardware and disk layout shared by every VM.
`hosts/installer.nix`	The installer ISO: NixOS minimal with the admin SSH keys on root.
`bin/install-host`	Plans and applies the VM, reads its address, runs nixos-anywhere.
`bin/new-quilt-staging-secrets`	Writes the two Quilt staging secrets with a new random password and `SECRET_KEY_BASE`. Prints no value.
`bin/new-krater-secrets`	Writes the two secrets of Krater staging or production with a new random database password and keys. Prints no value.
`hosts/central-db-schema.nix`	Apps and shared schemas of the central databases.
`hosts/alastor/`	Modules that `jaspermayone/infra` imports for alastor.
`modules/common/`	Accounts, SSH, sudo, Tailscale, Nix. Every host imports it. `keys.nix` holds the SSH keys.
`modules/ad-join/`	Joins the host to `patchwork.internal` with sssd. `modules/common` imports it.
`modules/auto-upgrade/`	Switches the host to each new commit on `main`. `modules/common` imports it.
`modules/central-db/`	One central PostgreSQL cluster and its access model.
`modules/weave/`	The Weave service module.
`modules/quilt/`	The Quilt service module.
`modules/krater/`	The Krater service module (portal, worker, screenshot storage; Krater Compute off by default). Exported as `nixosModules.krater`.
`modules/stash/`	The Stash service module (web, worker, archived files on a host directory).
`hosts/hank/stash.nix`	Stash production and staging on hank: instance settings and secrets.
`bin/new-stash-secrets`	Writes the two secrets of Stash staging or production with a new random database password and `SECRET_KEY_BASE`. Prints no value.
`modules/selvage/`	The Selvage service module (web and worker, production and an optional staging instance).
`hosts/twill/selvage.nix`	Selvage production and staging on twill: instance settings and secrets.
`bin/new-selvage-secrets`	Writes Selvage's restricted database password and env file. Reads the existing central password with your age identity. Prints no value.
`bin/new-selvage-staging-secrets`	The same for Selvage staging: the restricted staging password (hem:5433) and the staging env file. Reads the existing central staging password with your age identity. Prints no value.
`bin/set-env-secret`	Sets keys in an agenix env file, for example a Weave client id and secret. Asks for each value with the echo off. Prints no value.
`hosts/weft/krater.nix`	Krater staging and production on weft: instance settings and secrets. Production is off until `hosts/weft/krater-production.nix` turns it on.
`tests/`	NixOS VM tests.
`secrets/`	agenix secrets. `secrets.nix` lists the recipients of each file.
`ansible/`	Ansible for the Windows hosts and AD: `windows.yml`, `ad.yml`. See `ansible/README.md`.
`terraform/workspace/`	Terraform for Google Workspace: org units, groups, group settings. See `terraform/workspace/README.md`.
`ansible/vault/`	Git submodule `patchworklabsorg/infra-vault` (private): the Ansible vault.
`ad/`	The AD structure: `org.json` and `converge.ps1`. See `ad/README.md`.
`terraform/`	Terraform roots for resources that an API manages. See `terraform/README.md`.
`workspace/`	GAM script that sets up Google Workspace.
`terraform/github/`	Terraform for the `patchworklabsorg` GitHub org: repos, teams, members, rulesets. See `terraform/github/README.md`.
`terraform/proxmox/`	Terraform root for the Patchwork NixOS VMs on pve01. See `terraform/proxmox/README.md`.

Hosts

The lab hosts below are NixOS VMs on pve01, VLAN 90 (patchwork.internal). alastor is the only public edge. It terminates TLS and proxies to the lab hosts over the tailnet.

Host	VMID	Address	Memory	Runs
`warp`	356	10.100.90.56	6 GB	Misc: redirects and other low-risk shared things. No Docker. nginx on `tailscale0:80` for alastor.
`skein`	355	10.100.90.55	4 GB	`patchwork_central`, prod. Tailnet only.
`seam`	357	10.100.90.57	2 GB	`patchwork_central_restricted`, prod. Tailnet only.
`hem`	358	10.100.90.58	2 GB	Both staging clusters: central on 5432, restricted on 5433. Quilt staging uses the restricted one. Selvage staging uses both.
`batting`	360	10.100.90.60	4 GB	Quilt prod + staging. nginx on `tailscale0:80` for alastor.
`weft`	361	10.100.90.61	4 GB	Krater staging, and Krater production (off until its switch). nginx on `tailscale0:80` for alastor.
`hank`	362	10.100.90.62	6 GB	Stash prod + staging (declared, not built yet). 80 GB disk for the archived files. nginx on `tailscale0:80` for alastor.
`twill`	363	10.100.90.63	3 GB	Selvage prod + staging (declared, not built yet). nginx on `tailscale0:80` for alastor, Selvage's public paths only.

The Windows hosts are managed by Ansible (ansible/):

Host	VMID	Address	Runs
`loom`	310	10.100.90.10	DC, PDC emulator, DNS, DHCP
`spindle`	311	10.100.90.11	DC, DNS
`bobbin`	312	10.100.90.12	Issuing CA
`shuttle`	314	10.100.90.14	NPS / RADIUS

hem, batting and weft are built (2026-10-21 and 2026-10-22). weft has 4 GB and a 40 GB disk, because Krater builds its image on the host and runs SeaweedFS.

flowchart LR
  internet((Internet)) --> alastor["alastor (Traefik, TLS, Weave)"]
  alastor -- "tailnet, HTTP :80" --> warp["warp: misc"]
  alastor -- "tailnet, HTTP :80" --> batting["batting: Quilt"]
  alastor -- "tailnet, HTTP :80" --> weft["weft: Krater staging"]
  alastor -- "tailnet, HTTP :80" --> hank["hank: Stash (not built yet)"]
  batting -- "tailnet :5432" --> seam["seam: patchwork_central_restricted"]
  batting -. "staging :5433" .-> hem["hem: :5432 / :5433"]
  weft -. "staging :5432" .-> hem
  hank -- "tailnet :5432" --> skein["skein: patchwork_central"]
  hank -. "staging :5432" .-> hem
  alastor -- "tailnet, HTTP :80" --> twill["twill: Selvage (not built yet)"]
  twill -- "tailnet :5432" --> skein
  twill -- "tailnet :5432" --> seam

Login

Every host joins the patchwork.internal AD forest (modules/ad-join), with the same rules as roles/linux_ad_join in the jaspermayone/infra Ansible:

Members of lnx-<host>-users log in with an SSH key from their altSecurityIdentities (SSHKey:<key>). Passwords do not work over SSH.
Members of lnx-<host>-admins get sudo with their AD password.
The local accounts in modules/common are the break-glass way in.

Staff reach the lab over the thimble VPN or over Tailscale. thimble has WireGuard (one key per device) and, after the rollout in ansible/README.md ("VPN"), IKEv2 with AD login: NPS on shuttle checks the password and the vpn-users group. Use the daily account. Admin accounts in Protected Users cannot log in to the VPN.

Before a host is installed:

Add the host to hosts in flake.nix. Then create lnx-<host>-users and lnx-<host>-admins in AD: cd ansible && ansible-playbook ad.yml -t host-groups. The play reads the host names from the flake. If a group does not resolve, every AD login fails.
The same run adds the host's A record in AD DNS, from patchwork.lanAddress in its config (ad.yml -t host-dns runs only that part). sssd does not update DNS.
After the install, add the host key to secrets/ad-join.age and run agenix -r. ad-join.service then joins on the next boot or deploy.

loom and spindle must run. They are the DNS servers, the KDCs and the time source of every host.

Login banners

The lab hosts show a sewing-theme banner, in the style of thimble's. One banner shows before the SSH login. One shows after it. modules/banner makes both.

The banner is on by default. flake.nix adds the module to every host in hosts, so a new host gets it with no change. alastor does not get it, because modules/banner is not in nixosModules.

Give a new host its role in hosts/<host>/configuration.nix. Without a role, the banners show only the host name.

patchwork.banner.role = "app host"; # short, lower case

To turn the banner off for a host, set patchwork.banner.enable = false;.

The host name comes from networking.hostName. The option patchwork.banner.art replaces the art after login. Check the result with nix eval --raw .#nixosConfigurations.<host>.config.users.motd. The check banner tests the seam banner and the default.

Deploys from main

The lab hosts follow main of this repo. Every 5 minutes, patchwork-upgrade.timer reads the head of main. If the host runs another revision and CI (.github/workflows/ci.yml) passed on that commit, the host builds the commit and switches to it (modules/auto-upgrade). So a merged PR reaches every lab host a few minutes after CI passes. A red main stops every host. alastor is not a lab host. It follows jaspermayone/infra (see "Weave on alastor").

Krater staging also follows main of patchworklabsorg/krater. krater-follow.timer on weft reads the head of main every 5 minutes. When all of the commit's GitHub checks passed, it builds the commit and restarts Krater staging. Krater production does not follow a branch.

flowchart LR
  pr["PR merged to infra main"] --> ici["infra CI passes"]
  ici --> up["patchwork-upgrade.timer on each lab host"]
  up -- "nixos-rebuild switch --flake github:patchworklabsorg/infra/<sha>" --> host["lab host"]
  kpr["PR merged to krater main"] --> ci["krater CI passes"]
  ci --> follow["krater-follow.timer on weft"]
  follow -- "source-rev, krater-build, restart krater-web" --> staging["krater-staging.patchworklabs.org"]

Both timers pull from GitHub. GitHub needs no key and no way in to the lab.

Item	Infra	Krater staging
Log	`journalctl -u patchwork-upgrade`	`journalctl -u krater-follow`
Hold a host on another commit	`sudo touch /var/lib/patchwork-upgrade/hold`	`sudo touch /var/lib/krater/follow-hold`
A commit that fails	3 tries, then it waits for a new commit (`/var/lib/patchwork-upgrade/failed-rev`)	3 tries, then it waits for a new commit (`/var/lib/krater/follow-failed-rev`). After a failed build or start, the old version runs again.

Remove the failed-rev file to try a commit again. A manual deploy .#<host> still works, but the next run switches the host back to main. Create the hold file first to keep a branch on a host. Remove the hold file when you are done.

Install a host

The installer ISO has the admin SSH keys on root. Nobody types in the Proxmox console. The keys come from modules/common/keys.nix, the same file the host accounts use.

Build the ISO: nix build .#installer-iso -o result-installer. The build needs an x86_64-linux builder. On macOS, use a Linux builder in /etc/nix/machines, or pass --builders 'ssh-ng://<user>@<linux-host> x86_64-linux'. You can also run the command on a Linux host.
Create or update the VM: cd terraform/proxmox && tofu apply. Terraform uploads the ISO and boots the VM from it. The guest agent reports the DHCP address.
Read the address: tofu output ipv4_addresses. If the list is empty, wait one minute, run tofu refresh, and read again.
Install: nixos-anywhere --flake .#<host> --build-on remote --target-host root@<ip>.
Add the host key to secrets/ad-join.age. See "Login".

bin/install-host <host> runs steps 2 to 4. It stops if the plan deletes a resource.

Central databases

See the header of modules/central-db/default.nix for the full model. In short:

Each app gets a login role and a schema with the same name. The app owns the schema and runs its own migrations there.
A shared schema (for example people) has one owner app. Other apps get <schema>_read or <schema>_write through hosts/central-db-schema.nix.
Nobody can create objects in public. Nobody can connect to a database without a grant.
The restricted database is a separate cluster on a separate VM.

To add an app, edit hosts/central-db-schema.nix, add its password secrets, and deploy the DB hosts.

How the repos connect

This repo is public. Nothing here depends on jaspermayone/infra, which is private, so every org admin can build this repo.

flowchart LR
  pinfra["patchworklabsorg/infra (this repo)"]
  jinfra["jaspermayone/infra (private)"]
  pinfra -- "nixosModules.weave-alastor, edge-alastor" --> jinfra
  jinfra -- "deploy-rs" --> alastor
  pinfra -- "deploy-rs" --> lab["warp, skein, seam, hem"]

Repo	Owns
`patchworklabsorg/infra`	Every Patchwork host, Patchwork services and their secrets.
`patchworklabsorg/core`	Archived. It held the Core VM, which was never built; `warp` does that job.
`patchworklabsorg/dns`	Patchwork DNS.
`patchworklabsorg/infra-vault`	Private. The Ansible vault, as the submodule `ansible/vault`.
`jaspermayone/infra`	Jasper's hosts (alastor and others) and the lab's physical layer. It imports Patchwork services from this repo as the flake input `patchwork-infra`.

Weave on alastor

alastor is Jasper's VPS. jaspermayone/infra owns the host. This repo owns everything about Weave on it: the module, the instance settings, the CI deploy key, the Traefik routes and the secrets.

To change Weave on alastor:

Merge the change here.
In jaspermayone/infra, run nix flake update patchwork-infra.
Deploy alastor from jaspermayone/infra (deploy .#alastor).

alastor also rebuilds itself at 04:00 from jaspermayone/infra main. It uses the revision in that repo's flake.lock, so a merge here does nothing on alastor until step 2.

Krater on weft

Krater staging runs on weft. Krater production is configured on weft too, and its secrets exist, but it is off until hosts/weft/krater-production.nix turns it on (the checklist is in hosts/weft/krater.nix). weft is its own VM, so the people who run Krater cannot read the other apps' secrets. alastor routes krater-staging.patchworklabs.org and krater-staging-storage.patchworklabs.org to nginx on weft over the tailnet (nixosModules.edge-alastor).

Krater has no registry image. krater-build fetches one commit of patchworklabsorg/krater and builds it on weft with the repo's own Dockerfile. A deploy writes the commit to /var/lib/krater/source-rev, then runs systemctl start krater-build (the running site keeps serving) and systemctl restart krater-web. Production is the same with krater-prod in place of krater (/var/lib/krater-prod/source-rev, krater-prod-build, krater-prod-web). So Krater releases do not need a change here. Only the module and its settings do. Staging does this by itself for each commit on Krater's main (krater-follow, see "Deploys from main").

Staging deploys itself. The Deploy workflow in patchworklabsorg/krater runs after CI passes on main. It joins the tailnet as tag:ci-krater (an ephemeral node, OAuth client in the krater repo secrets), then sends the commit over sshd on weft port 2222 as krater-deploy. The key may run only the deploy command, which does the three steps above and puts the old commit back if the build fails. The public key is in hosts/weft/krater.nix. tag:ci-krater reaches tag:patchwork-krater:2222 and nothing else (jaspermayone/infra tailscale/policy.hujson). Production has no CI deploy yet.

krater-web requires krater-build, so a deploy that turns an instance on waits for its first build (minutes; TimeoutStartSec is 45 minutes). Run the build by hand first, or deploy with --magic-rollback false and a long --activation-timeout.

Krater staging's database is its schema krater in patchwork_central on hem (port 5432), like the other central apps. The module runs no Postgres of its own. weft reaches hem over the tailnet (tag:patchwork-krater to hem:5432). Sign-in is Weave staging.

Before Krater staging runs: bin/new-krater-secrets staging (writes krater-staging-env.age and central-staging-krater.age), hem deployed with Krater's role password, Krater's app in Weave staging (client id in hosts/weft/krater.nix, client secret and Slack values in krater-staging-env.age), and DNS for its two hostnames (krater-staging, krater-staging-storage, proxied to alastor). Krater's compute stays off, so no Vast key is needed. Krater's docs/dev/production.md has the full checklist.

Quilt on batting

Quilt runs on batting as two Docker containers, like Weave. hosts/batting/quilt.nix is the instance. modules/quilt is the service module. nginx on batting fronts the ports, and alastor routes quilt.patchworklabs.org and quilt-staging.patchworklabs.org to batting (nixosModules.edge-alastor). The contract with the app is docs/deploy.md in the quilt repo.

Item	Value
URL	`https://quilt.patchworklabs.org` (Traefik router `quilt`, option `traefik.hosts`)
Units	`quilt-web` (`./bin/rails server` on `127.0.0.1:3005`, migrates on start) and `quilt-worker` (`bundle exec good_job start`)
Image	`ghcr.io/patchworklabsorg/quilt:main` (option `image`). Pin it to `:<sha>` for a known build.
Database	`patchwork_central_restricted` on seam, role and schema `quilt`, over the tailnet
Secrets	`quilt-env.age`, `quilt-ghcr-token.age` (batting), `restricted-prod-quilt.age` (seam)

flowchart LR
  internet((Internet)) --> traefik["alastor: Traefik"]
  traefik -- "tailnet :80" --> nginx["batting: nginx"]
  nginx -- "127.0.0.1:3005" --> web[quilt-web]
  worker[quilt-worker] -- "waits for /up" --> web
  web -- "tailnet :5432" --> seam["seam: patchwork_central_restricted, schema quilt"]
  worker -- "tailnet :5432" --> seam

Both containers use the host network, so they reach seam.wildebeest-stargazer.ts.net through batting's Tailscale (tag:patchwork-quilt). The worker waits until GET /up on the web server answers. The web server answers only after the migrations ran.

RESTRICTED_DATABASE_URL in quilt-env.age holds the database password. It must be the same password as in restricted-prod-quilt.age, which seam sets on the role quilt.

To deploy a new main image, run sudo systemctl restart quilt-web on batting. That restarts the worker too. To change the module, follow the same steps as for Weave above.

Quilt staging

Quilt staging is a second instance on batting, next to production. It uses the Mercury sandbox, Weave staging and the restricted staging cluster on hem. It can never move real money: with RAILS_ENV=staging, Quilt refuses to boot unless MERCURY_API_BASE is the sandbox URL (patchworklabsorg/quilt#35). The option is staging in modules/quilt. hosts/batting/quilt.nix turns it on.

Item	Production	Staging
URL	`https://quilt.patchworklabs.org`	`https://quilt-staging.patchworklabs.org` (router `quilt-staging`)
Units	`quilt-web`, `quilt-worker`	`quilt-staging-web`, `quilt-staging-worker`
Port	`127.0.0.1:3005`	`127.0.0.1:3006`
`RAILS_ENV`	`production`	`staging`
Image	`ghcr.io/patchworklabsorg/quilt:main` (option `image`)	`ghcr.io/patchworklabsorg/quilt:main` (option `staging.image`)
Mercury	`https://api.mercury.com/api/v1/`, live tokens	`https://api-sandbox.mercury.com/api/v1/` (set in Nix), sandbox tokens
Weave	Weave production	`https://weave-staging.patchworklabs.org` (set in Nix), a Weave staging app
Database	seam:5432, `patchwork_central_restricted`	hem:5433, `patchwork_central_restricted` (the restricted staging cluster)
Env file	`quilt-env.age`	`quilt-staging-env.age`
Role password	`restricted-prod-quilt.age` (seam)	`restricted-staging-quilt.age` (hem)
Registry token	`quilt-ghcr-token.age`	The same file

flowchart LR
  internet((Internet)) --> traefik["alastor: Traefik"]
  traefik -- "tailnet :80" --> nginx["batting: nginx"]
  nginx -- "127.0.0.1:3006" --> web[quilt-staging-web]
  worker[quilt-staging-worker] -- "waits for /up" --> web
  web -- "tailnet :5433" --> hem["hem: restricted-staging container, schema quilt"]
  worker -- "tailnet :5433" --> hem
  worker -- sandbox only --> mercury["api-sandbox.mercury.com"]
  web -- OIDC --> weave["weave-staging.patchworklabs.org"]

MERCURY_API_BASE and WEAVE_ISSUER are in staging.extraEnv, not in the env file. extraEnv overrides the env file, so a value in the env file cannot point staging at the live Mercury API.

The database URL in quilt-staging-env.age:

RESTRICTED_DATABASE_URL=postgres://quilt:<password>@hem.wildebeest-stargazer.ts.net:5433/patchwork_central_restricted?sslmode=disable

<password> must be the value in restricted-staging-quilt.age. hem decrypts that file to /run/central-db-passwords/patchwork_central_restricted/quilt and binds the directory into the restricted-staging container, where central-db-setup sets the password on the role quilt. hosts/hem/configuration.nix explains why hem does not use /run/agenix for this. hem has no server TLS. Tailscale encrypts the path, so the URL uses sslmode=disable.

To deploy a new main image to staging, run sudo systemctl restart quilt-staging-web on batting. That restarts the staging worker too.

Go live

Do the steps in this order.

Merge patchworklabsorg/quilt#35. Wait until image.yml pushes ghcr.io/patchworklabsorg/quilt:main. An older image has no staging environment and does not boot.
Free 2 GiB of memory on pve01. Make sure that VMID 358 and 10.100.90.58 are free.
Prepare AD for hem. Run cd ansible && ansible-playbook ad.yml -t host-groups. Add the A record for hem in AD DNS. See "Login".
Create the VM: cd terraform/proxmox && tofu apply -target='proxmox_virtual_environment_vm.nixos["hem"]'. Read the address with tofu output ipv4_addresses.
Install hem: nixos-anywhere --flake .#hem --build-on remote --target-host root@<ip>.
Join hem to the tailnet. Make sure that the tailnet policy lets batting (tag:patchwork-quilt) reach hem:5433.
Get the host key: ssh-keyscan -t ed25519 10.100.90.58. Add it as hem in secrets/secrets.nix. Add hem to restricted-staging-quilt.age and ad-join.age. Remove the TODO.
Create the secrets: nix develop -c bin/new-quilt-staging-secrets. The script writes restricted-staging-quilt.age and quilt-staging-env.age with a random password and SECRET_KEY_BASE. It leaves the Weave client and the Mercury tokens empty.
In Weave staging (https://weave-staging.patchworklabs.org), register Quilt as a confidential app:
- Redirect URI: https://quilt-staging.patchworklabs.org/auth/callback.
- Scopes: openid profile email roles patches directory introspect. A superadmin allows directory and introspect. Quilt asks for patches at sign-in once patchworklabsorg/quilt#63 is deployed: without it in the app's scopes, every sign-in fails with invalid_scope.
- App roles: viewer, finance, admin.
In the Mercury sandbox dashboard (Settings, API Tokens), create a read-only token. Staging refuses to boot without MERCURY_API_TOKEN.
Add the values: cd secrets && agenix -e quilt-staging-env.age. Set WEAVE_CLIENT_ID, WEAVE_CLIENT_SECRET and MERCURY_API_TOKEN. Keep QUILT_TOPUPS=off.
Re-key: cd secrets && agenix -r. Commit the .age files and merge.
Deploy hem: deploy .#hem. In the container, make sure that central-db-setup succeeded and does not log "no password file for quilt": sudo nixos-container run restricted-staging -- journalctl -u central-db-setup.
From batting, test the connection: psql "<RESTRICTED_DATABASE_URL>" -c 'SHOW search_path'. The result must be quilt.
Merge the DNS record quilt-staging (A 129.213.163.213, proxied): patchworklabsorg/dns#11.
Deploy batting (deploy .#batting). alastor routes quilt-staging to batting (edge-alastor).
On batting, check systemctl status quilt-staging-web quilt-staging-worker and curl -i http://127.0.0.1:3006/up. Then sign in at https://quilt-staging.patchworklabs.org.

To test top-ups on staging, add a sandbox MERCURY_WRITE_TOKEN with batting's public IP on its allowlist, and set QUILT_TOPUPS to approval_only.

Selvage on twill

Selvage, the Patchwork Slack bot, runs on twill as two Docker containers, like Stash on hank. hosts/twill/selvage.nix is the instance. modules/selvage is the service module. The staging instance is in "Selvage staging" below.

Item	Value
URL	`https://selvage.patchworklabs.org` (router `selvage-twill` in `edge-alastor`). nginx on twill passes `POST /slack/*`, `GET /up`, and the moderator web UI under `/admin`, `/auth/`, `/assets/` and `/cable`. The app checks the weave role on every `/admin` request. Every other path is 404.
Units	`selvage-web` (`./bin/rails db:migrate` in a one-off container, then `./bin/rails server` on `127.0.0.1:3009`) and `selvage-worker` (`bundle exec good_job start`, on the web container's image)
Image	`ghcr.io/patchworklabsorg/selvage:main` (option `image`). Pin it to `:<sha>` for a known build.
Databases	`patchwork_central` on skein and `patchwork_central_restricted` on seam. Role and schema `selvage` in each.
Secrets	`selvage-env.age`, `selvage-ghcr-token.age` (twill), `central-prod-selvage.age` (skein), `restricted-prod-selvage.age` (seam)
Tailnet	`tag:patchwork` and `tag:patchwork-selvage`. The second tag reaches skein:5432 and seam:5432 only.

flowchart LR
  slack((Slack)) --> traefik["alastor: Traefik"]
  traefik -- "tailnet :80" --> nginx["twill: nginx, /slack/ and /up only"]
  nginx -- "127.0.0.1:3009" --> web[selvage-web]
  worker[selvage-worker] -- "waits for /up" --> web
  web -- "tailnet :5432" --> skein["skein: patchwork_central, schema selvage"]
  web -- "tailnet :5432" --> seam["seam: patchwork_central_restricted, schema selvage"]
  worker --> skein
  worker --> seam

The env file has CENTRAL_DATABASE_URL and RESTRICTED_DATABASE_URL. Their passwords must be the values in central-prod-selvage.age and restricted-prod-selvage.age. skein and seam set those values on the role selvage. The role has no CREATE on either database. selvage-web runs db:migrate, never db:prepare, and the migrations must not run CREATE SCHEMA or enable_extension.

The units do not start while CENTRAL_DATABASE_URL, RESTRICTED_DATABASE_URL, SECRET_KEY_BASE, SLACK_SIGNING_SECRET or SLACK_BOT_TOKEN is missing or empty in the env file (option requiredEnv). An empty signing secret would let anyone sign a Slack request.

To deploy a new main image, run sudo systemctl restart selvage-web on twill. That restarts the worker too.

Go live

Do the steps in this order. Steps 5 to 11 are on this repo's branch, before the merge.

Merge the tailnet policy change in jaspermayone/infra (tag:patchwork-selvage, with grants to skein:5432 and seam:5432).
Merge the DNS record selvage (proxied to alastor) in patchworklabsorg/dns.
In patchworklabsorg/selvage, merge the image workflow and make sure that image.yml pushed ghcr.io/patchworklabsorg/selvage:main. The migrations must not run CREATE SCHEMA or enable_extension.
Make sure that VMID 363 and 10.100.90.63 are free, and that pve01 has 3 GiB of memory to spare (terraform/proxmox/README.md, "Memory warning").
Prepare AD for twill: run .\converge.ps1 (it creates twill-admins), then cd ansible && ansible-playbook ad.yml -t host-groups. See "Login".
Create and install the VM: bin/install-host twill.
On twill, join the tailnet with both tags: sudo tailscale up --advertise-tags=tag:patchwork,tag:patchwork-selvage.
Get the host key: ssh-keyscan -t ed25519 10.100.90.63. Add it as twill in secrets/secrets.nix. Add twill to selvage-env.age, selvage-ghcr-token.age and ad-join.age. Remove the "not built yet" notes.
Write the secrets: nix develop -c bin/new-selvage-secrets. It writes restricted-prod-selvage.age, selvage-env.age and selvage-ghcr-token.age. It decrypts central-prod-selvage.age and quilt-ghcr-token.age with your age identity (~/.ssh/id_ed25519, or AGE_IDENTITY).

Re-encrypt only ad-join.age, so that twill can decrypt it. Do not run agenix -r: it re-encrypts every file, and the hash-based restart triggers then restart Stash and Krater with no change.

cd secrets
age -d -i ~/.ssh/id_ed25519 ad-join.age | age -e -o ad-join.age.new \
  -R <(nix eval --raw -f secrets.nix '"ad-join.age".publicKeys' --apply 'k: builtins.concatStringsSep "\n" k')
mv ad-join.age.new ad-join.age

Fill in the Slack and Weave values. Selvage does not start without SLACK_SIGNING_SECRET and SLACK_BOT_TOKEN: bin/set-env-secret selvage-env.age SLACK_SIGNING_SECRET SLACK_BOT_TOKEN SLACK_ADMIN_TOKEN WEAVE_CLIENT_ID WEAVE_CLIENT_SECRET SELVAGE_AUDIT_CHANNEL. Add SLACK_BROWSER_TOKEN and SLACK_BROWSER_COOKIE the same way when you need them.
Commit the .age files and secrets.nix, and merge after CI passes. The lab hosts follow main, so seam creates the role selvage with its password, and twill starts Selvage. To do it at once: deploy .#seam, then deploy .#twill.
On seam, make sure that central-db-setup succeeded: journalctl -u central-db-setup. On twill, check systemctl status selvage-web selvage-worker and curl -i http://127.0.0.1:3009/up.
Route the public name: in jaspermayone/infra, run nix flake update patchwork-infra, then deploy .#alastor.
Point the Slack app's request URLs at https://selvage.patchworklabs.org/slack/events, /slack/commands and /slack/interactions.

Selvage staging

Selvage staging is a second instance on twill, next to production. It serves the staging Slack workspace (the Slack developer sandbox) through its own Slack app. It uses Weave staging and both staging clusters on hem. The option is staging in modules/selvage. hosts/twill/selvage.nix turns it on.

Item	Production	Staging
URL	`https://selvage.patchworklabs.org` (router `selvage-twill`)	`https://selvage-staging.patchworklabs.org` (router `selvage-staging-twill`)
nginx vhost on twill	`selvage`	`selvage-staging`, from the shared list `selvageLocations`
Units	`selvage-web`, `selvage-worker`	`selvage-staging-web`, `selvage-staging-worker`
Port	`127.0.0.1:3009`	`127.0.0.1:3010`
`RAILS_ENV`	`production`	`staging`
Image	`ghcr.io/patchworklabsorg/selvage:main` (option `image`)	The same image (option `staging.image`)
Memory	No limit	512 MB for each container, no swap (option `staging.memory`)
Slack	The Patchwork workspace, the production Slack app	The staging workspace, a separate staging Slack app
Weave	`https://weave.patchworklabs.org`	`https://weave-staging.patchworklabs.org` (set in Nix), a Weave staging app
Central database	skein:5432, `patchwork_central`	hem:5432, `patchwork_central`
Restricted database	seam:5432, `patchwork_central_restricted`	hem:5433, `patchwork_central_restricted`
Env file	`selvage-env.age`	`selvage-staging-env.age`
Role passwords	`central-prod-selvage.age` (skein), `restricted-prod-selvage.age` (seam)	`central-staging-selvage.age`, `restricted-staging-selvage.age` (hem)
Registry token	`selvage-ghcr-token.age`	The same file

flowchart LR
  slack((Staging Slack workspace)) --> traefik["alastor: Traefik"]
  traefik -- "tailnet :80, Host selvage-staging" --> nginx["twill: nginx, vhost selvage-staging"]
  nginx -- "127.0.0.1:3010" --> web[selvage-staging-web]
  worker[selvage-staging-worker] -- "waits for /up" --> web
  web -- "tailnet :5432" --> central["hem: central-staging container, schema selvage"]
  web -- "tailnet :5433" --> restricted["hem: restricted-staging container, schema selvage"]
  worker --> central
  worker --> restricted
  web -- OIDC --> weave["weave-staging.patchworklabs.org"]

WEAVE_ISSUER and SELVAGE_HOSTS are in staging.extraEnv, not only in the env file. extraEnv overrides the env file, so a value in the env file cannot point staging at production Weave.

The staging units have the same checks as production: they do not start while a key in requiredEnv is missing or empty in the staging env file.

Memory: twill has 3 GB and zram swap. Production has no limit, so it keeps the room it has today. Each staging container (web, worker and the one-off migration) runs with --memory 512m --memory-swap 512m. Staging can then use at most about 1 GB at a time. If staging needs more, raise staging.memory in hosts/twill/selvage.nix, and check free -m on twill first.

To deploy a new main image to staging, run sudo systemctl restart selvage-staging-web on twill. That restarts the staging worker too.

Staging go live

Do "Selvage on twill" first: twill must exist, with its host key in secrets/secrets.nix. Then do these steps in this order.

In patchworklabsorg/selvage, merge the staging Rails environment (config/environments/staging.rb). Wait until image.yml pushes ghcr.io/patchworklabsorg/selvage:main. An older image has no staging environment and does not boot.
In jaspermayone/infra tailscale/policy.hujson, let tag:patchwork-selvage reach hem:5432 and hem:5433. Merge it.
Merge the DNS record selvage-staging (proxied to alastor) in patchworklabsorg/dns.
In the staging Slack workspace, create the staging Slack app. Use config/slack_manifest.yml of the selvage repo, with each selvage.patchworklabs.org changed to selvage-staging.patchworklabs.org. Install it in the staging workspace. The workspace is Enterprise Grid, so admin.* scopes exist there. Do not add them: production does not have them. See docs/deploy.md in the selvage repo.
In Weave staging (https://weave-staging.patchworklabs.org), register Selvage as a confidential app with the redirect URI https://selvage-staging.patchworklabs.org/auth/callback and the same scopes as the production Selvage app.
Add twill to selvage-staging-env.age in secrets/secrets.nix, if it is not there yet.
Write the secrets: nix develop -c bin/new-selvage-staging-secrets. It writes restricted-staging-selvage.age and selvage-staging-env.age. It decrypts central-staging-selvage.age with your age identity (~/.ssh/id_ed25519, or AGE_IDENTITY). To put the staging Slack and Weave values in at once, export them first with a STAGING_ prefix (STAGING_SLACK_BOT_TOKEN, STAGING_WEAVE_CLIENT_ID, ...). The script reads only the prefixed names, so production values left in the shell do not leak into staging.
Fill in the values that are still missing: bin/set-env-secret selvage-staging-env.age SLACK_SIGNING_SECRET SLACK_BOT_TOKEN SLACK_ADMIN_TOKEN WEAVE_CLIENT_ID WEAVE_CLIENT_SECRET SELVAGE_AUDIT_CHANNEL.
Commit the .age files, and merge after CI passes. hem sets the password of the role selvage on 5433, and twill starts Selvage staging. Do not run agenix -r: the new files already have their recipients.
On hem, make sure that central-db-setup does not log "no password file for selvage": sudo nixos-container run restricted-staging -- journalctl -u central-db-setup -b -n 30. Read only the lines after the deploy: earlier runs logged that line before the file existed.
On twill, check systemctl status selvage-staging-web selvage-staging-worker and curl -i http://127.0.0.1:3010/up.
Route the public name: in jaspermayone/infra, run nix flake update patchwork-infra, then deploy .#alastor.
In the staging Slack app, verify the request URL https://selvage-staging.patchworklabs.org/slack/events.

Commands

nix flake check
nix fmt
nix build .#checks.x86_64-linux.central-db   # VM test; needs a Linux builder with kvm
deploy .#<host>                               # deploy-rs over the tailnet
cd secrets && agenix -e <name>.age   # create or edit a secret
cd secrets && agenix -r              # re-key after changing recipients
git submodule update --init ansible/vault    # the Ansible vault (private repo)
cd ansible && ansible-playbook windows.yml   # Windows baseline
cd ansible && ansible-playbook ad.yml -t audit   # AD report, changes nothing

Navigation Menu

Repository navigation

About

Repository files navigation

infra

Layout

Hosts

Login

Login banners

Deploys from main

Install a host

Central databases

How the repos connect

Weave on alastor

Krater on weft

Quilt on batting

Quilt staging

Go live

Selvage on twill

Go live

Selvage staging

Staging go live

Commands

About

Releases

Packages

Used by

Contributors

Languages

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
ad		ad
ansible		ansible
bin		bin
hosts		hosts
modules		modules
secrets		secrets
terraform		terraform
tests		tests
workspace		workspace
.gitattributes		.gitattributes
.gitignore		.gitignore
.gitmodules		.gitmodules
LICENSE.md		LICENSE.md
README.md		README.md
flake.lock		flake.lock
flake.nix		flake.nix

Latest commit

History

Folders and files

Repository files navigation

infra

Layout

Hosts

Login

Login banners

Deploys from main

Install a host

Central databases

How the repos connect

Weave on alastor

Krater on weft

Quilt on batting

Quilt staging

Go live

Selvage on twill

Go live

Selvage staging

Staging go live

Commands

About

Resources

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages