diff --git a/docs/decisions/ADR-12_PostgreSQL_18_and_Data_Directory_Upgrade.md b/docs/decisions/ADR-12_PostgreSQL_18_and_Data_Directory_Upgrade.md
new file mode 100644
index 0000000..e4b4416
--- /dev/null
+++ b/docs/decisions/ADR-12_PostgreSQL_18_and_Data_Directory_Upgrade.md
@@ -0,0 +1,228 @@
+# ADR-12 PostgreSQL 18 and the Data Directory Upgrade / PostgreSQL 18 とデータディレクトリの更新
+
+> **Summary:** The developer environment moves from PostgreSQL 9.6 (end of life since 2021) to **PostgreSQL 18.6**, the newest released major, pinned by its index digest. osf.io's Django 1.11.28 with psycopg2 2.7.3 ran its migrations and 807 database tests on 18 with exactly the same outcome as on 9.6 (781 passed, the same 13 failed or erred on both), and CAS and fakecas logged in against it. CAS's JDBC driver 42.1.4 cannot answer SCRAM, so host authentication stays `md5` until CAS gets a newer driver. An existing 9.6 data directory is moved by a new command, `grdm-dev db upgrade`: a dump with the new major's `pg_dumpall` and a restore into a fresh directory, compared table by table, with the old directory kept beside it for `grdm-dev db rollback`; a durable journal and a lock shared with `up` and `reset` let the next command finish or roll back a process crash at any point (tested by injection at every boundary); durability across power loss rests on fsync ordering and was not tested physically. `up` refuses a data directory of another major (exit 15).
+> **概要:** 開発環境の PostgreSQL を 9.6(2021 年に EOL)から、リリース済みで最新の **PostgreSQL 18.6** へ移し、インデックスのダイジェストで固定する。osf.io の Django 1.11.28 と psycopg2 2.7.3 は 18 の上で migration と 807 件のデータベース試験を実行し、結果は 9.6 とまったく同じだった(781 件合格、両方で同じ 13 件が失敗かエラー)。CAS と fakecas も 18 に対してログインできた。CAS の JDBC ドライバー 42.1.4 は SCRAM に応えられないので、ホスト認証は CAS が新しいドライバーを持つまで `md5` のままにする。既存の 9.6 のデータディレクトリは新しいコマンド `grdm-dev db upgrade` で移す。新しい版の `pg_dumpall` でダンプし、新しいディレクトリへ復元し、表ごとに突き合わせ、旧ディレクトリは `grdm-dev db rollback` のために隣に残す。永続的な操作記録と、`up`・`reset` と共有するロックにより、どの時点のプロセスのクラッシュも次のコマンドが完了か巻き戻しできる(すべての境界で注入して試験)。電源断をまたぐ永続性は fsync の順序に依り、物理的には試験していない。`up` は別の版のデータディレクトリを拒む(終了 15)。
+
+**Jump:** [English](#english) · [日本語](#japanese)
+
+- Version / 版: 0.3 — 2026-11-10. **Learned:** recovery has failure modes of its own: a compensation can fail too, a rename is not durable until its directory is synced, and a lock that only some commands take excludes nothing; the journal is now retired only on a verified state, every recovery rename is synced before the record, and `up` and `reset` take the same lock (PR #42 review r2). / 復旧にも固有の失敗がある。補償も失敗しうるし、改名はそのディレクトリを sync するまで永続ではなく、一部のコマンドしか取らないロックは何も排除しない。そこで操作記録は確かめた状態でだけ消し、復旧の改名はすべて記録の前に sync し、`up` と `reset` も同じロックを取るようにした(PR #42 レビュー r2)。
+ 0.2 — 2026-11-10. **Learned:** a verified restore says nothing about a crash between two renames or a server that was never stopped: the operation now owns its containers and directories through a durable journal and a lock, every later `db` command, `reset` and `up` finish or roll it back, and a major that cannot be read is a refusal (PR #42 review r1). / 照合済みの復元は、二つの改名の間のクラッシュや止まらなかったサーバーについて何も言わない。そこで操作は永続的な操作記録とロックでコンテナとディレクトリを所有し、後続の `db` コマンド・`reset`・`up` が完了か巻き戻しを行い、読めない版は拒否にした(PR #42 レビュー r1)。
+ 0.1 — 2026-11-10. **Learned:** a database upgrade is proven by the clients, not by the server starting: the server and Django were fine on 18, while the CAS driver failed only once the password was stored as SCRAM, and PostgreSQL 18 rejected a catalogue query that 9.6 accepted (`text || "char"`), so the verification query itself had to be tested on both majors. / データベースの更新は、サーバーが起動したことではなくクライアントで証明される。サーバーと Django は 18 で問題なかったが、CAS のドライバーはパスワードが SCRAM で保存されたときにだけ失敗した。また PostgreSQL 18 は 9.6 が受け付けたカタログの問い合わせ(`text || "char"`)を拒んだので、照合の問い合わせ自体も両方の版で試す必要があった。
+- Status: Proposed (2026-11-10, M5 groundwork); the owner approves. Review date: when CAS moves to a JDBC driver with SCRAM, or when PostgreSQL 19 is released.
+- Records a decision for: [SPEC-02 §3](../spec/02_installer_cli.md#english) (`db`, the `up` refusal) and §6 (exit 15) · `manifest/upstreams.json` (`postgres`) · `installer/grdm_dev/templates/postgres.json`
+- Related: [docs/decisions/README.md](README.md) · [ADR-07 Native builds of EOL middleware](ADR-07_Native_Builds_of_EOL_Middleware.md) · [ADR-13 Retiring Elasticsearch 2 and MongoDB](ADR-13_Retiring_Elasticsearch_2_and_MongoDB.md) · [Status board](../status/README.md#english) · [SPEC-03 §3.3](../spec/03_security_gate.md#english)
+
+---
+
+
+## English
+
+[→ 日本語版へ](#japanese)
+
+### 1. Context
+
+The milestone M5 on the [status board](../status/README.md#english) asks for PostgreSQL 15 or later. The stack ran `postgres:9.6`, which its maker stopped fixing in November 2021. Three programs talk to it:
+
+```toon
+clients[3]{client,driver,measured_in}:
+ "osf.io (web, api, admin, worker, osf_init)","Django 1.11.28, psycopg2 2.7.3 linked against libpq 11.11","grdm-dev/osf.io:fe90e5c39e66"
+ "CAS (cas-overlay)","PostgreSQL JDBC 42.1.4 under Hibernate 4.3.10 and c3p0","WEB-INF/lib of grdm-dev/cas-overlay:22610b7681a2"
+ "fakecas","Go lib/pq v1.10.9","installer/images/fakecas/go.mod"
+```
+
+Django 1.11 documents PostgreSQL 9.3 and later with no upper bound, so only a measurement can say which major works. PostgreSQL 19 is still a beta on Docker Hub (`19beta1`), so 18 is the newest released major.
+
+### 2. Decision
+
+1. **Image.** `postgres:18.6` pinned by its index digest (`base_image_digest` in the manifest), for `linux/arm64` and `linux/amd64`. The one-shots `volume_owner_init` and `volume_access_check_*` run the same image; its GNU coreutils pass their checks (measured).
+2. **Template.** The 18 image moved `PGDATA` to `/var/lib/postgresql/18/docker` and declares a volume at `/var/lib/postgresql`. The template sets `PGDATA=/var/lib/postgresql/data`, the mount point of `postgres_data`, and puts a tmpfs on `/var/lib/postgresql` so that Compose creates no anonymous volume there (measured: the container has only the data volume and the init-script mount).
+3. **Authentication stays md5.** `POSTGRES_INITDB_ARGS` keeps `--auth-host=md5`; with it, initdb also writes `password_encryption = md5`, and the entrypoint's catch-all line is `host all all all md5`. With `scram-sha-256` instead, CAS failed every connection with `The authentication type 10 is not supported` (measured). SCRAM becomes the default when CAS ships a JDBC driver of 42.2 or later — a change in the integration repository and an owner item. PostgreSQL 18 logs a deprecation warning when an md5 password is set; it still works. This is a temporary compatibility exception accepted for this isolated developer network only, not a claim about internal networks in general. The way out (review r1): separate login roles for CAS and osf.io (both use `postgres` today), a SCRAM verifier set anew for osf.io's role (osf.io's libpq 11 supports SCRAM; changing `password_encryption` does not convert an existing verifier), and a currently patched JDBC driver for CAS rather than the historical minimum 42.2.
+4. **Moving existing data: `grdm-dev db upgrade`.** A server refuses a data directory of another major, so the data is dumped and restored, never changed in place. The operation owns everything it creates through a journal, so that a process crash at any point leaves either the old or the new cluster usable, or — when even a compensating rename fails — keeps the journal so that `up` refuses and `db recover` retries (review r1 R1-01, R1-02; r2 R2-01):
+ 1. **Checks, nothing started yet.** Every service stopped (`grdm-dev down`); the data major readable and known; free disk measured on the backups and the volumes filesystems (§5); a confirmation (`--yes` skips).
+ 2. **Journal and lock.** `runtime/db-operation.json` is written (fsync, atomic rename) before any container, volume or directory exists, and names all of them. An exclusive `flock` on `runtime/db-operation.lock` is held for the whole operation, including the service check and the confirmation; `up` takes the same lock from its checks until Compose has started the containers, and `reset` from its checks until the deletion ends, so none of them can run between another's checks and its action (R2-02); the kernel releases it when the process ends, however it ends, so a second `db` command waits for no dead process. SIGTERM and SIGHUP are handled like Ctrl-C.
+ 3. **Source.** The old major (`postgres:9.6`, pinned by digest) starts on the existing directory with no network and no TCP listener; its socket is in a throwaway Docker volume. The new major's `pg_dumpall` (the PostgreSQL rule: dump with the newer client) runs as the host user and writes `runtime/backups/postgres--to--/dumpall.sql` (0600 in a 0700 directory). Then the old server is stopped, removed, and proved gone with `docker ps -a` before anything else starts; a failure to prove it keeps the journal.
+ 4. **Target.** The new major initialises `runtime/volumes/postgres_data.pg-.new` with the rendered initdb arguments, `PGDATA` and command; the superuser password reaches initdb through a 0600 file deleted as soon as the container exists. The dump is restored with `ON_ERROR_STOP`; the only line removed is `CREATE ROLE postgres;` (initdb made that role; its `ALTER ROLE` with the old verifier stays). `vacuumdb --analyze-only` follows and the derived `restore.sql` is deleted. Both servers run with `log_min_error_statement=panic`, so a failing statement — which may carry row data or a password verifier — is not written to the container log.
+ 5. **Verification, and its limit.** Per database: every table's row count, every sequence's `last_value` and `is_called`, and the extension names; across the cluster: every role's attributes and an MD5 of its password verifier (never the verifier). Databases and roles are separate namespaces, so a database called `roles` keeps its check. Any difference stops before a rename. Equal results do not prove equal row contents or extension versions (extensions are recreated at the new server's default version); `pg_dumpall` with `ON_ERROR_STOP` is what moves the data.
+ 6. **Swap.** The new server is removed and proved gone; the journal is marked `swap` (fsync) and only then `postgres_data` is renamed to `postgres_data.pg-` and the new directory to `postgres_data`. A failed second rename is compensated by renaming the first back. Then `upgrade.json` is written and the journal deleted. The renames stay inside `runtime/volumes/`, because on Linux the data belongs to uid 999 and a host user may rename such a directory within its parent but not move it to another one.
+
+ **Recovery** (`grdm-dev db recover`, and first in `db upgrade`, `db rollback`, `reset` and `up`): every container and volume named in the journal is removed and proved gone; if Docker cannot answer or a container cannot be removed, the journal stays and `up` refuses. Before the `swap` mark the old directory was never written by the new major, so the operation is rolled back and the half-built new directory is named for deletion. After the mark the directories are inspected and the swap is finished (the data was already verified); if it cannot be finished, the journal is retired only when the old directory is proved back in place with its major, otherwise it stays (R2-01). Every rename of a recovery is followed by an fsync of `runtime/volumes/` before the record is written and the journal deleted, and a failed directory fsync is an error (only a filesystem that cannot sync directories, EINVAL or ENOTSUP, is accepted) (R2-03). An empty `postgres_data` that `compose render` created during the gap is removed first, so `up` never starts a fresh cluster in place of a pending swap. `grdm-dev db status` shows both majors and any pending operation. `grdm-dev db rollback` runs under the same lock and journal and puts the kept directory back; the old revision of this repository must then be checked out and `compose render` run, and until then `up` stops with exit 15.
+5. **`up` refuses a mismatch and fails closed.** After recovery and before Compose starts anything, `up` compares `PG_VERSION` with the major of the rendered image and stops with exit 15 when they differ, when `PG_VERSION` is unparseable or not ASCII (R2-04), when a non-empty directory has none, or when the version cannot be read at all — on Linux it is read by uid 999 in a throwaway container, and a failure of that container is a refusal, not a pass (R1-04). Only an absent or empty directory counts as "no cluster yet".
+6. **Hardening.** The `postgres` exception in `manifest/hardening.json` stays: the official entrypoint still starts as root. Both servers of `db upgrade` run with exactly that exception's capabilities, a read-only root and no network.
+
+### 3. Options considered
+
+```toon
+options[4]{option,what_it_does,why_not_or_why,chosen}:
+ A,"pg_upgrade (in place or copy)","needs both majors' binaries in one image (only third-party images ship that); --link writes the old directory; text indexes built under glibc 2.24 (Debian 9, the 9.6 image) would be reused under glibc 2.41 (Debian 13, the 18 image) although the collation rules changed in glibc 2.28","no"
+ B,"dump with the new pg_dumpall, restore into a new directory (this ADR)","two official images; the old directory is never written by the new major; indexes are rebuilt under the new glibc; a 290 MB development cluster moved in 37 seconds","yes"
+ C,"discard the data (grdm-dev reset)","always possible, loses every project and user of the developer","offered, not the default"
+ D,"stop at 15, 16 or 17","18 passed the same tests and is supported longest (to November 2030)","no"
+```
+
+### 4. Proof (2026-11-10, arm64, throwaway Compose projects `m5-*`)
+
+The running stack `grdm-dev` was never touched. The projects were derived from a file rendered by `compose render` into a throwaway checkout under `.m5-tmp/`; the 9.6 cluster came from a full `osf_init` (migrate, `populate_institutions`, `register_oauth_scopes`) plus one user created through osf.io.
+
+```toon
+results[10]{check,result}:
+ "osf_init on a new PostgreSQL 18.6 cluster","exit 0: 673 migrations, 68 institutions, OAuth scopes"
+ "osf.io DB test modules on 18 and on 9.6 (test_node, test_user, test_guid, test_institution, test_analytics, api_tests/institutions/views, osfstorage test_models)","both: 781 passed, 3 failed, 10 errors, 13 skipped; the failing and erroring tests are the same set on both majors (ES6 metrics fixtures and three existing failures)"
+ "CAS on 18 with md5","wrong password 200 without a ticket; right password 302 with a ticket; /p3/serviceValidate authenticationSuccess"
+ "CAS on 18 with scram-sha-256","every connection: PSQLException The authentication type 10 is not supported"
+ "fakecas on 18","known user and key 302 with a ticket; wrong key 404; unknown user 404"
+ "up on a 9.6 directory with the 18 template","exit 15 before any docker compose call"
+ "db upgrade 9.6 to 18 (real command, project renamed m5-pgup)","exit 0 in 37 s: 3 databases (osf 262 tables with 16030 rows, cas 9 tables), 531 tables, sequences and extensions equal before and after"
+ "the stack on the upgraded directory","osf_init: No migrations to apply; CAS login of the user created on 9.6 (302, serviceValidate success); fakecas 302"
+ "db rollback, then 9.6 on the directory","exit 0; PostgreSQL 9.6.24 starts, 68 institutions and 673 migrations, the user is there; up stops with exit 15 until the old revision is rendered"
+ "failure path (first attempts)","initdb without a password, then a catalogue query PostgreSQL 18 rejects: both stopped with exit 15 with postgres_data untouched; both fixed. Recovery from crashes and interruptions is shown in §4.1"
+```
+
+The first run at 512 MiB, the limit `compose render --memory-gib 16` gives `osf_init`, had the migrating Python process killed twice for memory; `osf_init` retried and finished. That limit is independent of the PostgreSQL major and is left for a separate change.
+
+#### 4.1 Recovery after review r1 (2026-11-10, a new 9.6 copy, the real command with the project renamed `m5-pgup`)
+
+The 9.6 cluster was rebuilt with `osf_init` and one user, plus a database named `roles` and a sequence set with `setval('s', 5, false)`, so that both review cases are in real data.
+
+```toon
+recovery[7]{check,result}:
+ "SIGKILL of grdm-dev while the 9.6 server ran","journal left (phase prepare), server still running; the next up removed it and its volume, rolled back, then stopped with exit 15 (9.6 data, 18 image); 0 helper containers or volumes left"
+ "SIGTERM while the 18 server ran","exit 15 in process: interrupted, rolled back; 0 helper containers or volumes; journal deleted; postgres_data still 9.6"
+ "db upgrade after both interruptions","exit 0; 524 objects equal across 4 databases (cas, osf, postgres, roles) and 1 role"
+ "the stack on the upgraded directory","osf_init: No migrations to apply; CAS login of the user created on 9.6 (302, serviceValidate success); fakecas 302"
+ "edge data","roles.t has 2 rows; sequence s keeps last_value 5 and is_called false, nextval returns 5"
+ "db rollback","exit 0; the record is marked rolled_back_at; 9.6.24 starts on the directory with 673 migrations and the user; up stops with exit 15"
+ "unit tests with injected failures","rc failure, KeyboardInterrupt and an uncatchable crash at each of 6 steps; both renames of upgrade and rollback (crash and EIO); each of 4 journal writes (ENOSPC); ENOSPC in dump, filter and restore; Docker not answering; a container that cannot be removed; a concurrent second process; an empty postgres_data created by compose render"
+```
+
+### 5. For developers
+
+Run from the root of your checkout, after pulling this change. Expected results are in the comments.
+
+```sh
+grdm-dev compose render # exit 0: the postgres service is now postgres:18.6
+grdm-dev down # exit 0
+grdm-dev db status # state: upgrade-needed, data_major: 9.6, image_major: 18
+grdm-dev db upgrade # asks first (--yes skips); exit 0, last line: PostgreSQL 9.6 → 18 done
+grdm-dev up # exit 0, ready at the web URL
+```
+
+If `db upgrade` stops (exit 15) or the terminal dies, run `grdm-dev db status`, then `grdm-dev db recover` or simply `grdm-dev up`: both finish or roll back the operation and say which. A developer who does not need the data may run `grdm-dev reset` instead.
+
+**Disk.** Before the old server starts, the command needs free space for twice the cluster plus 64 MiB on the filesystem of `runtime/backups/` (dump and its filtered copy) and twice the cluster plus 1 GiB on the filesystem of `runtime/volumes/` (the new cluster with rebuilt indexes, and WAL up to `max_wal_size`); on one filesystem, the sum. This is an estimate, not a guarantee: a logical dump can exceed the cluster (bytea is written as hex, compressed values are written expanded). A disk that fills anyway fails the step: before the swap the old cluster stays in place; after the swap (the record cannot be written) recovery finishes the new cluster and retries the record, keeping the journal until it is written. The kept directory and the dump may be deleted once the new cluster is trusted; `reset` deletes the kept directory with the rest of `runtime/volumes/`, and the dump under `runtime/backups/` survives it.
+
+### 6. Not verified
+
+An amd64 host; a Linux host where the data belongs to uid 999 (the container reads of `PG_VERSION` and of the size, and the in-parent renames, are unit-tested, not run there); a real crash between the two renames (unit-tested with an injected crash, not timed against a real kill); clusters with extensions other than `plpgsql`; SCRAM with a newer CAS driver. The production GRDM is not affected: this is the developer environment.
+
+---
+
+
+## 日本語
+
+[→ To English](#english)
+
+### 1. 背景
+
+[ステータスボード](../status/README.md#japanese)の節目 M5 は PostgreSQL 15 以降を求めている。スタックは `postgres:9.6` で動いており、これは 2021 年 11 月に作り手が修正をやめた版である。これに接続するプログラムは三つある。
+
+```toon
+clients[3]{client,driver,measured_in}:
+ "osf.io(web・api・admin・worker・osf_init)","Django 1.11.28、libpq 11.11 にリンクした psycopg2 2.7.3","grdm-dev/osf.io:fe90e5c39e66"
+ "CAS(cas-overlay)","Hibernate 4.3.10 と c3p0 の下の PostgreSQL JDBC 42.1.4","grdm-dev/cas-overlay:22610b7681a2 の WEB-INF/lib"
+ "fakecas","Go の lib/pq v1.10.9","installer/images/fakecas/go.mod"
+```
+
+Django 1.11 の文書は PostgreSQL 9.3 以降を挙げ、上限を書いていない。そのため、どの版が動くかは測定でしか分からない。Docker Hub の PostgreSQL 19 はまだベータ(`19beta1`)なので、リリース済みで最新の版は 18 である。
+
+### 2. 決定
+
+1. **イメージ。** `postgres:18.6` をインデックスのダイジェストで固定する(manifest の `base_image_digest`)。`linux/arm64` と `linux/amd64` の両方。一回限りの `volume_owner_init` と `volume_access_check_*` も同じイメージで動き、その GNU coreutils で確認を通った(測定)。
+2. **テンプレート。** 18 のイメージは `PGDATA` を `/var/lib/postgresql/18/docker` に移し、`/var/lib/postgresql` にボリュームを宣言している。テンプレートは `PGDATA=/var/lib/postgresql/data`(`postgres_data` のマウント先)を設定し、`/var/lib/postgresql` に tmpfs を置いて、Compose がそこに無名ボリュームを作らないようにする(測定: コンテナのマウントはデータのボリュームと初期化スクリプトだけ)。
+3. **認証は md5 のまま。** `POSTGRES_INITDB_ARGS` は `--auth-host=md5` を保つ。これで initdb は `password_encryption = md5` も書き、entrypoint が加える包括の行は `host all all all md5` になる。代わりに `scram-sha-256` にすると、CAS はすべての接続で `The authentication type 10 is not supported` で失敗した(測定)。SCRAM を既定にするのは、CAS が 42.2 以降の JDBC ドライバーを持ったときである。これは統合リポジトリの変更で、所有者の判断事項である。PostgreSQL 18 は md5 のパスワードを設定すると非推奨の警告をログに出すが、動作はする。これはこの隔離された開発ネットワークに限って受け入れる暫定の互換性の例外であり、内部ネットワーク一般の安全性を言うものではない。抜け出す道(レビュー r1): CAS と osf.io のログインロールを分け(今は両方が `postgres`)、osf.io のロールに SCRAM の検証子を設定し直し(osf.io の libpq 11 は SCRAM に対応する。`password_encryption` を変えても既存の検証子は変わらない)、CAS には歴史的な最低版 42.2 ではなく現在保守されている修正版の JDBC ドライバーを選ぶ。
+4. **既存データの移行: `grdm-dev db upgrade`。** サーバーは別の版のデータディレクトリを拒むので、データはダンプして復元し、その場では決して変えない。操作は自分が作るものすべてを操作記録で所有し、どの時点でプロセスがクラッシュしても旧か新のどちらかのクラスタが使える状態に残る。補償の改名まで失敗したときは操作記録を残し、`up` は拒み `db recover` が再試行する(レビュー r1 R1-01、R1-02、r2 R2-01)。
+ 1. **確認(まだ何も起動しない)。** 全サービスが停止している(`grdm-dev down`)。データの版が読めて既知である。バックアップとボリュームのファイルシステムの空き容量を測る(§5)。確認を得る(`--yes` で省く)。
+ 2. **操作記録とロック。** コンテナ・ボリューム・ディレクトリを一つも作る前に `runtime/db-operation.json` を書き(fsync と原子的な改名)、そのすべての名前を記す。サービスの確認と確認の問いを含め、操作の間ずっと `runtime/db-operation.lock` の排他 `flock` を持つ。`up` は確認から Compose がコンテナを起動するまで、`reset` は確認から削除が終わるまで同じロックを取るので、どれも他の確認と実行の間に割り込めない(R2-02)。プロセスがどう終わってもカーネルがそれを解放するので、二つ目の `db` コマンドが死んだプロセスを待つことはない。SIGTERM と SIGHUP は Ctrl-C と同じに扱う。
+ 3. **移行元。** 旧版(ダイジェスト固定の `postgres:9.6`)を既存のディレクトリの上で、ネットワークなし・TCP の待ち受けなしで起動する。ソケットは使い捨ての Docker ボリュームに置く。新しい版の `pg_dumpall`(PostgreSQL の原則: 新しい方のクライアントでダンプする)をホストの利用者として動かし、`runtime/backups/postgres-<旧>-to-<新>-<時刻>/dumpall.sql`(0700 のディレクトリの中の 0600)に書く。次に旧サーバーを止めて削除し、他の何かを起動する前に `docker ps -a` で消えたことを証明する。証明できなければ操作記録を残す。
+ 4. **移行先。** 新しい版が、生成済みの initdb 引数・`PGDATA`・コマンドで `runtime/volumes/postgres_data.pg<新>-<時刻>.new` を初期化する。スーパーユーザーのパスワードは、コンテナができた時点で消す 0600 のファイルで initdb に渡す。ダンプは `ON_ERROR_STOP` で復元する。取り除く行は `CREATE ROLE postgres;` だけ(initdb がそのロールを作った。旧検証子を持つ `ALTER ROLE` は残る)。続けて `vacuumdb --analyze-only` を実行し、派生した `restore.sql` を消す。二つのサーバーは `log_min_error_statement=panic` で動くので、失敗した文(行のデータやパスワードの検証子を含みうる)はコンテナのログに書かれない。
+ 5. **照合とその限界。** データベースごとに全表の行数、全シーケンスの `last_value` と `is_called`、拡張の名前を、クラスタ全体で全ロールの属性とパスワード検証子の MD5(検証子そのものではない)を比べる。データベースとロールは別の名前空間なので、`roles` という名のデータベースも自分の照合を保つ。違いが一つでもあれば改名の前に止まる。一致しても行の中身や拡張の版の一致は証明しない(拡張は新しいサーバーの既定の版で作り直される)。データを移すのは `ON_ERROR_STOP` 付きの `pg_dumpall` である。
+ 6. **切替。** 新サーバーを削除して消えたことを証明し、操作記録に `swap` の印を付けて(fsync)から初めて、`postgres_data` を `postgres_data.pg<旧>-<時刻>` に、新しいディレクトリを `postgres_data` に改名する。二つ目の改名に失敗したら一つ目を戻して補償する。その後 `upgrade.json` を書き、操作記録を消す。改名は `runtime/volumes/` の中に留める。Linux ではデータが uid 999 の所有で、ホストの利用者はその種のディレクトリを同じ親の中では改名できても、別の親へは移せないからである。
+
+ **復旧**(`grdm-dev db recover`。`db upgrade`・`db rollback`・`reset`・`up` も最初に行う): 操作記録にあるコンテナとボリュームをすべて削除し、消えたことを証明する。Docker が答えないか削除できなければ操作記録は残り、`up` は拒む。`swap` の印の前は、旧ディレクトリに新しい版が一度も書いていないので巻き戻し、作りかけの新ディレクトリを削除してよいものとして示す。印の後はディレクトリを調べて切替を完了する(データは照合済み)。完了できなければ、旧ディレクトリが版とともに元の場所に戻ったと確かめたときだけ操作記録を消し、そうでなければ残す(R2-01)。復旧の改名のたびに、記録を書き操作記録を消す前に `runtime/volumes/` を fsync し、ディレクトリの fsync の失敗は誤りとする(ディレクトリを sync できないファイルシステムの EINVAL・ENOTSUP だけは受け入れる)(R2-03)。その間に `compose render` が作った空の `postgres_data` は先に取り除くので、`up` が保留中の切替の代わりに新しい空のクラスタを作ることはない。`grdm-dev db status` は両方の版と保留中の操作を表示する。`grdm-dev db rollback` は同じロックと操作記録のもとで残したディレクトリを戻す。その後は本リポジトリの以前の版を取り出して `compose render` を実行する必要があり、それまで `up` は終了 15 で止まる。
+5. **`up` は不一致を拒み、閉じて失敗する。** 復旧の後、Compose が何かを起動する前に、`up` は `PG_VERSION` と生成済みイメージの版を比べ、違うとき、`PG_VERSION` が解析できないか ASCII でないとき(R2-04)、空でないディレクトリに `PG_VERSION` が無いとき、版がそもそも読めないときに終了 15 で止まる。Linux では使い捨てのコンテナの中で uid 999 が読み、そのコンテナの失敗は通過ではなく拒否である(R1-04)。「まだクラスタが無い」とみなすのは、ディレクトリが無いか空のときだけである。
+6. **堅牢化。** `manifest/hardening.json` の `postgres` の例外は残る。公式の entrypoint は今も root で始まるからである。`db upgrade` の二つのサーバーはちょうどその例外の capability と読み取り専用ルートで、ネットワークなしで動く。
+
+### 3. 検討した選択肢
+
+```toon
+options[4]{option,what_it_does,why_not_or_why,chosen}:
+ A,"pg_upgrade(その場、または複写)","両方の版のバイナリが一つのイメージに要る(それを配るのは第三者のイメージだけ)。--link は旧ディレクトリに書く。glibc 2.24(Debian 9、9.6 のイメージ)で作った文字列の索引を、照合規則が glibc 2.28 で変わったのに glibc 2.41(Debian 13、18 のイメージ)の下でそのまま使うことになる","いいえ"
+ B,"新しい pg_dumpall でダンプし新しいディレクトリへ復元(本 ADR)","公式イメージ二つ。旧ディレクトリに新しい版は一度も書かない。索引は新しい glibc の下で作り直される。290 MB の開発用クラスタが 37 秒で移った","はい"
+ C,"データを捨てる(grdm-dev reset)","常に可能だが、開発者のプロジェクトと利用者をすべて失う","選べるが既定ではない"
+ D,"15・16・17 で止める","18 は同じ試験に合格し、保守期間が最も長い(2030 年 11 月まで)","いいえ"
+```
+
+### 4. 実証(2026-11-10、arm64、使い捨ての Compose プロジェクト `m5-*`)
+
+稼働中のスタック `grdm-dev` には一度も触れていない。プロジェクトは `.m5-tmp/` の下の使い捨てのチェックアウトで `compose render` が生成したファイルから導いた。9.6 のクラスタは完全な `osf_init`(migrate、`populate_institutions`、`register_oauth_scopes`)と、osf.io で作った利用者一人から成る。
+
+```toon
+results[10]{check,result}:
+ "新しい PostgreSQL 18.6 のクラスタでの osf_init","終了 0: migration 673 件、機関 68、OAuth のスコープ"
+ "18 と 9.6 での osf.io の DB 試験モジュール(test_node、test_user、test_guid、test_institution、test_analytics、api_tests/institutions/views、osfstorage の test_models)","両方とも 781 合格、3 失敗、10 エラー、13 スキップ。失敗とエラーの試験は両方の版で同じ集合(ES6 のメトリクスの準備と既存の失敗 3 件)"
+ "md5 の 18 での CAS","誤ったパスワードは 200 でチケットなし。正しいパスワードは 302 でチケットあり。/p3/serviceValidate は authenticationSuccess"
+ "scram-sha-256 の 18 での CAS","すべての接続で PSQLException The authentication type 10 is not supported"
+ "18 での fakecas","既知の利用者と鍵で 302 とチケット。誤った鍵は 404。未知の利用者は 404"
+ "18 のテンプレートで 9.6 のディレクトリに up","docker compose を一度も呼ばずに終了 15"
+ "db upgrade 9.6 から 18(実際のコマンド、プロジェクト名を m5-pgup に変更)","37 秒で終了 0: データベース 3(osf は 262 表・16030 行、cas は 9 表)、表・シーケンス・拡張 531 件が前後で一致"
+ "更新したディレクトリでのスタック","osf_init: No migrations to apply。9.6 で作った利用者の CAS ログイン(302、serviceValidate 成功)。fakecas は 302"
+ "db rollback の後、そのディレクトリで 9.6","終了 0。PostgreSQL 9.6.24 が起動し、機関 68・migration 673 件・利用者がそろう。以前の版を生成するまで up は終了 15 で止まる"
+ "失敗の経路(最初の試行)","パスワードなしの initdb、次に PostgreSQL 18 が拒むカタログの問い合わせ。どちらも postgres_data を変えずに終了 15 で止まり、両方とも修正した。クラッシュと中断からの復旧は §4.1"
+```
+
+最初の実行では、`compose render --memory-gib 16` が `osf_init` に与える 512 MiB の上限で、migrate する Python のプロセスがメモリ不足で 2 回止められ、`osf_init` が再試行して完了した。この上限は PostgreSQL の版と無関係なので、別の変更に回す。
+
+#### 4.1 レビュー r1 の後の復旧(2026-11-10、新しい 9.6 の写し、プロジェクト名を `m5-pgup` にした実際のコマンド)
+
+9.6 のクラスタを `osf_init` と利用者一人で作り直し、`roles` という名のデータベースと `setval('s', 5, false)` で設定したシーケンスを加え、レビューの二つの例が実データにあるようにした。
+
+```toon
+recovery[7]{check,result}:
+ "9.6 サーバーの稼働中に grdm-dev を SIGKILL","操作記録(phase prepare)が残り、サーバーは動いたまま。次の up がそれとボリュームを削除して巻き戻し、終了 15(データ 9.6、イメージ 18)。補助のコンテナとボリュームは 0"
+ "18 サーバーの稼働中に SIGTERM","プロセス内で終了 15: 中断、巻き戻し。補助のコンテナとボリュームは 0、操作記録は削除、postgres_data は 9.6 のまま"
+ "二つの中断の後の db upgrade","終了 0。4 データベース(cas、osf、postgres、roles)とロール 1 で 524 件が一致"
+ "更新したディレクトリでのスタック","osf_init: No migrations to apply。9.6 で作った利用者の CAS ログイン(302、serviceValidate 成功)。fakecas は 302"
+ "境界のデータ","roles.t は 2 行。シーケンス s は last_value 5 と is_called false を保ち、nextval は 5 を返す"
+ "db rollback","終了 0。記録に rolled_back_at が付く。9.6.24 がそのディレクトリで起動し、migration 673 件と利用者がそろう。up は終了 15 で止まる"
+ "障害を注入した単体試験","6 段階それぞれで rc の失敗・KeyboardInterrupt・捕捉できないクラッシュ。更新と巻き戻しの両方の改名(クラッシュと EIO)。4 回の操作記録の書込みそれぞれ(ENOSPC)。ダンプ・加工・復元の ENOSPC。答えない Docker。削除できないコンテナ。同時に動く二つ目のプロセス。compose render が作った空の postgres_data"
+```
+
+### 5. 開発者へ
+
+この変更を取り込んだ後、自分のチェックアウトのルートで実行する。期待する結果はコメントに書いた。
+
+```sh
+grdm-dev compose render # 終了 0: postgres サービスは postgres:18.6 になる
+grdm-dev down # 終了 0
+grdm-dev db status # state: upgrade-needed、data_major: 9.6、image_major: 18
+grdm-dev db upgrade # 先に確認する(--yes で省く)。終了 0、最後の行: PostgreSQL 9.6 → 18 done
+grdm-dev up # 終了 0、web の URL で準備完了
+```
+
+`db upgrade` が止まった(終了 15)か端末が落ちたときは、`grdm-dev db status` の後に `grdm-dev db recover`、または単に `grdm-dev up` を実行する。どちらも操作を完了か巻き戻しし、どちらにしたかを示す。データが要らない開発者は代わりに `grdm-dev reset` を使ってよい。
+
+**容量。** 旧サーバーを起動する前に、`runtime/backups/` のファイルシステムにクラスタの 2 倍と 64 MiB(ダンプとその加工した写し)、`runtime/volumes/` のファイルシステムにクラスタの 2 倍と 1 GiB(索引を作り直した新しいクラスタと、`max_wal_size` までの WAL)の空きを求める。同じファイルシステムならその和。これは見積りであって保証ではない。論理ダンプはクラスタより大きくなりうる(bytea は 16 進で、圧縮された値は展開して書かれる)。それでも満杯になればその段階が失敗する。切替の前なら旧クラスタが元の場所に残り、切替の後(記録が書けない)なら復旧が新クラスタを完了させ、記録を書けるまで操作記録を残して再試行する。新しいクラスタを信頼できたら、残したディレクトリとダンプは消してよい。`reset` は残したディレクトリを `runtime/volumes/` の残りと一緒に消し、`runtime/backups/` の下のダンプは残す。
+
+### 6. 未確認
+
+amd64 のホスト。データが uid 999 の所有になる Linux のホスト(`PG_VERSION` と容量のコンテナでの読み取り、同じ親の中での改名は単体試験で確かめたが、そこでは動かしていない)。二つの改名の間の実際のクラッシュ(注入したクラッシュで単体試験したが、実際の kill で狙ってはいない)。`plpgsql` 以外の拡張を持つクラスタ。新しい CAS のドライバーでの SCRAM。本番の GRDM には影響しない。これは開発環境の話である。
+
+---
+
+## Glossary / 用語
+
+| Term | English note | 日本語注 |
+|---|---|---|
+| Major version | The first number of a PostgreSQL release (9.6, 18); a server reads only data directories of its own major. | PostgreSQL のリリースの最初の番号(9.6、18)。サーバーは自分と同じ版のデータディレクトリしか読まない。 |
+| Dump and restore | Writing every database out as SQL, then replaying it into a new server. | すべてのデータベースを SQL として書き出し、新しいサーバーで再生すること。 |
+| SCRAM | The password exchange PostgreSQL uses by default since 14; older drivers only know md5. | PostgreSQL が 14 から既定で使うパスワードのやり取り。古いドライバーは md5 しか知らない。 |
+| Data directory | The folder where PostgreSQL keeps a whole cluster; here `runtime/volumes/postgres_data`. | PostgreSQL がクラスタ全体を置くフォルダ。ここでは `runtime/volumes/postgres_data`。 |
diff --git a/docs/decisions/ADR-13_Retiring_Elasticsearch_2_and_MongoDB.md b/docs/decisions/ADR-13_Retiring_Elasticsearch_2_and_MongoDB.md
new file mode 100644
index 0000000..b1ccc7d
--- /dev/null
+++ b/docs/decisions/ADR-13_Retiring_Elasticsearch_2_and_MongoDB.md
@@ -0,0 +1,186 @@
+# ADR-13 Retiring Elasticsearch 2 and MongoDB / Elasticsearch 2 と MongoDB の退役
+
+> **Summary:** Measured findings and a proposal for the two remaining M5 removals. **Elasticsearch 2:** osf.io's search code cannot run on Elasticsearch 6.8 or OpenSearch 2.19, even after a shim for the missing Content-Type header: an index holds one mapping type only, and every query fails to parse. Turning search off (`SEARCH_ENGINE = None`) keeps writes working but makes five read endpoints answer 500 or `null`; a prototype patch of about 50 lines makes all of them answer well-formed empty results and moves contributor search to PostgreSQL with the same institution rule. A real port to Elasticsearch 8 or OpenSearch works on an affected code surface of about 5,300 lines in the integration repository. **MongoDB:** only sharejs uses it. osf.io's own reader (the `/draft/` route) resolves to `mongodb://postgres:5432/sharejs` in this environment and times out after 30 s. The only browser feature on sharejs is the file page's text editor, whose real-time co-editing is what retiring sharejs removes (since PR #40 sharejs is published on 127.0.0.1:7007; the editor was not exercised in a browser here); the exact steps are listed. On 2026-11-10 the owner chose the port to Elasticsearch 8 with full-text search, and the move of the file-page editor to y-websocket before sharejs and mongo are retired; both are in progress (§4).
+> **概要:** M5 に残る二つの撤去について、測定した事実と提案をまとめる。**Elasticsearch 2:** osf.io の検索コードは、欠けている Content-Type ヘッダーを補う互換層を入れても、Elasticsearch 6.8 でも OpenSearch 2.19 でも動かない。一つのインデックスは一つのマッピング型しか持てず、すべての問い合わせが構文解析で失敗する。検索を切る(`SEARCH_ENGINE = None`)と書き込みは動くが、読み取りの五つの入口が 500 か `null` を返す。約 50 行の試作パッチで、そのすべてが形の整った空の結果を返し、協力者の検索は同じ機関の規則のまま PostgreSQL に移る。Elasticsearch 8 か OpenSearch への本当の移植は、統合リポジトリの約 5,300 行の影響範囲での作業になる。**MongoDB:** 使うのは sharejs だけである。osf.io 自身の読み手(`/draft/` 経路)はこの環境では `mongodb://postgres:5432/sharejs` を指して 30 秒で時間切れになる。sharejs に乗るブラウザの機能はファイルページのテキストエディタだけで、sharejs の退役で失われるのはその実時間の共同編集である(PR #40 以降 sharejs は 127.0.0.1:7007 で公開されている。ここではブラウザでエディタを試していない)。正確な手順を挙げる。2026-11-10 に所有者は、全文検索を保った Elasticsearch 8 への移植と、sharejs と mongo の退役に先立つファイルページのエディタの y-websocket への移行を選んだ。どちらも進行中である(§4)。
+
+**Jump:** [English](#english) · [日本語](#japanese)
+
+- Version / 版: 0.3 — 2026-11-10. **Learned:** the header must say what the body records: once §4 held the owner's choices, "nothing yet" was wrong, while the record stays Proposed (PR #42 review r2). / 見出しは本文が記録するものを言わなければならない。§4 が所有者の選択を持った時点で「まだ無い」は誤りになったが、記録自体は提案中のまま(PR #42 レビュー r2)。
+ 0.2 — 2026-11-10. **Learned:** a finding is only as wide as its measurement: the drafts claim covered one reader, the line count is a code surface rather than work measured, and "every page" meant the probed entry points; and once the owner has chosen, the record keeps the options as the reasons for the choice (§4, PR #42 review r1). / 事実の広さは測定の広さまでである。下書きの主張は一つの読み手にしか及ばず、行数は作業量の実測ではなくコードの範囲で、「すべてのページ」は調べた入口のことだった。所有者が選んだ後は、選択肢を選んだ理由として記録に残す(§4、PR #42 レビュー r1)。
+ 0.1 — 2026-11-10. **Learned:** a setting that "disables" a backend is not a supported mode until every reader handles the empty answer: `SEARCH_ENGINE = None` was accepted by every write path and broke every read path, and a client library pinned to an old server fails on the first header before any real incompatibility shows. / バックエンドを「無効にする」設定は、すべての読み手が空の答えを扱えるまで、支えられた動作モードではない。`SEARCH_ENGINE = None` はすべての書き込みの経路に受け入れられ、すべての読み取りの経路を壊した。また古いサーバーに合わせたクライアントライブラリは、本当の非互換が見える前に最初のヘッダーで失敗する。
+- Status: Proposed (2026-11-10, M5 groundwork); the owner approves the record. The owner chose the paths on 2026-11-10 (§4); nothing is removed by this record.
+- Records a decision for: the owner's chosen direction of 2026-11-10 (§4) — the Elasticsearch 8 port and the move of the file-page editor to y-websocket before sharejs and mongo are retired. The record itself stays Proposed, and the implementations are other pull requests: this one removes nothing. It will change `manifest/upstreams.json` (`elasticsearch2`, `mongo`, `sharejs`), [SPEC-04](../spec/04_acceptance_tests.md#english) (C4 services) and the osf.io integration repository.
+- Related: [docs/decisions/README.md](README.md) · [ADR-07 §5](ADR-07_Native_Builds_of_EOL_Middleware.md#english) (sharejs on Node 24) · [ADR-12 PostgreSQL 18](ADR-12_PostgreSQL_18_and_Data_Directory_Upgrade.md) · [Status board](../status/README.md#english)
+
+---
+
+
+## English
+
+[→ 日本語版へ](#japanese)
+
+### 1. Context
+
+The status board's M5 row says "ES2 and mongo removed; ES8". Elasticsearch 2.4.6 runs today as a native build (ADR-07), MongoDB 3.4 as the official image with a hardening exception. Both are end of life. All measurements below used osf.io image `grdm-dev/osf.io:fe90e5c39e66` (integration commit `fe90e5c39e660f464bb8fcc9ae4028119b958d19`) on PostgreSQL 18 in the throwaway project `m5-pg18a`, with a probe that creates a project through API v2 and calls the search entry points: `/api/v1/search/`, `/search/`, contributor search (`website.search.search.search_contributor`, the function behind `/api/v1/user/search/`), `/v2/search/`, `/v2/search/users/`, `/v2/search/projects/`.
+
+### 2. Elasticsearch 2: findings
+
+```toon
+search_modes[7]{mode,create_index_and_writes,reads}:
+ "ES 2.4.6 (today)","all succeed","all 200 with results"
+ "search off (SEARCH_ENGINE = None)","project creation 201","/api/v1/search/ 500 TypeError; /v2/search/* 500 TypeError; contributor search returns null; /search/ page 200 but its requests fail"
+ "ES 6.8.23, unchanged client","every call: Content-Type header [] is not supported","every call fails the same way"
+ "ES 6.8.23 + Content-Type shim (8 lines)","index created; institutions indexed; projects and users rejected: Rejecting mapping update to [website] as the final mapping would have more than 1 type","every search 400: [query_string] cannot use [fields] parameter in conjunction with [default_field]"
+ "OpenSearch 2.19.5, unchanged client","index created; indexing fails in the client: TypeError string indices must be integers","Content-Type header [] is not supported"
+ "OpenSearch 2.19.5 + shim","same TypeError on indexing","every search 400 (same query error)"
+ "search off + prototype patch (about 50 lines)","project creation 201","every entry point 200 with a well-formed empty result; contributor search returns the user from PostgreSQL"
+```
+
+The client is `elasticsearch2` 2.5.0 (the old Python client under a renamed package); the ES 6 client in the image (`elasticsearch` 6.3.1) is used only for metrics. Code bound to the ES 2 client and query language: `website/search/` (elastic_search.py 1,951 lines, util.py 626, search.py 223, views.py 422), `website/search_migration/` (1,293), `api/search/views.py` (711) and `addons/metadata/search.py` (123) — about 5,300 lines. It keeps twelve document types (project, component, registration, user, file, wiki, comment, institution, preprint, group, metadata, collectionSubmission) in one index `website`, uses `query_string` with both `fields` and `default_field`, and the old `filtered` query.
+
+**Smallest supported path: search off with a patch.** The prototype replaces `website.search.search.search` with a function that returns `{'results': [], 'counts': {'total': 0}, 'aggs': {…: 0}, …}` (the API v2 paginator reads `aggs.total` and `counts.`) and `search_contributor` with a PostgreSQL query: active, registered, not merged users whose full name contains every word or whose username equals the query, excluding the given users, and limited to the current user's institutions when the user has any (the rule of the ES query). The search page, API v2 search and the private-search features then return nothing. No core acceptance scenario (C1–C6) calls a search endpoint. `osf_init` no longer needs to wait for a search engine.
+
+**Costs.**
+
+```toon
+paths[3]{path,where,cost,keeps}:
+ "A: keep ES 2.4.6 (native build, ADR-07)","nothing","an EOL engine stays past M5; the M5 row is not met","full search"
+ "B: search off + patch (prototype measured)","osf.io integration repository (about 50 lines in website/search, tests for the five entry points); installer: drop elasticsearch2, the osf_init wait loop and ELASTIC_URI","no full-text search of projects, files, wikis or metadata; contributor search becomes substring matching without kana normalisation or fuzziness","writes, contributor search with the institution rule; every probed entry point answers 200"
+ "C: port to Elasticsearch 8 or OpenSearch","osf.io integration repository: work on an affected code surface of about 5,300 lines — a size of the code bound to ES 2, not a measured count of lines to change — (one index per document type or a type field, the query builders, the client, the Japanese analyzer setup, the search migration)","weeks of work and its own review; a new image and data volume","full search on a supported engine"
+```
+
+### 3. MongoDB and sharejs: findings
+
+1. **Clients of mongo.** In the installer only `sharejs` connects (`SHAREJS_DB_URL=mongodb://mongo:27017/sharejs?ssl=true`). osf.io has a pymongo reader, `addons/wiki/utils.py` `share_db()`, but its URL is built from `DB_HOST` and `DB_PORT`, which this environment sets for PostgreSQL: it resolves to `mongodb://postgres:5432/sharejs`, and a read raised `ServerSelectionTimeoutError` after 30.3 s (measured).
+2. **Callers of that reader.** `WikiVersion.get_draft` and `get_sharejs_content`, both only through `wiki_page_draft` (route `/project//wiki//draft/`). The current wiki editor (`wikiPageMilkdown.js` of RDM-wiki-frontend-dev2027 at `5985d18b75da4105a610ac041b838c45f295de7d`) stores `urls.draft` but never requests it; only the legacy `ShareJSDoc.js`/`WikiEditor.js` do. `delete_share_doc` and `migrate_uuid` have no callers outside tests. `broadcast_to_sharejs` (on wiki rename and delete) posts to `localhost:7007` and ignores a connection error.
+3. **Browser.** The file page loads `///text.js` and `share.js` (`view_file.mako`), and its Edit panel (`filepage/editor.js`, `ShareJSDocFile.js`, 350 lines) uses ShareJS. When these measurements were taken (installer at `03c7358`) the browser was given `localhost:7007` and sharejs published no port, so the panel showed only a spinner (ADR-07 §5). PR #40 published sharejs on `127.0.0.1:7007` and gives the browser that URL, so the panel can now reach sharejs; it was not exercised in a browser for this record.
+4. **Acceptance.** SPEC-04 C4 lists `sharejs` among its services, but the scenario tests the wiki over `y_websocket` only.
+
+**What the file page loses:** real-time co-editing of a text file in the Edit panel. Choice (a) below drops it; choice (b) rebuilds it on y-websocket like the wiki.
+
+**Retirement plan (in this order).**
+
+1. *osf.io integration repository, first PR.* Choose the file-page editor: (a) a plain editor that saves through WaterButler as today, without co-editing, or (b) a Yjs document per file on y-websocket, with a token from osf.io (`generate_y_websocket_token` with a file-scoped room and a WRITE check) and an editor binding. Remove the sharejs script tags from `view_file.mako`; remove `wiki_page_draft` and its route, `WikiVersion.get_draft`, `get_sharejs_content`, `delete_share_doc`, `migrate_uuid`, `share_db`, `broadcast_to_sharejs` with its calls, and `update_active_sharejs`; the `SHAREJS_*` settings; `pymongo==3.7.1` from `requirements.txt`; the sharejs parts of the tests in `addons/wiki/tests/` (5 files) and `tests/test_addons.py`. Keep `wiki_private_uuids` and `get_sharejs_uuid`: y-websocket room names come from them.
+2. *Data.* Before removal, a developer who wants the drafts runs `mongoexport` of `sharejs.docs` into `runtime/backups/`; otherwise they are discarded. The measured wiki reader (`share_db`) cannot reach them here (item 1 above); that does not prove that no draft text reached a saved file, because the file page saves its text through its own HTTP PUT.
+3. *This repository, second PR.* Remove the `mongo` and `sharejs` components, `templates/mongo.json`, `templates/sharejs.json`, `installer/images/sharejs/`, `sharejs.env`, the published port 7007 and the browser's `SHAREJS_URL` from `compose render` (PR #40), the volumes `mongo_data` and `mongo_tls`, the `mongo` entry of `manifest/hardening.json`, `SHAREJS_SECRET` from the templates and `secrets.py` once osf.io stops reading it, and `sharejs` from the C4 services in SPEC-04. Bump the osf.io pin to the commit of step 1.
+
+### 4. Owner decisions (2026-11-10)
+
+1. **Elasticsearch 2: path C now.** osf.io search is ported to Elasticsearch 8, keeping full-text search; the work is in progress in the osf.io integration repository. Path B stays recorded as the measured fallback, not the plan.
+2. **File page: choice (b).** The file-page editor moves from sharejs to y-websocket; then sharejs and mongo are retired by the steps of §3 (in progress). The drafts question (step 2) follows from that work.
+3. **Also decided for M4 (recorded here because it was decided the same day):** freecad is removed from MFR, and the remaining OS packages for which the distribution publishes no fix are allow-listed.
+
+### 5. The prototype of path B (author-reported)
+
+The measurement logs and the prototype were throwaway files; their numbers in §2 are as reported by the author. The prototype, for reproduction, was loaded before Django set up the search module:
+
+```python
+import math
+from django.db.models import Q
+from website.search import search as _search
+
+
+def _empty(*args, **kwargs):
+ counts = dict.fromkeys(('total', 'users', 'projects', 'components', 'registrations', 'files',
+ 'institutions', 'collectionSubmissions'), 0)
+ return {'results': [], 'counts': {'total': 0}, 'aggs': counts, 'tags': [], 'typeAliases': {}}
+
+
+def search_contributor(query, page=0, size=10, exclude=None, current_user=None):
+ from osf.models import OSFUser
+ from website.profile.utils import profile_image_url
+ from website import settings
+ qs = OSFUser.objects.filter(is_active=True, is_registered=True, merged_by__isnull=True)
+ for word in (query or '').split():
+ qs = qs.filter(Q(fullname__icontains=word) | Q(username__iexact=word))
+ if exclude:
+ qs = qs.exclude(id__in=[u.id for u in exclude])
+ if current_user and current_user.affiliated_institutions.exists():
+ qs = qs.filter(affiliated_institutions__in=current_user.affiliated_institutions.all()).distinct()
+ total = qs.count()
+ users = [{'fullname': u.fullname, 'id': u._id, 'employment': None, 'education': None, 'social': u.social_links,
+ 'n_projects_in_common': -1 if current_user and current_user._id == u._id
+ else (current_user.n_projects_in_common(u) if current_user else 0),
+ 'profile_image_url': profile_image_url(settings.PROFILE_IMAGE_PROVIDER, u, use_ssl=True,
+ size=settings.PROFILE_IMAGE_MEDIUM),
+ 'profile_url': u.profile_url, 'registered': u.is_registered, 'active': u.is_active}
+ for u in qs.order_by('fullname')[page * size:(page + 1) * size]]
+ return {'users': users, 'total': total, 'pages': math.ceil(total / size), 'page': page}
+
+
+_search.search = _empty
+_search.search_contributor = search_contributor
+```
+
+---
+
+
+## 日本語
+
+[→ To English](#english)
+
+### 1. 背景
+
+ステータスボードの M5 の行は「ES2・mongo 廃止。ES8」と書く。Elasticsearch 2.4.6 は今ネイティブのビルド(ADR-07)で、MongoDB 3.4 は公式イメージと堅牢化の例外で動いている。どちらも EOL である。以下の測定はすべて、使い捨てのプロジェクト `m5-pg18a` の PostgreSQL 18 の上で osf.io のイメージ `grdm-dev/osf.io:fe90e5c39e66`(統合コミット `fe90e5c39e660f464bb8fcc9ae4028119b958d19`)を使った。調べる道具は API v2 でプロジェクトを作り、検索の入口を呼ぶ: `/api/v1/search/`、`/search/`、協力者の検索(`website.search.search.search_contributor`。`/api/v1/user/search/` の中身)、`/v2/search/`、`/v2/search/users/`、`/v2/search/projects/`。
+
+### 2. Elasticsearch 2: 事実
+
+```toon
+search_modes[7]{mode,create_index_and_writes,reads}:
+ "ES 2.4.6(現状)","すべて成功","すべて 200 で結果あり"
+ "検索を切る(SEARCH_ENGINE = None)","プロジェクト作成は 201","/api/v1/search/ は 500 TypeError。/v2/search/* は 500 TypeError。協力者の検索は null。/search/ のページは 200 だが中の要求が失敗"
+ "ES 6.8.23、クライアントは変更なし","すべての呼び出しが Content-Type header [] is not supported","すべて同じ理由で失敗"
+ "ES 6.8.23 と Content-Type の互換層(8 行)","インデックス作成と機関の登録は成功。プロジェクトと利用者は Rejecting mapping update to [website] as the final mapping would have more than 1 type で拒否","すべての検索が 400: [query_string] cannot use [fields] parameter in conjunction with [default_field]"
+ "OpenSearch 2.19.5、クライアントは変更なし","インデックス作成は成功。登録はクライアント内で TypeError string indices must be integers","Content-Type header [] is not supported"
+ "OpenSearch 2.19.5 と互換層","登録は同じ TypeError","すべての検索が 400(同じ問い合わせの誤り)"
+ "検索を切る+試作パッチ(約 50 行)","プロジェクト作成は 201","すべての入口が 200 で形の整った空の結果。協力者の検索は PostgreSQL から利用者を返す"
+```
+
+クライアントは `elasticsearch2` 2.5.0(古い Python クライアントを別名のパッケージにしたもの)である。イメージにある ES 6 のクライアント(`elasticsearch` 6.3.1)はメトリクスにしか使われない。ES 2 のクライアントと問い合わせ言語に結びついたコードは、`website/search/`(elastic_search.py 1,951 行、util.py 626、search.py 223、views.py 422)、`website/search_migration/`(1,293)、`api/search/views.py`(711)、`addons/metadata/search.py`(123)で、約 5,300 行になる。一つのインデックス `website` に十二種類の文書型(project・component・registration・user・file・wiki・comment・institution・preprint・group・metadata・collectionSubmission)を置き、`query_string` に `fields` と `default_field` を両方渡し、古い `filtered` の問い合わせを使う。
+
+**支えられる最小の道: 検索を切ってパッチを当てる。** 試作は `website.search.search.search` を `{'results': [], 'counts': {'total': 0}, 'aggs': {…: 0}, …}` を返す関数に置き換え(API v2 のページ分けは `aggs.total` と `counts.<型>` を読む)、`search_contributor` を PostgreSQL の問い合わせに置き換える。対象は有効・登録済み・未統合の利用者で、氏名がすべての語を含むか利用者名が問い合わせと一致するもの。指定された利用者を除き、今の利用者が機関に属していればその機関に限る(ES の問い合わせと同じ規則)。検索ページ、API v2 の検索、非公開検索の機能は何も返さなくなる。コアの受入シナリオ(C1〜C6)はどれも検索の入口を呼ばない。`osf_init` は検索エンジンを待つ必要がなくなる。
+
+**代償。**
+
+```toon
+paths[3]{path,where,cost,keeps}:
+ "A: ES 2.4.6 を残す(ネイティブのビルド、ADR-07)","なし","EOL のエンジンが M5 の後も残る。M5 の行を満たさない","検索のすべて"
+ "B: 検索を切る+パッチ(試作を測定済み)","osf.io の統合リポジトリ(website/search に約 50 行と、五つの入口の試験)。インストーラーでは elasticsearch2、osf_init の待ちの繰り返し、ELASTIC_URI を除く","プロジェクト・ファイル・wiki・メタデータの全文検索がなくなる。協力者の検索は部分一致になり、かなの正規化とあいまい一致がなくなる","書き込み、機関の規則つきの協力者の検索。調べた入口はすべて 200"
+ "C: Elasticsearch 8 か OpenSearch へ移植","osf.io の統合リポジトリ: 約 5,300 行の影響範囲での作業(ES 2 に結びついたコードの大きさであり、変える行数の実測ではない)(文書型ごとのインデックスか型の欄、問い合わせの組み立て、クライアント、日本語の解析器の設定、検索の移行)","数週間の作業と独自のレビュー。新しいイメージとデータボリューム","支えられたエンジンでの検索のすべて"
+```
+
+### 3. MongoDB と sharejs: 事実
+
+1. **mongo のクライアント。** インストーラーの中で接続するのは `sharejs` だけである(`SHAREJS_DB_URL=mongodb://mongo:27017/sharejs?ssl=true`)。osf.io には pymongo の読み手 `addons/wiki/utils.py` の `share_db()` があるが、その URL は `DB_HOST` と `DB_PORT` から作られ、この環境ではそれが PostgreSQL 向けに設定されている。結果は `mongodb://postgres:5432/sharejs` になり、読み取りは 30.3 秒後に `ServerSelectionTimeoutError` で失敗した(測定)。
+2. **その読み手を呼ぶもの。** `WikiVersion.get_draft` と `get_sharejs_content` で、どちらも `wiki_page_draft`(経路 `/project//wiki//draft/`)からしか呼ばれない。今の wiki エディタ(RDM-wiki-frontend-dev2027 の `5985d18b75da4105a610ac041b838c45f295de7d` の `wikiPageMilkdown.js`)は `urls.draft` を保持するが要求はしない。要求するのは旧来の `ShareJSDoc.js` と `WikiEditor.js` だけである。`delete_share_doc` と `migrate_uuid` を試験の外で呼ぶものは無い。`broadcast_to_sharejs`(wiki の改名と削除のとき)は `localhost:7007` へ送り、接続の誤りは無視する。
+3. **ブラウザ。** ファイルページは `///text.js` と `share.js` を読み込み(`view_file.mako`)、その編集パネル(`filepage/editor.js`、`ShareJSDocFile.js`、350 行)は ShareJS を使う。測定の時点(インストーラーは `03c7358`)ではブラウザに `localhost:7007` が渡され、sharejs はポートを公開しなかったので、パネルはスピナーのままだった(ADR-07 §5)。PR #40 は sharejs を `127.0.0.1:7007` で公開し、ブラウザにその URL を渡すので、パネルは今は sharejs に届きうる。この記録のためにブラウザでは試していない。
+4. **受入。** SPEC-04 の C4 はサービスに `sharejs` を挙げるが、シナリオが試すのは `y_websocket` 上の wiki だけである。
+
+**ファイルページが失うもの:** 編集パネルでのテキストファイルの実時間の共同編集。下の (a) ではそれを手放し、(b) では wiki と同じく y-websocket の上に作り直す。
+
+**退役の手順(この順で)。**
+
+1. *osf.io の統合リポジトリ、最初の PR。* ファイルページのエディタを選ぶ。(a) 共同編集なしで今と同じく WaterButler を通して保存する素のエディタ、または (b) y-websocket 上のファイルごとの Yjs 文書。トークンは osf.io から出し(ファイル単位の部屋と WRITE の確認を加えた `generate_y_websocket_token`)、エディタの結び付けを使う。`view_file.mako` から sharejs の script タグを除く。`wiki_page_draft` とその経路、`WikiVersion.get_draft`、`get_sharejs_content`、`delete_share_doc`、`migrate_uuid`、`share_db`、`broadcast_to_sharejs` とその呼び出し、`update_active_sharejs` を除く。`SHAREJS_*` の設定、`requirements.txt` の `pymongo==3.7.1`、`addons/wiki/tests/` の試験(5 ファイル)と `tests/test_addons.py` の sharejs の部分も除く。`wiki_private_uuids` と `get_sharejs_uuid` は残す。y-websocket の部屋の名前がそこから作られるからである。
+2. *データ。* 撤去の前に、下書きが欲しい開発者は `sharejs.docs` を `mongoexport` で `runtime/backups/` に書き出す。そうでなければ捨てる。測定した wiki の読み手(`share_db`)はここでは下書きに届かない(上の 1)。ただしファイルページは自分の HTTP PUT で本文を保存するので、下書きの文が保存済みのファイルに一度も届いていないことの証明にはならない。
+3. *本リポジトリ、二つ目の PR。* 部品 `mongo` と `sharejs`、`templates/mongo.json`、`templates/sharejs.json`、`installer/images/sharejs/`、`compose render` の `sharejs.env`、公開ポート 7007、ブラウザ向けの `SHAREJS_URL`(PR #40)、ボリューム `mongo_data` と `mongo_tls`、`manifest/hardening.json` の `mongo` の項目を除く。`SHAREJS_SECRET` は osf.io が読まなくなった後でテンプレートと `secrets.py` から除く。SPEC-04 の C4 のサービスから `sharejs` を除く。osf.io の固定を手順 1 のコミットに上げる。
+
+### 4. 所有者の判断(2026-11-10)
+
+1. **Elasticsearch 2: 道 C を今行う。** osf.io の検索を全文検索を保ったまま Elasticsearch 8 へ移植する。作業は osf.io の統合リポジトリで進行中である。道 B は測定した代替として記録に残すが、計画ではない。
+2. **ファイルページ: (b) を選ぶ。** ファイルページのエディタを sharejs から y-websocket へ移し、その後 §3 の手順で sharejs と mongo を退役させる(進行中)。下書きの扱い(手順 2)はその作業の中で決まる。
+3. **M4 についても同日に決定(同じ日の決定なのでここに記録する):** MFR から freecad を除き、配布元が修正を出していない残りの OS パッケージは許容リストに入れる。
+
+### 5. 道 B の試作(著者の報告)
+
+測定のログと試作は使い捨てのファイルだったので、§2 の数値は著者の報告による。再現のための試作を英語の節 §5 に載せた。Django が検索のモジュールを用意する前に読み込んだ。
+
+---
+
+## Glossary / 用語
+
+| Term | English note | 日本語注 |
+|---|---|---|
+| Mapping type | A kind of document inside an Elasticsearch index; from version 6 an index may hold only one. | Elasticsearch のインデックスの中の文書の種類。版 6 からは一つのインデックスに一つしか置けない。 |
+| Compatibility shim | A small piece of code that makes an old client speak to a newer server. | 古いクライアントを新しいサーバーと話せるようにする小さなコード。 |
+| ShareJS | The older real-time co-editing server; it keeps documents in MongoDB. | 旧来の実時間共同編集サーバー。文書を MongoDB に置く。 |
+| y-websocket | The co-editing server the wiki uses now. | wiki が今使う共同編集サーバー。 |
diff --git a/docs/decisions/README.md b/docs/decisions/README.md
index d52d526..57a040a 100644
--- a/docs/decisions/README.md
+++ b/docs/decisions/README.md
@@ -1,15 +1,16 @@
# Decision Records / 決定の記録
-> **Summary:** Documents explaining why we chose one way and not another: integration repository placement, network model, staged security gate, AI autonomy and roles, the test IdP, and the wiki-frontend overlay. Each record captures the context, options considered, and final decision.
-> **概要:** なぜその道を選び、ほかを選ばなかったかを説明する文書:統合リポジトリの置き場、ネットワーク方式、セキュリティ門の段階化、AI の自律度と役割、テスト IdP、wiki-frontend のオーバーレイ。各記録は文脈、検討した選択肢、最終決定を捉えます。
+> **Summary:** Documents explaining why we chose one way and not another: integration repository placement, network model, staged security gate, AI autonomy and roles, the test IdP, the wiki-frontend overlay, native builds of EOL middleware, PostgreSQL 18 with its data upgrade, and the retirement of Elasticsearch 2 and MongoDB. Each record captures the context, options considered, and final decision.
+> **概要:** なぜその道を選び、ほかを選ばなかったかを説明する文書:統合リポジトリの置き場、ネットワーク方式、セキュリティ門の段階化、AI の自律度と役割、テスト IdP、wiki-frontend のオーバーレイ、EOL ミドルウェアのネイティブビルド、PostgreSQL 18 とデータの更新、Elasticsearch 2 と MongoDB の退役。各記録は文脈、検討した選択肢、最終決定を捉えます。
**Jump:** [English](#english) · [日本語](#japanese)
-- Version / 版: 0.4 — 2026-11-06. **Learned:** a cold start turned an emulation cost into an architecture decision (ADR-07): EOL amd64-only middleware is built natively. / コールドスタートがエミュレーションのコストを設計判断(ADR-07)に変えた。amd64 のみの EOL ミドルウェアはネイティブで組む。
+- Version / 版: 0.5 — 2026-11-10. **Learned:** a data-tier change is two records, not one: the PostgreSQL move is proposed with its proof (ADR-12), while the removal of Elasticsearch 2 and MongoDB still needs owner choices (ADR-13); ADR-08 to ADR-11 stay reserved or planned for records on other branches. / データ層の変更は一つでなく二つの記録になる。PostgreSQL の移行は証明つきで提案し(ADR-12)、Elasticsearch 2 と MongoDB の撤去はまだ所有者の選択を要する(ADR-13)。ADR-08〜ADR-11 は他のブランチの記録のために予約・予定のまま残す。
+ 0.4 — 2026-11-06. **Learned:** a cold start turned an emulation cost into an architecture decision (ADR-07): EOL amd64-only middleware is built natively. / コールドスタートがエミュレーションのコストを設計判断(ADR-07)に変えた。amd64 のみの EOL ミドルウェアはネイティブで組む。
0.3 — 2026-11-06. **Learned:** a repository that cannot be built alone because its Dockerfile belongs to another tree is an architecture decision (overlay, not service) and needs its own record (ADR-06). / Dockerfile が別の木に属していて単独で組めないリポジトリは、サービスでなくオーバーレイという設計上の決定であり、独自の記録(ADR-06)が要ると分かった。
0.2 — 2026-11-06. **Learned:** a test fixture that logs people in is an architecture choice like any other service, so it needs its own record (ADR-05) rather than a note in a template. / 人をログインさせる試験用の部品も他のサービスと同じ設計上の選択であり、テンプレートの注記ではなく独自の記録(ADR-05)が要ると分かった。
0.1 — 2026-11-05. **Learned:** writing decisions as records turned subjective choices into objective, reviewable documents that explain the reasoning behind each rule. / 決定を記録として残すことで、主観的な選択を客観的でレビュー可能な文書にし、各ルールの背後にある推論を説明できるようになったと分かった。
-- Normative for: [ADR-01 Integration Repository Placement](./ADR-01_Integration_Repository_Placement.md) through [ADR-07 Native Builds of EOL Middleware](./ADR-07_Native_Builds_of_EOL_Middleware.md).
+- Normative for: [ADR-01 Integration Repository Placement](./ADR-01_Integration_Repository_Placement.md) through [ADR-07 Native Builds of EOL Middleware](./ADR-07_Native_Builds_of_EOL_Middleware.md), and [ADR-12 PostgreSQL 18 and Data Directory Upgrade](./ADR-12_PostgreSQL_18_and_Data_Directory_Upgrade.md) and [ADR-13 Retiring Elasticsearch 2 and MongoDB](./ADR-13_Retiring_Elasticsearch_2_and_MongoDB.md); ADR-08 to ADR-11 are reserved or planned.
- Related: [../README.md](../README.md) · [policy/](../policy/README.md) · [spec/](../spec/README.md) · [reference/REF-01_Glossary.md](./../reference/REF-01_Glossary.md)
---
@@ -32,14 +33,17 @@ A *decision record* (ADR) explains why we chose one way and not another. This fo
| ADR-05 | [ADR-05_Test_IdP_for_Shibboleth.md](./ADR-05_Test_IdP_for_Shibboleth.md) | Bundled test IdP for `--auth shibboleth`: our own SimpleSAMLphp image on `php:8.3-apache`, mock attributes prefixed `mock-`. |
| ADR-06 | [ADR-06_Wiki_Frontend_Overlay.md](./ADR-06_Wiki_Frontend_Overlay.md) | wiki-frontend is an overlay of osf.io (`overlay_of`): a bundle-builder image from the merged tree, tagged `-`, run once by `wiki_assets` into the volume `wiki_static`; no `wiki` server. |
| ADR-07 | [ADR-07_Native_Builds_of_EOL_Middleware.md](./ADR-07_Native_Builds_of_EOL_Middleware.md) | EOL middleware that only ships amd64 images is built natively for both architectures from the official artefact on a digest-pinned base (`image_build`, MC-15); first Elasticsearch 2.4.6, which took 38 minutes to start under emulation and 5 seconds natively (real install run 19). | Proposed 2026-11-06 |
+| ADR-12 | [ADR-12_PostgreSQL_18_and_Data_Directory_Upgrade.md](./ADR-12_PostgreSQL_18_and_Data_Directory_Upgrade.md) | PostgreSQL 9.6 → 18.6 (digest-pinned); md5 host auth until CAS has a SCRAM-capable JDBC driver; `grdm-dev db upgrade` dumps and restores with verification and keeps the old directory, `db rollback` puts it back, `up` refuses a mismatched major (exit 15). Status: Proposed. |
+| ADR-13 | [ADR-13_Retiring_Elasticsearch_2_and_MongoDB.md](./ADR-13_Retiring_Elasticsearch_2_and_MongoDB.md) | Measured: osf.io search does not run on ES 6.8 or OpenSearch; search off needs a ~50-line patch; mongo serves only sharejs, whose file-page co-editing is what retirement removes. Retirement steps; owner decisions of 2026-11-10: ES 8 port with full-text search, file-page editor on y-websocket. Status: Proposed. |
-Future decision records may cover:
+Reserved or planned numbers (no record in this tree yet):
-| Topic | Description |
+| Number | Topic |
|---|---|
-| ADR-07 | Repository branching strategy |
-| ADR-08 | Tool version policy (Docker, Python, Node.js) |
-| ADR-09 | Open source license compatibility |
+| ADR-08 | Reserved: Per-OS Installer Entry Points (on a separate branch) |
+| ADR-09 | Planned: repository branching strategy |
+| ADR-10 | Planned: tool version policy (Docker, Python, Node.js) |
+| ADR-11 | Planned: open source license compatibility |
### 2. Creating a new decision record
@@ -78,14 +82,17 @@ flowchart TD
| ADR-05 | [ADR-05_Test_IdP_for_Shibboleth.md](./ADR-05_Test_IdP_for_Shibboleth.md) | `--auth shibboleth` 用の同梱テスト IdP:`php:8.3-apache` 上の自前 SimpleSAMLphp イメージ、`mock-` 接頭辞の模擬属性。 |
| ADR-06 | [ADR-06_Wiki_Frontend_Overlay.md](./ADR-06_Wiki_Frontend_Overlay.md) | wiki-frontend は osf.io のオーバーレイ(`overlay_of`):重ねた木から束を組むイメージ(タグ `-`)を `wiki_assets` が一度動かしてボリューム `wiki_static` へ写す。`wiki` サーバは無い。 |
| ADR-07 | [ADR-07_Native_Builds_of_EOL_Middleware.md](./ADR-07_Native_Builds_of_EOL_Middleware.md) | amd64 のイメージしか無い EOL ミドルウェアは、公式成果物からダイジェスト固定のベース上に両アーキテクチャ向けにネイティブで組む(`image_build`、MC-15)。まず Elasticsearch 2.4.6。エミュレーションでは起動に 38 分、ネイティブでは 5 秒(実機第 19 回)。 | 提案 2026-11-06 |
+| ADR-12 | [ADR-12_PostgreSQL_18_and_Data_Directory_Upgrade.md](./ADR-12_PostgreSQL_18_and_Data_Directory_Upgrade.md) | PostgreSQL 9.6 → 18.6(ダイジェスト固定)。CAS が SCRAM に対応した JDBC ドライバーを持つまでホスト認証は md5。`grdm-dev db upgrade` は照合つきでダンプと復元を行い旧ディレクトリを残し、`db rollback` はそれを戻し、`up` は版の不一致を拒む(終了 15)。ステータス: 提案中。 |
+| ADR-13 | [ADR-13_Retiring_Elasticsearch_2_and_MongoDB.md](./ADR-13_Retiring_Elasticsearch_2_and_MongoDB.md) | 測定: osf.io の検索は ES 6.8 でも OpenSearch でも動かない。検索を切るには約 50 行のパッチが要る。mongo は sharejs にしか使われず、退役で失うのはそのファイルページの共同編集である。退役の手順。2026-11-10 の所有者の判断: 全文検索を保つ ES 8 への移植、ファイルページのエディタの y-websocket への移行。ステータス: 提案中。 |
-今後の決定記録として以下が考えられます:
+予約済み・予定の番号(この木にはまだ記録が無い):
-| トピック | 説明 |
+| 番号 | トピック |
|---|---|
-| ADR-07 | リポジトリのブランチ戦略 |
-| ADR-08 | 道具版方針(Docker、Python、Node.js) |
-| ADR-09 | オープンソースライセンスの互換性 |
+| ADR-08 | 予約: OS ごとのインストーラーの入口(別ブランチ) |
+| ADR-09 | 予定: リポジトリのブランチ戦略 |
+| ADR-10 | 予定: 道具版方針(Docker、Python、Node.js) |
+| ADR-11 | 予定: オープンソースライセンスの互換性 |
### 2. 新しい決定記録をつくるには
diff --git a/docs/reference/REF-01_Glossary.md b/docs/reference/REF-01_Glossary.md
index 958eaf6..fbe08d7 100644
--- a/docs/reference/REF-01_Glossary.md
+++ b/docs/reference/REF-01_Glossary.md
@@ -5,7 +5,8 @@
**Jump:** [English](#english) · [日本語](#japanese)
-- Version / 版: 0.1 — 2026-11-05. **Learned:** listing every term in one place showed that about a third of them (CAS, SAML, IdP/SP, Shibboleth, GakuNin) belong to one topic — login — and that newcomers stumble there first, so the login terms were written to read as one connected story. / 全部の語を一か所に並べてみると、約三分の一(CAS、SAML、IdP/SP、Shibboleth、GakuNin)が「ログイン」という一つの話題に属し、新入社員はまずそこでつまずくと分かった。そこでログインの語は一つの続いた話として読めるように書いた。
+- Version / 版: 0.2 — 2026-11-10. **Learned:** a glossary row that names a version goes stale with the first upgrade, so the PostgreSQL row now names the record that set the version (ADR-12). / 版を書いた用語の行は最初の更新で古くなるので、PostgreSQL の行は版を決めた記録(ADR-12)を示す。
+ 0.1 — 2026-11-05. **Learned:** listing every term in one place showed that about a third of them (CAS, SAML, IdP/SP, Shibboleth, GakuNin) belong to one topic — login — and that newcomers stumble there first, so the login terms were written to read as one connected story. / 全部の語を一か所に並べてみると、約三分の一(CAS、SAML、IdP/SP、Shibboleth、GakuNin)が「ログイン」という一つの話題に属し、新入社員はまずそこでつまずくと分かった。そこでログインの語は一つの続いた話として読めるように書いた。
- Applies to: every document in this repository. Documents still add a short note on first use and link here ([SPEC-06 §3](../spec/06_documentation_conventions.md)). `grdm-dev docs check` uses the **Term** column of the English table as its word list.
- Related: [docs/README.md](../README.md) · [SPEC-00 Overview](../spec/00_overview.md) · [SPEC-01 Manifest](../spec/01_manifest.md) · [SPEC-02 Installer CLI](../spec/02_installer_cli.md) · [SPEC-03 Security Gate](../spec/03_security_gate.md) · [SPEC-04 Acceptance Tests](../spec/04_acceptance_tests.md) · [SPEC-05 Upstream Tracking](../spec/05_upstream_tracking.md) · [POL-01](../policy/POL-01_Developer_Environment_2027_Position.md) · [ADR-01](../decisions/ADR-01_Integration_Repository_Placement.md) · [ADR-02](../decisions/ADR-02_Network_Model.md) · [ADR-03](../decisions/ADR-03_Staged_Security_Gate.md) · [ADR-04](../decisions/ADR-04_AI_Autonomy_and_Roles.md)
@@ -121,7 +122,7 @@
| rdmcp / MCP | A way for AI assistants to talk to GRDM through a standard plug. | Model Context Protocol server for GRDM (`RDM-rdmcp-private`). | [SPEC-00](../spec/00_overview.md) |
| AIPF | A GRDM data connector for the AI platform (private). | `RDM-aipf_mcp_data_grdm-private`, an MCP data source for GRDM. | [SPEC-00](../spec/00_overview.md) |
| Elasticsearch | The search engine inside GRDM. | ES 2 and 6.3.1 today; consolidated on ES 8 in M5. | [POL-01 §4](../policy/POL-01_Developer_Environment_2027_Position.md) |
-| PostgreSQL | The main database where GRDM keeps its records. | PostgreSQL 9.6 today; 15 or later in M5. | [POL-01 §4](../policy/POL-01_Developer_Environment_2027_Position.md) |
+| PostgreSQL | The main database where GRDM keeps its records. | PostgreSQL 18 (ADR-12); it was 9.6 until 2026-11. | [POL-01 §4](../policy/POL-01_Developer_Environment_2027_Position.md) |
| MongoDB | An older database GRDM still uses for a few things. | MongoDB 3.4 (sharejs); retired in M5. | [POL-01 §4](../policy/POL-01_Developer_Environment_2027_Position.md) |
| RabbitMQ | The message queue: a line where jobs wait for a worker. | AMQP broker used by Celery workers in osf.io. | [SPEC-00](../spec/00_overview.md) |
| Redis | A very fast memory store used for caching and sessions. | In-memory key-value store used by osf.io and wb/mfr. | [SPEC-00](../spec/00_overview.md) |
@@ -251,7 +252,7 @@
| rdmcp / MCP | AI アシスタントが標準の差し込み口で GRDM と話す仕組み。 | GRDM 向けの Model Context Protocol サーバ(`RDM-rdmcp-private`)。 | [SPEC-00](../spec/00_overview.md) |
| AIPF | AI 基盤向けの GRDM データ接続部品(非公開)。 | `RDM-aipf_mcp_data_grdm-private`。GRDM の MCP データソース。 | [SPEC-00](../spec/00_overview.md) |
| Elasticsearch | GRDM の中の検索エンジン。 | 現在は ES 2 と 6.3.1。M5 で ES 8 に集約。 | [POL-01 §4](../policy/POL-01_Developer_Environment_2027_Position.md) |
-| PostgreSQL | GRDM が記録を保存する主なデータベース。 | 現在は PostgreSQL 9.6。M5 で 15 以降。 | [POL-01 §4](../policy/POL-01_Developer_Environment_2027_Position.md) |
+| PostgreSQL | GRDM が記録を保存する主なデータベース。 | PostgreSQL 18(ADR-12)。2026-11 までは 9.6。 | [POL-01 §4](../policy/POL-01_Developer_Environment_2027_Position.md) |
| MongoDB | GRDM が一部でまだ使っている古いデータベース。 | MongoDB 3.4(sharejs 用)。M5 で廃止。 | [POL-01 §4](../policy/POL-01_Developer_Environment_2027_Position.md) |
| RabbitMQ | メッセージの待ち行列。仕事が作業者を待つ列。 | osf.io の Celery ワーカーが使う AMQP ブローカー。 | [SPEC-00](../spec/00_overview.md) |
| Redis | キャッシュやセッションに使う、とても速いメモリの保管庫。 | osf.io と wb/mfr が使うインメモリのキー値ストア。 | [SPEC-00](../spec/00_overview.md) |
diff --git a/docs/reviews/2026-11-10_RDM-developer-guide_PR42_PostgreSQL18_r1.md b/docs/reviews/2026-11-10_RDM-developer-guide_PR42_PostgreSQL18_r1.md
new file mode 100644
index 0000000..09e8813
--- /dev/null
+++ b/docs/reviews/2026-11-10_RDM-developer-guide_PR42_PostgreSQL18_r1.md
@@ -0,0 +1,123 @@
+# PR42 PostgreSQL 18 review r1 / PostgreSQL 18 レビュー r1
+
+> **Summary:** Changes requested. Failure recovery can leave the active data directory missing or a migration server running, and the version guard fails open when the version cannot be read.
+> **概要:** 要修正。障害復旧で稼働用データディレクトリが消えた状態や移行用サーバーが動いた状態が残り、版を読めない場合は起動の防護も通過する。
+
+**Jump:** [English](#english) · [日本語](#japanese)
+
+- Version / 版: 0.1 — 2026-11-10. **Learned:** successful restore checks do not establish recoverability across directory renames and process interruption. / 復元の検査が成功しても、ディレクトリの改名やプロセス中断をまたぐ復旧可能性は証明されない。
+- Applies to: RCOSDP/RDM-developer-guide PR #42, head `2b6a02a741b68cdf4bedfc369801ccb2bb0a8d68`; REV(担当レビュワ, Codex Astra).
+- Related: [Review index](README.md) · [SPEC-06](../spec/06_documentation_conventions.md) · [SPEC-02](../spec/02_installer_cli.md) · [ADR-12](../decisions/ADR-12_PostgreSQL_18_and_Data_Directory_Upgrade.md) · [ADR-13](../decisions/ADR-13_Retiring_Elasticsearch_2_and_MongoDB.md)
+- Used by: [Status board](../status/README.md).
+
+---
+
+
+## English
+
+[→ 日本語版へ](#japanese)
+
+### Verdict and scope
+
+**Changes requested.** Review used a clean scratch clone and synthetic fake-runner probes. No real Docker stack, existing runtime secrets, main checkout, commits, pushes, approvals or merges were used. Both ADRs remain Proposed; this review does not approve the owner decisions. Real migration, Linux uid 999 ownership, amd64 execution, image registry contents and browser acceptance were not independently exercised.
+
+### Required changes
+
+1. **R1-01 · P1 — Make swap and rollback recoverable.** In [database.py](../../installer/grdm_dev/database.py), lines 458–474 rename the current directory away, rename the replacement in, and only then write `upgrade.json`. Failure of the second rename returns 15 while claiming the current directory is unchanged, but that directory is absent. A fault-injection probe reproduced exactly this state with no rollback record. A failure writing the record after both renames raises an uncaught `OSError` and leaves a completed swap without its rollback record. Rollback lines 501–506 have the same two-rename gap. A subsequent `up` can interpret an absent path as a fresh installation. Persist a durable operation journal before mutation, recover or refuse incomplete operations, compensate recoverable rename failures, and prevent `up` from initializing a missing path during pending recovery. Add failure tests at both renames and at each journal write, including rollback. Individual renames do not make the pair atomic.
+
+2. **R1-02 · P1 — Verify shutdown and handle interruption before releasing migration ownership.** `_stop` (lines 280–283) ignores both return codes; lines 420–421 and 455–456 remove the server from tracking regardless. A probe returning failure for every stop/remove still completed the upgrade with exit 0 and a success record. The exception handler catches neither `KeyboardInterrupt` nor termination signals: interrupting dump left zero stop calls. The standalone migration servers are outside the Compose project checked by `_no_running_services`. Consequently a later command can start a second server against the same directory after interruption or failed shutdown, risking corruption. Require successful shutdown plus state verification before swapping; keep ownership tracking on failures; add cleanup for catchable interruption and a persistent lock/journal that blocks `up`, upgrade and rollback after a crash. Test interruption during source start, dump, destination start, restore, verification and swap. Forced termination needs recovery on the next invocation, not just a `finally` block.
+
+3. **R1-03 · P2 — Check disk capacity before starting the source.** Lines 399–415 start the old cluster without a disk-space check. Both `dumpall.sql` and its near-identical `restore.sql` remain alongside the new cluster and the old directory. ADR-12 §4's “twice the data” wording does not account clearly for two logical dump copies, new indexes and WAL, or separate filesystems for backups and volumes. Exhaustion can also prevent the post-swap record write described in R1-01. Check both destination filesystems with a conservative margin before any source startup, explain the estimate and its limitations, and test ENOSPC in dumping, filtering, restore and journal writes. Do not promise that a fixed multiple of physical cluster size is sufficient for arbitrary data.
+
+4. **R1-04 · P2 — Fail closed on an unreadable version.** `preflight` lines 157–159 treat `None` (failed inspection) like `''` (no cluster). A probe supplying an unreadable version returned `None`, permitting Compose instead of exit 15. This matters particularly for uid 999 directories when the helper container fails to run. Distinguish an established empty directory from an unknown version and refuse unknown/unparseable versions before Compose; test the fallback's failure as well as its success.
+
+5. **R1-05 · P2 — Cover the promised sequence and database inventory.** `SNAPSHOT_SQL` lines 64–74 records only sequence `last_value`, not `is_called`, even though identical values with different `is_called` yield different next IDs. `snapshot` line 248 overwrites a real database named `roles` with the role-name list, removing its table/sequence checks completely. Separate the database and role namespaces and compare sequence state including `is_called`; test both cases. Counts, extension names and role names are useful checks but do not prove row-content equality, role attributes or extension versions. State that boundary accurately rather than claiming that snapshot equality means every row moved intact. Normal `pg_dumpall`/`ON_ERROR_STOP` restoration is a sound starting point; these findings concern the advertised verification guarantee.
+
+### Authentication, hardening and documentation
+
+**MD5 judgment:** acceptable only as a temporary, explicitly accepted compatibility exception for this isolated developer network, not as a general security claim about internal networks. PostgreSQL documents MD5 as weaker and deprecated; an MD5 HBA rule automatically negotiates SCRAM for roles whose stored verifier is SCRAM. Separate CAS and osf.io login roles therefore permit CAS MD5 and osf.io SCRAM concurrently, after verifying osf.io's actual libpq supports SCRAM and resetting its password with SCRAM encryption. Merely changing a role setting does not convert an existing verifier. Currently `compose.py` lines 878 and 963–967 give both clients the same `postgres` role, so separation and credential/configuration changes are prerequisites. JDBC 42.2 introduced SCRAM for Java 8+, but a supported patched driver should be selected rather than treating the historical minimum as the upgrade target. Sources: [PostgreSQL password authentication](https://www.postgresql.org/docs/18/auth-password.html), [JDBC 42.2 release](https://www.postgresql.org/about/news/jdbc-4220-released-1825/).
+
+**Pinning and permissions:** the target manifest and source-image constant contain full SHA-256 pins. The template retains explicit PGDATA and the parent tmpfs. Migration servers use no network, no TCP listener, read-only roots, no-new-privileges and the five documented capabilities; the version helper uses uid 999 and a read-only mount. Backup mode 0700 and files 0600 are appropriate. The password is passed through an env file rather than argv, and the normal application report does not print it. These static checks do not verify registry digest provenance or Linux execution. Also test restoration failures for sensitive SQL/COPY data in PostgreSQL container logs: suppressing Python subprocess output alone cannot establish the requirement that secrets never reach logs.
+
+**ADR-13 accuracy:** pinned osf.io source was read through GitHub at `fe90e5c39e660f464bb8fcc9ae4028119b958d19`. The search decorator returns `None` with search disabled, ES2 code uses typed documents and filtered queries, wiki defaults build the Mongo URL from PostgreSQL settings unless explicitly overridden, and the file editor uses ShareJS and independently saves text by HTTP PUT. These support the broad migration concerns. However, the failed wiki draft reader does not prove that no ShareJS draft content ever reached saved files: narrow that claim to the measured wiki reader. Treat about 5,300 lines as the affected code surface, not a measured requirement to rewrite every line, and “every page loads” as limited to the measured endpoints. The prototype patch, complete measurement logs and post-PR40 browser result were not available in this PR, so their numerical and universal claims remain author-reported. No Mongo or ES service is removed by this PR. The new ADRs have bilingual sections and Proposed status; docs_check passes. ADR-12's inline operational command sequence should become a fenced procedure with working directory and expected results under SPEC-06 §3, and its unconditional failure/unchanged-directory promises must be corrected with R1-01/R1-02.
+
+### Validation evidence
+
+These are review measurements, not canonical manifest data. System Python was `/usr/bin/python3` 3.9.6.
+
+```toon
+checks[5]{check,exit,result}:
+ "full unit suite",1,"431 tests; 12 errors; 10 skipped"
+ "database unit suite",0,"17 tests passed"
+ "docs_check before review",0,"144 files; 8 exempt; 37 warnings"
+ "manifest_check",0,"0 findings; 665 tracked files scanned"
+ "synthetic failure probes",0,"second rename leaves current absent; stop failures report success; interruption skips cleanup; journal failure escapes; unknown version allowed"
+```
+
+All 12 suite errors arise from the Python 3.9-incompatible annotation in `test_compose.py:38` and dependent imports. That file is unchanged from base `48d0134347e681c00ebb95246eab26f9e7de3c92`; the full suite is nevertheless **not green** on system Python. The database tests do pass. Logs and the isolated probe script are in `/tmp/pr42-rev-yvCJXJ/`; they contain synthetic fixtures only.
+
+Reproduce from the scratch clone root; expected exits are shown in the measurement block:
+
+```sh
+PYTHONPATH=installer /usr/bin/python3 -m unittest discover -s installer/tests
+PYTHONPATH=installer /usr/bin/python3 -m unittest discover -s installer/tests -p test_database.py
+/usr/bin/python3 tools/docs_check.py
+/usr/bin/python3 tools/manifest_check.py
+/usr/bin/python3 /tmp/pr42-rev-yvCJXJ/probe.py
+```
+
+---
+
+
+## 日本語
+
+[→ To English](#english)
+
+### 判定と範囲
+
+**要修正(changes requested)。** 作業用の独立クローンと疑似 runner による障害注入を使った。実 Docker スタック、既存の runtime の秘密情報、主チェックアウトには接触せず、コミット・push・承認・マージを行っていない。両 ADR は Proposed のままで、所有者の判断を承認していない。実移行、Linux の uid 999 所有、amd64 実行、レジストリ内のイメージ内容、ブラウザ受入は独立に実施していない。
+
+### 必須修正
+
+1. **R1-01 · P1 — 切替と巻戻しを復旧可能にする。** [database.py](../../installer/grdm_dev/database.py) 458–474 行は現行ディレクトリを退避し、新ディレクトリを移し、その後で初めて `upgrade.json` を書く。二回目の改名に失敗させると終了 15 だが、報告の「現行は無変更」に反して現行パスが消え、巻戻し記録も無かった。切替後の記録書込み失敗は未捕捉の `OSError` となる。巻戻しの 501–506 行にも二回の改名の隙間がある。後続の `up` が空の新規環境と解釈しかねない。変更前に永続的な操作記録を書き、不完全な操作を復旧するか拒否し、回復可能な改名失敗を補償すること。復旧待ちの `up` に初期化させないこと。両改名と各記録書込みの失敗を巻戻しも含めて試験する。一回ずつの改名が原子的でも、二回全体は原子的ではない。
+
+2. **R1-02 · P1 — 停止を検証し、中断後も移行の占有を管理する。** `_stop`(280–283 行)は停止と削除の終了値を無視し、呼出し側は成功扱いで追跡対象から外す。すべての停止・削除を失敗させても移行は終了 0 と成功記録になった。ダンプ中の `KeyboardInterrupt` では停止呼出しが一度も無かった。例外処理はこれや終了シグナルを扱わず、単独起動した移行サーバーは Compose の稼働検査の外にいる。そのため次のコマンドが同じディレクトリに二つ目のサーバーを起動し、破損する危険がある。切替前に停止成功と状態確認を要求し、失敗時は追跡を保持すること。捕捉可能な中断の cleanup と、クラッシュ後に `up`・upgrade・rollback を止める永続ロック/操作記録が必要。旧版起動・ダンプ・新版起動・復元・照合・切替の各段階で中断試験を行うこと。強制終了には次回の復旧が必要で、`finally` だけでは足りない。
+
+3. **R1-03 · P2 — 旧版起動前に空き容量を検査する。** 399–415 行には容量検査がない。新旧クラスタに加えて、ほぼ同じ内容の `dumpall.sql` と `restore.sql` の両方が残る。ADR-12 §4 の「データの二倍」では二つの論理ダンプ、新しい索引、WAL、バックアップとボリュームが別ファイルシステムの場合を明確に説明していない。容量不足は R1-01 の切替後の記録にも影響する。両保存先に保守的な余裕を見て事前検査し、見積りの限界を説明すること。ダンプ・加工・復元・記録の ENOSPC を試験し、物理サイズの固定倍率で任意のデータに十分とは約束しないこと。
+
+4. **R1-04 · P2 — 版の読取り失敗を拒否する。** `preflight` 157–159 行は `None`(検査失敗)を空文字(クラスタなし)と同じ扱いにする。読取り不能を与えると拒否理由なしになり、終了 15 でなく Compose へ進めた。uid 999 所有のパスで補助コンテナが起動できない場合に問題となる。空であると確認できた場合と不明な場合を区別し、版不明・解析不能なら Compose 前に拒否すること。代替読取りの成功だけでなく失敗も試験すること。
+
+5. **R1-05 · P2 — シーケンスと DB 一覧の検証範囲を補う。** 64–74 行は `last_value` のみを記録し、次に発行される ID を変える `is_called` を比較しない。248 行は実在する `roles` という DB の検査結果をロール名一覧で上書きし、その表とシーケンスの比較を失う。DB とロールの名前空間を分け、`is_called` を含めて比較し、両例を試験すること。行数・拡張名・ロール名の比較は有用だが、行内容、ロール属性、拡張の版の一致まで証明しない。比較一致が全行の完全な移行を意味するという表現を狭めること。通常の `pg_dumpall` と `ON_ERROR_STOP` は妥当な出発点で、この指摘は検証保証の範囲に対するもの。
+
+### 認証・堅牢化・文書
+
+**MD5 の判断:** この隔離された開発ネットワークに限定し、明示的に受容する暫定互換性例外としては許容できる。内部ネットワーク一般の安全性を保証するものではない。PostgreSQL は MD5 を弱く非推奨とし、MD5 の HBA 規則でも保存済み検証子が SCRAM なら SCRAM を選ぶ。従って CAS と osf.io のログインロールを分け、osf.io の実際の libpq の対応を確認して SCRAM 形式でパスワードを設定し直せば併存できる。ロールの設定値だけ変えても既存検証子は変換されない。現状は `compose.py` 878、963–967 行で両者が `postgres` を共用しているため、ロール分離と資格情報・設定変更が前提。JDBC 42.2 は Java 8 以降向けに SCRAM を導入したが、更新先は歴史的最低版でなく保守済み修正版を選ぶべきである。根拠: [PostgreSQL 認証仕様](https://www.postgresql.org/docs/18/auth-password.html)、[JDBC 42.2 リリース](https://www.postgresql.org/about/news/jdbc-4220-released-1825/)。
+
+**固定と権限:** 移行先 manifest と移行元定数は完全な SHA-256 固定を持つ。テンプレートは明示的 PGDATA と親 tmpfs を維持する。移行サーバーはネットワーク・TCP なし、読取り専用 root、no-new-privileges、記録済みの五 capability を使う。版読取りは uid 999 と読取り専用 mount を使う。バックアップ 0700、ファイル 0600 は適切。通常経路では秘密値は argv でなく env file を通り、アプリケーションの報告に出ない。ただしレジストリの真正性や Linux 実行を確認したわけではない。復元失敗時に秘密を含む SQL/COPY データが PostgreSQL コンテナログへ出ないことも試験すべきで、Python 側の subprocess 出力を捨てるだけではログへの秘密流出禁止を証明できない。
+
+**ADR-13 の正確さ:** osf.io の固定コミット `fe90e5c39e660f464bb8fcc9ae4028119b958d19` の関連ソースを GitHub 経由で読んだ。無効時に検索 decorator が `None` を返すこと、ES2 の型と filtered query、環境変数で上書きされなければ PostgreSQL 設定から Mongo URL を組み立てる wiki 設定、ShareJS を使いつつ HTTP PUT で保存するファイルエディタは、移行上の懸念を裏付ける。ただし wiki の draft reader の失敗から、ShareJS の下書き内容が保存済みファイルに一度も到達していないとは証明できない。測定した wiki reader に主張を限定すること。約 5,300 行は影響するコード範囲であり全行書直しの実測量ではない。「すべてのページが表示」は測定した入口に限るべきである。試作パッチ、完全な測定ログ、PR40 後のブラウザ結果は PR 内にないため、数値と普遍的な主張は著者報告のまま。今回 Mongo/ES の撤去はない。新 ADR は英和構成と Proposed を備え docs_check は通る。ADR-12 のインライン操作列は SPEC-06 §3 に従う実行場所・期待結果付き fenced 手順にし、無条件の「失敗時無変更」の約束は R1-01/R1-02 と一緒に修正すること。
+
+### 検証結果
+
+レビューでの測定であり manifest の正本データではない。システム Python は `/usr/bin/python3` 3.9.6。
+
+```toon
+checks[5]{check,exit,result}:
+ "全単体試験",1,"431 件; エラー 12; スキップ 10"
+ "database 単体試験",0,"17 件合格"
+ "記録追加前 docs_check",0,"144 ファイル; 例外 8; 警告 37"
+ "manifest_check",0,"指摘 0; 追跡ファイル 665 件走査"
+ "疑似障害注入",0,"第二改名失敗で現行消失; 停止失敗でも成功; 中断で cleanup なし; 記録失敗は未捕捉; 版不明を許可"
+```
+
+全試験の 12 エラーは Python 3.9 で解釈できない `test_compose.py:38` の型注釈とその依存 import に由来する。このファイルは base `48d0134347e681c00ebb95246eab26f9e7de3c92` から無変更だが、システム Python で全試験合格とは報告しない。database の 17 件は合格した。ログと独立 probe は `/tmp/pr42-rev-yvCJXJ/` にあり、疑似データのみを扱う。
+
+独立クローンのルートで再現する。期待する終了値は測定表の通り。
+
+```sh
+PYTHONPATH=installer /usr/bin/python3 -m unittest discover -s installer/tests
+PYTHONPATH=installer /usr/bin/python3 -m unittest discover -s installer/tests -p test_database.py
+/usr/bin/python3 tools/docs_check.py
+/usr/bin/python3 tools/manifest_check.py
+/usr/bin/python3 /tmp/pr42-rev-yvCJXJ/probe.py
+```
diff --git a/docs/reviews/2026-11-10_RDM-developer-guide_PR42_PostgreSQL18_r2.md b/docs/reviews/2026-11-10_RDM-developer-guide_PR42_PostgreSQL18_r2.md
new file mode 100644
index 0000000..6b9ee7f
--- /dev/null
+++ b/docs/reviews/2026-11-10_RDM-developer-guide_PR42_PostgreSQL18_r2.md
@@ -0,0 +1,119 @@
+# PR42 PostgreSQL 18 review r2 / PostgreSQL 18 レビュー r2
+
+> **Summary:** Changes requested. Ordinary interruption recovery improves substantially, but recovery can discard its journal after failed compensation, and lifecycle operations do not share the migration lock throughout their critical sections.
+> **概要:** 要修正。通常の中断復旧は大きく改善したが、補償失敗後に復旧記録を捨てる経路があり、起動・リセットとの排他も完結していない。
+
+**Jump:** [English](#english) · [日本語](#japanese)
+
+- Version / 版: 0.1 — 2026-11-10. **Learned:** recovery itself needs failure tests and durability ordering; a journal and a lock are insufficient when a recovery error deletes the journal or another command bypasses the lock. / 復旧自身にも障害試験と永続化の順序が要る。復旧の失敗で記録を削除したり、別コマンドがロックを迂回したりすると、記録とロックがあるだけでは守れない。
+- Applies to: RCOSDP/RDM-developer-guide PR #42, diff `2b6a02a..5780b16`, head `5780b16e1592c070f62de6a56832b50c1914e649`; REV(担当レビュワ, Codex Astra).
+- Related: [Review index](README.md) · [r1](2026-11-10_RDM-developer-guide_PR42_PostgreSQL18_r1.md) · [SPEC-06](../spec/06_documentation_conventions.md) · [SPEC-02](../spec/02_installer_cli.md) · [ADR-12](../decisions/ADR-12_PostgreSQL_18_and_Data_Directory_Upgrade.md) · [ADR-13](../decisions/ADR-13_Retiring_Elasticsearch_2_and_MongoDB.md).
+- Used by: [Status board](../status/README.md).
+
+---
+
+
+## English
+
+[→ 日本語版へ](#japanese)
+
+### Verdict and required changes
+
+**Changes requested.** All supplied unit suites pass on system Python, but the independent probes reproduce two unsafe recovery/concurrency cases. References are to [database.py](../../installer/grdm_dev/database.py) and [lifecycle.py](../../installer/grdm_dev/lifecycle.py) at the reviewed head.
+
+1. **R2-01 · P1 — Keep the journal if compensation also fails (R1-01 remains open).** `recover` lines 576–581 and 589–594 catch every `OSError` from `_swap` and delete the journal while claiming the current directory is unchanged. `_swap` lines 606–611 can raise because both the replacement rename and the compensating rename failed, leaving the current directory absent. Reproduction: crash before the first swap rename, then recover with the first rename successful and the next two returning EIO. Both upgrade and rollback recovery return **0**, with **no current directory and no journal**. A later `up` can treat that missing directory as a new installation. Distinguish a verified compensated result from an incomplete swap; preserve the journal and return 15 unless the actual directory state establishes a safe completed outcome. Cover repeat failures during recovery, including failed compensation.
+
+2. **R2-02 · P1 — Serialize lifecycle changes with migration, including the period before journal creation (R1-02 remains open).** `recover_pending` lines 627–631 bypasses the lock when no journal exists, and releases it before `up` starts Compose or `reset` deletes volumes. Upgrade/rollback check running services before confirmation and before writing the journal. A second `up` can therefore start PostgreSQL after rollback checked for stopped services but before rollback swaps its data directories. An injected interleaving at `Context.confirm` reproduced a Compose `up` call while rollback held its lock; rollback then returned 0 and replaced the current 18 directory with 9.6. A simpler held-lock/no-journal probe also reached Compose. The fake health result was exit 40, not a DB refusal; no real server was started. Make lifecycle and DB commands use the same lock even when no journal exists, and hold it through the start/deletion operation and necessary checks. Test both directions of the race and `reset`.
+
+3. **R2-03 · P1 — Persist recovered renames before retiring the journal (R1-01 remains open).** Recovery's direct renames at lines 571, 573 and 596 never sync the volumes parent before writing the record and deleting the journal. The normal `_swap` path does sync it, showing the intended ordering. A probe recovering a crash between the two renames observed exit 0 and no volumes-parent sync for both upgrade and rollback. On a power loss, the record/journal deletion can survive without the recovered directory rename; process-exception tests cannot prove power-loss durability. Also `_fsync_dir` lines 284–294 silently ignores open/fsync errors, so even the initial write-ahead journal may be treated as durable after EIO. Sync every affected directory before declaring recovery complete, propagate unexpected durability errors while retaining recoverability, and test sync failures and ordering. No physical power-loss test was performed; the missing ordering is established by code inspection and call instrumentation.
+
+4. **R2-04 · P2 — Convert invalid encoding in PG_VERSION into exit 15 (R1-04 partially open).** `data_major` line 184 reads ASCII but catches only filesystem exceptions. A PG_VERSION containing byte `0xff` raises an uncaught `UnicodeDecodeError` through `up`; it does not return the documented refusal code 15. Compose is not reached, so this is an error-contract defect rather than a fail-open start. Treat decoding failures as an unknown version and add a test through the command entry point.
+
+### R1 closure and owner decisions
+
+**R1-03 closed for the reviewed scope:** disk capacity is checked before source startup; same/different filesystem budgets include both dump copies, the replacement cluster and margins; the documentation states the estimate is not a guarantee. Supplied low-space and ENOSPC tests pass. **R1-05 closed:** the SQL includes `is_called`, database snapshots use a separate namespace from roles, and tests detect a changed `roles` database and sequence state. Snapshot equality is correctly limited to counts/state/names/selected role attributes, not row-content equality. R1-01/R1-02 remain open for the findings above; R1-04 is otherwise improved and fails closed for ordinary unreadable, missing-in-nonempty and malformed ASCII cases.
+
+ADR-13 now records the owner's choices: Elasticsearch 8 with full-text search; the file-page editor on y-websocket before retiring sharejs/mongo; MFR freecad removal and allow-listing distribution packages without fixes. These are recorded choices, not implementations validated by PR #42. The draft export/discard decision remains for the migration work. The narrower claims about affected code size, probed endpoints and the wiki draft reader address r1; prototype measurements are explicitly author-reported. Minor editorial inconsistency: the header still says “Records a decision for: nothing yet” although §4 records choices; align that line while retaining the distinction between chosen direction and the Proposed record. No further owner approval is inferred by this review.
+
+ADR-12's unconditional claim that a crash at any point is recoverable is premature until R2-01–03 are resolved. Its disk paragraph also says a full disk leaves the old cluster in place, while the supplied post-swap ENOSPC tests intentionally finish the new cluster; describe both safe recovery outcomes.
+
+### Validation and limits
+
+Clean clone: `/tmp/pr42-rev-r2/repo`; system interpreter: `/usr/bin/python3` 3.9.6. These are review measurements, not canonical manifest data.
+
+```toon
+checks[5]{check,exit,result}:
+ "installer unit suite",0,"573 tests; 10 skipped"
+ "tools unit suite",0,"93 tests passed"
+ "docs_check before report",0,"145 files; 8 exempt; 36 warnings"
+ "manifest_check",0,"0 findings; 666 tracked files scanned; 63 skipped"
+ "independent synthetic probes",0,"single-crash boundaries recover; repeated rename failure, lock bypass, missing directory sync and decode exception reproduced"
+```
+
+The independent script exercised 162 injected crash boundaries (128 upgrade, 34 rollback), all recovered under the single-crash model. It wraps both sides of every observed Docker call, durable JSON write, rename/replace, fsync, mkdir/unlink, snapshot, readiness check, restore-file construction and journal removal in successful upgrade and rollback traces. Each injected uncatchable exception is followed by recovery and checks for a readable current major, no pending journal, no tracked server and an upgrade record when current is 18. It also probes repeated recovery failures and the lifecycle interleaving. This is a synthetic process-crash model, not emulated filesystem persistence or a real kill of Docker. Supplied tests additionally cover catchable interruption, SIGTERM, stop/removal failures, a competing DB process, disk checks and inventory comparisons.
+
+Logs and runnable probe are `/tmp/pr42-rev-r2/{units,tools-units,docs,manifest,probe}.log` and `/tmp/pr42-rev-r2/probe.py`. Reproduce from the clone root; expected exits are 0 (probe exit 0 means the reported behavior was reproduced, not that the implementation passed review):
+
+```sh
+PYTHONPATH=installer /usr/bin/python3 -m unittest discover -s installer/tests
+/usr/bin/python3 -m unittest discover -s tools/tests
+/usr/bin/python3 tools/docs_check.py
+/usr/bin/python3 tools/manifest_check.py
+/usr/bin/python3 /tmp/pr42-rev-r2/probe.py
+```
+
+No real grdm-dev Docker state or existing runtime/.env was accessed; unit fixtures use temporary synthetic credentials. No commit, push, approval, merge or closure was performed. Real migration, SQL execution on both PostgreSQL versions, Linux ownership, amd64 execution, power-loss persistence and browser acceptance were not independently exercised. The real-operation evidence in ADR-12 remains author-reported. The untracked report was also checked directly with `docs_check.check_file` (0 errors). Only this review file is added to the clone; the probe and logs are outside it.
+
+---
+
+
+## 日本語
+
+[→ To English](#english)
+
+### 判定と必須修正
+
+**要修正(changes requested)。** システム Python で提供された単体試験はすべて合格したが、独立試験で復旧と競合の危険な経路を再現した。行番号は対象 head の [database.py](../../installer/grdm_dev/database.py) と [lifecycle.py](../../installer/grdm_dev/lifecycle.py) を指す。
+
+1. **R2-01 · P1 — 補償も失敗した場合は記録を保持する(R1-01 未解決)。** `recover` 576–581、589–594 行は `_swap` の全 `OSError` を捕捉し、現行は無変更と報告して記録を削除する。しかし `_swap` 606–611 行は、置換の改名と補償の改名が両方失敗し、現行が消えた場合にも例外を送出する。最初の改名前にクラッシュさせ、次の復旧で一回目だけ成功、二・三回目を EIO にすると、upgrade と rollback の両方で **終了 0、現行パスなし、記録なし** となった。後続 `up` は新規環境として扱い得る。補償成功を実際のディレクトリ状態で確認し、不完全なら記録を保持して終了 15 にすること。復旧中の再障害・補償失敗を試験すること。
+
+2. **R2-02 · P1 — 記録作成前も含め、起動・リセットと移行を排他にする(R1-02 未解決)。** `recover_pending` 627–631 行は記録がなければロックを取らず、記録があっても Compose 起動やボリューム削除前に解放する。upgrade/rollback は停止検査後に確認を求め、その後で記録を書く。この間に別の `up` が起動すると、rollback は使用中のデータを改名する。`Context.confirm` で割り込ませると、rollback がロック保持中にも Compose `up` に到達し、rollback は終了 0 で現行を 18 から 9.6 へ替えた。ロックだけを保持する試験でも起動に到達した。疑似 health の終了値は 40 であり DB 拒否ではない。実サーバーは起動していない。記録の有無によらず共通ロックを取り、必要な検査から起動・削除まで保持すること。逆順の競合と `reset` も試験すること。
+
+3. **R2-03 · P1 — 復旧改名を永続化してから記録を捨てる(R1-01 未解決)。** 復旧の直接改名(571、573、596 行)には volumes 親の同期がなく、記録更新・削除へ進む。通常の `_swap` は同期している。二回の改名の間でクラッシュさせた後の復旧では、upgrade/rollback とも終了 0 だが volumes 親の同期は一度もなかった。電源断で記録更新・削除だけが残り、改名が残らない可能性がある。さらに `_fsync_dir` 284–294 行は open/fsync の失敗を無視し、先行記録の EIO でも永続化済み扱いになる。影響する親を同期してから完了とし、予期しない永続化失敗を返して復旧可能性を残すこと。同期の失敗と順序を試験すること。物理的な電源断は行っておらず、コードと呼出し記録で順序の欠落を確認した。
+
+4. **R2-04 · P2 — PG_VERSION の文字コード不正も終了 15 にする(R1-04 一部未解決)。** `data_major` 184 行は ASCII で読むが、捕捉するのはファイルシステムの例外だけである。`0xff` を含む PG_VERSION では `up` から未捕捉の `UnicodeDecodeError` が出る。Compose には到達しないため、誤起動ではなく終了値の契約違反である。版不明として扱い、コマンド入口から終了 15 を試験すること。
+
+### R1 の解決状況と所有者の判断
+
+**R1-03 は今回の範囲で解決。** 旧サーバー起動前に容量を検査し、同一/別ファイルシステムで二つのダンプ・新クラスタ・余裕を計算する。固定倍率は保証ではないと明示され、空き不足・ENOSPC の試験も合格した。**R1-05 は解決。** SQL は `is_called` を含み、DB とロールの名前空間を分け、`roles` DB とシーケンスの変化を検出する。比較は行数・状態・名前・一部ロール属性までで、行内容の同一性は保証しないと明記した。R1-01/R1-02 は前記の理由で未解決。R1-04 は通常の読取り不能・非空ディレクトリの版欠落・不正な ASCII については拒否するよう改善した。
+
+ADR-13 は所有者の選択として、全文検索を維持した Elasticsearch 8、sharejs/mongo 退役前のファイルエディタの y-websocket 移行、MFR freecad 撤去と配布元の修正がない OS パッケージの許容を記録した。PR #42 が実装・検証した結果ではない。下書きの書出し/破棄は移行作業で決める事項として残る。影響コード量・測定した入口・wiki の下書き reader へ主張を狭め、試作の測定を著者報告とした点は r1 に対応する。軽微な文言として、§4 に選択を記録しながらヘッダは “Records a decision for: nothing yet” のままなので、方向の決定と記録の Proposed 状態を区別して整合させたい。このレビューが追加の所有者承認を与えるものではない。
+
+ADR-12 の「どの時点のクラッシュも復旧」は R2-01–03 解決前には強すぎる。また容量不足なら旧クラスタがそのまま残るとの記述に対し、切替後 ENOSPC の単体試験は新版への復旧完了を期待する。安全な二つの復旧結果を説明すること。
+
+### 検証と限界
+
+独立クローンは `/tmp/pr42-rev-r2/repo`、システム Python は `/usr/bin/python3` 3.9.6。次はレビュー測定であり manifest の正本ではない。
+
+```toon
+checks[5]{check,exit,result}:
+ "installer 単体試験",0,"573 件; 10 件スキップ"
+ "tools 単体試験",0,"93 件合格"
+ "記録追加前 docs_check",0,"145 ファイル; 例外 8; 警告 36"
+ "manifest_check",0,"指摘 0; 追跡 666 ファイル走査; 63 件スキップ"
+ "独立疑似試験",0,"単発クラッシュから復旧; 連続改名失敗・ロック迂回・親同期欠落・デコード例外を再現"
+```
+
+独立スクリプトは 162 箇所(upgrade 128、rollback 34)にクラッシュを注入し、単発クラッシュモデルでは全件復旧した。成功する upgrade/rollback の全観測 Docker 呼出し、永続 JSON 書込み、rename/replace、fsync、mkdir/unlink、snapshot、起動待ち、復元ファイル作成、記録削除の前後に捕捉不能の例外を注入した。次回復旧で現行版が読めること、記録と追跡サーバーが残らないこと、現行 18 なら upgrade 記録があることを検査した。復旧の連続障害と起動の割込みも試した。これは疑似プロセスクラッシュであり、ファイルシステム永続化の模擬や実 Docker の kill ではない。提供試験では通常の中断、SIGTERM、停止・削除失敗、別 DB プロセス、容量、照合も検証した。
+
+ログは `/tmp/pr42-rev-r2/{units,tools-units,docs,manifest,probe}.log`、再現用スクリプトは `/tmp/pr42-rev-r2/probe.py`。クローンのルートで次を実行すると全コマンドの期待終了値は 0。probe の 0 は問題の再現成功であり、レビュー合格ではない。
+
+```sh
+PYTHONPATH=installer /usr/bin/python3 -m unittest discover -s installer/tests
+/usr/bin/python3 -m unittest discover -s tools/tests
+/usr/bin/python3 tools/docs_check.py
+/usr/bin/python3 tools/manifest_check.py
+/usr/bin/python3 /tmp/pr42-rev-r2/probe.py
+```
+
+実 grdm-dev の Docker 状態と既存 runtime/.env には触れていない。単体試験の資格情報は一時ディレクトリの疑似値である。コミット・push・承認・マージ・close は行っていない。実移行、両 PostgreSQL 版での SQL 実行、Linux 所有権、amd64、電源断の永続化、ブラウザ受入は独立検証していない。ADR-12 の実運用測定は著者報告として扱う。未追跡のレビュー記録も `docs_check.check_file` で直接検査し、エラー 0 を確認した。クローン内への追加はこのレビューのみで、probe とログは外に置いた。
diff --git a/docs/reviews/2026-11-10_RDM-developer-guide_PR42_PostgreSQL18_r3.md b/docs/reviews/2026-11-10_RDM-developer-guide_PR42_PostgreSQL18_r3.md
new file mode 100644
index 0000000..4919ee5
--- /dev/null
+++ b/docs/reviews/2026-11-10_RDM-developer-guide_PR42_PostgreSQL18_r3.md
@@ -0,0 +1,100 @@
+# PR42 PostgreSQL 18 review r3 / PostgreSQL 18 レビュー r3
+
+> **Summary:** Changes requested: the four r2 reproductions are fixed, but a durability error during pending recovery escapes the `db upgrade` command instead of returning 15.
+> **概要:** 要修正。r2 の四つの再現事象は修正済みだが、保留中の復旧で同期に失敗すると `db upgrade` は終了 15 を返さず例外を外へ出す。
+
+**Jump:** [English](#english) · [日本語](#japanese)
+
+- Version / 版: 0.1 — 2026-11-10. **Learned:** propagating durability errors also requires every command that invokes recovery to translate them into its documented exit code. / 永続化の失敗を伝える修正には、復旧を呼ぶすべてのコマンドで、その失敗を規定の終了値へ変換する必要がある。
+- Applies to: RCOSDP/RDM-developer-guide PR #42, diff `5780b16..7e61df3`, head `7e61df3c84207c9342745f5c80d18ac708f0bb63`; REV(担当レビュワ, Codex Astra).
+- Related: [Review index](README.md) · [r2](2026-11-10_RDM-developer-guide_PR42_PostgreSQL18_r2.md) · [SPEC-06](../spec/06_documentation_conventions.md) · [SPEC-02](../spec/02_installer_cli.md) · [ADR-12](../decisions/ADR-12_PostgreSQL_18_and_Data_Directory_Upgrade.md).
+- Used by: [Status board](../status/README.md).
+
+---
+
+
+## English
+
+[→ 日本語版へ](#japanese)
+
+### Verdict and finding
+
+**Changes requested. R3-01 · P2: translate recovery OSError in `db upgrade` to exit 15.** In [database.py](../../installer/grdm_dev/database.py), `run_upgrade` (lines 751–760) catches only `UpgradeError`, but `_upgrade_locked` calls `recover` at line 765 before entering its later error handler. With a pending swap whose two renames have finished, an EIO from `_fsync_dir(current.parent)` in `recover` (line 608) escapes through the actual `cli.main` entry point as an uncaught `OSError`. The CLI catches only KeyboardInterrupt. The new strict fsync behavior exposes this route; `run_recover` and `run_rollback` already catch both exception types.
+
+Independent reproduction: create a valid pending upgrade at the swap boundary, perform its two renames without retiring its journal (the state after a crash before record creation), inject EIO for the volumes-parent sync, then call `cli.main(['--root', fixture_root, '--yes', 'db', 'upgrade'])` with a fake Docker runner. It raises OSError rather than returning 15. Current remains PostgreSQL 18 and the journal remains; a later fault-free `db recover` succeeds. This is an error-contract defect, not observed data loss or a fail-open start. Catch OSError around the complete locked upgrade including initial recovery, retain the journal, and add an entry-point test for this state.
+
+### R2 closure and regression checks
+
+- **R2-01 closed:** both upgrade and rollback recovery return 15 and retain the journal when replacement and compensation renames both fail. Persistent subsequent failures also refuse `up` and recovery; removing the fault permits recovery to finish with the expected major and no journal.
+- **R2-02 closed:** the common lock excludes `up` even without a journal and throughout rollback confirmation. Independent callbacks during Compose start, reset shutdown and reset deletion reject all three DB actions with 15. Supplied tests also check the reverse reset direction and a competing process. No real Docker command was executed by these probes.
+- **R2-03 original reproduction fixed; R3-01 remains:** recovered upgrade/rollback renames sync the volumes parent before journal removal, and EIO is no longer swallowed. Supplied ordering and persistent-sync-error tests pass. The documented EINVAL/ENOTSUP exceptions remain intentional; process-crash tests do not prove power-loss durability.
+- **R2-04 closed:** non-ASCII PG_VERSION now returns 15 through `up`, without starting Compose; supplied tests also cover unreadable helper output.
+
+The original 162-point matrix (128 upgrade, 34 rollback) passes unchanged in boundary count. It injects an uncatchable exception before and after every observed Docker call, durable JSON write, rename/replace, fsync, mkdir/unlink, snapshot, readiness wait, restore-file construction and journal removal. Recovery checks a readable current major, absence of a pending journal and tracked servers, and an upgrade record when current is 18. Double failures and lifecycle interleavings are separate probes. ADR-12 now distinguishes pre-swap and post-swap disk-full outcomes; ADR-13 aligns its header with the recorded owner choices. No other blocking regression was found in the scoped diff.
+
+### Validation and limits
+
+Clean clone: `/tmp/pr42-rev-r3/repo`; system Python: `/usr/bin/python3` 3.9.6. Results are review measurements, not canonical manifest data.
+
+```toon
+checks[6]{check,exit,result}:
+ "installer unit suite",0,"583 tests; 10 skipped"
+ "tools unit suite",0,"93 tests"
+ "docs_check",0,"146 files; 8 exempt; 36 warnings"
+ "manifest_check",0,"0 findings; 667 tracked files scanned; 63 skipped"
+ "independent r2 probes",0,"162 crash boundaries plus double failures, lock, sync and encoding checks pass"
+ "extra probes",0,"concurrency passes; R3-01 uncaught exception reproduced; later recovery passes"
+```
+
+Run from the clone root; all listed script exits are 0. The extra probe's exit 0 means the defect was reproduced, not review approval. Logs and scripts are outside the clone in `/tmp/pr42-rev-r3/`.
+
+```sh
+PYTHONPATH=installer /usr/bin/python3 -m unittest discover -s installer/tests
+/usr/bin/python3 -m unittest discover -s tools/tests
+/usr/bin/python3 tools/docs_check.py
+/usr/bin/python3 tools/manifest_check.py
+/usr/bin/python3 /tmp/pr42-rev-r3/probe.py
+/usr/bin/python3 /tmp/pr42-rev-r3/extra.py
+```
+
+The untracked review was checked directly with `docs_check.check_file` (0 errors). Only this report is added to the clone; no commit, push, merge, approval or closure is performed. One Japanese PR comment reports the verdict. Existing runtime/.env and the grdm-dev Docker stack were not accessed or changed; fixtures use temporary synthetic credentials. Real PostgreSQL migration, Linux ownership, amd64, physical power loss and browser acceptance were not independently exercised. The matrix models process interruptions, not filesystem persistence or live Docker kills.
+
+---
+
+
+## 日本語
+
+[→ To English](#english)
+
+### 判定と指摘
+
+**要修正(changes requested)。R3-01 · P2: `db upgrade` の復旧中の OSError を終了 15 にする。** [database.py](../../installer/grdm_dev/database.py) の `run_upgrade`(751–760 行)は `UpgradeError` だけを捕捉するが、`_upgrade_locked` は後段の例外処理より前(765 行)で `recover` を呼ぶ。二つの改名が完了し操作記録が残っている状態で、`recover` の `_fsync_dir(current.parent)`(608 行)が EIO になると、実際の `cli.main` から未捕捉の OSError が出る。CLI が捕捉するのは KeyboardInterrupt だけである。今回の fsync 失敗を伝える修正でこの経路が表面化した。`run_recover` と `run_rollback` は両例外を捕捉済みである。
+
+独立再現では、切替直前の有効な操作記録を作り、二つの改名だけを完了させた(記録作成前のクラッシュ後に相当)。volumes 親の同期へ EIO を注入し、疑似 Docker を使って `cli.main(['--root', fixture_root, '--yes', 'db', 'upgrade'])` を呼ぶと、15 を返さず OSError が出た。現行は PostgreSQL 18、操作記録も残り、障害を除いた後の `db recover` は成功する。観測したのは終了値の契約違反であり、データ喪失や誤起動ではない。ロック内の初回復旧を含む全体で OSError を捕捉し、操作記録を保持して 15 を返すこと。この状態をコマンド入口から試験すること。
+
+### R2 の解決と回帰確認
+
+- **R2-01 解決。** upgrade/rollback とも置換・補償の改名が二重に失敗すると終了 15 で記録を保持した。障害が続く間は `up` と復旧を拒否し、除去後は期待する版へ復旧して記録が消える。
+- **R2-02 解決。** 記録なしでも共通ロックが `up` を拒否し、rollback の確認中も排他が続く。Compose 起動、reset の停止・削除中に三つの DB 操作を割り込ませる独立試験はすべて 15 で拒否した。提供試験は逆向きの reset と別プロセスの競合も検証する。独立試験で実 Docker コマンドは実行していない。
+- **R2-03 元の再現は修正済み、R3-01 が残る。** upgrade/rollback の復旧改名で記録削除前に volumes 親を同期し、EIO も無視しなくなった。提供された順序・同期失敗の試験も合格した。文書にある EINVAL/ENOTSUP の例外扱いは意図的である。プロセスクラッシュ試験は電源断の永続化を証明しない。
+- **R2-04 解決。** 非 ASCII の PG_VERSION は `up` から終了 15 になり Compose は起動しない。提供試験は補助コンテナ出力のデコード失敗も確認する。
+
+元の 162 箇所(upgrade 128、rollback 34)の試験は同じ件数で全件合格した。観測した Docker 呼出し、永続 JSON 書込み、rename/replace、fsync、mkdir/unlink、snapshot、起動待ち、復元ファイル作成、記録削除の前後で捕捉不能な例外を注入する。復旧後に現行版が読めること、記録と追跡サーバーが残らないこと、現行 18 なら upgrade 記録があることを検査した。二重障害と競合は別途試験した。ADR-12 は容量不足の切替前後の結果を区別し、ADR-13 は見出しを所有者の選択記録に合わせた。対象差分に他の必須修正は見つからなかった。
+
+### 検証と限界
+
+クリーン clone は `/tmp/pr42-rev-r3/repo`、システム Python は `/usr/bin/python3` 3.9.6。次はレビュー測定であり manifest の正本ではない。
+
+```toon
+checks[6]{check,exit,result}:
+ "installer 単体試験",0,"583 件; 10 件スキップ"
+ "tools 単体試験",0,"93 件"
+ "docs_check",0,"146 ファイル; 例外 8; 警告 36"
+ "manifest_check",0,"指摘 0; 追跡 667 ファイル走査; 63 件スキップ"
+ "独立 r2 試験",0,"162 クラッシュ箇所と二重障害・排他・同期・文字コード試験に合格"
+ "追加試験",0,"競合試験合格; R3-01 の例外を再現; 後続復旧は成功"
+```
+
+再現コマンドは英語節と同じで clone のルートから実行する。各スクリプトの終了値は 0。追加試験の 0 は不具合の再現成功であり、レビュー合格ではない。ログとスクリプトは clone 外の `/tmp/pr42-rev-r3/` に置いた。
+
+未追跡のレビューは `docs_check.check_file` で直接検査し、エラー 0 を確認した。clone 内への追加はこの記録のみで、commit・push・merge・approve・close は行わない。日本語の PR コメント一件で判定を報告する。既存 runtime/.env と grdm-dev の Docker 状態にはアクセスも変更もしておらず、試験の資格情報は一時ディレクトリの疑似値である。実 PostgreSQL 移行、Linux 所有権、amd64、物理電源断、ブラウザ受入は独立検証していない。この行列はプロセス中断の模擬であり、ファイルシステム永続化や実 Docker の kill ではない。
diff --git a/docs/reviews/2026-11-10_RDM-developer-guide_PR42_PostgreSQL18_r4.md b/docs/reviews/2026-11-10_RDM-developer-guide_PR42_PostgreSQL18_r4.md
new file mode 100644
index 0000000..a7c0848
--- /dev/null
+++ b/docs/reviews/2026-11-10_RDM-developer-guide_PR42_PostgreSQL18_r4.md
@@ -0,0 +1,94 @@
+# PR42 PostgreSQL 18 review r4 / PostgreSQL 18 レビュー r4
+
+> **Summary:** Pass. R3-01 is closed: recovery filesystem errors return 15 through every relevant command entry point, and pending recovery remains recoverable.
+> **概要:** 合格。R3-01 は解決。復旧中のファイルシステム障害は関連する全コマンド入口から終了 15 となり、保留操作は再復旧できる。
+
+**Jump:** [English](#english) · [日本語](#japanese)
+
+- Version / 版: 0.1 — 2026-11-10. **Learned:** the initial recovery needs the same error boundary as the migration itself; checking actual command entry points confirms the exit contract. / 初回復旧にも移行本体と同じ例外境界が必要であり、実コマンド入口の試験で終了値の契約を確認できる。
+- Applies to: RCOSDP/RDM-developer-guide PR #42, diff `7e61df3..d49945c`, head `d49945c78bd5d87efc5c7985142c0ecb5520a025`; REV(担当レビュワ, Codex Astra).
+- Related: [Review index](README.md) · [r3](2026-11-10_RDM-developer-guide_PR42_PostgreSQL18_r3.md) · [SPEC-06](../spec/06_documentation_conventions.md) · [SPEC-02](../spec/02_installer_cli.md) · [ADR-12](../decisions/ADR-12_PostgreSQL_18_and_Data_Directory_Upgrade.md).
+- Used by: [Status board](../status/README.md).
+
+---
+
+
+## English
+
+[→ 日本語版へ](#japanese)
+
+### Verdict
+
+**Pass; R3-01 closed.** No blocking regression found in the scoped diff. `run_upgrade` now catches OSError around the entire locked operation, including its initial recovery. The common filesystem-error message distinguishes a remaining journal from no pending operation; recover and rollback use it too. The fallback recovery message avoids reporting `None` after an operation has already retired its journal.
+
+The r3 reproduction now returns 15 from actual `cli.main`, retains the pending journal and current PostgreSQL 18 directory, then completes recovery after fault removal. Persistent fsync EIO was also injected through all five entry points: `db upgrade`, `db rollback`, `db recover`, `up --no-open`, and `reset`. Every entry returns 15 and retains the pending journal; later recovery succeeds. Unit tests separately inject the exact volumes-parent synchronization fault at those five entries.
+
+Every observed fsync on successful upgrade, rollback and pending-upgrade recovery was individually failed once with EIO: 10, 6 and 4 sites respectively. All return 15 through the CLI, mention the injected error, and permit later recovery with a readable current major, no journal and no tracked temporary server. Journal retention applies to unfinished recovery: a fault before initial journal publication or after journal unlink need not leave a journal. The patch accurately reports that distinction; these tests do not claim unconditional retention after completed retirement.
+
+The unchanged 162-point crash matrix passes (128 upgrade, 34 rollback). The r3 double-rename failures, lock-only refusal, non-ASCII version, recovery parent synchronization, rollback confirmation race and lifecycle interleavings also pass. The r3 extra probe was updated only to require exit 15 instead of expecting the former uncaught exception, and to import the new clone.
+
+### Validation and limits
+
+Clean clone: `/tmp/pr42-rev-r4/repo`; system Python: `/usr/bin/python3` 3.9.6. Logs and probe scripts: `/tmp/pr42-rev-r4/`. Results are review measurements, not canonical manifest data.
+
+```toon
+checks[8]{check,exit,result}:
+ "installer units",0,"587 tests; 10 skipped"
+ "tools units",0,"93 tests"
+ "docs_check",0,"147 files; 8 exempt; 36 warnings"
+ "manifest_check",0,"0 findings; 668 scanned; 63 skipped"
+ "r3 probes",0,"162 crash boundaries and regression probes pass"
+ "r3 extra",0,"R3-01 closed; lifecycle interleavings pass"
+ "EIO probes",0,"20 fsync sites; 5 persistent-error entry points pass"
+ "diff whitespace",0,"no errors"
+```
+
+Commands run from the clone root:
+
+```sh
+PYTHONPATH=installer /usr/bin/python3 -m unittest discover -s installer/tests
+/usr/bin/python3 -m unittest discover -s tools/tests
+/usr/bin/python3 tools/docs_check.py
+/usr/bin/python3 tools/manifest_check.py
+/usr/bin/python3 /tmp/pr42-rev-r4/probe.py
+/usr/bin/python3 /tmp/pr42-rev-r4/extra.py
+/usr/bin/python3 /tmp/pr42-rev-r4/eio.py
+git diff --check 7e61df3..d49945c
+```
+
+This untracked report was checked directly using `docs_check.check_file`: 0 errors and 0 warnings. The only clone addition is this report; no commit, push, approval, merge or closure is performed. One Japanese PR comment reports the verdict. Existing runtime/.env and the grdm-dev Docker stack were not accessed or changed. Synthetic fixtures and fake Docker runners do not validate real PostgreSQL migration, Linux ownership, amd64, browser acceptance or physical power-loss durability.
+
+---
+
+
+## 日本語
+
+[→ To English](#english)
+
+### 判定
+
+**合格(pass)、R3-01 解決。** 対象差分に必須修正となる回帰は見つからなかった。`run_upgrade` は初回復旧を含むロック内の操作全体で OSError を捕捉する。共通の障害メッセージは journal が残る場合と保留操作がない場合を区別し、recover と rollback も利用する。操作記録を削除済みの復旧結果が `None` と表示されることも防ぐ。
+
+r3 の再現は実際の `cli.main` から終了 15 となり、保留 journal と現行 PostgreSQL 18 を保持し、障害除去後の復旧に成功した。継続する fsync EIO を `db upgrade`、`db rollback`、`db recover`、`up --no-open`、`reset` の五つの入口へ注入し、すべて終了 15、保留 journal の保持、後続復旧の成功を確認した。提供単体試験は、この五入口で volumes 親ディレクトリ同期の元の障害も個別に検証する。
+
+正常経路で観測した fsync の全箇所へ一度ずつ EIO を注入した。upgrade 10、rollback 6、保留 upgrade の recover 4 箇所で、すべて CLI から 15 を返し、注入した障害を表示した。その後の復旧では現行版が読め、journal と追跡する一時サーバーが残らない。journal 保持は未完了の復旧に対するもので、最初の記録公開前や記録 unlink 後の障害では記録が残るとは限らない。修正後の表示はこの違いを正しく伝える。完了済み操作の記録が常に残ると主張する試験ではない。
+
+既存の 162 箇所の中断行列(upgrade 128、rollback 34)は同じ件数で合格した。r3 の二重 rename 失敗、記録なしロック拒否、非 ASCII 版番号、復旧時の親同期、rollback 確認中の競合、lifecycle との競合も合格した。r3 の追加プローブは新 clone の import と、旧未捕捉例外の期待を終了 15 の期待へ置き換えた箇所のみ更新した。
+
+### 検証と限界
+
+クリーン clone は `/tmp/pr42-rev-r4/repo`、システム Python は `/usr/bin/python3` 3.9.6。ログとプローブは `/tmp/pr42-rev-r4/`。結果は manifest 正本ではなくレビュー測定である。
+
+```toon
+checks[8]{check,exit,result}:
+ "installer units",0,"587 tests; 10 skipped"
+ "tools units",0,"93 tests"
+ "docs_check",0,"147 files; 8 exempt; 36 warnings"
+ "manifest_check",0,"0 findings; 668 scanned; 63 skipped"
+ "r3 probes",0,"162 crash boundaries and regression probes pass"
+ "r3 extra",0,"R3-01 closed; lifecycle interleavings pass"
+ "EIO probes",0,"20 fsync sites; 5 persistent-error entry points pass"
+ "diff whitespace",0,"no errors"
+```
+
+実行コマンドは英語節に記載した。未追跡の記録は `docs_check.check_file` で直接検査し、エラー 0・警告 0 を確認した。clone への追加は本記録のみで commit・push・approve・merge・close は行わず、日本語 PR コメント一件で判定を報告する。既存 runtime/.env と grdm-dev Docker 状態にはアクセスも変更もしていない。疑似 fixture と Docker runner の試験では、実 PostgreSQL 移行、Linux 所有権、amd64、ブラウザ受入、物理電源断の永続化は検証していない。
diff --git a/docs/spec/02_installer_cli.md b/docs/spec/02_installer_cli.md
index f3d7fc1..6e7d7c0 100644
--- a/docs/spec/02_installer_cli.md
+++ b/docs/spec/02_installer_cli.md
@@ -5,7 +5,10 @@
**Jump:** [English](#english) · [日本語](#japanese)
-- Version / 版: 0.19 — 2026-11-10. **Learned:** a fixed client is only as good as the address and trust anchor it is given: osf.io's mongo URL was never rendered and fell back to PostgreSQL's host and port, so `osf.env` now names the TLS URL and the CA file. / 直したクライアントも、渡される接続先と信頼の起点しだいである。osf.io の mongo の URL は一度も描画されず、PostgreSQL のホストとポートに落ちていたので、`osf.env` に TLS の URL と CA ファイルを書くことにした。
+- Version / 版: 0.22 — 2026-11-10. **Learned:** a lock that only the `db` commands take does not stop `up` from starting the server between their checks and their renames, so `up` and `reset` take it too (PR #42 review r2). / `db` のコマンドだけが取るロックでは、その確認と改名の間に `up` がサーバーを起動するのを止められないので、`up` と `reset` も同じロックを取る(PR #42 レビュー r2)。
+ 0.21 — 2026-11-10. **Learned:** a command that renames data directories needs a journal, a lock and a recovery path that every later command runs, not only checks before it starts (PR #42 review r1). / データディレクトリを改名するコマンドには、始める前の確認だけでなく、操作記録・ロック・後続のすべてのコマンドが通る復旧の道が要る(PR #42 レビュー r1)。
+ 0.20 — 2026-11-10. **Learned:** a new database major needs a command, not a note: a server refuses a data directory of another major, so `up` now says so with exit 15 and `db upgrade` moves the data by a verified dump and restore that never writes the old directory (ADR-12). / データベースの新しい版には注記でなくコマンドが要る。サーバーは別の版のデータディレクトリを拒むので、`up` はそれを終了 15 で告げ、`db upgrade` は旧ディレクトリに一度も書かない、照合つきのダンプと復元でデータを移す(ADR-12)。
+ 0.19 — 2026-11-10. **Learned:** a fixed client is only as good as the address and trust anchor it is given: osf.io's mongo URL was never rendered and fell back to PostgreSQL's host and port, so `osf.env` now names the TLS URL and the CA file. / 直したクライアントも、渡される接続先と信頼の起点しだいである。osf.io の mongo の URL は一度も描画されず、PostgreSQL のホストとポートに落ちていたので、`osf.env` に TLS の URL と CA ファイルを書くことにした。
0.18 — 2026-11-10. **Learned:** a build record says what was built, not what should run: after the pin of osf.io moved, `compose render` ran before `build` and the stack silently ran the old image, so render now trusts the record only when it equals the manifest pin and otherwise renders the pin and warns with the fix. / ビルドの記録は何を組んだかを示すだけで、何を動かすべきかは示さない。osf.io の固定が動いた後、`build` より前に `compose render` を実行したため、スタックは古いイメージを黙って動かした。そこで render は記録が manifest の固定と一致するときだけそれを信じ、違うときは固定を書いて対処を添えて警告することにした。
0.17 — 2026-11-10. **Learned:** a flag that changes what a server will serve belongs in the operating contract, not only in a template and a PR body, so the admin app's `--insecure` is now stated as a development-only, loopback-only exception (PR #40 review r1 R1-02). / サーバーが配信するものを変えるフラグは、テンプレートや PR 本文だけでなく運用の契約に書く。そのため管理画面の `--insecure` を、開発専用でループバック限定の例外として明記した(PR #40 レビュー r1 R1-02)。
0.16 — 2026-11-10. **Learned:** the browser opens the wiki and file-editor sockets itself, so y_websocket and sharejs need loopback ports like any other page the browser loads. / wiki とファイル編集のソケットはブラウザが自分で開くため、y_websocket と sharejs にも、ブラウザが読む他のページと同じくループバックのポートが要る。
@@ -97,7 +100,8 @@ flowchart TD
| `secrets init` / `secrets rotate` | `init` writes random secrets (database passwords, Django secret key, CAS keys) into `runtime/.env` with file mode `0600`. Never writes a default or example password. Refuses to overwrite an existing file unless `--force`. `rotate` regenerates every secret, changes the live PostgreSQL and RabbitMQ passwords, recreates the containers so they receive the new environment, and verifies that the old credentials are rejected; any failure restores the previous `runtime/.env` and exits 40. `init --force` and `rotate` drop retired credential keys. After an upgrade, run `compose render` before `rotate` or any recreate, because a previously rendered Compose file still refers to retired keys. Four keys have lengths the CAS image needs: `TGC_ENCRYPTION_KEY` is 43 characters, `TGC_SIGNING_KEY` is 86 characters, `OSF_API_INSTITUTIONS_AUTH_JWESECRET` is exactly 32 bytes and `OSF_API_INSTITUTIONS_AUTH_JWTSECRET` is at least 32 bytes; the last two are also osf.io's `JWE_SECRET` and `JWT_SECRET` ([ADR-07 §7](../decisions/ADR-07_Native_Builds_of_EOL_Middleware.md#english)). All four use only the base64url characters `A`–`Z`, `a`–`z`, `0`–`9`, `-` and `_` (no whitespace, quotes or padding), must not be a published default, and `TGC_ENCRYPTION_KEY` must decode canonically to 32 bytes. Used after a red-team capture ([SPEC-03 §6](03_security_gate.md#english)). | no / yes |
| `compose render` | Exits 2, naming the keys only, when `runtime/.env` lacks a generated key or holds a CAS key that CAS cannot use: wrong length, characters outside base64url, or a published default (fix either with `secrets rotate`). Writes `runtime/docker-compose.yml` from manifest + profile: the internal network `grdm` plus the publishing-only `grdm-edge` (`--egress ` attaches more, recorded in `runtime/.env`; [SPEC-03 §3.2](03_security_gate.md#english)), per-component `platform:` from the manifest's `build_platform`, ports published only on `127.0.0.1` ([§4](#2-en-ports)), non-root `user:` for every service, `read_only`, `cap_drop`, and resource limits (`mem_limit`, `cpus`, `pids_limit`) scaled from the smaller of the host memory `doctor` detected and the Docker engine's VM memory (`docker info`; on Docker Desktop the VM is a fraction of the host, real install run 10), where a template's `mem_min_mb` floor takes its share first and the others divide the rest, and from the CPU count the engine reports (`docker info`; the host's only when that is unknown), with per-service exceptions from `manifest/hardening.json` ([SPEC-03 §3.3](03_security_gate.md#english)), `platform:` chosen from `doctor`, named volumes mapped under `runtime/volumes/`, every `tmpfs` entry mounted with `mode=1777` (Compose's short form gives root-only `0755`, real install run 11), and the placeholders `{{HOST_PORT:}}`, `{{CONTAINER_PORT:}}` and `{{AUTH_URL}}` (fakecas's published URL, or web's when `--auth shibboleth` replaces fakecas) resolved in template values. The `image:` of a built component carries the tag of the last build (`runtime/reports/build_latest.json`) only when it equals the tag the manifest pins; a recorded build that differs from the pin is stale (the pin moved after the last `build`), so the pinned tag is rendered instead, the report lists the component under `stale_build_tags` (`component`, `pinned`, `recorded`, `image`), and stderr prints a warning block whose fix is `grdm-dev build`, then `grdm-dev up`. Render still exits 0 because `install` renders before it builds; an image that was never built makes `up` fail instead of running the old one. A component the manifest does not pin keeps the recorded tag. `osf.env` names osf.io's TLS connection to mongo, `SHAREJS_DB_URL=mongodb://mongo:27017/sharejs?ssl=true` with `SHAREJS_DB_CA_FILE=/mongo-tls/mongodb.crt`, and web, api, admin and worker mount `mongo_tls` read-only, as sharejs does ([ADR-07](../decisions/ADR-07_Native_Builds_of_EOL_Middleware.md)). Follows [ADR-02](../decisions/ADR-02_Network_Model.md). | no |
| `build` | Builds images with `docker buildx`. The platform is, in order of precedence, `--platform`, the component's `build_platform` in the manifest (for example `linux/amd64` for components whose EOL base cannot be built for arm64, run under emulation), otherwise the host platform; the image is loaded locally. A component with `prebuilt_image` ([SPEC-01](01_manifest.md)) is not built: the digest-pinned image is pulled once per requested platform and reported as `prebuilt` (`pull-failed` → exit 30). With `--push` a prebuilt image is never re-published to `ghcr.io/rcosdp` (it is a third party's artefact); the row is reported as `prebuilt-not-pushed` with `pushed: false`. `--push` pushes the multi-arch manifest to `ghcr.io/rcosdp/` (private). Exit 30 on failure. | yes |
-| `up` / `down` / `reset` | `up` starts the profile and waits for health; `down` stops containers and keeps volumes; `reset` stops and deletes `runtime/volumes/` after a confirmation (skipped by `--yes`), emptying it through a root container with only `DAC_OVERRIDE` and `FOWNER`, because on Linux the files belong to the containers' uids and the host user cannot remove them. Before anything is stopped or deleted the paths are validated on the host: a symlinked `runtime/` or volumes root, a root that resolves outside the checkout, a symlinked volume, or any mount at `runtime/`, at the volumes root or at any depth inside it (read from the host mount table, so same-filesystem bind mounts count; an unreadable table also refuses) stops with exit 14. `reset` repeats this after `down` and after the wipe, and stops if the volumes root has become a different directory. `compose render` and `up` apply the same validation with exit 13, because the owner step also runs as root against these paths. These are snapshots: Docker resolves the bind later and no check can close that window, so the checkout is assumed not to be changed concurrently by another user or process. | yes |
+| `up` / `down` / `reset` | `up` first finishes or rolls back an interrupted `db` operation, then stops with exit 15 when `runtime/volumes/postgres_data` was written by another PostgreSQL major than the rendered image or its major cannot be read (ADR-12), otherwise starts the profile and waits for health; `down` stops containers and keeps volumes; `reset` stops and deletes `runtime/volumes/` after a confirmation (skipped by `--yes`), emptying it through a root container with only `DAC_OVERRIDE` and `FOWNER`, because on Linux the files belong to the containers' uids and the host user cannot remove them. Before anything is stopped or deleted the paths are validated on the host: a symlinked `runtime/` or volumes root, a root that resolves outside the checkout, a symlinked volume, or any mount at `runtime/`, at the volumes root or at any depth inside it (read from the host mount table, so same-filesystem bind mounts count; an unreadable table also refuses) stops with exit 14. `reset` repeats this after `down` and after the wipe, and stops if the volumes root has become a different directory. `compose render` and `up` apply the same validation with exit 13, because the owner step also runs as root against these paths. These are snapshots: Docker resolves the bind later and no check can close that window, so the checkout is assumed not to be changed concurrently by another user or process. | yes |
+| `db status` / `db upgrade` / `db rollback` / `db recover` | `status` prints the PostgreSQL major of `runtime/volumes/postgres_data` and of the rendered `postgres` image and changes nothing. `upgrade` (all services stopped, after a confirmation that `--yes` skips) moves the data to the rendered major by dump and restore: the old major runs on the old directory with no network, the new major's `pg_dumpall` writes `runtime/backups/postgres--to--/`, a new directory is initialised and restored with `ON_ERROR_STOP`, every table's row count and sequence value is compared, and only then is the old directory renamed `postgres_data.pg-` beside the new one. `rollback` puts that directory back. Both hold an exclusive lock, which `up` and `reset` also take for their checks and their start or deletion, and write a durable journal (`runtime/db-operation.json`) before they create or rename anything; `recover`, and first every later `db` command, `reset` and `up`, finish or roll back an operation that a crash or an interrupt left behind, and a free-disk estimate is checked before the old server starts. Exit 15 on failure, with the old or the new cluster usable, or with the journal kept when a compensating rename also failed. See [ADR-12](../decisions/ADR-12_PostgreSQL_18_and_Data_Directory_Upgrade.md#english). | yes |
| `health` | Checks every service's health endpoint: an HTTP GET on the published loopback port (`/` for most services, `/login/` for `web`, because osf.io's `/` proxies to ember and answers 500 while ember is still building — real install run 14); a one-shot counts as healthy once it has exited 0. With `--network`, also lists listening sockets on the host and fails (exit 40) if any container port is bound to a non-loopback address, or if a wildcard listener on a project port has no visible owner (fail closed). A *named* host program that is not Docker on a project port (macOS AirPlay Receiver on `*:5000`, real install run 16) is reported as a `foreign-listener` warning, not a failure: it is not our leak, and `GRDM_PORT_` in `runtime/.env` moves our port. | read-only |
| `scan` | Runs the whole security gate of [SPEC-03](03_security_gate.md): baseline controls, Trivy SBOM (software bill of materials: the list of all parts inside an image) and CVE list, gitleaks, Semgrep, and OWASP ZAP; writes `runtime/reports/scan_.json` plus TOON, and diffs findings against `manifest/allowlist.json` and `manifest/allowlist.baseline.json` for the milestone in `manifest/milestone.json`. Gate per [ADR-03](../decisions/ADR-03_Staged_Security_Gate.md): M1–M3 report only, M4 exit 50 on HIGH, M5 exit 50 on CRITICAL or HIGH. Flags `--baseline`, `--secrets`, `--dynamic` run one part only. Exit 51 if a scanner cannot run, 52 if `--dynamic` was asked for while services are unhealthy. | read-only |
| `redteam preflight` | Verifies, before an AI red-team run, that the attacker container of the red-team Compose project is isolated (proved with a reachable canary, see [SPEC-03 §6](03_security_gate.md#english)), that only the containers named in `manifest/redteam-targets.json` are attached, and that the approval record is present and current. Exit 53 if either check fails. See [SPEC-03 §6](03_security_gate.md#english). | yes (starts and removes a canary container and network for the probe) |
@@ -155,7 +159,7 @@ The `admin` service runs Django's development server as `manage.py runserver 0.0
### 6. Exit codes
```toon
-exit_codes[16]{code,meaning}:
+exit_codes[17]{code,meaning}:
0,success
2,"usage error: unknown flag or bad value"
10,"host unsupported: OS, CPU, memory, or Python too old"
@@ -163,6 +167,7 @@ exit_codes[16]{code,meaning}:
12,a required port on 127.0.0.1 is in use and no free alternative was found
13,"install prerequisite missing: git, repository access, amd64 emulation, or a checkout path Docker cannot use"
14,reset could not delete runtime/volumes/
+ 15,"PostgreSQL data directory needs `grdm-dev db upgrade`, or the upgrade or rollback failed (the old directory is kept)"
20,manifest invalid or TOON excerpt disagrees with JSON
30,image build failed
40,"health check failed, or a port listens on a non-loopback address"
@@ -209,6 +214,7 @@ layout:
manifest.py: load and validate manifest/upstreams.json, TOON comparison
secrets.py: random secret generation, runtime/.env with mode 0600
compose.py: render runtime/docker-compose.yml from manifest and profile
+ database.py: PostgreSQL data directory status, upgrade by dump and restore, rollback (ADR-12)
build.py: buildx wrapper, platform list, optional push
health.py: service health and loopback-only check
scan.py: baseline controls, Trivy SBOM and CVE, gitleaks, Semgrep, ZAP, allow-list diff, staged gate (SPEC-03)
@@ -228,7 +234,7 @@ layout:
milestone.json: current milestone, owner-maintained (SPEC-03 §5.2)
hardening.json: per-service exceptions to read-only root and cap_drop, with reasons (SPEC-03 §3.3)
redteam-targets.json: container names the AI red team may touch (SPEC-03 §6)
- runtime/: generated, git-ignored (.env, docker-compose.yml, volumes/, reports/)
+ runtime/: generated, git-ignored (.env, docker-compose.yml, volumes/, reports/, backups/, db-operation.json and .lock)
```
### 10. Tests
@@ -304,7 +310,8 @@ flowchart TD
| `secrets init` / `secrets rotate` | `init` は乱数の秘密(DB パスワード、Django の秘密鍵、CAS の鍵)を `runtime/.env` にモード `0600` で書く。既定値や例のパスワードは決して書かない。既存ファイルは `--force` がなければ上書きしない。`rotate` はすべての秘密を作り直し、稼働中の PostgreSQL と RabbitMQ のパスワードを変更し、新しい環境を受け取るようコンテナを再作成し、旧い資格情報が拒否されることを確かめる。どこかで失敗したら以前の `runtime/.env` に戻して 40 で終わる。`init --force` と `rotate` は廃止した資格情報のキーを消す。更新後は、`rotate` や再作成の前に `compose render` を実行する。以前に生成した Compose ファイルは廃止したキーを参照し続けるからである。四つのキーには CAS のイメージが要する長さがある。`TGC_ENCRYPTION_KEY` は 43 文字、`TGC_SIGNING_KEY` は 86 文字、`OSF_API_INSTITUTIONS_AUTH_JWESECRET` はちょうど 32 バイト、`OSF_API_INSTITUTIONS_AUTH_JWTSECRET` は 32 バイト以上で、後の二つは osf.io の `JWE_SECRET` と `JWT_SECRET` でもある([ADR-07 §7](../decisions/ADR-07_Native_Builds_of_EOL_Middleware.md#japanese))。四つとも base64url の文字(`A`–`Z`、`a`–`z`、`0`–`9`、`-`、`_`。空白・引用符・パディングは不可)だけを使い、公開された既定値であってはならず、`TGC_ENCRYPTION_KEY` は 32 バイトへ正準に復号できなければならない。レッドチームに秘密を取られた後に使う([SPEC-03 §6](03_security_gate.md#japanese))。 | 触れない / 触れる |
| `compose render` | `runtime/.env` に生成すべきキーが無いとき、または CAS のキーが使えないとき(長さの誤り、base64url 以外の文字、公開された既定値)は、キーの名前だけを示して終了 2 で止まる(どちらも `secrets rotate` で直す)。manifest とプロファイルから `runtime/docker-compose.yml` を書く。内部ネットワーク `grdm` と公開専用の `grdm-edge`(`--egress ` で追加接続し `runtime/.env` に記録。[SPEC-03 §3.2](03_security_gate.md#japanese))、部品ごとの `platform:` は manifest の `build_platform` から、ポートは `127.0.0.1` にだけ公開([§4](#2-ja-ports))、全サービスに非 root の `user:`、`read_only`・`cap_drop`・資源制限(`mem_limit`・`cpus`・`pids_limit`。`doctor` が検出したホストのメモリと Docker エンジンの VM メモリ(`docker info`。Docker Desktop では VM はホストの一部。実機第 10 回)の小さい方に応じて決め、テンプレートの下限 `mem_min_mb` が先に取り分を確保し、残りを他が分け合う。CPU はエンジンが報告する数(`docker info`。不明なときだけホストの数)から決める)、サービスごとの例外は `manifest/hardening.json` から([SPEC-03 §3.3](03_security_gate.md#japanese))、`platform:` は `doctor` の結果から選ぶ、名前付きボリュームは `runtime/volumes/` の下、`tmpfs` の各項目は `mode=1777` で mount(Compose の短い記法は root 専用の `0755` になる。実機第 11 回)、テンプレート値のプレースホルダ `{{HOST_PORT:}}`・`{{CONTAINER_PORT:}}`・`{{AUTH_URL}}`(fakecas の公開 URL。`--auth shibboleth` で fakecas が外れるときは web の URL)を解決する。組み立てる部品の `image:` に最後のビルド(`runtime/reports/build_latest.json`)のタグを使うのは、manifest が固定するタグと同じときだけである。記録されたビルドが固定と違うときは古い(最後の `build` の後で固定が動いた)とみなし、代わりに固定のタグを書き、報告の `stale_build_tags`(`component`・`pinned`・`recorded`・`image`)に載せ、標準エラーに警告の囲みを出して対処(`grdm-dev build` を実行してから `grdm-dev up`)を示す。`install` は組む前に render するので、終了コードは 0 のままである。一度も組んでいないイメージは古いものを動かす代わりに `up` を失敗させる。manifest が固定しない部品は記録されたタグを使う。`osf.env` には osf.io から mongo への TLS 接続 `SHAREJS_DB_URL=mongodb://mongo:27017/sharejs?ssl=true` と `SHAREJS_DB_CA_FILE=/mongo-tls/mongodb.crt` を書き、web・api・admin・worker は sharejs と同じく `mongo_tls` を読み取り専用でマウントする([ADR-07](../decisions/ADR-07_Native_Builds_of_EOL_Middleware.md))。[ADR-02](../decisions/ADR-02_Network_Model.md) に従う。 | 触れない |
| `build` | `docker buildx` でイメージを作る。プラットフォームは優先順に `--platform`、manifest の部品の `build_platform`(例: EOL ベースで arm64 に組めない部品は `linux/amd64` にしてエミュレーションで動かす)、それもなければホストと同じ。イメージはローカルに読み込む。`prebuilt_image`([SPEC-01](01_manifest.md))を持つ部品は組まず、ダイジェスト固定のイメージを要求された platform ごとに取得して `prebuilt` と報告する(`pull-failed` は終了 30)。`--push` でも prebuilt のイメージは `ghcr.io/rcosdp` に再公開しない(第三者の成果物であるため)。行は `prebuilt-not-pushed`、`pushed: false` と報告する。`--push` で multi-arch の manifest を `ghcr.io/rcosdp/`(非公開)へ送る。失敗は終了 30。 | 触れる |
-| `up` / `down` / `reset` | `up` はプロファイルを起動し健康になるまで待つ。`down` はコンテナを止めボリュームを残す。`reset` は止めたうえで確認の後に `runtime/volumes/` を消す(`--yes` で確認を省く)。Linux ではファイルがコンテナの uid の所有でホストの利用者には消せないので、`DAC_OVERRIDE` と `FOWNER` だけを持つ root のコンテナで中身を空にする。何かを止めたり消したりする前にホスト側でパスを検証し、シンボリックリンクの `runtime/` やボリュームのルート、リポジトリの外へ解決されるルート、シンボリックリンクのボリューム、`runtime/` やボリュームのルートやその下の任意の深さの mount(ホストの mount 表から読むので同じファイルシステムの bind mount も含む。表が読めないときも拒む)は終了 14 で止まる。`reset` は `down` の後と消去の後にも繰り返し、ボリュームのルートが別のディレクトリに変わっていれば止まる。`compose render` と `up` も、所有者設定の段が同じパスに root で働くので、同じ検証を終了 13 で行う。これらは時点の確認であり、Docker は後で bind を解決するのでこの隙間はどの検査でも塞げない。リポジトリを他の利用者やプロセスが同時に変更しないことを前提とする。 | 触れる |
+| `up` / `down` / `reset` | `up` は中断された `db` の操作をまず完了か巻き戻しし、`runtime/volumes/postgres_data` が生成済みのイメージと別の PostgreSQL の版で書かれているか版を読めなければ終了 15 で止まり(ADR-12)、そうでなければプロファイルを起動し健康になるまで待つ。`down` はコンテナを止めボリュームを残す。`reset` は止めたうえで確認の後に `runtime/volumes/` を消す(`--yes` で確認を省く)。Linux ではファイルがコンテナの uid の所有でホストの利用者には消せないので、`DAC_OVERRIDE` と `FOWNER` だけを持つ root のコンテナで中身を空にする。何かを止めたり消したりする前にホスト側でパスを検証し、シンボリックリンクの `runtime/` やボリュームのルート、リポジトリの外へ解決されるルート、シンボリックリンクのボリューム、`runtime/` やボリュームのルートやその下の任意の深さの mount(ホストの mount 表から読むので同じファイルシステムの bind mount も含む。表が読めないときも拒む)は終了 14 で止まる。`reset` は `down` の後と消去の後にも繰り返し、ボリュームのルートが別のディレクトリに変わっていれば止まる。`compose render` と `up` も、所有者設定の段が同じパスに root で働くので、同じ検証を終了 13 で行う。これらは時点の確認であり、Docker は後で bind を解決するのでこの隙間はどの検査でも塞げない。リポジトリを他の利用者やプロセスが同時に変更しないことを前提とする。 | 触れる |
+| `db status` / `db upgrade` / `db rollback` / `db recover` | `status` は `runtime/volumes/postgres_data` と生成済みの `postgres` イメージの PostgreSQL の版を表示し、何も変えない。`upgrade`(全サービス停止中、`--yes` で省ける確認の後)はダンプと復元でデータを生成済みの版へ移す。旧版をネットワークなしで旧ディレクトリの上に動かし、新版の `pg_dumpall` が `runtime/backups/postgres-<旧>-to-<新>-<時刻>/` に書き、新しいディレクトリを初期化して `ON_ERROR_STOP` で復元し、全表の行数とシーケンスの値を突き合わせ、その後で初めて旧ディレクトリを新しいものの隣の `postgres_data.pg<旧>-<時刻>` に改名する。`rollback` はそのディレクトリを戻す。どちらも何かを作るか改名する前に排他ロック(`up` と `reset` も確認と起動・削除の間に取る)を取り、永続的な操作記録(`runtime/db-operation.json`)を書く。`recover`、そして後続のすべての `db` コマンド・`reset`・`up` は、クラッシュや中断が残した操作を最初に完了か巻き戻しする。旧サーバーの起動前に空き容量の見積りを確かめる。失敗すれば旧か新のクラスタが使える状態で終了 15。補償の改名まで失敗したときは操作記録を残して終了 15。[ADR-12](../decisions/ADR-12_PostgreSQL_18_and_Data_Directory_Upgrade.md#japanese) を参照。 | 触れる |
| `health` | 各サービスの健康確認用の URL を調べる。公開したループバックのポートへの HTTP GET(多くは `/`。`web` は `/login/`。osf.io の `/` は ember へ転送するので ember がビルド中は 500 を返す。実機第 14 回)で、一回限りのサービスは 0 で終了していれば健康と数える。`--network` を付けると、ホストで待ち受けているソケットも一覧し、コンテナのポートがループバック以外の番地に結び付いているか、プロジェクトのポート上のワイルドカードの待ち受けの所有者が見えなければ終了 40(安全側に失敗)。Docker でない*名前の分かる*ホストのプログラムがプロジェクトのポートにいる場合(macOS の AirPlay レシーバーの `*:5000`。実機第 16 回)は `foreign-listener` の警告として報告し、失敗にはしない。我々の漏れではなく、`runtime/.env` の `GRDM_PORT_` で我々のポートを移せる。 | 読むだけ |
| `scan` | [SPEC-03](03_security_gate.md) のセキュリティ門を全部動かす。基本対策、Trivy の SBOM(ソフトウェア部品表。イメージに入っている部品の一覧)と CVE の一覧、gitleaks、Semgrep、OWASP ZAP。`runtime/reports/scan_<日付>.json` と TOON を書き、`manifest/milestone.json` の節目に合わせて `manifest/allowlist.json` と `manifest/allowlist.baseline.json` と差分を取る。門は [ADR-03](../decisions/ADR-03_Staged_Security_Gate.md) どおり、M1〜M3 は報告のみ、M4 は HIGH で終了 50、M5 は CRITICAL か HIGH で終了 50。フラグ `--baseline`・`--secrets`・`--dynamic` は一部だけを動かす。スキャナーが動かなければ終了 51、サービスが健全でないのに `--dynamic` を求めたら終了 52。 | 読むだけ |
| `redteam preflight` | AI レッドチームの実行前に、レッドチーム用 Compose プロジェクトの攻撃側コンテナが隔離されていること(届くはずのカナリアで証明する。[SPEC-03 §6](03_security_gate.md#japanese))、`manifest/redteam-targets.json` に名前のあるコンテナだけが接続していること、承認記録があり新しいことを確かめる。どちらかが外れれば終了 53。詳細は [SPEC-03 §6](03_security_gate.md#japanese)。 | 触れる(検査用のカナリアのコンテナとネットワークを作って消す) |
@@ -362,7 +369,7 @@ ports[9]{service,host_port,profile,url}:
### 6. 終了コード
```toon
-exit_codes[16]{code,meaning}:
+exit_codes[17]{code,meaning}:
0,成功
2,指定の誤り。不明なフラグや値の形の誤り
10,対象外のホスト。OS・CPU・メモリ、または Python が古い
@@ -370,6 +377,7 @@ exit_codes[16]{code,meaning}:
12,必要なポートが 127.0.0.1 で使用中で、空きの代わりも見つからない
13,導入の前提が不足。git、リポジトリの読み取り権限、amd64 エミュレーション、または Docker が使えない配置先
14,reset が runtime/volumes/ を削除できない
+ 15,PostgreSQL のデータディレクトリに `grdm-dev db upgrade` が要る、または更新か巻き戻しに失敗(旧ディレクトリは残る)
20,manifest が不正、または TOON の抄録が JSON と一致しない
30,イメージのビルドに失敗
40,健康確認に失敗、またはループバック以外の番地で待ち受けている
@@ -416,6 +424,7 @@ layout:
manifest.py: manifest/upstreams.json の読み込みと検証、TOON との比較
secrets.py: 乱数の秘密の生成、モード 0600 の runtime/.env
compose.py: manifest とプロファイルから runtime/docker-compose.yml を生成
+ database.py: PostgreSQL のデータディレクトリの状態、ダンプと復元による更新、巻き戻し(ADR-12)
build.py: buildx の呼び出し、platform の一覧、任意の push
health.py: サービスの健康確認とループバック限定の確認
scan.py: 基本対策、Trivy の SBOM と CVE、gitleaks、Semgrep、ZAP、許容リストとの差分、段階的な門(SPEC-03)
@@ -435,7 +444,7 @@ layout:
milestone.json: 現在の節目。所有者が保守(SPEC-03 §5.2)
hardening.json: 読み取り専用ルートと cap_drop のサービスごとの例外と理由(SPEC-03 §3.3)
redteam-targets.json: AI レッドチームが触れてよいコンテナ名(SPEC-03 §6)
- runtime/: 生成物。Git は無視(.env、docker-compose.yml、volumes/、reports/)
+ runtime/: 生成物。Git は無視(.env、docker-compose.yml、volumes/、reports/、backups/、db-operation.json と .lock)
```
### 10. 試験
diff --git a/docs/status/README.md b/docs/status/README.md
index 7d811ec..9177140 100644
--- a/docs/status/README.md
+++ b/docs/status/README.md
@@ -5,7 +5,9 @@
**Jump:** [English](#english) · [日本語](#japanese)
-- Version / 版: 0.11 — 2026-11-10. **Learned:** an open row that a later all-green row depends on must be closed in the same change, or the board contradicts itself (PR #40 review r1 R1-03). / 後の全件合格の行が前提にしている未完了の行は、同じ変更で閉じなければボードが自己矛盾する(PR #40 レビュー r1 R1-03)。
+- Version / 版: 0.13 — 2026-11-10. **Learned:** an owner decision changes a waiting row into a working row the same day, so the M5 rows now name the chosen paths, and a merged-looking row keeps its review state (PR #42 review r1). / 所有者の判断は待ちの行をその日のうちに作業中の行に変えるので、M5 の行は選ばれた道を書き、マージ済みに見える行もレビューの状態を保つ(PR #42 レビュー r1)。
+ 0.12 — 2026-11-10. **Learned:** M5 work started before M2 closed, so M5 gets its own open-items table now (rule 3): one part is done and measured (PostgreSQL 18), the other two wait for owner decisions (ADR-13). / M5 の作業は M2 が閉じる前に始まったので、M5 にも今から専用の残件表を置く(規則 3)。一つは完了して測定済み(PostgreSQL 18)、残る二つは所有者の判断待ち(ADR-13)。
+ 0.11 — 2026-11-10. **Learned:** an open row that a later all-green row depends on must be closed in the same change, or the board contradicts itself (PR #40 review r1 R1-03). / 後の全件合格の行が前提にしている未完了の行は、同じ変更で閉じなければボードが自己矛盾する(PR #40 レビュー r1 R1-03)。
0.10 — 2026-11-10. **Learned:** a scenario that passes only on a stack built from the pinned commits is evidence; a pass on a stack still running older images is not, because Compose kept the tags recorded by the previous build. / 固定した版から組んだスタックで通った試験だけが証拠になる。古いイメージのままのスタックでの合格は証拠にならない。Compose は前回のビルドが記録したタグを使い続けたからである。
0.9 — 2026-11-10. **Learned:** acceptance counts per scenario belong on the board, because a single pass count hides which link is broken. / 受入の件数はシナリオごとにボードへ書く。合格数だけでは、どの連携が壊れているか分からない。
0.8 — 2026-11-06. **Learned:** a warm success and a cold-start success are two different rows: the cold start needed one more architecture decision (ADR-07) before it passed. / 温まった状態の成功とコールドスタートの成功は別の行である。コールドスタートは通るまでにもう一つ設計判断(ADR-07)を要した。
@@ -34,7 +36,7 @@
| M2 | Core 6 install with one command on Mac Studio; Playwright acceptance passes; amd64 builds | Report + allow-list (fewer entries than M1) | **In progress**: one-command install of the core profile succeeded on the Mac Studio in real install run 18 (PR #33, [DEMO-20](../demo/20_core_install_mac_studio.md#english)): build, up, health --network, scan, report all exit 0; Playwright acceptance not passed yet (first run 0 of 12, see §2b) |
| M3 | 5 subsystems start; one success + one rejection case each | Report + allow-list (fewer entries than M2) | Planned |
| M4 | Waterbutler, mfr, osf.io on Python 3.13 | Stop on HIGH or above; first AI red-team run | Planned |
-| M5 | ES2 and mongo removed; ES8 + postgres 15 or later; distributable abroad | Zero CRITICAL/HIGH; zero open AI red-team paths | Planned |
+| M5 | ES2 and mongo removed; ES8 + postgres 15 or later; distributable abroad | Zero CRITICAL/HIGH; zero open AI red-team paths | **Groundwork** (§2c): PostgreSQL 18 with `db upgrade` proposed ([ADR-12](../decisions/ADR-12_PostgreSQL_18_and_Data_Directory_Upgrade.md#english)); ES2 and mongo findings await owner decisions ([ADR-13](../decisions/ADR-13_Retiring_Elasticsearch_2_and_MongoDB.md#english)) |
### 2. Open items for M1
@@ -65,13 +67,24 @@
| Trivy image scan and allow-list count (fewer than M1) | ✅ Measured in run 18 (Trivy alone): 1835 CRITICAL / 8290 HIGH / 9763 MEDIUM / 3886 LOW over 22 targets; run 21 with the native Elasticsearch 2: 1769 / 8030 / 9550 / 3676; allow-list 0 entries (M1: 0). Semgrep, gitleaks and ZAP ran too (ZAP: 1 HIGH, 9 MEDIUM, 13 LOW to triage at M3) |
| Default branch switched to `develop` (owner) | ⏳ After M2 passes |
+
+### 2c. Open items for M5
+
+| Item | Status |
+|---|---|
+| PostgreSQL 9.6 → 18.6, `grdm-dev db status/upgrade/rollback`, `up` refuses a mismatched major (exit 15) | 🔄 PR #42 (review r1 changes addressed) ([ADR-12](../decisions/ADR-12_PostgreSQL_18_and_Data_Directory_Upgrade.md#english)): osf.io DB tests equal on 9.6 and 18 (781 passed, same 13 failed or erred); a 9.6 cluster upgraded in 37 s with 531 objects equal; CAS and fakecas log in; rollback measured; after review r1 a journal and a lock make a crash at any step recoverable (SIGKILL and SIGTERM measured, ADR-12 §4.1). PR #42. Not run on amd64 or on a Linux host |
+| SCRAM for PostgreSQL | ⏳ Owner: CAS needs a JDBC driver ≥ 42.2 in RDM-cas-overlay-dev2027 (42.1.4 fails with `authentication type 10`) |
+| Elasticsearch 2 retirement | 🔄 Owner decision 2026-11-10: port osf.io search to Elasticsearch 8, keeping full-text search; in progress in the osf.io integration repository ([ADR-13 §4](../decisions/ADR-13_Retiring_Elasticsearch_2_and_MongoDB.md#english)) |
+| MongoDB and sharejs retirement | 🔄 Owner decision 2026-11-10: the file-page editor moves from sharejs to y-websocket, then sharejs and mongo are retired by ADR-13 §3; in progress |
+| Elasticsearch 6 → 8 (metrics) | ⏳ Not started |
+
### 3. Rules
This board is the **single source of truth** for milestone progress:
1. Every document that mentions a milestone must link to this board.
2. The "Status" column above is authoritative; status updates require a PR with evidence (demo run or acceptance pass).
-3. "Open items" has one table per milestone that has started (§2 for M1, §2b for M2); later milestones get their own tables when they start.
+3. "Open items" has one table per milestone that has started (§2 for M1, §2b for M2, §2c for M5); later milestones get their own tables when they start.
---
@@ -88,7 +101,7 @@ This board is the **single source of truth** for milestone progress:
| M2 | コア 6 本が Mac Studio で一発導入。Playwright 受入合格。amd64 ビルド | 報告+許容リスト(M1 より件数を減らす) | **進行中**: core プロファイルの一発導入が実機第 18 回で Mac Studio 上で成功(PR #33、[DEMO-20](../demo/20_core_install_mac_studio.md#japanese)): build・up・health --network・scan・report がすべて終了 0。Playwright 受入は未合格(初回 12 中 0、§2b 参照) |
| M3 | サブシステム 5 本が起動。各 1 本の正常系+拒否ケース | 報告+許容リスト(M2 より件数を減らす) | 計画中 |
| M4 | waterbutler・mfr・osf.io を Python 3.13 へ | HIGH 以上で停止。AI レッドチーム初回 | 計画中 |
-| M5 | ES2・mongo 廃止。ES8+postgres 15 以降。海外配布可 | CRITICAL/HIGH ゼロ。AI レッドチームの未解決経路ゼロ | 計画中 |
+| M5 | ES2・mongo 廃止。ES8+postgres 15 以降。海外配布可 | CRITICAL/HIGH ゼロ。AI レッドチームの未解決経路ゼロ | **下準備**(§2c): `db upgrade` 付きの PostgreSQL 18 を提案([ADR-12](../decisions/ADR-12_PostgreSQL_18_and_Data_Directory_Upgrade.md#japanese))。ES2 と mongo の事実は所有者の判断待ち([ADR-13](../decisions/ADR-13_Retiring_Elasticsearch_2_and_MongoDB.md#japanese)) |
### 2. M1 の残件
@@ -119,13 +132,24 @@ This board is the **single source of truth** for milestone progress:
| Trivy のイメージ検査と許容リスト件数(M1 より少なく) | ✅ 第 18 回で計測(Trivy 単独): 22 対象で CRITICAL 1835 / HIGH 8290 / MEDIUM 9763 / LOW 3886。ネイティブ Elasticsearch 2 の第 21 回: 1769 / 8030 / 9550 / 3676。許容リスト 0 件(M1: 0)。Semgrep・gitleaks・ZAP も実行(ZAP: HIGH 1・MEDIUM 9・LOW 13 は M3 で精査) |
| 既定ブランチを `develop` へ切替(所有者) | ⏳ M2 合格後 |
+
+### 2c. M5 の残件
+
+| 項目 | ステータス |
+|---|---|
+| PostgreSQL 9.6 → 18.6、`grdm-dev db status/upgrade/rollback`、`up` は版の不一致を拒む(終了 15) | 🔄 PR #42(レビュー r1 の指摘に対応)([ADR-12](../decisions/ADR-12_PostgreSQL_18_and_Data_Directory_Upgrade.md#japanese)): osf.io の DB 試験は 9.6 と 18 で同じ(781 合格、同じ 13 件が失敗かエラー)。9.6 のクラスタを 37 秒で更新し 531 件が一致。CAS と fakecas はログインできた。巻き戻しも測定。レビュー r1 の後、操作記録とロックでどの段階のクラッシュも復旧できる(SIGKILL と SIGTERM を測定、ADR-12 §4.1)。amd64 と Linux のホストでは未実行 |
+| PostgreSQL の SCRAM | ⏳ 所有者: RDM-cas-overlay-dev2027 で CAS に 42.2 以降の JDBC ドライバーが要る(42.1.4 は `authentication type 10` で失敗) |
+| Elasticsearch 2 の退役 | 🔄 2026-11-10 の所有者の判断: 全文検索を保ったまま osf.io の検索を Elasticsearch 8 へ移植する。osf.io の統合リポジトリで進行中([ADR-13 §4](../decisions/ADR-13_Retiring_Elasticsearch_2_and_MongoDB.md#japanese)) |
+| MongoDB と sharejs の退役 | 🔄 2026-11-10 の所有者の判断: ファイルページのエディタを sharejs から y-websocket へ移し、その後 ADR-13 §3 の手順で sharejs と mongo を退役させる。進行中 |
+| Elasticsearch 6 → 8(メトリクス) | ⏳ 未着手 |
+
### 3. 規則
このボードは節目進行状況の**唯一の正本**です:
1. 節目を言及するすべての文書はこのボードへリンクしなければならない。
2. 上記の「ステータス」列が権威あるものである。ステータス更新には PR と証跡(実行結果または受入合格)が必要。
-3. 「残件」は着手した節目ごとに表を持つ(M1 は §2、M2 は §2b)。後の節目は着手時に専用の表を追加する。
+3. 「残件」は着手した節目ごとに表を持つ(M1 は §2、M2 は §2b、M5 は §2c)。後の節目は着手時に専用の表を追加する。
---
diff --git a/installer/grdm_dev/cli.py b/installer/grdm_dev/cli.py
index bbc818b..8d0fadd 100644
--- a/installer/grdm_dev/cli.py
+++ b/installer/grdm_dev/cli.py
@@ -10,7 +10,7 @@
# Each module exposes register(subparsers) and run(ctx, args). Order = help order.
SUBCOMMAND_MODULES = [
- "doctor", "plan", "secrets", "compose", "build", "lifecycle", "health",
+ "doctor", "plan", "secrets", "compose", "build", "lifecycle", "database", "health",
"scan", "redteam", "accept", "upstream", "manifest_cmd", "docs", "report_cmd", "install",
]
diff --git a/installer/grdm_dev/database.py b/installer/grdm_dev/database.py
new file mode 100644
index 0000000..ed10c01
--- /dev/null
+++ b/installer/grdm_dev/database.py
@@ -0,0 +1,984 @@
+"""grdm-dev db status / upgrade / rollback / recover — move the PostgreSQL data directory to the major of the
+rendered image (ADR-12).
+
+A PostgreSQL server refuses a data directory written by another major ("database files are incompatible with
+server"), so a new image needs the data moved. The move is a dump and restore, never an in-place change.
+
+status prints the major of runtime/volumes/postgres_data, the major of the rendered `postgres` service, whether
+ `db upgrade` is needed and whether an interrupted operation is pending. Changes nothing.
+upgrade with every service of the project stopped (`grdm-dev down`), enough free disk (disk_problems) and after a
+ confirmation (--yes skips):
+ 1. writes the operation journal runtime/db-operation.json (fsynced) naming every container, volume and
+ directory it will create, before creating any of them; an exclusive lock on runtime/db-operation.lock is
+ held for the whole operation, and the kernel drops it when the process dies, however it dies;
+ 2. starts the OLD major (SOURCE_IMAGES, digest pinned) on the existing directory, with no network and no
+ TCP listener; its socket lives in a throwaway Docker volume;
+ 3. dumps every database and role with the NEW major's pg_dumpall (the PostgreSQL rule: dump with the newer
+ client), as the host user, into runtime/backups/postgres--to--/dumpall.sql;
+ 4. records, per database, every table's row count, every sequence's last_value and is_called, the
+ extension names, and the roles with their attributes and a hash of their password verifier; then removes
+ the old server and proves it is gone before going on;
+ 5. initialises the NEW major in runtime/volumes/postgres_data.pg-.new and restores the
+ dump with ON_ERROR_STOP (only `CREATE ROLE postgres;`, the role initdb made, is dropped); both servers
+ run with log_min_error_statement=panic so a failing statement, which may carry data or a password
+ hash, is not written to the container log;
+ 6. records the same facts on the new cluster and stops unless they are identical, then removes the new
+ server and proves it is gone;
+ 7. marks the journal `swap`, renames postgres_data to postgres_data.pg- and the new
+ directory to postgres_data (a failed second rename is compensated by renaming the first back), writes
+ runtime/backups/.../upgrade.json, and only then deletes the journal.
+ Renames stay inside runtime/volumes/: on Linux the data belongs to uid 999, and a host user may rename such
+ a directory within its parent but may not move it to another one.
+rollback same lock and journal: renames the current postgres_data aside (postgres_data.pg-rolledback-)
+ and the directory kept by the last upgrade back, then marks that upgrade record rolled back.
+recover finishes or rolls back an operation that a crash, a kill or a failure left in the journal (below).
+
+Recovery (recover(), also run first by upgrade, rollback, reset and `up`): every container and volume named in the
+journal is removed and proved gone (if that fails the journal stays and `up` refuses). Before the `swap` mark the
+old directory was never touched, so the operation is rolled back: the journal is deleted and the half-built new
+directory is left aside and named. After the mark the directories are inspected and the swap is finished
+(roll-forward, the data was already verified); if it cannot be finished and the old directory is still in place,
+it is rolled back instead. An empty postgres_data that `compose render` created during the gap is removed first.
+
+The verification compares counts, sequence states, extension names and role attributes. It does not compare row
+contents, and extensions are recreated at the new server's default version.
+
+`up` calls recover_pending() and then preflight(): a pending operation is finished or rolled back, and a data
+directory of another major, or one whose major cannot be read, stops `up` with exit 15 before Compose starts.
+
+Runner: `runner(argv) -> (rc, stdout)`; tests inject a fake and never need Docker. Every container runs with
+cap_drop ALL and no-new-privileges; the two servers add back only the capabilities of the `postgres` hardening
+exception (manifest/hardening.json), because the official entrypoint starts as root and drops to postgres.
+"""
+from __future__ import annotations
+
+import contextlib
+import errno
+import json
+import os
+import re
+import shutil
+import signal
+import sys
+import time
+from pathlib import Path
+from typing import Any, Callable, Iterator
+
+from . import compose, exit_codes, health, report
+from .context import Context
+from .secrets import env_path, read_env
+
+try: # POSIX only; the supported hosts (macOS, Linux, WSL2) all have it
+ import fcntl
+except ImportError: # pragma: no cover
+ fcntl = None # type: ignore[assignment]
+
+SERVICE = "postgres"
+VOLUME = "postgres_data"
+DATA_TARGET = "/var/lib/postgresql/data"
+SOCKET_DIR = "/var/run/postgresql"
+BACKUP_TARGET = "/backup"
+BACKUPS_DIR = "backups"
+JOURNAL = "db-operation.json"
+LOCK = "db-operation.lock"
+# Majors this command can read. The image is the one the stack ran before ADR-12, pinned by its index digest.
+SOURCE_IMAGES: dict[str, str] = {
+ "9.6": "postgres:9.6@sha256:caddd35b05cdd56c614ab1f674e63be778e0abdf54e71a7507ff3e28d4902698",
+}
+SERVER_CAPS = ("CHOWN", "DAC_OVERRIDE", "FOWNER", "SETGID", "SETUID") # = the postgres exception in manifest/hardening.json
+POSTGRES_UID = "999:999" # the postgres user of the Debian-based official images, 9.6 and 18 alike
+BOOTSTRAP_ROLE_LINE = "CREATE ROLE postgres;"
+READY_TIMEOUT = 600.0
+MIB = 1024 * 1024
+# Disk estimate (disk_problems). A logical dump is usually smaller than the cluster, but not always (bytea is written
+# as hex, compressed TOAST values are written expanded), and dumpall.sql and restore.sql coexist until the restore
+# ends: twice the cluster plus 64 MiB on the backups filesystem. The new cluster rebuilds every index and writes WAL
+# up to max_wal_size (1 GiB by default): twice the cluster plus 1 GiB on the volumes filesystem. Both on one
+# filesystem: the sum. This is an estimate, not a guarantee; a disk that fills anyway is a failure like any other.
+BACKUP_FACTOR, BACKUP_MARGIN = 2, 64 * MIB
+VOLUME_FACTOR, VOLUME_MARGIN = 2, 1024 * MIB
+VERSION_RE = re.compile(r"^\d+(\.\d+)?$")
+# Every table's row count, every sequence's last_value and is_called, and the extension names, by name; the same
+# text on both majors (9.6 and 18). The casts matter: PostgreSQL 18 cannot choose an operator for text || "char"
+# (measured), 9.6 could.
+SNAPSHOT_SQL = (
+ "SELECT n.nspname::text || '.' || c.relname::text || ' ' || c.relkind::text || ' ' || "
+ "(xpath('/row/v/text()', query_to_xml(format(CASE c.relkind "
+ "WHEN 'S' THEN 'SELECT last_value::text || '' '' || is_called::text AS v FROM %I.%I' "
+ "ELSE 'SELECT count(*) AS v FROM %I.%I' END, n.nspname, c.relname), false, true, '')))[1]::text "
+ "FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace "
+ "WHERE c.relkind IN ('r', 'S') AND n.nspname NOT IN ('pg_catalog', 'information_schema') "
+ "AND n.nspname NOT LIKE 'pg_toast%' "
+ "UNION ALL SELECT 'extension ' || extname::text FROM pg_extension ORDER BY 1"
+)
+DATABASES_SQL = "SELECT datname FROM pg_database WHERE datallowconn AND NOT datistemplate ORDER BY 1"
+# Role attributes and a hash of the password verifier (the verifier itself is never printed).
+ROLES_SQL = (
+ "SELECT rolname::text || ' ' || rolsuper::text || ' ' || rolinherit::text || ' ' || rolcreaterole::text || ' ' || "
+ "rolcreatedb::text || ' ' || rolcanlogin::text || ' ' || rolreplication::text || ' ' || rolbypassrls::text || ' ' || "
+ "rolconnlimit::text || ' ' || md5(coalesce(rolpassword, '')) FROM pg_authid WHERE rolname !~ '^pg_' ORDER BY 1"
+)
+QUIET_SERVER = ("-c", "listen_addresses=", "-c", "log_min_error_statement=panic")
+
+Runner = Callable[[list[str]], tuple[int, str]]
+
+
+class UpgradeError(Exception):
+ """A step failed; the message says which."""
+
+
+def register(subparsers) -> None:
+ p = subparsers.add_parser("db", help="PostgreSQL data directory: status, upgrade to the rendered major, rollback, recover")
+ sub = p.add_subparsers(dest="db_command", metavar="")
+ sub.add_parser("status", help="show both majors and any pending operation; changes nothing")
+ sub.add_parser("upgrade", help="dump and restore runtime/volumes/postgres_data into the rendered major; keeps the old directory; exit 15 on failure")
+ sub.add_parser("rollback", help="put the directory kept by the last upgrade back in place; exit 15 on failure")
+ sub.add_parser("recover", help="finish or roll back an interrupted upgrade or rollback; exit 15 if it cannot")
+ p.set_defaults(_module=sys.modules[__name__])
+
+
+# --------------------------------------------------------------------------- facts
+
+
+def image_major(image: str | None) -> str | None:
+ """`postgres:18.6@sha256:…` → "18", `postgres:9.6` → "9.6" (before 10 a major had two parts). None if no tag."""
+ if not image:
+ return None
+ ref = image.split("@", 1)[0]
+ name, _, tag = ref.rpartition(":")
+ if not name or "/" in tag:
+ return None
+ m = re.match(r"^(\d+)(?:\.(\d+))?", tag)
+ if not m:
+ return None
+ first = int(m.group(1))
+ if first < 10:
+ return f"{first}.{m.group(2)}" if m.group(2) is not None else None
+ return str(first)
+
+
+def data_dir(ctx: Context) -> Path:
+ return ctx.runtime / compose.VOLUMES_DIR / VOLUME
+
+
+def journal_path(ctx: Context) -> Path:
+ return ctx.runtime / JOURNAL
+
+
+def rendered_service(ctx: Context) -> dict[str, Any] | None:
+ try:
+ data = health.load_compose(health.compose_path(ctx))
+ except (OSError, ValueError):
+ return None
+ return (data.get("services") or {}).get(SERVICE)
+
+
+def data_major(directory: Path, runner: Runner | None = None, image: str | None = None, platform: str | None = None) -> str | None:
+ """The major written in PG_VERSION; "" only when the directory is absent or empty (no cluster yet); None when
+ it cannot be established (unreadable, a non-empty directory without PG_VERSION, or an unparseable value).
+
+ On Linux the directory belongs to uid 999 with mode 0700, so the host user cannot read it; then it is read by the
+ postgres uid in a throwaway container of `image`, with no network and nothing writable."""
+ try:
+ if not directory.exists():
+ return ""
+ value = (directory / "PG_VERSION").read_text(encoding="ascii").strip()
+ return value if VERSION_RE.match(value) else None
+ except ValueError: # UnicodeDecodeError: bytes that are not ASCII are not a version (PR #42 review r2 R2-04)
+ return None
+ except FileNotFoundError:
+ try:
+ return "" if not any(directory.iterdir()) else None
+ except OSError:
+ pass
+ except (PermissionError, NotADirectoryError):
+ pass
+ except OSError:
+ return None
+ if runner is None or not image:
+ return None
+ argv = ["docker", "run", "--rm", "--network", "none", "--read-only", "--cap-drop", "ALL",
+ "--security-opt", "no-new-privileges:true", "--user", POSTGRES_UID]
+ if platform:
+ argv += ["--platform", platform]
+ argv += ["-v", f"{directory}:/data:ro", "--entrypoint", "sh", image, "-c",
+ 'if [ -e /data/PG_VERSION ]; then cat /data/PG_VERSION; elif [ -z "$(ls -A /data)" ]; then echo empty; '
+ "else echo nocluster; fi"]
+ try:
+ rc, out = runner(argv)
+ except ValueError: # output that does not decode
+ return None
+ value = out.strip() if rc == 0 else ""
+ if value == "empty":
+ return ""
+ return value if VERSION_RE.match(value) else None
+
+
+def preflight(ctx: Context, runner: Runner | None = None) -> str | None:
+ """Why `up` must not start postgres on the current data directory, or None. Fails closed: a major that cannot be
+ read is a refusal, not a pass (PR #42 review r1 R1-04)."""
+ svc = rendered_service(ctx)
+ if not svc:
+ return None
+ want = image_major(svc.get("image"))
+ if not want:
+ return None # not a postgres: image; nothing to compare against
+ have = data_major(data_dir(ctx), runner, svc.get("image"), svc.get("platform"))
+ if have is None:
+ return (f"cannot read the PostgreSQL major of {data_dir(ctx)} (PG_VERSION unreadable or missing in a non-empty "
+ f"directory); refusing to start it with PostgreSQL {want}; see `grdm-dev db status`")
+ if have and have != want:
+ return (f"{data_dir(ctx)} was written by PostgreSQL {have} but the rendered image is PostgreSQL {want}; "
+ f"run `grdm-dev db upgrade` (it keeps the old directory), or check out the revision that matches the data")
+ return None
+
+
+def tree_size(directory: Path, runner: Runner | None = None, image: str | None = None, platform: str | None = None) -> int | None:
+ """Bytes allocated under `directory`; read by uid 999 in a container when the host user cannot walk it."""
+ total = 0
+ try:
+ for root, dirs, files in os.walk(directory, onerror=_raise):
+ for name in files + dirs:
+ try:
+ total += os.lstat(os.path.join(root, name)).st_blocks * 512
+ except FileNotFoundError:
+ continue
+ return total
+ except OSError:
+ pass
+ if runner is None or not image:
+ return None
+ argv = ["docker", "run", "--rm", "--network", "none", "--read-only", "--cap-drop", "ALL",
+ "--security-opt", "no-new-privileges:true", "--user", POSTGRES_UID]
+ if platform:
+ argv += ["--platform", platform]
+ rc, out = runner(argv + ["-v", f"{directory}:/data:ro", "--entrypoint", "du", image, "-sk", "/data"])
+ try:
+ return int(out.split()[0]) * 1024 if rc == 0 else None
+ except (IndexError, ValueError):
+ return None
+
+
+def _raise(exc: OSError) -> None:
+ raise exc
+
+
+def disk_problems(size: int, backups: Path, volumes: Path, disk_usage=shutil.disk_usage) -> list[str]:
+ """Free space needed before anything starts (see the estimate at BACKUP_FACTOR)."""
+ need_backup = BACKUP_FACTOR * size + BACKUP_MARGIN
+ need_volume = VOLUME_FACTOR * size + VOLUME_MARGIN
+ try:
+ same = backups.stat().st_dev == volumes.stat().st_dev
+ free_backup, free_volume = disk_usage(backups).free, disk_usage(volumes).free
+ except OSError as exc:
+ return [f"cannot measure free disk space: {exc}"]
+ mib = lambda n: f"{n // MIB} MiB" # noqa: E731
+ if same and free_volume < need_backup + need_volume:
+ return [f"{volumes} needs {mib(need_backup + need_volume)} free for the dump and the new cluster "
+ f"(the cluster is {mib(size)}), has {mib(free_volume)}"]
+ problems = []
+ if not same and free_backup < need_backup:
+ problems.append(f"{backups} needs {mib(need_backup)} free for the dump, has {mib(free_backup)}")
+ if not same and free_volume < need_volume:
+ problems.append(f"{volumes} needs {mib(need_volume)} free for the new cluster, has {mib(free_volume)}")
+ return problems
+
+
+# --------------------------------------------------------------------------- journal and lock
+
+
+# A filesystem that cannot sync a directory says so with one of these; that is not a failed write.
+UNSYNCABLE = {errno.EINVAL, getattr(errno, "ENOTSUP", errno.EINVAL), getattr(errno, "EOPNOTSUPP", errno.EINVAL)}
+
+
+def _fsync_dir(path: Path) -> None:
+ """Make renames and unlinks in `path` durable. Any other error propagates (PR #42 review r2 R2-03): an entry
+ that may not be on disk must not be reported as done."""
+ fd = os.open(path, os.O_RDONLY)
+ try:
+ os.fsync(fd)
+ except OSError as exc:
+ if exc.errno not in UNSYNCABLE:
+ raise
+ finally:
+ os.close(fd)
+
+
+def write_durable(path: Path, data: dict[str, Any]) -> None:
+ """Write JSON so that after a crash the file is either the old or the new version, complete (fsync + rename)."""
+ tmp = path.with_name(path.name + ".tmp")
+ fd = os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
+ with os.fdopen(fd, "w", encoding="utf-8") as fh:
+ fh.write(json.dumps(data, ensure_ascii=False, indent=2) + "\n")
+ fh.flush()
+ os.fsync(fh.fileno())
+ os.replace(tmp, path)
+ _fsync_dir(path.parent)
+
+
+def read_journal(ctx: Context) -> dict[str, Any] | None:
+ path = journal_path(ctx)
+ if not path.exists():
+ return None
+ try:
+ data = json.loads(path.read_text(encoding="utf-8"))
+ except (OSError, ValueError) as exc:
+ raise UpgradeError(f"{path} cannot be read ({exc}); inspect it and runtime/volumes/ by hand") from exc
+ if not isinstance(data, dict) or data.get("op") not in ("upgrade", "rollback"):
+ raise UpgradeError(f"{path} is not a db operation journal; inspect it by hand")
+ return data
+
+
+def _drop_journal(ctx: Context) -> None:
+ journal_path(ctx).unlink(missing_ok=True)
+ _fsync_dir(ctx.runtime)
+
+
+LOCK_HELD = "another grdm-dev command (db upgrade, rollback, recover, up or reset) holds runtime/db-operation.lock; wait for it"
+
+
+def _acquire(ctx: Context) -> tuple[int | None, str | None]:
+ """(fd, None) with the exclusive lock held, or (None, refusal). The lock is a flock on its own open file
+ description, so it also excludes a second open in the same process, and the kernel drops it when the process
+ ends, however it ends."""
+ ctx.runtime.mkdir(parents=True, exist_ok=True)
+ fd = os.open(ctx.runtime / LOCK, os.O_RDWR | os.O_CREAT, 0o600)
+ if fcntl is not None:
+ try:
+ fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
+ except OSError:
+ os.close(fd)
+ return None, LOCK_HELD
+ return fd, None
+
+
+@contextlib.contextmanager
+def operation_lock(ctx: Context) -> Iterator[None]:
+ """Exclusive, non-blocking; raises UpgradeError when held elsewhere."""
+ fd, refusal = _acquire(ctx)
+ if refusal:
+ raise UpgradeError(refusal)
+ try:
+ yield
+ finally:
+ os.close(fd)
+
+
+@contextlib.contextmanager
+def signals_interrupt() -> Iterator[None]:
+ """SIGTERM and SIGHUP raise KeyboardInterrupt like Ctrl-C, so the cleanup path runs for all three."""
+ def handler(signum, frame): # noqa: ARG001
+ raise KeyboardInterrupt(f"signal {signum}")
+
+ saved = {}
+ for sig in (signal.SIGTERM, signal.SIGHUP):
+ try:
+ saved[sig] = signal.signal(sig, handler)
+ except (ValueError, OSError): # not the main thread
+ pass
+ try:
+ yield
+ finally:
+ for sig, old in saved.items():
+ signal.signal(sig, old)
+
+
+# --------------------------------------------------------------------------- docker argv
+
+
+def _auth_method(env: dict[str, str]) -> str:
+ """The host auth method a fresh install gets: initdb's --auth-host (which also sets password_encryption)."""
+ m = re.search(r"--auth-host[= ](\S+)", env.get("POSTGRES_INITDB_ARGS", ""))
+ return m.group(1) if m else "scram-sha-256"
+
+
+def server_argv(name: str, image: str, directory: Path, socket_volume: str, *, platform: str | None,
+ env: dict[str, str], command: list[str], tmpfs: list[str], backup: Path | None = None,
+ env_file: Path | None = None) -> list[str]:
+ argv = ["docker", "run", "-d", "--name", name, "--network", "none", "--read-only",
+ "--cap-drop", "ALL", *[x for cap in SERVER_CAPS for x in ("--cap-add", cap)],
+ "--security-opt", "no-new-privileges:true", "--label", "org.gakunin-rdm.component=postgres-upgrade"]
+ if platform:
+ argv += ["--platform", platform]
+ for path in tmpfs:
+ argv += ["--tmpfs", path]
+ argv += ["-v", f"{directory}:{DATA_TARGET}", "-v", f"{socket_volume}:{SOCKET_DIR}"]
+ if backup is not None:
+ argv += ["-v", f"{backup}:{BACKUP_TARGET}:ro"]
+ for key, value in env.items():
+ argv += ["-e", f"{key}={value}"]
+ if env_file is not None:
+ argv += ["--env-file", str(env_file)] # the password never appears in argv
+ # No TCP listener at all, and no failing statement (it may hold data or a password hash) in the server log.
+ return argv + [image, *command, *QUIET_SERVER]
+
+
+def exec_argv(name: str, *cmd: str) -> list[str]:
+ # root in the container keeps DAC_OVERRIDE, so it reads a 0700 backup directory owned by the host user
+ return ["docker", "exec", "-u", "0", name, *cmd]
+
+
+def psql_argv(name: str, database: str, sql: str) -> list[str]:
+ return exec_argv(name, "psql", "-X", "-q", "-A", "-t", "-v", "ON_ERROR_STOP=1", "-h", SOCKET_DIR, "-U", "postgres", "-d", database, "-c", sql)
+
+
+def dump_argv(image: str, socket_volume: str, backup: Path, platform: str | None, user: str) -> list[str]:
+ argv = ["docker", "run", "--rm", "--network", "none", "--read-only", "--cap-drop", "ALL",
+ "--security-opt", "no-new-privileges:true", "--user", user]
+ if platform:
+ argv += ["--platform", platform]
+ return argv + ["-v", f"{socket_volume}:{SOCKET_DIR}", "-v", f"{backup}:{BACKUP_TARGET}",
+ "--entrypoint", "pg_dumpall", image, "-h", SOCKET_DIR, "-U", "postgres", "-f", f"{BACKUP_TARGET}/dumpall.sql"]
+
+
+# --------------------------------------------------------------------------- containers
+
+
+def container_present(name: str, runner: Runner) -> bool:
+ """True if a container of that name exists (running or not). Raises when Docker cannot answer: unknown is not absent."""
+ rc, out = runner(["docker", "ps", "-a", "--filter", f"name=^/{name}$", "--format", "{{.Names}}"])
+ if rc != 0:
+ raise UpgradeError(f"cannot list containers (docker ps exited {rc}); {name} may still run")
+ return name in out.split()
+
+
+def remove_container(name: str, runner: Runner) -> None:
+ """Stop and remove, then prove it is gone (PR #42 review r1 R1-02); a server still holding a data directory
+ must never be forgotten."""
+ if not container_present(name, runner):
+ return
+ runner(["docker", "stop", "-t", "60", name])
+ runner(["docker", "rm", "-f", name])
+ if container_present(name, runner):
+ raise UpgradeError(f"container {name} could not be stopped and removed; remove it with `docker rm -f {name}`, "
+ f"then run `grdm-dev db recover`")
+
+
+def wait_ready(name: str, runner: Runner, sleep: Callable[[float], None], timeout: float = READY_TIMEOUT, interval: float = 2.0) -> None:
+ """Ready = the final server is PID 1 (the entrypoint's temporary init server never is) and accepts connections."""
+ probe = 'p=$(head -n1 "$PGDATA/postmaster.pid" 2>/dev/null); [ "$p" = 1 ] && pg_isready -q -h ' + SOCKET_DIR
+ waited = 0.0
+ while True:
+ rc, out = runner(["docker", "inspect", "-f", "{{.State.Running}}", name])
+ if rc != 0 or out.strip() != "true":
+ raise UpgradeError(f"container {name} is not running")
+ rc, _ = runner(exec_argv(name, "sh", "-c", probe))
+ if rc == 0:
+ return
+ if waited >= timeout:
+ raise UpgradeError(f"container {name} not ready after {int(timeout)}s")
+ sleep(interval)
+ waited += interval
+
+
+def snapshot(name: str, runner: Runner) -> dict[str, Any]:
+ """{"databases": {database: sorted lines}, "roles": [lines]}: two namespaces, so a database called `roles` keeps
+ its own check (R1-05). Equal snapshots mean equal row counts, sequence states (last_value and is_called),
+ extension names and role attributes; they do not prove equal row contents."""
+ rc, out = runner(psql_argv(name, "postgres", DATABASES_SQL))
+ if rc != 0:
+ raise UpgradeError(f"listing databases in {name} failed (psql exited {rc})")
+ databases: dict[str, list[str]] = {}
+ for db in [line.strip() for line in out.splitlines() if line.strip()]:
+ rc, rows = runner(psql_argv(name, db, SNAPSHOT_SQL))
+ if rc != 0:
+ raise UpgradeError(f"counting rows of {db} in {name} failed (psql exited {rc})")
+ databases[db] = [line for line in rows.splitlines() if line.strip()]
+ rc, out = runner(psql_argv(name, "postgres", ROLES_SQL))
+ if rc != 0:
+ raise UpgradeError(f"listing roles in {name} failed (psql exited {rc})")
+ return {"databases": databases, "roles": [line for line in out.splitlines() if line.strip()]}
+
+
+def snapshot_differences(old: dict[str, Any], new: dict[str, Any]) -> list[str]:
+ diffs: list[str] = []
+ od, nd = old.get("databases", {}), new.get("databases", {})
+ for db in sorted(set(od) | set(nd)):
+ if db not in nd:
+ diffs.append(f"database {db}: missing after restore")
+ elif db not in od:
+ diffs.append(f"database {db}: new after restore")
+ else:
+ a, b = set(od[db]), set(nd[db])
+ diffs += [f"database {db}: before `{x}`" for x in sorted(a - b)[:5]] + [f"database {db}: after `{x}`" for x in sorted(b - a)[:5]]
+ ro, rn = set(old.get("roles", [])), set(new.get("roles", []))
+ diffs += [f"role before `{x.split()[0]} …`" for x in sorted(ro - rn)[:5]] + [f"role after `{x.split()[0]} …`" for x in sorted(rn - ro)[:5]]
+ return diffs
+
+
+def write_restore_file(dump: Path, restore: Path) -> int:
+ """Copy the dump without the bootstrap role's CREATE (initdb made it; its ALTER with the password stays)."""
+ removed = 0
+ fd = os.open(restore, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
+ with dump.open(encoding="utf-8") as src, os.fdopen(fd, "w", encoding="utf-8") as dst:
+ for line in src:
+ if line.rstrip("\n") == BOOTSTRAP_ROLE_LINE:
+ removed += 1
+ continue
+ dst.write(line)
+ if removed != 1:
+ raise UpgradeError(f"expected one `{BOOTSTRAP_ROLE_LINE}` in {dump}, found {removed}")
+ return removed
+
+
+def _no_running_services(ctx: Context, runner: Runner) -> str | None:
+ from . import lifecycle
+
+ try:
+ running = lifecycle.running_services(ctx, runner)
+ except RuntimeError as exc:
+ return f"cannot tell whether the project is running ({exc})"
+ if running:
+ return f"services are running ({', '.join(sorted(running))}); run `grdm-dev down` first"
+ return None
+
+
+def latest_record(ctx: Context) -> tuple[Path, dict[str, Any]] | None:
+ """The newest upgrade record that has not been rolled back."""
+ base = ctx.runtime / BACKUPS_DIR
+ records = sorted(base.glob("postgres-*/upgrade.json")) if base.is_dir() else []
+ for path in reversed(records):
+ try:
+ data = json.loads(path.read_text(encoding="utf-8"))
+ except (OSError, ValueError):
+ continue
+ if not data.get("rolled_back_at"):
+ return path, data
+ return None
+
+
+# --------------------------------------------------------------------------- recovery
+
+
+def _major_of(path: Path, runner: Runner, journal: dict[str, Any]) -> str | None:
+ return data_major(path, runner, journal.get("image"), journal.get("platform"))
+
+
+def _drop_empty_placeholder(current: Path, runner: Runner, journal: dict[str, Any]) -> None:
+ """`compose render` creates a missing volume directory; while a swap is pending that empty directory is not data."""
+ if current.is_dir() and _major_of(current, runner, journal) == "":
+ try:
+ current.rmdir()
+ except OSError as exc:
+ raise UpgradeError(f"{current} is an empty placeholder that cannot be removed ({exc})") from exc
+
+
+def _write_record(journal: dict[str, Any]) -> Path:
+ record = Path(journal["backup"]) / "upgrade.json"
+ if not record.exists():
+ write_durable(record, {k: journal[k] for k in ("from", "to", "source_image", "image", "data_dir", "kept", "dump", "at", "steps") if k in journal})
+ return record
+
+
+def recover(ctx: Context, runner: Runner) -> str | None:
+ """Finish or roll back the journalled operation; None when nothing was pending. Raises UpgradeError (journal
+ kept) when it can do neither safely. The caller holds the lock."""
+ journal = read_journal(ctx)
+ if journal is None:
+ return None
+ for name in journal.get("containers", []):
+ remove_container(name, runner)
+ if journal.get("volume"):
+ runner(["docker", "volume", "rm", journal["volume"]]) # a leftover socket volume holds no data
+ current = Path(journal["data_dir"])
+ if journal["op"] == "upgrade":
+ new_dir, kept = Path(journal["new_dir"]), Path(journal["kept"])
+ if journal.get("phase") != "swap":
+ if not current.is_dir() or kept.exists():
+ raise UpgradeError(f"journal says the swap had not started, but {current} is missing or {kept} exists; inspect by hand")
+ _drop_journal(ctx)
+ left = f"; the unfinished {new_dir} can be deleted" if new_dir.exists() else ""
+ return f"rolled back an interrupted upgrade before its swap: {current} is the untouched PostgreSQL {journal['from']} cluster{left}"
+ _drop_empty_placeholder(current, runner, journal)
+ if not current.exists() and kept.is_dir() and new_dir.is_dir():
+ try:
+ _rename_durably(new_dir, current)
+ except OSError as exc:
+ _rename_durably(kept, current) # back to the old cluster; if this raises the journal stays
+ _undone(ctx, runner, journal, current, journal["from"], exc)
+ return f"the swap could not be finished; {current} is the old PostgreSQL {journal['from']} cluster again, {new_dir} is left aside"
+ elif current.is_dir() and not kept.exists() and new_dir.is_dir() and _major_of(current, runner, journal) == journal["from"]:
+ try:
+ _swap(current, new_dir, kept)
+ except OSError as exc:
+ _undone(ctx, runner, journal, current, journal["from"], exc)
+ return f"the swap could not be done ({exc}); {current} is the old PostgreSQL {journal['from']} cluster, {new_dir} is left aside"
+ elif not (current.is_dir() and kept.is_dir() and not new_dir.exists() and _major_of(current, runner, journal) == journal["to"]):
+ raise UpgradeError(f"cannot tell how far the swap went ({current}, {kept}, {new_dir}); inspect by hand")
+ _fsync_dir(current.parent) # the renames are on disk before the record and the journal say so
+ _write_record(journal)
+ _drop_journal(ctx)
+ return f"finished an interrupted upgrade: {current} is PostgreSQL {journal['to']}, the old cluster is {kept}"
+ aside, kept = Path(journal["aside"]), Path(journal["kept"])
+ _drop_empty_placeholder(current, runner, journal)
+ if current.is_dir() and kept.is_dir() and not aside.exists():
+ try:
+ _swap(current, kept, aside)
+ except OSError as exc:
+ _undone(ctx, runner, journal, current, journal["to"], exc)
+ return f"the rollback could not be done ({exc}); {current} is unchanged"
+ elif not current.exists() and aside.is_dir() and kept.is_dir():
+ _rename_durably(kept, current)
+ elif not (current.is_dir() and aside.is_dir() and not kept.exists()):
+ raise UpgradeError(f"cannot tell how far the rollback went ({current}, {kept}, {aside}); inspect by hand")
+ _fsync_dir(current.parent)
+ _mark_rolled_back(journal)
+ _drop_journal(ctx)
+ return f"finished an interrupted rollback: {current} is PostgreSQL {journal['from']} again"
+
+
+def _rename_durably(src: Path, dst: Path) -> None:
+ os.rename(src, dst)
+ _fsync_dir(dst.parent)
+
+
+def _undone(ctx: Context, runner: Runner, journal: dict[str, Any], current: Path, major: str, exc: OSError) -> None:
+ """A swap failed. Retire the journal only when the directory state proves the compensation worked: `current`
+ is there, holds `major`, and is on disk. Otherwise the journal stays and the error propagates (R2-01)."""
+ if current.is_dir() and _major_of(current, runner, journal) == major:
+ _fsync_dir(current.parent)
+ _drop_journal(ctx)
+ return
+ raise UpgradeError(f"a swap failed ({exc}) and so did putting {current} back; the journal is kept, "
+ f"`up` refuses, and `grdm-dev db recover` retries") from exc
+
+
+def _swap(current: Path, replacement: Path, aside: Path) -> None:
+ """current → aside, replacement → current; a failed second rename puts the first back."""
+ os.rename(current, aside)
+ try:
+ os.rename(replacement, current)
+ except OSError:
+ os.rename(aside, current) # if this raises too, `current` is missing and the caller must keep the journal
+ _fsync_dir(current.parent)
+ raise
+ _fsync_dir(current.parent)
+
+
+def _mark_rolled_back(journal: dict[str, Any]) -> None:
+ record = Path(journal["record"])
+ try:
+ data = json.loads(record.read_text(encoding="utf-8"))
+ except (OSError, ValueError):
+ return
+ data["rolled_back_at"] = journal["at"]
+ write_durable(record, data)
+
+
+def recover_pending(ctx: Context, runner: Runner) -> str | None:
+ """For `up` and `reset`, called while they hold operation_lock: finish or roll back a pending operation.
+ Returns a refusal, or None."""
+ try:
+ outcome = recover(ctx, runner)
+ except (UpgradeError, OSError) as exc:
+ return f"an interrupted `grdm-dev db` operation could not be recovered: {exc}"
+ if outcome:
+ sys.stderr.write(f"[grdm-dev] {outcome}\n")
+ return None
+
+
+@contextlib.contextmanager
+def lifecycle_lock(ctx: Context) -> Iterator[str | None]:
+ """`up` and `reset` hold the same lock as `db upgrade/rollback/recover` from their checks through starting or
+ deleting (PR #42 review r2 R2-02), so neither can run between the other's checks and its action. Yields a
+ refusal when another command holds it, else None while holding it."""
+ fd, refusal = _acquire(ctx)
+ try:
+ yield refusal
+ finally:
+ if fd is not None:
+ os.close(fd)
+
+
+# --------------------------------------------------------------------------- commands
+
+
+def run(ctx: Context, args, runner: Runner | None = None, sleep: Callable[[float], None] | None = None,
+ user: str | None = None, stamp: str | None = None, disk_usage=shutil.disk_usage) -> int:
+ action = getattr(args, "db_command", None)
+ runner = runner or health.default_runner
+ if action == "status":
+ return run_status(ctx, runner)
+ if action == "upgrade":
+ return run_upgrade(ctx, runner, sleep or time.sleep, user=user, stamp=stamp, disk_usage=disk_usage)
+ if action == "rollback":
+ return run_rollback(ctx, runner, stamp=stamp)
+ if action == "recover":
+ return run_recover(ctx, runner)
+ return report.error(ctx, exit_codes.USAGE, "choose status, upgrade, rollback or recover")
+
+
+def _target(ctx: Context) -> tuple[dict[str, Any] | None, str | None]:
+ svc = rendered_service(ctx)
+ if svc is None:
+ return None, f"{health.compose_path(ctx)} has no `{SERVICE}` service; run `grdm-dev compose render` first"
+ if not image_major(svc.get("image")):
+ return None, f"cannot tell the PostgreSQL major of the rendered image {svc.get('image')!r}"
+ return svc, None
+
+
+def run_status(ctx: Context, runner: Runner) -> int:
+ svc, problem = _target(ctx)
+ if problem:
+ return report.error(ctx, exit_codes.USAGE, problem)
+ want = image_major(svc.get("image"))
+ try:
+ pending = read_journal(ctx)
+ except UpgradeError as exc:
+ pending = {"op": "unreadable", "phase": str(exc)}
+ have = data_major(data_dir(ctx), runner, svc.get("image"), svc.get("platform"))
+ state = "empty" if have == "" else "unknown" if have is None else "current" if have == want else "upgrade-needed"
+ record = latest_record(ctx)
+ report.emit(ctx, "db_status", {
+ "data_dir": str(data_dir(ctx)), "data_major": have, "image": svc.get("image"), "image_major": want, "state": state,
+ "upgradable": state == "upgrade-needed" and have in SOURCE_IMAGES,
+ "pending_operation": f"{pending['op']} ({pending.get('phase')})" if pending else None,
+ "last_upgrade": str(record[0].parent) if record else None,
+ })
+ return exit_codes.OK
+
+
+def _os_failure(ctx: Context, exc: OSError) -> str:
+ """A filesystem error (EIO, ENOSPC, …) as an exit-15 message that says whether an operation is still pending."""
+ pending = journal_path(ctx).exists()
+ return (f"filesystem error: {exc}; " + (f"the journal {journal_path(ctx)} is kept, `up` refuses, and `grdm-dev db recover` "
+ f"finishes or rolls back once the error is gone" if pending else
+ "no operation is pending; see `grdm-dev db status`"))
+
+
+def run_recover(ctx: Context, runner: Runner) -> int:
+ try:
+ with operation_lock(ctx):
+ outcome = recover(ctx, runner)
+ except UpgradeError as exc:
+ return report.error(ctx, exit_codes.DB_UPGRADE_FAILED, f"recovery failed: {exc}; the journal {journal_path(ctx)} is kept")
+ except OSError as exc:
+ return report.error(ctx, exit_codes.DB_UPGRADE_FAILED, _os_failure(ctx, exc))
+ report.emit(ctx, "db_recover", {"pending": outcome is not None, "outcome": outcome})
+ return exit_codes.OK
+
+
+def run_upgrade(ctx: Context, runner: Runner, sleep: Callable[[float], None], user: str | None = None,
+ stamp: str | None = None, disk_usage=shutil.disk_usage) -> int:
+ svc, problem = _target(ctx)
+ if problem:
+ return report.error(ctx, exit_codes.USAGE, problem)
+ try:
+ with operation_lock(ctx):
+ return _upgrade_locked(ctx, svc, runner, sleep, user, stamp, disk_usage)
+ except UpgradeError as exc:
+ return report.error(ctx, exit_codes.DB_UPGRADE_FAILED, str(exc))
+ except OSError as exc: # e.g. an fsync that fails while the pending operation is recovered first (PR #42 review r3 R3-01)
+ return report.error(ctx, exit_codes.DB_UPGRADE_FAILED, _os_failure(ctx, exc))
+
+
+def _upgrade_locked(ctx: Context, svc: dict[str, Any], runner: Runner, sleep: Callable[[float], None],
+ user: str | None, stamp: str | None, disk_usage) -> int:
+ recovered = recover(ctx, runner)
+ if recovered:
+ sys.stderr.write(f"[grdm-dev] {recovered}\n")
+ image, platform = str(svc["image"]), svc.get("platform")
+ want = image_major(image)
+ current = data_dir(ctx)
+ bad = compose.volume_path_problems(ctx.root, ctx.runtime, [VOLUME])
+ if bad:
+ raise UpgradeError("unsafe volume path: " + "; ".join(bad))
+ have = data_major(current, runner, image, platform)
+ if have is None:
+ raise UpgradeError(f"cannot read the PostgreSQL major of {current}")
+ if have in ("", want):
+ report.emit(ctx, "db_upgrade", {"data_dir": str(current), "data_major": have or None, "image_major": want,
+ "changed": False, "recovered": recovered,
+ "reason": "no cluster yet" if have == "" else "already current"})
+ return exit_codes.OK
+ source = SOURCE_IMAGES.get(have)
+ if not source:
+ raise UpgradeError(f"no source image for PostgreSQL {have}; known: {', '.join(SOURCE_IMAGES)}")
+ password = read_env(env_path(ctx)).get("POSTGRES_PASSWORD")
+ if not password:
+ return report.error(ctx, exit_codes.USAGE, f"{env_path(ctx)} has no POSTGRES_PASSWORD; run `grdm-dev secrets init` first")
+ busy = _no_running_services(ctx, runner)
+ if busy:
+ raise UpgradeError(busy)
+ backups = ctx.runtime / BACKUPS_DIR
+ backups.mkdir(parents=True, exist_ok=True)
+ size = tree_size(current, runner, image, platform)
+ if size is None:
+ raise UpgradeError(f"cannot measure the size of {current}")
+ space = disk_problems(size, backups, current.parent, disk_usage)
+ if space:
+ raise UpgradeError("not enough free disk: " + "; ".join(space) + " (an estimate; see ADR-12 §5)")
+ if not ctx.confirm(f"Dump {current} (PostgreSQL {have}) and restore it into PostgreSQL {want}? The old directory is kept.", default=False):
+ if not ctx.yes and not ctx.stdin_is_tty:
+ return report.error(ctx, exit_codes.USAGE, "db upgrade needs a confirmation; add --yes when no terminal is attached")
+ sys.stderr.write("[grdm-dev] db upgrade cancelled\n")
+ return exit_codes.OK
+
+ ts = stamp or report.timestamp()
+ backup = backups / f"postgres-{have}-to-{want}-{ts}"
+ journal: dict[str, Any] = {
+ "op": "upgrade", "phase": "prepare", "at": ts, "from": have, "to": want, "source_image": source, "image": image,
+ "platform": platform, "data_dir": str(current), "new_dir": str(current.with_name(f"{VOLUME}.pg{want}-{ts}.new")),
+ "kept": str(current.with_name(f"{VOLUME}.pg{have}-{ts}")), "backup": str(backup), "dump": str(backup / "dumpall.sql"),
+ "containers": [f"grdm-dev-dbupgrade-old-{ts}", f"grdm-dev-dbupgrade-new-{ts}"],
+ "volume": f"grdm-dev-dbupgrade-{ts}".lower(), "steps": [],
+ }
+ try:
+ write_durable(journal_path(ctx), journal)
+ except OSError as exc:
+ journal_path(ctx).unlink(missing_ok=True)
+ raise UpgradeError(f"cannot write the journal ({exc}); nothing was started") from exc
+ try:
+ with signals_interrupt():
+ _upgrade_steps(ctx, svc, journal, runner, sleep, user or f"{os.getuid()}:{os.getgid()}", password)
+ except (UpgradeError, OSError, KeyboardInterrupt) as exc:
+ reason = "interrupted" if isinstance(exc, KeyboardInterrupt) else str(exc)
+ try:
+ outcome = recover(ctx, runner)
+ except (UpgradeError, OSError) as again:
+ report.emit(ctx, "db_upgrade", {"changed": False, "failed": reason, "steps": journal["steps"], "recovery": f"failed: {again}"})
+ return report.error(ctx, exit_codes.DB_UPGRADE_FAILED,
+ f"{reason}; recovery failed ({again}); the journal {journal_path(ctx)} is kept and `up` refuses until "
+ f"`grdm-dev db recover` succeeds")
+ outcome = outcome or f"nothing is pending any more; see `grdm-dev db status` for {journal['data_dir']}"
+ report.emit(ctx, "db_upgrade", {"changed": False, "failed": reason, "steps": journal["steps"], "recovery": outcome})
+ return report.error(ctx, exit_codes.DB_UPGRADE_FAILED, f"{reason}; {outcome}")
+ report.emit(ctx, "db_upgrade", {"changed": True, **{k: journal[k] for k in ("from", "to", "source_image", "image", "data_dir", "kept", "dump", "at", "steps")}})
+ sys.stderr.write(f"[grdm-dev] PostgreSQL {have} → {want} done; the old directory is {journal['kept']}; run `grdm-dev up`\n")
+ return exit_codes.OK
+
+
+def _upgrade_steps(ctx: Context, svc: dict[str, Any], journal: dict[str, Any], runner: Runner,
+ sleep: Callable[[float], None], user: str, password: str) -> None:
+ image, platform = journal["image"], journal.get("platform")
+ current, new_dir, kept = Path(journal["data_dir"]), Path(journal["new_dir"]), Path(journal["kept"])
+ backup = Path(journal["backup"])
+ old_name, new_name = journal["containers"]
+ socket_volume = journal["volume"]
+ env = {k: str(v) for k, v in (svc.get("environment") or {}).items() if k in ("POSTGRES_INITDB_ARGS", "PGDATA")}
+ command = [str(c) for c in (svc.get("command") or ["postgres"])]
+ tmpfs = [str(t).split(":", 1)[0] for t in (svc.get("tmpfs") or []) if not str(t).startswith(SOCKET_DIR)]
+
+ def step(name: str, **detail: Any) -> None:
+ journal["steps"].append({"step": name, "status": "ok", **detail})
+ sys.stderr.write(f"[grdm-dev] db upgrade: {name}\n")
+
+ backup.mkdir(mode=0o700)
+ os.chmod(backup, 0o700) # nosemgrep: python.lang.security.audit.insecure-file-permissions.insecure-file-permissions -- owner-only is the intended restriction
+ new_dir.mkdir(mode=0o755)
+ rc, _ = runner(["docker", "volume", "create", "--label", "org.gakunin-rdm.component=postgres-upgrade", socket_volume])
+ if rc != 0:
+ raise UpgradeError(f"docker volume create exited {rc}")
+ rc, _ = runner(server_argv(old_name, journal["source_image"], current, socket_volume, platform=platform,
+ env={"PGDATA": DATA_TARGET}, command=["postgres"], tmpfs=["/tmp"]))
+ if rc != 0:
+ raise UpgradeError(f"starting PostgreSQL {journal['from']} exited {rc}")
+ wait_ready(old_name, runner, sleep)
+ step("start-old", image=journal["source_image"])
+ before = snapshot(old_name, runner)
+ step("count-old", databases=sorted(before["databases"]))
+ rc, _ = runner(dump_argv(image, socket_volume, backup, platform, user))
+ dump = backup / "dumpall.sql"
+ if rc != 0 or not dump.is_file():
+ raise UpgradeError(f"pg_dumpall exited {rc}")
+ os.chmod(dump, 0o600) # it holds the roles' password hashes
+ step("dump", file=str(dump), bytes=dump.stat().st_size)
+ remove_container(old_name, runner)
+ step("stop-old")
+ restore = backup / "restore.sql"
+ write_restore_file(dump, restore)
+ new_env = {**env, "POSTGRES_HOST_AUTH_METHOD": _auth_method(env)}
+ # initdb needs a superuser password; the restore then sets the dumped verifier. It goes through a 0600 file in
+ # the 0700 backup directory, removed as soon as the container exists.
+ secret = backup / ".initdb.env"
+ fd = os.open(secret, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
+ with os.fdopen(fd, "w", encoding="utf-8") as fh:
+ fh.write(f"POSTGRES_PASSWORD={password}\n")
+ try:
+ rc, _ = runner(server_argv(new_name, image, new_dir, socket_volume, platform=platform, env=new_env,
+ command=command, tmpfs=tmpfs, backup=backup, env_file=secret))
+ finally:
+ secret.unlink(missing_ok=True)
+ if rc != 0:
+ raise UpgradeError(f"starting PostgreSQL {journal['to']} exited {rc}")
+ wait_ready(new_name, runner, sleep)
+ step("init-new", image=image, directory=str(new_dir))
+ rc, _ = runner(exec_argv(new_name, "psql", "-X", "-q", "-v", "ON_ERROR_STOP=1", "-h", SOCKET_DIR, "-U", "postgres",
+ "-d", "postgres", "-f", f"{BACKUP_TARGET}/restore.sql"))
+ if rc != 0:
+ raise UpgradeError(f"restore exited {rc}")
+ rc, _ = runner(exec_argv(new_name, "vacuumdb", "-q", "-h", SOCKET_DIR, "-U", "postgres", "--all", "--analyze-only"))
+ if rc != 0:
+ raise UpgradeError(f"vacuumdb --analyze-only exited {rc}")
+ restore.unlink(missing_ok=True) # derived from dumpall.sql, which stays
+ step("restore")
+ after = snapshot(new_name, runner)
+ diffs = snapshot_differences(before, after)
+ if diffs:
+ raise UpgradeError("restored data differs: " + "; ".join(diffs[:10]))
+ rows = sum(len(v) for v in after["databases"].values())
+ step("verify", objects_compared=rows, databases=sorted(after["databases"]), roles=len(after["roles"]))
+ remove_container(new_name, runner)
+ runner(["docker", "volume", "rm", socket_volume])
+ step("stop-new")
+ journal["phase"] = "swap"
+ write_durable(journal_path(ctx), journal) # from here on, recovery finishes the swap
+ _swap(current, new_dir, kept)
+ step("swap", kept=str(kept))
+ write_durable(journal_path(ctx), journal)
+ _write_record(journal)
+ _drop_journal(ctx)
+
+
+def run_rollback(ctx: Context, runner: Runner, stamp: str | None = None) -> int:
+ try:
+ with operation_lock(ctx):
+ return _rollback_locked(ctx, runner, stamp)
+ except UpgradeError as exc:
+ return report.error(ctx, exit_codes.DB_UPGRADE_FAILED, str(exc))
+ except OSError as exc:
+ return report.error(ctx, exit_codes.DB_UPGRADE_FAILED, _os_failure(ctx, exc))
+
+
+def _rollback_locked(ctx: Context, runner: Runner, stamp: str | None) -> int:
+ recovered = recover(ctx, runner)
+ if recovered:
+ sys.stderr.write(f"[grdm-dev] {recovered}\n")
+ found = latest_record(ctx)
+ if not found:
+ raise UpgradeError(f"no upgrade record under {ctx.runtime / BACKUPS_DIR}")
+ path, record = found
+ current = data_dir(ctx)
+ kept = current.with_name(Path(str(record.get("kept", ""))).name)
+ if not record.get("kept") or not kept.is_dir():
+ raise UpgradeError(f"the kept directory {kept} of {path} is gone")
+ bad = compose.volume_path_problems(ctx.root, ctx.runtime, [VOLUME, kept.name])
+ if bad:
+ raise UpgradeError("unsafe volume path: " + "; ".join(bad))
+ busy = _no_running_services(ctx, runner)
+ if busy:
+ raise UpgradeError(busy)
+ if not ctx.confirm(f"Put {kept} (PostgreSQL {record.get('from')}) back as {current}?", default=False):
+ if not ctx.yes and not ctx.stdin_is_tty:
+ return report.error(ctx, exit_codes.USAGE, "db rollback needs a confirmation; add --yes when no terminal is attached")
+ sys.stderr.write("[grdm-dev] db rollback cancelled\n")
+ return exit_codes.OK
+ ts = stamp or report.timestamp()
+ aside = current.with_name(f"{VOLUME}.pg{record.get('to')}-rolledback-{ts}")
+ journal = {"op": "rollback", "phase": "swap", "at": ts, "from": record.get("from"), "to": record.get("to"),
+ "image": record.get("image"), "data_dir": str(current), "kept": str(kept), "aside": str(aside),
+ "record": str(path), "containers": []}
+ write_durable(journal_path(ctx), journal)
+ try:
+ with signals_interrupt():
+ _swap(current, kept, aside)
+ except (OSError, KeyboardInterrupt) as exc:
+ outcome = recover(ctx, runner)
+ raise UpgradeError(f"rollback {'interrupted' if isinstance(exc, KeyboardInterrupt) else f'failed: {exc}'}; {outcome}") from None
+ _mark_rolled_back(journal)
+ _drop_journal(ctx)
+ report.emit(ctx, "db_rollback", {"data_dir": str(current), "restored_major": record.get("from"),
+ "set_aside": str(aside), "record": str(path), "recovered": recovered})
+ sys.stderr.write(f"[grdm-dev] {current} is PostgreSQL {record.get('from')} again; check out the revision that renders "
+ f"that major and run `grdm-dev compose render` before `up` (until then `up` stops with exit 15)\n")
+ return exit_codes.OK
diff --git a/installer/grdm_dev/exit_codes.py b/installer/grdm_dev/exit_codes.py
index d81ee3a..683fad1 100644
--- a/installer/grdm_dev/exit_codes.py
+++ b/installer/grdm_dev/exit_codes.py
@@ -11,6 +11,7 @@
PORT_IN_USE = 12
PREFLIGHT_FAILED = 13
RESET_FAILED = 14
+DB_UPGRADE_FAILED = 15
MANIFEST_INVALID = 20
BUILD_FAILED = 30
HEALTH_FAILED = 40
diff --git a/installer/grdm_dev/lifecycle.py b/installer/grdm_dev/lifecycle.py
index 9e24762..e7d97f7 100644
--- a/installer/grdm_dev/lifecycle.py
+++ b/installer/grdm_dev/lifecycle.py
@@ -1,7 +1,9 @@
"""grdm-dev up / down / reset — container lifecycle for the rendered profile (SPEC-02 §3, §7).
-up `docker compose up -d`, then wait for health (exit 40 on timeout), then open the browser at
- http://127.0.0.1: unless --no-open; the browser is only opened when stdin is a
+up finishes or rolls back an interrupted `db upgrade`/`db rollback`, then refuses (exit 15) a
+ runtime/volumes/postgres_data written by another PostgreSQL major than the rendered image, or one whose major
+ cannot be read (database.recover_pending, database.preflight, ADR-12); then `docker compose up -d`, then wait for health (exit 40 on timeout),
+ then open the browser at http://127.0.0.1: unless --no-open; the browser is only opened when stdin is a
terminal. Idempotent: healthy services are left alone by Compose.
down `docker compose down` — containers stop, volumes stay.
reset down, then delete runtime/volumes/ after ctx.confirm(). Non-interactive without --yes → exit 2.
@@ -22,7 +24,7 @@
import webbrowser
from pathlib import Path
-from . import compose, exit_codes, health, report
+from . import compose, database, exit_codes, health, report
from .context import Context
DEFAULT_TIMEOUT = 1800.0 # cold start: ember's first build under amd64 emulation took 13-25 min and osf_init runs migrations (real install runs 10-18; PR #33 review r1)
@@ -101,8 +103,16 @@ def run_up(ctx: Context, no_open: bool = False, timeout: float = DEFAULT_TIMEOUT
bad_paths = compose.volume_path_problems(ctx.root, ctx.runtime)
if bad_paths:
return report.error(ctx, exit_codes.PREFLIGHT_FAILED, "unsafe volume path: " + "; ".join(bad_paths))
- runner = runner or streaming_runner
- rc, out = runner(health.compose_base(ctx) + ["up", "-d", "--remove-orphans"])
+ # Under the lock of `grdm-dev db` (PR #42 review r2 R2-02), from the checks until Compose has started the
+ # containers: an interrupted `db upgrade`/`db rollback` is finished or rolled back first; then a server refuses a
+ # data directory of another major, or one whose major cannot be read (ADR-12).
+ checker = runner or health.default_runner
+ with database.lifecycle_lock(ctx) as held:
+ db_problem = held or database.recover_pending(ctx, checker) or database.preflight(ctx, checker)
+ if db_problem:
+ return report.error(ctx, exit_codes.DB_UPGRADE_FAILED, db_problem)
+ runner = runner or streaming_runner
+ rc, out = runner(health.compose_base(ctx) + ["up", "-d", "--remove-orphans"])
if rc != 0:
return report.error(ctx, exit_codes.HEALTH_FAILED, f"docker compose up exited {rc}")
kwargs = {"runner": runner, "prober": prober}
@@ -170,29 +180,38 @@ def refused() -> str | None:
bad = [f"{volumes_dir} changed during reset", *bad]
return "; ".join(bad) or None
- problem = refused()
- if problem:
- return report.error(ctx, exit_codes.RESET_FAILED, f"refusing to delete: {problem}")
- if health.compose_path(ctx).exists():
- rc = run_down(ctx, runner=runner, volumes=True)
- if rc != exit_codes.OK:
- return rc
- removed = False
- if volumes_dir.exists() or volumes_dir.is_symlink():
+ # Under the lock of `grdm-dev db` from the checks through the deletion (PR #42 review r2 R2-02): no upgrade or
+ # rollback can start on these directories while they are being deleted, and none is running now.
+ with database.lifecycle_lock(ctx) as held:
+ if held:
+ return report.error(ctx, exit_codes.DB_UPGRADE_FAILED, held)
problem = refused()
if problem:
return report.error(ctx, exit_codes.RESET_FAILED, f"refusing to delete: {problem}")
- owner_init = _owner_init_service(ctx)
- if owner_init.get("image"):
- # rc is not decisive: without the image, nothing was ever written by a container and the host can delete it all.
- (runner or streaming_runner)(wipe_argv(str(owner_init["image"]), volumes_dir, owner_init.get("platform")))
+ # a pending `grdm-dev db` operation may still own helper containers on these directories (ADR-12)
+ pending = database.recover_pending(ctx, runner or health.default_runner)
+ if pending:
+ return report.error(ctx, exit_codes.DB_UPGRADE_FAILED, pending)
+ if health.compose_path(ctx).exists():
+ rc = run_down(ctx, runner=runner, volumes=True)
+ if rc != exit_codes.OK:
+ return rc
+ removed = False
+ if volumes_dir.exists() or volumes_dir.is_symlink():
problem = refused()
if problem:
return report.error(ctx, exit_codes.RESET_FAILED, f"refusing to delete: {problem}")
- try:
- (rmtree or shutil.rmtree)(volumes_dir)
- except OSError as exc:
- return report.error(ctx, exit_codes.RESET_FAILED, f"could not delete {volumes_dir}: {exc}; remove it with `sudo rm -rf {volumes_dir}` / {volumes_dir} を削除できません。`sudo rm -rf {volumes_dir}` で消してください")
- removed = True
+ owner_init = _owner_init_service(ctx)
+ if owner_init.get("image"):
+ # rc is not decisive: without the image, nothing was ever written by a container and the host can delete it all.
+ (runner or streaming_runner)(wipe_argv(str(owner_init["image"]), volumes_dir, owner_init.get("platform")))
+ problem = refused()
+ if problem:
+ return report.error(ctx, exit_codes.RESET_FAILED, f"refusing to delete: {problem}")
+ try:
+ (rmtree or shutil.rmtree)(volumes_dir)
+ except OSError as exc:
+ return report.error(ctx, exit_codes.RESET_FAILED, f"could not delete {volumes_dir}: {exc}; remove it with `sudo rm -rf {volumes_dir}` / {volumes_dir} を削除できません。`sudo rm -rf {volumes_dir}` で消してください")
+ removed = True
report.emit(ctx, "reset", {"volumes_dir": str(volumes_dir), "removed": removed})
return exit_codes.OK
diff --git a/installer/grdm_dev/messages.tsv b/installer/grdm_dev/messages.tsv
index c560f5c..6cf8216 100644
--- a/installer/grdm_dev/messages.tsv
+++ b/installer/grdm_dev/messages.tsv
@@ -6,6 +6,7 @@ code en ja
12 a required port on 127.0.0.1 is in use and no free alternative was found 127.0.0.1 の必要なポートが使用中で、空きの代替がない
13 install prerequisite missing: git, repository access, amd64 emulation, or a checkout path Docker cannot use 導入の前提が不足: git、リポジトリの読み取り権限、amd64 エミュレーション、または Docker が使えない配置先
14 reset could not delete runtime/volumes/ reset が runtime/volumes/ を削除できない
+15 PostgreSQL data directory needs `grdm-dev db upgrade`, or the upgrade or rollback failed (the old directory is kept) PostgreSQL のデータディレクトリに `grdm-dev db upgrade` が要る、または更新か巻き戻しに失敗(旧ディレクトリは残る)
20 manifest invalid or TOON excerpt disagrees with JSON manifest が不正、または TOON 抄録が JSON と一致しない
30 image build failed イメージのビルドに失敗
40 health check failed, or a port listens on a non-loopback address 健康確認に失敗、またはループバック以外のアドレスでポートが待ち受けている
diff --git a/installer/grdm_dev/templates/postgres.json b/installer/grdm_dev/templates/postgres.json
index 957637d..0ba3f05 100644
--- a/installer/grdm_dev/templates/postgres.json
+++ b/installer/grdm_dev/templates/postgres.json
@@ -8,7 +8,8 @@
"POSTGRES_DB": "osf",
"POSTGRES_USER": "postgres",
"POSTGRES_PASSWORD": "${POSTGRES_PASSWORD}",
- "POSTGRES_INITDB_ARGS": "--auth-host=md5 --auth-local=trust"
+ "POSTGRES_INITDB_ARGS": "--auth-host=md5 --auth-local=trust",
+ "PGDATA": "/var/lib/postgresql/data"
},
"volumes": [
{
@@ -23,7 +24,8 @@
],
"tmpfs": [
"/tmp",
- "/var/run/postgresql"
+ "/var/run/postgresql",
+ "/var/lib/postgresql"
],
"weight": 3,
"mem_min_mb": 512,
@@ -33,5 +35,5 @@
"-c",
"max_connections=500"
],
- "note": "max_connections=500 (default 100): osf.io's development server leaks one connection per request that ends in an unhandled error (the Django teardown that closes connections is skipped), and the health probe plus a browser exhausted 100 within minutes in real install run 12. The limit is a dev-environment safety margin, not a fix for the leak."
+ "note": "max_connections=500 (default 100): osf.io's development server leaks one connection per request that ends in an unhandled error (the Django teardown that closes connections is skipped), and the health probe plus a browser exhausted 100 within minutes in real install run 12. The limit is a dev-environment safety margin, not a fix for the leak. PostgreSQL 18 (ADR-12): the image moved PGDATA to /var/lib/postgresql/18/docker and declares a VOLUME at /var/lib/postgresql; PGDATA is set back to the mount point of postgres_data, and a tmpfs on /var/lib/postgresql keeps Compose from creating an anonymous volume there. --auth-host=md5 stays on purpose: initdb then also sets password_encryption=md5, and CAS's JDBC driver 42.1.4 cannot answer SCRAM. A data directory of another major is moved with `grdm-dev db upgrade`; `up` refuses it (exit 15)."
}
diff --git a/installer/tests/test_compose.py b/installer/tests/test_compose.py
index c19fbb6..79ae160 100644
--- a/installer/tests/test_compose.py
+++ b/installer/tests/test_compose.py
@@ -1,3 +1,5 @@
+from __future__ import annotations
+
import argparse
import ast
import io
diff --git a/installer/tests/test_database.py b/installer/tests/test_database.py
new file mode 100644
index 0000000..bf46a72
--- /dev/null
+++ b/installer/tests/test_database.py
@@ -0,0 +1,883 @@
+import argparse
+import errno
+import io
+import json
+import os
+import tempfile
+import unittest
+from collections import namedtuple
+from contextlib import redirect_stderr, redirect_stdout
+from pathlib import Path
+from unittest import mock
+
+from grdm_dev import database, exit_codes, lifecycle
+from grdm_dev.context import Context
+
+TARGET = "postgres:18.6@sha256:e3d24f6906d0760bf4da0dae872415391ededefad23fd471c47dc53b0ddf3c42"
+TS = "20261110T000000Z"
+OLD, NEW = f"grdm-dev-dbupgrade-old-{TS}", f"grdm-dev-dbupgrade-new-{TS}"
+COMPOSE = {"services": {"postgres": {
+ "image": TARGET,
+ "platform": "linux/arm64",
+ "command": ["postgres", "-c", "max_connections=500"],
+ "environment": {"POSTGRES_DB": "osf", "POSTGRES_PASSWORD": "${POSTGRES_PASSWORD}", "PGDATA": "/var/lib/postgresql/data",
+ "POSTGRES_INITDB_ARGS": "--auth-host=md5 --auth-local=trust"},
+ "tmpfs": ["/tmp:mode=1777", "/var/run/postgresql:mode=1777", "/var/lib/postgresql:mode=1777"],
+}, "web": {"image": "grdm-dev/osf.io:abc"}}}
+ROWS = {"osf": ["public.osf_osfuser r 3", "public.osf_osfuser_id_seq S 3 true", "extension plpgsql"],
+ "cas": ["public.ticket r 0"], "postgres": ["extension plpgsql"]}
+ROLES = ["postgres true true true true true true true -1 0123abcd"]
+DUMP = "--\nSET default_transaction_read_only = off;\nCREATE ROLE postgres;\nALTER ROLE postgres WITH SUPERUSER PASSWORD 'md5x';\nCREATE DATABASE osf;\n"
+STEPS = ("start-old", "dump", "start-new", "restore", "verify", "stop-new")
+Usage = namedtuple("Usage", "total used free")
+
+
+class Crash(BaseException):
+ """A kill the process cannot catch (SIGKILL, power loss): nothing after it runs in this process."""
+
+
+class FakeDocker:
+ """`runner(argv) -> (rc, stdout)` for the docker calls of `db`: containers exist until removed, the new server's
+ initdb writes PG_VERSION, and `fail` makes one step return an error, raise KeyboardInterrupt or crash."""
+
+ def __init__(self, running=(), after=None, roles_after=None, dump=DUMP, fail=None, mode="rc", sticky=()):
+ self.calls, self.running, self.dump = [], list(running), dump
+ self.fail, self.mode, self.sticky = fail, mode, set(sticky)
+ self.present: set[str] = set()
+ self.rows = {"old": ROWS, "new": after if after is not None else ROWS}
+ self.roles = {"old": ROLES, "new": roles_after if roles_after is not None else ROLES}
+
+ def hit(self, label):
+ if self.fail != label:
+ return False
+ self.fail = None # an interrupt, a crash or an error happens once
+ if self.mode == "interrupt":
+ raise KeyboardInterrupt
+ if self.mode == "crash":
+ raise Crash(label)
+ return True
+
+ def __call__(self, argv):
+ self.calls.append(argv)
+ if argv[:2] == ["docker", "compose"] and "ps" in argv:
+ return 0, json.dumps([{"Service": s, "State": "running"} for s in self.running])
+ if argv[:3] == ["docker", "ps", "-a"]:
+ name = argv[argv.index("--filter") + 1][len("name=^/"):-1]
+ return 0, name + "\n" if name in self.present else ""
+ if argv[:3] == ["docker", "run", "-d"]:
+ name = argv[argv.index("--name") + 1]
+ if self.hit("start-old" if "-old-" in name else "start-new"):
+ return 1, ""
+ self.present.add(name)
+ if TARGET in argv: # the entrypoint's initdb
+ data = next(v.split(":")[0] for v in argv if v.endswith(":/var/lib/postgresql/data"))
+ Path(data, "PG_VERSION").write_text("18\n")
+ return 0, "id\n"
+ if argv[:3] == ["docker", "run", "--rm"] and "pg_dumpall" in argv:
+ if self.hit("dump"):
+ return 1, ""
+ backup = next(v.split(":")[0] for v in argv if v.endswith(":/backup"))
+ Path(backup, "dumpall.sql").write_text(self.dump, encoding="utf-8")
+ return 0, ""
+ if argv[:2] == ["docker", "inspect"]:
+ return 0, "true\n" if argv[-1] in self.present else "false\n"
+ if argv[:2] == ["docker", "exec"]:
+ side = "old" if "-old-" in argv[4] else "new"
+ if "-f" in argv:
+ return (3 if self.hit("restore") else 0), ""
+ if "psql" in argv and "-c" in argv:
+ sql, db = argv[-1], argv[argv.index("-d") + 1]
+ if sql == database.DATABASES_SQL:
+ if side == "new" and self.hit("verify"):
+ return 1, ""
+ return 0, "\n".join(sorted(self.rows[side])) + "\n"
+ if sql == database.ROLES_SQL:
+ return 0, "\n".join(self.roles[side]) + "\n"
+ return 0, "\n".join(self.rows[side][db]) + "\n"
+ return 0, ""
+ if argv[:2] == ["docker", "stop"]:
+ if self.hit("stop-" + ("old" if "-old-" in argv[-1] else "new")):
+ return 1, ""
+ return 0, ""
+ if argv[:2] == ["docker", "rm"]:
+ if argv[-1] in self.sticky:
+ return 1, ""
+ self.present.discard(argv[-1])
+ return 0, ""
+ return 0, ""
+
+ def runs(self):
+ return [c for c in self.calls if c[:3] == ["docker", "run", "-d"]]
+
+
+def roomy(path):
+ return Usage(10 ** 12, 0, 10 ** 12)
+
+
+class DatabaseTests(unittest.TestCase):
+ def setUp(self):
+ self.tmp = tempfile.TemporaryDirectory()
+ self.root = Path(self.tmp.name)
+ self.runtime = self.root / "runtime"
+ self.runtime.mkdir()
+ (self.runtime / "docker-compose.yml").write_text(json.dumps(COMPOSE))
+ (self.runtime / ".env").write_text("POSTGRES_PASSWORD=s3cret-value\n")
+ self.data = self.runtime / "volumes" / "postgres_data"
+ self.data.mkdir(parents=True)
+ (self.data / "PG_VERSION").write_text("9.6\n")
+ self.kept = self.data.with_name(f"postgres_data.pg9.6-{TS}")
+ self.new = self.data.with_name(f"postgres_data.pg18-{TS}.new")
+ self.backup = self.runtime / "backups" / f"postgres-9.6-to-18-{TS}"
+ self.journal = self.runtime / database.JOURNAL
+ self.out, self.err = io.StringIO(), io.StringIO()
+
+ def fresh(self):
+ self.tearDown()
+ self.setUp()
+
+ def tearDown(self):
+ for path in self.root.rglob("*"):
+ try:
+ path.chmod(0o755)
+ except OSError:
+ pass
+ self.tmp.cleanup()
+
+ def run_db(self, action, runner, yes=True, disk_usage=roomy, stamp=TS):
+ ctx = Context(root=self.root, yes=yes, stdin_is_tty=False)
+ with redirect_stdout(self.out), redirect_stderr(self.err):
+ return database.run(ctx, argparse.Namespace(db_command=action), runner=runner, sleep=lambda s: None,
+ user="501:20", stamp=stamp, disk_usage=disk_usage)
+
+ def run_up(self, runner):
+ ctx = Context(root=self.root, stdin_is_tty=False)
+ with redirect_stdout(self.out), redirect_stderr(self.err):
+ return lifecycle.run(ctx, argparse.Namespace(lifecycle_action="up", no_open=True, timeout=1.0), runner=runner,
+ prober=lambda url: (True, "HTTP 200"), sleep=lambda s: None)
+
+ def major(self, path=None):
+ return ((path or self.data) / "PG_VERSION").read_text().strip()
+
+ def assert_old_intact(self, runner):
+ self.assertEqual(self.major(), "9.6")
+ self.assertFalse(self.kept.exists())
+ self.assertFalse(self.journal.exists())
+ self.assertEqual(runner.present, set()) # every migration server proved gone
+
+ # ------------------------------------------------------------------ facts
+
+ def test_image_major_reads_one_part_from_10_and_two_before(self):
+ self.assertEqual(database.image_major(TARGET), "18")
+ self.assertEqual(database.image_major("postgres:9.6"), "9.6")
+ self.assertEqual(database.image_major("postgres:15-alpine"), "15")
+ self.assertEqual(database.image_major(database.SOURCE_IMAGES["9.6"]), "9.6")
+ self.assertIsNone(database.image_major("postgres"))
+ self.assertIsNone(database.image_major("postgres:latest"))
+ self.assertIsNone(database.image_major("localhost:5000/postgres"))
+
+ def test_status_reports_upgrade_needed_without_changing_anything(self):
+ runner = FakeDocker()
+ self.assertEqual(self.run_db("status", runner), exit_codes.OK)
+ self.assertIn("upgrade-needed", self.out.getvalue())
+ self.assertEqual(runner.calls, [])
+ self.assertEqual(self.major(), "9.6")
+
+ def test_data_major_distinguishes_empty_from_unknown(self):
+ self.assertEqual(database.data_major(self.data), "9.6")
+ (self.data / "PG_VERSION").write_text("garbage\n")
+ self.assertIsNone(database.data_major(self.data))
+ (self.data / "PG_VERSION").unlink()
+ (self.data / "base").mkdir()
+ self.assertIsNone(database.data_major(self.data)) # a non-empty directory without PG_VERSION is not "no cluster"
+ (self.data / "base").rmdir()
+ self.assertEqual(database.data_major(self.data), "")
+ self.assertEqual(database.data_major(self.root / "absent"), "")
+
+ def _unreadable(self):
+ self.data.chmod(0o000)
+ try:
+ os.listdir(self.data)
+ except PermissionError:
+ return
+ self.skipTest("permissions are not enforced for this user")
+
+ def test_unreadable_pg_version_is_read_by_the_postgres_uid_in_a_container(self):
+ calls = []
+
+ def runner(argv):
+ calls.append(argv)
+ return 0, "9.6\n"
+
+ self._unreadable()
+ self.assertEqual(database.data_major(self.data, runner, TARGET, "linux/arm64"), "9.6")
+ argv = calls[0]
+ self.assertEqual(argv[argv.index("--user") + 1], "999:999")
+ self.assertIn("--read-only", argv)
+ self.assertEqual(argv[argv.index("--network") + 1], "none")
+ self.assertIn(f"{self.data}:/data:ro", argv)
+
+ def test_up_refuses_when_the_container_read_fails(self):
+ calls = []
+
+ def runner(argv):
+ calls.append(argv)
+ return 125, "" # the helper container cannot start
+
+ self._unreadable()
+ self.assertEqual(self.run_up(runner), exit_codes.DB_UPGRADE_FAILED)
+ self.assertIn("cannot read the PostgreSQL major", self.err.getvalue())
+ self.assertFalse([c for c in calls if c[:2] == ["docker", "compose"]])
+
+ def test_up_refuses_an_unparseable_or_clusterless_directory(self):
+ (self.data / "PG_VERSION").write_text("nonsense\n")
+ self.assertEqual(self.run_up(FakeDocker()), exit_codes.DB_UPGRADE_FAILED)
+ (self.data / "PG_VERSION").unlink()
+ (self.data / "global").mkdir()
+ self.assertEqual(self.run_up(FakeDocker()), exit_codes.DB_UPGRADE_FAILED)
+ self.assertEqual(self.err.getvalue().count("cannot read the PostgreSQL major"), 2)
+
+ def test_up_stops_before_compose_on_a_data_directory_of_another_major(self):
+ runner = FakeDocker()
+ self.assertEqual(self.run_up(runner), exit_codes.DB_UPGRADE_FAILED)
+ self.assertIn("grdm-dev db upgrade", self.err.getvalue())
+ self.assertFalse([c for c in runner.calls if c[:2] == ["docker", "compose"]])
+
+ def test_preflight_accepts_matching_and_empty_directories(self):
+ ctx = Context(root=self.root, stdin_is_tty=False)
+ (self.data / "PG_VERSION").write_text("18\n")
+ self.assertIsNone(database.preflight(ctx))
+ (self.data / "PG_VERSION").unlink()
+ self.assertIsNone(database.preflight(ctx))
+
+ # ------------------------------------------------------------------ verification scope (R1-05)
+
+ def test_snapshot_keeps_a_database_named_roles_apart_from_the_roles(self):
+ rows = {**ROWS, "roles": ["public.t r 7"]}
+ runner = FakeDocker()
+ runner.rows["old"] = rows
+ snap = database.snapshot(OLD, runner)
+ self.assertEqual(snap["databases"]["roles"], ["public.t r 7"])
+ self.assertEqual(snap["roles"], ROLES)
+ changed = {**rows, "roles": ["public.t r 6"]}
+ diffs = database.snapshot_differences(snap, {"databases": changed, "roles": ROLES})
+ self.assertEqual(diffs, ["database roles: before `public.t r 7`", "database roles: after `public.t r 6`"])
+
+ def test_a_sequence_with_another_is_called_is_a_difference(self):
+ after = {**ROWS, "osf": ["public.osf_osfuser r 3", "public.osf_osfuser_id_seq S 3 false", "extension plpgsql"]}
+ runner = FakeDocker(after=after)
+ self.assertEqual(self.run_db("upgrade", runner), exit_codes.DB_UPGRADE_FAILED)
+ self.assertIn("osf_osfuser_id_seq S 3 false", self.err.getvalue())
+ self.assert_old_intact(runner)
+ self.assertIn("is_called", database.SNAPSHOT_SQL)
+
+ def test_a_changed_role_attribute_is_a_difference_without_printing_the_verifier(self):
+ runner = FakeDocker(roles_after=["postgres true true true true true true true -1 ffff0000"])
+ self.assertEqual(self.run_db("upgrade", runner), exit_codes.DB_UPGRADE_FAILED)
+ self.assertIn("role before `postgres …`", self.err.getvalue())
+ self.assertNotIn("0123abcd", self.err.getvalue() + self.out.getvalue())
+ self.assert_old_intact(runner)
+
+ # ------------------------------------------------------------------ upgrade
+
+ def test_upgrade_dumps_restores_verifies_and_keeps_the_old_directory(self):
+ runner = FakeDocker()
+ self.assertEqual(self.run_db("upgrade", runner), exit_codes.OK, self.err.getvalue())
+ self.assertEqual(self.major(self.kept), "9.6")
+ self.assertEqual(self.major(), "18")
+ self.assertFalse(self.journal.exists())
+ self.assertEqual(runner.present, set())
+ self.assertEqual(self.backup.stat().st_mode & 0o777, 0o700)
+ self.assertEqual((self.backup / "dumpall.sql").stat().st_mode & 0o777, 0o600)
+ self.assertFalse((self.backup / "restore.sql").exists()) # derived copy removed after the restore
+ record = json.loads((self.backup / "upgrade.json").read_text())
+ self.assertEqual((record["from"], record["to"], record["kept"]), ("9.6", "18", str(self.kept)))
+ old, new = runner.runs()
+ self.assertIn(database.SOURCE_IMAGES["9.6"], old)
+ self.assertIn(TARGET, new)
+ for argv in (old, new):
+ self.assertEqual(argv[argv.index("--network") + 1], "none")
+ self.assertEqual(argv[-4:], ["-c", "listen_addresses=", "-c", "log_min_error_statement=panic"])
+ self.assertEqual([argv[i + 1] for i, a in enumerate(argv) if a == "--cap-add"], list(database.SERVER_CAPS))
+ self.assertIn("--read-only", argv)
+ self.assertNotIn("s3cret-value", " ".join(argv))
+ secret = Path(new[new.index("--env-file") + 1])
+ self.assertEqual(secret.parent, self.backup)
+ self.assertFalse(secret.exists())
+ self.assertIn("POSTGRES_HOST_AUTH_METHOD=md5", new)
+ self.assertIn("max_connections=500", new)
+ dump = next(c for c in runner.calls if "pg_dumpall" in c)
+ self.assertIn(TARGET, dump)
+ self.assertEqual(dump[dump.index("--user") + 1], "501:20")
+ # the old server is proved gone before the new one starts
+ first_new = runner.calls.index(new)
+ self.assertIn(["docker", "rm", "-f", OLD], runner.calls[:first_new])
+
+ def test_upgrade_refuses_while_services_run(self):
+ runner = FakeDocker(running=["postgres", "web"])
+ self.assertEqual(self.run_db("upgrade", runner), exit_codes.DB_UPGRADE_FAILED)
+ self.assertIn("grdm-dev down", self.err.getvalue())
+ self.assertEqual(runner.runs(), [])
+
+ def test_upgrade_needs_the_database_password_and_a_confirmation(self):
+ (self.runtime / ".env").write_text("OTHER=1\n")
+ self.assertEqual(self.run_db("upgrade", FakeDocker()), exit_codes.USAGE)
+ (self.runtime / ".env").write_text("POSTGRES_PASSWORD=x\n")
+ runner = FakeDocker()
+ self.assertEqual(self.run_db("upgrade", runner, yes=False), exit_codes.USAGE)
+ self.assertEqual(runner.runs(), [])
+ self.assertFalse(self.journal.exists())
+
+ def test_upgrade_refuses_an_unknown_source_major_and_is_a_no_op_when_current(self):
+ (self.data / "PG_VERSION").write_text("12\n")
+ self.assertEqual(self.run_db("upgrade", FakeDocker()), exit_codes.DB_UPGRADE_FAILED)
+ self.assertIn("no source image for PostgreSQL 12", self.err.getvalue())
+ (self.data / "PG_VERSION").write_text("18\n")
+ self.assertEqual(self.run_db("upgrade", FakeDocker()), exit_codes.OK)
+ self.assertIn("already current", self.out.getvalue())
+
+ def test_upgrade_refuses_a_dump_without_exactly_one_bootstrap_role(self):
+ runner = FakeDocker(dump="CREATE DATABASE osf;\n")
+ self.assertEqual(self.run_db("upgrade", runner), exit_codes.DB_UPGRADE_FAILED)
+ self.assertIn("CREATE ROLE postgres;", self.err.getvalue())
+ self.assert_old_intact(runner)
+
+ # ------------------------------------------------------------------ failures, interruptions, crashes (R1-01, R1-02)
+
+ def test_a_failure_at_each_step_leaves_the_old_cluster_and_no_server(self):
+ for step in STEPS:
+ with self.subTest(step=step):
+ self.fresh()
+ runner = FakeDocker(fail=step, mode="rc")
+ code = self.run_db("upgrade", runner)
+ if step == "stop-new": # a failed `docker stop` is fine once removal is proved
+ self.assertEqual(code, exit_codes.OK, self.err.getvalue())
+ continue
+ self.assertEqual(code, exit_codes.DB_UPGRADE_FAILED)
+ self.assertEqual(self.major(), "9.6")
+ self.assertFalse(self.journal.exists())
+ self.assertEqual(runner.present, set())
+
+ def test_an_interrupt_at_each_step_is_cleaned_up_in_process(self):
+ for step in STEPS:
+ with self.subTest(step=step):
+ self.fresh()
+ runner = FakeDocker(fail=step, mode="interrupt")
+ self.assertEqual(self.run_db("upgrade", runner), exit_codes.DB_UPGRADE_FAILED)
+ self.assertIn("interrupted", self.err.getvalue())
+ self.assert_old_intact(runner)
+
+ def test_sigterm_is_handled_like_an_interrupt(self):
+ def term(argv):
+ if "pg_dumpall" in argv:
+ os.kill(os.getpid(), database.signal.SIGTERM)
+ return runner_inner(argv)
+
+ runner_inner = FakeDocker()
+ self.assertEqual(self.run_db("upgrade", term), exit_codes.DB_UPGRADE_FAILED)
+ self.assertIn("interrupted", self.err.getvalue())
+ self.assert_old_intact(runner_inner)
+
+ def test_a_crash_at_each_step_is_recovered_by_the_next_up_or_upgrade(self):
+ for i, step in enumerate(STEPS):
+ with self.subTest(step=step):
+ self.fresh()
+ runner = FakeDocker(fail=step, mode="crash")
+ with self.assertRaises(Crash):
+ self.run_db("upgrade", runner)
+ self.assertTrue(self.journal.exists()) # ownership survives the crash
+ runner.fail = None
+ if i % 2:
+ self.assertEqual(self.run_up(runner), exit_codes.DB_UPGRADE_FAILED) # recovered, then 9.6 ≠ 18
+ self.assertIn("rolled back an interrupted upgrade", self.err.getvalue())
+ self.assert_old_intact(runner)
+ else:
+ self.assertEqual(self.run_db("upgrade", runner, stamp=TS + "b"), exit_codes.OK, self.err.getvalue())
+ self.assertEqual(self.major(), "18")
+ self.assertEqual(runner.present, set())
+
+ def test_a_crash_between_the_renames_is_finished_by_recover_even_after_render(self):
+ real = os.rename
+ calls = []
+
+ def rename(a, b):
+ calls.append((a, b))
+ if len(calls) == 2:
+ raise Crash("between renames")
+ real(a, b)
+
+ runner = FakeDocker()
+ with mock.patch.object(database.os, "rename", side_effect=rename), self.assertRaises(Crash):
+ self.run_db("upgrade", runner)
+ self.assertFalse(self.data.exists())
+ self.data.mkdir() # what `compose render` does to a missing volume directory
+ self.assertEqual(self.run_up(runner), exit_codes.HEALTH_FAILED) # recovered, versions match, compose is fake
+ self.assertIn("finished an interrupted upgrade", self.err.getvalue())
+ self.assertEqual(self.major(), "18")
+ self.assertEqual(self.major(self.kept), "9.6")
+ self.assertTrue((self.backup / "upgrade.json").exists())
+ self.assertFalse(self.journal.exists())
+
+ def test_a_failed_second_rename_puts_the_old_cluster_back(self):
+ real = os.rename
+
+ def rename(a, b):
+ if Path(a) == self.new:
+ raise OSError(errno.EIO, "injected")
+ real(a, b)
+
+ runner = FakeDocker()
+ with mock.patch.object(database.os, "rename", side_effect=rename):
+ self.assertEqual(self.run_db("upgrade", runner), exit_codes.DB_UPGRADE_FAILED)
+ self.assertIn("swap could not be done", self.err.getvalue())
+ self.assert_old_intact(runner)
+ self.assertTrue(self.new.exists())
+
+ def test_a_crash_after_the_swap_before_the_record_is_finished(self):
+ real = database._write_record
+ runner = FakeDocker()
+ with mock.patch.object(database, "_write_record", side_effect=Crash("record")), self.assertRaises(Crash):
+ self.run_db("upgrade", runner)
+ self.assertEqual(self.major(), "18")
+ self.assertTrue(self.journal.exists())
+ self.assertEqual(self.run_db("recover", runner), exit_codes.OK)
+ self.assertTrue((self.backup / "upgrade.json").exists())
+ self.assertFalse(self.journal.exists())
+ self.assertIs(database._write_record, real)
+
+ def test_a_server_that_cannot_be_removed_keeps_the_journal_and_blocks_up(self):
+ runner = FakeDocker(sticky={OLD})
+ self.assertEqual(self.run_db("upgrade", runner), exit_codes.DB_UPGRADE_FAILED)
+ self.assertIn(f"docker rm -f {OLD}", self.err.getvalue())
+ self.assertTrue(self.journal.exists())
+ self.assertEqual(self.major(), "9.6")
+ self.assertEqual(self.run_up(runner), exit_codes.DB_UPGRADE_FAILED)
+ self.assertIn("could not be recovered", self.err.getvalue())
+ self.assertFalse([c for c in runner.calls if c[:2] == ["docker", "compose"] and "up" in c])
+ runner.sticky.clear()
+ self.assertEqual(self.run_db("recover", runner), exit_codes.OK)
+ self.assert_old_intact(runner)
+
+ def test_an_unanswering_docker_is_not_taken_for_an_absent_server(self):
+ runner = FakeDocker(fail="start-new", mode="crash")
+ with self.assertRaises(Crash):
+ self.run_db("upgrade", runner)
+ self.assertEqual(self.run_db("recover", lambda argv: (1, "")), exit_codes.DB_UPGRADE_FAILED)
+ self.assertTrue(self.journal.exists())
+
+ def test_a_second_operation_waits_for_the_lock(self):
+ ctx = Context(root=self.root, yes=True, stdin_is_tty=False)
+ runner = FakeDocker()
+ with database.operation_lock(ctx):
+ pid = os.fork()
+ if pid == 0: # a separate process, as a second terminal would be
+ with redirect_stdout(io.StringIO()), redirect_stderr(io.StringIO()):
+ code = database.run(ctx, argparse.Namespace(db_command="upgrade"), runner=runner,
+ sleep=lambda s: None, user="501:20", stamp=TS, disk_usage=roomy)
+ os._exit(code)
+ _, status = os.waitpid(pid, 0)
+ self.assertEqual(os.WEXITSTATUS(status), exit_codes.DB_UPGRADE_FAILED)
+ self.assertEqual(self.major(), "9.6")
+
+ # ------------------------------------------------------------------ disk (R1-03)
+
+ def test_disk_is_checked_before_any_server_starts(self):
+ runner = FakeDocker()
+ tight = lambda path: Usage(10 ** 9, 10 ** 9 - 100 * database.MIB, 100 * database.MIB) # noqa: E731
+ self.assertEqual(self.run_db("upgrade", runner, disk_usage=tight), exit_codes.DB_UPGRADE_FAILED)
+ self.assertIn("not enough free disk", self.err.getvalue())
+ self.assertEqual(runner.runs(), [])
+ self.assertFalse(self.journal.exists())
+
+ def test_disk_estimate_adds_up_on_one_filesystem_and_splits_on_two(self):
+ size = 100 * database.MIB
+ need_b, need_v = 2 * size + database.BACKUP_MARGIN, 2 * size + database.VOLUME_MARGIN
+ free = lambda n: (lambda path: Usage(0, 0, n)) # noqa: E731
+ backups, volumes = self.runtime, self.data.parent
+ self.assertEqual(database.disk_problems(size, backups, volumes, free(need_b + need_v)), [])
+ self.assertTrue(database.disk_problems(size, backups, volumes, free(need_b + need_v - 1)))
+ other = mock.Mock(st_dev=1), mock.Mock(st_dev=2)
+ with mock.patch.object(Path, "stat", side_effect=[other[0], other[1]]):
+ self.assertEqual(database.disk_problems(size, backups, volumes, free(max(need_b, need_v))), [])
+
+ def test_enospc_in_the_dump_the_filter_and_the_restore_leaves_the_old_cluster(self):
+ cases = {
+ "dump": lambda: FakeDocker(fail="dump"),
+ "restore": lambda: FakeDocker(fail="restore"),
+ }
+ for name, make in cases.items():
+ with self.subTest(step=name):
+ self.fresh()
+ runner = make()
+ self.assertEqual(self.run_db("upgrade", runner), exit_codes.DB_UPGRADE_FAILED)
+ self.assert_old_intact(runner)
+ self.fresh()
+ runner = FakeDocker()
+ with mock.patch.object(database, "write_restore_file", side_effect=OSError(errno.ENOSPC, "No space left on device")):
+ self.assertEqual(self.run_db("upgrade", runner), exit_codes.DB_UPGRADE_FAILED)
+ self.assertIn("No space left", self.err.getvalue())
+ self.assert_old_intact(runner)
+
+ def test_enospc_in_each_journal_write(self):
+ real = database.write_durable
+ for nth in (1, 2, 3, 4): # prepare, swap mark, after swap, record
+ with self.subTest(write=nth):
+ self.fresh()
+ count = []
+
+ def write(path, data, nth=nth):
+ count.append(path)
+ if len(count) == nth:
+ raise OSError(errno.ENOSPC, "No space left on device")
+ real(path, data)
+
+ runner = FakeDocker()
+ with mock.patch.object(database, "write_durable", side_effect=write):
+ code = self.run_db("upgrade", runner)
+ self.assertEqual(code, exit_codes.DB_UPGRADE_FAILED)
+ self.assertEqual(runner.present, set())
+ if nth == 1:
+ self.assertEqual(runner.runs(), []) # nothing started without a journal
+ self.assertEqual(self.major(), "9.6")
+ elif nth == 2:
+ self.assert_old_intact(runner) # the swap mark is written before any rename
+ else:
+ self.assertEqual(self.major(), "18") # swapped; the record write is retried by recovery
+ self.assertFalse(self.journal.exists())
+ self.assertTrue((self.backup / "upgrade.json").exists())
+
+ # ------------------------------------------------------------------ rollback
+
+ def _upgraded(self):
+ self.assertEqual(self.run_db("upgrade", FakeDocker()), exit_codes.OK, self.err.getvalue())
+
+ def test_rollback_puts_the_kept_directory_back_and_marks_the_record(self):
+ self._upgraded()
+ self.assertEqual(self.run_db("rollback", FakeDocker(), stamp=TS + "r"), exit_codes.OK, self.err.getvalue())
+ self.assertEqual(self.major(), "9.6")
+ self.assertTrue(json.loads((self.backup / "upgrade.json").read_text())["rolled_back_at"])
+ self.assertEqual(self.run_db("rollback", FakeDocker()), exit_codes.DB_UPGRADE_FAILED) # nothing left to roll back
+ self.assertIn("no upgrade record", self.err.getvalue())
+
+ def test_a_crash_between_the_rollback_renames_is_finished(self):
+ self._upgraded()
+ real = os.rename
+ calls = []
+
+ def rename(a, b):
+ calls.append(a)
+ if len(calls) == 2:
+ raise Crash("between renames")
+ real(a, b)
+
+ with mock.patch.object(database.os, "rename", side_effect=rename), self.assertRaises(Crash):
+ self.run_db("rollback", FakeDocker(), stamp=TS + "r")
+ self.assertFalse(self.data.exists())
+ self.assertEqual(self.run_db("recover", FakeDocker()), exit_codes.OK)
+ self.assertEqual(self.major(), "9.6")
+ self.assertFalse(self.journal.exists())
+
+ def test_a_failed_second_rollback_rename_is_compensated(self):
+ self._upgraded()
+ real = os.rename
+
+ def rename(a, b):
+ if Path(a) == self.kept:
+ raise OSError(errno.EIO, "injected")
+ real(a, b)
+
+ with mock.patch.object(database.os, "rename", side_effect=rename):
+ self.assertEqual(self.run_db("rollback", FakeDocker(), stamp=TS + "r"), exit_codes.DB_UPGRADE_FAILED)
+ self.assertEqual(self.major(), "18")
+ self.assertFalse(self.journal.exists())
+
+ def test_rollback_refuses_while_services_run(self):
+ self._upgraded()
+ self.assertEqual(self.run_db("rollback", FakeDocker(running=["postgres"])), exit_codes.DB_UPGRADE_FAILED)
+ self.assertEqual(self.major(), "18")
+
+ # ------------------------------------------------------------------ review r2
+
+ def _crash_before_swap(self, action):
+ runner = FakeDocker()
+ if action == "rollback":
+ self._upgraded()
+ with mock.patch.object(database, "_swap", side_effect=Crash("before swap")), self.assertRaises(Crash):
+ self.run_db(action, runner, stamp=TS + "r" if action == "rollback" else TS)
+ self.assertTrue(self.journal.exists())
+ return runner
+
+ def _renames_failing_after(self, ok):
+ real, count = os.rename, [0]
+
+ def rename(a, b):
+ count[0] += 1
+ if count[0] > ok:
+ raise OSError(errno.EIO, "replacement and compensation failed")
+ real(a, b)
+
+ return mock.patch.object(database.os, "rename", side_effect=rename)
+
+ def test_r2_01_a_failed_compensation_during_recovery_keeps_the_journal(self):
+ for action, final in (("upgrade", "18"), ("rollback", "9.6")):
+ with self.subTest(action=action):
+ self.fresh()
+ runner = self._crash_before_swap(action)
+ with self._renames_failing_after(1):
+ self.assertEqual(self.run_db("recover", runner), exit_codes.DB_UPGRADE_FAILED)
+ self.assertFalse(self.data.exists())
+ self.assertTrue(self.journal.exists()) # the only record of where the data went
+ self.assertIn("so did putting", self.err.getvalue())
+ with self._renames_failing_after(0): # it keeps failing: `up` and `recover` refuse, the journal stays
+ self.assertEqual(self.run_up(runner), exit_codes.DB_UPGRADE_FAILED) # never a fresh cluster
+ self.assertEqual(self.run_db("recover", runner), exit_codes.DB_UPGRADE_FAILED)
+ self.assertFalse([c for c in runner.calls if c[:2] == ["docker", "compose"] and "up" in c])
+ self.assertTrue(self.journal.exists())
+ self.assertEqual(self.run_db("recover", runner), exit_codes.OK)
+ self.assertEqual(self.major(), final)
+ self.assertFalse(self.journal.exists())
+
+ def test_r2_01_an_in_process_double_failure_keeps_the_journal(self):
+ real, count = os.rename, [0]
+
+ def rename(a, b):
+ count[0] += 1
+ if count[0] >= 2: # the replacement and the compensation, then every retry
+ raise OSError(errno.EIO, "injected")
+ real(a, b)
+
+ runner = FakeDocker()
+ with mock.patch.object(database.os, "rename", side_effect=rename):
+ self.assertEqual(self.run_db("upgrade", runner), exit_codes.DB_UPGRADE_FAILED)
+ self.assertFalse(self.data.exists())
+ self.assertTrue(self.journal.exists())
+ self.assertIn("recovery failed", self.err.getvalue())
+ self.assertEqual(self.run_db("recover", runner), exit_codes.OK)
+ self.assertEqual(self.major(), "18")
+
+ def test_r2_01_a_verified_compensation_retires_the_journal(self):
+ real = os.rename
+
+ def rename(a, b):
+ if Path(a) == self.new:
+ raise OSError(errno.EIO, "injected")
+ real(a, b)
+
+ runner = self._crash_before_swap("upgrade")
+ with mock.patch.object(database.os, "rename", side_effect=rename):
+ self.assertEqual(self.run_db("recover", runner), exit_codes.OK)
+ self.assertEqual(self.major(), "9.6")
+ self.assertFalse(self.journal.exists())
+
+ def test_r2_02_up_refuses_while_any_db_command_holds_the_lock(self):
+ (self.data / "PG_VERSION").write_text("18\n")
+ runner = FakeDocker()
+ with database.operation_lock(Context(root=self.root)):
+ self.assertEqual(self.run_up(runner), exit_codes.DB_UPGRADE_FAILED)
+ self.assertIn("db-operation.lock", self.err.getvalue())
+ self.assertFalse([c for c in runner.calls if c[:2] == ["docker", "compose"]])
+
+ def test_r2_02_up_cannot_start_between_the_checks_and_the_swap_of_a_rollback(self):
+ self._upgraded()
+ racer = FakeDocker()
+ codes = []
+
+ def confirm(ctx, *args, **kwargs): # the moment after rollback checked that nothing runs
+ codes.append(self.run_up(racer))
+ return True
+
+ with mock.patch.object(Context, "confirm", confirm):
+ self.assertEqual(self.run_db("rollback", FakeDocker(), stamp=TS + "r"), exit_codes.OK)
+ self.assertEqual(codes, [exit_codes.DB_UPGRADE_FAILED])
+ self.assertFalse([c for c in racer.calls if c[:2] == ["docker", "compose"] and "up" in c])
+ self.assertEqual(self.major(), "9.6")
+
+ def test_r2_02_a_db_command_cannot_start_while_up_starts_compose(self):
+ (self.data / "PG_VERSION").write_text("18\n")
+ self._upgraded_record_only()
+ codes = []
+ inner = FakeDocker()
+
+ def runner(argv):
+ if argv[:2] == ["docker", "compose"] and "up" in argv:
+ codes.append(self.run_db("rollback", FakeDocker()))
+ codes.append(self.run_db("upgrade", FakeDocker()))
+ codes.append(self.run_db("recover", FakeDocker()))
+ return inner(argv)
+
+ self.run_up(runner)
+ self.assertEqual(codes, [exit_codes.DB_UPGRADE_FAILED] * 3)
+ self.assertEqual(self.major(), "18")
+
+ def _upgraded_record_only(self):
+ self.kept.mkdir()
+ (self.kept / "PG_VERSION").write_text("9.6\n")
+ self.backup.mkdir(parents=True)
+ (self.backup / "upgrade.json").write_text(json.dumps({"from": "9.6", "to": "18", "kept": str(self.kept), "image": TARGET}))
+
+ def test_r2_02_reset_and_db_commands_exclude_each_other(self):
+ ctx = Context(root=self.root, yes=True, stdin_is_tty=False)
+ with database.operation_lock(ctx), redirect_stdout(self.out), redirect_stderr(self.err):
+ self.assertEqual(lifecycle.run(ctx, argparse.Namespace(lifecycle_action="reset"), runner=FakeDocker(), rmtree=lambda p: None),
+ exit_codes.DB_UPGRADE_FAILED)
+ self.assertEqual(self.major(), "9.6")
+ codes = []
+
+ def rmtree(path): # a db command attempted while reset deletes
+ codes.append(self.run_db("upgrade", FakeDocker()))
+ import shutil
+ shutil.rmtree(path)
+
+ with redirect_stdout(self.out), redirect_stderr(self.err):
+ self.assertEqual(lifecycle.run(ctx, argparse.Namespace(lifecycle_action="reset"), runner=FakeDocker(), rmtree=rmtree), exit_codes.OK)
+ self.assertEqual(codes, [exit_codes.DB_UPGRADE_FAILED])
+
+ def test_r2_03_recovery_syncs_the_renames_before_the_record_and_the_journal(self):
+ for action in ("upgrade", "rollback"):
+ with self.subTest(action=action):
+ self.fresh()
+ runner = FakeDocker()
+ if action == "rollback":
+ self._upgraded()
+ real, count = os.rename, [0]
+
+ def rename(a, b):
+ count[0] += 1
+ if count[0] == 2:
+ raise Crash("gap")
+ real(a, b)
+
+ with mock.patch.object(database.os, "rename", side_effect=rename), self.assertRaises(Crash):
+ self.run_db(action, runner, stamp=TS + "r" if action == "rollback" else TS)
+ order = []
+ real_sync, real_drop = database._fsync_dir, database._drop_journal
+ with mock.patch.object(database, "_fsync_dir", side_effect=lambda p: (order.append(("sync", Path(p))), real_sync(p))), \
+ mock.patch.object(database, "_drop_journal", side_effect=lambda c: (order.append(("drop", None)), real_drop(c))):
+ self.assertEqual(self.run_db("recover", runner), exit_codes.OK)
+ first_sync = order.index(("sync", self.data.parent))
+ self.assertLess(first_sync, order.index(("drop", None)))
+
+ def test_r2_03_a_failed_directory_sync_is_an_error_and_keeps_recovery_possible(self):
+ with mock.patch.object(database.os, "fsync", side_effect=OSError(errno.EIO, "disk error")):
+ with self.assertRaises(OSError):
+ database._fsync_dir(self.runtime)
+ with mock.patch.object(database.os, "fsync", side_effect=OSError(errno.EINVAL, "not supported")):
+ database._fsync_dir(self.runtime) # a filesystem that cannot sync directories is not a failure
+ # the first journal sync fails: nothing starts
+ runner = FakeDocker()
+ with mock.patch.object(database, "_fsync_dir", side_effect=OSError(errno.EIO, "disk error")):
+ self.assertEqual(self.run_db("upgrade", runner), exit_codes.DB_UPGRADE_FAILED)
+ self.assertEqual(runner.runs(), [])
+ self.assertFalse(self.journal.exists())
+ # a sync fails during recovery after a rename: the journal stays, a later recovery completes
+ runner = self._crash_before_swap("upgrade")
+ real_sync = database._fsync_dir
+
+ def sync(path):
+ if Path(path) == self.data.parent:
+ raise OSError(errno.EIO, "disk error")
+ real_sync(path)
+
+ with mock.patch.object(database, "_fsync_dir", side_effect=sync):
+ self.assertEqual(self.run_db("recover", runner), exit_codes.DB_UPGRADE_FAILED)
+ self.assertTrue(self.journal.exists())
+ self.assertEqual(self.run_db("recover", runner), exit_codes.OK)
+ self.assertEqual(self.major(), "18")
+ self.assertTrue((self.backup / "upgrade.json").exists())
+
+ def test_r2_04_a_non_ascii_pg_version_is_exit_15_not_a_traceback(self):
+ (self.data / "PG_VERSION").write_bytes(b"\xff\n")
+ self.assertIsNone(database.data_major(self.data))
+ runner = FakeDocker()
+ self.assertEqual(self.run_up(runner), exit_codes.DB_UPGRADE_FAILED)
+ self.assertIn("cannot read the PostgreSQL major", self.err.getvalue())
+ self.assertFalse([c for c in runner.calls if c[:2] == ["docker", "compose"]])
+ self.assertEqual(self.run_db("upgrade", FakeDocker()), exit_codes.DB_UPGRADE_FAILED)
+ self.assertEqual(self.run_db("status", FakeDocker()), exit_codes.OK)
+ self.assertIn("unknown", self.out.getvalue())
+
+ def undecodable(argv):
+ raise UnicodeDecodeError("utf-8", b"\xff", 0, 1, "invalid start byte")
+
+ self._unreadable()
+ self.assertIsNone(database.data_major(self.data, undecodable, TARGET))
+
+ # ------------------------------------------------------------------ review r3: the CLI entry point never leaks OSError
+
+ def cli(self, *args, runner):
+ from grdm_dev import cli, health
+ refuse_compose = lambda argv: (99, "") # noqa: E731 - `up` must never reach a real Docker in a unit test
+ with mock.patch.object(health, "default_runner", runner), mock.patch.object(lifecycle, "streaming_runner", refuse_compose), \
+ redirect_stdout(self.out), redirect_stderr(self.err):
+ return cli.main(["--root", str(self.root), "--yes", *args])
+
+ def _pending_after_both_renames(self):
+ runner = self._crash_before_swap("upgrade")
+ database._swap(self.data, self.new, self.kept) # the renames happened, the record and the journal removal did not
+ return runner
+
+ def test_r3_01_eio_on_the_recovery_sync_is_exit_15_at_every_entry_point(self):
+ for args in (("db", "upgrade"), ("db", "rollback"), ("db", "recover"), ("up", "--no-open"), ("reset",)):
+ with self.subTest(command=" ".join(args)):
+ self.fresh()
+ runner = self._pending_after_both_renames()
+ real = database._fsync_dir
+
+ def sync(path):
+ if Path(path) == self.data.parent:
+ raise OSError(errno.EIO, "injected volumes sync")
+ real(path)
+
+ with mock.patch.object(database, "_fsync_dir", side_effect=sync):
+ self.assertEqual(self.cli(*args, runner=runner), exit_codes.DB_UPGRADE_FAILED)
+ self.assertTrue(self.journal.exists())
+ self.assertEqual(self.major(), "18")
+ self.assertEqual(self.cli("db", "recover", runner=runner), exit_codes.OK)
+ self.assertFalse(self.journal.exists())
+ self.assertTrue((self.backup / "upgrade.json").exists())
+
+ def _eio_at_each_fsync(self, args, prepare):
+ """Run `args` through cli.main once to count os.fsync calls, then again with EIO at each one in turn."""
+ self.fresh()
+ runner = prepare()
+ real, calls = os.fsync, []
+ with mock.patch.object(database.os, "fsync", side_effect=lambda fd: (calls.append(fd), real(fd))):
+ self.assertEqual(self.cli(*args, runner=runner), exit_codes.OK, self.err.getvalue())
+ self.assertTrue(calls)
+ for nth in range(len(calls)):
+ with self.subTest(command=" ".join(args), fsync=nth + 1):
+ self.fresh()
+ runner = prepare()
+ count = []
+
+ def fsync(fd, nth=nth):
+ count.append(fd)
+ if len(count) == nth + 1:
+ raise OSError(errno.EIO, "injected fsync")
+ real(fd)
+
+ with mock.patch.object(database.os, "fsync", side_effect=fsync):
+ code = self.cli(*args, runner=runner)
+ self.assertEqual(code, exit_codes.DB_UPGRADE_FAILED, self.err.getvalue())
+ self.assertIn("injected fsync", self.err.getvalue())
+ self.assertEqual(self.cli("db", "recover", runner=runner), exit_codes.OK, self.err.getvalue())
+ self.assertFalse(self.journal.exists())
+ self.assertIn(self.major(), ("9.6", "18"))
+ self.assertEqual(runner.present, set())
+ return len(calls)
+
+ def test_r3_01_eio_at_each_fsync_of_db_upgrade_is_exit_15_through_the_cli(self):
+ self.assertGreaterEqual(self._eio_at_each_fsync(("db", "upgrade"), FakeDocker), 5)
+
+ def test_r3_01_eio_at_each_fsync_of_db_rollback_is_exit_15_through_the_cli(self):
+ def upgraded():
+ self._upgraded()
+ return FakeDocker()
+
+ self.assertGreaterEqual(self._eio_at_each_fsync(("db", "rollback"), upgraded), 4)
+
+ def test_r3_01_eio_at_each_fsync_of_db_recover_is_exit_15_through_the_cli(self):
+ self.assertGreaterEqual(self._eio_at_each_fsync(("db", "recover"), self._pending_after_both_renames), 3)
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/installer/tests/test_manifest.py b/installer/tests/test_manifest.py
index 7708aa7..cf5930c 100644
--- a/installer/tests/test_manifest.py
+++ b/installer/tests/test_manifest.py
@@ -45,7 +45,7 @@ def test_real_manifest_loads_and_validates(self):
self.assertTrue(osf.is_built)
self.assertEqual(len(osf.sha12), 12)
self.assertTrue(osf.local_image().startswith("grdm-dev/osf.io:"))
- self.assertEqual(man.component("postgres").image_reference(), "postgres:9.6")
+ self.assertEqual(man.component("postgres").image_reference(), "postgres:18.6@sha256:e3d24f6906d0760bf4da0dae872415391ededefad23fd471c47dc53b0ddf3c42") # ADR-12
# dict compatibility for modules written against raw JSON
self.assertEqual(osf["id"], "osf.io")
self.assertEqual(osf.get("integration_repo"), "RDM-osf.io-dev2027")
@@ -186,7 +186,7 @@ def test_service_fields(self):
self.assertEqual(services["admin"].profile, "core")
pg = services["postgres"]
self.assertIsNone(pg.build)
- self.assertEqual(pg.image, "postgres:9.6")
+ self.assertEqual(pg.image, "postgres:18.6@sha256:e3d24f6906d0760bf4da0dae872415391ededefad23fd471c47dc53b0ddf3c42")
self.assertEqual(pg.role, "infra")
def test_image_tag_override_from_build_report(self):
diff --git a/installer/tests/test_plan.py b/installer/tests/test_plan.py
index c2f579e..027ebb2 100644
--- a/installer/tests/test_plan.py
+++ b/installer/tests/test_plan.py
@@ -1,3 +1,5 @@
+from __future__ import annotations
+
import argparse
import io
import json
diff --git a/installer/tests/test_preflight.py b/installer/tests/test_preflight.py
index 0b4fe64..339ac88 100644
--- a/installer/tests/test_preflight.py
+++ b/installer/tests/test_preflight.py
@@ -1085,14 +1085,14 @@ def test_no_rendered_file_means_no_container(self):
class ExitCodeTests(unittest.TestCase):
def test_new_codes_have_messages_in_both_languages(self):
- for code in (exit_codes.PREFLIGHT_FAILED, exit_codes.RESET_FAILED):
+ for code in (exit_codes.PREFLIGHT_FAILED, exit_codes.RESET_FAILED, exit_codes.DB_UPGRADE_FAILED):
self.assertNotEqual(exit_codes.describe(code, "en"), "unknown exit code")
self.assertNotEqual(exit_codes.describe(code, "ja"), "不明な終了コード")
def test_spec_table_lists_them(self):
text = (REPO_ROOT / "docs" / "spec" / "02_installer_cli.md").read_text(encoding="utf-8")
- self.assertEqual(text.count("exit_codes[16]{code,meaning}:"), 2)
- self.assertEqual(len(exit_codes.messages()), 16)
+ self.assertEqual(text.count("exit_codes[17]{code,meaning}:"), 2)
+ self.assertEqual(len(exit_codes.messages()), 17)
if __name__ == "__main__":
diff --git a/manifest/hardening.json b/manifest/hardening.json
index fcb8b02..5dcf2fc 100644
--- a/manifest/hardening.json
+++ b/manifest/hardening.json
@@ -38,7 +38,7 @@
"reason": "The official postgres entrypoint starts as root, fixes ownership of the data directory (bind-mounted under runtime/volumes/), runs the init scripts, then drops to the postgres user with gosu. A fixed user: breaks first start on Linux hosts where the bind mount is owned by the developer.",
"review": "2027-01-31",
"service": "postgres",
- "target": "M5 (postgres:15 with a pre-created data volume)"
+ "target": "M5 (postgres:18 adopted by ADR-12; the entrypoint still starts as root, so the exception stays until the data volume is pre-created for uid 999)"
},
{
"cap_add": [
diff --git a/manifest/upstreams.json b/manifest/upstreams.json
index 7092639..aaa99d1 100644
--- a/manifest/upstreams.json
+++ b/manifest/upstreams.json
@@ -764,14 +764,14 @@
},
{
"adopted_prs": [],
- "base_image": "postgres:9.6",
- "base_image_digest": null,
+ "base_image": "postgres:18.6",
+ "base_image_digest": "sha256:e3d24f6906d0760bf4da0dae872415391ededefad23fd471c47dc53b0ddf3c42",
"candidate_prs": [],
"id": "postgres",
"integration_branch": null,
"integration_sha": null,
"integration_repo": null,
- "note": null,
+ "note": "PostgreSQL 18.6, the newest released major (19 is in beta), pinned by its index digest (ADR-12). osf.io's Django 1.11.28 with psycopg2 2.7.3 (libpq 11) ran its migrations, populate_institutions and 807 DB tests on it with the same outcome as on 9.6 (781 passed, the same 13 failed or erred on both); CAS's JDBC driver 42.1.4 cannot do SCRAM, so the template keeps --auth-host=md5, which also makes initdb set password_encryption=md5. A 9.6 data directory is moved by `grdm-dev db upgrade` (dump and restore, the old directory kept); `up` refuses a directory of another major.",
"platforms": [
"linux/arm64",
"linux/amd64"
@@ -786,7 +786,7 @@
],
"target_image": {
"M4": null,
- "M5": "postgres:15"
+ "M5": "postgres:18 (adopted early, ADR-12); SCRAM once CAS has JDBC 42.2 or later"
},
"upstreams": []
},