diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..86aee8e --- /dev/null +++ b/.gitattributes @@ -0,0 +1,16 @@ +# Files that run inside Linux (the shell, WSL2, or a container) must keep LF line endings even when the +# repository is cloned on Windows with core.autocrlf=true: a CRLF shebang fails before install.sh's own +# CRLF guard can run (ADR-08). The list is explicit on purpose: `* text=auto` would rewrite legacy guide +# files that are CRLF in the index. +*.sh text eol=lf +*.bash text eol=lf +Dockerfile text eol=lf +Dockerfile.* text eol=lf +docker-compose*.yml text eol=lf +installer/**/*.py text eol=lf +installer/grdm_dev/messages.tsv text eol=lf +installer/grdm_dev/templates/** text eol=lf +installer/images/** text eol=lf +installer/pyproject.toml text eol=lf +manifest/** text eol=lf +tools/**/*.py text eol=lf diff --git a/README.md b/README.md index 02e99ee..b65a5fc 100644 --- a/README.md +++ b/README.md @@ -5,6 +5,7 @@ **Jump:** [English](#english) · [日本語](#japanese) +- Version / 版: 0.3 — 2026-11-10. **Learned:** a decision index goes stale one record at a time, so each new ADR is added to the root list in the same commit (ADR-08). ／ 決定の索引は一件ずつ古くなるので、新しい ADR は同じコミットでルートの一覧に加える（ADR-08）。 - Version / 版: 0.2 — 2026-11-06. **Learned:** a glossary term is only explained for the reader when its note and REF-01 link sit at the first prose use, counting lower-case and plural forms, so first-use notes with links were added in both halves (SPEC-06 §3). ／ 用語注と REF-01 へのリンクは、小文字や複数形も含めた本文の初出に置かなければ読者に届かないと分かったため、英和両方の初出にリンク付きの用語注を補った（SPEC-06 §3）。 - Version / 版: 0.1 — 2026-11-05 - Normative for: `README.md` @@ -83,6 +84,7 @@ Each record explains one choice. Repository (a folder that remembers every chang - [ADR-05 Test IdP for Shibboleth](docs/decisions/ADR-05_Test_IdP_for_Shibboleth.md) - Test IdP for Shibboleth / Shibboleth 用テスト IdP - [ADR-06 Wiki Frontend Overlay](docs/decisions/ADR-06_Wiki_Frontend_Overlay.md) - Wiki frontend overlay / Wiki フロントエンドのオーバーレイ - [ADR-07 Native Builds of EOL Middleware](docs/decisions/ADR-07_Native_Builds_of_EOL_Middleware.md) - Native builds of EOL middleware / EOL ミドルウェアのネイティブビルド +- [ADR-08 Per-OS Installer Entry Points](docs/decisions/ADR-08_Per_OS_Installer_Entry_Points.md) - Per-OS installer entry points / OS ごとのインストーラーの入口 #### References / 参考資料 @@ -181,6 +183,7 @@ graph TD - [ADR-05 Shibboleth 用テスト IdP](docs/decisions/ADR-05_Test_IdP_for_Shibboleth.md) - Test IdP for Shibboleth / Shibboleth 用テスト IdP - [ADR-06 Wiki フロントエンドのオーバーレイ](docs/decisions/ADR-06_Wiki_Frontend_Overlay.md) - Wiki frontend overlay / Wiki フロントエンドのオーバーレイ - [ADR-07 EOL ミドルウェアのネイティブビルド](docs/decisions/ADR-07_Native_Builds_of_EOL_Middleware.md) - Native builds of EOL middleware / EOL ミドルウェアのネイティブビルド +- [ADR-08 OS ごとのインストーラーの入口](docs/decisions/ADR-08_Per_OS_Installer_Entry_Points.md) - Per-OS installer entry points / OS ごとのインストーラーの入口 #### 参考資料 diff --git a/docs/decisions/ADR-08_Per_OS_Installer_Entry_Points.md b/docs/decisions/ADR-08_Per_OS_Installer_Entry_Points.md new file mode 100644 index 0000000..aee6494 --- /dev/null +++ b/docs/decisions/ADR-08_Per_OS_Installer_Entry_Points.md @@ -0,0 +1,151 @@ +# ADR-08 Per-OS Installer Entry Points / OS ごとのインストーラーの入口 + +> **Summary:** `installer/install.sh` assumes that Python, git, Docker and the scanners already exist, and Windows has no entry point at all. We decide to add three thin, consent-based prerequisite scripts, one per OS, that prepare the host and always end by running `install.sh`, so there is still one decision path. This relaxes "never needs `sudo`" for those scripts only. +> **概要:** `installer/install.sh` は Python・git・Docker・スキャナーが既にあることを前提にしており、Windows には入口そのものがない。OS ごとに一つ、同意に基づく薄い前提条件スクリプトを三つ加え、ホストを準備したうえで必ず `install.sh` を実行して終わることを決める。判断の道筋は一つのままである。「`sudo` を必要としない」という規則は、これらのスクリプトに限って緩める。 + +**Jump:** [English](#english) · [日本語](#japanese) + +- Version / 版: 0.3 — 2026-11-10. **Learned:** the decision can land before its CI: the workflows of §9 need a token with the `workflow` scope, so they wait on branch `feat/per-os-installers` and the per-OS scripts are checked by hand and by DEMO-21 until then. ／ 決定は CI より先に入れられる。§9 のワークフローには `workflow` スコープのトークンが要るので、ブランチ `feat/per-os-installers` で待ち、それまで OS ごとのスクリプトは手と DEMO-21 で確かめる。 + 0.2 — 2026-11-10. **Learned:** the first real run of `install-ubuntu.sh`, on an Ubuntu 24.04 arm64 virtual machine, settled two rows of §5 and showed that a distribution's own emulator package is not a safe prerequisite: it crashed inside a build, so emulation now comes from a digest-pinned image ([DEMO-21](../demo/21_core_install_ubuntu_arm64_lima.md)). ／ `install-ubuntu.sh` の最初の実際の実行（Ubuntu 24.04 arm64 の仮想マシン）で §5 の二行が決着し、ディストリビューション自身のエミュレーターのパッケージは安全な前提条件ではないと分かった。ビルドの中で落ちたので、エミュレーションはダイジェストで固定したイメージから入れる（[DEMO-21](../demo/21_core_install_ubuntu_arm64_lima.md)）。 + 0.1 — 2026-11-10. **Learned:** "the installer checks each prerequisite" only moved the first failure later: without Trivy and gitleaks an otherwise perfect install still stops with exit 51 at its `scan` step, so prerequisites are everything the whole sequence needs, not only what its first step needs. ／ 「インストーラーが各前提条件を確認する」は最初の失敗を後ろへずらしただけだった。Trivy と gitleaks が無いと、ほかが完璧でも導入は `scan` の段で終了 51 になる。前提条件とは最初の段に要るものではなく、一連の手順全体に要るものである。 +- Status: Proposed for the owner's approval, 2026-11-10 (M2). Review date: after the first recorded run of each script on owner hardware. +- Records a decision for: [SPEC-02 §1, §2.3, §8, §9, §10](../spec/02_installer_cli.md#english) +- Parent index / 親索引: [Decision records](README.md) +- Related: [ADR-03 Staged security gate](ADR-03_Staged_Security_Gate.md) · [ADR-07 Native builds of EOL middleware](ADR-07_Native_Builds_of_EOL_Middleware.md) · [SPEC-04 §8](../spec/04_acceptance_tests.md#english) · [Environment guide](../guides/environment/README.md#english) +- Used by: [SPEC-02](../spec/02_installer_cli.md#english) · [Environment guide](../guides/environment/README.md#english) + +--- + + +## English + +[→ 日本語版へ](#japanese) + +### 1. Context + +An audit of the branch on 2026-11-10 found these gaps. Each was checked against the files. + +- Windows has no entry point, although [SPEC-02 §8](../spec/02_installer_cli.md#english) requires Windows 11 to work. +- No script installs Docker (the tool that builds images and runs containers; see [REF-01](../reference/REF-01_Glossary.md)), Python, buildx (Docker's helper that builds one image for several kinds of computer; see [REF-01](../reference/REF-01_Glossary.md)) or git. `install.sh` only prints a hint for Python. +- `grdm-dev install` stops with exit 51 at its `scan` step when `trivy` or `gitleaks` is missing, and nothing installs them. +- The repository had no `.gitattributes`. A clone with `core.autocrlf=true` turns the first line of `install.sh` into `#!/usr/bin/env bash` plus a CR (carriage return, the extra character Windows puts at the end of a line). The kernel then fails to find the interpreter, so the script's own CRLF guard never runs. The guard's hint also used `sed -i ""`, which only BSD `sed` accepts. +- No workflow runs `install.sh`, Compose (the Docker tool that starts many containers from one file; see [REF-01](../reference/REF-01_Glossary.md)), macOS, Windows or arm runners. The Windows CI (continuous integration: the checks that run automatically; see [REF-01](../reference/REF-01_Glossary.md)) build of SPEC-02 §8 and the amd64 (the CPU type of ordinary PCs and servers; see [REF-01](../reference/REF-01_Glossary.md)) build of SPEC-04 §8 did not exist. + +### 2. Decision + +1. **Three scripts, one decision path.** `installer/install-ubuntu.sh`, `installer/install-macos.sh` and `installer/install-windows.ps1` only prepare the host. Each ends by running `installer/install.sh`, passing other arguments unchanged. Windows reaches it through `install-ubuntu.sh` inside WSL2 (a way to run Linux inside Windows; see [REF-01](../reference/REF-01_Glossary.md)). No OS difference moves into `grdm-dev`. +2. **`sudo` is allowed here and only here.** `install.sh` and `grdm-dev` keep the rule that they never need `sudo` (a way to run a command as the computer's administrator). The prerequisite scripts may use it, after consent. +3. **Consent.** Detection is read-only. Each script lists its changes first and asks once; joining the `docker` group has its own question, because that group is equivalent to administrator rights. `--yes` (`-Yes` on Windows) gives consent up front. With no terminal and no `--yes` the script stops with exit 2. `--dry-run` (`-DryRun`) prints the plan and changes nothing. +4. **Exit codes.** The scripts reuse 2, 10 and 11 of `messages.tsv`. No number is added. +5. **Trusted sources only; nothing is piped into a shell.** + - Ubuntu: Docker's apt (Ubuntu's package installer) repository. The signing key is accepted only when its fingerprint is `9DC8 5822 9FC7 DD38 854A E2D8 8D81 803C 0EBF CD88`. Trivy (a scanner that reads our images and lists the holes it knows about; see [REF-01](../reference/REF-01_Glossary.md)) 0.75.0 and gitleaks 8.21.2 come from their GitHub releases and must match a SHA-256 (a file's fingerprint) pinned in the script, then go to `~/.local/bin`. + - macOS: Homebrew (a package manager for macOS) formulae, whose bottles Homebrew checks. Homebrew itself is not installed by the script, because its official one-line installer pipes a download into a shell; the script points to the signed `.pkg`. + - Windows: `wsl.exe` from Microsoft. Docker Desktop is detected, never installed, because its licence is the user's decision. The same applies on macOS. +6. **The clone stays out of `/mnt/c`.** `install-windows.ps1` accepts only one folder name under the WSL home directory, and `install-ubuntu.sh` refuses a checkout under `/mnt/` inside WSL. +7. **Tokens.** The integration repositories are private. The scripts say how to give git access (`gh auth login`, then `gh auth setup-git`) and never ask for, read or store a token. +8. **Line endings.** `.gitattributes` forces LF for shell scripts, Dockerfiles and the files that run in Linux containers. It is an explicit list, because `* text=auto` would rewrite legacy files. The CRLF hint now uses `perl -pi -e`, which works on macOS, Ubuntu and WSL. +9. **CI.** `installer-prereq-lint.yml` runs on every pull request: `bash -n`, shellcheck (a tool that finds mistakes in shell scripts), the PowerShell parser on `windows-latest`, and each script's dry run. `installer-full.yml` is `workflow_dispatch` only (started by hand) and does a full `minimum` install on `ubuntu-24.04` and `ubuntu-24.04-arm`. It reads the repository secret `GRDM_INTEGRATION_READ_TOKEN`, a read-only token that the owner creates, and passes it to git through a credential helper in one step's environment, so it is never written to disk. This workflow is what fulfils SPEC-04 §8's amd64 build, by hand. **Pending:** both workflows and `.github/scripts/Test-PowerShellSyntax.ps1` wait on branch `feat/per-os-installers` for a push with a token that has the `workflow` scope. + +### 3. Options considered + +```toon +options[4]{option,decision_path,risk,chosen}: + A,"only print better hints (today)",one,"every new user repeats the same manual steps; Windows stays unserved",no + B,"put OS bootstrap inside grdm-dev",one but mixed,"grdm-dev would need sudo and would break SPEC-02 §1; Python must exist first",no + C,"one cross-platform script",one,"PowerShell and bash cannot share a file; Windows must drive WSL from outside anyway",no + D,"thin per-OS scripts that end at install.sh (this ADR)",one,"three small scripts to keep working; only owner hardware proves them fully",yes +``` + +### 4. Consequences + +- A new developer on a supported OS needs one command, plus any manual step that the script names. +- The Docker signing-key fingerprint and four SHA-256 values must be updated by hand when those versions change. That is deliberate. +- The manual workflow must exist on the default branch before GitHub lets anyone start it. + + +### 5. What hosted runners cannot prove + +| Claim | Status on GitHub-hosted runners | +|---|---| +| Docker Engine installed from Docker's apt repository | Not exercised when the runner image already has Docker 27 or newer, because the script then skips it. **Proven on a clean Ubuntu 24.04 arm64 virtual machine** ([DEMO-21](../demo/21_core_install_ubuntu_arm64_lima.md)); amd64 still needs a clean machine. | +| WSL2 on `windows-latest` | **Unverified.** Nested virtualization (a virtual machine inside a virtual machine, which WSL2 needs) is doubtful on these runners, and no full run has been attempted. CI only parses the script and does a dry run. | +| UAC prompt, restart after enabling WSL, first-user creation, Docker Desktop's WSL integration | Need a person and real Windows 11 hardware. UAC is the Windows prompt that asks for administrator rights. | +| macOS with Docker Desktop | Hosted macOS runners provide no Docker engine, so CI is dry-run only. M2 acceptance on the Mac Studio remains the evidence. | +| amd64 emulation on arm64 Ubuntu | **Proven on a virtual machine** ([DEMO-21](../demo/21_core_install_ubuntu_arm64_lima.md)): Ubuntu's `qemu-user-static` 8.2.2 crashed, so the script registers QEMU from a digest-pinned `tonistiigi/binfmt` image. Not persistent across reboots. | + +The SPEC-02 §8 phrase "checked in CI as build-only" for Windows is withdrawn: CI checks the Windows script's syntax and dry run only. + +--- + + +## 日本語 + +[→ To English](#english) + +### 1. 背景 + +2026-11-10 にこのブランチを監査し、次の不足を見つけた。いずれもファイルと突き合わせて確かめた。 + +- [SPEC-02 §8](../spec/02_installer_cli.md#japanese) は Windows 11 で動くことを求めているのに、Windows の入口がない。 +- Docker（イメージを作りコンテナを動かす道具。[REF-01](../reference/REF-01_Glossary.md)）、Python、buildx（一つのイメージを何種類かの計算機向けに作る Docker の助手。[REF-01](../reference/REF-01_Glossary.md)）、git を入れるスクリプトがない。`install.sh` は Python の案内を表示するだけである。 +- `trivy` か `gitleaks` が無いと `grdm-dev install` は `scan` の段で終了 51 になるが、それらを入れるものがない。 +- `.gitattributes` が無かった。`core.autocrlf=true` で clone すると `install.sh` の 1 行目は `#!/usr/bin/env bash` の後に CR（Windows が行末に足す余分な文字）が付く。すると OS がインタープリタを見つけられず、スクリプト自身の CRLF 検査は動かない。その検査の案内も BSD の `sed` だけが受け付ける `sed -i ""` だった。 +- `install.sh`、Compose（一つのファイルから多くのコンテナを起こす Docker の道具。[REF-01](../reference/REF-01_Glossary.md)）、macOS、Windows、arm のランナーを動かすワークフローがなかった。SPEC-02 §8 の Windows の CI（自動で動く検査。[REF-01](../reference/REF-01_Glossary.md)）ビルドと SPEC-04 §8 の amd64（普通の PC やサーバーの CPU の種類。[REF-01](../reference/REF-01_Glossary.md)）ビルドは存在しなかった。 + +### 2. 決定 + +1. **スクリプトは三つ、判断の道筋は一つ。** `installer/install-ubuntu.sh`・`installer/install-macos.sh`・`installer/install-windows.ps1` はホストの準備だけを行う。どれも最後に `installer/install.sh` を実行し、ほかの引数はそのまま渡す。Windows は WSL2（Windows の中で Linux を動かす仕組み。[REF-01](../reference/REF-01_Glossary.md)）の中の `install-ubuntu.sh` を通ってそこへ至る。OS の違いを `grdm-dev` に持ち込まない。 +2. **`sudo` はここだけで認める。** `install.sh` と `grdm-dev` は引き続き `sudo`（コンピュータの管理者としてコマンドを動かす仕組み）を必要としない。前提条件スクリプトは同意の後に使ってよい。 +3. **同意。** 判定は読むだけである。各スクリプトは変更点を先に示し、一度尋ねる。`docker` グループへの追加は管理者権限と同等なので、別に尋ねる。`--yes`（Windows は `-Yes`）は同意を先に与える。端末が無く `--yes` も無ければ終了 2 で止まる。`--dry-run`（`-DryRun`）は予定を表示するだけで何も変えない。 +4. **終了コード。** `messages.tsv` の 2・10・11 を使い回し、番号を増やさない。 +5. **信頼できる配布元だけを使い、ダウンロードをシェルへ流し込まない。** + - Ubuntu: Docker の apt（Ubuntu のパッケージ導入の道具）リポジトリ。署名鍵は指紋が `9DC8 5822 9FC7 DD38 854A E2D8 8D81 803C 0EBF CD88` のときだけ受け入れる。Trivy 0.75.0 と gitleaks 8.21.2 は GitHub のリリースから取り、スクリプトに固定した SHA-256（ファイルの指紋）との一致を確かめて `~/.local/bin` に置く。 + - macOS: Homebrew（macOS のパッケージ管理の道具）の formula を使い、bottle の検証は Homebrew が行う。Homebrew 自体はスクリプトでは入れない。公式の一行の導入方法がダウンロードをシェルへ流し込むからであり、署名付きの `.pkg` を案内する。 + - Windows: Microsoft の `wsl.exe`。Docker Desktop は検出するだけで入れない。ライセンスは利用者が判断することだからである。macOS も同じである。 +6. **clone を `/mnt/c` に置かない。** `install-windows.ps1` は WSL のホームの下のフォルダ名一つだけを受け付け、`install-ubuntu.sh` は WSL の中で `/mnt/` の下のチェックアウトを拒む。 +7. **トークン。** 統合リポジトリは非公開である。スクリプトは git に権限を与える方法（`gh auth login` の後に `gh auth setup-git`）を案内するだけで、トークンを尋ねず、読まず、保存しない。 +8. **改行。** `.gitattributes` はシェルスクリプト、Dockerfile、Linux コンテナで動くファイルの改行を LF に固定する。`* text=auto` は旧いファイルを書き換えてしまうので、明示の一覧にする。CRLF の案内は macOS・Ubuntu・WSL で動く `perl -pi -e` に改めた。 +9. **CI。** `installer-prereq-lint.yml` はすべての PR で動く。`bash -n`、shellcheck（シェルスクリプトの誤りを見つける道具）、`windows-latest` での PowerShell パーサー、各スクリプトの dry run である。`installer-full.yml` は `workflow_dispatch` のみ（手で起動する）で、`ubuntu-24.04` と `ubuntu-24.04-arm` で `minimum` の完全な導入を行う。リポジトリのシークレット `GRDM_INTEGRATION_READ_TOKEN`（所有者が作る読み取り専用のトークン）を読み、一つの段の環境に置いた認証ヘルパーで git に渡すので、ディスクには書かれない。SPEC-04 §8 の amd64 ビルドは、このワークフローを手で動かすことで満たす。**保留中:** 二つのワークフローと `.github/scripts/Test-PowerShellSyntax.ps1` は、`workflow` スコープを持つトークンでの push をブランチ `feat/per-os-installers` で待つ。 + +### 3. 検討した選択肢 + +```toon +options[4]{option,decision_path,risk,chosen}: + A,"案内を良くするだけ（現状）",一つ,"新しい利用者が毎回同じ手作業を繰り返す。Windows は対応しないまま",いいえ + B,"OS の準備を grdm-dev に入れる",一つだが混在,"grdm-dev に sudo が要り SPEC-02 §1 を破る。Python が先に要る",いいえ + C,"全 OS 共通のスクリプト一つ",一つ,"PowerShell と bash は一つのファイルを共有できない。Windows はどのみち外から WSL を動かす",いいえ + D,"install.sh で終わる OS ごとの薄いスクリプト（本 ADR）",一つ,"小さなスクリプト三つの保守が要る。完全な証明は所有者の実機だけができる",はい +``` + +### 4. 影響 + +- 対応 OS の新しい開発者は、コマンド一つと、スクリプトが名指しする手作業だけで済む。 +- Docker の署名鍵の指紋と四つの SHA-256 は、版を変えるときに手で更新する。これは意図したことである。 +- 手動のワークフローは、既定のブランチに入ってからでないと GitHub で起動できない。 + + +### 5. ホスト型ランナーでは証明できないこと + +| 主張 | GitHub ホスト型ランナーでの状況 | +|---|---| +| Docker の apt リポジトリからの Docker Engine の導入 | ランナーのイメージに Docker 27 以上があると、スクリプトは飛ばすので試されない。**まっさらな Ubuntu 24.04 arm64 の仮想マシンで実証済み**（[DEMO-21](../demo/21_core_install_ubuntu_arm64_lima.md)）。amd64 はまだまっさらな機械が要る。 | +| `windows-latest` での WSL2 | **未検証。** WSL2 に要る入れ子の仮想化（仮想マシンの中の仮想マシン）はこのランナーでは疑わしく、完全な実行は試していない。CI はスクリプトの解析と dry run だけを行う。 | +| UAC の確認、WSL 有効化後の再起動、最初の利用者の作成、Docker Desktop の WSL 連携 | 人と実機の Windows 11 が要る。UAC は管理者権限を求める Windows の確認画面である。 | +| macOS と Docker Desktop | ホスト型の macOS ランナーには Docker エンジンが無いので、CI は dry run のみ。証拠は引き続き Mac Studio での M2 受入である。 | +| arm64 の Ubuntu での amd64 エミュレーション | **仮想マシンで実証済み**（[DEMO-21](../demo/21_core_install_ubuntu_arm64_lima.md)）。Ubuntu の `qemu-user-static` 8.2.2 は落ちたので、スクリプトはダイジェストで固定した `tonistiigi/binfmt` イメージから QEMU を登録する。再起動をまたいでは残らない。 | + +SPEC-02 §8 の Windows の「CI ではビルド確認のみ」という記述は取り下げる。CI が確かめるのは Windows のスクリプトの構文と dry run だけである。 + +--- + +## Glossary / 用語 + +| Term | English note | 日本語注 | +|---|---|---| +| Prerequisite | Something that must already be in place before a step can work. | ある段が動く前に揃っていなければならないもの。 | +| Consent | The user saying yes before a change is made. | 変更の前に利用者が「はい」と言うこと。 | +| Dry run | A practice run that only says what would happen. | 何が起きるかを言うだけの予行演習。 | +| Fingerprint | A short value that identifies one key or file exactly. | 一つの鍵やファイルを正確に見分ける短い値。 | +| CRLF / LF | The two ways of ending a line: Windows uses CRLF, Linux and macOS use LF. | 行の終わり方の二種類。Windows は CRLF、Linux と macOS は LF。 | +| BOM | Invisible bytes at the start of a file that say how its text is encoded. | ファイルの先頭にある、文字の符号化を示す見えないバイト。 | +| Hosted runner | A temporary machine that GitHub lends to run CI. | CI を動かすために GitHub が貸してくれる一時的な計算機。 | diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 57a040a..3efacd6 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -1,16 +1,17 @@ # Decision Records / 決定の記録 -> **Summary:** Documents explaining why we chose one way and not another: integration repository placement, network model, staged security gate, AI autonomy and roles, the test IdP, the wiki-frontend overlay, native builds of EOL middleware, PostgreSQL 18 with its data upgrade, and the retirement of Elasticsearch 2 and MongoDB. Each record captures the context, options considered, and final decision. -> **概要:** なぜその道を選び、ほかを選ばなかったかを説明する文書：統合リポジトリの置き場、ネットワーク方式、セキュリティ門の段階化、AI の自律度と役割、テスト IdP、wiki-frontend のオーバーレイ、EOL ミドルウェアのネイティブビルド、PostgreSQL 18 とデータの更新、Elasticsearch 2 と MongoDB の退役。各記録は文脈、検討した選択肢、最終決定を捉えます。 +> **Summary:** Documents explaining why we chose one way and not another: integration repository placement, network model, staged security gate, AI autonomy and roles, the test IdP, the wiki-frontend overlay, native builds of EOL middleware, per-OS installer entry points, PostgreSQL 18 with its data upgrade, and the retirement of Elasticsearch 2 and MongoDB. Each record captures the context, options considered, and final decision. +> **概要:** なぜその道を選び、ほかを選ばなかったかを説明する文書：統合リポジトリの置き場、ネットワーク方式、セキュリティ門の段階化、AI の自律度と役割、テスト IdP、wiki-frontend のオーバーレイ、EOL ミドルウェアのネイティブビルド、OS ごとのインストーラーの入口、PostgreSQL 18 とデータの更新、Elasticsearch 2 と MongoDB の退役。各記録は文脈、検討した選択肢、最終決定を捉えます。 **Jump:** [English](#english) · [日本語](#japanese) -- Version / 版: 0.5 — 2026-11-10. **Learned:** a data-tier change is two records, not one: the PostgreSQL move is proposed with its proof (ADR-12), while the removal of Elasticsearch 2 and MongoDB still needs owner choices (ADR-13); ADR-08 to ADR-11 stay reserved or planned for records on other branches. ／ データ層の変更は一つでなく二つの記録になる。PostgreSQL の移行は証明つきで提案し（ADR-12）、Elasticsearch 2 と MongoDB の撤去はまだ所有者の選択を要する（ADR-13）。ADR-08〜ADR-11 は他のブランチの記録のために予約・予定のまま残す。 +- Version / 版: 0.6 — 2026-11-10. **Learned:** a rule like "never needs sudo" has to name what it covers, so giving hosts a bootstrap became an architecture decision (ADR-08) instead of an exception nobody records. ／ 「sudo を必要としない」のような規則は対象を名指ししなければならない。ホストを準備する段を設けることは、誰も記録しない例外ではなく設計判断（ADR-08）になった。 + 0.5 — 2026-11-10. **Learned:** a data-tier change is two records, not one: the PostgreSQL move is proposed with its proof (ADR-12), while the removal of Elasticsearch 2 and MongoDB still needs owner choices (ADR-13); ADR-08 to ADR-11 stay reserved or planned for records on other branches. ／ データ層の変更は一つでなく二つの記録になる。PostgreSQL の移行は証明つきで提案し（ADR-12）、Elasticsearch 2 と MongoDB の撤去はまだ所有者の選択を要する（ADR-13）。ADR-08〜ADR-11 は他のブランチの記録のために予約・予定のまま残す。 0.4 — 2026-11-06. **Learned:** a cold start turned an emulation cost into an architecture decision (ADR-07): EOL amd64-only middleware is built natively. ／ コールドスタートがエミュレーションのコストを設計判断（ADR-07）に変えた。amd64 のみの EOL ミドルウェアはネイティブで組む。 0.3 — 2026-11-06. **Learned:** a repository that cannot be built alone because its Dockerfile belongs to another tree is an architecture decision (overlay, not service) and needs its own record (ADR-06). ／ Dockerfile が別の木に属していて単独で組めないリポジトリは、サービスでなくオーバーレイという設計上の決定であり、独自の記録（ADR-06）が要ると分かった。 0.2 — 2026-11-06. **Learned:** a test fixture that logs people in is an architecture choice like any other service, so it needs its own record (ADR-05) rather than a note in a template. ／ 人をログインさせる試験用の部品も他のサービスと同じ設計上の選択であり、テンプレートの注記ではなく独自の記録（ADR-05）が要ると分かった。 0.1 — 2026-11-05. **Learned:** writing decisions as records turned subjective choices into objective, reviewable documents that explain the reasoning behind each rule. ／ 決定を記録として残すことで、主観的な選択を客観的でレビュー可能な文書にし、各ルールの背後にある推論を説明できるようになったと分かった。 -- Normative for: [ADR-01 Integration Repository Placement](./ADR-01_Integration_Repository_Placement.md) through [ADR-07 Native Builds of EOL Middleware](./ADR-07_Native_Builds_of_EOL_Middleware.md), and [ADR-12 PostgreSQL 18 and Data Directory Upgrade](./ADR-12_PostgreSQL_18_and_Data_Directory_Upgrade.md) and [ADR-13 Retiring Elasticsearch 2 and MongoDB](./ADR-13_Retiring_Elasticsearch_2_and_MongoDB.md); ADR-08 to ADR-11 are reserved or planned. +- Normative for: [ADR-01 Integration Repository Placement](./ADR-01_Integration_Repository_Placement.md) through [ADR-08 Per-OS Installer Entry Points](./ADR-08_Per_OS_Installer_Entry_Points.md), and [ADR-12 PostgreSQL 18 and Data Directory Upgrade](./ADR-12_PostgreSQL_18_and_Data_Directory_Upgrade.md) and [ADR-13 Retiring Elasticsearch 2 and MongoDB](./ADR-13_Retiring_Elasticsearch_2_and_MongoDB.md); ADR-08 to ADR-11 are reserved or planned. - Related: [../README.md](../README.md) · [policy/](../policy/README.md) · [spec/](../spec/README.md) · [reference/REF-01_Glossary.md](./../reference/REF-01_Glossary.md) --- @@ -33,6 +34,7 @@ A *decision record* (ADR) explains why we chose one way and not another. This fo | ADR-05 | [ADR-05_Test_IdP_for_Shibboleth.md](./ADR-05_Test_IdP_for_Shibboleth.md) | Bundled test IdP for `--auth shibboleth`: our own SimpleSAMLphp image on `php:8.3-apache`, mock attributes prefixed `mock-`. | | ADR-06 | [ADR-06_Wiki_Frontend_Overlay.md](./ADR-06_Wiki_Frontend_Overlay.md) | wiki-frontend is an overlay of osf.io (`overlay_of`): a bundle-builder image from the merged tree, tagged `-`, run once by `wiki_assets` into the volume `wiki_static`; no `wiki` server. | | ADR-07 | [ADR-07_Native_Builds_of_EOL_Middleware.md](./ADR-07_Native_Builds_of_EOL_Middleware.md) | EOL middleware that only ships amd64 images is built natively for both architectures from the official artefact on a digest-pinned base (`image_build`, MC-15); first Elasticsearch 2.4.6, which took 38 minutes to start under emulation and 5 seconds natively (real install run 19). | Proposed 2026-11-06 | +| ADR-08 | [ADR-08_Per_OS_Installer_Entry_Points.md](./ADR-08_Per_OS_Installer_Entry_Points.md) | Thin, consent-based prerequisite scripts for Ubuntu, macOS and Windows that all end at `install.sh`; `sudo` is allowed there only. | Proposed 2026-11-10 | | ADR-12 | [ADR-12_PostgreSQL_18_and_Data_Directory_Upgrade.md](./ADR-12_PostgreSQL_18_and_Data_Directory_Upgrade.md) | PostgreSQL 9.6 → 18.6 (digest-pinned); md5 host auth until CAS has a SCRAM-capable JDBC driver; `grdm-dev db upgrade` dumps and restores with verification and keeps the old directory, `db rollback` puts it back, `up` refuses a mismatched major (exit 15). Status: Proposed. | | ADR-13 | [ADR-13_Retiring_Elasticsearch_2_and_MongoDB.md](./ADR-13_Retiring_Elasticsearch_2_and_MongoDB.md) | Measured: osf.io search does not run on ES 6.8 or OpenSearch; search off needs a ~50-line patch; mongo serves only sharejs, whose file-page co-editing is what retirement removes. Retirement steps; owner decisions of 2026-11-10: ES 8 port with full-text search, file-page editor on y-websocket. Status: Proposed. | @@ -40,7 +42,6 @@ Reserved or planned numbers (no record in this tree yet): | Number | Topic | |---|---| -| ADR-08 | Reserved: Per-OS Installer Entry Points (on a separate branch) | | ADR-09 | Planned: repository branching strategy | | ADR-10 | Planned: tool version policy (Docker, Python, Node.js) | | ADR-11 | Planned: open source license compatibility | @@ -82,6 +83,7 @@ flowchart TD | ADR-05 | [ADR-05_Test_IdP_for_Shibboleth.md](./ADR-05_Test_IdP_for_Shibboleth.md) | `--auth shibboleth` 用の同梱テスト IdP：`php:8.3-apache` 上の自前 SimpleSAMLphp イメージ、`mock-` 接頭辞の模擬属性。 | | ADR-06 | [ADR-06_Wiki_Frontend_Overlay.md](./ADR-06_Wiki_Frontend_Overlay.md) | wiki-frontend は osf.io のオーバーレイ（`overlay_of`）：重ねた木から束を組むイメージ（タグ `-`）を `wiki_assets` が一度動かしてボリューム `wiki_static` へ写す。`wiki` サーバは無い。 | | ADR-07 | [ADR-07_Native_Builds_of_EOL_Middleware.md](./ADR-07_Native_Builds_of_EOL_Middleware.md) | amd64 のイメージしか無い EOL ミドルウェアは、公式成果物からダイジェスト固定のベース上に両アーキテクチャ向けにネイティブで組む（`image_build`、MC-15）。まず Elasticsearch 2.4.6。エミュレーションでは起動に 38 分、ネイティブでは 5 秒（実機第 19 回）。 | 提案 2026-11-06 | +| ADR-08 | [ADR-08_Per_OS_Installer_Entry_Points.md](./ADR-08_Per_OS_Installer_Entry_Points.md) | Ubuntu・macOS・Windows の同意に基づく薄い前提条件スクリプト。どれも `install.sh` で終わる。`sudo` はここだけで認める。 | 提案 2026-11-10 | | ADR-12 | [ADR-12_PostgreSQL_18_and_Data_Directory_Upgrade.md](./ADR-12_PostgreSQL_18_and_Data_Directory_Upgrade.md) | PostgreSQL 9.6 → 18.6（ダイジェスト固定）。CAS が SCRAM に対応した JDBC ドライバーを持つまでホスト認証は md5。`grdm-dev db upgrade` は照合つきでダンプと復元を行い旧ディレクトリを残し、`db rollback` はそれを戻し、`up` は版の不一致を拒む（終了 15）。ステータス: 提案中。 | | ADR-13 | [ADR-13_Retiring_Elasticsearch_2_and_MongoDB.md](./ADR-13_Retiring_Elasticsearch_2_and_MongoDB.md) | 測定: osf.io の検索は ES 6.8 でも OpenSearch でも動かない。検索を切るには約 50 行のパッチが要る。mongo は sharejs にしか使われず、退役で失うのはそのファイルページの共同編集である。退役の手順。2026-11-10 の所有者の判断: 全文検索を保つ ES 8 への移植、ファイルページのエディタの y-websocket への移行。ステータス: 提案中。 | @@ -89,7 +91,6 @@ flowchart TD | 番号 | トピック | |---|---| -| ADR-08 | 予約: OS ごとのインストーラーの入口（別ブランチ） | | ADR-09 | 予定: リポジトリのブランチ戦略 | | ADR-10 | 予定: 道具版方針（Docker、Python、Node.js） | | ADR-11 | 予定: オープンソースライセンスの互換性 | diff --git a/docs/demo/21_core_install_ubuntu_arm64_lima.md b/docs/demo/21_core_install_ubuntu_arm64_lima.md new file mode 100644 index 0000000..98680c7 --- /dev/null +++ b/docs/demo/21_core_install_ubuntu_arm64_lima.md @@ -0,0 +1,475 @@ +# DEMO-21 Core Install on Ubuntu 24.04 arm64 / Ubuntu 24.04 arm64 でのコア導入記録 + +> **Summary:** What actually happened when `installer/install-ubuntu.sh --yes --profile core` was run on a fresh Ubuntu 24.04 arm64 virtual machine, as an ordinary user whose uid is not 1000. One entry per run; nothing is edited after the fact. +> **概要:** 新しい Ubuntu 24.04 arm64 の仮想マシンで、uid が 1000 でない一般利用者として `installer/install-ubuntu.sh --yes --profile core` を実行したときに実際に起きたこと。実行ごとに 1 節。後から書き換えない。 + +**Jump:** [English](#english) · [日本語](#japanese) + +- Version / 版: 0.4 — 2026-11-10. **Learned:** fix 4b changed files that cannot be published yet: `installer-prereq-lint.yml` and its helper wait on branch `feat/per-os-installers` for a token with the `workflow` scope, so on develop Semgrep has no workflow of this branch to read. ／ 修正 4b が変えたファイルはまだ公開できない。`installer-prereq-lint.yml` とその補助は `workflow` スコープのトークンをブランチ `feat/per-os-installers` で待つので、develop では Semgrep が読むこのブランチのワークフローは無い。 + 0.3 — 2026-11-10. **Learned:** the first Linux run of `grdm-dev db upgrade`, as uid 2001, moved PostgreSQL 9.6 to 18 in 12 seconds with 531 objects verified, and with develop's acceptance fixes the core suite passes 12 of 12 on Linux arm64 too; and the setup record had understated an exposure: the host's `~/.config/gh` was readable in the VM until the home mount was removed, so `install-ubuntu.sh` now stops when it finds credentials on a shared folder. ／ uid 2001 での `grdm-dev db upgrade` の Linux での最初の実行は、PostgreSQL 9.6 を 18 へ 12 秒で移し、531 個の対象を照合した。develop の受入試験の修正を入れると、コアの試験は Linux arm64 でも 12 件中 12 件合格する。また準備の記録は露出を小さく書いていた。ホームの mount を外すまで、ホストの `~/.config/gh` は VM から読めた。そこで `install-ubuntu.sh` は共有フォルダに資格情報を見つけると止まる。 + 0.2 — 2026-11-10. **Learned:** a Linux host user whose uid is not 1000 is the case that finds ownership bugs the Mac hides: after the volumes (PR #36) it was the ZAP report directory, and only the full sequence through `scan` and `report` (run 4, exit 0) shows that none is left on this host. ／ uid が 1000 でない Linux のホスト利用者こそ、Mac が隠す所有者の不具合を見つける。ボリューム（PR #36）の次は ZAP の報告ディレクトリだった。この機械に残りが無いと示せるのは、`scan` と `report` まで通した一連の実行（第 4 回、終了 0）だけである。 + 0.1 — 2026-11-10. **Learned:** the dry runs of [ADR-08](../decisions/ADR-08_Per_OS_Installer_Entry_Points.md) could not show three Linux facts that a real run did: Ubuntu's own amd64 emulator crashes inside a long build, QEMU makes the first ember build twice as slow as on the Mac, and the cloud image lacks a library that the acceptance browser's driver needs. ／ [ADR-08](../decisions/ADR-08_Per_OS_Installer_Entry_Points.md) の dry run では分からず、実際の実行で分かった Linux の事実が三つあった。Ubuntu 自身の amd64 エミュレーターは長いビルドの中で落ち、QEMU では ember の初回ビルドが Mac の 2 倍遅く、クラウドイメージには受入試験のブラウザのドライバーが要るライブラリが無い。 +- Normative source: [SPEC-02 Installer CLI](../spec/02_installer_cli.md) · [SPEC-04 Acceptance Tests](../spec/04_acceptance_tests.md) +- Related: [docs/demo/README.md](README.md) · [DEMO-20 Core install on Mac Studio](20_core_install_mac_studio.md) · [ADR-08 Per-OS installer entry points](../decisions/ADR-08_Per_OS_Installer_Entry_Points.md) + +--- + + +## English + +[→ 日本語版へ](#japanese) + +### Setup — 2026-11-10, branch `feat/per-os-installers` rebased onto develop 03c7358 + +The host is the Mac Studio of [DEMO-20](20_core_install_mac_studio.md) (128 GiB). The test machine is a Lima virtual machine (Lima runs Linux inside macOS with Apple's virtualization; a *virtual machine* is a whole computer simulated by software) named `grdm-ubuntu2404`: + +```toon +vm: + name: grdm-ubuntu2404 + tool: limactl 2.2.1, vmType vz, no Rosetta + image: ubuntu-24.04-server-cloudimg-arm64 (release-20260926), Ubuntu 24.04.5 LTS + cpus: 8 + memory: 40 GiB + disk: 120 GiB + user: "grdm, uid 2001 gid 2001, passwordless sudo (a developer with sudo, not root)" + checkout: "/home/grdm/RDM-developer-guide on the VM's own disk (cloned from a git bundle, so Linux owners and modes are real)" + shared_from_host: "/mnt/mirrors only, read-only: bare mirrors of the six integration repositories (git clone --mirror on the host)" + git_config: "url.file:///mnt/mirrors/.git.insteadOf https://github.com/RCOSDP/.git; no token in the VM" +``` + +> **Warning:** Lima's Ubuntu template also mounts the host home directory read-only, even when the configuration lists its own mounts, because the template's list is merged in. That would have shown the host's GitHub credentials inside the VM. The home mount was removed from the instance configuration before any repository was copied, and the VM restarted with only `/mnt/mirrors`. + +On arm64 (the CPU type of Apple Silicon and many servers; see [REF-01](../reference/REF-01_Glossary.md)), Linux runs the two amd64-only components (amd64 is the CPU type of most PCs and cloud servers; see [REF-01](../reference/REF-01_Glossary.md)), modular-file-renderer (the file preview service) and ember-osf-web (the newer web pages of GRDM), under emulation (running a program built for one kind of CPU on another, slower but working). Docker (the tool that builds images and runs containers; see [REF-01](../reference/REF-01_Glossary.md)) Desktop on the Mac emulates with Rosetta; plain Linux uses QEMU (a free CPU emulator) registered in the kernel's binfmt_misc table. + +### Run 1 — fresh VM, the one command + +```toon +run1: + command: ./installer/install-ubuntu.sh --yes --profile core + started: 2026-11-10T11:24+09:00 + result: exit 30 (image build failed) after 2868 s + prerequisites[6]{step,result}: + "apt: python3-venv",installed + "Docker apt key",fingerprint 9DC858229FC7DD38854AE2D88D81803C0EBFCD88 verified + "Docker Engine",29.9.3 with Compose 5.7.0 and buildx + "amd64 emulation","apt qemu-user-static 1:8.2.2+ds-0ubuntu1.18 (flags OPF)" + "trivy 0.75.0, gitleaks 8.21.2",SHA-256 verified into ~/.local/bin + "docker group","usermod, then this run continued under sg docker" + steps[5]{step,code,status}: + doctor,0,"ok: ubuntu 24.04 arm64, 39.1 GiB, 111.9 GiB free, official, no problems" + plan,0,ok + "secrets init",0,ok + "compose render",0,ok + build,30,failed + images[3]{component,platform,status,note}: + osf.io,linux/arm64,built,"about 30 min: pandas and scipy compiled from source on Alpine" + waterbutler,linux/arm64,built,"" + modular-file-renderer,linux/amd64,build-failed,"libgvc6 post-install: x86_64-binfmt-P: QEMU internal SIGSEGV, exit 139" +``` + +What this proves: + +- The prerequisite script works on a fresh Ubuntu 24.04 arm64 machine: signed Docker repository, pinned scanners, the docker group question, and the hand-over to `install.sh` inside `sg docker`. +- The private repositories were read from read-only mirrors through `insteadOf`; `doctor`'s `git ls-remote` check passed without any token. +- **Failure 1:** Ubuntu 24.04's `qemu-user-static` crashes inside modular-file-renderer's build, 48 minutes into the run. + +Isolation test, one minute each: `apt-get install libgvc6` in a `debian:buster-slim` amd64 container. + +```toon +qemu_isolation[4]{emulator,result}: + "Ubuntu qemu-user-static 8.2.2 (apt)","QEMU internal SIGSEGV, apt exit 100" + "tonistiigi/binfmt qemu-v8.1.5",installed + "tonistiigi/binfmt qemu-v9.2.2",installed + "tonistiigi/binfmt qemu-v10.2.3-68",installed +``` + +The crash belongs to Ubuntu's 8.2.2 build, not to a QEMU version range, so only that build is refused. + +**Fix 1** (`fix(installer)`): `install-ubuntu.sh` no longer installs `qemu-user-static`; it registers QEMU from `tonistiigi/binfmt:qemu-v10.2.3-68`, pinned by digest, and replaces a registered Ubuntu 8.2.2 handler (`--uninstall qemu-x86_64`, then `--install amd64`). `doctor` reads the registered interpreter's `--version` and stops with exit 13 before the build when it is the measured-bad build, printing the replacement command. + +### Run 2 — same VM, QEMU 10.2.3 registered by hand + +```toon +run2: + command: ./installer/install.sh --yes --profile core + result: exit 40 (health) after 3929 s + steps[6]{step,code,status}: + "secrets init",0,"skipped (runtime/.env exists)" + doctor,0,ok + plan,0,ok + "compose render",0,ok + build,0,"ok: 12 components, modular-file-renderer built under QEMU 10.2.3" + up,40,"not healthy after 1800s: ember_osf_web" + ownership: "volume_owner_init, volume_access_check_1000 and volume_access_check_999 completed with host uid 2001" + ember_first_build: "container start 03:43:05Z, port 4200 answered 04:27:20Z: 44 min 15 s, one core busy" + health_after_ember: "grdm-dev health exit 0, 24 services ok" +``` + +What this proves: + +- The ownership handling of PR (pull request: a proposed change waiting for review; see [REF-01](../reference/REF-01_Glossary.md)) #36 works for a host user whose uid is not 1000: every writer could use its volume, and the volumes stay owned by the container uids. +- **Failure 2:** QEMU builds ember about twice as slowly as Rosetta (13–25 minutes in DEMO-20). The single-threaded build cannot use more CPUs, so `up`'s 1800 s wait stopped an install that would have become healthy. + +**Fix 2** (`fix(installer)`): on Linux arm64, when the profile has amd64 services, `up` waits 5400 s by default (about twice the measurement); `--timeout` still overrides it. macOS and amd64 keep 1800 s. + +### Acceptance attempt 1 — stack of run 2 + +`grdm-dev health` exited 0. Installing the browser failed before any test: `playwright install --with-deps chromium` exited 127 because the Node.js driver of Playwright (a robot that drives a web browser to test a website; see [REF-01](../reference/REF-01_Glossary.md)) could not load `libatomic.so.1`, which the Ubuntu 24.04 cloud image does not contain. `grdm-dev accept` then exited 2 (no Chromium). + +**Failure 3, fix 3** (`fix(installer)`): `install-ubuntu.sh` adds `libatomic1` to its apt packages when `GRDM_WITH_ACCEPT=1`, and on Linux `grdm-dev accept` prints `sudo apt-get install -y libatomic1` before, and `playwright install-deps chromium` after, the existing install command. With `libatomic1` installed by hand, `playwright install --with-deps chromium` exited 0. + +### Acceptance run — stack of run 2 (osf.io 389415a4) + +| Run ID | Scope | Result | +|---|---|---| +| [`acc-20261110T043053Z-5c5dae`](runs/acc-20261110T043053Z-5c5dae/results.toon) | Core C1–C6 | 5 of 12 passed | + +C1 and C3 pass in both cases, C2's rejection passes. The other seven fail for the reasons DEMO-20 recorded on develop (project-name selector, wiki bundle wiring, preview, admin path, and ShareJS assets on port 7007); they are fixed on the separate branch `fix/accept-c2-c6`, which this branch does not contain. No failure here is specific to Linux. + +### Run 3 — the fixed scripts from the broken state, cold start + +Before the run: the checkout moved to commit 48859d7 (fixes 1–3), Ubuntu's 8.2.2 handler was registered again (`systemctl restart systemd-binfmt`), `libatomic1` was removed, and `grdm-dev reset --yes` emptied `runtime/volumes/` (exit 0; the files belonged to container uids). With Ubuntu's handler registered, `grdm-dev doctor` exited 13 and printed the replacement command. + +```toon +run3: + command: GRDM_WITH_ACCEPT=1 ./installer/install-ubuntu.sh --yes --profile core + started: 2026-11-10T04:40:11Z + result: exit 51 (scanner incomplete) after 2828 s + plan[2]: + "sudo: install apt packages: libatomic1" + "sudo: replace the registered amd64 emulator (qemu-x86_64 version 8.2.2 ...) with QEMU from tonistiigi/binfmt:qemu-v10.2.3-68@sha256:400a4873..." + steps[8]{step,code,status}: + "secrets init",0,"skipped (runtime/.env exists)" + doctor,0,ok + plan,0,ok + "compose render",0,ok + build,0,"ok (cached)" + up,0,"ok within the 5400 s wait; ember's first build again about 40 min" + "health --network",0,"ok, every port on 127.0.0.1" + scan,51,"semgrep incomplete; zap incomplete" +``` + +What this proves: fixes 1–3 work end to end, and the reset of PR #36 works for uid 2001. Two new failures stop `scan`: + +- **Failure 4a:** ZAP (the OWASP website scanner, which runs as uid 1000 in its container) could not write its report into `runtime/reports/`, owned by uid 2001; `zap-baseline` exited 0 without a report for all three targets. Docker Desktop's bind mounts ignore owners, so the Mac never showed it. +- **Failure 4b:** Semgrep (a tool that reads source code looking for dangerous patterns; see [REF-01](../reference/REF-01_Glossary.md)) 1.184.0 reported four "Internal matching error"s on `.github/workflows/installer-prereq-lint.yml`, a file this branch adds. Semgrep reads `run:` blocks as shell; the inline PowerShell of the Windows job breaks its engine (removing that job alone made the errors disappear). + +**Fix 4** (`fix(installer)`, commit 0c46a82): each ZAP target gets its own work directory under `runtime/reports/`, mode 0777 inside the 0700 `runtime/`, removed after the masked copy is saved; the PowerShell parse moved to `.github/scripts/Test-PowerShellSyntax.ps1`, and each PowerShell `run:` is a one-line call. Semgrep's `p/default` and `p/github-actions` then report no error on the workflows. + +### Acceptance run — stack of run 3 + +| Run ID | Scope | Result | +|---|---|---| +| [`acc-20261110T052932Z-f1da54`](runs/acc-20261110T052932Z-f1da54/results.toon) | Core C1–C6 | 5 of 12 passed (same cases as above) | + +Chromium was installed with the command that `grdm-dev accept` now prints on Linux: `playwright install chromium` exited 0 and `playwright install-deps chromium` exited 0. + +### Run 4 — commit 0c46a82, the one command: **exit 0** + +```toon +run4: + command: GRDM_WITH_ACCEPT=1 ./installer/install-ubuntu.sh --yes --profile core + started: 2026-11-10T05:34:32Z + result: exit 0 after 204 s (images cached, volumes kept from run 3) + prerequisites: "All prerequisites are already present; nothing to change." + steps[9]{step,code,status}: + "secrets init",0,"skipped (runtime/.env exists)" + doctor,0,ok + plan,0,ok + "compose render",0,ok + build,0,ok + up,0,ok + "health --network",0,ok + scan,0,"ok: trivy 23 targets, semgrep 4 packs, gitleaks, zap 3 targets; gate report-only, pass" + report,0,ok + url: "http://127.0.0.1:5000/" + health_after: "grdm-dev health exit 0" +``` + +Findings by severity (report-only at M1, [ADR-03](../decisions/ADR-03_Staged_Security_Gate.md)): 817 critical, 4355 high, 4392 medium, 2164 low, 5 info. + +### Correction to the setup — the home mount did expose the credentials + +The warning in the setup says the home mount "would have shown" the host's GitHub credentials. That understates it. The VM's first boot mounted `/Users/komiyama` read-only, and a directory listing confirmed it, so `~/.config/gh/hosts.yml` was readable inside the VM from the first boot until the mount was removed from `lima.yaml` and the VM restarted, a few minutes later. Nothing read the file, and no repository or tool had been copied into the VM by then. Since commit 45b49be, `install-ubuntu.sh` stops with exit 10 when a shared folder (virtiofs, 9p, sshfs and similar) holds `.config/gh/hosts.yml`, `.git-credentials`, `.netrc` or `.docker/config.json`. The environment guide now has the Lima steps: delete the `"~"` mount before the first start, then check `mount`. + +### Run 5 — rebased onto develop 50740b1 (PR #40–#43), database upgrade + +PostgreSQL (the main database where GRDM keeps its records; see [REF-01](../reference/REF-01_Glossary.md)) moved from 9.6 to 18.6 on develop (PR #42, [ADR-12](../decisions/ADR-12_PostgreSQL_18_and_Data_Directory_Upgrade.md)), so the existing data had to be upgraded before `up`. The branch was rebased twice; the commits named above became 8538324 (fixes 1–3, was 48859d7) and 05ff502 (fix 4, was 0c46a82). The host mirrors were refreshed with `git fetch`. The manifest now pins osf.io 8fc45de4; WaterButler (45e97a51) and modular-file-renderer (749e58d3) stay at their pinned commits. + +```toon +run5: + head: 45b49be + steps[10]{step,code,seconds,note}: + down,0,3,"" + doctor,0,0,"" + "compose render",0,1,"" + build,0,14,"images already built from the same pins" + "db status",0,10,"data 9.6, image 18 (postgres:18.6, digest pinned): upgrade-needed" + "db upgrade",0,12,"see below" + "db status",0,0,"data 18: current" + up,0,296,"healthy; ember's built pages were kept in their volume" + "secrets rotate",0,293,"every secret regenerated, containers recreated, healthy" + health,0,0,"" + db_upgrade: + user: "grdm, uid 2001 (the data directory belongs to uid 999)" + steps: "start-old, count-old, dump, stop-old, init-new, restore, verify, stop-new, swap: all ok" + dump_bytes: 4546929 + verified: "531 objects compared (row counts per table, sequences, extensions) in 3 databases (cas, osf, postgres) and 1 role" + kept: "runtime/volumes/postgres_data.pg9.6-20261110T055459Z" +``` + +### Acceptance run — stack of run 5: **12 of 12** + +| Run ID | Scope | Result | +|---|---|---| +| [`acc-20261110T060511Z-80ee06`](runs/acc-20261110T060511Z-80ee06/results.toon) | Core C1–C6 | **12 of 12 passed**, exit 0 | + +### Run 6 — the one command on the same stack + +`GRDM_WITH_ACCEPT=1 ./installer/install-ubuntu.sh --yes --profile core` exited 0 after 273 s: every step through `scan` (Trivy, Semgrep, gitleaks and ZAP all `ok`) and `report`. `grdm-dev health` exited 0 afterwards. Findings by severity: 808 critical, 4328 high, 4470 medium, 2267 low, 6 info. + +### What remains + +- **Ubuntu 24.04 amd64:** not run. No emulation is needed there, so failures 1, 2 and the arm64 part of 3 do not apply; failure 4a (ZAP as uid 1000) and the PR #36 ownership path do, and need a clean amd64 machine. +- **WSL2** (a way to run Linux inside Windows; see [REF-01](../reference/REF-01_Glossary.md)): not run. Docker comes from Docker Desktop, whose bind mounts behave like the Mac's, so 4a is not expected there; the checkout must be inside WSL. It needs Windows 11 hardware ([ADR-08 §5](../decisions/ADR-08_Per_OS_Installer_Entry_Points.md)). +- **Emulation after a reboot:** the QEMU registration is not persistent. While Ubuntu's `qemu-user-static` is installed, its 8.2.2 handler returns at boot; `doctor` then stops with exit 13, and running `install-ubuntu.sh` again replaces it. +- **Acceptance:** done in run 5 (12 of 12) after develop took the fixes of PR #40. +- **CI workflows:** `installer-prereq-lint.yml`, `installer-full.yml` and the helper of fix 4b, `.github/scripts/Test-PowerShellSyntax.ps1`, are pending on branch `feat/per-os-installers` until they are pushed with a token that has the `workflow` scope. Everything else of this record is on develop. +- **Cold start time:** with images already built, `up` from empty volumes took about 40 minutes on this VM, nearly all of it ember's first build under QEMU. Building the images from nothing took about 60 minutes more (runs 1 and 2). + +--- + + +## 日本語 + +[→ To English](#english) + +### 準備 — 2026-11-10、develop 03c7358 へ rebase したブランチ `feat/per-os-installers` + +ホストは [DEMO-20](20_core_install_mac_studio.md) の Mac Studio（128 GiB）。試験機は Lima の仮想マシン `grdm-ubuntu2404` である（Lima は Apple の仮想化機能で macOS の中に Linux を動かす道具。*仮想マシン* はソフトウェアで模した一台のコンピュータ）： + +```toon +vm: + name: grdm-ubuntu2404 + tool: limactl 2.2.1、vmType vz、Rosetta なし + image: ubuntu-24.04-server-cloudimg-arm64（release-20260926）、Ubuntu 24.04.5 LTS + cpus: 8 + memory: 40 GiB + disk: 120 GiB + user: "grdm、uid 2001・gid 2001、パスワードなしの sudo（root ではなく sudo を持つ開発者）" + checkout: "VM 自身のディスク上の /home/grdm/RDM-developer-guide（git bundle から clone。Linux の所有者とモードが本物になる）" + shared_from_host: "/mnt/mirrors だけを読み取り専用で共有: 統合リポジトリ 6 本の bare ミラー（ホストで git clone --mirror）" + git_config: "url.file:///mnt/mirrors/.git.insteadOf https://github.com/RCOSDP/.git。VM にトークンは置かない" +``` + +> **Warning:** Lima の Ubuntu テンプレートは、設定に独自の mount を書いてもテンプレート側の一覧が併合されるため、ホストのホームディレクトリを読み取り専用で mount する。そのままではホストの GitHub 資格情報が VM から見えてしまう。リポジトリを入れる前にインスタンスの設定からホームの mount を外し、`/mnt/mirrors` だけで VM を再起動した。 + +arm64（Apple Silicon や多くのサーバーの CPU の種類。[REF-01](../reference/REF-01_Glossary.md)）の Linux では、amd64 専用の部品 modular-file-renderer と ember-osf-web をエミュレーション（ある CPU 向けのプログラムを別の CPU で動かすこと。遅いが動く）で動かす。Mac の Docker Desktop は Rosetta でエミュレーションするが、素の Linux はカーネルの binfmt_misc 表に登録した QEMU（無償の CPU エミュレーター）を使う。 + +### 第 1 回 — 新しい VM で一つのコマンド + +```toon +run1: + command: ./installer/install-ubuntu.sh --yes --profile core + started: 2026-11-10T11:24+09:00 + result: 2868 秒後に終了 30（イメージのビルド失敗） + prerequisites[6]{step,result}: + "apt: python3-venv",導入 + "Docker の apt 鍵",指紋 9DC858229FC7DD38854AE2D88D81803C0EBFCD88 を検証 + "Docker Engine",29.9.3、Compose 5.7.0 と buildx + "amd64 エミュレーション","apt の qemu-user-static 1:8.2.2+ds-0ubuntu1.18（flags OPF）" + "trivy 0.75.0、gitleaks 8.21.2",SHA-256 を検証して ~/.local/bin へ + "docker グループ","usermod の後、この実行は sg docker の下で続行" + steps[5]{step,code,status}: + doctor,0,"ok: ubuntu 24.04 arm64、39.1 GiB、空き 111.9 GiB、公式、問題なし" + plan,0,ok + "secrets init",0,ok + "compose render",0,ok + build,30,failed + images[3]{component,platform,status,note}: + osf.io,linux/arm64,built,"約 30 分: Alpine で pandas と scipy をソースから組む" + waterbutler,linux/arm64,built,"" + modular-file-renderer,linux/amd64,build-failed,"libgvc6 の post-install: x86_64-binfmt-P: QEMU internal SIGSEGV、終了 139" +``` + +分かったこと: + +- 前提条件スクリプトは新しい Ubuntu 24.04 arm64 で動く。署名付きの Docker リポジトリ、固定したスキャナー、docker グループの確認、`sg docker` の中での `install.sh` への引き継ぎ。 +- 非公開リポジトリは `insteadOf` で読み取り専用のミラーから読めた。`doctor` の `git ls-remote` の確認はトークン無しで通った。 +- **失敗 1:** Ubuntu 24.04 の `qemu-user-static` が、実行開始から 48 分後、modular-file-renderer のビルドの中で落ちる。 + +切り分け（各 1 分）: amd64 の `debian:buster-slim` コンテナで `apt-get install libgvc6`。 + +```toon +qemu_isolation[4]{emulator,result}: + "Ubuntu の qemu-user-static 8.2.2（apt）","QEMU internal SIGSEGV、apt 終了 100" + "tonistiigi/binfmt qemu-v8.1.5",導入できた + "tonistiigi/binfmt qemu-v9.2.2",導入できた + "tonistiigi/binfmt qemu-v10.2.3-68",導入できた +``` + +落ちるのは QEMU の版の範囲ではなく Ubuntu の 8.2.2 のビルドなので、拒むのはそのビルドだけにした。 + +**修正 1**（`fix(installer)`）: `install-ubuntu.sh` は `qemu-user-static` を入れず、ダイジェストで固定した `tonistiigi/binfmt:qemu-v10.2.3-68` から QEMU を登録し、登録済みの Ubuntu 8.2.2 は置き換える（`--uninstall qemu-x86_64` の後に `--install amd64`）。`doctor` は登録済みインタープリターの `--version` を読み、落ちると実測したビルドならビルドの前に終了 13 で止まり、置き換えのコマンドを表示する。 + +### 第 2 回 — 同じ VM、QEMU 10.2.3 を手で登録 + +```toon +run2: + command: ./installer/install.sh --yes --profile core + result: 3929 秒後に終了 40（health） + steps[6]{step,code,status}: + "secrets init",0,"skipped（runtime/.env あり）" + doctor,0,ok + plan,0,ok + "compose render",0,ok + build,0,"ok: 12 部品。modular-file-renderer は QEMU 10.2.3 で組めた" + up,40,"1800 秒後も ember_osf_web が健康でない" + ownership: "volume_owner_init、volume_access_check_1000、volume_access_check_999 がホストの uid 2001 で完了" + ember_first_build: "コンテナ起動 03:43:05Z、4200 番が応答 04:27:20Z: 44 分 15 秒、使うのは 1 コア" + health_after_ember: "grdm-dev health 終了 0、24 サービスが ok" +``` + +分かったこと: + +- PR #36 の所有者処理は uid が 1000 でないホストの利用者でも働く。書き手はすべて自分のボリュームを使え、ボリュームはコンテナの uid の所有のままである。 +- **失敗 2:** QEMU は ember の組み立てに Rosetta（DEMO-20 で 13〜25 分）のおよそ 2 倍かかる。単一スレッドのビルドなので CPU を増やしても速くならず、`up` の 1800 秒の待ちが、健康になるはずの導入を止めた。 + +**修正 2**（`fix(installer)`）: Linux arm64 でプロファイルに amd64 のサービスがあるとき、`up` の既定の待ちは 5400 秒（実測のおよそ 2 倍）。`--timeout` で上書きできる。macOS と amd64 は 1800 秒のまま。 + +### 受入試験の試み 1 — 第 2 回のスタック + +`grdm-dev health` は終了 0。試験の前にブラウザの導入で失敗した。Playwright の Node.js ドライバーが `libatomic.so.1` を読めず、`playwright install --with-deps chromium` は終了 127 だった。Ubuntu 24.04 のクラウドイメージにはこのライブラリが無い。続く `grdm-dev accept` は終了 2（Chromium が無い）。 + +**失敗 3・修正 3**（`fix(installer)`）: `install-ubuntu.sh` は `GRDM_WITH_ACCEPT=1` のとき apt パッケージに `libatomic1` を加え、Linux の `grdm-dev accept` は既存の導入コマンドの前に `sudo apt-get install -y libatomic1`、後に `playwright install-deps chromium` を表示する。`libatomic1` を手で入れると `playwright install --with-deps chromium` は終了 0 だった。 + +### 受入試験の実行 — 第 2 回のスタック（osf.io 389415a4） + +| 実行 ID | 範囲 | 結果 | +|---|---|---| +| [`acc-20261110T043053Z-5c5dae`](runs/acc-20261110T043053Z-5c5dae/results.toon) | コア C1〜C6 | 12 件中 5 件合格 | + +C1 と C3 は両方のケースで合格し、C2 の拒否も合格した。残る 7 件の失敗は、DEMO-20 が develop で記録した理由と同じ（プロジェクト名のセレクタ、wiki バンドルの配線、プレビュー、管理画面のパス、7007 番の ShareJS の資産）で、このブランチが含まない別ブランチ `fix/accept-c2-c6` で直している。Linux に固有の失敗は無い。 + +### 第 3 回 — 壊れた状態から修正後のスクリプトで、コールドスタート + +実行前に、チェックアウトをコミット 48859d7（修正 1〜3）へ移し、Ubuntu の 8.2.2 を再登録し（`systemctl restart systemd-binfmt`）、`libatomic1` を削除し、`grdm-dev reset --yes` で `runtime/volumes/` を空にした（終了 0。ファイルはコンテナの uid の所有だった）。Ubuntu の登録がある状態で `grdm-dev doctor` は終了 13 となり、置き換えのコマンドを表示した。 + +```toon +run3: + command: GRDM_WITH_ACCEPT=1 ./installer/install-ubuntu.sh --yes --profile core + started: 2026-11-10T04:40:11Z + result: 2828 秒後に終了 51（スキャナーが不完全） + plan[2]: + "sudo: apt パッケージを入れる: libatomic1" + "sudo: 登録済みの amd64 エミュレーター（qemu-x86_64 version 8.2.2 ...）を tonistiigi/binfmt:qemu-v10.2.3-68@sha256:400a4873... の QEMU に置き換える" + steps[8]{step,code,status}: + "secrets init",0,"skipped（runtime/.env あり）" + doctor,0,ok + plan,0,ok + "compose render",0,ok + build,0,"ok（キャッシュ）" + up,0,"5400 秒の待ちの中で ok。ember の初回ビルドは今回も約 40 分" + "health --network",0,"ok。すべてのポートが 127.0.0.1" + scan,51,"semgrep が不完全、zap が不完全" +``` + +分かったこと: 修正 1〜3 は端から端まで働き、PR #36 の reset も uid 2001 で働く。新しい失敗が二つ `scan` を止めた。 + +- **失敗 4a:** ZAP（OWASP のウェブサイト検査器。コンテナの中では uid 1000 で動く）が、uid 2001 の所有の `runtime/reports/` に報告を書けず、`zap-baseline` は三つの対象すべてで報告なしに終了 0 だった。Docker Desktop の bind mount は所有者を無視するので、Mac では表に出なかった。 +- **失敗 4b:** Semgrep 1.184.0 が、このブランチで加えた `.github/workflows/installer-prereq-lint.yml` に「Internal matching error」を 4 件出した。Semgrep は `run:` をシェルとして読み、Windows のジョブのインラインの PowerShell でエンジンが壊れる（そのジョブだけを除くとエラーが消えた）。 + +**修正 4**（`fix(installer)`、コミット 0c46a82）: ZAP の対象ごとに `runtime/reports/` の下に専用の作業ディレクトリを作り、0700 の `runtime/` の中で 0777 にし、マスクした写しを保存した後に消す。PowerShell の構文確認は `.github/scripts/Test-PowerShellSyntax.ps1` に移し、PowerShell の `run:` は一行の呼び出しにした。その後、Semgrep の `p/default` と `p/github-actions` はワークフローにエラーを出さない。 + +### 受入試験の実行 — 第 3 回のスタック + +| 実行 ID | 範囲 | 結果 | +|---|---|---| +| [`acc-20261110T052932Z-f1da54`](runs/acc-20261110T052932Z-f1da54/results.toon) | コア C1〜C6 | 12 件中 5 件合格（上と同じケース） | + +Chromium は、Linux の `grdm-dev accept` が今は表示するコマンドで入れた。`playwright install chromium` は終了 0、`playwright install-deps chromium` も終了 0。 + +### 第 4 回 — コミット 0c46a82、一つのコマンド: **終了 0** + +```toon +run4: + command: GRDM_WITH_ACCEPT=1 ./installer/install-ubuntu.sh --yes --profile core + started: 2026-11-10T05:34:32Z + result: 204 秒後に終了 0（イメージはキャッシュ、ボリュームは第 3 回のまま） + prerequisites: "前提条件はすべて揃っています。変更するものはありません。" + steps[9]{step,code,status}: + "secrets init",0,"skipped（runtime/.env あり）" + doctor,0,ok + plan,0,ok + "compose render",0,ok + build,0,ok + up,0,ok + "health --network",0,ok + scan,0,"ok: trivy 23 対象、semgrep 4 パック、gitleaks、zap 3 対象。門は report-only で合格" + report,0,ok + url: "http://127.0.0.1:5000/" + health_after: "grdm-dev health 終了 0" +``` + +重大度ごとの検出数（M1 では報告のみ、[ADR-03](../decisions/ADR-03_Staged_Security_Gate.md)）: critical 817、high 4355、medium 4392、low 2164、info 5。 + +### 準備の訂正 — ホームの mount は資格情報を実際に見せていた + +準備の Warning は、ホームの mount がホストの GitHub 資格情報を「見せてしまうところだった」と書いたが、それは小さく書きすぎである。VM の最初の起動で `/Users/komiyama` が読み取り専用で mount され、ディレクトリの一覧で確かめた。最初の起動から、`lima.yaml` から mount を外して VM を再起動するまでの数分間、`~/.config/gh/hosts.yml` は VM の中から読めた。そのファイルを読んだものは無く、その時点で VM にはリポジトリも道具も入れていなかった。コミット 45b49be 以降、`install-ubuntu.sh` は共有フォルダ（virtiofs・9p・sshfs など）に `.config/gh/hosts.yml`・`.git-credentials`・`.netrc`・`.docker/config.json` があると終了 10 で止まる。環境ガイドには Lima の手順（最初の起動の前に `"~"` の mount を消し、`mount` で確かめる）を加えた。 + +### 第 5 回 — develop 50740b1（PR #40〜#43）へ rebase、PostgreSQL 18 + +develop で PostgreSQL（GRDM が記録を保つ主なデータベース。[REF-01](../reference/REF-01_Glossary.md)）が 9.6 から 18.6 に上がった（PR #42、[ADR-12](../decisions/ADR-12_PostgreSQL_18_and_Data_Directory_Upgrade.md)）ので、`up` の前に既存のデータを移す必要があった。ブランチを 2 度 rebase したので、上に書いたコミットは 8538324（修正 1〜3、旧 48859d7）と 05ff502（修正 4、旧 0c46a82）になった。ホストのミラーは `git fetch` で更新した。manifest は osf.io を 8fc45de4 に固定し、WaterButler（45e97a51）と modular-file-renderer（749e58d3）は固定したコミットのまま。 + +```toon +run5: + head: 45b49be + steps[10]{step,code,seconds,note}: + down,0,3,"" + doctor,0,0,"" + "compose render",0,1,"" + build,0,14,"同じ固定から組んだイメージが既にある" + "db status",0,10,"データ 9.6、イメージ 18（postgres:18.6、ダイジェスト固定）: upgrade-needed" + "db upgrade",0,12,"下を参照" + "db status",0,0,"データ 18: current" + up,0,296,"健康。ember の組み上がったページはボリュームに残っていた" + "secrets rotate",0,293,"秘密をすべて作り直し、コンテナを再作成して健康" + health,0,0,"" + db_upgrade: + user: "grdm、uid 2001（データディレクトリは uid 999 の所有）" + steps: "start-old、count-old、dump、stop-old、init-new、restore、verify、stop-new、swap: すべて ok" + dump_bytes: 4546929 + verified: "3 つのデータベース（cas、osf、postgres）と 1 つのロールで 531 個の対象（表ごとの行数、シーケンス、拡張）を照合" + kept: "runtime/volumes/postgres_data.pg9.6-20261110T055459Z" +``` + +### 受入試験の実行 — 第 5 回のスタック: **12 件中 12 件** + +| 実行 ID | 範囲 | 結果 | +|---|---|---| +| [`acc-20261110T060511Z-80ee06`](runs/acc-20261110T060511Z-80ee06/results.toon) | コア C1〜C6 | **12 件中 12 件合格**、終了 0 | + +### 第 6 回 — 同じスタックで一つのコマンド + +`GRDM_WITH_ACCEPT=1 ./installer/install-ubuntu.sh --yes --profile core` は 273 秒で終了 0。`scan`（Trivy・Semgrep・gitleaks・ZAP がすべて `ok`）と `report` まですべての段が通った。その後の `grdm-dev health` も終了 0。重大度ごとの検出数: critical 808、high 4328、medium 4470、low 2267、info 6。 + +### 残っていること + +- **Ubuntu 24.04 amd64:** 未実行。エミュレーションが要らないので、失敗 1・2 と 3 の arm64 の部分は当てはまらない。失敗 4a（uid 1000 の ZAP）と PR #36 の所有者処理は当てはまるので、まっさらな amd64 の機械が要る。 +- **WSL2:** 未実行。Docker は Docker Desktop から来て、その bind mount は Mac と同じく振る舞うので 4a は起きないと見込む。チェックアウトは WSL の中に置く。Windows 11 の実機が要る（[ADR-08 §5](../decisions/ADR-08_Per_OS_Installer_Entry_Points.md)）。 +- **再起動後のエミュレーション:** QEMU の登録は残らない。Ubuntu の `qemu-user-static` が入っている間は起動時に 8.2.2 が戻り、`doctor` が終了 13 で止まる。`install-ubuntu.sh` をもう一度実行すると置き換える。 +- **受入試験:** develop が PR #40 の修正を取り込んだ後、第 5 回で完了（12 件中 12 件）。 +- **CI のワークフロー:** `installer-prereq-lint.yml`・`installer-full.yml` と修正 4b の補助 `.github/scripts/Test-PowerShellSyntax.ps1` は、`workflow` スコープのトークンで push されるまでブランチ `feat/per-os-installers` で保留中。この記録のほかの部分はすべて develop にある。 +- **コールドスタートの時間:** イメージが組めている状態で、空のボリュームからの `up` はこの VM で約 40 分。ほぼすべてが QEMU での ember の初回ビルドである。何も無い状態からのイメージのビルドには、さらに約 60 分かかった（第 1・2 回）。 + +--- + +## Glossary / 用語 + +| Term | English note | 日本語注 | +|---|---|---| +| Virtual machine | A whole computer simulated by software, inside a real one. | 本物のコンピュータの中でソフトウェアが模した、一台のコンピュータ。 | +| Emulation | Running a program built for one kind of CPU on another kind, slower but working. | ある種類の CPU 向けに作ったプログラムを別の CPU で動かすこと。遅いが動く。 | +| QEMU | A free program that emulates a CPU; Linux uses it to run amd64 programs on arm64. | CPU を模す無償のプログラム。Linux は arm64 で amd64 のプログラムを動かすのに使う。 | +| binfmt_misc | The Linux table that says which program runs a file of a foreign CPU type. | 別の CPU 向けのファイルをどのプログラムで動かすかを決める Linux の表。 | +| Mirror | An identical copy of a repository, here read-only and made on the host. | リポジトリの同一の写し。ここではホストで作った読み取り専用のもの。 | diff --git a/docs/demo/runs/acc-20261110T043053Z-5c5dae/collector.json b/docs/demo/runs/acc-20261110T043053Z-5c5dae/collector.json new file mode 100644 index 0000000..e2c0fa4 --- /dev/null +++ b/docs/demo/runs/acc-20261110T043053Z-5c5dae/collector.json @@ -0,0 +1,134 @@ +[ + { + "nodeid": "test_c1_login.py::test_c1_login_success[chromium]", + "id": "C1", + "name": "Login via fakecas", + "case": "success", + "status": "passed", + "duration_ms": 6150, + "http_status": 200, + "evidence": null, + "known_external_blocked": 7 + }, + { + "nodeid": "test_c2_create_project.py::test_c2_create_project_success[chromium]", + "id": "C2", + "name": "Create project", + "case": "success", + "status": "failed", + "duration_ms": 34486, + "http_status": null, + "evidence": null, + "known_external_blocked": 7 + }, + { + "nodeid": "test_c2_create_project.py::test_c2_create_project_rejection[chromium]", + "id": "C2", + "name": "Create project", + "case": "rejection", + "status": "passed", + "duration_ms": 5448, + "http_status": 403, + "evidence": null, + "known_external_blocked": 8 + }, + { + "nodeid": "test_c3_file_upload.py::test_c3_file_upload_success[chromium]", + "id": "C3", + "name": "Upload and download file", + "case": "success", + "status": "passed", + "duration_ms": 8718, + "http_status": 200, + "evidence": null, + "known_external_blocked": 4 + }, + { + "nodeid": "test_c3_file_upload.py::test_c3_file_upload_rejection[chromium]", + "id": "C3", + "name": "Upload and download file", + "case": "rejection", + "status": "passed", + "duration_ms": 10933, + "http_status": 403, + "evidence": null, + "known_external_blocked": 8 + }, + { + "nodeid": "test_c4_wiki_collab.py::test_c4_wiki_collab_success[chromium]", + "id": "C4", + "name": "Wiki collaborative edit", + "case": "success", + "status": "failed", + "duration_ms": 66819, + "http_status": null, + "evidence": null, + "known_external_blocked": 13 + }, + { + "nodeid": "test_c4_wiki_collab.py::test_c4_wiki_collab_rejection[chromium]", + "id": "C4", + "name": "Wiki collaborative edit", + "case": "rejection", + "status": "failed", + "duration_ms": 65666, + "http_status": null, + "evidence": null, + "known_external_blocked": 13 + }, + { + "nodeid": "test_c5_file_preview.py::test_c5_file_preview_success[chromium]", + "id": "C5", + "name": "File preview", + "case": "success", + "status": "error", + "duration_ms": 69402, + "http_status": null, + "evidence": null, + "known_external_blocked": 8 + }, + { + "nodeid": "test_c5_file_preview.py::test_c5_file_preview_rejection[chromium]", + "id": "C5", + "name": "File preview", + "case": "rejection", + "status": "failed", + "duration_ms": 3182, + "http_status": 400, + "evidence": null, + "known_external_blocked": 4 + }, + { + "nodeid": "test_c6_admin.py::test_c6_admin_success[chromium]", + "id": "C6", + "name": "Institution admin screen", + "case": "success", + "status": "error", + "duration_ms": 737, + "http_status": 404, + "evidence": null, + "known_external_blocked": 0 + }, + { + "nodeid": "test_c6_admin.py::test_c6_admin_rejection[chromium]", + "id": "C6", + "name": "Institution admin screen", + "case": "rejection", + "status": "failed", + "duration_ms": 2408, + "http_status": null, + "evidence": null, + "known_external_blocked": 3 + }, + { + "nodeid": "test_c1_login.py::test_c1_login_rejection", + "id": "C1", + "name": "Login via fakecas", + "case": "rejection", + "status": "passed", + "duration_ms": 47, + "http_status": 401, + "evidence": null, + "known_external_blocked": 0 + } +] diff --git a/docs/demo/runs/acc-20261110T043053Z-5c5dae/junit.xml b/docs/demo/runs/acc-20261110T043053Z-5c5dae/junit.xml new file mode 100644 index 0000000..dd786e6 --- /dev/null +++ b/docs/demo/runs/acc-20261110T043053Z-5c5dae/junit.xml @@ -0,0 +1,298 @@ +logged_in_page = <Page url='http://127.0.0.1:5000/dashboard'> +base_url = 'http://127.0.0.1:5000', api_url = 'http://127.0.0.1:8000' +api_write = <function api_write.<locals>._write at 0xebde57f47ce0> +run_id = 'acc-20261110T043053Z-5c5dae' +record_property = <function record_property.<locals>.append_property at 0xebde57f46b60> + + @pytest.mark.case("success") + def test_c2_create_project_success(logged_in_page, base_url, api_url, api_write, run_id, record_property): + """A new project created in the UI appears with its title and the user as admin.""" + page = logged_in_page + title = f"acceptance C2 {run_id}" + page.goto(base_url + sel.PATHS["new_project"], wait_until="domcontentloaded") + page.click(sel.WEB["create_project_button"]) +> page.fill(sel.WEB["create_project_title"], title) + +installer/acceptance/test_c2_create_project.py:18: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +.venv/lib/python3.12/site-packages/playwright/sync_api/_generated.py:17731: in fill + self._sync( +.venv/lib/python3.12/site-packages/playwright/_impl/page.py:1229: in fill + return await self._main_frame.fill( +.venv/lib/python3.12/site-packages/playwright/_impl/frame.py:736: in fill + return await self._channel.fill( +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:387: in finish + raise e from None +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:352: in finish + result = await zone.run_async(lambda: awaitable) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = <playwright._impl.utils.zones.Zone object at 0xebde57f17530> +func = <function ChannelOwner._wrap_api_call.<locals>.finish.<locals>.<lambda> at 0xebde57f447c0> + + async def run_async(self, func: Callable[[], Awaitable[R]]) -> R: + # Python: run() of an async function. The coroutine runs when it is awaited, + # so the zone must be current while it is awaited, not while it is created. + token = *** + try: +> return await func() + ^^^^^^^^^^^^ +E playwright._impl._errors.TimeoutError: Page.fill: Timeout 30000ms exceeded. +E Call log: +E - waiting for locator("input#projectName, input[name='title']") + +.venv/lib/python3.12/site-packages/playwright/_impl/utils/zones.py:58: TimeoutErrortwo_contexts = (<Page url='http://127.0.0.1:5000/kd2pa/wiki/home/?edit&view&menu'>, <Page url='http://127.0.0.1:5000/dashboard'>) +login = <function login.<locals>._login at 0xebde567a8860> +user_a = {'email': '***', 'password': '***'} +user_b = {'email': '***', 'password': '***'} +base_url = 'http://127.0.0.1:5000', api_url = 'http://127.0.0.1:8000' +api_write = <function api_write.<locals>._write at 0xebde567ab560> +add_contributor = <function add_contributor.<locals>._add at 0xebde57b48220> +run_id = 'acc-20261110T043053Z-5c5dae' +record_property = <function record_property.<locals>.append_property at 0xebde57b48400> + + @pytest.mark.case("success") + def test_c4_wiki_collab_success(two_contexts, login, user_a, user_b, base_url, api_url, api_write, add_contributor, + run_id, record_property): + """Both users open the same wiki page; text typed by B appears in A's editor and text typed by A in B's.""" + page_a, page_b = two_contexts + login(page_a, user_a) + login(page_b, user_b) + + created = api_write(page_a.context, "POST", api_url + sel.API["nodes"], { + "data": {"type": "nodes", "attributes": {"title": "acceptance C4", "category": "project", "public": False}} + }) + assert created.status == 201, created.text()[:200] + node_id = created.json()["data"]["id"] + try: + add_contributor(page_a.context, node_id, user_id_of(page_b.context, api_url), "write") + +> resp_a = _open_editor(page_a, base_url, node_id) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +installer/acceptance/test_c4_wiki_collab.py:53: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +installer/acceptance/test_c4_wiki_collab.py:26: in _open_editor + page.wait_for_selector(sel.WEB["wiki_editor"], timeout=60_000) +.venv/lib/python3.12/site-packages/playwright/sync_api/_generated.py:21403: in wait_for_selector + self._sync( +.venv/lib/python3.12/site-packages/playwright/_impl/page.py:464: in wait_for_selector + return await self._main_frame.wait_for_selector( +.venv/lib/python3.12/site-packages/playwright/_impl/frame.py:418: in wait_for_selector + result = await self._channel.waitForSelector( +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:398: in _mapped + return mapper(await awaitable) + ^^^^^^^^^^^^^^^ +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:387: in finish + raise e from None +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:352: in finish + result = await zone.run_async(lambda: awaitable) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = <playwright._impl.utils.zones.Zone object at 0xebde577b4d10> +func = <function ChannelOwner._wrap_api_call.<locals>.finish.<locals>.<lambda> at 0xebde578a8f40> + + async def run_async(self, func: Callable[[], Awaitable[R]]) -> R: + # Python: run() of an async function. The coroutine runs when it is awaited, + # so the zone must be current while it is awaited, not while it is created. + token = *** + try: +> return await func() + ^^^^^^^^^^^^ +E playwright._impl._errors.TimeoutError: Page.wait_for_selector: Timeout 60000ms exceeded. +E Call log: +E - waiting for locator(".CodeMirror textarea, textarea#editor") to be visible + +.venv/lib/python3.12/site-packages/playwright/_impl/utils/zones.py:58: TimeoutErrortwo_contexts = (<Page url='http://127.0.0.1:5000/6d8pg/wiki/home/?edit&view&menu'>, <Page url='about:blank'>) +login = <function login.<locals>._login at 0xebde5796ca40> +user_a = {'email': '***', 'password': '***'} +base_url = 'http://127.0.0.1:5000', project = '6d8pg' +record_property = <function record_property.<locals>.append_property at 0xebde57933d80> + + @pytest.mark.case("rejection") + def test_c4_wiki_collab_rejection(two_contexts, login, user_a, base_url, project, record_property): + """A logged-out context connecting to the wiki websocket is closed with code 4401.""" + page_a, page_anon = two_contexts + login(page_a, user_a) +> _open_editor(page_a, base_url, project) + +installer/acceptance/test_c4_wiki_collab.py:72: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +installer/acceptance/test_c4_wiki_collab.py:26: in _open_editor + page.wait_for_selector(sel.WEB["wiki_editor"], timeout=60_000) +.venv/lib/python3.12/site-packages/playwright/sync_api/_generated.py:21403: in wait_for_selector + self._sync( +.venv/lib/python3.12/site-packages/playwright/_impl/page.py:464: in wait_for_selector + return await self._main_frame.wait_for_selector( +.venv/lib/python3.12/site-packages/playwright/_impl/frame.py:418: in wait_for_selector + result = await self._channel.waitForSelector( +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:398: in _mapped + return mapper(await awaitable) + ^^^^^^^^^^^^^^^ +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:387: in finish + raise e from None +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:352: in finish + result = await zone.run_async(lambda: awaitable) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = <playwright._impl.utils.zones.Zone object at 0xebde57bbcbc0> +func = <function ChannelOwner._wrap_api_call.<locals>.finish.<locals>.<lambda> at 0xebde57f47420> + + async def run_async(self, func: Callable[[], Awaitable[R]]) -> R: + # Python: run() of an async function. The coroutine runs when it is awaited, + # so the zone must be current while it is awaited, not while it is created. + token = *** + try: +> return await func() + ^^^^^^^^^^^^ +E playwright._impl._errors.TimeoutError: Page.wait_for_selector: Timeout 60000ms exceeded. +E Call log: +E - waiting for locator(".CodeMirror textarea, textarea#editor") to be visible + +.venv/lib/python3.12/site-packages/playwright/_impl/utils/zones.py:58: TimeoutErrorlogged_in_page = <Page url='http://127.0.0.1:5000/27efg/'>, project = 'c5q8w' +base_url = 'http://127.0.0.1:5000' +service_url = <function service_url.<locals>._url at 0xebde57f09260> +tmp_pdf = b'%PDF-1.4\n1 0 obj<</Type/Catalog/Pages 2 0 R>>endobj\n2 0 obj<</Type/Pages/Kids[3 0 R]/Count 1>>endobj\n3 0 obj<</Ty...%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n' +run_id = 'acc-20261110T043053Z-5c5dae' +record_property = <function record_property.<locals>.append_property at 0xebde57b1cfe0> + + @pytest.mark.case("success") + def test_c5_file_preview_success(logged_in_page, project, base_url, service_url, tmp_pdf, run_id, record_property): + """An uploaded PDF renders inside the mfr iframe (mfr answers 200 for the render request).""" + page = logged_in_page + entity = upload(page.context, service_url("wb"), project, f"c5-{run_id}.pdf", tmp_pdf) + mfr = service_url("mfr") + with page.expect_response(lambda r: same_origin(r.url, mfr) and "/render" in r.url, timeout=90_000) as render: + page.goto(_file_page_url(base_url, project, entity["attributes"]["path"]), wait_until="domcontentloaded") +> page.wait_for_selector(sel.WEB["file_preview_iframe"], timeout=60_000) + +installer/acceptance/test_c5_file_preview.py:24: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +.venv/lib/python3.12/site-packages/playwright/sync_api/_generated.py:21403: in wait_for_selector + self._sync( +.venv/lib/python3.12/site-packages/playwright/_impl/page.py:464: in wait_for_selector + return await self._main_frame.wait_for_selector( +.venv/lib/python3.12/site-packages/playwright/_impl/frame.py:418: in wait_for_selector + result = await self._channel.waitForSelector( +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:398: in _mapped + return mapper(await awaitable) + ^^^^^^^^^^^^^^^ +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:387: in finish + raise e from None +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:352: in finish + result = await zone.run_async(lambda: awaitable) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = <playwright._impl.utils.zones.Zone object at 0xebde57cbdb80> +func = <function ChannelOwner._wrap_api_call.<locals>.finish.<locals>.<lambda> at 0xebde57d839c0> + + async def run_async(self, func: Callable[[], Awaitable[R]]) -> R: + # Python: run() of an async function. The coroutine runs when it is awaited, + # so the zone must be current while it is awaited, not while it is created. + token = *** + try: +> return await func() + ^^^^^^^^^^^^ +E playwright._impl._errors.TimeoutError: Page.wait_for_selector: Timeout 60000ms exceeded. +E Call log: +E - waiting for locator("iframe#mfrIframe, iframe[src*='render']") to be visible + +.venv/lib/python3.12/site-packages/playwright/_impl/utils/zones.py:58: TimeoutErrorrequest = <SubRequest 'external_guard' for <Function test_c5_file_preview_success[chromium]>> + + @pytest.fixture(autouse=True) + def external_guard(request): + """Collect the loopback-guard offenders of one test and fail it if any guard (browser, websocket, + API) saw a URL that would have left the machine. Every browser context is created by our + `new_context` below (pytest-playwright's default `context` fixture calls it too), which attaches + the guard itself; requesting `context` from here created a fixture cycle + (external_guard -> context -> new_context -> external_guard, first real acceptance run).""" + offenders: list[str] = [] + before = len(_OFFENDERS) + known_before = len(_KNOWN_EXTERNAL) + yield offenders + offenders.extend(_OFFENDERS[before:]) + request.node.user_properties.append(("known_external_blocked", len(_KNOWN_EXTERNAL) - known_before)) + if offenders: +> pytest.fail(f"request(s) would have left the machine: {offenders[:5]}") +E Failed: request(s) would have left the machine: ['http://localhost:7007/text.js', 'http://localhost:7007/share.js'] + +installer/acceptance/conftest.py:274: Failedlogged_in_page = <Page url='http://127.0.0.1:5000/dashboard'>, project = 'x7uv2' +service_url = <function service_url.<locals>._url at 0xebde57f09260> +tmp_pdf = b'%PDF-1.4\n1 0 obj<</Type/Catalog/Pages 2 0 R>>endobj\n2 0 obj<</Type/Pages/Kids[3 0 R]/Count 1>>endobj\n3 0 obj<</Ty...%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n' +run_id = 'acc-20261110T043053Z-5c5dae' +api_context = <function api_context.<locals>._new at 0xebde57b48860> +record_property = <function record_property.<locals>.append_property at 0xebde5795ade0> + + @pytest.mark.case("rejection") + def test_c5_file_preview_rejection(logged_in_page, project, service_url, tmp_pdf, run_id, api_context, record_property): + """The preview URL of a private file without a session -> 401/403.""" + page = logged_in_page + wb = service_url("wb") + entity = upload(page.context, wb, project, f"c5-private-{run_id}.pdf", tmp_pdf) + file_url = download_link(entity, wb, project) + anon = api_context() + resp = anon.get(service_url("mfr") + sel.MFR["render"].format(file_url=file_url)) + record_property("http_status", resp.status) +> assert resp.status in (401, 403), f"anonymous preview returned {resp.status}" +E AssertionError: anonymous preview returned 400 +E assert 400 in (401, 403) +E + where 400 = <APIResponse url='http://127.0.0.1:7778/render?url=http://127.0.0.1:7777/v1/resources/x7uv2/providers/osfstorage/6af29f17c91ee600098cbf54' status=400 status_text='Bad Request'>.status + +installer/acceptance/test_c5_file_preview.py:39: AssertionErrorpage = <Page url='http://127.0.0.1:8001/institutions/1/users/'> +service_url = <function service_url.<locals>._url at 0xebde57f09260> +admin_user = {'email': '***', 'password': '***'} +record_property = <function record_property.<locals>.append_property at 0xebde569085e0> + + @pytest.mark.case("success") + def test_c6_admin_success(page, service_url, admin_user, record_property): + """The institution admin signs in to the admin app and sees the institution's user list.""" + admin_url = service_url("admin") + _admin_login(page, admin_url, admin_user) + # TODO(M2): the institution id of the seeded admin; "1" assumes the first institution created by --seed. + resp = page.goto(admin_url + sel.ADMIN["institution_users"].format(institution_id="1"), wait_until="domcontentloaded") + record_property("http_status", resp.status if resp else None) +> assert resp is not None and resp.status == 200, f"institution users page returned {resp and resp.status}" +E AssertionError: institution users page returned 404 +E assert (<Response url='http://127.0.0.1:8001/institutions/1/users/' request=<Request url='http://127.0.0.1:8001/institutions/1/users/' method='GET'>> is not None and 404 == 200) +E + where 404 = <Response url='http://127.0.0.1:8001/institutions/1/users/' request=<Request url='http://127.0.0.1:8001/institutions/1/users/' method='GET'>>.status + +installer/acceptance/test_c6_admin.py:27: AssertionErrorrequest = <SubRequest 'external_guard' for <Function test_c6_admin_success[chromium]>> + + @pytest.fixture(autouse=True) + def external_guard(request): + """Collect the loopback-guard offenders of one test and fail it if any guard (browser, websocket, + API) saw a URL that would have left the machine. Every browser context is created by our + `new_context` below (pytest-playwright's default `context` fixture calls it too), which attaches + the guard itself; requesting `context` from here created a fixture cycle + (external_guard -> context -> new_context -> external_guard, first real acceptance run).""" + offenders: list[str] = [] + before = len(_OFFENDERS) + known_before = len(_KNOWN_EXTERNAL) + yield offenders + offenders.extend(_OFFENDERS[before:]) + request.node.user_properties.append(("known_external_blocked", len(_KNOWN_EXTERNAL) - known_before)) + if offenders: +> pytest.fail(f"request(s) would have left the machine: {offenders[:5]}") +E Failed: request(s) would have left the machine: ['http://code.ionicframework.com/ionicons/2.0.1/css/ionicons.min.css', 'https://maxcdn.bootstrapcdn.com/font-awesome/4.3.0/css/font-awesome.min.css', 'http://code.ionicframework.com/ionicons/2.0.1/css/ionicons.min.css', 'https://maxcdn.bootstrapcdn.com/font-awesome/4.3.0/css/font-awesome.min.css', 'http://code.ionicframework.com/ionicons/2.0.1/css/ionicons.min.css'] + +installer/acceptance/conftest.py:274: Failedlogged_in_page = <Page url='http://127.0.0.1:5000/dashboard/'> +service_url = <function service_url.<locals>._url at 0xebde57f09260> +api_url = 'http://127.0.0.1:8000' +record_property = <function record_property.<locals>.append_property at 0xebde57933ce0> + + @pytest.mark.case("rejection") + def test_c6_admin_rejection(logged_in_page, service_url, api_url, record_property): + """An ordinary user opening the admin URL -> 403 or redirect to login; the API answers exactly 403.""" + client = api(logged_in_page.context) + admin_url = service_url("admin") + screen = client.get(admin_url + sel.ADMIN["institution_users"].format(institution_id="1"), max_redirects=0) +> assert screen.status in (301, 302, 303, 307, 403), f"admin screen answered {screen.status} to an ordinary user" +E AssertionError: admin screen answered 404 to an ordinary user +E assert 404 in (301, 302, 303, 307, 403) +E + where 404 = <APIResponse url='http://127.0.0.1:8001/institutions/1/users/' status=404 status_text='Not Found'>.status + +installer/acceptance/test_c6_admin.py:37: AssertionError \ No newline at end of file diff --git a/docs/demo/runs/acc-20261110T043053Z-5c5dae/pytest.log b/docs/demo/runs/acc-20261110T043053Z-5c5dae/pytest.log new file mode 100644 index 0000000..6e17875 --- /dev/null +++ b/docs/demo/runs/acc-20261110T043053Z-5c5dae/pytest.log @@ -0,0 +1,376 @@ +.F...FFFEFFEF. [100%] +==================================== ERRORS ==================================== +_________ ERROR at teardown of test_c5_file_preview_success[chromium] __________ + +request = > + + @pytest.fixture(autouse=True) + def external_guard(request): + """Collect the loopback-guard offenders of one test and fail it if any guard (browser, websocket, + API) saw a URL that would have left the machine. Every browser context is created by our + `new_context` below (pytest-playwright's default `context` fixture calls it too), which attaches + the guard itself; requesting `context` from here created a fixture cycle + (external_guard -> context -> new_context -> external_guard, first real acceptance run).""" + offenders: list[str] = [] + before = len(_OFFENDERS) + known_before = len(_KNOWN_EXTERNAL) + yield offenders + offenders.extend(_OFFENDERS[before:]) + request.node.user_properties.append(("known_external_blocked", len(_KNOWN_EXTERNAL) - known_before)) + if offenders: +> pytest.fail(f"request(s) would have left the machine: {offenders[:5]}") +E Failed: request(s) would have left the machine: ['http://localhost:7007/text.js', 'http://localhost:7007/share.js'] + +installer/acceptance/conftest.py:274: Failed +_____________ ERROR at teardown of test_c6_admin_success[chromium] _____________ + +request = > + + @pytest.fixture(autouse=True) + def external_guard(request): + """Collect the loopback-guard offenders of one test and fail it if any guard (browser, websocket, + API) saw a URL that would have left the machine. Every browser context is created by our + `new_context` below (pytest-playwright's default `context` fixture calls it too), which attaches + the guard itself; requesting `context` from here created a fixture cycle + (external_guard -> context -> new_context -> external_guard, first real acceptance run).""" + offenders: list[str] = [] + before = len(_OFFENDERS) + known_before = len(_KNOWN_EXTERNAL) + yield offenders + offenders.extend(_OFFENDERS[before:]) + request.node.user_properties.append(("known_external_blocked", len(_KNOWN_EXTERNAL) - known_before)) + if offenders: +> pytest.fail(f"request(s) would have left the machine: {offenders[:5]}") +E Failed: request(s) would have left the machine: ['http://code.ionicframework.com/ionicons/2.0.1/css/ionicons.min.css', 'https://maxcdn.bootstrapcdn.com/font-awesome/4.3.0/css/font-awesome.min.css', 'http://code.ionicframework.com/ionicons/2.0.1/css/ionicons.min.css', 'https://maxcdn.bootstrapcdn.com/font-awesome/4.3.0/css/font-awesome.min.css', 'http://code.ionicframework.com/ionicons/2.0.1/css/ionicons.min.css'] + +installer/acceptance/conftest.py:274: Failed +=================================== FAILURES =================================== +___________________ test_c2_create_project_success[chromium] ___________________ + +logged_in_page = +base_url = 'http://127.0.0.1:5000', api_url = 'http://127.0.0.1:8000' +api_write = ._write at 0xebde57f47ce0> +run_id = 'acc-20261110T043053Z-5c5dae' +record_property = .append_property at 0xebde57f46b60> + + @pytest.mark.case("success") + def test_c2_create_project_success(logged_in_page, base_url, api_url, api_write, run_id, record_property): + """A new project created in the UI appears with its title and the user as admin.""" + page = logged_in_page + title = f"acceptance C2 {run_id}" + page.goto(base_url + sel.PATHS["new_project"], wait_until="domcontentloaded") + page.click(sel.WEB["create_project_button"]) +> page.fill(sel.WEB["create_project_title"], title) + +installer/acceptance/test_c2_create_project.py:18: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +.venv/lib/python3.12/site-packages/playwright/sync_api/_generated.py:17731: in fill + self._sync( +.venv/lib/python3.12/site-packages/playwright/_impl/page.py:1229: in fill + return await self._main_frame.fill( +.venv/lib/python3.12/site-packages/playwright/_impl/frame.py:736: in fill + return await self._channel.fill( +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:387: in finish + raise e from None +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:352: in finish + result = await zone.run_async(lambda: awaitable) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = +func = .finish.. at 0xebde57f447c0> + + async def run_async(self, func: Callable[[], Awaitable[R]]) -> R: + # Python: run() of an async function. The coroutine runs when it is awaited, + # so the zone must be current while it is awaited, not while it is created. + token = *** + try: +> return await func() + ^^^^^^^^^^^^ +E playwright._impl._errors.TimeoutError: Page.fill: Timeout 30000ms exceeded. +E Call log: +E - waiting for locator("input#projectName, input[name='title']") + +.venv/lib/python3.12/site-packages/playwright/_impl/utils/zones.py:58: TimeoutError +____________________ test_c4_wiki_collab_success[chromium] _____________________ + +two_contexts = (, ) +login = ._login at 0xebde567a8860> +user_a = {'email': '***', 'password': '***'} +user_b = {'email': '***', 'password': '***'} +base_url = 'http://127.0.0.1:5000', api_url = 'http://127.0.0.1:8000' +api_write = ._write at 0xebde567ab560> +add_contributor = ._add at 0xebde57b48220> +run_id = 'acc-20261110T043053Z-5c5dae' +record_property = .append_property at 0xebde57b48400> + + @pytest.mark.case("success") + def test_c4_wiki_collab_success(two_contexts, login, user_a, user_b, base_url, api_url, api_write, add_contributor, + run_id, record_property): + """Both users open the same wiki page; text typed by B appears in A's editor and text typed by A in B's.""" + page_a, page_b = two_contexts + login(page_a, user_a) + login(page_b, user_b) + + created = api_write(page_a.context, "POST", api_url + sel.API["nodes"], { + "data": {"type": "nodes", "attributes": {"title": "acceptance C4", "category": "project", "public": False}} + }) + assert created.status == 201, created.text()[:200] + node_id = created.json()["data"]["id"] + try: + add_contributor(page_a.context, node_id, user_id_of(page_b.context, api_url), "write") + +> resp_a = _open_editor(page_a, base_url, node_id) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +installer/acceptance/test_c4_wiki_collab.py:53: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +installer/acceptance/test_c4_wiki_collab.py:26: in _open_editor + page.wait_for_selector(sel.WEB["wiki_editor"], timeout=60_000) +.venv/lib/python3.12/site-packages/playwright/sync_api/_generated.py:21403: in wait_for_selector + self._sync( +.venv/lib/python3.12/site-packages/playwright/_impl/page.py:464: in wait_for_selector + return await self._main_frame.wait_for_selector( +.venv/lib/python3.12/site-packages/playwright/_impl/frame.py:418: in wait_for_selector + result = await self._channel.waitForSelector( +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:398: in _mapped + return mapper(await awaitable) + ^^^^^^^^^^^^^^^ +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:387: in finish + raise e from None +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:352: in finish + result = await zone.run_async(lambda: awaitable) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = +func = .finish.. at 0xebde578a8f40> + + async def run_async(self, func: Callable[[], Awaitable[R]]) -> R: + # Python: run() of an async function. The coroutine runs when it is awaited, + # so the zone must be current while it is awaited, not while it is created. + token = *** + try: +> return await func() + ^^^^^^^^^^^^ +E playwright._impl._errors.TimeoutError: Page.wait_for_selector: Timeout 60000ms exceeded. +E Call log: +E - waiting for locator(".CodeMirror textarea, textarea#editor") to be visible + +.venv/lib/python3.12/site-packages/playwright/_impl/utils/zones.py:58: TimeoutError +___________________ test_c4_wiki_collab_rejection[chromium] ____________________ + +two_contexts = (, ) +login = ._login at 0xebde5796ca40> +user_a = {'email': '***', 'password': '***'} +base_url = 'http://127.0.0.1:5000', project = '6d8pg' +record_property = .append_property at 0xebde57933d80> + + @pytest.mark.case("rejection") + def test_c4_wiki_collab_rejection(two_contexts, login, user_a, base_url, project, record_property): + """A logged-out context connecting to the wiki websocket is closed with code 4401.""" + page_a, page_anon = two_contexts + login(page_a, user_a) +> _open_editor(page_a, base_url, project) + +installer/acceptance/test_c4_wiki_collab.py:72: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +installer/acceptance/test_c4_wiki_collab.py:26: in _open_editor + page.wait_for_selector(sel.WEB["wiki_editor"], timeout=60_000) +.venv/lib/python3.12/site-packages/playwright/sync_api/_generated.py:21403: in wait_for_selector + self._sync( +.venv/lib/python3.12/site-packages/playwright/_impl/page.py:464: in wait_for_selector + return await self._main_frame.wait_for_selector( +.venv/lib/python3.12/site-packages/playwright/_impl/frame.py:418: in wait_for_selector + result = await self._channel.waitForSelector( +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:398: in _mapped + return mapper(await awaitable) + ^^^^^^^^^^^^^^^ +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:387: in finish + raise e from None +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:352: in finish + result = await zone.run_async(lambda: awaitable) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = +func = .finish.. at 0xebde57f47420> + + async def run_async(self, func: Callable[[], Awaitable[R]]) -> R: + # Python: run() of an async function. The coroutine runs when it is awaited, + # so the zone must be current while it is awaited, not while it is created. + token = *** + try: +> return await func() + ^^^^^^^^^^^^ +E playwright._impl._errors.TimeoutError: Page.wait_for_selector: Timeout 60000ms exceeded. +E Call log: +E - waiting for locator(".CodeMirror textarea, textarea#editor") to be visible + +.venv/lib/python3.12/site-packages/playwright/_impl/utils/zones.py:58: TimeoutError +____________________ test_c5_file_preview_success[chromium] ____________________ + +logged_in_page = , project = 'c5q8w' +base_url = 'http://127.0.0.1:5000' +service_url = ._url at 0xebde57f09260> +tmp_pdf = b'%PDF-1.4\n1 0 obj<>endobj\n2 0 obj<>endobj\n3 0 obj<.append_property at 0xebde57b1cfe0> + + @pytest.mark.case("success") + def test_c5_file_preview_success(logged_in_page, project, base_url, service_url, tmp_pdf, run_id, record_property): + """An uploaded PDF renders inside the mfr iframe (mfr answers 200 for the render request).""" + page = logged_in_page + entity = upload(page.context, service_url("wb"), project, f"c5-{run_id}.pdf", tmp_pdf) + mfr = service_url("mfr") + with page.expect_response(lambda r: same_origin(r.url, mfr) and "/render" in r.url, timeout=90_000) as render: + page.goto(_file_page_url(base_url, project, entity["attributes"]["path"]), wait_until="domcontentloaded") +> page.wait_for_selector(sel.WEB["file_preview_iframe"], timeout=60_000) + +installer/acceptance/test_c5_file_preview.py:24: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +.venv/lib/python3.12/site-packages/playwright/sync_api/_generated.py:21403: in wait_for_selector + self._sync( +.venv/lib/python3.12/site-packages/playwright/_impl/page.py:464: in wait_for_selector + return await self._main_frame.wait_for_selector( +.venv/lib/python3.12/site-packages/playwright/_impl/frame.py:418: in wait_for_selector + result = await self._channel.waitForSelector( +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:398: in _mapped + return mapper(await awaitable) + ^^^^^^^^^^^^^^^ +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:387: in finish + raise e from None +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:352: in finish + result = await zone.run_async(lambda: awaitable) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = +func = .finish.. at 0xebde57d839c0> + + async def run_async(self, func: Callable[[], Awaitable[R]]) -> R: + # Python: run() of an async function. The coroutine runs when it is awaited, + # so the zone must be current while it is awaited, not while it is created. + token = *** + try: +> return await func() + ^^^^^^^^^^^^ +E playwright._impl._errors.TimeoutError: Page.wait_for_selector: Timeout 60000ms exceeded. +E Call log: +E - waiting for locator("iframe#mfrIframe, iframe[src*='render']") to be visible + +.venv/lib/python3.12/site-packages/playwright/_impl/utils/zones.py:58: TimeoutError +___________________ test_c5_file_preview_rejection[chromium] ___________________ + +logged_in_page = , project = 'x7uv2' +service_url = ._url at 0xebde57f09260> +tmp_pdf = b'%PDF-1.4\n1 0 obj<>endobj\n2 0 obj<>endobj\n3 0 obj<._new at 0xebde57b48860> +record_property = .append_property at 0xebde5795ade0> + + @pytest.mark.case("rejection") + def test_c5_file_preview_rejection(logged_in_page, project, service_url, tmp_pdf, run_id, api_context, record_property): + """The preview URL of a private file without a session -> 401/403.""" + page = logged_in_page + wb = service_url("wb") + entity = upload(page.context, wb, project, f"c5-private-{run_id}.pdf", tmp_pdf) + file_url = download_link(entity, wb, project) + anon = api_context() + resp = anon.get(service_url("mfr") + sel.MFR["render"].format(file_url=file_url)) + record_property("http_status", resp.status) +> assert resp.status in (401, 403), f"anonymous preview returned {resp.status}" +E AssertionError: anonymous preview returned 400 +E assert 400 in (401, 403) +E + where 400 = .status + +installer/acceptance/test_c5_file_preview.py:39: AssertionError +_______________________ test_c6_admin_success[chromium] ________________________ + +page = +service_url = ._url at 0xebde57f09260> +admin_user = {'email': '***', 'password': '***'} +record_property = .append_property at 0xebde569085e0> + + @pytest.mark.case("success") + def test_c6_admin_success(page, service_url, admin_user, record_property): + """The institution admin signs in to the admin app and sees the institution's user list.""" + admin_url = service_url("admin") + _admin_login(page, admin_url, admin_user) + # TODO(M2): the institution id of the seeded admin; "1" assumes the first institution created by --seed. + resp = page.goto(admin_url + sel.ADMIN["institution_users"].format(institution_id="1"), wait_until="domcontentloaded") + record_property("http_status", resp.status if resp else None) +> assert resp is not None and resp.status == 200, f"institution users page returned {resp and resp.status}" +E AssertionError: institution users page returned 404 +E assert (> is not None and 404 == 200) +E + where 404 = >.status + +installer/acceptance/test_c6_admin.py:27: AssertionError +______________________ test_c6_admin_rejection[chromium] _______________________ + +logged_in_page = +service_url = ._url at 0xebde57f09260> +api_url = 'http://127.0.0.1:8000' +record_property = .append_property at 0xebde57933ce0> + + @pytest.mark.case("rejection") + def test_c6_admin_rejection(logged_in_page, service_url, api_url, record_property): + """An ordinary user opening the admin URL -> 403 or redirect to login; the API answers exactly 403.""" + client = api(logged_in_page.context) + admin_url = service_url("admin") + screen = client.get(admin_url + sel.ADMIN["institution_users"].format(institution_id="1"), max_redirects=0) +> assert screen.status in (301, 302, 303, 307, 403), f"admin screen answered {screen.status} to an ordinary user" +E AssertionError: admin screen answered 404 to an ordinary user +E assert 404 in (301, 302, 303, 307, 403) +E + where 404 = .status + +installer/acceptance/test_c6_admin.py:37: AssertionError +=============================== warnings summary =============================== +test_c1_login.py::test_c1_login_success[chromium] + test_c1_login.py:10: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c2_create_project.py::test_c2_create_project_success[chromium] + test_c2_create_project.py:11: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c2_create_project.py::test_c2_create_project_rejection[chromium] + test_c2_create_project.py:39: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c3_file_upload.py::test_c3_file_upload_success[chromium] + test_c3_file_upload.py:29: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c3_file_upload.py::test_c3_file_upload_rejection[chromium] + test_c3_file_upload.py:43: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c4_wiki_collab.py::test_c4_wiki_collab_success[chromium] + test_c4_wiki_collab.py:37: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c4_wiki_collab.py::test_c4_wiki_collab_rejection[chromium] + test_c4_wiki_collab.py:67: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c5_file_preview.py::test_c5_file_preview_success[chromium] + test_c5_file_preview.py:16: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c5_file_preview.py::test_c5_file_preview_rejection[chromium] + test_c5_file_preview.py:29: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c6_admin.py::test_c6_admin_success[chromium] + test_c6_admin.py:19: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c6_admin.py::test_c6_admin_rejection[chromium] + test_c6_admin.py:31: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c1_login.py::test_c1_login_rejection + test_c1_login.py:20: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +=========================== short test summary info ============================ +ERROR installer/acceptance/test_c5_file_preview.py::test_c5_file_preview_success[chromium] +ERROR installer/acceptance/test_c6_admin.py::test_c6_admin_success[chromium] +FAILED installer/acceptance/test_c2_create_project.py::test_c2_create_project_success[chromium] +FAILED installer/acceptance/test_c4_wiki_collab.py::test_c4_wiki_collab_success[chromium] +FAILED installer/acceptance/test_c4_wiki_collab.py::test_c4_wiki_collab_rejection[chromium] +FAILED installer/acceptance/test_c5_file_preview.py::test_c5_file_preview_success[chromium] +FAILED installer/acceptance/test_c5_file_preview.py::test_c5_file_preview_rejection[chromium] +FAILED installer/acceptance/test_c6_admin.py::test_c6_admin_success[chromium] +FAILED installer/acceptance/test_c6_admin.py::test_c6_admin_rejection[chromium] +7 failed, 5 passed, 10 deselected, 12 warnings, 2 errors in 274.54s (0:04:34) diff --git a/docs/demo/runs/acc-20261110T043053Z-5c5dae/results.json b/docs/demo/runs/acc-20261110T043053Z-5c5dae/results.json new file mode 100644 index 0000000..64d2cd6 --- /dev/null +++ b/docs/demo/runs/acc-20261110T043053Z-5c5dae/results.json @@ -0,0 +1,167 @@ +{ + "run_id": "acc-20261110T043053Z-5c5dae", + "base_url": "http://127.0.0.1:5000", + "started": "2026-11-10T04:30:53Z", + "duration_ms": 288154, + "only": "core", + "seeded": true, + "precheck": { + "healthy": true, + "services": 24 + }, + "pytest_returncode": 1, + "evidence_dir": "docs/demo/runs/acc-20261110T043053Z-5c5dae", + "tracing": "off", + "trace_quarantine": null, + "expected_scenarios": [ + "C1", + "C2", + "C3", + "C4", + "C5", + "C6" + ], + "incomplete_scenarios": [], + "summary": { + "passed": 5, + "failed": 5, + "error": 2, + "skipped": 0, + "total": 12, + "known_external_blocked": 75 + }, + "tests": [ + { + "id": "C1", + "name": "Login via fakecas", + "case": "success", + "test": "test_c1_login_success[chromium]", + "status": "passed", + "duration_ms": 6150, + "http_status": 200, + "evidence": null, + "known_external_blocked": 7 + }, + { + "id": "C2", + "name": "Create project", + "case": "success", + "test": "test_c2_create_project_success[chromium]", + "status": "failed", + "duration_ms": 34486, + "http_status": null, + "evidence": null, + "known_external_blocked": 7 + }, + { + "id": "C2", + "name": "Create project", + "case": "rejection", + "test": "test_c2_create_project_rejection[chromium]", + "status": "passed", + "duration_ms": 5448, + "http_status": 403, + "evidence": null, + "known_external_blocked": 8 + }, + { + "id": "C3", + "name": "Upload and download file", + "case": "success", + "test": "test_c3_file_upload_success[chromium]", + "status": "passed", + "duration_ms": 8718, + "http_status": 200, + "evidence": null, + "known_external_blocked": 4 + }, + { + "id": "C3", + "name": "Upload and download file", + "case": "rejection", + "test": "test_c3_file_upload_rejection[chromium]", + "status": "passed", + "duration_ms": 10933, + "http_status": 403, + "evidence": null, + "known_external_blocked": 8 + }, + { + "id": "C4", + "name": "Wiki collaborative edit", + "case": "success", + "test": "test_c4_wiki_collab_success[chromium]", + "status": "failed", + "duration_ms": 66819, + "http_status": null, + "evidence": null, + "known_external_blocked": 13 + }, + { + "id": "C4", + "name": "Wiki collaborative edit", + "case": "rejection", + "test": "test_c4_wiki_collab_rejection[chromium]", + "status": "failed", + "duration_ms": 65666, + "http_status": null, + "evidence": null, + "known_external_blocked": 13 + }, + { + "id": "C5", + "name": "File preview", + "case": "success", + "test": "test_c5_file_preview_success[chromium]", + "status": "error", + "duration_ms": 69402, + "http_status": null, + "evidence": null, + "known_external_blocked": 8 + }, + { + "id": "C5", + "name": "File preview", + "case": "rejection", + "test": "test_c5_file_preview_rejection[chromium]", + "status": "failed", + "duration_ms": 3182, + "http_status": 400, + "evidence": null, + "known_external_blocked": 4 + }, + { + "id": "C6", + "name": "Institution admin screen", + "case": "success", + "test": "test_c6_admin_success[chromium]", + "status": "error", + "duration_ms": 737, + "http_status": 404, + "evidence": null, + "known_external_blocked": 0 + }, + { + "id": "C6", + "name": "Institution admin screen", + "case": "rejection", + "test": "test_c6_admin_rejection[chromium]", + "status": "failed", + "duration_ms": 2408, + "http_status": null, + "evidence": null, + "known_external_blocked": 3 + }, + { + "id": "C1", + "name": "Login via fakecas", + "case": "rejection", + "test": "test_c1_login_rejection", + "status": "passed", + "duration_ms": 47, + "http_status": 401, + "evidence": null, + "known_external_blocked": 0 + } + ] +} diff --git a/docs/demo/runs/acc-20261110T043053Z-5c5dae/results.toon b/docs/demo/runs/acc-20261110T043053Z-5c5dae/results.toon new file mode 100644 index 0000000..a0192b3 --- /dev/null +++ b/docs/demo/runs/acc-20261110T043053Z-5c5dae/results.toon @@ -0,0 +1,42 @@ +accept: + run_id: acc-20261110T043053Z-5c5dae + base_url: "http://127.0.0.1:5000" + started: "2026-11-10T04:30:53Z" + duration_ms: 288154 + only: core + seeded: true + precheck: + healthy: true + services: 24 + pytest_returncode: 1 + evidence_dir: docs/demo/runs/acc-20261110T043053Z-5c5dae + tracing: off + trace_quarantine: null + expected_scenarios[6]: + - C1 + - C2 + - C3 + - C4 + - C5 + - C6 + incomplete_scenarios[0]: + summary: + passed: 5 + failed: 5 + error: 2 + skipped: 0 + total: 12 + known_external_blocked: 75 + tests[12]{id,name,case,test,status,duration_ms,http_status,evidence,known_external_blocked}: + C1,Login via fakecas,success,"test_c1_login_success[chromium]",passed,6150,200,null,7 + C2,Create project,success,"test_c2_create_project_success[chromium]",failed,34486,null,null,7 + C2,Create project,rejection,"test_c2_create_project_rejection[chromium]",passed,5448,403,null,8 + C3,Upload and download file,success,"test_c3_file_upload_success[chromium]",passed,8718,200,null,4 + C3,Upload and download file,rejection,"test_c3_file_upload_rejection[chromium]",passed,10933,403,null,8 + C4,Wiki collaborative edit,success,"test_c4_wiki_collab_success[chromium]",failed,66819,null,null,13 + C4,Wiki collaborative edit,rejection,"test_c4_wiki_collab_rejection[chromium]",failed,65666,null,null,13 + C5,File preview,success,"test_c5_file_preview_success[chromium]",error,69402,null,null,8 + C5,File preview,rejection,"test_c5_file_preview_rejection[chromium]",failed,3182,400,null,4 + C6,Institution admin screen,success,"test_c6_admin_success[chromium]",error,737,404,null,0 + C6,Institution admin screen,rejection,"test_c6_admin_rejection[chromium]",failed,2408,null,null,3 + C1,Login via fakecas,rejection,test_c1_login_rejection,passed,47,401,null,0 diff --git a/docs/demo/runs/acc-20261110T052932Z-f1da54/collector.json b/docs/demo/runs/acc-20261110T052932Z-f1da54/collector.json new file mode 100644 index 0000000..eebc34a --- /dev/null +++ b/docs/demo/runs/acc-20261110T052932Z-f1da54/collector.json @@ -0,0 +1,134 @@ +[ + { + "nodeid": "test_c1_login.py::test_c1_login_success[chromium]", + "id": "C1", + "name": "Login via fakecas", + "case": "success", + "status": "passed", + "duration_ms": 5626, + "http_status": 200, + "evidence": null, + "known_external_blocked": 7 + }, + { + "nodeid": "test_c2_create_project.py::test_c2_create_project_success[chromium]", + "id": "C2", + "name": "Create project", + "case": "success", + "status": "failed", + "duration_ms": 34432, + "http_status": null, + "evidence": null, + "known_external_blocked": 7 + }, + { + "nodeid": "test_c2_create_project.py::test_c2_create_project_rejection[chromium]", + "id": "C2", + "name": "Create project", + "case": "rejection", + "status": "passed", + "duration_ms": 5452, + "http_status": 403, + "evidence": null, + "known_external_blocked": 8 + }, + { + "nodeid": "test_c3_file_upload.py::test_c3_file_upload_success[chromium]", + "id": "C3", + "name": "Upload and download file", + "case": "success", + "status": "passed", + "duration_ms": 13162, + "http_status": 200, + "evidence": null, + "known_external_blocked": 4 + }, + { + "nodeid": "test_c3_file_upload.py::test_c3_file_upload_rejection[chromium]", + "id": "C3", + "name": "Upload and download file", + "case": "rejection", + "status": "passed", + "duration_ms": 15381, + "http_status": 403, + "evidence": null, + "known_external_blocked": 8 + }, + { + "nodeid": "test_c4_wiki_collab.py::test_c4_wiki_collab_success[chromium]", + "id": "C4", + "name": "Wiki collaborative edit", + "case": "success", + "status": "failed", + "duration_ms": 66597, + "http_status": null, + "evidence": null, + "known_external_blocked": 13 + }, + { + "nodeid": "test_c4_wiki_collab.py::test_c4_wiki_collab_rejection[chromium]", + "id": "C4", + "name": "Wiki collaborative edit", + "case": "rejection", + "status": "failed", + "duration_ms": 66844, + "http_status": null, + "evidence": null, + "known_external_blocked": 13 + }, + { + "nodeid": "test_c5_file_preview.py::test_c5_file_preview_success[chromium]", + "id": "C5", + "name": "File preview", + "case": "success", + "status": "error", + "duration_ms": 64234, + "http_status": null, + "evidence": null, + "known_external_blocked": 8 + }, + { + "nodeid": "test_c5_file_preview.py::test_c5_file_preview_rejection[chromium]", + "id": "C5", + "name": "File preview", + "case": "rejection", + "status": "failed", + "duration_ms": 3587, + "http_status": 400, + "evidence": null, + "known_external_blocked": 4 + }, + { + "nodeid": "test_c6_admin.py::test_c6_admin_success[chromium]", + "id": "C6", + "name": "Institution admin screen", + "case": "success", + "status": "error", + "duration_ms": 670, + "http_status": 404, + "evidence": null, + "known_external_blocked": 0 + }, + { + "nodeid": "test_c6_admin.py::test_c6_admin_rejection[chromium]", + "id": "C6", + "name": "Institution admin screen", + "case": "rejection", + "status": "failed", + "duration_ms": 2277, + "http_status": null, + "evidence": null, + "known_external_blocked": 3 + }, + { + "nodeid": "test_c1_login.py::test_c1_login_rejection", + "id": "C1", + "name": "Login via fakecas", + "case": "rejection", + "status": "passed", + "duration_ms": 47, + "http_status": 401, + "evidence": null, + "known_external_blocked": 0 + } +] diff --git a/docs/demo/runs/acc-20261110T052932Z-f1da54/junit.xml b/docs/demo/runs/acc-20261110T052932Z-f1da54/junit.xml new file mode 100644 index 0000000..91be6e0 --- /dev/null +++ b/docs/demo/runs/acc-20261110T052932Z-f1da54/junit.xml @@ -0,0 +1,298 @@ +logged_in_page = <Page url='http://127.0.0.1:5000/dashboard'> +base_url = 'http://127.0.0.1:5000', api_url = 'http://127.0.0.1:8000' +api_write = <function api_write.<locals>._write at 0xf59ac21e3c40> +run_id = 'acc-20261110T052932Z-f1da54' +record_property = <function record_property.<locals>.append_property at 0xf59ac21e37e0> + + @pytest.mark.case("success") + def test_c2_create_project_success(logged_in_page, base_url, api_url, api_write, run_id, record_property): + """A new project created in the UI appears with its title and the user as admin.""" + page = logged_in_page + title = f"acceptance C2 {run_id}" + page.goto(base_url + sel.PATHS["new_project"], wait_until="domcontentloaded") + page.click(sel.WEB["create_project_button"]) +> page.fill(sel.WEB["create_project_title"], title) + +installer/acceptance/test_c2_create_project.py:18: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +.venv/lib/python3.12/site-packages/playwright/sync_api/_generated.py:17731: in fill + self._sync( +.venv/lib/python3.12/site-packages/playwright/_impl/page.py:1229: in fill + return await self._main_frame.fill( +.venv/lib/python3.12/site-packages/playwright/_impl/frame.py:736: in fill + return await self._channel.fill( +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:387: in finish + raise e from None +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:352: in finish + result = await zone.run_async(lambda: awaitable) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = <playwright._impl.utils.zones.Zone object at 0xf59ac2060f80> +func = <function ChannelOwner._wrap_api_call.<locals>.finish.<locals>.<lambda> at 0xf59ac21e3ec0> + + async def run_async(self, func: Callable[[], Awaitable[R]]) -> R: + # Python: run() of an async function. The coroutine runs when it is awaited, + # so the zone must be current while it is awaited, not while it is created. + token = *** + try: +> return await func() + ^^^^^^^^^^^^ +E playwright._impl._errors.TimeoutError: Page.fill: Timeout 30000ms exceeded. +E Call log: +E - waiting for locator("input#projectName, input[name='title']") + +.venv/lib/python3.12/site-packages/playwright/_impl/utils/zones.py:58: TimeoutErrortwo_contexts = (<Page url='http://127.0.0.1:5000/emu7y/wiki/home/?edit&view&menu'>, <Page url='http://127.0.0.1:5000/dashboard'>) +login = <function login.<locals>._login at 0xf59ac213fb00> +user_a = {'email': '***', 'password': '***'} +user_b = {'email': '***', 'password': '***'} +base_url = 'http://127.0.0.1:5000', api_url = 'http://127.0.0.1:8000' +api_write = <function api_write.<locals>._write at 0xf59ac18d5080> +add_contributor = <function add_contributor.<locals>._add at 0xf59ac18d5580> +run_id = 'acc-20261110T052932Z-f1da54' +record_property = <function record_property.<locals>.append_property at 0xf59ac18d5e40> + + @pytest.mark.case("success") + def test_c4_wiki_collab_success(two_contexts, login, user_a, user_b, base_url, api_url, api_write, add_contributor, + run_id, record_property): + """Both users open the same wiki page; text typed by B appears in A's editor and text typed by A in B's.""" + page_a, page_b = two_contexts + login(page_a, user_a) + login(page_b, user_b) + + created = api_write(page_a.context, "POST", api_url + sel.API["nodes"], { + "data": {"type": "nodes", "attributes": {"title": "acceptance C4", "category": "project", "public": False}} + }) + assert created.status == 201, created.text()[:200] + node_id = created.json()["data"]["id"] + try: + add_contributor(page_a.context, node_id, user_id_of(page_b.context, api_url), "write") + +> resp_a = _open_editor(page_a, base_url, node_id) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +installer/acceptance/test_c4_wiki_collab.py:53: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +installer/acceptance/test_c4_wiki_collab.py:26: in _open_editor + page.wait_for_selector(sel.WEB["wiki_editor"], timeout=60_000) +.venv/lib/python3.12/site-packages/playwright/sync_api/_generated.py:21403: in wait_for_selector + self._sync( +.venv/lib/python3.12/site-packages/playwright/_impl/page.py:464: in wait_for_selector + return await self._main_frame.wait_for_selector( +.venv/lib/python3.12/site-packages/playwright/_impl/frame.py:418: in wait_for_selector + result = await self._channel.waitForSelector( +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:398: in _mapped + return mapper(await awaitable) + ^^^^^^^^^^^^^^^ +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:387: in finish + raise e from None +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:352: in finish + result = await zone.run_async(lambda: awaitable) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = <playwright._impl.utils.zones.Zone object at 0xf59ac21fb7a0> +func = <function ChannelOwner._wrap_api_call.<locals>.finish.<locals>.<lambda> at 0xf59ac21e3f60> + + async def run_async(self, func: Callable[[], Awaitable[R]]) -> R: + # Python: run() of an async function. The coroutine runs when it is awaited, + # so the zone must be current while it is awaited, not while it is created. + token = *** + try: +> return await func() + ^^^^^^^^^^^^ +E playwright._impl._errors.TimeoutError: Page.wait_for_selector: Timeout 60000ms exceeded. +E Call log: +E - waiting for locator(".CodeMirror textarea, textarea#editor") to be visible + +.venv/lib/python3.12/site-packages/playwright/_impl/utils/zones.py:58: TimeoutErrortwo_contexts = (<Page url='http://127.0.0.1:5000/pk8qs/wiki/home/?edit&view&menu'>, <Page url='about:blank'>) +login = <function login.<locals>._login at 0xf59ac18d7c40> +user_a = {'email': '***', 'password': '***'} +base_url = 'http://127.0.0.1:5000', project = 'pk8qs' +record_property = <function record_property.<locals>.append_property at 0xf59ac0ab7380> + + @pytest.mark.case("rejection") + def test_c4_wiki_collab_rejection(two_contexts, login, user_a, base_url, project, record_property): + """A logged-out context connecting to the wiki websocket is closed with code 4401.""" + page_a, page_anon = two_contexts + login(page_a, user_a) +> _open_editor(page_a, base_url, project) + +installer/acceptance/test_c4_wiki_collab.py:72: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +installer/acceptance/test_c4_wiki_collab.py:26: in _open_editor + page.wait_for_selector(sel.WEB["wiki_editor"], timeout=60_000) +.venv/lib/python3.12/site-packages/playwright/sync_api/_generated.py:21403: in wait_for_selector + self._sync( +.venv/lib/python3.12/site-packages/playwright/_impl/page.py:464: in wait_for_selector + return await self._main_frame.wait_for_selector( +.venv/lib/python3.12/site-packages/playwright/_impl/frame.py:418: in wait_for_selector + result = await self._channel.waitForSelector( +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:398: in _mapped + return mapper(await awaitable) + ^^^^^^^^^^^^^^^ +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:387: in finish + raise e from None +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:352: in finish + result = await zone.run_async(lambda: awaitable) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = <playwright._impl.utils.zones.Zone object at 0xf59ac1fa6ff0> +func = <function ChannelOwner._wrap_api_call.<locals>.finish.<locals>.<lambda> at 0xf59ac21d5c60> + + async def run_async(self, func: Callable[[], Awaitable[R]]) -> R: + # Python: run() of an async function. The coroutine runs when it is awaited, + # so the zone must be current while it is awaited, not while it is created. + token = *** + try: +> return await func() + ^^^^^^^^^^^^ +E playwright._impl._errors.TimeoutError: Page.wait_for_selector: Timeout 60000ms exceeded. +E Call log: +E - waiting for locator(".CodeMirror textarea, textarea#editor") to be visible + +.venv/lib/python3.12/site-packages/playwright/_impl/utils/zones.py:58: TimeoutErrorlogged_in_page = <Page url='http://127.0.0.1:5000/e9fkz/'>, project = '3eqsj' +base_url = 'http://127.0.0.1:5000' +service_url = <function service_url.<locals>._url at 0xf59ac2137e20> +tmp_pdf = b'%PDF-1.4\n1 0 obj<</Type/Catalog/Pages 2 0 R>>endobj\n2 0 obj<</Type/Pages/Kids[3 0 R]/Count 1>>endobj\n3 0 obj<</Ty...%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n' +run_id = 'acc-20261110T052932Z-f1da54' +record_property = <function record_property.<locals>.append_property at 0xf59ac207a480> + + @pytest.mark.case("success") + def test_c5_file_preview_success(logged_in_page, project, base_url, service_url, tmp_pdf, run_id, record_property): + """An uploaded PDF renders inside the mfr iframe (mfr answers 200 for the render request).""" + page = logged_in_page + entity = upload(page.context, service_url("wb"), project, f"c5-{run_id}.pdf", tmp_pdf) + mfr = service_url("mfr") + with page.expect_response(lambda r: same_origin(r.url, mfr) and "/render" in r.url, timeout=90_000) as render: + page.goto(_file_page_url(base_url, project, entity["attributes"]["path"]), wait_until="domcontentloaded") +> page.wait_for_selector(sel.WEB["file_preview_iframe"], timeout=60_000) + +installer/acceptance/test_c5_file_preview.py:24: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +.venv/lib/python3.12/site-packages/playwright/sync_api/_generated.py:21403: in wait_for_selector + self._sync( +.venv/lib/python3.12/site-packages/playwright/_impl/page.py:464: in wait_for_selector + return await self._main_frame.wait_for_selector( +.venv/lib/python3.12/site-packages/playwright/_impl/frame.py:418: in wait_for_selector + result = await self._channel.waitForSelector( +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:398: in _mapped + return mapper(await awaitable) + ^^^^^^^^^^^^^^^ +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:387: in finish + raise e from None +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:352: in finish + result = await zone.run_async(lambda: awaitable) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = <playwright._impl.utils.zones.Zone object at 0xf59ac1b72d20> +func = <function ChannelOwner._wrap_api_call.<locals>.finish.<locals>.<lambda> at 0xf59ac207d440> + + async def run_async(self, func: Callable[[], Awaitable[R]]) -> R: + # Python: run() of an async function. The coroutine runs when it is awaited, + # so the zone must be current while it is awaited, not while it is created. + token = *** + try: +> return await func() + ^^^^^^^^^^^^ +E playwright._impl._errors.TimeoutError: Page.wait_for_selector: Timeout 60000ms exceeded. +E Call log: +E - waiting for locator("iframe#mfrIframe, iframe[src*='render']") to be visible + +.venv/lib/python3.12/site-packages/playwright/_impl/utils/zones.py:58: TimeoutErrorrequest = <SubRequest 'external_guard' for <Function test_c5_file_preview_success[chromium]>> + + @pytest.fixture(autouse=True) + def external_guard(request): + """Collect the loopback-guard offenders of one test and fail it if any guard (browser, websocket, + API) saw a URL that would have left the machine. Every browser context is created by our + `new_context` below (pytest-playwright's default `context` fixture calls it too), which attaches + the guard itself; requesting `context` from here created a fixture cycle + (external_guard -> context -> new_context -> external_guard, first real acceptance run).""" + offenders: list[str] = [] + before = len(_OFFENDERS) + known_before = len(_KNOWN_EXTERNAL) + yield offenders + offenders.extend(_OFFENDERS[before:]) + request.node.user_properties.append(("known_external_blocked", len(_KNOWN_EXTERNAL) - known_before)) + if offenders: +> pytest.fail(f"request(s) would have left the machine: {offenders[:5]}") +E Failed: request(s) would have left the machine: ['http://localhost:7007/text.js', 'http://localhost:7007/share.js'] + +installer/acceptance/conftest.py:274: Failedlogged_in_page = <Page url='http://127.0.0.1:5000/dashboard'>, project = '9s8zt' +service_url = <function service_url.<locals>._url at 0xf59ac2137e20> +tmp_pdf = b'%PDF-1.4\n1 0 obj<</Type/Catalog/Pages 2 0 R>>endobj\n2 0 obj<</Type/Pages/Kids[3 0 R]/Count 1>>endobj\n3 0 obj<</Ty...%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n' +run_id = 'acc-20261110T052932Z-f1da54' +api_context = <function api_context.<locals>._new at 0xf59ac18def20> +record_property = <function record_property.<locals>.append_property at 0xf59ac1e72e80> + + @pytest.mark.case("rejection") + def test_c5_file_preview_rejection(logged_in_page, project, service_url, tmp_pdf, run_id, api_context, record_property): + """The preview URL of a private file without a session -> 401/403.""" + page = logged_in_page + wb = service_url("wb") + entity = upload(page.context, wb, project, f"c5-private-{run_id}.pdf", tmp_pdf) + file_url = download_link(entity, wb, project) + anon = api_context() + resp = anon.get(service_url("mfr") + sel.MFR["render"].format(file_url=file_url)) + record_property("http_status", resp.status) +> assert resp.status in (401, 403), f"anonymous preview returned {resp.status}" +E AssertionError: anonymous preview returned 400 +E assert 400 in (401, 403) +E + where 400 = <APIResponse url='http://127.0.0.1:7778/render?url=http://127.0.0.1:7777/v1/resources/9s8zt/providers/osfstorage/6af2acda8924f6000909d245' status=400 status_text='Bad Request'>.status + +installer/acceptance/test_c5_file_preview.py:39: AssertionErrorpage = <Page url='http://127.0.0.1:8001/institutions/1/users/'> +service_url = <function service_url.<locals>._url at 0xf59ac2137e20> +admin_user = {'email': '***', 'password': '***'} +record_property = <function record_property.<locals>.append_property at 0xf59ac1e71da0> + + @pytest.mark.case("success") + def test_c6_admin_success(page, service_url, admin_user, record_property): + """The institution admin signs in to the admin app and sees the institution's user list.""" + admin_url = service_url("admin") + _admin_login(page, admin_url, admin_user) + # TODO(M2): the institution id of the seeded admin; "1" assumes the first institution created by --seed. + resp = page.goto(admin_url + sel.ADMIN["institution_users"].format(institution_id="1"), wait_until="domcontentloaded") + record_property("http_status", resp.status if resp else None) +> assert resp is not None and resp.status == 200, f"institution users page returned {resp and resp.status}" +E AssertionError: institution users page returned 404 +E assert (<Response url='http://127.0.0.1:8001/institutions/1/users/' request=<Request url='http://127.0.0.1:8001/institutions/1/users/' method='GET'>> is not None and 404 == 200) +E + where 404 = <Response url='http://127.0.0.1:8001/institutions/1/users/' request=<Request url='http://127.0.0.1:8001/institutions/1/users/' method='GET'>>.status + +installer/acceptance/test_c6_admin.py:27: AssertionErrorrequest = <SubRequest 'external_guard' for <Function test_c6_admin_success[chromium]>> + + @pytest.fixture(autouse=True) + def external_guard(request): + """Collect the loopback-guard offenders of one test and fail it if any guard (browser, websocket, + API) saw a URL that would have left the machine. Every browser context is created by our + `new_context` below (pytest-playwright's default `context` fixture calls it too), which attaches + the guard itself; requesting `context` from here created a fixture cycle + (external_guard -> context -> new_context -> external_guard, first real acceptance run).""" + offenders: list[str] = [] + before = len(_OFFENDERS) + known_before = len(_KNOWN_EXTERNAL) + yield offenders + offenders.extend(_OFFENDERS[before:]) + request.node.user_properties.append(("known_external_blocked", len(_KNOWN_EXTERNAL) - known_before)) + if offenders: +> pytest.fail(f"request(s) would have left the machine: {offenders[:5]}") +E Failed: request(s) would have left the machine: ['http://code.ionicframework.com/ionicons/2.0.1/css/ionicons.min.css', 'https://maxcdn.bootstrapcdn.com/font-awesome/4.3.0/css/font-awesome.min.css', 'http://code.ionicframework.com/ionicons/2.0.1/css/ionicons.min.css', 'https://maxcdn.bootstrapcdn.com/font-awesome/4.3.0/css/font-awesome.min.css', 'http://code.ionicframework.com/ionicons/2.0.1/css/ionicons.min.css'] + +installer/acceptance/conftest.py:274: Failedlogged_in_page = <Page url='http://127.0.0.1:5000/dashboard/'> +service_url = <function service_url.<locals>._url at 0xf59ac2137e20> +api_url = 'http://127.0.0.1:8000' +record_property = <function record_property.<locals>.append_property at 0xf59ac0680360> + + @pytest.mark.case("rejection") + def test_c6_admin_rejection(logged_in_page, service_url, api_url, record_property): + """An ordinary user opening the admin URL -> 403 or redirect to login; the API answers exactly 403.""" + client = api(logged_in_page.context) + admin_url = service_url("admin") + screen = client.get(admin_url + sel.ADMIN["institution_users"].format(institution_id="1"), max_redirects=0) +> assert screen.status in (301, 302, 303, 307, 403), f"admin screen answered {screen.status} to an ordinary user" +E AssertionError: admin screen answered 404 to an ordinary user +E assert 404 in (301, 302, 303, 307, 403) +E + where 404 = <APIResponse url='http://127.0.0.1:8001/institutions/1/users/' status=404 status_text='Not Found'>.status + +installer/acceptance/test_c6_admin.py:37: AssertionError \ No newline at end of file diff --git a/docs/demo/runs/acc-20261110T052932Z-f1da54/pytest.log b/docs/demo/runs/acc-20261110T052932Z-f1da54/pytest.log new file mode 100644 index 0000000..2d8fc8c --- /dev/null +++ b/docs/demo/runs/acc-20261110T052932Z-f1da54/pytest.log @@ -0,0 +1,376 @@ +.F...FFFEFFEF. [100%] +==================================== ERRORS ==================================== +_________ ERROR at teardown of test_c5_file_preview_success[chromium] __________ + +request = > + + @pytest.fixture(autouse=True) + def external_guard(request): + """Collect the loopback-guard offenders of one test and fail it if any guard (browser, websocket, + API) saw a URL that would have left the machine. Every browser context is created by our + `new_context` below (pytest-playwright's default `context` fixture calls it too), which attaches + the guard itself; requesting `context` from here created a fixture cycle + (external_guard -> context -> new_context -> external_guard, first real acceptance run).""" + offenders: list[str] = [] + before = len(_OFFENDERS) + known_before = len(_KNOWN_EXTERNAL) + yield offenders + offenders.extend(_OFFENDERS[before:]) + request.node.user_properties.append(("known_external_blocked", len(_KNOWN_EXTERNAL) - known_before)) + if offenders: +> pytest.fail(f"request(s) would have left the machine: {offenders[:5]}") +E Failed: request(s) would have left the machine: ['http://localhost:7007/text.js', 'http://localhost:7007/share.js'] + +installer/acceptance/conftest.py:274: Failed +_____________ ERROR at teardown of test_c6_admin_success[chromium] _____________ + +request = > + + @pytest.fixture(autouse=True) + def external_guard(request): + """Collect the loopback-guard offenders of one test and fail it if any guard (browser, websocket, + API) saw a URL that would have left the machine. Every browser context is created by our + `new_context` below (pytest-playwright's default `context` fixture calls it too), which attaches + the guard itself; requesting `context` from here created a fixture cycle + (external_guard -> context -> new_context -> external_guard, first real acceptance run).""" + offenders: list[str] = [] + before = len(_OFFENDERS) + known_before = len(_KNOWN_EXTERNAL) + yield offenders + offenders.extend(_OFFENDERS[before:]) + request.node.user_properties.append(("known_external_blocked", len(_KNOWN_EXTERNAL) - known_before)) + if offenders: +> pytest.fail(f"request(s) would have left the machine: {offenders[:5]}") +E Failed: request(s) would have left the machine: ['http://code.ionicframework.com/ionicons/2.0.1/css/ionicons.min.css', 'https://maxcdn.bootstrapcdn.com/font-awesome/4.3.0/css/font-awesome.min.css', 'http://code.ionicframework.com/ionicons/2.0.1/css/ionicons.min.css', 'https://maxcdn.bootstrapcdn.com/font-awesome/4.3.0/css/font-awesome.min.css', 'http://code.ionicframework.com/ionicons/2.0.1/css/ionicons.min.css'] + +installer/acceptance/conftest.py:274: Failed +=================================== FAILURES =================================== +___________________ test_c2_create_project_success[chromium] ___________________ + +logged_in_page = +base_url = 'http://127.0.0.1:5000', api_url = 'http://127.0.0.1:8000' +api_write = ._write at 0xf59ac21e3c40> +run_id = 'acc-20261110T052932Z-f1da54' +record_property = .append_property at 0xf59ac21e37e0> + + @pytest.mark.case("success") + def test_c2_create_project_success(logged_in_page, base_url, api_url, api_write, run_id, record_property): + """A new project created in the UI appears with its title and the user as admin.""" + page = logged_in_page + title = f"acceptance C2 {run_id}" + page.goto(base_url + sel.PATHS["new_project"], wait_until="domcontentloaded") + page.click(sel.WEB["create_project_button"]) +> page.fill(sel.WEB["create_project_title"], title) + +installer/acceptance/test_c2_create_project.py:18: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +.venv/lib/python3.12/site-packages/playwright/sync_api/_generated.py:17731: in fill + self._sync( +.venv/lib/python3.12/site-packages/playwright/_impl/page.py:1229: in fill + return await self._main_frame.fill( +.venv/lib/python3.12/site-packages/playwright/_impl/frame.py:736: in fill + return await self._channel.fill( +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:387: in finish + raise e from None +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:352: in finish + result = await zone.run_async(lambda: awaitable) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = +func = .finish.. at 0xf59ac21e3ec0> + + async def run_async(self, func: Callable[[], Awaitable[R]]) -> R: + # Python: run() of an async function. The coroutine runs when it is awaited, + # so the zone must be current while it is awaited, not while it is created. + token = *** + try: +> return await func() + ^^^^^^^^^^^^ +E playwright._impl._errors.TimeoutError: Page.fill: Timeout 30000ms exceeded. +E Call log: +E - waiting for locator("input#projectName, input[name='title']") + +.venv/lib/python3.12/site-packages/playwright/_impl/utils/zones.py:58: TimeoutError +____________________ test_c4_wiki_collab_success[chromium] _____________________ + +two_contexts = (, ) +login = ._login at 0xf59ac213fb00> +user_a = {'email': '***', 'password': '***'} +user_b = {'email': '***', 'password': '***'} +base_url = 'http://127.0.0.1:5000', api_url = 'http://127.0.0.1:8000' +api_write = ._write at 0xf59ac18d5080> +add_contributor = ._add at 0xf59ac18d5580> +run_id = 'acc-20261110T052932Z-f1da54' +record_property = .append_property at 0xf59ac18d5e40> + + @pytest.mark.case("success") + def test_c4_wiki_collab_success(two_contexts, login, user_a, user_b, base_url, api_url, api_write, add_contributor, + run_id, record_property): + """Both users open the same wiki page; text typed by B appears in A's editor and text typed by A in B's.""" + page_a, page_b = two_contexts + login(page_a, user_a) + login(page_b, user_b) + + created = api_write(page_a.context, "POST", api_url + sel.API["nodes"], { + "data": {"type": "nodes", "attributes": {"title": "acceptance C4", "category": "project", "public": False}} + }) + assert created.status == 201, created.text()[:200] + node_id = created.json()["data"]["id"] + try: + add_contributor(page_a.context, node_id, user_id_of(page_b.context, api_url), "write") + +> resp_a = _open_editor(page_a, base_url, node_id) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +installer/acceptance/test_c4_wiki_collab.py:53: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +installer/acceptance/test_c4_wiki_collab.py:26: in _open_editor + page.wait_for_selector(sel.WEB["wiki_editor"], timeout=60_000) +.venv/lib/python3.12/site-packages/playwright/sync_api/_generated.py:21403: in wait_for_selector + self._sync( +.venv/lib/python3.12/site-packages/playwright/_impl/page.py:464: in wait_for_selector + return await self._main_frame.wait_for_selector( +.venv/lib/python3.12/site-packages/playwright/_impl/frame.py:418: in wait_for_selector + result = await self._channel.waitForSelector( +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:398: in _mapped + return mapper(await awaitable) + ^^^^^^^^^^^^^^^ +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:387: in finish + raise e from None +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:352: in finish + result = await zone.run_async(lambda: awaitable) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = +func = .finish.. at 0xf59ac21e3f60> + + async def run_async(self, func: Callable[[], Awaitable[R]]) -> R: + # Python: run() of an async function. The coroutine runs when it is awaited, + # so the zone must be current while it is awaited, not while it is created. + token = *** + try: +> return await func() + ^^^^^^^^^^^^ +E playwright._impl._errors.TimeoutError: Page.wait_for_selector: Timeout 60000ms exceeded. +E Call log: +E - waiting for locator(".CodeMirror textarea, textarea#editor") to be visible + +.venv/lib/python3.12/site-packages/playwright/_impl/utils/zones.py:58: TimeoutError +___________________ test_c4_wiki_collab_rejection[chromium] ____________________ + +two_contexts = (, ) +login = ._login at 0xf59ac18d7c40> +user_a = {'email': '***', 'password': '***'} +base_url = 'http://127.0.0.1:5000', project = 'pk8qs' +record_property = .append_property at 0xf59ac0ab7380> + + @pytest.mark.case("rejection") + def test_c4_wiki_collab_rejection(two_contexts, login, user_a, base_url, project, record_property): + """A logged-out context connecting to the wiki websocket is closed with code 4401.""" + page_a, page_anon = two_contexts + login(page_a, user_a) +> _open_editor(page_a, base_url, project) + +installer/acceptance/test_c4_wiki_collab.py:72: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +installer/acceptance/test_c4_wiki_collab.py:26: in _open_editor + page.wait_for_selector(sel.WEB["wiki_editor"], timeout=60_000) +.venv/lib/python3.12/site-packages/playwright/sync_api/_generated.py:21403: in wait_for_selector + self._sync( +.venv/lib/python3.12/site-packages/playwright/_impl/page.py:464: in wait_for_selector + return await self._main_frame.wait_for_selector( +.venv/lib/python3.12/site-packages/playwright/_impl/frame.py:418: in wait_for_selector + result = await self._channel.waitForSelector( +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:398: in _mapped + return mapper(await awaitable) + ^^^^^^^^^^^^^^^ +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:387: in finish + raise e from None +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:352: in finish + result = await zone.run_async(lambda: awaitable) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = +func = .finish.. at 0xf59ac21d5c60> + + async def run_async(self, func: Callable[[], Awaitable[R]]) -> R: + # Python: run() of an async function. The coroutine runs when it is awaited, + # so the zone must be current while it is awaited, not while it is created. + token = *** + try: +> return await func() + ^^^^^^^^^^^^ +E playwright._impl._errors.TimeoutError: Page.wait_for_selector: Timeout 60000ms exceeded. +E Call log: +E - waiting for locator(".CodeMirror textarea, textarea#editor") to be visible + +.venv/lib/python3.12/site-packages/playwright/_impl/utils/zones.py:58: TimeoutError +____________________ test_c5_file_preview_success[chromium] ____________________ + +logged_in_page = , project = '3eqsj' +base_url = 'http://127.0.0.1:5000' +service_url = ._url at 0xf59ac2137e20> +tmp_pdf = b'%PDF-1.4\n1 0 obj<>endobj\n2 0 obj<>endobj\n3 0 obj<.append_property at 0xf59ac207a480> + + @pytest.mark.case("success") + def test_c5_file_preview_success(logged_in_page, project, base_url, service_url, tmp_pdf, run_id, record_property): + """An uploaded PDF renders inside the mfr iframe (mfr answers 200 for the render request).""" + page = logged_in_page + entity = upload(page.context, service_url("wb"), project, f"c5-{run_id}.pdf", tmp_pdf) + mfr = service_url("mfr") + with page.expect_response(lambda r: same_origin(r.url, mfr) and "/render" in r.url, timeout=90_000) as render: + page.goto(_file_page_url(base_url, project, entity["attributes"]["path"]), wait_until="domcontentloaded") +> page.wait_for_selector(sel.WEB["file_preview_iframe"], timeout=60_000) + +installer/acceptance/test_c5_file_preview.py:24: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +.venv/lib/python3.12/site-packages/playwright/sync_api/_generated.py:21403: in wait_for_selector + self._sync( +.venv/lib/python3.12/site-packages/playwright/_impl/page.py:464: in wait_for_selector + return await self._main_frame.wait_for_selector( +.venv/lib/python3.12/site-packages/playwright/_impl/frame.py:418: in wait_for_selector + result = await self._channel.waitForSelector( +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:398: in _mapped + return mapper(await awaitable) + ^^^^^^^^^^^^^^^ +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:387: in finish + raise e from None +.venv/lib/python3.12/site-packages/playwright/_impl/channel_owner.py:352: in finish + result = await zone.run_async(lambda: awaitable) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = +func = .finish.. at 0xf59ac207d440> + + async def run_async(self, func: Callable[[], Awaitable[R]]) -> R: + # Python: run() of an async function. The coroutine runs when it is awaited, + # so the zone must be current while it is awaited, not while it is created. + token = *** + try: +> return await func() + ^^^^^^^^^^^^ +E playwright._impl._errors.TimeoutError: Page.wait_for_selector: Timeout 60000ms exceeded. +E Call log: +E - waiting for locator("iframe#mfrIframe, iframe[src*='render']") to be visible + +.venv/lib/python3.12/site-packages/playwright/_impl/utils/zones.py:58: TimeoutError +___________________ test_c5_file_preview_rejection[chromium] ___________________ + +logged_in_page = , project = '9s8zt' +service_url = ._url at 0xf59ac2137e20> +tmp_pdf = b'%PDF-1.4\n1 0 obj<>endobj\n2 0 obj<>endobj\n3 0 obj<._new at 0xf59ac18def20> +record_property = .append_property at 0xf59ac1e72e80> + + @pytest.mark.case("rejection") + def test_c5_file_preview_rejection(logged_in_page, project, service_url, tmp_pdf, run_id, api_context, record_property): + """The preview URL of a private file without a session -> 401/403.""" + page = logged_in_page + wb = service_url("wb") + entity = upload(page.context, wb, project, f"c5-private-{run_id}.pdf", tmp_pdf) + file_url = download_link(entity, wb, project) + anon = api_context() + resp = anon.get(service_url("mfr") + sel.MFR["render"].format(file_url=file_url)) + record_property("http_status", resp.status) +> assert resp.status in (401, 403), f"anonymous preview returned {resp.status}" +E AssertionError: anonymous preview returned 400 +E assert 400 in (401, 403) +E + where 400 = .status + +installer/acceptance/test_c5_file_preview.py:39: AssertionError +_______________________ test_c6_admin_success[chromium] ________________________ + +page = +service_url = ._url at 0xf59ac2137e20> +admin_user = {'email': '***', 'password': '***'} +record_property = .append_property at 0xf59ac1e71da0> + + @pytest.mark.case("success") + def test_c6_admin_success(page, service_url, admin_user, record_property): + """The institution admin signs in to the admin app and sees the institution's user list.""" + admin_url = service_url("admin") + _admin_login(page, admin_url, admin_user) + # TODO(M2): the institution id of the seeded admin; "1" assumes the first institution created by --seed. + resp = page.goto(admin_url + sel.ADMIN["institution_users"].format(institution_id="1"), wait_until="domcontentloaded") + record_property("http_status", resp.status if resp else None) +> assert resp is not None and resp.status == 200, f"institution users page returned {resp and resp.status}" +E AssertionError: institution users page returned 404 +E assert (> is not None and 404 == 200) +E + where 404 = >.status + +installer/acceptance/test_c6_admin.py:27: AssertionError +______________________ test_c6_admin_rejection[chromium] _______________________ + +logged_in_page = +service_url = ._url at 0xf59ac2137e20> +api_url = 'http://127.0.0.1:8000' +record_property = .append_property at 0xf59ac0680360> + + @pytest.mark.case("rejection") + def test_c6_admin_rejection(logged_in_page, service_url, api_url, record_property): + """An ordinary user opening the admin URL -> 403 or redirect to login; the API answers exactly 403.""" + client = api(logged_in_page.context) + admin_url = service_url("admin") + screen = client.get(admin_url + sel.ADMIN["institution_users"].format(institution_id="1"), max_redirects=0) +> assert screen.status in (301, 302, 303, 307, 403), f"admin screen answered {screen.status} to an ordinary user" +E AssertionError: admin screen answered 404 to an ordinary user +E assert 404 in (301, 302, 303, 307, 403) +E + where 404 = .status + +installer/acceptance/test_c6_admin.py:37: AssertionError +=============================== warnings summary =============================== +test_c1_login.py::test_c1_login_success[chromium] + test_c1_login.py:10: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c2_create_project.py::test_c2_create_project_success[chromium] + test_c2_create_project.py:11: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c2_create_project.py::test_c2_create_project_rejection[chromium] + test_c2_create_project.py:39: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c3_file_upload.py::test_c3_file_upload_success[chromium] + test_c3_file_upload.py:29: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c3_file_upload.py::test_c3_file_upload_rejection[chromium] + test_c3_file_upload.py:43: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c4_wiki_collab.py::test_c4_wiki_collab_success[chromium] + test_c4_wiki_collab.py:37: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c4_wiki_collab.py::test_c4_wiki_collab_rejection[chromium] + test_c4_wiki_collab.py:67: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c5_file_preview.py::test_c5_file_preview_success[chromium] + test_c5_file_preview.py:16: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c5_file_preview.py::test_c5_file_preview_rejection[chromium] + test_c5_file_preview.py:29: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c6_admin.py::test_c6_admin_success[chromium] + test_c6_admin.py:19: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c6_admin.py::test_c6_admin_rejection[chromium] + test_c6_admin.py:31: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c1_login.py::test_c1_login_rejection + test_c1_login.py:20: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +=========================== short test summary info ============================ +ERROR installer/acceptance/test_c5_file_preview.py::test_c5_file_preview_success[chromium] +ERROR installer/acceptance/test_c6_admin.py::test_c6_admin_success[chromium] +FAILED installer/acceptance/test_c2_create_project.py::test_c2_create_project_success[chromium] +FAILED installer/acceptance/test_c4_wiki_collab.py::test_c4_wiki_collab_success[chromium] +FAILED installer/acceptance/test_c4_wiki_collab.py::test_c4_wiki_collab_rejection[chromium] +FAILED installer/acceptance/test_c5_file_preview.py::test_c5_file_preview_success[chromium] +FAILED installer/acceptance/test_c5_file_preview.py::test_c5_file_preview_rejection[chromium] +FAILED installer/acceptance/test_c6_admin.py::test_c6_admin_success[chromium] +FAILED installer/acceptance/test_c6_admin.py::test_c6_admin_rejection[chromium] +7 failed, 5 passed, 10 deselected, 12 warnings, 2 errors in 278.88s (0:04:38) diff --git a/docs/demo/runs/acc-20261110T052932Z-f1da54/results.json b/docs/demo/runs/acc-20261110T052932Z-f1da54/results.json new file mode 100644 index 0000000..a81d197 --- /dev/null +++ b/docs/demo/runs/acc-20261110T052932Z-f1da54/results.json @@ -0,0 +1,167 @@ +{ + "run_id": "acc-20261110T052932Z-f1da54", + "base_url": "http://127.0.0.1:5000", + "started": "2026-11-10T05:29:32Z", + "duration_ms": 292743, + "only": "core", + "seeded": true, + "precheck": { + "healthy": true, + "services": 24 + }, + "pytest_returncode": 1, + "evidence_dir": "docs/demo/runs/acc-20261110T052932Z-f1da54", + "tracing": "off", + "trace_quarantine": null, + "expected_scenarios": [ + "C1", + "C2", + "C3", + "C4", + "C5", + "C6" + ], + "incomplete_scenarios": [], + "summary": { + "passed": 5, + "failed": 5, + "error": 2, + "skipped": 0, + "total": 12, + "known_external_blocked": 75 + }, + "tests": [ + { + "id": "C1", + "name": "Login via fakecas", + "case": "success", + "test": "test_c1_login_success[chromium]", + "status": "passed", + "duration_ms": 5626, + "http_status": 200, + "evidence": null, + "known_external_blocked": 7 + }, + { + "id": "C2", + "name": "Create project", + "case": "success", + "test": "test_c2_create_project_success[chromium]", + "status": "failed", + "duration_ms": 34432, + "http_status": null, + "evidence": null, + "known_external_blocked": 7 + }, + { + "id": "C2", + "name": "Create project", + "case": "rejection", + "test": "test_c2_create_project_rejection[chromium]", + "status": "passed", + "duration_ms": 5452, + "http_status": 403, + "evidence": null, + "known_external_blocked": 8 + }, + { + "id": "C3", + "name": "Upload and download file", + "case": "success", + "test": "test_c3_file_upload_success[chromium]", + "status": "passed", + "duration_ms": 13162, + "http_status": 200, + "evidence": null, + "known_external_blocked": 4 + }, + { + "id": "C3", + "name": "Upload and download file", + "case": "rejection", + "test": "test_c3_file_upload_rejection[chromium]", + "status": "passed", + "duration_ms": 15381, + "http_status": 403, + "evidence": null, + "known_external_blocked": 8 + }, + { + "id": "C4", + "name": "Wiki collaborative edit", + "case": "success", + "test": "test_c4_wiki_collab_success[chromium]", + "status": "failed", + "duration_ms": 66597, + "http_status": null, + "evidence": null, + "known_external_blocked": 13 + }, + { + "id": "C4", + "name": "Wiki collaborative edit", + "case": "rejection", + "test": "test_c4_wiki_collab_rejection[chromium]", + "status": "failed", + "duration_ms": 66844, + "http_status": null, + "evidence": null, + "known_external_blocked": 13 + }, + { + "id": "C5", + "name": "File preview", + "case": "success", + "test": "test_c5_file_preview_success[chromium]", + "status": "error", + "duration_ms": 64234, + "http_status": null, + "evidence": null, + "known_external_blocked": 8 + }, + { + "id": "C5", + "name": "File preview", + "case": "rejection", + "test": "test_c5_file_preview_rejection[chromium]", + "status": "failed", + "duration_ms": 3587, + "http_status": 400, + "evidence": null, + "known_external_blocked": 4 + }, + { + "id": "C6", + "name": "Institution admin screen", + "case": "success", + "test": "test_c6_admin_success[chromium]", + "status": "error", + "duration_ms": 670, + "http_status": 404, + "evidence": null, + "known_external_blocked": 0 + }, + { + "id": "C6", + "name": "Institution admin screen", + "case": "rejection", + "test": "test_c6_admin_rejection[chromium]", + "status": "failed", + "duration_ms": 2277, + "http_status": null, + "evidence": null, + "known_external_blocked": 3 + }, + { + "id": "C1", + "name": "Login via fakecas", + "case": "rejection", + "test": "test_c1_login_rejection", + "status": "passed", + "duration_ms": 47, + "http_status": 401, + "evidence": null, + "known_external_blocked": 0 + } + ] +} diff --git a/docs/demo/runs/acc-20261110T052932Z-f1da54/results.toon b/docs/demo/runs/acc-20261110T052932Z-f1da54/results.toon new file mode 100644 index 0000000..1f7e559 --- /dev/null +++ b/docs/demo/runs/acc-20261110T052932Z-f1da54/results.toon @@ -0,0 +1,42 @@ +accept: + run_id: acc-20261110T052932Z-f1da54 + base_url: "http://127.0.0.1:5000" + started: "2026-11-10T05:29:32Z" + duration_ms: 292743 + only: core + seeded: true + precheck: + healthy: true + services: 24 + pytest_returncode: 1 + evidence_dir: docs/demo/runs/acc-20261110T052932Z-f1da54 + tracing: off + trace_quarantine: null + expected_scenarios[6]: + - C1 + - C2 + - C3 + - C4 + - C5 + - C6 + incomplete_scenarios[0]: + summary: + passed: 5 + failed: 5 + error: 2 + skipped: 0 + total: 12 + known_external_blocked: 75 + tests[12]{id,name,case,test,status,duration_ms,http_status,evidence,known_external_blocked}: + C1,Login via fakecas,success,"test_c1_login_success[chromium]",passed,5626,200,null,7 + C2,Create project,success,"test_c2_create_project_success[chromium]",failed,34432,null,null,7 + C2,Create project,rejection,"test_c2_create_project_rejection[chromium]",passed,5452,403,null,8 + C3,Upload and download file,success,"test_c3_file_upload_success[chromium]",passed,13162,200,null,4 + C3,Upload and download file,rejection,"test_c3_file_upload_rejection[chromium]",passed,15381,403,null,8 + C4,Wiki collaborative edit,success,"test_c4_wiki_collab_success[chromium]",failed,66597,null,null,13 + C4,Wiki collaborative edit,rejection,"test_c4_wiki_collab_rejection[chromium]",failed,66844,null,null,13 + C5,File preview,success,"test_c5_file_preview_success[chromium]",error,64234,null,null,8 + C5,File preview,rejection,"test_c5_file_preview_rejection[chromium]",failed,3587,400,null,4 + C6,Institution admin screen,success,"test_c6_admin_success[chromium]",error,670,404,null,0 + C6,Institution admin screen,rejection,"test_c6_admin_rejection[chromium]",failed,2277,null,null,3 + C1,Login via fakecas,rejection,test_c1_login_rejection,passed,47,401,null,0 diff --git a/docs/demo/runs/acc-20261110T060511Z-80ee06/collector.json b/docs/demo/runs/acc-20261110T060511Z-80ee06/collector.json new file mode 100644 index 0000000..a2bab7f --- /dev/null +++ b/docs/demo/runs/acc-20261110T060511Z-80ee06/collector.json @@ -0,0 +1,134 @@ +[ + { + "nodeid": "test_c1_login.py::test_c1_login_success[chromium]", + "id": "C1", + "name": "Login via fakecas", + "case": "success", + "status": "passed", + "duration_ms": 6656, + "http_status": 200, + "evidence": null, + "known_external_blocked": 7 + }, + { + "nodeid": "test_c2_create_project.py::test_c2_create_project_success[chromium]", + "id": "C2", + "name": "Create project", + "case": "success", + "status": "passed", + "duration_ms": 8540, + "http_status": 200, + "evidence": null, + "known_external_blocked": 14 + }, + { + "nodeid": "test_c2_create_project.py::test_c2_create_project_rejection[chromium]", + "id": "C2", + "name": "Create project", + "case": "rejection", + "status": "passed", + "duration_ms": 5779, + "http_status": 403, + "evidence": null, + "known_external_blocked": 8 + }, + { + "nodeid": "test_c3_file_upload.py::test_c3_file_upload_success[chromium]", + "id": "C3", + "name": "Upload and download file", + "case": "success", + "status": "passed", + "duration_ms": 9143, + "http_status": 200, + "evidence": null, + "known_external_blocked": 4 + }, + { + "nodeid": "test_c3_file_upload.py::test_c3_file_upload_rejection[chromium]", + "id": "C3", + "name": "Upload and download file", + "case": "rejection", + "status": "passed", + "duration_ms": 11009, + "http_status": 403, + "evidence": null, + "known_external_blocked": 8 + }, + { + "nodeid": "test_c4_wiki_collab.py::test_c4_wiki_collab_success[chromium]", + "id": "C4", + "name": "Wiki collaborative edit", + "case": "success", + "status": "passed", + "duration_ms": 12032, + "http_status": 200, + "evidence": null, + "known_external_blocked": 18 + }, + { + "nodeid": "test_c4_wiki_collab.py::test_c4_wiki_collab_rejection[chromium]", + "id": "C4", + "name": "Wiki collaborative edit", + "case": "rejection", + "status": "passed", + "duration_ms": 9106, + "http_status": 4401, + "evidence": null, + "known_external_blocked": 13 + }, + { + "nodeid": "test_c5_file_preview.py::test_c5_file_preview_success[chromium]", + "id": "C5", + "name": "File preview", + "case": "success", + "status": "passed", + "duration_ms": 10892, + "http_status": 200, + "evidence": null, + "known_external_blocked": 9 + }, + { + "nodeid": "test_c5_file_preview.py::test_c5_file_preview_rejection[chromium]", + "id": "C5", + "name": "File preview", + "case": "rejection", + "status": "passed", + "duration_ms": 3912, + "http_status": 401, + "evidence": null, + "known_external_blocked": 9 + }, + { + "nodeid": "test_c6_admin.py::test_c6_admin_success[chromium]", + "id": "C6", + "name": "Institution admin screen", + "case": "success", + "status": "passed", + "duration_ms": 1061, + "http_status": 200, + "evidence": null, + "known_external_blocked": 5 + }, + { + "nodeid": "test_c6_admin.py::test_c6_admin_rejection[chromium]", + "id": "C6", + "name": "Institution admin screen", + "case": "rejection", + "status": "passed", + "duration_ms": 2442, + "http_status": 403, + "evidence": null, + "known_external_blocked": 3 + }, + { + "nodeid": "test_c1_login.py::test_c1_login_rejection", + "id": "C1", + "name": "Login via fakecas", + "case": "rejection", + "status": "passed", + "duration_ms": 49, + "http_status": 401, + "evidence": null, + "known_external_blocked": 0 + } +] diff --git a/docs/demo/runs/acc-20261110T060511Z-80ee06/junit.xml b/docs/demo/runs/acc-20261110T060511Z-80ee06/junit.xml new file mode 100644 index 0000000..f322862 --- /dev/null +++ b/docs/demo/runs/acc-20261110T060511Z-80ee06/junit.xml @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/demo/runs/acc-20261110T060511Z-80ee06/pytest.log b/docs/demo/runs/acc-20261110T060511Z-80ee06/pytest.log new file mode 100644 index 0000000..3630ca3 --- /dev/null +++ b/docs/demo/runs/acc-20261110T060511Z-80ee06/pytest.log @@ -0,0 +1,40 @@ +............ [100%] +=============================== warnings summary =============================== +test_c1_login.py::test_c1_login_success[chromium] + test_c1_login.py:10: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c2_create_project.py::test_c2_create_project_success[chromium] + test_c2_create_project.py:11: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c2_create_project.py::test_c2_create_project_rejection[chromium] + test_c2_create_project.py:39: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c3_file_upload.py::test_c3_file_upload_success[chromium] + test_c3_file_upload.py:29: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c3_file_upload.py::test_c3_file_upload_rejection[chromium] + test_c3_file_upload.py:43: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c4_wiki_collab.py::test_c4_wiki_collab_success[chromium] + test_c4_wiki_collab.py:55: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c4_wiki_collab.py::test_c4_wiki_collab_rejection[chromium] + test_c4_wiki_collab.py:85: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c5_file_preview.py::test_c5_file_preview_success[chromium] + test_c5_file_preview.py:16: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c5_file_preview.py::test_c5_file_preview_rejection[chromium] + test_c5_file_preview.py:29: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c6_admin.py::test_c6_admin_success[chromium] + test_c6_admin.py:28: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c6_admin.py::test_c6_admin_rejection[chromium] + test_c6_admin.py:48: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c1_login.py::test_c1_login_rejection + test_c1_login.py:20: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +12 passed, 10 deselected, 12 warnings in 80.70s (0:01:20) diff --git a/docs/demo/runs/acc-20261110T060511Z-80ee06/results.json b/docs/demo/runs/acc-20261110T060511Z-80ee06/results.json new file mode 100644 index 0000000..aa62276 --- /dev/null +++ b/docs/demo/runs/acc-20261110T060511Z-80ee06/results.json @@ -0,0 +1,167 @@ +{ + "run_id": "acc-20261110T060511Z-80ee06", + "base_url": "http://127.0.0.1:5000", + "started": "2026-11-10T06:05:11Z", + "duration_ms": 102835, + "only": "core", + "seeded": true, + "precheck": { + "healthy": true, + "services": 24 + }, + "pytest_returncode": 0, + "evidence_dir": "docs/demo/runs/acc-20261110T060511Z-80ee06", + "tracing": "off", + "trace_quarantine": null, + "expected_scenarios": [ + "C1", + "C2", + "C3", + "C4", + "C5", + "C6" + ], + "incomplete_scenarios": [], + "summary": { + "passed": 12, + "failed": 0, + "error": 0, + "skipped": 0, + "total": 12, + "known_external_blocked": 98 + }, + "tests": [ + { + "id": "C1", + "name": "Login via fakecas", + "case": "success", + "test": "test_c1_login_success[chromium]", + "status": "passed", + "duration_ms": 6656, + "http_status": 200, + "evidence": null, + "known_external_blocked": 7 + }, + { + "id": "C2", + "name": "Create project", + "case": "success", + "test": "test_c2_create_project_success[chromium]", + "status": "passed", + "duration_ms": 8540, + "http_status": 200, + "evidence": null, + "known_external_blocked": 14 + }, + { + "id": "C2", + "name": "Create project", + "case": "rejection", + "test": "test_c2_create_project_rejection[chromium]", + "status": "passed", + "duration_ms": 5779, + "http_status": 403, + "evidence": null, + "known_external_blocked": 8 + }, + { + "id": "C3", + "name": "Upload and download file", + "case": "success", + "test": "test_c3_file_upload_success[chromium]", + "status": "passed", + "duration_ms": 9143, + "http_status": 200, + "evidence": null, + "known_external_blocked": 4 + }, + { + "id": "C3", + "name": "Upload and download file", + "case": "rejection", + "test": "test_c3_file_upload_rejection[chromium]", + "status": "passed", + "duration_ms": 11009, + "http_status": 403, + "evidence": null, + "known_external_blocked": 8 + }, + { + "id": "C4", + "name": "Wiki collaborative edit", + "case": "success", + "test": "test_c4_wiki_collab_success[chromium]", + "status": "passed", + "duration_ms": 12032, + "http_status": 200, + "evidence": null, + "known_external_blocked": 18 + }, + { + "id": "C4", + "name": "Wiki collaborative edit", + "case": "rejection", + "test": "test_c4_wiki_collab_rejection[chromium]", + "status": "passed", + "duration_ms": 9106, + "http_status": 4401, + "evidence": null, + "known_external_blocked": 13 + }, + { + "id": "C5", + "name": "File preview", + "case": "success", + "test": "test_c5_file_preview_success[chromium]", + "status": "passed", + "duration_ms": 10892, + "http_status": 200, + "evidence": null, + "known_external_blocked": 9 + }, + { + "id": "C5", + "name": "File preview", + "case": "rejection", + "test": "test_c5_file_preview_rejection[chromium]", + "status": "passed", + "duration_ms": 3912, + "http_status": 401, + "evidence": null, + "known_external_blocked": 9 + }, + { + "id": "C6", + "name": "Institution admin screen", + "case": "success", + "test": "test_c6_admin_success[chromium]", + "status": "passed", + "duration_ms": 1061, + "http_status": 200, + "evidence": null, + "known_external_blocked": 5 + }, + { + "id": "C6", + "name": "Institution admin screen", + "case": "rejection", + "test": "test_c6_admin_rejection[chromium]", + "status": "passed", + "duration_ms": 2442, + "http_status": 403, + "evidence": null, + "known_external_blocked": 3 + }, + { + "id": "C1", + "name": "Login via fakecas", + "case": "rejection", + "test": "test_c1_login_rejection", + "status": "passed", + "duration_ms": 49, + "http_status": 401, + "evidence": null, + "known_external_blocked": 0 + } + ] +} diff --git a/docs/demo/runs/acc-20261110T060511Z-80ee06/results.toon b/docs/demo/runs/acc-20261110T060511Z-80ee06/results.toon new file mode 100644 index 0000000..33db2f2 --- /dev/null +++ b/docs/demo/runs/acc-20261110T060511Z-80ee06/results.toon @@ -0,0 +1,42 @@ +accept: + run_id: acc-20261110T060511Z-80ee06 + base_url: "http://127.0.0.1:5000" + started: "2026-11-10T06:05:11Z" + duration_ms: 102835 + only: core + seeded: true + precheck: + healthy: true + services: 24 + pytest_returncode: 0 + evidence_dir: docs/demo/runs/acc-20261110T060511Z-80ee06 + tracing: off + trace_quarantine: null + expected_scenarios[6]: + - C1 + - C2 + - C3 + - C4 + - C5 + - C6 + incomplete_scenarios[0]: + summary: + passed: 12 + failed: 0 + error: 0 + skipped: 0 + total: 12 + known_external_blocked: 98 + tests[12]{id,name,case,test,status,duration_ms,http_status,evidence,known_external_blocked}: + C1,Login via fakecas,success,"test_c1_login_success[chromium]",passed,6656,200,null,7 + C2,Create project,success,"test_c2_create_project_success[chromium]",passed,8540,200,null,14 + C2,Create project,rejection,"test_c2_create_project_rejection[chromium]",passed,5779,403,null,8 + C3,Upload and download file,success,"test_c3_file_upload_success[chromium]",passed,9143,200,null,4 + C3,Upload and download file,rejection,"test_c3_file_upload_rejection[chromium]",passed,11009,403,null,8 + C4,Wiki collaborative edit,success,"test_c4_wiki_collab_success[chromium]",passed,12032,200,null,18 + C4,Wiki collaborative edit,rejection,"test_c4_wiki_collab_rejection[chromium]",passed,9106,4401,null,13 + C5,File preview,success,"test_c5_file_preview_success[chromium]",passed,10892,200,null,9 + C5,File preview,rejection,"test_c5_file_preview_rejection[chromium]",passed,3912,401,null,9 + C6,Institution admin screen,success,"test_c6_admin_success[chromium]",passed,1061,200,null,5 + C6,Institution admin screen,rejection,"test_c6_admin_rejection[chromium]",passed,2442,403,null,3 + C1,Login via fakecas,rejection,test_c1_login_rejection,passed,49,401,null,0 diff --git a/docs/guides/environment/README.md b/docs/guides/environment/README.md index 29197f6..db73f0e 100644 --- a/docs/guides/environment/README.md +++ b/docs/guides/environment/README.md @@ -5,6 +5,10 @@ **Jump:** [English](#english) · [日本語](#japanese) +- Version / 版: 1.5 — 2026-11-10. **Learned:** a safety check should say what it does not cover: the shared-folder check looks for four names directly under each shared mount point, decoded as the kernel escapes them, not for every credential anywhere ([PR #44 review r1](../../reviews/2026-11-10_RDM-developer-guide_PR44_Per_OS_Installers_r1.md) R1). ／ 安全のための検査は、何を見ないかも言うべきである。共有フォルダの検査は、カーネルのエスケープを復号した各共有 mount 先の直下にある四つの名前だけを探し、どこにある資格情報でも見つけるわけではない（[PR #44 レビュー r1](../../reviews/2026-11-10_RDM-developer-guide_PR44_Per_OS_Installers_r1.md) R1）。 + 1.4 — 2026-11-10. **Learned:** a test VM is only isolated if nothing of the host is shared by default: Lima's Ubuntu template mounted the host home and showed `~/.config/gh` inside the VM, so the Lima steps below share only read-only mirrors and `install-ubuntu.sh` stops when credentials are visible ([DEMO-21](../../demo/21_core_install_ubuntu_arm64_lima.md)). ／ テスト用の VM は、既定でホストの何も共有しないときだけ隔離されている。Lima の Ubuntu テンプレートはホームを mount し `~/.config/gh` を VM から見せたので、下の Lima の手順は読み取り専用のミラーだけを共有し、`install-ubuntu.sh` は資格情報が見えると止まる（[DEMO-21](../../demo/21_core_install_ubuntu_arm64_lima.md)）。 + 1.3 — 2026-11-10. **Learned:** on a Linux server image the acceptance browser needs two more steps than on a Mac, a library for Playwright's driver and Chromium's system libraries, so §3.9 lists them ([DEMO-21](../../demo/21_core_install_ubuntu_arm64_lima.md)). ／ Linux のサーバーイメージでは受入試験のブラウザに Mac より二つ多い手順、Playwright のドライバー用のライブラリと Chromium のシステムライブラリが要るので、§3.9 に載せた（[DEMO-21](../../demo/21_core_install_ubuntu_arm64_lima.md)）。 +- Version / 版: 1.2 — 2026-11-10. **Learned:** telling a reader that the installer checks each prerequisite is not the same as helping them get one, so the guide now starts from a script for each OS ([ADR-08](../../decisions/ADR-08_Per_OS_Installer_Entry_Points.md)). ／ 「インストーラーが前提条件を確認する」と伝えることは、それを揃える手助けにはならない。そこで手引きは OS ごとのスクリプトから始める（[ADR-08](../../decisions/ADR-08_Per_OS_Installer_Entry_Points.md)）。 - Version / 版: 1.1 — 2026-11-06. **Learned:** the Project Metadata edit page needs a waffle flag that no setup step turned on, so the legacy startup steps now set `ember_edit_draft_registration_page` once after the first migration (PR #17); the 2027 installer has no equivalent step yet. ／ Project Metadata の編集画面には、どの手順も有効にしていなかった waffle フラグが必要なため、旧ガイドの起動手順に初回マイグレーション後の `ember_edit_draft_registration_page` 設定を追加した（PR #17）。2027 インストーラーには相当する手順がまだ無いと分かった。 - Version / 版: 1.0 — 2026-11-06. **Learned:** full startup is not yet verified by a recorded run: the ember-osf-web (the browser interface) base image is end-of-life and current Docker (the tool that runs containers: boxes that each hold one program and everything it needs) refuses to pull it, so the manifest (the list of components and the exact source revisions they are built from) already configures a digest-pinned prebuilt image for ember-osf-web as a fallback until M4 modernizes its Dockerfile (DEMO-20, `manifest/upstreams.json`). ／ 完全な起動はまだ実行記録で検証できていない。ember-osf-web（ブラウザ画面）のベースイメージがサポート終了で現行 Docker（コンテナ＝一つのプログラムと必要なものを入れた箱を動かす道具）が取得を拒むため、M4 で Dockerfile を近代化するまでの代替として、manifest（部品と、それを組む正確なソース版の一覧）にはダイジェスト固定の配布済みイメージがすでに設定されている（DEMO-20、`manifest/upstreams.json`）。 - Version / 版: 0.1 — 2026-11-05. Initial bilingual skeleton pointing to the legacy guide. ／ 旧ガイドを指す最初の二言語の骨組み。 @@ -42,6 +46,41 @@ When the installer completes, `grdm-dev up` opens your browser at the web URL it > **Note:** The installer checks each prerequisite. It exits with code 10 if your host is unsupported and 11 if Docker is missing, not running, or lacks the Compose or buildx (multi-platform image builder) plugin. See [SPEC-02 §8](../../spec/02_installer_cli.md#english) for the full host list. +#### Prepare the host with the per-OS script + +If your computer does not have the prerequisites yet, run the script for your OS instead of `install.sh`. Each one shows what it will change, asks before any administrator action, and ends by running `install.sh`. Add `--yes` to consent to everything in advance, or `--dry-run` to only see the plan. Run them from the repository folder. + +| OS | Command | +|---|---| +| Ubuntu 24.04 | `./installer/install-ubuntu.sh` | +| macOS (Apple Silicon) | `./installer/install-macos.sh` | +| Windows 11 (in PowerShell) | `powershell -ExecutionPolicy Bypass -File .\installer\install-windows.ps1` | + +Expected result: the plan is printed, and after your answer the installer's report ends with the web URL. + +On Windows you only need this one script file from the repository. It enables WSL2, installs Ubuntu 24.04, checks Docker Desktop and its WSL integration, and clones the repository inside WSL, never under `/mnt/c`, because the Windows disk is slow from WSL and changes line endings. Read the script before you run it. `-ExecutionPolicy Bypass` applies to this one run only. + +> **Note:** The integration repositories are private. Before the build, give git your own access where the install runs (inside WSL on Windows), for example `gh auth login` and then `gh auth setup-git`. The scripts never ask for, read or store a token. + +#### Trying the Ubuntu script in a Lima VM on a Mac + +[Lima](https://lima-vm.io/) (a tool that runs Linux virtual machines on macOS) is a quick way to test the Ubuntu path on a Mac ([DEMO-21](../../demo/21_core_install_ubuntu_arm64_lima.md)). + +> **Warning:** Lima's Ubuntu template mounts your home directory into the VM (read-only), even when your configuration lists its own `mounts:`, because the template's list is merged in. Your `~/.config/gh/hosts.yml`, which holds your GitHub token, is then readable inside the VM. `install-ubuntu.sh` stops with exit 10 when one of `.config/gh/hosts.yml`, `.git-credentials`, `.netrc` or `.docker/config.json` exists directly under the mount point of a shared folder. It does not search deeper, so the steps below still matter. + +Write the VM configuration, create the VM, then remove the home mount **before** you start it. Run on the Mac: + +```bash +limactl create --name grdm-ubuntu2404 --tty=false my-vm.yaml # base: template:ubuntu-24.04, your own mounts +# open ~/.lima/grdm-ubuntu2404/lima.yaml and delete the line - location: "~" +limactl start grdm-ubuntu2404 +limactl shell grdm-ubuntu2404 -- mount | grep -E 'virtiofs|9p|sshfs' +``` + +Expected result: only the folders you chose are listed; no line ends in your home directory. To give the VM the private repositories without a token, share read-only bare mirrors (`git clone --mirror` on the Mac) and point git at them inside the VM with `git config --global url."file:///.git".insteadOf https://github.com/RCOSDP/.git`. + +> **Note:** Docker Desktop is never installed by these scripts, because its licence depends on your organisation. Homebrew is not installed either; `install-macos.sh` tells you where to get its signed package. See [ADR-08](../../decisions/ADR-08_Per_OS_Installer_Entry_Points.md) for the reasons. + --- ### 2. What you get @@ -226,6 +265,18 @@ The acceptance tests ([SPEC-04](../../spec/04_acceptance_tests.md)) are not run PLAYWRIGHT_BROWSERS_PATH=.venv/.playwright .venv/bin/playwright install chromium ``` +On Linux (Ubuntu 24.04) two more commands are needed, both with `sudo` ([DEMO-21](../../demo/21_core_install_ubuntu_arm64_lima.md)). Run this one **before** the two commands above, because Playwright's driver does not start without it: + +```bash +sudo apt-get install -y libatomic1 +``` + +and this one **after** them, to add Chromium's system libraries (it asks for `sudo` itself): + +```bash +PLAYWRIGHT_BROWSERS_PATH=.venv/.playwright .venv/bin/playwright install-deps chromium +``` + Then run the tests with `--seed`, which creates disposable test users, runs the browser scenarios, and removes those users again within the same run (there is no persistent seed step): ```bash @@ -332,7 +383,7 @@ sudo lsof -i :5000 `doctor` fails if Docker is not installed, not running, or lacks the Compose plugin. -**Fix:** Install Docker Desktop (macOS) or Docker Engine (Ubuntu). Version ≥ 27 with Compose v2 plugin is required. +**Fix:** Install Docker Desktop (macOS, Windows) or Docker Engine (Ubuntu). Version ≥ 27 with Compose v2 plugin is required. On Ubuntu, `./installer/install-ubuntu.sh` installs Docker Engine for you. #### Build failed (exit 30) @@ -401,6 +452,41 @@ cd RDM-developer-guide > **Note:** インストーラーは各前提条件を確認します。ホストが未対応なら終了コード 10、Docker が無い・動いていない・Compose や buildx（複数 CPU 向けイメージを作る機能）プラグインが無いなら 11 で終わります。対応ホストの一覧は [SPEC-02 §8](../../spec/02_installer_cli.md#japanese) を参照。 +#### OS ごとのスクリプトでホストを準備する + +前提条件がまだ揃っていないコンピュータでは、`install.sh` の代わりに OS に合ったスクリプトを実行します。どれも変更点を示し、管理者の操作の前に尋ね、最後に `install.sh` を実行します。`--yes` を付けると先にすべて同意し、`--dry-run` を付けると予定を見るだけです。リポジトリのフォルダで実行します。 + +| OS | コマンド | +|---|---| +| Ubuntu 24.04 | `./installer/install-ubuntu.sh` | +| macOS（Apple Silicon） | `./installer/install-macos.sh` | +| Windows 11（PowerShell で） | `powershell -ExecutionPolicy Bypass -File .\installer\install-windows.ps1` | + +期待する結果: 予定が表示され、答えた後にインストーラーの報告が web の URL で終わります。 + +Windows で必要なのは、リポジトリのこのスクリプトファイル一つだけです。WSL2 を有効にし、Ubuntu 24.04 を入れ、Docker Desktop と WSL 連携を確認し、リポジトリを WSL の中に clone します。Windows のディスクは WSL から遅く改行も変わるので、`/mnt/c` の下には決して置きません。実行する前にスクリプトを読んでください。`-ExecutionPolicy Bypass` はこの一回の実行だけに効きます。 + +> **Note:** 統合リポジトリは非公開です。ビルドの前に、導入を実行する場所（Windows では WSL の中）で git に自分の権限を設定してください。例: `gh auth login` の後に `gh auth setup-git`。スクリプトはトークンを尋ねず、読まず、保存しません。 + +#### Mac の Lima VM で Ubuntu のスクリプトを試す + +[Lima](https://lima-vm.io/)（macOS で Linux の仮想マシンを動かす道具）を使うと、Mac で Ubuntu の手順を手早く試せます（[DEMO-21](../../demo/21_core_install_ubuntu_arm64_lima.md)）。 + +> **Warning:** Lima の Ubuntu テンプレートは、設定に独自の `mounts:` を書いても一覧が併合されるため、ホームディレクトリを VM に（読み取り専用で）mount します。すると GitHub のトークンを持つ `~/.config/gh/hosts.yml` が VM の中から読めます。`install-ubuntu.sh` は、共有フォルダの mount 先の直下に `.config/gh/hosts.yml`・`.git-credentials`・`.netrc`・`.docker/config.json` のどれかがあると終了 10 で止まります。それより深くは探さないので、下の手順はやはり必要です。 + +VM の設定を書いて VM を作り、起動する**前に**ホームの mount を外します。Mac で実行します： + +```bash +limactl create --name grdm-ubuntu2404 --tty=false my-vm.yaml # base: template:ubuntu-24.04 と独自の mounts +# ~/.lima/grdm-ubuntu2404/lima.yaml を開き、 - location: "~" の行を消す +limactl start grdm-ubuntu2404 +limactl shell grdm-ubuntu2404 -- mount | grep -E 'virtiofs|9p|sshfs' +``` + +期待する結果: 自分で選んだフォルダだけが表示され、ホームディレクトリで終わる行はありません。トークン無しで VM に非公開リポジトリを渡すには、読み取り専用の bare ミラー（Mac で `git clone --mirror`）を共有し、VM の中で `git config --global url."file:///<ミラー>.git".insteadOf https://github.com/RCOSDP/<リポジトリ>.git` として git をそこへ向けます。 + +> **Note:** これらのスクリプトは Docker Desktop を入れません。ライセンスが所属組織によるからです。Homebrew も入れません。`install-macos.sh` が署名付きパッケージの入手先を案内します。理由は [ADR-08](../../decisions/ADR-08_Per_OS_Installer_Entry_Points.md) にあります。 + --- ### 2. 得られるもの @@ -585,6 +671,18 @@ manifest とプロファイルから Docker Compose ファイルを生成しま PLAYWRIGHT_BROWSERS_PATH=.venv/.playwright .venv/bin/playwright install chromium ``` +Linux（Ubuntu 24.04）では、`sudo` を使うコマンドがあと二つ要ります（[DEMO-21](../../demo/21_core_install_ubuntu_arm64_lima.md)）。次のコマンドは上の二つの**前に**実行します。これが無いと Playwright のドライバーが起動しません： + +```bash +sudo apt-get install -y libatomic1 +``` + +次のコマンドは上の二つの**後に**実行し、Chromium のシステムライブラリを入れます（自分で `sudo` を求めます）： + +```bash +PLAYWRIGHT_BROWSERS_PATH=.venv/.playwright .venv/bin/playwright install-deps chromium +``` + 次に `--seed` を付けて試験を実行します。`--seed` は使い捨ての試験利用者を作成し、ブラウザのシナリオを実行し、同じ一回の実行の中でその利用者を削除します（事前に別途「シード」しておく手順はありません）： ```bash @@ -691,7 +789,7 @@ sudo lsof -i :5000 `doctor` は Docker がインストールされていない、動いていない、または Compose プラグインがないと失敗します。 -**対応:** Docker Desktop（macOS）または Docker Engine（Ubuntu）をインストール。版 ≥ 27、Compose v2 プラグインが必要です。 +**対応:** Docker Desktop（macOS・Windows）または Docker Engine（Ubuntu）をインストール。版 ≥ 27、Compose v2 プラグインが必要です。Ubuntu では `./installer/install-ubuntu.sh` が Docker Engine を入れます。 #### ビルド失敗（終了コード 30） diff --git a/docs/reviews/2026-11-10_RDM-developer-guide_PR44_Per_OS_Installers_r1.md b/docs/reviews/2026-11-10_RDM-developer-guide_PR44_Per_OS_Installers_r1.md new file mode 100644 index 0000000..6da698d --- /dev/null +++ b/docs/reviews/2026-11-10_RDM-developer-guide_PR44_Per_OS_Installers_r1.md @@ -0,0 +1,103 @@ +# PR44 Per-OS installers review r1 / OS 別インストーラーレビュー r1 + +> **Summary:** REV reviewed PR #44 at c2607e6. Changes are requested for path validation and the ZAP permission boundary. +> **概要:** REV は PR #44 の c2607e6 を確認した。パス検証と ZAP の権限境界について修正を求める。 + +**Jump:** [English](#english) · [日本語](#japanese) + +- Version / 版: 0.1 — 2026-11-10. **Learned:** passing dry-run tests does not prove filesystem safety when mount names are escaped, paths are symlinks, or parent permissions differ from the assumed mode. ／ dry run の合格だけでは、mount 名のエスケープ、パスのシンボリックリンク、親の権限が想定と異なる場合の安全性は証明できない。 +- Applies to: [PR #44](https://github.com/RCOSDP/RDM-developer-guide/pull/44), `feat/per-os-installers-split`, head `c2607e60d3b9459d8a9132c662a72d838acb6f17`. +- Reviewer / レビュワ: REV（担当レビュワ, Codex Astra）; 2026-11-10. +- Parent index / 親索引: [Reviews](README.md). +- Related: [SPEC-02](../spec/02_installer_cli.md) · [SPEC-06](../spec/06_documentation_conventions.md) · [ADR-08](../decisions/ADR-08_Per_OS_Installer_Entry_Points.md) · [DEMO-21](../demo/21_core_install_ubuntu_arm64_lima.md). +- Used by / 利用先: PR #44 revision and coordinator review / PR #44 の修正と統括レビュー。 + +--- + + +## English + +[→ 日本語版へ](#japanese) + +### Verdict and scope + +**Changes requested.** Reviewed the four prerequisite scripts, attributes, Python changes and their tests, ADR-08, SPEC-02/04, environment guidance and DEMO-21 evidence against `origin/develop`. Work used a clean scratch clone. No commit, push, approval, merge, stack operation or inspection of a real runtime credential file was performed. Tests used temporary fixtures; focused reproductions used dummy credentials and stub commands only. + +### Required changes + +**R1 — P2: Decode mount-table paths before testing credentials.** In `installer/install-ubuntu.sh:89–95`, `read -r` preserves the kernel's `\040` encoding, but the existence test treats it as literal filename characters. A shared home containing a space therefore bypasses exit 10. A fixture with a real dummy `.config/gh/hosts.yml` under `host home` and an escaped virtiofs mount entry continued into the stub apt actions and eventually exited 11, without the credential refusal. Decode the mount-field escapes safely and test spaces, backslashes and the supported filesystem types. Keep checking existence only. The check also covers only four names directly below a mount root; describe that limit rather than implying recursive detection of all credentials. + +**R2 — P2: Resolve WSL destinations before cloning or invoking the installer.** `installer/install-windows.ps1:234–246` checks the textual HOME and accepts an existing target directory; `installer/prereq-lib.sh:22` computes a logical path with `pwd`, which `install-ubuntu.sh:77–79` compares with `/mnt/*`. A home or target symlink to `/mnt/c/...` passes these textual checks. A scratch alias confirmed that PREREQ_ROOT retains the symlink spelling while `pwd -P` points elsewhere. The Windows bootstrap can consequently clone on the Windows disk before a downstream permission probe rejects it. Resolve the physical home and existing target, validate the physical parent for a new target, and apply the same physical-path guard in Ubuntu. Add symlink cases; the current test only asserts that the guard text exists. This is static analysis plus a local path reproduction, not a Windows execution claim. + +**R3 — P2: Enforce the parent permission boundary before making ZAP work directories world-writable.** `installer/grdm_dev/scan.py:905–917` uses `mkdir(exist_ok=True)` and `chmod(0777)` without checking that runtime is an owned, real 0700 directory. Normal `secrets init` establishes 0700, but `scan --dynamic` does not enforce it and an existing/copied runtime can be 0755. A direct stubbed run confirmed runtime remained 0755 while each work directory became 0777. Other local users can then write into a known work directory during the scan, substituting report contents or report symlinks. A genuine 0700 ancestor prevents this cross-user access; 0777 is acceptable only while that invariant actually holds. Validate ownership, physical directory identity and permissions before widening access; create work directories exclusively with unpredictable names and reject output symlinks. Add permissive-parent and pre-existing-symlink tests. The current `nosemgrep` rationale assumes rather than establishes the invariant. + +### Other review observations + +Consent precedes the planned sudo, administrator and package operations, with a separate root-equivalence warning for docker-group membership. `--yes` is explicit consent. Existing prerequisites are skipped, unsupported platforms fail, and downloads are not piped into a shell. Docker's apt key requires exactly one primary key and the pinned fingerprint; its source uses Signed-By. Scanner archives are checked against embedded SHA-256 values before extraction. macOS delegates verification to Homebrew and Windows uses Microsoft WSL distribution installation. Upstream artifact bytes and signatures were not independently downloaded during this review. + +The binfmt image is digest-pinned identically in the script and doctor hint. The one-shot commands use `--rm`, install only amd64 and uninstall only qemu-x86_64, with no host-directory or Docker-socket mount. They still grant full privileged-container access and change a host-global kernel registration; this is not capability-minimal isolation. The plan explains sudo, the purpose, replacement and reboot lifetime, but should explicitly name the privileged-container trust and effect on other users' workloads. ADR-08 remains Proposed for owner approval. Clarify the environment guide's blanket privileged prohibition as applying to application services, with the narrowly documented host-setup exception. + +The longer Linux arm64 timeout preserves explicit overrides and is limited to profiles with amd64 services. The libatomic1 addition and Linux Playwright dependency guidance match the documented failure. Tokens are only discussed as guidance; the shared-folder guard checks existence, not contents. The attributes explicitly protect Linux-executed files from CRLF conversion. + +DEMO-21's final results record 12 passed, zero failed/error/skipped and pytest exit 0. Its earlier failed runs remain visible. The 531-object, uid-2001 database migration and successful installation are author-recorded VM evidence, not independently rerun here. Ubuntu amd64 and real Windows remain unverified; deferred workflows and their helper are explicitly documented as pending owner credentials. They are not credited as CI evidence. Bilingual layout and Proposed status are present; documentation checks pass with warnings. + +### Validation + +Commands ran from the scratch clone root using system `/usr/bin/python3` 3.9.6. Results are review observations, not a mirror of a canonical dataset. + +```toon +checks[6]{command,result,exit}: + "PYTHONPATH=installer /usr/bin/python3 -m unittest discover -s installer/tests","632 tests; 10 skipped; OK",0 + "/usr/bin/python3 -m unittest discover -s tools/tests","93 tests; OK",0 + "/usr/bin/python3 tools/docs_check.py","151 files; 8 exempt; 38 warnings; passed before adding review",0 + "/usr/bin/python3 tools/manifest_check.py","passed; zero policy findings",0 + "bash -n installer/install-ubuntu.sh installer/install-macos.sh installer/prereq-lib.sh installer/install.sh","passed",0 + "command -v shellcheck","not installed; not run",1 +``` + +PowerShell was unavailable, so no live parser or WSL execution was claimed. Full logs are retained beside the clone in `/tmp/pr44-rev-DJJ5gv/`. Re-run focused tests and obtain owner-platform evidence after fixes; no runtime installation is required to reproduce R1–R3. + +--- + + +## 日本語 + +[→ To English](#english) + +### 判定と範囲 + +**Changes requested（修正要求）。** 四つの前提条件スクリプト、属性、Python の変更と試験、ADR-08、SPEC-02/04、環境手引き、DEMO-21 の証跡を `origin/develop` と比較した。作業は scratch のクリーンな clone で行った。commit・push・承認・merge・スタック操作・実際の runtime 資格情報ファイルの閲覧はしていない。試験は一時 fixture を使い、個別再現はダミーの資格情報とスタブだけを使った。 + +### 必要な修正 + +**R1 — P2: 資格情報を調べる前に mount 表のパスを復号する。** `installer/install-ubuntu.sh:89–95` の `read -r` はカーネルの `\040` を保持するが、存在検査はそれを文字どおりのファイル名として扱う。共有ホーム名に空白があると終了 10 を通過する。`host home` の下にダミーの `.config/gh/hosts.yml` を置き、virtiofs の mount 行をエスケープした fixture は、資格情報の拒否を出さずに apt のスタブへ進み、最終的に終了 11 になった。mount フィールドのエスケープを安全に復号し、空白・バックスラッシュ・対応するファイルシステムを試験すること。内容は読まず存在確認を維持すること。また検出対象は mount 直下の四つの名前だけなので、全資格情報を再帰的に検出するとの印象を避け、この制限を明記すること。 + +**R2 — P2: clone や実行の前に WSL の実際の配置先を解決する。** `installer/install-windows.ps1:234–246` は HOME の文字列を検査して既存の対象を受け入れる。`installer/prereq-lib.sh:22` は `pwd` で論理パスを作り、`install-ubuntu.sh:77–79` はそれを `/mnt/*` と比較する。ホームや対象が `/mnt/c/...` へのシンボリックリンクなら、この文字列検査を通る。scratch の別名で、PREREQ_ROOT はリンク名を維持し、`pwd -P` は別の実体を指すことを確認した。後段の権限検査が拒否する前に、Windows の起動処理が Windows ディスクへ clone できる。ホームと既存対象の実体を解決し、新規対象は親の実体を検査すること。Ubuntu 側も実体パスを検査し、リンクの試験を加えること。現在の試験は検査文字列の存在を assert するだけである。これは静的解析とローカルのパス再現であり、Windows 実機試験ではない。 + +**R3 — P2: ZAP 作業ディレクトリを全員書込み可にする前に親の権限境界を保証する。** `installer/grdm_dev/scan.py:905–917` は、runtime が実行者所有の実ディレクトリで 0700 かを確認せず、`mkdir(exist_ok=True)` と `chmod(0777)` を行う。通常の `secrets init` は 0700 にするが、`scan --dynamic` はそれを強制せず、既存・コピー済みの runtime は 0755 の場合がある。直接のスタブ実行で runtime が 0755 のまま各作業ディレクトリが 0777 になることを確認した。この場合、別のローカル利用者がスキャン中の既知の作業ディレクトリへ書込み、報告の内容や報告ファイルのリンクを差し替えられる。実際に 0700 の祖先があれば利用者間のアクセスを防げるので、0777 を許せるのはその条件が成立する間だけである。権限を広げる前に所有者・実体・モードを検証し、予測できない名前で排他的に作業ディレクトリを作り、出力リンクを拒否すること。親が公開モードの場合と既存リンクの試験を加えること。現在の `nosemgrep` の理由は条件を保証せず仮定している。 + +### その他の確認 + +予定された sudo・管理者・パッケージ操作の前に同意を求め、docker グループへの追加では管理者と同等になるとの警告を別に示す。`--yes` は明示的な同意である。既存の前提条件は飛ばし、未対応ホストは拒否し、ダウンロードをシェルへ流し込まない。Docker の apt 鍵は主鍵一つと固定指紋の一致を要求し、配布元は Signed-By を使う。スキャナーは展開前に固定 SHA-256 を検査する。macOS は Homebrew の検証を使い、Windows は Microsoft の WSL 配布機構を使う。今回は上流の配布物や署名を独立にダウンロードして照合していない。 + +binfmt のイメージはスクリプトと doctor の案内で同じダイジェストに固定されている。一時コマンドは `--rm` を使い、amd64 だけを登録し、qemu-x86_64 だけを解除する。ホストのディレクトリや Docker ソケットの mount はない。ただしコンテナには完全な privileged 権限を与え、ホスト全体のカーネル登録を変えるため、最小 capability による隔離ではない。計画は sudo・目的・置換・再起動までの寿命を説明するが、privileged コンテナを信頼することと他の利用者の処理への影響も明示すべきである。ADR-08 は所有者承認待ちの Proposed のままである。環境手引きの全面的な privileged 禁止はアプリケーションサービスへの規則と明確にし、ホスト準備の狭い例外を記録するとよい。 + +Linux arm64 の待ち時間延長は明示の上書きを維持し、amd64 サービスを持つプロファイルに限定される。libatomic1 と Linux 向け Playwright 依存関係の案内は記録された失敗に対応する。トークンは案内だけで扱い、共有フォルダの検査は内容を読まない。属性は Linux で実行するファイルを明示的に CRLF 変換から保護する。 + +DEMO-21 の最終結果は 12 件合格、失敗・エラー・skip がゼロ、pytest 終了 0 を記録している。以前の失敗も残されている。uid 2001・531 対象の DB 移行と導入成功は著者の VM 記録であり、今回再実行していない。Ubuntu amd64 と Windows 実機は未検証で、ワークフローと補助ファイルは所有者の資格情報待ちと明記されているため、CI の証拠には数えない。英和構成と Proposed 状態を確認し、文書検査は警告付きで合格した。 + +### 検証 + +clone のルートからシステムの `/usr/bin/python3` 3.9.6 で実行した。結果はレビュー時の観測であり、別の正本データの写しではない。 + +```toon +checks[6]{command,result,exit}: + "PYTHONPATH=installer /usr/bin/python3 -m unittest discover -s installer/tests","632 件・10 skip・合格",0 + "/usr/bin/python3 -m unittest discover -s tools/tests","93 件・合格",0 + "/usr/bin/python3 tools/docs_check.py","レビュー追加前 151 ファイル・8 免除・38 警告・合格",0 + "/usr/bin/python3 tools/manifest_check.py","合格・ポリシー検出ゼロ",0 + "bash -n installer/install-ubuntu.sh installer/install-macos.sh installer/prereq-lib.sh installer/install.sh","合格",0 + "command -v shellcheck","未導入・実行せず",1 +``` + +PowerShell も無く、パーサーや WSL の実行はしていない。ログは clone と同階層の `/tmp/pr44-rev-DJJ5gv/` に保存した。修正後は対象試験と所有者の実機検証を行うこと。R1〜R3 の再現に実環境の導入は不要である。 diff --git a/docs/reviews/2026-11-10_RDM-developer-guide_PR44_Per_OS_Installers_r2.md b/docs/reviews/2026-11-10_RDM-developer-guide_PR44_Per_OS_Installers_r2.md new file mode 100644 index 0000000..28eeb05 --- /dev/null +++ b/docs/reviews/2026-11-10_RDM-developer-guide_PR44_Per_OS_Installers_r2.md @@ -0,0 +1,101 @@ +# PR44 Per-OS installers review r2 / OS 別インストーラーレビュー r2 + +> **Summary:** REV requests changes: trailing newlines still bypass the shared-credential check, and a FIFO report blocks ZAP report validation. +> **概要:** REV は修正を要求する。共有パス末尾の改行で資格情報検査を回避でき、FIFO の ZAP 報告では検証が停止する。 + +**Jump:** [English](#english) · [日本語](#japanese) + +- Version / 版: 0.1 — 2026-11-10. **Learned:** decoding an escaped path is insufficient if command substitution then strips its trailing newlines; validating a file after opening it is insufficient if opening can block. ／ 復号してもコマンド置換が末尾改行を消せばパスは保たれず、open 自体が停止する場合は open 後のファイル種別検証では足りない。 +- Applies to: [PR #44](https://github.com/RCOSDP/RDM-developer-guide/pull/44), diff `c2607e6..237387f`, head `237387f8767ddfc500e1ba71c1ab8fe50a4c42f2`. +- Reviewer / レビュワ: REV（担当レビュワ, Codex Astra）. +- Parent index / 親索引: [Reviews](README.md). +- Related: [r1](2026-11-10_RDM-developer-guide_PR44_Per_OS_Installers_r1.md) · [SPEC-02](../spec/02_installer_cli.md) · [SPEC-03](../spec/03_security_gate.md) · [SPEC-06](../spec/06_documentation_conventions.md). +- Used by / 利用先: PR #44 revision and coordinator review / PR #44 の修正と統括レビュー。 + +--- + + +## English + +[→ 日本語版へ](#japanese) + +### Verdict and required changes + +**Changes requested.** Reviewed the requested revision in a clean detached clone. R1 is only partially closed; R2's symlink fix passes the exercised cases; R3's permission and symlink protections pass, but its new regular-file reader has a blocking special-file case. + +**R2-01 — P2: Preserve trailing newlines in decoded mount paths.** At `installer/install-ubuntu.sh:105`, `mnt="$(decode_mount_field ...)"` removes all trailing newline bytes from the decoded result. A real fixture directory named `host` followed by a newline, containing a dummy `.netrc`, is represented in the mount table as `host\012`. The script checks `host/.netrc` instead. A non-dry invocation under stub commands bypassed exit 10, reached stub sudo, and eventually exited 11 for all six supported filesystem names. Space, tab, interior newline and a literal backslash followed by `012` were correctly detected; one and two trailing newlines were not. Assign the decoded value without lossy command substitution, or append/remove a non-newline sentinel. Add regression cases for trailing `\012` and repeated trailing `\012`; decoding must remain single-pass for `\134012`. + +**R2-02 — P2: Open special reports without blocking before checking their type.** At `installer/grdm_dev/scan.py:926`, `os.open(path, O_RDONLY | O_NOFOLLOW)` blocks indefinitely on a FIFO with no writer. The `fstat` regular-file check at lines 931–933 is never reached. An isolated child calling `_read_zap_report` on a temporary FIFO exceeded two seconds and was terminated by the test harness. A scanner-produced FIFO therefore stalls completion rather than reporting an incomplete scan, contrary to the updated SPEC-03 guarantee that nonregular reports are refused. Use a nonblocking, no-follow open followed by descriptor type validation, retaining refusal and cleanup; add a bounded FIFO test. This is a robustness defect, not a claim that another host user can cross the verified private parent. + +### Closure evidence and boundary variants + +R1 now correctly decodes spaces, tabs and backslashes and documents its four-name, mount-root-only scope. A synthetic shared-filesystem bind entry with a space in its mount point was refused with exit 10. The implementation reads `/proc/mounts`, not `/proc/self/mountinfo`; the fixtures used its actual four-field parsing contract. + +R2 resolves HOME and an existing target with `pwd -P`, validates before cloning, and uses the canonical target thereafter. Nested links with both absolute and relative destinations into the simulated Windows root exited 10. Replacing the original logical checkout alias after validation still executed the original safe canonical checkout. The repository tests also cover direct and symlinked Ubuntu entry points and an accepted Linux destination. `-Directory` is restricted to one folder name, so a new target's parent is the validated physical HOME. + +A physical-path prefix is not a filesystem-origin check: a Windows filesystem bind-mounted outside `/mnt` would retain that outside mount-point pathname under `pwd -P`. This is a static limitation, not an independently reproduced Linux bind-mount result or an additional proven regression. Actual bind mounts and WSL execution were unavailable on this Darwin host; no host mounts or Docker state were changed. + +R3 rejects a 0755 runtime before creating any 0777 work directory or running ZAP, uses exclusive unpredictable directories, and rejects report symlinks. Absolute, relative, nested and dangling report symlinks all returned a refusal. Replacing a regular report with a symlink immediately at `os.open` was also refused by `O_NOFOLLOW`. Existing tests verify scan exit 51, cleanup and that planted report findings are not accepted. + +A deliberate same-owner chmod from 0700 to 0755 between validation and use allowed creation of a 0777 work directory and a successful stubbed scan. Thus validation is a snapshot, not an invariant maintained against concurrent owner changes. No cross-user exploit is claimed: under the ordinary owner-controlled checkout and private runtime, an unrelated user cannot perform that chmod. Parent-path replacement by an actor able to rename checkout ancestors and bind aliases remain outside the demonstrated protection. Treat the owned checkout/ancestors as a precondition; do not claim general race-proof filesystem isolation. + +### Validation and reproduction + +System `/usr/bin/python3` 3.9.6 on Darwin was used. Results are observations, not a canonical dataset mirror. All scanner, installer and credential boundary probes used temporary fixtures and stub runners; no real credential file was inspected. No commit, push, approval, merge, close or grdm-dev stack mutation was performed. PowerShell was not installed, so its native parser and real WSL were not run. + +```toon +checks[7]{command,result,exit}: + "PYTHONPATH=installer /usr/bin/python3 -m unittest discover -s installer/tests","640 tests; 10 skipped; OK",0 + "/usr/bin/python3 -m unittest discover -s tools/tests","93 tests; OK",0 + "/usr/bin/python3 tools/docs_check.py","152 files; 8 exempt; 38 warnings; passed before review",0 + "/usr/bin/python3 tools/manifest_check.py","694 tracked text files scanned; 63 skipped; zero findings",0 + "bash -n installer/install-ubuntu.sh installer/install-macos.sh installer/prereq-lib.sh installer/install.sh","passed",0 + "/usr/bin/python3 /tmp/pr44-rev-r2-RXhzlP/variants.py","boundary observations including trailing-newline bypass and FIFO timeout",0 + "/usr/bin/python3 /tmp/pr44-rev-r2-RXhzlP/mount_variants.py","six filesystem bypasses; bind-table fixture refused",0 +``` + +Scripts and full logs are retained in `/tmp/pr44-rev-r2-RXhzlP/`; clone root is `/tmp/pr44-rev-r2-RXhzlP/repo`. From that root, run the last two commands to reproduce the boundary observations; their exit 0 means the observation harness completed, not that every product case passed. Fix R2-01/R2-02 and rerun the focused probes. Real platform and bind-mount coverage remains outstanding. + +--- + + +## 日本語 + +[→ To English](#english) + +### 判定と修正要求 + +**Changes requested（修正要求）。** 指定差分をクリーンな detached clone で確認した。R1 は部分解決、R2 の symlink 修正は今回の試験で合格、R3 の権限・symlink 防御も合格したが、新しい通常ファイル読込み処理には特殊ファイルで停止する問題がある。 + +**R2-01 — P2: 復号した mount パスの末尾改行を保持すること。** `installer/install-ubuntu.sh:105` の `mnt="$(decode_mount_field ...)"` は復号結果の末尾改行をすべて取り除く。`host` の後ろに改行がある実ディレクトリにダミーの `.netrc` を置き、mount 表に `host\012` と記すと、検査先は `host/.netrc` になってしまう。スタブ環境で dry run を付けずに実行すると、対応する全六種類の filesystem で終了 10 を回避し、sudo スタブへ進んで最終的に終了 11 となった。空白・タブ・途中の改行・文字どおりのバックスラッシュと `012` は検出できたが、末尾の改行一つ・二つは検出できなかった。値を直接代入するか、改行以外の番兵を付けてから除去し、末尾改行を保持すること。末尾 `\012` と連続する末尾 `\012` を回帰試験に加え、`\134012` を二重に復号しない性質も維持すること。 + +**R2-02 — P2: 特殊な報告ファイルを、種別検証前に停止せず開くこと。** `installer/grdm_dev/scan.py:926` の `os.open(path, O_RDONLY | O_NOFOLLOW)` は書き手のない FIFO で無期限に待つ。931–933 行の `fstat` による通常ファイル検証に到達しない。一時 FIFO を `_read_zap_report` に渡した隔離子プロセスは二秒を超え、試験側が終了させた。スキャナーが FIFO を残すと、不完全なスキャンとして報告せず停止するため、SPEC-03 の「通常ファイルでない報告は拒否」という追加保証を満たさない。nonblocking と no-follow で開いてから descriptor の種別を検証し、拒否と後片付けを維持すること。時間制限付きの FIFO 試験を加えること。これは堅牢性の問題であり、別のホスト利用者が private な親を越えられるという指摘ではない。 + +### 解決の証拠と境界ケース + +R1 は空白・タブ・バックスラッシュを正しく復号し、mount 直下の四つの名前だけを調べる制限を明記した。mount 先に空白を含む共有 filesystem の bind 行を模した fixture は終了 10 で拒否された。実装は `/proc/self/mountinfo` ではなく `/proc/mounts` を読み、fixture も実際の四フィールド解析に合わせた。 + +R2 は HOME と既存の対象を `pwd -P` で解決し、clone 前に検査して以後は実体の対象を使う。疑似 Windows root に至る多段リンクは、絶対・相対の両方のリンク先で終了 10 となった。検査後に元の論理 checkout 別名を差し替えても、元の安全な実体 checkout が実行された。リポジトリの試験も Ubuntu の直接・リンク経由の起動と Linux 配置の許可を含む。`-Directory` は単一フォルダ名に制限されるため、新規対象の親は検証済みの実体 HOME となる。 + +ただし実体パスの接頭辞は filesystem の出自を示さない。Windows filesystem を `/mnt` 外に bind mount した場合、`pwd -P` は外側の mount 先を返す。これは静的な制限であり、独立に再現した Linux bind mount の結果や追加の確定した回帰ではない。Darwin ホストのため実 bind mount と WSL 実行はできず、ホスト mount や Docker の状態は変更していない。 + +R3 は 0755 の runtime を、0777 の作業ディレクトリ作成と ZAP 実行より前に拒否し、予測できない名前で排他的に作業場所を作り、報告 symlink を拒否する。絶対・相対・多段・リンク先のない報告 symlink はすべて拒否された。`os.open` の直前で通常ファイルを symlink に差し替えても `O_NOFOLLOW` が拒否した。既存試験は終了 51、後片付け、植えられた報告の検出結果を受け入れないことも確認している。 + +同一所有者が検査と使用の間で意図的に 0700 を 0755 に chmod する試験では、0777 の作業ディレクトリが作成され、スタブのスキャンは成功した。検証は一時点の確認であり、所有者の並行変更に対して条件を維持する仕組みではない。通常の所有者管理の checkout と private runtime では無関係な利用者が chmod できないため、利用者間の攻撃を実証したとは扱わない。checkout の祖先を rename できる主体による親パス差替えや bind 別名も、今回証明した保護の外にある。checkout と祖先を所有者が管理することを前提とし、一般的な競合に耐える filesystem 隔離とは主張しないこと。 + +### 検証と再現 + +Darwin 上のシステム `/usr/bin/python3` 3.9.6 を使った。以下は観測結果で、正本データの写しではない。境界検査は一時 fixture とスタブを使い、実際の資格情報ファイルは調べていない。commit・push・承認・merge・close・grdm-dev スタック変更はしていない。PowerShell は未導入で、ネイティブの構文解析と実 WSL は未実施である。 + +```toon +checks[7]{command,result,exit}: + "PYTHONPATH=installer /usr/bin/python3 -m unittest discover -s installer/tests","640 件・10 skip・合格",0 + "/usr/bin/python3 -m unittest discover -s tools/tests","93 件・合格",0 + "/usr/bin/python3 tools/docs_check.py","レビュー追加前 152 ファイル・8 免除・38 警告・合格",0 + "/usr/bin/python3 tools/manifest_check.py","追跡対象テキスト 694 件検査・63 skip・検出ゼロ",0 + "bash -n installer/install-ubuntu.sh installer/install-macos.sh installer/prereq-lib.sh installer/install.sh","合格",0 + "/usr/bin/python3 /tmp/pr44-rev-r2-RXhzlP/variants.py","末尾改行回避と FIFO timeout を含む境界観測",0 + "/usr/bin/python3 /tmp/pr44-rev-r2-RXhzlP/mount_variants.py","六種類で回避・bind 表 fixture は拒否",0 +``` + +再現スクリプトと全ログは `/tmp/pr44-rev-r2-RXhzlP/`、clone は `/tmp/pr44-rev-r2-RXhzlP/repo` にある。clone ルートから最後の二コマンドを実行すると観測を再現できる。終了 0 は観測用スクリプトが完了した意味であり、製品の全ケースが合格した意味ではない。R2-01/R2-02 の修正後に個別試験を再実行すること。実プラットフォームと bind mount の検証は残っている。 diff --git a/docs/reviews/2026-11-10_RDM-developer-guide_PR44_Per_OS_Installers_r3.md b/docs/reviews/2026-11-10_RDM-developer-guide_PR44_Per_OS_Installers_r3.md new file mode 100644 index 0000000..7f59909 --- /dev/null +++ b/docs/reviews/2026-11-10_RDM-developer-guide_PR44_Per_OS_Installers_r3.md @@ -0,0 +1,87 @@ +# PR44 Per-OS installers review r3 / OS 別インストーラーレビュー r3 + +> **Summary:** REV passes the scoped revision: both r2 findings are closed, with no new regression found in the exercised cases. +> **概要:** REV は今回の差分を合格とする。r2 の二件は解決し、検証した範囲で新たな回帰は見つからなかった。 + +**Jump:** [English](#english) · [日本語](#japanese) + +- Version / 版: 0.1 — 2026-11-10. **Learned:** retaining decoded paths in a variable preserves trailing newlines, and nonblocking opens allow file-type checks to reject FIFOs before reading or writing. ／ 復号したパスを変数に保持すれば末尾改行を保て、nonblocking で開けば読み書きの前に FIFO を種別検査で拒否できる。 +- Applies to: [PR #44](https://github.com/RCOSDP/RDM-developer-guide/pull/44), diff `237387f..f789be9`, head `f789be9abb17c2ea172987ca83d3bb1ba13b74ac`. +- Reviewer / レビュワ: REV（担当レビュワ, Codex Astra）. +- Parent index / 親索引: [Reviews](README.md). +- Related: [r2](2026-11-10_RDM-developer-guide_PR44_Per_OS_Installers_r2.md) · [SPEC-02](../spec/02_installer_cli.md) · [SPEC-03](../spec/03_security_gate.md) · [SPEC-06](../spec/06_documentation_conventions.md). +- Used by / 利用先: [PR #44 revision and coordinator review / 修正と統括レビュー](https://github.com/RCOSDP/RDM-developer-guide/pull/44). + +--- + + +## English + +[→ 日本語版へ](#japanese) + +### Verdict and closure evidence + +**Pass.** A clean detached clone was reviewed; no required changes were found in the scoped revision. + +**R2-01 closed.** Direct assignment through `DECODED_MOUNT` avoids the command substitution that stripped trailing newlines. The original non-dry probe now exits 10 before stub sudo for all six filesystems: virtiofs, 9p, fuse.sshfs, vboxsf, fuse.vmhgfs-fuse and fuse.virtiofsd. The new repository test exercises one trailing newline, two trailing newlines and a literal backslash followed by `012` on each filesystem. The original space, tab, interior-newline, literal-backslash and repeated-trailing-newline probes also pass; the synthetic bind-table fixture still exits 10. + +**R2-02 closed.** `_SAFE_OPEN` combines `O_NOFOLLOW` and `O_NONBLOCK`; `_regular_fd` checks the opened descriptor before reading or truncating/writing. The original reader probe returns a regular-file refusal within its two-second limit. Additional bounded subprocess probes reject writing a FIFO both without a reader and with a reader; no bytes reach the reader. The repository integration test checks exit 51 when a FIFO occupies the masked ZAP report destination. Regular-file creation, shorter UTF-8 overwrite/truncation and symlink-write refusal preserve expected behavior. + +The original absolute, relative, nested, dangling and open-time replacement symlink probes still refuse reports. Nested WSL path aliases still exit 10, and replacing the logical alias after validation still reaches the original safe checkout. Runtime mode 0755 remains refused. The same-owner chmod gap still exists as recorded in r2; SPEC-03 now states the owner/ancestor stability precondition and does not claim race-proof isolation. This is not a new finding. + +### Validation and limits + +System `/usr/bin/python3` 3.9.6 on Darwin was used. Results are observed runs, not a canonical dataset mirror. Full logs and the added write probe are in `/tmp/pr44-rev-r3-r2SOON/`; the clean clone is `/tmp/pr44-rev-r3-r2SOON/repo`. The two original probe scripts remain in `/tmp/pr44-rev-r2-RXhzlP/` and were run from the new clone. Their exit 0 means the observation harness completed; their printed product results were inspected separately. + +```toon +checks[8]{command,result,exit}: + "PYTHONPATH=installer /usr/bin/python3 -m unittest discover -s installer/tests","643 tests; 10 skipped; OK",0 + "/usr/bin/python3 -m unittest discover -s tools/tests","93 tests; OK",0 + "/usr/bin/python3 tools/docs_check.py","153 files; 8 exempt; 38 warnings; passed before review",0 + "/usr/bin/python3 tools/manifest_check.py","695 tracked text files; 63 skipped; zero findings",0 + "bash -n installer/install-ubuntu.sh installer/install-macos.sh installer/prereq-lib.sh installer/install.sh","passed",0 + "/usr/bin/python3 /tmp/pr44-rev-r2-RXhzlP/variants.py","all targeted refusals observed; FIFO returned within 2 seconds",0 + "/usr/bin/python3 /tmp/pr44-rev-r2-RXhzlP/mount_variants.py","six filesystems exit 10 before sudo; bind-table fixture exit 10",0 + "/usr/bin/python3 /tmp/pr44-rev-r3-r2SOON/write_probes.py","FIFO refusal with/without reader; regular writes and symlink refusal passed",0 +``` + +The tracked-file documentation check also passed after writing the record; a direct `check_file` call validates this untracked review separately (zero errors, exit 0). + +No real platform install, native PowerShell/WSL execution or actual Linux bind mount was performed. Tests used temporary fixtures and stub runners; the grdm-dev stack and real runtime environment file were not touched. No commit, push, approval, merge or close was performed; the only GitHub write is one review comment. The review file is intentionally uncommitted and its index entry is left to the coordinator when incorporating the record. No further code changes are required for the two scoped findings; real-platform validation remains outside this review. + +--- + + +## 日本語 + +[→ To English](#english) + +### 判定と解決の証拠 + +**Pass（合格）。** クリーンな detached clone で指定差分を確認し、修正必須の指摘はなかった。 + +**R2-01 解決。** `DECODED_MOUNT` への直接代入により、末尾改行を消すコマンド置換を回避している。元の非 dry-run プローブは、virtiofs・9p・fuse.sshfs・vboxsf・fuse.vmhgfs-fuse・fuse.virtiofsd の全六種類で sudo スタブに到達せず終了 10 となった。追加されたリポジトリ試験は、各 filesystem で末尾改行一つ・二つ・文字どおりのバックスラッシュと `012` を検証する。元の空白・タブ・途中の改行・バックスラッシュ・連続末尾改行のプローブも合格し、bind 行を模した fixture も引き続き終了 10 となる。 + +**R2-02 解決。** `_SAFE_OPEN` は `O_NOFOLLOW` と `O_NONBLOCK` を組み合わせ、`_regular_fd` は読込み・切詰め・書込みより前に開いた記述子を検査する。元の読込みプローブは二秒以内に通常ファイルでない旨を返す。追加の時間制限付き子プロセス試験では、読み手がいない FIFO と読み手がいる FIFO の両方への書込みを拒否し、読み手には一バイトも渡らなかった。リポジトリの統合試験は、マスク済み ZAP 報告の保存先が FIFO の場合の終了 51 を確認する。通常ファイルの新規作成、短い UTF-8 内容への上書きと切詰め、symlink への書込み拒否も正常だった。 + +元の絶対・相対・多段・リンク先なし・open 直前差替えの報告 symlink は引き続き拒否される。WSL の多段別名は終了 10、検証後の論理別名差替えも元の安全な checkout に到達する。runtime の 0755 は拒否される。同一所有者による chmod の競合は r2 の記録どおり残るが、SPEC-03 は所有者と祖先の安定性を前提として明記し、競合に耐える隔離とは主張していない。新たな指摘ではない。 + +### 検証と制限 + +Darwin のシステム `/usr/bin/python3` 3.9.6 を使った。結果は実行時の観測であり、正本データの写しではない。全ログと追加書込みプローブは `/tmp/pr44-rev-r3-r2SOON/`、clone は `/tmp/pr44-rev-r3-r2SOON/repo` にある。元の二つのスクリプトは `/tmp/pr44-rev-r2-RXhzlP/` にあり、新しい clone を作業ディレクトリとして実行した。終了 0 は観測用スクリプトの完了を表し、出力された製品側の結果も別途確認した。 + +```toon +checks[8]{command,result,exit}: + "PYTHONPATH=installer /usr/bin/python3 -m unittest discover -s installer/tests","643 tests; 10 skipped; OK",0 + "/usr/bin/python3 -m unittest discover -s tools/tests","93 tests; OK",0 + "/usr/bin/python3 tools/docs_check.py","153 files; 8 exempt; 38 warnings; passed before review",0 + "/usr/bin/python3 tools/manifest_check.py","695 tracked text files; 63 skipped; zero findings",0 + "bash -n installer/install-ubuntu.sh installer/install-macos.sh installer/prereq-lib.sh installer/install.sh","passed",0 + "/usr/bin/python3 /tmp/pr44-rev-r2-RXhzlP/variants.py","all targeted refusals observed; FIFO returned within 2 seconds",0 + "/usr/bin/python3 /tmp/pr44-rev-r2-RXhzlP/mount_variants.py","six filesystems exit 10 before sudo; bind-table fixture exit 10",0 + "/usr/bin/python3 /tmp/pr44-rev-r3-r2SOON/write_probes.py","FIFO refusal with/without reader; regular writes and symlink refusal passed",0 +``` + +文書作成後の追跡ファイル検査も合格し、未追跡のレビュー文書自体は `check_file` を直接呼んで別途検証した（エラーゼロ・終了 0）。 + +実プラットフォームのインストール、ネイティブ PowerShell・WSL、Linux の実 bind mount は未実施。一時 fixture とスタブを使い、grdm-dev スタックと実環境の runtime 環境ファイルには触れていない。commit・push・承認・merge・close は行わず、GitHub への書込みはレビューコメント一件だけである。レビュー文書は意図どおり未コミットで、索引への追加は記録を取り込む統括側に委ねる。対象二件について追加のコード修正は不要であり、実プラットフォームの検証は今回のレビュー範囲外として残る。 diff --git a/docs/spec/02_installer_cli.md b/docs/spec/02_installer_cli.md index 6e7d7c0..97d355b 100644 --- a/docs/spec/02_installer_cli.md +++ b/docs/spec/02_installer_cli.md @@ -1,11 +1,16 @@ # SPEC-02 Installer CLI / インストーラー CLI -> **Summary:** The contract of the two entry points that install the GakuNin RDM developer environment: the thin bootstrap `installer/install.sh` and the Python command `grdm-dev` with its subcommands, profiles, flags, exit codes, and directory layout. -> **概要:** GakuNin RDM 開発環境を導入する二つの入口、薄い起動スクリプト `installer/install.sh` と Python コマンド `grdm-dev` の契約。サブコマンド、プロファイル、フラグ、終了コード、ディレクトリ構成を定める。 +> **Summary:** The contract of the optional per-OS prerequisite scripts and of the two entry points that install the GakuNin RDM developer environment: the thin bootstrap `installer/install.sh` and the Python command `grdm-dev` with its subcommands, profiles, flags, exit codes, and directory layout. +> **概要:** 任意の OS ごとの前提条件スクリプトと、GakuNin RDM 開発環境を導入する二つの入口、薄い起動スクリプト `installer/install.sh` と Python コマンド `grdm-dev` の契約。サブコマンド、プロファイル、フラグ、終了コード、ディレクトリ構成を定める。 **Jump:** [English](#english) · [日本語](#japanese) -- Version / 版: 0.22 — 2026-11-10. **Learned:** a lock that only the `db` commands take does not stop `up` from starting the server between their checks and their renames, so `up` and `reset` take it too (PR #42 review r2). ／ `db` のコマンドだけが取るロックでは、その確認と改名の間に `up` がサーバーを起動するのを止められないので、`up` と `reset` も同じロックを取る（PR #42 レビュー r2）。 +- Version / 版: 0.27 — 2026-11-10. **Learned:** decoding is not enough if the result passes through `$(...)`, which drops trailing newlines: a mount named `host\012` was checked as `host`, so the decoded name is kept in a variable ([PR #44 review r2](../reviews/2026-11-10_RDM-developer-guide_PR44_Per_OS_Installers_r2.md) R2-01). ／ 復号しても結果が `$(...)` を通ると末尾の改行が落ちる。`host\012` という mount が `host` として調べられたので、復号した名前は変数に保つ（[PR #44 レビュー r2](../reviews/2026-11-10_RDM-developer-guide_PR44_Per_OS_Installers_r2.md) R2-01）。 + 0.26 — 2026-11-10. **Learned:** a path check must read the path the way the system means it: the kernel escapes spaces in mount points (`\040`) and a symlink can carry a checkout onto the Windows disk, so the shared-folder check decodes mount fields and the WSL checks compare physical paths (`pwd -P`) ([PR #44 review r1](../reviews/2026-11-10_RDM-developer-guide_PR44_Per_OS_Installers_r1.md) R1, R2). ／ パスの検査は、システムが意味するとおりにパスを読まなければならない。カーネルは mount 先の空白を `\040` とエスケープし、シンボリックリンクはチェックアウトを Windows のディスクへ運びうる。そこで共有フォルダの検査は mount の欄を復号し、WSL の検査は実体のパス（`pwd -P`）で比べる（[PR #44 レビュー r1](../reviews/2026-11-10_RDM-developer-guide_PR44_Per_OS_Installers_r1.md) R1・R2）。 + 0.25 — 2026-11-10. **Learned:** a change can be blocked by the permission needed to publish one part of it: the token that pushes this repository may not change `.github/workflows/`, so the two workflows of §10 wait on branch `feat/per-os-installers` while the scripts, fixes and records land now. ／ 変更の一部を公開する権限が無いために全体が止まることがある。このリポジトリに push するトークンは `.github/workflows/` を変えられないので、§10 の二つのワークフローはブランチ `feat/per-os-installers` で待ち、スクリプト・修正・記録は先に入れる。 + 0.24 — 2026-11-10. **Learned:** a real Linux run disproves what a dry run cannot: Ubuntu 24.04's own `qemu-user-static` crashes inside an amd64 build after 48 minutes, QEMU builds ember about twice as slowly as Rosetta so the 1800 s wait stops a healthy install, and the cloud image lacks a library Playwright's driver needs; so emulation comes from a digest-pinned image, `doctor` refuses the measured-bad emulator, `up` waits longer on Linux arm64, and the accept hint names the Linux packages ([DEMO-21](../demo/21_core_install_ubuntu_arm64_lima.md)). ／ 実際の Linux での実行は dry run には分からないことを覆す。Ubuntu 24.04 自身の `qemu-user-static` は 48 分後に amd64 のビルドの中で落ち、QEMU は ember の組み立てに Rosetta のおよそ 2 倍かかるので 1800 秒の待ちは健全な導入を止め、クラウドイメージには Playwright のドライバーが要るライブラリが無い。そこでエミュレーションはダイジェストで固定したイメージから入れ、`doctor` は落ちると実測したエミュレーターを拒み、`up` は Linux arm64 では長く待ち、accept の案内は Linux のパッケージを示す（[DEMO-21](../demo/21_core_install_ubuntu_arm64_lima.md)）。 + 0.23 — 2026-11-10. **Learned:** a prerequisite is whatever the whole install sequence needs, not only its first step: Trivy and gitleaks are missing on a fresh host and stop an otherwise good install with exit 51 at `scan`, so the per-OS scripts of §2.3 provide them ([ADR-08](../decisions/ADR-08_Per_OS_Installer_Entry_Points.md)). ／ 前提条件とは最初の段ではなく一連の導入全体に要るものである。新しいホストには Trivy と gitleaks が無く、ほかが良くても `scan` で終了 51 になるため、§2.3 の OS ごとのスクリプトがそれらを用意する（[ADR-08](../decisions/ADR-08_Per_OS_Installer_Entry_Points.md)）。 + 0.22 — 2026-11-10. **Learned:** a lock that only the `db` commands take does not stop `up` from starting the server between their checks and their renames, so `up` and `reset` take it too (PR #42 review r2). ／ `db` のコマンドだけが取るロックでは、その確認と改名の間に `up` がサーバーを起動するのを止められないので、`up` と `reset` も同じロックを取る（PR #42 レビュー r2）。 0.21 — 2026-11-10. **Learned:** a command that renames data directories needs a journal, a lock and a recovery path that every later command runs, not only checks before it starts (PR #42 review r1). ／ データディレクトリを改名するコマンドには、始める前の確認だけでなく、操作記録・ロック・後続のすべてのコマンドが通る復旧の道が要る（PR #42 レビュー r1）。 0.20 — 2026-11-10. **Learned:** a new database major needs a command, not a note: a server refuses a data directory of another major, so `up` now says so with exit 15 and `db upgrade` moves the data by a verified dump and restore that never writes the old directory (ADR-12). ／ データベースの新しい版には注記でなくコマンドが要る。サーバーは別の版のデータディレクトリを拒むので、`up` はそれを終了 15 で告げ、`db upgrade` は旧ディレクトリに一度も書かない、照合つきのダンプと復元でデータを移す（ADR-12）。 0.19 — 2026-11-10. **Learned:** a fixed client is only as good as the address and trust anchor it is given: osf.io's mongo URL was never rendered and fell back to PostgreSQL's host and port, so `osf.env` now names the TLS URL and the CA file. ／ 直したクライアントも、渡される接続先と信頼の起点しだいである。osf.io の mongo の URL は一度も描画されず、PostgreSQL のホストとポートに落ちていたので、`osf.env` に TLS の URL と CA ファイルを書くことにした。 @@ -19,7 +24,7 @@ - Version / 版: 0.11 — 2026-11-10. **Learned:** a root process without capabilities still obeys file permissions, a container path does not confine the host path behind its bind mount, and a repair that trusts its top directory cannot finish a partial one; so volume paths are validated on the host, `unoconv` holds `DAC_OVERRIDE` for the shared directory, ownership is re-walked every time, and availability questions are asked of the filesystem or the engine rather than guessed from a path (PR #36 review r1). ／ capability を除いた root もファイル権限に従い、コンテナ内のパスは bind 元のホストパスを限定せず、一番上のディレクトリを信用する修復は途中で止まった修復を完了できない。そこでボリュームのパスはホスト側で検証し、`unoconv` は共有ディレクトリのために `DAC_OVERRIDE` を持ち、所有者は毎回全体を辿り、可否はパスから推測せずファイルシステムやエンジンに問うと分かった（PR #36 レビュー r1）。 - Version / 版: 0.10 — 2026-11-10. **Learned:** a check that fails after a 30-minute build teaches nothing, so `doctor` must prove every prerequisite up front: private repositories, scanners, memory floors, emulation and a checkout path Docker can see; and a fresh bind directory belongs to the host user, so ownership is one generic step derived from the writers instead of one init container per volume, while `reset` needs a container to delete what containers own (cross-OS audit, 2026-11-10; on Colima a bind mount from `/private/tmp` was measured to be silently empty). ／ 30 分のビルドの後で失敗する検査は何も教えない。だから `doctor` が非公開リポジトリ、スキャナー、メモリの下限、エミュレーション、Docker から見える配置先をすべて先に証明しなければならない。また新しい bind ディレクトリはホストの利用者の所有なので、所有者の設定はボリュームごとの init コンテナではなく書き手から導く一つの汎用の段にまとめ、`reset` はコンテナが所有するものをコンテナで消さねばならないと分かった（OS 横断の監査、2026-11-10。Colima では `/private/tmp` からの bind mount が黙って空になることを実測）。 0.9 — 2026-11-10. **Learned:** a retired credential key stays in `runtime/.env` forever unless it is dropped explicitly, and a Compose file rendered before the key was retired still refers to it, so `compose render` must run before `secrets rotate` or a recreate after an upgrade (PR #35 review r1). ／ 廃止した資格情報のキーは明示的に消さなければ `runtime/.env` に残り続け、廃止前に生成した Compose ファイルはそのキーを参照し続ける。更新後は `secrets rotate` や再作成の前に `compose render` が必要と分かった（PR #35 レビュー r1）。 -- Version / 版: 0.8 — 2026-11-06. **Learned:** a stack that builds is not a stack that starts: a fresh database needs a one-shot initialisation (`osf_init`), Compose's short `tmpfs` is root-only, the memory budget must respect the Docker VM, and a prebuilt SPA must be built once into a volume rather than at every start under emulation (real install runs 9–13). ／ 組めるスタックは起動するスタックではない。新しいデータベースには一回限りの初期化（`osf_init`）が要り、Compose の短い `tmpfs` は root 専用で、メモリ予算は Docker VM を尊重しなければならず、prebuilt の SPA はエミュレーション下で毎回組むのではなくボリュームへ一度組む（実機第 9〜13 回）。 + 0.8 — 2026-11-06. **Learned:** a stack that builds is not a stack that starts: a fresh database needs a one-shot initialisation (`osf_init`), Compose's short `tmpfs` is root-only, the memory budget must respect the Docker VM, and a prebuilt SPA must be built once into a volume rather than at every start under emulation (real install runs 9–13). ／ 組めるスタックは起動するスタックではない。新しいデータベースには一回限りの初期化（`osf_init`）が要り、Compose の短い `tmpfs` は root 専用で、メモリ予算は Docker VM を尊重しなければならず、prebuilt の SPA はエミュレーション下で毎回組むのではなくボリュームへ一度組む（実機第 9〜13 回）。 0.7 — 2026-11-06. **Learned:** a service row in the profile table must correspond to a process that exists, so the overlay repository wiki-frontend is represented by a one-shot `wiki_assets` that fills a named volume and its readers wait with `condition: service_completed_successfully` ([ADR-06](../decisions/ADR-06_Wiki_Frontend_Overlay.md)), while a template `command` must have `$` escaped as `$$` for Compose (PR #30 review r1); and, independently, a glossary term is only explained for the reader when its note and REF-01 link sit at the first prose use, counting lower-case and plural forms, so first-use notes with links were added in both halves (SPEC-06 §3). ／ プロファイル表のサービス行は実在するプロセスに対応していなければならないため、オーバーレイである wiki-frontend は名前付きボリュームを満たす一回限りの `wiki_assets` で表し、それを読むサービスは `condition: service_completed_successfully` で待つ（[ADR-06](../decisions/ADR-06_Wiki_Frontend_Overlay.md)）。またテンプレートの `command` は Compose 向けに `$` を `$$` にエスケープする（PR #30 レビュー r1）。また別に、用語注と REF-01 へのリンクは、小文字や複数形も含めた本文の初出に置かなければ読者に届かないと分かったため、英和両方の初出にリンク付きの用語注を補った（SPEC-06 §3）。 - Version / 版: 0.6 — 2026-11-06. **Learned:** a `command` in a template is written for the container shell, but Compose interpolates `$VAR` in every string of the file first, so the renderer must escape `$` as `$$` and a test must run the rendered file through Compose's own parser; and a fresh bind volume belongs to the host user, so a non-root writer needs an owner step (PR #30 review r1). ／ テンプレートの `command` はコンテナのシェル向けに書くが、Compose はファイル中の全文字列の `$VAR` を先に展開する。だから出力側で `$` を `$$` にエスケープし、試験は出力ファイルを Compose 自身の解析器に通さねばならない。また新しい bind ボリュームはホストの利用者の所有なので、非 root の書き手には所有者を決める段が要る（PR #30 レビュー r1）。 0.5 — 2026-11-06. **Learned:** a service row in the profile table must correspond to a process that exists: `wiki` had none, because the wiki-frontend repository is an overlay whose image only builds a webpack bundle (real install run 4, [ADR-06](../decisions/ADR-06_Wiki_Frontend_Overlay.md)); the honest row is a one-shot `wiki_assets` that fills a named volume, and the services that read the volume must wait for it with `condition: service_completed_successfully`. ／ プロファイル表のサービス行は実在するプロセスに対応していなければならないと分かった。`wiki` には無かった。wiki-frontend リポジトリは webpack の束を組むだけのイメージを持つオーバーレイだからである（実機第 4 回、[ADR-06](../decisions/ADR-06_Wiki_Frontend_Overlay.md)）。正直な行は名前付きボリュームを満たす一回限りの `wiki_assets` で、ボリュームを読むサービスは `condition: service_completed_successfully` でそれを待たねばならない。 @@ -28,8 +33,8 @@ 0.3 — 2026-11-05. **Learned:** `secrets rotate` had to become a coordinated database-password change plus container recreation, because `restart` keeps the old environment and leaves the captured credential valid (PR #21 review r1). ／ `restart` は旧い環境を保つため、捕られた資格情報が有効なまま残る。`secrets rotate` は DB パスワードの変更とコンテナ再作成を束ねた手順にする必要があると分かった（PR #21 レビュー r1）。 0.2 — 2026-11-05. **Learned:** a profile table and the acceptance criteria that depend on it must change together, and a TOON row whose text contains a comma or a colon must be quoted or the parser sees extra columns. ／ プロファイル表と、それに依存する受入基準は同時に変えなければならず、カンマやコロンを含む TOON の行は引用符で囲まないと列が増えて解析に失敗すると分かった。 - Version / 版: 0.1 — 2026-11-05. **Learned:** keeping the shell part to a bootstrap and putting every decision in one Python command is the only way to make the OS differences, the manifest check, and the reports testable with `unittest`. ／ シェルは起動だけに留め、判断をすべて一つの Python コマンドに集めないと、OS の違い・manifest の整合・報告書を `unittest` で試験できないと分かった。 -- Normative for: `installer/install.sh`, `installer/grdm_dev/`, `installer/tests/`, `runtime/`. -- Related: [SPEC-00 Overview](00_overview.md) · [SPEC-01 Manifest](01_manifest.md) · [SPEC-03 Security Gate](03_security_gate.md) · [SPEC-04 Acceptance Tests](04_acceptance_tests.md) · [SPEC-05 Upstream Tracking](05_upstream_tracking.md) · [SPEC-06 Documentation Conventions](06_documentation_conventions.md) · [ADR-02 Network Model](../decisions/ADR-02_Network_Model.md) · [ADR-03 Staged Security Gate](../decisions/ADR-03_Staged_Security_Gate.md) · [POL-01 Position](../policy/POL-01_Developer_Environment_2027_Position.md) · [docs/README.md](../README.md) +- Normative for: `installer/install.sh`, `installer/install-ubuntu.sh`, `installer/install-macos.sh`, `installer/install-windows.ps1`, `installer/prereq-lib.sh`, `.github/workflows/installer-prereq-lint.yml` and `.github/workflows/installer-full.yml` (pending on branch `feat/per-os-installers`), `installer/grdm_dev/`, `installer/tests/`, `runtime/`. +- Related: [SPEC-00 Overview](00_overview.md) · [SPEC-01 Manifest](01_manifest.md) · [SPEC-03 Security Gate](03_security_gate.md) · [SPEC-04 Acceptance Tests](04_acceptance_tests.md) · [SPEC-05 Upstream Tracking](05_upstream_tracking.md) · [SPEC-06 Documentation Conventions](06_documentation_conventions.md) · [ADR-02 Network Model](../decisions/ADR-02_Network_Model.md) · [ADR-03 Staged Security Gate](../decisions/ADR-03_Staged_Security_Gate.md) · [ADR-08 Per-OS Installer Entry Points](../decisions/ADR-08_Per_OS_Installer_Entry_Points.md) · [POL-01 Position](../policy/POL-01_Developer_Environment_2027_Position.md) · [docs/README.md](../README.md) --- @@ -40,11 +45,13 @@ ### 1. Scope and goal -The installer turns a fresh checkout of this repository into a running GakuNin RDM (GakuNin is Japan's academic network for logging in with a university account; see [REF-01](../reference/REF-01_Glossary.md)) developer environment with one command, on the hosts listed in [§8](#2-en-hosts). It never needs `sudo` (a way to run a command as the computer's administrator), never changes network settings on the host, and can be run again at any time without harm. Everything it creates lives under `runtime/`, which is ignored by Git. +The installer turns a fresh checkout of this repository into a running GakuNin RDM (GakuNin is Japan's academic network for logging in with a university account; see [REF-01](../reference/REF-01_Glossary.md)) developer environment with one command, on the hosts listed in [§8](#2-en-hosts). `install.sh` and `grdm-dev` never need `sudo` (a way to run a command as the computer's administrator), never change network settings on the host, and can be run again at any time without harm. Everything they create lives under `runtime/`, which is ignored by Git. The optional prerequisite scripts of [§2.3](#2-en-prereq) prepare the host beforehand and may use `sudo`, but only after showing the change and receiving consent ([ADR-08](../decisions/ADR-08_Per_OS_Installer_Entry_Points.md)). **Glossary notes.** An *installer* is a program that prepares other programs so they can run. A *CLI* (command line interface) is a program you drive by typing commands. A *container* is a box that holds one program and everything it needs, so it runs the same on any computer. *Docker* (the tool that builds images and runs containers; see [REF-01](../reference/REF-01_Glossary.md)) and its companion *Docker Compose* (see [REF-01](../reference/REF-01_Glossary.md)) run them; Compose is the tool that starts many containers together from one description file. A *manifest* is the list that says exactly which version of each source repository we use ([SPEC-01](01_manifest.md)). -### 2. Two entry points +### 2. Entry points + +There are two entry points that install the environment (§2.1, §2.2), and optional scripts that prepare a host for them ([§2.3](#2-en-prereq)). #### 2.1 `installer/install.sh` — thin bootstrap @@ -91,11 +98,33 @@ flowchart TD - **External tools** are called as subprocesses and checked by `doctor`: `docker` (27 or newer) with the Compose plugin (v2 or newer) and buildx (Docker's helper that builds one image for several kinds of computer; see [REF-01](../reference/REF-01_Glossary.md)), `trivy`, `gitleaks`, and `semgrep` for `scan`, the OWASP ZAP (a tool that pokes at a running website the way an attacker would; see [REF-01](../reference/REF-01_Glossary.md)) container image for `scan --dynamic`, and `mkdocs` for `docs`. A missing scanner makes `scan` exit 51 ([SPEC-03 §8](03_security_gate.md#english)). - Every subcommand supports `--json`, which prints the same report as machine-readable JSON instead of TOON (a compact text form of JSON, see [SPEC-06 §4](06_documentation_conventions.md#english)). + +#### 2.3 Per-OS prerequisite scripts + +These scripts prepare a host and then run `install.sh`. The reason for them is in [ADR-08](../decisions/ADR-08_Per_OS_Installer_Entry_Points.md). They hold no installation logic, so `install.sh` and `grdm-dev` still make every decision. + +| Script | Host | What it can provide | +|---|---|---| +| `installer/install-ubuntu.sh` | Ubuntu 24.04 amd64 or arm64; Ubuntu 24.04 inside WSL2 | apt packages (git, Python, `python3-venv`, curl, gnupg), Docker Engine with the Compose and buildx plugins from Docker's signed apt repository, `docker` group membership, amd64 emulation on arm64 (QEMU registered from the digest-pinned `tonistiigi/binfmt` image, replacing Ubuntu's `qemu-user-static` 8.2.2, which crashed in a real build: [DEMO-21](../demo/21_core_install_ubuntu_arm64_lima.md)), Trivy and gitleaks in `~/.local/bin`, and `libatomic1` when `GRDM_WITH_ACCEPT=1`. Inside WSL2 it skips Docker Engine and requires Docker Desktop. | +| `installer/install-macos.sh` | macOS on Apple Silicon | Homebrew formulae (`python@3.12`, git, Trivy, gitleaks); starts an installed Docker Desktop. It detects Docker Desktop, OrbStack and Colima, and installs neither a Docker runtime nor Homebrew. | +| `installer/install-windows.ps1` | Windows 11 amd64 | WSL2 and the `Ubuntu-24.04` distribution; checks Docker Desktop and its WSL integration; clones this repository under the WSL home directory, never `/mnt/c`; then runs `install-ubuntu.sh` inside WSL. | + +Rules that every script keeps: + +- It shows what it will change, and asks before any `sudo` or administrator action. Joining the `docker` group is asked separately. +- `--yes` (`-Yes` in PowerShell, which also accepts `--yes`) gives consent. Without a terminal and without `--yes` it stops with exit 2, as in [§7](#english). +- `--dry-run` (`-DryRun`) detects and prints the plan and changes nothing. +- It can be run again: a prerequisite that is already present is skipped. +- It stops with a clear message on failure and uses only the exit codes 2, 10 and 11 of [§6](#english). +- A download is accepted only through a signed source or a pinned checksum, and nothing is piped into a shell. +- It never asks for, reads or stores a token. It only says how to give git access to the private integration repositories. +- Its last action is `installer/install.sh`, with every other argument passed unchanged. + ### 3. Subcommands | Subcommand | Contract | Touches Docker | |---|---|---| -| `doctor` | Detects OS (macOS, Ubuntu, WSL2), CPU (`arm64`, `amd64`), memory, free disk, Docker ≥ 27, Compose plugin (v2 or newer), buildx, and proves what `install` needs so a problem appears before the long build: `git`; read access to the profile's private integration repositories, with `git ls-remote` that can never prompt (its output is never printed, a URL can hold a token); `trivy`, `gitleaks` and `semgrep`; the profile's memory floors against the smaller of the host and the Docker VM (`--memory-gib` turns this into a warning, a knowing overcommit); on Linux arm64 only, amd64 emulation (binfmt), printing the command that enables it; a checkout Docker can use: on WSL2 a filesystem that keeps a `chmod 0600` read back from a test file (a Windows drive cannot hold `runtime/.env` at `0600`); on macOS one the engine can see, asked by bind-mounting it read-only into the `postgres` image that `up` needs (the only image this subcommand may pull; a mount the engine refuses or that comes up empty is exit 13, a probe that could not run is a warning) because a bind mount from a path the Docker VM does not share is silently empty, with `GRDM_DOCKER_SHARED=1` to skip the question when it cannot be run. Prints a TOON report. Exit 10 if the host is unsupported or the memory floors do not fit, 11 if Docker is missing or not running, 13 if a prerequisite is missing, 51 if a scanner is missing. | read-only | +| `doctor` | Detects OS (macOS, Ubuntu, WSL2), CPU (`arm64`, `amd64`), memory, free disk, Docker ≥ 27, Compose plugin (v2 or newer), buildx, and proves what `install` needs so a problem appears before the long build: `git`; read access to the profile's private integration repositories, with `git ls-remote` that can never prompt (its output is never printed, a URL can hold a token); `trivy`, `gitleaks` and `semgrep`; the profile's memory floors against the smaller of the host and the Docker VM (`--memory-gib` turns this into a warning, a knowing overcommit); on Linux arm64 only, amd64 emulation (binfmt), printing the command that enables it, and refusing an emulator build measured to crash (Ubuntu 24.04's `qemu-user-static` 8.2.2, read from the registered interpreter's `--version`), printing the command that replaces it; a checkout Docker can use: on WSL2 a filesystem that keeps a `chmod 0600` read back from a test file (a Windows drive cannot hold `runtime/.env` at `0600`); on macOS one the engine can see, asked by bind-mounting it read-only into the `postgres` image that `up` needs (the only image this subcommand may pull; a mount the engine refuses or that comes up empty is exit 13, a probe that could not run is a warning) because a bind mount from a path the Docker VM does not share is silently empty, with `GRDM_DOCKER_SHARED=1` to skip the question when it cannot be run. Prints a TOON report. Exit 10 if the host is unsupported or the memory floors do not fit, 11 if Docker is missing or not running, 13 if a prerequisite is missing, 51 if a scanner is missing. | read-only | | `plan` | Resolves manifest + profile and prints which images would be built and which services started, with the published ports ([§4](#2-en-ports)). Changes nothing. Exit 20 if the manifest fails [SPEC-01](01_manifest.md) checks. | no | | `secrets init` / `secrets rotate` | `init` writes random secrets (database passwords, Django secret key, CAS keys) into `runtime/.env` with file mode `0600`. Never writes a default or example password. Refuses to overwrite an existing file unless `--force`. `rotate` regenerates every secret, changes the live PostgreSQL and RabbitMQ passwords, recreates the containers so they receive the new environment, and verifies that the old credentials are rejected; any failure restores the previous `runtime/.env` and exits 40. `init --force` and `rotate` drop retired credential keys. After an upgrade, run `compose render` before `rotate` or any recreate, because a previously rendered Compose file still refers to retired keys. Four keys have lengths the CAS image needs: `TGC_ENCRYPTION_KEY` is 43 characters, `TGC_SIGNING_KEY` is 86 characters, `OSF_API_INSTITUTIONS_AUTH_JWESECRET` is exactly 32 bytes and `OSF_API_INSTITUTIONS_AUTH_JWTSECRET` is at least 32 bytes; the last two are also osf.io's `JWE_SECRET` and `JWT_SECRET` ([ADR-07 §7](../decisions/ADR-07_Native_Builds_of_EOL_Middleware.md#english)). All four use only the base64url characters `A`–`Z`, `a`–`z`, `0`–`9`, `-` and `_` (no whitespace, quotes or padding), must not be a published default, and `TGC_ENCRYPTION_KEY` must decode canonically to 32 bytes. Used after a red-team capture ([SPEC-03 §6](03_security_gate.md#english)). | no / yes | | `compose render` | Exits 2, naming the keys only, when `runtime/.env` lacks a generated key or holds a CAS key that CAS cannot use: wrong length, characters outside base64url, or a published default (fix either with `secrets rotate`). Writes `runtime/docker-compose.yml` from manifest + profile: the internal network `grdm` plus the publishing-only `grdm-edge` (`--egress ` attaches more, recorded in `runtime/.env`; [SPEC-03 §3.2](03_security_gate.md#english)), per-component `platform:` from the manifest's `build_platform`, ports published only on `127.0.0.1` ([§4](#2-en-ports)), non-root `user:` for every service, `read_only`, `cap_drop`, and resource limits (`mem_limit`, `cpus`, `pids_limit`) scaled from the smaller of the host memory `doctor` detected and the Docker engine's VM memory (`docker info`; on Docker Desktop the VM is a fraction of the host, real install run 10), where a template's `mem_min_mb` floor takes its share first and the others divide the rest, and from the CPU count the engine reports (`docker info`; the host's only when that is unknown), with per-service exceptions from `manifest/hardening.json` ([SPEC-03 §3.3](03_security_gate.md#english)), `platform:` chosen from `doctor`, named volumes mapped under `runtime/volumes/`, every `tmpfs` entry mounted with `mode=1777` (Compose's short form gives root-only `0755`, real install run 11), and the placeholders `{{HOST_PORT:}}`, `{{CONTAINER_PORT:}}` and `{{AUTH_URL}}` (fakecas's published URL, or web's when `--auth shibboleth` replaces fakecas) resolved in template values. The `image:` of a built component carries the tag of the last build (`runtime/reports/build_latest.json`) only when it equals the tag the manifest pins; a recorded build that differs from the pin is stale (the pin moved after the last `build`), so the pinned tag is rendered instead, the report lists the component under `stale_build_tags` (`component`, `pinned`, `recorded`, `image`), and stderr prints a warning block whose fix is `grdm-dev build`, then `grdm-dev up`. Render still exits 0 because `install` renders before it builds; an image that was never built makes `up` fail instead of running the old one. A component the manifest does not pin keeps the recorded tag. `osf.env` names osf.io's TLS connection to mongo, `SHAREJS_DB_URL=mongodb://mongo:27017/sharejs?ssl=true` with `SHAREJS_DB_CA_FILE=/mongo-tls/mongodb.crt`, and web, api, admin and worker mount `mongo_tls` read-only, as sharejs does ([ADR-07](../decisions/ADR-07_Native_Builds_of_EOL_Middleware.md)). Follows [ADR-02](../decisions/ADR-02_Network_Model.md). | no | @@ -187,7 +216,7 @@ Numbers are never reused for a new meaning; a new cause gets a new number. The c - **Idempotent.** Running any subcommand twice gives the same result. `secrets init` keeps existing secrets, `compose render` overwrites only the generated file, `build` uses the buildx cache, `up` is a no-op for healthy services. - **No secrets in Git.** `runtime/` is in `.gitignore`. `docs check` and the CI secret scan fail if a file under `runtime/` or a value that looks like a secret is committed. - **No host network changes.** The loopback alias `192.168.168.167` from the old guide is not created; `compose render` rewrites the upstream `local.py` and `.docker-compose.*.env` settings to use service names inside the Compose network and `127.0.0.1` for the browser ([ADR-02](../decisions/ADR-02_Network_Model.md)). -- **First start is slow, later starts are not.** `up` waits up to 1800 s (`--timeout`) for health: on Apple Silicon (the chips Apple puts in its Macs since 2020; see [REF-01](../reference/REF-01_Glossary.md)) the first `up` builds ember-osf-web (the newer web pages of GRDM; see [REF-01](../reference/REF-01_Glossary.md)) under emulation (10–25 minutes) and runs `osf_init`; both leave their result in named volumes, so a second `up` is healthy within a minute or two (real install runs 10–18; the cold-start build took 13–25 minutes, which is why the default is 30 minutes). +- **First start is slow, later starts are not.** `up` waits up to 1800 s (`--timeout`) for health: on Apple Silicon (the chips Apple puts in its Macs since 2020; see [REF-01](../reference/REF-01_Glossary.md)) the first `up` builds ember-osf-web (the newer web pages of GRDM; see [REF-01](../reference/REF-01_Glossary.md)) under emulation (10–25 minutes) and runs `osf_init`; both leave their result in named volumes, so a second `up` is healthy within a minute or two (real install runs 10–18; the cold-start build took 13–25 minutes, which is why the default is 30 minutes). On Linux arm64 the emulation is QEMU, and ember's first build took 44 minutes on one core, so there the default is 5400 s when the profile has amd64 services ([DEMO-21](../demo/21_core_install_ubuntu_arm64_lima.md)); `--timeout` still overrides it. - **No `-private` code here.** The installer clones the integration repositories by name and SHA (the fingerprint of one commit; see [REF-01](../reference/REF-01_Glossary.md)) from the manifest at run time; this public repository holds no code from them ([POL-01](../policy/POL-01_Developer_Environment_2027_Position.md)). @@ -199,7 +228,7 @@ Numbers are never reused for a new meaning; a new cause gets a new number. The c |---|---|---| | macOS 14 or newer with Docker Desktop | arm64 | official (M2 acceptance runs here) | | Ubuntu 24.04 with Docker Engine | amd64, arm64 | official | -| Windows 11 with WSL2 (Ubuntu 24.04) and Docker Desktop | amd64 | required to work; checked in CI as build-only | +| Windows 11 with WSL2 (Ubuntu 24.04) and Docker Desktop | amd64 | required to work; CI parses and dry-runs `install-windows.ps1` only, and a real run needs owner hardware ([ADR-08 §5](../decisions/ADR-08_Per_OS_Installer_Entry_Points.md#en-limits)) | | Other Linux, macOS on Intel | any | best-effort; `doctor` warns and continues | ### 9. Directory layout @@ -207,6 +236,10 @@ Numbers are never reused for a new meaning; a new cause gets a new number. The c ```toon layout: installer/install.sh: bash 3.2 bootstrap + installer/install-ubuntu.sh: Ubuntu 24.04 and WSL2 prerequisites (§2.3) + installer/install-macos.sh: macOS prerequisites (§2.3) + installer/install-windows.ps1: Windows 11 prerequisites and WSL2 hand-over (§2.3) + installer/prereq-lib.sh: helpers shared by the two bash prerequisite scripts installer/pyproject.toml: package metadata, console script grdm-dev, extra accept installer/grdm_dev/: cli.py: argument parsing, subcommand dispatch, exit codes @@ -241,6 +274,10 @@ layout: `python3 -m unittest discover -s installer/tests` must pass on every PR (pull request: a proposed change waiting for review; see [REF-01](../reference/REF-01_Glossary.md)). Each module has a `test_.py` that covers at least one success case and one rejection case, and every rejection case asserts the exit code from [§6](#english). Docker is never required by unit tests: `host.py` and `build.py` take an injectable runner so tests pass fake `docker` output. Integration and acceptance tiers are defined in [SPEC-04](04_acceptance_tests.md). +The prerequisite scripts of [§2.3](#2-en-prereq) are checked on every PR by `installer-prereq-lint.yml`: `bash -n`, shellcheck, the PowerShell parser on `windows-latest`, and a dry run on each OS. Unit tests run them only with `--dry-run` or `--help`, under a `PATH` of stubs. `installer-full.yml`, started by hand, does a full `minimum` install on `ubuntu-24.04` and `ubuntu-24.04-arm` and needs the repository secret `GRDM_INTEGRATION_READ_TOKEN`. What these runners cannot prove is listed in [ADR-08 §5](../decisions/ADR-08_Per_OS_Installer_Entry_Points.md#en-limits). + +> **Note:** Both workflows, and their helper `.github/scripts/Test-PowerShellSyntax.ps1`, are **pending**: they wait on branch `feat/per-os-installers` until someone with a token that has the `workflow` scope pushes them. Until then the checks of this paragraph are run by hand, and the real Linux run is [DEMO-21](../demo/21_core_install_ubuntu_arm64_lima.md). + --- @@ -250,11 +287,13 @@ layout: ### 1. 範囲と目的 -インストーラーは、このリポジトリを取り出した直後の状態から、[§8](#2-ja-hosts) に挙げたホストで、コマンド一つで GakuNin RDM（GakuNin＝学認。大学のアカウントでログインするための日本の学術ネットワーク。[REF-01](../reference/REF-01_Glossary.md)）の開発環境を動かすものです。`sudo`（コンピュータの管理者としてコマンドを動かす仕組み）を使わず、ホストのネットワーク設定を変えず、何度やり直しても壊れません。作られるものはすべて `runtime/` の下にあり、Git は無視します。 +インストーラーは、このリポジトリを取り出した直後の状態から、[§8](#2-ja-hosts) に挙げたホストで、コマンド一つで GakuNin RDM（GakuNin＝学認。大学のアカウントでログインするための日本の学術ネットワーク。[REF-01](../reference/REF-01_Glossary.md)）の開発環境を動かすものです。`install.sh` と `grdm-dev` は `sudo`（コンピュータの管理者としてコマンドを動かす仕組み）を使わず、ホストのネットワーク設定を変えず、何度やり直しても壊れません。作られるものはすべて `runtime/` の下にあり、Git は無視します。任意の前提条件スクリプト（[§2.3](#2-ja-prereq)）は事前にホストを準備し、`sudo` を使うことがありますが、変更を示して同意を得た後に限ります（[ADR-08](../decisions/ADR-08_Per_OS_Installer_Entry_Points.md)）。 **用語注。** 「インストーラー」は、ほかのプログラムが動くように準備するプログラムです。「CLI」（コマンドラインインターフェース）は、文字でコマンドを打って使うプログラムです。「コンテナ」は、一つのプログラムと必要なものを一箱に入れ、どのコンピュータでも同じように動かすための箱です。「Docker」（イメージを作りコンテナを動かす道具。[REF-01](../reference/REF-01_Glossary.md)）とその相棒の「Docker Compose」（[REF-01](../reference/REF-01_Glossary.md)）がコンテナを動かします。Compose は一枚の設定ファイルから多くのコンテナをまとめて起動する道具です。「manifest」は、どのソースリポジトリのどの版を使うかを正確に書いた一覧です（[SPEC-01](01_manifest.md)）。 -### 2. 二つの入口 +### 2. 入口 + +環境を導入する入口は二つです（§2.1、§2.2）。そのためにホストを準備する任意のスクリプトがあります（[§2.3](#2-ja-prereq)）。 #### 2.1 `installer/install.sh` — 薄い起動スクリプト @@ -301,11 +340,33 @@ flowchart TD - **外部の道具** はサブプロセスとして呼び、`doctor` が有無を確かめます。`docker`（27 以上）と Compose プラグイン（v2 以上）・buildx（一つのイメージを何種類かの計算機向けに作る Docker の助手。[REF-01](../reference/REF-01_Glossary.md)）、`scan` 用の `trivy`・`gitleaks`・`semgrep`、`scan --dynamic` 用の OWASP ZAP（動いている Web サイトを攻撃者のやり方でつつく道具。[REF-01](../reference/REF-01_Glossary.md)）のコンテナイメージ、`docs` 用の `mkdocs` です。スキャナーが無いと `scan` は終了 51 で終わります（[SPEC-03 §8](03_security_gate.md#japanese)）。 - すべてのサブコマンドは `--json` を受け付け、TOON（JSON を短く書いた文字形式。[SPEC-06 §4](06_documentation_conventions.md#japanese)）の代わりに機械が読む JSON で同じ報告を出します。 + +#### 2.3 OS ごとの前提条件スクリプト + +これらのスクリプトはホストを準備してから `install.sh` を実行します。理由は [ADR-08](../decisions/ADR-08_Per_OS_Installer_Entry_Points.md) にあります。導入の判断は持たず、すべての判断は引き続き `install.sh` と `grdm-dev` が行います。 + +| スクリプト | ホスト | 用意できるもの | +|---|---|---| +| `installer/install-ubuntu.sh` | Ubuntu 24.04 の amd64・arm64、WSL2 の中の Ubuntu 24.04 | apt パッケージ（git、Python、`python3-venv`、curl、gnupg）、Docker の署名付き apt リポジトリからの Docker Engine と Compose・buildx プラグイン、`docker` グループへの追加、arm64 での amd64 エミュレーション（ダイジェストで固定した `tonistiigi/binfmt` イメージから QEMU を登録する。実際のビルドで落ちた Ubuntu の `qemu-user-static` 8.2.2 は置き換える。[DEMO-21](../demo/21_core_install_ubuntu_arm64_lima.md)）、`~/.local/bin` の Trivy と gitleaks、`GRDM_WITH_ACCEPT=1` のときの `libatomic1`。WSL2 の中では Docker Engine を飛ばし、Docker Desktop を求める。 | +| `installer/install-macos.sh` | Apple Silicon の macOS | Homebrew の formula（`python@3.12`、git、Trivy、gitleaks）。入っている Docker Desktop を起動する。Docker Desktop・OrbStack・Colima を検出し、Docker の実行基盤も Homebrew も入れない。 | +| `installer/install-windows.ps1` | Windows 11 の amd64 | WSL2 と `Ubuntu-24.04` のディストリビューション。Docker Desktop と WSL 連携を確認する。このリポジトリを WSL のホームの下に clone し（`/mnt/c` には決して置かない）、WSL の中で `install-ubuntu.sh` を実行する。 | + +どのスクリプトも次を守ります。 + +- 何を変えるかを示し、`sudo` や管理者の操作の前に尋ねる。`docker` グループへの追加は別に尋ねる。 +- `--yes`（PowerShell では `-Yes`。`--yes` も受け付ける）で同意する。端末が無く `--yes` も無ければ、[§7](#japanese) のとおり終了 2 で止まる。 +- `--dry-run`（`-DryRun`）は判定して予定を表示し、何も変えない。 +- 再実行できる。既にある前提条件は飛ばす。 +- 失敗したら分かりやすい文言で止まり、[§6](#japanese) の終了コード 2・10・11 だけを使う。 +- ダウンロードは署名された配布元か固定したチェックサムでだけ受け入れ、シェルへ流し込まない。 +- トークンを尋ねず、読まず、保存しない。非公開の統合リポジトリへの権限を git に与える方法を案内するだけである。 +- 最後に `installer/install.sh` を実行し、ほかの引数はそのまま渡す。 + ### 3. サブコマンド | サブコマンド | 契約 | Docker に触れるか | |---|---|---| -| `doctor` | OS（macOS・Ubuntu・WSL2）、CPU（`arm64`・`amd64`）、メモリ、空き容量、Docker 27 以上、Compose プラグイン（v2 以上）、buildx を判定し、長いビルドの前に問題が分かるよう `install` に要るものを確かめる。`git`、プロファイルの非公開統合リポジトリの読み取り権限（端末で問い合わせない `git ls-remote` で確かめ、URL にトークンが含まれ得るので出力は表示しない）、`trivy`・`gitleaks`・`semgrep`、Docker VM とホストの小さい方に対するプロファイルのメモリ下限（`--memory-gib` で警告に変わる。承知の上の過剰割当）、Linux arm64 に限り amd64 エミュレーション（binfmt。有効にするコマンドを表示）、Docker が使える配置先（WSL2 では試験ファイルの `chmod 0600` を読み返して確かめ、モードを保持するファイルシステムであること。Windows のドライブは `runtime/.env` を `0600` に保てない。macOS ではエンジンから見えること。`up` が使う `postgres` イメージにリポジトリを読み取り専用で bind して問い合わせる。このサブコマンドが取得してよいイメージはこれだけ。エンジンが mount を拒むか中身が空なら終了 13、問い合わせ自体ができなければ警告。Docker VM が共有していない場所からの bind mount は黙って空になるため。問い合わせができないときは `GRDM_DOCKER_SHARED=1` で省ける）。TOON で報告する。対象外のホストかメモリ下限が収まらないときは終了 10、Docker が無い・動いていないときは 11、前提が不足なら 13、スキャナーが無ければ 51。 | 読むだけ | +| `doctor` | OS（macOS・Ubuntu・WSL2）、CPU（`arm64`・`amd64`）、メモリ、空き容量、Docker 27 以上、Compose プラグイン（v2 以上）、buildx を判定し、長いビルドの前に問題が分かるよう `install` に要るものを確かめる。`git`、プロファイルの非公開統合リポジトリの読み取り権限（端末で問い合わせない `git ls-remote` で確かめ、URL にトークンが含まれ得るので出力は表示しない）、`trivy`・`gitleaks`・`semgrep`、Docker VM とホストの小さい方に対するプロファイルのメモリ下限（`--memory-gib` で警告に変わる。承知の上の過剰割当）、Linux arm64 に限り amd64 エミュレーション（binfmt。有効にするコマンドを表示し、落ちると実測したエミュレーター（Ubuntu 24.04 の `qemu-user-static` 8.2.2。登録済みインタープリターの `--version` で判定）は拒んで置き換えるコマンドを表示）、Docker が使える配置先（WSL2 では試験ファイルの `chmod 0600` を読み返して確かめ、モードを保持するファイルシステムであること。Windows のドライブは `runtime/.env` を `0600` に保てない。macOS ではエンジンから見えること。`up` が使う `postgres` イメージにリポジトリを読み取り専用で bind して問い合わせる。このサブコマンドが取得してよいイメージはこれだけ。エンジンが mount を拒むか中身が空なら終了 13、問い合わせ自体ができなければ警告。Docker VM が共有していない場所からの bind mount は黙って空になるため。問い合わせができないときは `GRDM_DOCKER_SHARED=1` で省ける）。TOON で報告する。対象外のホストかメモリ下限が収まらないときは終了 10、Docker が無い・動いていないときは 11、前提が不足なら 13、スキャナーが無ければ 51。 | 読むだけ | | `plan` | manifest とプロファイルを解決し、どのイメージを作りどのサービスを起こすか、公開ポート（[§4](#2-ja-ports)）とともに表示する。何も変えない。manifest が [SPEC-01](01_manifest.md) の検査に落ちれば終了 20。 | 触れない | | `secrets init` / `secrets rotate` | `init` は乱数の秘密（DB パスワード、Django の秘密鍵、CAS の鍵）を `runtime/.env` にモード `0600` で書く。既定値や例のパスワードは決して書かない。既存ファイルは `--force` がなければ上書きしない。`rotate` はすべての秘密を作り直し、稼働中の PostgreSQL と RabbitMQ のパスワードを変更し、新しい環境を受け取るようコンテナを再作成し、旧い資格情報が拒否されることを確かめる。どこかで失敗したら以前の `runtime/.env` に戻して 40 で終わる。`init --force` と `rotate` は廃止した資格情報のキーを消す。更新後は、`rotate` や再作成の前に `compose render` を実行する。以前に生成した Compose ファイルは廃止したキーを参照し続けるからである。四つのキーには CAS のイメージが要する長さがある。`TGC_ENCRYPTION_KEY` は 43 文字、`TGC_SIGNING_KEY` は 86 文字、`OSF_API_INSTITUTIONS_AUTH_JWESECRET` はちょうど 32 バイト、`OSF_API_INSTITUTIONS_AUTH_JWTSECRET` は 32 バイト以上で、後の二つは osf.io の `JWE_SECRET` と `JWT_SECRET` でもある（[ADR-07 §7](../decisions/ADR-07_Native_Builds_of_EOL_Middleware.md#japanese)）。四つとも base64url の文字（`A`–`Z`、`a`–`z`、`0`–`9`、`-`、`_`。空白・引用符・パディングは不可）だけを使い、公開された既定値であってはならず、`TGC_ENCRYPTION_KEY` は 32 バイトへ正準に復号できなければならない。レッドチームに秘密を取られた後に使う（[SPEC-03 §6](03_security_gate.md#japanese)）。 | 触れない / 触れる | | `compose render` | `runtime/.env` に生成すべきキーが無いとき、または CAS のキーが使えないとき（長さの誤り、base64url 以外の文字、公開された既定値）は、キーの名前だけを示して終了 2 で止まる（どちらも `secrets rotate` で直す）。manifest とプロファイルから `runtime/docker-compose.yml` を書く。内部ネットワーク `grdm` と公開専用の `grdm-edge`（`--egress ` で追加接続し `runtime/.env` に記録。[SPEC-03 §3.2](03_security_gate.md#japanese)）、部品ごとの `platform:` は manifest の `build_platform` から、ポートは `127.0.0.1` にだけ公開（[§4](#2-ja-ports)）、全サービスに非 root の `user:`、`read_only`・`cap_drop`・資源制限（`mem_limit`・`cpus`・`pids_limit`。`doctor` が検出したホストのメモリと Docker エンジンの VM メモリ（`docker info`。Docker Desktop では VM はホストの一部。実機第 10 回）の小さい方に応じて決め、テンプレートの下限 `mem_min_mb` が先に取り分を確保し、残りを他が分け合う。CPU はエンジンが報告する数（`docker info`。不明なときだけホストの数）から決める）、サービスごとの例外は `manifest/hardening.json` から（[SPEC-03 §3.3](03_security_gate.md#japanese)）、`platform:` は `doctor` の結果から選ぶ、名前付きボリュームは `runtime/volumes/` の下、`tmpfs` の各項目は `mode=1777` で mount（Compose の短い記法は root 専用の `0755` になる。実機第 11 回）、テンプレート値のプレースホルダ `{{HOST_PORT:}}`・`{{CONTAINER_PORT:}}`・`{{AUTH_URL}}`（fakecas の公開 URL。`--auth shibboleth` で fakecas が外れるときは web の URL）を解決する。組み立てる部品の `image:` に最後のビルド（`runtime/reports/build_latest.json`）のタグを使うのは、manifest が固定するタグと同じときだけである。記録されたビルドが固定と違うときは古い（最後の `build` の後で固定が動いた）とみなし、代わりに固定のタグを書き、報告の `stale_build_tags`（`component`・`pinned`・`recorded`・`image`）に載せ、標準エラーに警告の囲みを出して対処（`grdm-dev build` を実行してから `grdm-dev up`）を示す。`install` は組む前に render するので、終了コードは 0 のままである。一度も組んでいないイメージは古いものを動かす代わりに `up` を失敗させる。manifest が固定しない部品は記録されたタグを使う。`osf.env` には osf.io から mongo への TLS 接続 `SHAREJS_DB_URL=mongodb://mongo:27017/sharejs?ssl=true` と `SHAREJS_DB_CA_FILE=/mongo-tls/mongodb.crt` を書き、web・api・admin・worker は sharejs と同じく `mongo_tls` を読み取り専用でマウントする（[ADR-07](../decisions/ADR-07_Native_Builds_of_EOL_Middleware.md)）。[ADR-02](../decisions/ADR-02_Network_Model.md) に従う。 | 触れない | @@ -397,7 +458,7 @@ exit_codes[17]{code,meaning}: - **何度でも同じ結果。** どのサブコマンドも二度実行して同じ結果になる。`secrets init` は既存の秘密を残し、`compose render` は生成ファイルだけを上書きし、`build` は buildx のキャッシュを使い、`up` は健康なサービスには何もしない。 - **秘密を Git に入れない。** `runtime/` は `.gitignore` に入れる。`runtime/` の下のファイルや秘密に見える値がコミットされると、`docs check` と CI の秘密スキャンが不合格にする。 - **ホストのネットワークを変えない。** 旧手引きのループバック別名 `192.168.168.167` は作らない。`compose render` が上流の `local.py` と `.docker-compose.*.env` の設定を、Compose ネットワーク内ではサービス名、ブラウザ向けには `127.0.0.1` を使うよう書き換える（[ADR-02](../decisions/ADR-02_Network_Model.md)）。 -- **初回の起動は遅く、二度目からは速い。** `up` は health を最長 1800 秒（`--timeout`）待つ。Apple Silicon では初回の `up` が ember-osf-web をエミュレーション下で組み（10〜25 分）、`osf_init` を実行する。どちらも結果を名前付きボリュームに残すので、二度目の `up` は 1〜2 分で健康になる（実機第 10〜18 回。初回ビルドは 13〜25 分かかったので既定を 30 分にした）。 +- **初回の起動は遅く、二度目からは速い。** `up` は health を最長 1800 秒（`--timeout`）待つ。Apple Silicon では初回の `up` が ember-osf-web をエミュレーション下で組み（10〜25 分）、`osf_init` を実行する。どちらも結果を名前付きボリュームに残すので、二度目の `up` は 1〜2 分で健康になる（実機第 10〜18 回。初回ビルドは 13〜25 分かかったので既定を 30 分にした）。Linux arm64 ではエミュレーションが QEMU で、ember の初回ビルドは 1 コアで 44 分かかったため、プロファイルに amd64 のサービスがあるときの既定は 5400 秒とする（[DEMO-21](../demo/21_core_install_ubuntu_arm64_lima.md)）。`--timeout` で上書きできるのは同じ。 - **`-private` のコードを置かない。** インストーラーは実行時に manifest の名前と SHA（コミット一つの指紋。[REF-01](../reference/REF-01_Glossary.md)）で統合リポジトリを取得する。この公開リポジトリにはそれらのコードを置かない（[POL-01](../policy/POL-01_Developer_Environment_2027_Position.md)）。 @@ -409,7 +470,7 @@ exit_codes[17]{code,meaning}: |---|---|---| | macOS 14 以上 と Docker Desktop | arm64 | 正式（M2 の受入はここで実行） | | Ubuntu 24.04 と Docker Engine | amd64・arm64 | 正式 | -| Windows 11 の WSL2（Ubuntu 24.04）と Docker Desktop | amd64 | 動くことを必須とする。CI ではビルド確認のみ | +| Windows 11 の WSL2（Ubuntu 24.04）と Docker Desktop | amd64 | 動くことを必須とする。CI は `install-windows.ps1` の解析と dry run のみで、実際の実行には所有者の実機が要る（[ADR-08 §5](../decisions/ADR-08_Per_OS_Installer_Entry_Points.md#ja-limits)） | | その他の Linux、Intel の macOS | 任意 | できる範囲で対応。`doctor` が警告して続行 | ### 9. ディレクトリ構成 @@ -417,6 +478,10 @@ exit_codes[17]{code,meaning}: ```toon layout: installer/install.sh: bash 3.2 の起動スクリプト + installer/install-ubuntu.sh: Ubuntu 24.04 と WSL2 の前提条件（§2.3） + installer/install-macos.sh: macOS の前提条件（§2.3） + installer/install-windows.ps1: Windows 11 の前提条件と WSL2 への受け渡し（§2.3） + installer/prereq-lib.sh: bash の前提条件スクリプト二つが共有する部品 installer/pyproject.toml: パッケージ情報、コンソールスクリプト grdm-dev、追加 accept installer/grdm_dev/: cli.py: 引数の解釈、サブコマンドの振り分け、終了コード @@ -451,6 +516,10 @@ layout: `python3 -m unittest discover -s installer/tests` はすべての PR（プルリクエスト。レビュー待ちの変更の提案。[REF-01](../reference/REF-01_Glossary.md)）で合格していなければなりません。各モジュールに `test_<モジュール>.py` を置き、成功ケースと拒否ケースを少なくとも一つずつ含め、拒否ケースでは [§6](#japanese) の終了コードを assert します。単体試験は Docker を要求しません。`host.py` と `build.py` は実行器を差し替えられる作りにし、試験では偽の `docker` の出力を渡します。統合と受入の段は [SPEC-04](04_acceptance_tests.md) で定めます。 +[§2.3](#2-ja-prereq) の前提条件スクリプトは、すべての PR で `installer-prereq-lint.yml` が確かめます。`bash -n`、shellcheck、`windows-latest` の PowerShell パーサー、各 OS での dry run です。単体試験はスタブだけの `PATH` の下で `--dry-run` か `--help` でだけ実行します。手で起動する `installer-full.yml` は `ubuntu-24.04` と `ubuntu-24.04-arm` で `minimum` の完全な導入を行い、リポジトリのシークレット `GRDM_INTEGRATION_READ_TOKEN` を必要とします。これらのランナーで証明できないことは [ADR-08 §5](../decisions/ADR-08_Per_OS_Installer_Entry_Points.md#ja-limits) に挙げています。 + +> **Note:** 二つのワークフローとその補助 `.github/scripts/Test-PowerShellSyntax.ps1` は**保留中**です。`workflow` スコープを持つトークンで誰かが push するまで、ブランチ `feat/per-os-installers` で待ちます。それまでこの段落の確認は手で行い、実際の Linux での実行は [DEMO-21](../demo/21_core_install_ubuntu_arm64_lima.md) が示します。 + --- ## Glossary / 用語 diff --git a/docs/spec/03_security_gate.md b/docs/spec/03_security_gate.md index 3d7c066..f8f7144 100644 --- a/docs/spec/03_security_gate.md +++ b/docs/spec/03_security_gate.md @@ -5,7 +5,9 @@ **Jump:** [English](#english) · [日本語](#japanese) -- Version / 版: 0.11 — 2026-11-10. **Learned:** a hardening exception that blames an image's start command expires with that image, so building CAS from source removed it entirely, while the secure-cookie default had to be switched off explicitly and in one place because the stack is plain HTTP on loopback (ADR-07 §7). ／ イメージの起動コマンドを理由にした堅牢化の例外はそのイメージとともに期限が切れる。CAS をソースから組むとこの例外はまるごと消えた。一方、スタックはループバック上の平文 HTTP なので、Secure Cookie の既定は一か所で明示して切る必要があった（ADR-07 §7）。 +- Version / 版: 0.13 — 2026-11-10. **Learned:** a type check that runs after a blocking open never runs: a FIFO in place of a report hung the scan, so report files are opened with `O_NONBLOCK` and `O_NOFOLLOW` and their descriptor type is checked before any read or write; and the private-parent check is a snapshot, so the user's own checkout and its ancestors are a precondition, not something the scan defends ([PR #44 review r2](../reviews/2026-11-10_RDM-developer-guide_PR44_Per_OS_Installers_r2.md)). ／ 止まる open の後の型検査は実行されない。報告の代わりの FIFO でスキャンが止まったので、報告ファイルは `O_NONBLOCK` と `O_NOFOLLOW` で開き、読み書きの前に記述子の型を確かめる。また非公開の親の確認は一時点のもので、利用者自身のチェックアウトと祖先は前提条件であり、スキャンが守るものではない（[PR #44 レビュー r2](../reviews/2026-11-10_RDM-developer-guide_PR44_Per_OS_Installers_r2.md)）。 + 0.12 — 2026-11-10. **Learned:** a world-writable directory is safe only while its private parent really is private, so the ZAP work directories are created only after `runtime/` is verified as a 0700 real directory owned by the user, under unpredictable names, and a report that is a symlink is refused ([PR #44 review r1](../reviews/2026-11-10_RDM-developer-guide_PR44_Per_OS_Installers_r1.md) R3). ／ 誰でも書けるディレクトリが安全なのは、非公開の親が実際に非公開である間だけである。そこで ZAP の作業ディレクトリは、`runtime/` が利用者の所有の実ディレクトリで 0700 だと確かめてから、予測できない名前で作り、シンボリックリンクの報告は拒む（[PR #44 レビュー r1](../reviews/2026-11-10_RDM-developer-guide_PR44_Per_OS_Installers_r1.md) R3）。 + 0.11 — 2026-11-10. **Learned:** a hardening exception that blames an image's start command expires with that image, so building CAS from source removed it entirely, while the secure-cookie default had to be switched off explicitly and in one place because the stack is plain HTTP on loopback (ADR-07 §7). ／ イメージの起動コマンドを理由にした堅牢化の例外はそのイメージとともに期限が切れる。CAS をソースから組むとこの例外はまるごと消えた。一方、スタックはループバック上の平文 HTTP なので、Secure Cookie の既定は一か所で明示して切る必要があった（ADR-07 §7）。 0.10 — 2026-11-10. **Learned:** a converter behind HTTP needs no shared volume and no capability, so the exception for its old image could be removed once the client was changed. ／ HTTP の背後の変換サービスは共有ボリュームも capability も要らないので、クライアントを変えた時点で旧イメージの例外を除けた。 0.9 — 2026-11-10. **Learned:** a native rebuild removed the elasticsearch6 hardening entry as well as its Log4Shell findings, while a scan count alone would have approved a unoconv image that MFR's client cannot use, so an image change is accepted only with its client's real path proven (ADR-07 §5). ／ ネイティブの組み直しで elasticsearch6 の堅牢化の項目も Log4Shell の検出も消えた。一方、検出数だけなら MFR のクライアントが使えない unoconv イメージを通していたので、イメージの変更はクライアントの実経路を証明して初めて受け入れる（ADR-07 §5）。 0.8 — 2026-11-10. **Learned:** per-volume init containers multiplied hardening exceptions and still missed redis and the other uid-1000 volumes, so one generic owner step, derived from each volume's writers, replaces them (cross-OS audit, 2026-11-10). ／ ボリュームごとの init コンテナは堅牢化の例外を増やすだけで redis や他の uid 1000 のボリュームを取りこぼしていた。各ボリュームの書き手から導く一つの汎用の所有者設定の段に置き換えると分かった（OS 横断の監査、2026-11-10）。 @@ -110,7 +112,7 @@ Because this repository is public, every pull request and every push to `develop #### 4.3 Dynamic scan: OWASP ZAP baseline -A *dynamic scan* pokes the running web application like a browser would. `grdm-dev scan --dynamic` starts OWASP ZAP (a tool that pokes at a running website the way an attacker would; see [REF-01](../reference/REF-01_Glossary.md)) in a container attached to the Compose internal network ([ADR-02 §3](../decisions/ADR-02_Network_Model.md#english)) and runs the baseline scan against the Compose service names and container ports: `http://web:5000` (web), `http://admin:8001` (admin), and `http://ember_osf_web:4200` (ember). It never targets `127.0.0.1`: inside the ZAP container, loopback is ZAP itself, not a service and not the host. The container ports are the ports the services listen on inside the network; their numbers equal the default host ports in the ports table of [SPEC-02 §4](02_installer_cli.md#2-en-ports), but that table defines *host* ports, which browsers, acceptance tests, and `grdm-dev health --network` use. When `compose render` remaps a host port on collision and records it as `GRDM_PORT_` in `runtime/.env` ([SPEC-02 §4](02_installer_cli.md#2-en-ports)), host-side checks read the new number from there and the ZAP targets do not change. ZAP risk High maps to HIGH, Medium to MEDIUM. The scan runs only after `grdm-dev health` reports all services healthy: a full `grdm-dev scan` skips the dynamic part and notes it in the report, while an explicit `scan --dynamic` exits 52. +A *dynamic scan* pokes the running web application like a browser would. `grdm-dev scan --dynamic` starts OWASP ZAP (a tool that pokes at a running website the way an attacker would; see [REF-01](../reference/REF-01_Glossary.md)) in a container attached to the Compose internal network ([ADR-02 §3](../decisions/ADR-02_Network_Model.md#english)) and runs the baseline scan against the Compose service names and container ports: `http://web:5000` (web), `http://admin:8001` (admin), and `http://ember_osf_web:4200` (ember). It never targets `127.0.0.1`: inside the ZAP container, loopback is ZAP itself, not a service and not the host. The container ports are the ports the services listen on inside the network; their numbers equal the default host ports in the ports table of [SPEC-02 §4](02_installer_cli.md#2-en-ports), but that table defines *host* ports, which browsers, acceptance tests, and `grdm-dev health --network` use. When `compose render` remaps a host port on collision and records it as `GRDM_PORT_` in `runtime/.env` ([SPEC-02 §4](02_installer_cli.md#2-en-ports)), host-side checks read the new number from there and the ZAP targets do not change. ZAP risk High maps to HIGH, Medium to MEDIUM. The scan runs only after `grdm-dev health` reports all services healthy: a full `grdm-dev scan` skips the dynamic part and notes it in the report, while an explicit `scan --dynamic` exits 52. ZAP runs as its own uid, so each target gets a work directory writable by every uid, created exclusively under an unpredictable name below `runtime/reports/` and removed after the report is copied. This is allowed only while `runtime/` is a real directory owned by the user with mode 0700 and `reports/` is a real directory owned by the user; otherwise ZAP is not run and the dynamic part is incomplete (exit 51). A report that is not a regular file, such as a symlink or a FIFO, is refused: report files are opened with `O_NOFOLLOW` and `O_NONBLOCK`, and the descriptor's type is checked before anything is read or written. The checks are taken once before the run; they assume that the user's checkout and its ancestors are not changed by someone else meanwhile, and they do not make the scan race-proof against the owner. #### 4.4 Dependency update bots @@ -339,7 +341,7 @@ Exit codes are defined in [SPEC-02 §6](02_installer_cli.md#english); this table #### 4.3 動的スキャン: OWASP ZAP ベースライン -「動的スキャン」は、動いている Web アプリをブラウザのように突いて調べます。`grdm-dev scan --dynamic` は Compose の内部ネットワークにつないだコンテナで OWASP ZAP（動いている Web サイトを攻撃者のやり方でつつく道具。[REF-01](../reference/REF-01_Glossary.md)）を起動し（[ADR-02 §3](../decisions/ADR-02_Network_Model.md#japanese)）、Compose のサービス名とコンテナ側ポート、すなわち `http://web:5000`（web）、`http://admin:8001`（admin）、`http://ember_osf_web:4200`（ember）に対してベースラインスキャンを行います。`127.0.0.1` を対象にすることはありません。ZAP のコンテナの中では、ループバックはサービスでもホストでもなく ZAP 自身だからです。コンテナ側ポートはサービスがネットワークの中で待ち受けるポートで、番号は [SPEC-02 §4](02_installer_cli.md#2-ja-ports) のポート表の既定のホスト側ポートと同じですが、その表が定めるのは「ホスト側」のポートで、ブラウザ・受入試験・`grdm-dev health --network` が使います。`compose render` が衝突でホスト側ポートを付け替えて `runtime/.env` に `GRDM_PORT_` として記録したとき（[SPEC-02 §4](02_installer_cli.md#2-ja-ports)）、ホスト側の確認はそこから新しい番号を読み、ZAP の対象は変わりません。ZAP のリスク High は HIGH、Medium は MEDIUM に対応させます。`grdm-dev health` が全サービス健全と報告した後にだけ動かします。`grdm-dev scan` 全体では動的部分を省略して報告に記し、`scan --dynamic` を明示したときは終了 52 で終わります。 +「動的スキャン」は、動いている Web アプリをブラウザのように突いて調べます。`grdm-dev scan --dynamic` は Compose の内部ネットワークにつないだコンテナで OWASP ZAP（動いている Web サイトを攻撃者のやり方でつつく道具。[REF-01](../reference/REF-01_Glossary.md)）を起動し（[ADR-02 §3](../decisions/ADR-02_Network_Model.md#japanese)）、Compose のサービス名とコンテナ側ポート、すなわち `http://web:5000`（web）、`http://admin:8001`（admin）、`http://ember_osf_web:4200`（ember）に対してベースラインスキャンを行います。`127.0.0.1` を対象にすることはありません。ZAP のコンテナの中では、ループバックはサービスでもホストでもなく ZAP 自身だからです。コンテナ側ポートはサービスがネットワークの中で待ち受けるポートで、番号は [SPEC-02 §4](02_installer_cli.md#2-ja-ports) のポート表の既定のホスト側ポートと同じですが、その表が定めるのは「ホスト側」のポートで、ブラウザ・受入試験・`grdm-dev health --network` が使います。`compose render` が衝突でホスト側ポートを付け替えて `runtime/.env` に `GRDM_PORT_` として記録したとき（[SPEC-02 §4](02_installer_cli.md#2-ja-ports)）、ホスト側の確認はそこから新しい番号を読み、ZAP の対象は変わりません。ZAP のリスク High は HIGH、Medium は MEDIUM に対応させます。`grdm-dev health` が全サービス健全と報告した後にだけ動かします。`grdm-dev scan` 全体では動的部分を省略して報告に記し、`scan --dynamic` を明示したときは終了 52 で終わります。ZAP は自分の uid で動くので、対象ごとに誰でも書ける作業ディレクトリを、`runtime/reports/` の下に予測できない名前で排他的に作り、報告を写した後に消します。これを許すのは、`runtime/` が利用者の所有の実ディレクトリでモード 0700、`reports/` が利用者の所有の実ディレクトリである間だけです。そうでなければ ZAP を動かさず、動的部分は不完全（終了 51）とします。通常のファイルでない報告（シンボリックリンクや FIFO など）は拒みます。報告ファイルは `O_NOFOLLOW` と `O_NONBLOCK` で開き、読み書きの前に記述子の型を確かめます。確認は実行前の一度だけで、その間に利用者のチェックアウトと祖先を他人が変えないことを前提とし、所有者自身の変更に対して競合に強いとは主張しません。 #### 4.4 依存更新ボット diff --git a/docs/spec/04_acceptance_tests.md b/docs/spec/04_acceptance_tests.md index 32fdc9f..43881e0 100644 --- a/docs/spec/04_acceptance_tests.md +++ b/docs/spec/04_acceptance_tests.md @@ -5,6 +5,7 @@ **Jump:** [English](#english) · [日本語](#japanese) +- Version / 版: 0.9 — 2026-11-10. **Learned:** an install command that works on a desktop can fail on a server image: on the Ubuntu 24.04 cloud image Playwright's own driver would not start without `libatomic1`, and Chromium needs system libraries, so on Linux the printed command installs both ([DEMO-21](../demo/21_core_install_ubuntu_arm64_lima.md)). ／ デスクトップで動く導入コマンドがサーバーのイメージでは失敗しうる。Ubuntu 24.04 のクラウドイメージでは Playwright 自身のドライバーが `libatomic1` 無しに起動せず、Chromium にはシステムのライブラリが要る。そこで Linux では表示するコマンドが両方を入れる（[DEMO-21](../demo/21_core_install_ubuntu_arm64_lima.md)）。 - Version / 版: 0.8 — 2026-11-10. **Learned:** a known-asset exception written as a path prefix also accepts lookalike and unlisted paths, so a false pass was possible; each entry now names a whole path, an exact endpoint or a version plus file name (PR #35 review r1, R1-02). ／ パスの先頭で書いた既知資産の例外は、似たパスや一覧にないパスも受け入れ、偽の合格を生み得ると分かった。各項目はパス全体（正確な endpoint、または版とファイル名）で定める（PR #35 レビュー r1、R1-02）。 - Version / 版: 0.7 — 2026-11-09. **Learned:** the GRDM pages of today load a few third-party assets (first real login, acceptance run 9), so "abort and fail" alone would keep every browser scenario red; rule 4 now keeps blocking them and counts only a fixed list that must be empty at M5. ／ 現在の GRDM の画面は少数の外部資産を読み込む（初の実ログイン、受入第 9 回）ため、「中断して失敗」だけではすべてのブラウザシナリオが不合格のままになる。規則 4 は遮断を続け、M5 までに空にする固定の一覧だけを件数の記録にとどめる。 - Version / 版: 0.6 — 2026-11-06. **Learned:** a glossary term is only explained for the reader when its note and REF-01 link sit at the first prose use, counting lower-case and plural forms, so first-use notes with links were added in both halves (SPEC-06 §3). ／ 用語注と REF-01 へのリンクは、小文字や複数形も含めた本文の初出に置かなければ読者に届かないと分かったため、英和両方の初出にリンク付きの用語注を補った（SPEC-06 §3）。 @@ -29,7 +30,7 @@ Acceptance tests drive a real browser against the environment that [SPEC-02](02_ - Location: `installer/acceptance/`, one file per scenario, named `test_.py`. - Runner: `grdm-dev accept [--profile ...] [--only ] [--json]`. It runs `pytest` inside the same `.venv/` that `install.sh` created ([SPEC-02 §2.2](02_installer_cli.md#english)) and exits 60 if any test fails. -- Browsers: `grdm-dev accept` checks that Chromium is present under `PLAYWRIGHT_BROWSERS_PATH=.venv/.playwright/` and, if not, exits 2 and prints the exact install command (`pip install -e "installer[accept]"` then `playwright install chromium` with that variable). The download is a one-time setup step, never part of a test run. Nothing is installed outside the venv (a private box of Python tools for one project; see [REF-01](../reference/REF-01_Glossary.md)). +- Browsers: `grdm-dev accept` checks that Chromium is present under `PLAYWRIGHT_BROWSERS_PATH=.venv/.playwright/` and, if not, exits 2 and prints the exact install command (`pip install -e "installer[accept]"` then `playwright install chromium` with that variable; on Linux it is preceded by `sudo apt-get install -y libatomic1` and followed by `playwright install-deps chromium`, which asks for `sudo` to add Chromium's system libraries). The download is a one-time setup step, never part of a test run. Nothing is installed outside the venv (a private box of Python tools for one project; see [REF-01](../reference/REF-01_Glossary.md)). - Target URL: always `http://127.0.0.1:` as published by `compose render` ([ADR-02](../decisions/ADR-02_Network_Model.md); defaults in the ports table of [SPEC-02 §4](02_installer_cli.md#2-en-ports)). Tests read the ports from `runtime/.env` (`GRDM_PORT_`); they never hard-code them. ### 2. Test tiers @@ -150,7 +151,7 @@ The Mac Studio self-hosted runner is registered only with the private `-dev2027` - 置き場: `installer/acceptance/`。シナリオごとに一ファイル、名前は `test_<シナリオ>.py`。 - 実行器: `grdm-dev accept [--profile ...] [--only <シナリオ>] [--json]`。`install.sh` が作った同じ `.venv/` の中で `pytest` を動かし（[SPEC-02 §2.2](02_installer_cli.md#japanese)）、一件でも失敗すれば終了 60。 -- ブラウザ: `grdm-dev accept` は `PLAYWRIGHT_BROWSERS_PATH=.venv/.playwright/` に Chromium があるかを確かめ、なければ終了コード 2 で導入コマンド（`pip install -e "installer[accept]"` のあと同じ変数を付けて `playwright install chromium`）をそのまま表示する。ダウンロードは一度だけの準備であり、試験の実行には含めない。venv（プロジェクトごとの Python 道具箱。[REF-01](../reference/REF-01_Glossary.md)）の外には何も入れない。 +- ブラウザ: `grdm-dev accept` は `PLAYWRIGHT_BROWSERS_PATH=.venv/.playwright/` に Chromium があるかを確かめ、なければ終了コード 2 で導入コマンド（`pip install -e "installer[accept]"` のあと同じ変数を付けて `playwright install chromium`。Linux では前に `sudo apt-get install -y libatomic1`、後に Chromium のシステムライブラリを `sudo` で入れる `playwright install-deps chromium` が付く）をそのまま表示する。ダウンロードは一度だけの準備であり、試験の実行には含めない。venv（プロジェクトごとの Python 道具箱。[REF-01](../reference/REF-01_Glossary.md)）の外には何も入れない。 - 対象 URL: 常に `compose render` が公開した `http://127.0.0.1:<ポート>`（[ADR-02](../decisions/ADR-02_Network_Model.md)。既定値は [SPEC-02 §4](02_installer_cli.md#2-ja-ports) のポート表）。ポートは `runtime/.env`（`GRDM_PORT_`）から読み、決め打ちしない。 ### 2. 試験の段 diff --git a/docs/spec/README.md b/docs/spec/README.md index 4eaee18..46ebb5d 100644 --- a/docs/spec/README.md +++ b/docs/spec/README.md @@ -5,6 +5,7 @@ **Jump:** [English](#english) · [日本語](#japanese) +- Version / 版: 0.3 — 2026-11-10. **Learned:** the installer specification also governs the scripts that run before the installer, so an index line must name them ([ADR-08](../decisions/ADR-08_Per_OS_Installer_Entry_Points.md)). ／ インストーラーの仕様は、インストーラーの前に動くスクリプトも定めるので、索引の行にも名指しする（[ADR-08](../decisions/ADR-08_Per_OS_Installer_Entry_Points.md)）。 - Version / 版: 0.2 — 2026-11-06. **Learned:** a glossary term is only explained for the reader when its note and REF-01 link sit at the first prose use, counting lower-case and plural forms, so first-use notes with links were added in both halves (SPEC-06 §3). ／ 用語注と REF-01 へのリンクは、小文字や複数形も含めた本文の初出に置かなければ読者に届かないと分かったため、英和両方の初出にリンク付きの用語注を補った（SPEC-06 §3）。 - Version / 版: 0.1 — 2026-11-05. **Learned:** the installer, the security gate and the acceptance tests had to be written down as rules before any code, otherwise each implementer would fill the gaps differently. ／ インストーラー、セキュリティの門、受入試験は、コードより先に規則として書いておかないと、実装する人ごとに隙間の埋め方が変わると分かった。 - Normative for: [SPEC-00 Overview](./00_overview.md) through [SPEC-06 Documentation Conventions](./06_documentation_conventions.md). @@ -25,7 +26,7 @@ A *specification* is a rule book that implementers must follow. This folder cont |---|---|---| | SPEC-00 | [00_overview.md](./00_overview.md) | Component map, milestones, roles, the big picture. | | SPEC-01 | [01_manifest.md](./01_manifest.md) | The manifest: which upstream commits and PRs the installer builds. | -| SPEC-02 | [02_installer_cli.md](./02_installer_cli.md) | The `grdm-dev` CLI and `install.sh`: host checks, secrets, compose, build, health, reports. | +| SPEC-02 | [02_installer_cli.md](./02_installer_cli.md) | The `grdm-dev` CLI, `install.sh` and the per-OS prerequisite scripts: host checks, secrets, compose, build, health, reports. | | SPEC-03 | [03_security_gate.md](./03_security_gate.md) | SBOM, CVE scan, static analysis, ZAP, AI red team, per-milestone thresholds. | | SPEC-04 | [04_acceptance_tests.md](./04_acceptance_tests.md) | Playwright Python acceptance: success and rejection cases per feature. | | SPEC-05 | [05_upstream_tracking.md](./05_upstream_tracking.md) | Daily mirror and diff report, weekly AI integration PR, COS reference tracking. | @@ -63,7 +64,7 @@ flowchart TD |---|---|---| | SPEC-00 | [00_overview.md](./00_overview.md) | 部品の地図、節目、役割、全体像。 | | SPEC-01 | [01_manifest.md](./01_manifest.md) | マニフェスト：どの上流のコミットと PR を組むか。 | -| SPEC-02 | [02_installer_cli.md](./02_installer_cli.md) | `grdm-dev` CLI と `install.sh`：ホスト判定、秘密、compose、ビルド、健康確認、報告。 | +| SPEC-02 | [02_installer_cli.md](./02_installer_cli.md) | `grdm-dev` CLI、`install.sh`、OS ごとの前提条件スクリプト：ホスト判定、秘密、compose、ビルド、健康確認、報告。 | | SPEC-03 | [03_security_gate.md](./03_security_gate.md) | SBOM、CVE スキャン、静的解析、ZAP、AI レッドチーム、節目ごとの基準。 | | SPEC-04 | [04_acceptance_tests.md](./04_acceptance_tests.md) | Playwright Python 受入：機能ごとの成功・拒否ケース。 | | SPEC-05 | [05_upstream_tracking.md](./05_upstream_tracking.md) | 日次ミラーと差分報告、週次の AI 統合 PR、COS 参照追跡。 | diff --git a/installer/README.md b/installer/README.md index a164f1e..17e901d 100644 --- a/installer/README.md +++ b/installer/README.md @@ -5,6 +5,7 @@ **Jump:** [English](#english) · [日本語](#japanese) +- Version / 版: 0.3 — 2026-11-10. **Learned:** a host that lacks the prerequisites needs a script before `install.sh`, and that script must still end at `install.sh` so that the package stays the only decision path ([ADR-08](../docs/decisions/ADR-08_Per_OS_Installer_Entry_Points.md)). ／ 前提条件の無いホストには `install.sh` の前にスクリプトが要るが、パッケージを唯一の判断の道筋に保つため、そのスクリプトは必ず `install.sh` で終わる（[ADR-08](../docs/decisions/ADR-08_Per_OS_Installer_Entry_Points.md)）。 - Version / 版: 0.2 — 2026-11-06. **Learned:** a glossary term is only explained for the reader when its note and REF-01 link sit at the first prose use, counting lower-case and plural forms, so first-use notes with links were added in both halves (SPEC-06 §3). ／ 用語注と REF-01 へのリンクは、小文字や複数形も含めた本文の初出に置かなければ読者に届かないと分かったため、英和両方の初出にリンク付きの用語注を補った（SPEC-06 §3）。 - Version / 版: 0.1 — 2026-11-05. **Learned:** the module contract (`register`/`run`, injectable runner) had to be fixed before several people could implement subcommands in parallel without stepping on each other. ／ 複数人が同時にサブコマンドを実装してもぶつからないよう、モジュールの契約（`register`/`run`、差し替え可能な runner）を先に固定する必要があると分かった。 - Related: [docs/README.md](../docs/README.md) · [SPEC-02](../docs/spec/02_installer_cli.md) · [SPEC-04](../docs/spec/04_acceptance_tests.md) diff --git a/installer/grdm_dev/accept.py b/installer/grdm_dev/accept.py index 7acaffb..64987bc 100644 --- a/installer/grdm_dev/accept.py +++ b/installer/grdm_dev/accept.py @@ -83,6 +83,16 @@ "chromium_headless_shell*/chrome-headless-shell-*/chrome-headless-shell", ) INSTALL_HINT = 'pip install -e "installer[accept]" && PLAYWRIGHT_BROWSERS_PATH=.venv/.playwright playwright install chromium' +# Linux also needs Chromium's system libraries (`install-deps`, which asks for sudo), and Playwright's own Node.js +# driver needs libatomic1, which the Ubuntu 24.04 cloud image lacks: without it even `playwright install` fails +# with "libatomic.so.1: cannot open shared object file" (DEMO-21, arm64). +LINUX_INSTALL_HINT = ("sudo apt-get install -y libatomic1 && " + INSTALL_HINT + + " && PLAYWRIGHT_BROWSERS_PATH=.venv/.playwright playwright install-deps chromium") + + +def install_hint(system: str | None = None) -> str: + """The command that installs the [accept] extra and Chromium on this kind of host.""" + return LINUX_INSTALL_HINT if (system if system is not None else sys.platform).startswith("linux") else INSTALL_HINT REQUIRED_MODULES = ("pytest", "playwright", "pytest_playwright") # Disposable-user mailbox domain. `.test` is reserved (RFC 2606): never a real person (POL-01). SEED_DOMAIN = "acceptance.test" @@ -611,7 +621,7 @@ def fail(code: int, detail: str) -> int: missing = missing_modules(importer) if missing: - return fail(exit_codes.USAGE, f"missing {', '.join(missing)}; run: {INSTALL_HINT}") + return fail(exit_codes.USAGE, f"missing {', '.join(missing)}; run: {install_hint()}") base_url = (getattr(args, "base_url", None) or default_base_url(ctx)).rstrip("/") problem = loopback_problem(base_url) @@ -620,11 +630,11 @@ def fail(code: int, detail: str) -> int: browsers = ctx.root / BROWSERS_PATH if chromium_executable(browsers) is None: - return fail(exit_codes.USAGE, f"no Chromium executable under {browsers}; run: {INSTALL_HINT}") + return fail(exit_codes.USAGE, f"no Chromium executable under {browsers}; run: {install_hint()}") suite_dir = ACCEPTANCE_DIR if not (suite_dir / "conftest.py").is_file(): - return fail(exit_codes.USAGE, f"acceptance suite not found at {suite_dir}; reinstall: {INSTALL_HINT}") + return fail(exit_codes.USAGE, f"acceptance suite not found at {suite_dir}; reinstall: {install_hint()}") selection = select_tests(getattr(args, "only", None), suite_dir) if selection is None: return fail(exit_codes.USAGE, f"--only {args.only!r} matches no scenario in {suite_dir}") diff --git a/installer/grdm_dev/host.py b/installer/grdm_dev/host.py index b10443c..9f0531c 100644 --- a/installer/grdm_dev/host.py +++ b/installer/grdm_dev/host.py @@ -46,7 +46,20 @@ SCANNER_TOOLS = ("trivy", "gitleaks", "semgrep") WITHOUT_SCAN_ENV = "GRDM_WITHOUT_SCAN" # install.sh's opt-out of the [scan] extra: semgrep is then absent on purpose BINFMT_AMD64_FILES = ("/proc/sys/fs/binfmt_misc/qemu-x86_64", "/proc/sys/fs/binfmt_misc/rosetta") -ENABLE_AMD64_EMULATION = "docker run --privileged --rm tonistiigi/binfmt --install amd64" +# The QEMU registration image, pinned by digest (tag qemu-v10.2.3-68). Kept in step with BINFMT_IMAGE in +# installer/install-ubuntu.sh; a test compares the two. +BINFMT_IMAGE = "tonistiigi/binfmt:qemu-v10.2.3-68@sha256:400a4873b838d1b89194d982c45e5fb3cda4593fbfd7e08a02e76b03b21166f0" +ENABLE_AMD64_EMULATION = f"docker run --privileged --rm {BINFMT_IMAGE} --install amd64" +# Replacing a registered handler: the kernel refuses a second entry with the same name, so remove it first. +REPLACE_AMD64_EMULATION = (f"docker run --privileged --rm {BINFMT_IMAGE} --uninstall qemu-x86_64 && " + f"docker run --privileged --rm {BINFMT_IMAGE} --install amd64") +# Emulator builds measured to crash while building a core image, matched against ` --version`. +# Ubuntu 24.04's qemu-user-static 1:8.2.2+ds-0ubuntu1.18 stops modular-file-renderer's build with +# "QEMU internal SIGSEGV" in the libgvc6 post-install script, while tonistiigi/binfmt's 8.1.5, 9.2.2 and +# 10.2.3 install the same package (DEMO-21 run 1). Only the measured build is listed; the version alone is not the cause. +KNOWN_BAD_QEMU = ( + (re.compile(r"qemu-x86_64 version 8\.2\.2 \(Debian 1:8\.2\.2\+ds-0ubuntu"), "Ubuntu 24.04 qemu-user-static 8.2.2"), +) def default_runner(argv: list[str]) -> tuple[int, str]: @@ -82,6 +95,7 @@ class Probe: tools: Optional[dict[str, bool]] = None # name -> found, for REQUIRED_TOOLS + SCANNER_TOOLS; None = not probed repo_root: str = "" # resolved repository root; "" = not probed binfmt_amd64: Optional[bool] = None # Linux: an enabled amd64 handler in binfmt_misc; None = unknown + binfmt_amd64_qemu: str = "" # Linux: first line of the registered qemu-x86_64 interpreter's --version; "" = unknown repo_modes_enforced: Optional[bool] = None # WSL2: the checkout's filesystem keeps chmod 0600; None = unknown @classmethod @@ -103,6 +117,7 @@ def from_system(cls, root: Path | None = None, runner: Runner | None = None) -> tools=_probe_tools(), repo_root=str(root.resolve()), binfmt_amd64=_binfmt_amd64() if system == "Linux" else None, + binfmt_amd64_qemu=_binfmt_amd64_qemu(runner or default_runner) if system == "Linux" else "", ) if classify_os(p)[0] == "wsl2": p.repo_modes_enforced = _modes_enforced(root) @@ -167,6 +182,33 @@ def _binfmt_amd64() -> Optional[bool]: return any(_read_text(path).splitlines()[:1] == ["enabled"] for path in BINFMT_AMD64_FILES) +def _binfmt_amd64_qemu(runner: Runner, entry: str = BINFMT_AMD64_FILES[0]) -> str: + """The `--version` line of the registered qemu-x86_64 interpreter when that file is on this host, else "". + A handler registered from a container with the F flag (tonistiigi/binfmt) has no host file: unknown, not bad. + The path is resolved first: Ubuntu's x86_64-binfmt-P refuses `--version` under its own name.""" + interpreter = next((line.split(None, 1)[1].strip() for line in _read_text(entry).splitlines() + if line.startswith("interpreter ") and len(line.split(None, 1)) == 2), "") + if not interpreter: + return "" + path = Path(interpreter) + try: + path = path.resolve(strict=True) + except (OSError, RuntimeError): + return "" + if not path.is_file(): + return "" + rc, out = runner([str(path), "--version"]) + return (out or "").strip().splitlines()[0] if rc == 0 and (out or "").strip() else "" + + +def known_bad_qemu(version_line: str) -> Optional[str]: + """The name of a measured-bad emulator build matching `version_line`, else None.""" + for pattern, name in KNOWN_BAD_QEMU: + if pattern.search(version_line or ""): + return name + return None + + def _modes_enforced(root: Path) -> Optional[bool]: """Does this filesystem keep a chmod? The fact behind `runtime/.env` at 0600, wherever it is mounted: a Windows drive without the `metadata` option reports 0777. None when no file can be created to find out.""" @@ -360,6 +402,13 @@ def check_amd64_emulation(probe: Probe, info: HostInfo, services: list[str]) -> f"amd64 emulation is not enabled but {names} need it; run this (and again after a reboot): {ENABLE_AMD64_EMULATION} / " f"amd64 エミュレーションが無効ですが {names} に必要です。次を実行してください（再起動のたびに必要）: {ENABLE_AMD64_EMULATION}" ) + elif bad := known_bad_qemu(probe.binfmt_amd64_qemu): + info.prerequisite_problems.append( + f"the registered amd64 emulator is {bad} ({probe.binfmt_amd64_qemu}), which crashes with `QEMU internal SIGSEGV` while building {names}; " + f"replace it (and again after a reboot, while qemu-user-static is installed): {REPLACE_AMD64_EMULATION} / " + f"登録されている amd64 エミュレーターは {bad}（{probe.binfmt_amd64_qemu}）で、{names} のビルド中に `QEMU internal SIGSEGV` で落ちます。" + f"次で置き換えてください（qemu-user-static が入っている間は再起動のたびに必要）: {REPLACE_AMD64_EMULATION}" + ) # --- entry points ------------------------------------------------------------------------------ diff --git a/installer/grdm_dev/lifecycle.py b/installer/grdm_dev/lifecycle.py index e7d97f7..9be7916 100644 --- a/installer/grdm_dev/lifecycle.py +++ b/installer/grdm_dev/lifecycle.py @@ -4,7 +4,8 @@ runtime/volumes/postgres_data written by another PostgreSQL major than the rendered image, or one whose major cannot be read (database.recover_pending, database.preflight, ADR-12); then `docker compose up -d`, then wait for health (exit 40 on timeout), then open the browser at http://127.0.0.1: unless --no-open; the browser is only opened when stdin is a - terminal. Idempotent: healthy services are left alone by Compose. + terminal. Idempotent: healthy services are left alone by Compose. Without --timeout the wait is + DEFAULT_TIMEOUT, or QEMU_TIMEOUT on Linux arm64 when the profile has amd64 services (default_timeout). down `docker compose down` — containers stop, volumes stay. reset down, then delete runtime/volumes/ after ctx.confirm(). Non-interactive without --yes → exit 2. On Linux the files belong to the containers' uids, so the host user cannot remove them: the contents @@ -19,15 +20,38 @@ from __future__ import annotations import json +import platform import shutil import sys import webbrowser from pathlib import Path -from . import compose, database, exit_codes, health, report +from . import compose, database, exit_codes, health, manifest as _manifest, plan, report from .context import Context DEFAULT_TIMEOUT = 1800.0 # cold start: ember's first build under amd64 emulation took 13-25 min and osf_init runs migrations (real install runs 10-18; PR #33 review r1) +# Linux arm64 emulates amd64 with QEMU, which is slower than Rosetta: ember's first build took 44 min there, on one +# core, so more CPUs do not help, and 1800 s stopped an otherwise healthy install with exit 40 (DEMO-21 run 2). +# About twice the measurement. +QEMU_TIMEOUT = 5400.0 +_ARM64 = ("aarch64", "arm64") + + +def default_timeout(ctx: Context, system: str | None = None, machine: str | None = None) -> float: + """The health wait when --timeout is not given: QEMU_TIMEOUT on Linux arm64 when a service of the profile runs as + linux/amd64 (manifest build_platform, as doctor counts them), otherwise DEFAULT_TIMEOUT. On macOS Docker Desktop + reports Darwin here and emulates with Rosetta, so it keeps the default. An unreadable manifest keeps the default.""" + system = system if system is not None else platform.system() + machine = (machine if machine is not None else platform.machine()).lower() + if system != "Linux" or machine not in _ARM64: + return DEFAULT_TIMEOUT + try: + man = _manifest.load(ctx) + services = _manifest.services_for_profile(man, plan.resolve_profile(ctx)) + except (_manifest.ManifestError, ValueError, OSError): + return DEFAULT_TIMEOUT + emulated = [s for s in services if "amd64" in str(man.component(s.component).get("build_platform") or "")] + return QEMU_TIMEOUT if emulated else DEFAULT_TIMEOUT def default_runner(argv: list[str]) -> tuple[int, str]: @@ -68,7 +92,8 @@ def recreate(ctx: Context, runner=None) -> int: def register(subparsers) -> None: up = subparsers.add_parser("up", help="start the profile and wait for health; exit 40 on failure") up.add_argument("--no-open", dest="no_open", action="store_true", help="do not open the browser") - up.add_argument("--timeout", type=float, default=DEFAULT_TIMEOUT, help=f"seconds to wait for health (default {int(DEFAULT_TIMEOUT)})") + up.add_argument("--timeout", type=float, default=None, + help=f"seconds to wait for health (default {int(DEFAULT_TIMEOUT)}; {int(QEMU_TIMEOUT)} on Linux arm64 with amd64 services)") up.set_defaults(_module=sys.modules[__name__], lifecycle_action="up") down = subparsers.add_parser("down", help="stop containers, keep volumes") down.set_defaults(_module=sys.modules[__name__], lifecycle_action="down") @@ -79,7 +104,9 @@ def register(subparsers) -> None: def run(ctx: Context, args, runner=None, prober=None, opener=None, sleep=None, rmtree=None) -> int: action = getattr(args, "lifecycle_action", None) if action == "up": - return run_up(ctx, no_open=bool(getattr(args, "no_open", False)), timeout=float(getattr(args, "timeout", DEFAULT_TIMEOUT)), + given = getattr(args, "timeout", None) + timeout = float(given) if given is not None else default_timeout(ctx) + return run_up(ctx, no_open=bool(getattr(args, "no_open", False)), timeout=timeout, runner=runner, prober=prober, opener=opener, sleep=sleep) if action == "down": return run_down(ctx, runner=runner) diff --git a/installer/grdm_dev/scan.py b/installer/grdm_dev/scan.py index 8a97b58..4a3f80c 100644 --- a/installer/grdm_dev/scan.py +++ b/installer/grdm_dev/scan.py @@ -33,11 +33,15 @@ from __future__ import annotations import datetime as _dt +import errno import json +import os import platform import re import shutil +import stat import subprocess +import tempfile import sys from pathlib import Path from typing import Any, Callable @@ -618,11 +622,32 @@ def mask_secrets(data: Any, secrets: list[str]) -> Any: # --------------------------------------------------------------------------- scanners +# Opening a FIFO blocks until the other end appears, and a symlink is followed, both before any type check could +# run. So report files are opened without following links and without blocking, and the descriptor's type is +# checked before a byte is read or written (PR #44 review r2 R2-02). +_SAFE_OPEN = getattr(os, "O_NOFOLLOW", 0) | getattr(os, "O_NONBLOCK", 0) + + +def _regular_fd(fd: int, path: Path) -> int: + if not stat.S_ISREG(os.fstat(fd).st_mode): + os.close(fd) + raise OSError(errno.EINVAL, "not a regular file", str(path)) + return fd + + +def _write_report_file(path: Path, text: str) -> None: + """Write `text` to `path` only if it is, or becomes, a regular file; a FIFO, symlink or device raises OSError.""" + fd = _regular_fd(os.open(path, os.O_WRONLY | os.O_CREAT | _SAFE_OPEN, 0o666), path) + with os.fdopen(fd, "w", encoding="utf-8") as fh: + fh.truncate(0) + fh.write(text) + + def _save_raw(ctx: Context, name: str, text: str, secrets: list[str]) -> Path: """Save a scanner's raw output under runtime/reports/ with runtime/.env values masked.""" ctx.reports_dir.mkdir(parents=True, exist_ok=True) path = ctx.reports_dir / name - path.write_text(mask_secrets(text, secrets), encoding="utf-8") + _write_report_file(path, mask_secrets(text, secrets)) return path @@ -892,6 +917,47 @@ def services_healthy(ctx: Context, runner: Runner, which: Callable[[str], bool], return True +def private_reports_problem(ctx: Context) -> str | None: + """Why the ZAP work directories may not be made writable for every uid, or None. + + A 0777 work directory is safe only below an ancestor that no other host user can enter. That ancestor is + runtime/: it must be a real directory (not a symlink), owned by this user, with no group or other bits; and + reports/ below it must be a real directory owned by this user. `secrets init` creates runtime/ this way, but a + copied or older runtime/ may be 0755, so the invariant is checked, not assumed (PR #44 review r1 R3).""" + uid = os.getuid() + for path, private in ((ctx.runtime, True), (ctx.reports_dir, False)): + try: + st = os.lstat(path) + except OSError as exc: + return f"{path} cannot be inspected ({exc.strerror})" + if stat.S_ISLNK(st.st_mode): + return f"{path} is a symbolic link" + if not stat.S_ISDIR(st.st_mode): + return f"{path} is not a directory" + if st.st_uid != uid: + return f"{path} belongs to uid {st.st_uid}, not to this user (uid {uid})" + if private and stat.S_IMODE(st.st_mode) & 0o077: + return f"{path} has mode {oct(stat.S_IMODE(st.st_mode))}; it must be 0700 (chmod 700 {path})" + return None + + +def _read_zap_report(path: Path) -> tuple[Any, str | None]: + """The parsed report, refusing anything but a regular file (a symlink planted in the work directory is not + followed). (None, None) when there is no file.""" + try: + fd = os.open(path, os.O_RDONLY | _SAFE_OPEN) + except FileNotFoundError: + return None, None + except OSError as exc: + return None, f"ZAP report {path.name} could not be opened safely ({exc.strerror}); a symlink is refused" + try: + fd = _regular_fd(fd, path) + except OSError: + return None, f"ZAP report {path.name} is not a regular file (a FIFO, device or socket is refused)" + with os.fdopen(fd, "r", encoding="utf-8") as fh: + return _json_or_empty(fh.read()), None + + def run_zap(ctx: Context, runner: Runner, which: Callable[[str], bool], ts: str, secrets: list[str] | None = None) -> tuple[list[dict], dict, list[str]]: _require(which, "docker") @@ -904,23 +970,44 @@ def run_zap(ctx: Context, runner: Runner, which: Callable[[str], bool], ts: str, ctx.reports_dir.mkdir(parents=True, exist_ok=True) findings: list[dict] = [] problems: list[str] = [] + unsafe = private_reports_problem(ctx) + if unsafe: # fail closed: no world-writable directory, no ZAP run, the dynamic part is incomplete (exit 51) + problems.append(f"ZAP not run: {unsafe}; its work directories must sit below a private runtime/") + return [], _tool("zap", "stable", len(targets), problems), notes for svc, port in targets.items(): url = f"http://{svc}:{port}" # service name, never 127.0.0.1 (SPEC-03 §4.3) out_name = f"zap_{svc}_{ts}.json" - argv = ["docker", "run", "--rm", "--network", network, "-v", f"{ctx.reports_dir}:/zap/wrk:rw", + # ZAP writes as its own uid (1000, "zap"). On Linux a host directory owned by another uid is not writable + # for it, so zap-baseline exited 0 without a report (DEMO-21 run 3, host uid 2001). Each target gets its + # own work directory, writable by any uid: created exclusively under an unpredictable name, below the + # runtime/ whose 0700 mode private_reports_problem() has just verified, holding only this report, and + # removed once the masked copy is saved in reports/. + wrk = Path(tempfile.mkdtemp(prefix=f"zap_wrk_{svc}_{ts}_", dir=ctx.reports_dir)) + os.chmod(wrk, 0o777) # nosemgrep: python.lang.security.audit.insecure-file-permissions.insecure-file-permissions -- exclusive, unpredictable ZAP work directory below a runtime/ verified as 0700 and owned by this user; removed in `finally` + argv = ["docker", "run", "--rm", "--network", network, "-v", f"{wrk}:/zap/wrk:rw", ZAP_IMAGE, "zap-baseline.py", "-t", url, "-J", out_name, "-I"] - rc, out = runner(argv) - if rc == 127: - raise ScannerMissing("docker", "docker not found") - out_path = ctx.reports_dir / out_name - data = _json_or_empty(out_path.read_text(encoding="utf-8")) if out_path.exists() else _json_or_empty(out) + try: + rc, out = runner(argv) + if rc == 127: + raise ScannerMissing("docker", "docker not found") + data, refused = _read_zap_report(wrk / out_name) + if refused: + problems.append(refused) + continue + if data is None: + data = _json_or_empty(out) + finally: + shutil.rmtree(wrk, ignore_errors=True) if data is None: problems.append(f"no ZAP report for {url} (rc {rc}); image {ZAP_IMAGE} pull or the scan may have failed") continue if rc not in ZAP_OK_CODES: problems.append(f"zap-baseline exited {rc} for {url}") - if out_path.exists(): + try: _save_raw(ctx, out_name, json.dumps(data, ensure_ascii=False, indent=2), secrets or []) + except OSError as exc: + problems.append(f"ZAP report {out_name} could not be saved in reports/ ({exc.strerror}); only a regular file is written") + continue findings.extend(normalize_zap(data, url)) return findings, _tool("zap", "stable", len(targets), problems), notes diff --git a/installer/install-macos.sh b/installer/install-macos.sh new file mode 100755 index 0000000..4ca5a08 --- /dev/null +++ b/installer/install-macos.sh @@ -0,0 +1,126 @@ +#!/usr/bin/env bash +# Prerequisites for macOS on Apple Silicon, then installer/install.sh (ADR-08, SPEC-02 §2.3). +# It only prepares the host: every installation decision stays in install.sh and grdm-dev. +# +# What it may change, after showing it and asking: Homebrew formulae (python@3.12, git, trivy, gitleaks; each +# bottle is checked by Homebrew against the checksum in its formula) and starting an already installed Docker +# Desktop. It does not install Homebrew or a Docker runtime: Homebrew's own one-line installer pipes a +# download into a shell, which ADR-08 rejects, and Docker Desktop's licence is the user's decision. +# Runs on the bash that ships with macOS (3.2). +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=installer/prereq-lib.sh +. "${SCRIPT_DIR}/prereq-lib.sh" +PREREQ_NAME="install-macos.sh" +DOCKER_WAIT_SECONDS=120 +APPLICATIONS="${GRDM_PREREQ_APPLICATIONS:-/Applications}" + +prereq_usage() { + cat <<'EOF' +Usage: ./installer/install-macos.sh [--yes] [--dry-run] [grdm-dev install options...] + + Prepares macOS on Apple Silicon, then runs installer/install.sh. + --yes consent to every change listed in the plan + --dry-run detect and print what would happen; change nothing + Other options go to install.sh unchanged, for example --profile minimum. + +使い方: ./installer/install-macos.sh [--yes] [--dry-run] [grdm-dev install のオプション...] + + Apple Silicon の macOS を準備し、installer/install.sh を実行します。 + --yes 計画に載せたすべての変更に同意する + --dry-run 判定して予定を表示するだけで、何も変更しない + ほかのオプションはそのまま install.sh に渡します。例: --profile minimum +EOF +} + +prereq_parse_args "$@" + +# --- host check (read-only) ------------------------------------------------------------------------ +if [ "$(uname -s)" != "Darwin" ]; then + prereq_fail 10 "this script is for macOS; on Ubuntu use installer/install-ubuntu.sh / このスクリプトは macOS 用です。Ubuntu では installer/install-ubuntu.sh を使ってください" +fi +if [ "$(uname -m)" != "arm64" ]; then + prereq_say "Warning: Intel Macs are best-effort only (SPEC-02 §8)." "警告: Intel の Mac はできる範囲での対応です（SPEC-02 §8）。" +fi + +# --- Homebrew -------------------------------------------------------------------------------------- +BREW="" +for candidate in ${GRDM_PREREQ_BREW_CANDIDATES-/opt/homebrew/bin/brew /usr/local/bin/brew}; do + if [ -x "${candidate}" ]; then BREW="${candidate}"; break; fi +done +if [ -z "${BREW}" ] && command -v brew >/dev/null 2>&1; then BREW="$(command -v brew)"; fi +if [ -n "${BREW}" ]; then + PATH="$(dirname "${BREW}"):${PATH}" + export PATH +else + prereq_stop 10 "Homebrew is required and was not found. Install it from the signed .pkg on its official releases page, then run this script again: https://github.com/Homebrew/brew/releases/latest" \ + "Homebrew が必要ですが見つかりません。公式のリリースページにある署名付き .pkg から入れて、もう一度実行してください: https://github.com/Homebrew/brew/releases/latest" +fi + +# --- detection (read-only) ------------------------------------------------------------------------- +FORMULAE="" +if ! prereq_python_ok; then FORMULAE="${FORMULAE} python@3.12"; fi +if ! git --version >/dev/null 2>&1; then FORMULAE="${FORMULAE} git"; fi +if ! command -v trivy >/dev/null 2>&1; then FORMULAE="${FORMULAE} trivy"; fi +if ! command -v gitleaks >/dev/null 2>&1; then FORMULAE="${FORMULAE} gitleaks"; fi +FORMULAE="${FORMULAE# }" + +docker_up() { docker info >/dev/null 2>&1; } + +START_DESKTOP=0 +if ! command -v docker >/dev/null 2>&1 || ! docker_up; then + if [ -d "${APPLICATIONS}/Docker.app" ]; then + START_DESKTOP=1 + elif [ -d "${APPLICATIONS}/OrbStack.app" ]; then + prereq_stop 11 "OrbStack is installed but its Docker engine is not answering. Start OrbStack, then run this script again." \ + "OrbStack は入っていますが Docker エンジンが応答しません。OrbStack を起動して、もう一度実行してください。" + elif command -v colima >/dev/null 2>&1; then + prereq_stop 11 "Colima is installed but its Docker engine is not answering. Run 'colima start', then run this script again." \ + "Colima は入っていますが Docker エンジンが応答しません。'colima start' を実行して、もう一度実行してください。" + else + prereq_stop 11 "No Docker runtime was found. Install Docker Desktop from https://www.docker.com/products/docker-desktop/ (check that its licence fits your organisation), then run this script again." \ + "Docker の実行基盤が見つかりません。https://www.docker.com/products/docker-desktop/ から Docker Desktop を入れ（ライセンスが所属組織に合うか確認してください）、もう一度実行してください。" + fi +fi + +# --- plan and consent ------------------------------------------------------------------------------ +if [ -n "${FORMULAE}" ]; then + prereq_plan "brew install ${FORMULAE}" "brew install ${FORMULAE} を実行する" +fi +if [ "${START_DESKTOP}" = "1" ]; then + prereq_plan "start Docker Desktop and wait up to ${DOCKER_WAIT_SECONDS} seconds" "Docker Desktop を起動し、最長 ${DOCKER_WAIT_SECONDS} 秒待つ" +fi +prereq_show_plan +prereq_token_note +if [ "${#PREREQ_PLAN_EN[@]}" -gt 0 ]; then + prereq_confirm "Make these changes?" "これらの変更を行いますか?" +fi + +# --- changes --------------------------------------------------------------------------------------- +if [ -n "${FORMULAE}" ]; then + # shellcheck disable=SC2086 # FORMULAE is a list of formula names + prereq_step 10 "install Homebrew formulae: ${FORMULAE}" brew install ${FORMULAE} +fi + +if [ "${START_DESKTOP}" = "1" ]; then + prereq_step 11 "start Docker Desktop" open -a Docker + if [ "${PREREQ_DRY_RUN}" = "0" ]; then + waited=0 + until docker_up; do + if [ "${waited}" -ge "${DOCKER_WAIT_SECONDS}" ]; then + prereq_fail 11 "Docker Desktop did not answer within ${DOCKER_WAIT_SECONDS} seconds; finish its first-start steps and run this script again" \ + "Docker Desktop が ${DOCKER_WAIT_SECONDS} 秒以内に応答しませんでした。初回起動の手順を済ませて、もう一度実行してください" + fi + sleep 5 + waited=$((waited + 5)) + done + fi +fi + +if [ "${PREREQ_DRY_RUN}" = "0" ] && { ! prereq_docker_client_ok || ! prereq_docker_plugins_ok; }; then + prereq_fail 11 "Docker 27 or newer with the Compose and buildx plugins is required; update your Docker runtime" \ + "Docker 27 以上と Compose・buildx プラグインが必要です。Docker の実行基盤を更新してください" +fi + +prereq_handover diff --git a/installer/install-ubuntu.sh b/installer/install-ubuntu.sh new file mode 100755 index 0000000..a0f6677 --- /dev/null +++ b/installer/install-ubuntu.sh @@ -0,0 +1,308 @@ +#!/usr/bin/env bash +# Prerequisites for Ubuntu 24.04 (amd64, arm64) and for Ubuntu 24.04 inside WSL2, then installer/install.sh +# (ADR-08, SPEC-02 §2.3). It only prepares the host: every installation decision stays in install.sh and grdm-dev. +# +# What it may change, after showing it and asking: apt packages (git, python3, python3-venv, curl, gnupg, +# ca-certificates, and libatomic1 with GRDM_WITH_ACCEPT=1), Docker Engine with the Compose and buildx plugins +# from Docker's signed apt repository, the docker group, amd64 emulation on arm64 (QEMU registered by the pinned +# BINFMT_IMAGE, not Ubuntu's qemu-user-static, whose 8.2.2 build crashed while building modular-file-renderer: +# DEMO-21 run 1), and trivy and gitleaks in ~/.local/bin. +# Inside WSL2 Docker comes from Docker Desktop, so the Docker Engine steps are skipped. +# +# Trust: the apt signing key is accepted only when its fingerprint equals DOCKER_SIGNER_FPR, the scanner +# tarballs only when their SHA-256 equals the values below (copied from each release's checksums file), and +# the emulator image only by its digest. +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=installer/prereq-lib.sh +. "${SCRIPT_DIR}/prereq-lib.sh" +PREREQ_NAME="install-ubuntu.sh" + +DOCKER_SIGNER_FPR="9DC858229FC7DD38854AE2D88D81803C0EBFCD88" +DOCKER_APT_URL="https://download.docker.com/linux/ubuntu" +DOCKER_APT_PGP="/etc/apt/keyrings/docker.asc" +DOCKER_APT_SOURCES="/etc/apt/sources.list.d/docker.sources" +DOCKER_PACKAGES="docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin" +DOCKER_CONFLICTS="docker.io docker-doc docker-compose docker-compose-v2 podman-docker containerd runc" +TRIVY_VERSION="0.75.0" +GITLEAKS_VERSION="8.21.2" +LOCAL_BIN="${HOME}/.local/bin" +# Same value as BINFMT_IMAGE in installer/grdm_dev/host.py (a test compares them). Tag qemu-v10.2.3-68. +BINFMT_IMAGE="tonistiigi/binfmt:qemu-v10.2.3-68@sha256:400a4873b838d1b89194d982c45e5fb3cda4593fbfd7e08a02e76b03b21166f0" +# Emulator builds measured to crash; same pattern as KNOWN_BAD_QEMU in host.py. +KNOWN_BAD_QEMU_RE='qemu-x86_64 version 8\.2\.2 \(Debian 1:8\.2\.2\+ds-0ubuntu' + +prereq_usage() { + cat <<'EOF' +Usage: ./installer/install-ubuntu.sh [--yes] [--dry-run] [grdm-dev install options...] + + Prepares Ubuntu 24.04 (amd64 or arm64), then runs installer/install.sh. + --yes consent to every change listed in the plan, including sudo steps and the docker group + --dry-run detect and print what would happen; change nothing + Other options go to install.sh unchanged, for example --profile minimum. + +使い方: ./installer/install-ubuntu.sh [--yes] [--dry-run] [grdm-dev install のオプション...] + + Ubuntu 24.04（amd64 または arm64）を準備し、installer/install.sh を実行します。 + --yes 計画に載せたすべての変更（sudo の段と docker グループを含む）に同意する + --dry-run 判定して予定を表示するだけで、何も変更しない + ほかのオプションはそのまま install.sh に渡します。例: --profile minimum +EOF +} + +prereq_parse_args "$@" + +# --- host check (read-only) ------------------------------------------------------------------------ +if [ "$(uname -s)" != "Linux" ]; then + prereq_fail 10 "this script is for Ubuntu 24.04; on macOS use installer/install-macos.sh / このスクリプトは Ubuntu 24.04 用です。macOS では installer/install-macos.sh を使ってください" +fi +OS_RELEASE="${GRDM_PREREQ_OS_RELEASE:-/etc/os-release}" +os_value() { awk -F= -v key="$1" '$1 == key { gsub(/"/, "", $2); print $2; exit }' "${OS_RELEASE}" 2>/dev/null || true; } +if [ "$(os_value ID)" != "ubuntu" ] || [ "$(os_value VERSION_ID)" != "24.04" ]; then + prereq_fail 10 "only Ubuntu 24.04 is supported here; on other Linux prepare the host yourself and run installer/install.sh / ここで対応するのは Ubuntu 24.04 だけです。他の Linux では自分で準備して installer/install.sh を実行してください" +fi +case "$(uname -m)" in + x86_64) ARCH="amd64"; TRIVY_ASSET="Linux-64bit"; GITLEAKS_ASSET="linux_x64" + TRIVY_SUM="c6e65abddb348e25f10549df887045629cf28cc72453cd1c63acb717316b3f3f" + GITLEAKS_SUM="5bc41815076e6ed6ef8fbecc9d9b75bcae31f39029ceb55da08086315316e3ba" ;; + aarch64|arm64) ARCH="arm64"; TRIVY_ASSET="Linux-ARM64"; GITLEAKS_ASSET="linux_arm64" + TRIVY_SUM="a1ee9f6ffb7d112b64ff726a2a0717c21175c1114361391f4a132956751a13b3" + GITLEAKS_SUM="654c935542c89f565aabe7bf7c6c500830f116c114f0aeb509d2460c1ac2e6da" ;; + *) prereq_fail 10 "unsupported CPU: $(uname -m) / 対応していない CPU です: $(uname -m)" ;; +esac + +IS_WSL=0 +if [ -n "${WSL_DISTRO_NAME:-}" ] || grep -qi microsoft "${GRDM_PREREQ_PROC_VERSION:-/proc/version}" 2>/dev/null; then IS_WSL=1; fi +# PREREQ_ROOT is the physical path, so a checkout reached through a symlink into /mnt/c is refused too (PR #44 +# review r1 R2). GRDM_PREREQ_WSL_WINDOWS_ROOT exists for the unit tests, which cannot create /mnt. +WSL_WINDOWS_ROOT="${GRDM_PREREQ_WSL_WINDOWS_ROOT:-/mnt}" +WSL_WINDOWS_ROOT="$(cd "${WSL_WINDOWS_ROOT}" 2>/dev/null && pwd -P || printf '%s' "${WSL_WINDOWS_ROOT}")" +if [ "${IS_WSL}" = "1" ]; then + case "${PREREQ_ROOT}/" in + "${WSL_WINDOWS_ROOT}"/*) prereq_fail 10 "this checkout is on the Windows disk (${PREREQ_ROOT}); clone it inside WSL, for example under ~/ / このチェックアウトは Windows のディスク上にあります（${PREREQ_ROOT}）。~/ など WSL の中に clone してください" ;; + esac +fi + +# A virtual machine must not see the host's credentials. Lima's Ubuntu template mounts the host home directory +# read-only even when the instance lists its own mounts (the template's list is merged in), which showed +# ~/.config/gh to the VM in DEMO-21. Only existence is tested; no file is read. The check is deliberately narrow: it +# looks for the four names below directly under each shared mount point (a shared home directory), not for every +# credential anywhere below it. +SHARED_FS_RE='^(virtiofs|9p|fuse\.sshfs|vboxsf|fuse\.vmhgfs-fuse|fuse\.virtiofsd)$' +CREDENTIAL_FILES=".config/gh/hosts.yml .git-credentials .netrc .docker/config.json" +decode_mount_field() { # sets DECODED_MOUNT; the kernel writes space, tab, newline and backslash as \040 \011 \012 \134 + # A variable, not $(...): command substitution strips trailing newlines, so "host\012" would become "host" + # (PR #44 review r2 R2-01). + DECODED_MOUNT="$1" + DECODED_MOUNT="${DECODED_MOUNT//\\040/ }" + DECODED_MOUNT="${DECODED_MOUNT//\\011/$'\t'}" + DECODED_MOUNT="${DECODED_MOUNT//\\012/$'\n'}" + DECODED_MOUNT="${DECODED_MOUNT//\\134/\\}" # last, so a decoded backslash cannot start another escape (review r1 R1) +} +EXPOSED="" +while read -r _src raw_mnt fstype _rest; do + if printf '%s\n' "${fstype}" | grep -Eq "${SHARED_FS_RE}"; then + decode_mount_field "${raw_mnt}" + mnt="${DECODED_MOUNT}" + for c in ${CREDENTIAL_FILES}; do + if [ -e "${mnt}/${c}" ]; then EXPOSED="${EXPOSED} '${mnt}/${c}'"; fi + done + fi +done < "${GRDM_PREREQ_MOUNTS:-/proc/mounts}" +if [ -n "${EXPOSED}" ]; then + prereq_stop 10 "host credentials are visible in this machine through a shared folder:${EXPOSED}" \ + "Remove that mount, then run this script again. Lima: delete the \"~\" entry under mounts: in ~/.lima//lima.yaml, then limactl stop && limactl start ." \ + "共有フォルダを通じてホストの資格情報がこの機械から見えます。その mount を外してから、もう一度実行してください。Lima では ~/.lima//lima.yaml の mounts: から \"~\" の項目を消し、limactl stop && limactl start を実行します。" +fi + +# --- detection (read-only) ------------------------------------------------------------------------- +pkg_installed() { [ "$(dpkg-query -W -f='${Status}' "$1" 2>/dev/null || true)" = "install ok installed" ]; } + +APT_PACKAGES="git python3 python3-venv curl gnupg ca-certificates" +# GRDM_WITH_ACCEPT=1 (install.sh's [accept] extra): Playwright's Node.js driver needs libatomic1, which the Ubuntu 24.04 +# cloud image lacks (DEMO-21). Chromium's own libraries come later from `playwright install-deps`. +if [ "${GRDM_WITH_ACCEPT:-0}" = "1" ]; then APT_PACKAGES="${APT_PACKAGES} libatomic1"; fi +APT_MISSING="" +for pkg in ${APT_PACKAGES}; do + if ! pkg_installed "${pkg}"; then APT_MISSING="${APT_MISSING} ${pkg}"; fi +done +APT_MISSING="${APT_MISSING# }" + +NEED_ENGINE=0 +if ! command -v docker >/dev/null 2>&1 || ! prereq_docker_client_ok || ! prereq_docker_plugins_ok; then + if [ "${IS_WSL}" = "1" ]; then + prereq_stop 11 "Docker is not usable inside this WSL distribution. Install Docker Desktop on Windows and enable Settings > Resources > WSL Integration for this distribution." \ + "この WSL ディストリビューションで Docker を使えません。Windows に Docker Desktop を入れ、Settings > Resources > WSL Integration でこのディストリビューションを有効にしてください。" + else + NEED_ENGINE=1 + fi +fi +if [ "${NEED_ENGINE}" = "1" ]; then + CONFLICTS="" + for pkg in ${DOCKER_CONFLICTS}; do + if pkg_installed "${pkg}"; then CONFLICTS="${CONFLICTS} ${pkg}"; fi + done + if [ -n "${CONFLICTS}" ]; then + prereq_stop 11 "these packages conflict with Docker's official packages:${CONFLICTS}" \ + "Remove them yourself (your images and volumes stay in /var/lib/docker), then run this script again: sudo apt-get remove${CONFLICTS}" \ + "これらは Docker 公式のパッケージと衝突します。自分で削除して（イメージとボリュームは /var/lib/docker に残ります）、もう一度実行してください。" + fi +fi + +qemu_version_line() { # the registered interpreter's `--version` line when it is a file on this host, else nothing + interp="$(awk '$1 == "interpreter" { print $2; exit }' "${binfmt}" 2>/dev/null || true)" + if [ -z "${interp}" ]; then return 0; fi + real="$(readlink -f "${interp}" 2>/dev/null || true)" # Ubuntu's x86_64-binfmt-P refuses --version under its own name + if [ -z "${real}" ] || [ ! -f "${real}" ]; then return 0; fi + "${real}" --version 2>/dev/null | head -n 1 || true +} +emulation_ok() { # enabled, with the F flag (usable from inside a container), and not a build measured to crash + grep -q '^enabled' "${binfmt}" 2>/dev/null && grep -q '^flags:.*F' "${binfmt}" 2>/dev/null \ + && ! qemu_version_line | grep -Eq "${KNOWN_BAD_QEMU_RE}" +} +NEED_EMULATION=0 +BAD_EMULATOR="" +binfmt="${GRDM_PREREQ_BINFMT_DIR:-/proc/sys/fs/binfmt_misc}/qemu-x86_64" +if [ "${ARCH}" = "arm64" ] && [ "${IS_WSL}" = "0" ]; then + if ! emulation_ok; then NEED_EMULATION=1; fi + if qemu_version_line | grep -Eq "${KNOWN_BAD_QEMU_RE}"; then BAD_EMULATOR="$(qemu_version_line)"; fi +fi + +export PATH="${LOCAL_BIN}:${PATH}" +NEED_TRIVY=0; command -v trivy >/dev/null 2>&1 || NEED_TRIVY=1 +NEED_GITLEAKS=0; command -v gitleaks >/dev/null 2>&1 || NEED_GITLEAKS=1 + +# --- plan and consent ------------------------------------------------------------------------------ +if [ -n "${APT_MISSING}" ]; then + prereq_plan "sudo: install apt packages: ${APT_MISSING}" "sudo: apt パッケージを入れる: ${APT_MISSING}" +fi +if [ "${NEED_ENGINE}" = "1" ]; then + prereq_plan "sudo: add Docker's signed apt repository and install ${DOCKER_PACKAGES}, then start the docker service" \ + "sudo: Docker の署名付き apt リポジトリを追加し、${DOCKER_PACKAGES} を入れて docker サービスを起動する" +fi +if [ -n "${BAD_EMULATOR}" ]; then + prereq_plan "sudo: replace the registered amd64 emulator (${BAD_EMULATOR}), which crashes while building modular-file-renderer, with QEMU from ${BINFMT_IMAGE}" \ + "sudo: 登録済みの amd64 エミュレーター（${BAD_EMULATOR}）は modular-file-renderer のビルド中に落ちるため、${BINFMT_IMAGE} の QEMU に置き換える" +elif [ "${NEED_EMULATION}" = "1" ]; then + prereq_plan "sudo: register QEMU from ${BINFMT_IMAGE} so that amd64 images can run on this arm64 host (until the next reboot)" \ + "sudo: この arm64 ホストで amd64 イメージを動かすため ${BINFMT_IMAGE} の QEMU を登録する（次の再起動まで有効）" +fi +if [ "${NEED_TRIVY}" = "1" ]; then + prereq_plan "download trivy ${TRIVY_VERSION} (SHA-256 checked) into ${LOCAL_BIN}" "trivy ${TRIVY_VERSION} を（SHA-256 を検証して）${LOCAL_BIN} に入れる" +fi +if [ "${NEED_GITLEAKS}" = "1" ]; then + prereq_plan "download gitleaks ${GITLEAKS_VERSION} (SHA-256 checked) into ${LOCAL_BIN}" "gitleaks ${GITLEAKS_VERSION} を（SHA-256 を検証して）${LOCAL_BIN} に入れる" +fi +prereq_show_plan +prereq_token_note +if [ "${#PREREQ_PLAN_EN[@]}" -gt 0 ]; then + prereq_confirm "Make these changes?" "これらの変更を行いますか?" +fi + +# --- changes --------------------------------------------------------------------------------------- +if [ -n "${APT_MISSING}" ]; then + prereq_step 10 "update the apt package lists" prereq_as_root apt-get update + # shellcheck disable=SC2086 # APT_MISSING is a list of package names + prereq_step 10 "install ${APT_MISSING}" prereq_as_root env DEBIAN_FRONTEND=noninteractive apt-get install -y ${APT_MISSING} +fi + +verify_docker_signer() { # verify_docker_signer : exactly one primary key, with the expected fingerprint + GNUPGHOME="${PREREQ_TMP}/gnupg" + export GNUPGHOME + mkdir -m 700 -p "${GNUPGHOME}" + listing="$(gpg --show-keys --with-colons --with-fingerprint "$1" 2>/dev/null || true)" + primaries="$(printf '%s\n' "${listing}" | grep -c '^pub:' || true)" + fingerprint="$(printf '%s\n' "${listing}" | awk -F: '$1 == "fpr" { print $10; exit }')" + if [ "${primaries}" != "1" ] || [ "${fingerprint}" != "${DOCKER_SIGNER_FPR}" ]; then + prereq_fail 11 "the downloaded Docker signing key has an unexpected fingerprint: refusing to trust it" \ + "expected ${DOCKER_SIGNER_FPR}, got ${fingerprint:-none} (${primaries} primary keys)" \ + "ダウンロードした Docker の署名鍵の指紋が想定と違うため、信頼しません。" + fi +} + +if [ "${NEED_ENGINE}" = "1" ]; then + if [ "${PREREQ_DRY_RUN}" = "1" ]; then + prereq_say "[dry-run] download ${DOCKER_APT_URL}/gpg, require fingerprint ${DOCKER_SIGNER_FPR}, install it as ${DOCKER_APT_PGP}, write ${DOCKER_APT_SOURCES} (noble stable ${ARCH})" + else + prereq_tmp + prereq_step 11 "download Docker's apt signing key" curl --proto '=https' --tlsv1.2 -fsSL -o "${PREREQ_TMP}/docker.asc" "${DOCKER_APT_URL}/gpg" + verify_docker_signer "${PREREQ_TMP}/docker.asc" + prereq_say "Docker signing key verified: ${DOCKER_SIGNER_FPR}" "Docker の署名鍵を検証しました: ${DOCKER_SIGNER_FPR}" + cat > "${PREREQ_TMP}/docker.sources" < + if [ "${PREREQ_DRY_RUN}" = "1" ]; then + prereq_say "[dry-run] download $2, require SHA-256 $3, install $1 into ${LOCAL_BIN}" + return 0 + fi + prereq_tmp + archive="${PREREQ_TMP}/$1.tar.gz" + prereq_step 10 "download $1" curl --proto '=https' --tlsv1.2 -fsSL -o "${archive}" "$2" + if ! printf '%s %s\n' "$3" "${archive}" | sha256sum -c --status; then + prereq_fail 10 "$1: the download does not match its pinned SHA-256; nothing was installed / $1: ダウンロードが固定した SHA-256 と一致しないため、入れていません" + fi + mkdir -p "${PREREQ_TMP}/$1" "${LOCAL_BIN}" + prereq_step 10 "unpack $1" tar -xzf "${archive}" -C "${PREREQ_TMP}/$1" "$1" + prereq_step 10 "install $1 into ${LOCAL_BIN}" install -m 0755 "${PREREQ_TMP}/$1/$1" "${LOCAL_BIN}/$1" +} +if [ "${NEED_TRIVY}" = "1" ]; then + install_scanner trivy "https://github.com/aquasecurity/trivy/releases/download/v${TRIVY_VERSION}/trivy_${TRIVY_VERSION}_${TRIVY_ASSET}.tar.gz" "${TRIVY_SUM}" +fi +if [ "${NEED_GITLEAKS}" = "1" ]; then + install_scanner gitleaks "https://github.com/gitleaks/gitleaks/releases/download/v${GITLEAKS_VERSION}/gitleaks_${GITLEAKS_VERSION}_${GITLEAKS_ASSET}.tar.gz" "${GITLEAKS_SUM}" +fi + +# --- docker group: root-equivalent, so it has its own question ------------------------------------- +# Only a socket this user cannot open is a group problem; a stopped daemon is for `grdm-dev doctor` to report. +USE_SG="" +DOCKER_SOCK="/var/run/docker.sock" +if [ "${IS_WSL}" = "0" ] && [ "$(id -u)" != "0" ] \ + && { [ "${NEED_ENGINE}" = "1" ] || { [ -S "${DOCKER_SOCK}" ] && [ ! -w "${DOCKER_SOCK}" ]; }; } \ + && ! docker info >/dev/null 2>&1; then + me="$(id -un)" + if id -nG "${me}" 2>/dev/null | tr ' ' '\n' | grep -qx docker; then + USE_SG="sg" + else + prereq_say "Warning: members of the docker group can act as root on this computer." \ + "警告: docker グループの利用者は、このコンピュータの管理者と同じことができます。" + prereq_confirm "Add ${me} to the docker group?" "${me} を docker グループに加えますか?" + prereq_step 11 "add ${me} to the docker group" prereq_as_root usermod -aG docker "${me}" + USE_SG="sg" + prereq_say "This run continues in the docker group. For new terminals, log out and back in once." \ + "この実行は docker グループで続けます。新しい端末のために、一度ログアウトしてログインし直してください。" + fi +fi + +prereq_handover ${USE_SG} diff --git a/installer/install-windows.ps1 b/installer/install-windows.ps1 new file mode 100644 index 0000000..4a2e072 --- /dev/null +++ b/installer/install-windows.ps1 @@ -0,0 +1,269 @@ +# Prerequisites for Windows 11 (amd64), then installer/install-ubuntu.sh and installer/install.sh inside WSL2 +# (ADR-08, SPEC-02 §2.3). It only prepares the host: every installation decision stays in install.sh and grdm-dev. +# +# What it may change, after showing it and asking: the WSL feature (administrator, through a UAC prompt), the +# Ubuntu-24.04 distribution, and a clone of this repository inside the WSL file system. It never clones under +# /mnt/c, and it does not install Docker Desktop, whose licence is the user's decision. +# +# Compatible with Windows PowerShell 5.1. The file is UTF-8 with a BOM: without one, 5.1 reads it in the system +# code page and the Japanese strings break the parser. Exit codes are those of installer/grdm_dev/messages.tsv: +# 2 refused or no consent, 10 host, 11 Docker. +[CmdletBinding(PositionalBinding = $false)] +param( + [switch]$Yes, + [switch]$DryRun, + [switch]$Help, + [string]$Branch = 'develop', + [string]$Directory = 'RDM-developer-guide', + [Parameter(ValueFromRemainingArguments = $true)] + [string[]]$InstallArgs = @() +) + +$ErrorActionPreference = 'Stop' +$Distro = 'Ubuntu-24.04' +$RepoUrl = 'https://github.com/RCOSDP/RDM-developer-guide.git' +$Script:Plan = New-Object System.Collections.ArrayList + +function Write-Both([string]$English, [string]$Japanese) { + Write-Host "[install-windows.ps1] $English" + if ($Japanese) { Write-Host "[install-windows.ps1] $Japanese" } +} + +function Stop-Prereq([int]$Code, [string]$English, [string]$Japanese) { + if ($DryRun) { + Write-Both "[dry-run] would stop with exit ${Code}: $English" $Japanese + return + } + [Console]::Error.WriteLine("[install-windows.ps1] exit ${Code}: $English") + if ($Japanese) { [Console]::Error.WriteLine(" $Japanese") } + exit $Code +} + +function Test-Interactive { + return ([Environment]::UserInteractive -and -not [Console]::IsInputRedirected) +} + +function Confirm-Step([string]$English, [string]$Japanese) { + if ($Yes) { return } + if ($DryRun) { + if (-not (Test-Interactive)) { + Write-Both '[dry-run] would stop with exit 2: no terminal and no -Yes' '[dry-run] 端末が無く -Yes も無いため終了 2 で止まります' + } + return + } + if (-not (Test-Interactive)) { + Stop-Prereq 2 'no terminal to ask on: add -Yes to consent' '確認する端末がありません。同意するには -Yes を付けてください' + } + Write-Host $English + $answer = Read-Host "$Japanese [y/N]" + if ($answer -notmatch '^(y|yes)$') { + Stop-Prereq 2 'declined; nothing further was changed' '同意がなかったため、これ以上は変更しません' + } +} + +function Get-WslText([string[]]$Arguments) { + # wsl.exe prints UTF-16; read as text it arrives with NUL characters between the letters. + $previous = $ErrorActionPreference + $ErrorActionPreference = 'Continue' + try { + $output = & wsl.exe @Arguments 2>$null + $code = $LASTEXITCODE + } finally { + $ErrorActionPreference = $previous + } + $text = (($output | ForEach-Object { "$_" }) -join "`n") -replace "`0", '' + return @{ Code = $code; Text = $text } +} + +function ConvertTo-BashWord([string]$Value) { + return "'" + ($Value -replace "'", "'\''") + "'" +} + +if ($Help) { + @' +Usage: powershell -ExecutionPolicy Bypass -File installer\install-windows.ps1 [-Yes] [-DryRun] [-Branch ] [-Directory ] [grdm-dev install options...] + + Prepares WSL2 with Ubuntu 24.04, checks Docker Desktop, clones this repository inside WSL, and runs the installer there. + -Yes consent to every change listed in the plan (--yes also works) + -DryRun detect and print what would happen; change nothing (--dry-run also works) + -Branch branch to clone (default: develop) + -Directory folder under your WSL home directory (default: RDM-developer-guide) + Other options go to install.sh unchanged, for example --profile minimum. + +使い方: powershell -ExecutionPolicy Bypass -File installer\install-windows.ps1 [-Yes] [-DryRun] [-Branch <名前>] [-Directory <名前>] [grdm-dev install のオプション...] + + WSL2 と Ubuntu 24.04 を準備し、Docker Desktop を確認し、WSL の中にこのリポジトリを clone して、そこでインストーラーを実行します。 + -Yes 計画に載せたすべての変更に同意する（--yes も可） + -DryRun 判定して予定を表示するだけで、何も変更しない（--dry-run も可） + -Branch clone するブランチ（既定: develop） + -Directory WSL のホームディレクトリの下のフォルダ（既定: RDM-developer-guide） + ほかのオプションはそのまま install.sh に渡します。例: --profile minimum +'@ | Write-Host + exit 0 +} + +# Unix-style flags, so that the same habits work on every OS. +$Forward = New-Object System.Collections.ArrayList +foreach ($argument in $InstallArgs) { + switch ($argument) { + '--yes' { $Yes = $true } + '-y' { $Yes = $true } + '--dry-run' { $DryRun = $true } + default { [void]$Forward.Add($argument) } + } +} +if ($Yes) { [void]$Forward.Add('--yes') } + +# The folder must be a plain name under the WSL home: that is what keeps the clone off /mnt/c. +if ($Directory -notmatch '^[A-Za-z0-9._-]+$' -or $Directory -match '^\.+$') { + [Console]::Error.WriteLine('[install-windows.ps1] exit 2: -Directory must be one folder name under the WSL home directory, never a path such as /mnt/c/... / -Directory は WSL のホームの下のフォルダ名一つにしてください。/mnt/c/... のようなパスは使えません') + exit 2 +} +if ($Branch -notmatch '^[A-Za-z0-9._/-]+$') { + [Console]::Error.WriteLine('[install-windows.ps1] exit 2: -Branch contains characters that are not allowed / -Branch に使えない文字が含まれています') + exit 2 +} + +# --- host check (read-only) -------------------------------------------------------------------------- +$build = [Environment]::OSVersion.Version.Build +if ([Environment]::OSVersion.Platform -ne 'Win32NT' -or $build -lt 22000) { + Stop-Prereq 10 "Windows 11 is required (this is build $build)." "Windows 11 が必要です（このホストのビルドは $build）。" +} +$cpu = if ($env:PROCESSOR_ARCHITEW6432) { $env:PROCESSOR_ARCHITEW6432 } else { $env:PROCESSOR_ARCHITECTURE } +if ($cpu -ne 'AMD64') { + Stop-Prereq 10 "Only amd64 Windows is supported (this is $cpu)." "対応するのは amd64 の Windows だけです（このホストは $cpu）。" +} + +# --- detection (read-only) --------------------------------------------------------------------------- +$wslReady = $false +if (Get-Command wsl.exe -ErrorAction SilentlyContinue) { + $wslReady = (Get-WslText @('--status')).Code -eq 0 +} +$distroPresent = $false +$distroVersion = '' +if ($wslReady) { + foreach ($line in ((Get-WslText @('--list', '--verbose')).Text -split "`n")) { + $fields = ($line -replace '^\s*\*?\s*', '') -split '\s+' + if ($fields.Count -ge 3 -and $fields[0] -eq $Distro) { + $distroPresent = $true + $distroVersion = $fields[2] + } + } +} +if (-not $wslReady) { + [void]$Script:Plan.Add(@('administrator: enable WSL2 (wsl.exe --install --no-distribution); a restart may be needed', '管理者: WSL2 を有効にする（wsl.exe --install --no-distribution）。再起動が要ることがあります')) +} +if (-not $distroPresent) { + [void]$Script:Plan.Add(@("install the $Distro distribution from Microsoft's catalog", "Microsoft のカタログから $Distro を入れる")) +} elseif ($distroVersion -ne '2') { + [void]$Script:Plan.Add(@("convert $Distro to WSL version 2", "$Distro を WSL 版 2 に変換する")) +} +[void]$Script:Plan.Add(@("inside WSL: install git with sudo if it is absent, clone $RepoUrl (branch $Branch) to ~/$Directory if absent, then run installer/install-ubuntu.sh, which lists its own changes", "WSL の中で: git が無ければ sudo で入れ、無ければ $RepoUrl（ブランチ $Branch）を ~/$Directory に clone し、installer/install-ubuntu.sh を実行する（変更点はそのスクリプトが示します）")) + +Write-Both 'This script will make these changes:' 'このスクリプトは次の変更を行います:' +$number = 0 +foreach ($item in $Script:Plan) { + $number++ + Write-Host " $number. $($item[0])" + Write-Host " $($item[1])" +} +Write-Both 'Note: the integration repositories are private. This script never asks for, reads or stores a token.' '注意: 統合リポジトリは非公開です。このスクリプトはトークンを尋ねず、読まず、保存しません。' +Write-Both 'Inside WSL, before the build, give git your own access, for example: gh auth login, then gh auth setup-git' 'WSL の中で、ビルドの前に git へ自分の権限を設定してください。例: gh auth login の後に gh auth setup-git' +Confirm-Step 'Make these changes?' 'これらの変更を行いますか?' + +# --- changes ----------------------------------------------------------------------------------------- +if (-not $wslReady) { + if ($DryRun) { + Write-Both '[dry-run] administrator (UAC prompt): wsl.exe --install --no-distribution' + } else { + $process = Start-Process -FilePath 'wsl.exe' -ArgumentList '--install', '--no-distribution' -Verb RunAs -Wait -PassThru + if ($process.ExitCode -ne 0) { + Stop-Prereq 10 "wsl.exe --install failed with code $($process.ExitCode)." "wsl.exe --install がコード $($process.ExitCode) で失敗しました。" + } + Stop-Prereq 10 'WSL was enabled. Restart Windows, then run this script again.' 'WSL を有効にしました。Windows を再起動して、もう一度実行してください。' + } +} + +if (-not $distroPresent) { + if ($DryRun) { + Write-Both "[dry-run] wsl.exe --install --distribution $Distro --no-launch" + } else { + & wsl.exe --install --distribution $Distro --no-launch + if ($LASTEXITCODE -ne 0) { + Stop-Prereq 10 "Could not install $Distro (code $LASTEXITCODE)." "$Distro を入れられませんでした（コード $LASTEXITCODE）。" + } + } +} elseif ($distroVersion -ne '2') { + if ($DryRun) { + Write-Both "[dry-run] wsl.exe --set-version $Distro 2" + } else { + & wsl.exe --set-version $Distro 2 + if ($LASTEXITCODE -ne 0) { + Stop-Prereq 10 "Could not convert $Distro to WSL version 2 (code $LASTEXITCODE)." "$Distro を WSL 版 2 に変換できませんでした（コード $LASTEXITCODE）。" + } + } +} + +# Everything below talks to the distribution, which a dry run may not have. +if (-not $DryRun) { + $identity = Get-WslText @('--distribution', $Distro, '--exec', 'id', '-u') + if ($identity.Code -ne 0 -or $identity.Text.Trim() -eq '0') { + Stop-Prereq 10 "$Distro has no regular user yet. Run 'wsl -d $Distro' once, create your user, then run this script again." "$Distro にまだ一般利用者がいません。'wsl -d $Distro' を一度実行して利用者を作り、もう一度実行してください。" + } +} + +$desktop = Join-Path $env:ProgramFiles 'Docker\Docker\Docker Desktop.exe' +$desktopHelp = 'https://www.docker.com/products/docker-desktop/' +$engineAnswers = $false +if (-not $DryRun) { + $engineAnswers = (Get-WslText @('--distribution', $Distro, '--exec', 'docker', 'info', '--format', 'ok')).Code -eq 0 +} +if (-not $engineAnswers) { + if (-not (Test-Path -LiteralPath $desktop)) { + Stop-Prereq 11 "Docker Desktop was not found. Install it from $desktopHelp (check that its licence fits your organisation), start it, then run this script again." "Docker Desktop が見つかりません。$desktopHelp から入れ（ライセンスが所属組織に合うか確認してください）、起動してから、もう一度実行してください。" + } elseif (-not $DryRun) { + Stop-Prereq 11 "Docker does not answer inside $Distro. Start Docker Desktop and enable Settings > Resources > WSL Integration > $Distro, then run this script again." "$Distro の中で Docker が応答しません。Docker Desktop を起動し、Settings > Resources > WSL Integration > $Distro を有効にして、もう一度実行してください。" + } +} + +# The boot script travels base64-encoded in one environment variable, so no quoting layer of PowerShell, wsl.exe +# or the login shell can change it, and the terminal stays attached for sudo and the installer's questions. +$forwardWords = ($Forward | ForEach-Object { ConvertTo-BashWord $_ }) -join ' ' +$bootLines = @( + 'set -euo pipefail', + 'cd "$HOME"', + '# Physical paths (pwd -P): a home or checkout that is a symlink into /mnt/c is refused too (PR #44 review r1 R2).', + '# GRDM_WSL_WINDOWS_ROOT exists for the unit tests, which cannot create /mnt.', + 'win="${GRDM_WSL_WINDOWS_ROOT:-/mnt}"; win="$(cd "$win" 2>/dev/null && pwd -P || printf %s "$win")"', + 'home_real="$(pwd -P)"', + 'case "$home_real/" in "$win"/*) echo "the WSL home directory is on the Windows disk ($home_real); refusing to clone there" >&2; exit 10 ;; esac', + "target=`"`$HOME/$Directory`"", + 'if [ -e "$target" ] || [ -L "$target" ]; then', + ' target_real="$(cd "$target" 2>/dev/null && pwd -P)" || { echo "$target exists but is not a directory; refusing" >&2; exit 10; }', + 'else', + ' target_real="$home_real/${target##*/}"', + 'fi', + 'case "$target_real/" in "$win"/*) echo "the checkout $target is on the Windows disk ($target_real); refusing" >&2; exit 10 ;; esac', + 'if ! command -v git >/dev/null 2>&1; then', + ' sudo apt-get update && sudo apt-get install -y git', + 'fi', + 'if [ -d "$target_real/.git" ]; then', + ' echo "using the existing checkout at $target_real; it is not changed"', + 'else', + " git clone --branch $(ConvertTo-BashWord $Branch) $(ConvertTo-BashWord $RepoUrl) `"`$target_real`"", + 'fi', + 'cd "$target_real"', + "exec bash installer/install-ubuntu.sh $forwardWords" +) +$boot = ($bootLines -join "`n") + "`n" + +if ($DryRun) { + Write-Both "[dry-run] would run inside ${Distro}:" + Write-Host $boot + exit 0 +} + +$env:GRDM_WSL_BOOT = [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes($boot)) +$env:WSLENV = if ($env:WSLENV) { "$($env:WSLENV):GRDM_WSL_BOOT" } else { 'GRDM_WSL_BOOT' } +& wsl.exe --distribution $Distro --cd '~' --exec bash -c 'source <(base64 -d <<<$GRDM_WSL_BOOT)' +exit $LASTEXITCODE diff --git a/installer/install.sh b/installer/install.sh index e44d60a..0822ef7 100755 --- a/installer/install.sh +++ b/installer/install.sh @@ -11,7 +11,7 @@ # [accept] extra (pytest-playwright); GRDM_SKIP_EXEC=1 stop after the venv (tests only). # --- 0. CRLF guard. bash misreads a CRLF file, so this runs before `set` and stays on ONE line that ends in a # comment: the stray \r then lands inside the comment and the line still parses. -if grep -q "$(printf '\r')" "${BASH_SOURCE[0]}" 2>/dev/null; then printf '[install.sh] exit 2: %s\n this script has CRLF line endings; fix: sed -i "" "s/\\r$//" installer/install.sh\n このスクリプトの改行が CRLF です。修正: sed -i "" "s/\\r$//" installer/install.sh\n' "$(awk -F '\t' '$1 == "2" { printf "%s / %s", $2, $3; exit }' "$(dirname "${BASH_SOURCE[0]}")/grdm_dev/messages.tsv" 2>/dev/null || echo 'usage error')" >&2; exit 2; fi # one line, see above +if grep -q "$(printf '\r')" "${BASH_SOURCE[0]}" 2>/dev/null; then printf '[install.sh] exit 2: %s\n this script has CRLF line endings; fix: perl -pi -e "s/\\r\\$//" installer/install.sh\n このスクリプトの改行が CRLF です。修正: perl -pi -e "s/\\r\\$//" installer/install.sh\n' "$(awk -F '\t' '$1 == "2" { printf "%s / %s", $2, $3; exit }' "$(dirname "${BASH_SOURCE[0]}")/grdm_dev/messages.tsv" 2>/dev/null || echo 'usage error')" >&2; exit 2; fi # one line, see above set -euo pipefail ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" diff --git a/installer/prereq-lib.sh b/installer/prereq-lib.sh new file mode 100644 index 0000000..7fc2e47 --- /dev/null +++ b/installer/prereq-lib.sh @@ -0,0 +1,181 @@ +# shellcheck shell=bash +# Shared helpers for the per-OS prerequisite scripts (ADR-08, SPEC-02 §2.3). Sourced, never run. +# Stays on bash 3.2 (the macOS bash): no associative arrays, no mapfile, no ${var,,}. +# +# Contract kept by every script that sources this file: +# - detection is read-only; each change is listed first and needs consent (--yes gives it up front); +# - --dry-run performs detection and prints what would happen, and changes nothing; +# - the last action is always installer/install.sh, so there is one decision path; +# - exit codes are the ones in installer/grdm_dev/messages.tsv: 2 refused or no consent, 10 host or +# Python/git prerequisite, 11 Docker prerequisite. No new number is introduced. +# - no token is read, stored or passed; prereq_token_note only says how to provide one. + +PREREQ_NAME="prereq" +PREREQ_YES=0 +PREREQ_DRY_RUN=0 +PREREQ_FORWARD=() +PREREQ_TMP="" +PREREQ_PLAN_EN=() +PREREQ_PLAN_JA=() +# Physical path (pwd -P): a symlinked checkout must be judged by where its files really are (PR #44 review r1 R2). +PREREQ_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd -P)" +PREREQ_MESSAGES="${PREREQ_ROOT}/installer/grdm_dev/messages.tsv" + +prereq_say() { # prereq_say [] + printf '[%s] %s\n' "${PREREQ_NAME}" "$1" + if [ $# -gt 1 ]; then printf '[%s] %s\n' "${PREREQ_NAME}" "$2"; fi +} + +prereq_cleanup() { + if [ -n "${PREREQ_TMP}" ]; then rm -rf "${PREREQ_TMP}"; PREREQ_TMP=""; fi +} +trap prereq_cleanup EXIT + +prereq_tmp() { # creates the private temporary directory once; the path is left in PREREQ_TMP + if [ -z "${PREREQ_TMP}" ]; then PREREQ_TMP="$(mktemp -d)"; fi +} + +prereq_message() { # the canonical "en / ja" sentence for an exit code + if [ -f "${PREREQ_MESSAGES}" ]; then + awk -F '\t' -v code="$1" '$1 == code { printf "%s / %s\n", $2, $3; exit }' "${PREREQ_MESSAGES}" + else + printf 'exit %s\n' "$1" + fi +} + +prereq_fail() { # prereq_fail ...: always stops + code="$1"; shift + printf '[%s] exit %s: %s\n' "${PREREQ_NAME}" "${code}" "$(prereq_message "${code}")" >&2 + if [ $# -gt 0 ]; then printf '%s\n' "$@" | sed 's/^/ /' >&2; fi + exit "${code}" +} + +prereq_stop() { # like prereq_fail, but --dry-run reports the stop and goes on, to show the rest of the plan + if [ "${PREREQ_DRY_RUN}" = "1" ]; then + code="$1"; shift + printf '[%s] [dry-run] would stop with exit %s: %s\n' "${PREREQ_NAME}" "${code}" "$(prereq_message "${code}")" + if [ $# -gt 0 ]; then printf '%s\n' "$@" | sed 's/^/ /'; fi + return 0 + fi + prereq_fail "$@" +} + +prereq_parse_args() { # sets PREREQ_YES and PREREQ_DRY_RUN; every other argument goes to install.sh unchanged + for a in "$@"; do + case "${a}" in + -h|--help) prereq_usage; exit 0 ;; + --dry-run) PREREQ_DRY_RUN=1 ;; + -y|--yes) PREREQ_YES=1; PREREQ_FORWARD+=("--yes") ;; + *) PREREQ_FORWARD+=("${a}") ;; + esac + done +} + +prereq_plan() { # prereq_plan : one planned change + PREREQ_PLAN_EN+=("$1") + PREREQ_PLAN_JA+=("$2") +} + +prereq_show_plan() { + if [ "${#PREREQ_PLAN_EN[@]}" -eq 0 ]; then + prereq_say "All prerequisites are already present; nothing to change." "前提条件はすべて揃っています。変更するものはありません。" + return 0 + fi + prereq_say "This script will make these changes:" "このスクリプトは次の変更を行います:" + i=0 + while [ "${i}" -lt "${#PREREQ_PLAN_EN[@]}" ]; do + printf ' %d. %s\n %s\n' "$((i + 1))" "${PREREQ_PLAN_EN[${i}]}" "${PREREQ_PLAN_JA[${i}]}" + i=$((i + 1)) + done +} + +prereq_confirm() { # prereq_confirm : returns 0 on consent, otherwise stops with exit 2 + if [ "${PREREQ_YES}" = "1" ]; then return 0; fi + if [ "${PREREQ_DRY_RUN}" = "1" ]; then + if [ ! -t 0 ]; then + prereq_say "[dry-run] would stop with exit 2: no terminal and no --yes" "[dry-run] 端末が無く --yes も無いため終了 2 で止まります" + fi + return 0 + fi + if [ ! -t 0 ]; then + prereq_fail 2 "no terminal to ask on: add --yes to consent / 確認する端末がありません。同意するには --yes を付けてください" + fi + printf '%s\n%s [y/N] ' "$1" "$2" + answer="" + read -r answer || answer="" + case "${answer}" in + y|Y|yes|YES|Yes) return 0 ;; + *) prereq_fail 2 "declined; nothing further was changed / 同意がなかったため、これ以上は変更しません" ;; + esac +} + +prereq_step() { # prereq_step : prints or runs; a failure stops with the code + code="$1"; what="$2"; shift 2 + if [ "${PREREQ_DRY_RUN}" = "1" ]; then + shown="$*" + printf '[%s] [dry-run] %s: %s\n' "${PREREQ_NAME}" "${what}" "${shown/#prereq_as_root/sudo}" + return 0 + fi + prereq_say "${what}" + if ! "$@"; then + prereq_fail "${code}" "failed: ${what} / 失敗しました: ${what}" "command: $*" + fi +} + +prereq_as_root() { # runs a command with sudo; without a terminal sudo must not ask, so it fails instead of hanging + if [ "$(id -u)" = "0" ]; then + "$@" + elif [ -t 0 ]; then + sudo "$@" + else + sudo -n "$@" + fi +} + +prereq_python_ok() { # 0 when an interpreter of 3.11 or newer is on PATH (same candidates as install.sh) + for cand in python3 python3.13 python3.12 python3.11; do + if command -v "${cand}" >/dev/null 2>&1 \ + && "${cand}" -c 'import sys; sys.exit(0 if sys.version_info[:2] >= (3, 11) else 1)' >/dev/null 2>&1; then + return 0 + fi + done + return 1 +} + +prereq_docker_client_ok() { # 0 when the docker client is version 27 or newer (the daemon may be unreachable) + version="$(docker version --format '{{.Client.Version}}' 2>/dev/null || true)" + major="${version%%.*}" + case "${major}" in + ''|*[!0-9]*) return 1 ;; + esac + [ "${major}" -ge 27 ] +} + +prereq_docker_plugins_ok() { + docker compose version >/dev/null 2>&1 && docker buildx version >/dev/null 2>&1 +} + +prereq_token_note() { + prereq_say "Note: the integration repositories are private. This script never asks for, reads or stores a token." \ + "注意: 統合リポジトリは非公開です。このスクリプトはトークンを尋ねず、読まず、保存しません。" + prereq_say "Before the build, give git your own access, for example: gh auth login && gh auth setup-git" \ + "ビルドの前に git へ自分の権限を設定してください。例: gh auth login && gh auth setup-git" +} + +prereq_handover() { # prereq_handover [sg]: the last action of every script. "sg" runs it in the docker group. + prereq_cleanup + target="${PREREQ_ROOT}/installer/install.sh" + if [ "${PREREQ_DRY_RUN}" = "1" ]; then + printf '[%s] [dry-run] would run: %s' "${PREREQ_NAME}" "${target}" + if [ "${#PREREQ_FORWARD[@]}" -gt 0 ]; then printf ' %s' "${PREREQ_FORWARD[@]}"; fi + printf '\n' + exit 0 + fi + if [ "${1:-}" = "sg" ]; then + # shellcheck disable=SC2068 # the ${a[@]+"${a[@]}"} form is what keeps an empty array safe on bash 3.2 + quoted="$(printf '%q ' bash "${target}" ${PREREQ_FORWARD[@]+"${PREREQ_FORWARD[@]}"})" + exec sg docker -c "${quoted}" + fi + # shellcheck disable=SC2068 # see above + exec bash "${target}" ${PREREQ_FORWARD[@]+"${PREREQ_FORWARD[@]}"} +} diff --git a/installer/tests/test_accept.py b/installer/tests/test_accept.py index 7fa6b6d..4a07b81 100644 --- a/installer/tests/test_accept.py +++ b/installer/tests/test_accept.py @@ -155,6 +155,15 @@ def test_missing_playwright_is_usage_error_with_hint(self): self.assertIn("playwright install chromium", err) self.assertEqual(runner.calls, []) + def test_linux_hint_adds_libatomic_and_system_libraries(self): + # DEMO-21: on the Ubuntu 24.04 arm64 cloud image Playwright's driver could not start without libatomic1. + linux = accept.install_hint("linux") + self.assertTrue(linux.startswith("sudo apt-get install -y libatomic1 && ")) + self.assertIn("playwright install chromium", linux) + self.assertIn("playwright install-deps chromium", linux) + self.assertEqual(accept.install_hint("darwin"), accept.INSTALL_HINT) + self.assertNotIn("apt-get", accept.install_hint("darwin")) + def test_non_loopback_base_url_is_refused(self): for url in ("http://localhost:5000", "http://192.168.1.10:5000", "https://127.0.0.1:5000", "http://example.com"): runner = FakeRunner() diff --git a/installer/tests/test_lifecycle.py b/installer/tests/test_lifecycle.py index 9f3b7c3..3cc10fa 100644 --- a/installer/tests/test_lifecycle.py +++ b/installer/tests/test_lifecycle.py @@ -119,3 +119,41 @@ def test_missing_compose_is_usage_error(self): if __name__ == "__main__": unittest.main() + + +class DefaultTimeoutTests(unittest.TestCase): + """DEMO-21 run 2: on Linux arm64 QEMU took 44 min for ember's first build, so 1800 s stopped a good install.""" + + def setUp(self): + import shutil + self.tmp = tempfile.TemporaryDirectory() + self.addCleanup(self.tmp.cleanup) + self.root = Path(self.tmp.name) + shutil.copytree(Path(__file__).resolve().parents[2] / "manifest", self.root / "manifest") + + def timeout(self, profile, system, machine): + return lifecycle.default_timeout(Context(root=self.root, profile=profile), system=system, machine=machine) + + def test_linux_arm64_with_amd64_services_waits_longer(self): + self.assertEqual(self.timeout("core", "Linux", "aarch64"), lifecycle.QEMU_TIMEOUT) + self.assertGreaterEqual(lifecycle.QEMU_TIMEOUT, 2 * 44 * 60) + + def test_other_hosts_and_profiles_keep_the_default(self): + self.assertEqual(self.timeout("core", "Linux", "x86_64"), lifecycle.DEFAULT_TIMEOUT) + self.assertEqual(self.timeout("core", "Darwin", "arm64"), lifecycle.DEFAULT_TIMEOUT) + self.assertEqual(self.timeout("minimum", "Linux", "aarch64"), lifecycle.DEFAULT_TIMEOUT) + + def test_unreadable_manifest_keeps_the_default(self): + (self.root / "manifest" / "upstreams.json").write_text("{not json") + self.assertEqual(self.timeout("core", "Linux", "aarch64"), lifecycle.DEFAULT_TIMEOUT) + + def test_explicit_timeout_wins_and_none_means_default(self): + seen = [] + original = lifecycle.run_up + self.addCleanup(setattr, lifecycle, "run_up", original) + lifecycle.run_up = lambda ctx, **kw: seen.append(kw["timeout"]) or 0 + ctx = Context(root=self.root, profile="core") + lifecycle.run(ctx, argparse.Namespace(lifecycle_action="up", no_open=True, timeout=12.0)) + lifecycle.run(ctx, argparse.Namespace(lifecycle_action="up", no_open=True, timeout=None)) + self.assertEqual(seen[0], 12.0) + self.assertEqual(seen[1], lifecycle.default_timeout(ctx)) diff --git a/installer/tests/test_preflight.py b/installer/tests/test_preflight.py index 339ac88..0f07e1e 100644 --- a/installer/tests/test_preflight.py +++ b/installer/tests/test_preflight.py @@ -240,9 +240,24 @@ def test_enabled_emulation_passes(self): def test_missing_emulation_is_exit_13_and_prints_the_command(self): code, _, err = self.doctor(self.arm(binfmt_amd64=False), preflight_runner()) self.assertEqual(code, exit_codes.PREFLIGHT_FAILED) - self.assertIn("docker run --privileged --rm tonistiigi/binfmt --install amd64", err) + self.assertIn(host.ENABLE_AMD64_EMULATION, err) + self.assertIn("tonistiigi/binfmt:qemu-v10.2.3-68@sha256:", err) self.assertIn("mfr", err) + def test_measured_bad_ubuntu_qemu_is_exit_13_and_prints_the_replacement(self): + # DEMO-21 run 1: Ubuntu 24.04's qemu-user-static crashed in libgvc6's post-install script after a 48-minute run. + bad = "qemu-x86_64 version 8.2.2 (Debian 1:8.2.2+ds-0ubuntu1.18)" + code, _, err = self.doctor(self.arm(binfmt_amd64=True, binfmt_amd64_qemu=bad), preflight_runner()) + self.assertEqual(code, exit_codes.PREFLIGHT_FAILED) + self.assertIn("QEMU internal SIGSEGV", err) + self.assertIn("--uninstall qemu-x86_64", err) + self.assertIn(host.REPLACE_AMD64_EMULATION, err) + + def test_other_or_unknown_qemu_builds_pass(self): + for line in ("", "qemu-x86_64 version 10.2.3", "qemu-x86_64 version 8.2.2", "qemu-x86_64 version 9.2.2 (Debian 1:9.2.2+ds-1)"): + with self.subTest(line=line): + self.assertEqual(self.doctor(self.arm(binfmt_amd64=True, binfmt_amd64_qemu=line), preflight_runner())[0], exit_codes.OK) + def test_unreadable_binfmt_is_a_warning(self): code, out, _ = self.doctor(self.arm(binfmt_amd64=None), preflight_runner()) self.assertEqual(code, exit_codes.OK) @@ -253,6 +268,45 @@ def test_not_probed_on_amd64_or_macos(self): self.assertEqual(self.doctor(self.mac(binfmt_amd64=False), preflight_runner())[0], exit_codes.OK) +class QemuProbeTests(unittest.TestCase): + def setUp(self): + self.tmp = tempfile.TemporaryDirectory() + self.addCleanup(self.tmp.cleanup) + self.root = Path(self.tmp.name) + + def entry(self, interpreter): + path = self.root / "qemu-x86_64" + path.write_text(f"enabled\ninterpreter {interpreter}\nflags: OPF\noffset 0\n") + return str(path) + + def test_resolves_the_interpreter_and_reads_its_version(self): + real = self.root / "qemu-x86_64-static" + real.write_text("") + link = self.root / "x86_64-binfmt-P" + link.symlink_to(real) + seen = [] + + def runner(argv): + seen.append(argv) + return 0, "qemu-x86_64 version 8.2.2 (Debian 1:8.2.2+ds-0ubuntu1.18)\nCopyright\n" + + line = host._binfmt_amd64_qemu(runner, self.entry(str(link))) + self.assertEqual(seen, [[str(real.resolve()), "--version"]]) + self.assertEqual(host.known_bad_qemu(line), "Ubuntu 24.04 qemu-user-static 8.2.2") + + def test_container_registered_handler_is_unknown_not_bad(self): + def runner(argv): + raise AssertionError("must not run a file that is not on this host") + + self.assertEqual(host._binfmt_amd64_qemu(runner, self.entry("/usr/bin/qemu-x86_64-does-not-exist")), "") + self.assertEqual(host._binfmt_amd64_qemu(runner, str(self.root / "missing-entry")), "") + + def test_failing_version_is_unknown(self): + real = self.root / "qemu" + real.write_text("") + self.assertEqual(host._binfmt_amd64_qemu(lambda argv: (1, "has to be run using kernel binfmt-misc"), self.entry(str(real))), "") + + class RepoPathTests(PreflightBase): NO_SUCH_FILE = (2, "ls: cannot access '/probe/upstreams.json': No such file or directory\n") diff --git a/installer/tests/test_prereq_scripts.py b/installer/tests/test_prereq_scripts.py new file mode 100644 index 0000000..fa2661f --- /dev/null +++ b/installer/tests/test_prereq_scripts.py @@ -0,0 +1,561 @@ +"""Tests for the per-OS prerequisite scripts (ADR-08, SPEC-02 §2.3). + +Every run below is `--dry-run` or `--help` under a PATH of stubs, so no test can change the machine. A stub `sudo`, +`brew`, `apt-get`, `curl`, `open` or `usermod` records its call; the tests assert that nothing was recorded. +The PowerShell script is only inspected as text here: the parser runs in CI on windows-latest. +""" +import os +import re +import shutil +import subprocess +import tempfile +import unittest +from pathlib import Path + +REPO = Path(__file__).resolve().parents[2] +INSTALLER = REPO / "installer" +UBUNTU = INSTALLER / "install-ubuntu.sh" +MACOS = INSTALLER / "install-macos.sh" +LIB = INSTALLER / "prereq-lib.sh" +WINDOWS = INSTALLER / "install-windows.ps1" + +MUTATING = ("sudo", "apt-get", "curl", "gpg", "usermod", "systemctl", "brew", "open", "sg", "wsl.exe") +UBUNTU_RELEASE = 'ID=ubuntu\nVERSION_ID="24.04"\n' +BINFMT_IMAGE = "tonistiigi/binfmt:qemu-v10.2.3-68@sha256:400a4873b838d1b89194d982c45e5fb3cda4593fbfd7e08a02e76b03b21166f0" +DOCKER_OK = """#!/bin/sh +case "$*" in + "version --format"*) echo 29.1.0 ;; +esac +exit 0 +""" +DOCKER_DOWN = """#!/bin/sh +case "$*" in + "version --format"*) echo 29.1.0 ;; + "compose version"|"buildx version") exit 0 ;; +esac +exit 1 +""" + + +def _bash(): + return "/bin/bash" if Path("/bin/bash").exists() else shutil.which("bash") + + +class ScriptRun(unittest.TestCase): + def setUp(self): + self.tmp = tempfile.TemporaryDirectory() + self.addCleanup(self.tmp.cleanup) + self.root = Path(self.tmp.name) + self.bin = self.root / "bin" + self.home = self.root / "home" + self.calls = self.root / "calls.log" + self.bin.mkdir() + self.home.mkdir() + for tool in ("awk", "sed", "grep", "dirname", "mktemp", "rm", "tr", "id", "cat", "bash", "mkdir", "sh", "head", "readlink"): + src = shutil.which(tool) + if src: + os.symlink(src, self.bin / tool) + for tool in MUTATING: + self._stub(tool, f'#!/bin/sh\necho "{tool} $*" >> "{self.calls}"\nexit 0\n') + + def _stub(self, name, body): + path = self.bin / name + if path.exists() or path.is_symlink(): + path.unlink() + path.write_text(body) + path.chmod(0o755) + + def _host(self, system, machine): + self._stub("uname", f'#!/bin/sh\ncase "$1" in -s) echo {system} ;; -m) echo {machine} ;; esac\n') + + def _run(self, script, *args, env_extra=None): + env = {"PATH": str(self.bin), "HOME": str(self.home), "LANG": "C"} + env.update(env_extra or {}) + return subprocess.run([_bash(), str(script), *args], env=env, capture_output=True, text=True, + timeout=60, stdin=subprocess.DEVNULL) + + def assert_nothing_changed(self): + recorded = self.calls.read_text() if self.calls.exists() else "" + self.assertEqual(recorded, "", f"a mutating command ran in a dry run:\n{recorded}") + + +class UbuntuTests(ScriptRun): + def setUp(self): + super().setUp() + self.release = self.root / "os-release" + self.release.write_text(UBUNTU_RELEASE) + self.mounts = self.root / "mounts" + self.mounts.write_text("/dev/vda1 / ext4 rw,relatime 0 0\n") + self.env = {"GRDM_PREREQ_OS_RELEASE": str(self.release), "GRDM_PREREQ_PROC_VERSION": str(self.root / "none"), + "GRDM_PREREQ_BINFMT_DIR": str(self.root / "binfmt"), "GRDM_PREREQ_MOUNTS": str(self.mounts)} + self._stub("dpkg-query", "#!/bin/sh\nexit 1\n") + + def test_fresh_amd64_plans_engine_and_scanners(self): + self._host("Linux", "x86_64") + cp = self._run(UBUNTU, "--dry-run", "--yes", "--profile", "minimum", env_extra=self.env) + self.assertEqual(cp.returncode, 0, cp.stderr) + self.assertIn("9DC858229FC7DD38854AE2D88D81803C0EBFCD88", cp.stdout) + self.assertIn("noble stable amd64", cp.stdout) + self.assertIn("docker-buildx-plugin docker-compose-plugin", cp.stdout) + self.assertIn("trivy_0.75.0_Linux-64bit.tar.gz", cp.stdout) + self.assertIn("gitleaks_8.21.2_linux_x64.tar.gz", cp.stdout) + self.assertNotIn("qemu-user-static", cp.stdout) + self.assertRegex(cp.stdout, r"would run: \S*installer/install\.sh --yes --profile minimum\n") + self.assert_nothing_changed() + + def test_arm64_plans_emulation_and_arm_assets(self): + self._host("Linux", "aarch64") + cp = self._run(UBUNTU, "--dry-run", "--yes", env_extra=self.env) + self.assertEqual(cp.returncode, 0, cp.stderr) + # QEMU comes from the pinned image, never from Ubuntu's qemu-user-static (DEMO-21 run 1). + self.assertIn(f"sudo docker run --privileged --rm {BINFMT_IMAGE} --install amd64", cp.stdout) + self.assertNotIn("--uninstall", cp.stdout) + self.assertNotIn("apt-get install -y qemu-user-static", cp.stdout) + self.assertIn("trivy_0.75.0_Linux-ARM64.tar.gz", cp.stdout) + self.assertIn("gitleaks_8.21.2_linux_arm64.tar.gz", cp.stdout) + self.assert_nothing_changed() + + def _binfmt_entry(self, flags, version_line=None): + """A fake binfmt_misc entry; with version_line, its interpreter is a host file (reached through a symlink, + like Ubuntu's x86_64-binfmt-P) that prints that line for --version.""" + interpreter = "/usr/bin/qemu-x86_64-from-a-container" + if version_line is not None: + real = self.root / "qemu-x86_64-static" + real.write_text(f'#!/bin/sh\n[ "$1" = --version ] && echo "{version_line}" && echo Copyright\n') + real.chmod(0o755) + link = self.root / "x86_64-binfmt-P" + link.symlink_to(real) + interpreter = str(link) + (self.root / "binfmt").mkdir() + (self.root / "binfmt" / "qemu-x86_64").write_text(f"enabled\ninterpreter {interpreter}\nflags: {flags}\n") + + def _arm_ready(self): + self._host("Linux", "aarch64") + self._stub("dpkg-query", '#!/bin/sh\nprintf "install ok installed"\n') + self._stub("docker", DOCKER_OK) + self._stub("trivy", "#!/bin/sh\nexit 0\n") + self._stub("gitleaks", "#!/bin/sh\nexit 0\n") + + def test_arm64_measured_bad_ubuntu_qemu_is_replaced(self): + self._arm_ready() + self._binfmt_entry("OPF", "qemu-x86_64 version 8.2.2 (Debian 1:8.2.2+ds-0ubuntu1.18)") + cp = self._run(UBUNTU, "--dry-run", "--yes", env_extra=self.env) + self.assertEqual(cp.returncode, 0, cp.stderr) + self.assertIn("replace the registered amd64 emulator (qemu-x86_64 version 8.2.2", cp.stdout) + self.assertIn(f"sudo docker run --privileged --rm {BINFMT_IMAGE} --uninstall qemu-x86_64", cp.stdout) + self.assertIn(f"sudo docker run --privileged --rm {BINFMT_IMAGE} --install amd64", cp.stdout) + self.assert_nothing_changed() + + def test_arm64_good_emulation_is_kept(self): + for version_line in (None, "qemu-x86_64 version 10.2.3"): + with self.subTest(version_line=version_line): + shutil.rmtree(self.root / "binfmt", ignore_errors=True) + for name in ("qemu-x86_64-static", "x86_64-binfmt-P"): + (self.root / name).unlink(missing_ok=True) + self._arm_ready() + self._binfmt_entry("POCF", version_line) + cp = self._run(UBUNTU, "--dry-run", "--yes", env_extra=self.env) + self.assertEqual(cp.returncode, 0, cp.stderr) + self.assertIn("nothing to change", cp.stdout) + self.assertNotIn("binfmt", cp.stdout) + self.assert_nothing_changed() + + def test_arm64_handler_without_f_flag_is_registered_again(self): + self._arm_ready() + self._binfmt_entry("OP") + cp = self._run(UBUNTU, "--dry-run", "--yes", env_extra=self.env) + self.assertEqual(cp.returncode, 0, cp.stderr) + self.assertIn("register QEMU from", cp.stdout) + self.assertIn("--uninstall qemu-x86_64", cp.stdout) + self.assert_nothing_changed() + + def test_accept_extra_adds_libatomic(self): + self._host("Linux", "x86_64") + cp = self._run(UBUNTU, "--dry-run", "--yes", env_extra=dict(self.env, GRDM_WITH_ACCEPT="1")) + self.assertEqual(cp.returncode, 0, cp.stderr) + self.assertIn("libatomic1", cp.stdout) + cp = self._run(UBUNTU, "--dry-run", "--yes", env_extra=self.env) + self.assertNotIn("libatomic1", cp.stdout) + self.assert_nothing_changed() + + def _shared_home(self, fstype, with_credentials): + home = self.root / "host-home" + (home / ".config" / "gh").mkdir(parents=True) + if with_credentials: + (home / ".config" / "gh" / "hosts.yml").write_text("not read by the script\n") + self.mounts.write_text(f"/dev/vda1 / ext4 rw 0 0\nlima-abc {home} {fstype} ro,relatime 0 0\n") + return home + + def test_shared_host_credentials_stop_with_exit_10(self): + # DEMO-21: Lima's template mounted the host home read-only, so ~/.config/gh was visible in the VM. + self._host("Linux", "x86_64") + home = self._shared_home("virtiofs", True) + cp = self._run(UBUNTU, "--yes", env_extra=self.env) + self.assertEqual(cp.returncode, 10, cp.stdout + cp.stderr) + self.assertIn(f"{home}/.config/gh/hosts.yml", cp.stderr) + self.assertIn("lima.yaml", cp.stderr) + self.assert_nothing_changed() + cp = self._run(UBUNTU, "--dry-run", "--yes", env_extra=self.env) + self.assertEqual(cp.returncode, 0, cp.stderr) + self.assertIn("would stop with exit 10", cp.stdout) + + def test_trailing_newlines_in_mount_names_are_kept(self): + # PR #44 review r2 R2-01: $(...) stripped the trailing newline of "host\012", so host/.netrc was checked. + self._host("Linux", "x86_64") + cases = (("host\n", "host\\012"), ("host\n\n", "host\\012\\012"), ("host\\012", "host\\134012")) + for fstype in ("virtiofs", "9p", "fuse.sshfs", "vboxsf", "fuse.vmhgfs-fuse", "fuse.virtiofsd"): + for name, encoded in cases: + with self.subTest(fstype=fstype, name=repr(name)): + base = self.root / f"nl-{fstype}" + (base / name).mkdir(parents=True, exist_ok=True) + (base / name / ".netrc").write_text("dummy\n") + (base / "host").mkdir(exist_ok=True) # the wrong, stripped path exists but holds nothing + self.mounts.write_text(f"share {base}/{encoded} {fstype} ro 0 0\n") + cp = self._run(UBUNTU, "--yes", env_extra=self.env) + self.assertEqual(cp.returncode, 10, cp.stdout + cp.stderr) + self.assertIn("host credentials are visible", cp.stderr) + self.assert_nothing_changed() + + def test_shared_folder_without_credentials_or_local_disk_passes(self): + self._host("Linux", "x86_64") + for fstype, creds in (("virtiofs", False), ("ext4", True)): + with self.subTest(fstype=fstype): + shutil.rmtree(self.root / "host-home", ignore_errors=True) + self._shared_home(fstype, creds) + cp = self._run(UBUNTU, "--dry-run", "--yes", env_extra=self.env) + self.assertEqual(cp.returncode, 0, cp.stderr) + self.assertNotIn("would stop with exit 10", cp.stdout) + + def test_binfmt_image_and_bad_build_match_doctor(self): + import sys + sys.path.insert(0, str(INSTALLER)) + from grdm_dev import host + text = UBUNTU.read_text() + self.assertIn(f'BINFMT_IMAGE="{host.BINFMT_IMAGE}"', text) + self.assertRegex(host.BINFMT_IMAGE, r"@sha256:[0-9a-f]{64}$") + self.assertIn(f"KNOWN_BAD_QEMU_RE='{host.KNOWN_BAD_QEMU[0][0].pattern}'", text) + + def test_second_run_is_a_no_op(self): + self._host("Linux", "x86_64") + self._stub("dpkg-query", '#!/bin/sh\nprintf "install ok installed"\n') + self._stub("docker", DOCKER_OK) + self._stub("trivy", "#!/bin/sh\nexit 0\n") + self._stub("gitleaks", "#!/bin/sh\nexit 0\n") + cp = self._run(UBUNTU, "--dry-run", env_extra=self.env) + self.assertEqual(cp.returncode, 0, cp.stderr) + self.assertIn("nothing to change", cp.stdout) + self.assertNotIn("apt-get", cp.stdout) + self.assertNotIn("would stop", cp.stdout) + self.assert_nothing_changed() + + def test_no_terminal_without_yes_is_refused(self): + self._host("Linux", "x86_64") + cp = self._run(UBUNTU, "--dry-run", env_extra=self.env) + self.assertEqual(cp.returncode, 0, cp.stderr) + self.assertIn("would stop with exit 2", cp.stdout) + self.assert_nothing_changed() + + def test_unsupported_release_exits_10(self): + self._host("Linux", "x86_64") + self.release.write_text('ID=ubuntu\nVERSION_ID="22.04"\n') + cp = self._run(UBUNTU, "--dry-run", "--yes", env_extra=self.env) + self.assertEqual(cp.returncode, 10, cp.stderr) + self.assertIn("exit 10", cp.stderr) + self.assertIn("host unsupported", cp.stderr) + self.assert_nothing_changed() + + def test_macos_host_exits_10(self): + self._host("Darwin", "arm64") + cp = self._run(UBUNTU, "--dry-run", "--yes", env_extra=self.env) + self.assertEqual(cp.returncode, 10, cp.stderr) + self.assertIn("install-macos.sh", cp.stderr) + + def test_conflicting_package_is_reported_not_removed(self): + self._host("Linux", "x86_64") + self._stub("dpkg-query", '#!/bin/sh\ncase "$*" in *docker.io*) printf "install ok installed" ;; *) exit 1 ;; esac\n') + cp = self._run(UBUNTU, "--dry-run", "--yes", env_extra=self.env) + self.assertEqual(cp.returncode, 0, cp.stderr) + self.assertIn("would stop with exit 11", cp.stdout) + self.assertIn("sudo apt-get remove docker.io", cp.stdout) + self.assert_nothing_changed() + + def test_wsl_uses_docker_desktop_and_never_installs_engine(self): + self._host("Linux", "x86_64") + env = dict(self.env, WSL_DISTRO_NAME="Ubuntu-24.04") + cp = self._run(UBUNTU, "--dry-run", "--yes", env_extra=env) + self.assertEqual(cp.returncode, 0, cp.stderr) + self.assertIn("would stop with exit 11", cp.stdout) + self.assertIn("WSL Integration", cp.stdout) + self.assertNotIn("download.docker.com", cp.stdout) + self.assert_nothing_changed() + + def _copy_checkout(self, dest): + (dest / "installer" / "grdm_dev").mkdir(parents=True) + for name in ("install-ubuntu.sh", "prereq-lib.sh"): + shutil.copy2(INSTALLER / name, dest / "installer" / name) + shutil.copy2(INSTALLER / "grdm_dev" / "messages.tsv", dest / "installer" / "grdm_dev" / "messages.tsv") + return dest + + def _wsl_run(self, checkout_link): + self._host("Linux", "x86_64") + env = dict(self.env, WSL_DISTRO_NAME="Ubuntu-24.04", GRDM_PREREQ_WSL_WINDOWS_ROOT=str(self.root / "win")) + return self._run(checkout_link / "installer" / "install-ubuntu.sh", "--dry-run", "--yes", env_extra=env) + + def test_wsl_refuses_a_checkout_on_the_windows_disk_even_through_a_symlink(self): + # PR #44 review r1 R2: the logical path ~/rdm passed the /mnt/* check while the files were on the Windows disk. + real = self._copy_checkout(self.root / "win" / "c" / "Users" / "dev" / "rdm") + link = self.home / "rdm" + link.symlink_to(real, target_is_directory=True) + for checkout in (real, link): + with self.subTest(checkout=str(checkout)): + cp = self._wsl_run(checkout) + self.assertEqual(cp.returncode, 10, cp.stdout + cp.stderr) + self.assertIn("on the Windows disk", cp.stderr) + self.assert_nothing_changed() + + def test_wsl_accepts_a_checkout_inside_wsl(self): + (self.root / "win").mkdir() + checkout = self._copy_checkout(self.home / "rdm") + cp = self._wsl_run(checkout) + self.assertNotIn("on the Windows disk", cp.stderr) + + def test_mount_escapes_are_decoded_before_the_credential_check(self): + # PR #44 review r1 R1: /proc/mounts writes "host home" as host\040home, which the check used literally. + self._host("Linux", "x86_64") + for name, encoded in (("host home", "host\\040home"), ("back\\slash", "back\\134slash"), ("tab\there", "tab\\011here")): + for fstype in ("virtiofs", "9p", "fuse.sshfs"): + with self.subTest(name=name, fstype=fstype): + base = self.root / f"share-{fstype}" + home = base / name + (home / ".config" / "gh").mkdir(parents=True, exist_ok=True) + (home / ".config" / "gh" / "hosts.yml").write_text("not read\n") + self.mounts.write_text(f"/dev/vda1 / ext4 rw 0 0\nshare {base}/{encoded} {fstype} ro 0 0\n") + cp = self._run(UBUNTU, "--yes", env_extra=self.env) + self.assertEqual(cp.returncode, 10, cp.stdout + cp.stderr) + self.assertIn(f"{home}/.config/gh/hosts.yml", cp.stderr) + self.assert_nothing_changed() + + +class MacosTests(ScriptRun): + def setUp(self): + super().setUp() + self._host("Darwin", "arm64") + self._stub("python3", "#!/bin/sh\nexit 1\n") + self._stub("git", "#!/bin/sh\nexit 1\n") + # Without these the script would look at the real Homebrew and /Applications of the machine running the tests. + self.env = {"GRDM_PREREQ_BREW_CANDIDATES": "", "GRDM_PREREQ_APPLICATIONS": str(self.root / "apps")} + + def _run(self, script, *args, env_extra=None): + return super()._run(script, *args, env_extra=dict(self.env, **(env_extra or {}))) + + def test_plan_uses_homebrew(self): + self._stub("docker", DOCKER_OK) + cp = self._run(MACOS, "--dry-run", "--yes", "--profile", "minimum") + self.assertEqual(cp.returncode, 0, cp.stderr) + self.assertIn("brew install python@3.12 git trivy gitleaks", cp.stdout) + self.assertRegex(cp.stdout, r"would run: \S*installer/install\.sh --yes --profile minimum\n") + self.assert_nothing_changed() + + def test_no_docker_runtime_stops_with_guidance(self): + cp = self._run(MACOS, "--dry-run", "--yes") + self.assertEqual(cp.returncode, 0, cp.stderr) + self.assertIn("would stop with exit 11", cp.stdout) + self.assertIn("Docker Desktop", cp.stdout) + self.assert_nothing_changed() + + def test_linux_host_exits_10(self): + self._host("Linux", "x86_64") + cp = self._run(MACOS, "--dry-run", "--yes") + self.assertEqual(cp.returncode, 10, cp.stderr) + self.assertIn("install-ubuntu.sh", cp.stderr) + + +class SharedRules(unittest.TestCase): + def test_syntax_on_the_bash_that_ships_with_macos(self): + for script in (UBUNTU, MACOS, LIB): + cp = subprocess.run([_bash(), "-n", str(script)], capture_output=True, text=True) + self.assertEqual(cp.returncode, 0, f"{script.name}: {cp.stderr}") + + def test_help_is_english_then_japanese(self): + for script in (UBUNTU, MACOS): + cp = subprocess.run([_bash(), str(script), "--help"], capture_output=True, text=True, timeout=30) + self.assertEqual(cp.returncode, 0, cp.stderr) + self.assertLess(cp.stdout.index("Usage:"), cp.stdout.index("使い方:")) + + def test_scripts_are_executable_lf_and_end_at_install_sh(self): + for script in (UBUNTU, MACOS): + self.assertTrue(os.access(script, os.X_OK), f"{script.name} must be executable") + self.assertNotIn(b"\r", script.read_bytes()) + self.assertEqual(script.read_text(encoding="utf-8").strip().splitlines()[-1].split()[0], "prereq_handover") + self.assertIn('target="${PREREQ_ROOT}/installer/install.sh"', LIB.read_text(encoding="utf-8")) + + def test_no_pipe_to_shell_and_no_token_handling(self): + pipe_to_shell = re.compile(r"(curl|wget)[^\n|]*\|\s*(sudo\s+)?(ba|z)?sh\b|Invoke-Expression|\biex\b") + token_read = re.compile(r"(GH_TOKEN|GITHUB_TOKEN|ACCESS_TOKEN|Read-Host\s+-AsSecureString|read\s+-s)") + for script in (UBUNTU, MACOS, LIB, WINDOWS): + text = script.read_text(encoding="utf-8-sig") + self.assertIsNone(pipe_to_shell.search(text), script.name) + self.assertIsNone(token_read.search(text), script.name) + + def test_downloads_are_pinned(self): + text = UBUNTU.read_text(encoding="utf-8") + self.assertEqual(len(re.findall(r'_SUM="[0-9a-f]{64}"', text)), 4) + self.assertIn("sha256sum -c --status", text) + self.assertRegex(text, r'DOCKER_SIGNER_FPR="[0-9A-F]{40}"') + + def test_no_new_exit_codes(self): + known = {line.split("\t")[0] for line in (INSTALLER / "grdm_dev" / "messages.tsv").read_text(encoding="utf-8").splitlines()} + for script in (UBUNTU, MACOS, LIB): + text = script.read_text(encoding="utf-8") + for code in re.findall(r"prereq_(?:fail|stop|step) (\d+)", text): + self.assertIn(code, known, f"{script.name}: exit {code}") + for code in re.findall(r"Stop-Prereq (\d+)", WINDOWS.read_text(encoding="utf-8-sig")): + self.assertIn(code, known, f"{WINDOWS.name}: exit {code}") + + +class WindowsBootScript(unittest.TestCase): + """Runs the bash that install-windows.ps1 sends into WSL, taken from its $bootLines, under bash with a fake + Windows-disk root (PR #44 review r1 R2: a home or checkout symlinked into /mnt/c passed a textual check).""" + + def setUp(self): + self.tmp = tempfile.TemporaryDirectory() + self.addCleanup(self.tmp.cleanup) + self.root = Path(self.tmp.name) + self.win = self.root / "win" + (self.win / "c").mkdir(parents=True) + self.boot = self._boot_text() + + @staticmethod + def _boot_text(): + text = WINDOWS.read_text(encoding="utf-8-sig") + block = text[text.index("$bootLines = @(") + len("$bootLines = @("):] + block = block[:block.index("\n)")] + lines = [] + for entry in block.strip().splitlines(): + entry = entry.strip().rstrip(",") + if entry.startswith("'"): + lines.append(entry[1:-1]) + else: # a PowerShell expandable string: the four substitutions it uses + entry = entry[1:-1].replace("`\"", "\"").replace("`$", "$") + entry = entry.replace("$(ConvertTo-BashWord $Branch)", "develop").replace("$(ConvertTo-BashWord $RepoUrl)", "/nonexistent.git") + entry = entry.replace("$Directory", "rdm").replace("$forwardWords", "--dry-run") + lines.append(entry) + return "\n".join(lines) + "\n" + + def _run(self, home): + env = {"PATH": os.environ.get("PATH", "/usr/bin:/bin"), "HOME": str(home), "GRDM_WSL_WINDOWS_ROOT": str(self.win)} + return subprocess.run([_bash(), "-c", self.boot], env=env, cwd=str(home), capture_output=True, text=True, timeout=60) + + def _checkout(self, path): + (path / ".git").mkdir(parents=True) + (path / "installer").mkdir() + (path / "installer" / "install-ubuntu.sh").write_text('echo "reached install-ubuntu.sh $*"\n') + return path + + def test_symlinked_checkout_on_the_windows_disk_is_refused(self): + home = self.root / "home" + home.mkdir() + (home / "rdm").symlink_to(self._checkout(self.win / "c" / "rdm"), target_is_directory=True) + cp = self._run(home) + self.assertEqual(cp.returncode, 10, cp.stdout + cp.stderr) + self.assertIn("is on the Windows disk", cp.stderr) + self.assertNotIn("reached", cp.stdout) + + def test_symlinked_home_on_the_windows_disk_is_refused(self): + real_home = self.win / "c" / "Users" / "dev" + real_home.mkdir(parents=True) + home = self.root / "home" + home.symlink_to(real_home, target_is_directory=True) + cp = self._run(home) + self.assertEqual(cp.returncode, 10, cp.stdout + cp.stderr) + self.assertIn("the WSL home directory is on the Windows disk", cp.stderr) + + def test_checkout_inside_wsl_is_used(self): + home = self.root / "home" + self._checkout(home / "rdm") + cp = self._run(home) + self.assertEqual(cp.returncode, 0, cp.stdout + cp.stderr) + self.assertIn("reached install-ubuntu.sh --dry-run", cp.stdout) + + +class WindowsScriptText(unittest.TestCase): + """Text-level rules only; `Parser.ParseFile` and the dry run happen on windows-latest (installer-prereq-lint.yml).""" + + def setUp(self): + self.raw = WINDOWS.read_bytes() + self.text = self.raw.decode("utf-8-sig") + + def test_bom_is_present(self): + self.assertTrue(self.raw.startswith(b"\xef\xbb\xbf"), "Windows PowerShell 5.1 needs a BOM to read the Japanese strings") + + def test_consent_and_dry_run_flags(self): + for needle in ("[switch]$Yes", "[switch]$DryRun", "'--yes'", "'--dry-run'", "Confirm-Step", "-Verb RunAs"): + self.assertIn(needle, self.text) + + def test_clone_target_is_a_single_folder_name(self): + self.assertIn(r"$Directory -notmatch '^[A-Za-z0-9._-]+$'", self.text) + self.assertIn('home_real="$(pwd -P)"', self.text) # the physical checks themselves run in WindowsBootScript + + def test_runs_the_ubuntu_script_inside_wsl(self): + self.assertIn("exec bash installer/install-ubuntu.sh", self.text) + self.assertIn("--distribution $Distro --cd '~' --exec bash -c", self.text) + + def test_checks_wsl_integration(self): + self.assertIn("'docker', 'info'", self.text) + self.assertIn("WSL Integration", self.text) + + def test_quotes_and_braces_balance_outside_strings(self): + depth = 0 + quote = None + lines = self.text.splitlines() + in_here = False + for line in lines: + stripped = line.strip() + if in_here: + if stripped.startswith("'@"): + in_here = False + continue + if stripped.endswith("@'"): + in_here = True + continue + i = 0 + while i < len(line): + ch = line[i] + if quote: + if ch == "`" and quote == '"': + i += 2 + continue + if ch == quote: + if i + 1 < len(line) and line[i + 1] == quote: + i += 2 + continue + quote = None + elif ch in "'\"": + quote = ch + elif ch == "#": + break + elif ch == "{": + depth += 1 + elif ch == "}": + depth -= 1 + self.assertGreaterEqual(depth, 0) + i += 1 + self.assertEqual(depth, 0) + self.assertIsNone(quote) + + +class LineEndingRules(unittest.TestCase): + def test_gitattributes_forces_lf_where_linux_runs(self): + text = (REPO / ".gitattributes").read_text(encoding="utf-8") + for pattern in ("*.sh ", "Dockerfile ", "installer/grdm_dev/templates/**", "installer/images/**", "manifest/**"): + self.assertRegex(text, re.escape(pattern) + r"\s*text eol=lf") + self.assertNotRegex(text, r"(?m)^\*\s+text=auto") + + def test_crlf_hint_is_portable(self): + text = (INSTALLER / "install.sh").read_text(encoding="utf-8") + self.assertNotIn('sed -i ""', text) + self.assertIn("perl -pi -e", text) + + +if __name__ == "__main__": + unittest.main() diff --git a/installer/tests/test_scan.py b/installer/tests/test_scan.py index 706c3a7..6c1e355 100644 --- a/installer/tests/test_scan.py +++ b/installer/tests/test_scan.py @@ -10,6 +10,8 @@ import json import os import shutil +import subprocess +import sys import tempfile import unittest from argparse import Namespace @@ -134,6 +136,7 @@ def write_compose(self, services=None, networks=None): services = services if services is not None else {"web": hardened(ports=["127.0.0.1:5000:5000"], networks=["grdm", "grdm-edge"])} doc = {"services": services, "networks": networks if networks is not None else NETWORKS} (self.root / "runtime").mkdir(exist_ok=True) + os.chmod(self.root / "runtime", 0o700) # nosemgrep: python.lang.security.audit.insecure-file-permissions.insecure-file-permissions -- test fixture: the private runtime/ that `secrets init` leaves (self.root / "runtime" / "docker-compose.yml").write_text(json.dumps(doc)) def run_scan(self, runner, which=all_tools, **flags): @@ -686,6 +689,112 @@ def test_full_scan_skips_dynamic_when_unhealthy(self): self.assertEqual(code, exit_codes.OK) self.assertTrue(any("dynamic (ZAP) part skipped" in n for n in data["notes"])) + def test_zap_writes_into_its_own_world_writable_directory_that_is_removed(self): + # DEMO-21 run 3: ZAP (uid 1000) could not write into runtime/reports owned by host uid 2001. + self.write_compose() + ps = json.dumps([{"Service": "web", "State": "running", "Health": "healthy"}]) + base = fake_runner(compose_ps=ps) + seen = [] + + def runner(argv): + if argv[:2] == ["docker", "run"]: + mount = Path(argv[argv.index("-v") + 1].split(":")[0]) + seen.append((mount, oct(mount.stat().st_mode & 0o777))) + return base(argv) + + code, data, _ = self.run_scan(runner, dynamic=True) + self.assertEqual(code, exit_codes.OK, data["tools"]) + reports = self.root / "runtime" / "reports" + self.assertEqual(len(seen), 1) + mount, mode = seen[0] + self.assertEqual(mount.parent, reports) + self.assertEqual(mode, "0o777") + self.assertFalse(mount.exists()) + self.assertEqual(len(list(reports.glob("zap_web_*.json"))), 1) + self.assertEqual(list(reports.glob("zap_wrk_*")), []) + + def _zap_ready(self): + self.write_compose() + return json.dumps([{"Service": "web", "State": "running", "Health": "healthy"}]) + + def test_zap_refuses_a_runtime_that_others_can_enter(self): + # PR #44 review r1 R3: with runtime/ at 0755 the 0777 work directory was reachable by other local users. + ps = self._zap_ready() + os.chmod(self.root / "runtime", 0o755) # nosemgrep: python.lang.security.audit.insecure-file-permissions.insecure-file-permissions -- test fixture: the permissive runtime/ the scan must refuse + calls = [] + code, data, _ = self.run_scan(fake_runner(compose_ps=ps, calls=calls), dynamic=True) + self.assertEqual(code, exit_codes.SCANNER_UNAVAILABLE) + self.assertIn("must be 0700", data["tools"][0]["result"]) + self.assertEqual([c for c in calls if c[:2] == ["docker", "run"]], []) + self.assertEqual(list((self.root / "runtime" / "reports").glob("zap_wrk_*")), []) + + def test_zap_refuses_symlinked_runtime_or_reports(self): + ps = self._zap_ready() + elsewhere = self.root / "elsewhere" + elsewhere.mkdir(mode=0o700) + reports = self.root / "runtime" / "reports" + reports.symlink_to(elsewhere, target_is_directory=True) + calls = [] + code, data, _ = self.run_scan(fake_runner(compose_ps=ps, calls=calls), dynamic=True) + self.assertEqual(code, exit_codes.SCANNER_UNAVAILABLE) + self.assertIn("symbolic link", data["tools"][0]["result"]) + self.assertEqual([c for c in calls if c[:2] == ["docker", "run"]], []) + self.assertEqual(list(elsewhere.glob("zap_wrk_*")), []) + + def test_zap_work_directory_names_are_unpredictable_and_reports_symlinks_are_refused(self): + ps = self._zap_ready() + outside = self.root / "outside.json" + outside.write_text('{"site": [{"alerts": [{"pluginid": "1", "riskcode": "3", "alert": "planted"}]}]}') + base = fake_runner(compose_ps=ps) + mounts = [] + + def runner(argv): + if argv[:2] == ["docker", "run"]: + wrk = Path(argv[argv.index("-v") + 1].split(":")[0]) + mounts.append(wrk.name) + (wrk / argv[argv.index("-J") + 1]).symlink_to(outside) # what another user could plant + return 0, "" + return base(argv) + + code, data, _ = self.run_scan(runner, dynamic=True) + self.assertEqual(code, exit_codes.SCANNER_UNAVAILABLE) + self.assertIn("symlink is refused", data["tools"][0]["result"]) + self.assertNotIn("planted", json.dumps(data["findings"])) + self.assertEqual(len(mounts), 1) + self.assertRegex(mounts[0], r"^zap_wrk_web_\S+_[A-Za-z0-9_]{6,}$") + self.assertTrue(outside.exists()) + + def test_fifo_report_is_refused_without_blocking(self): + # PR #44 review r2 R2-02: os.open on a FIFO with no writer blocked before the type check. Bounded by a child. + fifo = self.root / "report.json" + os.mkfifo(fifo) + code = ("import sys; from pathlib import Path; from grdm_dev import scan; " + "print(scan._read_zap_report(Path(sys.argv[1]))[1])") + cp = subprocess.run([sys.executable, "-c", code, str(fifo)], capture_output=True, text=True, timeout=10, + env=dict(os.environ, PYTHONPATH=str(Path(__file__).resolve().parents[1]))) + self.assertEqual(cp.returncode, 0, cp.stderr) + self.assertIn("not a regular file", cp.stdout) + + def test_zap_does_not_write_into_a_fifo_in_reports(self): + ps = self._zap_ready() + reports = self.root / "runtime" / "reports" + reports.mkdir() + base = fake_runner(compose_ps=ps) + planted = [] + + def runner(argv): + if argv[:2] == ["docker", "run"] and not planted: + name = argv[argv.index("-J") + 1] + os.mkfifo(reports / name) # a FIFO where the masked copy will be saved + # A reader keeps the test bounded: without it a regression would block in open() instead of failing. + planted.append(os.open(reports / name, os.O_RDONLY | os.O_NONBLOCK)) + self.addCleanup(os.close, planted[-1]) + return base(argv) + + code, data, _ = self.run_scan(runner, dynamic=True) + self.assertEqual(code, exit_codes.SCANNER_UNAVAILABLE) + self.assertIn("only a regular file is written", data["tools"][0]["result"]) + def test_dynamic_targets_service_names_on_compose_network(self): self.write_compose({ "web": hardened(ports=["127.0.0.1:5000:5000"], networks=["grdm", "grdm-edge"]),