diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml index 253f705..9630e38 100644 --- a/.github/workflows/ci.yaml +++ b/.github/workflows/ci.yaml @@ -57,6 +57,10 @@ jobs: run: pixi run test - name: Generated field tables and support matrix are current run: pixi run fields-check + - name: EB/KI catalogue fragments are current and every cited ID has an entry + run: pixi run catalog-check + - name: Example figures are current + run: pixi run figures-check gsasii: # GSAS-II suite incl. the slow regressions against the committed references. diff --git a/AGENTS.md b/AGENTS.md index 26cd373..19b5ccf 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,243 +1,150 @@ # AGENTS.md — PowderLine -Guidance for coding agents (and a quick orientation for human contributors). -Tool-specific files (`CLAUDE.md`) only point here. - -Automated powder X-ray diffraction refinement driven from a JSON "recipe". -PowderLine is the application layer over interchangeable refinement engines: -a recipe names **one engine and one workflow** in its `schema_name` -(`gsasii.rietveld`, `gsasii.spf`, `topas.rietveld`, `topas.spf`, -`easydiffraction.rietveld`), is written in that engine's own conventions and -units, and `powderline.run()` / the `powderline` CLI hand it to that engine's -**gateway** and return standardized tables with uniform fit statistics. -Nothing is translated between engines. Example-driven: real refinement cases -drive schema evolution and are the regression references. - -**Authoritative sources for a change:** `docs/policies.md` (versioning, -deliberate changes, frozen code, engine pins, review, "ask, don't -assume"), `docs/SCHEMA_HISTORY.md` (schema facts), `docs/engine-behavior.md` -(what the engines do and PowderLine's policy), `docs/known_issues.md` -(`KI-NN`). Maintainers who work with the private design records (the devkit) -also follow the devkit's own agent guidance. **If a decision is not written down, -stop and ask the maintainer; do not infer it from the code or from another -engine.** - -## Architecture (`src/powderline/`) +A path map for coding agents (and a quick orientation for human +contributors). Tool-specific files (`CLAUDE.md`) only point here. This file +links the page that owns each topic instead of restating it. + +PowderLine runs powder diffraction refinements from a **recipe** (structured +data: a dict, serialised as JSON). A recipe names one engine and one workflow +in `schema_name` (`gsasii.rietveld`, `gsasii.spf`, `topas.rietveld`, +`topas.spf`, `easydiffraction.rietveld`) and is written in that engine's own +terms and units. `powderline.run()` and the `powderline` CLI hand it to the +engine's **gateway** and return standardized tables with uniform fit +statistics. Nothing is translated between engines. + +## Authoritative sources + +- `docs/development/policies.md` — versioning, deliberate changes, frozen + code, engine pins, review, "ask, don't assume". +- `docs/reference/versions.md` — schema versions, history, support matrix. +- `docs/engines/*.md` + `docs/catalog/engine_behaviour.yaml` — what each + engine does and PowderLine's policy (`EB-NN`). +- `docs/reference/known-issues.md` + `docs/catalog/known_issues.yaml` — + known, deliberately deferred behaviours (`KI-NN`). Check before "fixing" a + quirk. +- The code under `src/` and `pixi.toml`: where a page and the code disagree, + the code wins; report the discrepancy. + +**If a decision is not written down, stop and ask the maintainer.** Do not +infer it from the code or from another engine. Maintainers working with the +private devkit also follow its own agent guidance. + +## Code map (`src/powderline/`) Core (engine-free; `import powderline` imports no engine): -- `__init__.py` — the public API, `__all__`: `run`, `validate`, `ValidationResult`, - `load_recipe_asset`, `support_matrix`, `PowderLineError`, - `GatewayNotInstalledError`, `EngineNotAvailableError`, `EngineVersionError`, - `EngineExecutionError`, `ExecutionBackendError`. -- `api.py` — `validate(recipe)` → `ValidationResult(recipe=, - warnings=[...])` through the gateway's engine-free schema layer; - `run(recipe, output_dir, *, verbose, validate_only, **engine_options)` - dispatches on `schema_name` (`.` → `registry.get(engine)`); - engine options are the gateway's explicit keywords (gsasii `execution_mode`; - topas `topas_dir`, `topas_version`, `topas_timeout`); `allow_unsupported_engine_version` is - refused (test-only, on the gateway's own `run`). -- `kicker.py` — the one CLI: `powderline [--output DIR] - [--validate-only] [--verbose] [--use-server|--no-server] [--server-timeout SECONDS] - [--topas-dir DIR] [--topas-version V] [--topas-timeout SECONDS]` and `powderline --doctor` (`doctor.py`: gateways, - engine versions vs pins, server state; non-zero on a stale install, KI-13); - `pixi run kicker` / `pixi run doctor` run the same `main`. Refuses - templates; an engine flag for another engine's recipe exits 1. -- `registry.py` — gateways from the `powderline.gateways` entry-point group - (lazy `ep.load`), the frozen `Gateway` dataclass (`name`, `capabilities`, - `validate`, `run`), `register_gateway` escape hatch, `check_engine_version`. - Entry points only; no in-code fallback (KI-13). -- `schema_core.py` — core schema **0.1.0**: `CoreRecipe` frame - (`schema_name`, `core_schema_version`, `engine_schema_version`, `metadata` - ≤ 1 MiB, `payload`), `RefinableParameter` `[value, flag]` / - `BoundedRefinableParameter` `[value, flag, min, max]`, `XRDData`, `FitRange`, - `ChebyshevBackground`, the phase block `Phase[P]` (`space_group` canonical - gemmi name, `unit_cell`, `atoms`; every symmetry rule: cell check, special - positions, tie groups and flags, derived values held exactly), `CoreName`, - `collect_warnings`, `parameters_requested`, `fit_limits_on_data`, and the - engine schema building blocks (`EngineRecipe`, `recipe_validator`, - `refined_flags`, `custom_error`/`raise_problems`, `omit_absent`, - `absent_not_null`, `unique_phase_names`, `fit_window`, `fit_range_problems`, - `outside_window_problems`, `refined_flag_problems`, `unit_schema`) every - gateway schema is composed from. Every - model `extra='forbid'`; units in `json_schema_extra={"unit": ...}`. -- `symmetry.py` (gemmi), `compat.py` (declared schema compatibility, - `support_matrix()`), `fitstats.py` (uniform Rwp/Rexp/GoF/χ²_red), `reports.py` - (standardized report writers), `recipes.py`, `exceptions.py` (also - `WARNING_CODES`: every warning code with owner and level, asserted against - `src/` and the engine-behaviour table), - `config_loader.py` (simulation defaults), `_status.py`. -- `results.py` — the run-result envelope (`ENVELOPE_VERSION`, `REASONS`, the - failed/generated envelopes, `raise_if_failed`); `roundtrip.py` — the output - recipe (`output_recipe`, `output_recipe_valid`, `parameter_dispositions`; - tie propagation, KI-04); `doctor.py` — the `--doctor` report (KI-13). - -Gateways (`src/powderline/gateways//`), each a **schema layer** -(`schema.py`: the engine schema 0.1.0 on core 0.1.0, `DECLARATIONS`, -`SCHEMA_NAMES`, `validate_recipe`; imports no engine) plus a **runtime -layer** (`gateway.py` factory `get_gateway()` with `capabilities()`, -`validate()`, `run()`, `ENGINE_VERSION_SPEC`, `ENV`/`INSTALL_COMMAND`; engine -imported lazily from `run()`): -- `gsasii/` (GSAS-II `==5.7.9`, env `gsasii`): the **frozen** setup/execution - code `helpers.py`, `project.py`, `setters.py`, `executors.py`, - `extractors.py`, `constraints.py` (moved verbatim from `kicker.py` at the - verified baseline `daace24`; code-hashed); non-frozen `routing.py`, `fit_report.py` - (uniform statistics, `engine_details`, convergence, divergence = failure), - `sites.py` (GSAS-II's reading of each site vs core's), `report_rows.py`, - `paths.py` (GSAS-II parameter names → recipe paths for the output recipe), - `console.py` (the engine's console tail into a failed envelope, KI-05), - `scattering.py` (GSAS-II's ion table, `La3+` → `La+3`); - `server.py` / `client.py` (persistent FastAPI server + `GSASClient`, the - `auto`/`server` execution modes; per-user state dir, token-signed requests - and responses, `POWDERLINE_NO_SERVER=1` disables it — v0.1.2, KI-14; output files travel - back inside the response, no shared filesystem — KI-12); `simulation_builder.py` - (a crystal-structure dict → a `gsasii.rietveld` simulation recipe; the Materials Project - fetch/export tool that used it left PowderLine in 0.2.0; RandomXRD carries its own). -- `topas/` (TOPAS 6; recipes state `engine_version: "6"`, runs need the - installed version declared): `native_writer.py` (model → INP + `.xye`; - limits on data points; `= p/q;` constants), `native_run.py` (run `tc.exe`, - strict file reads, the background-only second run, result), `peak_list.py`, - `runner.py`, `scattering.py` (TOPAS 6's ion table, `La3+` → `occ La+3`). - No Python engine: generate-only - without `tc.exe`. -- `easydiffraction/` (`==0.21.1`, env `easydiffraction`, Python ≥ 3.12): - `native_builder.py`, `native_run.py`, `space_group_table.py`, - `space_group_support.py` (per-calculator verified settings), `scattering.py` - (CrysPy's ion table). - -Docs: `docs/schemas.md`, `docs/fields.md` (generated tables), `docs/input-guidance.md`, -`docs/engine-behavior.md`, `docs/policies.md`, `docs/migration-0.1-to-0.2.md`, -`docs/support-matrix.md`, `docs/cross-engine-comparison.md`, `docs/DEVELOPMENT.md` -("Adding a refinement engine"). Scripts: `scripts/gen_schema_field_tables.py` -(field tables + support matrix → `docs/_generated/`), `scripts/cross_engine_comparison.py`, -`scripts/convert_recipe_026.py` (0.26.0 → `gsasii.*` migration tool; -fixtures `tests/data/convert_026/`), `scripts/update_code_hash.py`. -## Key commands (pixi) - -Environments: `default` = core only (validate anything, run nothing); -`gsasii`, `easydiffraction`, `all` (both engines), `dev` (all + docs/test -tooling). `pixi install -e `. -- `pixi run kicker --validate-only` — any env; engine-free. -- `pixi run -e gsasii kicker ` — run a `gsasii.*` recipe - (`--use-server`/`--no-server`, `--output DIR`, `--verbose`); - `pixi run -e easydiffraction kicker ` for `easydiffraction.*`; - `pixi run kicker --topas-version 6 [--topas-dir DIR] - [--topas-timeout SECONDS]` for `topas.*` (runs `tc.exe` on Windows; when no - `tc.exe` is found the run is generate-only: the INP and `.xye` are written, - the result has status `generated` and the CLI exits 0). - After `pixi reinstall powderline` the installed command is `powderline …`. -- `pixi run test` — the engine-free suites (`core or topas_gen`, any env); - `pixi run -e gsasii test-gsasii`, `pixi run -e easydiffraction - test-easydiffraction` per engine; `pixi run test-topas` — `topas_gen` + - `topas_exec` (the exec tests run `tc.exe` only when `POWDERLINE_TOPAS_DIR` - and `POWDERLINE_TOPAS_VERSION` are set, on the TOPAS Windows machine; else - they skip); `pixi run test-all` — everything (`dev`). -- `pixi run -e gsasii gsas-server start|status|stop|restart|logs` — the - persistent GSAS-II server (task in the `gsasii` feature; without `-e` pixi - reports it ambiguous). -- `pixi run docs` / `pixi run docs-clean` — Sphinx HTML with `-W` (zero warnings). -- `PYTHONPATH=src python scripts/gen_schema_field_tables.py [--check]` — after - editing a schema module or a gateway's declarations (regenerates - `docs/_generated/fields_*.md`, `support_matrix.md`); `pixi run fields-check` - runs the `--check` form (fails when the committed fragments are stale). -- `pixi reinstall powderline --all` — after editing `[project.entry-points]` - or `[project.scripts]` in `pyproject.toml`: entry-point metadata is written - at install time, per env; the registry finds nothing until the editable - package is re-installed (KI-13). -- `pixi run update-code-hash` — regenerate `src/powderline/_code_hash.json` - after a **signed** edit to a frozen gsasii module (guarded by `tests/test_code_hash.py`). - -## Frozen GSAS-II code (the rule stays) +- `__init__.py` — public API (`__all__`): `run`, `validate`, `ValidationResult`, `load_recipe_asset`, `support_matrix`, the exceptions. +- `api.py` — `validate()` / `run()`: `schema_name` → `registry.get()`; clears fixed outputs, writes `failure.json`; refuses `allow_unsupported_engine_version`. +- `kicker.py` — the CLI (`powderline`); refuses templates; an engine flag for another engine's recipe exits 1. +- `doctor.py` — `powderline --doctor` (KI-13: stale install). +- `registry.py` — entry-point group `powderline.gateways`, `Gateway` dataclass, `register_gateway`, `check_engine_version`, `validate_only_summary`. No in-code fallback. +- `schema_core.py` — core schema 0.1.0: `CoreRecipe`, `RefinableParameter` / `BoundedRefinableParameter`, `XRDData`, `FitRange`, `ChebyshevBackground`, `Phase[P]`, `EngineRecipe`, `recipe_validator`, the building blocks. +- `symmetry.py` — gemmi: canonical space groups, cell ties, special positions. +- `compat.py` — declared schema compatibility, `schema_declarations`, `support_matrix`. +- `fitstats.py` — uniform Rwp / Rexp / GoF / χ²_red. +- `reports.py` — report writers, `FIXED_OUTPUT_NAMES` / `FIXED_OUTPUT_PATTERNS`, `clear_fixed_outputs`. +- `results.py` — result envelope, `REASONS`, `raise_if_failed`. +- `roundtrip.py` — output recipe and parameter dispositions. +- `exceptions.py` — exceptions and `WARNING_CODES`. +- `recipes.py`, `config_loader.py` (`.powderline_config.yaml`), `_status.py`, `_code_hash.json`. + +Gateways (`src/powderline/gateways//`): schema layer `schema.py` +(`DECLARATIONS`, `SCHEMA_NAMES`, `validate_recipe`; imports no engine) + +runtime layer `gateway.py` (`get_gateway()`, `ENGINE_VERSION_SPEC`, `ENV` / +`INSTALL_COMMAND`; engine imported lazily from `run()`). + +- `gsasii/` (GSAS-II `==5.7.9`, env `gsasii`) — frozen: `helpers`, `project`, `setters`, `executors`, `extractors`, `constraints`; around them: `routing`, `fit_report`, `sites`, `paths`, `report_rows`, `console`, `scattering`, `server` / `client` (persistent server, `POWDERLINE_NO_SERVER=1` disables it), `simulation_builder`. +- `topas/` (TOPAS 6, no Python engine; generate-only without `tc.exe`) — `native_writer`, `native_run`, `peak_list`, `runner`, `scattering`. +- `easydiffraction/` (`==0.21.1`, env `easydiffraction`, Python ≥ 3.12) — `native_builder`, `native_run`, `space_group_table`, `space_group_support`, `scattering`. + +Docs (`docs/`, MyST): `getting-started/`, `concepts/`, `how-to/`, +`examples/`, `engines/`, `reference/`, `development/`, `about/`; +`_generated/` (script output, never hand-edited); `catalog/` (EB/KI data). +`docs/reference/messages.md` holds the warning-codes table that +`tests/test_warning_codes.py` reads. + +Scripts (`scripts/`): `gen_schema_field_tables.py`, `gen_catalog.py`, +`gen_example_figures.py`, `cross_engine_comparison.py`, +`convert_recipe_026.py` (0.26.0 → `gsasii.*`), `check_engine_version_candidate.py`, +`update_code_hash.py`, `verify_easydiffraction_settings.py`, +`generate_easydiffraction_space_groups.py`, `hooks/pre-commit`. + +## Key commands + +Environments: `default` (core only: validates everything, runs nothing), +`gsasii`, `easydiffraction`, `all`, `dev` (all + docs and test tooling). +Install with `pixi install --frozen -e `. + +- `pixi run powderline --validate-only` — any env. +- `pixi run -e gsasii powderline [--use-server|--no-server] [--server-timeout S] [--output DIR] [--verbose]`. +- `pixi run -e easydiffraction powderline `. +- `pixi run powderline --topas-version 6 [--topas-dir DIR] [--topas-timeout S]` — without `tc.exe`: status `generated`, exit 0. +- `pixi run powderline --doctor`. The tasks `kicker` and `doctor` run the same `main`. +- `pixi run test` (engine-free: `core or topas_gen`); `pixi run -e gsasii test-gsasii`; `pixi run -e easydiffraction test-easydiffraction`; `pixi run test-topas` (runs `tc.exe` only with `POWDERLINE_TOPAS_DIR` and `POWDERLINE_TOPAS_VERSION` set); `pixi run test-all` (`dev`). +- `pixi run -e gsasii gsas-server start|status|stop|restart|logs`. `start` and `restart` run the server in the foreground until stopped; to replace a stale server, `stop` it and the next run starts a fresh one. +- `pixi run docs` / `pixi run docs-clean` — Sphinx, `-W`, zero warnings. +- `pixi run fields-check`, `pixi run catalog-check`, `pixi run figures-check` — generated docs are current; regenerate with the matching script without `--check`. +- `pixi run update-code-hash` — after any edit to a frozen gsasii module. +- `pixi reinstall powderline --all` — after editing `[project.entry-points]` or `[project.scripts]` (KI-13). + +Detail: `docs/development/contributing.md`. + +## Frozen GSAS-II code `src/powderline/gateways/gsasii/{helpers,project,setters,executors,extractors,constraints}.py` -are **move-only**: they do GSAS-II's project-dictionary work exactly as the -pre-refactor code did. Never change what they do to the GSAS-II project dict. -Additive or input-side edits — and output-side control flow that leaves the -project-dictionary work untouched (the failure propagation, the checks -before any result table is written) — go **only** through the maintainers' -signed-off record of frozen-code edits, each entry with its proof, then -`pixi run update-code-hash`. An agent never makes such an edit on its own: -propose it to the maintainer. New behaviour goes in -the non-frozen modules (`fit_report.py`, `sites.py`, `routing.py`, -`gateway.py`). Restart the GSAS-II server after any edit it runs (the client -compares the server's source-tree hash with its own and refuses a stale one). - -## Engines / environment notes - -- `GSAS-II` is built as a **conda package via `pixi-build`** from the - `tacaswell/GSAS-II@enh/pixi_pkg` fork (pinned to a commit in `pixi.toml`, - feature `gsasii`); it resolves to **v5.7.9**, which the gateway's - `ENGINE_VERSION_SPEC` checks at run time. easydiffraction is pinned - `==0.21.1`; gemmi `==0.7.5` (the canonical space-group names depend on it). - Moving a pin is a deliberate change with a compatibility run - (`docs/policies.md`): GSAS-II (or scipy/BLAS) changes shift refined values - at the last digit, so `refined_parameters.csv` and the unit-cell reports are - compared within tolerance (`docs/regression-tolerance.md`; - `tests/subprocess_utils.py` `refined_params_match` / `unit_cell_match`) and - peak lists by row count. TOPAS and easydiffraction goldens live under - `tests/data/{topas,easydiffraction}/native/golden/` (TOPAS's by version, - `golden/6/`); TOPAS runs only in the `topas_exec` tests on the TOPAS machine - (goldens from the TOPAS machine). -- The schema layer of every gateway, `powderline.api` and `powderline.kicker` - import no engine — enforced by subprocess tests that block the engine - imports. Keep it that way: engine imports live inside `run()`-time code. -- Platforms: `linux-64`, `win-64`, `osx-arm64`. **All functionality must work - on Linux and Windows** (sole exception: TOPAS *execution* needs `tc.exe` on - win-64). Avoid Unix-only shell syntax in pixi tasks — tasks set `PYTHONPATH` - through the task-level `env` dict; see `docs/cross-platform-guide.md`. -- **Testing caveat**: some `test_esd_extraction` / `test_example_*_regression` - cases require a working GSAS-II refinement; they can fail for environment - reasons unrelated to a given change. Compare against the baseline rather - than assuming a failure is new. +do GSAS-II's project-dictionary work. Rules (`docs/development/policies.md#frozen-gsas-ii-code`): + +- A behaviour-changing edit needs the maintainers' explicit consent. An agent + never makes one on its own: propose it, with its reason and proof. +- Comment and docstring edits need no consent. Don't leave outdated comments. +- After **any** edit, `pixi run update-code-hash` (the hash covers raw bytes; + `tests/test_code_hash.py` guards it). +- New behaviour goes into the non-frozen modules (`fit_report`, `sites`, + `routing`, `paths`, `gateway`). +- Replace the GSAS-II server after any edit it runs (a stale server is + detected: skipped in `auto` mode, refused in `server` mode): + `gsas-server stop`; the next run starts a fresh one. + `gsas-server restart` blocks in the foreground, so never chain a command + after it. + +## Environment notes + +- GSAS-II is a conda package built by `pixi-build` from a fork pinned to a + commit in `pixi.toml`; it resolves to v5.7.9. Use `--frozen` so it is not + re-locked. +- Pins: GSAS-II `==5.7.9`, easydiffraction `==0.21.1`, gemmi `==0.7.5`. Moving + a pin is a deliberate change with a compatibility run (policies page). +- Regression references: `examples/**/output/` (gsasii, easydiffraction), + compared within tolerance (`tests/subprocess_utils.py` + `refined_params_match` / `unit_cell_match`); goldens under + `tests/data/{topas,easydiffraction}/native/golden/` (TOPAS by version). +- The schema layers, `powderline.api` and `powderline.kicker` import no engine; + subprocess tests enforce it. +- Platforms: `linux-64`, `win-64`, `osx-arm64`. Everything works on Linux and + Windows except TOPAS execution (`tc.exe`, Windows). pixi tasks set + `PYTHONPATH` through the task-level `env` dict; no Unix-only shell syntax. +- Some GSAS-II regression and ESD tests can fail for environment reasons: + compare against a baseline run on `main`. ## Conventions -- Parameters are lists: `[value, refine_flag]` in `gsasii.*`, - `[value, refine_flag, min, max]` in `topas.*` and `easydiffraction.*`; the - flag is a JSON boolean; `null` bounds mean open. Blocks that may be left out - are left out, never `null`; unknown keys are errors. -- **Validation never changes a recipe beyond its stated tolerance, and never - silently**: anything an engine would silently alter is a validation error - naming the value to write; a value stated within tolerance of its exact - symmetry-derived form is *held* in that form and reported - (`symmetry_value_derived`, level `info`); documented defaults (gsasii - size/strain 10 µm / 0, background none, 5 cycles) and informative findings - are structured warnings `{code, message, field_path[, level]}` (`level` is - `"warning"` when absent, or `"info"`). -- Space groups are gemmi's canonical names (`"P m -3 m"`, `"F d -3 m:2"`); - every member of a symmetry tie group is stated, with the same flag. -- Results: uniform `rwp`/`r_exp`/`gof`/`chi2_red` from core `fitstats` over the - window's weighted points; the engine's own values and - `parameters_requested`/`parameters_varied` under `engine_details`; every - result carries `status` (`ok` / `generated`), `converged`, `output_recipe` + - `output_recipe_valid` + `parameter_dispositions` (KI-04: the resolved - recipe with refined values written back, built by `powderline.roundtrip`) and - `envelope_version` (no `success` key: `status` is the outcome); **every failure - raises**: `EngineExecutionError` (the engine ran and failed) or - `ExecutionBackendError` (it did not run), with `.reason` from - `powderline.results.REASONS` and `.result` the failed envelope (the - GSAS-II server speaks the same envelope). -- Output uses fixed filenames (`refined_parameters.csv`, - `_unit_cell_report.csv`, `_peak_list_report.csv`, - `fit_profile.txt`, `output_recipe.json`, `parameter_dispositions.csv`; - gsasii `dummy.gpx`/`dummy.lst`; topas `.inp` etc.). - `unit_cell_data`/`peak_list_data` from `run()` are dicts keyed by phase name. - `powderline.run()` removes PowderLine's own fixed-name files from the output - directory before every run (`reports.clear_fixed_outputs`) — only once the - recipe has validated and the engine is importable, so a typo or a wrong - environment costs nothing — and, when a run fails, writes the failed envelope - to `failure.json` beside the engine's own files; no result table is written - for a failed run on any engine. -- Known, deliberately-deferred behaviours are registered with stable `KI-NN` - ids in `docs/known_issues.md` — check it before "fixing" a quirk; engine - quirks and PowderLine's policy are on `docs/engine-behavior.md`. -- Every intentional behaviour change is a deliberate change (recorded, - reviewed, in the CHANGELOG); regression references under - `examples/**/output/` **are** tracked and move only with an equivalence - record — do not gitignore or regenerate them casually. +- Parameter shapes, bounds, units, unknown keys, blocks left out (never + `null`) — `docs/reference/recipe-format.md`. +- Validation never changes a recipe silently; space groups are gemmi's + canonical names; tie groups are stated in full — `docs/concepts/validation.md`. +- Structured warnings `{code, message, field_path[, level]}`; every code in + `exceptions.WARNING_CODES` and `docs/reference/messages.md`. +- Result keys, report files, `failure.json`, failure reasons — + `docs/reference/outputs.md`; uniform statistics and the output recipe — + `docs/concepts/results.md`. +- Every failure raises `EngineExecutionError` or `ExecutionBackendError`; a + gateway never returns a failed result — `docs/development/adding-an-engine.md`. +- Every intentional behaviour change is deliberate; references move only with + an equivalence check — `docs/development/policies.md`. +- Docs follow the writing rules in + `docs/development/contributing.md#writing-documentation`. - Don't commit secrets: `.powderline_config.yaml` is gitignored. -- `_dev/` is a reserved, gitignored path (the maintainers' private design - records). Never `git add` it or anything under it. +- `_dev/` is reserved and gitignored. Never commit it or anything under it. - Never commit TOPAS's own files (console text, `tc.log`, `TOPAS.INC`, `ATMSCAT.CPP`): they are licensed with TOPAS. @@ -247,15 +154,15 @@ PowderLine follows the NSLS-II policy for AI coding assistants: - A human is responsible for every change, reviews every line, and confirms that the tests passed. AI output gets no shortcut through review. -- Agents work on a feature branch, commit **unsigned** if they commit at all, - and **never** sign, push, tag, force-push or rewrite published history; the +- Agents work on a feature branch and commit unsigned, if they commit at all. + They never sign, push, tag, force-push or rewrite published history; the maintainer signs and pushes. - Commit messages follow Conventional Commits and end with one trailer line per model that contributed: `Assisted-by: :` (e.g. `Assisted-by: claude-code:claude-opus-5-5`). Never `Co-authored-by` for an AI, never `Signed-off-by` on a human's behalf. - Never put secrets, tokens or credentials in the repository or in a - prompt; `.powderline_config.yaml` is gitignored for that reason. + prompt. - Re-run every reported test count yourself before stating it, and compare against the baseline rather than assuming a failure is new. - Before calling documentation complete, grep for the old wording. diff --git a/CHANGELOG.md b/CHANGELOG.md index 416486b..ab3a2d3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,7 +5,7 @@ follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and PowderLine versions follow [Semantic Versioning](https://semver.org/). The recipe **schemas** are versioned separately from the package and have their -own history, [`docs/SCHEMA_HISTORY.md`](docs/SCHEMA_HISTORY.md). +own history, on the [versions page](docs/reference/versions.md). ## [Unreleased] @@ -14,15 +14,15 @@ own history, [`docs/SCHEMA_HISTORY.md`](docs/SCHEMA_HISTORY.md). PowderLine 0.2.0 is a clean break from the unified recipe schema 0.26.0: a recipe names one engine and is written in that engine's own conventions, and nothing is translated between engines. Recipes written for 0.1.x do not -validate; see the [migration guide](docs/migration-0.1-to-0.2.md). Known -limitations are listed in the [known-issues register](docs/known_issues.md). +validate; see the [migration guide](docs/development/migrating-from-0.1.md). Known +limitations are listed in the [known-issues register](docs/reference/known-issues.md). ### Highlights - **One recipe, one engine:** `gsasii.*`, `topas.*` and `easydiffraction.*` recipes, each in its engine's own names and units, through one API and one CLI. - **New schemas, starting at 0.1.0:** a core schema plus one schema per engine - replace the single 0.1–0.26.0 schema ([schemas](docs/schemas.md)). + replace the single 0.1–0.26.0 schema ([engines](docs/engines/index.md)). - **Validation states what the engine computes:** symmetry, engine floors and silent defaults become errors or structured warnings, never silent changes. - **Uniform fit statistics:** `rwp`, `r_exp`, `gof`, `chi2_red` are computed by @@ -34,7 +34,7 @@ limitations are listed in the [known-issues register](docs/known_issues.md). ### Upgrading -- Read the [migration guide](docs/migration-0.1-to-0.2.md): a 0.26.0 recipe +- Read the [migration guide](docs/development/migrating-from-0.1.md): a 0.26.0 recipe (`GSASII_Rietveld` / `GSASII_SPF`) is refused at `schema_name`. - Convert gsasii recipes with `python scripts/convert_recipe_026.py IN.json -o OUT.json`; it reports every change and stops where 0.1.1's refinement cannot @@ -52,10 +52,10 @@ limitations are listed in the [known-issues register](docs/known_issues.md). - Engine schemas `gsasii.rietveld` / `gsasii.spf` (GSAS-II 5.7.9), `topas.rietveld` / `topas.spf` (TOPAS 6) and `easydiffraction.rietveld` - (easydiffraction 0.21.1), on a shared core phase block ([fields](docs/fields.md)). + (easydiffraction 0.21.1), on a shared core phase block ([fields](docs/reference/fields.md)). - Declared schema compatibility: each gateway states the core and engine schema versions it accepts, `validate()` enforces them, and `powderline.support_matrix()` - lists them ([support matrix](docs/support-matrix.md)). + lists them ([support matrix](docs/reference/versions.md)). - `powderline.validate()`, engine-free, returning `ValidationResult(recipe, warnings)`; an invalid recipe raises pydantic's `ValidationError` listing every problem at its field path. @@ -74,7 +74,7 @@ limitations are listed in the [known-issues register](docs/known_issues.md). with value and ESD, by GSAS-II's own name. - Ionic `scattering_types` (CIF spelling, e.g. `La3+`) per atom label in the gsasii and easydiffraction (`cryspy`) phase blocks, checked against each - engine's ion table ([input guidance](docs/input-guidance.md); EB-94–EB-96). + engine's ion table ([scattering types by engine](docs/engines/index.md); EB-94–EB-96). - March–Dollase `preferred_orientation` (`hkl`, `ratio`) in the gsasii and easydiffraction (`cryspy`) phase blocks (EB-97, EB-98). - TOPAS accepts ionic `scattering_types` (TOPAS 6's table, 113 ions, written as @@ -95,16 +95,26 @@ limitations are listed in the [known-issues register](docs/known_issues.md). - Examples: per-engine recipes under `examples//{topas,easydiffraction}/`, `example_LaB6_temperature_series` (sequential refinement), `example_LaB6_ionic` and `example_LaB6_textured`, with easydiffraction references. -- Documentation: [schemas](docs/schemas.md), [fields](docs/fields.md), [input - guidance](docs/input-guidance.md), [engine behaviour](docs/engine-behavior.md), - [cross-engine comparison](docs/cross-engine-comparison.md), [policies](docs/policies.md). -- Engine citations in `CITATION.cff` and on the [citing page](docs/citing.md) (#12). +- Documentation, organised by reader: Get started, Concepts, How-to guides, + Examples, Engines, Reference and Development, on pydata-sphinx-theme with + diagrams and per-engine tabs ([docs](https://nsls2.github.io/PowderLine/)). + Among its pages: [outputs](docs/reference/outputs.md), + [fields](docs/reference/fields.md), [validation](docs/concepts/validation.md), + [engine behaviour](docs/engines/index.md), + [cross-engine comparison](docs/concepts/comparing-engines.md) and + [policies](docs/development/policies.md). +- Every engine behaviour (`EB-NN`) and known issue (`KI-NN`) has its own + entry and link, listed in the [ID index](docs/reference/ids.md). Known + issues KI-19 to KI-21 cover the GSAS-II server's lifetime, its state + directory and a race in `gsas-server stop` + ([known issues](docs/reference/known-issues.md)). +- Engine citations in `CITATION.cff` and on the [citing page](docs/about/citing.md) (#12). ### Changed - The recipe schema is split: the single 0.1–0.26.0 schema is replaced by a core schema and one schema per engine, each starting at **0.1.0** and told - apart by `schema_name`; the 0.x bump rule is in [policies](docs/policies.md). + apart by `schema_name`; the 0.x bump rule is in [policies](docs/development/policies.md). - A recipe names its engine and workflow in `schema_name` (`.`) and states that engine's own terms and units; unknown keys are errors at every level. @@ -178,6 +188,14 @@ limitations are listed in the [known-issues register](docs/known_issues.md). the LaB6 examples drop their undeterminable background peak. - The simulation recipe builder writes SH/L 0.002 (GSAS-II's floor) and emits a `gsasii.rietveld` 0.1.0 recipe. +- CLI and GSAS-II server messages name the current commands: `pixi run + powderline …`, `pixi install --frozen -e `, `pixi run -e gsasii + gsas-server stop` for a stale server (the next run starts a fresh one), + `pixi run -e gsasii gsas-server start`, and `pixi reinstall powderline + --all`. +- Messages say what they mean: an atom near a special position reports the + distance to its symmetry image, and the GSAS-II background-default warning + says GSAS-II's own default background is fixed. ### Removed diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index b51b213..c10ccca 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,76 +1,37 @@ # Contributing to PowderLine -Thanks for your interest! This page is the map; the detail lives in the linked -guides. +Thanks for your interest. This page is the short version. The full guide is +the [Development section](https://nsls2.github.io/PowderLine/development/) of +the docs: architecture, setup, tests, the docs build, adding an engine, and +the policies. -## Dev setup +## Set up ```bash -git clone https://github.com/NSLS2/PowderLine.git -cd PowderLine -pixi install -e dev # every engine + docs/test tooling (GSAS-II is built as a conda package) -pixi run test # full suite (dev env) — run it before and after your change -pixi run docs # Sphinx with -W; zero warnings is the standard +pixi install --frozen -e dev # every engine + docs and test tooling +git config core.hooksPath scripts/hooks # the pre-commit guard, once per clone +pixi run test # the engine-free suites ``` -The default environment is core-only (it validates recipes for every engine -and runs none); `pixi install -e gsasii` / `-e easydiffraction` install one -engine. After pulling a change to `pyproject.toml`'s entry points, run -`pixi reinstall powderline --all`. - -[pixi](https://pixi.sh) is the only tool you install yourself. See the -[README](README.md) for platforms and the [Quickstart](docs/quickstart.md) for -a first refinement run. - -## Where things are - -- **[docs/DEVELOPMENT.md](docs/DEVELOPMENT.md)** — developer guide: - architecture (core + gateways), the schema layer, **how to add a refinement - engine or a capability**, GSAS-II integration, workflow. -- **[docs/policies.md](docs/policies.md)** — what binds every change: - versioning and declared compatibility, deliberate behaviour changes, the - frozen GSAS-II modules, engine-version pins, review, "ask, don't - assume". -- **[docs/engine-behavior.md](docs/engine-behavior.md)** — what each engine - does with a recipe and PowderLine's policy toward it. -- **[docs/known_issues.md](docs/known_issues.md)** — the known-issues register - (`KI-NN` ids, referenced from code and docs): check it before "fixing" a - quirk. See also [docs/regression-tolerance.md](docs/regression-tolerance.md) - (why numeric comparisons are tolerance-based) and - [docs/cross-platform-guide.md](docs/cross-platform-guide.md). -- **[examples/CONTRIBUTING_EXAMPLES.md](examples/CONTRIBUTING_EXAMPLES.md)** — - how to contribute a new example recipe (examples drive schema evolution). - -## Design records - -Maintainers keep design records in a separate private repository. -`_dev/` is reserved and gitignored; never commit it. - -Dev-doc scaffold paths (`docs/superpowers/`, `docs/plans/`, `docs/specs/`, -`docs/dev/`), `.gpx` binaries, `fort.*` scratch, and `_dev/` are all rejected by -the pre-commit guard so they never reach `main`. +[pixi](https://pixi.sh) is the only tool you install yourself. ## Ground rules -- Branch off `main`; keep commits reviewable and self-contained. -- Enable the pre-commit guard once per clone: - `git config core.hooksPath scripts/hooks`. It blocks accidental commits of - data files outside the sanctioned dirs, `.gpx` binaries, and the private - `_dev/` workspace. -- Run `pixi run test` before opening a PR. Some regression cases need a working - GSAS-II refinement — compare failures against a baseline run, don't assume - they're yours. +- Branch off `main` and keep commits reviewable and self-contained. +- Run the tests before and after your change, and compare against `main`: + some GSAS-II regression cases can fail for environment reasons. +- Keep `pixi run docs` free of warnings. After editing a schema or a gateway's + declarations, regenerate the field tables (`pixi run fields-check` tells you + when they are stale). - The frozen GSAS-II modules (`src/powderline/gateways/gsasii/{helpers,project,setters,executors,extractors,constraints}.py`) - change only through the maintainers' signed-off record of frozen-code - edits; after such an edit run `pixi run update-code-hash`. Restart the GSAS-II server after - editing anything it runs (`pixi run -e gsasii gsas-server restart`). -- After editing a schema module or a gateway's version declarations, run - `PYTHONPATH=src python scripts/gen_schema_field_tables.py` and commit the - regenerated `docs/_generated/` fragments. -- If a decision is not written down (policies page, schema history), ask the maintainer rather than inferring it. -- `examples/**/output/` is tracked on purpose (regression references) — don't - regenerate it casually. + change behaviour only with the maintainers' explicit consent. Comment and + docstring edits are fine. After any edit, run `pixi run update-code-hash`. +- `examples/**/output/` is tracked on purpose (regression references). It + moves only as a deliberate, reviewed change. +- If a decision is not written down, ask the maintainers rather than infer + it. +- AI-assisted commits end with an `Assisted-by: :` trailer. - Questions and bug reports: open a GitHub issue. ## Changelog diff --git a/README.md b/README.md index dd32838..a56898f 100644 --- a/README.md +++ b/README.md @@ -1,258 +1,112 @@ # PowderLine -[![Docs](https://img.shields.io/badge/docs-live-blue)](https://nsls2.github.io/PowderLine/) - -Automated powder X-ray diffraction refinement. You describe a refinement in a -JSON "recipe"; PowderLine runs it through the refinement engine the recipe -names — [GSAS-II](https://gsasii.github.io/), Bruker TOPAS 6, or the -open-source [easydiffraction](https://easydiffraction.org) — and returns -standardized result tables. - -**Documentation:** [nsls2.github.io/PowderLine](https://nsls2.github.io/PowderLine/) -· **New here? Start with the [Quickstart](docs/quickstart.md).** - -PowderLine gives you a programmatic, reproducible way to run Rietveld refinements -and single-peak fits without hand-driving the underlying refinement software. - -- **One recipe = one engine.** A recipe's `schema_name` names the engine and - workflow (`gsasii.rietveld`, `gsasii.spf`, `topas.rietveld`, `topas.spf`, - `easydiffraction.rietveld`) and is written in that engine's own - conventions and units; nothing is translated between engines. -- **Refine what you need:** unit cell, atom coordinates, occupancies, isotropic and - anisotropic displacement parameters, background, peak broadening and - March–Dollase preferred orientation, with ionic scattering types per atom — - each parameter individually flagged, and checked against the space-group - symmetry before any engine runs. -- **Honest statistics:** every engine returns the same uniform Rwp / Rexp / - GoF over the same data points, with the engine's own values kept alongside. -- **Strict validation, engine-free:** a recipe validates in a core-only - environment; every problem is reported at its field, and engine quirks are - refused or warned about, never papered over. -- **Standardized outputs:** fixed-filename CSV/TXT reports plus each engine's - native artifacts (the GSAS-II `.gpx`, the TOPAS `.inp` and output files). - -## 1. Install +**A complete powder diffraction refinement, as structured data that any +person, script or machine can write, check and run.** + +[Documentation](https://nsls2.github.io/PowderLine/) · +[Get started](https://nsls2.github.io/PowderLine/getting-started/) · +[How it works](https://nsls2.github.io/PowderLine/concepts/how-powderline-works.html) · +[Paper](https://doi.org/10.48550/arXiv.2608.17009) + +## What and why + +Rietveld refinement extracts structure, phase fractions and microstructure from +powder data. Doing it reliably takes expertise in a particular program, and +doing it at scale has meant custom code for each project. No refinement +program offered a single, validated, machine-writable description of a whole +refinement. PowderLine provides one: the **recipe**. + +```mermaid +flowchart LR + D["GUIs · notebooks · pipelines
autonomous experiments · LLM agents"] --> R["Recipe
structured · versioned"] + R --> V{{Validate}} --> X{{Run}} + X <--> E["GSAS-II · TOPAS
easydiffraction"] + X --> O["Results
tables · uniform Rwp · provenance"] + O -. "output recipe" .-> R +``` -PowderLine uses [pixi](https://pixi.sh) for reproducible conda environments. -The **default environment is core-only** (schema validation, the CLI, no -engine); each engine lives in its own environment: +1. **Describe.** A recipe holds the data, the structural model, the + instrument, what to refine and any metadata you attach. It is structured + data: a dictionary in Python, JSON when stored or sent. It can come from a + file, a database record or a live data stream. +2. **Validate.** PowderLine checks the recipe against a versioned schema and + against the crystallography: the cell against the space group, atoms on + special positions, symmetry-tied parameters. No engine needs to be + installed. Each problem is reported at its field. +3. **Run.** PowderLine hands the recipe to the engine it names and returns + the same structured results whichever engine ran. A recipe is written for + one engine, in that engine's own terms and units. PowderLine never + translates a recipe between engines. +4. **Chain.** Every result includes an *output recipe* with the refined + values written back and provenance attached: the next step's input. + +What that gives you: + +- **One interface for every driver.** A GUI, a notebook, a beamline pipeline, + an autonomous experiment and an LLM agent drive PowderLine the same way and + get the same result. +- **A component, not a silo.** PowderLine takes data from your reduction step + and hands structured results to whatever comes next: plots, databases, the + decision about the next measurement. +- **Machine-checkable.** Each problem is reported at its field, and + PowderLine's own checks say what to write instead, so a program or an LLM + can write a recipe, check it and correct it. +- **Traceable.** Your metadata travels with the recipe into the results, and + every run records what produced it, so data and analyses stay linked. +- **Comparable.** Every engine reports Rwp, Rexp and GoF computed the same way + over the same points. + +PowderLine handles constant-wavelength X-ray powder data on a 2θ axis. +Neutron and time-of-flight data are not supported; see +[Scope and limitations](https://nsls2.github.io/PowderLine/reference/scope.html). + +## Try it + +You need [pixi](https://pixi.sh) and git. ```bash git clone https://github.com/NSLS2/PowderLine.git cd PowderLine -pixi install # core only: validate recipes, no engine -pixi install -e gsasii # + GSAS-II (built as a conda package by pixi-build) -pixi install -e easydiffraction # + easydiffraction (Python >= 3.12) -pixi install -e all # every engine -# fresh checkout: add --frozen (e.g. `pixi install --frozen -e gsasii`) so the committed lock is used as is -``` - -Platforms: `linux-64`, `win-64`, `osx-arm64`. TOPAS has no Python package: -`topas.*` recipes validate everywhere and their input files are written -everywhere, but running them needs a Windows machine with TOPAS 6 (`tc.exe`). - -## 2. Getting started - -Run the bundled LaB6 calibrant example with GSAS-II: - -```bash -pixi run -e gsasii kicker examples/example_LaB6/input.json -``` - -Results are written next to the recipe in `examples/example_LaB6/output/`: - -| File | What it is | -|---|---| -| `refined_parameters.csv` | Every refined parameter with its ESD (9-column table) | -| `LaB6_unit_cell_report.csv` | Per-phase unit-cell parameters + ESDs | -| `LaB6_peak_list_report.csv` | Reflection list (hkl, d-spacing, 2θ, intensity) | -| `fit_profile.txt` | Observed / calculated / background / difference | -| `output_recipe.json`, `parameter_dispositions.csv` | The recipe with the refined values written back, and how each value came about | -| `dummy.lst` | Human-readable GSAS-II refinement log (Rwp, parameter tables) | -| `dummy.gpx` | GSAS-II project — reopenable in the GSAS-II GUI | - -PowderLine removes its own fixed-name files from the output directory before -each run, so a reused directory holds one run's results. A failed run raises, -and its directory holds the engine's own files plus `failure.json` (the reason, -the message and the engine's last console lines) — never a result table. - -The same case as a TOPAS and an easydiffraction recipe sits in -`examples/example_LaB6/topas/` and `examples/example_LaB6/easydiffraction/`. - -> `examples/example_template/` holds fill-in-the-blanks skeletons for the -> three engines, not runnable recipes — copy one to start your own. - -**Bringing your own CIF, calibrant and data files?** PowderLine has no -built-in recipe builder; [Building a recipe](docs/input-guidance.md#building-a-recipe) -walks through writing one and validating it. - -## 3. Ways to use PowderLine - -**Command line** — one command for every engine; the recipe's `schema_name` -decides where it runs: - -```bash -pixi run kicker --validate-only # check a recipe; needs no engine -pixi run -e gsasii kicker # run a gsasii.* recipe -pixi run -e easydiffraction kicker # run an easydiffraction.* recipe -pixi run kicker --topas-dir C:/TOPAS6 --topas-version 6 # run a topas.* recipe (tc.exe on Windows) -pixi run -e gsasii doctor # what this environment can do (gateways, engines, server) +pixi install --frozen -e gsasii +pixi run -e gsasii powderline examples/example_LaB6/input.json --output my_lab6 ``` -After `pixi reinstall powderline` the same program is the installed command -`powderline …`. Flags: `--output DIR` (default: `output/` next -to the recipe), `--verbose`, `--validate-only`; gsasii `--use-server` / -`--no-server` / `--server-timeout SECONDS` (default: no limit); topas -`--topas-dir DIR` / `--topas-version V` / -`--topas-timeout SECONDS` (default 3600; 0 = no limit). A flag given -for a recipe of another engine is an error. +The results are in `my_lab6/`. +[Your first refinement](https://nsls2.github.io/PowderLine/getting-started/first-refinement.html) +walks through them. -**Python API** — load a recipe, run it, get pandas DataFrames back: +## Engines -```python -import powderline, tempfile -from pathlib import Path +| Engine | Workflows | Runs on | +|---|---|---| +| GSAS-II 5.7.9 | Rietveld, single-peak fitting | Linux, Windows, macOS\* | +| TOPAS 6 | Rietveld, single-peak fitting | Windows with a TOPAS licence (input files are written anywhere) | +| easydiffraction 0.21.1 | Rietveld | Linux, Windows, macOS\* | -recipe = powderline.load_recipe_asset(Path("examples/example_LaB6/input.json")) +\* macOS (Apple Silicon) is expected to work but is not routinely tested. -# Engine-free check: every problem at its field path; warnings for defaults applied -checked = powderline.validate(recipe) -print(checked.recipe.schema_name, checked.warnings) - -# run() writes its file reports into the directory you give it; the engine is -# the one the recipe names, so this needs the gsasii environment: -result = powderline.run(recipe, Path(tempfile.mkdtemp())) - -print(result["rwp"]) # PowderLine's uniform Rwp (%) -print(result["engine_details"]) # GSAS-II's own statistics, parameters requested vs varied -print(result["refined_parameters"]) # DataFrame of refined values + ESDs -``` - -The returned dict holds the fit metadata (`status`, `converged`, `rwp`, `r_exp`, -`gof`, `chi2_red`, `simulation_mode`, `engine_details`, `warnings`, elapsed time, -which execution path ran, the output recipe with its parameter dispositions; a -failed run raises instead of returning) plus every -tabular report in memory — the refined-parameter table with ESDs, the fit -profile, per-phase unit-cell and peak-list tables (dicts keyed by phase -name), and the SPF peak tables when single-peak fitting. Everything the -CSV/TXT files contain, without touching disk yourself. - -The public API (`powderline.__all__`) is `run`, `validate`, -`ValidationResult`, `load_recipe_asset`, `support_matrix` and the exceptions -`PowderLineError`, `GatewayNotInstalledError`, `EngineNotAvailableError`, -`EngineVersionError`, `EngineExecutionError`. `import powderline` imports no -engine. - -**The GSAS-II server (used by default for gsasii recipes)** — refinements go -through a persistent server that keeps GSAS-II resident in memory -(auto-started as needed), a large speedup across many refinements. Manage it -directly, or opt out per run with `--no-server`: - -```bash -pixi run -e gsasii gsas-server status # start | status | stop | restart | logs -pixi run -e gsasii kicker --no-server # force an isolated in-process run -``` - -The server is per-user: it listens on loopback only and accepts only requests -signed with a secret token kept in a directory only you can read; your client -in turn only accepts results signed with that token, so other users on the -same machine can neither use your server nor impersonate it. To switch it off -— e.g. on a shared workstation — set `POWDERLINE_NO_SERVER=1`: default -(`auto`) runs then go in-process, explicit server mode (`--use-server` / -`execution_mode='server'`) fails, and nothing is auto-started. - -## 4. Refinement engines - -| Engine | Schemas | Environment | Notes | -|---|---|---|---| -| **GSAS-II 5.7.9** | `gsasii.rietveld`, `gsasii.spf` | `gsasii` | The reference engine; the committed examples carry its outputs as regression references. | -| **TOPAS 6** (Bruker) | `topas.rietveld`, `topas.spf` | any (writing the INP); Windows + `tc.exe` (running) | The recipe states `engine_version: "6"`; the run needs the installed version and location declared (`--topas-version 6 --topas-dir DIR`, or `topas.version` / `topas.dir` in `.powderline_config.yaml`); PowderLine never guesses where `tc.exe` is. | -| **easydiffraction 0.21.1** | `easydiffraction.rietveld` | `easydiffraction` | The recipe states the calculator (`cryspy` or `crysfml`); no single-peak fitting. | - -The structural part of a recipe (space group, cell, atoms) is identical in -every engine; the instrument, background and controls are each engine's -own, with its own units. The -[cross-engine comparison](docs/cross-engine-comparison.md) refines the same -two data sets with every engine and explains why the numbers differ: -**compare lattice parameters, and read Rwp as a property of the model, not -of the engine.** What each engine does with your recipe, and what PowderLine -refuses on its behalf, is on the -[engine behaviour page](docs/engine-behavior.md). - -For the engine architecture and how to add a backend, see -[Adding a refinement engine](docs/DEVELOPMENT.md#adding-a-refinement-engine). -Maintainers keep design records in a separate private repository. - -## Documentation - -- **[Quickstart](docs/quickstart.md)** — clone → install → run → read the output. -- **[Getting Started](docs/getting-started.rst)** — recipe structure, output - files, and the Python API. -- **[Recipe schemas](docs/schemas.md)**, **[Fields and features](docs/fields.md)**, - **[Input guidance](docs/input-guidance.md)** — the five schemas, every field - with its unit, how to supply data / structures / instruments. -- **[Engine behaviour notes](docs/engine-behavior.md)** — what each engine - does, and PowderLine's policy toward it. -- **[Troubleshooting](docs/TROUBLESHOOTING.md)** — common errors and fixes. -- **[Citing PowderLine](docs/citing.md)** — PowderLine, the engines and the - libraries behind them. -- **[Migration guide](docs/migration-0.1-to-0.2.md)** — from PowderLine 0.1.1 - (unified schema 0.26.0) to 0.2.0 (core + engine schemas 0.1.0). -- **[Schema History](docs/SCHEMA_HISTORY.md)** and the - **[support matrix](docs/support-matrix.md)** — schema versions and declared - compatibility. -- **[Using PowderLine as a library](docs/integration-guide.md)** — install it into - another project. -- **[Developer Guide](docs/DEVELOPMENT.md)** and **[Policies](docs/policies.md)** - — architecture, how to add an engine, versioning, frozen code, review - gates. See also the [known-issues register](docs/known_issues.md) - (`KI-NN` ids referenced from code and docs). - -## Citation +## Cite If you use PowderLine in your research, please cite the preprint: > A. A. Corrao, J. A. Perez, J. D. Langhout, M. M. Butala, T. A. Caswell, and -> D. Olds. *PowderLine: a programmatic powder diffraction analysis application.* -> arXiv:2608.17009 (2026). +> D. Olds. *PowderLine: a programmatic powder diffraction analysis +> application.* arXiv:2608.17009 (2026). > [doi:10.48550/arXiv.2608.17009](https://doi.org/10.48550/arXiv.2608.17009) -Machine-readable metadata is in [`CITATION.cff`](CITATION.cff). +**Please cite the engine you used too:** -### Cite the refinement engine too - -PowderLine drives an underlying refinement engine: the one your recipe's -`schema_name` names. Please cite the engine you used in addition to -PowderLine. - -- **GSAS-II** (`gsasii.*` recipes; GSAS-II 5.7.9 in the `gsasii` environment): - > B. H. Toby and R. B. Von Dreele. *GSAS-II: the genesis of a modern - > open-source all purpose crystallography software package.* J. Appl. Cryst. - > **46**, 544–549 (2013). - > [doi:10.1107/S0021889813003531](https://doi.org/10.1107/S0021889813003531) - - PowderLine drives GSAS-II through its scripting interface, so please also cite: - > J. H. O'Donnell, R. B. Von Dreele, M. K. Y. Chan, and B. H. Toby. *A - > scripting interface for GSAS-II.* J. Appl. Cryst. **51**, 1244–1250 (2018). - > [doi:10.1107/S1600576718008075](https://doi.org/10.1107/S1600576718008075) -- **TOPAS** (`topas.*` recipes; TOPAS 6): - > A. A. Coelho. *TOPAS and TOPAS-Academic: an optimization program - > integrating computer algebra and crystallographic objects written in C++.* - > J. Appl. Cryst. **51**, 210–218 (2018). - > [doi:10.1107/S1600576718000183](https://doi.org/10.1107/S1600576718000183) -- **easydiffraction** (`easydiffraction.*` recipes): the easydiffraction - project asks you to cite the specific version you used, through its - [Zenodo archive](https://doi.org/10.5281/zenodo.16806521) (all versions). - PowderLine's `easydiffraction` environment installs version 0.21.1: - > A. Sazonov, E. Lindgren, P. Rozyczko, and M. Seth. *easydiffraction - > 0.21.1.* Zenodo (2026). - > [doi:10.5281/zenodo.22905049](https://doi.org/10.5281/zenodo.22905049) +| Engine | Cite | +|---|---| +| GSAS-II | Toby & Von Dreele, *J. Appl. Cryst.* **46**, 544 (2013), [doi](https://doi.org/10.1107/S0021889813003531); O'Donnell et al., *J. Appl. Cryst.* **51**, 1244 (2018), [doi](https://doi.org/10.1107/S1600576718008075) | +| TOPAS | Coelho, *J. Appl. Cryst.* **51**, 210 (2018), [doi](https://doi.org/10.1107/S1600576718000183) | +| easydiffraction | easydiffraction 0.21.1, Zenodo (2026), [doi](https://doi.org/10.5281/zenodo.22905049), plus the calculator your recipe names (CrysPy or CrysFML) | - Please also cite the calculator your recipe names in - `refinement_controls.calculator`: CrysPy (`cryspy`) or CrysFML (`crysfml`). - The references are on the [citing page](docs/citing.md), which also covers - gemmi, the library behind PowderLine's space-group checks. +Full references, including the calculators and gemmi, are on +[Citing PowderLine](https://nsls2.github.io/PowderLine/about/citing.html). +Machine-readable metadata is in [`CITATION.cff`](CITATION.cff). -## License +## Contributing and licence -BSD 3-Clause — see [LICENSE](LICENSE). +See the [developer guide](https://nsls2.github.io/PowderLine/development/). +BSD 3-Clause licence, see [LICENSE](LICENSE). diff --git a/docs/DEVELOPMENT.md b/docs/DEVELOPMENT.md deleted file mode 100644 index 9015396..0000000 --- a/docs/DEVELOPMENT.md +++ /dev/null @@ -1,719 +0,0 @@ -# PowderLine Developer Guide - -This guide provides context for developers working on PowderLine (and for AI -coding assistants used alongside them). Read this first to understand the -architecture, the design decisions, and the development workflows. The rules -that bind every change (versioning, deliberate changes, the frozen GSAS-II -code, the engine pins, review gates, "ask, don't assume") are on the -[policies page](policies.md); this guide shows where things are and how to -work on them. - -## Project Overview - -**What is PowderLine?** -PowderLine automates crystallographic refinement from a declarative JSON -"recipe" and drives interchangeable refinement engines — GSAS-II, Bruker -TOPAS 6 and the open-source easydiffraction — through one API and one CLI. -A recipe names its engine and workflow in `schema_name` and is written in -that engine's own conventions; PowderLine validates it strictly, hands it to -the engine's **gateway**, and returns standardized result tables with uniform -fit statistics. The goal is to make powder diffraction refinements -reproducible, automatable, and accessible. - -**Scope:** -- One CLI (`powderline `) and one Python API (`powderline.run`, - `powderline.validate`) for every engine; the recipe decides where it runs -- A **core schema** (the shared frame and structural models) plus one - **engine schema** per gateway, all pydantic, `extra='forbid'`, engine-free -- Three gateways under `src/powderline/gateways/`, discovered through entry - points, each split into an engine-free schema layer and a lazily imported - runtime layer -- Single-phase and multi-phase Rietveld refinement; single peak fitting - (GSAS-II, TOPAS) -- Standard output reports plus each engine's native artifacts; uniform - Rwp / Rexp / GoF computed once in core, the engine's own values alongside -- Example-driven design: real refinement cases drive the schemas and are the - regression references - -**Design Philosophy:** -- **Example-driven**: don't build features speculatively; add them when real examples need them -- **Schema-first**: every rule a recipe must obey lives in the schema models, so validation - reports every problem at its field before any engine runs -- **Honest**: an engine's silent default, floor or conversion is either refused at - validation (with the value to write) or reported as a structured warning; never papered over -- **Engine-native**: no translation between engines; each recipe states the numbers its engine computes with -- **Reproducible**: the same recipe produces the same refinement; the references are tracked - -## Architecture (`src/powderline/`) - -### Core (engine-free) - -| Module | Role | -|---|---| -| `__init__.py` | the public API, `__all__`: `run`, `validate`, `ValidationResult`, `load_recipe_asset`, `support_matrix`, the exceptions. `import powderline` imports no engine | -| `api.py` | `validate()` and `run()`: resolve the gateway from `schema_name` (`.`) through the registry and hand the recipe over; `ValidationResult` | -| `kicker.py` | the one CLI (`powderline `; `pixi run kicker` runs the same `main`): loads the recipe, refuses templates, maps the engine flags onto the gateway's run options, prints the summary and the structured warnings | -| `registry.py` | gateway discovery from the `powderline.gateways` entry-point group, lazy loading, `Gateway` dataclass (`name`, `capabilities`, `validate`, `run`), `register_gateway` escape hatch, the engine-version check | -| `schema_core.py` | core schema 0.1.0: `CoreRecipe` (frame), `RefinableParameter` / `BoundedRefinableParameter`, `XRDData`, `FitRange`, `ChebyshevBackground`, `UnitCell`, `Atom`, `UanisoTensor`, `Phase[P]` (the phase block with every symmetry rule), `CoreFloat`/`CoreInt`/`CoreName`, `collect_warnings`, `parameters_requested`, `fit_limits_on_data`; the engine schema building blocks: `EngineRecipe`, `recipe_validator`, `refined_flags`, `custom_error` / `raise_problems`, `omit_absent`, `absent_not_null`, `unique_phase_names`, `fit_window`, `fit_range_problems`, `outside_window_problems`, `refined_flag_problems`, `unit_schema` | -| `symmetry.py` | gemmi-backed symmetry: canonical space-group names, cell tie groups and the cell check, special-position analysis, coupling groups for coordinates and Uij | -| `compat.py` | declared schema-version compatibility: `check_engine_schema_version`, `support_matrix()`, `schema_declarations()` (a gateway's `DECLARATIONS`) | -| `fitstats.py` | the uniform Rwp / Rexp / GoF / χ²_red over the fit window with positive weights | -| `reports.py` | the standardized report writers (`refined_parameters.csv`, unit-cell and peak-list reports, `fit_profile.txt`): one column order, UTF-8, LF; the fixed-name list PowderLine owns (`FIXED_OUTPUT_NAMES` / `FIXED_OUTPUT_PATTERNS`, `FAILURE_FILE`) and `clear_fixed_outputs()` | -| `recipes.py` | `load_recipe_asset` (JSON / YAML) | -| `exceptions.py` | `PowderLineError`, `SymmetryError`, `GatewayNotInstalledError`, `EngineNotAvailableError`, `EngineVersionError`, `EngineExecutionError`, `ExecutionBackendError` (both with `.reason` / `.result`), the `StructuredWarning` shape; `WARNING_CODES`, the registry of every structured-warning code with its owner and level (asserted against `src/` and the engine-behaviour table) | -| `results.py` | the run-result envelope: `ENVELOPE_VERSION`, the reason-code sets, `failure_envelope`, `generated_envelope`, `with_status_fields`, `raise_if_failed` | -| `config_loader.py` | `.powderline_config.yaml` (the `simulation_defaults` block; engine-free) | -| `_status.py` | console status glyphs | - -### Gateways (`src/powderline/gateways//`) - -Each gateway is a package with the same two layers: - -- **Schema layer** — `schema.py`: the engine schema models built on - `schema_core` (the engine's `Phase` subclass, instrument, background, - controls, payload and recipe models), the version declarations - (`ENGINE_SCHEMA_VERSION`, `SUPPORTED_ENGINE_SCHEMAS`, `REQUIRES_CORE_SCHEMA`, - exposed as `DECLARATIONS`), `SCHEMA_NAMES`, and `validate_recipe()`. Imports - no engine; this is what `powderline.validate()` runs. -- **Runtime layer** — `gateway.py` (the entry-point factory: `capabilities()`, - `validate()`, `run()`, `ENGINE_VERSION_SPEC`, and for pixi-installed engines - `ENV` / `INSTALL_COMMAND`) plus the modules that import the engine, imported - lazily from `run()`. - -| Gateway | Runtime modules | -|---|---| -| `gsasii/` (GSAS-II 5.7.9) | the **frozen** setup/execution code moved verbatim from `kicker.py` at the baseline `daace24`: `helpers.py`, `project.py`, `setters.py`, `executors.py`, `extractors.py`, `constraints.py` (code-hashed; see [policies](policies.md#frozen-gsas-ii-code)); non-frozen additions `routing.py` (one entry for in-process, server and client), `fit_report.py` (uniform statistics, `engine_details`, convergence, divergence), `sites.py` (GSAS-II's reading of each site vs core's), `report_rows.py`, `paths.py` (GSAS-II parameter names → recipe paths for the output recipe), `console.py` (the engine's console tail into a failed envelope), `scattering.py` (GSAS-II's ion table and spelling); `server.py` / `client.py` (the persistent FastAPI server and `GSASClient`: per-user, loopback, ephemeral port, signed requests and responses, `POWDERLINE_NO_SERVER=1` disables it — v0.1.2); `simulation_builder.py` (structure dict → simulation recipe; see [the builder](#the-simulation-recipe-builder)) | -| `topas/` (TOPAS 6) | `native_writer.py` (model → INP + `.xye`), `native_run.py` (run `tc.exe`, read TOPAS's files strictly, build the result; the background-only second run), `peak_list.py`, `runner.py` (find `tc.exe`, read `.powderline_config.yaml`) | -| `easydiffraction/` (0.21.1) | `native_builder.py` (model → easydiffraction `Project`; the CrysPy template rule; anisotropic atoms first), `native_run.py` (calculate or lmfit fit, the result), `space_group_table.py` / `space_group_support.py` (the 531 settings and the per-calculator verified sets), `scattering.py` (CrysPy's ion table) | - -The entry points are declared in `pyproject.toml` -(`[project.entry-points."powderline.gateways"]`); the metadata is written at -install time, so after editing that table run `pixi reinstall powderline` -(every environment has its own install: `--all`). There is deliberately no -in-code fallback (KI-13): the in-repo gateways use exactly the mechanism a -third-party engine would. - -### The execution stack - -``` -powderline / powderline.run(recipe, output_dir, **engine_options) - │ schema_name -> registry.get("") ; gateway.run(...) - ▼ -gateway.run() validates (schema layer), checks the engine version, - │ then the runtime layer: - ├─ gsasii: routing.run_refinement() in this process, or GSASClient - │ -> the FastAPI server (GSAS-II resident) -> routing.run_refinement() - │ -> executors.run_refinement() -> GSASIIscriptable - │ -> fit_report (uniform statistics) ; reports (files) - ├─ topas: native_writer -> tc.exe (twice: fit, background) -> native_run reads the files - └─ easydiffraction: native_builder -> lmfit / calculate() -> native_run - ▼ -the standardized result dict (tables as DataFrames) + the report files + native artifacts -``` - -`execution_mode` (gsasii only: `'auto'` | `'server'` | `'subprocess'`) is the -public option; `method` is the result key saying which path ran. The server -keeps GSAS-II imported in one process (~2.6 s import amortized); it caches -code, so **restart it after editing anything it runs** -(`pixi run -e gsasii gsas-server restart`). - -## The schema layer - -### Shape of a recipe - -``` -schema_name "." -core_schema_version "0.1.0" -engine_schema_version "0.1.0" -metadata free-form, echoed, never read (<= 1 MiB) -payload the engine schema's payload model -``` - -Core owns the frame, `xrd_data`, `fit_range`, `ChebyshevBackground` and the -**phase block** (`space_group`, `unit_cell`, `atoms`), generic over the -parameter type `P` (`RefinableParameter` for gsasii, `BoundedRefinableParameter` -for topas and easydiffraction). The engine schema owns everything else -(instrument, background placement, controls, engine phase fields such as -`scale` and `peak_broadening`) and may add checks, but can never redefine, -skip or rewrite core's (guarded when the class is defined: no reuse of a -core field, validator or private-attribute name, no `'before'`/`'wrap'` -validator on a core field, `extra='forbid'` kept; `tests/test_phase_canary.py` -runs core's must-fail recipes through every engine phase model). - -User-facing rules: [schemas](schemas.md), [fields](fields.md) (generated -tables), [input guidance](input-guidance.md), [SCHEMA_HISTORY](SCHEMA_HISTORY.md). - -### Validation model - -- Every recipe check is a pydantic validator raising `ValueError` (or - `SymmetryError`), so an invalid recipe raises **one** `pydantic.ValidationError` - listing every problem at its field path. There is no PowderLine wrapper - exception for invalid recipes. -- **Validation never changes a recipe.** The validated model holds exactly the - recipe's values plus documented defaults and the symmetry-derived values; a - value an engine would silently replace (a floor, a snapped coordinate, a - limit) is an **error naming the value to write**, never an adjustment. -- **Structured warnings** (`{code, message, field_path[, level]}`; `level` absent = `"warning"`, or `"info"`) are collected once - from the validated model (`collect_warnings`) and travel with it: a default - applied, a symmetry remark, a unit conversion the gateway will make. They - never change what the recipe means, so re-validating reproduces them. -- Engine quirks that reach a user are recorded in the maintainers' - engine-behaviour register and rendered on the - [engine behaviour page](engine-behavior.md); a schema rule that exists - because of a quirk cites its `EB-nn` in the error message. - -## Adding a refinement engine - -The gateway seam is the contract the refactor exists to provide. A new engine -is a package `src/powderline/gateways//` (in-repo) or a separate -distribution (out of tree) that provides: - -1. **A schema module** (`schema.py`), engine-free: - - the engine's `Phase` subclass (`class MyPhase(Phase[BoundedRefinableParameter])` - or `Phase[RefinableParameter]` if the engine has no usable bounds), adding - the engine's phase fields with their own validator names and `'after'` - validators only; - - instrument, background and controls models in the **engine's native - terms and units**, every quantity carrying `json_schema_extra={"unit": ...}` - (the [fields page](fields.md) is generated from it), `extra='forbid'`, - no `null` for a block that may be left out, and **no default that - replaces an engine behaviour silently** (either the value is required, or - the default is documented and reported by a warning); - - the recipe model(s) subclassing `powderline.schema_core.EngineRecipe` - with `schema_name` narrowed to a `Literal` and two class variables, - `gateway_name` and `schema_declarations = DECLARATIONS`, where - `DECLARATIONS = compat.schema_declarations(engine_schema_version="0.1.0", - supported_engine_schemas="==0.1.0", requires_core_schema="==0.1.0")` - (widened explicitly later, never by range); `EngineRecipe` runs the - version check; - - `validate_recipe = schema_core.recipe_validator({: , ...}, - engine="", title="Recipe")`: accepts a dict or an - already-validated model (idempotent) and reports an unknown - `schema_name` at `schema_name`; - - every engine rule as a validator: refuse what the engine cannot express - (settings, features) and what it would silently alter (floors, limits), - naming the value to write; apply the three shared value rules - (wavelength > 0, scale per engine, peak positions inside the window) and - the fit window on data points (`fit_limits_on_data`). The shared pieces - are core building blocks, not copies: `absent_not_null` for a - block that may be left out but never be `null`, `omit_absent` in the - wrap serializer that drops it from dumps, `unique_phase_names`, - `fit_range_problems` / `fit_window` / `outside_window_problems` / - `refined_flag_problems` for the payload rules, `refined_flags`, - `custom_error` + `raise_problems` to report every problem at once; - - every warning code the engine emits registered in - `powderline.exceptions.WARNING_CODES` (owner, level) and listed in the - engine-behaviour table; a core test holds the three in agreement. -2. **A gateway module** (`gateway.py`) exporting `get_gateway()` → - `registry.Gateway(name, capabilities, validate, run)`: - - `capabilities()` returns at least `name`, `workflows` (the schema names), - `engine_version_spec` (PEP 440) and the schema `DECLARATIONS`; - - `validate(recipe, *, verbose=False)` (exactly these keywords) returns the - validated model; - - `run(recipe, output_dir, *, verbose=False, validate_only=False, - allow_unsupported_engine_version=False, **own_options)`: validates, checks - the installed engine version against `engine_version_spec` - (`registry.check_engine_version`; raise `EngineNotAvailableError` with - `env` and `install_command` when the engine is missing — only on a real - run, never for `validate_only`), imports the engine **lazily**, runs, and - hands `powderline.roundtrip` its refined values by recipe path (`{path: (value, esd)}` - in recipe units, plus the flagged paths it did not vary and those that ended at a - limit; `roundtrip.build_output` / `attach` do the rest: tie propagation, the - multiplicity invariant, `output_recipe`, `output_recipe_valid`, `parameter_dispositions`, - the two files) and - returns the standardized result dict with the same keys and types as the - other gateways (`status`, `converged`, `envelope_version`, `run_id`, `rwp`, `r_exp`, `gof`, `chi2_red`, - `simulation_mode`, `engine_details` with `parameters_requested` / - `parameters_varied` / `n_points`, `warnings`, `elapsed_time`, `method`, - `output_files`, `fit_profile`, `unit_cell_data`, `peak_list_data`, - `refined_parameters`, `spf_peaks`, `spf_convergence_diagnostics`; gsasii adds - `peak_fit` for a peak fit). A gateway **never returns a failed - result**: it raises `EngineExecutionError` (the engine ran and failed) or - `ExecutionBackendError` (it did not run) with a `reason` from - `powderline.results.REASONS`, or returns - `results.raise_if_failed(envelope)` on an envelope it received; the - GSAS-II server/client show the wire form (`results.failure_envelope`, - `results.with_status_fields`). Statistics come from `powderline.fitstats` over - the window's weighted points with P read from the engine after the run; - the engine's own values go under `engine_details`; divergence (a - non-finite calculated pattern) raises `EngineExecutionError`. - - report files through `powderline.reports`; native artifacts kept. -3. **An entry point** in `pyproject.toml` - (`[project.entry-points."powderline.gateways"] = "...gateway:get_gateway"`), - then `pixi reinstall powderline`. A pixi feature/environment for the - engine's dependencies if the default environment must not carry them - (every in-repo engine has one); `ENV` / `INSTALL_COMMAND` on the gateway. -4. **Tests**, in the pattern of the existing gateways: schema tests - (`test__schema.py`: every rule, every refusal, with the error - path), the engine phase model registered in `tests/test_phase_canary.py` - `ENGINE_PHASES`, an import-block test proving the schema layer and - `import powderline` load no engine module, a space-group translation test - over every setting the engine accepts (orbit / operator comparison against - gemmi), fixtures + goldens for the runtime (hand-derived native recipes - with a README of the derivations; goldens recording the engine's version), - and the result-contract test (same keys and types as the other engines). - Tests that need the engine are marked / skipped so the core-only suite - stays green. -5. **Freeze-equivalent expectations**: the engine's goldens are the - references; a change in what the engine is asked to do (the INP, the - project, the input dictionary) is proved against them, and anything the - engine does silently is written up in the engine-behaviour register with - its policy before the gateway ships. -6. **Docs**: the schemas, fields (regenerate with - `scripts/gen_schema_field_tables.py`), input-guidance and engine-behaviour - pages; a SCHEMA_HISTORY section; the support matrix regenerates itself. - -### Adding a capability to an existing engine - -The same discipline, smaller: a new field or block goes into the engine -schema with its unit, its validators and its refusals; the gateway consumes it -from the validated model (for gsasii, an **input-side read in a frozen module -is a signed-off frozen-code edit**, otherwise the read happens in the non-frozen modules); a -fixture and a golden prove it reaches the engine and does what the recipe -says; the engine-behaviour register records anything the engine does with it -silently; SCHEMA_HISTORY and the generated field tables record the field. If -the field changes what an existing recipe means, that is a deliberate change -(recorded, CHANGELOG, migration note) — and if the engine's own behaviour -is unknown, that is a question for the maintainer, not an inference. - -Worked example — March–Dollase preferred orientation: - -1. **Ground truth first.** Read and run each engine: GSAS-II's HAP - `Pref.Ori.` entry and `GetPrefOri`; easydiffraction's - `preferred_orientation` rows, which only CrysPy reads, and only under the - lowercase structure key (EB-97); CrysPy's un-normalised correction (EB-98). -2. **One core building block, engine fields.** `schema_core.MarchDollase[P]` - carries the shape and the engine-independent rules (`hkl` not all zero, - `ratio > 0`); each engine's phase model adds `preferred_orientation` in - its own parameter shape (gsasii `[value, flag]`, easydiffraction bounded - plus the required `march_random_fract`, topas bounded); topas declared the - field and refused it until its half was verified on a TOPAS 6 machine - (the `PO` macro against the Dollase formula, then the writer and the - read-back of r). -3. **The gateway reads the model.** gsasii: an additive frozen setter and its - call, two signed-off frozen-code edits with proofs (the project-dict diff shows only - `Pref.Ori.`), and the refined name `p:h:MD` in `paths.py`; - easydiffraction: the builder's row plus the `_free` bookkeeping that maps - the refined value back to the recipe path. -4. **A gate, then a demonstration.** The pass/fail gate is a formula check on a - simulated pattern (each engine's textured/untextured intensity ratio equals - Dollase's correction over the reflection's equivalents); the example - (`example_LaB6_textured`) is the demonstration that refinement recovers the - programmed value, with its caveats written down (the start at exactly 1.0 - aborts, EB-99). -5. **Record.** Fields tables regenerated, SCHEMA_HISTORY per engine, - engine-behaviour register rows, CHANGELOG, the T-list row for the TOPAS - half, the P-row. - -## GSAS-II integration notes - -### The frozen modules and the project dictionary - -The GSAS-II setup/execution code manipulates the `proj.data` dictionary of a -GSAS-II project directly (explicit control, debuggable, brittle to GSAS-II -internals). It moved verbatim out of `kicker.py` at the baseline -`daace24` (not on `main`) and is -frozen behind a code hash; input-side edits go through the maintainers' -signed-off record of frozen-code edits ([policies](policies.md#frozen-gsas-ii-code)). -The module headers still read "moved verbatim"; that record, not the headers, -is the record of the additive edits made since (ionic types in `project.py`, -the preferred-orientation setter in `setters.py`, the simulation ESDs in -`extractors.py`). -Key access patterns, for reading the code: - -```python -# Instrument parameters (list format: [current_value, value, refine_flag]); the native -# gateway builds Iparm1 from the recipe's instrument block (schema.gsasii_iparm1) -proj.data[hist.name]['Instrument Parameters'][0]['Lam'][1] = wavelength -proj.data[hist.name]['Instrument Parameters'][0]['Lam'][2] = refine_flag - -# Background -proj.data[hist.name]['Background'] = [ - [chebyshev, no_peaks, background_type], # Background metadata - {'nDebye': 0, 'debyeTerms': [], ...}, # Debye-Scherrer params - {'nPeaks': n, 'peaksList': [...]}, # Single peaks -] - -# Histogram-atom parameters (HAP): phase scale, size, microstrain -proj.data['Phases'][phase_name]['Histograms'][hist.name]['Scale'][0] = value -proj.data['Phases'][phase_name]['Histograms'][hist.name]['Size'][1][0] = value # isotropic size -proj.data['Phases'][phase_name]['Histograms'][hist.name]['Mustrain'][1][0] = value # isotropic microstrain - -# Phase parameters -proj.data['Phases'][phase_name]['General']['Cell'] = [refine_cell, a, b, c, alpha, beta, gamma, volume] -``` - -GSAS-II exposes lumped refine flags (one for the cell, `'X'` for all three -coordinates of an atom, `'U'` for all six Uij); `constraints.py` sets the -lumped flag and emits GSAS-II "Hold" constraints for the degrees of freedom -the recipe does not refine, so per-parameter flags are honoured. The native -schema has already refused any flag pattern GSAS-II could not honour (a strict -subset of a tie group or of a GSAS-II cell group), so no flag is ever silently -overridden. - -### What the native gsasii path adds around the frozen code - -- `sites.py` compares GSAS-II's `(site symmetry, multiplicity)` of every atom - with core's after phase setup and stops with an error naming an equivalent - position when GSAS-II misreads the site (some 2-fold sites of R32/R-3m/R-3c) - or cannot name it while a coordinate / `Uaniso` is refined. -- `fit_report.py` computes the uniform statistics over the stated window, - reads GSAS-II's stored messages into `gsasii_refinement_message` warnings, - its convergence flag into `engine_details.converged`, counts - `parameters_varied` from GSAS-II (`Nvars`; the Peak List count after - singular drops), and raises on a non-finite calculated pattern. -- The documented defaults (size/strain 10 µm / 0 with a warning; background - none) are pydantic defaults in `schema.py`, so the frozen setters receive - explicit values and GSAS-II's own 1 µm / 1000 / constant-1.0 defaults are - never used. - -### Known GSAS-II quirks - -All on the [engine behaviour page](engine-behavior.md); the ones a developer -meets first: the background-peak width `BkPksig` is a **variance** (the -schema names it `pv_gaussian_sigma_sq`); SH/L is floored at 0.002 in the -profile calculation; `Refine` returns nothing and stores its messages in the -project; `Nvars` counts parameters dropped as singular; `GOF` means two -different things in Rietveld and Peak List fits; limits are applied after the -cycles (why gsasii 0.1.0 exposes no bounds). - -### Output files - -Fixed filenames in every engine (`refined_parameters.csv`, -`_unit_cell_report.csv`, `_peak_list_report.csv`, -`fit_profile.txt`; single peak fitting adds a peak report), written by -`powderline.reports` with one column order and LF line endings so the files -are byte-stable across platforms. gsasii keeps `dummy.gpx`, `dummy.bak0.gpx` -and `dummy.lst`; topas keeps the INP, `.xye` and every file TOPAS wrote; -easydiffraction writes the reports only. Refinement runs add -`output_recipe.json` and `parameter_dispositions.csv`. `powderline.run()` -removes every PowderLine-owned fixed name (`reports.FIXED_OUTPUT_NAMES` / -`FIXED_OUTPUT_PATTERNS`) from the directory before dispatch and never the -engine's files; a failed run leaves the engine's files plus `failure.json` -(the envelope with a `powderline_provenance` block) and no result table — -in gsasii the checks run before the tables are written. - - The `refined_parameters.csv` -columns: `parameter_name` (the engine's identifier), `descriptive_name`, -`phase_name`, `phase_idx`, `atom_name`, `atom_idx`, `value`, `esd`, -`category`. gsasii's ESDs come from GSAS-II's covariance matrix -(`proj.data['Covariance']`); unit-cell ESDs are converted from the -reciprocal-metric tensor with `extractors.calculate_cell_esds_from_A_matrix`. - -## Domain essentials - -**Rietveld refinement** fits a calculated powder pattern to the observed one -by adjusting structural, profile and background parameters. The headline -quality metric, in PowderLine's uniform form (percent, over the fit window's -points with positive weight): - -$$R_{wp} = 100 \sqrt{\frac{\sum_i w_i (y_{obs,i} - y_{calc,i})^2}{\sum_i w_i y_{obs,i}^2}}, \quad -R_{exp} = 100 \sqrt{\frac{N - P}{\sum_i w_i y_{obs,i}^2}}, \quad -\mathrm{GoF} = R_{wp}/R_{exp}$$ - -with N the included points and P the engine's count of varied parameters. -Lower Rwp is a better fit of *that model* to *those points*; across engines -compare lattice parameters, not Rwp (the models differ: background peaks, -asymmetry treatment, sample broadening). - -**Parameter types** (every one a `[value, flag]` or `[value, flag, min, max]` -list): structural (cell, coordinates, occupancies, ADPs: the core phase block, -identical across engines), intensity (phase `scale`), peak shape (sample -size/strain in gsasii and topas; the instrument profile in every engine), -background (Chebyshev; gsasii/topas peaks), instrument (profile terms, -wavelength, zero shift, polarization, asymmetry: the engine's own names and -units). - -**Phase vs histogram parameters**: phase parameters belong to a crystal -structure (cell, atoms, scale, sample broadening); histogram parameters belong -to the measurement (instrument profile, background) and are shared across -phases in a multi-phase refinement. - -**Glossary**: *Recipe* — PowderLine's JSON input. *Gateway* — the per-engine -package behind the registry. *Schema layer / runtime layer* — the engine-free -and engine-importing halves of a gateway. *Tie group* — parameters the -symmetry makes one parameter (cubic a, b, c). *Golden* — a committed engine -output a test compares against. *GPX / LST* — the GSAS-II project and log. -*INP / `.xye`* — TOPAS's input and data files. - -## Development workflow - -### Environments and tasks - -```bash -pixi install -e dev # every engine + docs/test tooling (contributors) -pixi install -e gsasii # GSAS-II only -pixi run kicker --validate-only # default env: engine-free -pixi run -e gsasii kicker # run a gsasii.* recipe -pixi run -e easydiffraction kicker # run an easydiffraction.* recipe -pixi run test # engine-free suites (core + topas_gen), any env -pixi run -e dev test-all # every suite, the pre-merge gauntlet -pixi run -e gsasii pytest tests/test_example_LaB6_regression.py -q # one engine's suite -pixi run docs # Sphinx with -W (zero warnings is the standard) -pixi run -e gsasii gsas-server start|status|stop|restart|logs -pixi run update-code-hash # after a signed edit to a frozen gsasii module -pixi reinstall powderline --all # after editing entry points / [project.scripts] -``` - -The default feature carries only PowderLine's runtime dependencies plus -pytest; `gsasii`, `easydiffraction`, `all` and `dev` are features/environments -in `pixi.toml`. `test` and `update-code-hash` live in the default feature and run anywhere; -tasks only `dev` has (`test-all`, `docs`, `docs-clean`) resolve without `-e`; engine-bound tasks live in their engine's feature, so pixi -reports them as ambiguous without `-e` and prints the form to use. Tasks that -import PowderLine set `PYTHONPATH` through the task-level `env` dict (cross-platform; -[cross-platform guide](cross-platform-guide.md)). - -### Execution modes (gsasii) - -| Mode | Command | Behaviour | -|---|---|---| -| auto (default) | `pixi run -e gsasii kicker input.json` | tries the server, auto-starts it, falls back to in-process | -| server | `... --use-server` | server only; fails if it cannot be started | -| in-process | `... --no-server` | skips the server; GSAS-II imported in this process | - -The server amortizes GSAS-II's import (~2.6 s) over many runs (batch jobs: -0.1–0.3 s per run against 1–2 s in-process). It **caches code**, so restart it -after editing anything under `src/` it executes; since 0.2.0 a forgotten -restart is caught: `/health` reports the PowderLine version and an MD5 of the -source tree the server loaded (`server.source_tree_hash()`), the client -compares both with its own and treats a mismatch as a **stale** server — -`auto` mode says so and runs in-process, `--use-server` fails with -`server_unavailable` naming `gsas-server restart`, and `gsas-server status` -prints `Code: STALE`. - -**No shared filesystem (KI-12):** the client sends the recipe inline and -the server runs each job in a private scratch directory under its state dir; -every output file comes back base64-encoded inside the signed response -(`files`, GSAS-II's `*.bak*.gpx` backups excepted) and the client writes it -into its own `output_dir` (atomic replace, retried on Windows). A server on -another node or in a container therefore behaves like a local one; a client -that cannot write its directory fails with `backend_error`, a malformed or -oversized transfer with `communication_failed` (auto mode then runs -in-process). - -**Server state and security (per user, v0.1.2):** the server binds `127.0.0.1` -on a free ephemeral port and writes its lock, PID, log, and an endpoint file -(port + random secret token) — all mode 0600 — to a private per-user state -directory: `$XDG_RUNTIME_DIR/powderline` (if it is a real directory owned by -you), else `/powderline-` on POSIX (mode 0700; refused if it is a -symlink, owned by someone else, or group/other-accessible, or if the temp -directory would let others replace it), and `%LOCALAPPDATA%\powderline` on -Windows. On POSIX the files are opened relative to a re-verified, no-follow -directory handle. - -The token never travels over the wire: every request carries a fresh nonce, a -timestamp and an HMAC-SHA256 over method, path, nonce, timestamp and body, and -the server rejects anything unsigned, wrongly signed, stale, or replayed — on -every route, including 404/405/422 — before reading the body. Every response -is signed back over the nonce, status and body, and the client discards -unsigned responses. Other users on a shared host therefore can neither submit -jobs to your server nor impersonate it (e.g. by taking over the port of a -crashed server). `gsas-server start` takes a kernel-held lock (`flock` / -`msvcrt.locking` on `server.lock`) for the server's lifetime; the OS releases -it when the process exits, so concurrent auto-starts never spawn duplicates -and leftover PID files can never make a dead server look alive. - -**One refinement at a time:** the server runs `/simulate` requests strictly in -sequence (GSAS-II's module-level state is not thread-safe); concurrent clients -queue (up to the ASGI thread pool's size, 40 by default; `/health` is not -behind the lock). **Port problems are reported, not crashed:** a malformed -`POWDERLINE_SERVER_PORT` is refused up front by `gsas-server start` and by a -client about to auto-start (`ExecutionBackendError`, `server_unavailable`, -naming the variable); a busy port fails the start before GSAS-II is imported, -with the port named, and a client whose auto-started server exits at start -reports the exit and points at the server log. - -**Pin the server port** (only if needed — the default ephemeral port avoids -collisions between users): -```bash -export POWDERLINE_SERVER_PORT=19472 -pixi run -e gsasii gsas-server start -pixi run -e gsasii kicker input.json # the client reads the port from your state directory -``` - -**Disable the server entirely** (shared workstations, locked-down deployments): -```bash -export POWDERLINE_NO_SERVER=1 # also: true / yes / on -``` -With it set, `execution_mode='auto'` (and the CLI default) runs in-process and -never starts a server, `execution_mode='server'` / `--use-server` fail with a -clear error, and `pixi run -e gsasii gsas-server start|restart` refuse. - -**View server logs** (`server.log` in the state directory; `status` prints the path): -```bash -pixi run -e gsasii gsas-server logs -``` - -### Adding a new recipe field (any engine) - -1. **Schema**: add the field to the engine's `schema.py` (unit metadata, - validators, refusals; for a structural quantity, to `schema_core`). -2. **Gateway**: consume it from the validated model in the runtime layer - (gsasii: a non-frozen module, or a signed-off edit for a frozen read). -3. **Fixture + golden**: a native recipe stating it and the engine's output. -4. **Example** if it is a real case: `examples//` with `DESCRIPTION.md` - and, for gsasii, the committed output (regression reference). -5. **Docs**: `scripts/gen_schema_field_tables.py`, SCHEMA_HISTORY, the - input-guidance / engine-behaviour pages as needed, CHANGELOG. - -### Adding a new example - -Examples drive schema evolution; see `examples/CONTRIBUTING_EXAMPLES.md`. -Directory `examples//` with `input.json` (the `gsasii.*` recipe), -`DESCRIPTION.md`, `output/` (committed GSAS-II results as regression -references), and optionally `topas/input.json`, `easydiffraction/input.json` -with a README each. An `easydiffraction/output/` folder holds that engine's -committed reference results (checked against the easydiffraction goldens); -TOPAS references live as goldens under `tests/data/topas/native/golden//` -(they come from a TOPAS machine; the directory is the fixture's `engine_version`). - -### Running tests - -Every test carries exactly one suite marker (`pyproject.toml` -`[tool.pytest.ini_options] markers`; `tests/conftest.py` refuses a collection -where one is missing or doubled), and the pixi tasks select on them: - -```bash -# Engine-free suites (core + topas_gen): any env, the default one included. -# Installs the core guard: an engine import or a tc.exe run fails loudly. -pixi run test - -# Engine suites, each in its env (the slow regressions are part of test-gsasii) -pixi run -e gsasii test-gsasii -pixi run -e easydiffraction test-easydiffraction - -# TOPAS: generation everywhere; the topas_exec tests run tc.exe on the TOPAS -# machine when POWDERLINE_TOPAS_DIR and POWDERLINE_TOPAS_VERSION are set -# (PowerShell: $env:POWDERLINE_TOPAS_DIR='C:\TOPAS6'; $env:POWDERLINE_TOPAS_VERSION='6') -pixi run test-topas - -# Everything, in the dev env -- the pre-merge gauntlet -pixi run test-all - -# Extra pytest arguments pass through -pixi run test -x -k schema -pixi run -e gsasii test-gsasii -m "gsasii and not slow" -``` - -Markers: `core` (imports no engine, runs no engine subprocess -- the schema -layer of every gateway included), `gsasii`, `topas_gen` (engine-free TOPAS -input generation), `topas_exec` (tc.exe), `easydiffraction`, plus the optional -`slow` (a full refinement runs). `POWDERLINE_CORE_GUARD=1` forces the guard on -for a path or `-k` selection; `=0` turns it off. - -**Regression test strategy:** -- Rwp, unit cell parameters and refined values compared within the tolerances - of `regression-tolerance.md`; peak lists by row count -- Detects any change in the engine's refinement result -- Reference outputs committed to git for comparison -- If a test fails: inspect the `.lst` file to determine whether the change is - expected or a bug; a new engine version goes through `engine-versions.md` - - -Suite map: `test_schema_core`, `test_symmetry_core`, -`test_symmetry_characterization` (core rules; all 230 groups against -GSAS-II's site tables), `test_phase_canary` (engine phase models cannot -weaken core), `test_api`, `test_cli`, `test_registry`, `test_compat`, -`test_exceptions`, `test_fitstats`, `test_reports` (core), `test_gsasii_schema`, -`test_gsasii_schema_engine` (GSAS-II's reading of every setting), -`test_gsasii_native_run`, `test_example_LaB6_regression`, `test_esd_extraction`, -`test_per_parameter_flags`, `test_run_refinement_missing_keys`, -`test_constraints_mapping`, `test_code_hash`, the server/client tests -(gsasii), `test_topas_schema`, `test_topas_native`, `test_topas_native_golden` -(12 INPs byte-identical to the TOPAS 6 goldens), -`test_topas_runner` (topas), -`test_easydiffraction_schema`, `test_easydiffraction_native`, -`test_easydiffraction_space_groups` (easydiffraction), -`test_convert_recipe_026` (the migration tool), `test_cross_engine_comparison`, -`test_simulation_builder` (the structure → simulation-recipe builder). - -**Regression strategy:** the gsasii examples' outputs are compared within -the tolerances of [regression-tolerance.md](regression-tolerance.md) -(`tests/subprocess_utils.py`: `refined_params_match`, `unit_cell_match`), -peak lists by row count, GSAS-II's own `.lst` wR as a freeze tripwire and the -uniform `rwp` against a recorded value; TOPAS and easydiffraction against -their goldens (easydiffraction cell rtol 1e-4, values 1e-3). If a regression -test fails, inspect the engine's log and the deliberate-changes policy before -touching a reference. Some gsasii cases need a working GSAS-II refinement -and can fail for environment reasons: compare against the baseline run. - -### Documentation - -`pixi run docs` builds with `-W --keep-going` and `nitpicky`; zero warnings -is the standard. Pages under `docs/_generated/` are written by scripts -(`gen_schema_field_tables.py`: field tables and support matrix; -`cross_engine_comparison.py`: the comparison tables) and pulled in with MyST -`{include}`; regenerate them with the change that affects them. `api.rst` -documents the public API; the schema contract is the generated field tables. - -### Template file detection - -The CLI refuses a recipe whose path contains `template` or whose payload lacks -the blocks its workflow needs (`xrd_data`, `instrument`, `refinement_controls` -and `phases` or `single_peaks`), pointing at the examples. It is a CLI-only -guard: `powderline.run()` trusts its caller. - -## Key gotchas - -- **Restart the GSAS-II server** after code changes; a stale server is the - usual cause of "my fix has no effect". -- **`pixi reinstall powderline --all`** after touching entry points or - `[project.scripts]`, or when the registry reports no gateways (KI-13). -- **Frozen modules** change only through signed-off, recorded edits; `test_code_hash` guards - them; `pixi run update-code-hash` after a signed edit. -- **Generated docs** go stale silently unless regenerated - (`gen_schema_field_tables.py --check`). -- **Parameter correlation warnings** from GSAS-II (`gsasii_refinement_message`: - "parameters 0 and 1 are 99.8% correlated") are normal for - over-parameterized models (background terms, U/V/W, size vs strain); - they matter when a fit does not converge or values are unphysical. -- **Chebyshev coefficients** are indexed from the constant term; their basis - spans the fit window's first and last data points in every engine. -- **Single peak fitting**: `gsasii.spf` widths are σ² (centidegree²) and γ - (centidegree), each ≥ 0.001; with `use_instrument_profile: true` the - widths come from U..Z and the peaks' own width flags must be `false` - (GSAS-II would ignore them silently); `topas.spf` peaks are `xo_Is` with - the instrument peak type convolved with their own `gauss_fwhm` / `lor_fwhm`. - gsasii writes `single_peaks_report.txt` (position, intensity, σ, γ, FWHMs, - integral breadths, convergence) and `peak_convergence_diagnostics.txt` only - when a peak did not converge. -- **Error handling**: validation errors are pydantic's, listed together; - PowderLine's own errors derive from `PowderLineError`; the CLI exits 1 on - any failure. - -## The simulation recipe builder - -`powderline.gateways.gsasii.simulation_builder.SimulationRecipeBuilder` turns a -crystal-structure dict (`space_group`, `unit_cell`, `atoms`, `formula`) into a -`gsasii.rietveld` 0.1.0 simulation recipe: the conventional cell reduced to -symmetry-unique sites with an orbit self-check, two-origin groups moved to -origin choice 2, the canonical space-group name, every default stated, SH/L -0.002, instrument defaults from `.powderline_config.yaml`'s -`simulation_defaults`. The Materials Project fetch and `.chi` export that -drove it (`mp-simulate`) left PowderLine in 0.2.0; RandomXRD carries -its own copy. The recipe's `metadata.materials_project` block (material id, -formula, the space group as stated) keeps its name: it is what the tests and -RandomXRD's copy read, and a source-neutral key is a shape decision for the -maintainer. - -## Future roadmap - -- A later minor gsasii - schema version with bounds (clamp-freeze-refit); TOPAS 7 after a - compatibility run on a TOPAS 7 machine; `unsafe_mode` for deliberately - relaxed checks; a recipe builder / CIF importer upstream of the schemas; - a structure-source plugin (Materials Project, CIF) upstream of the builder. - -## Contributing - -Key principles for future work: -- Add features when real examples need them (not speculatively) -- Every rule in the schema; every engine quirk refused or warned about, and - recorded in the engine-behaviour register -- Behaviour changes are deliberate, recorded and reviewed; references move - only with an equivalence record -- Frozen modules only through signed-off, recorded edits; "ask, don't assume" on any open - decision ([policies](policies.md)) -- Add regression tests, fixtures and goldens for new capabilities - -Questions? Open an issue or discussion on GitHub. diff --git a/docs/SCHEMA_HISTORY.md b/docs/SCHEMA_HISTORY.md deleted file mode 100644 index 83d9b55..0000000 --- a/docs/SCHEMA_HISTORY.md +++ /dev/null @@ -1,812 +0,0 @@ -# Schema Evolution History - -PowderLine recipes are validated against a **core schema** (the shared top level -and structural models, owned by PowderLine) plus one **engine schema** per -gateway (`gsasii`, `topas`, `easydiffraction`), each in its engine's native -conventions. Every schema has its own version. A recipe states both -(`core_schema_version`, `engine_schema_version`), and each PowderLine release -**declares** which versions it accepts. Compatibility is never inferred from the -version numbers (`powderline.support_matrix()` lists the declarations). - -> **PowderLine 0.2.0** reads these schemas only. The unified 0.26.0 schema of -> PowderLine 0.1.1 is documented in the archive at the bottom of this page and -> is no longer accepted; `scripts/convert_recipe_026.py` converts a 0.26.0 -> GSAS-II recipe (see the [migration guide](migration-0.1-to-0.2.md)). The -> field tables generated from these schemas are on the [fields page](fields.md). - -## Version numbering: a new line starting at 0.1.0 - -PowderLine 0.2.0 replaces the single recipe schema of PowderLine 0.1.x — -versions 0.1 to **0.26.0**, `schema_name` `GSASII_Rietveld` / `GSASII_SPF` — -with a **core schema** and **one schema per engine** (`gsasii`, `topas`, -`easydiffraction`). Each is versioned separately, and each starts at -**0.1.0** in PowderLine 0.2.0 (`"core_schema_version": "0.1.0"`, -`"engine_schema_version": "0.1.0"`). - -- The old line **ends at 0.26.0**. It is told apart from the new schemas by - `schema_name` (`GSASII_Rietveld` versus `gsasii.rietveld`) and by the split - into `core_schema_version` + `engine_schema_version` (instead of one - `schema_version`), **not by the number**: a new schema at 0.1.0 is not - older than 0.26.0, it is a different schema. -- The new schemas are 0.x because PowderLine is pre-production and they will - still evolve; the bump rule for 0.x versions is in - [policies](policies.md#versioning-and-compatibility). -- `scripts/convert_recipe_026.py` migrates a 0.26.0 GSAS-II recipe to - `gsasii.*` 0.1.0; there is no converter to `topas.*` or `easydiffraction.*` - (see the [migration guide](migration-0.1-to-0.2.md)). - ---- - -## Core schema - -### 0.1.0 — introduced in PowderLine 0.2.0 - -Defined in `src/powderline/schema_core.py`; every engine schema builds on it. - -> **Pre-release revision (not a new version).** Before its first -> release, core 0.1.0 replaced its structure-only phase model (`PhaseStructure` -> with plain values, plus `null`-valued "structure" parameters) with the -> **phase block** described below: one block per phase, every structural -> quantity a refinable parameter, flags and bounds checked against the symmetry, -> and `space_group` in gemmi's canonical spelling. No recipe was ever written -> against the earlier draft. - -- **Recipe frame** (`CoreRecipe`): `schema_name` (`"."`), - `core_schema_version`, `engine_schema_version`, `metadata`, `payload` (typed - by the engine schema). Unknown keys are errors at every level. -- **`metadata`**: free-form and never interpreted; must be JSON-serializable and - at most **1 MiB** as UTF-8 JSON. The cap is the constant - `schema_core.METADATA_MAX_BYTES`; change it there and record the change here. -- **Parameters**: `[value, refine_flag]` (no bounds) or - `[value, refine_flag, min, max]` (bounds the engine honors; `min`/`max` may - be `null` = open). A bounded list where bounds aren't supported is an error. - `value` is always a number and `refine_flag` a JSON boolean; neither is ever - `null`. -- **Data and ranges**: `xrd_data` (2θ in degrees, weights 1/σ², validated as in - 0.26.0); `fit_range` `[min, max]`, with `max > min`, inside the data's 2θ range, holding at least two points with a positive weight. -- **Background**: Chebyshev (`num_coefficients`, `coefficients`, `refine_flag`). - Coefficient semantics are documented per engine. -- **Units** are fixed per field in the schema, never written in a recipe. -- **Numbers** are JSON numbers: a quoted number (`"0.25"`) or a boolean is an - error, never converted. Integer fields (`Multiplicity`, `num_coefficients`) - take a whole number (`4` or `4.0`), not `4.5`. -- **Accepted core versions** (this release): `==0.1.0`. Compatibility is - declared, never inferred ([support matrix](support-matrix.md)). - -- **Phase block** (`Phase`): one block per phase, keyed by the phase name in - the engine payload's `phases` (no `phase_name` field). Core owns - `space_group`, `unit_cell` and `atoms`; each engine schema adds its own phase - fields next to them (e.g. gsasii `scale`, `peak_broadening`) and may add - its own checks, but never redefine, skip or rewrite core's fields and - validation (checked when the engine schema is defined, and by a test). Every structural quantity is a parameter (`[value, flag]`, - or `[value, flag, min, max]` in engines with bounds): cell `a`–`gamma`; atom - `x`, `y`, `z`, `occupancy`, `Uiso` or `U11`…`U23`. `element`, `ADP`, - `Multiplicity` and `space_group` are plain values. Nothing is `null` and - nothing has a default: `occupancy` must be stated; `Multiplicity` (derived, - so optional) is left out when not stated, never `null`. - **Names:** atom labels (the `atoms` keys) and phase names (the engine - payload's `phases` keys) start with a letter and contain only ASCII letters, - digits and `_` (`O1`, `LaB6_a`); phase names, and the atom labels of a - phase, must also differ by more than case. This is the form every engine can use as-is: GSAS-II renames a phase - with surrounding spaces or non-ASCII characters (and its settings were then - skipped), easydiffraction accepts no other atom label, TOPAS writes the names - into its input file, phase names become report file names (which ignore case - on Windows and macOS), and problems are reported at `.`-joined paths - (`atoms.O1.Uiso`). So `"2H-MoS2"` is written e.g. `MoS2_2H`. - Structural interpretation is checked once, in core, so every engine gets the - same structure. **The recipe is the record of the refinement intent**, and - validation never changes what it means. Values fixed or tied by symmetry - follow one rule (see *Symmetry-determined values* below): the first member - of each tie group is stated freely and kept as written; every other value is - derived from the symmetry and must be stated, either exactly (a constant - with a decimal form, such as `0.5` or `0`) or to the precision that states it - beyond doubt (a third, `0.333333`; a value derived from another stated - value). The validated model, which every engine receives, holds the derived - values. PowderLine never rewrites a recipe file; only a Python dump of a - validated model shows a derived value's full spelling - (`0.3333333333333333`), which means the same and validates the same. All problems - are reported together, each at its own field - (`unit_cell`, `atoms.