diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 91b1313d..0e4844e2 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -7,13 +7,13 @@ The vocabulary was a set of base sets plus marker subsets (`particles` and `part - The vocabulary redesign comes before adding many packs: it is designed for 3.0, and what is safe is pulled forward in four phases, (1) the data format and a compiler, snapshot-neutral, (2) derived ambiguity with the gate on every reading kind, (3) pack semantics, (4) the rules menu, order and shapes, and the output model — 2026-10-15/16. - Each word is described by its readings (word → reading → properties), each kind belonging to one category that decides the compiled field it feeds; detailed kinds are recorded even where the kinds of a category behave alike, generation and credential being separate kinds among the suffix category's six (`generation`, `credential`, `honorific`, `professional_honorific`, `status`, `courtesy`) — 2026-10-15/16; 2026-10-18 #737. - A language file holds the words the parser might meet in names written in that language, immigrant particles included, not the words of its etymology, and a word sits in the file of every language whose names it occurs in, a Latin transliteration in its source language's file; `mul.toml` holds the words shared on purpose instead (the generations `i` to `v` and `2`, `&`, and the European titles below) — 2026-10-15/16; 2026-10-18 #737; 2026-10-19 #749; 2026-10-19 #750, title kinds. -- The default loads every language and keeps working for names of any language. Choosing a language is to narrow it in phase 3 (#753): a pack is to settle only the clashes where its language has a vocabulary reading, a strict pack to drop other languages' name-level readings too, and a pack's preferred reading to be able to outrank one a shape rule derives — 2026-10-15/16. +- The default loads every language and keeps working for names of any language. Choosing a language is to narrow it in phase 3 (#753): a pack is to settle only the clashes where its language has a vocabulary reading, a strict choice to read only the chosen languages' files and native-script vocabulary, and a pack's preferred reading to be able to outrank one a shape rule derives — 2026-10-15/16; 2026-10-30 #753, file-strict. - `e`, `y` and `i` stay in the default; choosing a language settles their one-case fork for that language's own reading, `may_be_initial` not holding on a chosen file's conjunction, as `locales.PT` and `locales.PT_BR` do for `e` — 2026-10-15/16; 2026-10-29 #753, phase 3b; 2026-10-30 #753, phase 3g. - A patronymic field is to come in 3.0, as a view that names its words without emptying `family`, and the particle kinds already draw the `clan`/`patronymic` line for it: a patronymic is followed by a parent's name, a clan prefix by a family name the lineage shares — 2026-10-15/16; 2026-10-19 #749; 2026-10-28 #754 (Derek); landed as rules/R9, 2026-10-29 #754, sub-phase 4c. - A shipped pack that is only a choice of languages declares no `DEVIATES`, its surface the words its choice changes; `ga` turns on `nic`, `mhic`, `mic` and `uí` as clan particles, `gd` `nic`, and `pt-BR` `sb` as Sobrinho's spelling, while the single letters `f` and `n` are excluded, the pack included, the letter abbreviating a surname more often than the agnomen in the filings that write it — 2026-10-30 #753, phase 3h. - `es` turns on the courtesy, academic and professional titles that lead a Spanish name with no complement (`lic`, `licenciado`, `licenciada`, `lcdo`, `lcda`, `ingeniero`, `ingeniera`, `arq`, `mtro`, `mtra`, `doctora`, `profesor`, `profesora`, `doña`) and `pt` `doutor`, `doutora`, `professora` and `profa`, `pt-BR` with it; `don`, `dona` and `dom` are excluded, the packs included, names opening with each as a name more often than as the title; `maestro`, a family name whose form of address is written `Mtro.`, is excluded too, and so is `h`, the hijo an initial is written as; `sr` keeps a generation reading beside its salutation in both files, so a choice leaves it the dual position decides; no connective continues a title run, the shape unwritten in name data, and the office titles that take a complement stay out — 2026-10-30 #753, phase 3g. - Of the romanized titles a pack's phase lists, one whose native spelling is a default title turns on unless, counted in persons and pooled across every population measured, names open with it more often than it opens one as the title; a spelling outside the phase's list (`Shreemati`, `Srimati`) stays open until a phase writes it as a reading: `hi` turns on `smt`, `shrimati` and `shriyut`, `bn` `smt`, `shrimati` and `adhyapak` (`shriyut` with no bearer anywhere, `adhyapak` with family-name bearers only, every one where the title claim, acting at the head, does not reach), while `shri`, opening given names in every population measured, stays excluded, and `sushri`, an Odia given name, waits for strict with no reading written, a plain choice being made for data that holds those names — 2026-10-30 #753, phase 3e. -- A language pack is a choice of languages on `Locale.languages` selecting which readings compile, a tag choosing its own file and its ancestors', and a reading index, loaded on its first lookup, carries the languages to run time; strict waits for 3i, and third-party language files are deferred — 2026-10-27 #753 (Derek); 2026-10-29 #753, phase 3a (Derek); 2026-10-29 #753, phase 3b. +- A language pack is a choice of languages on `Locale.languages` selecting which readings compile, a tag choosing its own file and its ancestors', and a reading index, loaded on its first lookup, carries the languages to run time; strict is file-strict, the chosen files and native-script vocabulary only, `mul.toml` and `und.toml` not among them, so it needs no 3i, and its API beyond the `strict=True` argument is not yet decided; third-party language files are deferred — 2026-10-27 #753 (Derek); 2026-10-29 #753, phase 3a (Derek); 2026-10-29 #753, phase 3b; 2026-10-30 #753, file-strict. - A choice is matched without regard to case, a tag whose file does not exist selects its ancestors' (`pt-PT` selects `pt`), `mul` and `und` are no language to choose, and a reading turned on also prefers, so `de`'s generation `i` drops `ca`'s and `pl`'s conjunction under a `de` choice; a choice is one call's: one chained through `base=` applies on top of the base's rather than unioning with it, and a choice drops the caller's own base entries it removes — 2026-10-29 #753, phase 3b; 2026-10-29 #753, phase 3b, chained choices (Derek). - A reading right in its own language and wrong only as a default, because another language written in the same script collides with it, is written `default = false` in its file, never excluded, so that after the packs an exclusion means wrong everywhere: cs `st` since phase 2h, and cs `ml`, hr, sk and sl `ml` and `st` and fr `père` since 3c, the rest in their packs' PRs — 2026-10-27 #753 (Derek); 2026-10-29 #753, phase 3c. - A name reading written `default = false` derives no marker in the default and counts for no check guarding the default's compile; a particle whose word's only `given` readings are off is refused until a pack needs one — 2026-10-29 #753, phase 3a. @@ -54,13 +54,13 @@ The vocabulary was a set of base sets plus marker subsets (`particles` and `part Not standing: the plans the classifications carried out, phase 1's mechanical conversion of the unclassified categories into `und.toml` (2026-10-18 #737) and the shared readings left in place until their categories were classified (2026-10-19 #749), both done by 2026-10-20 #751, the conjunctions; and #747's reason for exempting `und.toml`, its entries keeping the Python lists' order for its vetting comments to point along (2026-10-19 #747), overtaken once it held two entries and no such comment. -Open: #913 (the title words phase 2j left, read as titles though borne as names), #753 (phase 3, language packs; whether a `Parser` records its choice, so that chained choices union, `Parser.languages`, decided with the configure-by-reading method names), #754 (phase 4, order, shapes and the output model). +Open: #913 (the title words phase 2j left, read as titles though borne as names), #753 (phase 3, language packs; file-strict's API beyond `strict=True`, a strict call choosing no language and whether bare choices get a constructor, and which mul.toml words move so that strict `en` can shed them; whether a `Parser` records its choice, so that chained choices union, `Parser.languages`, decided with the configure-by-reading method names), #754 (phase 4, order, shapes and the output model). ## Log - 2026-10-21 #762 (Derek) — `und.toml` STAYS EXEMPT FROM CHECK 10 BECAUSE IT HOLDS NO LANGUAGE'S WORDS, the ground nameparser/vocab/README.md gives. The 2026-10-19 #747 entry exempted it so its entries could keep the Python lists' order that its vetting comments pointed along; #749 to #751 moved every word out but `bon` and `dí` and its per-entry comments with them, so that reason no longer holds, and tools/vocab/build.py's `UNORDERED` comment now gives this one. No compiled set or check moves. -- **2026-10-15/16 — the model (Derek).** Decided over a brainstorm whose notes are gitignored; the durable decisions are these. +- **2026-10-15/16 — the model (Derek)** (decision 6's strict variant, dropping other languages' name readings only, superseded by 2026-10-30 #753, file-strict). Decided over a brainstorm whose notes are gitignored; the durable decisions are these. 1. The vocabulary redesign comes before adding many packs; it is designed for 3.0 and whatever is safe is pulled forward. 2. Each word is described by its READINGS (word → reading → properties), replacing the base-plus-marker pairs. 3. Detailed kinds are recorded during the migration even where every kind of a category behaves alike at first; generation and credential are separate kinds (#326). @@ -254,7 +254,7 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) - **2026-10-26 #752, phase 2l — THE BRACKET ESCAPE READS THE DERIVED CLASH**, carrying out item 16 of the 2026-10-23 #752 entry above (#810). A member of the ambiguous class, the class 2a derives from name readings, alone in brackets or quotes after the name is the suffix where its writing settles it (`John Smith (MA)`, `John Smith (M.A)` → suffix), read in one pass, and stays the nickname otherwise (`John Smith (Ed)`, `JOHN SMITH (MA)`, `Jeffrey (JD) Bricken`). It is recorded as rules/S13, a rule of its own rather than the amendment to S1 item 16 planned, the position it reads being evidence S1's signal does not read; the writing is S3's and S6's evidence, which S13 cites. The record, the measurements and what was declined are decisions/S13's 2026-10-26 #810 entry. - **2026-10-26 #752, phase 2m — THE READING NOT TAKEN SURFACES AS `Ambiguity.alternative`, WITHOUT ITS LANGUAGES**, carrying out item 13 of the 2026-10-23 #752 entry above and item 7's source with it: a word the credential's run took by position alone reports its name reading's source as `position`, a fourth source beside the three the item named, and a word with a declared name reading reports `declared`. The pickle guard was widened as item 13 asked, every skew the narrow guard refused still refused. The reading's languages were left out: the per-word index costs about 0.35 ms of an 18.2 ms import (decisions/A1) and describes the default vocabulary, not a pack's or a caller's lexicon, so they wait for 3.0 with `Token.reading`. The record, the measurements and what was declined are decisions/A1's 2026-10-26 #752 entry. -- **2026-10-27 #753 (Derek) (its item 10's clause making ZH's surname list and the RU and TR_AZ endings rule data for `[rules]` superseded by 2026-10-29 #754, sub-phase 4f) — PHASE 3'S DESIGN IS AGREED: A CHOICE OF LANGUAGES SELECTS THE POOL, ONE MECHANISM FOR TURNING READINGS ON AND PREFER; NAME READINGS IN A PACK KEEP THE GATE ON; A READING INDEX CARRIES THE LANGUAGES TO RUN TIME; STRICT WAITS.** Derek accepted every item as written in chat on 2026-10-28, items 1, 2, 4, 5, 6 and 8 first, then 3, 7 and 9 to 12. One design inside item 11 stays open: 3g's connective, which accepting the work order did not decide. The design overview is docs/design/vocabulary-model-rfc.md's Phase 3 section. Decisions 15 and 20 are kept as written; decision 6's strict variant is reshaped to a property of the whole choice and deferred (items 1 and 4). Phase 3 is 3.0 work since the 2026-10-28 re-plan (item 12). The inputs are #753's comments of 2026-10-27: the inventory of 411 kept-out candidate readings with their verdicts, and the probe of foreign titles and function-word collisions. Measurements were taken 2026-10-27 on master `cfe8bf92` (Python 3.11.17) with throwaway Lexicon edits standing in for a pack, so each is evidence for the proposal and each sub-phase re-measures on its own base. Local-corpus figures are counts. +- **2026-10-27 #753 (Derek) (its item 10's clause making ZH's surname list and the RU and TR_AZ endings rule data for `[rules]` superseded by 2026-10-29 #754, sub-phase 4f; item 2's definition of strict, item 4, and 3i in item 11, superseded by 2026-10-30 #753, file-strict) — PHASE 3'S DESIGN IS AGREED: A CHOICE OF LANGUAGES SELECTS THE POOL, ONE MECHANISM FOR TURNING READINGS ON AND PREFER; NAME READINGS IN A PACK KEEP THE GATE ON; A READING INDEX CARRIES THE LANGUAGES TO RUN TIME; STRICT WAITS.** Derek accepted every item as written in chat on 2026-10-28, items 1, 2, 4, 5, 6 and 8 first, then 3, 7 and 9 to 12. One design inside item 11 stays open: 3g's connective, which accepting the work order did not decide. The design overview is docs/design/vocabulary-model-rfc.md's Phase 3 section. Decisions 15 and 20 are kept as written; decision 6's strict variant is reshaped to a property of the whole choice and deferred (items 1 and 4). Phase 3 is 3.0 work since the 2026-10-28 re-plan (item 12). The inputs are #753's comments of 2026-10-27: the inventory of 411 kept-out candidate readings with their verdicts, and the probe of foreign titles and function-word collisions. Measurements were taken 2026-10-27 on master `cfe8bf92` (Python 3.11.17) with throwaway Lexicon edits standing in for a pack, so each is evidence for the proposal and each sub-phase re-measures on its own base. Local-corpus figures are counts. 1. **A pack is a choice of languages carried by `Locale`.** `Locale` gains `languages`, a set of BCP 47 tags; `parser_for` unions the sets of the packs it is given, so the choice is order-independent while policy patches still fold in order; strict is an argument of the whole choice (`parser_for(locales.NE, locales.HI, strict=True)`), since a strict pack beside a preferring one has no meaning, reshaping decision 6's "strict variant". The shipped language packs are `Locale` constants with an empty lexicon fragment (`locales.NE`, `locales.PT_BR`); what a fragment holds once 3.0 configures vocabulary by reading is the 2026-10-28 #754 entry on the `Lexicon` surface. The dependency this rested on, `Locale` and `Policy` keeping their 2.x shape once the facade is gone, is answered by the 2026-10-28 3.0 freeze: both keep it, with additions. Weighed against: `Parser(languages=…)` beside `Locale`, two ways to choose one thing, and a `Policy` field, `Policy` being behavior and vocabulary being `Lexicon`'s. 2. **One mechanism.** The chosen languages select which readings the compiler compiles, the default being that compiler with every language and no choice. Turning on, prefer and strict are derived from the file each reading sits in, and nothing is written per pack beyond its language's readings: a chosen language's `default = false` readings join; prefer drops other languages' vocabulary readings of a word on which a chosen language holds one, never `mul.toml`'s (`Haddad, Abd`, suffix `Abd` today, reads given `Abd` with the en credential gone); strict drops other languages' name readings and the markers they derive. A clash a property records with a SHAPE reading is settled for the chosen language's own reading, decision 20 applied: `may_be_initial` does not hold on it, so a `pt` choice reads `ANA SILVA E SOUZA` family `SILVA E SOUZA` where the default reads middle `SILVA E` with a report. This answers decisions/3-0-reevaluations' 2026-09-13 bullet that a pack cannot remove `e`, and its 2026-08-17 bullet's "overrides rather than additions": a pack overrides by selection, never by a removal list. That removes base entries all the same, so mechanisms.md#LOCALE-PACKS-PURE-DATA's add-only contract is amended, in 3b's PR, which builds the selection (items 11 and 12), not by this decision. The shape-clash rule follows from decision 7, which decision 20 permits; it reaches every reading `may_be_initial` stands on (pt and it `e`, ca and pl `i`, the Italian one unjudged), and it forecloses an `nl` pack marking `y` (the 2026-08-18 bullet there), which waits with `fru` below. New beside decision 6: `mul.toml`'s exemption from prefer. Left open rather than built: a property a language's own reading should take only under its pack, Swedish `fru` and Icelandic `frú` as after-given salutations (rules/H8), refused in the default for the Cameroonian surname, the two readings in the inventory needing it. 3. **`default = false` is allowed on `given` and `family` readings, and turned on they derive markers like any name reading, never settling (decision 15 unchanged).** Measured on nepal-candidates' 126,981 Devanagari and romanized pairs, comparing every word's role across the two spellings: 123,440 read alike in the default; `given` readings on `श्री`, `guru` and `गुरु`, which make each a title only inside a title phrase (rules/H9), give 123,636 (198 fixed, 2 broken, both pairs whose romanization opens with a listed title); a `family` reading on `dc` as well, making the credential ambiguous, gives 123,639, the count still leaving `Ram Bahadur DC` a suffix; dropping `dc`'s credential, which settling by a name reading would do, gives 123,658, 19 more. Settling is not taken: the 19 pairs are the accepted price of decision 15, whose `do` case it protects. `bk`, `kc`, `gc` and `bc` read as the family in every shape tried (spoken order, a family comma, `FAMILY_FIRST`, dotted and Title case), so a reading would move nothing and none is added. Recompute: the readings moved into or out of `Lexicon` fields by hand, the pairs compared by `tuple(t.role for t in parser.parse(s).tokens)`. @@ -376,7 +376,7 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) - **The default does not move.** The build reports `_compiled.py` current; the committed snapshot gained the three names rules/S11's new examples add to the rules corpus and no existing entry moved; `snapshot.py --local tse-donors --base` against master `40675146` matches the base. - **What each choice moves,** measured 2026-10-30 on `40675146` plus this change, Python 3.11. Committed corpora (`snapshot.py --languages TAG`): `ga` and `gd` 4 entries each, every one `mc` (`John Smith MC` family `MC`, where the default reads the Military Cross), the prefer of the phase 3b entry's turning-on bullet on the `mc` both files already held; `pt-BR` 40, `pt`'s 38 (1 held only by `corpus_cldr.jsonl`) and the two of rules/S11's new examples that read `SB` (`JOSE CARLOS SILVA SB`, `Ana SB`). CLDR's locale view (`cldr_agreement.py`, `ga`, `gd` and `pt` mapped to the packs, CLDR's `pt` being Brazilian): `ga` 14/24, `gd` 15/24, `pt` 17/24 renderings agreeing with and without the pack, recorded. Fielded registers, scored over every record holding a word the choice changes (`local_agreement.py`'s `read` under each parser): PSC, `ga` 4 fixed (`Clodagh Nic Lochlainn`, `Orlaith Mhic Mhathuna`, and `Damien Patrick Mc` by the `mc` prefer) and 1 broken (`Nicholas Nic Rawlings Limited Rawlings`, a company filed as a person), `gd` 3 and the same 1; ORCID, `ga` 24 fixed and none broken over every country (23 in Ireland and Great Britain, `Pauline Uí Dhuibhir`, `Éimí Mhic an Ridire`), `gd` 20 and none (19 in Ireland and Great Britain). `pt-BR`'s moves there are all `pt`'s: PSC 12 broken, each a trailing `Sr` read as the title Senhor (`Gary Holloway Sr` family `Sr`), ORCID 21 fixed (the one-case `e`) and 11 broken. - **Open for 3g:** a `pt` choice prefers `pt.toml`'s title `sr` (Senhor) over English's generation, so a trailing `Sr` reads as a title or the family name under `pt` and `pt-BR` (PSC's 12 above); whether the generation survives a Portuguese choice is 3g's, which owns `pt`'s titles. **Accepted: under the pack a written S.B. degree reads as Sobrinho's generation.** In ctgov-officials (`snapshot.py --local ctgov-officials --languages pt-BR --base`, against `--languages pt`), 17 names move beyond `pt`'s: 12 only in `capitalized(force=True)`, a mid-name `SB` keeping its capitals, and 5 where the degree after a comma or a credential now fills `generation` rather than `credentials` (`Howard S. Gordon, MD SB`, `Waleed Riad, MD, AB, SB`), one of them moving fields (`Silvio Brugger, SB`, given `SB` by default, suffix under the pack). The degree is a suffix either way; which view holds it is the cost of a choice made for Brazilian data. The recipe for the fielded figures is the scratch scorer this entry describes: `local_agreement.read` per record under `Parser()` and `parser_for(pack)`, over the records whose name holds a word of `changed_words(selected(pack.languages))`. -- **2026-10-30 #753, phase 3e — THE HINDI AND BENGALI PACKS, `locales.HI` AND `locales.BN`: THE ROMANIZED `smt`, `shrimati` AND `shriyut` (HI) AND `smt`, `shrimati` AND `adhyapak` (BN) TURN ON AS TITLES; `shri` AND `sushri` STAY OUT, MEASURED; THE DEFAULT DOES NOT MOVE.** Carries out 3e of item 11 of the 2026-10-27 #753 entry, for #345's Latin forms. +- **2026-10-30 #753, phase 3e (its `sushri` bullet's wait for 3i superseded by 2026-10-30 #753, file-strict) — THE HINDI AND BENGALI PACKS, `locales.HI` AND `locales.BN`: THE ROMANIZED `smt`, `shrimati` AND `shriyut` (HI) AND `smt`, `shrimati` AND `adhyapak` (BN) TURN ON AS TITLES; `shri` AND `sushri` STAY OUT, MEASURED; THE DEFAULT DOES NOT MOVE.** Carries out 3e of item 11 of the 2026-10-27 #753 entry, for #345's Latin forms. - **What each pack turns on.** `locales.HI` chooses `hi` and `locales.BN` `bn`, each with an empty lexicon fragment and no policy patch, so each is a choice alone and declares no `DEVIATES` (the phase 3d entry's item 2). hi.toml gains `salutation` readings `default = false` for `shrimati` (f), `smt` (its `same_as`) and `shriyut` (m), each `after_given` as its Devanagari spelling is; bn.toml gains `shrimati` and `smt` the same way and `adhyapak` as an `academic` reading `default = false`. None of the words has another reading in any file, so the choice turns each on and prefers nothing: `changed_words` is `{shrimati, shriyut, smt}` for `hi` and `{adhyapak, shrimati, smt}` for `bn` (pinned by `test_hi_and_bn_pack_contents`). A word turned on is read wherever the title vocabulary is: at the head, behind a listed title, after a family comma, ahead of one with a name word behind it (rules/H6, `Shrimati Gandhi, Indira` → family `Gandhi`), where it ends the given part after one (rules/H8, `Sitharaman, Nirmala Smt`), written with its period at the end of the name (rules/H5, `Indira Gandhi Shrimati.` → family `Gandhi`), and as a chunk of a period-joined token (the Accepted bullet below). Written with its period, rules/H2's shape already reads it as a title by default at the head, behind a listed title, after a family comma and alone after one (`Smt. Nirmala Sitharaman`, `Dr. Smt. Rekha Sharma`, `Sitharaman, Smt. Nirmala`, `Rekha Sharma, Smt.`), so a dotted spelling moves wherever else the title vocabulary is read, which a sweep of 18 shapes (head, behind a title, mid-name, end, end before a suffix, and each comma shape with a one-word, full and empty head, a second comma and a credential beside it) finds to be: ending the name, a suffix peeled behind it or not (H5, `Rekha Sharma Smt. Jr.` → title `Smt.`, family `Sharma`); ending the given part after a family comma (H8, `Sitharaman, Nirmala Smt.`); ahead of a family comma with a name word behind it (H6, `Smt. Gandhi, Indira`); ending what follows an empty head, which is not a family comma (rules/C15), so H8 does not name it (`, Rekha Smt.` → title `Smt.`, family `Rekha`, where the default reads given `Rekha`, middle `Smt.`); any comma part after the name that holds it beside a credential or past a second comma (rules/C2, `Rekha Sharma, Jr., Smt.` → title `Smt.`, suffix `Jr.`, where the default reads suffix `Jr., Smt.`; `Rekha Sharma, PhD Smt.`; `Sharma, Rekha, Smt.`); and a period-joined chunk; bracketed at the end it counts as ending the name, its period keeping it from the nickname (`Rekha Sharma (Smt.)` → title `Smt.`, family `Sharma`, while `(Smt)` stays the nickname); mid-name it stays a name word (`Rekha Smt. Sharma`); bare at the end of the name it stays a name word, as every bare title does (`Indira Gandhi Shrimati`). Each reading's counts are on hi.toml's reading, bn.toml's pointing to them, from `tools/name_usage.py smt shrimati sushri shriyut shri adhyapak --boxes --country IN --country BD` (2026-10-30) and a pass over ORCID's whole extract by box and position. - **`shrimati` ships, weighed against its two name bearers.** ORCID's whole extract has 3 given boxes holding it: `SHRIMATI`|`SHETTY` and `Shrimati`|`Shetty`, likely one person, the word alone as the given name, and `Shrimati Datta`; under either pack `Shrimati Shetty` reads title `Shrimati`, family `Shetty`, and no given name, a cost. The title side is nepal-candidates' 16 romanizations, whose Devanagari is the salutation in every one (`श्रीमती` 11, `श्रीमति` 4, the spaced `श्री मती` 1), and the word's Devanagari twin, a default title. That data is Nepali, not the Hindi or Bengali data the packs are for, which no corpus here holds, so the measured split is 16 leading uses as the title against 1 person bearing it as a name (the Shetty pair; `Shrimati Datta`, a given box with no family box, reads equally as the salutation and a surname), from populations beside the pack's own. That is the reverse of `shri`'s 198 openers against 1 fix, and at this volume the title reading holds under the Standing criterion, which pools the populations in persons: 16 openings as the title against 1 name, though within ORCID alone the name (the Shetty pair) is at least as many as the title (`Shrimati Datta` at most). `srimati`, the same word in Bengali spelling, is not judged by the same counts (Open below): it was not measured as a reading. - **`shri` is left out: its exclusion stays in hi.toml, with the counts.** In every population that writes it, a name opens with it far more often than the salutation does. ORCID's whole extract (24,990,909 records) has 198 given boxes opening with `Shri`, 52 of them `Shri` alone and most of the rest a given name written in two words (`Shri Kant`, `Shri Ram`, `Shri Lakshmi`, `Shri Vaishnavi`), Hindi and Tamil names alike; nam_dict bands it 2 in India, and Wikidata holds 7 given. Turned on (measured on a tree whose hi.toml held it, its exclusion removed, as a `salutation` reading `default = false`, and bn.toml the same reading), it broke 20 filed names and fixed none: scored box by box, PSC 2 (`Mr Shri Ram Neupane`, its forename `Shri`), ORCID's local corpus 5 and PatentsView 4 (`Shri Ram Yadav`), and FEC 9, each `SHRI` the first name after the family comma (`BANGALEY, SHRI` → title `SHRI`, no given). It moved nepal-candidates' 10 `Shri` romanizations to the title, 4 of them the one Devanagari word `श्रीनारायण` (`Shri Narayan Paswan`), and in jrc-names fixed 1 (`Shri Dharmendra Pradhan`). No CLDR rendering writes it (CLDR's `hi_Latn` writes the salutation `Shree`). This is not a clash prefer settles, since the word has no other vocabulary reading, nor one strict would: its bearers are the pack's own language's. It is the same-language clash the RFC's phase 3 section sends down phase 2j's route, a `given` reading and the title phrase (rules/H9), as default work; Open below. @@ -413,3 +413,36 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) - **Recompute:** `uv run pytest -n0 tests/v2/test_shape_inventory.py`; the counts by order from `ARRANGEMENTS` and `SHAPES` (each row's order is `SHAPES[row.shape].order`); the corpus and snapshot counts from `git diff --numstat` against `1f6bdf1d` after `uv run python tools/differential/build_shapes_corpus.py` and `uv run python tools/differential/snapshot.py --record`; the control by parsing each family-first row with `Parser()`. - **Disclosed, not added:** the two title-particles at the head written with periods (`Freiherr St. van Berg MA`, `Freiherr. St. van Berg MA`) read as the bare spelling does, the title run whole, under every order. - **Open, unweighed and held by no issue, so these rows carry no mark:** an ambiguous particle opening a family-first name is left where position puts it and reported, as root AGENTS.md's particles paragraph describes, though no decision weighs it there (`van Gogh Vincent` under `FAMILY_FIRST`: family `van`, given `Gogh`; `du Ménil Thi Mai`, its letter decomposed, under `FAMILY_FIRST_GIVEN_LAST`: family `du`, middle `Ménil Thi`); rules/P1 covers a never-given particle only. And `Vega y Lopez Jr., PhD Jones` under the family-first orders reads given `y`, middle `Lopez` (`FAMILY_FIRST`) and given `Lopez`, middle `y` (`FAMILY_FIRST_GIVEN_LAST`): the default order's division of the words laid out by the order, which decisions/C3 records under the default order only. And a Hungarian title after a married `-né` word, before the woman's birth family name (`Bakainé dr. Korbács Melinda`), 102 of 37,250 names in the 2024 municipal candidates and every one misread today (decisions/H5's #938 entry has the measurement): what licenses the title there is the `-né` word, so it is a question for a future `hu` pack, which no sub-phase schedules, and not for a general rule. +- **2026-10-30 #753, file-strict — STRICT IS FILE-STRICT: A STRICT CHOICE READS ONLY THE CHOSEN LANGUAGES' FILES AND NATIVE-SCRIPT VOCABULARY, NOT `mul.toml` OR `und.toml`, SO IT NEEDS NO 3i; THE `strict=True` ARGUMENT STANDS, THE REST OF ITS API IS PROPOSED, NOT DECIDED, AND NOTHING IS BUILT.** SUPERSEDES: decision 6's strict variant in the 2026-10-15/16 entry, and in the 2026-10-27 #753 entry item 2's definition of strict (other languages' name readings and their markers leave), item 4 (strict waits until name readings say whose names they are) and 3i in item 11. Item 1's `strict=True` argument of `parser_for`, over the whole choice, stands. The definition and items 1 to 3 were decided in chat on 2026-10-30; the measurements were taken on 2026-10-31 on master `b577e9a8`. The design overview is docs/design/vocabulary-model-rfc.md's "Strict: the chosen files and nothing else". + 1. **The definition.** A strict choice compiles only the chosen languages' files, each tag selecting its own and its ancestors', plus native-script vocabulary, which stays on because it selects itself by script (the CJK honorifics; the Korean surname list, outside the per-word model, is in no choice's compile and stays too); everything else leaves, vocabulary and name readings alike. A word with no reading in what is compiled is read by position and shape alone, so under strict `vi`, `do`, which only en.toml and pt.toml hold, is a name word, while the shape rules still act (`Dr. Minh Do` keeps title `Dr.`, rules/H2; `Minh Do, M.D., Ph.D.` keeps suffix `Ph.D.` past the second comma, rules/C2, but reads given `M.D.`). For a credential, the harm item 4 of the 2026-10-27 entry guarded against (`Minh Do` → suffix `Do`) does not arise today, measured 2026-10-31: no strict choice of one tag ungates a credential acronym, each credential's gating name reading sitting in its own file. It holds only while name readings are added to more files and never moved: moving `do`'s Đỗ readings from en.toml to vi.toml would ungate its credential under strict `en`. For the other kinds it can, where the vocabulary reading is in a chosen file and the gating name reading only in a dropped one (the ungated-words bullet below; strict `pt` reads `Do Van Minh` family `Do Van Minh`). 3i, the attribute naming each name reading's bearers' languages, is still not needed for strict: under file-strict that case is remedied by data, a name reading written in the chosen file whose names carry the word. Name-strict, the earlier definition, under which other languages' titles, credentials and particles stayed and only their name readings left, is set aside with 3i. + 2. **`mul.toml` and `und.toml` are not in a strict choice.** `mul.toml` stays as it is: its entries name several languages each or "the European languages" (`dr`, `prof`), or carry no note (the numerals, `&`), so under file-strict it behaves as one more language's file. Open: entries may later move between en.toml and mul.toml, or into the files of the languages that write them, so that strict `en` can shed problem words; which words those are is not yet known. + 3. **Dropping English credentials under strict `vi` is accepted in principle.** Vietnamese writes its degrees in Vietnamese, before the name (`TS.`, `ThS.`, `GS.`, `PGS.`, `BS.`, stacked as `PGS.TS.`), and has no generational suffix; English `PhD` and `MD` come with English-context writing. Measured below rather than assumed. + - **Measured: ORCID agreement per country** (`local_agreement.read` box by box over each country's capped records in the cached extract, fetched 2026-10-27; the records whose given box opens with a title word left out, 1,717 for IN, 437 for NP and under 70 for every other country, since ORCID has no title box and a holder who typed `Dr` there scores a fix whenever `dr` leaves the vocabulary; fixed and broken against the default). "choice" is the shipped pack carrying the language, else a bare choice of the tag: + + | Country | records | default | choice | strict | strict + en | + |---|---|---|---|---|---| + | VN | 20,000 | 65.48% | none, no file holds `vi` | 64.28% (+349 −589) | 64.34% (+306 −534) | + | KR | 20,000 | 92.42% | 92.42% (+0 −0) | 92.54% (+41 −17) | 92.51% (+33 −16) | + | CN | 20,000 | 84.54% | 84.54% (+0 −0) | 84.65% (+23 −3) | 84.61% (+16 −3) | + | JP | 20,000 | 95.59% | 95.59% (+0 −0) | 95.64% (+30 −20) | 95.54% (+11 −20) | + | CZ | 20,000 | 95.12% | 96.54% (+635 −352) | 96.42% (+626 −367) | 96.43% (+625 −364) | + | SK | 12,962 | 95.01% | 96.58% (+347 −144) | 96.53% (+347 −150) | 96.59% (+348 −143) | + | PL | 20,000 | 96.88% | 96.88% (+0 −0) | 96.74% (+4 −32) | 96.78% (+4 −25) | + | DE | 20,000 | 95.01% | 95.00% (+0 −1) | 94.42% (+8 −125) | 94.44% (+7 −120) | + | ES | 20,000 | 48.04% | 93.47% (+9,295 −215) | 93.21% (+9,286 −258) | 93.22% (+9,285 −255) | + | MX | 20,000 | 31.33% | 93.01% (+12,534 −204) | 92.93% (+12,524 −210) | 92.93% (+12,523 −209) | + | BR | 20,000 | 47.30% | 47.33% (+5 −0) | 46.90% (+18 −99) | 47.22% (+9 −26) | + | PT | 20,000 | 69.99% | 69.99% (+0 −0) | 69.87% (+1 −25) | 69.90% (+1 −19) | + | NP | 17,541 | 90.85% | 90.86% (+1 −0) | 90.74% (+18 −37) | 90.85% (+3 −3) | + | IN | 20,000 | 79.82% | 79.82% (+0 −0) | 79.85% (+23 −17) | 79.80% (+6 −10) | + + The ES, MX, CZ and SK gains are the packs' `two_family_names`, in the choice and strict alike; strict against the choice is CZ +9 −33, SK +5 −11, ES +11 −63, MX +6 −22, BR +13 −99, PT +1 −25, NP +17 −37 and IN +23 −17, its breaks mostly other languages' words borne in that country's names, counted as the breaks whose name holds the word (DE `de` 45 and `van` 39 of 124, BR `junior` 62 and `jr` 12 of 99, both in en.toml, pt.toml holding only `júnior`, ES the Catalan `i` 39 of 63). VN's loss is ORCID's family box holding the middle names, which the default's French `le` and Dutch `van` happen to land on (`Manh Le Ngo Duc`, family box `Le Ngo Duc`); holders who write `Văn`, which no vocabulary reads, box it with the family 20 times of 73, against 144 of 802 for `van`. KR, CN and JP gain from Latin words that are their names (`bin`, `do`, `gen`, `de`, `du`). + - **Measured: the `van` chain.** ORCID VN, `van` before the last word, the name split at spaces only: 802 records (803, 404 and 3 split at commas too, `Master Thuy Van, TT` joining), the default reading it into the family in 403 and the holders in 144; strict reads it there in 2, and the 802 agree 583 times against the default's 479 (boxed with the given names, 658: 354 to 583; boxed with the family, 144: 125 to 0). PSC's Vietnamese filers, which box the middle name apart: 156, 5 boxing `van` as the surname; the default reads it into the family in 97 and agrees 62 times, strict + en in none and 147 times. Over all 1,499 Vietnamese PSC filers, strict + en fixes 93 and breaks 36, 91 of the fixes holding `van` and 20 of the breaks `le`. + - **Measured: the cost of other languages' credentials and salutations.** PSC: strict `vi` moves 865 of 1,499 Vietnamese filers off their boxes (the parse's given name `Mr` in 466, `Ms` 213, `Mrs` 98, `Miss` 73, `Mr Anh Cong Tran` → given `Mr`), strict + en 36; South Korean, 390 of 556 and 13. ClinicalTrials.gov officials, distinct names, whose name before the first comma holds one of 18 common Vietnamese surnames (nguyen tran le pham hoang huynh phan vu vo dang bui do ho ngo duong ly truong dinh): 1,697, 1,247 with a suffix by default; strict `vi` changes 1,112 of those suffixes and reads one of the default's suffix words (the suffix split at spaces and commas) as the given name in 1,079 (`Ngoc P Ly, MD` → given `MD`), strict + en 9 and 0. With 18 Korean surnames (kim park choi jung jeong kang cho yoon yun jang lim oh seo shin kwon hwang ahn yoo): 5,686, 4,224 with a suffix; strict `ko` 3,678 and 3,569, strict + en 5 and 0. Counted over raw.jsonl's records instead, which repeat an official per study, the figures are 2,319, 1,764, 1,603 and 1,562, and 8,068, 6,208, 5,550 and 5,426. The slice is rough and the corpus holds no ground truth: these are moves. + - **Measured: what each file lacks to stand alone,** the words a country's ORCID or PSC records write that its strict choice loses: nine of the ten rows a bare `dr` and three `prof` (mul.toml; `Dr.` stays a title by its period; the pt rows rank neither), and in PSC the salutations (en.toml); cs and sk also `ing` (mul.toml), `phd`, `md`, `msc`, `mph`, `jr` (en.toml); pl `maj`, `phd`, `md`; de `v` (mul.toml), `md` and the immigrant particles; es `i` (ca, de, mul and pl), `e` (it and pt), `phd`; pt and pt-BR `junior`, `jr` and the Italian particles; ne and hi `phd`, `md`. The losses that are gains, Latin words that are that country's names: ko `jo` (fil), `se`, `uk`, `ma`; zh `ma`, `meng`, `si`, `chi`, `che`, `du`; ja `gen`; ne and hi `kc`, `das`. + - **Measured: files with no readings.** hu.toml, kk.toml, kok.toml, vi.toml and zu.toml hold only exclusions, so `selected` refuses each tag; strict `vi` needs vi.toml readings first. 26 more hold no Latin-script vocabulary reading, so strict of one is the native-script pool and that file's name readings (am, as, be, bg, el, gu, ja, km, kn, ko, lo, mk, ml, mr, my, ne, or, sd, shn, si, ta, te, th, ti, uk, zh). + - **Measured: a derived marker does not always leave with the reading it gates.** Where a vocabulary reading is in a chosen file and the name reading gating it only in a dropped one, strict keeps the reading ungated: 16 words, 17 readings (`st` both as a particle and as a generation), 27 word-and-tag pairs over 13 tags, each strict choice of one tag compared with the plain choice of the same tag: `cheikh` and `sultan` (ar), `la`, `marquis`, `marquise` and `st` (fr), `mac` (ga, gd), `nic` (gd, its name readings in ga.toml), `del`, `la`, `san` and `santa` (it), `bin` and `sultan` (ms), `den` and `tho` (nl), `da`, `do` and `santa` (pt, pt-BR), and the generation `st` turned on by `cs`, `hr`, `sk` and `sl` (its `given` reading in en.toml). Under the plain `cs` choice `Novák, St` reads given `St`, reported suffix-or-name, under strict `cs` suffix `St` with no report; under `gd`, `Nic Smith` reads given `Nic`, reported particle-or-given, under strict `gd` family `Nic Smith`. Strict `pt` reads `Da Costa` family `Da Costa` (given `Da` by default and under the choice); strict `ar` reads `Sultan Ahmed Khan` title `Sultan`, `sultan`'s `given` reading sitting in tr.toml though the name is Arabic too. The remedy under file-strict is data, a name reading in each file whose names carry the word. + - **Proposed, not decided (the RFC section has the arguments):** a strict call raising where it chooses no language; no `Locale.strict` field; the shipped language-choice constants kept for packs that carry something, ZH, JA, RU and TR_AZ as they are, with `locales.choose(*tags)` for a bare choice; a strict choice turning on the chosen files' `default = false` readings and no other file's; patches and fragments untouched, so `locales.ES`'s given-name list (1,000 entries under strict `es`, measured) and `locales.ZH`'s surnames stay; `Lexicon.given_names` and `Lexicon.surnames` outside the compile, untouched; stacking a union; a caller's own listing of a dropped word dropped, as 3b's delta drops it (a base adding `dr` and `zzz` keeps `zzz` under strict `vi`, measured), a caller adding it after the call. + - **3i's evidence, kept** from the RFC paragraph this entry retires, should 3i be wanted again: all measured 2026-10-28 on master `c8357e74`, not on this entry's tree: of the 227 name readings, 165 words carried one; `do`'s family box led by Brazil in ORCID (none of Brazil's 7,227 `do` alone, 2,629 opening with it and 4,598 holding it inside) but by share Vietnam (1.90% against 0.81%; PatentsView 1.78% and 0.64%), Vietnam having 618 boxes of `do` alone and Korea 151; `kim`'s family box 17.67% of ORCID's Korean holders and 13.30% of PatentsView's, its given box 0.93% of PatentsView's Danish inventors. Recompute: `uv run python tools/name_usage.py do kim --no-wikidata --boxes`, and the whole-box split from the cached ORCID extract's `family` and `country` keys. + - **Candidate next steps, not taken here:** Vietnamese degree prefixes as vi.toml readings, each checked for collisions and bearers per nameparser/vocab/AGENTS.md first; `ông` and `bà` stay excluded, Ông being a Vietnamese surname, a clash within Vietnamese strict does not settle; 3h's `nl` prepositions and 3e's `sushri` as data with the clashing name readings declared in ko.toml, vi.toml and or.toml, gated under the plain choice and not under strict, though the plain `nl` choice still reads `Park In Hwan` family `In Hwan` and `Nguyen To Nga` family `To Nga`, item 4's recorded cost, the marker acting at the head only (re-run 2026-10-31), which a follow-up must settle before shipping them; `_reading_index`'s `UNCHOOSABLE` comment and `selected`'s refusal text, which say no choice removes a reading of `mul` or `und`, updated with 3j; the 16 ungated words read one by one. + - **Recompute:** a strict choice of the tags `T` is `compile_readings(choose(rows, T))` from `nameparser._reading_index`, `rows` the `index()` rows whose tag is in `T` or whose text has a letter and no letter whose `unicodedata.name` contains `LATIN`, applied to `Lexicon.default()` as `_parser._apply_choice` applies `_choice_delta` (the delta taken against `compile_choice(frozenset())`, the markers of a dropped base entry removed), and a pack's own patch and fragment applied by `parser_for(dataclasses.replace(pack, languages=frozenset()), base=Parser(lexicon=that))`. Count a word's share of the moves as the moves whose name, split at spaces and commas with edge periods dropped and casefolded, holds it. Score ORCID with `local_agreement.read(record, local_agreement.FIELDED["orcid-researchers"], parser, local_agreement._title_words(parser.lexicon))` over the cache's `raw.jsonl` records of each country, and PSC the same with `"psc-individuals"` and the `nationality` key; the title-word filter is the given box's first word, split at spaces and periods and casefolded, in {dr, prof, professor, mr, mrs, ms, miss, dra, ing, mgr, assoc, assist, er, engr, drs, ir}. The ungated words: for each choosable tag, the words in a marker (`particles_ambiguous`, `suffix_acronyms_ambiguous`, `suffix_words_ambiguous`, `titles_ambiguous`) of the plain choice of that tag (`parser_for(Locale(code="x", lexicon=Lexicon.empty(), languages={tag})).lexicon`) that stay in the base set of the strict lexicon and leave the marker, or join `titles`; compared with `Lexicon.default()`'s markers instead, it misses the readings a choice turns on (`st` under cs, hr, sk and sl, `nic` under gd) and finds 15 words. The scripts are in no tree: they are a few dozen lines each from this recipe. diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index 29d003be..8608cda1 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -26,7 +26,7 @@ Four phases, in a fixed order: |---|---|---|---| | 1 | The per-word TOML format, compiled to today's sets | no | shipped in 2.6 (#737, #744); #749–#751 gave the words in `und.toml` their languages, leaving two particles of unrecorded origin | | 2 | Ambiguity derived from clashing readings; the ambiguity gate extended to every reading kind | yes, the first phase that does | #752, design agreed 2026-10-23; 2a, the two name-clash markers derived, landed without moving a parse (decisions/vocabulary-model, 2026-10-25) | -| 3 | Language packs that prefer or restrict their language | yes, for callers who choose one | #753, for 3.0, needs 1 and 2; design agreed 2026-10-28, 3g's connective declined 2026-10-30; 3a, the reading index and the shared compiler, 3b, the choice on `Locale.languages`, and 3c, the pack-shaped exclusions as readings off by default, landed without moving a default parse (decisions/vocabulary-model, 2026-10-29) | +| 3 | Language packs that prefer or restrict their language | yes, for callers who choose one | #753, for 3.0, needs 1 and 2; design agreed 2026-10-28, 3g's connective declined 2026-10-30; 3a, the reading index and the shared compiler, 3b, the choice on `Locale.languages`, and 3c, the pack-shaped exclusions as readings off by default, landed without moving a default parse (decisions/vocabulary-model, 2026-10-29); strict redefined on 2026-10-30 as file-strict, the chosen files and native-script vocabulary only, which needs no 3i, the rest of its API proposed and the definition measured on 2026-10-31 (decisions/vocabulary-model's 2026-10-30 #753, file-strict entry) | | 4 | Order and shape; the rules a pack can choose; the output model | yes | #754, 3.0 work; its output model agreed 2026-10-28, ahead of or alongside phase 3's mechanism; 4a to 4d, the reading and the views `generation`, `credentials`, `patronymic` and `surname2`, landed without moving a parse, 4e, the render defaults and `ParsedName.order`, moved the default `str()`, and 4f, `[rules]` compiled into a pack's policy patch, moved no parse (decisions/vocabulary-model, decisions/R6 and decisions/R8 to R10, 2026-10-29) | Two commitments hold through all four. **The default loads every language and keeps working for names of any language, as today**; choosing a language narrows it (decision 5 in decisions/vocabulary-model). And **languages own vocabulary and defaults, the core owns structure**: a pack configures rules the core implements, and never ships code (mechanisms.md#LOCALE-PACKS-PURE-DATA). @@ -164,20 +164,20 @@ Each sub-phase is one PR, measured on its own base: the committed snapshot read ## Phase 3: packs that prefer or restrict -Status: **agreed** (proposed 2026-10-27, #753; Derek accepted every item on 2026-10-28, six and then the other six); **3a landed 2026-10-29**, the reading index, the compiler moved into the package and `default = false` on name readings (decisions/vocabulary-model's 2026-10-29 entry, which records what 3a decided and measured, and the index's lazy load, which Derek chose on 2026-10-29 over loading it with `Lexicon.default()`; "Where the provenance lives at runtime" below describes it). **3b landed 2026-10-29**, the choice: `Locale.languages`, its tag selection, turning on, prefer and the shape clash of `may_be_initial`, the derived deviation surface, `snapshot.py --languages`, and mechanisms.md#LOCALE-PACKS-PURE-DATA amended (decisions/vocabulary-model's 2026-10-29 phase 3b entry, which records what it decided, measured and left open). **3c landed 2026-10-29**, the eight pack-shaped exclusions (cs `ml`, hr, sk and sl `st` and `ml`, fr `père`) written as `generation` readings `default = false`, the default's compile unchanged (decisions/vocabulary-model's 2026-10-29 phase 3c entry, which records what each choice now moves). **3d landed 2026-10-30**, the first language pack, `locales.NE`, a choice of `ne` turning on `given` readings of `श्री`, `guru` and `गुरु` written `default = false` in `ne.toml`, its deviations derived rather than declared, and `local_corpora.py alike --languages` for the pair measure (decisions/vocabulary-model's 2026-10-30 phase 3d entry, which records what it moves and the one defect left open). **3f landed 2026-10-30**, the Czech, Slovak, Slovenian and Croatian packs, `locales.CS`, `SK`, `SL` and `HR`, one `Locale` per language as the API section below has it, each, like `NE`, a choice of its one tag and nothing else: 3c's generations, cs and pl `pan` and cs `roz` converted from exclusions, and the Czech and Slovak academic titles and post-nominals, all readings written `default = false`, the default's compile unchanged (decisions/vocabulary-model's 2026-10-30 phase 3f entry, which records what each pack moves and settles 3c's three open questions). **3h landed 2026-10-30**, the packs `ga` (`nic`, `mhic`, `mic`, `uí`), `gd` (`nic`) and `pt-BR` (`sb`), each a choice with no fragment and no patch, its deviations derived; `F` and `N` stay excluded, the pack included, measured wrong more often than right in the filings that write them (decisions/vocabulary-model's 2026-10-30 phase 3h entry). **3e landed 2026-10-30**, the packs `hi` (`smt`, `shrimati`, `shriyut`) and `bn` (`smt`, `shrimati`, `adhyapak`), each a choice alone; `shri` stays excluded and `sushri` unwritten, each opening given names more often than it is the title (decisions/vocabulary-model's 2026-10-30 phase 3e entry). **3g landed 2026-10-30**, the packs `es` (the courtesy, academic and professional titles that lead a name, `lic`, `licenciado`, `doctora`, `profesora`, `mtro`, `doña` among them, and the Spanish given-name list as a lexicon fragment) and `pt` (`doutor`, `doutora`, `professora`, `profa`), the titles written `default = false`; `don`, `dona`, `dom` and `maestro` stay excluded, `sr` keeps its generation under both, and 3g's connective was declined on measurement (decisions/vocabulary-model's 2026-10-30 phase 3g entry). The record is decisions/vocabulary-model's 2026-10-27 #753 entry; this section describes its items. Phase 3 is 3.0 work (re-planned 2026-10-28): 2.8 is the last 2.x feature release, and 3.0 ships phases 3 and 4 with the facade's removal. Packs stay opt-in, so a caller who chooses no language parses as the default does, and nothing in this phase moves the default's parses. One standing text is amended with it, in 3b's PR, which builds the selection: mechanisms.md#LOCALE-PACKS-PURE-DATA's contract that a pack is add-only and declares its deviations in its docstring (a pack adds, and selects among the default's readings by language, its deviations derived for a vocabulary pack). decisions/3-0-reevaluations' line that removing base vocabulary through a pack is part of 3.0's pack design is carried out rather than moved. +Status: **agreed** (proposed 2026-10-27, #753; Derek accepted every item on 2026-10-28, six and then the other six); **3a landed 2026-10-29**, the reading index, the compiler moved into the package and `default = false` on name readings (decisions/vocabulary-model's 2026-10-29 entry, which records what 3a decided and measured, and the index's lazy load, which Derek chose on 2026-10-29 over loading it with `Lexicon.default()`; "Where the provenance lives at runtime" below describes it). **3b landed 2026-10-29**, the choice: `Locale.languages`, its tag selection, turning on, prefer and the shape clash of `may_be_initial`, the derived deviation surface, `snapshot.py --languages`, and mechanisms.md#LOCALE-PACKS-PURE-DATA amended (decisions/vocabulary-model's 2026-10-29 phase 3b entry, which records what it decided, measured and left open). **3c landed 2026-10-29**, the eight pack-shaped exclusions (cs `ml`, hr, sk and sl `st` and `ml`, fr `père`) written as `generation` readings `default = false`, the default's compile unchanged (decisions/vocabulary-model's 2026-10-29 phase 3c entry, which records what each choice now moves). **3d landed 2026-10-30**, the first language pack, `locales.NE`, a choice of `ne` turning on `given` readings of `श्री`, `guru` and `गुरु` written `default = false` in `ne.toml`, its deviations derived rather than declared, and `local_corpora.py alike --languages` for the pair measure (decisions/vocabulary-model's 2026-10-30 phase 3d entry, which records what it moves and the one defect left open). **3f landed 2026-10-30**, the Czech, Slovak, Slovenian and Croatian packs, `locales.CS`, `SK`, `SL` and `HR`, one `Locale` per language as the API section below has it, each, like `NE`, a choice of its one tag and nothing else: 3c's generations, cs and pl `pan` and cs `roz` converted from exclusions, and the Czech and Slovak academic titles and post-nominals, all readings written `default = false`, the default's compile unchanged (decisions/vocabulary-model's 2026-10-30 phase 3f entry, which records what each pack moves and settles 3c's three open questions). **3h landed 2026-10-30**, the packs `ga` (`nic`, `mhic`, `mic`, `uí`), `gd` (`nic`) and `pt-BR` (`sb`), each a choice with no fragment and no patch, its deviations derived; `F` and `N` stay excluded, the pack included, measured wrong more often than right in the filings that write them (decisions/vocabulary-model's 2026-10-30 phase 3h entry). **3e landed 2026-10-30**, the packs `hi` (`smt`, `shrimati`, `shriyut`) and `bn` (`smt`, `shrimati`, `adhyapak`), each a choice alone; `shri` stays excluded and `sushri` unwritten, each opening given names more often than it is the title (decisions/vocabulary-model's 2026-10-30 phase 3e entry). **3g landed 2026-10-30**, the packs `es` (the courtesy, academic and professional titles that lead a name, `lic`, `licenciado`, `doctora`, `profesora`, `mtro`, `doña` among them, and the Spanish given-name list as a lexicon fragment) and `pt` (`doutor`, `doutora`, `professora`, `profa`), the titles written `default = false`; `don`, `dona`, `dom` and `maestro` stay excluded, `sr` keeps its generation under both, and 3g's connective was declined on measurement (decisions/vocabulary-model's 2026-10-30 phase 3g entry). **Strict was redefined on 2026-10-30** as file-strict, the chosen files and native-script vocabulary only, needing no 3i; the rest of its API is proposed and the definition measured below, nothing built (decisions/vocabulary-model's 2026-10-30 #753, file-strict entry). The record is decisions/vocabulary-model's 2026-10-27 #753 entry; this section describes its items. Phase 3 is 3.0 work (re-planned 2026-10-28): 2.8 is the last 2.x feature release, and 3.0 ships phases 3 and 4 with the facade's removal. Packs stay opt-in, so a caller who chooses no language parses as the default does, and nothing in this phase moves the default's parses. One standing text is amended with it, in 3b's PR, which builds the selection: mechanisms.md#LOCALE-PACKS-PURE-DATA's contract that a pack is add-only and declares its deviations in its docstring (a pack adds, and selects among the default's readings by language, its deviations derived for a vocabulary pack). decisions/3-0-reevaluations' line that removing base vocabulary through a pack is part of 3.0's pack design is carried out rather than moved. **Phase 4's shape questions came first.** The output model decides what a pack can express: a patronymic field, a second surname, credentials apart from generation suffixes (#326), and what `middle` means (#704). A pack reading of a kind with no field to land in is a reading nothing can surface, so those questions were to be settled before or with phase 3's mechanism, by the same shape-before-value rule, and the output model was agreed on 2026-10-28 (decisions/vocabulary-model's 2026-10-28 #754 entry): the seven roles stay, and `generation`, `credentials`, `patronymic` and `surname2` are views read from `Token.reading` or rules/O6's tag. Of the work order below, 3a (the reading index and the shared compiler) and 3c (the pack-shaped exclusions converted) are independent of the output model: the index carries whatever kinds the files hold, and 3c moves data between two encodings that compile alike. 3b's selection semantics are independent too, and its API's dependency, what `Locale` and `Policy` look like once the facade is gone, is answered (below). The packs depend on the output model in part, each on the phase 4 sub-phase that builds its view: `cs`'s generations and `pt-BR`'s agnomens on 4b's `generation` (3f and 3h), `es` and `pt` on 4d's `surname2` and 4g's compound given name (3g); `ne`'s readings do not. The semantics were fixed in 2026-10-15/16, and phase 3 keeps them. A pack is a choice of languages, in two strengths (decision 6): - **prefer** settles a clash where the chosen language has a vocabulary reading: the word keeps that language's readings and loses other languages' vocabulary readings. Under Portuguese, `do` is a particle only. -- **strict** also drops other languages' name readings. It is for single-origin data. +- **strict** reads only the chosen languages' files and native-script vocabulary, so every other language's readings leave, names and vocabulary alike (file-strict, redefined 2026-10-30; "Strict: the chosen files and nothing else" below). It is for single-origin data. The asymmetry is deliberate (decision 15): **vocabulary readings settle a clash; name readings only keep the gate on.** `do` shows why. Its Vietnamese reading is a name only (Đỗ). If name readings settled clashes, preferring Vietnamese would drop the credential reading, and `John Smith DO`, a suffix today, would read `DO` as the family name. Under the rule, Vietnamese holds no vocabulary reading of `do`, so preferring it settles nothing: the credential and particle readings stay, gated as in the default. Consequences that follow from the definitions: -- **Order does not matter.** A choice is a set of languages, so combining packs is commutative. Decided 2026-10-28 (below): strictness belongs to the whole choice rather than to one pack, reshaping decision 6's "strict variant", and strict waits until name readings say whose names they are. +- **Order does not matter.** A choice is a set of languages, so combining packs is commutative. Decided 2026-10-28 (below): strictness belongs to the whole choice rather than to one pack, reshaping decision 6's "strict variant"; strict was to wait until name readings said whose names they are, which file-strict no longer needs (2026-10-30, below). - **Two chosen languages that each hold a vocabulary reading both keep it**, and the word stays gated, as in the default. - **Non-Latin vocabulary stays on under any choice**, because it selects itself by script (`씨`, `さん`, Cyrillic titles). Almost every clash between languages is within one script: `do`, `e`, `van` are all Latin. - **Whether a word needs a pack depends on where its clash is.** `do` clashes between languages, so a pack settles it. `fils` clashes within French, a generation word and a French surname, so no pack settles it and the phase 2 gate does the work, reversing the premise of #684. @@ -194,7 +194,7 @@ Everything a pack does is one operation: **the chosen languages select which rea | turn on | a chosen language's `default = false` readings join the pool | cs `ml` (*mladší*, once 3c makes it a reading): `Jan Novák ml.` reads family `ml.` today; under `cs`, suffix `ml.` | | prefer | on a word where a chosen language holds a vocabulary reading, other languages' vocabulary readings leave; `mul.toml`'s readings never do, being every language's | `Haddad, Abd` reads suffix `Abd` and no given name today; under `ar` the bound given reading stays and the en credential leaves; cs `st`: `Jiří Dienstbier st.` reads title `st.` today (rules/H5), and under `cs` its generation reading joins and the Saint readings leave, suffix `st.`, reported as suffix-or-name since en's `given` reading of `st` still gates it | | prefer, against a shape | a clash a property records with a SHAPE reading is settled for the chosen language's own reading, by decision 20: `may_be_initial` does not hold on it | `ANA SILVA E SOUZA` reads middle `SILVA E` with a conjunction-or-initial report today; under `pt` the conjunction holds, family `SILVA E SOUZA` | -| strict, deferred (below) | other languages' name readings leave, and with them the markers they derive | an `nl` `in` turned on, with Korean In declared as a `given` reading, is a particle that may be given under `nl` and never-given under strict `nl` | +| strict (file-strict: the definition decided 2026-10-30, its API proposed, below) | every reading outside the chosen files and native-script vocabulary leaves, and with it the markers it derives | `Do Van Minh` under `FAMILY_FIRST` reads given `Van Minh` today, Dutch `van` chaining the Vietnamese middle name Văn; under strict `vi` (simulated: `vi` cannot be chosen until vi.toml holds a reading), given `Van`, middle `Minh`; an `nl` `in` turned on, with Korean In declared as a `given` reading in ko.toml, is a particle that may be given under `nl` and never-given under strict `nl` | The third row answers decisions/3-0-reevaluations' "a pack cannot remove `e`": `may_be_initial` records a clash between a conjunction and the initial a letter spells, and the initial is a shape reading, so a pack that prefers Portuguese settles it, as decision 7 says choosing a language does and decision 20 permits. It reaches every reading the property stands on, pt and it `e` and ca and pl `i`, the Italian one unjudged by the inventory; and it forecloses the other direction of the 2026-08-18 bullet there, an `nl` pack marking `y` as an initial, which waits with the `fru` question below. Two parts of the mechanism are new beside decision 6: the exemption of `mul.toml` from prefer (under `ca`, `i` keeps its mul generation), and strictness as a property of the whole choice. No property is toggled per pack, and no reading is written twice. @@ -213,9 +213,90 @@ Properties ride on their readings, so a choice that keeps a reading keeps its pr The 2 broken are pairs whose romanization opens with a listed title, which reads the `श्री` as a salutation (#870 counted `Mr` written for it twice). So the gate route delivers the `श्री` and `guru` fix without amending decision 15, and the 19 `dc` pairs it leaves are the price of keeping it. `bk`, `kc`, `gc` and `bc` need nothing: they read as the family in every shape tried (`Ram Bahadur BK`, `Ram Bahadur Kc`, `BK, Ram Bahadur`, `KC Sita` under `FAMILY_FIRST`), so a reading would move nothing. -**Strict cannot be built on today's files.** A name reading sits in the file whose source counted it, not the file of the language whose names carry it: 129 of the 227 name readings are in en.toml (2026-10-28, `c8357e74`), `do`'s Vietnamese Đỗ readings among them, and vi.toml holds no `do`. A strict `vi` choice built on the files would drop exactly those readings, and with them the gate: simulated, `Minh Do` reads suffix `Do` and `Do Van Minh` family `Do Van Minh`, the harm decision 15 is there to prevent. Nor does a census count belong to one language. So strict waits (decided 2026-10-28), out of the first packs, until name readings say whose names they are, and the candidates that need it wait with it. The `nl` prepositions show they would need more than strict: turned on as possibly-given particles, `in` and `to` already read `Park In Hwan` family `In Hwan` and `Nguyen To Nga` family `To Nga`, the marker acting at the head only, and check 11 lets their author write `may_be_given = false` instead of declaring the Korean and Vietnamese readings. +**Name-strict could not be built on the files, and is replaced.** This section first defined strict as *name-strict*: other languages' titles, credentials and particles stay, and only their name readings leave. A name reading sits in the file whose source counted it, though, not the file of the language whose names carry it: 129 of the 227 name readings were in en.toml (2026-10-28, `c8357e74`), `do`'s Vietnamese Đỗ readings among them. Built on the files, name-strict `vi` drops those readings and keeps the `en` credential `do`, so `Minh Do` reads suffix `Do`, the harm decision 15 is there to prevent. Building it needed 3i, an attribute on each name reading naming its bearers' languages, filled by a person word by word, since a box count says where a word was typed and not whose name it is (decisions/vocabulary-model's 2026-10-27 #753 entry, item 4, and the 2026-10-30 file-strict entry's 3i bullet hold the counts). On 2026-10-30 strict was redefined so that it needs none of this (the next section). -**Open: nothing in the work order says whose names a name reading's are.** Strict waits on it, and none of 3a–3h does it. Moving readings between files would not: a census count belongs to no one language, and a word is borne in several, `do` as a Vietnamese and a Korean name, `kim` as a Korean surname and a Scandinavian given name. What it points at is an attribute on each name reading naming the languages of its bearers (`do`'s family reading `vi` and `ko`; the key's name still to choose), compiling to nothing until strict reads it. The counts to fill it exist, but none fills it mechanically. Of the 227 name readings (2026-10-28, `c8357e74`), 188 count a source under a `us_` key and 21 none, and what per-country evidence they carry is prose in a `note`, as `do`'s nam_dict bands are (Korea 8, Vietnam 6, the Netherlands 1), the notes of 40 naming nam_dict. `name_usage.py --boxes` gives `orcid-researchers`' and `patentsview-inventors`' counts by box and country, and a box count says where a word was typed, not whose name it is: `do`'s family box in ORCID is led by Brazil, 7,227 holders against Vietnam's 876, and none of Brazil's is `do` alone (2,629 open with it, 4,598 hold it inside), the particle of a Brazilian surname, where Vietnam has 618 boxes of `do` alone and Korea 151. A share of each country's holders ranks them the other way (Vietnam 1.90%, Brazil 0.81%; PatentsView 1.78% and 0.64%), and `kim` reads cleanly either way, its family box 17.67% of ORCID's Korean holders and 13.30% of PatentsView's, its given box 0.93% of PatentsView's Danish inventors (2026-10-28, master `c8357e74`). So the fill is a person reading each word with a name reading (165 on 2026-10-28, `c8357e74`), by share and by whole box, not a script. Two further consequences: the attribute names languages, filled from countries by that reader, since a country-to-language table read at run time is the macrolanguage table the tag matching below refuses; and it is the first provenance anything computes a reading from, where counts today are evidence nothing reads. Decided with it: a sub-phase of its own, 3i, after 3a, independent of the output model and moving no default parse, with strict and the candidates waiting on it after it. Recompute: count the `given` and `family` readings per file, per count key and by whether their `note` names nam_dict, with `tomllib` over `nameparser/vocab/*.toml`; `name_usage.py do kim --no-wikidata --boxes`; the whole-box split by reading the cached ORCID extract's `family` and `country` keys. +### Strict: the chosen files and nothing else + +Status: **the definition decided 2026-10-30; the API beyond 2026-10-28's `strict=True` argument proposed, and the definition measured, on 2026-10-31** (decisions/vocabulary-model's 2026-10-30 #753, file-strict entry, which holds every measurement below with its recipe). Nothing here is built. + +**File-strict.** A strict choice compiles only the chosen languages' files, each tag selecting its own and its ancestors' as any choice does, plus native-script vocabulary, and nothing else. Native-script vocabulary stays on because it selects itself by script (the CJK honorifics, the Cyrillic and Devanagari titles; the Korean surname list stays too, but because it is outside the per-word model and so outside any choice's compile, not by this filter); a native-script reading is one whose word has a letter and no Latin letter, read from the word as the script always is. `mul.toml` and `und.toml` are not in a strict choice. A word with no reading in what is compiled is read by position and shape alone: under strict `vi`, `do` is in no chosen file (en.toml and pt.toml hold it) and is a name word wherever it stands, while the shape rules still act, so `Dr. Minh Do` keeps title `Dr.` by its period (rules/H2) and in `Minh Do, M.D., Ph.D.` the part past the second comma stays the suffix, `Ph.D.`, holding no name word (rules/C2), but the first credential becomes the given name (given `M.D.`, family `Minh Do`), as `Minh Do, PhD` reads given `PhD`: stacked credentials do not survive. For a credential, the harm that made name-strict wait does not arise today: measured 2026-10-31, no strict choice of one tag ungates a credential acronym, because each credential's gating name reading sits in the credential's own file, so the two leave together (`Minh Do` stays family `Do`). That holds only while the remedy below adds name readings to more files and never moves one: moving `do`'s Đỗ readings from en.toml to vi.toml would ungate `do`'s credential under strict `en`. For the other kinds it can: where a chosen file holds the vocabulary reading and only a dropped file the gating name reading, strict keeps the reading ungated ("The derived markers under strict" below; strict `pt` reads `Do Van Minh` family `Do Van Minh`, the harm item 4 of the 2026-10-27 entry named). Strict still needs no 3i, because under file-strict that case is remedied by data, a name reading written in the chosen file whose names carry the word, not by an attribute naming bearers' languages. + +Two consequences were accepted with the definition: + +- **`mul.toml` stays as it is, and under file-strict it is one more language's file.** Its entries each name several languages (`doctor` four, `docent` six, `professor` English, German, Dutch, Portuguese and the Scandinavian languages), or "the European languages" (`dr`, `prof`), or carry no note (the numerals `2` and `i` to `v`, `&`), and `ing`, the Czech and Slovak engineering degree, sits there too. Open: entries may later move between en.toml and mul.toml, or into the files of the languages that write them, so that strict `en` can shed problem words; which words those are is not yet known. +- **Strict `vi` drops English credentials, accepted in principle.** Vietnamese writes its degrees in Vietnamese and before the name (`TS.`, `ThS.`, `GS.`, `PGS.`, `BS.`, stacked as `PGS.TS.`) and has no generational suffix, so `PhD` and `MD` come with English-context writing, where the caller chooses `en` too. What this costs is measured below, not assumed. + +**What it does, measured** (2026-10-31 on master `b577e9a8`, Python 3.11, the cached local corpora only). The simulation compiles a strict choice through the shipped compiler, `_reading_index.choose` and `compile_readings` over the index rows of the chosen tags and the native-script rows, and applies it to the default lexicon as a choice's delta is applied; a pack's policy patch and fragment apply as `parser_for` applies them. Each ORCID country's capped 20,000 records (12,962 for SK, 17,541 for NP) are scored box by box with `local_agreement.read`, leaving out the records whose given box opens with a title word (`Dr`, `Prof`, `Mr` and their kin; 1,717 for IN, 437 for NP, under 70 elsewhere), since ORCID has no title box and a holder who typed `Dr` into the given box scores a fix whenever `dr` leaves the vocabulary. "choice" is the shipped pack where one carries the language, else a bare choice of the tag; "strict + en" adds en.toml to the strict choice. Fixed and broken are against the default. + +| Country | default | choice | strict | strict + en | +|---|---|---|---|---| +| VN | 65.48% | none: no file holds `vi` | 64.28% (+349 −589) | 64.34% (+306 −534) | +| KR | 92.42% | 92.42% (+0 −0) | 92.54% (+41 −17) | 92.51% (+33 −16) | +| CN | 84.54% | 84.54% (+0 −0) | 84.65% (+23 −3) | 84.61% (+16 −3) | +| JP | 95.59% | 95.59% (+0 −0) | 95.64% (+30 −20) | 95.54% (+11 −20) | +| CZ | 95.12% | 96.54% (+635 −352) | 96.42% (+626 −367) | 96.43% (+625 −364) | +| SK | 95.01% | 96.58% (+347 −144) | 96.53% (+347 −150) | 96.59% (+348 −143) | +| PL | 96.88% | 96.88% (+0 −0) | 96.74% (+4 −32) | 96.78% (+4 −25) | +| DE | 95.01% | 95.00% (+0 −1) | 94.42% (+8 −125) | 94.44% (+7 −120) | +| ES | 48.04% | 93.47% (+9,295 −215) | 93.21% (+9,286 −258) | 93.22% (+9,285 −255) | +| MX | 31.33% | 93.01% (+12,534 −204) | 92.93% (+12,524 −210) | 92.93% (+12,523 −209) | +| BR | 47.30% | 47.33% (+5 −0) | 46.90% (+18 −99) | 47.22% (+9 −26) | +| PT | 69.99% | 69.99% (+0 −0) | 69.87% (+1 −25) | 69.90% (+1 −19) | +| NP | 90.85% | 90.86% (+1 −0) | 90.74% (+18 −37) | 90.85% (+3 −3) | +| IN | 79.82% | 79.82% (+0 −0) | 79.85% (+23 −17) | 79.80% (+6 −10) | + +How to read it: + +- **Strict moves little where a choice already exists, and what it breaks is immigrant names.** The large ES, MX, CZ and SK gains are the packs' `two_family_names` (rules/O6), present in the choice and strict alike. Against the choice, over the eight countries a shipped pack covers, strict fixes 1 to 23 records per country and breaks 11 to 99 (CZ +9 −33, SK +5 −11, ES +11 −63, MX +6 −22, BR +13 −99, PT +1 −25, NP +17 −37, IN +23 −17), and against a bare choice DE +8 −124 and PL +4 −32, and most of what it breaks is a word of another language borne in that country's names, counted as the breaks whose name holds the word: DE's `de` 45 and `van` 39 of 124 (`Anton van Essen` → middle `van`; DE's choice is a bare `de`), BR's `junior` 62 and `jr` 12 of 99 (en.toml holds both; pt.toml holds only `júnior`), ES's Catalan `i` 39 of 63 (ca.toml and mul.toml). ORCID records a country of affiliation, not a language, which is the single-origin premise strict asks the caller to vouch for. +- **VN's ORCID loss is a box convention, not a misreading.** Vietnamese holders often type the middle names into the family box, and the default's French `le` and Dutch `van`, chaining the next word into the family, happen to land on that box: `Manh Le Ngo Duc`, family box `Le Ngo Duc`, is the default's family and strict's middle `Le Ngo`. The control is the spelling no vocabulary reads: holders who write `Văn` put it in the family box 20 times of 73, against 144 of 802 for `van`. PSC's three boxes separate the middle name, and there strict `vi` + `en` fixes 93 Vietnamese filers and breaks 36 (91 of the fixes hold `van`). +- **KR, CN and JP gain a little, from Latin words that are their names**: `bin` (Korean and Chinese Bin), `do`, `gen` (Japanese Gen, en.toml's military title), `de` and `du` (Chinese De and Du). + +**The `van` chaining fix.** ORCID's Vietnamese records hold `van` before the last word 802 times (the name split at spaces only; splitting at commas too adds `Master Thuy Van, TT`, giving 803, 404 and 3). The default reads it into the family in 403 of them; the holders put it there in 144. Strict reads it into the family in 2, and the 802 agree with their boxes 583 times against the default's 479 (the 658 that box it with the given names: 354 to 583; the 144 that box it with the family: 125 to 0). PSC holds 156 such Vietnamese filers, 5 boxing `van` as the surname; the default reads it into the family in 97 and agrees 62 times, strict + en reads it there in none and agrees 147 times. + +**The cost of dropping other languages' credentials and salutations.** In English-context data the cost is large and almost all of it is en.toml's: + +- PSC's Vietnamese filers: strict `vi` moves 865 of 1,499 off their boxes, the parse's given name being `Mr` in 466 of them, `Ms` in 213, `Mrs` in 98 and `Miss` in 73 (`Mr Anh Cong Tran` → given `Mr`); strict + en moves 36. South Korean filers: 390 of 556, and 13 with en. +- ClinicalTrials.gov officials, distinct names, whose name before the first comma holds one of 18 common Vietnamese surnames: 1,697, a suffix read in 1,247 by default. Strict `vi` changes 1,112 of those suffixes and reads one of the default's suffix words as the given name in 1,079 (`Ngoc P Ly, MD` → given `MD`, family `Ngoc P Ly`); strict + en changes 9 and reads none as the given name. With 18 Korean surnames: 5,686, 4,224 with a suffix; strict `ko` changes 3,678 and reads 3,569 as the given name (`Minyoung Kim, MD, PhD` → given `MD`), strict + en 5 and 0. The surname slice is rough (`Ho`, `Do`, `Lim` are borne elsewhere too) and the corpus holds no ground truth, so these are moves, not errors; each move counted is a credential the default read as one. + +**What each language file is missing to stand alone.** The words a country's own records write that its strict choice loses, from ORCID (O) and PSC (P), with the file holding them: + +| Strict | Loses, with the file holding it | +|---|---| +| `vi` | `le` fr, `van` nl, `do` en and pt, `the` en, `tho` de and nl (the intended effect); `Mr`, `Ms`, `Mrs`, `Miss` en (P); `dr` mul | +| `ko` | `md` en; `jo` fil and `se`, `uk`, `ma` en (Korean names: a gain); the salutations en (P); `dr` mul | +| `zh` | `ma`, `meng`, `si`, `chi`, `che` en, `du` fr (Chinese names: a gain); the salutations en (P); `dr` mul | +| `ja` | `md`, `gen` en (`gen` a gain); the salutations en (P); `dr` mul | +| `cs`, `sk` | `dr`, `prof`, `ing` mul; `phd`, `md`, `msc`, `mph`, `jr` en | +| `pl` | `dr` mul; `phd`, `md`, `maj` en | +| `de` | `dr`, `prof`, `v` mul; `md` en; `de`, `van`, `da`, `le`, `la` (immigrant particles) | +| `es` | `i` ca, de, mul and pl; `e` it and pt; `dr`, `v` mul; `phd` en | +| `pt`, `pt-BR` | `junior`, `jr` en; `del`, `dal`, `di`, `della` es and it; `v`, `i` mul | +| `ne`, `hi` | `dr`, `prof`, `v` mul; `phd`, `md` en; `kc`, `das` (Nepali and Indian surnames: a gain) | + +Nine of the ten rows lose a bare `dr` and three `prof` (`Dr.` stays a title by its period; the pt rows' records write neither often enough to rank), and every English-context population the salutations, so a strict choice made for real data is in practice strict of the language and `en`, or needs those words in its own file; this is the mul.toml Open above. + +**Files that hold no readings.** hu.toml, kk.toml, kok.toml, vi.toml and zu.toml hold only exclusions (vi.toml's are `ông` and `bà`), so `selected` refuses each tag, and a strict `vi` needs readings in vi.toml before it can be asked for (the simulation bypasses the refusal). 26 further files hold no Latin-script vocabulary reading (am, as, be, bg, el, gu, ja, km, kn, ko, lo, mk, ml, mr, my, ne, or, sd, shn, si, ta, te, th, ti, uk, zh), so strict of one of them is the native-script pool plus that file's name readings: strict `ko` is the native-script vocabulary and nothing Latin. + +**The derived markers under strict.** A marker derived from a name reading leaves with the reading it gates where both sit in a dropped file, and stays where both sit in a chosen one. It does not hold in general: where the vocabulary reading is in a chosen file and the name reading that gates it only in a dropped one, strict keeps the reading and drops its gate. The vocabulary holds 16 such words, 17 readings (`st` both as a particle and as a generation), 27 word-and-tag pairs over 13 tags (2026-10-31, each strict choice of one tag compared with the plain choice of the same tag): `cheikh` and `sultan` under `ar` (their `given` readings in fr.toml and tr.toml), `la`, `marquis`, `marquise` and `st` under `fr`, `mac` under `ga` and `gd`, `nic` under `gd` (its name readings in ga.toml), `del`, `la`, `san` and `santa` under `it`, `bin` and `sultan` under `ms`, `den` and `tho` under `nl`, `da`, `do` and `santa` under `pt` and `pt-BR`, and the generation `st` the choice turns on under `cs`, `hr`, `sk` and `sl` (its `given` reading in en.toml). Each becomes a never-given particle, an unconditional title or a generation that reports nothing (under the plain `cs` choice `Novák, St` reads given `St` and reports suffix-or-name; under strict `cs`, suffix `St` and no report; `Nic Smith` under `gd`, given `Nic` reported particle-or-given, and under strict `gd` family `Nic Smith`): under strict `pt`, `Da Costa` reads family `Da Costa` where the default and the `pt` choice read given `Da`; under strict `ar`, `Sultan Ahmed Khan` reads title `Sultan`, where the default reads given `Sultan`. Some of these are what strict is for (a Portuguese `Do` is the particle), and some are the file-placement fact that sank name-strict, at small scale: Sultan is an Arabic given name whose reading sits in tr.toml. Under file-strict the remedy is data, a name reading written in each file whose names carry it, as a word already sits in the file of every language whose names it occurs in; the list is derivable from the index, as a choice's changed words are, so a test can hold it to a reviewed set. + +**The API (the `strict=True` argument decided 2026-10-28; the rest proposed, not decided).** + +- **How a caller asks: an argument of `parser_for`, `strict=True`, over the whole choice, decided 2026-10-28 and unchanged by file-strict.** What is proposed is the rest. A `Locale.strict` field, the alternative, is not recommended (proposed, like the rest): the choice is the union of the packs' languages, and a strict pack stacked with a plain one has no meaning, a plain choice keeping every file. A strict call whose packs choose no language (`parser_for(locales.ZH, strict=True)`) would keep only native-script vocabulary; it should raise, naming the packs and asking for a language, rather than strip a caller's Latin vocabulary in silence. A strict choice is one call's, as a choice is (decisions/vocabulary-model's chained-choices entry): as a base it records nothing, so a later plain choice applies its delta to an already stripped lexicon. +- **Shipped `Locale` constants: keep them for packs that carry something, and add a constructor for the rest (recommended).** A constant per language file would be 78 constants, one per tag a file holds a reading of beside `mul` and `und` (2026-10-31), most of them a tag and nothing else. Today a caller writes `Locale(code="de", lexicon=Lexicon.empty(), languages=frozenset({"de"}))`, which is verbose for a choice and nothing more. A `locales.choose("vi", "en")` returning that Locale (an addition), validated by `selected`, covers every language with a file; the constants stay for the language-choice packs with a policy patch, a fragment, or a measured docstring (NE, CS, SK, SL, HR, GA, GD, PT_BR, HI, BN, ES, PT), and ZH, JA, RU and TR_AZ, which choose no language and carry a fragment or a policy patch, stay as they are. Folding a bare tag into `locales.get` was weighed and is not recommended: `get` returns a shipped pack, and a choice it built would be one no test has measured. +- **Turned-on readings:** a strict choice turns on the chosen files' `default = false` readings, as a choice does; another file's off reading stays off, a native-script one included. +- **A pack's `[rules]` patch and its fragment** are untouched: the patch is policy, and the fragment is unioned after the strict compile, so `locales.ES`'s given-name list and `locales.ZH`'s surnames stay (measured: the 1,000 entries of ES's list under strict `es`). A fragment is the pack's own data, which strict keeps by definition. +- **`Lexicon.given_names` and `Lexicon.surnames`** are not compiled from the files, so no choice touches them, strict included: the Korean surname list, outside the per-word model, stays under every strict choice (measured). +- **Stacking packs:** the union of their languages, then strict over the union; commutative, as a choice is. +- **A caller's own `Lexicon.add`:** strict applies as a delta, as 3b's choice does, so it removes from the base every entry it drops, a caller's own listing of a dropped word included, and keeps words no file holds: under strict `vi`, a base that added `dr` and `zzz` keeps `zzz` and loses `dr` (measured). A caller who wants `dr` adds it after, in a fragment or to the finished parser's lexicon. Keeping a caller's additions by subtracting the default was weighed and is not recommended: a caller's listing of a word the default holds cannot be told from the default's. + +**Candidate next steps, not in this design's PR:** + +- **Vietnamese degree prefixes as vi.toml readings** (`TS.`, `ThS.`, `GS.`, `PGS.`, `BS.`, and the stacked `PGS.TS.`), which would also give vi.toml its first readings and make a `vi` choice possible. Each needs its collisions and bearer counts checked first, per nameparser/vocab/AGENTS.md: `Ts`, `Gs` and `Bs` are short enough to be initials or other languages' words. +- **`ông` and `bà` stay excluded.** Ông is a Vietnamese surname, a clash within Vietnamese that strict does not settle. +- **The candidates that waited for strict become buildable as data.** 3h's `nl` prepositions: `in` and `to` turned on `default = false` in nl.toml, with Korean In and Vietnamese To declared as `given` readings in ko.toml and vi.toml, are possibly-given particles under the plain `nl` choice and never-given under strict `nl`. The plain choice still carries the cost item 4 of the 2026-10-27 entry recorded: the marker acts at the head only, so `Park In Hwan` reads family `In Hwan` and `Nguyen To Nga` family `To Nga` (re-run 2026-10-31 with `particles` and `particles_ambiguous` holding `in` and `to`), and a follow-up must not ship them under the plain `nl` choice without settling it. 3e's `sushri`: a hi.toml salutation `default = false` with the Odia given name declared in or.toml is a title only inside a title phrase under the plain `hi` choice (rules/H9) and a title under strict `hi`. The build has not been asked whether it accepts either; that is each candidate's own PR. +- **The 16 strict-ungated words** each read, and a name reading added to the chosen file where that language's names carry the word. + +3i is left with no consumer: nothing else in the work order reads the bearers' languages it would record. ### Where the provenance lives at runtime @@ -229,7 +310,7 @@ A choice is applied as a delta against the default's compile, so it composes wit ### The API and tags -This rested on `Locale` and `Policy` keeping their 2.x shape after the facade's removal, and the 2026-10-28 3.0 freeze answers it: both keep it, with additions. A choice is carried by `Locale`, which gains a `languages` field, a set of BCP 47 tags. `parser_for` unions the `languages` of the packs it is given, so the choice is order-independent while their policy patches still fold in order. Strict, once it can be built, is a property of the whole choice, not of one pack, and an argument: `parser_for(locales.NE, locales.HI, strict=True)`. The shipped language packs are `Locale` constants like the four today (`locales.NE`, `locales.get("ne")`), each with an empty lexicon fragment (what a fragment holds once 3.0 configures vocabulary by reading is decisions/vocabulary-model's the 2026-10-28 #754 entry on the `Lexicon` surface); a code keeps `Locale`'s `[a-z0-9_]+`, so `pt-BR` is `locales.PT_BR`. +This rested on `Locale` and `Policy` keeping their 2.x shape after the facade's removal, and the 2026-10-28 3.0 freeze answers it: both keep it, with additions. A choice is carried by `Locale`, which gains a `languages` field, a set of BCP 47 tags. `parser_for` unions the `languages` of the packs it is given, so the choice is order-independent while their policy patches still fold in order. Strict is a property of the whole choice, not of one pack, and an argument: `parser_for(locales.NE, locales.HI, strict=True)` (file-strict, the rest of its API proposed in "Strict: the chosen files and nothing else" above). The shipped language packs are `Locale` constants like the four today (`locales.NE`, `locales.get("ne")`), each with an empty lexicon fragment (what a fragment holds once 3.0 configures vocabulary by reading is decisions/vocabulary-model's the 2026-10-28 #754 entry on the `Lexicon` surface); a code keeps `Locale`'s `[a-z0-9_]+`, so `pt-BR` is `locales.PT_BR`. A chosen tag selects its own file and its ancestors', never its descendants': `pt-BR` selects `pt-BR.toml` and `pt.toml`, and `pt` selects only `pt.toml`, so a caller in Portugal does not turn on the Brazilian agnomens (#843 left the single-letter `F` and `N` to a `pt-BR` pack). No macrolanguage table is read: Malay and Indonesian are separate tags, though BCP 47 counts Indonesian within the Malay macrolanguage, and a caller with names in both chooses both. The Czech, Slovak, Slovenian and Croatian files each hold the words they share, so each language is a pack of its own. Script subtags need no matching rule yet, since no file carries one. @@ -252,7 +333,7 @@ The TOML names `prefer` and `strict`, reserved for phase 3 since phase 1, stay r ### The order of work -Shape first, then the packs, each one PR measured on its own base, as phase 2's were, interleaved with phase 4's sub-phases as its work order says: a pack needing a view waits for the phase 4 sub-phase that builds it, and 4f, which compiles `[rules]` into a `Locale`'s policy patch, lands with or after 3b, both deciding what a shipped `Locale` holds. The first three move no default parse, and no pack exists until 3d. Strict is not in this order (above); 3i, the bearers' languages it would read, is. +Shape first, then the packs, each one PR measured on its own base, as phase 2's were, interleaved with phase 4's sub-phases as its work order says: a pack needing a view waits for the phase 4 sub-phase that builds it, and 4f, which compiles `[rules]` into a `Locale`'s policy patch, lands with or after 3b, both deciding what a shipped `Locale` holds. The first three move no default parse, and no pack exists until 3d. Strict was not in this order while it waited on 3i, the bearers' languages name-strict would read; file-strict needs no 3i (2026-10-30), and is 3j, below. | Sub-phase | What | Needs | |---|---|---| @@ -264,7 +345,8 @@ Shape first, then the packs, each one PR measured on its own base, as phase 2's | 3f | `cs`, `sk`, `sl`, `hr`: the generations of 3c, `pan`, `roz` and the academic abbreviations #848 left | 3b, 3c, 4b | | 3g | `es` and `pt`: the occupational titles and `Don`/`Doña`/`Dona`, with a connective that counts only inside a title run, its own rule; `pt` also settles the one-case `e`. Landed with the titles that take no complement, `Doña` alone of the courtesy titles, the one-case `e` settled by the `pt` choice (3b's rule), and no connective, declined on measurement (the 2026-10-30 phase 3g entry) | 3b, 4d, 4g | | 3h | `ga` and `gd` (`nic`, `mhic`, `mic`, `uí`), `pt-BR` (`F`, `N`, `sb`); `nl`'s prepositions wait with strict. Landed with `mhic` and `mic` in `ga` only and `pt-BR` as `sb` alone (the 2026-10-30 phase 3h entry) | 3b, 4b | -| 3i | the languages of each name reading's bearers, an attribute compiling to nothing, filled word by word from per-country shares and whole-box counts (above); strict and its candidates after it | 3a (neutral) | +| 3i | the languages of each name reading's bearers, an attribute compiling to nothing, filled word by word from per-country shares and whole-box counts, for name-strict to read; set aside 2026-10-30 with name-strict, file-strict needing none, and left with no consumer | — | +| 3j | file-strict: `parser_for(..., strict=True)` (the argument decided 2026-10-28, the definition 2026-10-30) compiling only the chosen files and native-script vocabulary; proposed 2026-10-31 and not decided: refusing a strict call that chooses no language, `locales.choose`, and a derived list of the words strict ungates, each strict choice compared with the plain choice of the same tags, never with the default's markers (which miss a reading the choice turns on), held by a test; and `_reading_index`'s `UNCHOOSABLE` comment and `selected`'s refusal text, which say no choice removes a reading of `mul` or `und`, which strict does. The candidates that waited on strict follow it, each in its own PR | 3b; a `vi` choice also needs readings in vi.toml | **Declined 2026-10-30 (the phase 3g entry in decisions/vocabulary-model):** 3g's connective, which needed a design of its own, not decided with the work order, because the pair that separates it is a count: `Profesora de Historia Ana López` should read title `Profesora de Historia`, while `Doctora de la Vega`, the same title, preposition and following word, has no name behind it and must keep family `de la Vega`. With `profesora` and `ministro` added as titles and no rule, the first reads family `de Historia Ana López` (the rules/P1 fold), and `Ministro da Fazenda Carlos Souza` reads given `da`. The rule must reach only readings a pack turns on, so the default's parses stay where they are. No rule was written: over 8,526,439 names of the ten local and ten committed corpora measured, no leading title and connective is a title phrase with a name behind it, the 16 that open that way being particle-led surnames and given names behind a title, the costs, a given name that is no title (`Cardeal`), one office with no name behind it and a fragment; Spanish and Portuguese write an office lowercase in apposition, and the office titles that take a complement stay out of the packs. @@ -394,7 +476,7 @@ Each sub-phase is one PR, measured on its own base as phase 2's and phase 3's ar | 4g | the compound given name, a rule kind (#704): built 2026-10-30, `Policy.two_given_names` (rules/O7) | 4f | | 4h | the shape inventory per order (#469), starting with the title between a family and a given name under `FAMILY_FIRST`. Landed as an inventory, no reading moved: the arrangements each order is written in, listed with what each reads (decisions/vocabulary-model's 2026-10-30 #754, sub-phase 4h entry); the title between, and bare after, under a family-first order stays a name word, #938 declined on a survey of real names, which write the title in front (decisions/H5's 2026-10-30 #938 entry) | — | -**Interleaved with phase 3** (its sub-phases 3a to 3i, above): 4a can start beside 3a, its `languages` waiting for 3a's index; 4d needs nothing of phase 3, and 4b and 4c need 3a's index only for a listed word's kind, a suffix word's and a particle's; they unblock its packs, `cs`'s generations (3f) and `pt-BR`'s agnomens (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; 4f lands with or after 3b, since both decide what a shipped `Locale` holds; an `is` pack, which no phase 3 sub-phase lists, needs 3b, 4c and 4f. 3i, the bearers' languages strict would read, needs 3a alone, and 4e and 4h stand apart from both. So the order is 3a and 4a, then 4b to 4d with 3b and 3c (4b after 3a), then 4f, then the packs with 4g, with 4e at any point and 3i at any point after 3a. +**Interleaved with phase 3** (its sub-phases 3a to 3j, above): 4a can start beside 3a, its `languages` waiting for 3a's index; 4d needs nothing of phase 3, and 4b and 4c need 3a's index only for a listed word's kind, a suffix word's and a particle's; they unblock its packs, `cs`'s generations (3f) and `pt-BR`'s agnomens (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; 4f lands with or after 3b, since both decide what a shipped `Locale` holds; an `is` pack, which no phase 3 sub-phase lists, needs 3b, 4c and 4f. 3j, file-strict, needs 3b alone (3i, which name-strict would have read, is set aside), and 4e and 4h stand apart from both. So the order is 3a and 4a, then 4b to 4d with 3b and 3c (4b after 3a), then 4f, then the packs with 4g, with 4e at any point and 3j at any point after 3b. **Left open.** With no sub-phase yet: phase 1's four reserved names said to be phase 4's (`separators` with phase 4's glued words, and `bare_only`, `leading_only` and `trailing_only`, which this RFC's phase 2 section found not needed), whose build messages still name this phase. The first unit's view of a family read as two, and its name, with each unit's particles and base, which CLDR's `surname-prefix` and `surname-core` and a sort key per surname would read: 4d built `surname2` alone (decisions/R10's Open).