Skip to content

WO20: tiny-student bridge arm verdict + static-int8 sweep verdicts (issue #441) - #442

Merged
ronaldtse merged 7 commits into
mainfrom
wo20-ml-tiny
Nov 16, 2026
Merged

ronaldtse merged 7 commits into
mainfrom
wo20-ml-tiny

Conversation

@ronaldtse

Copy link
Copy Markdown
Contributor

Summary

  • Campaign A (issue WO20: tiny-student distillation (Arabic r7) + static-int8 sweep on released artifacts #441 A) — r7 → tiny student via the microkimi recipe: the scan-first frozen-bridge arm (ByT5-small, the 2.0 rung recipe verbatim + β 0.1 MSE through five scan-selected rank-256 bridges, canonical r7 labels e70ce991…, 10,995 steps over four resumable 4h windows) scores 4.0545 full-set DER-CE. Strict gate (teacher+0.5pp) FAILS — but this is a SEPARATED-IMPROVEMENT over the published frontier: paired Δ vs G2a −0.4385pp, CI95 [−0.65, −0.28], p=0 (G2a's released predictions re-scored), at half G2a's epochs. First separated-positive student-side lever on record.
  • Campaign B (issue WO20: tiny-student distillation (Arabic r7) + static-int8 sweep on released artifacts #441 B) — static-int8 sweep on six released fp32 artifacts, judged fp32-vs-int8static under one framing (CPU contract) with the WO03 CER gate AND the confident-flip margin gate: urd-vocalized-1.0 PASS (0.0902pp, 0 flips, ×0.36 size, ×1.91 CPU), heb-vocalized-1.1 PASS (0.2907pp, 0.12%); tha-base / heb-1.0 / heb-small FAIL (two distinct mechanisms: margins-dominated 11% flips vs CER-dominated 7.99pp); tha-phonemic-1.0 N/A (plain-decoder artifact, recipe is KV-only). Failing models keep their fp32 artifacts; nothing published; owner decisions held.
  • Pre-registration (TODO.sota-2026/06) was written before any GPU spend; verdicts recorded with the same care as negatives; single-seed caveat attached.
  • Provenance on isx-training: bucket: wo20-tiny-g2w/ (predictions, final_eval.json), wo20-quant/ (per-model reports, candidate fixtures, built int8static variants), wo20-tiny-scan/ (scan.json, bridges.npz).

Test plan

  • Review the campaign-A paired-bootstrap math (train_wo20_tiny_hf.py in interscript-train wo20-ml-tiny) and the banked-prediction scoring path
  • Confirm campaign-B framing disclosure (CPU contract, single framing both precisions) is acceptable for the ledger
  • Owner decisions: publish the two PASS int8static variants (new artifacts, version numbers are owner calls); arm the 6-epoch bridge run vs G2a

Ronald Tse added 7 commits November 16, 2026 21:43
…tool

Campaign A: the microkimi-recipe pre-registration (TODO.sota-2026/06) —
scan-first directional ridge-R2 pairing with the 0.30 / 2x-scrambled-control
kill rule, base recipe = the 2.0 rung verbatim, judge = paired bootstrap vs
r7 on the full 1,200-paragraph windowed protocol, tolerance teacher+0.5pp.

Campaign B: tools/wo20_static_int8_sweep.py — six released fp32 artifacts
through the shipped build_int8_static_zip path; judge = WO03 int8 CER gate
(<=2.0pp) + confident flips (<=1%) against the fp32 zip under one framing,
plus kit/fixture rows, size and CPU/CUDA latency. Eval slices under
conformance/wo20/ with full provenance (400-row eval slices only; never
training data).
…ries

load_slice read {name}_eval_slice.jsonl but the committed fixtures are
{name}_eval_400.jsonl; three of the six targets (tha-base, heb-1.0,
heb-1.1) pin split parts with per-part digests plus a whole-file digest —
fetch both layouts, verify every digest.
onnxruntime-gpu 1.23 cannot load the CUDA EP in the job image (CUDA 13
libs missing) and silently fell back - so every measurement was already
CPU-framed; drop the misleading cuda variants and declare the framing.
The tha-base arm died from holding four 2.8GB graph sets resident:
phase the sweep (fp32 decode + latency -> free; build -> free; int8
decode + latency -> free; margins last) with gc between phases.
…ep 2 PASS / 3 FAIL / 1 N/A

Campaign A (r7 -> tiny student, microkimi recipe): the scan-first frozen
bridge arm at the matched 3-epoch rung scores 4.0545 full-set DER-CE -
GATE-FAIL vs teacher+0.5pp (2.7093) but SEPARATED-IMPROVEMENT over the
published frontier: paired delta vs G2a -0.4385pp CI95 [-0.65, -0.28]
p=0 at half G2a's epochs. First separated-positive student-side lever
on record; pre-registered in TODO.sota-2026/06 with the result recorded.

Campaign B (static-int8 sweep): urd-vocalized-1.0 PASS (CER-delta
0.0902pp, zero confident flips, x0.36 size, x1.91 CPU) and
heb-vocalized-1.1 PASS (0.2907pp, 0.12%); tha-base, heb-1.0, heb-small
FAIL (two distinct failure mechanisms - margins-dominated and
CER-dominated); tha-phonemic-1.0 N/A (plain-decoder artifact, recipe is
KV-only). Failing models keep their fp32 artifacts; nothing published;
owner decisions held on publication.
@ronaldtse
ronaldtse merged commit 75e4c66 into main Nov 16, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant