You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
WO20: tiny-student distillation (Arabic r7) + static-int8 sweep on released artifacts #441
Sharpen the ML claims with zero new data: two executable campaigns on the EXISTING canonical teachers and released artifacts.
A. Distill the canonical Arabic teacher (r7) to tiny students
The durable distillation pipeline (microkimi recipe: scan-first ridge R2, closed-form stitch/bridge across widths, paired bootstrap eval) applied to r7's outputs → small byte-level students.
Judge: paired bootstrap against r7 on the held-out eval; report WER + latency/size deltas honestly. Publish-worthy only if the student holds within the pipeline's established tolerance; negative results are recorded as negatives (the program's convention).
B. Static-int8 quantization sweep on released models
The DeepSeek round's one positive lever (static int8) applied to the other released artifacts (urd-diac-1.0, the ByT5 Thai line, Hebrew if artifacts exist).
Judge: conformance vectors per model MUST stay within the established tolerance (the existing conformance kits); size/latency deltas reported. Any model failing tolerance keeps its fp32 artifact — record and move on.
Rules: training/distillation runs on the established infra (HF jobs, --timeout 4h + step checkpoints, mounted data — never local corpora); license gates untouched (urd-g2p stays owner-gated; r7 outputs are in-program). Report under 300 words per campaign with commands + outputs. Branch: wo20-ml-tiny (and any quant branch names it needs).
Stage 1 (pre-registered kill gate): the microkimi scan — closed-form directional ridge R² of r7 decoder-layer activations from ByT5-small decoder-layer streams on a calibration slice (minutes, no gradients). Kill rule: if the best directional R² does not clear the scan's alignment bar, no distillation GPU spend is armed; the scan result is recorded as the negative.
Stage 2 (scan-gated): ByT5-small student, base 2.0 recipe (Muon, r7 labels, sequence KD) + frozen scan-selected low-rank bridges as FitNets-style hidden-MSE projectors — the one mechanism in the bridge/stitch family not yet in the lever ledger.
Judge: paired bootstrap against r7 on the full 1,200-paragraph SadeedDiac-25 windowed zero-skip protocol (teacher decoded in-job on the same paragraphs; documents = resampling unit). Tolerance: the pipeline's established student gate, teacher + 0.5pp = 2.7864. Verdict recorded honestly either way.
Campaign B — static-int8 sweep: fp32 released artifacts urd-vocalized-1.0 (urd-diac-1.0), tha-phonemic-base-1.0 + tha-phonemic-1.0 (ByT5 Thai line), heb-vocalized-1.0, heb-vocalized-small-1.0, heb-vocalized-1.1 (Hebrew). Per model: build via the shipped build_int8_static_zip path (dynamic-int8 encoder + static-calibrated decoder), judge against the conformance-kit vectors where pinned and the WO03 precision gate (CER-delta ≤ 2.0pp int8, confident flips ≤ 1%) on banked fp32-reference decode; size + latency deltas reported. Any model failing tolerance keeps its fp32 artifact. urd-g2p stays owner-gated; nothing is published without owner sign-off.
Milestone: both campaigns are in flight (branches wo20-ml-tiny pushed on interscript-models + interscript-train).
Campaign A scan (kill-gate): job wo20-scan rhtse/6afb07bef7633f04f3556de5, t4-small, --timeout 4h. Verifies the r7 teacher + teacher_labels_r7.jsonl (sha e70ce991…) from teachers-arabic-v1, captures decoder streams, closed-form directional ridge-R² scan with scrambled controls, applies the pre-registered kill rule (R² ≥ 0.30 and ≥ 2× control). ARMED → bridges solved and the distill job launches; KILLED → recorded as the negative, no distillation spend.
Campaign B sweep: job wo20-quant rhtse/6afb0851f7633f04f3556e3d, a10g-small, --timeout 4h, resume-safe per-model markers. Six released fp32 artifacts → build_int8_static_zip → WO03 int8 gate (CER-Δ ≤ 2.0pp) + confident-flip gate (≤ 1%) vs the fp32 zip under one framing, kit/fixture conformance rows, size + CPU/CUDA latency.
Two earlier scan launches are recorded as failed starts, both infra: the teacher zip's checkpoint lives at run-007-news/best/ (first launch pointed at the parent dir) and the labels file's target column is teacher, not tgt — both fixed in train_wo20_tiny_hf.py; no GPU numbers were produced by either.
Milestone: Campaign A scan verdict — ARMED; distillation launched.
Scan job wo20-scan (rhtse/6afb07bef7633f04f3556de5) completed. The closed-form directional ridge-R² scan over all 5×7 (student layer, teacher layer) pairings cleared the pre-registered kill rule — best R² 1.0000 (student L0 → teacher L0) vs scrambled control 0.2539 (rule: ≥ 0.30 and ≥ 2× control). All five per-student-layer best pairings armed at rank-256 bridges: L0→L0 (1.0000), L1→L0 (0.9985), L2→L2 (0.9939), L3→L2 (0.9928), L4→L2 (0.9906). Honest caveat, recorded now: the near-1.0 R² across nearly all early-layer pairs is the microkimi scan's documented "early states align with everything" inflation — teacher-forced decoder streams share token-identity structure, so the kill rule passes easily; the informative test is the downstream gate, not the scan.
Distill job wo20-distill rhtse/6afb09f9f7633f04f3556f31 (a100-large, --timeout 4h, step checkpoints every 500 + resume): the 2.0 rung verbatim (Muon 0.01 + AdamW 1e-4, sequence KD α 0.5 T 2.0, BS 8, 3 epochs, canonical r7 labels sha e70ce991…) + the single pre-registered delta β·MSE through the frozen bridges. Epilogue: full 1,200-paragraph SadeedDiac-25 windowed zero-skip decode for student AND in-job teacher, paired bootstrap vs r7 (documents = resampling unit), gate = teacher + 0.5pp = 2.7864, secondary read = delta vs the 4.8218 base rung.
Campaign B sweep continues on wo20-quant (urd reference decode done; int8static build in progress).
Milestone: distillation training started — recipe now bit-verified against the published 2.0 rung.
Job wo20-distill rhtse/6afb0ba1f7633f04f3556fc0 (a100-large, --timeout 4h, step ckpts): 29,322 usable label pairs × 3 epochs = 10,995 steps — exactly the published run-006 step count — Muon 0.01 (128 matrix params) + AdamW 1e-4 (42 embedding-like), sequence KD α 0.5 T 2.0, gradient checkpointing, plus the single pre-registered delta β 0.1·MSE through the five frozen scan-selected bridges. Two earlier distill launches are recorded as failed starts, both my wiring bugs (armed_pairs format; a 384-B data filter copied from the wrong lineage call site — the ara spec's real caps are units ≤ 1450 B, labels ≤ 2×max_len); no GPU numbers were produced by either.
Campaign B sweep wo20-quant (rhtse/6afb0ad5f7633f04f3556f93): urd-vocalized-1.0 past reference decode, int8static build in progress; the first sweep launch is recorded as a failed start (per-row session rebuild in the margin pass — fixed to build once per zip).
Milestone: first campaign-B verdict + measured training cadence.
urd-vocalized-1.0 (urd-diac-1.0): PASS — int8static viable. Parity CER-Δ 0.0902pp (gate ≤ 2.0pp), zero confident flips, row-exact int8-vs-fp32 disclosed in the report; artifact ×0.359 size (1.42 GiB → 510 MiB), ×1.91 CPU decode speed. Report + the built variant sit on the bucket at wo20-quant/.
Training cadence measured on wo20-distill: ~5 steps/min (teacher-forced fwd + checkpointed bwd at BS 8 × 1450-B windows, teacher fp16 in the loop). The 10,995-step arm therefore spans ~36h of GPU time — it will run as a sequence of --timeout 4h jobs resuming from the 500-step bucket checkpoints (labels.sha-pinned), the established resume pattern. Training is healthy: step 200, loss 12.92 → 2.98 (ce 1.57, kd 1.45, bridge-MSE 27.7 → 14.7 as the frozen bridges pull student activations toward teacher geometry).
One more sweep fix recorded: the tha slice rows key as word/phonemic (tha-phonemic corpora convention), not src/tgt — loader normalizes; urd's PASS marker made the relaunch skip it.
Milestone: second campaign-B verdict — the confident-flip gate bites.
tha-phonemic-base-1.0: FAIL — keeps its fp32 artifact. Parity CER-Δ 0.6583pp (inside the ≤ 2.0pp int8 gate) but confident flips 11.01% vs the ≤ 1% margin gate — exactly the failure mode the teacher-forced margin analysis exists to catch: quantization noise flipping confident argmaxes where CER on the surface stays quiet. Size ×0.368, CPU speed ×1.87 were healthy; the fragility gate fails the artifact. Recorded, moving on.
Sweep now on tha-phonemic-1.0 (ByT5-small line). Distill at step 900, loss 0.615 (ce 0.136, kd 0.128, bridge-MSE 4.83, still descending).
Framing note, corrected on the banked reports: the job image's onnxruntime-gpu cannot load the CUDA EP (needs CUDA 13 libs), so every measurement is on the CPU contract framing, both precisions, one framing — the urd report on the bucket now says so explicitly.
heb-vocalized-1.0 (heb-diac-1.0): FAIL — keeps its fp32 artifact. Parity CER-Δ 3.2769pp breaches the ≤ 2.0pp int8 gate outright, and confident flips 1.46% also breach the ≤ 1% margin gate — both judges reject the static-int8 variant.
tha-phonemic-1.0: the build crashed with StopIteration — that artifact is a plain-decoder zip (no decoder-kv.onnx), and the static recipe re-quantizes the KV decode path. The sweep now short-circuits such artifacts as N/A — keeps fp32 (committed); it re-records on the next sweep relaunch via the marker chain.
heb-vocalized-small-1.0 in flight; heb-vocalized-1.1 + the tha-1.0 N/A rerun queued.
Distill: step 5350/10995, loss 0.351 (ce 0.075, kd 0.042, bridge-MSE 2.92) — roughly the halfway mark, resume chain continues.
heb-vocalized-small-1.0 (heb-diac-small-1.0): FAIL — keeps its fp32 artifact. Parity CER-Δ 7.9907pp — nearly 4x the ≤ 2.0pp int8 gate (confident flips 0.16% pass, but the CER gate is decisive). Notable asymmetry vs heb-vocalized-1.0 (big model: flips-dominated fail at 1.46%, CER-Δ 3.28pp; small model: CER-dominated fail with clean margins) — static-int8 degrades the two Hebrew artifacts through different mechanisms.
Sweep verdicts so far: urd PASS / tha-base FAIL / tha-1.0 N/A (pending rerun; plain-decoder) / heb-1.0 FAIL / heb-small FAIL. heb-vocalized-1.1 downloading; the sweep job's 4h window closes soon and the marker chain resumes it for the remainder.
Milestone: training complete; eval epilogue in progress.
All 10,9995/10,995 steps done — final train loss 0.35 (ce 0.075, kd 0.037, bridge-MSE 2.79), checkpoints at step-10000/10500 on the bucket. The job resumed cleanly from step-8500 across the 4h window boundary ("resume from step-8500" in the log — the labels.sha-pinned checkpoint chain works exactly as designed).
The eval epilogue is running: full 1,200-paragraph SadeedDiac-25 windowed zero-skip decode for the student (in flight; early windows show real well-voweled Arabic) then the in-job teacher on the same paragraphs, then the paired bootstrap. Measured decode cadence ~5 windows/min (batch-8, the protocol's framing) → the dual decode spans roughly a day across several --timeout 4h windows; every window is checkpointed to the bucket and resumes losslessly.
Campaign B: 5 of 6 verdicts banked (urd PASS; tha-base, heb-1.0, heb-small FAIL; tha-1.0 N/A plain-decoder). heb-vocalized-1.1 fp32 reference decode in flight — the last target.
WO20 COMPLETE — both campaigns landed. Verdicts below; full entries in RESULTS.md (PR #442, branch wo20-ml-tiny).
Campaign A — r7 → tiny student (microkimi recipe)
Scan: ARMED (best directional R² 1.0000 vs control 0.2539; the near-1.0 inflation across pairs is recorded as the scan's documented early-layer effect).
Bridge arm (2.0 rung verbatim + β 0.1 frozen-bridge MSE, canonical r7 labels, 10,995 steps): full-set DER-CE 4.0545 (teacher in-run 2.2093 ✓ reproduction of published 2.2864).
But the frontier MOVED: paired Δ vs G2a = −0.4385pp, CI95 [−0.65, −0.28], p = 0 (G2a's released predictions re-scored under the identical evaluator), at HALF G2a's epochs (3 vs 6), and ~−0.77pp vs the matched 3-epoch 2.0 rung. The first separated-positive student-side lever on record — the ledger's standing rule was satisfied by the pre-registered novel mechanism (scan-first directional ridge pairing + frozen bridges).
Campaign B — static-int8 sweep on released artifacts
model
verdict
CER-Δ (≤2.0)
confident flips (≤1%)
size
CPU
urd-vocalized-1.0
PASS
0.0902
0.00%
×0.359
×1.91
tha-phonemic-base-1.0
FAIL
0.66 ✓
11.01% ✗
×0.368
×1.87
tha-phonemic-1.0
N/A
plain-decoder artifact — recipe is KV-only
heb-vocalized-1.0
FAIL
3.28 ✗
1.46% ✗
×0.368
×1.35
heb-vocalized-small-1.0
FAIL
7.99 ✗
0.16% ✓
×0.359
×1.35
heb-vocalized-1.1
PASS
0.2907
0.12%
×0.368
×1.59
Key finding: the CER gate alone would have shipped two bad artifacts (tha-base and heb-1.0 pass CER while failing margins — two different failure mechanisms). Confident-flip gating stays mandatory. Every failing model keeps its fp32 artifact; urd-g2p untouched (owner-gated); nothing published, no versions picked — publication of the two PASS variants is the owner's call. Reports + built variants on isx-training:/wo20-quant/. Run: wo20-quant (final + predecessors).
Activity
ronaldtse commentedon Nov 16, 2026
WO20 opened — milestone plan (branch
wo20-ml-tiny, both repos).Campaign A — r7 → tiny students (microkimi recipe):
run-007-news(r7, 2.2864 full-set DER-CE) +teacher_labels_r7.jsonlfromteachers-arabic-v1(shae70ce991…verified in-job).Campaign B — static-int8 sweep: fp32 released artifacts
urd-vocalized-1.0(urd-diac-1.0),tha-phonemic-base-1.0+tha-phonemic-1.0(ByT5 Thai line),heb-vocalized-1.0,heb-vocalized-small-1.0,heb-vocalized-1.1(Hebrew). Per model: build via the shippedbuild_int8_static_zippath (dynamic-int8 encoder + static-calibrated decoder), judge against the conformance-kit vectors where pinned and the WO03 precision gate (CER-delta ≤ 2.0pp int8, confident flips ≤ 1%) on banked fp32-reference decode; size + latency deltas reported. Any model failing tolerance keeps its fp32 artifact. urd-g2p stays owner-gated; nothing is published without owner sign-off.Runs post here as they land.
ronaldtse commentedon Nov 16, 2026
Milestone: both campaigns are in flight (branches
wo20-ml-tinypushed on interscript-models + interscript-train).wo20-scanrhtse/6afb07bef7633f04f3556de5, t4-small,--timeout 4h. Verifies the r7 teacher +teacher_labels_r7.jsonl(shae70ce991…) fromteachers-arabic-v1, captures decoder streams, closed-form directional ridge-R² scan with scrambled controls, applies the pre-registered kill rule (R² ≥ 0.30 and ≥ 2× control). ARMED → bridges solved and the distill job launches; KILLED → recorded as the negative, no distillation spend.wo20-quantrhtse/6afb0851f7633f04f3556e3d, a10g-small,--timeout 4h, resume-safe per-model markers. Six released fp32 artifacts →build_int8_static_zip→ WO03 int8 gate (CER-Δ ≤ 2.0pp) + confident-flip gate (≤ 1%) vs the fp32 zip under one framing, kit/fixture conformance rows, size + CPU/CUDA latency.Two earlier scan launches are recorded as failed starts, both infra: the teacher zip's checkpoint lives at
run-007-news/best/(first launch pointed at the parent dir) and the labels file's target column isteacher, nottgt— both fixed intrain_wo20_tiny_hf.py; no GPU numbers were produced by either.ronaldtse commentedon Nov 16, 2026
Milestone: Campaign A scan verdict — ARMED; distillation launched.
Scan job
wo20-scan(rhtse/6afb07bef7633f04f3556de5) completed. The closed-form directional ridge-R² scan over all 5×7 (student layer, teacher layer) pairings cleared the pre-registered kill rule — best R² 1.0000 (student L0 → teacher L0) vs scrambled control 0.2539 (rule: ≥ 0.30 and ≥ 2× control). All five per-student-layer best pairings armed at rank-256 bridges: L0→L0 (1.0000), L1→L0 (0.9985), L2→L2 (0.9939), L3→L2 (0.9928), L4→L2 (0.9906). Honest caveat, recorded now: the near-1.0 R² across nearly all early-layer pairs is the microkimi scan's documented "early states align with everything" inflation — teacher-forced decoder streams share token-identity structure, so the kill rule passes easily; the informative test is the downstream gate, not the scan.Distill job
wo20-distillrhtse/6afb09f9f7633f04f3556f31 (a100-large,--timeout 4h, step checkpoints every 500 + resume): the 2.0 rung verbatim (Muon 0.01 + AdamW 1e-4, sequence KD α 0.5 T 2.0, BS 8, 3 epochs, canonical r7 labels shae70ce991…) + the single pre-registered delta β·MSE through the frozen bridges. Epilogue: full 1,200-paragraph SadeedDiac-25 windowed zero-skip decode for student AND in-job teacher, paired bootstrap vs r7 (documents = resampling unit), gate = teacher + 0.5pp = 2.7864, secondary read = delta vs the 4.8218 base rung.Campaign B sweep continues on
wo20-quant(urd reference decode done; int8static build in progress).ronaldtse commentedon Nov 16, 2026
Milestone: distillation training started — recipe now bit-verified against the published 2.0 rung.
Job
wo20-distillrhtse/6afb0ba1f7633f04f3556fc0 (a100-large,--timeout 4h, step ckpts): 29,322 usable label pairs × 3 epochs = 10,995 steps — exactly the published run-006 step count — Muon 0.01 (128 matrix params) + AdamW 1e-4 (42 embedding-like), sequence KD α 0.5 T 2.0, gradient checkpointing, plus the single pre-registered delta β 0.1·MSE through the five frozen scan-selected bridges. Two earlier distill launches are recorded as failed starts, both my wiring bugs (armed_pairs format; a 384-B data filter copied from the wrong lineage call site — the ara spec's real caps are units ≤ 1450 B, labels ≤ 2×max_len); no GPU numbers were produced by either.Campaign B sweep
wo20-quant(rhtse/6afb0ad5f7633f04f3556f93): urd-vocalized-1.0 past reference decode, int8static build in progress; the first sweep launch is recorded as a failed start (per-row session rebuild in the margin pass — fixed to build once per zip).ronaldtse commentedon Nov 16, 2026
Milestone: first campaign-B verdict + measured training cadence.
urd-vocalized-1.0(urd-diac-1.0): PASS — int8static viable. Parity CER-Δ 0.0902pp (gate ≤ 2.0pp), zero confident flips, row-exact int8-vs-fp32 disclosed in the report; artifact ×0.359 size (1.42 GiB → 510 MiB), ×1.91 CPU decode speed. Report + the built variant sit on the bucket atwo20-quant/.wo20-distill: ~5 steps/min (teacher-forced fwd + checkpointed bwd at BS 8 × 1450-B windows, teacher fp16 in the loop). The 10,995-step arm therefore spans ~36h of GPU time — it will run as a sequence of--timeout 4hjobs resuming from the 500-step bucket checkpoints (labels.sha-pinned), the established resume pattern. Training is healthy: step 200, loss 12.92 → 2.98 (ce 1.57, kd 1.45, bridge-MSE 27.7 → 14.7 as the frozen bridges pull student activations toward teacher geometry).word/phonemic(tha-phonemic corpora convention), notsrc/tgt— loader normalizes; urd's PASS marker made the relaunch skip it.ronaldtse commentedon Nov 16, 2026
Milestone: second campaign-B verdict — the confident-flip gate bites.
tha-phonemic-base-1.0: FAIL — keeps its fp32 artifact. Parity CER-Δ 0.6583pp (inside the ≤ 2.0pp int8 gate) but confident flips 11.01% vs the ≤ 1% margin gate — exactly the failure mode the teacher-forced margin analysis exists to catch: quantization noise flipping confident argmaxes where CER on the surface stays quiet. Size ×0.368, CPU speed ×1.87 were healthy; the fragility gate fails the artifact. Recorded, moving on.tha-phonemic-1.0(ByT5-small line). Distill at step 900, loss 0.615 (ce 0.136, kd 0.128, bridge-MSE 4.83, still descending).ronaldtse commentedon Nov 16, 2026
Milestone: third campaign-B verdict.
heb-vocalized-1.0(heb-diac-1.0): FAIL — keeps its fp32 artifact. Parity CER-Δ 3.2769pp breaches the ≤ 2.0pp int8 gate outright, and confident flips 1.46% also breach the ≤ 1% margin gate — both judges reject the static-int8 variant.tha-phonemic-1.0: the build crashed with StopIteration — that artifact is a plain-decoder zip (nodecoder-kv.onnx), and the static recipe re-quantizes the KV decode path. The sweep now short-circuits such artifacts as N/A — keeps fp32 (committed); it re-records on the next sweep relaunch via the marker chain.heb-vocalized-small-1.0in flight;heb-vocalized-1.1+ the tha-1.0 N/A rerun queued.ronaldtse commentedon Nov 16, 2026
Milestone: fourth campaign-B verdict.
heb-vocalized-small-1.0(heb-diac-small-1.0): FAIL — keeps its fp32 artifact. Parity CER-Δ 7.9907pp — nearly 4x the ≤ 2.0pp int8 gate (confident flips 0.16% pass, but the CER gate is decisive). Notable asymmetry vsheb-vocalized-1.0(big model: flips-dominated fail at 1.46%, CER-Δ 3.28pp; small model: CER-dominated fail with clean margins) — static-int8 degrades the two Hebrew artifacts through different mechanisms.heb-vocalized-1.1downloading; the sweep job's 4h window closes soon and the marker chain resumes it for the remainder.ronaldtse commentedon Nov 16, 2026
Milestone: training complete; eval epilogue in progress.
All 10,9995/10,995 steps done — final train loss 0.35 (ce 0.075, kd 0.037, bridge-MSE 2.79), checkpoints at step-10000/10500 on the bucket. The job resumed cleanly from step-8500 across the 4h window boundary ("resume from step-8500" in the log — the labels.sha-pinned checkpoint chain works exactly as designed).
The eval epilogue is running: full 1,200-paragraph SadeedDiac-25 windowed zero-skip decode for the student (in flight; early windows show real well-voweled Arabic) then the in-job teacher on the same paragraphs, then the paired bootstrap. Measured decode cadence ~5 windows/min (batch-8, the protocol's framing) → the dual decode spans roughly a day across several
--timeout 4hwindows; every window is checkpointed to the bucket and resumes losslessly.Campaign B: 5 of 6 verdicts banked (urd PASS; tha-base, heb-1.0, heb-small FAIL; tha-1.0 N/A plain-decoder).
heb-vocalized-1.1fp32 reference decode in flight — the last target.ronaldtse commentedon Nov 16, 2026
WO20 COMPLETE — both campaigns landed. Verdicts below; full entries in RESULTS.md (PR #442, branch
wo20-ml-tiny).Campaign A — r7 → tiny student (microkimi recipe)
wo20-distill(final job + predecessors), scanwo20-scan. Provenance:isx-training:/wo20-tiny-g2w/(preds, final_eval.json),/wo20-tiny-scan/(scan.json, bridges.npz). Single-seed caveat attached.Campaign B — static-int8 sweep on released artifacts
Key finding: the CER gate alone would have shipped two bad artifacts (tha-base and heb-1.0 pass CER while failing margins — two different failure mechanisms). Confident-flip gating stays mandatory. Every failing model keeps its fp32 artifact; urd-g2p untouched (owner-gated); nothing published, no versions picked — publication of the two PASS variants is the owner's call. Reports + built variants on
isx-training:/wo20-quant/. Run:wo20-quant(final + predecessors).