Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
0d4de47
0.3.1a0: cluster-locality default-on, architecture-adaptive expert pa…
FonaTech Nov 7, 2026
de46962
0.3.1a1: new Web UI, file picking/upload, LAN access, lazy-expert probe
FonaTech Nov 7, 2026
ae9cecf
Make the lazy-probe sweep test independent of installed backends
FonaTech Nov 7, 2026
5ae7b08
0.3.1a2: move the en-zh translation demo out of the kernel
FonaTech Nov 7, 2026
1fb9175
0.3.1a3: segment clustering, MLX multimodal, tool-call SFT, MLX cache…
FonaTech Nov 8, 2026
6a79c98
shards: keep -100 labels under NumPy 2 for uint16 .lmask records
FonaTech Nov 8, 2026
4e5214c
serve: parse tool calls; MLX: multimodal DPO/ORPO/GRPO/distill
FonaTech Nov 8, 2026
ee1b2d2
0.3.2a0: native kernel, compiled MLX step, MLX activation recomputation
FonaTech Nov 8, 2026
acf9ed6
Add chronos-enzh-50m-sft model package (en->zh, research only)
FonaTech Nov 8, 2026
c83ed56
chronos-enzh-50m-sft: certifi CAs for --download, refuse before loading
FonaTech Nov 8, 2026
8ce2b8d
0.3.2a1: long-context MLX training (blockwise attention, chunked CE/M…
FonaTech Nov 9, 2026
b27ef98
0.3.2a2: MLP-scope recompute, lookahead options, web UI fields, compi…
FonaTech Nov 9, 2026
86dff81
0.3.2a3: compile segment-mode cluster locality, recover from a failed…
FonaTech Nov 10, 2026
2c63130
Reject token ids beyond vocab_size before MLX reads the embedding table
FonaTech Nov 10, 2026
206a4a7
0.3.3a0: block-order loop, sigmoid step size, general 32k tokenizer, …
FonaTech Nov 11, 2026
4a92c06
0.3.3a1: every loop knob in planner and auto-tune, probe scores cross…
FonaTech Nov 12, 2026
b0f62bc
0.3.3: stable release with the integrated documentation set
FonaTech Nov 13, 2026
bfddacc
merge origin/main into release-0.3.3
FonaTech Nov 13, 2026
782d125
ci(python-app): install fastapi, uvicorn, httpx for lazy-probe webui …
FonaTech Nov 13, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
7 changes: 6 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ on:
push:
branches: [main, dev]
pull_request:
branches: [main]
branches: [main, dev]

jobs:
test:
Expand Down Expand Up @@ -36,3 +36,8 @@ jobs:
- name: Unit tests
run: |
python -m pytest tests/ -v --tb=short

- name: Demo driver tests
run: |
# The translation demo's model driver; the kernel suite above stays task-neutral.
python -m pytest demos/translation/tests/test_model_enzh_50m_sft.py -v --tb=short
2 changes: 1 addition & 1 deletion .github/workflows/python-app.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ jobs:
run: |
python -m pip install --upgrade pip
pip install torch==2.10.0 --index-url https://download.pytorch.org/whl/cpu
pip install -c .github/ci-constraints.txt flake8 pytest psutil -r requirements.txt
pip install -c .github/ci-constraints.txt flake8 pytest psutil fastapi uvicorn httpx -r requirements.txt
- name: Lint with flake8
run: |
# stop the build if there are Python syntax errors or undefined names
Expand Down
386 changes: 386 additions & 0 deletions CHANGELOG.md

Large diffs are not rendered by default.

4 changes: 3 additions & 1 deletion MANIFEST.in
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
include README.md README_zh.md LICENSE THIRD_PARTY_NOTICES.md CHANGELOG.md requirements.txt
recursive-include chronos/monitor/static *.html *.js *.css
recursive-include chronos/webui/static *.html *.js *.css *.svg
recursive-include chronos/webui/i18n *.json
include ui/event_contract.js
global-exclude __pycache__ *.py[cod] .DS_Store
prune runs
Expand All @@ -10,5 +12,5 @@ prune results
prune reports
prune .archive
prune Translation?Training?Data
prune demos
prune chronos/mlx/training/recipes
prune models
199 changes: 109 additions & 90 deletions README.md

Large diffs are not rendered by default.

184 changes: 109 additions & 75 deletions README_zh.md

Large diffs are not rendered by default.

47 changes: 19 additions & 28 deletions THIRD_PARTY_NOTICES.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,36 +13,26 @@ below or linked.
**License**: Apache License, Version 2.0
**Copyright**: Copyright jingyaogong and MiniMind Contributors

**Files derived from MiniMind**:
**Status since 0.3.2: no runtime dependency, no bundled code.** Releases up to
0.3.1a3 imported a minimind checkout at runtime (`chronos.deps` cloned
`master` into `~/.cache/chronos/minimind-master/` and put it on `sys.path`).
0.3.2 removes that dependency. The pieces Chronos used are re-implemented in
`chronos/model/kernel.py` (`ChronosBaseConfig`, RoPE/YaRN tables, gated FFN,
logging and resume utilities). Nothing is downloaded.

| Chronos file | Derived from MiniMind |
|---|---|
| `chronos/model/config.py` | `model/model_minimind.py` — `MiniMindConfig` |
| `chronos/model/model_chronos.py` | `model/model_minimind.py` — `MiniMindModel`, `MiniMindForCausalLM`, `MiniMindBlock` |
| `chronos/model/moe_chronos.py` | `model/model_minimind.py` — `MOEFeedForward`, `FeedForward` |
| `chronos/model/hybrid_attention.py` | `model/model_minimind.py` — `Attention`, `apply_rotary_pos_emb`, `repeat_kv`, `RMSNorm` |
| `chronos/model/lookahead_router.py` | Architecture inspired by MiniMind's routing design |
| `chronos/trainer/chronos_trainer.py` | `trainer/train_pretrain.py` — pretraining loop structure |
| `chronos/trainer/sft_trainer.py` | `trainer/train_full_sft.py` — SFT loop + chat-template loss masking |
| `chronos/trainer/dpo_trainer.py` | `trainer/train_dpo.py` — log-σ(β·logits) DPO formulation, `_logits_to_log_probs` helper |
| `chronos/trainer/grpo_trainer.py` | `trainer/train_grpo.py` — GRPO group-relative PG + KL formulation |
| `chronos/trainer/distill_trainer.py` | `trainer/train_distillation.py` — temperature-scaled KL distillation |
| `chronos/trainer/reward.py` | `trainer/trainer_utils.py` — `LMForRewardModel` ABI (`get_score`) wrapper |
| `chronos/eval/io_profiler.py` | `eval_llm.py` — evaluation patterns |

**Nature of modifications**:
**What still derives from MiniMind**:

- `ChronosConfig` extends `MiniMindConfig` with new fields: `lookahead_steps`, hybrid attention dims, λ loss coefficients (balance / temporal / lookahead / router-anchor), VRAM budget, pinned RAM fraction, `storage_format`, `cluster_manifest_path`.
- `ChronosForCausalLM` replaces `MiniMindForCausalLM` with hybrid attention, LookaheadRouter injection, dual-layer routing hooks, an `available_expert_masks` forward kwarg, and `_tied_weights_keys` / `all_tied_weights_keys` shims for HF safetensors round-trips.
- `ChronosMOEFeedForward` extends `MOEFeedForward` with always-resident shared experts and compile-safe soft gating (`avail[i] * expert_out + (1−avail[i]) * shared_out`).
- `MLAAttention` and `SlidingWindowAttention` are new implementations sharing RMSNorm and RoPE utilities from MiniMind's `Attention`.
- The full Stage 1–6 trainer family (Pretrain / SFT / DPO / ORPO / GRPO / Distill) wraps the corresponding minimind loops where they exist (Pretrain / SFT / DPO / GRPO / Distill) and adds **ORPO from scratch** (Hong et al. 2024 — minimind has no ORPO trainer). Every wrapper injects `chronos_loss_term` (lookahead + temporal + balance) and an optional `router_kl_anchor` term.
- `ChronosGRPOTrainer` is **self-contained**: it ships its own rollout (vanilla `model.generate` style sampling) and accepts a pluggable `reward_fn`. It does not require minimind's sglang-based `rollout_engine`.
| Chronos file | What it keeps from MiniMind |
|---|---|
| `chronos/model/kernel.py` | Field names and defaults of `MiniMindConfig`, the rotate-half RoPE/YaRN layout and numerics, and the `gate_proj/up_proj/down_proj` FFN naming. They are kept so that checkpoints trained before 0.3.2 load bit-identically. The code is a new implementation. |
| `chronos/assets/tokenizers/legacy_6400/` | `tokenizer.json` and `tokenizer_config.json`, copied unchanged (data files only) from commit `24896cd2c4e9328a0911cf69ff61ace17a459b71`, with its `LICENSE` and a `NOTICE` that records the sha256 of each file. It is the default tokenizer only for old checkpoints that never recorded one. |
| `chronos/trainer/sft_trainer.py`, `dpo_trainer.py`, `grpo_trainer.py`, `distill_trainer.py` | The loop structure and loss formulations first followed minimind's `train_full_sft.py`, `train_dpo.py`, `train_grpo.py` and `train_distillation.py` (chat-template loss masking, log-σ(β·Δ) DPO, group-relative policy gradient with KL, temperature-scaled KD). These trainers are self-contained Chronos code. ORPO was written from scratch (Hong et al. 2024). |
| `chronos/trainer/reward.py` | The `get_score(tokenizer, messages)` reward-model ABI wrapper. |
| `tools/compare_minimind_chronos*.py` | Optional comparison harnesses that run minimind as an external baseline. They need a checkout the user provides through `--minimind-dir` or `CHRONOS_MINIMIND_DIR`. |

**Automatic download**: When `minimind-master` is not found locally, Project Chronos
automatically clones it from `https://github.com/jingyaogong/minimind` into
`~/.cache/chronos/minimind-master/`. This download is governed by the Apache-2.0
license of the MiniMind project.
Chronos-specific parts (MLA/SWA hybrid attention, recurrent-depth controller,
LookaheadRouter, shared-expert MoE, lazy expert paging, speculative decoding,
MLX backend, tokenizer builder) are original work.

**Apache-2.0 License Text**:

Expand All @@ -60,7 +50,8 @@ See the License for the specific language governing permissions and
limitations under the License.
```

Full license: https://github.com/jingyaogong/minimind/blob/master/LICENSE
Full license: https://github.com/jingyaogong/minimind/blob/master/LICENSE (also shipped as
`chronos/assets/tokenizers/legacy_6400/LICENSE`)

---

Expand Down
33 changes: 20 additions & 13 deletions benchmark_compare.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
"""
benchmark_compare.py — Background benchmark: minimind (pure RAM) vs Chronos (SSD+DRAM)
benchmark_compare.py — Background benchmark: dense baseline (pure RAM) vs Chronos (SSD+DRAM)

Runs in background, writes results to benchmark_results.json
Usage: python benchmark_compare.py
Expand All @@ -11,7 +11,6 @@
if _pkg_root not in sys.path:
sys.path.insert(0, _pkg_root)

import chronos.deps # auto-bootstrap minimind on sys.path
import torch
import psutil

Expand Down Expand Up @@ -68,18 +67,25 @@ def measure_tokens_per_sec(model, vocab_size, device, n_tokens=50, seq_len=32):
elapsed = time.monotonic() - t0
return n_tokens / max(elapsed, 1e-6)

# ── Benchmark 1: minimind pure RAM ────────────────────────────────
# ── Benchmark 1: dense baseline (pure RAM) ─────────────────────────
# A conventional transformer built from the same native kernel: full causal
# GQA (no MLA/SWA), one pass per layer (no recurrence), all experts resident.
# Before 0.3.2 this slot ran minimind; tools/compare_minimind_chronos*.py
# still compare against an external minimind checkout on request.

def bench_minimind():
log('=== Benchmark 1: MiniMind (pure RAM) ===')
from model.model_minimind import MiniMindConfig, MiniMindForCausalLM
def bench_baseline():
log('=== Benchmark 1: dense baseline (pure RAM) ===')
from chronos.model.config import ChronosConfig
from chronos.model.model_chronos import ChronosForCausalLM

cfg = MiniMindConfig(
cfg = ChronosConfig(
hidden_size=512, num_hidden_layers=8, use_moe=True,
num_experts=4, num_experts_per_tok=1,
num_experts=4, num_experts_per_tok=1, num_shared_experts=0,
use_hybrid_attention=False, recurrent_enabled=False,
lambda_lookahead=0.0, lambda_temporal=0.0,
)
ram_before = ram_used_gb()
model = MiniMindForCausalLM(cfg)
model = ChronosForCausalLM(cfg)
model.eval()
ram_after = ram_used_gb()
ram_model_gb = ram_after - ram_before
Expand All @@ -93,7 +99,7 @@ def bench_minimind():
train_losses = []
for step in range(5):
x = torch.randint(0, cfg.vocab_size, (2, 64))
out = model(x, labels=x)
out, _lookahead = model(x, labels=x)
loss = out.loss + out.aux_loss
loss.backward()
optimizer.step()
Expand All @@ -111,13 +117,14 @@ def bench_minimind():
log(f' TPS={tps:.1f} RAM_train={ram_train:.3f}GB RAM_infer={ram_infer:.3f}GB')

result = {
'name': 'MiniMind (pure RAM)',
'name': 'Dense baseline (GQA, pure RAM)',
'params_m': round(params_m, 1),
'ram_model_gb': round(ram_model_gb, 3),
'ram_train_gb': round(ram_train, 3),
'ram_infer_gb': round(ram_infer, 3),
'tokens_per_sec': round(tps, 1),
'train_losses': [round(l, 4) for l in train_losses],
'kv_cache_type': 'GQA (full K/V)',
}
del model
gc.collect()
Expand Down Expand Up @@ -241,9 +248,9 @@ def bench_chronos():

results = {}
try:
results['minimind'] = bench_minimind()
results['baseline'] = bench_baseline()
except Exception as e:
log(f'minimind bench failed: {e}')
log(f'baseline bench failed: {e}')
import traceback; log(traceback.format_exc())

try:
Expand Down
6 changes: 1 addition & 5 deletions chronos/__init__.py
Original file line number Diff line number Diff line change
@@ -1,9 +1,5 @@
# Project Chronos: On-device low-latency lookahead dual-layer MoE inference
__version__ = "0.3.0"

# Bootstrap minimind dependency before any submodule imports
from chronos.deps import ensure_minimind as _ensure_minimind
_ensure_minimind()
__version__ = "0.3.3"

# Register with HuggingFace AutoXXX factories so
# AutoModelForCausalLM.from_pretrained("path/to/chronos") works out of the box.
Expand Down
3 changes: 1 addition & 2 deletions chronos/app.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,6 @@
from ui.gradio_compat import gr
from ui.event_contract import UIEventContract, event_contract_app_kwargs

import chronos.deps # auto-bootstrap minimind
from ui.i18n import (
LANGUAGE_CHOICES, DEFAULT_LANGUAGE,
t, set_current_lang, build_language_update, get_registered_components,
Expand Down Expand Up @@ -78,7 +77,7 @@ def _hero_html() -> str:
"<span class='app-meta-chip'>Apache-2.0</span>"
"<span class='app-meta-chip'>Python 3.10+</span>"
"<span class='app-meta-chip'>PyTorch 2.1+</span>"
"<span class='app-meta-chip'><a href='https://github.com/jingyaogong/minimind' target='_blank'>MoE kernel: minimind</a></span>"
"<span class='app-meta-chip'>Kernel: Chronos native</span>"
"</div>"
"</div>"
)
Expand Down
Empty file added chronos/assets/__init__.py
Empty file.
Loading
Loading