Skip to content

Add PROGRAM-KV: program-aware request/KV controller - #4

Open
Anjiangy wants to merge 1 commit into
RIDE-Lab:mainfrom
Anjiangy:add-program-kv
Open

Anjiangy wants to merge 1 commit into
RIDE-Lab:mainfrom
Anjiangy:add-program-kv

Conversation

@Anjiangy

Copy link
Copy Markdown

Add PROGRAM-KV: program-aware request/KV controller

新增一条 portfolio 条目:Program-Aware Request/KV Controller(code: PROGRAM-KV)。

一句话定位

以 program 为单位的准入与 KV 常驻控制:按块预算把驻留集合装箱到多卡,按需换出/换回某个 program 的 KV,
并随超容比在细粒度换入换出与整批轮转之间切换。

机制

  • 准入:Σp_resident + p_next ≤ (1 − headroom)·C,否则排队(把死锁变成排队)
  • 装箱:按每个 program 的块数(p = ceil(L/128)+2)贪心分卡
  • 常驻:KV 降级到 host 内存池(/kv/park)与按需取回(/kv/restore),192 KB/token
  • 两档策略:K/N ≤ 0.55 时用整批轮转(批内不换入换出);否则细粒度

实测(2 × Ascend 910B2,Qwen2.5-14B,892 块/卡)

负载(每卡超容比) 对手 本方案 结论
异质 16 program(2.1×) 双卡轮询 51.9 / 请求级路由器 5.4 step/min 74.4 +43% / 13.8×
同质 24×20k(2.1×) 引擎 LRU 30.2 60.7~61.3 +100%
同质 12×20k(1.05×) 裸 TA 62.1 47.7 这一档我们输

收益随超容比下降:1.15× +13.2% / 1.53× +12.0% / 1.91× +6.6% / 2.35× reload 型失效。

变更

只改 data/control-plane-projects.json,在 projects 数组末尾追加一项(见 diff)。

@Anjiangy
Anjiangy force-pushed the add-program-kv branch 2 times, most recently from c39abd2 to c1bb32e Compare November 14, 2026 08:35

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant