Kyojin: open-source engine that runs the largest open models on one mini PC or one graphics card, about 2× faster than llama.cpp. AMD (ROCm) and NVIDIA (CUDA), built on ExLlamaV3.
-
Updated
Nov 1, 2026 - Python
#
exl3
Here are 90 public repositories matching this topic...
DeepSeek-V4.1-Flash-1017 EXL3 2.9bpw on 2x NVIDIA DGX Spark (GB10) with TensorFold: 2.1-2.3x the vLLM kit's decode speed.
-
Updated
Nov 1, 2026 - Python
An environment where open models work as agents: a first-level harness with routing, skills, memory and sandboxed tools, on an engine that drafts and verifies its own tokens.
self-hosted quantization ai-agents inference-engine llm-inference local-ai qwen speculative-decoding nvfp4 dgx-spark exl3
-
Updated
Oct 30, 2026 - Python
GLM-5.3-Flash EXL3 (320B MoE) on 2x NVIDIA DGX Spark — production serving kit, 1M context, 97%+ multi-session prefix caching, DFlash2 spec decode
self-hosted glm sparse-attention llm-serving vllm llm-inference flashinfer gb10 dgx-spark tensor-parallel exl3
-
Updated
Oct 18, 2026 - Python
Validated GLM-5.3 Flash recipe for 2x NVIDIA RTX PRO 6000 Blackwell 96GB: 262K context, EXL3/TR3, adaptive MTP, tools, and vision.
multimodal long-context vllm local-llm speculative-decoding nvidia-blackwell rtx-pro-6000 exl3 glm-5-3
-
Updated
Sep 20, 2026 - Python
A tiered-memory system design for workloads that don't fit in RAM: measure the working set, pin the hot tier, stream the cold tier from flash. Ships the residency calculator, measurement harnesses, and the build recipes behind it. Predictions validated against public benchmarks.
mac memory nvidia diagnostics quantization wafer mx mlx low-resource mixture-of-experts memory-optimization memory-tiering llm llama-cpp vllm gptq gguf vllm-serve nvfp4 exl3
-
Updated
Nov 1, 2026 - Python
Serve EXL3 (ExLlamaV3 trellis) quantized models on vLLM fork runtimes — any architecture, mixed per-layer bitrates, composable with source-format non-routed weights
-
Updated
Oct 24, 2026 - Python
Qwen3.8-27B EXL3 (4.00 bpw) + DFlash2 speculative decoding for ExLlamaV3, validated at 262k context on a 24 GB RTX 3090
-
Updated
Sep 20, 2026 - Python
vLLM for NVIDIA CMP 170HX (Ampere sm_80): fast, repeatable model serving on one card or several
cuda nvidia glm ampere multi-gpu pipeline-parallelism tensor-parallelism sparse-attention vllm llm-inference qwen speculative-decoding deepseek exl3 cmp-170hx sm80
-
Updated
Nov 1, 2026 - Python
DeepSeek 1017 EXL3 on 3-4 DGX Sparks. TP4 4.0.2: 4,264 cold prefill tok/s; C16 code 320.1 / prose 285.4 native decode sum. Smart KV cache controller + Smart Adaptive speculation. C16/R48, 9M KV, 3.5 GiB checkpoints. TP3 1.2.8 (+58% cold prefill) preserved. Exact recipes, source and benchmarks.
inference benchmarks local-llm speculative-decoding deepseek prefix-caching dgx-spark exl3 tensorfold
-
Updated
Nov 1, 2026 - Python
EXL3 inference engine for Qwen3.8-Flash-Next on AMD Strix Halo: Ryzen AI Max+ 395, Radeon 8060S, gfx1151, ROCm/HIP in C++20. Exact speculative decoding, OpenAI-compatible server, and speed claims stated against measured hardware ceilings.
amd hip quantization rocm inference-engine cpp20 model-inference llm-serving local-llm llm-inference qwen speculative-decoding openai-compatible qwen3 amd-apu strix-halo gfx1151 ryzen-ai-max radeon-8060s exl3
-
Updated
Nov 1, 2026 - C++
GLM-5.3-Flash EXL3 on 4x RTX PRO 6000 with TensorFold: one-command recipe and engine patches. By Aevonix Research in collaboration with Mia's AI Lab.
-
Updated
Oct 8, 2026 - Python
Qwen3.8-Flash-Next EXL3 3.05bpw on ONE DGX Spark (GB10): ~2.4x prefill (700->1777 t/s), prompt-lookup drafter, native vision/video, 262k ctx. Unofficial TensorFold v0.6.5 fork, 18 patches, byte-exact receipts.
-
Updated
Oct 11, 2026 - Python
Add this topic to your repo
To associate your repository with the exl3 topic, visit your repo's landing page and select "manage topics."