A GGUF port of DFlash speculative decoding. Standalone C++/CUDA stack on top of ggml, runs on a single 24 GB RTX 3090, hosts the new Qwen3.6-27B.
~1.98x mean over autoregressive on Qwen3.6 across HumanEval / GSM8K / Math500, with zero retraining.
If you have CUDA 12+ and an NVIDIA GPU like RTX 3090 / 4090 / 5090, then all you need to do is
clone the repo
cd lucebox-hub/dflash
cmake -B build -S . -DCMAKE_BUILD_TYPE=Release
cmake --build build --target test_dflash -j
fetch target (~16 GB)
hf download unsloth/Qwen3.6-27B-GGUF Qwen3.6-27B-Q4_K_M.gguf --local-dir models/
matched 3.6 draft is gated: accept terms + set HF_TOKEN first
hf download z-lab/Qwen3.6-27B-DFlash --local-dir models/draft/
run
DFLASH_TARGET=models/Qwen3.6-27B-Q4_K_M.gguf python3 scripts/run.py --prompt "def fibonacci(n):"
That's it. No Python runtime in the engine, no llama.cpp install, no vLLM, no SG... Show more...A GGUF port of DFlash speculative decoding. Standalone C++/CUDA stack on top of ggml, runs on a single 24 GB RTX 3090, hosts the new Qwen3.6-27B.
~1.98x mean over autoregressive on Qwen3.6 across HumanEval / GSM8K / Math500, with zero retraining.
If you have CUDA 12+ and an NVIDIA GPU like RTX 3090 / 4090 / 5090, then all you need to do is
clone the repo
cd lucebox-hub/dflash
cmake -B build -S . -DCMAKE_BUILD_TYPE=Release
cmake --build build --target test_dflash -j
fetch target (~16 GB)
hf download unsloth/Qwen3.6-27B-GGUF Qwen3.6-27B-Q4_K_M.gguf --local-dir models/
matched 3.6 draft is gated: accept terms + set HF_TOKEN first
hf download z-lab/Qwen3.6-27B-DFlash --local-dir models/draft/
run
DFLASH_TARGET=models/Qwen3.6-27B-Q4_K_M.gguf python3 scripts/run.py --prompt "def fibonacci(n):"
That's it. No Python runtime in the engine, no llama.cpp install, no vLLM, no SGLang.
Luce DFlash will:
- Load Qwen3.6-27B Q4_K_M target weights (~16 GB) plus the matched DFlash bf16 draft (~3.46 GB) and run DDTree tree-verify speculative decoding (block size 16, default budget 22, greedy verify).
- Compress the KV cache to TQ3_0 (3.5 bpv, ~9.7x vs F16) and roll a 4096-slot target_feat ring so 256K context fits in 24 GB. Q4_0 is the legacy path and tops out near 128K.
- Auto-bump the prefill ubatch from 16 to 192 for prompts past 2048 tokens (~913 tok/s prefill on 13K prompts).
- Apply sliding-window flash attention at decode (default 2048-token window, 100% speculative acceptance retained) so 60K context still decodes at 89.7 tok/s instead of 25.8 tok/s.
- Serve over an OpenAI-compatible HTTP endpoint or a local chat REPL.
Running on RTX 3090, Qwen3.6-27B UD-Q4_K_XL (unsloth Dynamic 2.0) target, 10 prompts/dataset, n_gen=256:
Bench AR tok/s DFlash tok/s AL Speedup
HumanEval 34.90 78.16 5.94 2.24x
Math500 35.13 69.77 5.15 1.99x
GSM8K 34.89 59.65 4.43 1.71x
Mean 34.97 69.19 5.17 1.98x
Lucebox optimization hub: hand-tuned LLM inference, built for specific consumer hardware. - Luce-Org/lucebox-hub
GitHub
Mister_Hangman
in reply to ☆ Yσɠƚԋσʂ ☆ • • •☆ Yσɠƚԋσʂ ☆
in reply to Mister_Hangman • • •HiddenLayer555
in reply to Mister_Hangman • • •doo
in reply to ☆ Yσɠƚԋσʂ ☆ • • •