KIRIGAMI

A 35B MoE that won't fit

Qwen3.6-35B-A3B is a fine-grained Mixture-of-Experts model: 40 layers, 256 routed experts per layer, top-8 routing. Quantized to NVFP4 it is ~21 GB of weights — and on a 24 GB card it fails to allocate a usable KV cache by a small, fixed margin:

The deficit is context-insensitive — it's a fixed weight overflow, not a KV problem. The model is simply ~1.5 GB too big.

The expert pool is the redundant part

In a fine-grained MoE, only top-k experts fire per token — the routed-expert pool is the most redundant parameter class in the model. So instead of quantizing harder or offloading, we remove whole experts: keep the most important ones, drop the rest.

Importance-ranked expert keep — zero training

One shard-streamed CPU pass over the checkpoint. Per MoE layer, score every expert with a calibration-free importance:

Three rungs that fit — and fly

Footprint and headroom — measured on a single RTX 5090 laptop (24 GB), vLLM compiled:

Benchmarks — every workload we measured

All figures measured 2026-07-15 on the v2 MTP-aware carves, single RTX 5090 (24 GB), Blackwell sm120, vLLM compiled. Throughput in tok/s; TTFT = time to first token.

Short 256-token generations undersell aggregate throughput ~40% vs long gen — prefill isn't amortized. See profile 2 for the 1024-token picture.

Quick local inference

The rungs are NVFP4 checkpoints — load one straight from the Hub with transformers and generate:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "kshitijthakkar/Kirigami-Qwen3.6-24B-A3B-NVFP4"   # or 28B / 20B
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="cuda", dtype="auto")

msgs = [{"role": "user", "content": "Summarize this incident trace."}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=256)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

NVFP4 needs a Blackwell GPU (RTX 5090 / RTX Pro 6000 / B200) for native FP4 — the ~15–20 GB of weights fit a single 24 GB card. For production serving, use the vLLM recipe below.

Or, once a rung is up under vLLM (see 06), talk to it over the OpenAI-compatible API — same client you already use for GPT/Claude, just repoint base_url:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
r = client.chat.completions.create(
    model="kirigami-28b",   # = --served-model-name
    messages=[{"role": "user", "content": "Summarize: API p99 jumped 40x after the 14:02 deploy."}],
    max_tokens=512, temperature=0.3,
)
print(r.choices[0].message.content)

Or straight curl — chat, plus streaming with one extra field:

curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "kirigami-28b",
  "messages": [{"role":"user","content":"Explain this incident timeline."}],
  "max_tokens": 512, "temperature": 0.3
}'
# streaming: add  "stream": true

Tip: for direct answers (skip the thinking preamble) pass "chat_template_kwargs": {"enable_thinking": false} in the request body.

Run a rung on your own GPU

Each rung fits a single 24 GB card. Pick a rung, then serve it under vLLM — the recommended per-rung config, then the full copy-paste command.

Full serve command (28B shown — swap repo / name / ctx per the table above):

vllm serve kshitijthakkar/Kirigami-Qwen3.6-28B-A3B-NVFP4 \
  --served-model-name kirigami-28b \
  --kv-cache-dtype fp8 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.94 \
  --max-num-seqs 16 \
  --limit-mm-per-prompt '{"image":0,"video":0}' \
  --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":4}' \
  --port 8000

• Requires a vLLM build with Blackwell (sm120) NVFP4-MoE support — these checkpoints are NVFP4 (Blackwell-native).
• Never set moe_backend=triton — NVFP4 only accepts cutlass / flashinfer; let auto pick.
• --gpu-memory-utilization 0.94 is the safe ceiling (0.95 fails startup). --limit-mm-per-prompt '{"image":0,"video":0}' disables the vision tower and reclaims ~1.6 GB.
• First boot of a new (model, ctx, seqs) combo pays a one-time torch.compile (~25 min); a persistent compile cache makes repeat boots ~4 min. Allow ~3–4 min warmup before sending requests.
• The MTP draft head (--speculative-config) is optional and adds +41–54% decode via speculative decoding (99.5% draft acceptance).
• If KV won't fit on a smaller card, drop --max-num-seqs to 8 or 4.