Qwen3.8-Flash-Next ternary experts

Ternary routed-expert overlay for Qwen3.8-Flash-Next. Hadamard-rotated ternary packs (ternary-g128-hadamard-gptq, 48 layers). Attention, router, shared experts, and MTP stay in the base checkpoint (not re-uploaded here).

Drafter

Default: NEXTN with base MTP (--speculative-num-steps 3, --speculative-num-draft-tokens 4).

Serve: single GPU, concurrency 1

git clone https://github.com/meshapplied/goethe
cd goethe
pip install -e ./ternary-kernels

import ternary_runtime
m = ternary_runtime.fetch("meshapplied/Qwen3.8-Flash-Next-Ternary-Latest")
ternary_runtime.bootstrap()

CUDA_VISIBLE_DEVICES=0 python -m sglang.launch_server \
  --model-path "$TERNARY_MODEL_DIR" \
  --tp-size 1 \
  --dtype bfloat16 \
  --trust-remote-code \
  --kv-cache-dtype fp8_e4m3 \
  --speculative-algorithm NEXTN \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --speculative-token-map /path/to/hot_self_32k.pt \
  --mem-fraction-static 0.90 \
  --context-length 16384 \
  --max-running-requests 1 \
  --chunked-prefill-size 4096 \
  --page-size 64 \
  --cuda-graph-backend-decode full \
  --cuda-graph-backend-prefill disabled \
  --cuda-graph-max-bs-decode 1 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen \
  --served-model-name qwen38-flash-next-ternary \
  --host 127.0.0.1 --port 31003

TERNARY_PACK_DIR / TERNARY_PROFILE=flash_next must be set (via fetch / bootstrap). Example TOML: Goethe examples/ternary-flash-next.toml.

Serve: single GPU, concurrency 4

Same command with:

  --max-running-requests 4 \
  --cuda-graph-max-bs-decode 4

Performance

Hardware unless noted: single RTX PRO 6000 96 GB, TP1, Goethe ternary_runtime + SGLang, FP8 KV, NEXTN 3/4 + hot vocab, T=0. Expert overlay: meshapplied/Qwen3.8-Flash-Next-Ternary-Latest on base Qwen/Qwen3.8-Flash-Next.

VRAM

Serve knobs for this snapshot: mem 0.90, chunk 4096, context 65536, max_running 1, --disable-radix-cache.

Agentic coding throughput

Replay of 10 SWE-bench Pro / DeepSWE mini-swe-agent conversations × 6 turns each. Full chat history + tool observations fed back each turn; 256 output tokens/turn; prefix cache on; hot32; NEXTN 3/4. Headline concurrency-1 decode rate: 168 tok/s.

Random prompts

sglang.benchmark.serving backend sglang-oai-chat, 32 prompts, in=512, out=256, radix cache on, hot32.

Cold prefill and long-context decode

Same GPU/stack with --disable-radix-cache and --flush-cache between lengths (no hot map on this cold boot). sglang.benchmark.serving random, 8 prompts, out=256, concurrency 1. Prefill tok/s ≈ prompt_len / mean_TTFT.

Decode slows as context grows; ITL stays flat; TTFT scales with prompt length.

Speculative decoding

Format

• ternary_config.json: profile flash_next, gate_up_down layout, 48 layers
• layer-00.safetensors … layer-47.safetensors
• manifest-layer-*.json, eval.json
• Hadamard-rotated ternary GPTQ, group 128. Fused gate_up_proj / down_proj.

License

• This pack / card: Apache-2.0
• Base model: Qwen license / community terms on the base repo