Experimental Sushi-format quantization made directly from the BF16 weights of
orcarouter/Qwen3.8-Flash-Next-Uncensored,
revision e096800036ec20da7e2442dcd4044a004d4e99fa.
The abliteration comes from that upstream checkpoint; this conversion adds no
further abliteration or fine-tuning. A community refusal/KLD measurement is
summarized below.
ngram_table.bin.151539c77abc7ab7425d30da7a4e8e3c5c154e7b, with an isolated MCG-window15 encoder
adaptation and parallel dispatch for quantized experts alongside BF16 linears.This is a Sushi pack, not a general ExLlamaV3 or Transformers checkpoint. 2bpw
refers to the routed-expert code rate, not every tensor or the total file-size
average. No lossless/BF16-equivalent quality claim is made. The private Sashimi
quantizer and its calibration recipe are not reproduced.
The complete download is about 64.87 GiB (including the 29.8 GiB disk-backed
n-gram sidecar). Keep every shard, tokenizer file and ngram_table.bin together.
Download with the Hugging Face CLI:
hf download sanasol2008/Qwen3.8-Flash-Next-Abliterated-Sushi-2bpw \
--local-dir ~/.sushi/models/Qwen3.8-Flash-Next-Abliterated-Sushi-2bpw
Use an official Sushi release, 1.2.0 or newer. Since 1.2.0, Sushi supports
Qwen vision together with expert streaming (--ssd-budget-gb) — enable it with
--vision. No patched build is needed any more.
Tested setups:
| Machine | Sushi | Mode | Result |
|---|---|---|---|
| M3 Max / 48 GiB (author) | 1.2.1 (stock) | 22 GiB streaming, --vision, --no-mtp |
text, image OCR, tool calls OK |
| M1 Max / 32 GiB (community report) | 1.2.4 (stock) | 23 GiB streaming, --no-vision, --no-mtp, 100k ctx |
text, Japanese, tool calls OK; ~19–21 tok/s warm |
The author's serving configuration (48 GiB Mac):
sushi serve --model ~/.sushi/models/Qwen3.8-Flash-Next-Abliterated-Sushi-2bpw \
--host 127.0.0.1 --port 12345 --ssd-budget-gb 22 --no-mtp --vision \
--kv-quant 8 --ctx-size 262144 --prefill-chunk 2048 --metrics \
--max-tokens 8192 --prefix-cache-entries 2 --prefix-cache-mem 1GB \
--prefix-cache-disk 10GB --temp 1
The 256k context is a configured ceiling, not a measured full-context memory or
quality guarantee. Memory usage grows with context and image inputs. On a 32 GiB Mac lower
--ssd-budget-gb and --ctx-size (see the table above). Sushi ≥ 1.2.0 also
supports MTP with streaming, but MTP and video inference were not validated on this pack.
The measurements below were taken on Sushi 1.1.1 with the (now superseded) vision-streaming patch; current releases were not re-benchmarked here. Controlled text comparison used sequential fresh servers, 22 GiB streaming, KV8, 16k context / prefill chunk512, prefix-cache entries0, temperature0, thinking off, the same 40-token prompt, and 256 generated tokens. No downloads or checksum jobs ran during measurement.
| Pack | Cold decode | Warm decode, mean of runs2–3 |
|---|---|---|
| Original Sushi 2bpw | 22.89 tok/s | 31.60 tok/s |
| This BF16-derived pack | 16.78 tok/s | 26.48 tok/s |
These are short-prompt expert-cache measurements, not representative long-context
agent throughput. On the final serving settings, arithmetic and Russian text
checks passed; an image OCR check correctly transcribed three fields including
all numbers (915 prompt tokens, 9.91s total request time). A tool-call check
correctly returned get_weather with city=Belgrade.
These smoke checks establish basic functionality, not model-wide quality, BF16 equivalence, or a measured refusal rate.
These are consecutive real agent requests captured from the local server log on the same M3 Max / 48 GiB, before the later diagnostic test run. Settings: 22 GiB streaming budget, KV8, 256k configured context, prefill chunk 2048, thinking enabled, temperature 1, prefix cache 2 entries / 1 GiB RAM / 10 GiB disk. They include tool results and a growing conversation, rather than repeating a small benchmark prompt.
| Total input tokens | Cached input tokens | Output tokens | Prefill tok/s | Decode tok/s | End reason |
|---|---|---|---|---|---|
| 39,652 | 39,252 | 417 | 83.7 | 10.9 | tool_calls |
| 40,269 | 39,621 | 704 | 107.8 | 11.8 | tool_calls |
| 40,638 | 40,238 | 360 | 80.0 | 11.4 | tool_calls |
| 41,338 | 40,607 | 320 | 84.0 | 10.6 | tool_calls |
| 41,723 | 41,307 | 192 | 78.7 | 11.0 | tool_calls |
| 47,393 | 41,692 | 542 | 218.9 | 13.2 | client_disconnect |
Prefill rates are the server-reported rates with prefix reuse; they are not uncached processing rates for the full 40–47k context. Observed decode was 10.6–13.2 tok/s here, substantially below the short warm benchmark's 26.48 tok/s. The last row is an interrupted request, not a completed reply or a server-crash record. These observations are not a controlled long-context comparison with the original model. Private conversation content is not published.
Additional functional diagnostics: identical 12-case, thinking-off checks passed 8/12 for this pack and 6/12 for the original Sushi pack. A separate six-case follow-up on this pack passed 6/6 with thinking enabled and two prompts clarified. The original failures remain part of the result; the follow-up changed conditions. Checks included exact tool arguments, retrieval from 15,336-token inputs, and two Python functions each passing 100 randomized cases (interval merging passed on the clarified follow-up). This small hand-authored set is not a standardized benchmark or a BF16 fidelity measurement.
From discussion #1 (M1 Max / 32 GiB, stock Sushi 1.2.4, thinking off, temperature 0, final answer only, keyword-based refusal classifier, 220 prompts per pack):
| Pack | Harmful refusal ↓ (80, JA) | Benign false refusal ↓ (60, JA) | Over-refusal ↓ (80, EN, OR-Bench) |
|---|---|---|---|
| Original Sushi 2bpw | 65.0% | 13.3% | 7.5% |
| This pack | 5.0% | 1.7% | 0.0% |
Residual refusals were mostly copyright requests. sushi kld against the original
Sushi 2bpw pack (52 benign prompts × 32 greedy tokens): mean KLD 0.232, top-1
agreement 0.839, cosine 0.930. That KLD includes both abliteration and the
different quantization source, so it is not abliteration drift alone. Refusal
labels are a keyword heuristic, not an LLM judge, and refusal rate is not a
measure of output harm.
Before Sushi 1.2.0, Qwen vision was disabled under expert streaming. A local patch
(7e8684f,
earlier 1671383)
was used for the original tests and proposed in
beamivalice/sushi#18; upstream then
added official support. Use a current Sushi release instead of the patch.
The upstream LICENSE is included verbatim: Qwen Community License1.0. The upstream
model card labels Apache2.0, but its actual bundled license differs; this repository
preserves the bundled license rather than asserting a new Apache license.
See LICENSE for the complete terms. Credit to Qwen for the base model, OrcaRouter
for the BF16 derivative, turboderp-org for ExLlamaV3, and beamivalice for Sushi.
ngram_table.bin 32 GBtrunk.safetensors 5.3 GBvision.safetensors 898 MBexperts-10.safetensors 639 MBexperts-11.safetensors 639 MBexperts-12.safetensors 639 MBexperts-13.safetensors 639 MBexperts-14.safetensors 639 MBexperts-15.safetensors 639 MBexperts-16.safetensors 639 MBexperts-17.safetensors 639 MBexperts-18.safetensors 639 MBexperts-19.safetensors 639 MBexperts-20.safetensors 639 MBexperts-21.safetensors 639 MBexperts-22.safetensors 639 MBexperts-23.safetensors 639 MBexperts-24.safetensors 639 MBexperts-25.safetensors 639 MBexperts-26.safetensors 639 MBexperts-27.safetensors 639 MBexperts-28.safetensors 639 MBexperts-29.safetensors 639 MBexperts-30.safetensors 639 MBexperts-31.safetensors 639 MBexperts-32.safetensors 639 MBexperts-33.safetensors 639 MBexperts-34.safetensors 639 MBexperts-35.safetensors 639 MBexperts-36.safetensors 639 MBexperts-37.safetensors 639 MBexperts-38.safetensors 639 MBexperts-39.safetensors 639 MBexperts-40.safetensors 639 MBexperts-41.safetensors 639 MBexperts-42.safetensors 639 MBexperts-43.safetensors 639 MBexperts-44.safetensors 639 MBexperts-45.safetensors 639 MBexperts-46.safetensors 639 MBexperts-47.safetensors 639 MBexperts-0.safetensors 639 MBexperts-1.safetensors 639 MBexperts-2.safetensors 639 MBexperts-3.safetensors 639 MBexperts-4.safetensors 639 MBexperts-5.safetensors 639 MBexperts-6.safetensors 639 MBexperts-7.safetensors 639 MBexperts-8.safetensors 639 MBexperts-9.safetensors 639 MB有文件?提交种子。
提交种子orcarouter/Qwen3.8-Flash-Next-Uncensored