fable-traces-NVFP4

[!WARNING]

This is a quantization of a joke model. The upstream model card explicitly says:

"This is a joke. This is not an actual model."

Do not use this checkpoint for real reasoning, Japanese QA, coding, evaluation,

production, or any task where correctness matters.

NVFP4 W4A4 compressed-tensors quantization of

AliesTaha/fable-traces, a

dense Qwen3 4B causal language model.

This repository is meant only as a Blackwell/vLLM/NVFP4 format experiment. It

preserves the upstream joke behavior and only changes the tensor format. Quick

Japanese reasoning checks failed badly, often drifting into the same unrelated

Egypt/Australia line, so treat the benchmark numbers below as throughput tests,

not evidence of model quality.

Practical Warning

• Not a serious instruction model
• Not reliable for Japanese reasoning
• Not reliable for arithmetic, logic, coding, or factual answers
• Useful only for testing NVFP4 loading, serving, KV-cache behavior, and raw TPS

Quantization

• Source precision: BF16 safetensors
• Output format: compressed-tensors
• NVFP4 format: nvfp4-pack-quantized
• Scheme: NVFP4 W4A4
• Targets: Linear
• Ignored modules: lm_head
• Calibration: neuralmagic/calibration, LLM, 512 samples, max sequence 4096
• Tooling: llmcompressor, compressed-tensors

The resulting model.safetensors is about 2.82 GB.

vLLM Usage

For a 128K context run with FP8 KV cache:

vllm serve sakamakismile/fable-traces-NVFP4 \
  --max-model-len 131072 \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.92

For Python:

from vllm import LLM, SamplingParams

llm = LLM(
    model="sakamakismile/fable-traces-NVFP4",
    max_model_len=131072,
    kv_cache_dtype="fp8",
    gpu_memory_utilization=0.92,
)

messages = [[{"role": "user", "content": "Tell me something interesting."}]]
outputs = llm.chat(messages, SamplingParams(max_tokens=128, temperature=0.0))
print(outputs[0].outputs[0].text)

Benchmark

Measured locally with vLLM 0.21.0+cu129 on NVIDIA RTX PRO 2000 Blackwell

16 GB GPUs, driver 595.71.05.

Common settings:

• max_model_len=131072
• kv_cache_dtype=fp8
• max_num_seqs=8
• max_tokens=256
• repeats=3, warmup=1
• enforce_eager=True

TP=1

vLLM reported 156,576 GPU KV-cache tokens with FP8 KV cache.

| TP | Parallel | Mean output tok/s | Best output tok/s | Mean total tok/s |

|---:|---:|---:|---:|---:|

| 1 | 1 | 69.03 | 71.45 | 118.07 |

| 1 | 2 | 129.30 | 131.40 | 219.80 |

| 1 | 4 | 238.04 | 244.81 | 404.32 |

| 1 | 8 | 449.17 | 461.41 | 770.48 |

TP=2

vLLM reported 359,488 GPU KV-cache tokens with FP8 KV cache.

| TP | Parallel | Mean output tok/s | Best output tok/s | Mean total tok/s |

|---:|---:|---:|---:|---:|

| 2 | 1 | 60.37 | 60.72 | 103.26 |

| 2 | 2 | 100.86 | 122.85 | 166.75 |

| 2 | 4 | 234.43 | 236.21 | 388.02 |

| 2 | 8 | 360.84 | 415.11 | 613.43 |

For this short-output benchmark, TP=1 is faster; TP=2 mainly buys more KV-cache

headroom for long-context use.

Raw benchmark files are included under benchmarks/.