Hemmingway-1 heretic with MTP — native FP8

This checkpoint merges arelath/Hemmingway-1-heretic-adapter into Altworld/Hemmingway-1, then quantizes the merged BF16 weights to FP8 E4M3FN with 128 × 128 block scales. It uses native Hugging Face Safetensors and quant_method: fp8, not GGUF.

The exact source revisions match the provenance of arelath/Hemmingway-1-heretic-GGUF/Hemmingway-1-heretic-MTP-Q8_0.gguf:

• Base: 777059f912d4615fb06efca94a0d2d26f6641c88
• Adapter: d13a9e28e6e09526bf539db44bac62468db1b2a6

The conversion starts from those original weights, not from the quantized GGUF. FP8 and Q8_0 are different quantizers, so outputs need not match exactly. All 128 adapter target matrices are merged before quantization. All 15 original MTP tensors are included; MTP attention/MLP projections use FP8, while its fusion projection and normalization weights retain their original precision.

Embeddings, output head, normalization, convolution and the small GDN a/b projections retain original precision. The quantization layout follows Qwen's official FP8 release. This is the text-only model, without vision weights.

vLLM

Use a recent vLLM release supporting Qwen3_5ForCausalLM and block FP8 (the user's deployment reports vLLM 0.28.0). Use an FP8-capable GPU with sufficient memory for the weights plus cache and execution buffers. The local conversion machine cannot hold the full model in GPU memory, so full-model vLLM generation has not been validated for this artifact. Structural and numerical conversion checks are recorded in validation.json and provenance.json; these do not constitute a quality benchmark.

vllm serve arelath/Hemmingway-1-heretic-MTP-FP8 \
  --served-model-name hemmingway-1 \
  --quantization fp8 \
  --dtype bfloat16 \
  --max-model-len 8192 \
  --max-num-seqs 4

For RunPod's vLLM worker, set MODEL_NAME to this repository and use the Safetensors/auto loader. Remove previous GGUF-specific overrides and stale revision pins from the other repository. Start with a modest context and batch size; the model's advertised maximum context is not a memory-sizing default.

MTP is included but is only used when speculative decoding is enabled. After confirming ordinary generation, add:

--speculative-config '{"method":"mtp","num_speculative_tokens":1}'

MTP execution is also untested for this artifact and requires additional memory.

Reproduction

publish_native_fp8.py processes one source shard at a time without writing a second full merged checkpoint. Install PyTorch with CUDA support, safetensors, and huggingface_hub. The conversion run used PyTorch 2.14.0+cu126; exact source revisions and conversion details are recorded in provenance.json. Place this README beside the script as FP8-MODEL-CARD.md, then run:

python publish_native_fp8.py

Publishing requires HF_TOKEN and --upload; the script's destination is the repository named above. It does not provision or alter a serving endpoint.