Whisper inference weights

Inference-oriented conversions of OpenAI Whisper checkpoints. The model architecture and learned parameters come from OpenAI Whisper and the corresponding openai/whisper-* repositories.

This repository uses branches to keep each model at the conventional root path model.safetensors:

Always select a revision. The main branch is an index and does not contain weights.

Each model branch also provides an experimental model-fp8.safetensors. Linear matrices use OCP E4M3 with one FP16 scale per output channel. Token and positional embeddings, convolutions, biases, and normalization parameters remain FP16/FP32. FP8 reduces storage but requires runtime support for the companion *.weight_scale tensors.

Transformation

The conversion is deterministic and does not train or otherwise alter the model:

• Linear, convolution, embedding, and other compute weights are stored as IEEE FP16.
• Positional embeddings and LayerNorm affine parameters remain FP32, matching Whisper's mixed-precision inference behavior.
• Hugging Face Transformers parameter paths are normalized to the concise OpenAI Whisper layout, for example model.decoder.layers.0.self_attn.q_proj.weight becomes decoder.blocks.0.attn.query.weight.
• Tied token embeddings are stored once in FP16 and used for both token lookup and vocabulary projection.
• Non-floating tensors are preserved without conversion.
• Tokenizer, generation, preprocessing, and architecture configuration files are copied from each source revision.

The exact converter is included as convert.py and can be run without PyTorch:

uv run --with numpy --with safetensors python convert.py \
  source/model.safetensors model.safetensors \
  --source openai/whisper-medium@abdf7c39ab9d0397620ccaea8974cc764cd0953e

Add --with ml-dtypes and --compute-dtype float8_e4m3fn to produce the scaled FP8 variant.

On AMD gfx1151, FP8 is emulated as FP16 because the architecture has no native FP8 WMMA path. It therefore saves storage but is slower: Medium took 8.90s versus 4.70s and changed a short phrase; Large V3 took 6.20s versus 2.74s while preserving the tested transcript. Use FP16 weights for speed on gfx1151.

Validation status: Large V3 preserved the tested transcript; Medium remained coherent but was not transcript-identical; Tiny produced repetitive output on the tested Russian clip; Small has only format/structure validation. All FP8 files are therefore marked experimental rather than parity-equivalent.

The mixed storage cuts checkpoint size and prevents runtimes from repeatedly converting a large FP32 vocabulary matrix during autoregressive decoding. FP16 conversion introduces the expected rounding relative to the source FP32 checkpoints.

Because parameter names are normalized, these weights are not a drop-in transformers checkpoint. Consumers must understand the OpenAI-style paths. Svod can load this layout directly.

Loading

With Svod, pass the branch as the Hub revision and use dimensions matching that branch:

let model = Whisper::from_hub(
    "vpermilp/whisper",
    "medium",
    ModelDimensions::for_size(WhisperSize::Medium),
)?;

Select the experimental FP8 file explicitly:

let model = Whisper::from_hub_with_weights(
    "vpermilp/whisper",
    "large-v3",
    "model-fp8.safetensors",
    ModelDimensions::for_size(WhisperSize::LargeV3),
)?;

With the inference example, use --weights model-fp8.safetensors.

For generic safetensors consumers:

from huggingface_hub import hf_hub_download

path = hf_hub_download(
    "vpermilp/whisper",
    "model.safetensors",
    revision="medium",
)

Model

Whisper is a Transformer encoder-decoder model for multilingual speech recognition and speech translation to English. It was trained by OpenAI on 680,000 hours of weakly supervised audio data. See the paper Robust Speech Recognition via Large-Scale Weak Supervision, the original repository, and each source model card for architecture details, evaluation results, intended uses, training data, and complete limitations.

Limitations

These converted weights retain the source models' behavior and limitations. Whisper can hallucinate text not present in the audio, generate repetitions, perform unevenly across languages and accents, and make consequential transcription errors. Evaluate the selected model on the target language, domain, acoustic conditions, and hardware before deployment. Do not use transcription or inferred attributes for high-risk decisions, and do not transcribe people without an appropriate legal basis or consent.

No independent benchmark suite is claimed for these conversions. Results reported by OpenAI apply to the source checkpoints and evaluation procedures; FP16 rounding can produce small output differences.

License and attribution

The source checkpoints are distributed under the Apache License 2.0. This repository preserves that license and attributes OpenAI and the Whisper authors. Model-card content is adapted in part from the OpenAI/Hugging Face Whisper cards.

@article{radford2022robust,
  title={Robust Speech Recognition via Large-Scale Weak Supervision},
  author={Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg and McLeavey, Christine and Sutskever, Ilya},
  journal={arXiv preprint arXiv:2212.04356},
  year={2022}
}