Feather 0.2B - Chinese ONNX (INT8)

[!NOTE] Feather is a family of compact speech-recognition models for low-latency, on-device transcription. This Chinese-specialized checkpoint is a streaming Conformer-Transducer (RNN-T) with a LiteConformer encoder, exported to ONNX and packaged for onnxruntime-genai.

The encoder is quantized to INT8 weight-only k-quant. The smaller RNN-T decoder and joint network remain FP32. This package is optimized for CPU inference with ONNX Runtime.

LiteConformer

LiteConformer is a CPU-oriented simplification of the standard Nemotron Conformer encoder. Each layer uses one feed-forward block instead of the two Macaron feed-forward blocks, RMSNorm for the transformer sublayers, and GELU instead of Swish/SiLU. The convolution modules retain LayerNorm. Across this model's 20-layer encoder, those changes reduce the number and cost of normalization and projection operations while retaining the streaming Conformer-RNN-T structure.

Model details

Evaluation

The custom ONNX Runtime evaluator used greedy RNN-T decoding on CPU without VAD, processing the Mandarin Chinese FLEURS test split in 560 ms encoder chunks. It computed NeMo mel features for each complete utterance before chunked decoding. These results are not a full-set ONNX Runtime GenAI or Foundry Local benchmark.

Character error rate (CER) is computed after NFKC normalization, lowercasing, punctuation removal, and removal of whitespace for character scoring. It does not normalize spoken Chinese numbers to Arabic digits. CER is used rather than word error rate because Chinese word boundaries are not explicit.

Error breakdown: 11.24% substitutions, 5.33% deletions, and 0.88% insertions (4,007 substitutions, 1,901 deletions, and 314 insertions over 35,655 reference characters). Evaluation completed with zero failed samples.

Example

Input audio: 10.38-second FLEURS Mandarin sample.

Reference: 这 并 不 是 告 别 这 是 一 个 篇 章 的 结 束 也 是 新 篇 章 的 开 始
Prediction: 这并不是告别这是一个篇章的结束也是新篇章的开始
CER:        0.00%

Usage

Install the tested runtime and audio dependencies:

pip install "onnxruntime-genai==0.17.1" numpy soundfile huggingface_hub

Run this self-contained example with a 16 kHz WAV file. It downloads the package and transcribes it on CPU:

import numpy as np
import soundfile as sf
import onnxruntime_genai as og
from huggingface_hub import snapshot_download

model_dir = snapshot_download("onnx-community/feather-0.2b-zh-onnx-int8")
audio, sample_rate = sf.read("chinese-16khz-mono.wav", dtype="float32")
if sample_rate != 16000:
    raise ValueError("Resample the audio to 16 kHz before transcription.")
if audio.ndim == 2:
    audio = audio.mean(axis=1)
audio = np.ascontiguousarray(audio)

model = og.Model(model_dir)
tokenizer = og.Tokenizer(model)
processor = og.StreamingProcessor(model)
processor.set_option("use_vad", "false")
generator = og.Generator(model, og.GeneratorParams(model))
token_stream = tokenizer.create_stream()
pieces = []

def decode_chunk(inputs):
    if inputs is None:
        return
    generator.set_inputs(inputs)
    while not generator.is_done():
        generator.generate_next_token()
        tokens = generator.get_next_tokens()
        if len(tokens):
            pieces.append(token_stream.decode(tokens[0]))

for start in range(0, len(audio), 8960):
    decode_chunk(processor.process(audio[start:start + 8960]))
decode_chunk(processor.flush())
print("".join(pieces).strip())

The model expects 16 kHz mono audio. Its cache shapes are fixed for 560 ms streaming chunks. The GenAI tokenizer can emit a leading space; the example trims outer whitespace. The first five FLEURS test samples were smoke-tested with this runtime and matched the saved evaluator predictions after trimming. The 17.45% full-test CER above is from the separate custom evaluator.

Foundry Local

For integration through Foundry Local's Nemotron Speech ASR support, use a compatible runtime and preserve the model-discovery metadata.

For a manual override, stop/unload the model and back up its cache first. Use the generic CPU Nemotron Speech variant and replace the complete runtime payload, not just the encoder:

• encoder.onnx and encoder.onnx.data
• decoder.onnx and decoder.onnx.data
• joint.onnx and joint.onnx.data
• genai_config.json
• audio_processor_config.json
• tokenizer.model, tokenizer.json, tokenizer_config.json, and vocab.txt
• silero_vad.onnx

Keep Foundry Local's catalog metadata files in place so its local model scanner can still discover the model, then restart the Foundry Local service. Replace the complete payload as a set: Feather uses different model dimensions, tokenizer vocabulary, and streaming cache shapes from the original Nemotron checkpoint. Verify a known Chinese recording before relying on the override.

This is a manual local override rather than a separately registered Foundry Local catalog model, so Foundry Local will continue to display the original Nemotron model identity. Cache refreshes or re-downloads may undo the override.

Files

Limitations

• The model is specialized for Mandarin Chinese; other languages are unsupported.
• It does not restore punctuation or reliably preserve written capitalization.
• Accuracy can degrade for dialects, code-switching, noisy audio, distant microphones, overlapping speakers, or domains unlike the training data.
• Quantization can produce small accuracy differences from the FP32 model.
• The model is optimized for ONNX Runtime CPU execution; other execution providers may run it but are not the intended or validated target.

License

The model package is released under the MIT License.

Acknowledgements

• Architecture implementation: NeMo Conformer-RNN-T with custom LiteConformer modifications.
• Runtime: ONNX Runtime GenAI.
• Voice activity detection: Silero VAD, packaged from onnx-community/silero-vad.

Author

Nenad Banfic, while working in CoreAI at Microsoft.