Audio8 compressed: complete mixed Q3/Q4 streaming model

Complete weights and runnable CPU, Apple MLX, CUDA and Vulkan consumers. Experimental: strict one-second latency and universal real-time performance are not established. This is not ternary.

The CPU and MLX model is 1.882 GB on disk, including the encoder, decoder, projector, tied embedding/output weights and original small BF16 tensors. All 108 decoder MLP matrices use full eight-level, group-64 Q3 GPTQ; other packed matrices retain the previous calibrated Q4 weights. It saves 304,349,184 bytes relative to the earlier Q4 payload. CPU and MLX read the same file without expanding entire matrices to floating-point weights.

The CUDA/Vulkan GGUF stores these exact same reconstructed weights in Q4_0, at 2.330 GB. It is not a three-bit GGUF. Every one of its 4,075,552,768 packed float32 weight values was independently checked for bitwise equality with the native model; original BF16 tensor bytes also match. Use the included Audio8-specific GGML runtime: stock llama.cpp does not implement this architecture.

Fresh public downloads of both full model files and the shipped launchers passed CPU and MLX English/Chinese streaming checks, CUDA on RTX 5080 Laptop, and Vulkan on NVIDIA and AMD. See post-publication verification. These checks bind the immutable release commit c49eca996a633fa64c0ec326436ce2cd131800dc; those checks apply to that revision. The current Mac launcher uses the separately qualified batch-2 profile described below; model bytes remain unchanged; the current runtime update is described below.

Current runtime memory update

A fresh public HTTP download of the complete model and current runtime passed English and Chinese streaming-stdin checks, with exact token parity to the preceding public runtime: consumer verification, pinned to fddfb1086894106166b4d4a525ca93458f143643.

The Mac runtime now materializes encoder cache rebases one layer at a time. All tokens and window schedules matched the previous runtime on all 11 previously evaluated recordings; this is a runtime regression check, not a new held-out accuracy result. Weights and inference settings are unchanged.

Current mature-cache work/audio ratio: 0.863–0.864. These memory measurements are separate and must not be added. Serial measurements are not a controlled speed comparison. The soft memory target remains exceeded. Complete regression and unchanged error counts.

Long CUDA and Vulkan streaming checks

Both shipped GPU launchers were checked on the same complete, previously evaluated 319.715-second meeting with actual 20 ms paced stdin and the frozen causal silence boundaries. These are exposed-case backend checks, not a new held-out qualification. Hardware: RTX 5080 Laptop GPU.

The original BF16 CUDA control had 171/478 word errors on this recording; the published MLX model had 179/478. GPU kernel differences can change decoder decisions even with identical reconstructed weights. The original global text-alignment reports retain their late cross-session matches; the supplementary timing audit cannot turn omissions or misrecognitions into latency successes. Peak device memory was not measured. The existing AMD Vulkan smoke remains slower than real time, and the GPU binaries retain their narrow GLIBC 2.43 compatibility. CUDA evidence, Vulkan evidence.

Earlier batch-2 accuracy qualification

The preceding batch-2 runtime passed English and Chinese streamed-stdin checks from a fresh public download, including the complete model file and hash verification: consumer report, pinned to commit 18c3e32a29402802af4c2053d9928d3ca61b1101. That verification predates the cache-rebase runtime change.

The Mac launcher now uses batch 2, selected on a previously exposed recording before a separate frozen 11-recording evaluation (471.630 seconds). This is a different corpus from the earlier batch-4 table, so the two tables are not a paired batch-speed comparison. The original CUDA BF16 model used the same full references and causal silence boundaries.

The timed meeting contains 327 correctly matched words, including 1 ambiguous matches; 0 matched words exceeded one second. Omissions and substitutions remain errors. This supports measured latency on these matched words, not a guarantee that every spoken word appears within one second. Memory remains above the soft target; per-case RSS and footprint are in the complete batch-2 report. The selection record retains the exposed batch comparisons and cross-reset alignment caveat. No model weights were changed.

Earlier batch-4 and CPU qualification

The candidate weights, runtime sources, profiles, original-model control and input identities were frozen before running all 11 reserved clips: 470.249 seconds, comprising three English recordings and eight short Chinese clips. No cases were dropped. The original CUDA BF16 model used identical frozen silence boundaries and full references. These are practical cross-backend comparisons, not an isolated weight-only experiment.

Memory values use decimal GB. RSS and macOS footprint are different measurements and must not be added. The soft 1–2 GB RAM target is narrowly exceeded on CPU and more substantially on Mac; file size is not RAM usage. Mature-cache work includes trims and excludes EOF padding; it is measured over eligible speech windows after the cache fills, not the elapsed duration of a paced recording. CPU uses eight physical cores and AVX512 BF16 on the tested Ryzen. No real-time claim extends to other CPUs or the AMD Vulkan route.

Word-end labels exist for the AMI recording, not every reserved clip. Native timing covers 423 matched reference words, including 13 ambiguous matches; MLX covers 422, also including 13 ambiguous matches. Native had one match above one second, with ambiguous alignment; no unambiguous match exceeded one second (maximum 0.770 s). MLX had three matches above one second, including two unambiguous matches (maximum 1.101 s). Missing/substituted words remain errors, not latency successes. Annotation and overlapping-speaker uncertainty remain. A universal maximum of one second after each spoken word is not proven.

The supplied GLIBC 2.35 CPU executable separately matched the qualified host build's English and Chinese token IDs, winning-logit bits, and clocks exactly, both with fresh prefixes and restored prefixes. Full reserved timing used the pinned host build; the portability check is not a second full performance evaluation.

See the complete paired report and the prospective freeze. Per-case raw events, host receipts, original-model results and boundary evidence are included. The smaller training and exposed diagnostic reports are also preserved, clearly separate from the reserved evaluation. Evaluation audio is not redistributed here. Upstream pretraining overlap is unknown.

Download only the format you need

The launchers additionally need their runtime files and assets/. MLX also needs the small JSON assets in model/. A full repository download contains both alternative weight formats; they are not loaded together. The previous Q4 release remains available at its immutable revision.

Audio input and streaming captions

Input is mono 16 kHz float32 little-endian PCM. Convert audio with a separately installed FFmpeg:

ffmpeg -i input.wav -f f32le -acodec pcm_f32le -ar 16000 -ac 1 audio.f32le

Every launcher accepts - instead of a filename for streaming stdin. Append the text field of flushed caption_text_delta JSON lines. A live producer should wait for caption_stream_ready before starting its audio clock. Model loading and optional projection warmup occur before readiness; they are not included in warmed caption delay. An unpaced file run is not a live latency measurement.

Silero VAD runs continuously in 512-sample frames. After two seconds of silence it resets the ASR state, retaining 400 ms of preroll; the VAD state itself continues. Only global EOF receives synthetic drain frames. All reference words, including speech missed by this gate, remain in accuracy scoring. This is streaming with silence-driven resets, not a guarantee of arbitrary-duration accuracy.

Linux CPU

Requires Linux x86_64, GLIBC 2.35 or newer, and Python 3 for physical-core selection. BF16 arithmetic and the measured fast path require suitable CPU instructions; other hardware and fallbacks are not performance-qualified. Prepare an audio-independent prefix once on the destination machine:

chmod +x bin/linux-x86_64/audio8-stream
sh ./run-cpu-linux-x86_64.sh --prepare-prefix en ./prefix-en.a8pfx
sh ./run-cpu-linux-x86_64.sh audio.f32le en ./prefix-en.a8pfx

Use zh and its own prefix for Chinese. Prefixes are bound to the binary, model, frontend, CPU/OS, language and execution profile. Recreate them after any identity change. The two-argument run without a prefix also works but computes the conditioning prefix at speech start. Prefix preparation is separate from audio capture and never reads microphone data.

The launcher selects eight distinct physical cores within its allowed mask. It preserves the existing mask if that topology is unavailable, reporting the fallback. It uses BF16 activations/cache, RT18/240 ms, batch 1, eight workers, four tasks and tiled encoder attention. Use the launcher: the raw binary's legacy A8 default is not the validated arithmetic profile.

Build sources, Cargo locks and vendored dependencies are in sources/native-source.tar.gz:

mkdir native-source
 tar -xzf sources/native-source.tar.gz -C native-source
cargo build --manifest-path native-source/native/Cargo.toml --locked --release --features x86-vnni,silero --bin audio8-stream

Apple Silicon MLX

Tested on Apple M2, macOS 26.5.1, Python 3.12 and MLX 0.32.2:

python3.12 -m venv .venv
. .venv/bin/activate
python -m pip install -r requirements-mlx.txt
sh ./run-mlx.sh audio.f32le en

The selected profile is compiled float16 compute, BF16 cache, fused projections, projection warmup and batch 2 after the first batch 1. The 4 GiB allocator limit is not a process-memory guarantee. MLX is Apple-specific; the GGUF GPU route provides CUDA/Vulkan separately.

CUDA and Vulkan GGUF

These Linux x86_64 binaries are narrower than the CPU package: GLIBC 2.43, matching system GPU libraries and drivers are required. CUDA is built with CUDA 13.3 for SM120a and tested on RTX 5080 Laptop only. Other NVIDIA architectures are not qualified. Vulkan was exercised on that NVIDIA GPU and AMD Radeon 890M with explicit devices; AMD was slower than real time on the smoke clip.

chmod +x gguf/bin/linux-x86_64-cuda/audio8-gguf
chmod +x gguf/bin/linux-x86_64-vulkan/audio8-gguf
sh ./run-cuda.sh audio.f32le en 0
sh ./run-vulkan.sh audio.f32le en 0

The CUDA 13.3 runtime libraries must be discoverable, for example through the system loader or LD_LIBRARY_PATH=/usr/local/cuda-13.3/lib64. Proprietary drivers are not bundled. The executable resolves the adjacent libaudio8-bridge.so. Device selection is explicit and no silent CPU ASR fallback is allowed. Floating-activation sentinels and backend counters verify the GPU path; no activation quantization is silently substituted.

All three routes completed the same exposed 5.855-second English clip via actual 20 ms stdin chunks, producing the complete same transcription. This is a functionality/packaging check, not a long-stream GPU quality or maximum-latency qualification. CPU process RSS excludes device allocation; these GPU smokes did not measure peak VRAM. The exact GGUF conversion preserves weights, but different kernels can still produce different decoder decisions. CUDA/Vulkan source envelopes and inherited backend build evidence are included under sources/ and provenance/.

Provenance and integrity

Source: Edge0/Audio8-ASR-Infinite, revision b4413de154ed6bdef0a4011028b1ebd12aca8152, Apache 2.0. The Q3 MLP fit uses full-correlation GPTQ with group-64 FP16 scales, eight codes representing signed values -4 through 3, and original column order. Calibration used only the declared training clips and RT18/240 ms clocks. The encoder remains Q4; there was no claim that an untrained ternary conversion preserves accuracy.

execution_profile.json records runtime bindings. release_manifest.json and SHA256SUMS identify shipped bytes. Historical export records retain their export-time accuracy_validated: false and production_ready: false; later measured results are in the evaluation reports rather than retroactively rewriting those records.

shasum -a 256 -c SHA256SUMS

Known limitations: residual accuracy loss, speech omissions, some late captions, CPU/MLX memory above the soft target, slow AMD Vulkan, narrow prebuilt GPU compatibility, and no broad multilingual or arbitrary-hardware qualification. The release is complete and usable on the tested routes, but these remaining requirements are not represented as passed.