Qwen3.8-Flash-Next-ternary-mlx-K50-16GB

Download and run. This repository contains everything needed to generate: the files of Jakevin/Qwen3.8-Flash-Next-ternary-mlx-K50 that the runner reads (copied unchanged from revision 87268af47d1250cc3e0a4a3d3f701533a680e212), the experts already converted to the native layout (nat/), and the runner. No separate K50 download and no conversion are needed. Total size is about 65 GB.

Quick start

Requires Apple Silicon, macOS 15 or later, Python 3.11, and about 66 GB of free disk space plus room for the Python environment and the system.

python3.11 -m venv k50-env && source k50-env/bin/activate
pip install "omlx @ https://github.com/jundot/omlx/releases/download/v0.7.2/omlx-0.7.2-cp311-cp311-macosx_15_0_universal2.whl" "mlx-ternary @ git+https://github.com/Jakevin/mlx-ternary@e6270bc268b09df8dfa73c96124a4ed73134b731"

Then download and run in one line:

hf download Jakevin/Qwen3.8-Flash-Next-ternary-mlx-K50-16GB --local-dir K50-16GB && python K50-16GB/runner/run.py --max-tokens 256 --prompt "Write a Python function that ..."

runner/run.py uses the repository root as the model directory and nat/ as the native experts. On the first run it checks nat/ against the K50 codes in this repository and creates a stamp file in nat/, so the downloaded directory must be writable; then it starts generation. If the check fails, generation is not started. Later runs use a matching stamp; a malformed or nonmatching existing stamp is never overwritten and generation is refused. The check took about two minutes in earlier runs on the developer's Macs; it has not been timed for this packaged layout on the 16 GB Mac.

Validation status: the author reports an 8-output-token smoke test on a 16 GB Mac for each of two prompts, using run.py with a locally prepared single-directory copy of the model, native files and merged runner. For both prompts, the release and c73 baseline output text matched the text decoded from the corresponding c73 token reference. This is a text-byte comparison, not proof of token-ID equality; it was not a clean-download test and has not been independently verified. The measurements below are from an earlier c73 development runner with separate local weight directories, not a benchmark of this merged release. Generated text goes to stdout; logs and a final tokens/s line go to stderr.

Dependencies (all public; the oMLX app is not needed): oMLX 0.7.2 (Apache-2.0) release wheel, which provides the qwen4_exp model code and pins MLX 0.32.4, mlx-lm (commit 94cdcae1) and mlx-vlm (MIT, commit 97902318); mlx-ternary 0.3.0 (Apache-2.0), commit e6270bc2.

What the runner does

• Native packed experts. The ternary routed experts of K50 were repacked once (with runner/convert_k50_nat.py) from the on-disk base-3 layout to mlx-ternary's native 2-bit layout; the result is nat/ (about 7.7 GB). At run time those experts are read without repacking. The values are unchanged: the converter and verify_nat.py compare every record byte for byte with a repack of the source.
• Batched synchronization. During decode, each selected routed expert still runs as its own matrix product (there is no fused multi-expert kernel), but the runtime synchronizes (mx.eval) once after every four experts instead of after every expert. Prefill uses the original path, which synchronizes after every expert.
• Resident hot experts. Expert records marked hot in the K50 index are loaded, in index order, into a resident cache of --resident-gb gigabytes (default 1.5) before the prompt is processed. Other experts are read from SSD when the router selects them.
• Layer-local prefetch. In the c73 configuration used for the measurements below, four worker threads prefetch selected experts up to two experts ahead within the current layer. The temporary prefetch buffer is limited to 64 MiB. The measured configuration used PF_ON=1, PF_VERIFY=0 and PF_MAX_MB=64. run.py supplies those values only when the variables are absent; values already in the environment take precedence. Invoking generate.py directly does not enable prefetch by itself. This is not prediction of later-layer routes. The 64 MiB cap covers prefetch data buffers, not all process memory or object overhead.

What is in this repository

Why the K50 files are needed: the runner reads the PLE tables (about 30 GB), the non-ternary (affine) experts and dense weights in codes.bin, the scales and biases, the non-expert weights, and the tokenizer from them. codes.bin also still holds the ternary experts in base-3 form (18,753 records, 6,144,983,040 bytes, about 6.1 GB); they are not read during generation but the native-file check compares nat/ with them. K50 files the runner does not read (its own runtime copies, EXPORT.json, PRUNE.json, K50_kept.json, router_k50.safetensors, its README) are not included.

Speed and memory

c73 development measurements on a 16 GB Mac

The following are author measurements from an Apple M4 Mac mini with 16 GB of memory, with K50 and its native files on the internal SSD. The c73 development runner was launched directly with resident expert cache 1.5 GB, prefetch depth 2, four worker threads, PF_ON=1, PF_VERIFY=0, PF_MAX_MB=64, and 8-token prefill chunks. The prompts were run separately with greedy decoding and 32 output tokens: the first comes from prefill, and throughput times the remaining 31 decode steps. Each prompt was measured three times, interleaved with the comparison runner. These results have not been independently verified.

The code prompt asked for a Python palindrome checker, and the essay prompt asked about calm autumn evenings. The generated token sequences matched their respective reference sequences on all six runs; this is evidence for these two prompts only. The comparison used the same model and prompt settings with two prefetch workers. Its paired median speed ratios for four workers over two were 1.072 for code and 1.077 for essay; the smallest of the six ratios was 1.063.

During the six c73 runs, the highest sampled process footprint was between 8.20 GB and 8.37 GB; system swap usage remained 1114.25 MB at the recorded samples. Footprint and swap were sampled approximately once per second, so brief peaks may be missed. The test monitor did not set a runtime memory limit. Speeds depend on the prompt, hardware, page cache, system load and software environment.

Scope: These results were collected with the c73 development runner and separately located local weight directories. The author reports matching 259 model and 144 native files to the repository snapshot's SHA-256 manifest, and the limited 8-token smoke test described above; those reports have not been independently verified. The merged one-directory run.py path has not been benchmarked. The numbers above are not a quality evaluation or a guarantee for other prompts or devices.

The native-file check

generate.py refuses to start unless nat/nat_verified.json matches the files it opens:

• verify_nat.py checks every native record field by field against the K50 index (projection, expert, shape, offsets, byte counts), checks that the records cover every ternary record exactly once, compares every record byte for byte with a repack of the K50 codes, and checks the SHA-256 values recorded by the converter. Each large file is hashed through one open handle whose identity and status are compared before and after hashing and again just before the stamp is written.
• At load, the runtime compares the SHA-256 of the small index files and the identity and status (device, inode, size, modification and change time) of the codes.bin and codes_nat.bin handles it actually opened with the stamp.

After a download, a copy to another disk, or an edit, the stamp can cease to match. If a nonmatching stamp exists, run.py refuses to start instead of overwriting it; use a fresh copy of the model and native files without an old stamp or its temporary files and verify it again. Verification requires a filesystem that supports hard links. It retains the uniquely named temporary stamp file on both success and failure; a successful stamp and that retained file are read-only hard links to the same inode. Do not edit either file. If creating the hard link fails, verification refuses and reports the retained temporary file. The check took about two minutes in earlier runs on the developer's Macs, not measured for this package on the 16 GB Mac. Renaming or moving a directory within the same volume keeps the same files and does not by itself invalidate the stamp. This check catches mistakes such as a wrong source, a partial conversion, or edited files; it does not protect against someone who deliberately rewrites the files, the stamp, and the program. It covers only the expert index.json and codes.bin files and the native files. It does not check the other files (non-expert weights, PLE, scales, biases, tokenizer, configuration). Do not modify any of these files while the runner is running.

python K50-16GB/tests/test_verify_nat.py <new directory> runs the counterexample tests on a small synthetic model. The directory must not exist yet; the test refuses an existing directory, even an empty one.

To check files listed in the manifest, run shasum -a 256 -c SHA256SUMS in the repository directory. The manifest excludes itself and .gitattributes; this command does not detect extra unlisted files or verify the manifest itself.

Memory and resource limits

• generate.py has no watchdog. It does not watch the memory footprint, swap, or disk space of the system.
• runner/runtime.py sets fixed limits: an MLX memory limit of 19 GiB and, at a few checkpoints while the model loads, an abort if resident memory (RSS) plus MLX active memory exceeds 20 GiB. This check does not run during generation and is not the same as the memory footprint. These limits are not derived from the memory of the Mac; on a 16 GB Mac they do not stop the process before macOS starts swapping.
• --resident-gb (default 1.5) is the size of the resident expert cache only, not a limit for the whole process. During the c73 measurements above, the highest sampled process footprint was 8.20–8.37 GB.
• If other programs use much of the memory, macOS may swap and generation becomes slower.

Limitations

• Apple Silicon and MLX only. Greedy decoding only.
• Inherits all limitations of the K50 model.
• About 65 GB to download.
• The c73 development speed and output equality were measured on two prompts of 32 tokens each. The merged release has only the author-reported two-prompt, 8-token text smoke test described above; it has not been benchmarked or independently verified.
• The research, benchmarking, and profiling programs are not included.

License

The model files in this repository (everything except runner/, tests/, and the license and notice files) are derived from Qwen/Qwen3.8-Flash-Next and are under the Qwen Community License 1.0. See LICENSE (the original license file, unchanged) and NOTICE. The files from K50 are unchanged copies; the files in nat/ hold the same ternary codes as K50 in another byte layout.

This model is a derivative work of Qwen/Qwen3.8-Flash-Next. Copyright (c) 2026 Qwen. As required by the license, the copyright notice and permission notice are included in LICENSE. Summary only; the text of LICENSE governs: if this model or a derivative is used in a commercial product or service with more than 100,000,000 monthly active users or US$20,000,000 monthly revenue, the model name must be prominently displayed in that product's user interface. If the licensee or any of its affiliates runs a Model-as-a-Service or AI Work Assistant business, it needs a separate license from Qwen for commercial use, except for qualifying internal use as defined in LICENSE.

The code in runner/ and tests/ is Copyright 2026 Jakevin and licensed under the Apache License 2.0 (runner/LICENSE). Using it with Qwen-derived weights does not change the license of those weights.