Clover-35B-A3B-Preview BF16 GGUF

Clover quantization by IsValorum

Hugging Face | Ko-fi

BF16 GGUF of Diffuspace/Clover-35B-A3B-Preview at revision 4b0402b65e523d0183a14c431955438714cb16ff.

[!WARNING] This file does not load in vanilla llama.cpp. Clover uses the clover_moe architecture, which upstream does not implement. Use the precompiled builds in IsValorum/llama.cpp-clover-moe, which ships Linux x86_64 CPU and CUDA 12 builds plus a Windows x86_64 universal package (NVIDIA CUDA 12.8, Vulkan for AMD and Intel GPUs, and CPU), or compile the runtime from the patched branch (PR #31306, commit 4838ffb0c8ade92f34d9157deea38d92758ab824). The CUDA builds need an NVIDIA GPU; the CPU builds run on any x86-64 machine.

The Linux x86_64 archives require GLIBC_2.38 or newer, so run them on Ubuntu 24.04 or equivalent; the Windows package has no glibc requirement.

[!NOTE] What the prebuilt runtimes cover. Linux x86_64: any CPU, and NVIDIA CUDA 12 for sm_80/86/89/90. Windows x86_64, in a single universal package: NVIDIA CUDA 12.8 from sm_50 through sm_120, AMD and Intel GPUs through Vulkan 1.4.363, and any CPU with per-CPU backends. Not covered: macOS, ARM64, ROCm, and Blackwell on Linux; those need a build from the same branch and commit.

export LD_LIBRARY_PATH=/path/to/llama-clover-bin:$LD_LIBRARY_PATH
llama-cli -m Clover-35B-A3B-Preview-BF16.gguf -ngl 99 -fa on -c 4096 -p "Hello"

Drop -ngl 99 on the CPU build.

A 1.25M I-Matrix will be published separately.