ELYZA-Thinking-1.0-llm-jp-4-32b-a3b-GGUF

GGUF quantizations of elyza/ELYZA-Thinking-1.0-llm-jp-4-32b-a3b.

[!IMPORTANT] Stock llama.cpp cannot chat with this model yet: /v1/chat/completions fails. Apply scripts/llmjp4-space-after-control.patch to llama.cpp and rebuild. See NOTES.md for why.

[!TIP] The model is sensitive to long runs of blank lines in the prompt (for example a raw terminal screen). If you pass such text, collapse runs of 3+ newlines first. See NOTES.md.

Files

Quantized with an importance matrix (imatrix.gguf). KLD is the mean KL divergence from BF16 on Japanese and English Wikipedia text; lower is closer to the original. -dyn files keep attention and output tensors at higher precision and spend the low bits on the MoE experts.

Usage

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git apply /path/to/llmjp4-space-after-control.patch
cmake -B build -DGGML_VULKAN=ON   # or -DGGML_CUDA=ON, etc.
cmake --build build -j --target llama-server

./build/bin/llama-server -m ELYZA-Thinking-1.0-llm-jp-4-32b-a3b-IQ4_XS-dyn.gguf \
  --jinja -ngl 99 -c 65536 -fa on --temp 0.6 --top-p 0.95

The patch was tested on llama.cpp 4ebdf2c. Recommended sampling is the same as the original model: temperature=0.6, top_p=0.95.

License and credits

Apache License 2.0, the same as the original model. LICENSE and NOTICE are included.

• Original model: elyza/ELYZA-Thinking-1.0-llm-jp-4-32b-a3b (ELYZA, Inc.)
• Base model: llm-jp/llm-jp-4-32b-a3b-base (LLM-jp)
• imatrix data: TFMC/imatrix-dataset-for-japanese-llm (ODC-BY) and bartowski's calibration_datav3