diodel/Qwen3.5-0.8B-Q4_K_M-GGUF

Quantization

This model is created by quantizing Qwen/Qwen3.5-0.8B to Q4_K_M using convert_hf_to_gguf.py and llama-quantize from llama.cpp.

Target Users

This model supports CPU and heterogeneous (CPU+GPU) inference deployment, making it suitable for users without a GPU.

Usage Steps

1. Download

SDK Download

# Install ModelScope
pip install modelscope

# Download the model with SDK
from modelscope import snapshot_download
model_dir = snapshot_download('diodel/Qwen3.5-0.8B-Q4_K_M-GGUF')

Git Download

# Download the model with Git
git clone https://www.modelscope.cn/diodel/Qwen3.5-0.8B-Q4_K_M-GGUF.git

For more download methods, please refer to the official documentation: https://modelscope.cn/docs/models/download

2. Prepare llama.cpp dependencies

3. Build and Start the Service

# Build and start the service
./llama.cpp/build/bin/llama-server \
    --model ./Qwen3.5-0.8B-Q4_K_M.gguf \
    --ctx-size 2048 \
    --host 0.0.0.0 \
    --port 8000