modelopt NVFP4 quantized GLM-4.6
Tested (but not extensively validated) on 4x RTX Pro 6000 Blackwell via:
inference:
image: vllm/vllm-openai:nightly
container_name: inference
ports:
- "0.0.0.0:8000:8000"
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: -1 # use all GPUs
capabilities: [gpu]
environment:
- NVIDIA_VISIBLE_DEVICES=all
- NVIDIA_DRIVER_CAPABILITIES=compute,utility
- NCCL_IB_DISABLE=1
- NCCL_NVLS_ENABLE=0
- NCCL_P2P_DISABLE=0
- NCCL_SHM_DISABLE=0
- VLLM_USE_V1=1
- VLLM_USE_FLASHINFER_MOE_FP4=1
- OMP_NUM_THREADS=8
volumes:
- /models/GLM-4.6-NVFP4-4:/GLM-4.6:ro
command:
- /GLM-4.6
- --enable-expert-parallel
- --enable-prefix-caching
- --enable-chunked-prefill
- --served-model-name
- "GLM-4.6"
- --tensor-parallel-size
- "4"
- --gpu-memory-utilization
- "0.95"
- --max-num-batched-tokens
- "16384"
- --dtype
- "auto"
- --max-num-seqs
- "8"
- --kv-cache-dtype
- fp8
- --enable-auto-tool-choice
- --tool-call-parser
- glm45
- --host
- "0.0.0.0"
- --port
- "8000"```
model-00020-of-00041.safetensors 5.0 GBmodel-00016-of-00041.safetensors 5.0 GBmodel-00024-of-00041.safetensors 5.0 GBmodel-00012-of-00041.safetensors 5.0 GBmodel-00028-of-00041.safetensors 5.0 GBmodel-00032-of-00041.safetensors 5.0 GBmodel-00008-of-00041.safetensors 5.0 GBmodel-00004-of-00041.safetensors 5.0 GBmodel-00001-of-00041.safetensors 5.0 GBmodel-00036-of-00041.safetensors 5.0 GBmodel-00015-of-00041.safetensors 5.0 GBmodel-00019-of-00041.safetensors 5.0 GBmodel-00023-of-00041.safetensors 5.0 GBmodel-00027-of-00041.safetensors 5.0 GBmodel-00031-of-00041.safetensors 5.0 GBmodel-00011-of-00041.safetensors 5.0 GBmodel-00007-of-00041.safetensors 5.0 GBmodel-00038-of-00041.safetensors 5.0 GBmodel-00034-of-00041.safetensors 5.0 GBmodel-00030-of-00041.safetensors 5.0 GBmodel-00026-of-00041.safetensors 5.0 GBmodel-00022-of-00041.safetensors 5.0 GBmodel-00018-of-00041.safetensors 5.0 GBmodel-00014-of-00041.safetensors 5.0 GBmodel-00006-of-00041.safetensors 5.0 GBmodel-00010-of-00041.safetensors 5.0 GBmodel-00009-of-00041.safetensors 5.0 GBmodel-00013-of-00041.safetensors 5.0 GBmodel-00017-of-00041.safetensors 5.0 GBmodel-00021-of-00041.safetensors 5.0 GBmodel-00025-of-00041.safetensors 5.0 GBmodel-00029-of-00041.safetensors 5.0 GBmodel-00033-of-00041.safetensors 5.0 GBmodel-00037-of-00041.safetensors 5.0 GBmodel-00005-of-00041.safetensors 5.0 GBmodel-00003-of-00041.safetensors 5.0 GBmodel-00002-of-00041.safetensors 5.0 GBmodel-00039-of-00041.safetensors 5.0 GBmodel-00035-of-00041.safetensors 5.0 GBmodel-00040-of-00041.safetensors 4.4 GBmodel-00041-of-00041.safetensors 1.6 GBHave the files? Submit a torrent.
Submit a torrentHave the complete files and permission to share them? Replace the example folder path. The script checks the files, makes a torrent and submits its magnet and checksums. Keep your torrent client seeding. Get your community key from your settings, and read the script before running it.
curl -fsSL https://pirateface.co/package.sh | bash -s -- --repo lukealonso/GLM-4.6-NVFP4 ./model-folderzai-org/GLM-4.6