中文Sign up

alphakek/GLM-4.7-Flash-heretic-NVFP4

🤗 Hugging Face sourcetext-generationmit17.1B params18 GBsafetensors✓ 2 checksumsupdated today
Submit in one command

Run it next to your model folder. It makes the torrent, checks your files against Hugging Face, and submits it. You just start seeding and paste your key from your account. It only reads your files and never changes them. Read the script first if you like.

curl -fsSL https://pirateface.co/package.sh | bash -s -- --repo alphakek/GLM-4.7-Flash-heretic-NVFP4 ./model-folder
Needs a seeder →

GLM-4.7-Flash-heretic NVFP4

NVFP4 post-training quantization of Olafangensan/GLM-4.7-Flash-heretic for long-context multi-GPU inference with vLLM.

[!IMPORTANT] The Hugging Face UI "Model size" badge is auto-inferred from packed NVFP4 safetensors and may show an incorrect parameter count for this repo. Use the architecture statement below as source of truth: GLM-4.7-Flash (30B-A3B MoE).

This release uses NVFP4 (4-bit) quantization, not 8-bit quantization.

Model Size

  • Base architecture: GLM-4.7-Flash (30B-A3B MoE)
  • Parameter count for this release: unchanged from the base model architecture
  • Note: the ~17.8GB model.safetensors file size is a quantized checkpoint size and does not mean the model is 18B parameters.

Runtime Compatibility

Known issue on some stock vLLM 0.16.x + vllm-node setups:

  • assistant content may be null
  • output may be dumped into reasoning fields with broken formatting

Required Docker image (build this first)

This model requires a custom vLLM image. Do not use stock vllm/vllm-openai as-is.

1) Create Dockerfile

Save as Dockerfile.vllm-glm4lite:

FROM vllm/vllm-openai:latest

ARG TRANSFORMERS_COMMIT=393b4b3d28e29b4b05b19b4b7f3242a7fc893637

RUN apt-get update && apt-get install -y --no-install-recommends git && rm -rf /var/lib/apt/lists/*
RUN pip install --no-cache-dir -U "huggingface_hub==1.4.0"
RUN pip install --no-cache-dir -U --no-deps "git+https://github.com/huggingface/transformers.git@${TRANSFORMERS_COMMIT}"

2) Build image

docker build -t vllm-glm:parser-only-r2 -f Dockerfile.vllm-glm4lite .

3) Serve model with that image

docker run --rm --name glm47   --gpus all   --ipc=host   -p 8000:8000   -v /path/to/hf_cache:/hf_cache   -v /path/to/models:/models   -e HF_HOME=/hf_cache   -e HUGGINGFACE_HUB_CACHE=/hf_cache   -e HF_HUB_CACHE=/hf_cache   -e TRANSFORMERS_CACHE=/hf_cache   -e VLLM_NVFP4_GEMM_BACKEND=marlin   vllm-glm:parser-only-r2   --model /models/GLM-4.7-Flash-heretic-NVFP4   --served-model-name glm-4.7-flash   --quantization modelopt_fp4   --dtype bfloat16   --tensor-parallel-size 4   --max-model-len 131072   --enable-auto-tool-choice   --tool-call-parser glm47   --reasoning-parser glm45   --default-chat-template-kwargs '{"enable_thinking": true}'   --generation-config vllm   --override-generation-config '{"temperature":0.7,"top_p":1.0}'

4) Verify

curl http://127.0.0.1:8000/v1/models

Expected: model list includes glm-4.7-flash.

Performance Notes (RTX 3090 / Ampere)

These measurements were taken on a 4x RTX 3090 host with vLLM and this NVFP4 export.

Best single-session setup found so far

  • Use --optimization-level 3 (O3), not default O2.
  • Use TP2 on an NVLink-connected pair (CUDA_VISIBLE_DEVICES=0,2 + --tensor-parallel-size 2).
  • Keep --max-model-len 131072 (128k context remains supported).

Observed impact (single-session decode):

  • O3 vs O2: ~+16% tokens/sec
  • TP2 NVLink vs TP2 non-NVLink: ~+10% tokens/sec

What did not help in this environment

  • MTP speculative decoding (--speculative-config {"method":"mtp",...}) was slower than baseline.
  • VLLM_MARLIN_USE_ATOMIC_ADD=1 was slightly slower (~4-5%).
  • FP8 KV cache variants were not viable on this stack:
    • --kv-cache-dtype fp8
    • --kv-cache-dtype fp8 --calculate-kv-scales
    • --kv-cache-dtype fp8_ds_mla These failed due to no valid MLA attention backend on this Ampere path.

Throughput vs single-session speed

  • Increasing --max-num-seqs / --max-num-batched-tokens improved aggregate throughput significantly.
  • Those changes did not materially improve single-session latency/tokens-sec.
  • If your goal is one chat/session feeling faster, prioritize O3 + TP2 on NVLink.

Repository Contents

This model repo intentionally contains only serving-required artifacts:

  • model.safetensors
  • config.json
  • generation_config.json
  • tokenizer.json
  • tokenizer_config.json
  • chat_template.jinja
  • hf_quant_config.json
  • README.md
  • QUANTIZATION.md
  • LICENSE

No training checkpoints, raw calibration corpora, or temporary files are included.

License and Provenance

  • Base model: Olafangensan/GLM-4.7-Flash-heretic
  • Upstream lineage: decensored derivative of zai-org/GLM-4.7-Flash
  • Base license: MIT (per upstream model card)
  • This repo: quantized derivative for inference; no architecture changes

Please review and comply with upstream licenses and terms for your use case.

Reproducibility

Quantization recipe, command, and environment details are documented in QUANTIZATION.md.

At a glance:

  • Quantization method: ModelOpt NVFP4 (group_size=16, lm_head excluded)
  • Calibration mix: switch_turnflow_sanitized,open_code_reasoning
  • Calibration sizes: 1536,512 (sequence length 2048)
  • Export format: Hugging Face

Quick Start (OpenAI-compatible)

curl http://127.0.0.1:8000/v1/chat/completions   -H 'Content-Type: application/json'   -d '{
    "model": "glm-4.7-flash",
    "messages": [{"role": "user", "content": "hello"}],
    "max_tokens": 256
  }'

Integrity

  • model.safetensors SHA256: 3b5aca2db60c472e9dbcb44e79ab4f69442d9a83315bbfab7a3f39ab8b004116