Ling-3.0-flash-VL NVFP4 — GGUF

NVFP4 GGUF conversion of Ling-3.0-flash-VL, a next-generation native multimodal model from InclusionAI.

This repository provides the NVFP4 (NVIDIA 4-bit floating-point) GGUF files, making the model usable with llama.cpp and other GGUF-compatible inference engines.

Model Overview

Ling-3.0-flash-VL is a native multimodal model built upon Ling-3.0-flash. It integrates visual information into the complete process of understanding, reasoning, acting, and verification, advancing beyond simple perception to solving real-world tasks through vision.

The model features a sparse MoE architecture with 124B total parameters, but only 5.5B activated parameters per token, delivering powerful multimodal reasoning with exceptional efficiency.

Model Architecture

Property	Value
Architecture	Sparse MoE (KDA & Gated MLA)
Total Parameters	124B
Active Parameters	5.5B
Context Length	256K tokens
Visual Encoder	ViT with MLP Projector
Temporal Encoding	VideoRoPE (Spatial + Temporal)
Backbone	42-layer hybrid (KDA/MLA 5:1 ratio)
Supported Modalities	Text, Image, Video

GGUF Files

File	Format	Description
`ling-3.0-flash-vl-nvfp4.gguf`	NVFP4	NVFP4 quantized model
`mmproj.gguf`	—	Multimodal projector (vision)

A chat_template.jinja file is also provided for use with chat-based inference.

About NVFP4: NVFP4 is NVIDIA's 4-bit floating-point format (E2M1) with microscaled per-block scaling factors. It offers significant memory savings while preserving quality close to bfloat16. NVFP4 inference is accelerated on NVIDIA Blackwell GPUs and supported via llama.cpp on compatible hardware.

Note on mmproj: The mmproj.gguf file contains the visual projector needed for multimodal (image/video) inference.

Usage

llama.cpp (CLI)

# Run text-only inference
./llama-cli \
  -m ling-3.0-flash-vl-nvfp4.gguf \
  -p "Explain the significance of multimodal LLMs." \
  --temp 0.6 --top-k 20 --top-p 0.95

llama-server (OpenAI-compatible API)

# Text-only
./llama-server \
  -m ling-3.0-flash-vl-nvfp4.gguf \
  --host [IP_ADDRESS] --port 8080

# Multimodal (image + video)
./llama-server \
  -m ling-3.0-flash-vl-nvfp4.gguf \
  --mmproj mmproj.gguf \
  --host [IP_ADDRESS] --port 8080

Multimodal (Image / Video)

For image and video inputs, use llama-server or llama-cli with the --mmproj flag pointing to mmproj.gguf. Refer to your inference engine's documentation for passing visual data alongside text prompts.

Generation Parameters

Recommended parameters based on the official configuration:

Parameter	Value	Note
Temperature	0.6	1.0 recommended for some generative tasks
Top-K	20
Top-P	0.95

Thinking Mode

Ling-3.0-flash-VL has thinking mode enabled by default via the chat template.

Output format: When thinking is enabled, the model outputs internal reasoning (chain-of-thought) followed by the final answer.
Control: Thinking can typically be disabled per request via the chat_template_kwargs (e.g., "enable_thinking": false).

Key Features

Native Multimodal: Deep integration of vision into reasoning and acting.
Efficient Inference: MoE architecture activates only 5.5B parameters per token.
Long Context: Support for up to 256K tokens.
Video Understanding: VideoRoPE allows the model to understand temporal changes and event localization.
Agentic Capabilities: Designed to interact with interfaces and translate visual info into action sequences.

Acknowledgements

Original model: inclusionAI/Ling-3.0-flash-VL

License

This project is licensed under the MIT License.

WhiskyAKM/Ling-3.0-Flash-VL-NVFP4-GGUF