This model is a quantized version of thinkingmachines/Inkling, a 975B total / 41B active parameter multimodal Mixture-of-Experts model that accepts text, image, and audio inputs and generates text outputs.
This model was obtained by quantizing the weights and activations of thinkingmachines/Inkling to FP8 data type using dynamic per-token quantization, ready for inference with vLLM. This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%.
Weights are quantized statically using per-channel FP8 scaling, and activations are quantized dynamically at inference time using per-token scaling. Only the weights and activations of the linear (attention and MoE expert) layers within the language backbone are quantized using LLM Compressor. The vision encoder, audio encoder, token embedding and unembedding layers, normalization layers, biases, MoE routing/gating logic, shared experts, and the model's early dense MLP layers are kept in their original precision.
This model can be deployed using vLLM.
Note: FP8 quantization support for Inkling requires vllm-project/vllm#48876.
vllm serve RedHatAI/Inkling-FP8-dynamic \
--tensor-parallel-size 8 \
--max-model-len 131072 \
--gpu-memory-utilization 0.90 \
--limit-mm-per-prompt '{"image": 4, "audio": 1}'
Tip: For text-only workloads, pass
--limit-mm-per-prompt '{"image": 0, "audio": 0}'to skip the vision/audio encoder memory allocation and free up GPU memory for a longer context window.
from openai import OpenAI
openai_api_key = "EMPTY"
openai_api_base = "http://<your-server-host>:8000/v1"
client = OpenAI(
api_key=openai_api_key,
base_url=openai_api_base,
)
model = "RedHatAI/Inkling-FP8-dynamic"
messages = [
{"role": "user", "content": "Explain quantum mechanics clearly and concisely."},
]
outputs = client.chat.completions.create(
model=model,
messages=messages,
)
generated_text = outputs.choices[0].message.content
print(generated_text)
model-00107-of-00108.safetensors 19 GBmodel-00001-of-00108.safetensors 15 GBmodel-00026-of-00108.safetensors 10 GBmodel-00092-of-00108.safetensors 10 GBmodel-00011-of-00108.safetensors 10 GBmodel-00049-of-00108.safetensors 10 GBmodel-00105-of-00108.safetensors 10 GBmodel-00072-of-00108.safetensors 10 GBmodel-00059-of-00108.safetensors 10 GBmodel-00036-of-00108.safetensors 10 GBmodel-00021-of-00108.safetensors 10 GBmodel-00037-of-00108.safetensors 10 GBmodel-00076-of-00108.safetensors 10 GBmodel-00042-of-00108.safetensors 10 GBmodel-00045-of-00108.safetensors 10 GBmodel-00069-of-00108.safetensors 10 GBmodel-00104-of-00108.safetensors 10 GBmodel-00094-of-00108.safetensors 10 GBmodel-00032-of-00108.safetensors 10 GBmodel-00080-of-00108.safetensors 10 GBmodel-00003-of-00108.safetensors 10 GBmodel-00028-of-00108.safetensors 10 GBmodel-00087-of-00108.safetensors 10 GBmodel-00023-of-00108.safetensors 10 GBmodel-00082-of-00108.safetensors 10 GB 10 GBmodel-00053-of-00108.safetensorsmodel-00025-of-00108.safetensors 10 GBmodel-00040-of-00108.safetensors 10 GBmodel-00018-of-00108.safetensors 10 GBmodel-00079-of-00108.safetensors 10 GBmodel-00016-of-00108.safetensors 10.0 GBmodel-00070-of-00108.safetensors 10.0 GBmodel-00067-of-00108.safetensors 10.0 GBmodel-00020-of-00108.safetensors 10.0 GBmodel-00058-of-00108.safetensors 10.0 GBmodel-00006-of-00108.safetensors 10.0 GBmodel-00088-of-00108.safetensors 10.0 GBmodel-00029-of-00108.safetensors 10.0 GBmodel-00078-of-00108.safetensors 10.0 GBmodel-00073-of-00108.safetensors 10.0 GBmodel-00048-of-00108.safetensors 10.0 GBmodel-00057-of-00108.safetensors 9.9 GBmodel-00098-of-00108.safetensors 9.9 GBmodel-00096-of-00108.safetensors 9.9 GBmodel-00099-of-00108.safetensors 9.9 GBmodel-00010-of-00108.safetensors 9.9 GBmodel-00061-of-00108.safetensors 9.9 GBmodel-00043-of-00108.safetensors 9.9 GBmodel-00004-of-00108.safetensors 9.9 GBmodel-00041-of-00108.safetensors 9.9 GBmodel-00007-of-00108.safetensors 9.9 GBmodel-00095-of-00108.safetensors 9.9 GBmodel-00034-of-00108.safetensors 9.9 GBmodel-00106-of-00108.safetensors 9.9 GBmodel-00097-of-00108.safetensors 9.8 GBmodel-00091-of-00108.safetensors 9.8 GBmodel-00102-of-00108.safetensors 9.8 GBmodel-00033-of-00108.safetensors 9.8 GBmodel-00024-of-00108.safetensors 9.8 GBmodel-00060-of-00108.safetensors 9.8 GBmodel-00027-of-00108.safetensors 9.8 GBmodel-00064-of-00108.safetensors 9.8 GBmodel-00062-of-00108.safetensors 9.8 GBmodel-00056-of-00108.safetensors 9.8 GBmodel-00054-of-00108.safetensors 9.8 GBmodel-00085-of-00108.safetensors 9.8 GBmodel-00015-of-00108.safetensors 9.8 GBmodel-00051-of-00108.safetensors 9.8 GBmodel-00065-of-00108.safetensors 9.8 GBmodel-00039-of-00108.safetensors 9.7 GBmodel-00038-of-00108.safetensors 9.7 GBmodel-00005-of-00108.safetensors 9.7 GBmodel-00083-of-00108.safetensors 9.7 GBmodel-00093-of-00108.safetensors 9.7 GBmodel-00035-of-00108.safetensors 9.7 GBmodel-00030-of-00108.safetensors 9.7 GBmodel-00086-of-00108.safetensors 9.7 GBmodel-00066-of-00108.safetensors 9.7 GBmodel-00047-of-00108.safetensors 9.7 GBmodel-00012-of-00108.safetensors 9.7 GBmodel-00008-of-00108.safetensors 9.7 GBmodel-00014-of-00108.safetensors 9.7 GBmodel-00063-of-00108.safetensors 9.7 GBmodel-00074-of-00108.safetensors 9.7 GBmodel-00101-of-00108.safetensors 9.7 GBmodel-00044-of-00108.safetensors 8.2 GBmodel-00089-of-00108.safetensors 7.4 GBmodel-00046-of-00108.safetensors 6.4 GBmodel-00090-of-00108.safetensors 6.0 GBmodel-00052-of-00108.safetensors 5.9 GBmodel-00002-of-00108.safetensors 5.7 GBmodel-00022-of-00108.safetensors 5.4 GBmodel-00050-of-00108.safetensors 5.3 GBmodel-00100-of-00108.safetensors 5.3 GBmodel-00031-of-00108.safetensors 5.2 GBmodel-00071-of-00108.safetensors 5.2 GBmodel-00108-of-00108.safetensors 5.0 GBmodel-00084-of-00108.safetensors 5.0 GBmodel-00009-of-00108.safetensors 5.0 GBmodel-00081-of-00108.safetensors 5.0 GBmodel-00017-of-00108.safetensors 4.9 GBmodel-00103-of-00108.safetensors 4.9 GBmodel-00013-of-00108.safetensors 4.9 GBmodel-00075-of-00108.safetensors 4.8 GBmodel-00077-of-00108.safetensors 4.8 GBmodel-00068-of-00108.safetensors 4.8 GBmodel-00019-of-00108.safetensors 4.8 GBmodel-00055-of-00108.safetensors 717 MB有文件?提交种子。
提交种子thinkingmachines/Inkling