This model has been quantized using GPTQModel.
from transformers import AutoTokenizer
from gptqmodel import GPTQModel
tokenizer = AutoTokenizer.from_pretrained("ModelCloud/QwQ-32B-Preview-gptqmodel-4bit-vortex-v2")
model = GPTQModel.load("ModelCloud/QwQ-32B-Preview-gptqmodel-4bit-vortex-v2")
messages = [
{"role": "system", "content": "You are a helpful and harmless assistant. You are Qwen developed by Alibaba. You should think step-by-step."},
{"role": "user", "content": "How can I design a data structure in C++ to store the top 5 largest integer numbers?"},
]
input_tensor = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
outputs = model.generate(input_ids=input_tensor.to(model.device), max_new_tokens=512)
result = tokenizer.decode(outputs[0][input_tensor.shape[1]:], skip_special_tokens=True)
print(result)
model-00001-of-00067.safetensors 1.6 GBmodel-00003-of-00067.safetensors 1.6 GBmodel-00014-of-00067.safetensors 282 MBmodel-00015-of-00067.safetensors 282 MBmodel-00016-of-00067.safetensors 282 MBmodel-00017-of-00067.safetensors 282 MBmodel-00018-of-00067.safetensors 282 MBmodel-00019-of-00067.safetensors 282 MBmodel-00020-of-00067.safetensors 282 MBmodel-00021-of-00067.safetensors 282 MBmodel-00022-of-00067.safetensors 282 MBmodel-00023-of-00067.safetensors 282 MBmodel-00024-of-00067.safetensors 282 MBmodel-00025-of-00067.safetensors 282 MBmodel-00026-of-00067.safetensors 282 MBmodel-00027-of-00067.safetensors 282 MBmodel-00028-of-00067.safetensors 282 MBmodel-00029-of-00067.safetensors 282 MBmodel-00030-of-00067.safetensors 282 MBmodel-00031-of-00067.safetensors 282 MBmodel-00032-of-00067.safetensors 282 MBmodel-00033-of-00067.safetensors 282 MBmodel-00034-of-00067.safetensors 282 MBmodel-00035-of-00067.safetensors 282 MBmodel-00036-of-00067.safetensors 282 MBmodel-00037-of-00067.safetensors 282 MBmodel-00038-of-00067.safetensors 282 MBmodel-00039-of-00067.safetensors 282 MBmodel-00040-of-00067.safetensors 282 MBmodel-00041-of-00067.safetensors 282 MBmodel-00042-of-00067.safetensors 282 MBmodel-00043-of-00067.safetensors 282 MBmodel-00044-of-00067.safetensors 282 MBmodel-00045-of-00067.safetensors 282 MBmodel-00046-of-00067.safetensors 282 MBmodel-00047-of-00067.safetensors 282 MBmodel-00048-of-00067.safetensors 282 MBmodel-00049-of-00067.safetensors 282 MBmodel-00050-of-00067.safetensors 282 MBmodel-00051-of-00067.safetensors 282 MBmodel-00052-of-00067.safetensors 282 MBmodel-00053-of-00067.safetensors 282 MBmodel-00054-of-00067.safetensors 282 MBmodel-00055-of-00067.safetensors 282 MBmodel-00056-of-00067.safetensors 282 MBmodel-00057-of-00067.safetensors 282 MBmodel-00058-of-00067.safetensors 282 MBmodel-00059-of-00067.safetensors 282 MBmodel-00060-of-00067.safetensors 282 MBmodel-00061-of-00067.safetensors 282 MBmodel-00062-of-00067.safetensors 282 MBmodel-00063-of-00067.safetensors 282 MBmodel-00064-of-00067.safetensors 282 MBmodel-00065-of-00067.safetensors 282 MBmodel-00066-of-00067.safetensors 282 MBmodel-00067-of-00067.safetensors 282 MBmodel-00004-of-00067.safetensors 282 MBmodel-00005-of-00067.safetensors 282 MBmodel-00006-of-00067.safetensors 282 MBmodel-00007-of-00067.safetensors 282 MBmodel-00008-of-00067.safetensors 282 MBmodel-00009-of-00067.safetensors 282 MBmodel-00010-of-00067.safetensors 282 MBmodel-00011-of-00067.safetensors 282 MBmodel-00012-of-00067.safetensors 282 MBmodel-00013-of-00067.safetensors 282 MBmodel-00002-of-00067.safetensors 10.4 kB有文件?提交种子。
提交种子Qwen/QwQ-32B-Preview