So I want an LLM to teach me Japanese and I have an NVIDIA Thor. I have compressed models text layers to fit into available memory. I also included patched llmcompessor, inference script and a proof of concept script that does a multi turn voice conversation with vllm-omni in extras.
This model quantizes all layers which works Ok for voice recognition and text generation, but results in garbled voice. Keeping it because it's useful with external TTS that can give more voice options. Use catplusplus/Qwen3-Omni-30B-A3B-Instruct-NVFP4-talker-safe for voice generation, the tradeoff is larger size.