North-Mini-Code-1.0-GGUF
GGUF conversions and quantizations of CohereLabs/North-Mini-Code-1.0 for use with:
- llama.cpp
- LM Studio
- Ollama
- Jan
- KoboldCpp
- Text Generation WebUI
- Open WebUI
- Other GGUF-compatible runtimes
About the Model
North-Mini-Code-1.0 is a code-focused Mixture-of-Experts (MoE) model released by CohereLabs.
This repository provides ready-to-use GGUF conversions for local inference across a range of hardware configurations.
Available Files
Full Precision
North-Mini-Code-1.0-F16.gguf
Quantized Versions
North-Mini-Code-1.0-Q4_K_M.ggufNorth-Mini-Code-1.0-Q5_K_M.ggufNorth-Mini-Code-1.0-Q6_K.ggufNorth-Mini-Code-1.0-Q8_0.gguf
Recommended Quantization
For most users:
North-Mini-Code-1.0-Q4_K_M.gguf
It offers the best balance of:
- Quality
- Memory usage
- Inference speed
If you have more available RAM/VRAM, consider:
North-Mini-Code-1.0-Q5_K_M.gguf
or
North-Mini-Code-1.0-Q6_K.gguf
for slightly higher output quality.
Approximate File Sizes
F16 ~60+ GB
Q4_K_M ~20 GB
Q5_K_M ~23 GB
Q6_K ~27 GB
Q8_0 ~34 GB
Actual sizes may vary slightly depending on conversion tooling versions.
Usage
llama.cpp
Prompt mode:
./llama-cli \
-m North-Mini-Code-1.0-Q4_K_M.gguf \
-p "Write a Python function that reverses a linked list."
Chat mode:
./llama-cli \
-m North-Mini-Code-1.0-Q4_K_M.gguf \
-cnv
LM Studio
- Download your preferred GGUF file.
- Open LM Studio.
- Import the model.
- Start chatting.
Ollama
Create a Modelfile:
FROM North-Mini-Code-1.0-Q4_K_M.gguf
Create the model:
ollama create north-mini-code -f Modelfile
Run it:
ollama run north-mini-code
Hardware Recommendations
Q4_K_M
Recommended minimum:
24 GB RAM
Q5_K_M
Recommended minimum:
32 GB RAM
Q6_K
Recommended minimum:
32-40 GB RAM
Q8_0
Recommended minimum:
48+ GB RAM
F16
Recommended minimum:
80+ GB RAM
Prompting Tips
This model is optimized for programming-related tasks.
Example prompts:
Implement a fast Rust HTTP server.
Explain this C++ compiler error.
Write comprehensive unit tests for the following Python code.
Convert this JavaScript function to TypeScript.
Optimize this SQL query.
Base Model
Base model:
CohereLabs/North-Mini-Code-1.0
All training, architecture, benchmarks, licensing terms, and usage restrictions belong to the original model authors.
Please refer to the original repository for official documentation and licensing information.
Conversion Details
Converted using:
llama.cpp
Generated quantizations:
F16
Q4_K_M
Q5_K_M
Q6_K
Q8_0
A tokenizer compatibility workaround was applied during conversion to support current GGUF conversion tooling.
Credits
- Base Model: CohereLabs
- GGUF Conversion & Quantization: NANI-Nithin
- Tooling: llama.cpp