Llama.cpp imatrix quantization of mistralai/Mistral-Nemo-Instruct-2407
Original Model: mistralai/Mistral-Nemo-Instruct-2407
Original dtype: BF16 (bfloat16)
Quantized by: llama.cpp b3445
IMatrix dataset: here
Status: β
Available
Link: here
| Filename | Quant type | File Size | Status | Uses IMatrix | Is Split |
|---|---|---|---|---|---|
| Mistral-Nemo-Instruct-2407.Q8_0.gguf | Q8_0 | 13.02GB | β Available | βͺ Static | π¦ No |
| Mistral-Nemo-Instruct-2407.Q6_K.gguf | Q6_K | 10.06GB | β Available | βͺ Static | π¦ No |
| Mistral-Nemo-Instruct-2407.Q4_K.gguf | Q4_K | 7.48GB | β Available | π’ IMatrix | π¦ No |
| Mistral-Nemo-Instruct-2407.Q3_K.gguf | Q3_K | 6.08GB | β Available | π’ IMatrix | π¦ No |
| Mistral-Nemo-Instruct-2407.Q2_K.gguf | Q2_K | 4.79GB | β Available | π’ IMatrix | π¦ No |
| Filename | Quant type | File Size | Status | Uses IMatrix | Is Split |
|---|---|---|---|---|---|
| Mistral-Nemo-Instruct-2407.BF16.gguf | BF16 | 24.50GB | β Available | βͺ Static | π¦ No |
| Mistral-Nemo-Instruct-2407.FP16.gguf | F16 | 24.50GB | β Available | βͺ Static | π¦ No |
| Mistral-Nemo-Instruct-2407.Q8_0.gguf | Q8_0 | 13.02GB | β Available | βͺ Static | π¦ No |
| Mistral-Nemo-Instruct-2407.Q6_K.gguf | Q6_K | 10.06GB | β Available | βͺ Static | π¦ No |
| Mistral-Nemo-Instruct-2407.Q5_K.gguf | Q5_K | 8.73GB | β Available | βͺ Static | π¦ No |
| Mistral-Nemo-Instruct-2407.Q5_K_S.gguf | Q5_K_S | 8.52GB | β Available | βͺ Static | π¦ No |
| Mistral-Nemo-Instruct-2407.Q4_K.gguf | Q4_K | 7.48GB | β Available | π’ IMatrix | π¦ No |
| Mistral-Nemo-Instruct-2407.Q4_K_S.gguf | Q4_K_S | 7.12GB | β Available | π’ IMatrix | π¦ No |
| Mistral-Nemo-Instruct-2407.IQ4_NL.gguf | IQ4_NL | 7.10GB | β Available | π’ IMatrix | π¦ No |
| Mistral-Nemo-Instruct-2407.IQ4_XS.gguf | IQ4_XS | 6.74GB | β Available | π’ IMatrix | π¦ No |
| Mistral-Nemo-Instruct-2407.Q3_K.gguf | Q3_K | 6.08GB | β Available | π’ IMatrix | π¦ No |
| Mistral-Nemo-Instruct-2407.Q3_K_L.gguf | Q3_K_L | 6.56GB | β Available | π’ IMatrix | π¦ No |
| Mistral-Nemo-Instruct-2407.Q3_K_S.gguf | Q3_K_S | 5.53GB | β Available | π’ IMatrix | π¦ No |
| Mistral-Nemo-Instruct-2407.IQ3_M.gguf | IQ3_M | 5.72GB | β Available | π’ IMatrix | π¦ No |
| Mistral-Nemo-Instruct-2407.IQ3_S.gguf | IQ3_S | 5.56GB | β Available | π’ IMatrix | π¦ No |
| Mistral-Nemo-Instruct-2407.IQ3_XS.gguf | IQ3_XS | 5.31GB | β Available | π’ IMatrix | π¦ No |
| Mistral-Nemo-Instruct-2407.IQ3_XXS.gguf | IQ3_XXS | 4.95GB | β Available | π’ IMatrix | π¦ No |
| Mistral-Nemo-Instruct-2407.Q2_K.gguf | Q2_K | 4.79GB | β Available | π’ IMatrix | π¦ No |
| Mistral-Nemo-Instruct-2407.Q2_K_S.gguf | Q2_K_S | 4.49GB | β Available | π’ IMatrix | π¦ No |
| Mistral-Nemo-Instruct-2407.IQ2_M.gguf | IQ2_M | 4.44GB | β Available | π’ IMatrix | π¦ No |
| Mistral-Nemo-Instruct-2407.IQ2_S.gguf | IQ2_S | 4.14GB | β Available | π’ IMatrix | π¦ No |
| Mistral-Nemo-Instruct-2407.IQ2_XS.gguf | IQ2_XS | 3.92GB | β Available | π’ IMatrix | π¦ No |
| Mistral-Nemo-Instruct-2407.IQ2_XXS.gguf | IQ2_XXS | 3.59GB | β Available | π’ IMatrix | π¦ No |
| Mistral-Nemo-Instruct-2407.IQ1_M.gguf | IQ1_M | 3.22GB | β Available | π’ IMatrix | π¦ No |
| Mistral-Nemo-Instruct-2407.IQ1_S.gguf | IQ1_S | 3.00GB | β Available | π’ IMatrix | π¦ No |
If you do not have hugginface-cli installed:
pip install -U "huggingface_hub[cli]"
Download the specific file you want:
huggingface-cli download legraphista/Mistral-Nemo-Instruct-2407-IMat-GGUF --include "Mistral-Nemo-Instruct-2407.Q8_0.gguf" --local-dir ./
If the model file is big, it has been split into multiple files. In order to download them all to a local folder, run:
huggingface-cli download legraphista/Mistral-Nemo-Instruct-2407-IMat-GGUF --include "Mistral-Nemo-Instruct-2407.Q8_0/*" --local-dir ./
# see FAQ for merging GGUF's
<s>[INST] {user_prompt}[/INST] {assistant_response}</s>[INST] {next_user_prompt}[/INST]
<s>[INST] {user_prompt}[/INST] {assistant_response}</s>[INST] {system_prompt}
{next_user_prompt}[/INST]
llama.cpp/main -m Mistral-Nemo-Instruct-2407.Q8_0.gguf --color -i -p "prompt here (according to the chat template)"
According to this investigation, it appears that lower quantizations are the only ones that benefit from the imatrix input (as per hellaswag results).
gguf-split availablegguf-split, navigate to https://github.com/ggerganov/llama.cpp/releasesgguf-splitMistral-Nemo-Instruct-2407.Q8_0)gguf-split --merge Mistral-Nemo-Instruct-2407.Q8_0/Mistral-Nemo-Instruct-2407.Q8_0-00001-of-XXXXX.gguf Mistral-Nemo-Instruct-2407.Q8_0.ggufgguf-split to the first chunk of the split.Got a suggestion? Ping me @legraphista!
Have the files? Submit a torrent.
Submit a torrentHave the complete files and permission to share them? Replace the example folder path. The script checks the files, makes a torrent and submits its magnet and checksums. Keep your torrent client seeding. Get your community key from your settings, and read the script before running it.
curl -fsSL https://pirateface.co/package.sh | bash -s -- --repo legraphista/Mistral-Nemo-Instruct-2407-IMat-GGUF ./model-foldermistralai/Mistral-Nemo-Instruct-2407