中文Sign up

second-state/Nous-Hermes-2-Mixtral-8x7B-SFT-GGUF

🤗 Hugging Face sourcetext-generationapache-2.0352 GBGGUF✓ 12 checksumsupdated today
Submit in one command

Run it next to your model folder. It makes the torrent, checks your files against Hugging Face, and submits it. You just start seeding and paste your key from your account. It only reads your files and never changes them. Read the script first if you like.

curl -fsSL https://pirateface.co/package.sh | bash -s -- --repo second-state/Nous-Hermes-2-Mixtral-8x7B-SFT-GGUF ./model-folder
Needs a seeder →

Nous-Hermes-2-Mixtral-8x7B-SFT-GGUF

Original Model

NousResearch/Nous-Hermes-2-Mixtral-8x7B-SFT

Run with LlamaEdge

  • LlamaEdge version: v0.2.8 and above

  • Prompt template

    • Prompt type: chatml

    • Prompt string

      <|im_start|>system
{system_message}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant

  • Context size: 4096

  • Run as LlamaEdge service

    wasmedge --dir .:. --nn-preload default:GGML:AUTO:Nous-Hermes-2-Mixtral-8x7B-SFT-Q5_K_M.gguf llama-api-server.wasm -p chatml

  • Run as LlamaEdge command app

    wasmedge --dir .:. --nn-preload default:GGML:AUTO:Nous-Hermes-2-Mixtral-8x7B-SFT-Q5_K_M.gguf llama-chat.wasm -p chatml

Quantized GGUF Models

Name Quant method Bits Size Use case
Nous-Hermes-2-Mixtral-8x7B-SFT-Q2_K.gguf Q2_K 2 17.3 GB smallest, significant quality loss - not recommended for most purposes
Nous-Hermes-2-Mixtral-8x7B-SFT-Q3_K_L.gguf Q3_K_L 3 24.2 GB small, substantial quality loss
Nous-Hermes-2-Mixtral-8x7B-SFT-Q3_K_M.gguf Q3_K_M 3 22.5 GB very small, high quality loss
Nous-Hermes-2-Mixtral-8x7B-SFT-Q3_K_S.gguf Q3_K_S 3 20.4 GB very small, high quality loss
Nous-Hermes-2-Mixtral-8x7B-SFT-Q4_0.gguf Q4_0 4 26.4 GB legacy; small, very high quality loss - prefer using Q3_K_M
Nous-Hermes-2-Mixtral-8x7B-SFT-Q4_K_M.gguf Q4_K_M 4 28.4 GB medium, balanced quality - recommended
Nous-Hermes-2-Mixtral-8x7B-SFT-Q4_K_S.gguf Q4_K_S 4 26.7 GB small, greater quality loss
Nous-Hermes-2-Mixtral-8x7B-SFT-Q5_0.gguf Q5_0 5 32.2 GB legacy; medium, balanced quality - prefer using Q4_K_M
Nous-Hermes-2-Mixtral-8x7B-SFT-Q5_K_M.gguf Q5_K_M 5 33.2 GB large, very low quality loss - recommended
Nous-Hermes-2-Mixtral-8x7B-SFT-Q5_K_S.gguf Q5_K_S 5 32.2 GB large, low quality loss - recommended
Nous-Hermes-2-Mixtral-8x7B-SFT-Q6_K.gguf Q6_K 6 38.4 GB very large, extremely low quality loss
Nous-Hermes-2-Mixtral-8x7B-SFT-Q8_0.gguf Q8_0 8 49.6 GB very large, extremely low quality loss - not recommended