nightmedia/Ling-3.0-tiny-q8-hi-mlx

🤗 Hugging Face sourcetext-generationmit7.9B params16 GBsafetensors✓ 3 checksumsupdated today

Submit in one command

Run it next to your model folder. It makes the torrent, checks your files against Hugging Face, and submits it. You just start seeding and paste your key from your account. It only reads your files and never changes them. Read the script first if you like.

curl -fsSL https://pirateface.co/package.sh | bash -s -- --repo nightmedia/Ling-3.0-tiny-q8-hi-mlx ./model-folder

Needs a seeder →

Ling-3.0-tiny-q8-hi-mlx

Brainwaves

          arc   arc/e boolq hswag obkqa piqa  wino
bf16      0.507,0.649,0.821,0.634,0.374,0.768,0.610
q8-hi     0.503,0.650,0.828,0.634,0.380,0.768,0.594
q6-hi     0.509,0.657,0.822,0.634,0.380,0.763,0.595
q5-hi     0.499,0.652,0.827,0.631,0.376,0.766,0.589
q4-hi     0.487,0.608,0.812,0.622,0.392,0.761,0.589

Quant     Perplexity      Peak Memory   Tokens/sec
bf16      5.614 ± 0.043   20.49 GB      4459
q8-hi     5.614 ± 0.043   14.04 GB      3761
q6-hi     5.630 ± 0.043   12.19 GB      4020
q5-hi     5.645 ± 0.043   11.27 GB      3838
q4-hi     5.725 ± 0.044   10.35 GB      3672

This model Ling-3.0-tiny-q8-hi-mlx was converted to MLX format from inclusionAI/Ling-3.0-tiny using mlx-lm version 0.32.0.

Use with mlx

pip install mlx-lm

from mlx_lm import load, generate

model, tokenizer = load("Ling-3.0-tiny-q8-hi-mlx")

prompt = "hello"

if tokenizer.chat_template is not None:
    messages = [{"role": "user", "content": prompt}]
    prompt = tokenizer.apply_chat_template(
        messages, add_generation_prompt=True, return_dict=False,
    )

response = generate(model, tokenizer, prompt=prompt, verbose=True)