🦅 Supra Mini v5 8M

Supra Mini v5 8M is a very small model trained on 5 billion tokens of Fineweb-Edu for 2 epochs as the fifth version of our Supra Mini series. SupraMini-8M shows improvements across all benchmarks because of its larger size and training budget.

Model Config

• Parameters: 7,867,584 (8M)
• Architecture: Llama
• Vocab size with custom BPE tokenizer: 16384
• Hidden Size: 191
• Intermediate Size: 768
• Hidden Layers: 8
• Attention Heads: 4
• Max Position Embeddings: 1024
• Learning rate: 2e-4
• Weight Decay: 0.01
• Trained in bfloat16

Final Loss

This model reached a final CrossEntropy loss (on the train set) of 4.414.

Benchmarks

All benchmarks were executed using lm-eval.

Examples

Prompt: "Artificial intelligence is "
Output:: "Artificial intelligence is _______. I was a scientist at the University of Cambridge in London, and I had been looking for ways to get them back into the first time. It’s not just about learning how to use it but also about the world around us. The study has led to more than 150 studies showing that people who are able to learn new things like computers, laptops, and smartphones can be found in many different types of computer systems. The researchers have demonstrated that this type of technology works well with a wide range of computing applications. They believe they need to know what they want to do, such as a laptop or iPad, which means you might find themselves doing everything on your own. This could mean we would take up to 3 times an hour if we didn't see any problems, including anything from something else. So now there is no idea how much data will go through so far. We don’t understand why these algorithms aren’t actually working on their devices. And if you think about it, then here are some of the most important things we can do to help make our decisions. What does it say? What happens when someone comes across the internet? Why doesn’t they really feel like the Internet? How should you connect"

Prompt: "The main concept of physics is "
Output: "The main concept of physics is _______. This principle, which means that the universe has no energy; it should be a matter of time in its own right and therefore to be able to achieve its maximum potential. It must have been used as an alternative for quantum mechanics but not only in terms of power but also in the form of electrical systems such as magnetic fields or electric currents. The theory would include some processes like this: "We are using mechanical machines." In fact, we need to work with other types of computers, including those who do so by their computer. This is because they can actually use them to generate electricity from our devices. Thus, if you want to learn how to manufacture electronic circuits, then you will see a very simple solution - the way you think about what happens when you're trying to build up on your device. So, why does that sound? Because there's something new than a few different things, and I'm sure that you don't know exactly where it was. But, just let me know if there were many ways to create the same thing. You might say that it could make sense to you how much one is doing at all, even though it isn't going to happen! So, if you'd ask yourself to write down the problem"

Prompt: "Once upon a time, "
Output: "Once upon a time, .. ... - "The word 'pot' is not used in the past.") (A.D., A.D.) - "It's like a thing of course that we're going to think about it. The phrase "these" means what you see and how much are there?"). - "You have no idea where I'm so, but if my son has any thought or understanding of his name, he will be able to understand him by saying something more than one day. This way they can tell us when you've got me at home. That is why I want your child to know which words are most important for them: "If you get this language from another person, then you'll find yourself in the same place as you read it," says Mike McNamara, who was born with an English friend, Jennifer Batharinee, who had been diagnosed with dementia during her lifetime. He said he would learn how to say things such as "a lot of things," and "you don't really need to do anything else." It may seem simple because he didn't feel good before he went out and asked whether he could make sense of it. But he wanted to take advantage of the fact"

Usage

To use our model, just run this code:

from transformers import pipeline
import torch

print("Loading Supra Mini v5 8M model from Hugging Face...")
pipe = pipeline(
    "text-generation", 
    model="SupraLabs/Supra-Mini-v5-8M",
    device_map="auto",
    torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32
)

def generate_text(prompt, max_length=150):
    result = pipe(
        prompt, 
        max_new_tokens=max_length,
        do_sample=True,
        temperature=0.5,
        top_k=25,
        top_p=0.9,
        repetition_penalty=1.2,
        pad_token_id=pipe.tokenizer.pad_token_id,
        eos_token_id=pipe.tokenizer.eos_token_id
    )
    return result[0]['generated_text']

test_prompt = "The importance of education is"
print(f"\nPrompt: {test_prompt}")
print("-" * 30)
print("\nOutput:\n" + generate_text(test_prompt))

Use cases

1. Educational research
2. deployment or testing/fine-tuning on edge environments
3. Or more simply, for fun

Limitations

1. Cannot reason, chat, or code
2. Incoherent more often than not
3. Mostly unfactual

Training guide

We trained Supra Mini v5 8M on a single NVIDIA RTX 5060 Ti 16GB in ~11 hours for 2 epochs.
The full training code can be found in this repo as train_tokenizer.py (train costum BPE tokenizer with vocab size of 16384), train.py (train the model) and inference.py (test the model).
The model was trained on the first 5 billion tokens of Sample-10BT from Fineweb-Edu.