simonko912/simllama-1-instruct

A simple model trained from scratch on the llama chat format. Its trained without a base model, fully from scratch as a experiment. Trained within 16h, loss 3.088 and trained for 2 epoches. Made for extreme speed for interference, and easy finetuning. Sometimes gets facts wrong, but is able to do some instruction following.

Perplexity and loss based off my script (againsts training data): Loss: 2.7869
Perplexity: 16.2313

These ware tested by generating synthetic text and using it for perplexity calculation. Perplexity vs llama3.2 (data from llama3.2 in aidata.txt): Loss: 3.0716
Perplexity: 21.5765

Perplexity vs openhermes:
Loss: 3.0375
Perplexity: 20.8523

If you find a use for this model, let me know in discussions!

Benchmark

Simllama 1 is this model, oasst2 llama is my older, less language knowing model.
These benchmarks ware done 0 shot.

Context: 2048 tokens total
Size: 0.2B
Extra info: Dense model, llama like arch.

Examples:

(Stuff in [] are comments ) Prompt: "What is 1+1?"

1+1=2
The answer to this question is: 1+1=2
The answer to this question is: 1+1=2
The answer to this question is: 1+1=2
The answer to this question is: 1+1=2
[and so on (cropped)]

Prompt: "She sells sea shells on the sea shore. What does she sell?"

She sells sea shells on the sea shore.
She has a long, deep shade of sea shells on the sea shore.
She is a bright, deep shade of greenhouse gas, she is a beacon of light and orange that spread across the sea.
She is a tall, imposing figure with a thin, green apple and a warm smile.
She is a bright, deep blue light that transports all her knowledge and knowledge from the sea.
[cropped]

Prompt: "Who are you?" (This one was hilarious so didnt crop anything)

Hello! I am an AI assistant developed by Meta AI, a large language model
that can be used to generate human-like text based on the input I receive. I am
trained on a massive dataset of text from the internet and can generate human-like responses to a wide range of topics and questions. I can be used to generate
responses that are not considered harmful or inappropriate.
Hello, I am Open Assistant, an AI language model developed by Meta AI, which is
a type of AI that uses language that is designed to generate human-like text based on the input I receive. I can generate responses that are not considered harmful or inappropriate, and I cannot provide responses that are harmful or inappropriate.
Hello, I am Open Assistant, an open-source language model developed by Meta AI,
aims to provide an easy-to-use API for building and training large language models.
Hello! I am Open Assistant, an open-source AI language model developed by Meta AI. My purpose is to assist and provide helpful responses to a wide range of questions and topics, and I cannot provide responses that promote harmful or inappropriate or harm

Prompt: "If you eat a apple, what do you eat?"

If you eat a apple, it's important to take regular breaks and
eat them in a warm and light-hearted way. Here are some general guide
lines to help you fall asleep:
1. Avoid alcohol and alcohol: Avoid consuming alcohol from the alcoho
l and drinking it in a cold and warm environment, especially if you h
ave a cold drink.
2. Avoid alcohol: Avoid consuming alcohol by drinking too much alcohol, as this can negatively impact the quality of your sleep.
[cropped]

Prompt: "What is python?"

Python is a popular programming language that is used for developing applications. It is a language that is widely used in various fields, including web development, scientific computing, and scientific computing. However, it is not a widely used language for developing applications, as it is a fundamental language for creating applications and applications. It is also used in a variety of industries, including finance, healthcare, and finance.
Python is a high-level programming language that is used in various fields, including science, engineering, and finance. It is a popular language that is used in many different fields and has many applications in various fields such as data science, machine learning, and scientific computing. Python is also used in a wide range of industries, including finance, healthcare, and finance.
Python is a high-level, interpreted programming language that is used for
developing applications in various fields. It is a broad and broad field that is used to study and develop, and it is used to develop a wide range of applications in various fields.
Python is a popular language that is used for developing applications in various fields, including web development, data science, and scientific computing. It is used in a wide range of fields, including business, science, and
[cropped]

Prompt: "What is github?"

GitHub is an open-source software platform for building, testing, and deploying code. It is designed to be easy to use and run on a wide range of platforms, including Windows, macOS, and Linux. However, it is not a
perfect solution for developers who want to build a website that is simple to use and built on a single platform.
Open-source software is the standard for anyone to develop, test, and maintain.
GitHub is a free and open-source software platform that is well-suited for developing applications. It is also widely used in a variety of applications, including web development, data analysis, and content creation.
GitHub is an open-source sof
tware platform that is well-suited for developing applications.



[cropped]