Veyra 30M Instruct

Veyra-30M-Instruct is the first public instruction-tuned checkpoint in the Veyra 30M line.

It is built on top of veyra-ai/Veyra-30M-Base and is intended to test how far a 30M-parameter English base model can be pushed with masked ChatML supervised fine-tuning.

Model Details

Base model: veyra-ai/Veyra-30M-Base
Parameters: approximately 34.6M
Language: English
Context length: 1024 tokens
Architecture: decoder-only causal language model
Chat format: ChatML
License: Apache 2.0
Status: experimental instruct release

Architecture

Veyra-30M is a compact decoder-only transformer-style model.

Vocabulary size: 8,192
Hidden size: 512
Layers: 8
Layer pattern: alternating attention and MLP-only blocks
Query heads: 8
Key/value heads: 2
Attention type: grouped-query attention
MLP: SwiGLU
Normalization: RMSNorm
Positional encoding: RoPE
Context length: 1024 tokens
Tied input/output embeddings
KV cache support for generation

Training

This checkpoint was trained with masked ChatML supervised fine-tuning.

Base checkpoint: veyra-ai/Veyra-30M-Base
SFT tokens: 100M non-padding SFT tokens
Supervised assistant tokens: approximately 35M supervised assistant tokens
Loss masking: system and user turns were used as context but masked from loss; assistant responses were supervised.

The released checkpoint was selected from intermediate SFT milestones based on qualitative behavior, not only lowest training loss. Later SFT checkpoints reached lower loss but showed more generic refusal/template behavior, so checkpoint selection was based on output quality.

Intended Use

Tiny-model instruction-following experiments
Local assistant experiments
ChatML behavior testing
Simple formatting and JSON experiments
Basic Python/helpfulness prompts
Studying SFT behavior in very small language models

Not Intended For

Production applications
Safety-critical use cases
Medical, legal, financial, or security advice
Reliable factual QA
Long-context reasoning
Robust arithmetic
User-facing deployment without additional safeguards

Known Limitations

This is a 30M-parameter model and has significant limitations.

Known failure modes include:

Weak variable binding
Weak arithmetic
Weak multi-turn recall
Occasional repetition
Confident but incorrect answers
Generic refusal or disclaimer behavior
Tool-call or reasoning-template contamination
Sensitivity to prompt wording
Fluent nonsense on unfamiliar prompts

This model is not fully safety-tuned. It may refuse some harmful requests, but refusal behavior is not reliable.

Prompt Format

This model uses ChatML.

<|im_start|>system
You are Veyra, a tiny local instruction model. Be concise, useful, casual, and lightly playful. Correct mistakes gently.<|im_end|>
<|im_start|>user
What does a tokenizer do?<|im_end|>
<|im_start|>assistant

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "veyra-ai/Veyra-30M-Instruct"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)

messages = [
    {
        "role": "system",
        "content": "You are Veyra, a tiny local instruction model. Be concise, useful, casual, and lightly playful."
    },
    {
        "role": "user",
        "content": "Explain what an API is using a simple analogy."
    },
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=120,
        temperature=0.3,
        top_k=40,
        do_sample=True,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=[tokenizer.eos_token_id, tokenizer.convert_tokens_to_ids("<|im_end|>")],
        use_cache=True,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=False))

Special Tokens

Important tokenizer special tokens include:

<|bos|>
<|eos|>
<|pad|>
<|unk|>
<|im_start|>
<|im_end|>
<|tool_call|>
<|tool_result|>
<|context|>
<|reasoning|>
<|end_reasoning|>
<|answer|>
<|fim_prefix|>
<|fim_middle|>
<|fim_suffix|>

For this checkpoint, standard ChatML with <|im_start|> and <|im_end|> is the recommended format.

Citation / Attribution

If you use or build on this model, please retain attribution to Veyra AI.

License

Apache 2.0.

veyra-ai/Veyra-30M-Instruct