Lily-Qwen1.5-0.5B Preview

Lily-Qwen1.5-0.5B Preview is a fine-tuned version of the Qwen1.5-0.5B model, optimized for enhanced performance in natural language processing tasks. Built on the Transformer architecture with SwiGLU activation, attention QKV bias, and group query attention, it offers improved multilingual support and stable handling of up to 32K context length. This model excels in text generation, conversational dialogue, and language understanding, making it ideal for chatbots, content creation, and interactive applications.

Installation

Ensure you have the required dependencies installed:

pip install transformers>=4.37.0 torch

Usage Examples

1. Loading the Model and Tokenizer

Load the model and tokenizer using the Hugging Face transformers library:

from transformers import AutoModelForCausalLM, AutoTokenizer

# Load model and tokenizer
model = AutoModelForCausalLM.from_pretrained("kulia-moon/Lily-Qwen1.5-0.5B-Preview", torch_dtype="auto", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("kulia-moon/Lily-Qwen1.5-0.5B-Preview")

2. Generating Text with a Prompt

Generate text based on a simple prompt:

# Define prompt
prompt = "Write a short story about a magical garden."

# Create chat template
messages = [
    {"role": "system", "content": "You are a creative assistant with a passion for storytelling."},
    {"role": "user", "content": prompt}
]

# Apply chat template
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

# Tokenize input
model_inputs = tokenizer([text], return_tensors="pt")

# Generate response
generated_ids = model.generate(model_inputs.input_ids, max_new_tokens=512)

# Decode output
response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(response)

Expected Output (example):

In a hidden valley, there bloomed a magical garden where flowers sang softly under the moonlight. Lily, a young explorer, stumbled upon it one evening...

3. Engaging in Conversational Dialogue

Use the model for interactive chat:

# Define conversation
messages = [
    {"role": "system", "content": "You are Lily, a friendly assistant who loves nature."},
    {"role": "user", "content": "What's your favorite flower?"}
]

# Apply chat template
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

# Tokenize and generate
model_inputs = tokenizer([text], return_tensors="pt")
generated_ids = model.generate(model_inputs.input_ids, max_new_tokens=100)

# Decode response
response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(response)

Expected Output (example):

Oh, I adore cherry blossoms! Their delicate pink petals remind me of spring's gentle embrace.

4. Language Translation

Translate text into another language:

# Define translation prompt
prompt = "Translate 'The garden blooms with vibrant colors' into Chinese."

messages = [
    {"role": "system", "content": "You are a multilingual assistant."},
    {"role": "user", "content": prompt}
]

# Apply chat template
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

# Tokenize and generate
model_inputs = tokenizer([text], return_tensors="pt")
generated_ids = model.generate(model_inputs.input_ids, max_new_tokens=50)

# Decode response
response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(response)

Expected Output (example):

花园盛开着鲜艳的色彩。

5. Fine-Tuning the Model

For custom tasks, fine-tune the model using Supervised Fine-Tuning (SFT):

from transformers import Trainer, TrainingArguments, DataCollatorForLanguageModeling
from datasets import load_dataset

# Load dataset
dataset = load_dataset("your_dataset")

# Define training arguments
training_args = TrainingArguments(
    output_dir="./lily-finetuned",
    per_device_train_batch_size=4,
    num_train_epochs=3,
    learning_rate=5e-5,
)

# Initialize trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=dataset["train"],
    data_collator=DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False),
)

# Start fine-tuning
trainer.train()

Limitations

• Context Length: Supports up to 32K tokens, which may be insufficient for extremely long documents.
• GQA Support: Lacks General Question Answering (GQA) support for most model sizes, potentially limiting performance on complex queries.
• Common Sense: May occasionally miss nuanced human behavior or real-world context.

Resources

• Hugging Face Model Hub
• Qwen1.5 Documentation
• Transformers Library

Citation BETA

If you use Lily-Qwen1.5-0.5B in your work, please cite:

@misc{lily-qwen1.5-0.5b,
  author = {kulia-moon},
  title = {Lily-Qwen1.5-0.5B: A Fine-Tuned Qwen1.5 Model},
  year = {2025},
  publisher = {Hugging Face},
  journal = {Hugging Face Model Hub}
}

Enjoy exploring the capabilities of Lily-Qwen1.5-0.5B and bring your creative ideas to life!