Qwen3.5-2B Saudi Dialect

This model is a Saudi dialect conversational fine-tune of unsloth/Qwen3.5-2B:

• Model: https://huggingface.co/AyoubChLin/Qwen3.5-2B-saudi-dialect
• LoRA adapters: https://huggingface.co/AyoubChLin/Qwen3.5-2B-saudi-dialect-lora
• Dataset: https://huggingface.co/datasets/HeshamHaroon/saudi-dialect-conversations
• Base model: https://huggingface.co/unsloth/Qwen3.5-2B

The training setup uses Unsloth + TRL SFTTrainer with LoRA adapters and then merges the adapters back into the base model for easier deployment.

Model Details

• Base model: unsloth/Qwen3.5-2B
• Fine-tuning method: LoRA SFT
• Language: Arabic, focused on Saudi dialect conversations
• Training modality in this run: text-only conversational SFT
• Dataset split: 3545 total examples -> 3366 train / 179 eval
• System prompt used in training: أنت مساعد مفيد يتحدث باللهجة السعودية العامية.
• Tracking: Weights & Biases
• W&B run: https://wandb.ai/cherguelainea/qwen-saudi-dialect/runs/6udmlaan

Training Arguments

Training Results

Loss and Metrics

Trainable Parameters

Hardware

Recorded memory numbers above are GPU memory / VRAM measurements taken from the training run. The notebook did not record host system RAM.

Data Preparation

The dataset examples are conversation turns stored under messages. During preprocessing, a Saudi Arabic system prompt is prepended to each conversation before fine-tuning. The training notebook keeps only valid conversations and then performs a 5% evaluation split with seed 3407.

Usage

Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "AyoubChLin/Qwen3.5-2B-saudi-dialect"

tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    torch_dtype="auto",
    device_map="auto",
)

messages = [
    {"role": "system", "content": "أنت مساعد مفيد يتحدث باللهجة السعودية العامية."},
    {"role": "user", "content": "كيف حالك اليوم؟"},
]

input_ids = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    enable_thinking=False,
    return_tensors="pt",
).to(model.device)

outputs = model.generate(
    input_ids,
    max_new_tokens=200,
    temperature=0.7,
    top_p=0.9,
)

print(tokenizer.decode(outputs[0][input_ids.shape[-1]:], skip_special_tokens=True))

Unsloth

Install

%%capture
import re, torch

v = re.match(r"[\d]{1,}\.[\d]{1,}", str(torch.__version__)).group(0)
xformers = "xformers==" + {
    "2.10": "0.0.34",
    "2.9": "0.0.33.post1",
    "2.8": "0.0.32.post2",
}.get(v, "0.0.34")

!pip install sentencepiece protobuf "datasets>=2.18.0" "huggingface_hub>=0.34.0" hf_transfer wandb
!pip install --no-deps unsloth_zoo bitsandbytes accelerate {xformers} peft trl triton unsloth
!pip install -q "transformers>=5.0.0"
!pip install -q --no-deps "trl>=0.15.0"

Run

from unsloth import FastLanguageModel

repo_id = "AyoubChLin/Qwen3.5-2B-saudi-dialect"
max_seq_length = 4096

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name=repo_id,
    max_seq_length=max_seq_length,
    load_in_4bit=False,  # this repo was pushed as merged_16bit
)

FastLanguageModel.for_inference(model)

messages = [
    {
        "role": "system",
        "content": [
            {"type": "text", "text": "أنت مساعد مفيد يتحدث باللهجة السعودية العامية."}
        ],
    },
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "كيف حالك اليوم؟"}
        ],
    },
]

input_ids = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    enable_thinking=False,
    return_tensors="pt",
).to(model.device)

output_ids = model.generate(
    input_ids=input_ids,
    max_new_tokens=200,
    use_cache=True,
    temperature=0.7,
    top_p=0.9,
)

response = tokenizer.decode(
    output_ids[0][input_ids.shape[-1]:],
    skip_special_tokens=True,
)
print(response)

Notes

• This repository contains the merged full model pushed with save_method="merged_16bit".
• A separate LoRA adapter repository is also available: AyoubChLin/Qwen3.5-2B-saudi-dialect-lora.
• The base checkpoint is multimodal-capable, but this fine-tune was trained on text-only dialogue data.
• The training data is conversational and dialect-specific, so outputs may reflect biases or stylistic patterns present in the source dataset.