Qwen3-4B-Instruct-2507-PII-RL
Qwen3-4B-Instruct-2507-PII-RL is a LoRA reinforcement learning fine-tune of Qwen/Qwen3-4B-Instruct-2507, trained for 740 policy updates on batches sampled from AdamLucek/open-pii-masking-en-us-30k using the adamlucek/pii-masking environment.
Qwen3-4B-Instruct-2507-PII-RL has been trained to mask PII data. Given an input phrase it will output the same phrase with all PII instances replaced by [PII].
Training
This model was trained using Tinker and the adamlucek/pii-masking environment with the following specs:
| Parameter | Value |
|---|---|
| Method | LoRA (rank=32) |
| Environment | pii-masking verifiers environment |
| Batch size | 256 trajectory groups (groups_per_batch=32 × group_size=8) |
| Max sequence length | 512 tokens |
| Optimizer | Adam (lr=1e-5, β1=0.9, β2=0.95, ε=1e-8) |
| Scheduler | Constant learning rate |
| Dataset | Full training set (num_train_examples=-1) |
Over 740 training steps, the following reward curve was produced:
Rewards
The reward function is a weighted combination of three components:
| Component | Weight | Description |
|---|---|---|
exact_match_reward |
1.0 | Binary reward (1.0 if the parsed masked output exactly matches the expected answer character-by-character, 0.0 otherwise) |
pii_count_reward |
0.5 | Binary reward (1.0 if the number of [PII] tags in the output matches the expected count, 0.0 otherwise) |
format_reward |
0.1 | Parser-generated format reward ensuring the output is properly formatted with valid XML tags (<masked_output>...</masked_output>) |
The final reward is calculated as:
reward = (1.0 × exact_match_reward) + (0.5 × pii_count_reward) + (0.1 × format_reward)
Reward Range: The reward can range from 0.0 (worst) to 1.6 (best), where:
- 1.6: Perfect match with correct PII count and valid format
- 1.0: Exact match but incorrect PII count or invalid format
- 0.5-0.6: Correct PII count but inexact match, with/without format compliance
- 0.0-0.1: No match, incorrect count, or invalid format
Usage
Loading and using the model via Transformers
from transformers import AutoModelForCausalLM, AutoTokenizer
# Replace with the actual repository name you used
model_id = "AdamLucek/Qwen3-4B-Instruct-2507-PII-RL"
# Load the merged model and tokenizer from the Hugging Face Hub
model = AutoModelForCausalLM.from_pretrained(model_id)
tokenizer = AutoTokenizer.from_pretrained(model_id)
system_prompt = """Replace all personally identifiable information (PII) in the text with [PII] tags.
PII includes: names, dates, phone numbers, SSNs, account numbers, addresses, email addresses, and any other identifying information.
Examples:
Input: Ticket Reservation for Florije: 'one ticket for Madame on October 8th, 1990'
Output: Ticket Reservation for [PII]: 'one ticket for [PII] on [PII]'
Input: User account recovery: "Hi Arljind Komla, your account recovery key is 426220045."
Output: User account recovery: "Hi [PII], your account recovery key is [PII]."
Return ONLY the masked text wrapped in masked_outputXML tags:
<masked_output>
[Your masked text here]
</masked_output>"""
# Prepare a prompt for inference using the messages format
messages = [
{"role": "system", "content": system_prompt},
{"role": "user", "content": "Hi Balian, we are reaching out to confirm your gaming preferences. Your account, EL@protonmail.com, has been inactive for 46 months. Please verify your account details, including 72611183194555."},
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
input_ids = tokenizer(text, return_tensors="pt").input_ids
# Perform inference
output = model.generate(input_ids, max_length=512)
# Decode and print the generated text
print(tokenizer.decode(output[0], skip_special_tokens=True))
Output:
<masked_output>
Hi [PII], we are reaching out to confirm your gaming preferences. Your account, [PII], has been inactive for [PII] months. Please verify your account details, including [PII].
</masked_output>
LoRA Adapter
The unmerged LoRA adapter is available in lora_adapter.
Additional Information
For all other information about the base model and usage, refer to the original Qwen/Qwen3-4B-Instruct-2507 page.