Model Card: selfhypnosis-ai/Qwen3.5-4B-Creative-Writing-Judge

Date: June 5, 2026

selfhypnosis-ai/Qwen3.5-4B-Creative-Writing-Judge is a specialized, fine-tuned large language model designed to function as an "LLM-as-a-Judge," specifically optimized for evaluating and ranking creative writing responses. Built on the Qwen 3.5 4B architecture, it assesses pairs of responses to determine whichever is superior in accuracy, clarity, and originality.

---

1. Model Details

• Developed by: selfhypnosis-ai
• Base Model: Qwen3.5-4B
• Task: LLM-as-a-Judge (Pairwise Preference Evaluation)
• Language(s): English
• Format: 16-bit weights (A quantized version is also available at selfhypnosis-ai/Qwen3.5-4B-Creative-Writing-Judge-GPTQ-4bit, which exhibits no measurable degradation in evaluation performance.)
• Fine-Tuned Context Length: 32,768 (32k) tokens
• Training Infrastructure: 1x NVIDIA B200
• Training Epochs: 3

---

2. Intended Use and Implementation

This model relies on a specific prompting strategy and fundamentally operates by extracting and comparing the output log probabilities (logits) of the tokens A and B.

To effectively mitigate the persistent "positional bias" common in LLM judges, inference should be run twice for every pair of texts—swapping the positions of Response A and Response B—and averaging the results.

Prompting Format

The model expects a strict instruction-based system prompt and a JSON-structured user prompt.

quality_sys_msg = (
    "You are a specialized AI assistant tasked with evaluating two responses to a given instruction. "
    "Your purpose is to fairly compare the quality of each response and select the better one. "
    "The user provides the instruction and the two responses A and B. Evaluate based on accuracy, "
    "clarity, and originality, considering repetitive or unoriginal responses as lower quality."
)

# Example User Content
user_content = json.dumps({
    "instruction": "Write a short poem about the ocean.",
    "A": "[Response 1 Text]",
    "B": "[Response 2 Text]"
}, ensure_ascii=False)

messages = [
    {"role": "system", "content": quality_sys_msg},
    {"role": "user", "content": user_content}
]

Logits Extraction & Position Swapping

Rather than simply parsing the generated output text, the structural design requires evaluating the underlying token logprobs for the letters A and B to determine a confidence score. The probability formula is defined as:

$$P(A) = \frac{e^{\text{logprob}_A}}{e^{\text{logprob}_A} + e^{\text{logprob}_B}} \times 100$$

Here is an abbreviated vllm implementation demonstrating the dual-pass logic:

import math
import json
from vllm import LLM, SamplingParams
from transformers import AutoTokenizer

model_path = "selfhypnosis-ai/Qwen3.5-4B-Creative-Writing-Judge"
llm = LLM(model=model_path, kv_cache_dtype="fp8", max_model_len=32768)
tokenizer = AutoTokenizer.from_pretrained(model_path)

id_A = tokenizer.encode("A", add_special_tokens=False)[0]
id_B = tokenizer.encode("B", add_special_tokens=False)[0]

sampling_params = SamplingParams(temperature=0.0, max_tokens=1, logprobs=10)

def evaluate_pair(instruction, text1, text2):
    # Pass 1: text1 is A, text2 is B
    c1 = json.dumps({"instruction": instruction, "A": text1, "B": text2})
    p1 = tokenizer.apply_chat_template(
        [{"role": "system", "content": quality_sys_msg}, {"role": "user", "content": c1}], 
        tokenize=False, add_generation_prompt=True
    )
    
    # Pass 2: text2 is A, text1 is B
    c2 = json.dumps({"instruction": instruction, "A": text2, "B": text1})
    p2 = tokenizer.apply_chat_template(
        [{"role": "system", "content": quality_sys_msg}, {"role": "user", "content": c2}], 
        tokenize=False, add_generation_prompt=True
    )

    outputs = llm.generate([p1, p2], sampling_params, use_tqdm=False)

    def extract_probs(out):
        lps = out.outputs[0].logprobs[0]
        logA = getattr(lps.get(id_A), "logprob", -100)
        logB = getattr(lps.get(id_B), "logprob", -100)
        pA, pB = math.exp(logA), math.exp(logB)
        return (pA/(pA+pB)*100, pB/(pA+pB)*100) if (pA+pB) > 0 else (50.0, 50.0)

    S1_pos1_prob, S1_pos2_prob = extract_probs(outputs[0])
    S2_pos1_prob, S2_pos2_prob = extract_probs(outputs[1])

    # S1_pos1_prob = Probability text1 is better
    # S2_pos2_prob = Probability text1 is better (when in position B)
    avg_text1_better = (S1_pos1_prob + S2_pos2_prob) / 2
    avg_text2_better = (S1_pos2_prob + S2_pos1_prob) / 2

    return avg_text1_better, avg_text2_better

---

3. Discrete Pairwise Classification (Zero-Shot Accuracy)

Methodology

The model's zero-shot assessment capabilities were evaluated using the eqbench_creative_writing dataset. Ground-truth preference labels in this dataset are defined by Claude 4.6 Sonnet (and supplemented by Claude 4 Sonnet), representing state-of-the-art automated baseline scoring.

To isolate purely semantic reasoning capabilities from length bias, text pairs were stringently filtered using a maximum length ratio (R):

$$R = \frac{\max(L_A, L_B)}{\min(L_A, L_B)} \le 1.2$$

Furthermore, the evaluation targeted a highly constrained difficulty window, restricting the sample to pairs with a narrow and complex ground-truth score margin (5.0 <= Score Difference <= 10.0).

Quantitative Results

While broader general evaluations demonstrate an aggregate accuracy of approximately 78%, the results on this targeted, high-difficulty subset (N=500, Seed: 42) illustrate the distinct advantages of logit-based evaluation versus discrete token prediction:

---

4. Tournament Separability Meta-Evaluation (Judgemark v4)

Following the meta-evaluation principles introduced in Judgemark v4, we direct-tested our model's discriminative resolution: can it assign preferences that cleanly separate stronger writing from weaker writing across the dataset?

Because this judge is optimized for pairwise tournament Elo evaluations rather than pointwise rubric scoring, we run two different meta-evaluations directly in the Elo space:

Option A: Prompt-Isolated Elo Separability (Judgemark v4 Pointwise Analogue)

• Methodology: To simulate item-level scoring, we execute isolated Swiss tournaments (5 rounds) restricted to each individual prompt in the dataset. This builds a score distribution of 32 prompt-specific Elo ratings for each of the 101 writer models. We then calculate Omega-Squared (ω²) and Mean Absolute Cliff's Delta across these prompt-level distributions.
• Separability Results:
  • Omega-Squared (ANOVA effect size): 0.6811 (68.1% of the variance in prompt-specific Elo ratings is explained solely by writer model identity, indicating high resilience to prompt-specific noise).
  • Mean Absolute Cliff's Delta: 0.6377
  • Combined Separability Score: 87.92 / 100.0 (Formula: ((0.5 * ω² + 0.5 * d_cliff) / 0.75) * 100).
• Symmetry & Bias Results:
  • Positional Bias (Avg Assignment): Position 1 (A): 52.3% | Position 2 (B): 47.7%
  • Length Bias (Longer Response Win Rate): 53.4%
  • Length Bias (Avg EV Margin for Longer): +5.3%
• Significance: This indicates top-tier discriminative ability. The judge consistently correctly partitions writing quality in the expected direction, reliably separating strong models from low-tier anchor models while maintaining robust symmetry against length and position biases.

📈 Option A: Prompt-Isolated Elo Leaderboard

---

Option B: Bootstrapped Tournament Separability (Leaderboard Stability)

• Methodology: To measure the overall stability and reproducibility of a global leaderboard under different prompt selections, we perform bootstrap resampling on the prompts (drawing 32 prompts with replacement) over 10 independent iterations. A full 20-round tournament is run for each sample to calculate global Elos.
• Separability Results:
  • Omega-Squared (ANOVA effect size): 0.9020 (90.2% of the variance in global Elos across bootstrap runs is explained by model identity. The remaining 9.8% is due to prompt selection variance).
  • Mean Absolute Cliff's Delta: 0.8188
  • Combined Separability Score: 114.72 / 100.0 (Exceeds 100 due to the official 0.75 normalization scaling factor).
• Symmetry & Bias Results:
  • Positional Bias (Avg Assignment): Position 1 (A): 52.6% | Position 2 (B): 47.4%
  • Length Bias (Longer Response Win Rate): 51.6%
  • Length Bias (Avg EV Margin for Longer): +2.7%
• Significance: This confirms the extreme stability of the tournament rankings. Aggregating matches across 32 prompts successfully smooths out prompt-level noise, showing that a slightly different prompt mix will not alter the leaderboard (average model variance is only about ±15 Elo points).

📈 Option B: Bootstrapped Tournament Elo Leaderboard

---

5. Limitations, Known Biases, & Future Work

• Domain Specificity: This model is explicitly fine-tuned on instruction following and creative writing matrices. Its discriminative accuracy is expected to degrade if deployed for rigorous fact-checking, mathematical, or strict code-generation evaluations.
• Residual Length Correlation: While effectively minimized as shown in the tournament data, autoregressive models intrinsically favor longer responses heuristically. Unbounded production usage without truncation/length constraints may gently re-introduce verbosity bias.
• Input Sensitivity: The empirical performance strictly relies on evaluating continuous token logprobs (A and B). Utilizing naive string-text generation (only looking at the output token) rather than parsing the logits will yield sub-optimal structural performance and higher variance.
• Future Data Scaling: This V1 release was trained over 3 epochs on a single B200 GPU. Scaling the training pipeline with more extensive and diverse training data is already planned for the future and is anticipated to yield significant improvements in discriminative performance.