CLM-Qwen3.5-2B

• 131K-token context: 4× Jev's 32K.
• Beats Jev at spotting multi-document files: 97.7% vs 96.1%.
• Within five points of Jev on 200+ document types: 94.1% vs 98.7%, at 2B parameters.
• 2× faster than Jev: 359 ms vs 710 ms per five-question case.
• Level with Jev on typed decisions once fitted: 72.4% vs 72.7%.

Open weights · Apache 2.0 · a 2B encoder plus 12.6M trained parameters · built on Contrastive Language Models (Kwok et al., 2026).

What this is, and whose shoulders it stands on

A Contrastive Language Model (CLM) is a frozen LLM encoder with two small projection heads, a state head and an action head. The heads are trained with a bidirectional InfoNCE objective so that a situation and the right answer to it land close together in one embedding space. Answering a question takes one encoder pass: every candidate answer is scored by cosine similarity, and a softmax over the candidates is the answer's probability distribution. States and candidates are encoded separately, so candidate embeddings are computed once and reused across requests (CLM's state & action caching).

The method, the training objective, the open-source code and the serving stack (contrastive-lm) all come from Jacky Kwok, Hangoo Kang, Tarun Suresh, Jon Saad-Falcon, Marco Pavone, Christopher Ré and Azalia Mirhoseini. Their release, CLM-v0.1-8B, sits on Qwen3-8B. This model re-runs their recipe on Qwen3.5-2B and adds what it took to make a model a quarter the size strong at typed decisions and documents:

• An answer-position layout. The context, the question and its options are rendered as a chat turn, and the state vector is read where the model is about to answer, rather than at the end of the question text. The heads see the model at the moment it decides.
• A typed-decision training stage on top of the original three stages (Q&A pre-training, hard-negative mid-training, agent post-training). It covers 37 public human-labelled tasks, hundreds of synthetic decision workflows labelled with soft answer distributions by reasoning teachers, and synthetic documents across hundreds of types in about twenty domains. The documents come with the mess real files have: missing headers, OCR noise, partial pages, several documents stapled together.
• Long states. The encoder reads up to 131K tokens, 4× the 32K that Jev accepts for a state and its questions, so long document packages fit in one pass.
• Calibration. One temperature is fitted on held-out validation data (never on a benchmark), so the probabilities mean something out of the box.
• Generalization over memorization. Checkpoints were chosen only on held-out validation data, including whole workflows and whole document taxonomies the model never trained on. No benchmark was used for training or checkpoint selection; the one exception is the typed-decisions specialist file, which is fitted on that benchmark's train split.

Training at a glance: pre-trained on ~31M Nemotron-CC Q&A pairs, mid-trained on ~3.2M judged synthetic hard negatives, post-trained on ~1M agent steps (including states longer than 32K tokens), then ~380K typed questions.

Compared with CLM-v0.1-8B

CLM-v0.1-8B is the authors' general-purpose model and the starting point for this one. It targets agentic decisions and fine-tuned verification. There the authors report zero-shot results on par with Jev on computer-use, gaming and tool-calling tasks, and state-of-the-art verifier results with fine-tuned heads: 81.6% on DeepSWE and 87.6% on Terminal-Bench 2.1. We did not re-measure those. This model narrows the target to typed decisions and documents, and on those it comes out ahead with a quarter of the encoder.

• The document rows measure what the added typed-decision stage buys, not a flaw in the original. CLM-8B was never trained to sort documents into 200+ types. It ran zero-shot in its own layout on 8K-token inputs.
• The fitted typed-decisions row is the closest head-to-head: the same fitting recipe on the same data, each model in its own layout. The 2B model leads on both accuracy and calibration.
• The last two rows use held-out splits of our own data, which favour this model. The authors' evaluation set is not public.
• Zero-shot on the four typed-decisions workflows, the 8B is slightly ahead.

Results

Typed decisions: the public leaderboard

LocalLLaMA/typed-decisions, test split: 400 cases and 2,000 decisions. Each case is sent whole (the state plus all five questions) in one request and scored against the full gold distributions, exactly as the leaderboard does. Our scorer reproduces the card's Uniform row to the digit (KL 0.444, Brier 0.238).

Rows without "our run" are copied from the leaderboard as of 2026-09-30; see it for every general-track entry. † = measured on a laptop by the leaderboard, not hosted.

• Fitted, it is the strongest specialist on the board. A few minutes of head training on the train split takes the 2B model to 72.4% accuracy with a KL of 0.135. That beats both specialist rows and the 4× larger CLM-8B fitted the same way, and sits near the 0.735 ceiling set by the teacher's own self-agreement. It is level with Jev's 72.7% on accuracy, and its distributions are more than 10× closer to the gold on KL. The leaderboard rightly says fitted and zero-shot numbers are not directly comparable, so read this as "a 2B encoder plus minutes of fitting reaches Jev-level agreement on these workflows", not as a head-to-head. The fitted heads ship in this repo, so the row can be reproduced.
• Zero-shot on these four workflows it does not transfer yet (0.339, below the label prior). The original CLM-8B shows the same gap (0.358). See Limitations.
• It is fast. 359 ms p50 end to end for a whole five-question case: one L40S on Modal, measured from a client over the public internet, one request at a time. That is lower than every hosted row on the leaderboard and about half of Jev's 710 ms. Providers, hardware and client locations differ, so treat latency comparisons as indicative.

Document classification: private benchmark

The benchmark has 373 synthetic financial documents over 200+ document types: tax forms, bank and pay records, disclosures, insurance, identity and property records. They include clean, header-less, OCR-noised, scanned, long, packaged, partial and out-of-scope documents. Fifteen more files bundle several documents together, for 388 in all on the "more than one document?" question. Every model gets the identical request: one choice over all 200+ types plus one yes/no question. Both models were measured through their hosted APIs: Jev 1.13.0 on 2026-09-25, and this model on 2026-09-30. Our offline evaluation of the same heads scored 95.2% on document type.

• Within five points of a leading commercial classifier at 2B parameters with open weights. The right type is in the top three 98.7% of the time.
• Better than Jev at spotting stapled-together files: 97.7% vs 96.1% on whether a file holds more than one document.
• Robust to messy input. It scores 100% on OCR-noised documents and 94% on scanned ones, and it reads long packages whole: up to 131K tokens in one pass.
• A second pass adds more. In our offline evaluation, re-ranking the top five candidates with short per-type descriptions lifted document type from 95.2% to 96.3%. That step is not part of this release.

The original CLM skills

We also measured the three skills the CLM recipe trains, on held-out splits of our own data. Retrieval uses held-out Q&A pairs. The hard-negative test uses 10,676 questions, each with ten LLM-written wrong answers. The agent test uses held-out steps from agent trajectories.

These questions come from the same public sources our training data does, and our generator wrote the negatives, so the comparison favours this model. Read it as a check that the original skills survived the extra training, not as a head-to-head. The authors' own evaluation set is not public.

Generalization

On held-out validation data, never trained on:

• Document taxonomies it has never seen (whole domains held out): 79.8% top-1.
• Held-out examples of 37 public human-labelled tasks (entailment, reading comprehension, topic, intent, emotion, ratings and more): 71.9%.

Usage

The model runs on the authors' contrastive-lm stack. You need three pieces: a vLLM server producing last-token-pooled Qwen3.5-2B embeddings, the heads in this repo, and the layout in clm_choice/. It was tested with vllm==0.30.0 and contrastive-lm==0.1.0.

pip install "vllm==0.30.0" "contrastive-lm==0.1.0"
hf download jayavibhav/clm-qwen3.5-2b --local-dir clm-qwen3.5-2b && cd clm-qwen3.5-2b

# 1. The frozen encoder: last-token-pooled Qwen3.5-2B embeddings
vllm serve Qwen/Qwen3.5-2B --served-model-name qwen3.5-2b --runner pooling --convert embed \
  --pooler-config '{"seq_pooling_type": "LAST", "use_activation": true}' \
  --max-model-len 131072 --port 8090 &

# 2. The API: the authors' clm-serve with this model's layout (serve.py is in this repo)
python serve.py        # or: python serve.py specialists/typed-decisions/clm_heads.pt

# 3. Ask typed questions about any state
from clm import CLMClient, Choice, Noul, Score

client = CLMClient()  # http://127.0.0.1:8700
answer = client.system_one(
    state={"ticket": "I was charged twice for my March invoice and nobody answers the phone!"},
    questions={
        "team": Choice(instructions="Which team should handle this?",
                       criteria={"billing": "Charges, invoices, refunds",
                                 "technical": "Bugs and outages",
                                 "sales": "Plans and upgrades"}),
        "refund_due": Noul(instructions="The customer is owed a refund."),
        "frustration": Score(instructions="How frustrated is the customer?",
                             criteria=["Calm", "Annoyed", "Very angry"]),
    },
)
print(answer.answers["team"].probabilities)

The server speaks the System One API: POST /v1/systemone for typed questions, and POST /v1/rank for ranking free-form candidates. It takes the same request body as the typed-decisions leaderboard sends.

• Temperature. The default heads are calibrated for judgment calls. For crisp classification, such as document types or intents, pass temperature=0.5. That restores the sharper confidences the model was trained with. Accuracy is identical either way.
• Fitting to your task. Only the heads train, so fitting on a few hundred labelled cases takes minutes on one GPU. That is how the 72.4% typed-decisions row was made. Those heads are in specialists/typed-decisions/clm_heads.pt; pass that path to serve.py to reproduce it.

Files

Limitations

• Zero-shot on brand-new decision workflows is its weak spot. On the four typed-decisions workflows it scores 33.9% zero-shot, below simply predicting the most common label. The original CLM-8B shows the same gap. Fitting the heads to a few hundred cases fixes it (72.4%). For a new decision workflow, plan on that small fit.
• Still behind Jev on document type, by 4.6 points. Its weakest inputs are partial pages, documents that mention other document types, and documents outside the label set. The benchmark documents are synthetic, so real-world numbers may differ.
• Encoder-locked. The heads need Qwen3.5-2B last-token embeddings in the layout from clm_choice/.
• English only.
• Relative probabilities. Like every CLM, it only scores the candidates you give it, and its probabilities are relative to that set.
• One temperature is a compromise between judgment calls and crisp classification. See the temperature note above.

Training pipeline and data: available on request

The full pipeline is available on request for research and evaluation. It covers data generation, encoding at up to 131K tokens, the four training stages, calibration and the evaluation harness. The curated training data is available the same way. It includes the synthetic documents, decision workflows with soft labels and judged hard negatives, written with open Qwen3.5 models and Claude. Open a discussion on this repository saying who you are and what you would like to use them for.

Acknowledgements

• The CLM authors, for the method, the code, the contrastive-lm serving package and the CLM-v0.1-8B reference model. Please cite their work (below) if this model is useful to you.
• The Qwen team, for Qwen3.5-2B.
• The typed-decisions maintainers, for a benchmark that measures calibration and not just accuracy.
• Public data: NVIDIA Nemotron-CC (Diverse-QA), Endless-Terminals, LiteCoder and SWE-smith agent trajectories. Also GLUE, SuperGLUE, MultiNLI, SNLI, ANLI, SciTail, BoolQ, RACE, ARC, OpenBookQA, CommonsenseQA, QASC, SciQ, AG News, DBpedia-14, Yahoo Answers Topics, 20 Newsgroups, CLINC150, dair-ai/emotion, Yelp, SST-5, TweetEval, IMDB, Rotten Tomatoes, PAWS, SMS Spam, WinoGrande, HellaSwag, LexGLUE and PubMed RCT.

Citation

This model follows the CLM method. Please cite:

@misc{kwok2026contrastivelanguagemodels,
  title={Contrastive Language Models: A System One Model for Fast and Generalizable Decision-Making},
  author={Jacky Kwok and Hangoo Kang and Tarun Suresh and Jon Saad-Falcon and Marco Pavone and Christopher Ré and Azalia Mirhoseini},
  year={2026},
  note={Notion Blog},
  url={https://contrastive-lm.notion.site}
}

License

Apache 2.0, the same as the CLM code and weights and the Qwen3.5-2B encoder.