UltraGliClass v0.1

UltraGliClass is a zero-shot text classifier with about 150M parameters. It uses the GLiClass uni-encoder head on top of the dmis-lab/UltraBERT-150M encoder. The official GLiClass base models use ModernBERT-base or DeBERTa-v3-base instead.

You pass a text, a list of label strings and, optionally, an instruction. The model scores every label in one forward pass. It works for single-label and multi-label classification, intent detection, and choice / yes-no / score decisions over long inputs.

This release is the final checkpoint of Stage 3. Training ran in three stages. The last stage trained on inputs of up to 8,192 tokens and up to 256 labels.

License notice. The code and the weights are released under Apache-2.0. Stages 1 and 2 trained on rows of tasksource/tasksource-jev-typed-decisions that are licensed CC-BY-SA. Stage 3 removed share-alike rows from its new data, but it starts from the Stage 2 weights and replays Stage 2 data. These weights are therefore not free of the CC-BY-SA question. Whether share-alike terms apply to model weights is legally unsettled. If you need weights with no share-alike training data, do not use v0.1. Details are in Licenses and training data.

• Developed by: Srimon Danguria
• Model type: GLiClass uni-encoder (zero-shot sequence classification)
• Base model: dmis-lab/UltraBERT-150M (Apache-2.0)
• Language: English
• Code, training recipe and result files: github.com/srimon12/ultragliclass

Results at a glance

All numbers below come from one evaluation run on full test splits, with the same harness and the same GPU (Kaggle 2× T4) for every model, on 2 November 2026.

¹ Leaves out massive and financial_phrasebank, which are close to our training data. On these 12 sets v0.1 ties gliclass-base-v3.0 (Δ −0.005, 95% CI [−0.011, +0.000]). ² Held-out Jev rows whose state is at least 6,000 characters. This slice is not zero-shot for v0.1, because other Jev rows are in its training data. It is zero-shot for the two other models.

What this means in practice:

• Against gliclass-modern-base-v3.0 (same encoder size), v0.1 is ahead on the suite by +0.130 (95% CI [+0.124, +0.136]).
• Against gliclass-base-v3.0, v0.1 is ahead by +0.015 [+0.009, +0.020] on 14 sets and tied on the 12 strictly zero-shot sets. Its clear gains are on many-label intent detection and long inputs.
• On long decision inputs at 8,192 tokens, v0.1 beats modern-base by +0.230 [+0.167, +0.300] (McNemar p = 7.1e-13) and base-v3 by +0.248 [+0.180, +0.316] (p = 1.3e-14).
• gliclass-base-v3.0 is better on sst5 (0.338 vs 0.239), 20 Newsgroups (0.505 vs 0.420), spam (0.625 vs 0.581), cap_sotu (0.465 vs 0.435) and snips (0.926 vs 0.913).
• CLINC-150 accuracy (0.636) is below the bar we set before Stage 3 (0.650, the Stage 2 value). We accepted this for v0.1.

Context length

• v0.1 was trained and evaluated at 8,192 tokens. That is the length behind every long-input number on this card.
• The UltraBERT encoder supports 32,768 positions, and nothing in the model stops you from passing longer inputs.
• Inputs longer than 8,192 tokens have not been tested with this checkpoint. If you use 16K or 32K windows, measure accuracy, memory and latency on your own data first.
• The shipped tokenizer config sets model_max_length to 8,192 so that the default matches what was trained. Pass a larger max_length yourself if you want to try longer inputs.

How to use

UltraBERT is supported natively in Transformers 5.20 and later, so no trust_remote_code is needed. All training and evaluation used transformers==5.20.2 and gliclass==0.1.21.

pip install "transformers>=5.20.2" "gliclass==0.1.21"

import torch
from gliclass import GLiClassModel, ZeroShotClassificationPipeline
from transformers import AutoTokenizer

repo = "Srimon12/ultragliclass-v0.1"
model = GLiClassModel.from_pretrained(repo)
tokenizer = AutoTokenizer.from_pretrained(repo, add_prefix_space=False)  # trained without a prefix space

pipe = ZeroShotClassificationPipeline(
    model, tokenizer,
    classification_type="multi-label",   # or "single-label"
    device="cuda:0" if torch.cuda.is_available() else "cpu",
    max_length=8192,                     # trained length; gliclass defaults to 1,024
    max_classes=256,                     # trained label count; gliclass defaults to 25
)

text = "The invoice total does not match the purchase order and the vendor is not on the approved list."
labels = ["approve", "reject", "escalate to finance"]
print(pipe(text, labels, threshold=0.5)[0])

Notes:

• Always pass max_length=8192 and max_classes=256 to the pipeline. The gliclass pipeline does not read these from the config, and its defaults (1,024 tokens, 25 labels) are lower than what the model was trained for.
• Load the tokenizer with add_prefix_space=False. That is how the model was trained and evaluated.
• Older Transformers versions (before 5.20) have not been tested.
• This is a classifier, not a Sentence Transformers embedding model.
• For fine-tuning, use the GLiClass train.py recipe (uni-encoder) with an encoder learning rate of about 5e-6 to 1e-5, a head learning rate about 2.5–3× the encoder rate, and fp16 or bf16.

Intended use

• A starting point for fine-tuning your own classification, intent, routing or decision tasks, especially when inputs are long.
• Zero-shot classification over label sets you define at run time, single-label or multi-label.
• Typed decisions (choice, yes/no, score) over long inputs such as documents, records or logs, up to the 8,192 tokens it was trained at.

Not recommended:

• High-stakes decisions without human review, such as medical, legal, credit or hiring decisions.
• Non-English text. Training and evaluation were English only.
• Inputs longer than 8,192 tokens without your own testing (see Context length).
• Reporting the typed-decisions score as zero-shot (see Limitations).

Model details

Training

All three stages ran on Kaggle with 2× T4, fp16 and an effective batch of 64 rows per step. The optimizer was AdamW (β 0.9/0.98, ε 1e-6) with weight decay 0.1 on the encoder, 0.01 on the heads and none on the RMSNorm gains, with a linear schedule. Each stage starts from the final model of the stage before it.

Data hygiene:

• Train rows whose text exactly matched a validation or eval text were dropped: 865 in Stage 2 and 774 in Stage 3.
• Stage 3 had 0 train texts in the long-Jev eval slice.
• The LocalLLaMA/typed-decisions test split was never read. A code guard stops the data build if it is.
• The suite's eval sets were blocked from training with a forbid_ids guard. Two of them still have close in-domain training data; they are marked below.

Stage 3 compute: about 7.27 Kaggle GPU-hours (smoke 1.16 + full 6.11), a median of about 24 s per step at 8,192 tokens, and 5.47 GB peak reserved memory per T4.

The full recipe, with configs, data-build scripts and data checksums, is in docs/TRAINING_RECIPE.md.

Evaluation

Protocol

• Full test splits of every set. cap_sotu has no test split, so its only split (23,040 rows) is used, as in the official GLiClass script. About 82k suite rows, plus 3,076 Banking77 rows and 4,500 CLINC-150 rows.
• Single-label softmax argmax. Macro-F1 for the suite, accuracy for the intent sets. Label strings are the ones in the official test_gliclass.py. Inputs up to 1,024 tokens (8,192 for long Jev), fp16.
• Each set is scored with 3 label orders (the canonical order plus 2 seeded shuffles, seed 0). Tables show the canonical-order score; the mean over orders is also given.
• Same harness and same GPU (Kaggle 2× T4) for all three models. Each model uses its authors' tokenizer setting: add_prefix_space=False for UltraGliClass and True for the two Knowledgator models. Calibration uses one global temperature fitted on 2,000 rows of calib/stage1_val.jsonl.
• Comparison models, pinned by Hub commit: knowledgator/gliclass-modern-base-v3.0 (ModernBERT-base, commit ac369222) and knowledgator/gliclass-base-v3.0 (DeBERTa-v3-base, commit 77a70e6c).
• Software: torch 2.11.0+cu128, transformers 5.20.2, gliclass 0.1.21, datasets 5.1.0.
• Paired confidence intervals are bootstrapped over the same rows. Intent and long-Jev p-values are exact McNemar tests.

Harness check. Our run reproduces the published averages of both comparison models on the same 14 sets: modern-base 0.5577 (published 0.5577, Δ −0.0000) and base-v3 0.6728 (published 0.6764, Δ −0.0036).

Zero-shot text classification (macro-F1, full test splits)

The v0.1 column is bold for readability, not because it is always highest.

¹ Not zero-shot for UltraGliClass. massive shares 56 of its 59 intents with HWU64, which was in Stage 2 training. financial_phrasebank uses the same 3 sentiment labels as the financial-tweets sentiment set, also in Stage 2 training. ² Leaves out massive and financial_phrasebank.

Where v0.1 is behind:

• vs modern-base: sst5 (−0.037) and spam (−0.009).
• vs base-v3: sst5 (−0.099), 20_newsgroups (−0.085), spam (−0.044), cap_sotu (−0.030), snips (−0.013), financial_phrasebank (−0.003), CR (−0.002).

Intent detection, all labels in one pass (accuracy, full test splits)

CLINC-150: before Stage 3 we set the bar "CLINC-150 ≥ Stage 2 (0.650)". On the full 4,500-row test split v0.1 scores 0.636 (95% CI 0.622–0.650), so it misses that bar. The earlier 500-row Stage 3 check gave 0.640. v0.1 is still ahead of both comparison models on CLINC-150 (+0.163 over modern-base, p = 6.1e-93; +0.028 over base-v3, p = 1.1e-4).

Long-context decisions: long Jev at 8,192 tokens (accuracy)

The eval slice (calib/jev_long_test.jsonl in the GitHub repo):

• Rows from Jev typed decisions (config full) whose state is at least 6,000 characters, taken from the test (177) and validation (254) splits. Training used only the train split. No state or group is shared with training, and Stage 3 had 0 train texts in this slice.
• 431 rows in 344 groups: choice 390, yes/no 28, score 13. By license_use: commercial 324, unspecified 83, non-commercial 24. These rows were used only for evaluation.
• Length with the UltraBERT tokenizer: median 3,463 tokens, p90 7,648, max 23,508.
• Not zero-shot for UltraGliClass (other Jev rows are in its training data); zero-shot for the other two models. Confidence intervals use a cluster bootstrap by state.

Breakdown (accuracy):

On the same slice, Stage 2 scored 0.552. Stage 3's long-context training added +0.111 (cluster 95% CI [+0.068, +0.157], McNemar exact p = 1.9e-7). This Stage 2 vs Stage 3 comparison comes from the training-time eval (stage3_longjev_compare.json).

Paired significance on the suite (same rows): v0.1 − other model

Other measurements (same run)

• How often the prediction changes across 3 label orders (v0.1 / modern-base / base-v3): Banking77 0.248 / 0.420 / 0.214, CLINC-150 0.198 / 0.496 / 0.278, massive 0.116 / 0.506 / 0.455, 20 Newsgroups 0.165 / 0.384 / 0.256.
• Calibration: mean ECE over the 14 suite sets after one global temperature is 0.114 for v0.1 (T = 1.81), 0.137 for modern-base (T = 3.19) and 0.161 for base-v3 (T = 3.63).

Earlier training-time checks (kept for history)

During training, go/no-go decisions used a smaller protocol: at most 500 examples per set, the same label strings, 1,024 tokens, fp16 on T4, with the modern-base baseline re-run on CPU in fp32. These numbers only show progress between stages. Use the full-split tables above for any comparison.

Typed decisions (LocalLLaMA/typed-decisions test, 400 cases / 2,000 decisions; training-time check, not re-run in the full evaluation)

v0.1 by workflow: customer_service 0.710 ³, agent_trace_observability 0.612 ³, security_incidents 0.584, invoice_processing 0.390. Stage 2 by workflow: 0.726 / 0.626 / 0.592 / 0.518.

³ In-domain. Stage 2 trained on 1,500 rows from each of these two workflows, taken from the dataset's train split. The overall typed score is therefore not zero-shot. Only security_incidents and invoice_processing are unseen workflows.

v0.1 is lower than Stage 2 on typed decisions (−0.042), mostly on invoice_processing (−0.128). One possible cause is the larger share of procedural-style rows in the Stage 3 data (about 13.9%, against about 4.9% in Stage 2). We have not confirmed this.

Latency (training-time check): median 5.8 ms per example, batched, fp16 on a T4, at 1,024 tokens.

Limitations

• CLINC-150 misses the Stage 3 bar: 0.636 on the full test split against a bar of 0.650 (the Stage 2 value on the 500-row protocol). Stage 2 was also slightly better on Banking77 and typed decisions in the training-time checks. v0.1 gives up these small short-input points for the long-context gain.
• Not better than gliclass-base-v3.0 on strictly zero-shot short text. On the 12 sets that are zero-shot for all models, v0.1 (0.663) ties base-v3 (0.668; Δ −0.005, 95% CI [−0.011, +0.000]). Part of the 14-set lead (+0.015) comes from the two in-domain sets.
• Typed decisions are partly in-domain (footnote ³), and the typed score went down from Stage 2. Do not report it as zero-shot.
• Two suite sets are not zero-shot (massive, financial_phrasebank), and the long-Jev slice is not zero-shot for v0.1. The 12-set average is given for this reason.
• Weak on fine-grained 5-way sentiment (sst5 0.239; base-v3 0.338) and on 20 Newsgroups (0.420; base-v3 0.505).
• Many-label tasks are sensitive to label order (prediction changes for 0.20 of CLINC-150 rows and 0.25 of Banking77 rows). Averaging logits over a few label orders helps.
• Long inputs: accuracy is lowest for states of 12k–16k characters (0.500). Inputs over 8,192 tokens are cut off, and cut-off rows score lower (0.583, n = 24). Nothing was tested beyond 8,192 tokens.
• UltraBERT was pre-trained in bf16. All our training and evaluation used fp16 with no NaN losses. bf16 inference with this checkpoint was not tested.
• Possible benchmark overlap: the upstream sources of gliclass-v2.0 are not documented, so overlap with eval domains cannot be ruled out. The official GLiClass cards carry the same caveat.
• English only. Small sets (CR n = 376, rotten_tomatoes n = 1,066) have wide per-set intervals, so a difference of 0.02–0.03 on one small set is within noise.

Licenses and training data

Release license: Apache-2.0 for the code and the weights, with the CC-BY-SA caveat below.

The CC-BY-SA caveat, step by step:

1. Stages 1 and 2: Jev rows were filtered to license_use == "commercial". That bucket still includes share-alike sources (CC-BY-SA-3.0 and 4.0). So Stage 1 (100,000 Jev rows) and Stage 2 (436,985 Jev rows) trained on some CC-BY-SA text.
2. Stage 3: the 21,184 new long-Jev rows are commercial-only and exclude every row whose license contains -sa, by-sa, gpl or odbl. That filter removed about 1.8k rows, so Stage 3 added no new share-alike or copyleft text.
3. The 40,000 Stage 3 replay rows were sampled from the Stage 2 mix, and v0.1 starts from the Stage 2 weights. So v0.1 is not free of the CC-BY-SA caveat.
4. Whether weights trained on CC-BY-SA text carry share-alike duties is not settled law. If this matters for your use, get legal advice. Jev rows labeled unspecified (284k) or non-commercial (115k) were never used for training; rows of every license_use value appear only in the long-Jev eval slice.

A later version trained without any share-alike data would remove this caveat. v0.1 does not.

Typed-decisions overlap: Stage 2 used 3,000 rows from the LocalLLaMA/typed-decisions train split (customer_service 1,500, agent_trace_observability 1,500). They share the test split's schema, workflows and 57 option strings, so typed results must be reported as partly in-domain.

Eval sets (SST, CR, IMDB, 20NG, Enron, AG News, emotion, cap_sotu, rotten_tomatoes, financial_phrasebank, CLINC, Banking77, MASSIVE, SNIPS) were used only for evaluation. Several of them have unclear or non-commercial licenses, and none were trained on.

Files

The weights and tokenizer.json are byte-identical to the Stage 3 training output. Two config values were changed for this release so the defaults match training: max_num_classes 50 → 256 in config.json, and max_length 1,024 → 8,192 and model_max_length 32,768 → 8,192 in tokenizer_config.json (a machine-local is_local key was also dropped). These changes do not affect the scores: the gliclass pipeline sizes label slots from the labels you pass and takes max_length as an argument. The original training-output hashes are config.json 748a02d905618b1dec94511bdcfc0a743e0f4748aedc16ca3f5af608a684785e and tokenizer_config.json 949f9464da03d1e6a54deb9d50ba9016304ee5c3d8fb0d5b89f52e2a82951387.

Where the numbers come from

Every number on this card is taken from a result file in the GitHub repo (paths relative to the repo root; index in docs/PROOFS.md):

• Full evaluation (main tables): proofs/hf_readme_eval/ — TABLES.md, STATUS.json (complete, nothing missing, 1.217 GPU-hours), summary.json (per-set scores, paired tests, long Jev), models.json (comparison-model commits), protocol.json, env.json, longjev/*.json (per-row long-Jev results, no text). Harness: scripts/hf_readme_eval.py. (The footnote in TABLES.md says the long-Jev rows are from the test split; the slice manifest calib/jev_long_test_manifest.json shows test 177 + validation 254, which is what this card states.)
• Stage 3 training-time checks, typed decisions, latency: proofs/stage_summaries/s3/stage3_final_quick.json
• Stage 3 vs Stage 2 on long Jev: proofs/stage_summaries/s3/stage3_final_longjev_8192.json, stage3_baseline_s2_longjev_8192.json, stage3_longjev_compare.json
• Stage 3 status and data: proofs/stage_summaries/s3/STATUS.json, stage3_full_summary_s1.json, stage3_smoke_summary_s1.json, stage3_data_manifest.json, stage3_data_check.json
• Stage 2: proofs/stage_summaries/s2/stage2_full_C_final_quick.json, stage2_full_data_manifest.json, STATUS.json, STATUS_session1.json
• Stage 1: proofs/stage_summaries/s1/stage1_ultrabert150m_quick.json, stage1_data_manifest.json, STATUS.json
• Compute: proofs/stage_summaries/gpu_hours_log.tsv
• ModernBERT training-time baseline (CPU): proofs/stage_summaries/baselines/stage1_fair_modernbert_v3_cpu.json
• Licenses: docs/DATASETS.md, configs/data_stage3.yaml
• Recipe: docs/TRAINING_RECIPE.md

Citation

@misc{danguria2026ultragliclass,
  title        = {UltraGliClass v0.1: GLiClass on UltraBERT-150M with long-context decision training},
  author       = {Danguria, Srimon},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/Srimon12/ultragliclass-v0.1}}
}

@misc{stepanov2025gliclass,
  title         = {GLiClass: Generalist Lightweight Model for Sequence Classification Tasks},
  author        = {Stepanov, Ihor and Shtopko, Mykhailo and Vodianytskyi, Dmytro and Lukashov, Oleksandr and Yavorskyi, Alexander and Yaroshenko, Mykyta},
  year          = {2025},
  eprint        = {2508.07662},
  archivePrefix = {arXiv}
}

@misc{park2026ultrabert,
  title         = {UltraBERT: Beyond the Limit of BERT Pre-Training},
  author        = {Park, Jungwoo and Lee, Taewhoo and Kang, Jaewoo},
  year          = {2026},
  eprint        = {2610.15028},
  archivePrefix = {arXiv}
}

If you use results on Jev typed decisions, please also cite the dataset as its card asks: tasksource/tasksource-jev-typed-decisions.