llmon-span-tagger

Model Summary: llmon-span-tagger is a 149M-parameter token-classification model finetuned from ModernBERT-base that splits a natural-language prompt into typed spans (instruction, constraint and data) so the prompt can be rendered as LLMON (LLM Object Notation) markup. It tags every token with a BIO label and decodes spans from runs of tags. Its span boundaries therefore do not depend on punctuation, and it can cut inside a single sentence: "Write a poem about dogs | using less than 50 words" is one instruction and one constraint. It is the default backend of the lang.unstructured_text segmenter in the LLMON repository.

• Developers: IBM Research
• GitHub Repository: generative-computing/llmon (code in lang/unstructured_text)
• Paper: LLMON: An LLM-native Markup Language to Leverage Structure and Semantics at the LLM Interface
• Release Date: November 12, 2026
• License: Apache 2.0
• Supported Languages: English only.

Intended use: The model converts prompts written as plain text into LLMON for models trained to consume it, such as Granite-4.1-3B-LLMON. It is meant for prompts whose author you trust, and is useful for:

• Converting existing prompts, prompt libraries or instruction datasets into LLMON.
• Inspecting how a prompt decomposes into instructions, constraints and data, for example while learning LLMON or debugging a prompt.
• Research on prompt structure and instruction/constraint segmentation.

Out of scope: The model is not a security control. It must not be used to decide which parts of untrusted input are trustworthy; see Limitations. It is also not intended for non-English text, for general-purpose named-entity or document tagging, or for prompts longer than 2,048 tokens.

Label scheme

The head predicts 7 BIO tags: O, plus B-/I- for each of three classes. Each class renders to an LLMON tag:

O marks tokens that belong to no span, such as inter-span whitespace and trailing punctuation, which annotators left unlabeled.

Annotations nest; the most common case is a Constraint inside an Instruction. A token inside several spans was trained on its innermost span, which is what teaches the model to split within a sentence. The model's output is a flat, non-overlapping list of spans. The authoritative class definitions are in docs/guidelines.html.

Usage

The checkpoint is a fine-tuned encoder plus a separate linear head. It is not a transformers AutoModelForTokenClassification checkpoint, and pipeline("token-classification") will not load it. Use the llmon package, which also applies the post-processing the reported scores were measured with. The standalone snippet further down loads the model without it.

With the llmon package (recommended)

pip install "llmon[unstructured-text] @ git+https://github.com/generative-computing/llmon"

from lang.unstructured_text import segment

result = segment(
    "You are a pirate. Write a poem about dogs using less than 50 words. "
    "Do not mention cats.",
    model="ibm-research/llmon-span-tagger",
)

for span in result.spans:
    print(f"{span.label:<12} {span.confidence:.2f}  {span.text}")

constraint   0.87  You are a pirate
instruction  0.97  Write a poem about dogs
constraint   0.91  using less than 50 words
constraint   0.99  Do not mention cats

The same spans can be rendered as LLMON (with the special tokens a model reads) or as human-readable Shorthand:

>>> print(result.to_short())
\constraint:a\ You are a pirate /constraint:a/
\instr:a\ Write a poem about dogs /instr:a/
\constraint:b\ using less than 50 words /constraint:b/
\constraint:c\ Do not mention cats /constraint:c/

>>> print(result.to_llmon())
<|open|>constraint<|:|>a<|close|> You are a pirate <|open_end|>constraint<|:|>a<|close|>
...

Every span satisfies prompt[span.start:span.end] == span.text. Each span also carries a confidence (the mean of the per-token maximum probabilities) and a requires_review flag, which is set when confidence is below 0.60. A command-line tool is included as well:

export LLMON_SEG_TAGGER_REPO=ibm-research/llmon-span-tagger
echo "Summarize this article in 3 bullets. Do not use commas." | llmon-segment --format spans
llmon-segment --repl        # interactive; the model stays loaded

Standalone, with transformers only

This snippet produces the raw per-token tags without the post-processing described below, so expect more fragmented spans than segment() returns.

import torch
from huggingface_hub import snapshot_download
from torch import nn
from transformers import AutoModel, AutoTokenizer

path = snapshot_download("ibm-research/llmon-span-tagger")
tokenizer = AutoTokenizer.from_pretrained(path)
encoder = AutoModel.from_pretrained(path, subfolder="encoder").eval()

state = torch.load(f"{path}/head.pt", map_location="cpu", weights_only=True)
tags = state["tags"]  # ['O', 'B-instruction', 'I-instruction', ...]
head = nn.Linear(encoder.config.hidden_size, len(tags))
head.load_state_dict(state["head"])

text = "Write a poem about dogs using less than 50 words."
enc = tokenizer(text, return_tensors="pt", return_offsets_mapping=True,
                truncation=True, max_length=2048)
offsets = enc.pop("offset_mapping")[0].tolist()
with torch.no_grad():
    pred = head(encoder(**enc).last_hidden_state)[0].argmax(-1).tolist()

for (start, end), p in zip(offsets, pred):
    if end > start:  # skip special tokens
        print(repr(text[start:end]), tags[p])

Repository files

Post-processing

segment() applies two measured defaults on top of the raw decode. Both were swept on the holdout across three training seeds:

The requires_review gate (confidence < 0.60) was calibrated on the same holdout. It flags 17% of spans, and 97% of the flagged spans are wrong. All thresholds can be changed through LLMON_SEG_* settings; see the repository's .env.example.

Evaluation

All figures come from a single released checkpoint, evaluated on a 134-prompt holdout stratified across the four source benchmarks. Spans are micro-averaged over the three classes and scored against top-level (outermost) ground truth. The baseline is the repository's dependency-free, regex-based rules backend.

Span accuracy (three-class)

Per-class F1

Boundary recall. A predicted cut within 3 characters of a ground-truth boundary counts. Mid-sentence boundaries are about 31% of all boundaries.

Read these figures with the following caveats:

• Seed variance. Training seed alone moved holdout scores by up to 8.8 points. Each figure above is one sample, not a mean.
• The holdout was reused. The same holdout was used to tune the post-processing thresholds and, following the project's documented procedure, to choose the released checkpoint among training seeds. The scores are therefore likely optimistic for unseen prompts.
• Small, non-public test set. 134 prompts (535 labeled spans), not distributed. Running the evaluation tooling on your own data measures your data; it does not reproduce these numbers.
• Over-splitting. The model predicts 665 spans against 535 ground-truth labels (+24%), which lowers precision. This is the main open accuracy problem.

Inference costs roughly 100 ms per prompt on CPU. Full tables and methodology are in docs/RESULTS.md.

Training Data

The model was trained on 1,353 prompts (~6,000 spans) taken from four public benchmarks and annotated with span labels for this project:

Prompts were annotated in a nine-class taxonomy following the project's annotation guidelines, then collapsed to three classes when loaded. The data was split into 1,219 training prompts and the 134-prompt holdout, stratified by source. The annotations are not distributed. The prompt text remains subject to each benchmark's original terms.

Training Procedure

The encoder and head were fine-tuned together, with full-parameter updates, using token-level cross-entropy over the BIO tags (special tokens excluded from the loss).

The training, evaluation and threshold-sweep code is open source. docs/TRAINING.md covers retraining on your own corpus, and docs/CORPUS.md gives the corpus schema and a browser-based annotation tool.

Infrastructure

We trained llmon-span-tagger on IBM's computing cluster, BlueVela, the same cluster used for the Granite-based LLMON models. Each training run used a single NVIDIA H100 80GB GPU.

Limitations

• Do not run it over untrusted content to decide what is trusted. In LLMON, an instr block is treated as authoritative by a downstream model. If this tagger labels attacker-controlled text as an instruction, LLMON rendering promotes that text to a command. Apply the tagger only to text whose author you trust. Wrap retrieved documents, user uploads and tool results as data in your application; do not ask the tagger to classify them.
• The injection check is a tripwire, not a defense. The model has no notion of prompt injection. The llmon package runs seven English keyword patterns alongside it and refuses to render when one matches. A paraphrase, another language or an encoding gets through. A negative result is not evidence of safety.
• Text past 2,048 tokens is silently dropped. Input is truncated to 2,048 tokens, and rendering emits only labeled spans. Anything past the limit is missing from the LLMON output, and no error is raised. Check the input length, or split long prompts, before rendering.
• Narrow training distribution. Training prompts come from four English instruction-following and QA benchmarks. System prompts, agent and tool-use prompts, multi-turn conversations, code-heavy prompts and conversational user writing are under-represented, so expect lower accuracy on them.
• Three classes, flat output. The model cannot represent the nine-class distinctions (for example Constraint vs. Output Format) or nested spans.
• Over-splitting and boundary errors. The model produces about 24% more spans than the ground truth, and constraint is its weakest class.
• Confidence is a heuristic. Span confidence is the mean of per-token maximum softmax probabilities. It ranks errors well on the holdout but is not a calibrated probability, and the 0.60 gate may not transfer to other data.

Ethical Considerations and Risks

The model inherits the considerations of its base model, ModernBERT-base, including biases present in its pre-training data. Its fine-tuning data comes from public benchmarks and was not collected from users. The main risk specific to this model is misuse as a security control (see Limitations): a mislabeled span changes what a downstream LLM treats as an instruction. We urge the community to use this model responsibly, to keep untrusted content out of its input, and to evaluate it on prompts representative of their own deployment before relying on it.

Citation

@misc{hind2026llmon,
  title         = {LLMON: An LLM-native Markup Language to Leverage Structure and Semantics at the LLM Interface},
  author        = {Hind, Michael and Shbita, Basel and Wu, Bo and Ahmed, Farhan and DeLuca, Chad and Fulton, Nathan and Cox, David and Gutfreund, Dan},
  year          = {2026},
  eprint        = {2603.22519},
  archivePrefix = {arXiv},
  primaryClass  = {cs.SE},
  url           = {https://arxiv.org/abs/2603.22519}
}