EditRank-Multidomain-4B (Research Preview)

Status: Research Preview. Not a production model.

• Do not use this model alone to automatically edit or publish content. Every selected edit needs external fact verification and human approval before use.
• The model can rank a fact-violating edit above the original. In the bundled example, a rewrite that invents a version number, a latency and a statistic scores +3.750, above the unchanged original at +3.312.
• Fact-violating top-1 rate: 10.3% on the documentation test set (automated judge).
• No effect on live AI search has been shown. All results come from simulated answer engines; improved exposure, citation or recommendation in ChatGPT, Gemini, Perplexity, Google AI Overviews or any other product is unverified.

At a glance

What it does

Given a search query, an original document and a revised version, the model outputs a score logit("yes") - logit("no") estimating whether the revised version would be ranked higher than the original as a source by an AI answer engine for that query, while keeping every fact of the original unchanged. Score every candidate, including the unchanged original, and take the highest. If the original wins, keep it. See example_inference.py for the exact prompt format; scores are only comparable within one query.

In the bundled example (bf16, one GPU), the model scores a faithful rewrite +3.938, a rewrite that invents a version number, a latency and a statistic +3.750, and the unchanged original +3.312. The fact-inventing rewrite outranks the original. This is the failure mode behind the unsafe rates below: the model is not a fact checker.

Results

Utility = mean rank gain of the selected candidate over the original document under a simulated answer engine (unsafe candidates receive a penalty below the worst candidate in the group). Keeping the original = 0. Two engines are reported separately and never averaged: a fixed in-house Qwen3.8-27B engine (the same model family that produced the training labels) and an independent Gemma-4-31B engine that produced no training label. Decisions used the Gemma engine. Comparisons were pre-registered with fixed thresholds before training; the model was trained once, with no hyperparameter search.

Technical documentation (184 held-out query groups, independent)

Query groups come from Kubernetes and Python documentation pages whose document families (versions, translations, sibling pages of one topic) were kept entirely out of training. Queries are synthetic (see Limitations).

Paired bootstrap vs EditRank-0.6B, mean difference [95% CI]:

Over-edit loss is the mean utility lost when the chosen edit is worse than leaving the document unchanged (max(0, u_original - u_selected)). Unsafe top-1 is the share of queries whose selected candidate an automated judge flagged as changing or adding facts; a stricter numeric/quote judge flags 4.9% (EditRank-0.6B: 6.5%).

Retail (624 query groups, development comparison)

These Amazon ESCI queries were already used in earlier decisions about this model family, so this is a development comparison, not an independent test.

Difference vs EditRank-0.6B on the Gemma engine: -0.007 [-0.092, +0.075]. Retail quality is at the level of the 0.6B model, not above it; an internal 4B variant trained on retail and video games only scored higher on retail (0.530) but did not improve on documentation (1.746) and almost never kept the original (3%).

Video games: not independently evaluated

There is no independent video-game test set. A 37-group exploratory set shares documents with training and has labels from the label engine only; it is not used for any claim. Video-game generalization is unverified.

Limitations

• Simulated engines only. All numbers come from two fixed open-weight engines, a fixed prompt and a fixed candidate set. They do not show improved exposure in ChatGPT, Gemini, Perplexity, Google AI Overviews or any other live product, and the documentation gain above must not be read as a visibility or traffic increase.
• Synthetic documentation queries. The documentation queries were generated from the documentation sections by an in-house open-weight Qwen3.8-27B model. They are not real user queries.
• The decision to keep the original is still weak. On retail groups where the original is strictly the best candidate (25 groups), the model kept it 20% of the time (EditRank-0.6B: 32%). The documentation test set has only 2 such groups, so it barely tests this decision.
• Not a fact checker. 10.3% unsafe top-1 on documentation and 12.8% on retail. The automated judge is itself imperfect, so the true rates are uncertain in both directions.
• Untested domains. SaaS and marketing content, news, other languages and long documents were not tested (documents are clipped to 1,500 characters, the original to 600; inputs truncated at 1,280 tokens). Video games: unverified.
• Some documentation sections start mid-context (for example with a code example) because of chunk boundaries.
• Answer engines change; rankings learned against these engines may not transfer.

Training

• Base: Qwen/Qwen3-Reranker-4B, Hub revision 22e683669bc0f0bd69640a1354a6d0aebcfeede5 (shard sha256 values in eval_manifest.json, checked against the Hub before training).
• LoRA r 16, alpha 32 on q/k/v/o/gate/up/down, lr 1e-4, 3 epochs, gradient accumulation 4, tau 0.25, bf16; epoch selected on dev top-1 utility (epoch 1). Merged into the base weights. One NVIDIA B200 GPU, 50 minutes. One run, hyperparameters fixed before training.
• Training groups: retail 400, video games 368, technical documentation 460 (dev: 45 / 31 / 46). Each group has the original document and 9 rewrites (structure, query alignment, fact-preserving restructuring, and strategies that can introduce unsupported facts). Candidates flagged as fact-violating get the lowest utility, so they are never the training target.
• Rewrites, rankings and documentation queries were produced with an in-house deployment of an open-weight Qwen3.8-27B model. No outputs of commercial LLM APIs were used.
• Training and evaluation data are not released.

Data sources and licenses

Attribution and changes. The Kubernetes and Python documentation was split into sections of 400 to 1,400 characters (long code blocks removed); synthetic queries and rewritten candidates were generated from those sections and used only as training and evaluation signal. Per-file paths, licenses and git blob hashes were recorded for all 844 source pages. No documentation text, query or rewrite is included in this repository, and neither the Kubernetes Authors nor the Python Software Foundation endorses this model.

This repository contains only model weights, configuration, tokenizer files, this card, eval_manifest.json and an example script. The license field (Apache-2.0) covers this repository's contents and follows the base model; it is not a legal determination about the upstream texts.

@inproceedings{puerto2025cseo,
  title={C-{SEO} Bench: Does Conversational {SEO} Work?},
  author={Puerto, Haritz and Gubri, Martin and Green, Tommaso and Oh, Seong Joon and Yun, Sangdoo},
  booktitle={NeurIPS 2025 Datasets and Benchmarks Track},
  year={2025}
}
@article{reddy2022shopping,
  title={Shopping Queries Dataset: A Large-Scale {ESCI} Benchmark for Improving Product Search},
  author={Reddy, Chandan K. and M{\`a}rquez, Llu{\'i}s and Valero, Fran and Rao, Nikhil and Zaragoza, Hugo and
          Bandyopadhyay, Sambaran and Biswas, Arnab and Xing, Anlu and Subbian, Karthik},
  journal={arXiv preprint arXiv:2206.06588},
  year={2022}
}

Inference cost

Retail set, 624 groups of 10 candidates, one B200, bf16: group latency p50 168 ms, p95 257 ms, about 56 candidates/s, peak memory 12.5 GiB. (Timing on the documentation set is not reported because an evaluation-file overwrite in the scoring job made that measurement run over a different group set; documentation scores were unaffected.)

Release verification

Before publication, this repository's files were downloaded from the private Hub revision in a separate process with an empty cache and checked offline on one NVIDIA B200 GPU against the original training checkpoint and the frozen scores behind the pre-registered verdict. The verdict itself was not recomputed. Every check passed:

The sha256 values of the model files are listed in eval_manifest.json.