ECP-Q7-27B-Research

🇰🇷 한국어 카드: MODEL_CARD.ko.md

Research Preview · Not Safety Certified. This adapter does not always refuse when evidence is insufficient (see the known failure below and the unsupported-answer rates). Do not use it as a safety control.

Research preview — not a certified risk control. An evidence-decision adapter: a 2,162,688-parameter LoRA adapter (r = 8 on the MLP projections of decoders 44–47 of Qwen/Qwen3.8-27B) plus a small action head read at layer 48. It decides, after retrieval and before generation, whether the passages support an answer (Answer), more retrieval is needed (Retrieve), or the system should stop (Stop). The stack is truncated after decoder 47, so a decision costs a partial forward pass. With the two heads (20,484 parameters) the release is 2,183,172 parameters in four files, one per training seed. All four seeds are released; none was picked as "best".

How to use

ecp_q7.py is required: this is not a drop-in PEFT adapter (it truncates the stack and reads a hidden state). The card's numbers are tied to one runtime. Install it exactly:

pip install --index-url https://download.pytorch.org/whl/cu130 --extra-index-url https://pypi.org/simple -r requirements.txt

That is torch 2.9.0 built for CUDA 13.0 with flash-linear-attention 0.5.2 (and no causal-conv1d). In a clean environment built this way the loader reproduced the reference probabilities bit-for-bit. Other runtimes also work but give different probabilities: on 64 development rows per seed, torch 2.9.0+cu128 without flash-linear-attention moved probabilities by up to .144 and with it by up to .094, with no decision flipped in that sample.

from ecp_q7 import ECPQ7
m = ECPQ7.from_pretrained("Qwen/Qwen3.8-27B (local dir)", "ecp_q7_s1.safetensors", expected_sha256="<MANIFEST.json>")
d = m.decide("Which river flows through the city where Ada Lin was born?",
             [{"title": "Ada Lin", "text": "Ada Lin is a painter. She was born in Varden."},
              {"title": "Varden", "text": "Varden is a town on the river Osk."}], retrieval_available=True)
print(d.action, d.probs)   # ANSWER / RETRIEVE_MORE / ABSTAIN

decide() never generates text. Load the unmodified backbone separately if you also want an answer. examples/public_inputs.json holds six synthetic inputs and examples/expected_outputs.json the outputs of all four seeds in the runtime above. Example 6 is a known miss: every seed answers although the second passage describes a different Ines Varga (a painter) and says nothing about an instrument.

Results (development split only)

All numbers are on the 1,436-base MuSiQue-derived development split, which was also used to select each seed's checkpoint — they are optimistic. The sealed test and calibration splits have not been read for this adapter. Four seeds; the seed-mean is the headline, not the best seed. Numbers below are produced exactly the way the loader runs: one input per forward pass, no padding (batch = 1).

* Best supported coverage when the Answer margin threshold is swept on this same split so that UAR ≤ 5% — an in-sample diagnostic, not a calibrated threshold.

Historical A41 Protocol (right-padded batches of 8, same checkpoints): seed-mean accuracy .928, UAR .066. Batch=1 minus A41: -.001 [-.002, .000]; prediction agreement .995. Checkpoint selection used the A41 protocol and was not redone.

Reference on the same development split, from a different model and a different protocol (label-token logits at the last position, left-padded batches of 8), so not a direct batch = 1 comparison: the full LoRA fine-tune RAG-Gate-27B reaches .957; zero-shot Qwen3.8-27B .793. This adapter is lighter and shared across backbones; it is not more accurate. Under the A41 protocol, compared with its predecessor recipe (Q5) the accuracy change is .000 [-.007, .008] — no detectable change.

Zero-shot transfer to derived 27B checkpoints

Trained on Qwen3.8-27B only, applied unchanged (seed-mean, 4 seeds, paired bootstrap over 110 components). These are A41-protocol numbers (right-padded batches of 8); they were not re-measured at batch = 1.

Derivative-A to -D are fine-tuned Qwen3.8-27B checkpoints with identical geometry; their names are withheld, so this transfer result cannot be reproduced from public weights. † Derivative-D's lineage is not fully verified: the number is a measurement, not a compatibility claim. The derived checkpoints share Qwen's structure and sit close to it at layer 48 (prediction agreement ≥ .958), so this shows transfer within one model family, not across architectures.

Limitations

• Known failure: in synthetic example 6 the second passage is about a different person with the same name and has no instrument, yet all four seeds choose Answer. Insufficient evidence is not always refused.
• Development-split numbers only; selection used the same split. No sealed-test result exists.
• Unsupported-answer rate (UAR) is above 5% for 3 of 4 seeds; this is not a 5% risk guarantee. No independent statistical upper bound on UAR (e.g. on a held-out calibration split) has been certified.
• English, synthetic MuSiQue-derived passages. Korean enterprise documents are untested.
• ChainCheck numbers come from MuSiQue-derived pairs only; no independent replication exists yet (an attempt to build one from 2WikiMultihopQA failed its independence gates and produced no data).
• Shared vLLM serving of the truncated readout is not implemented; inference needs the custom loader. No product serving validation has been done.
• Reported probabilities depend on the runtime (CUDA build of torch, presence of flash-linear-attention); see How to use. ecp_q7.py emits a RuntimeWarning when the runtime differs from the verified one.
• The v0.1 loader officially supports batch = 1 only. Shape-dependent numerical differences were observed: padded batches changed probabilities by up to .066 (no decision flipped in the diagnostic sample). Changing only the neighbouring rows at a fixed shape changed nothing in that sample. The root cause (kernel, masking or otherwise) has not been identified.

License and data

The adapter, heads and ecp_q7.py are released under Apache-2.0, the same license as the Qwen/Qwen3.8-27B backbone (not included; download it separately). Training and development data were derived from MuSiQue (Trivedi et al., 2022), distributed under CC BY 4.0; the dgslibisey/MuSiQue Hub entry is a mirror without its own license file. ChainCheck is released under a mixed CC BY 4.0 / Apache-2.0 license. No training or evaluation rows are included in this repository; examples/public_inputs.json contains only synthetic inputs written for this card.