🇰🇷 한국어 카드: MODEL_CARD.ko.md

Qwen3-1.7B-Base-CareQ-2.4bpw

Qwen3-1.7B-Base (1,720,574,976 parameters) compressed to 2.397 bits per weight: one 515,564,507-byte file, about 15% of the 3.44 GB BF16 weights. Every linear layer is a binary low-rank core plus a small low-rank residual. The model was recovered by quantization-aware distillation from the BF16 base, with an FP32 master copy of the trainable parameters.

On the sealed GSM8K test set (1,319 questions, 5-shot, strict match, run once) it scores 23.05%. Our earlier 2.394-BPW checkpoint scored 14.86% and our 2.785-BPW checkpoint 20.09%. The BF16 base scores 67.7%, so this is a research checkpoint for studying the ultra-low-bit regime. It is not a drop-in replacement for the base model.

⚠️ Custom format, custom loader. The weights are stored in a packed research format (careq-v1). Stock transformers, vLLM and llama.cpp cannot read it. Use the loader in this repository (§1). There is no fused kernel yet, so inference is slower than a dense model of the same size.

0. Results

All numbers below were measured by us with one harness: lm-eval 0.4.9, GSM8K 5-shot greedy strict-match, and ARC/HellaSwag/PIQA 0-shot. Every quantized model was loaded from its packed file only. Where a row has no number, we did not measure it with this harness.

Paired comparisons (bootstrap over questions, 10,000 resamples):

The K3 row is the controlled comparison. K3 started from the same checkpoint and saw the same data in the same order; the difference is that K3 kept its trainable parameters in BF16. The P3 and P2-A rows compare final models whose starting points, rank allocation and parameter counts differ, so they do not isolate the effect of the FP32 master.

Other tasks for this model (0-shot, acc_norm): ARC-Challenge 0.319, HellaSwag 0.493, PIQA 0.670. The BF16 base scores 0.453, 0.665 and 0.754 on the same harness. WikiText-2 PPL 13.63, teacher KL 0.477.

How the test number was produced

The model and file (SHA-256 below) were fixed before the test was run. The test used the same strict protocol as the P3 and P2-A test runs, and it was run exactly once. The result was not used for any further training or model selection. GSM8K DEV is a separate 256-question split held out from training, and it is the set the model was promoted on.

Generation structure

We counted three failure shapes in the greedy 5-shot generations. final means the output contains a #### answer line. loop means some 8-gram appears three or more times. early means the output is shorter than 16 tokens.

On DEV the gap to K3 is not significant (loop +2.3 pp, CI [−1.6, +6.6], McNemar p = 0.35; final −3.1 pp, CI [−7.4, +1.2], p = 0.23). Loops still cost real answers: 107 test generations loop and only 4 of them are correct, and none of the 128 test generations without a final line is correct. Earlier test runs (P3, P2-A) did not store generations, so there is no test-set loop baseline for them.

Examples (real test outputs, greedy)

Correct, GSM8K test doc 1 (robe):

The robe takes 2 bolts of blue fiber and 2/2 = 1 bolt of white fiber.
So, the robe takes 2 + 1 = 3 bolts of fiber in total.
#### 3

Correct, GSM8K test doc 23 (candle):

The candle burns for 4 hours, so it burns 4 x 2 = 8 centimeters.
The candle is 8 centimeters shorter than it was when it was first lit.
#### 8

Failure, GSM8K test doc 10. The model starts a calculator annotation and repeats it until the 256-token limit:

The number of downloads in the third month was 180-30% = <<180-30/100=180-30/100=180-30/100=...

Failure, GSM8K test doc 12. The model repeats one sentence:

He will have to pay $3 a year to water and feed the tree, so he will have to pay $3 a year
for that. So he will have to pay $3 a year for water and feed the tree, so he will have to
pay $3 a year for water and feed the tree, ...

1. How to use

Tested environment: Python 3.12, CUDA GPU, torch 2.9.0, transformers 4.51.3, and NanoQuant at commit a9e0a430881ff80d83b622c3129e330dc33c04f5 with the one-line patch in patches/. The patch only touches NanoQuant's BF16 loader, which this model does not use.

git clone https://huggingface.co/ThakiCloud/Qwen3-1.7B-Base-CareQ-2.4bpw && cd Qwen3-1.7B-Base-CareQ-2.4bpw
pip install -r requirements-inference.txt
git clone https://github.com/SamsungLabs/NanoQuant.git && cd NanoQuant
git checkout a9e0a430881ff80d83b622c3129e330dc33c04f5
git apply ../patches/nanoquant-a9e0a430-torch_dtype.patch.txt && pip install --no-deps -e . && cd ..
python load_careq.py .        # loads, checks SHA-256, runs one greedy generation

From Python:

import sys, torch
sys.path.insert(0, ".")                 # the cloned repository
from load_careq import load

model, tok, info = load(".", device="cuda")
prompt = "Question: Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?\nAnswer:"
x = tok(prompt, return_tensors="pt").input_ids.to(model.device)
with torch.no_grad():
    y = model.generate(x, max_new_tokens=256, do_sample=False, pad_token_id=tok.eos_token_id)
print(tok.decode(y[0, x.shape[1]:], skip_special_tokens=True))

Decoding: every number in this card is greedy (do_sample=False). GSM8K used the lm-eval 5-shot prompt with a 256-token limit. This is a base model with no instruction tuning or chat template. Prompt it with plain text or few-shot examples, and stop on \n\nQuestion: or on the #### line.

model.careq.pt is a torch pickle. load() first opens it with weights_only=True, which refuses any pickle that references code, and then builds the model. It also checks the SHA-256 before loading.

2. Method

Each of the 196 linear layers (28 blocks × q, k, v, o, gate, up, down) is represented as

y = NanoQuant(x) + F.linear(F.linear(x, rb), ra)

NanoQuant(x) is a binary low-rank core: sign matrices packed one bit per entry, with per-row and per-column scales. ra · rb is a low-rank residual. The tied embedding and LM head are a 4-bit embedding plus a rank-56 residual. The file breaks down as follows.

Effective BPW is file bytes × 8 / 1,720,574,976 = 2.397. It counts everything, including the embedding.

Training was quantization-aware distillation from the BF16 base:

• Start: the binary low-rank initialization from NanoQuant, then 1,500 steps of an earlier recovery run (K3, BF16 master). This model branches from step 1,500 and continues.
• Trainable: only the scales, the layer residuals and the embedding residual. The binary signs stay frozen.
• FP32 master: the 196 middle scales and both residual factors (16.5 M values) are kept and updated in FP32, then cast for the forward pass. This is the only change against K3.
• Loss: KL to the BF16 teacher (weight 1.0), next-token cross-entropy (0.1) and hidden-state matching (0.1). AdamW (betas 0.9, 0.95, no weight decay), learning rate 1e-4 for scales and 5e-5 for residuals.
• Length: 9,156 steps in total (steps 1,500 to 9,156 took 5.0 hours on one B200), seed 0. No non-finite steps.
• Data: a mix of fineweb-edu, finemath, opc-annealing and math word-problem data (§4). GSM8K DEV and test were never trained on.

Before the run we registered the promotion gates, and the model passed all six on DEV: BPW ≤ 2.400, packed-only integrity 9/9, S3 SHA read-back, C4 PPL ≤ 26.05, held-out KL ≤ 0.549 and DEV strict ≥ 37/256.

3. Files

4. Data and license

• License: Apache-2.0, the same as Qwen3-1.7B-Base and NanoQuant.
• Training data is not included in this repository and is not redistributed.
• Sources used for training: fineweb-edu, finemath and opc-annealing (ODC-By; finemath is also subject to the CommonCrawl terms of use), GSM8K train and MATH train (MIT), and kuotient/orca-math-word-problems-193k-korean (CC-BY-SA 4.0), a Korean machine translation of microsoft/orca-math-word-problems-200k (MIT; answers generated by GPT-4-Turbo). 18,809 filtered examples (8.06M tokens) of it were used, so part of the math training text is Korean. An earlier version of this card misidentified this source as the English original; we confirmed the actual source by exact question+answer match against revision 9d2edbed8d73.
• A Qwen3.8-27B model was used only to check answers during data filtering. It did not generate training text.

5. Limitations

• Far below the base model on reasoning: 23.05% against 67.7% on GSM8K test, and lower on ARC, HellaSwag and PIQA.
• About 8% of greedy generations loop until the token limit, and those are almost never correct. Use a repetition stop or a small max_new_tokens.
• No instruction tuning or alignment. Outputs are base-model continuations and are not safety-filtered.
• English math evaluation only. We did not measure other languages or tasks.
• One seed (0). The pre-registered design forbids further seed, rank or data search, so we claim no spread across seeds.
• Custom format and no fused kernel. We have not published speed or memory numbers.
• Logits depend slightly on the torch/CUDA build. Re-loading this file with the same build (torch 2.9.0+cu130) gave bit-identical logits. With the PyPI build (2.9.0+cu128) greedy outputs and argmax were identical on our probes, but logits differed by up to 0.31 (about two bf16 steps at logit size 28).

6. Citation

@misc{thakicloud2026careq24,
  title  = {Qwen3-1.7B-Base-CareQ-2.4bpw: binary low-rank quantization with FP32-master distillation},
  author = {ThakiCloud},
  year   = {2026},
  url    = {https://huggingface.co/ThakiCloud/Qwen3-1.7B-Base-CareQ-2.4bpw}
}