BeeNara Big 🐝 v0.5
Suggest the right folder in a whole folder tree, or say that none fits.
BeeNara Big reads a document (its text and, if available, the page image) together with the complete folder tree of a user (from 3 to several hundred folders) and returns a calibrated probability for every folder and for "none fits". It is the large sibling of BeeNara, which decides between at most 9 folders on a CPU.
Status: sorts on its own in trees of up to 49 folders, suggests in larger ones. For trees of 3–49 folders v0.5 has risk thresholds for automatic sorting (target: at most 1 % wrong among automatic decisions, see Results). For larger trees the calibration found no such threshold, so v0.5 suggests folders there and the user confirms.
What is new in v0.5
- More accurate on the same held-out tasks: 87.4 % of 2,400 tasks, v0.4 83.6 % on the same tasks. The gain is largest in trees of 50–199 folders (90 % instead of 84 %).
- Sorts on its own in larger trees. v0.4 had a risk threshold for 3–9 folders only. v0.5 has thresholds for 3–9, 10–24 and 25–49 folders and sorts 72 %, 57 % and 49 % of those tasks automatically; 0.3 %, 0.9 % and 1.0 % of them were wrong on the benchmark.
- Recognizes a missing folder more often: "none fits" recall 85.8 % (v0.4: 75.7 % on the same tasks).
- Knows which part of a tree a document belongs to. Many trees have several areas with
similar subfolders, for example a household and a company, each with
Steuer/2025. In the earlier training data the document often did not say whose it was, so the model learned to guess. In v0.5 every area has an identity (a person, a family, a firm, an association), the documents name their owner, and the top folder of an area carries that name in about 60 % of the areas. - Asks when the text cannot decide. Where the tree does not name its areas and a document fits two folders equally well, v0.5 was trained to split the probability between them. Its median confidence on such tasks is 0.50 (v0.4: 0.72), so the risk rule does not sort them automatically: 1 of 152 such benchmark tasks was sorted automatically (correctly).
- A third teacher model. Gemma 4 rewrote the owner details of the existing documents; the documents themselves are those of v0.4.
Quickstart
pip install torch "transformers>=5.17" peft safetensors tokenizers huggingface_hub pillow
hf download Kwokou/BeeNara-Big beenara_big.py --local-dir .
from beenara_big import BeeNaraBig
bb = BeeNaraBig.from_pretrained("Kwokou/BeeNara-Big") # adapter 164 MB + Qwen3.5-2B-Base 4.6 GB, GPU if available
s = bb.suggest(
"Einkommensteuerbescheid 2025\nFinanzamt Musterstadt\nFestgesetzte Einkommensteuer: 1.234,00 EUR",
["Finanzen/Steuer/2024", "Finanzen/Steuer/2025", "Finanzen/Bank", "Wohnung/Miete", "Arbeit/Verträge"],
)
s.suggestions # [('Finanzen/Steuer/2025', ...), ('Finanzen/Steuer/2024', ...), ...]
s.none # p("none of these folders fits")
s.category # the folder, when the risk threshold allows automatic sorting; else None
s.ask # True: show the suggestions and let the user decide
- Folder names are free text; paths like
Finanzen/Steuer/2025work best. A description can follow the name:"Wohnung/Miete — Mietvertrag, Nebenkosten". - If a tree has several areas (two people, a household and a firm), name them in the top
folders, for example
Familie Berger/Steuer/2025andBerger GmbH/Steuer/2025. Then the model can tell them apart; otherwise it asks. images=[...]adds page images (file paths or PIL images);language="en"switches the internal prompt to English.- On a GPU the base model loads in bf16, on a CPU in fp32.
Earlier versions stay available under the tags v0.4 and v0.3.
Key points
- Whole folder trees. One forward pass over the complete tree, with folder paths such as
Finanzen/Steuer/2025. In our tests a tree of 644 folders still fit into the context; one of about 750 folders did not. - Best up to about 200 folders: 89–95 % accuracy on held-out trees with 3–199 folders, 66 % for trees of 200 folders and more.
- Reads the page. The page image (letterhead, logo, layout) is used together with the text in half of the training tasks; it also works with text only.
- Calibrated. Expected calibration error 0.019.
- Stable under reordering. Shuffling the folder list changes the decision in 2.3 % of the tested tasks.
Results
Held-out "world": folder trees, folder names and documents that were never used for training. 2,400 tasks, 400 per list size, including tasks where no folder fits, where only a similar neighbour folder is present, and where the parent folder is the right answer. Calibrated, PyTorch bf16. v0.5 was trained twice with different seeds; this repository ships seed 1, chosen on the separate selection set, not on this benchmark.
The held-out documents now name their owner, like the training documents. The tasks therefore differ from those in the v0.4 card. The v0.4 column below is the shipped v0.4 model with its own calibration, evaluated again on the new tasks.
| Folders in the tree | Tasks | Accuracy, seed 1 (shipped) | Accuracy, seed 2 | v0.4, same tasks | Sorted automatically, seed 1 (wrong) |
|---|---|---|---|---|---|
| 3–9 | 400 | 95.2 % | 95.0 % | 94.8 % | 71.8 % (0.3 %) |
| 10–24 | 400 | 93.5 % | 92.0 % | 90.2 % | 57.0 % (0.9 %) |
| 25–49 | 400 | 89.2 % | 88.5 % | 89.5 % | 49.2 % (1.0 %) |
| 50–99 | 400 | 90.5 % | 89.8 % | 84.5 % | – |
| 100–199 | 400 | 90.2 % | 87.5 % | 83.5 % | – |
| 200 and more | 400 | 65.8 % | 65.2 % | 59.0 % | – |
| All | 2,400 | 87.4 % | 86.3 % | 83.6 % |
"–": the calibration found no threshold for this list size; the model suggests, the user confirms. v0.4 sorts 59.2 % of the tasks with 3–9 folders automatically (none wrong) and none in larger trees.
The thresholds beyond 24 folders depend on the seed: for 25–49 folders the shipped seed is at the limit of the target on the benchmark (2 of 197 automatic decisions wrong), and seed 2 found no threshold there (but one for 50–99 folders: 35 % automatic, none wrong). Treat automatic sorting in trees of 25–49 folders as less certain than in smaller ones.
| Metric | Seed 1 (shipped) | Seed 2 | v0.4, same tasks |
|---|---|---|---|
| "None fits" recall | 85.8 % | 83.8 % | 75.7 % |
| "None fits" precision | 75.4 % | 73.2 % | 73.3 % |
| Wrongly said "none fits" | 8.5 % | 9.3 % | 8.3 % |
| Calibration error (ECE) | 0.019 | 0.030 | 0.026 |
| AUROC, right vs. wrong answer | 0.91 | 0.91 | 0.87 |
| Accuracy, German / English | 86.9 % / 88.1 % | 86.2 % / 86.5 % | 83.7 % / 83.4 % |
| Decision changes when the folder order is shuffled (300 tasks, 3–9 folders) | 2.3 % | 1.0 % | not measured |
Clear and ambiguous tasks. In 152 of the 2,400 tasks (6.3 %) the list holds a folder of another area into which the document fits just as well, judged by everything written in it. There is no single right answer for these; the benchmark still counts only the folder the document was generated for.
| Seed 1 (shipped) | Seed 2 | v0.4, same tasks | |
|---|---|---|---|
| Accuracy, clear tasks (2,248) | 90.9 % | 89.9 % | 85.3 % |
| Ambiguous tasks (152): first suggestion is the reference folder | 35.5 % | 32.9 % | 58.6 % |
| … reference folder among the first two suggestions | 82.2 % | 78.9 % | 82.2 % |
| … median confidence | 0.50 | 0.53 | 0.72 |
| … sorted automatically | 1 (correct) | 0 | 0 |
On ambiguous tasks v0.5 puts both candidates first with about half the probability each, so its first suggestion is the reference folder less often than in v0.4. That is intended: it does not sort these documents on its own. Most ambiguous tasks (126 of 152) come from trees of 50 folders and more, where no version sorts automatically; only 26 fall into list sizes with a threshold.
Speed and memory.
| GPU memory, bf16, 644 folders (16,250 tokens) | 6.0 GB |
| GPU memory, 8-bit weights, 644 folders | 4.5 GB |
| CPU, 4 threads, 7 folders | 2.8 s |
| CPU, 4 threads, 43 folders | 21 s |
| CPU, 4 threads, 112 folders, with page image | 128 s |
| CPU, 4 threads, 630 folders | 195 s |
On a CPU it is too slow for interactive use; plan for a GPU. CPU times depend strongly on the page image and the length of the folder names. In PollySort a laptop GPU (RTX PRO 1000, 8 GB) needed 0.12–0.19 s per document with the v0.3 weights; v0.5 has the same architecture and size.
How it works
- Base: Qwen3.5-2B-Base (revision
b1485b2f), a hybrid model with linear and full attention and a vision encoder. - Input (variant "L2"):
[task] [page image] [document text] [folder list] [folder list with one marker per folder] [none marker]. The folder list appears twice so that every folder marker sees the document and the complete list. Markers are existing special tokens; no new vocabulary. - Head: a small decision head reads the hidden states at the markers and gives one score per folder and one for "none fits"; together they form one probability distribution.
- Limits: context up to 16,384 tokens; document text up to 2,048 tokens (start and end are kept); up to 32 tokens per folder name; page images up to about 1024 × 1448 pixels.
- Calibration: temperature scaling and risk thresholds per list size (Learn-then-Test, ≤ 1 %
wrong among automatic decisions, δ = 0.1) on 16,159 tasks from the held-out world, about
3,000 per list size (1,160 for trees of 200 folders and more). Separate calibration files for
bf16 and for 8-bit weights (
kalibrierung.json,kalibrierung_8bit.json), each bound to its weights by SHA-256. The list-size buckets are stored in the calibration file.
Training
- Data. Only synthetic documents, no user data. The documents and folder trees are those of
v0.4 (37,735 documents in 3,000 trees of 13–499 folders, 32 document types, page images
rendered for all documents, part of them degraded and read with Tesseract). New in v0.5:
- An identity for every area. In 1,553 of the 3,000 training trees the areas received an identity. 22,447 documents had their owner or recipient replaced accordingly: 21,157 were rewritten by a teacher model, changing only this detail, and checked against the facts the document was built from (numbers, dates, mandatory details); for the 1,290 rewrites that failed the check a fixed wording was used. 384 documents received an extra line, 200 are of kinds that carry no owner (receipts, notes, recipes), 14,970 were left unchanged. The held-out worlds for selection, calibration and the benchmark were treated the same way.
- Ambiguous tasks are labelled. A task counts as ambiguous when the list holds a folder of another area into which the document fits equally well. In the selection world the share of such tasks fell from 3–61 % per list size to 1–14 % through the identities.
- Episodes. As in v0.4: every step draws a task per document: the whole tree or a part of it (3–300 folders, 5 % up to 500), with folder descriptions in 30 % and spelling variants in 20 % of the tasks; in 25 % the right folder is missing, half of those with a similar neighbour present; in 15 % only the parent of the right folder is in the list, and the parent is the answer. New: in ambiguous tasks the target probability is shared equally between the right folder and the equally fitting ones.
- Kept apart by tests. Separate names, identities and trees for training and for the held-out world; normalized and substring overlap checks. No leakage was found.
- Setup: LoRA (r = 32, α = 64, dropout 0.05) on attention, linear attention and MLP of the language model; vision encoder frozen. Learning rate 1e-4 for LoRA and 3e-4 for the head, 3 % warmup, 3,000 steps of 16 tasks (v0.4: 4,716), no label smoothing, gradient checkpointing, bf16. Best checkpoint by validation NLL (step 2,250).
- Hardware: one NVIDIA RTX A6000 (48 GB), about 18 hours.
Limitations
- Automatic sorting only up to 49 folders, and less certain from 25. For larger trees the model is sure of some wrong answers, so no threshold keeps automatic decisions at ≤ 1 % wrong. Use it for suggestions there. For 25–49 folders see the note under the first table.
- Very large trees are harder. Accuracy is about 90 % up to 199 folders and 66 % for trees of 200 folders and more. Very large trees (in our tests, about 750 folders) do not fit into the context.
- Areas need names. If two areas of a tree have the same subfolders and neither the folder names nor the document say whose it is, the model cannot know. It then shows both folders and asks; its first suggestion is right in about one of three such cases.
- Wrongly answers "none fits" in 8.5 % of the tasks that have a fitting folder. A missing folder is recognized least reliably when its whole branch is absent (80 %).
- Weaker document types: vehicle papers, letters from authorities, notes, school letters and pension documents (71–76 %, some with few test documents), and OCR text from degraded scans (78 %).
- Slow on a CPU (seconds to minutes per document), heavy with a page image.
- Synthetic training data. Real archives are messier. Expect a gap until the model is checked on real, consented documents.
- German and English only.
Files
| File | Content | Size |
|---|---|---|
lora.safetensors |
LoRA weights for the language model | 125 MB |
kopf.safetensors |
decision head | 19 MB |
big_config.json |
base model and revision, LoRA settings, input limits, variant | < 1 KB |
kalibrierung.json |
calibration for bf16 weights | 9 KB |
kalibrierung_8bit.json |
calibration for 8-bit weights | 9 KB |
tokenizer/ |
Qwen3.5 tokenizer | 20 MB |
bildprozessor/ |
image preprocessing settings | < 1 KB |
beenara_big.py |
loader and inference: input format, head, calibration | 15 KB |
The base model weights are not included; they come from
Qwen/Qwen3.5-2B-Base at the revision in
big_config.json.
Environmental impact
We did not have a power meter. The numbers are an upper-bound estimate following Lacoste et al. 2019.
| Compute | Energy, at most | CO₂eq, at most | |
|---|---|---|---|
| This model (one training run) | about 18 h on one GPU | 12 kWh | 4.5 kg |
| The whole v0.5 run | about 56 GPU hours on two RTX A6000 (teacher model ~9 h, both Big training runs ~36 h, calibration, the small models and evaluation ~11 h) over 58 hours | 37 kWh | 14 kg |
Assumptions: at most 300 W per GPU while in use, plus 350 W for the rest of the machine over the whole wall time; grid 0.38 kg CO₂eq/kWh (German mix 2023/24). Not included: the pretraining of the base and teacher models, a few GPU hours of pilot experiments, and the documents re-used from v0.4 and v0.3 (counted there).
Credits
| Source | What BeeNara Big takes from it |
|---|---|
| Qwen3.5-2B-Base, Qwen team, Apache-2.0 | the base model |
| LoRA, Hu et al. 2021 | low-rank fine-tuning |
| laya, convaiinnovations, Apache-2.0 | the idea of one marker per option, scored in one pass |
| Guo et al. 2017 | temperature scaling |
| Angelopoulos et al. 2021, Learn then Test | the risk calibration |
| Xia et al., Understanding Why Label Smoothing Degrades Selective Classification and How to Fix It | the effect that made us drop label smoothing |
Gemma 4 (26B, 4-bit, Ollama tag gemma4:26b-a4b-it-q4_K_M), Google, Apache-2.0 |
rewrote the owner details of the training and held-out documents for v0.5; run locally |
| Qwen3.8-27B, Qwen team, Apache-2.0 | wrote training and held-out documents and drafted folder trees for v0.4; checked the other teacher's names, trees and document types; run locally |
| gpt-oss-120b, OpenAI, Apache-2.0 | wrote training and held-out documents, all folder names, and drafted folder trees for v0.4; checked the other teacher's trees and document types; run locally |
| Tesseract OCR, Apache-2.0 | the OCR text in training |
License
Apache-2.0, see LICENSE. The adapter derives from Qwen3.5-2B-Base, which is Apache-2.0
licensed; see NOTICE. Developed by Akinara as part of PollySort.