Qwen3.5-9B · Policy Structure (LoRA)

Give it a policy document one page image at a time, and it returns the whole document as one nested JSON tree. Text is copied verbatim and nested the way the document is printed: sections, subsections, numbered clauses, lettered items and definitions. Content that runs across a page break is joined back up.

• Live demo (side-by-side with the original model): policy-structure-extractor.vercel.app
• Base model: Qwen/Qwen3.5-9B
• Code: extract.py (PDF → JSON tree) · page_merge.py (page reassembly)
• Results: results/ (every number in this card, as CSV)
• Training data: Ashish-Ranjan/Policy-Structure-Pages (gated, non-commercial research only)
• Author: Ashish Ranjan

Model summary

Highlights

• Structure: node F1 of 0.89, against 0.36 for the original model, on two insurance policies held out from training (34 pages).
• Nesting: 87% of nodes sit under the right parent, against 59% for the original model, and 95% have the right type.
• Verbatim text: 99.8% whole-document word similarity. Only 4 of the 1,301 reference nodes are missing.
• Beyond the training domain: on six unlabelled documents it had never seen (88 pages, including commercial forms, a card agreement and a federal regulation), it captured 98% of the printed words, against 43% for the original model.
• Behaves normally without the extraction prompt: no ordinary question was answered with JSON, visual-QA accuracy is 32/40 against 34/40, and thinking mode is intact.
• Fast and cheap: 8.6 s per page on one H100. One training run took 33 minutes; the whole project cost an estimated $12 of cloud GPU time.

Contents

How it works · Getting started · Evaluation · Uses · Bias, risks and limitations · Training details · Technical specifications · Environmental impact · Troubleshooting · Files · License and data provenance · Citation · Changelog · Glossary

How it works

A policy is too long for one generation, and its nesting runs across pages. The model therefore reads one page per request. Each request also carries a short description of where the previous page left off, called the open path.

page 1 ─► model ─► page tree ─► merge ─► document tree ─► open path ─┐
                                                                     │
page 2 + open path ◄─────────────────────────────────────────────────┘ ─► model ─► page tree ─► merge ─► …

1. The open path is the chain of nodes still open at the end of the previous page, from the document root down through the last child at each level. Each entry holds the node's type and its label, or its title, or the first words of its text. The deepest entry also gets a tail (its last 150 characters), so the model can tell whether the new page continues that sentence.
2. The page tree. The model writes only what is printed on the current page. A node that continues from an earlier page is written as a stub along the left edge, for example {"cont": true, "type": "clause", "label": "12."}. A stub never repeats earlier text. If the page starts mid-sentence, the deepest stub's text holds just the continuation.
3. Merging is done by deterministic Python code, not by the model (page_merge.py):
   • Each stub is matched to the open node at the same level.
   • Continuation text is appended to that node, and new nodes are attached after it.
   • The next page's open path is then read off the merged tree.
   • Slicing the training documents into pages and merging them back reproduces every original tree exactly.
4. Order matters. Pages of one document are processed in order, because each prompt is built from the model's own output for the earlier pages. Several documents can still be processed side by side in one batch.

Example: held-out GEICO auto policy, page 5

Page 5 of GEICO's Maine Family Automobile Insurance Policy (form A-30ME), a publicly filed policy form, shown here for illustration only (not affiliated with or endorsed by GEICO). This policy was held out from training.

The open path given to the model was built from its own output for pages 3–4:

[{"type": "document"},
 {"type": "section", "title": "Section I — Liability Coverages"},
 {"type": "subsection", "title": "Exclusions — When Section I Does Not Apply"},
 {"type": "clause", "label": "12.", "tail": "of a non-owned auto or temporary substitute auto that is designed for use principally off public roads that is not registered for use on public roads."}]

The start of the model's output, after the bracket repair:

{"cont": true, "type": "document", "children": [
 {"cont": true, "type": "section", "title": "Section I — Liability Coverages", "children": [
  {"cont": true, "type": "subsection", "title": "Exclusions — When Section I Does Not Apply", "children": [
   {"type": "clause", "label": "13.", "text": "We do not cover bodily injury or property damage that results from nuclear exposure or explosion including resulting fire, radiation or contamination is not covered."},
   {"type": "clause", "label": "14.", "text": "We do not cover bodily injury or property damage that results from bio-chemical attack or exposure to bio-chemical agents is not covered."},
   {"type": "clause", "label": "15.", "text": "Section I does not apply:", "children": [
    {"type": "item", "label": "(a)", "text": "To bodily injury or property damage caused by an auto driven in or preparing for any racing, speed or demolition contest or stunting activity of any nature, whether or not prearranged or organized."},
    {"type": "item", "label": "(b)", "text": "To the operation or use of a motor vehicle on a track designed primarily for racing or high speed driving. ..."}]},
   …

How to read this:

• The page continues the open Exclusions subsection, so it starts with three stubs and then adds exclusions 13 onwards.
• The repeated "…is not covered." in exclusions 13 and 14 is printed that way in the policy; the model copies text exactly as printed.
• The raw output had a closing-bracket miscount, which repair_json fixed. The raw and parsed outputs are both in examples/geico_p5.json.

Getting started

Requirements

• Hardware: a Linux machine with an NVIDIA GPU with more than 24 GB of memory. With the default settings the model needs about 21.5 GB: 17.8 GB of weights once loaded, plus vLLM's working memory and the cache for one page.
• Software: vLLM 0.30.0, pypdfium2 and Pillow (see requirements.txt).
• Tested end to end from this repository (2026-10-16, fresh install on one H100): download, then extract.py on pages 3–8 of the held-out GEICO policy.
  • All 6 pages attached, and 4 of the 6 outputs were byte-identical to the evaluation run.
  • Node F1 on those pages was 0.868, against 0.861 in the evaluation run.
  • Greedy decoding on GPUs isn't bit-exact across library versions and batch sizes, so expect small differences like these.
  • v1.1 changed only the parsing code. Replaying the saved model outputs through the v1.1 extract.py reproduces the re-scored trees exactly for all 8 test documents.

Which GPU? Measured 2026-10-16 with the published extract.py on held-out GEICO pages:

*At Modal's on-demand prices. The H100 is the cheapest per page because it generates almost 4× faster.

Other GPUs (calculated, not tested): any card with 32 GB or more should work. 24 GB consumer cards (RTX 3090 / 4090) have slightly more usable memory than the L4, so they're borderline. 16 GB cards can't hold the weights.

Run on a PDF

pip install -U "huggingface_hub[cli]"
hf download Ashish-Ranjan/Qwen3.5-9B-Policy-Structure-LoRA --local-dir policy-structure
cd policy-structure
pip install -r requirements.txt
python extract.py my_policy.pdf --out-dir out

What it does:

• Renders each page at 200 DPI, the resolution the model was trained on.
• Runs the page loop.
• Writes out/my_policy.json, the merged document tree.
• Writes out/my_policy.pages.json, which holds for every page the raw model output, whether its brackets were repaired, merge repairs, output tokens and timing.

Options:

• --pages 3-18 processes only some pages.
• Passing several PDFs batches them together.
• --adapter <folder> loads the adapter from a local copy.

Use from Python

from extract import render_pdf, extract, vllm_generator

generate = vllm_generator("Ashish-Ranjan/Qwen3.5-9B-Policy-Structure-LoRA")              # loads Qwen3.5-9B + this adapter in vLLM
docs = {"policy": render_pdf("policy.pdf", pages=range(3, 19))}
result = extract(docs, generate)                   # pages in order; documents batched together
tree = result["policy"]["tree"]

Prompt format (for your own loop)

Everything the evaluation used is in this repo.

Keep the bracket repair. With free decoding the model sometimes miscounts closing brackets at the end of deeply nested pages (9 of 16 GEICO pages and 7 of 18 Encompass pages needed it).

Output format

Every node is {type, label, title, text, children}, and empty fields are left out. There are no ids; assign them downstream if you need them.

Conventions the model learned:

• Headings of equal visual weight are siblings.
• A list is nested under the heading or lead-in sentence (ending in ":") that introduces it.
• The first enumerated level under a section or subsection is a clause; deeper levels are items.
• Page furniture is left out: running headers and footers, page numbers, form numbers, copyright lines, insurer address blocks and endorsement banners.

Evaluation

Testing data, factors and metrics

Testing data:

Factors:

• Layout: one, two or three columns.
• Document type: personal-lines insurance, commercial insurance, a card agreement, a federal regulation.
• Page kind: body pages, covers, contents pages.
• Decoding and merge settings, for the ablation.

Metrics:

All runs went page by page, exactly as in How it works. Each page's prompt carried the open path built from the model's own earlier pages, never from the reference.

Results: held-out documents

• Original model: the same prompts with JSON-constrained decoding, its best setting. With free decoding it scores lower (F1 0.22 / 0.36), because most of its pages don't parse.
• Fine-tuned model: the released pipeline: free decoding, then bracket repair, then the tolerant merge.

More on the metrics

• Right parent, strict: the matched node's parent is the matched counterpart of its reference parent. This check cascades: if a parent is written slightly differently (for example "4" instead of "4."), all of its children count as wrong.
• Right parent, relaxed: the parent is the same node, ignoring label punctuation, or has the same heading or text start. Only 1.5% (GEICO) and 4% (Encompass) of nodes are genuinely under the wrong node.
• Unmatched reference nodes (66 GEICO, 54 Encompass): almost all are present but merged into a neighbouring node's text, or split differently. Only 4 of 1,301 reference nodes are missing outright.

Results: unseen documents

On GEICO, where the right answer is known, the reference tree scores 99.8% captured / 0.1% not on page, and the fine-tuned model 99.7% / 0.1%. So these two checks track quality closely.

What we found:

• Text capture generalises. 79 of 88 pages scored ≥ 95%, across a three-column layout, a watermark, the small print of a federal regulation and a pricing table.
• Nesting is the weak spot on unfamiliar layouts. On some pages the model closes one bracket level too few in the middle of the page. Every later node on that page then lands one level too deep, for example Part B nested inside Part A. It happened most on Regulation E (9 of 15 pages) and the commercial general liability form (8 of 20).
• Contents pages: one was lost, because the model copied the dot leaders ("Agreement . . . .") until it hit the 4,096-token cap. Another came out as empty headings.
• Regulation-style run-in labels such as "(b) Electronic fund transfer—(1) Definition." are nested incorrectly.
• A second form inside the same PDF is attached beside the first form's sections instead of under a new node.

The live demo shows these outputs page by page under "More documents"; per-document numbers are in results/unseen_docs.csv.

Ablation: what each part of the pipeline adds

Constrained decoding guarantees valid JSON, but on dense two-column pages it stops at a premature close and loses the right-hand column. The repair-based pipeline keeps that column.

Behaviour outside extraction

Learning-rate sweep and checkpoint selection

• Validation loss is teacher-forced on the 16 GEICO pages. It was 0.0623 for the original model.
• F1 in this table is plain free decoding before bracket repair. It is dominated by pages lost to bracket errors, which the released pipeline recovers.
• Selection: lowest validation loss together with the best full-document F1. Because the checkpoint was chosen on the same two held-out documents, their scores are slightly optimistic; the six unseen documents were not used for any choice.

Uses

Direct use

Turning born-digital, English insurance policies and similarly nested contracts into one JSON tree per document, with extract.py or your own loop (see Prompt format).

Downstream use

• Structure-aware chunking for retrieval (RAG): one chunk per section or clause, with its heading path.
• Clause lookup and side-by-side comparison of policies.
• A starting point for fine-tuning on other nested document types with the same page-by-page format.

Out-of-scope use

• Automated coverage, claims, underwriting or legal decisions without human review.
• Legal or insurance advice.
• Scanned, handwritten or non-English documents, and table-heavy documents: grids are not supported, and the schema has no table type.
• General chat or image questions. These still work, but use the base model for them.

Bias, risks and limitations

• Small reference-scored test. Structure is scored against reference trees on only two documents (34 pages), both US insurance policies. The six unseen documents (88 pages) were checked with label-free measures only.
• Insurance-centred training. Training used 15 documents, mostly US personal-lines insurance plus one credit-card agreement. Layouts and drafting styles from insurers or document types outside that set nest less reliably. Flat or lightly structured documents may come out over-nested.
• Pages the model hasn't seen. Cover pages, contents pages, declarations pages and data tables were not in training. Contents pages with dot leaders can make the model copy the leaders until the token cap, losing the page. Key-value tables are flattened into items.
• Granularity and nesting. The model sometimes:
  • merges two paragraphs into one node;
  • folds several heading levels into one title;
  • attaches a lead-in sentence at the wrong level;
  • drops a label's punctuation ("4" vs "4."), because the training labels were inconsistent about this.
• Brackets. repair_json fixes missing or extra closers at the end of a page. It can't fix a pair missing in the middle of a page, which pushes the rest of that page one level too deep.
• Error propagation. Pages are processed in order, so an early nesting mistake carries into later prompts.
• Downstream risk. A clause attached to the wrong parent can make a RAG system cite the wrong section or apply an exclusion to the wrong coverage.
• Personal data. Run on a real policyholder's document, the output reproduces any personal data on the page verbatim (names, addresses, policy numbers). Handle the outputs like the source document.

Recommendations

• Always parse with parse_page and merge with merge_lenient (extract.py does). Together they repair miscounted brackets, never drop a key the model wrote twice, and never drop a page.
• Skip contents pages, or stop generation on a long run of . . ., and split PDFs that bundle several forms into one document per form.
• Check outputs cheaply without labels:
  • compare the output's words with the PDF text layer (the "words captured" check);
  • flag numbering gaps or restarts among siblings ("1, 2, 4" or a list starting at 3), which reliably point to nesting slips.
• Before relying on it for a new document type, score a few documents against hand-checked trees.
• Keep a human in the loop for any decision that depends on the extracted structure.

Training details

Training data

Training documents: 15 policy forms and agreements: forms from US insurers (plus one Canadian policy), some of them ISO standard forms, and one bank credit-card agreement. Declarations pages and other policyholder-specific pages were excluded. The pages, targets, held-out reference trees and the exact training mix are available for non-commercial research in the gated dataset Policy-Structure-Pages. There, training documents are identified by kind only.

13 of the 15 documents are two-column layouts. Drafting styles range from ISO standard forms to insurers' own plain-language and narrative styles. Enumeration styles include A. / 1. / a. / (1) / (a) / 1) / a).

• Data collection method: hybrid. The extraction data comes from public policy documents, mostly from a US state insurance regulator's policy-comparison filings, and one bank's published agreement. The retention prompts and images come from public datasets.
• Labelling method:
  • Extraction targets: human. The author wrote a reference tree for each document from its page images, using the PDF text layer for exact wording.
  • Contrast and retention answers: synthetic. The original Qwen3.5-9B generated them itself (self-distillation), about 75% with thinking on, so the adapter doesn't unlearn reasoning.

Training procedure

Preprocessing:

1. Each reference tree was aligned to the page breaks and sliced into per-page targets with continuation stubs. Merging the slices reproduces every tree exactly.
2. Node types were normalised with one rule across documents.
3. Five near-duplicate pages were dropped (4-gram Jaccard ≥ 0.7).
4. Pages were rendered at 200 DPI. A page image is 3,657 tokens and is never downscaled.

Hyperparameters (ms-swift 4.5.3; full arguments in training/swift_args.json):

Speeds, sizes, times

The training log is in training/train_log.jsonl.

Technical specifications

Model architecture and objective

• Base model: Qwen3.5-9B, a natively multimodal model. Its language model mixes Gated DeltaNet (linear attention) layers with full-attention layers and supports a 262K-token context.
• What the adapter changes: low-rank updates to every linear layer of the language model. The vision encoder and projector are unchanged.
• Objective: supervised fine-tuning with next-token cross-entropy on the assistant's tokens.

Compute infrastructure

• Hardware: NVIDIA H100 80 GB on Modal, one GPU per training run. Self-distillation answers were generated on an L40S.
• Software:
  • training: ms-swift 4.5.3, transformers 5.16.1, peft 0.20.0, flash-linear-attention 0.5.2;
  • inference and evaluation: vLLM 0.30.0.

Environmental impact

Estimated with the Machine Learning CO₂ Impact method (Lacoste et al., 2019): GPU hours × power draw × grid carbon intensity.

Troubleshooting

• vLLM fails while compiling a FlashInfer sampler (no nvcc on the machine): set VLLM_USE_FLASHINFER_SAMPLER=0. The evaluation ran this way.
• Out of memory while vLLM starts: the GPU is too small. The model needs about 21.5 GB, so a 24 GB L4 fails; use a 32 GB or larger card. On a borderline card, --gpu-memory-utilization 0.95 gives vLLM a little more room. max_model_len is 12,288 tokens: one page image plus prompt plus up to 4,096 output tokens.
• A page is missing from the tree: look at its entry in <name>.pages.json. An error means the output couldn't be parsed even after repair. output_tokens = 4096 means generation hit the cap, usually on a contents page. A non-empty duplicate_keys means the model wrote a field twice on one node; both values were kept (see Changelog).
• Output is prose instead of JSON: make sure the system message is exactly system_prompt.txt, the adapter is loaded and thinking is off.

Files in this repository

License and data provenance

• Licence: the adapter weights and the code in this repository are released under Apache-2.0, the same licence as the base model, Qwen3.5-9B.
• Training documents: insurance policy forms (most from a US state insurance regulator's public policy-comparison filings) and a public card agreement. Their copyrights stay with their owners, and some include ISO copyrighted material. This model repository doesn't contain the documents. The training and evaluation data is available separately in the gated, research-only dataset Policy-Structure-Pages. The single example page above is shown for illustration.
• Trademarks and affiliation: this project is not affiliated with or endorsed by any insurer or other company named in this card. Names are used only to identify the publicly filed documents the model was tested on.
• Retention data: prompts and images come from HuggingFaceM4/the_cauldron, whose subsets carry their own licences, and from OpenAssistant/oasst2 (Apache-2.0). The answers were generated by Qwen3.5-9B. None of that data is redistributed here.

Changelog

v1.1 (2026-10-18): parsing fix, same model weights.

• What was wrong: when a clause has text both before and after its list, the model sometimes writes the node's text field twice, the lead-in before children and the closing words after them. Standard JSON parsing keeps only the last value of a repeated key, so v1.0's parser silently dropped the lead-in.
• Where it happened: on the held-out Encompass policy it removed "Injury arising out of a premises:" from a personal-injury exclusion. In the unseen-documents test it also overwrote a list of 3 nodes (Liberty Mutual homeowners) and 3 table row names (Capital One).
• The fix: parse_page keeps every value. A second text becomes a trailing paragraph, which is how the reference trees store such text, and a second children list is appended. Each page's log reports any repeated keys, so this can't happen silently.
• Effect (saved outputs re-scored, no retraining):
  • the Encompass clause now matches the reference exactly;
  • Encompass node F1 0.921 → 0.923 and text similarity 0.997 → 0.998;
  • unseen documents: Liberty Mutual homeowners 99.6% → 99.7% of words captured, Capital One 99.6% → 99.7%;
  • nothing else changed: 118 of 122 pages parse exactly as before.

v1.0 (2026-10-16): first release.

Citation

BibTeX:

@misc{ranjan2026policystructure,
  title        = {Qwen3.5-9B Policy Structure: page-by-page nested JSON extraction of insurance policies with a LoRA adapter},
  author       = {Ranjan, Ashish},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/Ashish-Ranjan/Qwen3.5-9B-Policy-Structure-LoRA}}
}

APA: Ranjan, A. (2026). Qwen3.5-9B Policy Structure: page-by-page nested JSON extraction of insurance policies with a LoRA adapter [Computer software]. Hugging Face. https://huggingface.co/Ashish-Ranjan/Qwen3.5-9B-Policy-Structure-LoRA

Glossary

Model card authors and contact

Written by Ashish Ranjan. For questions, problems or results on other document types, open a discussion in this repository's Community tab.

Acknowledgements

• Qwen team: Qwen3.5-9B.
• ModelScope: ms-swift.
• The vLLM project.
• Modal: the GPUs.
• HuggingFaceM4 (the_cauldron) and OpenAssistant (oasst2): retention data.
• Thinking Machines' "LoRA Without Regret": the LoRA hyperparameter choices.