summerAIO/Qwen3.8-27B-Longbook27-v2

---

🍵 Overview

Qwen3.8-27B-Longbook27-v2 is a fine-tuned 27B model for understanding and generating long-form text such as books, papers, and lengthy reports. Version 2 improves on v1 in behavior and output quality.

The model uses a custom architecture (qwen3_5_gdn24) with Gated DeltaNet linear-attention layers, shipped as remote code in this repository.

⚠️ trust_remote_code=True is required. Loading without it will fail or prompt for confirmation.

🌿 Highlights

🧩 Model Details

💾 Requirements

Load the whole model on the GPU. CPU offloading is not supported by the custom kernels and fails with Tensor on device meta.

pip install -U transformers accelerate bitsandbytes
# Optional: much faster linear-attention kernels
pip install flash-linear-attention
pip install causal-conv1d --no-build-isolation

Without the optional packages the model still runs, using a slower PyTorch reference implementation.

🛠️ Quick Start

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig

model_id = "summerMC/Qwen3.8-27B-Longbook27-v2"
USE_4BIT = True  # False for bf16 on 80 GB GPUs

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)  # required

if USE_4BIT:
    model = AutoModelForCausalLM.from_pretrained(
        model_id,
        quantization_config=BitsAndBytesConfig(
            load_in_4bit=True,
            bnb_4bit_quant_type="nf4",
            bnb_4bit_compute_dtype=torch.bfloat16,
            bnb_4bit_use_double_quant=True,
        ),
        device_map={"": 0},       # keep everything on GPU 0
        trust_remote_code=True,   # required
    )
else:
    model = AutoModelForCausalLM.from_pretrained(
        model_id,
        torch_dtype=torch.bfloat16,
        device_map={"": 0},
        trust_remote_code=True,   # required
    )

book_text = open("book.txt", encoding="utf-8").read()
messages = [{"role": "user", "content":
    "Summarize this book chapter by chapter. Only use facts stated in the text.\n\n" + book_text}]

prompt = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True,
    enable_thinking=False,        # True: reason before answering
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=800,
        do_sample=False,          # greedy for faithful summaries
        repetition_penalty=1.05,
        pad_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Tips

• Pass the real text, not a placeholder, and keep prompts specific.
• Use greedy decoding (do_sample=False) for summarization and QA. For creative writing, use do_sample=True, temperature=0.7, top_p=0.8, top_k=20.
• Keep repetition_penalty around 1.05 to avoid loops.
• Use enable_thinking=True for harder questions at the cost of more tokens.

🎯 Use Cases

• 📘 Summarizing books, papers, and long reports
• 🔍 Question answering over long documents
• ✍️ Long-form writing and continuation

⚠️ Limitations

• Memory use grows with input length.
• Summaries may add details not present in the source; verify against the original text.
• trust_remote_code=True executes code from this repository (modeling_qwen35_gdn24.py, fused_ops.py). Review it and pin a revision in sensitive environments.

📜 License

Follows the license of the base Qwen model.

---

_{Made with 🍵 by summerMC}