Laguna S 2.1 — attention-Q8 target + Q4_K_M DFlash (GGUF)
Hardware-tuned GGUF pair for running Poolside Laguna S 2.1 on a single 22 GB RTX 2080 Ti with CPU/GPU hybrid offload:
- a layer-aware target derived from Poolside's official imatrix Q4_K_M GGUF, converting only its 240 F16 attention tensors to Q8_0;
- a Q4_K_M conversion of Poolside's official Laguna S 2.1 DFlash draft model.
This is a modified quantized redistribution, not a fine-tune and not an official Poolside release. The target's existing Q4_K, Q6_K, router, norm, and bias tensors were byte-preserved; only the F16 attention tensors were changed.
Files
| File | Size | Tensor layout | Purpose |
|---|---|---|---|
laguna-s-2.1-attnQ8.gguf |
67.6 GiB | 239 Q4_K, 48 Q6_K, 240 Q8_0, 287 F32 | Layer-aware target |
laguna-s-2.1-DFlash-Q4_K_M.gguf |
622 MiB | 45 Q4_K, 4 Q6_K, 27 F32 | Speculative-decoding draft |
The original official target GGUF is about 75 GB. Converting its 240 F16 attention tensors to Q8_0 reduced tensor payload by 2,506 MiB while preserving every already-quantized tensor without requantization.
Why this layout
Laguna S 2.1 is a 118B-A8B MoE. Its official Q4_K_M GGUF deliberately leaves attention in F16, while experts use Q4_K/Q6_K and sensitive router/norm/bias tensors remain F32. On a 22 GB Turing card, target weights are mostly CPU-resident, so reducing attention traffic lets llama.cpp fit more target data on GPU.
The target conversion was intentionally narrow:
- F16 attention → Q8_0
- Q4_K / Q6_K / F32 → COPY unchanged
- no expert-layer requantization
- no router, norm, or bias quantization
RTX 2080 Ti results
AI Lab test system: EPYC 7402, 128 GB eight-channel DDR4 ECC, one modified RTX 2080 Ti 22 GB for Laguna, Poolside's laguna llama.cpp branch, 262K context, one slot.
Target-only decode
| Prompt language | Official Q4_K_M | attention-Q8 | Change |
|---|---|---|---|
| English | 19.640 tok/s | 21.048 tok/s | +7.2% |
| Traditional Chinese | 19.733 tok/s | 21.259 tok/s | +7.7% |
With Q4_K_M DFlash
| Prompt language | Official target + Q4 draft | attention-Q8 + Q4 draft | Change |
|---|---|---|---|
| English | 27.812 tok/s | 28.959 tok/s | +4.1% |
| Traditional Chinese | 18.906 tok/s | 19.498 tok/s | +3.1% |
Strict JSON, native tool-call safety, and executable min_window coding gates passed after target quantization. Laguna's known failure mode—spending the full completion budget in reasoning and returning no final content—was unchanged and should be handled at the agent layer.
Serving with llama.cpp
Use Poolside's llama.cpp fork, laguna branch. Generic upstream llama.cpp does not yet contain the complete Laguna DFlash decoder contract required by this pair.
./build/bin/llama-server \
-m laguna-s-2.1-attnQ8.gguf \
-md laguna-s-2.1-DFlash-Q4_K_M.gguf \
--spec-type draft-dflash \
--spec-draft-n-max 4 \
-ngld 99 \
-c 262144 \
-ctk q4_0 -ctv q4_0 \
-ctkd f16 -ctvd f16 \
-t 24 -tb 24 -td 12 -tbd 12 \
-fa on \
-fit on -fitt 7680 -fitc 262144 \
--parallel 1 \
--jinja \
--reasoning on --reasoning-format deepseek \
--host 0.0.0.0 --port 8095
Important DFlash caveats
Keep draft KV in F16. Q4_0 draft KV reduced proposal acceptance below 1% and decode to roughly 5–6 tok/s. Q4_K_M draft weights and Q4_0 runtime KV are independent quantizations; their errors do not cancel.
Tune --spec-draft-n-max for the hardware and draft precision. Poolside's official BF16 example uses n=15. With this Q4_K_M draft on the tested Turing hybrid-offload system, n=4 was the best measured point. Larger values reduced acceptance enough to lose throughput.
The target KV can remain Q4_0; the precision warning applies specifically to the DFlash draft K/V cache (-ctkd and -ctvd).
Context and reasoning
This package was validated at 262,144 tokens. The native target was trained up to 1M context; see Poolside's official model card for the 1M YaRN override and quality caveats.
For tool/coding action lanes, consider disabling thinking or adding an empty-final repair. A short max_tokens cap alone can consume the whole budget inside reasoning_content before a final answer is emitted.
Credits
- Base model and architecture: poolside/Laguna-S-2.1
- Official GGUF and importance matrix: poolside/Laguna-S-2.1-GGUF
- DFlash draft: poolside/Laguna-S-2.1-DFlash
- Layer-aware target conversion, Q4_K_M draft conversion, Turing tuning, and validation: coolthor
License
Distributed under the OpenMDW-1.1 License, inherited from Laguna S 2.1. A complete copy is included as LICENSE.md. The files in this repository were modified through quantization; Poolside does not endorse or maintain this redistribution.
☕ If this saved you GPU hours, you can buy me a coffee.
Made by Thor Lin. Full benchmarks, setup notes and the runs that failed are on ai-muninn.com. More quantized models: huggingface.co/coolthor.
Provenance
This section documents release ea6cd8f8fd070f48a8c9bdbb8ee96838d1639cfd, inspected on 2026-11-14. Where the exact upstream commit used at build time is unknown, it is listed as not recorded rather than backfilled with today's upstream HEAD.
Source lineage
| Source repository | Revision used for this build | Role |
|---|---|---|
| poolside/Laguna-S-2.1-GGUF | not recorded | Immediate official imatrix Q4_K_M target GGUF |
| poolside/Laguna-S-2.1-DFlash | not recorded | Immediate DFlash draft input |
| poolside/Laguna-S-2.1 | not recorded | Original target lineage |
Build details
Tools and versions: Poolside llama.cpp fork, laguna branch; exact build-time commit/build not recorded.
For the target model, only the 240 F16 attention tensors were converted to Q8_0; the 239 Q4_K, 48 Q6_K and 287 F32 tensors were copied without changes. The draft model was converted to Q4_K_M. The published target recipe is copied below; the exact draft-conversion command is not recorded.
The existing records cover quantization and format conversion only; they don't mention any fine-tuning or merging by coolthor. That reflects the documentation, not an independent weight diff.
Recorded conversion command (published recipe; not executed for this provenance review):
llama-quantize \
--token-embedding-type Q4_K \
--output-tensor-type Q6_K \
--tensor-type '^blk\.[0-9]+\.attn_(q|k|v|output|gate)\.weight$=Q8_0' \
--tensor-type '^blk\.(0|1|2|3|4|5|8|11|14|17|20|23|26|29|32|35|38|41|42|43|44|45|46|47)\.ffn_down=Q6_K' \
--tensor-type 'ffn_down=Q4_K' \
--tensor-type 'ffn_gate=Q4_K' \
--tensor-type 'ffn_up=Q4_K' \
laguna-s-2.1-Q4_K_M.gguf laguna-s-2.1-attnQ8.gguf Q8_0 24
Weight file SHA-256 checksums
The values below are Hugging Face LFS object SHA-256 values for the pinned release above. File sizes and digests agree with pinned tree metadata. They identify full file contents, not LFS pointer text or Git blob IDs. Calibration imatrices are included when present.
| File | Bytes | SHA-256 |
|---|---|---|
laguna-s-2.1-DFlash-Q4_K_M.gguf |
652160160 | 872be33424231b9e70ad95caf5c080825d110a0a4411bd66fa13c379651b56b9 |
laguna-s-2.1-attnQ8.gguf |
72545160800 | 2d3d6f5a0a4fa534aad7bd1b67b00da59c2961a6c8bbe364ab8dbb376e842b54 |
Verifying the files
Download the desired files from revision ea6cd8f8fd070f48a8c9bdbb8ee96838d1639cfd and compare sha256sum <file> (Linux) or shasum -a 256 <file> (macOS) with the table. If cloning with Git LFS, hash the downloaded weight files, not the LFS pointer files. A matching hash confirms your files match this release; it does not prove the weights came only from the listed base model with no other changes.
Checking how the weights were derived requires the exact upstream commit, tool versions, recipe and any calibration or training data, then re-running the build and diffing the weights with a format-aware tool (or comparing file hashes where serialization is deterministic). Because of the gaps listed below, this build can't be reproduced exactly today. Quantized files are expected to differ numerically from BF16 sources, and a raw file hash cannot establish a base-to-quant weight diff. No quantization rerun or weight-diff audit was performed for this provenance review.
Known gaps: Target/draft build-time upstream commits; Exact quantizer commit/build; Complete DFlash conversion invocation.
Sources: existing model card and HF metadata.