Kataguru-Sceptic-35B-A3B-v2.6-AWQ (W4A16 Marlin / compressed-tensors)

Kataguru-Sceptic-35B-A3B-v2.6-AWQ is a bilingual (English & Finnish) 35-billion parameter (3B active parameters per token, 254 routed experts per layer) multimodal Mixture-of-Experts (MoE) model quantized to W4A16 (compressed-tensors / Marlin INT4, group_size=128) for vLLM clusters across NVIDIA Ampere, Ada Lovelace, Hopper, and Blackwell GPUs.

---

1. Specialized Capabilities & Architectural Scope

1. Dual-Stage 2D Zero-Swap Epistemic Triage (System 1 Reflex):
   • Evaluates candidate training samples in < 20 ms (measured 18.2 ms single-call latency on RTX 5090) using 1-token logprob scoring (max_tokens=1, top_logprobs=10) with 0 generated output tokens.
   • Computes both sample quality $E[\text{sample}]$ (decision mode) and prompt learning potential $E[\text{topic}]$ (topic mode) to fast-reject trivial data while routing high-potential STEM prompts with flawed answers to the Surgeon.
2. Autonomous Self-Healing Data Surgeon:
   • In surgeon mode ([MODE: SURGEON] / [TILA: KIRURGI]), rewrites flawed mathematical, physical, or software responses from first principles so that they can be verified by an external execution sandbox (e.g., isolated Python 3.12 subprocess execution with assert checks).
3. Industrial Automation & PLC/CNC Code Generation:
   • Trained on curriculum mixtures covering Beckhoff TwinCAT 3 Structured Text (ST), Siemens TIA Portal SCL, Fanuc / Siemens 840D CNC G-code, and OPC UA (IEC 62541) diagnostic schemas.
4. Multimodal Visual & SCADA Diagnostic Agent (Zero-Swap Unified Pipeline):
   • Why Integrated Multimodality? Hosting separate vision and coding/reasoning models either doubles VRAM consumption or introduces model-swapping latency. Integrating the vision tower (visual) directly into the 254-expert MoE backbone allows a single model instance to read an HMI alarm screenshot or wiring schematic, issue XML tool calls (qwen3_xml / qwen3_coder), and generate PLC ST/SCL diagnostic code without swapping weights.
   • Hardware & Memory Scope: Running the unified multimodal pipeline on a single 16–24 GB GPU without CPU offloading requires the 3-bit quantized GGUF release (APEX-Small 12.0 GiB or APEX-Medium 13.5 GiB + mmproj 0.84 GiB). The NVFP4 (25.44 GB / 23.70 GiB) and AWQ (27.85 GB / 25.94 GiB) safetensors packages do not fit in 16 GB VRAM without CPU offloading; they target 2x 24 GB / 2x 32 GB GPUs (TP=2) or a 32 GB+ GPU.
5. Layer 40 Multi-Token Prediction (MTP) Speculative Decoding:
   • Preserves the 254-expert Layer 40 MTP head (mtp.*), providing multi-token speculative decoding in both vLLM (--speculative-config) and llama.cpp (--spec-type draft-mtp, measured +12.0% tok/s on 1x RTX 5090).
6. Bilingual Finnish & Technical English Focus:
   • Trained to avoid English-to-Finnish translation calques and morphological case errors in technical prose. (Note: Evaluated via internal curriculum filtering gates during data curation; a standalone public Finnish morphological benchmark dataset is not yet bundled in eval/.)
7. Epistemic Directness & ASD-STE100 Technical Style:
   • Suppresses sycophantic filler and conversational preambles, following ASD-STE100 (Simplified Technical English) writing principles (short sentences, active voice, one instruction per sentence).
   • Important Safety Notice: ASD-STE100 is a technical writing clarity standard; adhering to it improves human readability of comments and diagnostics but does not guarantee functional software or machinery safety. All generated code, PLC logic, and G-code must be validated in external deterministic sandboxes, vendor compilers (TwinCAT / TIA Portal), and hardware-in-the-loop safety checks before deployment.

---

2. Empirical Benchmark Comparison (Same-Backbone & Cross-Backbone Baselines)

To separate the effect of Kataguru post-training from base model differences, the table below compares Kataguru-Sceptic-35B-A3B-v2.6 (NVFP4) against:

1. Qwen3.5-35B-A3B Pre-SFT Base (cp0): The same 35B-A3B MoE backbone after initial SOMA/ARA orthogonalization before SFT curriculum training (isolates cumulative MoE post-training impact on tasks 1–11; raw log in eval/qwen35_35b_a3b_cp0_base_summary.json).
2. Qwen 3.8 27B Base (Untuned Dense): Cross-architecture dense baseline evaluated on the same 13-task harness.

All differences ($\Delta$) are reported in percentage points (pp).

*Note on GSM8K in cp0: The pre-SFT base checkpoint lacked the #### <number> final-answer formatting convention required by lm-eval strict/flex regex extractors, causing extraction failures (1.06% strict / 12.36% flex). Therefore, both $\Sigma_{10}$ (tasks 1–10 excluding GSM8K, +2.34 pp) and $\Sigma_{11}$ (tasks 1–11, +10.16 pp) are reported above.

Evaluation Methodology & Published Raw Logs (eval/)

• Evaluation Harness & Versions: Executed on 2x NVIDIA RTX 5090 32GB Blackwell (sm_120, TP=2, port 8003, CUDAGraphs enabled) using lm-evaluation-harness 0.4.13 (recorded in eval/benchmark_13_summary.json) for tasks 1–11 and task 13 (ifeval), and a standalone zero-shot evaluator for task 12 (gpqa_diamond).
• Complete Item-by-Item Logs Included in Repository:
  • eval/benchmark_13_summary.json: Aggregated metrics, runtime configurations, and lm-eval 0.4.13 metadata for all 13 tasks.
  • eval/<task>/kataguru__Kataguru-Sceptic-v2.6-Symmetric-CP100-NVFP4/samples_<task>_*.jsonl: Complete question-by-question prompts, model completions, and scores for all 12 lm-eval tasks (gsm8k, ifeval, medqa_4options, arc_challenge, arc_easy, boolq, hellaswag, openbookqa, piqa, truthfulqa_mc1, truthfulqa_mc2, winogrande).
  • eval/gpqa_diamond.json: Complete item-by-item completions and extracted answers for all 198 GPQA Diamond questions (121/198 = 61.11%).
  • eval/qwen35_35b_a3b_cp0_base_summary.json: Historical evaluation summary for the Qwen3.5-35B-A3B pre-SFT base (jackripper-sovereign).

---

2.1 Preliminary Functional Smoke Tests (Data Surgeon, PLC & CNC)

To verify end-to-end runtime functionality of the specialized modes, a small preliminary smoke test suite (eval/specialized_smoke_tests/benchmark_specialized_capabilities.py) was executed against the live NVFP4 endpoint:

Caveat: The PLC ($N=2$), CNC ($N=2$), and Surgeon ($N=5$) results above are small-scale functional smoke tests demonstrating pipeline mechanics and external sandbox integration, not statistically large domain benchmarks.

---

3. Hardware & VRAM Allocation Breakdown (AWQ W4A16 on vLLM)

The serialized W4A16 safetensors footprint is 27.85 GB decimal (25.94 GiB binary) across 6 shards (model-00001-of-00006.safetensors to model-00006-of-00006.safetensors):

• 40 Routed MoE Expert Layers (10,160 experts in INT4 W4A16): ~18.4 GB
• Unquantized BF16 Modules (visual tower ~1.7 GB + Layer 40 254-expert mtp speculative head ~3.3 GB + embed_tokens / lm_head / attention / shared experts ~4.4 GB): ~9.4 GB

Per-GPU Memory Requirements (TP=2)

• Model Weights Loaded (TP=2): ~13.0 GiB / GPU
• Non-Torch Runtime + Peak Activations + CUDAGraphs: ~3.6 GiB / GPU
• Minimum GPU Footprint (Before KV Cache): ~16.6 GiB / GPU
• Recommended Hardware: 2x 24 GB GPUs (TP=2), 2x 32 GB GPUs (TP=2), or 1x 48 GB+ GPU.
• Single 16 GB or 24 GB GPU: Because the W4A16 weights alone occupy 25.94 GiB, this package cannot fit on a single 16 GB or 24 GB GPU without CPU offloading (--cpu-offload-gb). For single-GPU 16–24 GB inference without CPU offload, use kataguru/Kataguru-Sceptic-35B-A3B-v2.6-GGUF.

---

4. Native 6 Operating Modes (Kataguru Master Chat Template)

The embedded chat_template.jinja recognizes both English and Finnish trigger prefixes (case-insensitive) at the start of the user prompt, or via chat_template_kwargs={"sceptic_mode": "..."}:

---

5. Serving with vLLM (TP=2)

vllm serve "kataguru/Kataguru-Sceptic-35B-A3B-v2.6-AWQ" \
  --tensor-parallel-size 2 \
  --max-model-len 32768 \
  --quantization compressed-tensors \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.85 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --host 0.0.0.0 --port 8003

---

---

Suomenkielinen Yhteenveto & Käyttöopas

Kataguru-Sceptic-35B-A3B-v2.6 on suomen- ja englanninkielinen 35 miljardin parametrin (3B aktiivista parametria / token, 254 reititettyä asiantuntijaa / kerros) multimodaalinen Mixture-of-Experts (MoE) -malli, joka yhdistää kuusi toimintatilaa (decision, topic, inspect, curate, surgeon, chat), visioprojektorin ja Layer 40 MTP-spekulaatiopään yhteen mallipakettiin.

Keskeiset ominaisuudet ja rajaukset

• 2D System 1 Episteeminen Pikatarkastus (decision & topic): Pisteyttää näytteen laadun ($E[\text{sample}]$) ja aiheen potentiaalin ($E[\text{topic}]$) yhden tokenin logprob-jakaumasta (< 20 ms) ilman <think>-hukkatokeneita.
• Autonominen Datan Kirurgi ([TILA: KIRURGI] / [MODE: SURGEON]): Korjaa virheelliset STEM-johdot ja kaatuvat Python-koodilohkot siten, että korjattu koodi voidaan todentaa ulkoisessa Python-suoritusportissa (alustavassa toimivuustestissä 3/5 viallisesta näytteestä läpäisi ulkoisen Python assert -ajon yhdellä yrityksellä).
• Teollisuusautomaatio & Multimodaalinen Diagnostiikka: Tukee Beckhoff TwinCAT 3 ST-, Siemens TIA Portal SCL- ja ISO CNC G-koodin generointia sekä kytkentäkaavioiden ja HMI-hälytysruutujen lukua.
  • Miksi integroitu multimodaalisuus? Yhdistämällä visio suoraan MoE-runkoon vältetään kahden erillisen mallin yhtäaikainen muistirasitus tai mallien latausviiveet. Yhden 16–24 Gt näytönohjaimen käyttö ilman CPU-siirtoa edellyttää GGUF-versiota (APEX-Small 12,0 GiB tai APEX-Medium 13,5 GiB); NVFP4- (25,44 GB) ja AWQ-versiot (27,85 GB) on tarkoitettu 2x 24 Gt / 2x 32 Gt (TP=2) tai 32 Gt+ ympäristöihin.
• Suomen kieli ja ASD-STE100 -selkeysstandardi: Malli on koulutettu tuottamaan teknistä suomea ja englantia ilman käännösanglismeja sekä noudattamaan koodikommenteissa ja diagnostiikassa ilmailualan ASD-STE100 (Simplified Technical English) -selkeyssääntöjä (lyhyet virkkeet, aktiivimuoto, ei täytesanoja).
  • Turvallisuushuomautus: ASD-STE100 on teknisen kirjoittamisen selkeysstandardi; se ei takaa ohjelman tai logiikan toiminnallista turvallisuutta. Kaikki tuotettu koodi sekä PLC- ja CNC-ohjelmat on aina todennettava ulkoisella suoritusportilla ja kohdejärjestelmän omalla kääntäjällä.