Qwen3.8-27b-CIRU-EXL3-SGLANG

Packed EXL3 weights, served through SGLang with DFlash2, image input, and a 262,144-token configured context on one RTX 5090.

CIRU built this inference package around r0b0tlab's 4.00-bpw Qwen3.8-27B EXL3 target and its EXL3 DFlash2 draft. The target and draft weights are unchanged: this release is a serving integration, with no additional training or requantization.

What makes this build different

• CIRU's loader executes the packed EXL3 projections directly inside SGLang, including the 6-bit output head. It maps the model's fused attention and Gated DeltaNet components without exporting a dense or NVFP4 copy of the weights.
• SGLang supplies request scheduling, attention, CUDA graph decode, and shared KV/recurrent-state management. DFlash2 verifies eight-token draft blocks.
• The release profile uses NVFP4 target KV, FP8 draft KV, FP32 recurrent state, and a 270,336-token shared pool to serve a 262,144-token request on a single 32 GB RTX 5090.
• Vision mode loads the checkpoint's image encoder, including 83 packed EXL3 projections. Text mode remains the default configuration used for the published speed measurements.

Benchmark highlights

Measured on one RTX 5090. Speed figures are medians of three valid requests or batches; the throughput row counts actual simultaneous requests and includes prompt processing in the batch time.

The speed comparisons are between complete serving packages, with different runtime and KV settings; they do not isolate quantization. The original-model Hermes runs used lower default turn ceilings, so their scores are not a matched-limit comparison to the CIRU score above. The full benchmark report contains the complete context ladder, concurrency sweeps, protocols, and raw results, including rows not selected for this short card.

Install and connect

The CIRU runner contains the pinned container recipe, CIRU integration source, model download locks, and the exact serving arguments. It downloads the credited target and draft from their upstream Hugging Face repositories; this page does not duplicate their weight files.

NixOS x86_64 is the validated host. The RTX 5090 system uses Docker and NixOS's NVIDIA Container Toolkit. For an RTX 5090, add these settings inside /etc/nixos/configuration.nix:

virtualisation.docker.enable = true;
hardware.graphics.enable = true;
services.xserver.videoDrivers = [ "nvidia" ];
hardware.nvidia = {
  modesetting.enable = true;
  open = true;
  package = config.boot.kernelPackages.nvidiaPackages.latest;
};
hardware.nvidia-container-toolkit.enable = true;

Run sudo nixos-rebuild switch, ensure your account can use Docker, and start a new login session if its group membership changed. Check docker info and /var/run/cdi/nvidia-container-toolkit.json; the runner selects the nvidia.com/gpu=all CDI device automatically. The NixOS CUDA container guide explains the CDI setup. Other NVIDIA GPUs may require a different driver selection.

On Ubuntu x86_64, install Docker Engine and the NVIDIA Container Toolkit. On Fedora x86_64, use the Docker Engine Fedora guide with the same NVIDIA toolkit guide. Then run the commands below. These other host install paths have not yet been validated with this packaged runner.

git clone https://github.com/ciru-ai/qwen38-exl3-sglang-runner.git
cd qwen38-exl3-sglang-runner
bash setup.sh install
bash setup.sh run

For image input, stop the text server with Ctrl-C and run:

CIRU_MODE=vision bash setup.sh run

In a second terminal, ask about an image:

python3 scripts/ask_image.py /path/to/photo.png "What is in this picture?"

Use http://127.0.0.1:8000/v1 as the OpenAI-compatible base URL and Qwen3.8-27b-CIRU-EXL3-SGLANG as the model name. The API is bound to localhost and has no authentication. bash setup.sh check checks the running endpoint; bash setup.sh stop stops it. Windows users can run the Linux container from WSL2 with Docker Desktop GPU support. The tested host was NixOS x86_64 with an RTX 5090; other hosts are not yet validated. Native macOS, AMD, CPU, and native Windows execution are unsupported.

The container packages one hash-checked patch to SGLang 0.5.20, one small ExLlamaV3 CUDA graph interoperability patch, and CIRU's EXL3 target, draft, and vision bridge. Users do not need to assemble multiple forks. The runner pins its Python/CUDA dependencies and verifies downloaded weight hashes.

Vision checks

Vision mode successfully processed three PNG images and one JPEG on the same RTX 5090: a robot illustration, a dense chart, a pagoda screenshot, and an illustrated poster. It identified the visible subjects and chart/page titles, and a text-only request also completed in vision mode. These are functionality checks; the speed and quality figures above are from text mode. A full-length vision context sweep has not been run.

Scope and attribution

This is a thinking-off, single-GPU runner with text and image modes. The 262,144-token configured limit includes image, text, and generated tokens. It was validated on NixOS x86_64 with an RTX 5090 32 GB and NVIDIA driver 610.43.02. Performance, memory use, and successful installation on another system can differ.

Credit Qwen for the base model; r0b0tlab for the EXL3 target, DFlash2 draft, and community ExLlamaV3 runtime; and SatelliteDown for the SGLang NVFP4-KV patch. CIRU built the EXL3-to-SGLang serving integration and runner. SGLang, FlashInfer, PyTorch, and NVIDIA provide the underlying inference stack. The relevant upstream licenses and modification notices are retained by the runner.