Jev_Qwen3.8-27B EXL3 4.0 bpw
EXL3 4.0 bpw conversion of SargeDev/Jev_Qwen3.8-27B.
Build provenance
- Source revision: dbe134f5cfd084b7a89dc7851db31c83fa8d7434
- Architecture: Qwen3_5ForCausalLM
- ExLlamaV3: 1.5.2+cu128.torch2.10.0
- PyTorch: 2.10.0+cu128
- Target bitrate: 4.00 bpw
- Source license: Apache-2.0
- Artifact size: approximately 14.67 GiB
RTX 4090 results
TabbyAPI, RTX 4090 only, FP16 KV cache, max batch size 1. Thinking was disabled for the context throughput sweep so decode numbers measure visible generation rather than variable reasoning length.
| Target context | Actual prompt | Prefill tok/s | Decode tok/s | Wall time | Needle |
|---|---|---|---|---|---|
| ~1K | 1,035 | 885 | 53.7 | 1.62s | PASS |
| ~4K | 4,113 | 2,654 | 53.2 | 2.01s | PASS |
| ~8K | 8,198 | 2,277 | 52.5 | 4.08s | PASS |
| ~16K | 16,390 | 2,656 | 51.1 | 6.65s | PASS |
| ~31K | 32,008 | 2,227 | 48.3 | 15.03s | PASS |
All five retrieval needles passed through approximately 32K prompt tokens.
Observed decode throughput stayed near 54 tok/s at short context and remained 48.3 tok/s around 32K.
Observed GPU0 memory during the sweep was approximately 16.4-16.7 GiB total device usage. This is an observed whole-device figure, not isolated model allocation.
Serving validation
The artifact passed direct ExLlamaV3 load and generation, TabbyAPI model load, OpenAI-compatible chat completions, streaming responses, structured tool-call parsing, two simultaneous API requests, the 8K TabbyAPI smoke, and the 32K retrieval sweep.
This checkpoint is text-only, so vision validation was correctly skipped.
The short warmed TabbyAPI smoke generated at 53.52 tok/s.
Repetition / generation stability smoke
Three sampled generations were tested with thinking disabled and up to 1,024 output tokens.
- Obvious repeated n-gram loops detected: 0/3
- Longest observed completion in this small sample: 489 tokens
- Decode throughput across these samples: 53.7-54.1 tok/s
These samples ended naturally before 1,024 tokens, so this is a stability smoke test rather than an EOS or exhaustive anti-loop benchmark.
Raw benchmark data: benchmarks/rtx4090-tabbyapi.json.
Notes on the numbers
These are single-machine local smoke measurements, not standardized cross-system benchmark scores. Prefill can vary with kernel compilation, autotuning, prompt shape, cache state, and runtime configuration.
The direct minimal ExLlamaV3 probe measured 40.475 tok/s. The warmed TabbyAPI path was materially faster, so the context table above is the more useful serving-oriented measurement.
TabbyAPI configuration used
model:
backend: exllamav3
max_seq_len: 32768
cache_size: 32768
cache_mode: FP16
gpu_split_auto: false
gpu_split: [23, 0]
max_batch_size: 1
vision: false
reasoning: true
tool_format: qwen3_5
Upstream
This is an independent community quantization. Refer to the upstream model card for model purpose, limitations, and license terms.