Gemma 4 31B QAT Q4_0_V2
Quantization of Google's Gemma 4 31B QAT checkpoint using a modified Q4_0 encoder that better preserves QAT weight geometry under standard GGUF FP16 block scales.
Q4_0_V2 achieves approximately 7× lower KL divergence than standard Q4_0 while preserving the same storage format and runtime characteristics.
Q4_0_V2 will run in normal mainline llama.cpp.
Gemma 4 31B — KL divergence vs unquantized BF16 QAT checkpoint
| Rank | Model | PPL ratio | Mean KLD | Δp RMS | Same top-p |
|---|---|---|---|---|---|
| 🥇 | pajari/gemma-4-31B-it-qat-Q4_0_V2 | 1.00533 | 0.001990 | 1.408% | 99.416% |
| 🥈 | unsloth/gemma-4-31B-it-qat-GGUF | 1.00580 | 0.002198 | 1.537% | 99.339% |
| 🥉 | google/gemma-4-31B-it-qat-q4_0-gguf | 0.99484 | 0.013254 | 3.477% | 97.666% |
| 4 | Stock Q4_0 | 0.99870 | 0.013962 | 3.582% | 97.564% |
Lower is better for KLD and Δp RMS, higher is better for Same top-p, PPL ratio is best closest to 1.0. Bold marks the best score in each column.
Models are ranked by Mean KLD (lower is better).
Logs
.\llama.cpp\build_hip\bin\llama-perplexity.exe --override-kv gemma4.final_logit_softcapping=float:30.0,tokenizer.ggml.add_bos_token=bool:false --device rocm0 -ngl 30 -c 8192 -m .\model\gemma-4-31B-it-qat-q4_0-unquantized\gemma-4-31B-it-qat-q4_0-unquantized-gguf-model-bf16.gguf -f .\dataset\calibration\gemma-4-think-single-chat.txt --kl-divergence-base .\model\base_logits.kld
HIP Library Path: C:\WINDOWS\SYSTEM32\amdhip64_7.dll
0.00.178.215 I common_init_result: fitting params to device memory ...
0.00.178.218 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
0.02.053.395 W load: control-looking token: 212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
0.02.088.101 W load: gemma4: removing </s> (212) from EOG (normal text token)
0.21.847.120 W llama_context: n_ctx_seq (8192) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
0.22.113.053 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
0.38.521.077 I
0.38.526.884 I system_info: n_threads = 12 (n_threads_batch = 12) / 24 | ROCm : NO_VMM = 1 | PEER_MAX_BATCH_SIZE = 128 | FA_ALL_QUANTS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
0.38.528.042 I perplexity: saving all logits to .\model\base_logits.kld
0.38.528.050 I perplexity: tokenizing the input ..
0.38.689.954 I perplexity: tokenization took 161.891 ms
0.38.690.086 I perplexity: calculating perplexity over 12 chunks, n_ctx=8192, batch_size=2048, n_seq=1
2.36.876.928 I perplexity: 116.76 seconds per pass - ETA 23.35 minutes
[1]1.7538,[2]1.9196,[3]1.9008,[4]2.0169,[5]2.0069,[6]2.0836,[7]2.0426,[8]2.0783,[9]2.1363,[10]2.2036,[11]2.3006,[12]2.3660,
22.26.827.359 I Final estimate: PPL = 2.3660 +/- 0.02526
Unsloth q4_0
.\llama.cpp\build_hip\bin\llama-perplexity.exe --override-kv gemma4.final_logit_softcapping=float:30.0,tokenizer.ggml.add_bos_token=bool:false -c 8192 --device rocm0 -ngl all -m .\model\gemma-4-31B-it-qat-q4_0-unquantized\unsloth-gemma-4-31B-it-qat-UD-Q4_K_XL.gguf --kl-divergence-base .\model\base_logits.kld --kl-divergence -f .\dataset\calibration\gemma-4-think-single-chat.txt
HIP Library Path: C:\WINDOWS\SYSTEM32\amdhip64_7.dll
0.00.204.407 I common_init_result: fitting params to device memory ...
0.00.204.410 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
0.01.408.473 W load: control-looking token: 212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
0.01.439.222 W load: gemma4: removing </s> (212) from EOG (normal text token)
0.12.907.460 W llama_context: n_ctx_seq (8192) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
0.12.998.779 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
0.13.112.335 I
0.13.112.386 I system_info: n_threads = 12 (n_threads_batch = 12) / 24 | ROCm : NO_VMM = 1 | PEER_MAX_BATCH_SIZE = 128 | FA_ALL_QUANTS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
0.13.505.695 I kl_divergence: computing over 12 chunks, n_ctx=8192, batch_size=2048, n_seq=1
0.25.381.788 I kl_divergence: 11.87 seconds per pass - ETA 2.37 minutes
chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p
1 1.7560 ± 0.0510 0.00166 ± 0.00096 0.00107 ± 0.00029 1.223 ± 0.363 % 99.634 ± 0.094 %
2 1.9218 ± 0.0429 0.00212 ± 0.00076 0.00099 ± 0.00017 1.071 ± 0.212 % 99.475 ± 0.080 %
3 1.9028 ± 0.0347 0.00220 ± 0.00070 0.00168 ± 0.00042 1.402 ± 0.234 % 99.487 ± 0.064 %
4 2.0198 ± 0.0332 0.00364 ± 0.00101 0.00184 ± 0.00039 1.515 ± 0.250 % 99.426 ± 0.059 %
5 2.0095 ± 0.0297 0.00404 ± 0.00096 0.00172 ± 0.00032 1.432 ± 0.213 % 99.443 ± 0.052 %
6 2.0853 ± 0.0290 0.00374 ± 0.00089 0.00165 ± 0.00027 1.417 ± 0.191 % 99.451 ± 0.047 %
7 2.0444 ± 0.0261 0.00380 ± 0.00082 0.00172 ± 0.00025 1.394 ± 0.169 % 99.445 ± 0.044 %
8 2.0801 ± 0.0251 0.00412 ± 0.00080 0.00188 ± 0.00025 1.501 ± 0.156 % 99.396 ± 0.043 %
9 2.1380 ± 0.0249 0.00465 ± 0.00078 0.00194 ± 0.00026 1.487 ± 0.143 % 99.411 ± 0.040 %
10 2.2051 ± 0.0249 0.00509 ± 0.00079 0.00231 ± 0.00030 1.596 ± 0.133 % 99.387 ± 0.039 %
11 2.3031 ± 0.0253 0.00556 ± 0.00076 0.00225 ± 0.00028 1.557 ± 0.124 % 99.354 ± 0.038 %
12 2.3683 ± 0.0253 0.00578 ± 0.00080 0.00220 ± 0.00026 1.537 ± 0.117 % 99.339 ± 0.037 %
====== Perplexity statistics ======
Mean PPL(Q) : 2.368275 ± 0.025304
Mean PPL(base) : 2.354614 ± 0.024749
Cor(ln(PPL(Q)), ln(PPL(base))): 99.73%
Mean ln(PPL(Q)/PPL(base)) : 0.005785 ± 0.000800
Mean PPL(Q)/PPL(base) : 1.005802 ± 0.000804
Mean PPL(Q)-PPL(base) : 0.013660 ± 0.001925
====== KL divergence statistics ======
Mean KLD: 0.002198 ± 0.000257
Maximum KLD: 5.927129
99.9% KLD: 0.257504
99.0% KLD: 0.013956
95.0% KLD: 0.003716
90.0% KLD: 0.002043
Median KLD: 0.000007
10.0% KLD: -0.000000
5.0% KLD: -0.000001
1.0% KLD: -0.000002
0.1% KLD: -0.000024
Minimum KLD: -0.000109
====== Token probability statistics ======
Mean Δp: -0.002 ± 0.007 %
Maximum Δp: 75.997%
99.9% Δp: 9.896%
99.0% Δp: 2.580%
95.0% Δp: 0.668%
90.0% Δp: 0.163%
75.0% Δp: 0.002%
Median Δp: 0.000%
25.0% Δp: -0.003%
10.0% Δp: -0.183%
5.0% Δp: -0.713%
1.0% Δp: -2.528%
0.1% Δp: -9.386%
Minimum Δp: -96.647%
RMS Δp : 1.537 ± 0.117 %
Same top p: 99.339 ± 0.037 %
Google q4_0
.\llama.cpp\build_hip\bin\llama-perplexity.exe --override-kv gemma4.final_logit_softcapping=float:30.0,tokenizer.ggml.add_bos_token=bool:false -c 8192 --device rocm0 -ngl all -m .\model\gemma-4-31B-it-qat-q4_0-unquantized\google-gemma-4-31B_q4_0-it.gguf --kl-divergence-base .\model\base_logits.kld --kl-divergence -f .\dataset\calibration\gemma-4-think-single-chat.txt
HIP Library Path: C:\WINDOWS\SYSTEM32\amdhip64_7.dll
0.00.171.986 I common_init_result: fitting params to device memory ...
0.00.171.989 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
0.01.364.326 W load: control-looking token: 212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
0.01.395.934 W load: gemma4: removing </s> (212) from EOG (normal text token)
0.13.204.258 W llama_context: n_ctx_seq (8192) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
0.13.289.629 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
0.13.385.253 I
0.13.385.285 I system_info: n_threads = 12 (n_threads_batch = 12) / 24 | ROCm : NO_VMM = 1 | PEER_MAX_BATCH_SIZE = 128 | FA_ALL_QUANTS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
0.13.760.755 I kl_divergence: computing over 12 chunks, n_ctx=8192, batch_size=2048, n_seq=1
0.25.446.538 I kl_divergence: 11.69 seconds per pass - ETA 2.33 minutes
chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p
1 1.7425 ± 0.0500 -0.00604 ± 0.00319 0.00848 ± 0.00119 2.724 ± 0.254 % 98.535 ± 0.188 %
2 1.9103 ± 0.0422 -0.00388 ± 0.00267 0.00985 ± 0.00095 3.139 ± 0.272 % 98.388 ± 0.139 %
3 1.8894 ± 0.0340 -0.00486 ± 0.00210 0.00986 ± 0.00072 3.099 ± 0.212 % 98.372 ± 0.114 %
4 2.0048 ± 0.0326 -0.00380 ± 0.00198 0.01016 ± 0.00057 3.143 ± 0.173 % 98.132 ± 0.106 %
5 1.9948 ± 0.0292 -0.00332 ± 0.00180 0.01035 ± 0.00051 3.175 ± 0.149 % 98.129 ± 0.095 %
6 2.0686 ± 0.0284 -0.00428 ± 0.00167 0.01117 ± 0.00046 3.235 ± 0.132 % 97.998 ± 0.089 %
7 2.0270 ± 0.0256 -0.00478 ± 0.00151 0.01066 ± 0.00041 3.171 ± 0.122 % 98.050 ± 0.082 %
8 2.0628 ± 0.0246 -0.00422 ± 0.00143 0.01101 ± 0.00045 3.246 ± 0.119 % 97.973 ± 0.078 %
9 2.1186 ± 0.0244 -0.00447 ± 0.00141 0.01169 ± 0.00046 3.331 ± 0.117 % 97.916 ± 0.074 %
10 2.1848 ± 0.0244 -0.00415 ± 0.00136 0.01227 ± 0.00047 3.386 ± 0.107 % 97.836 ± 0.072 %
11 2.2792 ± 0.0247 -0.00487 ± 0.00132 0.01296 ± 0.00044 3.443 ± 0.101 % 97.707 ± 0.071 %
12 2.3425 ± 0.0247 -0.00518 ± 0.00132 0.01325 ± 0.00042 3.477 ± 0.095 % 97.666 ± 0.068 %
====== Perplexity statistics ======
Mean PPL(Q) : 2.342454 ± 0.024702
Mean PPL(base) : 2.354614 ± 0.024749
Cor(ln(PPL(Q)), ln(PPL(base))): 99.21%
Mean ln(PPL(Q)/PPL(base)) : -0.005178 ± 0.001324
Mean PPL(Q)/PPL(base) : 0.994836 ± 0.001318
Mean PPL(Q)-PPL(base) : -0.012160 ± 0.003110
====== KL divergence statistics ======
Mean KLD: 0.013254 ± 0.000418
Maximum KLD: 6.996171
99.9% KLD: 1.077398
99.0% KLD: 0.153186
95.0% KLD: 0.054351
90.0% KLD: 0.031315
Median KLD: 0.000065
10.0% KLD: 0.000000
5.0% KLD: -0.000000
1.0% KLD: -0.000001
0.1% KLD: -0.000005
Minimum KLD: -0.000027
====== Token probability statistics ======
Mean Δp: -0.076 ± 0.016 %
Maximum Δp: 92.678%
99.9% Δp: 27.057%
99.0% Δp: 9.345%
95.0% Δp: 2.392%
90.0% Δp: 0.618%
75.0% Δp: 0.011%
Median Δp: 0.000%
25.0% Δp: -0.007%
10.0% Δp: -0.759%
5.0% Δp: -3.051%
1.0% Δp: -10.076%
0.1% Δp: -30.930%
Minimum Δp: -95.720%
RMS Δp : 3.477 ± 0.095 %
Same top p: 97.666 ± 0.068 %
Stock llama.cpp q4_0
.\llama.cpp\build_hip\bin\llama-perplexity.exe --override-kv gemma4.final_logit_softcapping=float:30.0,tokenizer.ggml.add_bos_token=bool:false -c 8192 --device rocm0 -ngl all -m .\model\gemma-4-31B-it-qat-q4_0-unquantized\gemma-4-31B-it-qat-q4_0-unquantized-gguf-model-Q4_0.gguf --kl-divergence-base .\model\base_logits.kld --kl-divergence -f .\dataset\calibration\gemma-4-think-single-chat.txt
HIP Library Path: C:\WINDOWS\SYSTEM32\amdhip64_7.dll
0.00.172.402 I common_init_result: fitting params to device memory ...
0.00.172.405 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
0.01.361.246 W load: control-looking token: 212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
0.01.393.176 W load: gemma4: removing </s> (212) from EOG (normal text token)
0.12.161.758 W llama_context: n_ctx_seq (8192) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
0.12.253.706 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
0.12.376.249 I
0.12.376.298 I system_info: n_threads = 12 (n_threads_batch = 12) / 24 | ROCm : NO_VMM = 1 | PEER_MAX_BATCH_SIZE = 128 | FA_ALL_QUANTS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
0.12.937.129 I kl_divergence: computing over 12 chunks, n_ctx=8192, batch_size=2048, n_seq=1
0.24.418.515 I kl_divergence: 11.48 seconds per pass - ETA 2.28 minutes
chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p
1 1.7455 ± 0.0503 -0.00433 ± 0.00322 0.00887 ± 0.00121 2.758 ± 0.256 % 98.413 ± 0.195 %
2 1.9156 ± 0.0425 -0.00115 ± 0.00285 0.01025 ± 0.00115 3.013 ± 0.274 % 98.242 ± 0.145 %
3 1.8952 ± 0.0343 -0.00176 ± 0.00221 0.01032 ± 0.00087 3.103 ± 0.231 % 98.282 ± 0.117 %
4 2.0093 ± 0.0328 -0.00157 ± 0.00206 0.01053 ± 0.00067 3.080 ± 0.181 % 98.046 ± 0.108 %
5 1.9987 ± 0.0293 -0.00137 ± 0.00185 0.01100 ± 0.00065 3.214 ± 0.169 % 98.017 ± 0.097 %
6 2.0729 ± 0.0286 -0.00220 ± 0.00173 0.01191 ± 0.00059 3.334 ± 0.158 % 97.875 ± 0.092 %
7 2.0316 ± 0.0257 -0.00251 ± 0.00156 0.01135 ± 0.00052 3.267 ± 0.143 % 97.952 ± 0.084 %
8 2.0686 ± 0.0247 -0.00140 ± 0.00148 0.01177 ± 0.00054 3.345 ± 0.135 % 97.875 ± 0.080 %
9 2.1260 ± 0.0246 -0.00096 ± 0.00144 0.01227 ± 0.00053 3.401 ± 0.128 % 97.827 ± 0.076 %
10 2.1919 ± 0.0245 -0.00092 ± 0.00139 0.01279 ± 0.00050 3.453 ± 0.117 % 97.734 ± 0.074 %
11 2.2881 ± 0.0249 -0.00097 ± 0.00136 0.01367 ± 0.00051 3.540 ± 0.113 % 97.611 ± 0.072 %
12 2.3516 ± 0.0249 -0.00130 ± 0.00136 0.01396 ± 0.00048 3.582 ± 0.106 % 97.564 ± 0.070 %
====== Perplexity statistics ======
Mean PPL(Q) : 2.351559 ± 0.024908
Mean PPL(base) : 2.354614 ± 0.024749
Cor(ln(PPL(Q)), ln(PPL(base))): 99.18%
Mean ln(PPL(Q)/PPL(base)) : -0.001298 ± 0.001357
Mean PPL(Q)/PPL(base) : 0.998702 ± 0.001355
Mean PPL(Q)-PPL(base) : -0.003055 ± 0.003192
====== KL divergence statistics ======
Mean KLD: 0.013962 ± 0.000476
Maximum KLD: 7.633423
99.9% KLD: 1.140293
99.0% KLD: 0.153066
95.0% KLD: 0.056004
90.0% KLD: 0.032574
Median KLD: 0.000080
10.0% KLD: 0.000001
5.0% KLD: 0.000000
1.0% KLD: -0.000001
0.1% KLD: -0.000002
Minimum KLD: -0.000011
====== Token probability statistics ======
Mean Δp: -0.047 ± 0.016 %
Maximum Δp: 95.999%
99.9% Δp: 27.728%
99.0% Δp: 9.559%
95.0% Δp: 2.588%
90.0% Δp: 0.666%
75.0% Δp: 0.012%
Median Δp: 0.000%
25.0% Δp: -0.007%
10.0% Δp: -0.688%
5.0% Δp: -2.970%
1.0% Δp: -10.155%
0.1% Δp: -32.910%
Minimum Δp: -99.760%
RMS Δp : 3.582 ± 0.106 %
Same top p: 97.564 ± 0.070 %
q4_0_v2
.\llama.cpp\build_hip\bin\llama-perplexity.exe --override-kv gemma4.final_logit_softcapping=float:30.0,tokenizer.ggml.add_bos_token=bool:false -c 8192 --device rocm0 -ngl all -m .\model\gemma-4-31B-it-qat-q4_0-unquantized\gemma-4-31B-it-qat-gguf-model-Q4_0_V2_QAT.gguf --kl-divergence-base .\model\base_logits.kld --kl-divergence -f .\dataset\calibration\gemma-4-think-single-chat.txt
HIP Library Path: C:\WINDOWS\SYSTEM32\amdhip64_7.dll
0.00.312.798 I common_init_result: fitting params to device memory ...
0.00.312.801 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
0.01.448.995 W load: control-looking token: 212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
0.01.487.202 W load: gemma4: removing </s> (212) from EOG (normal text token)
0.09.133.145 W llama_context: n_ctx_seq (8192) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
0.09.217.400 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
0.09.307.267 I
0.09.307.299 I system_info: n_threads = 12 (n_threads_batch = 12) / 24 | ROCm : NO_VMM = 1 | PEER_MAX_BATCH_SIZE = 128 | FA_ALL_QUANTS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
0.09.515.347 I kl_divergence: computing over 12 chunks, n_ctx=8192, batch_size=2048, n_seq=1
0.20.486.565 I kl_divergence: 10.97 seconds per pass - ETA 2.18 minutes
chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p
1 1.7530 ± 0.0509 -0.00003 ± 0.00083 0.00093 ± 0.00027 0.728 ± 0.070 % 99.829 ± 0.065 %
2 1.9206 ± 0.0429 0.00149 ± 0.00075 0.00089 ± 0.00016 0.795 ± 0.050 % 99.670 ± 0.063 %
3 1.9027 ± 0.0347 0.00216 ± 0.00072 0.00164 ± 0.00044 1.277 ± 0.231 % 99.617 ± 0.056 %
4 2.0180 ± 0.0331 0.00277 ± 0.00097 0.00151 ± 0.00033 1.255 ± 0.187 % 99.560 ± 0.052 %
5 2.0078 ± 0.0297 0.00317 ± 0.00092 0.00142 ± 0.00027 1.191 ± 0.159 % 99.570 ± 0.046 %
6 2.0843 ± 0.0290 0.00325 ± 0.00086 0.00141 ± 0.00024 1.152 ± 0.138 % 99.577 ± 0.041 %
7 2.0436 ± 0.0261 0.00338 ± 0.00080 0.00150 ± 0.00022 1.165 ± 0.122 % 99.574 ± 0.038 %
8 2.0793 ± 0.0251 0.00373 ± 0.00078 0.00159 ± 0.00020 1.280 ± 0.114 % 99.509 ± 0.039 %
9 2.1367 ± 0.0249 0.00405 ± 0.00076 0.00169 ± 0.00022 1.309 ± 0.106 % 99.517 ± 0.036 %
10 2.2043 ± 0.0249 0.00474 ± 0.00077 0.00215 ± 0.00028 1.482 ± 0.114 % 99.490 ± 0.035 %
11 2.3017 ± 0.0253 0.00495 ± 0.00075 0.00209 ± 0.00026 1.447 ± 0.106 % 99.438 ± 0.035 %
12 2.3672 ± 0.0253 0.00532 ± 0.00079 0.00199 ± 0.00024 1.408 ± 0.100 % 99.416 ± 0.034 %
====== Perplexity statistics ======
Mean PPL(Q) : 2.367174 ± 0.025279
Mean PPL(base) : 2.354614 ± 0.024749
Cor(ln(PPL(Q)), ln(PPL(base))): 99.73%
Mean ln(PPL(Q)/PPL(base)) : 0.005320 ± 0.000791
Mean PPL(Q)/PPL(base) : 1.005334 ± 0.000795
Mean PPL(Q)-PPL(base) : 0.012560 ± 0.001900
====== KL divergence statistics ======
Mean KLD: 0.001990 ± 0.000237
Maximum KLD: 5.916455
99.9% KLD: 0.229264
99.0% KLD: 0.013512
95.0% KLD: 0.003680
90.0% KLD: 0.001975
Median KLD: 0.000007
10.0% KLD: -0.000000
5.0% KLD: -0.000001
1.0% KLD: -0.000003
0.1% KLD: -0.000027
Minimum KLD: -0.000136
====== Token probability statistics ======
Mean Δp: -0.010 ± 0.006 %
Maximum Δp: 76.061%
99.9% Δp: 8.657%
99.0% Δp: 2.482%
95.0% Δp: 0.641%
90.0% Δp: 0.147%
75.0% Δp: 0.002%
Median Δp: 0.000%
25.0% Δp: -0.004%
10.0% Δp: -0.184%
5.0% Δp: -0.717%
1.0% Δp: -2.585%
0.1% Δp: -8.989%
Minimum Δp: -74.365%
RMS Δp : 1.408 ± 0.100 %
Same top p: 99.416 ± 0.034 %