GGUF Quantization Quality Benchmark: Methodology

What is KL divergence?

KL divergence measures how different the quantized model’s token probability distribution is from the original model’s. A KL of 0 means the quant is identical to the original. Higher values mean more information is lost.

Unlike perplexity, KL divergence directly compares two models against each other. It answers the question: “how much does this quant change the model’s output?”

Why not perplexity?

Perplexity measures how well a model predicts text. It doesn’t directly answer “how much did quantization change this model?” Two models can have similar perplexity but very different token distributions. KL divergence captures exactly what was lost.

Why are these numbers higher than usual?

Most KL divergence benchmarks use Wikipedia with ~2048 token context. It’s a lot easier to score well there. This benchmark uses real-world inputs up to ~30k tokens across 6 task categories, which is much harder on quantized models. If you’ve seen KL values of 0.01-0.03 for Q4_K_M elsewhere, that’s why.

KL divergence is dataset-dependent, same as perplexity. The difference is that KL divergence compares two models against each other, while perplexity only measures one model against the evaluation text.

Inference

All measurements are collected using TextGen’s OpenAI-compatible API.

llama.cpp doesn’t natively support extracting logprobs from the prompt, so I patched it. The patch is included in the llama.cpp binary shipped with TextGen. Source: commit on my llama.cpp fork.

Reference model

The BF16 GGUF from unsloth is used as the reference for each model. BF16 is lossless relative to the original weights.

Dataset

~250,000 tokens across 6 categories:

• Coding: programming tasks and code generation

• General chat: typical assistant conversations

• Tool calling: samples with tool definitions and tool_calls, rendered through the model’s chat template

• Science: technical and scientific questions

• Non-Latin scripts: Chinese, Japanese, Korean, Arabic, Cyrillic, and others

• Long documents: extended context inputs up to ~30k tokens

Every sample is a full OpenAI-compatible messages array, not raw text. The samples go through the model’s Jinja2 chat template, so the benchmark captures how quantization affects real usage including special tokens, system prompts, and tool formatting.

Measurement process

KL divergence is computed on prompt tokens, not on generated text. The model processes the prompt and returns log-probabilities for each token position, which are then compared against the reference.

1. The Jinja2 chat template is taken from the official model release (not from the GGUF metadata). The same template is used for every quant of that model.

2. Each sample’s messages are converted to a prompt string using the /v1/internal/chat-prompt endpoint.

3. Each prompt is sent to /v1/completions with echo: true and logprobs: 40, which returns the top-40 token log-probabilities for every token in the prompt.

4. For each token position, the KL is computed over the union of tokens that appear in either model’s top-40 list. For tokens missing from one list, a floor logprob is used (lowest observed logprob in that distribution minus 2).

5. The per-token KL values are averaged across all tokens and all prompts to produce the final number.

Why top-40?

Top-40 covers virtually all the probability mass in practice. Tokens outside the top 40 contribute negligible probability mass, so the approximation error is negligible.

Top-1 agreement

In addition to KL divergence, each benchmark reports top-1 agreement: the percentage of tokens where the quantized model’s most likely token matches the reference model’s most likely token. A top-1 agreement of 85% means that for 85 out of every 100 tokens, the quant would pick the same token as the original under greedy decoding.