Are compressed models faster? We timed 27 copies.
In our demo game the 4-bit copy of Qwen2.5-1.5B usually answers first. We went through 222 GPU runs to see whether speed can expose a copy anywhere else.
Kartikeya Sharma, Bhag Labs
- 9 of 27
- compressed copies ran slower than the original on the same GPU and settings
- 13×
- slowdown from one honest server setting, on the real Qwen2.5-7B
- 47 to 50%
- how often a single 4-bit answer was the shorter one
The game on our home page puts two providers side by side on one NVIDIA L4. One serves Qwen2.5-1.5B-Instruct. The other serves a 4-bit copy while claiming to be the same model. Players noticed that the copy usually finishes first, so we checked whether speed gives a copy away in general.
We used 222 runs of 14 open models on Google Cloud A100, H100 and L4 GPUs, plus 87 live SPOT checks of our demo providers. Speed below is tokens generated per second in our long-answer test: 25 prompts, 5,000 answers of up to 50 tokens each, 64 requests in flight, the way a provider serves many users at once.
A third of the copies were slower
We ran 12 4-bit copies (AWQ, GPTQ, W4A16, QAT) and 15 8-bit copies (FP8, W8A8, GPTQ) of 11 models, each against the official weights on the same GPU, engine and settings. 18 were faster and 9 were slower. The largest gain was 1.41×, for an 8-bit Llama 3.1 8B. The 4-bit Gemma 3 27B ran at 0.91× and the 8-bit GPTQ Qwen2.5 7B at 0.84×.
| Gemma 3 27B | 8-bit FP8 | A100 | 0.92× |
| Gemma 3 27B | 4-bit W4A16 | A100 | 0.91× |
| Gemma 4 26B-A4B | 8-bit FP8 | A100 | 1.19× |
| Gemma 4 26B-A4B | 4-bit AWQ | A100 | 1.22× |
| Gemma 4 26B-A4B | 8-bit FP8 | A100 | 1.09× |
| Gemma 4 26B-A4B | 4-bit AWQ | A100 | 1.13× |
| Gemma 4 31B | 8-bit FP8 | A100 | 0.87× |
| Gemma 4 31B | 4-bit QAT | A100 | 0.90× |
| Llama 3.1 8B | 4-bit AWQ | A100 | 0.96× |
| Llama 3.1 8B | 8-bit W8A8 | A100 | 1.41× |
| Llama 3.1 8B | 8-bit FP8 | A100 | 0.98× |
| Mistral Nemo 12B | 4-bit AWQ | A100 | 0.98× |
| Mistral Nemo 12B | 8-bit W8A8 | A100 | 1.40× |
| Nemotron 3.5 30B-A3B | 8-bit FP8 | A100 | 1.08× |
| Qwen3 30B-A3B | 4-bit AWQ | A100 | 1.32× |
| Qwen3 30B-A3B | 8-bit FP8 | A100 | 1.20× |
| Qwen3.6 27B | 4-bit GPTQ | A100 | 1.12× |
| Qwen3.6 27B | 8-bit FP8 | A100 | 1.02× |
| Qwen3.6 27B | 4-bit AWQ | A100 | 1.11× |
| Qwen3.6 35B-A3B | 4-bit AWQ | A100 | 1.06× |
| Qwen3.6 35B-A3B | 4-bit GPTQ | A100 | 1.12× |
| Qwen3.8 27B | 8-bit FP8 (eager) | A100 | 1.24× |
| Gemma 4 26B-A4B | 8-bit FP8 | H100 | 1.23× |
| Gemma 4 26B-A4B | 8-bit FP8 (SGLang) | H100 | 0.96× |
| Qwen2.5 7B | 8-bit GPTQ | L4 | 0.84× |
| Qwen2.5 7B | 4-bit AWQ | L4 | 1.13× |
| Qwen2.5 7B | 8-bit FP8 | L4 | 1.35× |
Compressed weights are unpacked before every multiply. With 64 requests in flight the GPU is busy with arithmetic, so the smaller download often doesn't cover the cost of unpacking.
Honest providers vary more than copies do
We also ran the official weights the ways an honest provider might: SGLang or a newer vLLM, prefix caching off, CUDA graphs off, other seeds and batch limits. On the same A100, the official Qwen3.6 27B produced 182 tokens per second on SGLang and 885 to 956 on vLLM. On an L4, Qwen2.5 7B ran 13× slower with the server set to run 8 answers at a time instead of 256.
If anything faster than the official default counts as a copy, 27 of 104 honest runs get called copies too. Speed does carry some signal: 15 of the 27 copies beat every honest setup we ran for that model on that GPU. To use it, a buyer would need to know the provider's GPU and the fastest honest speed for the model on it.
The GPU matters more than the weights
The official Gemma 4 26B-A4B produced 10,063 tokens per second on an H100 and 5,701 on an A100. Its 4-bit copy on the A100 reached 6,981, still 31% behind the original on the newer card. Gemma 4 31B ran 2.3× faster on the H100 than on the A100.
Live checks overlap
Our command-line demo serves Qwen2.5-7B-Instruct and its 4-bit AWQ copy from one machine. On 1 November, 13 SPOT checks of the real model on an L4 took 22.7 to 25.0 seconds and 6 checks of the copy took 21.7 to 23.3. On 21 October the copy was clearly faster: 10.5 seconds against 13.6 to 14.4. On 28 October, on an A100, the two ranges were 8.2 to 32.3 and 7.7 to 32.0 seconds.
Answer length doesn't help either
For every prompt in our tests we paired single answers from Qwen2.5 7B and its 4-bit copy at random and compared their length in characters. The copy's answer was the shorter one 47% to 50% of the time, depending on the test. Per SPOT check, the copy used 26,693 to 27,044 completion tokens and the real model 26,347 to 26,967.
Why the game looks different
In the game, one person chats with two models on one GPU. With a single request in flight, the L4 spends most of each step reading weights from memory, and a 4-bit copy has about a third of the bytes to read. Providers batch many users, run different GPUs and engines, and can add delay to any answer.
authenticated.si ignores timing. SPOT sends secret questions and compares the answers with the official model's. On the same 14 models it flagged 82% of checks on fake setups and 10.3% of checks on honest ones.
How we measured
- Runs: October 2026, vLLM and SGLang, temperature 1.0, on Google Cloud A100 80GB, H100 80GB and L4 GPUs. Each run logs its GPU, driver and library versions.
- Copies are public quantised releases of each model. For Qwen3.6 35B-A3B the official release is FP8, so its copies are compared with that.
- Two copies ran with non-default settings (CUDA graphs off, or SGLang). In the first chart they are compared with the official weights under the same settings; the second chart shows them against our default setup.
- Single-user speed was measured only for Qwen2.5 7B on an L4. None of these runs used a third-party provider.
- Live check times come from SPOT's signed certificates. Ask for the tables at hello@authenticated.si.