← back to normiebench

about normiebench

normiebench is a simple, lightweight, open-source benchmark for comparing real-world LLM inference speeds across everyday hardware. the goal is to make it easy for anyone — not just developers — to see how fast their machine runs a standard AI model and compare it to others.

what it does

run one command and it handles everything automatically:

• detects your CPU, GPU, RAM, and backend (CUDA, Vulkan, Metal, or CPU)
• downloads a pinned llama.cpp release for reproducible comparisons
• downloads the official Qwen2.5 7B Q4_K_M model from Hugging Face
• verifies every download and runs the short Basic suite or optional Advanced suite
• saves your result locally and submits it unless you select local-only mode

the benchmark

every leaderboard machine runs the same test: Qwen2.5 7B Instruct, Q4_K_M quantization, using pinned model bytes, a pinned llama.cpp build, and fixed batch settings. two primary numbers are measured:

• pp256 — prompt processing speed: how fast the model reads a 256-token prompt (tokens/second)
• tg32 — token generation speed: how fast the model generates 32 tokens (tokens/second)

Basic Score converts those rates into the estimated completion time of one representative prompt-and-response workload. Advanced Score combines chat, writing, summarization, and long-document shapes. results also include variation, raw samples, phase timings, effective settings, hardware details, and optional temperature telemetry. sampling temperature does not apply because llama-bench does not sample output text.

same model, same quant, same tool — across every machine. apples to apples.

how to run it

linux / mac:

windows:

downloads are stored in the user cache so subsequent runs are fast. the verified binary, engine, and model are only downloaded once, and interrupted model downloads resume.

is it safe?

yes. the scripts are minimal and fully open source — you can read every line before running them. here is exactly what they do:

• download llama.cpp from github.com/ggml-org/llama.cpp/releases (official releases only)
• download the model from the official Qwen Hugging Face repository
• run llama-bench locally on your machine — no data leaves your machine during the benchmark itself
• submit allowlisted hardware specs and raw benchmark samples unless local-only mode is requested

nothing is installed system-wide. everything goes into ~/.normiebench/ and can be deleted at any time.

open source

normiebench is fully open source and licensed under the GNU General Public License v3.0 (GPL-3.0). the source code for the benchmark scripts, leaderboard website, and submission pipeline is all public:

gitlab.com/brando69/normiebench

contributions, bug reports, and forks are welcome.

submission pipeline

when you submit results, this is what happens:

• your hardware specs and scores are POSTed to a Cloudflare Worker
• the Worker validates the immutable benchmark profile and allowlists every stored field
• a private random machine ID is converted to a non-reversible public ID with a server-side HMAC
• the accepted result is written directly to Cloudflare D1; no repository issue, CI job, or automated commit is created
• the leaderboard reads the results API and falls back to a checked-in snapshot during an outage

if you run the benchmark again on the same machine, your previous entry is replaced with the latest result. see the privacy policy for full details on what is and isn't collected.

credits

normiebench is built on top of llama.cpp by Georgi Gerganov and contributors — the engine that does the actual inference. the benchmark model and GGUF quantization are published by the Qwen team under Apache 2.0.