about normiebench

free, open source, one command. it measures how fast a local llm actually runs on your machine, whether that's a mini pc, a pi-class board, an old gaming gpu or a current one.

why this exists

i wanted to know how my own machines stacked up and there was no simple way to find out. so this is a shot at the most consistent, reliable, boring way to measure normal consumer hardware running a strict, controlled llm, without dumping junk on your computer.

that shaped all of it. one command. same fixed work on every machine. nothing installed system wide, no background service, no account, no admin rights, no analytics, no tracking. i'm against telemetry, flat out. the only thing that leaves your machine is the benchmark result itself, and NORMIEBENCH_NO_SUBMIT=1 kills even that. the privacy policy lists every field that gets sent.

uninstalling is deleting a folder. runner, engine, model and saved results all live in one per-user cache dir:

• windows: %LOCALAPPDATA%\normiebench
• macos: ~/Library/Caches/normiebench
• linux: ~/.cache/normiebench, or $XDG_CACHE_HOME/normiebench if you've set that

delete it and it's gone. the only id is a random one written inside that folder so repeat runs from the same box group together. it isn't built from your hardware, your network or anything about you, and it dies with the folder.

the point is the database, not the ranking. a big pile of real readings from real consumer machines, charted so you can actually use them: what do i need to run this, how does my rig compare, what can the thing i already own actually do. that beats marketing numbers and one reviewer's test bench. it only works if plenty of ordinary hardware shows up, which is why running it has to be dead easy.

this is a community project. help, ideas, corrections, all welcome. the runner, the engine profile, the scoring and the server side validation are public and open to argument. if something here is wrong or could be measured better, say so. this is our software.

what it does

one command picks a cpu or gpu backend, grabs the verified pieces, warms up, runs the same five fixed measurements three times, saves the full result locally and sends a sanitized copy to the leaderboard.

you can also force it. cpu or gpu for the device class, or name a single backend: cuda, vulkan, metal, rocm, sycl, cpu. handy for comparing two backends on the same card. naming one means no fallback, so it either measures that engine or tells you it can't run. auto is the default and it's what the leaderboard expects.

what it downloads

so the first run pulls roughly 4.4 GiB on cpu, vulkan and metal boxes, ~4.5 GiB on arc or linux amd, and ~4.7-5.0 GiB on windows amd or nvidia. if an accelerated backend flunks its warmup the fallback engine is another download. leave 8 GiB of free disk for download plus extraction plus headroom. you need a 64-bit os, 6 GiB of ram total and 5 GiB free.

on windows the llama.cpp engines need the microsoft visual c++ redistributable, which isn't part of windows, so a fresh install often doesn't have it. normiebench checks before downloading anything and offers to pull the official package straight from microsoft. we don't redistribute it ourselves, and windows update patches it afterward.

runner and model files get sha-256 checked before use. engine archives get checked before extraction and are only marked done after a clean install. everything caches in your normal per-user cache dir, interrupted model downloads resume, half-finished engine installs repair themselves, and nothing goes in system wide. repeat runs just use the cache.

the standardized test

the whole idea is that two results are actually comparable, so nearly everything is nailed down:

• one model file. Qwen2.5 7B Instruct Q4_K_M, pinned by sha-256, 4,683,073,632 bytes exactly. no swapping in a different quant or a "close enough" build.
• one engine. llama.cpp b10173 from the official release, checksummed before it runs. same commit on every machine.
• fixed engine settings. same batch sizes, f16 kv cache, memory mapping on, one thread per physical core.
• fixed work. the same five rows, the same token counts, three measured repetitions, everywhere. no time limit, so a slow machine does exactly the same job as a fast one and just takes longer.

there's no seed setting and no temperature, because neither applies here. llama-bench feeds the model deterministic synthetic tokens and skips tokenization and sampling completely, so there's nothing random to seed and nothing being sampled. every machine chews the identical tokens in the identical order. that's the point: if the number moves, it's the hardware.

what does change per machine is the route, and that's deliberate. backend, flash attention, and how many layers fit on the gpu all get picked during warmup. the engine starts with inherited tuning variables removed, then the runner applies and records the exact plan. the work stays byte for byte the same either way: same job for everyone, each machine gets its best verified route through it.

the five rows, three measured reps each:

• prompt processing: 2,048 prompt tokens, reported as prompt tokens/s
• token generation: 128 generated tokens, reported as generation tokens/s
• chat: 512 prompt + 256 generated
• general: 2,048 prompt + 256 generated
• document: 4,096 prompt + 256 generated

those names are just familiar context lengths, not quality tests. nothing here grades chat or code or retrieval output. prompt and generation stay on separate rows because the two halves of "does this feel fast" are different, and they don't get blended into some "effective tokens/s" number since prompt tokens and output tokens cost wildly different amounts.

the three combined ones are what get scored. score is 1,000 x a fixed 143.36 second reference basket, divided by the sum of your three measured median times. one of each interaction, no extra weighting. finish the basket twice as fast, get exactly twice the score.

what the score does and doesn't tell you

prompt processing is compute bound. generating tokens is memory bandwidth bound. consumer hardware varies enormously in that balance, roughly 3:1 on an n100-class cpu vs roughly 50:1 on a high end gpu. so the same fixed basket ends up mostly a generation test on a fast gpu and mostly a prompt test on a slow cpu. that's the hardware, not a bug in the test, and no fixed workload can dodge it. shifting the prompt-to-output ratio just slides the emphasis around.

so one number doesn't answer everything, and we don't pretend it does. prompt and generation throughput sit right next to every score. paste in long documents? watch the prompt number. ask short questions and read long answers? watch generation. a machine can win one and lose the other, and compare mode exists to show exactly that.

256 tokens of output was picked because it's a paragraph or two. it's not a claim that every answer is that long. if your work is all giant outputs, generation matters more to you than this score says.

each machine runs whatever backend suits it: cuda on windows nvidia, rocm/hip on discrete amd, sycl on discrete arc, metal on apple silicon, bundled vulkan on intel macs, vulkan for everything else. nothing is assumed. every candidate has to finish a real warmup and actually report accelerated execution before it counts, and anything that fails or quietly falls back to cpu gets skipped for the next option. so a machine with a busted driver ends up slower, not broken.

worth knowing: nvidia cards on linux run vulkan instead of cuda, because llama.cpp ships no linux cuda package and we only use official builds. the cost depends a lot on the card. on an rtx 4070 it was about 1.6%, basically nothing. on a gtx 1070, same driver so only the backend changed, cuda won by 21%. the split underneath is sharper: cuda did prompts 2.1x faster while vulkan generated 18% faster. cuda only wins the combined score because the basket has way more prompt tokens than generated ones. so older nvidia gear gives up real performance on linux, and a generation-heavy workload might give up nothing. since the same gpu lands on different backends across operating systems, the backend is printed next to every score.

flash attention works the same way. once the offload level is settled the warmup reruns with it on, and it's kept only if it's clearly faster. same math, same output, just a different route. across five machines it swung scores anywhere from -2.7% to +10.1% purely on how well each backend implements it. forcing it on would punish the machines it slows down, forcing it off understated apple silicon by ~9%. letting each machine keep the faster one punishes nobody.

inside the chosen backend it tries full offload first. if the 4k warmup won't fit it tries 24, 16, 8, 4, then 1 layers and uses the highest that passes. low-vram gpus still get to contribute instead of being thrown out. auto only drops to cpu when nothing works at any layer count.

one selected accelerator is still the safe default. v7 can use a tested whole-machine plan where multiple accelerators are how that complete machine runs the same model best. these are explicit hardware plans, not open-ended auto tuning. the first is the dual firepro d700 mac pro 6,1 on linux with radv/vulkan: both d700s, tensor split, flash attention, full offload. it came from repeatable full-basket tests, not a short probe. set NORMIEBENCH_SINGLE_GPU=1 to bypass a whole-machine plan and measure only the selected gpu.

the work is fixed, not time limited. fast machines finish sooner, slow ones do the exact same token counts. figure one to two minutes of measured work on typical accelerated consumer hardware. cpu-only takes way longer and varies a lot: about half an hour on a power-limited laptop cpu, closer to an hour on an n100. that's just what a 7b model on a slow cpu costs, and we don't shorten it per device because then nothing would be comparable. after the warmup the runner tells you how long the measured part will take, so a long run is never a surprise.

stability comes from the full scored basket time across the three reps, which is what the headline score is made of. both the robust variation and the full spread have to pass, because with only three reps the robust stat alone can't tell a clean run from one where a single rep ran way slow. max row variation is shown too, for diagnostics. fail either gate and it retries once, then stays off the public board if it still fails.

plug laptops in and close the heavy stuff. normiebench checks ac/battery at the start and end when the os reports it. a run caught on battery still saves locally but isn't ranked. unknown power state stays eligible so desktops and sbcs without battery telemetry aren't punished.

model download, engine startup, model load, tokenization and app ui time are all outside the score. prompt speed is a decent proxy for first-response wait but it isn't full app time to first token. this measures repeatable single-stream inference speed. not answer quality, not multi-user serving, not capacity for bigger models, not power efficiency.

what your machine can run

expand a result and it also shows which model sizes fit and roughly how each would feel. only the 7b is measured. everything else is math on top of that, and it's labelled as such.

igpus and apple silicon share memory with the system, and the engine reports whatever slice the driver hands over: measured at 57% of system ram on a radeon 780m and a bc-250, 89% on a ryzen 5500u, 81% on an m1 max. that slice is the real ceiling for the gpu path so that's what the estimates use, but it's labelled as unified memory and shown against total ram, because the cpu path on the same box can address more.

whether a model fits is memory: weights, plus kv cache for 4,096 tokens of context, plus about 0.7 GB of working room. whether it's pleasant is speed, and speed scales predictably. generation is bandwidth bound so it drops roughly with model bytes, prompt is compute bound so it drops roughly with parameter count. that's why benchmarking 14b or 32b separately isn't needed. your 7b result already predicts them, as long as the bigger model fits at all.

time to first token is shown for 512, 2,048 and 4,096 token prompts. it's derived, not measured, but the expensive part isn't guesswork: prefill is exactly what the pp2048 row measures, tokenization takes microseconds, and the rest is one token of generation. three lengths because it scales with prompt size, so a single number would be meaningless without saying how long the prompt was. model load is excluded, it happens once per session in a real app.

generation speed also gets translated into something you can feel. people read around 5.5 tokens/s, so 40 t/s is about seven times faster than you can read, and 2 t/s isn't practical for a conversation no matter what the score says.

estimates get hidden when the run used partial gpu offload, since a bigger model would offload even less and the guess would mislead. a machine that doesn't report accelerator memory shows no estimates at all rather than making something up.

how it sits next to other benchmarks

MLPerf Client is the deep industry standard: multiple models, real prompt categories, quality validation, time to first token and output speed. it also asks a lot more of your disk, dependencies and hardware. UL Procyon AI Text Generation covers several real model/prompt combos with solid reporting, but it's a paid windows product with much narrower hardware support. llamafile and llama.cpp give you great portable engines and low level benchmark tools, just not a one-command leaderboard.

normiebench sits in the light middle on purpose: one open model, one frozen engine profile, one command, broad cpu/gpu coverage, transparent raw timing checks and a leaderboard you can actually read. it trades multi-model and quality coverage for reach and repeatability.

results and past runs

the board shows each machine's best accepted score with its prompt and generation throughput. hit compare on any two for an overlaid view. expand a machine for workload times, stability, hardware, backend, execution plan, temps when available and phase timings. the last ten runs are kept so you can see what a tuning change did. v6 and v7 runs share one board because they use the same model, fixed work, repetitions and score formula. community results are unverified self-reports. "official" means a machine the project controls and tested itself.

the server recomputes every score from the raw timing samples and rejects anything internally inconsistent or claiming throughput this model physically can't hit on that memory and compute. a score way above the field gets held for review instead of published. that raises the cost of faking a result. it can't prove the stated hardware ran the test, and honestly no benchmark at this hardware range can.

run it

normal user shell. no admin, no root, no sudo.

linux and macos:

windows:

set NORMIEBENCH_DEVICE=cpu or gpu to force a device class. on a multi-gpu box set NORMIEBENCH_GPU_INDEX=1 (or another zero-based index) to pick a different card. set NORMIEBENCH_SINGLE_GPU=1 to bypass an approved whole-machine plan. set NORMIEBENCH_NO_SUBMIT=1 for a local-only run, which also turns off fatal-error reports.

source

the bootstrap, rust runner, worker api, site, profile and tests are all public at gitlab.com/brando69/normiebench. benchmarking happens on your machine. only allowlisted hardware metadata, raw measurements, settings, timings and optional temperature readings get sent. details in the privacy policy.

license and credits

normiebench is MIT licensed. it uses llama.cpp, also MIT. Qwen2.5 model files are Apache 2.0. the self-hosted JetBrains Mono typeface is under the SIL Open Font License 1.1.