Tokens Per Second: What Speed to Expect From Local LLMs on Your Hardware

The first question after "will the model fit in my VRAM?" is "will it be fast enough that I actually want to use it?" Those are different questions with different answers. A model can fit comfortably and still crawl, or run on modest hardware and feel instant.

The number you care about is tokens per second (tok/s): how fast the model generates text once it starts replying. You can predict it without trusting anyone's benchmark, because single-stream generation obeys a simple, durable rule. This guide explains that rule, turns it into a "fast enough?" verdict, and points you at the tokens per second calculator to run the numbers for your exact setup.

The one formula that survives every new GPU

Per-card benchmark tables rot. A new driver, a new GPU, a new quantization format, and last year's numbers are wrong. The underlying physics does not change, so anchor to that instead.

When an LLM generates one token at batch size 1, it reads every active weight out of memory exactly once. The arithmetic is cheap; the time goes into moving weights from VRAM into the compute units. That makes single-stream decoding memory-bandwidth bound. The whole behaviour collapses into:

tokens/sec ≈ memory_bandwidth ÷ bytes_read_per_token

Where bytes_read_per_token is just the model's size on disk in your chosen quantization: parameters times bytes-per-parameter. An 8B model at FP16 (2 bytes each) reads about 16 GB per token. The same model at 4-bit reads about 4 GB. Divide your card's bandwidth by that and you have a ceiling that no benchmark can contradict, because it is set by how fast the memory bus can move bytes.

This is why the calculator asks for two things first: a GPU (which fills in its bandwidth) and a quantization (which sets bytes per parameter). Everything else is a refinement.

A worked example you can reproduce

Take an 8B model at Q4_K_M on an RTX 4090. Q4_K_M is not exactly 0.5 bytes per parameter. K-quants keep some components at higher precision, so the effective figure is closer to 0.58 bytes. The 4090's published bandwidth is 1008 GB/s.

bytes per token = 8 × 0.58 = 4.64 GB
theoretical max = 1008 ÷ 4.64 ≈ 217 tok/s
realistic (80%) ≈ 174 tok/s

The 80% is an efficiency factor: real inference kernels sustain roughly 70-85% of a card's advertised peak bandwidth, never 100%. The calculator puts this setup at roughly 174 tok/s with its default 80% slider. Treat that as a model-derived estimate, not a measured benchmark; your real number moves with your runtime, batch settings, and context length.

Want it faster? Either move bytes faster (more bandwidth) or read fewer bytes (smaller model, heavier quantization). There is no third lever for single-stream decode.

What counts as 'fast enough'

Raw tok/s only means something once you map it to what you are doing with the output. Two regimes matter, and they have very different thresholds.

You are reading the output. Average adult silent reading runs around 200-300 words per minute, roughly 3-5 words per second. Tokens are not words; English runs around 1.3 tokens per word, so one token is a bit less than one word. That means somewhere around 7-10 tok/s already produces text faster than most people read it. At that rate a chat reply feels live. The tokens per word chart shows how that ratio shifts across model families if you want to be precise.

You are waiting on the output. A coding agent, a batch summarizer, or anything where you are not reading along but waiting for a finished result has no patience budget tied to reading speed. Here you want tens of tok/s, because a 2,000-token answer at 10 tok/s is over three minutes of staring, while at 80 tok/s it is under thirty seconds. The same model that feels fine for chat can feel painful as an agent.

Rough guide, hedged because it depends on the person and the task:

  • Under about 5 tok/s: usable for short questions, tedious for anything long.
  • About 7-15 tok/s: fine for interactive chat you read in real time.
  • About 30-60 tok/s: comfortable for longer generations and light agent use.
  • 80+ tok/s: snappy enough that latency stops being the thing you notice.

Quantization makes models faster, not just smaller

Most coverage frames quantization as a memory trick: shrink the weights so they fit. That undersells it. Because speed is bytes-per-token divided into bandwidth, halving the bytes roughly doubles the speed on the same card.

Move an 8B model from FP16 (2 bytes) to 4-bit (about 0.5-0.58 bytes) and you cut the bytes read per token by roughly four. On identical hardware that is roughly a 4x jump in tok/s, on top of the VRAM savings. This is why people running local models reach for Q4 and Q5 quants by default: the quality cost is usually small, and the speed gain is large and free.

The trade-off is precision. Heavier quantization (Q2_K and below) degrades output more, so the practical sweet spot for most local use sits around Q4_K_M to Q5_K_M, balancing speed against fidelity. If you generate quantized variants for Ollama, the Ollama Modelfile generator can scaffold the config.

Why the first reply feels slow, then accelerates

You have probably noticed a model pause before it starts typing, then speed up. That is the difference between two phases, and only one of them is bandwidth-bound.

Prefill (prompt processing) reads your entire input. The model can process all those tokens in parallel, which makes prefill compute-bound and very fast per token. But a long prompt still takes real wall-clock time before the first output token appears. This is your time-to-first-token.

Decode (generation) produces output one token at a time, each one bandwidth-bound, and this is the tok/s the formula predicts. So total latency is prefill time + (output tokens ÷ decode rate). A huge system prompt can dominate the felt delay even when decode is quick, which is why trimming context helps responsiveness, not just cost.

There is a second, growing cost during decode: the KV cache. Each generated token attends to all previous tokens, and the cache holding those keys and values consumes both VRAM and a little extra bandwidth as context grows. The bandwidth formula captures the weight reads, which dominate at typical context lengths, but at very long contexts the cache reads start to matter. The KV cache calculator sizes that for your context length.

Two cases that break the simple intuition

Mixture-of-experts models. An MoE has a large total parameter count but only activates a fraction per token. A model with 47B total and roughly 13B active reads memory like a 13B dense model, so it decodes at roughly 13B speed while occupying VRAM like a 47B model. Fast to run, expensive to fit. Enter the active-parameter count in the calculator (or work it out with the MoE active parameters calculator) and you get the realistic speed instead of the misleadingly slow total-size estimate.

Batched serving. The formula describes one request in isolation. A server batching many requests reads each weight once and reuses it across the whole batch, so aggregate throughput per GPU climbs far above the single-stream number, while each individual user still sees roughly the per-request latency the formula predicts. If you are deciding between a local card and a hosted API, that throughput math is central; the self-hosting vs API calculator and the GPU cloud price comparison cover the cost side.

One more lever worth knowing: speculative decoding uses a small draft model to propose tokens that the big model verifies in parallel, which can beat the plain bandwidth ceiling for a single stream. The speculative decoding calculator estimates the speedup.

How to use the calculator

Open the tokens per second calculator and:

  • Pick your GPU to auto-fill its memory bandwidth, or choose Custom and type the figure from the spec sheet. Apple Silicon is in the list too; unified-memory bandwidth, not TFLOPS, is what makes a Mac competitive for inference.
  • Set parameters and quantization. Together these are your bytes-per-token. Toggle between FP16 and a 4-bit quant to watch the speed jump.
  • Add active parameters only for MoE models; leave it at 0 for dense.
  • Adjust efficiency if you have a benchmark you trust. Default 80% is reasonable; raise it to match a measured result, lower it for an unoptimized runtime.

Read the result as a planning estimate. It tells you whether a card will feel snappy or sluggish before you spend money, and it stays correct as new hardware ships, because it is grounded in bandwidth rather than someone's leaderboard. To confirm a model also fits, pair it with the LLM VRAM calculator.

Use the tool

Skip the manual work. The companion tool runs this in your browser, with nothing uploaded.

Tokens per second calculator

Frequently asked questions

What tok/s is 'usable' for chat versus a coding agent?

For chat you read in real time, roughly 7-15 tok/s is comfortable because that already outpaces typical reading speed. For a coding agent or any task where you wait for a finished result rather than reading along, you want tens of tok/s; a long answer at single-digit tok/s feels slow even though the same speed is fine for chat.

Does a faster GPU speed up a small model?

Only if it has higher memory bandwidth. Single-stream decode speed is bandwidth divided by bytes-per-token, so two cards with the same bandwidth run the same model at roughly the same tok/s regardless of their compute. Extra TFLOPS mostly help prompt processing and batched serving, not single-stream generation.

Why is my measured speed different from the estimate?

The formula gives single-stream decode under ideal-ish conditions. Speculative decoding, flash-attention, and a higher real efficiency push you above it; an unoptimized runtime, a very long context, or a cold cache push you below. Set the efficiency slider to whatever reproduces a benchmark you trust for your exact setup.

Does quantization actually make a model faster, or just smaller?

Both. Because speed equals bandwidth divided by bytes read per token, cutting the bytes roughly proportionally raises tok/s. Moving from FP16 to 4-bit reads about a quarter of the bytes, so on the same card you get roughly four times the speed in addition to the VRAM savings.

Is decode speed the whole latency story?

No. Total latency is prefill time plus output tokens divided by decode rate. Prefill (reading your prompt) is compute-bound and fast per token but still adds real delay for long prompts, which is why a big system prompt can make the first token feel slow even when generation itself is quick.

Why can a Mac keep up with a discrete GPU for inference?

Because single-stream inference is bound by memory bandwidth, not raw compute. Apple Silicon's unified memory offers several hundred GB/s, which is enough to generate at competitive tok/s for many models despite modest TFLOPS. Plug your Mac's bandwidth into the calculator to see where it lands.

Related tools