Local LLM Performance Predictor

Model

Preset (or Custom)

KV cache dtype

Quantization & Workload

Model quantization

Context length (tokens)

Prompt tokens

Output tokens

Streams (batch)

Speculative decoding (small draft model)

α = avg accepted draft tokens per step (0.5–3 typical; measure with llama.cpp -ngl speculative runs).

Hardware & Calibration

Unified RAM (GB)

Usable fraction

GPU mem bandwidth (GB/s)

SSD read (GB/s, for swap)

GPU GEMM (TFLOPS fp16)

Compute efficiency

Efficiency = the share of the measured fp16 GEMM rate that prompt processing actually realizes (typically 0.3–0.8; inference is far from theoretical peak, and the TFLOPS above are themselves already measured). Tune it until one model you benchmarked with llama-bench matches its pp column — the rest then follows.

Sustained power (W)

Decode BW efficiency

How the prediction works

Decode (tok/s): each generated token reads the active weights once from unified memory, so tok/s ≈ effective bandwidth ÷ active-weight bytes (MoE uses active params). Batching splits weights across streams until compute caps aggregate throughput.

Prompt / TTFT: compute-bound — TFLOPS × efficiency ÷ (2 × active params).

Memory: weights + KV cache (2 × layers × KV-heads × head-dim × dtype × tokens) + ~1.2 GB runtime overhead. If it doesn’t fit in usable RAM, the fraction that spills is read from SSD at the swap rate — that is what kills tokens/s.

Uncertainty: ±12% (±25% when swapping) covers runtime differences (llama.cpp vs MLX), quant overhead and thermals.

Prediction

…

– tok/s

Prompt processing

–

tok/s

Time to first token

–

Generate 512 tok

–

wall time

Context that fits

–

KV + weights in RAM

Memory used

–

Free headroom

–

Energy

–

Wh per 1 000 output tokens

Throughput

–

output tokens / hour

Memory budget

Decode speed vs quantization

Log scale. Swap-bound quantizations collapse toward the SSD rate.

Verify on this machine

Run this and compare the pp (prompt) / tg (generation) columns against the prediction. If your runtime consistently differs, adjust Compute efficiency (prompt) or Decode BW efficiency (generation) so one known model matches — then trust the rest.