The Real World Performance of Large Language Models

The primary metric for LLM performance is tokens per second (tps). A "token" is a unit of text, roughly equivalent to ¾ of a word. A higher tps means a faster, more fluid stream of text generation.

However, not all tps are created equal, and focusing solely on this metric can be misleading.

Latency vs. Throughput:   

• Latency is the time it takes from submitting a prompt to receiving the first token of output (often called "time to first token" or TTFT). Low latency is crucial for chat and real-time applications, where users expect an immediate response 
• Throughput is the number of tokens processed per second, especially when handling multiple requests or generating long outputs. High throughput is essential for batch processing, summarization of large documents, or serving many users in parallel

 These two factors are often in tension: maximising throughput (by processing many requests in parallel) can increase latency for individual users, and vice versa. The right balance depends on your use case. 

LLM providers often report tps numbers, but these can refer to different things:

• API Performance: This is the raw speed at which a cloud service (like Azure AI, OpenAI API or Groq) can process requests. This can be extremely high—Groq, for example, reports over 1,200 tps for Llama 3 8B, and NVIDIA H100 servers can reach up to 19,000 tps in optimal conditions.  
• User Interface (UI) Performance: The speed you experience in a web interface like the consumer version of ChatGPT. This is often deliberately slower and isn't a technical limitation; it’s a design choice to make the interaction feel more natural and readable, preventing a giant wall of text from appearing instantly.

 For most business applications that we build, we care about the raw API performance, as this determines the true speed of the underlying workflow. 

The promise of running LLMs locally for privacy is appealing, but performance hinges entirely on the hardware. Here’s a practical look at what different setups can realistically achieve when running a popular open-source model like Meta's Llama 3 (8B Parameter version).

All figures below refer to Meta’s Llama 3 8B model, unless otherwise noted. 

*Assumes 100KB ≈ 22,755 tokens and 1MB ≈ 233,016 tokens.

Key Takeaways from Benchmarks

• Apple M-series chips (e.g., M4) offer much better LLM performance than typical small form-factor PCs, thanks to their integrated Neural Engine. They should not be grouped with generic "NUCs" or low-end desktops. 
• Mobile devices can process long content; the only limitation is how long the user is willing to wait, not any technical cutoff on content length. 
• Enterprise servers achieve high throughput by rapidly switching between user requests, not by processing all users' tokens simultaneously. 
• Processing large files: For example, a standard office PC would take 2–6 hours to process a 100KB file, while an enterprise server could process it in seconds to a couple of minutes. 

To improve performance on local hardware, models are often "quantized."

In simple terms, quantization reduces the precision of the mathematical calculations within the model, making it smaller and faster. It’s like saving a high-resolution photo as a lower-quality JPEG to reduce the file size.

• The Impact: Quantization can dramatically increase tps, often doubling performance or more. 
• The Cost: This speed comes at a real cost to accuracy. For many general tasks, the drop in quality is modest, but not negligible—typically in the 5–10% range for benchmark accuracy, and potentially more for complex reasoning or nuanced tasks. If the loss were negligible, quantized models would always be used in production, but for high-stakes applications, full-precision models are still preferred.

 Choosing the right level of quantization is a technical decision that requires balancing your specific needs for speed versus accuracy. 

The performance gap between local and cloud models will shrink but is unlikely to disappear. We expect to see rapid improvements in model efficiency and even more specialised hardware (like Apple's Neural Engine) designed to run these tasks.

However, for the foreseeable future, a fundamental trade-off remains: the ultimate performance and power of flagship commercial models will reside in the cloud. The key is to have an expert partner who can help you benchmark your needs and design a solution with the right performance profile for your users and your budget.