LLM Inference
12 posts tagged #llm-inference
12 posts on LLM Inference, 9 of them in the Local LLM Hardware pillar, with 1 in Production AI Engineering — most read: “Apple Silicon vs NVIDIA GPU for Local LLMs” and “LLM Quantization Levels Compared: Q4_K_M vs Q8_0 vs FP16”.
AI and Machine Learning Mercury 2.5 770 tok/s Benchmark: The Production Playbook
Mercury 2.5 is clocking ~780 tok/s on Artificial Analysis. At that speed, decode stops being your bottleneck. Here’s what actually changes in production: batching, streaming UX, context packing, cost controls, and evals that don’t lie.
AI and Machine Learning RTX 4060 Ti vs RTX 4070 for Local LLM Inference
I'd pick the RTX 4060 Ti if you're running sub-13B models solo on a tight budget, and the RTX 4070 if VRAM headroom and generation speed actually matter to your workflow. The $150 price gap is real, but so is the performance cliff you hit at 16GB models.
AI and Machine Learning Groq vs Together AI: Inference Speed, Price and Models
I'd pick Groq when raw token throughput is the make-or-break metric — it's still the fastest hosted inference I've tested at under $1/M tokens for Llama 3. I'd pick Together AI when model variety, fine-tuning, or multimodal pipelines matter more than milliseconds.
AI and Machine Learning LLM Quantization Levels Compared: Q4_K_M vs Q8_0 vs FP16
The practitioner's guide to choosing between Q4_K_M, Q5_K_S, Q8_0, and FP16 quantization for local LLMs — with real perplexity numbers, throughput benchmarks, and per-use-case recommendations.
![GGUF vs GPTQ vs EXL2: LLM Quantization Compared [2026]](https://img.kunalganglani.com/images/vzekdneq/production/34e429bea58e21d5119b5baad6b9efe44974be77-1200x675.webp?w=600&q=75&auto=format)
AI and Machine Learning GGUF vs GPTQ vs EXL2: LLM Quantization Formats Compared
A head-to-head comparison of GGUF, GPTQ, and EXL2 quantization formats with real quality, speed, and VRAM trade-offs — updated for the 2026 Hugging Face acquisition of ggml.ai.
AI and Machine Learning M4 Max vs M5 Max for Local AI: Which Should You Buy?
The M5 Max wins for serious local AI workloads, with 614GB/s of memory bandwidth against the M4 Max's 546GB/s and a Neural Accelerator in every GPU core. The M4 Max remains the smart buy for budget-conscious developers who don't need the extra speed.
AI and Machine Learning Mac Studio M4 Max vs RTX 4090 PC for Local LLMs
The Mac Studio M4 Max wins for plug-and-play local LLM work with massive unified memory; the RTX 4090 PC wins for raw CUDA throughput and flexibility. Your budget, workflow, and model size determine which is worth every dollar.
AI and Machine Learning Raspberry Pi 5 vs Jetson Orin Nano for Edge AI
The Jetson Orin Nano wins for serious edge AI workloads with its dedicated GPU and CUDA ecosystem, while the Raspberry Pi 5 wins for cost-sensitive prototyping, general computing, and hobbyist projects. Neither is universally better — it depends entirely on whether you need inferencing horsepower or affordability.
Developer Tools vLLM vs Ollama: Production Power or Developer Ease?
vLLM wins for high-throughput production deployments where every token/second counts; Ollama wins for local developer workflows where setup speed and portability matter most. Pick wrong and you'll either over-engineer a side project or under-power a real API.
AI and Machine Learning RTX 4090 vs RX 7900 XTX for Local LLMs: CUDA vs ROCm
The RTX 4090 wins for serious local LLM inference thanks to superior CUDA ecosystem support and faster throughput; the RX 7900 XTX wins on price-per-GB for budget-conscious builders willing to navigate ROCm. Your choice hinges almost entirely on ecosystem tolerance and how much you value plug-and-play setup.
AI and Machine Learning Apple Silicon vs NVIDIA GPU for Local LLMs
NVIDIA wins on raw throughput and ecosystem depth for serious multi-GPU workloads; Apple Silicon wins on memory bandwidth per dollar and zero-friction local inference for solo developers. Your budget and batch size decide the rest.
AI and Machine Learning Apple M4 vs M4 Max for Local LLMs: Which Should You Buy?
The M4 Max wins for serious local LLM work thanks to its unified memory ceiling and bandwidth advantage; the base M4 wins for portability and budget-conscious inference on smaller models. Here's exactly where the line falls.
Keep exploring
Close to this topic, not on this page.
