llama.cpp
13 posts tagged #llama-cpp
13 posts on llama.cpp, 12 of them in the Local LLM Hardware pillar, with 1 in AI Agent Architecture & Frameworks — most read: “ROCm vs CUDA for Local AI: AMD's CUDA Equivalent Compared” and “How to Run Local Agentic AI on Your Mac With MLX”.
AI and Machine Learning Phone as GPU for Local LLM: Tested vs Buying RAM
Can a Galaxy S23 act as an extra GPU node for local LLM inference? I break down the real llama.cpp RPC mechanism, the setup, and whether it beats just buying RAM.
Technology Intel Arc B‑Series Local LLM Benchmark: Harness and Quants
A reproducible Intel Arc B‑series local LLM benchmark harness (TTFT, tok/s, VRAM, power), plus a Linux/Windows compatibility matrix and Arc-friendly GGUF quants that won’t OOM at 8K context.
Developer Tools KoboldCpp GGUF Setup Guide: When It Beats Ollama
Run your existing .gguf models in KoboldCpp in under 15 minutes. I’ll show the exact settings for context, GPU layers, streaming, and when Ollama still wins.
AI and Machine Learning How to Run Qwen 35B on 16GB VRAM: Flags + Quants
A reproducible 16GB recipe for “Qwen 35B”: which Qwen2.5-32B quants fit, how to budget KV cache, and the exact serving flags that stop OOMs.
AI and Machine Learning RTX 4060 Ti vs RTX 4070 for Local LLM Inference
I'd pick the RTX 4060 Ti if you're running sub-13B models solo on a tight budget, and the RTX 4070 if VRAM headroom and generation speed actually matter to your workflow. The $150 price gap is real, but so is the performance cliff you hit at 16GB models.
AI and Machine Learning LLM Quantization Levels Compared: Q4_K_M vs Q8_0 vs FP16
The practitioner's guide to choosing between Q4_K_M, Q5_K_S, Q8_0, and FP16 quantization for local LLMs — with real perplexity numbers, throughput benchmarks, and per-use-case recommendations.
![GGUF vs GPTQ vs EXL2: LLM Quantization Compared [2026]](https://img.kunalganglani.com/images/vzekdneq/production/34e429bea58e21d5119b5baad6b9efe44974be77-1200x675.webp?w=600&q=75&auto=format)
AI and Machine Learning GGUF vs GPTQ vs EXL2: LLM Quantization Formats Compared
A head-to-head comparison of GGUF, GPTQ, and EXL2 quantization formats with real quality, speed, and VRAM trade-offs — updated for the 2026 Hugging Face acquisition of ggml.ai.
AI and Machine Learning How to Run Local Agentic AI on Your Mac With MLX
Apple's WWDC 2026 MLX session was 13 minutes and skipped the hard parts. Here's the full setup: model selection, MTP speculative decoding, multimodal support, and wiring it all to a coding agent.
Developer Tools Text Generation WebUI vs Ollama: Which Local LLM Tool Fits?
Ollama wins for developers who want a fast, CLI-first runtime with clean API integration; Text Generation WebUI wins for researchers and power users who need deep model control and a browser-based interface. Your choice hinges on whether you optimize for simplicity or configurability.
Developer Tools Ollama vs llama.cpp for Local LLMs: Speed, Setup, Control
Ollama wins for developers who want a fast, polished setup with REST APIs and model management. llama.cpp wins for power users squeezing every last token of performance from their hardware.
Developer Tools Ollama vs Llamafile: Server Daemon or Single-File LLM?
Ollama wins for developers who want a persistent API server with a rich model library and ecosystem integrations. Llamafile wins for anyone who needs zero-install portability — one executable, any machine, no setup.
Developer Tools LM Studio vs Jan: Which Local LLM App Should You Use?
LM Studio wins for polished UX and OpenAI-compatible APIs; Jan wins for open-source transparency and offline-first privacy. Here's exactly when to pick each.
Technology ROCm vs CUDA for Local AI: AMD's CUDA Equivalent Compared
After months of running LLMs on AMD hardware, here's the brutally honest comparison of ROCm vs CUDA that no marketing page will give you — covering real performance, setup friction, and which GPUs actually work.
Keep exploring
Close to this topic, not on this page.
