Local LLM
60 posts tagged #local-llm
60 posts on Local LLM, 58 of them in the LLM Hardware & Local AI pillar, with 2 in AI Agents — most read: “Local LLM vs Claude for Coding: When a Local GPU Is Enough” and “Portable LLM on a USB Stick: Run Ollama or LM Studio Offline”.
Cybersecurity Local LLM Privacy Risk: What Your Setup Leaks
Two viral videos claim your local AI isn't private. I ran packet captures on Ollama to verify the three real leak vectors: open APIs, 0.0.0.0 binding, and client telemetry.
AI and Machine Learning Qwen 4 27B? Why Alibaba Skipped It [2026 Benchmark]
You searched for a Qwen 4 27B local LLM benchmark — but there's no Qwen 4. Here's the real Alibaba release, how the Apache-licensed 27B stacks up against Llama 4 and Mistral, and whether it's worth switching.
AI and Machine Learning Phone as GPU for Local LLM: Tested [2026]
Can a Galaxy S23 act as an extra GPU node for local LLM inference? I break down the real llama.cpp RPC mechanism, the setup, and whether it beats just buying RAM.
Technology Anthropic Warns AI Models Could Soon Self-Replicate
Anthropic warns AI models could soon copy themselves onto small devices without a data center. Here's what 'self-replication' actually tests for, and how worried you should be.
Developer Tools How to Build a Local Model Registry for GGUF
Stop downloading random weights from links. Build a local model registry with GGUF versioning, provenance attestations, eval gates, and a real rollback plan.
AI and Machine Learning How to Run Qwen 3.8 Flash Next 125B in Strata on an RTX 4090
A reproducible Strata setup for Qwen 3.8 Flash Next (125B) on RTX 4090, plus the KV cache knobs that decide whether you hit ~100 tok/s or hit OOM.
AI and Machine Learning Kolibri Open Weight Model Benchmark: 1M Context Reality Check [2026]
Aleph Alpha’s Kolibri ships downloadable weights under Apache-2.0 and claims a 1M-token context. Here’s what that changes for EU/on-prem teams and how to benchmark it reproducibly on an A10/A100.
Developer Tools ds4 dwarfstar [2026 Review]: Redis-Style Local LLM Loops
ds4 (DwarfStar 4) turns local LLM inference into a Redis-like workflow: one long-lived server, deterministic prompt-keyed KV caching, and fast edit-run-edit loops without a GUI circus.
Developer Tools LM Studio Power User Setup: Profiles, Quants, LAN Serving
My opinionated LM Studio setup: reproducible model profiles, quant hygiene rules, scripted loads via lms + /api/v1, and a secure multi-user LAN server.
Technology Intel Arc B‑Series Local LLM Benchmark [2026]: Arc Done Right
A reproducible Intel Arc B‑series local LLM benchmark harness (TTFT, tok/s, VRAM, power), plus a Linux/Windows compatibility matrix and Arc-friendly GGUF quants that won’t OOM at 8K context.
![ollama terminal macbook laptop screen — illustration for article on Ollaya Ollama Decision Model Setup [2026]:](https://img.kunalganglani.com/images/vzekdneq/production/a6ede8c9d4b1546ce5b4e28f601f2cf78d2b4815-1200x675.webp?auto=format&fit=max&q=75&w=600)
Developer Tools Ollaya Setup: Local Decision-Model Routing and Benchmarks
Set up Ollaya locally, write safe routing questions, point the TypeSafe SDK at localhost, and benchmark p50/p95 (warm + cold) vs LLM JSON routing.
Cybersecurity How to Secure Local LLM Inference: Sandbox + Egress
A practical blueprint for secure local LLM inference: sandbox inference hard, default-deny outbound network, stage allowlisted downloads, scan artifacts, and isolate tools.
Developer Tools KoboldCpp GGUF Setup Guide [2026]: When It Beats Ollama
Run your existing .gguf models in KoboldCpp in under 15 minutes. I’ll show the exact settings for context, GPU layers, streaming, and when Ollama still wins.
AI and Machine Learning How to Run Qwen 35B on 16GB VRAM [2026]: Flags + Quants
A reproducible 16GB recipe for “Qwen 35B”: which Qwen2.5-32B quants fit, how to budget KV cache, and the exact serving flags that stop OOMs.
Cybersecurity Verify GGUF Model Hashes: 10-Step Supply Chain Workflow
A team-ready workflow to verify GGUF integrity: compute SHA256, require signed manifests, handle mirrors safely, scan sidecars, and ship updates via an internal registry.
![nvidia gpu graphics card close-up — illustration for article on Local LLM Break-Even Math [2026]: Power,](https://img.kunalganglani.com/images/vzekdneq/production/76beafb78b6f19b7ed6235232857078a023801c2-1200x675.webp?auto=format&fit=max&q=75&w=600)
AI and Machine Learning Local LLM Break-Even Math: Power, Idle, Depreciation
Most “GPU price ÷ tokens” break-even math is wrong. Here’s a spreadsheetable local LLM total-cost model that includes idle power, utilization, depreciation, failures, and opportunity cost.
AI and Machine Learning 9 Picks for the Best Local Model for Agentic Coding [2026]
A practical decision tree for agentic coding: pick local models by repo size and VRAM tier (16GB/24GB/48GB+/CPU), with context, tool-calling reliability, and quantization rules that actually hold up.
Technology Local LLM Benchmark Methodology [2026]: TTFT vs tok/s Done Right
Stop screenshot-benchmarking. Here’s a reproducible local LLM benchmark methodology for 2026 that separates TTFT from throughput and reports rerunnable results.
Developer Tools Run Local LLMs in VS Code: No Copilot Plan [2026]
VS Code now wires Ollama and LM Studio straight into Copilot Chat's model picker — no CLI, no Continue.dev, no Copilot subscription. Here's the full 2026 setup, both paths compared, plus the troubleshooting nobody documents.
AI and Machine Learning Gemma 4 26B CPU Inference Benchmark: 5 tok/s Production Math [2026]
A $300 Xeon from 2013 runs Gemma 4 26B at 5 tok/s with no GPU. Here's the memory bandwidth math, quantization tradeoffs, and production decision framework nobody else is covering.
AI and Machine Learning RTX 4060 Ti vs RTX 4070 for Local LLM Inference in 2026
I'd pick the RTX 4060 Ti if you're running sub-13B models solo on a tight budget, and the RTX 4070 if VRAM headroom and generation speed actually matter to your workflow. The $150 price gap is real, but so is the performance cliff you hit at 16GB models.
AI and Machine Learning AI Hardware Guide: GPUs, Apple Silicon and Edge Boards
A guide to AI hardware — covering GPUs, Apple Silicon, edge devices, CPUs, and the semiconductor supply chain — so you can choose the right setup for running local LLMs and AI workloads without wasting money.
Technology LM Studio vs Ollama 2026: 3 Shifts That Change Everything
LM Studio and Ollama have converged so much in 2026 that every old comparison is wrong — here's what actually matters now for choosing your local LLM tool.
AI and Machine Learning Local LLM Cost vs Cloud API: 2026 Break-Even Math [Calculator]
A workload-specific break-even framework with real per-token math — hardware amortization vs. API spend — for coding, RAG, and batch workloads in 2026.
AI and Machine Learning LLM Quantization Levels Compared: Q4_K_M vs Q8_0 vs FP16 [2026]
The practitioner's guide to choosing between Q4_K_M, Q5_K_S, Q8_0, and FP16 quantization for local LLMs — with real perplexity numbers, throughput benchmarks, and per-use-case recommendations.
![GGUF vs GPTQ vs EXL2: LLM Quantization Compared [2026]](https://img.kunalganglani.com/images/vzekdneq/production/34e429bea58e21d5119b5baad6b9efe44974be77-1200x675.webp?auto=format&fit=max&q=75&w=600)
AI and Machine Learning GGUF vs GPTQ vs EXL2: LLM Quantization Compared [2026]
A head-to-head comparison of GGUF, GPTQ, and EXL2 quantization formats with real quality, speed, and VRAM trade-offs — updated for the 2026 Hugging Face acquisition of ggml.ai.
![Fine-Tune Open-Source LLMs: LoRA, QLoRA, Gemma 4 [2026]](https://img.kunalganglani.com/images/vzekdneq/production/bbdfb150ddb4280a3411f8cbcf4d7f0f54cfdf63-1200x675.webp?auto=format&fit=max&q=75&w=600)
AI and Machine Learning Fine-Tune Open-Source LLMs: LoRA, QLoRA, Gemma 4 [2026]
A practical 2026 guide to fine-tuning open-source LLMs with LoRA and QLoRA using Unsloth + Gemma 4 — including GPU requirements, hyperparameter defaults, evaluation setup, and when to just prompt instead.
![Linux vs Windows vs macOS for Local AI [2026 Compared]](https://img.kunalganglani.com/images/vzekdneq/production/649947e748ba7040e11ca2a4012c16e535f4ca37-1200x675.webp?auto=format&fit=max&q=75&w=600)
AI and Machine Learning Linux vs Windows vs macOS for Local AI [2026 Compared]
Your OS choice affects local LLM inference speed more than your model pick. Here's the benchmark-driven breakdown of Linux, Windows, and macOS for local AI in 2026.
![GLM-5.2 vs Claude Fable 5: Open-Source AI Challenges the Throne [2026]](https://img.kunalganglani.com/images/vzekdneq/production/def90c1fb8ed7133b55d3c4e9e53af34f4fb6312-1200x675.webp?auto=format&fit=max&q=75&w=600)
AI and Machine Learning GLM-5.2 vs Claude Fable 5: How Close Is Open Source?
Zhipu AI's 753B open-weight GLM-5.2 is the highest-ranking open-source model on lmarena.ai, challenging Claude Fable 5 across WebDev and Agent benchmarks — and it's already runnable locally via Ollama.
Developer Tools Homelab AI Coding Server: Run OpenCode Across All Devices [2026]
How to set up OpenCode as a persistent, PR-gated AI coding server on your homelab — accessible from every device, isolated from production, and free from vendor lock-in.
Showing the 30 newest of 60. See the full LLM Hardware & Local AI pillar for the rest.
Keep exploring
Close to this topic, not on this page.
