LLM Serving
6 posts tagged #llm-serving
6 posts on LLM Serving, 5 of them in the Local LLM Hardware pillar, with 1 in Production AI Engineering — most read: “Aleph Alpha Kolibri Benchmark: 1M Context Reality Check” and “ds4 DwarfStar Review: Redis-Style Local LLM Loops”.
AI and Machine Learning Aleph Alpha Kolibri Benchmark: 1M Context Reality Check
Aleph Alpha’s Kolibri ships downloadable weights under Apache-2.0 and claims a 1M-token context. Here’s what that changes for EU/on-prem teams and how to benchmark it reproducibly on an A10/A100.
Developer Tools ds4 DwarfStar Review: Redis-Style Local LLM Loops
ds4 (DwarfStar 4) turns local LLM inference into a Redis-like workflow: one long-lived server, deterministic prompt-keyed KV caching, and fast edit-run-edit loops without a GUI circus.
Developer Tools LM Studio Power User Setup: Profiles, Quants, LAN Serving
My opinionated LM Studio setup: reproducible model profiles, quant hygiene rules, scripted loads via lms + /api/v1, and a secure multi-user LAN server.
Cloud and DevOps vLLM Self-Hosted LLM Production Checklist: Auth + Quotas
A production-first checklist for self-hosting an OpenAI-compatible vLLM endpoint: auth, per-tenant quotas, streaming SSE, queueing, and redaction-safe logs.
Cloud and DevOps How to Serve a Local LLM to Multiple Users
If your local LLM server “works” but falls apart at 30–50 concurrent chats, this guide shows the real limiter (KV cache) and the exact knobs in vLLM, SGLang, and TGI that move P99.
Developer Tools vLLM vs Ollama: Production Power or Developer Ease?
vLLM wins for high-throughput production deployments where every token/second counts; Ollama wins for local developer workflows where setup speed and portability matter most. Pick wrong and you'll either over-engineer a side project or under-power a real API.
Keep exploring
Close to this topic, not on this page.
