LLMOps
36 posts tagged #llmops
36 posts on LLMOps, spread across AI in Production (17), AI Agents (10), Dev Tools & AI Workflow (3) — most read: “How to Build a Local Model Registry for GGUF [2026]” and “How to Tell If You Need an AI Agent or a Workflow (7 Smells) [2026]”.
Developer Tools How to Build a Local Model Registry for GGUF [2026]
Stop downloading random weights from links. Build a local model registry with GGUF versioning, provenance attestations, eval gates, and a real rollback plan.
AI and Machine Learning How to Tell If You Need an AI Agent or a Workflow (7 Smells) [2026]
Most “AI agents” in production are deterministic workflows wearing an LLM costume. Here’s the teardown rubric I use to simplify designs, cut cost, and make on-call sane.
Cybersecurity How to Build a Prompt Injection Scanner for RAG [2026]
A practical, reproducible way to scan retrieved RAG context for indirect prompt injection, benchmark false positives, and gate regressions in GitHub Actions without blocking your team all day.
Developer Tools How to Follow arXiv Updated Rate Limit Policy [2026]
arXiv tightened rate limits on Oct 1, 2026. Here’s a compliant ingestion blueprint for RAG: incremental harvest, caching, backoff, bulk S3, WARC snapshots, and version tracking.
Developer Tools MCP vs Function Calling in Agents [2026]: When to Say No
MCP backlash is real. Here’s my decision framework for when to use MCP vs function calling in agents, and when a bespoke tool API beats both on ops, security, and debugging.
AI and Machine Learning How to Do Synthetic Data for RAG Evaluation [2026 Pipeline]
A leakage-first, adversarial-negative pipeline to generate, label, and validate a synthetic RAG evaluation set without building a benchmark that only tests your generator.
AI and Machine Learning How to Use MLflow LLM Evaluation Tracing [2026] (Spans → Gates)
Turn MLflow traces into living eval datasets, rubric scorecards, and CI/CD regression gates for tool-calling agents. Stop debugging with random logs.
AI and Machine Learning Jev Models Explained: Faster Routing, Reranking, JSON
Jev-style decision models are non-autoregressive models for routing, reranking, and structured outputs. Here’s how they work, where they fail, and how to evaluate them like a builder.
AI and Machine Learning 7 AI Agent Swarm Coordination Patterns [2026]: Failure Modes Included
AI agent swarm coordination patterns decide whether your multi-agent system converges or spirals. Here are the topologies, stop conditions, budget caps, and failure modes most teams don’t test in 2026.
AI and Machine Learning How to Test Non-Deterministic AI Systems and LLMs
A release-gating playbook for LLMs and agents: golden sets, metamorphic tests, variance control, tolerance bands, cohort diffs, and eval-regression postmortems.
AI and Machine Learning RAG Evaluation Metrics for Retrieval Quality: My Production Playbook
If your RAG app got worse after an embeddings or chunking change, grading answers won’t tell you why. Here’s a retrieval-first eval workflow: recall@k, MRR, citation accuracy, leakage tests, and a frozen offline corpus so regressions are real—not vibes.
AI and Machine Learning How to Ship OpenAI Agents API Guardrails in 1 Day [2026]
Production tool-use fails on governance, not prompts. Here’s a practical setup for OpenAI Agents API guardrails: allowlists, layered rate limits, and audit logs you can actually use in incidents.
AI and Machine Learning Execution Trace Tree for AI Agents: Build One in 60 Minutes
Stop drowning in agent logs. Instrument a deterministic execution trace tree (spans, tool calls, checkpoints) so you can replay failures and diff runs like real engineering.
Cloud and DevOps How to Build Vendor-Neutral LLM Observability Monitoring [2026]
A practical blueprint for llm observability monitoring vendor neutral: an OpenTelemetry pipeline, an open trace schema, redaction boundaries, tail sampling, and cost controls that survive vendor swaps.
AI and Machine Learning Agent Evaluation Roadmap for Small Teams [2026]: The 30-Min/Week Plan
A pragmatic map of offline vs online vs HITL evals, what to measure beyond “response quality,” and the smallest program that actually prevents agent regressions.
AI and Machine Learning How to Start an AI Agent Evaluation Program (5-Task Scorecard)
Ship agents with a regression safety net: pick 5 real tasks, define pass/fail, run evals weekly in CI, and publish a scorecard that forces better decisions.
AI and Machine Learning Agent Tool Call Failure Testing: A Deterministic CI Harness
Build a deterministic CI harness that breaks your agent’s tools on purpose: timeouts, 429s, partial writes, retries, and stuck loops. Then regress it with golden traces.
Cybersecurity AI Security Leader Playbook [2026]: 10 Controls That Ship
A practical AI security leader playbook you can implement this quarter: inventory, approval gates, agent threat modeling, OWASP LLM Top 10 controls, vendor review, and incident response.
Cloud and DevOps ChatGPT Down? 8 Fallback Patterns for API Outages
When ChatGPT goes down, your app shouldn’t. A production playbook for detection, retries with jitter, circuit breakers, bulkheads, throttling, and multi-provider failover.
AI and Machine Learning RAG Data Leakage Test Suite [2026]: CI Red-Team Setup
Build an automated red-team suite for RAG apps: canary tokens, regex + similarity detectors, multi-step prompt-injection attacks, and a CI risk score that blocks risky merges.
Cloud and DevOps How to Pick LLM Application Observability Metrics [2026]
Token logs are table stakes. Here’s the minimum set of LLM application observability metrics for tools, RAG, caching, refusals, and privacy-safe logging that actually debugs production incidents.
AI and Machine Learning Data Privacy in RAG Redaction and Retention [2026 Playbook]
A practical engineering playbook for data privacy in RAG: what to redact, where privacy leaks happen, default retention periods, and audit trails you can defend in an audit without wrecking debugging.
AI and Machine Learning AI Engineering Evals: Regression Gates for Prompts, Tools, RAG [2026]
Stop letting prompt tweaks and model upgrades silently break production. Here’s a CI-style regression gate system for prompts, tool calling, and RAG with golden sets, schemas, shadow evals, and failure budgets.
AI and Machine Learning AI Agent Observability Logging Schema [2026]: OTel + Redaction
A copy‑paste JSON logging contract for agent runs (spans, tool I/O, cost, retries) plus a minimal OpenTelemetry mapping you can implement in a weekend—without leaking secrets.
AI and Machine Learning LLM Latency Benchmark Methodology: Streaming UX Metrics [2026]
A UX-first LLM latency benchmark methodology for streaming chat and agent apps: measure chunk cadence, jitter, tool-call stall time, and end-to-end time-to-usable—not just TTFT.
AI and Machine Learning AI Agent Evaluation Framework 2026: 8 Metrics Beyond Task Success
If your agent eval is just “did it finish the task?”, you’re flying blind. Here’s a 2026-ready scorecard for tool correctness, recovery, safety, and cost-per-success—plus a regression suite blueprint you can actually run in CI.
Cybersecurity Prevent Sensitive Data Leakage in RAG: A 7-Step Playbook
RAG leaks rarely happen in the model. They happen in logs, traces, and vector stores. Here’s a practical playbook to ship redaction, least-context retrieval, and auditable controls end-to-end.
AI and Machine Learning Agent Evaluation Harness [2026]: Replay, Rubrics, CI Gates
Most agent failures aren’t “bad prompts”. They’re multi-step tool cascades. Here’s how I build an agent evaluation harness that actually prevents regressions.
Developer Tools AI Coding Assistant Reviews 2026: The Only Buyer Framework That Holds Up
In 2026, the best AI coding assistant isn’t the smartest model. It’s the one that hits latency SLOs, finds the right code, and ships with governance that won’t get you fired.
![developer monitor terminal logs tracing — illustration for article on OpenTelemetry Instrumentation for AI Agents [2026]:](https://img.kunalganglani.com/images/vzekdneq/production/f126a9eb93656ea60adeae440cf4b74ed0b5fb93-1200x675.webp?auto=format&fit=max&q=75&w=600)
AI and Machine Learning OpenTelemetry Instrumentation for AI Agents [2026]: Ship It
A vendor-neutral tracing schema for AI agents: model LLM calls, retrieval, tool runs, retries, and token cost as spans. Then dashboard latency, error tax, and cost per successful task.
Showing the 30 newest of 36. See the full AI in Production pillar for the rest.
Keep exploring
Close to this topic, not on this page.
