A/B testing, in the context of model serving, involves directing different subsets of live production traffic to distinct model versions to compare their real-world performance. The primary goal is to validate which model (e.g., a baseline Model A versus a new Model B) performs better against specific business metrics, such as conversion rates, user engagement, or fraud detection accuracy, rather than solely relying on offline ML metrics. This requires a robust traffic routing mechanism, typically at the API gateway or service mesh level, and careful statistical analysis to determine if observed differences are significant and warrant a full rollout of the winning model.
Shadow deployments, often called dark launches, provide a low-risk method for validating a new model's operational stability and predictive behavior under live traffic conditions without impacting end-users. With a shadow deployment, all or a percentage of incoming production requests are duplicated and sent to both the currently active model and the new 'shadow' model. The key difference is that only the active model's predictions are returned to the user; the shadow model's predictions are discarded or logged for comparison. This allows MLOps teams to monitor the shadow model's latency, error rates, resource consumption, and the quality of its predictions (e.g., comparing its outputs to the live model's outputs for significant divergence) before it ever serves a user.
Practically, shadow deployments often precede A/B tests. First, you'd deploy a new model in shadow mode to ensure it's stable and performs as expected operationally under real load. Once validated, you might then move to an A/B test, gradually exposing a small percentage of users to the new model to measure its impact on business-critical metrics. Tools like service meshes (e.g., Istio, Linkerd) or cloud-native load balancers are instrumental in managing the complex traffic routing and mirroring required for these advanced deployment strategies.
Key Takeaways
- A/B testing compares live model performance using split production traffic, focusing on business metrics to determine user impact.
- Shadow deployments validate new models under live traffic without impacting users, primarily for operational stability and prediction health checks.
- Shadow deployments often precede A/B tests, de-risking new model rollouts by verifying operational readiness first.
- Both strategies require robust traffic management (e.g., service mesh) and comprehensive monitoring.
- The ultimate goal is to safely and strategically deploy models that drive better business outcomes.
Code Example
apiVersion: networking.k8s.io/v1beta1
kind: VirtualService
metadata:
name: ml-inference-service
spec:
hosts:
- "ml-api.example.com"
gateways:
- ml-gateway
http:
- route:
- destination:
host: ml-model-v1 # Baseline model (A)
port:
number: 80
weight: 90
- destination:
host: ml-model-v2 # Challenger model (B)
port:
number: 80
weight: 10How this code works
This YAML configures a VirtualService to manage how user requests are routed to different versions of an ML model, enabling A/B testing or gradual rollouts. Specifically, it directs traffic to either ml-model-v1 (the baseline model) or ml-model-v2 (the challenger). This allows engineers to compare the performance of a new model (v2) with the existing one (v1) using real user traffic, without fully exposing v2 to everyone immediately. The service exposed via ml-api.example.com acts as the single entry point for all inference requests, abstracting away the underlying model versions.
Inside the spec, the hosts entry ml-api.example.com defines the public-facing URL for the ML service, while gateways links it to the cluster's ingress. The core routing logic is under http, where a list of route objects specifies traffic distribution. Each destination points to an internal Kubernetes service (like ml-model-v1 or ml-model-v2) which represents a specific deployment of a model version, usually accessed on port 80. The crucial part is weight: ml-model-v1 receives 90% of requests, and ml-model-v2 gets 10%. A common pitfall is forgetting that these weight values for all destinations within a single http block must always sum to 100 to ensure all traffic is accounted for, otherwise some requests might not be routed at all.