Phase 5: Platform Engineering

Service mesh, network policies & ingress reliability

Advanced ~3 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

Imagine a really busy restaurant kitchen, but instead of just a few chefs, you have hundreds of chefs (we'll call them "services") working on tiny parts of many different meals all at once. One chef specializes in making burger patties, another in toasting buns, another in slicing tomatoes, and yet another in making fries.

A "Service Mesh" is like having a super-smart kitchen manager who isn't a chef themselves, but stands between all these busy chefs. This manager makes sure everyone talks nicely and securely – like ensuring the burger chef's secret recipe for a special sauce isn't accidentally shared with the fry chef. The manager also handles complex tasks: "This customer wants two burgers, make sure both patties and buns come from the same line of chefs, and if the bun chef is too slow, send that order to the backup bun chef." This manager tracks everything: how long each part of the meal takes, if any chef is struggling, and if all the ingredients reach the right place safely.

Now, even with a smart manager, you still need rules. "Network Policies" are like special sticky notes or chalkboards in the kitchen that say, "The dessert station can only send finished cakes to the serving counter, not to the meat prep area." Or, "Only the head chef can talk directly to the ingredient supplier, no one else." These rules are invisible security guards that make sure even if one chef accidentally makes a mistake, they can't mess up another part of the kitchen or peek at things they shouldn't. It keeps everyone working in their own safe zone.

Finally, "Ingress Reliability" is all about how reliable and steady the main front door of the restaurant is. This is where all the customer orders come in, and where the finished meals go out to the dining room. If the front door gets jammed, or the person taking orders gets overwhelmed, no one gets their food, no matter how good the chefs are. So, ensuring "Ingress Reliability" means having a super sturdy front door, multiple friendly people taking orders, and backup plans for everything, so that no matter how many hungry customers show up, orders always flow smoothly into the kitchen, and delicious food always flows out to the tables without any delays or mishaps.

So, with our smart kitchen manager, strict kitchen rules, and a super reliable front door, you can run a restaurant that's incredibly busy, super secure, and always delivers delicious food to happy customers without major meltdowns. This means you can add new dishes, change recipes, or bring in new chefs without worrying that the whole system will fall apart.

For advanced SREs operating Kubernetes at scale, a service mesh (like Istio or Linkerd) becomes indispensable for managing inter-service communication. It extends Kubernetes with an application-aware network, abstracting away critical concerns such as mutual TLS (mTLS) for secure communication, fine-grained traffic routing (canary deployments, A/B testing), robust reliability patterns (retries, timeouts, circuit breakers), and unparalleled observability via golden signals for every service interaction. This elevates your control plane from basic L4 load balancing to sophisticated L7 traffic management and security. Complementing the service mesh are Kubernetes Network Policies. These operate at L3/L4, acting as internal firewalls to restrict pod-to-pod and namespace-to-namespace communication based on labels. They are a fundamental security primitive, enforcing a least-privilege network posture within your cluster and preventing unauthorized lateral movement, even if a single pod is compromised.

Ensuring ingress reliability is paramount, as it represents the critical entry point to your cluster. This involves more than just deploying an Ingress Controller; it requires a deep SRE focus on its high availability, scalability, and robust configuration. You'll need to strategically implement redundancy (multiple controller instances across zones), perform intelligent load balancing at the edge (e.g., sticky sessions, weighted routing), and integrate robust health checks and auto-scaling. Beyond basic routing, consider features like rate limiting, Web Application Firewall (WAF) integration, DDoS protection, and end-to-end TLS termination/re-encryption at the ingress layer. A reliable ingress setup also often involves external DNS integration (e.g., ExternalDNS) for automated record management and robust certificate management (e.g., Cert-Manager) for seamless TLS lifecycle, ensuring that external traffic always reaches your services securely and efficiently.

These three components — service mesh, network policies, and ingress reliability — are not isolated but form a layered defense and control strategy. Network policies establish a strong L3/L4 perimeter; the service mesh provides deep L7 traffic management, security, and observability between services; and a reliable ingress ensures the cluster's edge is robust and secure. As an SRE, mastering their interplay allows you to build highly resilient, secure, and observable distributed applications, moving beyond basic Kubernetes constructs to address the complex operational challenges of production systems at scale. This holistic approach is crucial for achieving high SLOs and maintaining operational excellence.

Key Takeaways

  • Service mesh (L7) provides mTLS, fine-grained traffic control, and deep observability for microservices.
  • Network Policies (L3/L4) enforce least-privilege network access between pods for internal security.
  • Ingress reliability demands highly available, scalable, and secure external access to the cluster.
  • These components form a layered, complementary strategy for security, traffic management, and observability.
  • SREs leverage this stack to build resilient, secure, and observable distributed applications at scale.

Code Example

yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-backend
  namespace: default # Or your specific application namespace
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: frontend
      ports:
        - protocol: TCP
          port: 8080

How this code works

This Kubernetes NetworkPolicy establishes a critical security rule: it explicitly allows network traffic only from frontend application pods to backend pods on a specific port. Its main job is to restrict communication, ensuring that only trusted components can talk to each other within the application's network. This helps build a more resilient and secure service architecture by limiting the potential attack surface.

The podSelector targets pods labeled app: backend, meaning this policy applies to all instances of the backend service. The policyTypes: - Ingress declares that this rule governs incoming connections to these backend pods. The ingress section then specifies what traffic is allowed: only from pods with app: frontend labels, and only on TCP port: 8080. A subtle but crucial point for beginners is Kubernetes' security model: once any NetworkPolicy selects a pod, all network traffic not explicitly permitted by a policy is automatically denied. This means this policy doesn't just add an allowance; it creates a secure, "default deny" environment for the backend, where everything is blocked unless precisely allowed.

Resources

Related Articles

Hand-picked reads to go deeper on kubernetes for sre.

Back to Site Reliability Engineer Roadmap