As an SRE, understanding how systems fail is crucial for building resilient infrastructure. Three common and often interconnected failure modes are cascading failures, thundering herd, and split brain. Cascading failures are like a domino effect: the failure of one component triggers subsequent failures in dependent components, eventually bringing down a large part or even the entire system. For example, an overloaded database might cause application servers to exhaust their connection pools and crash, which then overloads other services trying to connect to those now-failed application servers, creating a widespread outage from a single point of failure.
The thundering herd problem occurs when a large number of clients or processes simultaneously attempt to access a limited resource or perform an identical operation. A classic example is when a cache entry expires, and hundreds or thousands of requests concurrently try to rebuild the cache by hitting the underlying (often slower) database. This sudden spike in demand can overwhelm the database, leading to slow responses, timeouts, and potentially triggering a cascading failure. Similarly, when a primary server fails, multiple standby servers might simultaneously attempt to become the new primary, overwhelming the election mechanism.
Split brain is a specific issue for distributed systems, especially those relying on leader election or shared state management. It happens when network issues or component failures cause different parts of a cluster to lose communication and independently decide they are the authoritative 'leader' or have the most up-to-date state. This can lead to multiple 'leaders' trying to write conflicting data, process transactions independently, or allocate resources duplicate, resulting in data corruption, inconsistency, or services behaving erratically. Preventing split-brain typically involves robust consensus algorithms (like Raft or Paxos) and fencing mechanisms to ensure only one leader can actively write data.
Key Takeaways
- Cascading failures spread disruption; prevent with isolation (bulkheads, circuit breakers) and graceful degradation.
- Thundering herd overloads shared resources; mitigate with request coalescing (single-flight), intelligent caching, and exponential backoff.
- Split brain causes conflicting states in distributed systems; ensure strong consensus algorithms and fencing to maintain data integrity.
- These failure modes often combine, exacerbating outages and making recovery more complex.
Code Example
import threading
import time
cache = {}
cache_lock = threading.Lock() # Protects access to the cache
db_calls = 0
def get_data_from_db(key):
global db_calls
time.sleep(0.1) # Simulate an expensive database call
db_calls += 1
return f"data_for_{key}"
def get_data_with_single_flight(key):
# If the key is already in cache, return immediately
if key in cache:
return cache[key]
# Only one thread/process should rebuild the cache for a given key
with cache_lock:
# Double-checked locking: after acquiring lock, check again
if key in cache:
return cache[key]
# Perform the expensive operation and update cache
data = get_data_from_db(key)
cache[key] = data
return data
# Without single-flight, multiple threads might hit get_data_from_db
# With single-flight, only one thread will perform the expensive call for 'key'How this code works
This code demonstrates "single-flight" caching, a pattern that prevents multiple simultaneous requests from overwhelming a backend when refreshing a cache entry. Its core job is to ensure that an expensive operation, like fetching data from a database using get_data_from_db, is performed only once for a specific key at any given time, even if many clients ask for that key simultaneously. This prevents a "thundering herd" of requests hitting the database unnecessarily.
The get_data_with_single_flight function first checks if key is already in the cache. If it is, the data is returned immediately. If not, a cache_lock is acquired using with cache_lock, ensuring only one thread can proceed to rebuild the cache for any key at this point. Crucially, a second if key in cache check is performed after acquiring the lock. This "double-checked locking" handles the subtle case where another thread might have just finished populating the cache for this key while the current thread was waiting for the lock. Only if the key is still missing from the cache does the code then call get_data_from_db and update the cache, incrementing db_calls just once per cache miss.