What will happen to a food item if a large group of animals (a herd) starts running towards it, all at once?
Problem Statement
Now think of a same scenario for a system or a component of a system. If a large number of requests arrives at once trying to acquire a resource, the system will be overloaded if not handled well. The sudden traffic spike can cause issues like high CPU utilisation, memory leaks, etc.
Thundering Herd is often used in-relation to cache and databases. For a cache-miss scenario, all the requests trying to access same item will be routed to the database. This overloads the database if database is not scaled to handle this load. This causes a cascading failure in the system bringing down the entire system.
Resolution
The resolution has two directions, one is mitigation (come out of situation if thundering herd occurs) and other is prevention (system is designed in a way so thundering herd doesn’t occur). Let’s explore both of these directions.
Thundering Herd Mitigation
It’s really hard to come out of Thundering Herd situation. This can cause cascading failures and bring the entire infrastructure down. The best is dropping the user requests. This could be for all the users or a percentage of users depending on the situation.
The database might have queued up all the requests, so restart the database. Same applies to other infra components.
Thundering Herd Prevention
The system can be designed in a way ensuring it can come out of these situations easily if they happen and enhancements ensuring this situation doesn’t happen.
Discard Requests
The first solution is load shedding. Serving some requests is always better than serving none of them. This is the idea behind load shedding because an overloaded system can cause cascading failures.
Here are few suggestions on discarding requests:
Discard lower priority requests. This could be some back-office jobs that can be scheduled for later point of time.
Discard % of user requests. If above doesn’t help, the system should drop some percentage of user requests.
Keep a request buffer. The requests are put in a queue for certain time and served if capacity is available (or system recovers).
Watch the dive deep on Uber’s Load Shedding Framework QALM (QoS Load Management Framework) and Cinnamon architecture (Part I, Part II, Part III) for discussion on how these systems are built.
Jitter in Retries
If a request fails, what you do? Hit the refresh button. The retry could be user enforced (such as refresh) or system invoked on request failures. Retries are great to recover from transient failures but this can overload the system in case of thundering herd scenarios.
If the system is not able to process the requests, and all the users retried at the same time. This makes the situation even worse. Introducing Jitter means there is random lag between the retries ensuring not all requests arrive at the same time.
Exponential backoff in retries is only good if applied with Jitter else it doesn’t solve the problem it is supposed to. In summary, don’t retry at fixed intervals. Jitter is must, exponential backoff is good to have.
What’s Important Should Stay Alive?
Big software architecture has a lot of components (or services). Each component does some job and might depend on other components for part of the job. There are also some good have components. Take an example of video feed generated by Netflix, it’s customised for an user.
In overloaded system scenarios, it’s ok to reflect same video feed to all users and stop the work from recommendation system. This is good to have and not critical. The critical component is video play. The video should start once user clicks on play button.
The recommendation is clear division between the services. General criteria followed by many companies is tier strategy. Tier 1 service is most critical, Tier 2 is less critical and so on.
The chaos and stress testing should focus majorly on Tier 1 services ensuring system operations in unknown scenarios.
Scaling
Automated Scaling is one of suggestions that floats around to unburden the servers from this load. For thundering herd, it doesn’t help much. Bringing up a server takes some time (even with light-weight application binaries and low startup time).
Pre-scaling is one recommendation if you’re aware of some upcoming event and loosening up the auto-scaling thresholds so the system doesn’t wait for hit upper bounds can help.
Request Coalescing
We discussed thundering herd occurs if a huge number of requests are trying to access the same resource. Can we deploy a mechanism so as only one request is processed and others wait for for its results, see Figure 1.
Figure 1 Request Coalescing
All the requests are trying get(x), only one request actually fetch the result and rest of them use the same. This shields the backend systems, resolving thundering herd issue. At the caching layer, this problem is solved with a cache promise.
Cache Promise works on the similar concept, it stores a promise till the results are retried by the primary query and other requests wait for this promise. Once the promise is fulfilled, all requests are served.
Interested in practical implementation of these concepts, STAY TUNED. Something exciting is coming — https://msdeepsingh.com/books/