Are you working on an application that uses one of the Google APIs. Have you encounter one of the following errors?
- 403 userRateLimitExceeded
- 403 rateLimitExceeded
- 429 RESOURCE_EXHAUSTED
These errors are basically flood protection. Your application is running to fast. When you see these errors Google recommends you implement Exponential Backoff.
Implementing exponential backoff
[wp_ad_camp_3]
Exponential Backoff is where your application periodically retries a failed request over an increasing amount of time. It is a standard error handling strategy for network applications. The Google API’s are designed with the expectation that clients which choose to retry failed requests do so using exponential backoff. Besides being “required”, using exponential backoff increases the efficiency of bandwidth usage, reduces the number of requests required to get a successful response, and maximizes the throughput of requests in concurrent environments.
The flow for implementing simple exponential backoff is as follows.
- Make a request to the API
- Receive an error response that has a retry-able error code
- Wait 1s + random_number_milliseconds seconds
- Retry request
- Receive an error response that has a retry-able error code
- Wait 2s + random_number_milliseconds seconds
- Retry request/li>
- Receive an error response that has a retry-able error code
- Wait 4s + random_number_milliseconds seconds
- Retry request
- Receive an error response that has a retry-able error code
- Wait 8s + random_number_milliseconds seconds
- Retry request
- Receive an error response that has a retry-able error code
- Wait 16s + random_number_milliseconds seconds
- Retry request
- If you still get an error, stop and log the error.
In the above flow, random_number_milliseconds is a random number of milliseconds less than or equal to 1000. This is necessary to avoid certain lock errors in some concurrent implementations. random_number_milliseconds must be redefined after each wait. If you are using any of the Google Client libraries you shouldn’t have to worry about this most of them have already implemented this for you.
Taking it one step farther
Exponential backoff works just fine. My application detects an error and waits a few milliseconds and then tries it again. The great thing is that 90% of the time I only need to wait and retry once before I get data back. The problem is that it still bothers me that I am getting the initial error message to begin with.
So I have come up with a way of effectually keeping track of how many request my application was making and then slowing it down just a little before Google would begin to detect flooding.
GoogleFlood Buster
This class is designed to allow you to keep track of the number of request you are making to Google. The time each request is made is logged by the class it tracks how many requests have been made within a given amount of time. If to many have been made then it sleeps just enough to prevent flooding the server. This does not work 100% of the time but it will reduce the number of flooding errors you get significantly.
[wp_ad_camp_5]
Conclusion
Google recommends that we implement Exponential Backoff when we encounter any of the RateLimitExceeded errors. I personally like to take things one step further and try to limit the amount of flooding my applications do. For the most part Google allows us to access there APIs free of charge I like to be polite when accessing their systems.