Handle network failures

When making HTTP requests in Ruby, it‘s inevitable that some of those requests will occasionally fail. Maybe the API you‘re hitting is experiencing an outage, maybe you‘ve been rate limited, or maybe there was just a temporary network glitch. Whatever the reason, failed requests are a fact of life in the world of web programming.

Luckily, Ruby makes it easy to detect and retry failed requests. In this guide, we‘ll go over everything you need to know to gracefully handle failures and ensure your Ruby scripts keep humming along smoothly. Let‘s get started!

Why Requests Fail

First, it‘s important to understand the different reasons an HTTP request might fail. Some common failure scenarios include:

• Timeouts – The server takes too long to respond and the request times out
• Rate limiting – You‘ve exceeded the number of allowed requests in a given time period
• Server errors – The API returns a 5xx status code indicating an internal server error
• Network issues – There are problems with the network connection
• Invalid requests – Your request is malformed or missing required parameters

When a failure occurs, Ruby will typically raise an exception or return a non-200 level HTTP status code. It‘s up to you to detect these failure signals and decide how to proceed.

Checking for Failures

The simplest way to check if a request failed is by inspecting the HTTP status code. Anything in the 200-299 range indicates success, while codes in the 400-599 range indicate an error occurred. You can access the numeric status code on the response object:


response = HTTP.get("http://api.example.com/users")
status = response.code


Ruby will also raise an exception for certain kinds of failures, like network timeouts or invalid URLs. To handle these, you can wrap your request code in a begin/rescue block:


begin
response = HTTP.get("http://api.example.com/users")
rescue HTTP::ConnectionError

end


Retrying with Exponential Backoff

Once you‘ve detected a failed request, you‘ll typically want to retry it. However, it‘s important not to retry immediately, as this can exacerbate problems like rate limiting or server overload.

Instead, best practice is to use a technique called exponential backoff. The idea is simple: after each failed attempt, you double the amount of time you wait before trying again. This gives the server time to recover and reduces your chances of being rate limited.

Here‘s an example of how you might implement exponential backoff in Ruby:


max_retries = 5
retries = 0
base_delay = 0.5 # seconds

begin
response = HTTP.get("http://api.example.com/users")
rescue HTTP::Error
if retries < max_retries
delay = base_delay * (2 ** retries)
sleep delay
retries += 1
retry
else
raise
end
end


In this code:

1. We set a maximum number of retry attempts, so we don‘t keep trying indefinitely
2. We use a base delay as our initial wait time
3. When a request fails, we check if we‘ve reached the retry limit
4. If not, we calculate the wait time by doubling the base delay for each previous attempt
5. We sleep for the calculated delay, increment the retry counter, and re-run the request
6. If we‘ve exceeded the max retries, we give up and re-raise the exception

This will give the problematic API a chance to recover between attempts, without wasting time on requests that are likely to fail.

Logging Failed Requests

Even with retries and exponential backoff, you may occasionally run into requests that stubbornly refuse to succeed. In these cases, it‘s valuable to record some information about the failures, so you can debug the underlying issue.

At minimum, you‘ll want to log:

• The URL of the failed request
• The HTTP status code
• The exception message
• A full stack trace

You can write this information to a dedicated log file, or print it to the console if you‘re running the script manually. For example:


def with_logging(url)
yield
rescue => e
puts "[#{Time.now}] Request failed:"
puts "URL: #{url}"
puts "Status code: #{e.response.status}"
puts "Exception: #{e.message}"
puts e.backtrace.join("\n")
raise
end

with_logging("http://api.example.com/users") do
HTTP.get("http://api.example.com/users")
end


This will output detailed information about any failed requests, while still surfacing the exception so it can be handled upstream.

Reusable Retry Logic

Since most Ruby projects make HTTP requests from multiple places, it makes sense to encapsulate your retry logic in a reusable method. For example, you could write a ‘with_retries‘ method that accepts a block to execute:


def with_retries(max_retries: 3, base_delay: 0.5)
retries = 0
begin
yield
rescue HTTP::Error
if retries < max_retries
delay = base_delay * (2 ** retries)
sleep delay
retries += 1
retry
else
raise
end
end
end


Then use it like:


with_retries do
HTTP.get("http://api.example.com/users")
end


This keeps your code DRY and makes it easy to adjust your retry parameters in a single location.

The retriable Gem

If you don‘t want to write your own retry logic from scratch, there are several excellent gems available. One of the most popular is simply called ‘retriable‘.

With retriable, you can wrap any block of code with robust retry behavior in just a few lines:


require ‘retriable‘

Retriable.retriable(on: HTTP::Error, tries: 5, base_interval: 0.5) do
HTTP.get("http://api.example.com/users")
end


This will automatically retry failed requests using exponential backoff, with sane defaults for things like the maximum number of attempts and delay between tries. Of course, you can customize these to your liking.

Retriable can also be configured globally and supports more advanced features like callbacks and context-sensitive parameters. It‘s a great option if you find yourself writing a lot of retry logic.

Conclusion

Transient failures are an unavoidable challenge when working with APIs and other web services. Fortunately, Ruby provides the tools you need to detect and gracefully recover from these failures.

By implementing techniques like exponential backoff, logging, and reusable retry logic, you can keep your applications running smoothly in the face of server outages, network issues, and other common problems. With a bit of clever coding, your Ruby scripts will be all but unbreakable. Now get out there and start retrying those requests!