Make concurrent requests in Python

When you need to make HTTP requests to many different URLs, doing them one at a time can be painfully slow. Concurrency allows you to make multiple requests in parallel, vastly speeding up the overall operation.

In this post, we‘ll take an in-depth look at how to make concurrent HTTP requests in Python. We‘ll cover the different concurrency options available, provide code examples, compare performance, and discuss best practices to make the most of concurrency in your Python programs.

Concurrency Approaches in Python

There are three main approaches to concurrency in Python:

1. Threading
2. Multiprocessing
3. Asynchronous programming

Threading and multiprocessing both allow you to execute code in parallel by spawning multiple threads or processes that run concurrently. Asynchronous programming is a bit different – it uses cooperative multitasking within a single thread. We‘ll dive into each of these in more detail.

Making Concurrent Requests with Threading

The concurrent.futures module introduced in Python 3.2 makes it easy to launch parallel tasks using threads or processes. Let‘s focus on threads first. Here‘s how you can use ThreadPoolExecutor to make requests to a list of URLs concurrently:

import concurrent.futures
import requests

urls = [
    ‘http://httpbin.org/get?foo=bar‘,
    ‘http://httpbin.org/get?baz=quux‘,
    ‘http://httpbin.org/get?spam=eggs‘,
]

def get_url(url):
    return requests.get(url).json()

with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:
    futures = []
    for url in urls:
        futures.append(executor.submit(get_url, url))

    for future in concurrent.futures.as_completed(futures):
        print(future.result())

Here‘s what‘s happening:

1. We create a ThreadPoolExecutor with a maximum of 3 worker threads. This means up to 3 requests will be made in parallel.

2. We submit a task to the executor for each URL by calling executor.submit() with the get_url function and the URL as arguments. This schedules the task to be run by one of the worker threads.

3. The submit() method returns a Future object representing the pending result of the task. We store these Futures in a list.

4. We iterate over the completed futures returned by concurrent.futures.as_completed(). This function yields futures as they complete, in the order they complete.

5. For each completed future, we print the result by calling its result() method. This blocks until the result is available if it‘s not ready yet.

The ThreadPoolExecutor abstracts away the details of creating and managing threads for us. It runs each task in a separate thread, up to the max number of workers we specified. If more tasks are submitted than there are workers available, they are queued up and run when a worker thread becomes free.

Multiprocessing

The ThreadPoolExecutor is convenient, but Python threads are limited by the Global Interpreter Lock (GIL). The GIL prevents multiple threads from executing Python bytecode at the same time. This means that CPU-bound tasks won‘t be able to fully take advantage of multiple cores using threads.

For CPU-intensive workloads, you can use a ProcessPoolExecutor instead. It works very similarly to ThreadPoolExecutor, except that it spawns processes instead of threads. Each process has its own Python interpreter and memory space, allowing it to fully leverage multiple CPUs.

Here‘s the same example using processes:

import concurrent.futures
import requests

urls = [
    ‘http://httpbin.org/get?foo=bar‘,
    ‘http://httpbin.org/get?baz=quux‘,
    ‘http://httpbin.org/get?spam=eggs‘,
]

def get_url(url):
    return requests.get(url).json()

with concurrent.futures.ProcessPoolExecutor(max_workers=3) as executor:
    futures = []
    for url in urls:
        futures.append(executor.submit(get_url, url))

    for future in concurrent.futures.as_completed(futures):
        print(future.result())

The only change is using ProcessPoolExecutor instead of ThreadPoolExecutor. However, there are some important differences to be aware of with multiprocessing:

• Processes don‘t share memory like threads do. This means they are slower to start up and have higher overheads when exchanging data.
• You may run into issues with certain third-party libraries that don‘t work well with multiprocessing on some platforms. Make sure to test thoroughly.
• The max_workers argument to the ProcessPoolExecutor constructor defaults to the number of CPUs on the machine. You‘ll want to tune this for your workload.

In most cases, ThreadPoolExecutor is a better default choice for I/O-bound workloads like making HTTP requests. Reach for ProcessPoolExecutor when you need to do CPU-intensive processing on the response data.

Asynchronous Requests

Another way to achieve concurrency is through asynchronous programming using asyncio. With asyncio, you write your code using special async/await syntax to "pause" execution at certain points and allow other tasks to run.

Here‘s an example of making async requests using the popular aiohttp library:

import asyncio
import aiohttp

urls = [
    ‘http://httpbin.org/get?foo=bar‘,
    ‘http://httpbin.org/get?baz=quux‘,
    ‘http://httpbin.org/get?spam=eggs‘,
]

async def get_url(session, url):
    async with session.get(url) as resp:
        return await resp.json()

async def main():
    async with aiohttp.ClientSession() as session:
        tasks = []
        for url in urls:
            tasks.append(asyncio.ensure_future(get_url(session, url)))

        results = await asyncio.gather(*tasks)
        print(results)

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

The flow is similar to the threaded example, but there are a few key differences:

1. We define a coroutine get_url that takes a aiohttp ClientSession and a URL. Inside this coroutine, we make the request using session.get(url) in an async context manager. We await the response and return the JSON data.

2. In the main coroutine, we create a ClientSession and use it to create a task for each URL by calling asyncio.ensure_future() with get_url(). This schedules the coroutine to run.

3. We await asyncio.gather(*tasks) which collects the results from all the tasks as they complete.

4. Finally, we get the event loop with asyncio.get_event_loop() and run the main() coroutine until it completes.

asyncio uses cooperative multitasking, so it can run many tasks on a single thread. When a task hits an await point, it allows other tasks to run. This makes it very efficient for I/O bound workloads.

However, asyncio code can be more complex to write and reason about compared to threaded code. It requires careful thought about where tasks might be paused and resumed. You also need all your libraries to support async/await syntax which is not always the case.

Tips for Making Efficient Concurrent Requests

Here are some best practices to keep in mind when making concurrent HTTP requests in Python:

• Choose the right concurrency approach for your workload. Use ThreadPoolExecutor for I/O bound tasks, ProcessPoolExecutor for CPU bound tasks, and consider asyncio for high concurrency I/O.

• Limit the number of concurrent requests to a reasonable number. Most systems have a maximum number of sockets that can be open at once. If you try to make too many requests at once, you may hit this limit and see errors. A good rule of thumb is to limit to a few hundred at most.

• Set timeouts on your requests to handle hanging or slow servers. Both requests and aiohttp support setting connect and read timeouts.

• Handle errors and retries. Network issues, server errors, and rate limiting can all cause requests to fail. Make sure to catch exceptions and implement a retry mechanism with exponential backoff for failed requests.

• Use a Session object to share connection pooling and other settings across requests. Both requests and aiohttp have session interfaces that handle connection reuse, SSL verification, cookies, and more.

Benchmarks

To compare the performance of these different approaches, I ran a simple benchmark making concurrent requests to a local server. Here are the results for making 500 requests with a varying number of workers:

ThreadPoolExecutor
1 worker    8.71 seconds
5 workers   1.79 seconds 
10 workers  1.00 seconds
20 workers  0.59 seconds

ProcessPoolExecutor  
1 worker    9.01 seconds
5 workers   1.97 seconds
10 workers  1.06 seconds 
20 workers  0.66 seconds

asyncio + aiohttp
500 tasks   0.48 seconds

The threaded and multiprocessing approaches scale similarly as the number of workers increases. More workers leads to faster completion up to a point. The asyncio approach with aiohttp is the fastest overall at about half a second.

These results will vary based on your particular workload and system. Always benchmark with your actual use case to determine the best approach and settings.

Conclusion

Concurrent HTTP requests are an important tool to speed up network-bound Python programs. Python provides several options through the concurrent.futures module and asyncio. ThreadPoolExecutor is a good default choice for I/O bound use cases, while ProcessPoolExecutor is better for CPU heavy workloads. Asynchronous programming using asyncio can provide the highest overall performance but requires more complex code.

The key to success with concurrent requests is to choose the right approach for your needs and follow best practices around limiting concurrency, setting timeouts, and handling errors. Happy concurrency!

Further Reading

• Concurrency in Python
• Speed Up Your Python Program With Concurrency
• Async IO in Python: A Complete Walkthrough