Using HTTP Headers with Axios: The Complete Guide

When making HTTP requests from your code, whether it‘s for fetching data from an API, submitting a form, or scraping a webpage, it‘s critical to understand and utilize HTTP headers properly. Headers provide additional context and instructions for both the client sending a request and the server sending a response. Setting the appropriate headers can make your requests more efficient, help you integrate with APIs seamlessly, and make your scraping bots look more like authentic browser traffic.

In this in-depth guide, we‘ll focus on how to work with HTTP headers when using the popular Axios HTTP client library for JavaScript. We‘ll cover everything you need to know, including an overview of key HTTP headers, how to set request headers with Axios, and best practices and tips to make the most of headers in your projects.

HTTP Headers 101

Before we dive into using headers with Axios, let‘s make sure we‘re on the same page about what HTTP headers are and why they‘re important.

HTTP headers are additional pieces of information sent along with an HTTP request or response. Headers are key-value pairs that provide metadata about the request or response, give instructions for how it should be handled, provide context about the sender, and more.

Some key things HTTP headers are used for:

• Specifying the content type of the request/response body
• Setting caching policies
• Providing authentication credentials
• Describing the user agent (browser/device) making the request
• Controlling CORS (Cross-Origin Resource Sharing) behavior

Headers are a fundamental part of the HTTP protocol, so having a solid grasp on them is essential for any kind of web development.

Common HTTP Headers to Know

There are dozens of different HTTP headers, some standardized and some custom. Here are a few of the most common and important headers to be familiar with:

• Content-Type: Indicates the media type (format) of the request or response body, like application/json or text/html
• Authorization: Contains credentials for authenticating the sender of the request, commonly a Bearer token
• User-Agent: Describes the client application making the request, like a web browser or bot
• Accept: Tells the server what media types the client can handle in the response
• Cache-Control: Sets caching behavior, like no-cache or max-age=3600
• Origin: Indicates the origin (scheme, hostname, and port) of the request sender, used in CORS
• Referer: The URL of the previous web page that linked to the currently requested page

Having a general awareness of these and other common headers will help you understand how to inspect and construct your own HTTP requests.

Sending HTTP Headers with Axios

Now that we‘ve covered the fundamentals of HTTP headers, let‘s look at how to use them with the Axios HTTP client library.

Axios provides a simple and intuitive API for making HTTP requests and dealing with responses. It has several advantages over built-in browser methods like fetch(), including better error handling, the ability to cancel requests, and default protection against XSRF. You can make requests with Axios using its get, post, put, and delete methods (among others).

Setting Request Headers

To set HTTP headers on a request with Axios, you pass a headers configuration object to the request method. This headers object is a key-value mapping of header names to header values. Here‘s what a simple GET request with custom headers looks like:

const axios = require(‘axios‘);

const headers = {
  ‘Authorization‘: ‘Bearer my-access-token‘,
  ‘Content-Type‘: ‘application/json‘,
  ‘User-Agent‘: ‘my-app/1.0‘
};

axios.get(‘https://api.example.com/data‘, {headers})
  .then(response => {
    console.log(response.data);
  });

In this example, we‘re setting the Authorization, Content-Type, and User-Agent headers on the request by passing a headers configuration object to axios.get(). Note that the second argument to axios.get() is an options object where we can put configuration like headers, query parameters, and more.

We can use the same approach to set headers on other types of requests like POST:

axios.post(‘https://api.example.com/login‘, 
  {username: ‘user123‘, password: ‘secret‘},
  {headers: {‘Content-Type‘: ‘application/json‘}} 
)
  .then(response => {
    console.log(response.data);
  });

Here we‘re making a POST request with a JSON payload and explicitly setting the Content-Type header to application/json. Even though this is the default Content-Type for Axios, it‘s a good practice to set it explicitly in your code to make the expected behavior clear.

Setting Default Headers

If you want certain headers to be set on every request your app makes through Axios, you can set them as defaults in the Axios configuration:

axios.defaults.headers.common[‘Authorization‘] = ‘Bearer my-access-token‘;
axios.defaults.headers.post[‘Content-Type‘] = ‘application/json‘;

Now, the Authorization header will be included on all requests, and the Content-Type header will default to application/json for POST requests. You can override these defaults on individual requests by setting different values for those headers.

Making Requests Look Browser-Like

One of the most common use cases for custom HTTP headers is making your programmatic requests look more like they came from a real web browser. This is particularly relevant for web scraping, where sites may block requests from obvious bots and scripts.

To make your requests seem browser-like, here are a few key headers to set:

• User-Agent: Set this to a real browser‘s user agent string, like Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36
• Accept: Include values that browsers commonly ask for, like text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
• Accept-Language: Represents the human language the client prefers, like en-US,en;q=0.5
• Referer: Set this to the URL of the page where the request would have originated from in a browser
• Origin: Set this to the full origin of the referring page, if the request would be a cross-origin browser request

Here‘s an example with realistic browser-like headers:

axios.defaults.headers.common[‘User-Agent‘] = ‘Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36‘;
axios.defaults.headers.common[‘Accept‘] = ‘text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8‘;
axios.defaults.headers.common[‘Accept-Language‘] = ‘en-US,en;q=0.5‘;

axios.get(‘https://example.com/data‘, {
  headers: {
    ‘Referer‘: ‘https://example.com‘, 
    ‘Origin‘: ‘https://example.com‘
  }
})
  .then(response => {
    console.log(response.data);
  });

With headers like this, your request will look much more authentic and are more likely to succeed against bot protection.

Note that while carefully setting headers goes a long way in making requests look real, it‘s not a complete scraping solution by itself. Websites can use other signals like cookies, browser fingerprints, and CAPTCHAs to detect and block undesired bots. For large scale scraping, you‘ll likely need a battle-tested, comprehensive solution like ScrapingBee to avoid blocks and gather data reliably.

Conclusion

HTTP headers are a key tool for controlling the behavior of HTTP requests and responses. When using an HTTP client like Axios to make requests from your JavaScript code, it‘s important to understand how to inspect, set, and customize headers to meet your needs.

In this guide, we covered the fundamentals of HTTP headers, showed how to set default and per-request headers with Axios, and gave tips for using headers to make requests look more authentic for scraping purposes. With the knowledge you‘ve gained here, you‘re well-equipped to tackle almost any HTTP request use case in your projects.

While mastering HTTP headers is a critical piece of the puzzle, remember that there‘s a lot more that goes into successful scraping and other advanced HTTP request scenarios. If you want to save time and headaches, consider a dedicated tool like ScrapingBee that handles header best practices, CAPTCHAs, rate limits, and more out-of-the-box, letting you focus on your core development goals.