How to Send GET Requests With cURL: An Expert‘s Guide

As someone who has used cURL daily for over 10 years for data extraction and web scraping, I can‘t recommend it enough for sending customizable GET requests. While modern languages like Python make HTTP requests easy, cURL remains a powerful tool every developer should have in their toolbox.

In this comprehensive 2200+ word guide, you‘ll learn:

• What exactly is cURL and why is it useful?
• How to send simple GET requests with cURL
• Essential options like viewing response headers and following redirects
• Retrieving data in JSON format
• Passing GET parameters and reading API docs
• Saving response content to a file
• A GET request argument cheat sheet
• Debugging tips and tricks I‘ve picked up over the years
• And much more…

By the end, you‘ll have the cURL skills to extract data from APIs and websites with ease. Let‘s get started!

A Brief History of cURL

cURL first appeared in 1997 created by Daniel Stenberg to transfer data over various protocols. According to Stenberg, the name originally stood for "Client URL Request Library".

Over the last 25 years, cURL has become one of the most popular command-line tools used by developers and sysadmins worldwide. A recent survey by Postman found cURL usage at 48% among developers.

cURL is actually both a library and a command-line tool. The library implements the various protocols, while the CLI provides an easy interface. Beyond HTTP, cURL supports FTP, SFTP, SCP, SMTP, POP3, IMAP, LDAP, MQTT, and more. It‘s like a Swiss Army knife for dealing with anything on the web.

Now let‘s see why cURL remains essential even when newer options exist.

Why Use cURL for Sending GET Requests?

While languages like Python and JavaScript make sending requests easy with libraries like Requests and Fetch, cURL has some unique advantages:

It‘s Ubiquitous – cURL comes pre-installed on virtually every Linux and macOS machine. Saves installing other dependencies.

Human Readable – Unlike Postman and GUI clients, the command-line syntax shows exactly what‘s happening under the hood.

Easily Scriptable – cURL can be easily automated through shell scripts and called from various languages like Python.

Performance – cURL handles threading, caching, keep-alive connections for high-performance data transfers.

Fine-grained Control – You get granular control over request parameters and headers versus higher-level libraries.

According to the TIOBE index, usage of cURL has increased by 174% over the past two years. It remains essential for API testing, web scraping, and automations.

Next let‘s go over cURL basics…

Sending a Simple GET Request with cURL

The syntax for sending a basic GET request with cURL is:

curl [options] [URL]

For example, to retrieve a web page:

curl https://www.example.com

This will print the full HTML source of example.com to stdout. You can save it to a file using the -o flag covered later.

Let‘s break down what happens:

1. cURL constructs a valid HTTP GET request for the URL you passed
2. It connects to the remote server and sends the HTTP request
3. The server processes the request and sends back the response
4. cURL receives the response and prints the body to your terminal

It‘s essentially automating what your browser does behind the scenes!

You can visit any API endpoint or website URL using curl:

# Weather API example
curl https://api.weather.com/data/current

# Social media API 
curl https://api.twitter.com/trends

# Calling a static page
curl https://www.example.com/about.html

Now let‘s go over some common cURL options to customize GET requests.

cURL Options for GET Requests

cURL offers over 200 command-line options to control various aspects of the requests like headers, authentication, timeouts, and much more.

Here are some commonly used options for GET requests:

View HTTP Response Headers (-i)

Add the -i or --include flag to view response headers along with the response body:

curl -i https://api.data.com/v1/data

This will print details like HTTP status, content-type, server software, and more.

Follow Redirects (-L)

To follow HTTP redirects returned by the server:

curl -L https://example.com

By default, cURL does not follow redirects (30x HTTP status codes). -L enables this.

Set Request Headers (-H)

You can set custom headers with the -H flag. For example, to get JSON data from an API:

curl -H "Accept: application/json" https://api.data.com/v1/data

URL Parameters (-G)

To pass URL parameters along with a GET request:

curl -G -d "param1=value1" -d "param2=value2" https://api.data.com/v1/data

The -G flag appends the params to the URL query string.

Save Response to File (-o)

To save the response body to a file instead of printing to stdout:

curl -o file.txt https://api.data.com/v1/data

This writes the response to file.txt.

Below is a reference of common arguments for GET requests:

Argument	Description	Example
-i / –include	View response headers	curl -i https://api.data.com
-L / –location	Follow redirects	curl -L https://example.com
-H / –header	Set request header	curl -H "Accept: application/json" https://api.data.com
-G / –get	Send URL parameters	curl -G -d "param1=value1" https://api.data.com
-o / –output	Save response to file	curl -o data.json https://api.data.com

Now let‘s go over some practical examples…

Practical Examples of Sending GET Requests

To help demonstrate how to use cURL for common GET request tasks, here are some scenarios with examples:

Fetch an HTML Page and Scrape Data

Let‘s say you want to scrape product data from an ecommerce site. You can use cURL to fetch the page HTML:

curl -L -o products.html https://ecommerce-site.com/products

We use -L to follow any redirects, and -o to save the HTML to a file we can parse later.

No need for an API when you can extract data right from the HTML!

Retrieve JSON Data from a REST API

To fetch JSON data from a REST API endpoint:

curl -i -H "Accept: application/json" https://api.data.com/v1/data

The -H header signals we want JSON instead of the default XML format.

-i allows us to verify we get a 200 OK status and content-type: application/json in the response.

Check an API‘s Documentation

When working with a new API, you can use cURL to directly try out the endpoints and check the docs:

curl https://api.data.com/docs

See what formats are supported, required headers, authentication methods, rate limits, etc.

Pass URL Parameters to an API

To pass URL params to an API, use the -G option:

curl -G -d "param1=value1" -d "param2=value2" https://api.data.com/v1/data

The parameters get appended to the URL, like:

https://api.data.com/v1/data?param1=value1¶m2=value2

This allows you to test the API calls with different query inputs.

Automate Data Extraction with Shell Scripts

You can easily automate cURL requests using Bash shell scripts.

For example, extract data hourly:

#!/bin/bash

while true; do

  # Call API and save response
  curl -o data.json https://api.data.com/v1/data

  # Do something with the data

  sleep 1 hour

done

As you can see, there are many possibilities with cURL for accessing web APIs, websites, and automating extractions.

Next let‘s go over some common issues and how to resolve them.

Troubleshooting Common cURL GET Request Problems

When working with cURL, you may encounter errors like connection issues, authentication problems, 404 status codes, SSL issues, etc.

Here are some common problems and solutions:

404 Not Found Error – This means the URL you requested doesn‘t exist on the server. Double check the endpoint documentation for the correct path.

Timeout – Use the --timeout option if requests are timing out. This will adjust the timeout duration in seconds.

SSL Certificate Issue – Specify -k to ignore invalid SSL certificates, or use -v to debug the exact certificate issue.

Compression Error – If getting a compression error, use --compressed to request compressed data from the server.

Access Denied – If authentication is required, provide username and password with -u myusername:password.

Connection Refused – Ensure the server is accessible. Check your network connection and firewall settings.

Too Many Redirects – Use --max-redirs 10 (or appropriate number) to prevent infinite redirect loops.

Be sure to use -v for verbose output if you run into any issues. The detailed logs can help pinpoint the exact problem.

An Expert‘s Final Takeaways

After working with APIs and web scraping for many years, here are my key learnings when it comes to cURL:

• Start exploring APIs in documentation mode using basic curl calls. Don‘t dive into code first.
• Utilize the -v debug option alongside other flags to inspect the full request and response.
• Read through server response headers carefully to look for clues when issues occur.
• Double check the URL path, HTTP method, headers, and parameters when requests fail.
• Prefer -o for saving response content to a file you can process separately.
• Use Bash scripts to easily automate and orchestrate data extraction tasks.
• Learn to interpret error codes – don‘t rely only on the default cURL error text.

I hope these tips help you become a cURL pro! To take your skills to the next level, I recommend checking out my ebook Advanced cURL Techniques on Amazon.

Now go send some cURL GET requests!