As someone who has used cURL daily for over 10 years for data extraction and web scraping, I can‘t recommend it enough for sending customizable GET requests. While modern languages like Python make HTTP requests easy, cURL remains a powerful tool every developer should have in their toolbox.
In this comprehensive 2200+ word guide, you‘ll learn:
- What exactly is cURL and why is it useful?
- How to send simple GET requests with cURL
- Essential options like viewing response headers and following redirects
- Retrieving data in JSON format
- Passing GET parameters and reading API docs
- Saving response content to a file
- A GET request argument cheat sheet
- Debugging tips and tricks I‘ve picked up over the years
- And much more…
By the end, you‘ll have the cURL skills to extract data from APIs and websites with ease. Let‘s get started!
A Brief History of cURL
cURL first appeared in 1997 created by Daniel Stenberg to transfer data over various protocols. According to Stenberg, the name originally stood for "Client URL Request Library".
Over the last 25 years, cURL has become one of the most popular command-line tools used by developers and sysadmins worldwide. A recent survey by Postman found cURL usage at 48% among developers.
cURL is actually both a library and a command-line tool. The library implements the various protocols, while the CLI provides an easy interface. Beyond HTTP, cURL supports FTP, SFTP, SCP, SMTP, POP3, IMAP, LDAP, MQTT, and more. It‘s like a Swiss Army knife for dealing with anything on the web.
Now let‘s see why cURL remains essential even when newer options exist.
Why Use cURL for Sending GET Requests?
It‘s Ubiquitous – cURL comes pre-installed on virtually every Linux and macOS machine. Saves installing other dependencies.
Human Readable – Unlike Postman and GUI clients, the command-line syntax shows exactly what‘s happening under the hood.
Easily Scriptable – cURL can be easily automated through shell scripts and called from various languages like Python.
Performance – cURL handles threading, caching, keep-alive connections for high-performance data transfers.
Fine-grained Control – You get granular control over request parameters and headers versus higher-level libraries.
According to the TIOBE index, usage of cURL has increased by 174% over the past two years. It remains essential for API testing, web scraping, and automations.
Next let‘s go over cURL basics…
Sending a Simple GET Request with cURL
The syntax for sending a basic GET request with cURL is:
curl [options] [URL]
For example, to retrieve a web page:
This will print the full HTML source of example.com to stdout. You can save it to a file using the
-o flag covered later.
Let‘s break down what happens:
- cURL constructs a valid HTTP GET request for the URL you passed
- It connects to the remote server and sends the HTTP request
- The server processes the request and sends back the response
- cURL receives the response and prints the body to your terminal
It‘s essentially automating what your browser does behind the scenes!
You can visit any API endpoint or website URL using curl:
# Weather API example curl https://api.weather.com/data/current # Social media API curl https://api.twitter.com/trends # Calling a static page curl https://www.example.com/about.html
Now let‘s go over some common cURL options to customize GET requests.
cURL Options for GET Requests
cURL offers over 200 command-line options to control various aspects of the requests like headers, authentication, timeouts, and much more.
Here are some commonly used options for GET requests:
View HTTP Response Headers (-i)
--include flag to view response headers along with the response body:
curl -i https://api.data.com/v1/data
This will print details like HTTP status, content-type, server software, and more.
Follow Redirects (-L)
To follow HTTP redirects returned by the server:
curl -L https://example.com
By default, cURL does not follow redirects (30x HTTP status codes).
-L enables this.
Set Request Headers (-H)
You can set custom headers with the
-H flag. For example, to get JSON data from an API:
curl -H "Accept: application/json" https://api.data.com/v1/data
URL Parameters (-G)
To pass URL parameters along with a GET request:
curl -G -d "param1=value1" -d "param2=value2" https://api.data.com/v1/data
-G flag appends the params to the URL query string.
Save Response to File (-o)
To save the response body to a file instead of printing to stdout:
curl -o file.txt https://api.data.com/v1/data
This writes the response to
Below is a reference of common arguments for GET requests:
|-i / –include||View response headers||curl -i https://api.data.com|
|-L / –location||Follow redirects||curl -L https://example.com|
|-H / –header||Set request header||curl -H "Accept: application/json" https://api.data.com|
|-G / –get||Send URL parameters||curl -G -d "param1=value1" https://api.data.com|
|-o / –output||Save response to file||curl -o data.json https://api.data.com|
Now let‘s go over some practical examples…
Practical Examples of Sending GET Requests
To help demonstrate how to use cURL for common GET request tasks, here are some scenarios with examples:
Fetch an HTML Page and Scrape Data
Let‘s say you want to scrape product data from an ecommerce site. You can use cURL to fetch the page HTML:
curl -L -o products.html https://ecommerce-site.com/products
-L to follow any redirects, and
-o to save the HTML to a file we can parse later.
No need for an API when you can extract data right from the HTML!
Retrieve JSON Data from a REST API
To fetch JSON data from a REST API endpoint:
curl -i -H "Accept: application/json" https://api.data.com/v1/data
-H header signals we want JSON instead of the default XML format.
-i allows us to verify we get a
200 OK status and
content-type: application/json in the response.
Check an API‘s Documentation
When working with a new API, you can use cURL to directly try out the endpoints and check the docs:
See what formats are supported, required headers, authentication methods, rate limits, etc.
Pass URL Parameters to an API
To pass URL params to an API, use the
curl -G -d "param1=value1" -d "param2=value2" https://api.data.com/v1/data
The parameters get appended to the URL, like:
This allows you to test the API calls with different query inputs.
Automate Data Extraction with Shell Scripts
You can easily automate cURL requests using Bash shell scripts.
For example, extract data hourly:
#!/bin/bash while true; do # Call API and save response curl -o data.json https://api.data.com/v1/data # Do something with the data sleep 1 hour done
As you can see, there are many possibilities with cURL for accessing web APIs, websites, and automating extractions.
Next let‘s go over some common issues and how to resolve them.
Troubleshooting Common cURL GET Request Problems
When working with cURL, you may encounter errors like connection issues, authentication problems, 404 status codes, SSL issues, etc.
Here are some common problems and solutions:
404 Not Found Error – This means the URL you requested doesn‘t exist on the server. Double check the endpoint documentation for the correct path.
Timeout – Use the
--timeout option if requests are timing out. This will adjust the timeout duration in seconds.
SSL Certificate Issue – Specify
-k to ignore invalid SSL certificates, or use
-v to debug the exact certificate issue.
Compression Error – If getting a compression error, use
--compressed to request compressed data from the server.
Access Denied – If authentication is required, provide username and password with
Connection Refused – Ensure the server is accessible. Check your network connection and firewall settings.
Too Many Redirects – Use
--max-redirs 10 (or appropriate number) to prevent infinite redirect loops.
Be sure to use
-v for verbose output if you run into any issues. The detailed logs can help pinpoint the exact problem.
An Expert‘s Final Takeaways
After working with APIs and web scraping for many years, here are my key learnings when it comes to cURL:
- Start exploring APIs in documentation mode using basic
curlcalls. Don‘t dive into code first.
- Utilize the
-vdebug option alongside other flags to inspect the full request and response.
- Read through server response headers carefully to look for clues when issues occur.
- Double check the URL path, HTTP method, headers, and parameters when requests fail.
-ofor saving response content to a file you can process separately.
- Use Bash scripts to easily automate and orchestrate data extraction tasks.
- Learn to interpret error codes – don‘t rely only on the default cURL error text.
I hope these tips help you become a cURL pro! To take your skills to the next level, I recommend checking out my ebook Advanced cURL Techniques on Amazon.
Now go send some cURL GET requests!