If you work with APIs or do any kind of web development, there‘s a great tool you should have in your toolkit: cURL. cURL is a versatile command-line utility for transferring data using various protocols. It‘s especially handy for testing REST APIs by making HTTP requests like GET, POST, PUT and DELETE.
In this guide, we‘ll take an in-depth look at how to use cURL to send GET requests. We‘ll cover everything from basic syntax to advanced options, troubleshooting, and real-world examples. By the end, you‘ll be a cURL pro! Let‘s dive in.
What is cURL?
cURL (pronounced "curl" – think curling your hair) stands for "Client URL". It‘s an open source command line tool and library for transferring data with URLs. Originally developed in Sweden, cURL now ships standard with most operating systems.
With cURL, you can download or upload data using protocols like HTTP, HTTPS, FTP, FTPS, SFTP, and more. It supports a huge range of features from proxies and cookies to authentication and SSL connections. This makes it an indispensable tool for web development, API testing, and data transfer scripting.
Installing cURL
Before we get started, let‘s make sure you have cURL installed. Most MacOS, Linux, and Unix systems come with cURL pre-installed. Open a terminal and type curl --version. If you see version information, you‘re good to go!
On Windows, you‘ll need to install cURL. The easiest way is to download a pre-compiled binary from the official cURL site: https://curl.se/windows/. Unzip the package and add the directory to your system PATH variable so you can access cURL from any command prompt.
Sending a Basic GET Request
The simplest way to make a GET request with cURL is to just provide a URL:
curl https://api.example.com/usersThis will send an HTTP GET request to the specified URL and output the response body to your terminal. By default, cURL will use HTTP/1.1 and include an Accept: */* header indicating it will accept any content type in the response.
Here‘s what a real example looks like using the JSONPlaceholder API:
$ curl https://jsonplaceholder.typicode.com/posts/1
{
"userId": 1,
"id": 1,
"title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit",
"body": "quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto"
}The response body is a JSON object representing a blog post. cURL outputs it directly to the terminal.
Adding Query Parameters
Many APIs require parameters to be sent as part of the GET request URL. To include query parameters with cURL, just append them to the URL using the standard ?key=value syntax:
curl https://api.example.com/users?sort=name&order=ascYou can include multiple parameters by separating them with an &. If your parameter values contain special characters like spaces, you should URL encode them first.
Here‘s an example using the GitHub API to search for repositories:
$ curl https://api.github.com/search/repositories?q=cURL+language:php&sort=stars&order=desc
{
"total_count": 142,
"incomplete_results": false,
"items": [
{
"id": 1362490,
"node_id": "MDEwOlJlcG9zaXRvcnkxMzYyNDkw",
"name": "curl",
...
},
...
]
}This searches for PHP repositories mentioning "cURL", sorts by star count descending. The response is a JSON object with the search results.
Inspecting the Response
By default, cURL just dumps out the response body. But there‘s a lot more information in the response that can be useful for debugging and validation.
To see the response headers and status code, use the -i or --include flag:
$ curl -i https://jsonplaceholder.typicode.com/posts/1
HTTP/2 200
date: Fri, 19 May 2024 21:08:31 GMT
content-type: application/json; charset=utf-8
content-length: 292
x-powered-by: Express
varies: Origin, Accept-Encoding
access-control-allow-credentials: true
cache-control: max-age=43200
pragma: no-cache
expires: -1
x-content-type-options: nosniff
etag: W/"124-yiKdLzqO5gfBrJFrcdJ8Yq0LGnU"
via: 1.1 vegur
cf-cache-status: HIT
age: 6691
server-timing: cf-q-config;dur=5.9999992036635e-06
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=DCIJw0%2F2X8sTtSSpjTZ7lL1IKdZUhY%2FPL6d0WMJr2mNB%2BVu98YRmYPFj02S9nTP8idtFBzgAtSV%2B%2Bh1zVGsTqVtLiUuIU6NwAC9%2FQow%3D"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
server: cloudflare
cf-ray: 7c9db1dcdb04289c-IAD
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
{
"userId": 1,
"id": 1,
"title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit",
"body": "quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto"
}The -i flag includes the response headers and status line before the body. This shows the response was a 200 OK and includes details like the content type, cache headers, and more. Very handy for debugging!
You can also use -I or --head to send a HEAD request, which asks for just the response headers without the body:
$ curl -I https://jsonplaceholder.typicode.com/posts/1
HTTP/2 200
date: Fri, 19 May 2024 21:14:14 GMT
content-type: application/json; charset=utf-8
content-length: 292
x-powered-by: Express
varies: Origin, Accept-Encoding
access-control-allow-credentials: true
cache-control: max-age=43200
pragma: no-cache
expires: -1
x-content-type-options: nosniff
etag: W/"124-yiKdLzqO5gfBrJFrcdJ8Yq0LGnU"
via: 1.1 vegur
cf-cache-status: HIT
age: 12374
server-timing: cf-q-config;dur=6.0000002122251e-06
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=wKQPJxs9Ju3SM1bYZE4w8Jx2z7a8AMuMvPcBae3XSB7pUNm2H0zyzbmJx72l%2B1UcEjpTtcBs7JaQnfYEr%2FDmdfkDstAE%2FFIl3KvecFM%3D"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
server: cloudflare
cf-ray: 7c9db6bb0f3828f0-IAD
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400This just retrieves the headers, which can be useful to check things like content type, content length, caching, etc. without actually downloading the body.
Following Redirects
By default, if cURL receives a 3xx redirect status code (like 301 Moved Permanently or 302 Found), it will output the headers but won‘t actually follow the redirect to the new location. You‘ll just see the 3xx status and headers:
$ curl http://httpbingo.org/redirect-to?url=http://example.com
HTTP/2 302
date: Fri, 19 May 2024 21:16:10 GMT
content-type: text/html; charset=utf-8
content-length: 0
location: http://example.comTo make cURL automatically follow redirects to the final destination URL, use the -L or --location flag:
$ curl -L http://httpbingo.org/redirect-to?url=http://example.com
<!doctype html>
<html>
<head>
<title>Example Domain</title>
...
</head>
<body>
<div>
<p>This domain is for use in illustrative examples in documents. You may use this
domain in literature without prior coordination or asking for permission.</p>
<p><a href="https://www.iana.org/domains/example">More information...</a></p>
</div>
</body>
</html>With -L, cURL follows the redirect to example.com and outputs the HTML response body.
Setting Custom Headers
You can set custom request headers using the -H or --header flag. This is useful for things like authentication, content type negotiation, etc.
For example, to set a custom User-Agent header:
curl -H "User-Agent: MyCoolApp/1.0" https://api.example.com/usersTo set multiple headers, just use multiple -H flags:
curl -H "X-Auth-Token: abc123" -H "Content-Type: application/json" https://api.example.com/users Authentication
Many APIs require authentication to access protected resources. cURL supports a wide variety of auth mechanisms.
To send basic auth credentials (username/password), use the -u or --user flag:
curl -u "username:password" https://api.example.com/userscURL will automatically encode the credentials and send them in the Authorization header.
For bearer token auth, set the token in the Authorization header:
curl -H "Authorization: Bearer abc123" https://api.example.com/usersFor APIs that use custom auth headers, you can set those using -H as described above. cURL also has flags for more advanced schemes like digest auth, kerberos, client certs, and more. Check the man page for details.
Saving Output to a File
Instead of printing the response to your terminal, you can save it directly to a file using the -o or --output flag:
curl -o users.json https://api.example.com/usersThis will save the response body to users.json (overwriting if the file already exists). To have cURL automatically determine the filename from the URL, use -O:
curl -O https://api.example.com/avatars/john-smith.jpgThis will save the response to john-smith.jpg in the current directory.
Using a Proxy
If you need to tunnel your requests through an HTTP proxy, use the -x or --proxy flag:
curl -x http://proxy.example.com:3128 https://api.example.com/usersThis will send the request through the proxy at proxy.example.com on port 3128. You can also specify the proxy protocol – http, https, socks4, or socks5. For example:
curl -x socks5://127.0.0.1:9150 https://api.example.com/usersThis tunnels the request through a local SOCKS5 proxy on port 9150. Proxies are useful for security, compliance, performance, and getting around firewalls.
Troubleshooting
If you run into issues sending requests with cURL, here are a few tips:
-
Use
-vor--verboseto enable verbose output for debugging. This will show you the full headers sent and received, connection details, SSL info, and more. -
Check your URL for typos and make sure you‘re using the right protocol –
httpvshttpsetc. -
If you get SSL certificate errors, try using
-kor--insecureto ignore cert validation (warning: this is insecure and should only be used for testing). -
Make sure your auth credentials and tokens are valid and have the necessary permissions for the API you‘re accessing.
-
Check the API docs for the required headers, parameters, data formats, etc. Use
-Hto set headers and add query params to the URL. -
If the response is garbled, try setting an explicit
Acceptheader to specify the content type you want, e.g.-H "Accept: application/json" -
When sending POST or PUT requests with data, make sure you set the appropriate
Content-Typeheader and format the data correctly.
Examples
Here are a few more examples to illustrate practical uses of cURL:
Download a file and show download progress:
curl -O -# https://example.com/file.zipTest an API login:
curl -v -X POST -H "Content-Type: application/json" -d ‘{"username":"john","password":"secret"}‘ https://api.example.com/loginDownload Google‘s robots.txt file:
curl https://www.google.com/robots.txt Get the latest Hacker News headlines:
curl https://hacker-news.firebaseio.com/v0/topstories.jsonPost a message to Slack:
curl -X POST -H ‘Content-type: application/json‘ -d ‘{"text":"Hello from cURL!"}‘ https://hooks.slack.com/services/YOUR_WEBHOOK_URLConclusion
As you can see, cURL is a powerful tool for working with HTTP APIs and transferring data. With its flexible options and extensive protocol support, it can handle almost any task you throw at it. Mastering cURL is a valuable skill for developers, DevOps, and anyone who works with web services.
To learn more, dig into the man page (man curl) and check out the official cURL documentation at https://curl.se/docs/. You can also find lots of community guides, tutorials, and examples online.
Happy cURLing!
