Skip to content

How to Send a GET Request Using cURL: The Ultimate Guide

If you work with APIs or do any kind of web development, there‘s a great tool you should have in your toolkit: cURL. cURL is a versatile command-line utility for transferring data using various protocols. It‘s especially handy for testing REST APIs by making HTTP requests like GET, POST, PUT and DELETE.

In this guide, we‘ll take an in-depth look at how to use cURL to send GET requests. We‘ll cover everything from basic syntax to advanced options, troubleshooting, and real-world examples. By the end, you‘ll be a cURL pro! Let‘s dive in.

What is cURL?

cURL (pronounced "curl" – think curling your hair) stands for "Client URL". It‘s an open source command line tool and library for transferring data with URLs. Originally developed in Sweden, cURL now ships standard with most operating systems.

With cURL, you can download or upload data using protocols like HTTP, HTTPS, FTP, FTPS, SFTP, and more. It supports a huge range of features from proxies and cookies to authentication and SSL connections. This makes it an indispensable tool for web development, API testing, and data transfer scripting.

Installing cURL

Before we get started, let‘s make sure you have cURL installed. Most MacOS, Linux, and Unix systems come with cURL pre-installed. Open a terminal and type curl --version. If you see version information, you‘re good to go!

On Windows, you‘ll need to install cURL. The easiest way is to download a pre-compiled binary from the official cURL site: https://curl.se/windows/. Unzip the package and add the directory to your system PATH variable so you can access cURL from any command prompt.

Sending a Basic GET Request

The simplest way to make a GET request with cURL is to just provide a URL:

curl https://api.example.com/users

This will send an HTTP GET request to the specified URL and output the response body to your terminal. By default, cURL will use HTTP/1.1 and include an Accept: */* header indicating it will accept any content type in the response.

Here‘s what a real example looks like using the JSONPlaceholder API:

$ curl https://jsonplaceholder.typicode.com/posts/1

{
  "userId": 1,
  "id": 1,
  "title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit",
  "body": "quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto"
}

The response body is a JSON object representing a blog post. cURL outputs it directly to the terminal.

Adding Query Parameters

Many APIs require parameters to be sent as part of the GET request URL. To include query parameters with cURL, just append them to the URL using the standard ?key=value syntax:

curl https://api.example.com/users?sort=name&order=asc

You can include multiple parameters by separating them with an &. If your parameter values contain special characters like spaces, you should URL encode them first.

Here‘s an example using the GitHub API to search for repositories:

$ curl https://api.github.com/search/repositories?q=cURL+language:php&sort=stars&order=desc

{
  "total_count": 142,
  "incomplete_results": false,
  "items": [
    {
      "id": 1362490,
      "node_id": "MDEwOlJlcG9zaXRvcnkxMzYyNDkw",
      "name": "curl",
      ...
    },
    ...
  ]
}

This searches for PHP repositories mentioning "cURL", sorts by star count descending. The response is a JSON object with the search results.

Inspecting the Response

By default, cURL just dumps out the response body. But there‘s a lot more information in the response that can be useful for debugging and validation.

To see the response headers and status code, use the -i or --include flag:

$ curl -i https://jsonplaceholder.typicode.com/posts/1

HTTP/2 200 
date: Fri, 19 May 2023 21:08:31 GMT
content-type: application/json; charset=utf-8
content-length: 292
x-powered-by: Express
varies: Origin, Accept-Encoding
access-control-allow-credentials: true
cache-control: max-age=43200
pragma: no-cache
expires: -1
x-content-type-options: nosniff
etag: W/"124-yiKdLzqO5gfBrJFrcdJ8Yq0LGnU"
via: 1.1 vegur
cf-cache-status: HIT
age: 6691
server-timing: cf-q-config;dur=5.9999992036635e-06
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=DCIJw0%2F2X8sTtSSpjTZ7lL1IKdZUhY%2FPL6d0WMJr2mNB%2BVu98YRmYPFj02S9nTP8idtFBzgAtSV%2B%2Bh1zVGsTqVtLiUuIU6NwAC9%2FQow%3D"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
server: cloudflare
cf-ray: 7c9db1dcdb04289c-IAD
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400

{
  "userId": 1,
  "id": 1, 
  "title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit",
  "body": "quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto"
}

The -i flag includes the response headers and status line before the body. This shows the response was a 200 OK and includes details like the content type, cache headers, and more. Very handy for debugging!

You can also use -I or --head to send a HEAD request, which asks for just the response headers without the body:

$ curl -I https://jsonplaceholder.typicode.com/posts/1    

HTTP/2 200 
date: Fri, 19 May 2023 21:14:14 GMT
content-type: application/json; charset=utf-8
content-length: 292
x-powered-by: Express
varies: Origin, Accept-Encoding
access-control-allow-credentials: true
cache-control: max-age=43200
pragma: no-cache
expires: -1
x-content-type-options: nosniff
etag: W/"124-yiKdLzqO5gfBrJFrcdJ8Yq0LGnU"
via: 1.1 vegur
cf-cache-status: HIT
age: 12374
server-timing: cf-q-config;dur=6.0000002122251e-06
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=wKQPJxs9Ju3SM1bYZE4w8Jx2z7a8AMuMvPcBae3XSB7pUNm2H0zyzbmJx72l%2B1UcEjpTtcBs7JaQnfYEr%2FDmdfkDstAE%2FFIl3KvecFM%3D"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
server: cloudflare
cf-ray: 7c9db6bb0f3828f0-IAD
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400

This just retrieves the headers, which can be useful to check things like content type, content length, caching, etc. without actually downloading the body.

Following Redirects

By default, if cURL receives a 3xx redirect status code (like 301 Moved Permanently or 302 Found), it will output the headers but won‘t actually follow the redirect to the new location. You‘ll just see the 3xx status and headers:

$ curl http://httpbingo.org/redirect-to?url=http://example.com

HTTP/2 302
date: Fri, 19 May 2023 21:16:10 GMT 
content-type: text/html; charset=utf-8
content-length: 0
location: http://example.com

To make cURL automatically follow redirects to the final destination URL, use the -L or --location flag:

$ curl -L http://httpbingo.org/redirect-to?url=http://example.com

<!doctype html>
<html>
<head>
    <title>Example Domain</title>
    ...
</head>

<body>
<div>

    <p>This domain is for use in illustrative examples in documents. You may use this
    domain in literature without prior coordination or asking for permission.</p>
    <p><a href="https://www.iana.org/domains/example">More information...</a></p>
</div>
</body>
</html>

With -L, cURL follows the redirect to example.com and outputs the HTML response body.

Setting Custom Headers

You can set custom request headers using the -H or --header flag. This is useful for things like authentication, content type negotiation, etc.

For example, to set a custom User-Agent header:

curl -H "User-Agent: MyCoolApp/1.0" https://api.example.com/users

To set multiple headers, just use multiple -H flags:

curl -H "X-Auth-Token: abc123" -H "Content-Type: application/json" https://api.example.com/users

Authentication

Many APIs require authentication to access protected resources. cURL supports a wide variety of auth mechanisms.

To send basic auth credentials (username/password), use the -u or --user flag:

curl -u "username:password" https://api.example.com/users

cURL will automatically encode the credentials and send them in the Authorization header.

For bearer token auth, set the token in the Authorization header:

curl -H "Authorization: Bearer abc123" https://api.example.com/users

For APIs that use custom auth headers, you can set those using -H as described above. cURL also has flags for more advanced schemes like digest auth, kerberos, client certs, and more. Check the man page for details.

Saving Output to a File

Instead of printing the response to your terminal, you can save it directly to a file using the -o or --output flag:

curl -o users.json https://api.example.com/users

This will save the response body to users.json (overwriting if the file already exists). To have cURL automatically determine the filename from the URL, use -O:

curl -O https://api.example.com/avatars/john-smith.jpg

This will save the response to john-smith.jpg in the current directory.

Using a Proxy

If you need to tunnel your requests through an HTTP proxy, use the -x or --proxy flag:

curl -x http://proxy.example.com:3128 https://api.example.com/users

This will send the request through the proxy at proxy.example.com on port 3128. You can also specify the proxy protocol – http, https, socks4, or socks5. For example:

curl -x socks5://127.0.0.1:9150 https://api.example.com/users

This tunnels the request through a local SOCKS5 proxy on port 9150. Proxies are useful for security, compliance, performance, and getting around firewalls.

Troubleshooting

If you run into issues sending requests with cURL, here are a few tips:

  • Use -v or --verbose to enable verbose output for debugging. This will show you the full headers sent and received, connection details, SSL info, and more.

  • Check your URL for typos and make sure you‘re using the right protocol – http vs https etc.

  • If you get SSL certificate errors, try using -k or --insecure to ignore cert validation (warning: this is insecure and should only be used for testing).

  • Make sure your auth credentials and tokens are valid and have the necessary permissions for the API you‘re accessing.

  • Check the API docs for the required headers, parameters, data formats, etc. Use -H to set headers and add query params to the URL.

  • If the response is garbled, try setting an explicit Accept header to specify the content type you want, e.g. -H "Accept: application/json"

  • When sending POST or PUT requests with data, make sure you set the appropriate Content-Type header and format the data correctly.

Examples

Here are a few more examples to illustrate practical uses of cURL:

Download a file and show download progress:

curl -O -# https://example.com/file.zip

Test an API login:

curl -v -X POST -H "Content-Type: application/json" -d ‘{"username":"john","password":"secret"}‘ https://api.example.com/login

Download Google‘s robots.txt file:

curl https://www.google.com/robots.txt

Get the latest Hacker News headlines:

curl https://hacker-news.firebaseio.com/v0/topstories.json

Post a message to Slack:

curl -X POST -H ‘Content-type: application/json‘ -d ‘{"text":"Hello from cURL!"}‘ https://hooks.slack.com/services/YOUR_WEBHOOK_URL

Conclusion

As you can see, cURL is a powerful tool for working with HTTP APIs and transferring data. With its flexible options and extensive protocol support, it can handle almost any task you throw at it. Mastering cURL is a valuable skill for developers, DevOps, and anyone who works with web services.

To learn more, dig into the man page (man curl) and check out the official cURL documentation at https://curl.se/docs/. You can also find lots of community guides, tutorials, and examples online.

Happy cURLing!

Join the conversation

Your email address will not be published. Required fields are marked *

Related Posts

How to Use XPath Selectors for Web Scraping in Python

There are multiple ways for using XPath selectors in Python. One popular option is to use lxml and BeautifulSoup and pair it with requests. And the second option is to use Selenium.
Here is some sample code for using lxml, BeautifulSoup, and Requests for opening up the ScrapingBee homepage and extracting the text from h1 tag using XPath:
import requests from lxml import etree from bs4 import BeautifulSoup html = requests.

How to Select Elements by Text in XPath

You can select elements by text in XPath by using the contains(text(), "Text string") or text()="Text string" expression.
The first expression will match any element that contains the "Text string" sub-string. However, the second expression will match only those elements that contain this exact string.
Here is some sample code for using lxml, BeautifulSoup, and Requests for opening up the ScrapingBee homepage and extracting the text from h1 tag using the text-based XPath selector:

How to Select Elements by Class in XPath: The Ultimate Guide

You can select elements by class in XPath by using the contains(@class, "class-name") or @class="class-name" expressions.
The first expression will match any element that contains class-name. Even if the element has additional classes defined it will still match. However, the second expression will match the elements that only have one class named class-name and no additional classes.
Here is some Selenium XPath sample code that extracts the h1 tag from the ScrapingBee website using the class name: