How to download a file using cURL? | ScrapingBee

How to Download Files Using cURL: The Ultimate Guide

If you need a powerful and flexible tool for downloading files from the command line, look no further than cURL. Short for "Client URL", cURL is a versatile open source utility for transferring data using a variety of protocols.

While cURL can be used for uploading data and interacting with web APIs, one of its most useful features is the ability to easily download files from the web directly from your terminal. Whether you need to automate downloads, access files on headless servers, or just prefer working from the command line, cURL makes it simple to fetch files using HTTP, HTTPS, FTP and many other protocols.

In this guide, we‘ll walk through everything you need to know to become a cURL download expert. Let‘s get started!

Using cURL to Download a File

At its most basic, downloading a file with cURL is as simple as passing a URL to the cURL command. By default, cURL will fetch the contents of the URL and output it directly in your terminal.

For example, to download the example.com homepage, you would use:

curl https://example.com 

This will print out the raw HTML from example.com right in your command prompt. Not super useful for most file downloads. To actually save the contents to a file, you need to use the -o or –output option followed by the filename where you want to write the downloaded data.

So to save example.com to a file named example.html, the command would be:

curl -o example.html https://example.com

cURL will download the contents of the URL and save it to example.html in the current directory. You can specify a different directory path to save the file elsewhere.

That‘s the fundamental way to download files with cURL! But there are many more options and techniques to cover that can help you handle different situations and optimize your cURL downloads. Let‘s go through some key ones.

Following Redirects

One important thing to be aware of is that by default, cURL does not follow HTTP redirects. So if you try to download from a URL that redirects to another location, cURL will just output the contents of the redirect response, usually a small HTML document with a link to the new URL.

To have cURL automatically follow the redirect and download from the final URL, you need to pass the -L or –location option:

curl -L -o example.html https://example.com 

The -L flag will make cURL follow any redirects, while still saving the contents of the final URL to the output filename you specified. This is a good default to use for most downloads.

Downloading to a Specific Filename

We already saw how to save downloads to a file using the -o option. But sometimes the URL you are downloading from might contain special characters like & or spaces that can get misinterpreted by your shell.

To avoid issues, you can specify the output file separately using -o after your URL:

curl -o myfile.zip https://example.com/download?file=my+archive.zip

This lets you save the contents to a different filename than what the URL might suggest. It‘s a good way to make sure your output file has a valid name.

Resuming Downloads

If you‘re downloading a large file, you may run into a situation where the download gets interrupted before it can complete. Instead of starting from scratch, cURL provides an easy way to resume where you left off.

By passing the -C – or –continue-at – option, cURL will attempt to resume the previous download and continue where it got cut off:

curl -C - -o bigfile.zip https://example.com/bigfile.zip

Note the dash after -C, which tells cURL to auto-detect where the download should resume from. If you wanted to start from a specific byte offset, you could pass that instead of -.

This is super useful for dealing with spotty connections or large downloads that may get interrupted. Just re-run the same command with -C – and cURL will pick up where it stopped before.

Downloading in the Background

If you need to download a big file but don‘t want to sit around waiting for it to finish, you can have cURL run the download in the background. This way you can start the download then close your terminal or do other things while cURL handles the transfer behind the scenes.

To background a cURL download, just add an ampersand & at the end of your command:

curl -o bigfile.zip https://example.com/bigfile.zip &

cURL will still output progress meter and info to your terminal, but you can end the session and cURL will keep running in the background.

To avoid the output messing with your terminal, it‘s a good idea to redirect the output to a log file when backgrounding:

curl -o bigfile.zip https://example.com/bigfile.zip > download.log 2>&1 &  

This will save all the cURL output to download.log so you can check on the status later.

Scripting cURL Commands

So far we‘ve been running cURL commands directly in the shell. But if you want to combine multiple cURL transfers or integrate it with other CLI tools, you can write the cURL commands in a shell script.

For example, you could write a script called download-albums.sh that looks like:

#!/bin/bash

curl -L -o thriller.zip https://example.com/thriller.zip
curl -L -o off_the_wall.zip https://example.com/off_the_wall.zip  
curl -L -o bad.zip https://example.com/bad.zip

Then you can execute the entire script at once:

bash download-albums.sh

This will run the cURL commands sequentially in a single script. You could get fancier and introduce loops, variables, and other logic to do more advanced downloading workflows.

The key is that cURL is a command line tool that can be composed and combined just like any other Unix utility. Scripting allows you to automate complex use cases beyond one-off downloads.

Wrapping Up

As you can see, cURL provides an incredibly powerful Swiss Army knife for all kinds of file transfer needs. While we focused on downloading in this guide, cURL can do uploads, authentication, cookies, proxies and much more.

Mastering cURL is a key skill for any developer, sysadmin, or power user that works heavily in command line. We hope this guide gives you a solid foundation for using cURL to handle a variety of downloading scenarios.

Here are a few of the key techniques and options we covered:

• Use curl URL -o filename to save the contents of a URL to a specific file
• Pass the -L option to follow redirects
• Specify -C – to resume interrupted downloads
• Add & at the end to background long-running downloads
• Combine cURL commands in shell scripts for automation

With these fundamentals in hand, you‘ll be ready to take advantage of cURL‘s power for all your CLI downloading needs. It‘s an indispensable tool to have in your toolbox.