Skip to content

Building a Robust Web Scraper in Golang: A Comprehensive Tutorial

Web scraping is growing exponentially as organizations race to extract and analyze data from the web. As the scale and sophistication of scraping projects grow, Python, the long-time go-to language for scraping, is starting to show limitations in areas like performance, concurrency, and scalability.

Golang has emerged as a modern alternative, especially well-suited for large-scale, high-performance scraping tasks. Companies like Cloudflare, Twitch, and Amazon all use Golang for mission-critical services, and it‘s gaining traction for web scraping too.

In this comprehensive tutorial, we’ll walk step-by-step through building a robust web scraper in Golang from the ground up, including:

  • Setting up the Go environment
  • Choosing a web scraping framework
  • Extracting data from HTML
  • Handling pagination
  • Debugging and optimization
  • Storing scraped data
  • Integrating proxies
  • Scheduling recurring scraping

By the end, you’ll have all the knowledge needed to leverage the power of Golang for your own web scraping projects, whether it’s a simple hobby scraper or a mission-critical commercial application.

Why Use Golang for Web Scraping?

Before we dive in, let’s highlight some of the key advantages of using Golang specifically for web scraping:

Performance

Golang was designed for speed and efficiency. Benchmarks typically show Go scraping faster than equivalent Python scripts:

Go vs Python Scraping Benchmark

This performance advantage widens as the scale and complexity of scrapes grow.

Concurrency

Golang has built-in concurrency through goroutines, making it easy to parallelize scraping tasks across multiple cores and threads without relying on external libraries.

Compilation

Go compiles down to standalone native binaries, unlike Python which relies on interpreted execution. This means faster execution speeds, simpler deployment, and cross-platform portability.

Scalability

Go utilizes a lightweight thread model, enabling high throughput scraping at scale compared to Python’s bulkier process-based model.

Simplicity

Go uses a simple, minimalist syntax without classes or inheritance like other OOP languages. This makes Go easy to learn for beginners while still providing the control needed for advanced use cases.

So in summary, if you need to scrape large amounts of data quickly and efficiently, especially in a production setting, Golang is likely a better choice than Python.

Now let‘s see how to build a scraper in Go from start to finish!

Installing Go

Since Go compiles down to standalone binaries, the first step is to install Go on your development machine based on your OS:

Windows

Download the MSI installer from golang.org and run the exe to install Go.

Alternatively, use the Chocolatey package manager:

choco install golang

MacOS

Download the PKG installer from golang.org, or install using Homebrew:

brew install go 

Linux

Download the tarball, then extract it to /usr/local:

tar -C /usr/local -xzf go1.19.2.linux-amd64.tar.gz  

Then add Go to your PATH:

export PATH=$PATH:/usr/local/go/bin

Check it worked:

go version # should print version 

With Go installed, let‘s look at setting up a development environment.

Setting Up Your Go Environment

Go developers use a wide range of text editors and IDEs for coding, including:

VS Code is a popular free editor with excellent Go support through the Go extension.

Install VS Code, then search for "Go" in the Extensions panel:

Installing Go extension in VS Code

Click Install. This will add code completion, debugging, linting, and other Go tools to VS Code.

Now we‘re ready to start coding!

Choosing a Golang Web Scraping Framework

While Go‘s standard library supports HTTP requests and HTML parsing, for professional web scraping, a battle-tested framework simplifies the process.

Some popular options include:

  • Colly – Full-featured scraping framework similar to Python‘s Scrapy
  • GoQuery – jQuery-style HTML parser and selector
  • Rod – Headless browser automation for dynamic sites
  • Ferret – Declarative web scraping

We‘ll use Colly for this tutorial since it provides a robust toolset for production scraping.

Let‘s create a new folder for our scraper, initialize Go mod, and install Colly:

mkdir tutorial && cd tutorial
go mod init scraper
go get github.com/gocolly/colly

This adds Colly to the go.mod file as a dependency.

Time to write our first scraper!

Scraping a Simple Page

Let‘s start by scraping book data from the demo site books.toscrape.com.

Create a file main.go and import Colly:

package main

import (
  "github.com/gocolly/colly" 
)

Next, we‘ll instantiate a Collector which handles making requests and traversing pages:

c := colly.NewCollector()

Collectors emit events we can listen to like OnRequest, OnResponse, and OnHTML:

c.OnRequest(func(r *colly.Request) {
  fmt.Println("Visiting", r.URL) 
})

c.OnResponse(func(r *colly.Response) {
  fmt.Println(r.StatusCode)
})

Let‘s visit the books page and extract titles:

c.OnHTML(".product_pod", func(e *colly.HTMLElement) {

  // Extract title 
  title := e.ChildText(".image_container img")

  fmt.Println(title)

})

c.Visit("https://books.toscrape.com/")

We use the .product_pod CSS selector to match each book container, then extract the <img> title.

Run it:

go run main.go

This prints all book titles!

Let‘s expand our scraper to extract more info.

Extracting Book Data

To extract additional data like price, we can create a Book struct:

type Book struct {
  Title string
  Price string
}

Then populate it from HTML:

c.OnHTML(".product_pod", func(e *colly.HTMLElement) {

  book := Book{}

  book.Title = e.ChildAttr(".image_container img", "alt")
  book.Price = e.ChildText(".price_color") 

})

We use CSS selectors like .image_container img to target elements.

Rather than just printing, let‘s save results to a CSV file:

file, _ := os.Create("books.csv")
writer := csv.NewWriter(file)
headers := []string{"Title", "Price"}

writer.Write(headers)

c.OnHTML(".product_pod", func(e *colly.HTMLElement) {

  //... extract book data

  row := []string{book.Title, book.Price}
  writer.Write(row)

})

Now when run, it saves all books to a CSV!

Handling Pagination

To scrape multiple pages, we first need to find the "Next" links on each page:

c.OnHTML(".next > a", func(e *colly.HTMLElement) {
  nextPage := e.Request.AbsoluteURL(e.Attr("href"))
  c.Visit(nextPage) 
})

This recursively follows pagination links to scrape all pages.

Debugging Common Issues

Some common issues when building scrapers include:

  • Unexpected HTML layouts
  • JavaScript rendering data
  • Bot protection barriers

Debugging techniques include:

Logging

Use fmt.Println or a logger like logrus to output info during a scrape for debugging.

Capturing Traffic

Inspect raw HTTP requests and responses using a tool like mitmproxy.

Testing Selectors

Use browser DevTools to test and refine CSS selectors when they don‘t match expected elements.

Handling JS Sites

For JavaScript rendering, use a headless browser like chromedp to evaluate pages.

Slowing Down

Add delays between requests using time.Sleep to avoid protections against too-fast scraping.

With some debugging know-how, you can overcome most obstacles.

Storing Scraped Data

Besides saving to CSV, Go provides many options for storage:

  • Save to databases like PostgreSQL, MySQL or MongoDB using their Go drivers.
  • Use an object storage service like S3 to store scraped data in the cloud. The minio library integrates with S3-compatible backends.
  • Index data in a search engine like Elasticsearch using the olivere/elastic package.
  • Cache scraped data in Redis for low latency lookups later.

The Go ecosystem provides clients for virtually any data store.

Integrating Proxies

To scale up scraping and prevent blocks, we can proxy requests through services like Oxylabs.

First, sign up for residential proxies. Then integrate them:

import (
  "github.com/oxylabs/oxy/proxy"
)

pool, _ := proxy.NewPoolFromURL("http://user:pass@proxy:port")

c.OnRequest(func(r *colly.Request) {
  proxy := pool.GetProxy()
  r.ProxyURL = p.URL()
})

This will rotate proxies randomly per request.

Specific proxy types like datacenter and residential provide different benefits:

  • Datacenter – Extremely fast throughput needed for large scraping volumes
  • Residential – Rotate thousands of real residential IPs great for sites sensitive to datacenter traffic

Features like backconnects, sticky sessions, and load balancing help optimize proxy performance.

Integrating smart proxying dramatically improves scale, resiliency and transparency.

Scheduling Recurring Scrapes

For ongoing scraping, we can scheduled jobs using cron or libraries like gocron:

import "github.com/go-co-op/gocron"

scheduler := gocron.NewScheduler(time.UTC)

scheduler.Every(1).Day().At("09:00").Do(scrapeBooks) 

scheduler.StartAsync() 

This runs scrapeBooks daily at 9am UTC.

We can also containerize the scraper as a Docker image and run it on a Kubernetes cluster alongside cron jobs.

Conclusion

In this comprehensive Golang web scraping tutorial, we covered:

  • Setting up Go and editors like VS Code
  • Using frameworks like Colly to extract data
  • Debugging common scraping issues
  • Storing results in databases, cloud storage and search engines
  • Integrating proxies for large-scale scraping
  • Scheduling recurring jobs for continuous scraping

The key advantages of Go for web scraping include its performance, concurrency, compile speeds, scalability, and simplicity. Together with mature scraping packages, it‘s ready to power everything from hobbyist scrapers to mission-critical commercial applications.

The full code for this tutorial is available on Github at github.com/colly-scraper.

We‘ve only scratched the surface of capabilities like headless browsers, distributed scraping, and advanced patterns. For production scraping, Oxylabs provides advanced support, infrastructure, and proxies for Golang and other languages.

Hopefully this provides a solid foundation for you to start leveraging Golang for your own robust web scraping projects! Let us know if you have any other questions.

Tags:

Join the conversation

Your email address will not be published. Required fields are marked *