1. What Is Colly and Why Use It?

Colly is a powerful, high-performance web scraping library for the Go programming language. It provides a clean, asynchronous, and easily extensible interface for extracting data from websites. Built on top of the net/http and goquery packages (Go's jQuery equivalent), Colly lets developers use familiar CSS selector syntax to navigate HTML documents.

Why use this library? In today's world, data is the new oil. Web scraping automates data collection: competitor price monitoring, news aggregation, SEO metric analysis, product database creation, and much more. Colly stands out from other solutions thanks to its speed (asynchronous architecture using goroutines), built-in caching support, automatic delay management (to respect site policies), and a flexible callback system.

The library is ideal for both small scripts and large-scale distributed data collection systems. Thanks to Go's strong typing and lack of runtime overhead (unlike Python/Node.js), Colly applications compile into a single binary, simplifying deployment and scaling.

2. Installation

Install Colly using Go's standard package manager. Make sure you have Go version 1.18 or higher installed.

go get -u github.com/gocolly/colly/v2

After running the command, the library will be added to your go.mod file. It's also recommended to install goquery for HTML parsing (it will be pulled in automatically as a dependency):

go get -u github.com/PuerkitoBio/goquery

3. Quick Start: Minimal Example

Let's create a simple scraper that prints the <h1> tags from the homepage of example.com.

package main

import (
    "fmt"
    "github.com/gocolly/colly/v2"
)

func main() {
    // Create a new collector
    c := colly.NewCollector()

    // Callback for visited pages
    c.OnHTML("h1", func(e *colly.HTMLElement) {
        fmt.Println("Title:", e.Text)
    })

    // Error handler
    c.OnError(func(r *colly.Response, err error) {
        fmt.Println("Error:", err)
    })

    // Start scraping
    c.Visit("https://example.com")
}

Run the code with go run main.go. You'll see the page title. This is the minimal foundation to build upon.

4. Key Methods and Functions (Minimum 8)

Let's explore the core Colly methods that cover 90% of web scraping tasks.

4.1 colly.NewCollector()

Signature: func NewCollector(options ...func(*Collector)) *Collector

What it does: Creates a new collector instance—the main object for managing scraping. Accepts optional configuration functions.

// Creating with options
c := colly.NewCollector(
    colly.AllowedDomains("example.com"), // Only allow example.com
    colly.CacheDir("./cache"),           // Cache requests
)

4.2 Collector.Visit()

Signature: func (c *Collector) Visit(URL string) error

What it does: Initiates an HTTP GET request to the specified URL. This is the entry point for scraping.

err := c.Visit("https://example.com")
if err != nil {
    log.Fatal(err)
}

4.3 Collector.OnHTML()

Signature: func (c *Collector) OnHTML(selector string, f func(*HTMLElement))

What it does: Registers a callback that fires when an HTML element matching the CSS selector is found. The most commonly used method for data extraction.

c.OnHTML("div.product", func(e *colly.HTMLElement) {
    name := e.ChildText("h2")
    price := e.ChildText("span.price")
    fmt.Printf("Product: %s, Price: %s\n", name, price)
})

4.4 Collector.OnRequest()

Signature: func (c *Collector) OnRequest(f func(*Request))

What it does: Registers a callback that fires before each request. Useful for setting headers, logging, or modifying request parameters.

c.OnRequest(func(r *colly.Request) {
    r.Headers.Set("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64)")
    fmt.Println("Visiting:", r.URL)
})

4.5 Collector.OnResponse()

Signature: func (c *Collector) OnResponse(f func(*Response))

What it does: Registers a callback that fires after receiving a response. Useful for inspecting response headers, status codes, or raw body content.

c.OnResponse(func(r *colly.Response) {
    fmt.Println("Status:", r.StatusCode)
    fmt.Println("Content length:", len(r.Body))
})

4.6 Collector.OnError()

Signature: func (c *Collector) OnError(f func(*Response, error))

What it does: Registers a callback that fires when an error occurs during a request. Essential for robust error handling.

c.OnError(func(r *colly.Response, err error) {
    fmt.Println("Request failed:", err)
    if r != nil {
        fmt.Println("Status code:", r.StatusCode)
    }
})

4.7 Collector.OnScraped()

Signature: func (c *Collector) OnScraped(f func(*Response))

What it does: Registers a callback that fires after all callbacks for a page have been executed. Perfect for cleanup, saving data, or triggering next steps.

c.OnScraped(func(r *colly.Response) {
    fmt.Println("Finished scraping:", r.Request.URL)
    // Save data to database or file here
})

4.8 Collector.Limit()

Signature: func (c *Collector) Limit(limit *LimitRule) error

What it does: Applies rate limiting rules to avoid overwhelming target servers. Critical for ethical scraping and avoiding IP bans.

err := c.Limit(&colly.LimitRule{
    DomainGlob:  "*",
    Delay:       1 * time.Second, // 1 second delay between requests
    Parallelism: 2,               // Max 2 concurrent requests
})

4.9 Collector.Clone()

Signature: func (c *Collector) Clone() *Collector

What it does: Creates a deep copy of the collector with all its settings and callbacks. Useful for creating specialized collectors from a base configuration.

baseCollector := colly.NewCollector(colly.AllowedDomains("example.com"))
productCollector := baseCollector.Clone()
productCollector.OnHTML("div.product", productHandler)

4.10 HTMLElement.ChildText()

Signature: func (h *HTMLElement) ChildText(selector string) string

What it does: Extracts the text content of the first child element matching the CSS selector. A convenient shorthand for nested data extraction.

c.OnHTML("div.item", func(e *colly.HTMLElement) {
    title := e.ChildText("h3.title")
    description := e.ChildText("p.desc")
    fmt.Printf("%s: %s\n", title, description)
})

5. Advanced Features

Colly offers several advanced features for complex scraping scenarios:

• Automatic link following: Use c.LinkRevisit = false to avoid revisiting already scraped URLs.
• Cookie and session management: Colly automatically handles cookies across requests.
• Proxy support: Configure HTTP/HTTPS proxies for distributed scraping.
• Async scraping: Colly's goroutine-based architecture handles thousands of requests concurrently.
• Custom storage backends: Implement the Storage interface for custom caching solutions.

// Example: Async scraping with proxy
c := colly.NewCollector()
c.SetProxy("http://proxy.example.com:8080")
c.Async = true

// Start multiple requests concurrently
c.Visit("https://site1.com")
c.Visit("https://site2.com")
c.Visit("https://site3.com")

c.Wait() // Wait for all requests to complete

6. Best Practices and Tips

• Respect robots.txt: Use colly.AllowedDomains() and implement polite crawling delays.
• Use caching: Enable disk caching with colly.CacheDir() to avoid redundant requests during development.
• Handle pagination: Use OnHTML to find "next page" links and call Visit() recursively.
• Set proper User-Agent: Always identify your scraper with a realistic User-Agent header.
• Monitor rate limits: Use Limit() to stay within acceptable request rates.
• Error recovery: Implement retry logic with exponential backoff for transient failures.

// Example: Pagination handling
c.OnHTML("a.next", func(e *colly.HTMLElement) {
    nextPage := e.Attr("href")
    if nextPage != "" {
        e.Request.Visit(nextPage)
    }
})

7. Conclusion

Colly is a robust, production-ready web scraping framework for Go that combines performance with ease of use. Its callback-based architecture, built-in rate limiting, and caching make it suitable for projects of any scale. Whether you're building a simple data extraction script or a complex distributed scraping system, Colly provides the tools you need to get the job done efficiently.

Start with the minimal example, experiment with different callbacks, and gradually add advanced features as your requirements grow. The Go ecosystem, combined with Colly's capabilities, offers a powerful platform for all your web scraping needs.