Queues and Background Tasks in Go: Goroutines, Channels, and Advanced Patterns

Why Go is the Ideal Language for Queues and Background Tasks?

If you've ever written a backend in Python or Node.js, you've likely encountered this problem: the application is processing a request, but at the same time it needs to send an email, generate a report, or process an uploaded file. If you do this synchronously — the user waits. If you do it in a separate thread — you start dancing with locks and race conditions.

Go solves this problem elegantly and natively. Goroutines and channels are built-in concurrency mechanisms that make developing queues and background tasks in Go simple and safe. In this article, we'll explore how to organize background processing using both the standard library and popular tools like RabbitMQ. You'll see real code examples that you can immediately apply in your project.

Basics: Goroutines and Channels

Before diving into queues, let's review the basics. A goroutine is a lightweight thread of execution launched with the go keyword. A channel is a typed "pipeline" through which goroutines communicate with each other.

Simple Example: Asynchronous Email Sending

package main\n\nimport (\n    "fmt"\n    "time"\n)\n\nfunc sendEmail(email string) {\n    time.Sleep(2 * time.Second) // simulating a long operation\n    fmt.Println("Email sent to", email)\n}\n\nfunc main() {\n    fmt.Println("Start")\n    go sendEmail("user@example.com") // background launch\n    fmt.Println("Continue processing...")\n    time.Sleep(3 * time.Second) // wait for the goroutine to finish\n}

Here, sendEmail runs in the background, and the main thread is not blocked. But what if you need to process 1000 emails? Launching 1000 goroutines is not a good idea — they will start competing for resources. This is where channels and the Worker Pool pattern come to the rescue.

Worker Pool Pattern in Go

Worker Pool is a classic approach for organizing queues and background tasks in Go. You have a fixed number of workers (goroutines) that listen to a single task channel. Tasks come from the main thread or from an HTTP handler.

Worker Pool Implementation

package main\n\nimport (\n    "fmt"\n    "sync"\n    "time"\n)\n\ntype Task struct {\n    ID   int\n    Data string\n}\n\nfunc worker(id int, tasks <-chan Task, wg *sync.WaitGroup) {\n    defer wg.Done()\n    for task := range tasks {\n        fmt.Printf("Worker %d processing task %d: %s\\n", id, task.ID, task.Data)\n        time.Sleep(1 * time.Second) // simulating work\n    }\n}\n\nfunc main() {\n    const numWorkers = 3\n    tasks := make(chan Task, 10) // buffered channel\n\n    var wg sync.WaitGroup\n\n    // Start workers\n    for i := 1; i <= numWorkers; i++ {\n        wg.Add(1)\n        go worker(i, tasks, &wg)\n    }\n\n    // Send tasks\n    for i := 1; i <= 10; i++ {\n        tasks <- Task{ID: i, Data: fmt.Sprintf("Task %d", i)}\n    }\n    close(tasks) // close the channel so workers can finish\n\n    wg.Wait()\n    fmt.Println("All tasks completed")\n}

This code creates 3 workers that process 10 tasks. The buffered channel allows sending tasks without blocking until the buffer is full. This pattern is perfect for background tasks in Golang in web applications: you receive a request, put a task into the channel, and immediately respond to the user.

Queues in Go with RabbitMQ

For distributed systems, in-memory channels are not enough — you need an external queue storage. RabbitMQ is one of the most popular message brokers. Let's look at how to connect it to a Go application.

Installation and Connection

First, install the amqp library:

go get github.com/rabbitmq/amqp091-go

Producer (Sending Tasks)