Task Queues in Go: A Complete Guide for Beginners

Introduction to Task Queues in Go

Task queues are one of the key patterns when developing high-load applications. They allow you to organize asynchronous data processing, distribute load, and increase system fault tolerance. In the Go language, thanks to built-in concurrency mechanisms — goroutines and channels — implementing queues becomes incredibly elegant and performant.

In this article, we will break down how to create task queues in Go from scratch. You will learn what goroutines and channels are, how to organize a worker pool, and how to process tasks in parallel. The material is designed for beginner developers familiar with the basics of Go syntax.

Basics: Goroutines and Channels

Before moving on to queues, it is important to understand two fundamental Go concepts:

\n
• Goroutine — a lightweight thread of execution. It is launched with the keyword go before a function call.
\n
• Channel — a typed "conduit" for passing data between goroutines. It allows synchronizing their work.
\n

Example of a simple goroutine and channel:

package main\n\nimport "fmt"\n\nfunc main() {\n    ch := make(chan string)\n\n    go func() {\n        ch <- "Hello from goroutine!"\n    }()\n\n    msg := <-ch\n    fmt.Println(msg)\n}\n

In this example, we created a channel ch, launched an anonymous goroutine that sends a message to it, and the main goroutine received that message. Task queues are built on this exact mechanism.

Implementing a Simple Task Queue

The simplest task queue is a channel into which we place tasks, and one or more goroutines that process them. Let's look at a basic example:

package main\n\nimport (\n    "fmt"\n    "time"\n)\n\nfunc worker(id int, tasks <-chan int) {\n    for task := range tasks {\n        fmt.Printf("Worker %d started task %d\\n", id, task)\n        time.Sleep(time.Second) // simulating work\n        fmt.Printf("Worker %d completed task %d\\n", id, task)\n    }\n}\n\nfunc main() {\n    tasks := make(chan int, 10) // buffered channel\n\n    // launch one worker\n    go worker(1, tasks)\n\n    // send 5 tasks\n    for i := 1; i <= 5; i++ {\n        tasks <- i\n    }\n\n    close(tasks) // close the channel\n    time.Sleep(2 * time.Second) // give time for processing\n}\n

Here, worker is a function that reads from the tasks channel. The channel is buffered (capacity 10), so we can send several tasks without blocking. After closing the channel, the for range loop will terminate automatically.

Note: we used <-chan int — this is a receive-only channel. This is good practice, explicitly indicating that the worker should not send data to this channel.

Buffered and Unbuffered Channels

In the example above, we used a buffered channel (make(chan int, 10)). The difference between channel types:

\n
• Unbuffered channel — sending blocks until the receiver reads the data. Ideal for synchronization.
\n
• Buffered channel — sending blocks only when the buffer is full. Allows accumulating tasks and reduces latency.
\n

For task queues, buffered channels are typically used so that the producer can quickly send a task and continue working without waiting for its execution.

Worker Pool in Go

One worker is good, but several are better. A worker pool allows processing tasks in parallel, utilizing all CPU cores. Here's how to implement it:

package main\n\nimport (\n    "fmt"\n    "sync"\n    "time"\n)\n\nfunc worker(id int, tasks <-chan int, wg *sync.WaitGroup) {\n    defer wg.Done()\n    fo