Task Queues in Go: Overview of async, gocelery, and machinery Libraries

Task Queues for Go (Golang): Overview and Practice

In modern web applications and microservices, there is often a need to perform resource-intensive or long-running operations asynchronously: sending emails, generating reports, processing images, interacting with external APIs. Task queues are used for these purposes. In the Go (Golang) ecosystem, there are several mature libraries that allow implementing such an architecture. In this article, we will look at three of the most popular tools: machinery, gocelery, and asynq, analyze their features, and provide code examples.

Why do you need a task queue in Go?

A task queue solves several key problems:

• Separation of concerns: The HTTP handler does not wait for the background task to complete.
• Scalability: You can run multiple workers on different servers.
• Reliability: If a worker fails, the task is not lost and can be reprocessed.
• Scheduling: The ability to delay task execution for a specific time.

1. Machinery — A Powerful and Flexible Library

Machinery is an asynchronous task queue for Go, written from scratch. It supports multiple brokers (Redis, RabbitMQ, AWS SQS) and result backends (Redis, MongoDB, Memcache). Machinery provides a rich API for creating tasks, chains, groups, and recurring tasks.

Installation and Example

go get github.com/RichardKnox/machinery/v2

Example of a simple worker and client:

package main

import (    "github.com/RichardKnox/machinery/v2"    "github.com/RichardKnox/machinery/v2/config"    "github.com/RichardKnox/machinery/v2/tasks")

func main() {    // Configuration with Redis broker    cnf := &config.Config{        Broker:        "redis://localhost:6379",        DefaultQueue:  "machinery_tasks",        ResultBackend: "redis://localhost:6379",    }

    server, err := machinery.NewServer(cnf)    if err != nil {        panic(err)    }

    // Registering a task    err = server.RegisterTasks(map[string]interface{}{        "add": func(a, b int) (int, error) {            return a + b, nil        },    })    if err != nil {        panic(err)    }

    // Starting the worker    worker := server.NewWorker("worker1", 2) // 2 concurrent tasks    go worker.Launch()

    // Sending a task    task := tasks.Signature{        Name: "add",        Args: []tasks.Arg{            {Type: "int", Value: 5},            {Type: "int", Value: 7},        },    }    result, _ := server.SendTask(&task)    // Waiting for the result    if res, err := result.Get(10); err == nil {        println("Result:", res.(int)) // Output: 12    }}

Pros: Support for complex task graphs, retry capability, monitoring via web interface.

Cons: Relatively complex setup, documentation sometimes lags behind the code.

2. Asynq — A Modern Queue with a Rich UI

Asynq is a library from Google developers, based on Redis. It provides a convenient API, built-in scheduling, task priorities, and a built-in web interface for monitoring. Asynq is actively developed and has excellent documentation.

Installation and Example

go get github.com/hibiken/asynq

Example of a simple client and worker:

package main

import (    "context"    "log"    "github.com/hibiken/asynq")

// Typed taskconst TypeEmailDelivery = "email:deliver"

func main() {    // Client for sending tasks    client := asynq.NewClient(asynq.RedisClientOpt{Addr: "localhost:6379"})    defer client.Clos