Task Queues in Go: Overview of Libraries and Tools

Task Queues for Go (Golang): Overview and Comparison of Tools

In modern web development and building high-load services in Go (Golang), there is almost always a need to execute tasks asynchronously, outside the main HTTP request processing thread. Sending email notifications, generating PDF reports, image processing, interacting with external APIs — all of this requires offloading to background processes. This is where task queues come to the rescue.

A task queue is a mechanism that allows you to defer the execution of an operation, guarantee its retry in case of failure, and balance the load across multiple workers. In the Go ecosystem, there are several mature and actively developed libraries for solving this problem. In this article, we will take a detailed look at three of the most popular approaches: Asynq, Machinery, and working directly with Redis Streams.

1. Asynq: Simplicity and Power on Top of Redis

Asynq by developer hibiken is arguably the most popular task queue library in Go at the moment. It is built on top of Redis and offers rich functionality out of the box.

Key Features of Asynq:

• Monitoring: Built-in web interface (Asynqmon) for tracking queues, task execution, and errors.
• Scheduling: Ability to delay task execution for a specific time (cron-like syntax).
• Priorities: Support for multiple queues with different priorities.
• Retry and Dead Letters: Automatic retries and a dead letter queue for unprocessable tasks.
• Middleware: Support for middleware for logging, metrics, etc.

Example of Using Asynq:

package main

import (    "context"    "log"    "time"    "github.com/hibiken/asynq")

// Define task typeconst TypeEmailDelivery = "email:deliver"

// Client for sending tasksfunc main() {    client := asynq.NewClient(asynq.RedisClientOpt{Addr: "localhost:6379"})    defer client.Close()

    // Create a task    task, err := asynq.NewTask(TypeEmailDelivery, []byte(`{"user_id": 42, "template": "welcome"}`))    if err != nil {        log.Fatal(err)    }

    // Enqueue the task with a 5-minute delay    info, err := client.Enqueue(task, asynq.ProcessIn(5*time.Minute))    if err != nil {        log.Fatal(err)    }    log.Printf("Task [%s] enqueued", info.ID)}

// Worker for processing tasksfunc handler(ctx context.Context, t *asynq.Task) error {    log.Printf("Processing task: %s", t.Type())    // Email sending logic here    return nil}

func startWorker() {    srv := asynq.NewServer(        asynq.RedisClientOpt{Addr: "localhost:6379"},        asynq.Config{Concurrency: 10},    )

    mux := asynq.NewServeMux()    mux.HandleFunc(TypeEmailDelivery, handler)

    if err := srv.Run(mux); err != nil {        log.Fatal(err)    }}

Asynq is ideal for most projects where Redis is already in use. It is easy to set up and provides an excellent developer UX.

2. Machinery: Distributed Tasks with Brokers

Machinery is an older but very powerful library, inspired by the Python library Celery. Unlike Asynq, Machinery is not tightly coupled to Redis and supports multiple message brokers: Redis, RabbitMQ, AWS SQS, and Memory.

Key Features of Machinery:

• Broker Flexibility: Easily switch between different message brokers.
• Task Results: Built-in support for saving and retrieving task execution results (e.g., in Redis).