Task Queues in Rust: Overview of Libraries for Asynchronous Processing

Task Queues for Rust: A Complete Overview of Tools and Libraries

Rust is a systems programming language gaining popularity in web development and building high-load microservices. One of the key requirements for modern backend applications is asynchronous processing of background tasks: sending emails, generating reports, processing uploaded files, or interacting with external APIs. In this article, we will look at how to organize a task queue in Rust, what libraries exist for this, and how to use them in practice.

Why Do You Need a Task Queue?

In a synchronous application, each operation is executed sequentially. If a user initiates a long-running task (e.g., video processing), the server will be blocked until it completes. A task queue solves this problem: tasks are placed into a broker (queue), and special workers asynchronously pick them up and execute them. This allows you to:

• Offload the main application thread;
• Scale processing horizontally (add workers);
• Guarantee task execution even in case of failures (thanks to acknowledgments and retries).

Popular Libraries for Organizing Queues in Rust

The Rust ecosystem offers several mature solutions. Let's look at three main approaches.

1. Celery-rs (Rust Celery)

Celery-rs is a Rust implementation of the popular Python Celery. The library allows using Redis or RabbitMQ brokers. It is ideal if you are migrating from Python to Rust or want compatibility with an existing Celery infrastructure.

Example of a basic worker:

use celery::Celery;use celery::task::Task;

#[celery::task]fn add(x: i32, y: i32) -> i32 {    x + y}

#[tokio::main]async fn main() -> Result<(), Box<dyn std::error::Error>> {    let celery = Celery::builder("redis://localhost:6379/0")        .with_task(add)        .build().await?;

    // Start the worker    celery.run().await?;

    // Send a task    let task_id = add.delay(&celery, 1, 2).await?;    println!("Task ID: {}", task_id);

    Ok(())}

Pros: familiar API for Python developers, support for complex scenarios (retry, ETA, chord tasks).
Cons: dependency on Redis/RabbitMQ, somewhat heavy for simple cases.

2. Lapin + Tokio (direct work with RabbitMQ)

Lapin is a high-performance AMQP client for Rust, running on Tokio. If you need full control over queues, exchanges, and routing keys, Lapin is the best choice. It does not provide task abstractions "out of the box," but offers flexibility.

Example of sending and receiving a message:

use lapin::{options::*, types::FieldTable, Connection, ConnectionProperties};use tokio_amqp::*;

#[tokio::main]async fn main() -> Result<(), Box<dyn std::error::Error>> {    let conn = Connection::connect(        "amqp://guest:guest@localhost:5672",        ConnectionProperties::default().with_tokio(),    ).await?;

    let channel = conn.create_channel().await?;    let queue = channel.queue_declare(        "task_queue",        QueueDeclareOptions::default(),        FieldTable::default(),    ).await?;

    // Send a task    let payload = b"Hello, task!";    channel.basic_publish(        "",        "task_queue",        BasicPublishOptions::default(),        payload,        BasicProperties::default(),    ).await?;

    // Receive a task (worker)    let consumer = channel.basic_consume(        "task_queue",        "consumer",        BasicConsumeOptions::default(),        FieldTable::default(),    ).await?;

    while let Some(delivery) = consumer.next().await {        let delivery = delivery?;        println!("Received task: {:?}", delivery.data);        delivery.a