Task Queues in JavaScript (Node.js): A Complete Guide

Working with Task Queues in JavaScript (Node.js)

In modern web applications and microservice architectures, there is often a need to execute background tasks that should not block the main execution thread. Task queues are a powerful tool that helps manage asynchronous operations, ensuring reliability, scalability, and fault tolerance. In this article, we will thoroughly examine how to work with task queues in the Node.js environment, review the main libraries, and write practical code examples.

What is a Task Queue and Why is it Needed?

A Task Queue is a data structure operating on the FIFO (First In, First Out) principle. Tasks are placed in the queue and processed by workers in the order they arrive. The main use cases include:

• Sending email notifications — no need to make the user wait while the email is being sent.
• Generating reports or PDFs — resource-intensive operations that are better performed asynchronously.
• Processing uploaded images — resizing, compression, adding watermarks.
• Integration with external APIs — executing requests with rate limiting.
• Scheduled tasks (cron) — regular database cleanup or cache updates.

Using queues allows you to offload the main server, increase fault tolerance (if one worker fails, the task will be transferred to another), and guarantee task execution even during temporary failures.

Simple In-Memory Queue in Node.js

To start, let's implement a simple queue without external dependencies. This approach is suitable for prototyping or very small projects, but is not recommended for production due to the lack of persistence.

class SimpleQueue {    constructor() {        this.queue = [];        this.processing = false;    }

    enqueue(task) {        this.queue.push(task);        this.processNext();    }

    async processNext() {        if (this.processing || this.queue.length === 0) return;        this.processing = true;        const task = this.queue.shift();        try {            await task();        } catch (err) {            console.error('Task execution error:', err);        } finally {            this.processing = false;            this.processNext();        }    }}

// Usage exampleconst queue = new SimpleQueue();

queue.enqueue(async () => {    console.log('Task 1: Sending email');});

queue.enqueue(async () => {    console.log('Task 2: Generating report');});

console.log('Tasks added to queue');

This code demonstrates the basic mechanics: tasks are executed sequentially, one after another. However, when the server restarts, all unprocessed tasks will be lost.

Bull — Professional Queue on Redis

For serious projects in Node.js, the de facto standard is the Bull library. It uses Redis as a backend, providing persistence, priority support, retries, and delayed tasks.

Installation:

npm install bull

Example of a simple producer and consumer:

const Queue = require('bull');

// Create a queue named 'email-queue'const emailQueue = new Queue('email-queue', {    redis: { host: 'localhost', port: 6379 }});

// Worker function: processes tasksemailQueue.process(async (job) => {    const { to, subject, body } = job.data;    console.log(`Sending email to ${to}: ${subject}`);    // Here would be the actual email sending code    await new Promise(resolve => setTimeout(resolve, 1000));    return { success: true };});

// Adding tasks