Working with Task Queues in JavaScript in Practice with Node.js

Introduction: Why Do You Need Task Queues in JavaScript?

In modern web development with Node.js, tasks often arise that cannot be completed instantly within a single HTTP request. Sending email newsletters, generating PDF reports, processing images, synchronizing with external APIs — all these operations require time and resources. If you execute them synchronously, the user will wait too long for a response, and the server may become overloaded.

Job Queues solve this problem. They allow you to defer heavy operations, distribute the load, guarantee reprocessing in case of failures, and scale the system horizontally. In this article, we will cover the practical aspects of working with queues in JavaScript (Node.js): from theory to writing working code using popular libraries.

Basic Concepts of Task Queues

Before moving to the code, it is important to understand the key elements of any task queue:

\n
• Job — a unit of work that needs to be performed. It usually contains data (payload) and meta-information (priority, attempts, status).
\n
• Queue — a data structure (often based on Redis) that stores tasks in the order they arrive.
\n
• Worker — a process or thread that picks up tasks from the queue and executes them.
\n
• Producer — the part of the system that adds tasks to the queue.
\n
• Consumer — a synonym for a worker that processes tasks.
\n

In Node.js, the most popular libraries for working with queues are Bull and Kue. Both use Redis as a message broker. Bull is more modern and performant, while Kue is simpler to start with. In this article, we will focus on Bull as the recommended solution for production.

Practical Implementation of a Queue Using Bull

Installation and Setup

First, install Redis (locally or via Docker) and add the library to your project:

npm install bull

Basic queue initialization looks like this:

const Queue = require('bull');\nconst imageQueue = new Queue('image processing', {\n  redis: { host: 'localhost', port: 6379 }\n});

Adding Tasks (Producer)

The producer adds a task with data and options:

app.post('/upload', async (req, res) => {\n  const job = await imageQueue.add({\n    imageUrl: req.body.url,\n    userId: req.user.id\n  }, {\n    attempts: 3,        // number of retries on error\n    backoff: 5000,      // delay between retries (ms)\n    removeOnComplete: true // remove the task after success\n  });\n  res.json({ jobId: job.id });\n});

Processing Tasks (Worker)

The worker is a separate process (or the same process, but asynchronous) that listens to the queue:

imageQueue.process(async (job, done) => {\n  try {\n    const { imageUrl, userId } = job.data;\n    console.log(`Processing image ${imageUrl} for user ${userId}`);\n    // Your logic here: download, resize, save\n    await processImage(imageUrl);\n    done(); // signal successful completion\n  } catch (error) {\n    done(new Error('Processing error: ' + error.message));\n  }\n});

Error Handling and Retries

Bull automatically retries a task if the worker completes with an error. You can configure the retry strategy:

const job = await imageQueue.add(data, {\n  attempts: 5,\n  backoff: { type: 'exponential', delay: 2000 }\n});

You can also listen to queue events for monitoring:

imageQueue.on('completed', (job, result) => {\n  console.log(`Task ${job.id} completed`);\n});\nimageQueue.on('failed', (job, err) => {\n  console.error(`Task ${job.id} failed: ${err.mes