Working with Kafka in JavaScript (Node.js): node-rdkafka library

Apache Kafka and Node.js: Introduction to node-rdkafka

Apache Kafka is a distributed event streaming platform that has become the de facto standard for building reliable and scalable data exchange systems. In the Node.js ecosystem, one of the most performant and mature libraries for working with Kafka is node-rdkafka. It provides native bindings to the librdkafka library (written in C/C++), ensuring maximum speed and stability.

In this article, we will thoroughly cover how to set up and use node-rdkafka for sending (producer) and reading (consumer) messages. You will learn how to integrate Kafka into your Node.js application, handle errors, and manage data streams.

Installing node-rdkafka

To install the library, you need Node.js version 12 or higher and a C++ compiler (gcc, clang, or MSVC). Installation is done using the standard package manager:

npm install node-rdkafka

If the build fails with an error, make sure you have build-essential installed (Linux) or Python and build tools for Windows (npm install --global windows-build-tools).

Creating a Producer (Message Sender)

A Producer in Kafka is responsible for publishing messages to topics. Let's look at a basic example:

const Kafka = require('node-rdkafka');

// Create a producerconst producer = new Kafka.Producer({  'metadata.broker.list': 'localhost:9092',  'dr_cb': true  // Callback for delivery confirmation});

// Event handler on connectionproducer.on('event.log', function(event) {  console.log('Log:', event.message);});

// Error handlerproducer.on('event.error', function(err) {  console.error('Producer error:', err);});

// Connect to brokerproducer.connect();

// After connection, send messagesproducer.on('ready', function() {  try {    // Send message to topic 'test-topic'    producer.produce(      'test-topic',  // topic name      null,          // partition (null = automatic distribution)      Buffer.from('Hello, Kafka!'),  // message body (Buffer)      'my-key',      // message key (optional)      Date.now()     // timestamp (optional)    );    console.log('Message sent');  } catch (err) {    console.error('Error sending:', err);  }});

// Don't forget to call poll to process callbackssetInterval(() => {  producer.poll();}, 1000);

// Close connection on process terminationprocess.on('SIGINT', () => {  producer.disconnect();  process.exit();});

Note: the producer.produce() method accepts the message body as a Buffer. This allows passing both strings and binary data. The parameter dr_cb: true enables delivery report callbacks.

Creating a Consumer (Message Receiver)

A Consumer subscribes to one or more topics and processes incoming messages. For working with consumer groups, KafkaConsumer is used:

const Kafka = require('node-rdkafka');

const consumer = new Kafka.KafkaConsumer({  'group.id': 'my-group',  'metadata.broker.list': 'localhost:9092',  'enable.auto.commit': true}, {});

consumer.on('event.error', function(err) {  console.error('Consumer error:', err);});

consumer.on('ready', function() {  console.log('Consumer ready');  // Subscribe to topic  consumer.subscribe(['test-topic']);  // Start consumption loop  consumer.consume();});

// Process each messageconsumer.on('data', function(data) {  console.log('Message received:');  console.log('  Topic:', data.topic);  console.log('  Partition:', data.partition);  console.log('  Key:', data.key ? data.key.toString() : null);  console.log('  Value:', data.value.toString());