Working with Apache Kafka in Node.js: KafkaJS Library

Working with Apache Kafka in Node.js using the KafkaJS Library

Apache Kafka is a distributed event streaming platform that has become the de facto standard for building high-load messaging systems. In the Node.js ecosystem, one of the most popular and stable libraries for interacting with Kafka is KafkaJS. It is a modern, fully asynchronous client written in pure JavaScript that supports all key Kafka features: producers, consumers, consumer groups, offset management, and transactions.

In this article, we will thoroughly cover how to install KafkaJS, configure a connection to a Kafka cluster, send and read messages, and also discuss best practices for working with this library in a production environment.

Installing and Configuring KafkaJS

The library is installed using the standard npm package manager:

npm install kafkajs

After installation, you need to create a client by specifying the broker addresses and a client ID. Example of minimal configuration:

const { Kafka } = require('kafkajs');

const kafka = new Kafka({  clientId: 'my-app',  brokers: ['localhost:9092']});

For a production environment, it is recommended to specify multiple brokers separated by commas to ensure fault tolerance, and also configure SSL and SASL authentication:

const kafka = new Kafka({  clientId: 'my-production-app',  brokers: ['broker1:9092', 'broker2:9092', 'broker3:9092'],  ssl: true,  sasl: {    mechanism: 'scram-sha-256',    username: 'myuser',    password: 'mypassword'  }});

Producer: Sending Messages to a Topic

The producer is responsible for publishing messages to a specified Kafka topic. KafkaJS provides two modes of operation for the producer: simple (send) and transactional (for exactly-once delivery). Let's look at a basic example of sending a message:

const producer = kafka.producer();

async function sendMessage() {  await producer.connect();

  await producer.send({    topic: 'test-topic',    messages: [      { key: 'user1', value: 'Hello Kafka from Node.js!' },      { key: 'user2', value: JSON.stringify({ orderId: 123, amount: 2500 }) }    ]  });

  await producer.disconnect();}

sendMessage().catch(console.error);

Key points:

• topic — the name of the topic to which the message is sent.
• messages — an array of objects, each of which can contain key (optional) and value (string or Buffer).
• It is recommended to call producer.connect() once when the application starts and reuse the connection.

To improve performance in production, use a connection pool and configure acks:

const producer = kafka.producer({  allowAutoTopicCreation: false,  transactionTimeout: 30000});

await producer.send({  topic: 'orders',  messages: [{ value: 'New order' }],  acks: -1  // Wait for acknowledgment from all replicas});

Consumer: Reading Messages from a Topic

The consumer allows reading messages from one or more topics. The most important concept is consumer groups, which provide load balancing and ensure that each message is processed by only one consumer within the group.

const consumer = kafka.consumer({ groupId: 'my-group' });

async function consumeMessages() {  await consumer.connect();  await consumer.subscribe({ topic: 'test-topic', fromBeginning: true });

  await consumer.run({    eachMessage: async ({ topic, partition, message }) => {      console.log({        partition,        offset: message.offset,        key: message.key.toString(),        value: message.valu