Kafka in TypeScript: KafkaJS for Node.js

KafkaJS Library: Working with Apache Kafka in TypeScript

Apache Kafka is a distributed streaming platform widely used for building high-throughput messaging systems. In the TypeScript (and Node.js) ecosystem, the KafkaJS library has become the de facto standard. It provides a clean, modern API with full TypeScript support, allowing developers to write type-safe code for Kafka producers and consumers.

In this article, we will thoroughly cover the installation, configuration, and main usage scenarios of KafkaJS. You will learn how to send and receive messages, handle errors, and properly configure the client for a production environment.

Installing and Configuring KafkaJS

To get started, install the library via npm or yarn:

npm install kafkajs# oryarn add kafkajs

KafkaJS is written in TypeScript, so types are built-in — no need to install @types/kafkajs separately.

Now, let's create a Kafka client instance. It is recommended to specify multiple brokers for fault tolerance:

import { Kafka, logLevel } from 'kafkajs';

const kafka = new Kafka({  clientId: 'my-app',  brokers: ['localhost:9092', 'localhost:9093'],  logLevel: logLevel.INFO, // Log level  retry: {    initialRetryTime: 100,    retries: 8  }});

Producer: Sending Messages to Kafka

The producer is responsible for writing messages to Kafka topics. KafkaJS provides a convenient API for sending both individual messages and batches.

Example of a basic producer:

import { Kafka, Producer } from 'kafkajs';

async function produceMessage(producer: Producer) {  await producer.send({    topic: 'orders',    messages: [      {         key: 'order-123',         value: JSON.stringify({ userId: 42, amount: 1500 })       },    ],  });  console.log('Message sent');}

async function runProducer() {  const producer = kafka.producer();  await producer.connect();

  // Sending multiple messages  await producer.send({    topic: 'orders',    messages: [      { key: 'order-124', value: '{"item":"laptop"}' },      { key: 'order-125', value: '{"item":"mouse"}' },    ],  });

  await produceMessage(producer);  await producer.disconnect();}

runProducer().catch(console.error);

Important: Always call disconnect() after finishing work to free up resources. For long-running applications, the producer can be kept connected permanently.

Consumer: Reading Messages from Kafka

The consumer subscribes to one or more topics and processes messages. KafkaJS supports consumer groups for parallel processing.

Example of a consumer with message processing:

import { Kafka, Consumer, EachMessagePayload } from 'kafkajs';

async function runConsumer() {  const consumer: Consumer = kafka.consumer({     groupId: 'order-processor-group'   });

  await consumer.connect();  await consumer.subscribe({     topic: 'orders',     fromBeginning: true // Read from the very first message  });

  await consumer.run({    eachMessage: async ({ topic, partition, message }: EachMessagePayload) => {      const key = message.key?.toString();      const value = message.value?.toString();      console.log({        partition,        offset: message.offset,        key,        value: value ? JSON.parse(value) : null,      });    },  });}

runConsumer().catch(console.error);

Note the fromBeginning parameter: if set to true, the consumer will read all previously accumulated messages. For production environments, false is typically used (only new messages).