Kafka Client for Java: Complete Guide with Examples

Introduction to Apache Kafka Client for Java

Apache Kafka is a distributed event streaming platform that allows you to publish, store, and process data streams in real time. The Kafka Client for Java library is the official client providing an API for interacting with Kafka brokers. It allows Java applications to act as producers (send messages) and consumers (read messages), as well as manage metadata and administrative tasks.

This library is indispensable when building microservice architectures, log processing systems, ETL pipelines, real-time analytics systems, and any other scenarios requiring reliable, scalable, and fault-tolerant data transfer between components.

Installing Kafka Client for Java

Kafka Client for Java is distributed via Maven Central Repository. To add the library to a project, you need to specify the dependency in the pom.xml file (for Maven) or build.gradle (for Gradle).

Maven

<dependency>\n    <groupId>org.apache.kafka</groupId>\n    <artifactId>kafka-clients</artifactId>\n    <version>3.6.0</version> <!-- Use the current version -->\n</dependency>

Gradle

implementation 'org.apache.kafka:kafka-clients:3.6.0'

After adding the dependency and updating the project, all library classes will be available for import.

Quick Start: Producer and Consumer

Let's look at a minimal example of sending and receiving a message.

Producer (Sender)

import org.apache.kafka.clients.producer.*;\nimport java.util.Properties;\n\npublic class SimpleProducer {\n    public static void main(String[] args) {\n        // 1. Configure properties\n        Properties props = new Properties();\n        props.put("bootstrap.servers", "localhost:9092");\n        props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");\n        props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");\n\n        // 2. Create producer\n        Producer<String, String> producer = new KafkaProducer<>(props);\n\n        // 3. Send message\n        ProducerRecord<String, String> record = new ProducerRecord<>("my-topic", "key1", "Hello, Kafka!");\n        producer.send(record);\n\n        // 4. Close producer\n        producer.close();\n    }\n}

Consumer (Receiver)

import org.apache.kafka.clients.consumer.*;\nimport java.time.Duration;\nimport java.util.Collections;\nimport java.util.Properties;\n\npublic class SimpleConsumer {\n    public static void main(String[] args) {\n        // 1. Configure properties\n        Properties props = new Properties();\n        props.put("bootstrap.servers", "localhost:9092");\n        props.put("group.id", "my-group");\n        props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");\n        props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");\n        props.put("auto.offset.reset", "earliest");\n\n        // 2. Create consumer\n        Consumer<String, String> consumer = new KafkaConsumer<>(props);\n\n        // 3. Subscribe to topic\n        consumer.subscribe(Collections.singletonList("my-topic"));\n\n        // 4. Read messages\n        while (true) {\n            ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));\n            for (ConsumerRecord<String, String> record : records) {\n                System.out.printf("offset = %d, key = %s, value = %s%n", record.offset(), record.key(), record.value());\n            }\n        }\n    }\n}

Main Methods and Functions of the Library

Consider