Working with Apache Kafka in Java: A Complete Developer's Guide

Working with Apache Kafka in Java: From Basics to Advanced Usage

Apache Kafka is a distributed data streaming platform that has become the de facto standard for building event-driven architectures. In this guide, we will break down how to effectively work with Kafka in Java: from installing the necessary libraries to writing a producer and consumer with error handling.

Why Java and Kafka are the Perfect Pair?

Java is one of the primary languages for working with Kafka for several reasons:

• The official Kafka client is written in Java and provides maximum performance
• A rich ecosystem of libraries (Spring Kafka, Kafka Streams, Akka Streams)
• Strong typing and the ability for fine-tuning through configuration objects

Installing and Adding Dependencies

To work with Kafka in Java, you will need to add a dependency to pom.xml (Maven) or build.gradle (Gradle).

Maven

<dependency>    <groupId>org.apache.kafka</groupId>    <artifactId>kafka-clients</artifactId>    <version>3.6.0</version></dependency>

Gradle

implementation 'org.apache.kafka:kafka-clients:3.6.0'

Creating a Kafka Producer

The producer is responsible for sending messages to Kafka topics. Here is a basic configuration example:

import org.apache.kafka.clients.producer.*;import org.apache.kafka.common.serialization.StringSerializer;import java.util.Properties;

public class SimpleProducer {    public static void main(String[] args) {        // 1. Configure producer properties        Properties props = new Properties();        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());                // 2. Create the producer        KafkaProducer<String, String> producer = new KafkaProducer<>(props);                // 3. Send a message        ProducerRecord<String, String> record = new ProducerRecord<>(            "my-topic",             "key-1",             "Hello, Kafka from Java!"        );                // 4. Asynchronous send with callback        producer.send(record, (metadata, exception) -> {            if (exception == null) {                System.out.println("Message sent: " +                     "topic=" + metadata.topic() +                     ", partition=" + metadata.partition() +                     ", offset=" + metadata.offset());            } else {                System.err.println("Send error: " + exception.getMessage());            }        });                // 5. Close the producer        producer.close();    }}

Creating a Kafka Consumer

The consumer reads messages from topics. It is important to correctly configure the consumer group for load balancing.

import org.apache.kafka.clients.consumer.*;import org.apache.kafka.common.serialization.StringDeserializer;import java.time.Duration;import java.util.Collections;import java.util.Properties;

public class SimpleConsumer {    public static void main(String[] args) {        // 1. Configure consumer properties        Properties props = new Properties();        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");        props.put(ConsumerConfig.GROUP_ID_CONFIG, "my-consumer-group");        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());        props.put(Consu