Working with Kafka in Kotlin: Libraries and Examples

Working with Kafka in Kotlin: Library Overview and Practical Examples

Apache Kafka is a distributed streaming platform that has become the de facto standard in microservice architecture. Kotlin, being a modern JVM language, is fully compatible with the Java ecosystem, allowing the use of all popular Kafka libraries. In this article, we will look at three main approaches: the official Java Kafka client, Spring Kafka, and the lightweight Kafkakit library. All examples are written in Kotlin.

1. Official Kafka Client (Java Client)

The kafka-clients library from Apache is the foundation on which all other wrappers are built. It provides a low-level API for producers and consumers. For Kotlin, simply add the dependency to build.gradle.kts:

implementation("org.apache.kafka:kafka-clients:3.6.0")

Kotlin Producer Example

import org.apache.kafka.clients.producer.KafkaProducerimport org.apache.kafka.clients.producer.ProducerRecordimport java.util.Properties

fun createProducer(): KafkaProducer<String, String> {    val props = Properties().apply {        put("bootstrap.servers", "localhost:9092")        put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer")        put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer")    }    return KafkaProducer(props)}

fun main() {    val producer = createProducer()    val record = ProducerRecord("my-topic", "key1", "Hello from Kotlin!")    producer.send(record) { metadata, exception ->        if (exception != null) {            println("Send error: ${exception.message}")        } else {            println("Sent to partition ${metadata.partition()} offset ${metadata.offset()}")        }    }    producer.close()}

Kotlin Consumer Example

import org.apache.kafka.clients.consumer.ConsumerRecordsimport org.apache.kafka.clients.consumer.KafkaConsumerimport java.time.Durationimport java.util.Properties

fun createConsumer(): KafkaConsumer<String, String> {    val props = Properties().apply {        put("bootstrap.servers", "localhost:9092")        put("group.id", "kotlin-group")        put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer")        put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer")        put("auto.offset.reset", "earliest")    }    return KafkaConsumer(props)}

fun main() {    val consumer = createConsumer()    consumer.subscribe(listOf("my-topic"))

    while (true) {        val records: ConsumerRecords<String, String> = consumer.poll(Duration.ofMillis(100))        records.forEach { record ->            println("Received: key=${record.key()}, value=${record.value()}, offset=${record.offset()}")        }    }}

2. Spring Kafka with Kotlin

If you are using Spring Boot, Spring Kafka is the natural choice. It provides a declarative approach through annotations and automatic configuration. Add the dependency:

implementation("org.springframework.kafka:spring-kafka")

Configuration in application.yml

spring:  kafka:    bootstrap-servers: localhost:9092    producer:      key-serializer: org.apache.kafka.common.serialization.StringSerializer      value-serializer: org.apache.kafka.common.serialization.StringSerializer    consumer:      group-id: spring-kotlin-group      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer      value-deserializer: org.apache.kafka.common.serialization.StringDeserializer      auto-offset-reset: earliest

Producer using KafkaTemplate

import org.springfram