Working with Kafka in Go (Golang): segmentio/kafka-go library

Working with Kafka in Go (Golang) with the segmentio/kafka-go library

Apache Kafka is a distributed streaming platform that has become the de facto standard for building high-load message exchange systems. When developing in Go (Golang), one of the most popular and reliable libraries for interacting with Kafka is segmentio/kafka-go. In this article, we will thoroughly cover how to install, configure, and use this library to create producers and consumers.

Why segmentio/kafka-go?

There are several Go libraries for Kafka on the market (e.g., Sarama and Confluent Go Client). segmentio/kafka-go stands out with the following advantages:

• Pure Go — no C dependencies (unlike Confluent), simplifying build and deployment.
• Kafka Protocol v2 support — efficient compression, batching, and transaction support.
• Simple API — intuitive interfaces for producers, consumers, and topic administration.
• Active community — regular updates and good documentation.

Installing the library

To get started, add the library to your project using go get:

go get github.com/segmentio/kafka-go

Make sure you have a Kafka instance running (locally or via Docker). Example minimal Docker Compose configuration:

version: '3'services:  zookeeper:    image: confluentinc/cp-zookeeper:latest    environment:      ZOOKEEPER_CLIENT_PORT: 2181  kafka:    image: confluentinc/cp-kafka:latest    depends_on:      - zookeeper    ports:      - "9092:9092"    environment:      KAFKA_BROKER_ID: 1      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092

Creating a Producer

A producer sends messages to a Kafka topic. Let's look at a basic example:

package main

import (    "context"    "fmt"    "log"    "time"

    "github.com/segmentio/kafka-go")

func main() {    // Configure the producer    writer := &kafka.Writer{        Addr:     kafka.TCP("localhost:9092"),        Topic:    "test-topic",        Balancer: &kafka.LeastBytes{}, // Load balancing by message size        BatchSize: 10,                 // Send batch of 10 messages        BatchTimeout: 100 * time.Millisecond,    }    defer writer.Close()

    // Send a message    err := writer.WriteMessages(context.Background(),        kafka.Message{            Key:   []byte("key-1"),            Value: []byte("Hello, Kafka from Go!"),        },    )    if err != nil {        log.Fatal("Error sending message:", err)    }

    fmt.Println("Message sent successfully")}

Key parameters of kafka.Writer:

• Addr — Kafka broker address.
• Topic — topic name.
• Balancer — message distribution strategy across partitions (LeastBytes, RoundRobin, Hash).
• BatchSize and BatchTimeout — performance optimization.

Creating a Consumer

A consumer reads messages from a topic. A simple example with manual offset management:

package main

import (    "context"    "fmt"    "log"

    "github.com/segmentio/kafka-go")

func main() {    // Configure the reader (consumer)    reader := kafka.NewReader(kafka.ReaderConfig{        Brokers:   []string{"localhost:9092"},        Topic:     "test-topic",        GroupID:   "my-consumer-group",        MinBytes:  10, // Minimum batch size        MaxBytes:  10e6, // 10 MB        StartOffset: kafka.FirstOffset, // Start