Sarama for Go: Complete Guide with Examples

What is Sarama and why is it needed?

Sarama is a high-performance, pure Go library for working with Apache Kafka. It provides a full set of tools for interacting with a Kafka cluster, including producers for sending messages, consumers for reading messages, as well as administrative functions for managing topics, partitions, and configurations.

Apache Kafka is a distributed streaming platform used for building real-time data pipelines, messaging systems, event processing, and log storage. Sarama allows Go applications to easily integrate with Kafka, providing reliable and efficient data transfer.

Why use Sarama?

\n
• Pure Go: Sarama is written entirely in Go without external C/C++ dependencies, simplifying build and deployment.
\n
• High performance: The library is optimized for working with large volumes of data and supports asynchronous message sending.
\n
• Flexibility: Supports various producer modes (synchronous, asynchronous) and consumer modes (simple, consumer group).
\n
• Active community: Sarama is one of the most popular Kafka libraries for Go with a large number of contributors and extensive documentation.
\n
• Full control: Gives the developer complete control over configuration, including offset management, serialization/deserialization, and error handling.
\n

Installation

Installing Sarama in a Go project is done using the standard method through the go get package manager.

go get github.com/IBM/sarama

After executing this command, the library will be added to your go.mod file, and you will be able to import it in your code.

Important: Make sure you have Go version 1.18 or higher installed, as Sarama uses modern language features.

Quick start: sending and receiving messages

Let's write a simple example that will send a message to a Kafka topic and read it.

1. Producer — sending a message

package main\n\nimport (\n    "fmt"\n    "log"\n\n    "github.com/IBM/sarama"\n)\n\nfunc main() {\n    // 1. Producer configuration\n    config := sarama.NewConfig()\n    config.Producer.Return.Successes = true // Required for synchronous mode\n    config.Producer.RequiredAcks = sarama.WaitForAll // Wait for acknowledgment from all replicas\n\n    // 2. Creating a synchronous producer\n    producer, err := sarama.NewSyncProducer([]string{"localhost:9092"}, config)\n    if err != nil {\n        log.Fatalf("Failed to create producer: %v", err)\n    }\n    defer producer.Close()\n\n    // 3. Creating a message\n    msg := &sarama.ProducerMessage{\n        Topic: "my-topic",\n        Key:   sarama.StringEncoder("my-key"),\n        Value: sarama.StringEncoder("Hello, Kafka!"),\n    }\n\n    // 4. Sending the message\n    partition, offset, err := producer.SendMessage(msg)\n    if err != nil {\n        log.Fatalf("Failed to send message: %v", err)\n    }\n\n    fmt.Printf("Message sent to partition %d with offset %d\\n", partition, offset)\n}\n

2. Consumer — receiving a message

package main\n\nimport (\n    "fmt"\n    "log"\n    "os"\n    "os/signal"\n    "syscall"\n\n    "github.com/IBM/sarama"\n)\n\nfunc main() {\n    // 1. Consumer configuration\n    config := sarama.NewConfig()\n    config.Consumer.Return.Errors = true\n    config.Consumer.Offsets.Initial = sarama.OffsetOldest // Read from the beginning\n\n    // 2. Creating a consumer\n    consumer, err := sarama.NewConsumer([]string{"localhost:9092"}, config)\n    if err != nil {\n        log.Fatalf("Failed to create consumer: %v", err)\n    }\n