Working with Kafka in Rust: Libraries and Examples for Beginners

Working with Kafka in Rust: Library Overview and Practical Examples

Apache Kafka is a distributed streaming platform widely used for building reliable and scalable messaging systems. The Rust language, thanks to its performance and memory safety, is becoming an increasingly popular choice for writing high-load applications that work with Kafka. In this article, we will review the main libraries for working with Kafka in Rust, their installation, and provide simple code examples.

Popular Libraries for Kafka in Rust

Currently, there are two main approaches to working with Kafka in the Rust ecosystem:

• rdkafka — a wrapper over the popular C library librdkafka. It provides maximum performance and functionality, including support for all modern Kafka features (transactions, idempotence, exactly-once semantics).
• Kafka-Rust (kafka) — a pure Rust implementation of the Kafka protocol, without external dependencies. Easier to build, but may lag behind rdkafka in functionality.

For most production projects, it is recommended to use rdkafka, as it is more mature, performant, and actively maintained by the community.

Installing rdkafka

To get started, add the dependency to your Cargo.toml:

[dependencies]rdkafka = { version = "0.36", features = ["cmake-build"] }tokio = { version = "1", features = ["full"] }  # for async workserde = { version = "1", features = ["derive"] }  # for data serializationserde_json = "1"  # for working with JSON

The rdkafka library requires librdkafka to be installed on your system. You can install it via your system package manager or use the cmake-build feature as in the example above — in this case, the library will be compiled automatically.

Example: Basic Producer

A producer sends messages to a specified Kafka topic. Here is a simple example of an async producer using Tokio:

use rdkafka::config::ClientConfig;use rdkafka::producer::{FutureProducer, FutureRecord};use std::time::Duration;

#[tokio::main]async fn main() {    // Create producer configuration    let producer: FutureProducer = ClientConfig::new()        .set("bootstrap.servers", "localhost:9092")        .set("message.timeout.ms", "5000")        .create()        .expect("Producer creation error");

    // Send a message to the "my-topic" topic    let delivery_status = producer        .send(            FutureRecord::to("my-topic")                .payload("Hello, Kafka from Rust!")                .key("message-key"),            Duration::from_secs(0),        )        .await;

    match delivery_status {        Ok(partition_offset) => println!("Message sent: {:?}", partition_offset),        Err((e, _)) => eprintln!("Send error: {}", e),    }}

Example: Basic Consumer

A consumer reads messages from a topic. Below is an example of a synchronous consumer with manual commit:

use rdkafka::config::ClientConfig;use rdkafka::consumer::{Consumer, StreamConsumer};use rdkafka::message::Message;use rdkafka::util::get_rdkafka_version;

fn main() {    let consumer: StreamConsumer = ClientConfig::new()        .set("group.id", "my-consumer-group")        .set("bootstrap.servers", "localhost:9092")        .set("enable.auto.commit", "false")  // manual commit        .set("auto.offset.reset", "earliest")  // read from the beginning        .create()        .expect("Consumer creation error");

    consumer.subscribe(&["my-topic"]).expect("Subscription error");

    loop {        match consumer.recv() {            Ok(msg) => {                let paylo