Working with Kafka in Rust: rdkafka and Franz Libraries

Working with Kafka in Rust: Library Overview

Apache Kafka is a distributed streaming platform widely used in microservice architecture, event processing, and real-time systems. The Rust language, thanks to its performance and memory safety, is well-suited for writing high-load Kafka applications. In this article, we will look at two main libraries for working with Kafka in Rust: rdkafka (based on librdkafka) and Franz (pure Rust).

Why choose Rust for working with Kafka?

• High performance — Rust compiles to native code, minimizing overhead.
• Memory safety — the ownership system prevents data races and memory leaks.
• Zero-cost abstractions — allows writing efficient code without losing expressiveness.
• Asynchrony — built-in async/await support simplifies working with streams.

The rdkafka Library

rdkafka is the most popular library for working with Kafka in Rust. It is a wrapper around librdkafka — a high-performance C library used in many projects (including Node.js, Python, Go).

Installing rdkafka

Add the dependency to Cargo.toml:

[dependencies]rdkafka = { version = "0.36", features = ["cmake-build"] }tokio = { version = "1", features = ["full"] }

Example: Producer (sending messages)

use rdkafka::producer::{FutureProducer, FutureRecord};use rdkafka::ClientConfig;use std::time::Duration;

#[tokio::main]async fn main() {    let producer: FutureProducer = ClientConfig::new()        .set("bootstrap.servers", "localhost:9092")        .create()        .expect("Producer creation error");

    let record = FutureRecord::to("my-topic")        .payload("Hello from Rust!")        .key("key-1");

    let delivery_status = producer        .send(record, Duration::from_secs(0))        .await;

    match delivery_status {        Ok(delivery) => println!("Sent: {:?}", delivery),        Err(e) => eprintln!("Error: {:?}", e),    }}

Example: Consumer (reading messages)

use rdkafka::consumer::{Consumer, StreamConsumer};use rdkafka::ClientConfig;use rdkafka::message::Message;use futures::stream::StreamExt;

#[tokio::main]async fn main() {    let consumer: StreamConsumer = ClientConfig::new()        .set("bootstrap.servers", "localhost:9092")        .set("group.id", "my-group")        .set("auto.offset.reset", "earliest")        .create()        .expect("Consumer creation error");

    consumer.subscribe(&["my-topic"]).unwrap();

    let mut stream = consumer.stream();    while let Some(message) = stream.next().await {        match message {            Ok(msg) => {                let payload = msg.payload_view::<str>().unwrap_or(None);                if let Some(text) = payload {                    println!("Received: {}", text);                }            }            Err(e) => eprintln!("Error: {}", e),        }    }}

The Franz Library

Franz is a modern Kafka library written entirely in Rust (without C bindings). It provides a safe and ergonomic API, uses async/await, and supports the latest versions of the Kafka protocol (including KIP-848).

Installing Franz

[dependencies]franz = "0.7"tokio = { version = "1", features = ["full"] }

Example: Producer in Franz

use franz::client::Client;use franz::record::Record;use franz::producer::Producer;

#[tokio::main]async fn main() -> Result<(), Box<dyn std::error::Error>> {    l