Working with Apache Kafka in C++: Libraries and Examples

Working with Apache Kafka in C++: Libraries, Configuration, and Examples

Apache Kafka is a distributed streaming platform widely used in microservice architecture, logging systems, and analytics. For integration with C++, there are several mature libraries, the most popular being librdkafka (low-level) and cppkafka (C++ wrapper). In this article, we will thoroughly examine both libraries, show code examples for producer and consumer, and provide recommendations for selection.

1. librdkafka — Foundation for Working with Kafka in C

librdkafka is a high-performance C library from Confluent that implements the Apache Kafka protocol. It supports all key features: asynchronous message sending, consumer groups, exactly-once semantics (via transactions), and SSL/SASL. The library is written in C but can be used from C++ directly or through C++ wrappers.

Installing librdkafka

On Ubuntu/Debian:

sudo apt-get install librdkafka-dev

On macOS via Homebrew:

brew install librdkafka

Building from source:

git clone https://github.com/confluentinc/librdkafka.gitcd librdkafka./configure --prefix=/usr/localmakesudo make install

Producer Example in C Using librdkafka

A simple producer sending a message to a topic:

#include <librdkafka/rdkafka.h>#include <iostream>#include <string>#include <cstring>

int main() {    rd_kafka_t *rk;          // producer handle    rd_kafka_conf_t *conf;   // configuration    char errstr[512];

    conf = rd_kafka_conf_new();    rd_kafka_conf_set(conf, "bootstrap.servers", "localhost:9092", errstr, sizeof(errstr));    rd_kafka_conf_set(conf, "acks", "all", errstr, sizeof(errstr));

    rk = rd_kafka_new(RD_KAFKA_PRODUCER, conf, errstr, sizeof(errstr));    if (!rk) {        std::cerr << "Failed to create producer: " << errstr << std::endl;        return 1;    }

    std::string topic_name = "test-topic";    std::string message = "Hello from C++ and librdkafka!";

    rd_kafka_resp_err_t err = rd_kafka_producev(        rk,        RD_KAFKA_V_TOPIC(topic_name.c_str()),        RD_KAFKA_V_MSGFLAGS(RD_KAFKA_MSG_F_COPY),        RD_KAFKA_V_VALUE((void*)message.c_str(), message.size()),        RD_KAFKA_V_END    );

    if (err) {        std::cerr << "Failed to produce message: " << rd_kafka_err2str(err) << std::endl;    } else {        std::cout << "Message sent successfully!" << std::endl;    }

    // Wait for delivery and cleanup    rd_kafka_flush(rk, 10000);    rd_kafka_destroy(rk);    return 0;}

Consumer Example in C Using librdkafka

A simple consumer reading messages from a topic:

#include <librdkafka/rdkafka.h>#include <iostream>#include <string>

int main() {    rd_kafka_t *rk;    rd_kafka_conf_t *conf;    rd_kafka_topic_partition_list_t *topics;    char errstr[512];

    conf = rd_kafka_conf_new();    rd_kafka_conf_set(conf, "bootstrap.servers", "localhost:9092", errstr, sizeof(errstr));    rd_kafka_conf_set(conf, "group.id", "my-group", errstr, sizeof(errstr));    rd_kafka_conf_set(conf, "auto.offset.reset", "earliest", errstr, sizeof(errstr));

    rk = rd_kafka_new(RD_KAFKA_CONSUMER, conf, errstr, sizeof(errstr));    if (!rk) {        std::cerr << "Failed to create consumer: " << errstr << std::endl;        return 1;    }

    rd_kafka_poll_set_consumer(rk);    topics = rd_kafka_topic_partition_list_new(1);    rd_kafka_topic_partition_list_add(topics, "test-topic", RD_KAFKA_PARTITION_UA);    rd_kafka_consumer_subscribe(rk, topics);

    std::cout << "Waiting for messages..." <<