Working with Kafka in PHP: Libraries, Connection, and Code Examples

Working with Kafka in PHP: Libraries, Connection, and Code Examples

Apache Kafka is a distributed streaming platform that allows you to publish, store, and process streams of messages in real time. In the PHP ecosystem, the php-rdkafka extension is used to work with Kafka — a wrapper around the librdkafka library written in C. In this article, we will cover how to install the tool, write a producer and consumer, and look at typical usage scenarios.

What is php-rdkafka?

php-rdkafka is the official PHP extension for working with Apache Kafka. It provides a high-performance interface for sending and receiving messages. The extension supports all key Kafka features: topics, partitions, consumer groups, manual commits, compression, and much more.

Key features:

• Creating producers to send messages to topics.
• Creating consumers to read messages from topics.
• Support for consumer groups and automatic rebalancing.
• Working with partitions and manual offset management.
• Message compression (gzip, snappy, lz4, zstd).
• Asynchronous sending and batch processing.

Installing php-rdkafka

The extension can be installed via PECL or built from source. The recommended method is via PECL:

# Installation via PECLpecl install rdkafka

# For Ubuntu/Debian, first install librdkafka-devsudo apt-get install librdkafka-devpecl install rdkafka

# Add the extension to php.iniextension=rdkafka.so

After installation, verify that the extension is loaded:

php -m | grep rdkafka

If you are using Composer, you can additionally install the arnaud-lb/php-rdkafka package for autoloading:

composer require arnaud-lb/php-rdkafka

Producer Example

The producer sends messages to a specified Kafka topic. Let's look at a basic example with synchronous sending:

<?phprequire_once 'vendor/autoload.php';

$conf = new RdKafka\\Conf();$conf->set('metadata.broker.list', 'localhost:9092');

$producer = new RdKafka\\Producer($conf);$topic = $producer->newTopic('test_topic');

// Sending a message$topic->produce(RD_KAFKA_PARTITION_UA, 0, 'Hello, Kafka from PHP!');

// Wait for all messages to be sent$producer->flush(1000);echo "Message sent";

Code explanation:

• metadata.broker.list — the address of the Kafka broker (default localhost:9092).
• RD_KAFKA_PARTITION_UA — automatic partition selection.
• flush(1000) — blocks execution until all messages are sent (timeout 1 second).

Consumer Example

The consumer reads messages from a topic. It is important to specify the consumer group and the offset commit strategy:

<?phprequire_once 'vendor/autoload.php';

$conf = new RdKafka\\Conf();$conf->set('metadata.broker.list', 'localhost:9092');$conf->set('group.id', 'my_consumer_group');$conf->set('auto.offset.reset', 'earliest');

$consumer = new RdKafka\\KafkaConsumer($conf);$consumer->subscribe(['test_topic']);

echo "Waiting for messages...";

while (true) {    $message = $consumer->consume(120 * 1000); // timeout 2 minutes    switch ($message->err) {        case RD_KAFKA_RESP_ERR_NO_ERROR:            echo "Received message: " . $message->payload . "";            break;        case RD_KAFKA_RESP_ERR__PARTITION_EOF:            echo "End of partition reached";            break;        case RD_KAFKA_RESP_ERR__TIMED_OUT:            echo "Timeout";            break;        defau