Processing Big Data in PHP: From Myth to Real Project

PHP and Big Data: Not an Oxymoron, but Reality

When you hear "big data processing," the first things that come to mind are Spark, Hadoop, or Python with Pandas. PHP in this list is usually perceived as "a language for landing pages and forums." But let's face the truth: if your project already lives on PHP, rewriting it entirely in Go for a single task of processing a gigabyte-sized CSV is like using a sledgehammer to crack a nut.

PHP handles Big Data perfectly well if you approach the issue wisely. In this article, we'll break down real techniques: how not to kill your server when working with files over 10 GB, how to process millions of rows from a database without out of memory, and how to turn PHP into a real data pipeline. No fluff, only code and architectural solutions.

Why PHP "Doesn't Like" Big Data (and How to Fix It)

The main problem with PHP in the context of Big Data is its execution model. The script runs sequentially, loading everything into memory. Try using file_get_contents() on a 5 GB file — the server will simply crash.

But there's good news: PHP has built-in mechanisms for stream processing. You just need to forget about "barbaric" methods like file() or file_get_contents() and switch to generators, streams, and chunk queries.

Generators: Your Main Tool

Generators (the yield keyword) allow you to return data one element at a time without loading the entire set into memory. This is ideal for reading large files or database result sets.

Example: reading a 5 GB CSV file line by line without killing memory:

<?phpfunction readLargeCSV(string $filePath): Generator{    $handle = fopen($filePath, 'r');    if ($handle === false) {        throw new RuntimeException("Failed to open file: $filePath");    }

    // Skip BOM if present    $bom = fread($handle, 3);    if ($bom !== "\\xEF\\xBB\\xBF") {        rewind($handle); // If no BOM, go back to the beginning    }

    while (($row = fgetcsv($handle, 0, ',')) !== false) {        yield $row;    }

    fclose($handle);}

// Usageforeach (readLargeCSV('/data/orders_2024.csv') as $index => $row) {    // Process each row one by one    if ($index === 0) {        continue; // skip header    }    processOrder($row);}?>

Why does this work? The generator does not load the entire file into an array. It reads one line — yields it — reads the next. Memory consumption stays at a few kilobytes, regardless of the file size.

Working with Large Databases: Chunk Queries

If you need to process 5 million records from MySQL, don't try SELECT * FROM table — it will kill both the DB and PHP. Instead, use cursors or chunk queries.

Here's the correct approach using PDO and chunk iteration:

<?phpfunction processLargeTable(PDO $pdo, int $chunkSize = 1000, callable $callback): void{    $offset = 0;    $lastId = 0;

    // Use ORDER BY id for stability    $stmt = $pdo->prepare("SELECT * FROM orders WHERE id > :lastId ORDER BY id LIMIT :limit");

    do {        $stmt->execute(['lastId' => $lastId, 'limit' => $chunkSize]);        $rows = $stmt->fetchAll(PDO::FETCH_ASSOC);

        if (empty($rows)) {            break;        }

        foreach ($rows as $row) {            $callback($row);            $lastId = $row['id']; // Remember the last processed ID        }

        // Free memory        unset($rows);        $stmt->closeCursor();

    } while (true);}

// Usage$pdo = new PDO('mysql:host=localhost;dbname=big_data', 'user', 'pass');$pdo->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);

p