Cache-Friendly Code in C++: What It Is and Why It Matters for Speed

Your code runs slowly, even though the algorithm is perfect? It might not be "friendly" with the cache

Have you ever written code in C++ that should theoretically run lightning-fast, but in practice lags? Or noticed that traversing a two-dimensional array by rows and by columns yields different speeds, even though the algorithmic complexity is the same? If so, you've encountered cache-friendly code and its opposite — cache-unfriendly code.

Today, we'll break down what this means at the hardware level, why the cache isn't just "another memory" but a key to performance, and how to write code that the processor will "love." No complex abstractions — just C++ and clear examples.

Why can't the processor just read from RAM?

Processor speed has increased thousands of times over the past decades, while RAM (DRAM) speed has only increased tens of times. Imagine a Formula 1 driver (the processor) who has to stop every second to get fuel (data) from a truck moving at 30 km/h (RAM). Sad and slow.

The solution is the CPU cache: a small but very fast memory built directly into the chip. It stores copies of frequently used data. When the processor needs data, it first checks the cache (L1 — the fastest, then L2, L3). Only if the data isn't there (cache miss) does it go to the slow RAM.

Here are approximate latencies (in CPU cycles):

\n
• L1 cache: ~3-5 cycles
\n
• L2 cache: ~10-20 cycles
\n
• L3 cache: ~30-50 cycles
\n
• RAM: ~100-300 cycles
\n

The difference between L1 and RAM is 50-100 times. This is why cache-friendly code in C++ can be many times faster than its "unfriendly" counterpart.

The principle of locality: how the cache decides what to keep

The cache doesn't copy a single byte — it works in blocks (cache lines), typically 64 bytes. When you read one variable, the processor loads the entire block containing that variable. And it bets on two principles:

\n
• Spatial locality: if you read an array element arr[i], you'll likely need arr[i+1] soon.
\n
• Temporal locality: if you read data now, you'll probably access it again in the near future.
\n

Cache-friendly code is code that maximally leverages these principles. Let's see what this looks like in practice.

Example 1: Traversing a two-dimensional array — a classic

Cache-unfriendly (bad code)

const int SIZE = 1024;\nint matrix[SIZE][SIZE];\n\n// Column-major traversal\nfor (int j = 0; j < SIZE; ++j) {\n    for (int i = 0; i < SIZE; ++i) {\n        matrix[i][j] = i + j;\n    }\n}\n

In C++, arrays are stored in row-major order. That is, elements of a row matrix[0][0], matrix[0][1], matrix[0][2]... lie sequentially in memory. The outer loop over j (columns) forces the processor to jump across the entire row at each step — each access to matrix[i][j] is most likely a cache miss. The processor spends time loading new blocks, while previous data is evicted from the cache.

Cache-friendly (good code)

// Row-major traversal\nfor (int i = 0; i < SIZE; ++i) {\n    for (int j = 0; j < SIZE; ++j) {\n        matrix[i][j] = i + j;\n    }\n}\n

Now we traverse memory sequentially. As soon as we load the block with matrix[0][0], the next 15 elements (with a 4-byte int) are already in the cache! This yields a 5-10x speed improvement