Criterion Library for Rust: Statistical Benchmarking

In the world of systems programming, performance is not just a wish, but often a strict requirement. Rust, being a language that gives the developer control over memory and execution time, requires tools capable of precisely measuring this control. This is where Criterion comes onto the scene — a library for statistical benchmarking that transforms the performance measurement process from a rough "stopwatch" into a scientific experiment.

Why is this library needed? Standard benchmarking in Rust (via #[bench]) is useful, but it has a fundamental flaw: it does not account for statistical variance. Modern processors have complex architectures with caches, branch predictors, and turbo boost. The same code can execute in different amounts of time depending on operating system "noise", CPU temperature, or random cache state. Criterion solves this problem by running each test many times, collecting a sample of execution times, and applying statistical methods (mean, median, standard deviation, percentiles) to it. The result is not just "time in microseconds", but a confidence interval that says: "With 95% probability, the true execution time lies in the range from X to Y".

Furthermore, Criterion provides powerful tools for regression analysis. You can compare the performance of two versions of your code (for example, before and after optimization) and get a statistically significant answer to the question: "Did the new version become faster, or is the difference within the margin of error?". The library generates beautiful HTML reports with charts, histograms, and tables, making performance analysis visual and compelling. This is an indispensable tool for developers of libraries, game engines, databases, cryptographic algorithms, and any other code where every nanosecond counts.

Installation

Installing Criterion in a Rust project is done via the cargo package manager. You need to add a dependency to your project's Cargo.toml file. It is recommended to add it as a dev-dependency, since benchmarks are not part of the final application.

[dev-dependencies]\ncriterion = { version = "0.5", features = ["html_reports"] }\n\n[[bench]]\nname = "my_benchmark"\nharness = false

Explanation:

\n
• version = "0.5" — specifies the library version. At the time of writing, version 0.5.x is current.
\n
• features = ["html_reports"] — enables generation of HTML reports with charts. This is an optional but very useful feature.
\n
• [[bench]] — a section that tells Cargo we have a benchmark named my_benchmark.
\n
• harness = false — disables Cargo's built-in benchmark runner. This is necessary so that Criterion can take control.
\n

After this, create a file benches/my_benchmark.rs (the path must match the name specified in [[bench]]).

Quick Start: Minimal Working Example

Let's write a simple benchmark that measures the speed of calculating the sum of numbers from 1 to 1000.

// benches/my_benchmark.rs\nuse criterion::{black_box, criterion_group, criterion_main, Criterion};\n\nfn sum_up_to(n: u64) -> u64 {\n    let mut sum = 0;\n    for i in 1..=n {\n        sum += i;\n    }\n    sum\n}\n\nfn bench_sum(c: &mut Criterion) {\n    // black_box prevents compiler optimization\n    c.bench_function("sum 1000", |b| b.iter(|| sum_up_to(black_box(1000))));\n}\n\ncriterion_group!(benches, bench_sum);\ncriterion_main!(benches);

How to run:

cargo bench

By