A/B Testing in C++: How Not to Break Production and Find the Truth

Why A/B Testing Is Not Just About Web and Marketing

When we hear "A/B testing," landing pages, "Buy" buttons, and Google Analytics conversions come to mind. But what if your product is not a website, but a high-load C++ server, an embedded system, or a game engine? Can A/B tests be applied here? Absolutely! And it's not just a trendy term, but a real tool for making data-driven decisions rather than relying on intuition.

In C++, A/B testing helps answer questions like: "Is the new sorting algorithm really faster?", "Is it worth introducing a thread pool?", "Will the new serialization format improve API response time?". Without an A/B test, you risk implementing an "optimization" that turns out to be a regression. In this article, we'll break down how to properly embed an experimentation mechanism into C++ code without turning it into a mess of if-else statements.

A/B Test Architecture in C++: Without Pain and Suffering

The main danger of A/B testing in C++ is cluttering production code with conditional operators. To avoid this, you need to separate the experiment logic from the business logic. We use the Feature Flag pattern and a strategy. Let's look at the basic architecture.

Step 1: Centralized Configuration

Store all flags and experiment parameters in one place. This could be a JSON file, a database, or a remote config server. For simplicity, let's use JSON.

// experiment_config.json\n{\n  "experiments": {\n    "sort_algorithm_v2": {\n      "enabled": true,\n      "variants": {\n        "control": 50,\n        "treatment": 50\n      }\n    },\n    "memory_pool_size": {\n      "enabled": true,\n      "variants": {\n        "default": 33,\n        "pool_256": 33,\n        "pool_512": 34\n      }\n    }\n  }\n}

Traffic split is specified as percentages. In C++, we load this config once at application startup.

Step 2: Experiment Manager Class

Let's create a thread-safe ExperimentManager class that determines which group an object falls into based on a key (e.g., user ID or transaction ID).

#include <string>\n#include <unordered_map>\n#include <random>\n#include <shared_mutex>\n#include <nlohmann/json.hpp> // for JSON parsing\n\nclass ExperimentManager {\npublic:\n    explicit ExperimentManager(const std::string& config_path);\n    \n    // Returns the variant name for a given experiment and key\n    std::string getVariant(const std::string& experiment_name, \n                           const std::string& user_id) const;\n\nprivate:\n    struct Experiment {\n        bool enabled;\n        std::vector<std::pair<std::string, int>> variants; // name, weight\n        int total_weight;\n    };\n    \n    std::unordered_map<std::string, Experiment> experiments_;\n    mutable std::shared_mutex mutex_;\n    \n    // Deterministic hashing for stable distribution\n    size_t hashAndMod(const std::string& key, int mod) const;\n};\n\n// Implementation (simplified)\nstd::string ExperimentManager::getVariant(const std::string& exp_name,\n                                          const std::string& user_id) const {\n    std::shared_lock lock(mutex_);\n    auto it = experiments_.find(exp_name);\n    if (it == experiments_.end() || !it->second.enabled) {\n        return "control"; // fallback\n    }\n    const auto& exp = it->second;\n    size_t bucket = hashAndMod(user_id, exp.total_weight);\n    int cumulative = 0;\n    for (const auto& [name, weight] : exp.variants) {\n        cumulative += weight;\n        if (bucket < cumulative) {\n            return name;\n        }\n    }\n    return "control";\n}

Important: Use deterministic hashing — the same user always ends up in the same group. This is critical for test correctness.

Practical Example: A/B Test of a New Algorithm