A/B Testing in Java: How to Know for Sure, Not Guess

Introduction: Why Your Application is Losing Money and You Don't Notice It

Have you ever wondered how many users drop off from your Java application because of a poorly designed "Buy" button? Or maybe the new recommendation system you spent a month on actually worsened conversion? In the development world, there are plenty of "obvious" solutions that fail spectacularly. The only way to stop guessing is through A/B testing in Java.

In this article, we'll break down how to implement a full-fledged A/B test directly in your Java application. You'll learn how to split users into groups, run an experiment, and most importantly — avoid statistical errors. No fluff, just code and practice. Let's go!

What is A/B Testing and Why Does a Java Developer Need It?

A/B testing (or split testing) is a method of comparing two versions (A and B) of an interface, algorithm, or logic to determine which one performs better. Simply put: you show 50% of users the old button and 50% the new one, and see which gets more clicks.

This is especially relevant for Java applications:

• Backend logic: Changing product sorting algorithms, recommendations, or pricing.
• APIs and microservices: Testing a new endpoint version on a portion of traffic.
• UI components: If your Java framework (e.g., Vaadin or Spring MVC) renders HTML.

The key idea: A/B testing in Java applications allows you to make decisions based on data, not intuition. This reduces risks and improves product metrics.

A/B Test Architecture: Think Like an Engineer

Before writing code, let's design the system. We need:

1. Variant assignment service — who goes into which group.
2. Experiment storage — where settings are stored (which experiment is active, what percentage of traffic).
3. Event collector — logging user actions.
4. Analyzer — an external tool or script for calculating statistics.

In this article, we'll focus on the first two points, as collection and analysis are topics for separate articles.

Randomization Service: Don't Let a Bug in random() Ruin Everything

The simplest way to split traffic is to use Math.random(). But this is unreliable: when the JVM restarts, a user might end up in a different group. We need deterministic randomization based on the user ID.

The best approach is hashing the userId and taking the remainder of division. This way, a user will always see the same version as long as the experiment is active.

public class RandomizationService {

    public String assignVariant(String userId, String experimentId, int splitPercent) {        // Use userId + experimentId for determinism        String seed = userId + ":" + experimentId;        int hash = Math.abs(seed.hashCode());        int bucket = hash % 100;

        if (bucket < splitPercent) {            return "B"; // New version        } else {            return "A"; // Control version        }    }}

Important: hashCode() can be negative, so we use Math.abs(). Also, keep in mind that the distribution won't be perfectly uniform with a small number of users, but for 1000+ users, it's fine.

Real Example: A/B Test on Spring Boot

Let's say we have an online store built on Spring Boot. We want to test a new pricing model: show a discount only if the user has been on the site for more than 30 seconds (version B) versus a regular discount (version A).