History of A/B Testing

Updated: Oct 1, 2021

“Experimentation is the least arrogant method of gaining knowledge.” — Isaac Asimov

Let’s lay the foundations of A/B testing with its definition and brief history.

A/B testing, at its simplest, is a way to compare two versions of something, in our digital product case it is a web page or an app, to figure out which performs better.

In A/B tests, users are randomly split between the two groups where one group sees the original version (Control) and the other sees the new version (Variation) as shown in the figure below:

During the tests, we collect the necessary metrics and in the end we prove or disprove our hypothesis by analyzing the data we collected.

How it all started


Scurvy was the leading cause of death among seamen in the 18th century. James Lind, a physician in the British Royal Navy, ran a test to prove his theory that citrus fruits could cure scurvy. For this purpose, he divided the 12 sick sailors into six pairs, and provided each of them with a different supplement in their diet: cider, vitriolic elixir, vinegar, sea water, two oranges and a lemon, or a purgative mixture.

As a result of what some have considered the first clinical trial in history, only the two sailors who took the citrus fruits improved. Of course it was a bit far away from today’s A/B testing standards and statistics since it was based on a small number of observations, but in the end the result was convincing and decisive enough since the impact was so dramatic.