The Conversion Rate Optimisation Industry Sold You the Wrong Problem

Most organisations that have run CRO programmes will recognise this pattern. The agency presents a roadmap of tests. The first few produce wins. Button colours change, CTAs are repositioned, form fields are reduced. The results look good in monthly reporting. Then the wins become harder to find. The tests run longer. More of them are inconclusive. The conversion rate settles back toward where it was before the programme began, and the team debates whether to extend the contract or find a different agency.

Only 22% of A/B tests produce a statistically significant result at 95% confidence. The other 78% are inconclusive. The difference between variants is not large enough, or the sample is not sufficient to declare a winner. The CRO industry built an entire methodology around the 22% and sold it as a growth system. It is not. It is a test management process, and its relationship to business outcomes is considerably weaker than its relationship to test outputs.

This is not an argument against testing. It is an argument about what the testing should be designed to answer.

What the Industry Sold

The canonical CRO offer is a sequence of A/B tests aimed at surface elements of the website. Change the headline, change the button, shorten the form, remove a distraction, test a new layout. 58% of companies still make website changes based on opinion rather than data. CRO was positioned as the evidence-based, statistically validated, and repeatable alternative. That positioning was correct. The problem was that the evidence being generated was evidence about surface elements, not about why visitors behave the way they do.

Headline tests win most often at 31%, the highest win rate of any test element. Button colour tests almost never produce a real winner. This tells you something important about what actually drives conversion decisions, and what does not. A headline test wins when it better communicates the value proposition to the visitor. The headline is the surface. The message beneath it is the substance. Testing the headline without interrogating the message produces a finding about one variant of the surface, not about whether the message itself is the right one. ConversionTeam

The problem the industry framed was: how do we convert more of these visitors? The question that should have preceded it was: why are these visitors not converting, and is the reason something a page element can fix?

Those are different questions. They produce different programmes. The first one produces a test backlog. The second one produces behavioural intelligence that can inform the entire organisation.

The Part of the Journey Nobody Optimised

Standard CRO concentrated its effort on two surfaces of what is typically a six-stage customer journey: the acquisition landing page and the conversion form. Both are legitimate optimisation surfaces. Neither is where the majority of growth is lost.

Every visitor who arrives, reads, and leaves without converting made that decision somewhere in the experience. The question is where. Was it the acquisition experience, where the page failed to match what the ad had implied? Was it the website experience, where navigation friction made the information they needed hard to find? Was it the conversion experience, where an unclear value proposition or a trust deficit stopped the form being completed? Was it the activation experience, where the first interaction after converting left them uncertain they had made the right decision? Or the retention experience, where a gap between what was promised and what was delivered began to erode the relationship before the renewal conversation even started?

Each of these represents a distinct growth surface with its own friction points, its own evidence sources, and its own optimisation priorities. Most CRO programmes addressed one or two of them and measured the result as if it represented the whole.

The consequence is consistent. Conversion rate on the landing page improves. Lead quality does not change. CAC rises. Revenue growth stays flat. The optimisation worked on the metric it targeted. It did not touch the system that metric sits inside.

Growth is rarely limited by traffic. Most organisations that have invested heavily in SEO, paid media, social, and content have more traffic than they need to grow significantly. The limitation is almost always what happens to that traffic after it arrives.

What the Problem Definition Gets Wrong

Marketers who prioritise CRO are 3.5 times more likely to report revenue growth year-over-year. Brands running structured CRO programmes see average ROI of 223%. These figures are real. They also hide a significant caveat, which is that structured programmes and isolated A/B tests are not the same thing, and the outcomes are not interchangeable.

The difference is in what precedes the test. A structured programme begins with evidence. Analytics funnels, session recordings, heatmaps, voice of customer research, sales call data, support tickets. It uses that evidence to form a specific hypothesis about a specific friction point in the customer journey. It designs an experiment that tests that hypothesis directly. It measures both the business outcome and the behavioural signal, regardless of whether the experiment wins or loses.

An isolated A/B test begins with an idea about what might perform better. The idea might be informed by data, or it might be informed by what another company did, or what the designer preferred, or what the quarterly planning session decided to focus on. Tests under 14 days have a 61% false positive rate. They find winners that are not real. Without a minimum viable run time, without a clear hypothesis, without evidence-based prioritisation, the test result is as likely to mislead as to inform.

The deeper problem is what happens when the test completes. An isolated test produces a result and the programme moves to the next test. A structured programme produces a learning that changes the understanding of how customers behave, informs the next hypothesis, and compounds over time into a knowledge base that makes every subsequent experiment more likely to identify real friction and less likely to waste capacity on surface variation.

Most organisations have never treated the customer journey as a complete object of study. They have treated individual pages as isolated problems, applied localised solutions, and measured the result on a metric that does not always connect to the business outcome they were trying to change. This is not a failure of the people running the programme. It is a failure of the problem definition they were given.

What a Different Problem Definition Produces

The alternative framing does not abandon experimentation. It changes what experimentation is designed to answer.

Every assumption in an organisation about why visitors behave the way they do is an assumption. Most decisions about what to change on a website, what to test, what to prioritise, are made on accumulated opinion. Some of those opinions are accurate. Some are wrong in ways that would be obvious with 15 minutes of session recording review. The difference between a CRO programme and a growth optimisation programme is that the latter starts from the principle that every assumption carries risk until it has been converted into evidence, and that evidence should precede the hypothesis, not follow from it.

The client work that produces the strongest and most durable results begins with a question, not a test. Why are visitors who match our ICP leaving the pricing page without converting? What do users do immediately before they abandon the form? What search terms are people using who then do not find what they came for? The answers come from the data that is already being collected and mostly not read: session recordings, funnel drop-off reports, search queries, form abandonment data, voice of customer.

That evidence produces a prioritised understanding of where friction exists and why. That understanding produces a hypothesis. That hypothesis produces an experiment designed to validate or refute it. The experiment produces a learning: not just a result, but a piece of reliable evidence about customer behaviour that informs everything downstream.

Testing velocity is the single strongest predictor of CRO programme success. Brands running 24 or more tests per year see three to four times the cumulative improvement of brands running fewer than ten. That finding is accurate and important. It is also incomplete without its prerequisite, which is that velocity without evidence generates a lot of inconclusive tests quickly. The velocity that compounds is the velocity of hypotheses built on strong behavioural evidence, tested with sufficient sample sizes, and measured against business outcomes rather than element-level metrics. ConversionTeam

The CRO industry was not wrong about experimentation. It was wrong about what experimentation is for. Testing is a method for generating reliable evidence about a question worth asking. The question worth asking is not which button performs better. It is 

1. Why the visitor who arrived with genuine intent to solve a problem did not find what they needed.
2. What changes, across which surfaces of the journey, would make that outcome less common.

That is a considerably harder problem than a button colour. It is also the actual problem.

Sources: VWO and Optimizely research via roast.page April 2026 · Marketing LTB CRO Statistics 2026 · DRIP Agency Experiment Database 2026 · ExperimentHQ CRO Benchmarks 2026 · ConversionTeam A/B Test Win Rate Audit 2026 (2,288 tests)