Everyone loves a good experiment. You get to try new things. You gather data. You learn. Sounds great, right? But there’s a catch — testing everything can actually cause more harm than good.
Let’s talk about why smart experiments need smart rules. And more importantly, let’s dive into why an Experiment Review Board might be your team’s new secret weapon.
What’s an Experiment Review Board?
An Experiment Review Board (ERB) is a group of people who review and approve experiments before they go live. They ask questions like:
- Is this experiment safe?
- Does it align with our goals?
- Is it ethical?
- Do we even need to run this test?
Think of them as science fair judges — but cooler, smarter, and better with spreadsheets.
The Problem with “Test Everything”
In many teams, the approach is simple: if you can test it, you should. But this fast and loose strategy backfires. Here’s why:
- Your users become guinea pigs for half-baked ideas.
- You waste time testing things that don’t matter.
- Your data gets messy and hard to trust.
- Your team gets overwhelmed with low-impact work.
It’s like cooking spaghetti by throwing every possible ingredient into the pot. Just because you can add pineapple doesn’t mean you should.
Why ERBs Save the Day
Adding a review board to your experiment process takes a little extra time. But the benefits? Huge.
Here’s what an ERB helps you do:
- Filter out bad ideas – Not every idea deserves data.
- Focus on priorities – Run tests that actually support your business goals.
- Avoid user fatigue – Don’t bombard customers with tiny, pointless changes.
- Stay ethical – Some experiments cross lines they shouldn’t.
- Learn faster – Fewer tests means more clarity on results.
Instead of being the team that runs 100 mediocre experiments, be the one that runs 10 truly brilliant ones.
Signs You’re Stuck in the “Test Everything” Trap
- You test button shades even when conversion hasn’t changed in months.
- You have a long list of “inconclusive” test results.
- Your customers are part of 5+ experiments at once.
- Your team forgets why a test was even started.
- You hear “let’s test it” more than “what’s the goal?”.
If this feels familiar, don’t worry. You’re not alone. Many teams fall into the trap of testing for testing’s sake.
But remember: not all ideas need validation through an A/B test. Some need strategy. Some need common sense.
What a Good ERB Process Looks Like
Ready to set up your own board? Here’s a simple, fun process.
- Gather a group — usually product managers, analysts, designers, maybe someone from ethics or legal.
- Set meeting times — weekly or biweekly reviews work well.
- Require an experiment brief — include hypotheses, success metrics, and potential risks.
- Score or vote — use a checklist or thumbs-up system to approve or delay tests.
- Document decisions — write down why something was approved or rejected.
It’s not about control. It’s about clarity. Everyone wins when we’re deliberate.
Make it Easy for Everyone
People hate extra work. So keep the ERB process light and friendly.
Use fun forms. Share results in Slack. Celebrate tests that had high impact (or dramatic fails with great lessons).
And most of all — train your team. Help them understand what makes a good experiment. The better they get, the fewer ideas end up on the cutting room floor.
Here’s a mini checklist your team can use before submitting a test idea:
- Does this test tie to a real goal?
- Is the sample size big enough?
- If it fails or wins, do we know what to do next?
- Can this test affect lots of users?
- Would we be okay if users talked about this on Twitter?
Real-World Example: The Button That Broke Everything
Let’s say your growth team wants to test a new sign-up button style. Harmless, right?
But they forget to review the test. It goes live on the checkout page… but has a bug. The new button breaks for 20% of mobile users. Cart abandonment shoots up. Revenue drops.
Why? No one caught it during review. It was “just a button” — until it caused a crash.
If an ERB had seen that test beforehand, someone might have asked, “Has this been QA’d?” or “Can we soft launch to a tiny user group first?”
Disaster avoided.
From Chaos to Clarity
Let’s be honest — testing without a plan feels productive. You get to say, “We’re data-driven!”
But in reality? It often leads to confusion. Mixed signals. Team burnout.
With an Experiment Review Board, you flip the script. You make testing meaningful. Strategic. Focused. You learn faster while keeping users happy and your product stable.
Final Tips for a Great ERB
- Keep it small – 4 to 6 people is ideal.
- Set ground rules – Know what gets auto-approved vs. needs full review.
- Rotate roles – Share the responsibility and avoid bottlenecks.
- Review results too – Celebrate what worked. Learn from what didn’t.
- Keep leveling up – Tweak your process as your team grows.
The Big Picture
Being curious is great. But being deliberate is powerful.
Don’t fall for the “test everything” mindset just because it sounds smart or scientific. Be bold. Be thoughtful. Put guardrails around experiments so they actually help you grow.
Build an ERB — and go make magic with your data.