Testing is More than Just A/B

February 26, 2018 • 1 min read

Colette Nataf

Founder & CEO


I was a panelist in a discussion by Branch Metrics last spring on A/B testing. One of the other members was from Instagram. She described the perfect testing environment - randomly sample 1% of the total audience. Sort that group into experiments where one variable (and that variable only) changes. See what engagement and other metrics look like. Analyze, decide, and continue.

That’s all well and good if you’re Instagram. But what about the rest of us? When you’re working your way up from nothing, starting with a testing sample of 1% is impossible.

I learned about experimental design and testing in my undergraduate days as a research assistant for a quantitative psychology lab. We would develop a hypothesis, create a test around that hypothesis, recruit participants and then analyze the results and hopefully find something interesting.

And then the real world hit. I was at a startup where we were testing email messages. I remember the first test,  just the title. It was me saying “lowercase vs title case?” and the rest of the team saying “beach vacation or tax audit?”

A/B testing is the crux of experimentation, but we’re doing something more than that in the world of startups. We’re drawing on a blank canvas, throwing spaghetti on a wall and fishing for insights. And it works.

We experienced this the most while creating and running ads for Live Fantasy TV. We started from nothing, so we tested everything.

alt text

The winner? Margaritas on the beach. I can’t tell you why, but I can tell you it gave us a starting place. We could take the image, learn from it and create dozens of similar ads to iterate on the same concept. We were able to win — because we tried everything.

At the end of the day, testing for startups will always be about more than building a perfect experiment. It’s about the same values companies are founded on — risk, intuition and math.

Share this post

Other posts you might like


What is a good CPM rate?

Colette Nataf

March 01, 2018 • 1 min read

Cost per mille (CPM), is a commonly used measurement in advertising. It refers to the cost an advertiser pays for one thousand views or impressions of an...