What I Learned from Making a Fake Product

I originally envisioned BreakStreak helping me and my friends be better about getting into a routine such as cooking more, working out, running, etc.  Rather than build a full fledged product, with a mobile app, complex reminders and check ins, and payment processing, I thought it’d be great to run an experiment first.  BreakStreak.com makes its users think they’re using a real product, complete with all sorts of fancy features.  But really they’re using a totally empty, fake product that’s tracking their usages in the background.  Cruel, I know, but this experiment let me disprove my original hypothesis that BreakStreak is a good implementation of my original vision.

The Product

BreakStreak is essentially a wizard that takes the user through a few steps.  The user is asked what routine they want to get into, along with a desired per-week frequency.  Then they’re prompted with a description of the product, and a few mechanisms for tuning their incentives — reminders, donations to charity if they fail, and guilt from friends.  Finally, they’re asked to create an account.  After creating an account, they’re directed to a page that explains this experiment, apologizing that BreakStreak isn’t actually a real service.

The Experiment

I wanted to see how far along in the wizard customers would get.  So I built the wizard such that only the user’s progress was stored, not the actual data associated with their routine.  For every single user I know exactly what part of the wizard they stopped at, what data fields they filled out (but not the values), and when they gave up with the wizard.

The Results

The results were very surprising to me.  Of 271 page visits, only 4 (1.5%) actually created a real routine and a real account (another 8 created test accounts).  Keep in mind, too, that those 271 page visits came from friends who found BreakStreak through my Facebook and Twitter posts — they weren’t a fair representation of the average consumer.

Furthermore, of those 271 page visits, 119 (43%) actually filled out the name of a routine (running, gym, etc.).  And of those 119, many were test routines such as “make sexy time,” “test,” and “masturbate.”  I built a little dashboard that models the funnel and the choice of incentive:

(You’ll notice in the chart that 12, not 4 finished.  8 of those finishes were tests, not real routines.)

Conclusions

My BreakStreak experiment proves two things: people want to be better about getting into a routine, and BreakStreak isn’t the solution.  The 43% conversion rate when prompted for a routine is very high.  However, the 1.5% final conversion rate shows that the product has fallen short of its original vision.

I built the BreakStreak experiment in about two days, whereas the real app would have taken me many weeks.  I was able to save tons of time otherwise spent on development by running an incredibly meaningful survey.  Instead of asking my users if they’d pay for something, I gave them a product experience that was real and let the results dictate their buying patterns.

I’m incredibly happy that I made the decision to run this experiment instead of wasting my time building the whole thing.  I’m stoked about the results, even with with (mostly) negative outcome.  Now I can rethink the original BreakStreak intention and either build a new/improved product or move on to my next idea.  (For now I’m choosing the latter.)

I’ve always been a fan of the Lean Startup movement, and this experiment drives home the importance of experimentation, at least for someone like me who’s relatively inexperienced building consumer apps.  Not to mention my sub-par design skills ;).