December 16, 2011

Tales of Split Testing

Split testing is a way of improving the performance of your site. The main focus of split testing tends to be literally on conversion i.e. how successful the checkout process is or the colour of the buy now button. There are other ways that split testing can be used from testing an algorithm or tweaking the price of a product. Here are some examples of how some companies use it. This all ultimately leads to more conversions and perhaps better retention.

Amazon Shopping Cart

Amazon makes 20 – 30% of its sales from recommendations. You could say that Google itself is one great big recommendation system. Greg Linden is credited with inventing the system at Amazon. He had an idea to suggest other items that people might want to buy when checking out on Amazon. He spent some time working on a prototype. The general reaction was positive but senior vice president wasn’t too happy. Greg was told not to work on the project anymore. He tested it anyway. The senior vice president was very happy but it was hard to stops test at Amazon. Because of the results the shopping cart recommendations was launched.

41 Shades of Blue

This is a now famous story about Marissa Mayer employee number 20 at Google, currently the Vice President of Location and Local Services. The story goes that a designer had chosen a colour but a product manager tested a different colour. To solve this situation Ms Mayer picked a colour half between the two to use. She also got them to test 41 different shades to see which colour the users preferred. This reliance on testing at Google as one the reasons why the designer Douglas Bowman who created the ground breaking hotwired site left Google in 2009

Beating the pirates

Valve Corp makes computer games. They have a games distribution system called Steam. This is similar to ITunes but distributes games and associated media. This means they have complete control of their online distribution system. The games industry has an ongoing issue with piracy and different companies approach it is different ways. Some companies include a DRM system that makes users register their product online – this can cause issues for legitimate users and in some cases has led to the companies themselves producing DRM removal tools. Valve looked at the problem and decided that instead of adding DRM to their products – offering a better service was the key. They found that by running tests on the prices of their products the amount of income did not change and in some cases went up when they reduced the price.

You might like this

Netflix are a DVD rental/video streaming company in the US.  They are probably most recently know for the rise and fall the Qwikster where they planned splitting the DVD rental and stream parts of the company into two. Qwikster would have been the DVD rental company and Netflix would have remained the streaming service. Due the unpopularity of this idea amongst their users Netflix scrapped the idea.  Neflix do a lot of ab testing. They test interfaces, they test algorithms, they test database queries.