Sep 1, 2009
Why crowd sourcing doesnt work for most review sites
As wikipedia was being build, ‘crowd sourcing’ became a buzz word and the worlds entrepreneurs started to dream up ideas where the work of a thousand small participations could out-compete major companies. While this method for sure has been helpful for some projects (e.g. wiki, digg, openstreetmap) other projects like ‘user reviews sites’ (yelp, amazon reviews) are now getting to a point where obvious disadvantages are becoming clear. This is especially the case when used for review of restaurants, gadgets, hotels and so on. My three main concerns with user submitted content are:
1. One mans cool is another mans stuffy. Reviews work when there are clear and shared criteria for what ‘quality’ is – but that is often not the case. More often that not, I’ll check out a place online which will have e.g. 25 ‘5-stars’ and then 10 ‘1-star’. So what does that mean? That the place is an medium place? Probably not. Its most likely the case that people like different things. e.g. some like posh up-market places where others find them stuffy.
2. Secondly I’ll find that peoples willingness to express themselves when they have experienced something negative far outweighs people with good things to say. So I tend more for ‘neutral data’ rather than getting concerned if a place have a few ‘bad experiences’. For example, a good indications of a good youtube clip is often more ‘most viewed’ than ‘most popular’. (especially if most viewed include a qualifier like ‘most viewed with more than 75% of the show completed’)
3. Cheating. Multiple examples have been publicized where a company hire a group of freelancers to trawl around the web and writing positive reviews of their products or places. Making, at least I, pretty skeptical about the validity of user reviews.
For these three reasons, which in my view will only become worse, I think that user reviews needs new development. This can either be with the introduction of the social graph (e.g. only read reviews from people you know), using more passive data collection with qualifies (see post about behavior generated content) to make the data more objective or we invent new methods to highly reviews form people who are more like yourselves.