Sorry, but I'm slightly obsessed with this. I'll try not to math it up or anything…
Online reviews aren't always a meaningful measure of the consumer experience for a number of reasons that I obsess over constantly. They are useful for one useless purpose: determining whether or not a given place of business is still in business. It's also useful to select professionals because, in the case of restaurants, it can be used to track food poisoning outbreaks and shut down the worst offenders.1)
Idiots are the biggest problem out there. There are two types: the first fails to understand the question, the second fails to understand the product being sold. A big problem that Amazon, eBay, Jet, and other online retailers have is that the user is reviewing the service and not the product. We'd do well to just automatically remove any product review that has the word “shipping” in it. Of course there's no getting around the insistence of idiots to participate, so the form should clearly designate separate areas for the product and shipping/handling. Others who come to look at the product can then know about the product and also about the shipping. The other idiot doesn't understand the product, and by that I mean, they ordered the wrong thing, didn't read the specifics, and take their anger out on the product they specifically ordered. This is the hot-headed idiot, a fragile animal that has to be treated with care. It's hard to automatically sort these out, because users can legitimately receive the wrong product. Remember, we're dealing with idiots too and they may not consider contacting customer support to sort things out first, but just take getting burned in stride, and yell about it in the reviews.
Fake reviews are a minor problem by comparison. Glowing praise among a sea of bad reviews is suspect and usually in place just to bolster a rating. It can come from various mysterious sources, none of which particularly concern me. It's easy to automatically figure these out: one-time registrations that pimp one establishment or product, or serial high scorers that have no geographic ties to the establishment in question or no verified purchase of the product (or any product, ever). Counterfeits are usually easy to spot. High-quality counterfeits are another problem, but still, automation can find them: verified reviews of establishments in town with short and sweet comments, but long-winded praise (or hate) for one outlier, usually registered right before the praised establishment opened for business.
The scale problem is well known. Give people five stars, they will generally choose one or five, ignoring the others in between. You can't just give people two options though, because you have to give people a choice in order to coax them into participating and how else would you graphically represent the like/dislike ratio?2)
Mediocre experiences are important too but never register in any form of customer feedback. There are crazy people in the world that will review every cup of coffee they drink. That might be valuable to some, but given the type of person prone to doing such a thing, it's not particularly valuable to the product. That crazy person is going to drink coffee every day and we already knew that. The consumer we're trying to reach is the one that tries the new coffee, thinks it's only “OK”, and either buys it until they're bored with it, or never buys it again. We'd like to know why it's merely “OK”, but people are generally less-inclined to actively review products unless they love or hate them.
Fanboys are noisy and pollute reviews. There are tons of examples, but I like Crystal Pepsi, because Crystal Pepsi is gross. The market proved that point, but the damaged and perverted people who like it won't ever shut the fuck up about it. In general, the Internet has this problem where management tends to think of it as a replacement for fieldwork. Rather than actually have a focus group and testing products with real people, it's now common practice to just listen to social media… of course, not everyone uses it, but it's so cheap! I've got a whole article about that too…
Confirmation bias pollutes everything from the other direction. The maker of any product is more-likely to focus on positive reviews and ignore the negative. They too are aware of the pollution caused by idiots, but can't be convinced about fanboy pollution.