New Deakin method for cracking down on review saboteurs and outliers

Deakin School of Information Technology Lecturers Professor Gleb Beliakov, Dr Simon James and Dr Tim Wilkin have developed an algorithm that sorts through ratings and eliminates outliers, putting together a much more accurate overall score.

"This would definitely be a way to make reviews better reflect what the real user experience is actually like - whether that's a review for a restaurant, a film, music or an app," Dr James said.

Dr James said users often assumed online reviews had been submitted in a genuine way, but unfortunately "that's not always the case".

"People can submit reviews to make a restaurant or a movie seem worse than it is, or on the other hand a lot of promotion can be done to try to make a restaurant or a movie look better," he said.

Dr James said the team's work produced more legitimate aggregated reviews by digging into a spread of ratings based on the highest density.

"A common way this has been approached in the past is you look at standard deviations - you have all of your reviews and you look at those on average and group those that differ from each other by about two points, then if something is three points or more away it can be classified as an outlier," he said.

"The problem with that approach is that if you have a lot of outliers they increase the standard deviation and then they don’t appear to be outliers anymore, because they're affecting the spread score as much as they're affecting the average score - the outliers have effectively masked themselves from detection.

"So what we do instead is measure where the densest majority of data is located and how spread out that majority is. That could be low or high, but it won’t be affected by outliers."

Dr James said the approach could help produce more accurate review aggregation on sites such as Yelp, IMDB or Rotten Tomatoes, though their algorithms were often closely guarded to prevent users from gaming the system.

He said one example of the "weaponisation" of reviews was the 2016 Ghostbusters re-make, when online trolls launched a campaign to give it the lowest score on film site IMBD - posting bad faith reviews when no one apart from a small pool of critics had seen the movie.

More recently, e-commerce giant Amazon was forced to remove hundreds of politically-motivated reviews of presidential candidate and former US Secretary of State Hilary Clinton’s campaign memoir What Happened, after it was revealed less than a third of early reviewers had purchased the book.

Dr James said cyber security experts were always on the lookout for "unusual review activity" such as surges of users logging on, or waves of reviews all being posted during the same short timeframe.

"If it's a book then sometimes more reviews have been submitted than copies could even have been sold, or at a restaurant you get 1000 opening night reviews even though the restaurant only has 20 seats in it," he said.

While Dr James and his colleagues' work could give a much clearer picture of the public consensus regarding a film, album or television series, he said we shouldn’t overlook individual opinion and subjectivity.

"Take any review with a grain of salt, approach those numbers with a healthy scepticism and don't let it discourage you from something that you might have thought was a good idea - it's always best to find out for yourself," he said.

The full algorithm and article "Robustifying OWA operators for aggregating data with outliers" will soon be published in the journal IEEE Transactions on Fuzzy Systems.