Beware Before You Book: Computer Program Helps Detect Opinion Spam in Online Hotel Reviews
February 03, 2012
By Stephanie K. Kelly In the world of online hotel reviews, finding an honest opinion can be about as rare as scoring a complimentary upgrade to the Presidential Suite. While we may think we’re pretty good at sniffing out bogus hotel reviews online, one group of Cornell researchers found that the average consumer might not be able to detect these “embellished truths” as easily as one might think. Claire Cardie, a professor of computer science at Cornell University, worked with a team of researchers who looked at 800 Chicago hotel reviews to find out just how accurate our perceptions are when it comes to online “opinion spam.” “We wanted to answer two questions,” Cardie explains. “One was, ‘Can people tell the difference between real and fake reviews?’ The second question was, ‘Can we build a (computer) system that can do better than people in telling the difference between real and deceptive reviews?’” To test this, Cardie and her colleagues first looked at positive reviews for 20 Chicago hotels on the popular travel website, TripAdvisor.com. The researchers collected and carefully verified truthful opinions, while also asking independent writers to create false positive reviews of the same hotels. Three human judges, undergraduate volunteers at the University, were then asked to assess the entries, based on which they believed to be truthful and which they thought were deceptive. “We wanted to answer two questions,” Cardie explains. “One was, ‘Can people tell the difference between real and fake reviews?’ The second question was, ‘Can we build a (computer) system that can do better than people in telling the difference between real and deceptive reviews?’”
The study found that participants did not perform much better than by chance, guessing the validity of the reviews with just over 50% accuracy. What’s more, the individuals could not agree on which entries they believed were true and which were false.
The researchers then set out to develop a computer program that could analyze the reviews and accurately identify truth and deception. They trained a computer to identify subtle features in the text that help indicate the validity of a statement. For example, truthful hotel reviews were found to be more likely to use concrete descriptive words giving spatial information such as: “bathroom,” “floor” or “the hotel.” Deceptive reviews, on the other hand, focused more on the reason for going on the trip, with words such as “business,” “vacation” or “my husband.” By looking at these keywords and analyzing the bigrams, or pairs of words, in the reviews, the computer was able to identify deceptive reviews with almost 90% accuracy. As Cardie explains, these results make for some exciting possibilities for future research in online reviews. “Given that these systems do work very well at detecting deceptive versus truthful positive reviews,” she says, “we thought that one direction would be to use that to try to determine how much deception really exists on these sites.” Using the program, Cardie says, “We’ll be conducting new studies to look at the prevalence of deception on TripAdvisor, where anyone can submit a review, versus Hotels.com- where you have to have booked through the site and stayed at the hotel in order to submit a review.” So be honest: How much trust do you put in online hotel ratings? Did your most recent vacation live up to the hype you heard on TripAdvisor? Or did your “Room with a View” bear an uncomfortable resemblance to the Bates Motel? In any case, you might want to check out the website that Cardie’s team created after the study. It’s called ReviewSkeptic.com. Here, you can paste the text from any online hotel review, and the site will use the methods used in the study to assess its truthfulness. Though the site is for entertainment purposes only, it’s certainly fun and a handy way to try your hand at a little online lie-detecting before your next vacation. Photo credits: Johan Larsson, Steve Snodgrass About the Author
Stephanie Kelly is Florida writer and web communications specialist with a passion for cooking, classical music, and the Oxford comma. She graduated with honors from the University of Florida, earning a Bachelor of Science in Advertising and a minor in Anthropology, and has since written for the websites of the Florida Museum of Natural History, Elfster.com, Fembody Nutrition and Gainesville Magazine. You can see more of Stephanie’s work at http://www.stephaniekelly.co.
Comments?
|
Search by typing & pressing enter