Social Listening and Flawed Sentiment Analysis – Don't Throw Baby …
I was giving a Pulse Analytics demonstration last week, and it got side-tracked with a deep discussion on the flaws of sentiment analysis. To set further context, this gentleman was one of the most intelligent people I’ve met in recent memory. He nailed everything that is currently wrong with Natural Language Processing (NLP) – with slang and context two of the biggest stumbling blocks. He finished his outstanding explanation to the other people in the room, and then he looked at me.
My response: You are absolutely right, sir.
As part of Pulse Analytics ongoing development, we periodically kick the tires on different text analysis solutions. And although they continue to get better, none of them are infallible. Let’s start with slang first. Guess how sentiment analysis would score the following:
Oskar Blues Ten Fidy is the F****** Bomb!
Sentiment analysis would see “bomb”, and it would see the profane adjective, and it would score the sentiment as strongly negative. But we have to evaluate the tone of the author, and the preferred communication style of the social channel, and conclude that this brand mention is actually strongly positive (I concur, Ten Fidy is a great Russian Imperial Stout beer). In contrast, how should sentiment analysis score the following?
Related Resources from B2C
» Free Webcast: Native Advertising – What Is It & Why Content Marketers Should Care
My employer dropped the F****** Bomb on me – I got laid off. Side note: Now replace “I got laid off” with “I got the promotion”.
What’s a poor sentiment analysis engine to do?!? Now, let’s look at a less profane example of context from an actual Pulse Analytics collection:
In this case, there is zero context. The post literally was “sick shot”. By default, the sentiment analysis engine scored the document as strongly negative. However, clicking on the document from within Pulse Analytics takes the user to the original source document. By viewing the picture posted to Instagram, and the context of the other comments, we can conclude that the original author was paying a compliment to the photographer! Let’s stick with the “sick shot” phrase for another example:
We have more context in this example, and the sentiment analysis engine “correctly” relates “sick shot” to the word “awesome”. But what if we were monitoring for the opposing team – is the score still correct in regards to brand health? Even if a true hockey fan has an appreciation of a “sick shot”, regardless of who scored the goal, we may not want this document to skew the sentiment results of our overall brand health. In this case, we may elect to mark them as Neutral or Exclude them from the sentiment score.
These are just a few examples where sentiment analysis can go awry. But do we throw away a social listening strategy (the baby), or even the sentiment scores for the entire collection, based upon a few exceptions (the bathwater)?
My answer is a resounding no!
Social listening is about so much more than a sentiment score. People are talking about your brand. Do you know…
WHO is talking about you?WHAT they are saying?WHERE these conversations are?WHEN they prefer to engage on social?WHY they are talking about your brand?HOW they feel as they are talking about your brand?
Sentiment analysis just covers that last point! There is a lot of actionable intelligence in those other 5 points even if you never considered the sentiment score. And that actionable intelligence includes allowing the brand to find the conversations and engage the authors with the appropriate context to improve their customer experience.
One final note: I still see value in sentiment scoring that is mostly accurate. It gives the brand a headstart on judging overall brand health. Pulse Analytics even allows users to change the sentiment on an individual basis (like my exclusion of the word Johnson in the screenshot above) or en masse. I could search for the string “sick shot” and change the sentiment for all mentions of the phrase to be positive, about 1,200 for this example, with just a few clicks.
So what is your stance on brands needing to adopt a social listening strategy? What are your thoughts regarding sentiment analysis?
I love my wife and two daughters. I am blessed in that I also love my job as a principal and EVP of the Rocky Mountain Region for Mantis Technology Group. I am very excited about our Pulse Analytics offering for social media listening and analytics. I try… View full profile
This article originally appeared on Brian Vickery – Social Media Sport Analogies and has been republished with permission.
Find out how to syndicate your content with Business 2 Community.
Related White Papers
The Final Hire: How To Complete The Hiring Process6 Steps to Successful Lead NurturingTracking Phone Leads: The Missing Piece of Marketing Automation
Recommended Reading
Continue reading:
Social Listening and Flawed Sentiment Analysis – Don't Throw Baby …
