Tweets on-demand: analyzing what they're saying, as they say it

You may remember that earlier this year, we reported the results from our survey, which found that almost two-thirds (63%) of the people who responded said they are "undecided" about the value of data collected from social media sites like Twitter and Facebook to help them understand more about their organization or customers. Another 23% called social media "overrated," saying "there are not as many customer conversations going on as the media would have us believe." Only 14% of the survey’s respondents said they wanted to incorporate data from Twitter and other sites as part of their ongoing data analysis efforts.

We made the point at the time that social media was producing huge amounts of real-time data that would need to be incorporated into any analytic work, of things such as consumer sentiment.

Guess what. Google’s already there. It’s just launched what it calls its “Realtime Search,” which “lets you see up-to-the-second social updates, news articles and blog posts about hot topics around the world.” So, if I’m a large network/cable provider like Comcast, I now have another valuable tool that allows me to see what people are saying about my service. For a company that’s had its share of unhappy customers who’ve taken to Twitter to vent and let others know their feelings, this can be an invaluable tool. (A typical Tweet reads, “You know what's annoying? Paying to have @comcastcares install internet in your new apartment and IT NOT WORKING CORRECTLY. Die.”)

“Sentiment analysis” is a relatively new idea, but guys like Seth Grimes have already thought through its importance, in articles like this. He argues that such analyses will require a deep dive, and significant technical focus to cover areas such as “the ability to see beyond keywords,” and understand the context in which they’re being used.

Well, that’s going to require far more than bright people capable of going through lots of data. That’s going to require a business intelligence deployment, with ultra-powerful databases and analytic software, capable of analyzing data on the fly, even as it’s incorporating more data. (In storage circles, they refer to it as CDP: Continuous Data Protection. We could easily refer to it as CDI: Continuous Data Infusion.)

The tools to do this are here, and they’re getting more and more powerful. That’s good news, because even though most of the BI professionals who answered our survey weren’t sure as to whether the analysis of Tweets and Facebook posts was important, the means to immediately gather the information is here. And the demand for its analysis won’t be far behind.