Tweets on-demand: analyzing what they're saying, as they say it

You may remember that earlier this year, we reported the results from our survey, which found that almost two-thirds (63%) of the people who responded said they are "undecided" about the value of data collected from social media sites like Twitter and Facebook to help them understand more about their organization or customers. Another 23% called social media "overrated," saying "there are not as many customer conversations going on as the media would have us believe." Only 14% of the survey’s respondents said they wanted to incorporate data from Twitter and other sites as part of their ongoing data analysis efforts.

We made the point at the time that social media was producing huge amounts of real-time data that would need to be incorporated into any analytic work, of things such as consumer sentiment.

Guess what. Google’s already there. It’s just launched what it calls its “Realtime Search,” which “lets you see up-to-the-second social updates, news articles and blog posts about hot topics around the world.” So, if I’m a large network/cable provider like Comcast, I now have another valuable tool that allows me to see what people are saying about my service. For a company that’s had its share of unhappy customers who’ve taken to Twitter to vent and let others know their feelings, this can be an invaluable tool. (A typical Tweet reads, “You know what's annoying? Paying to have @comcastcares install internet in your new apartment and IT NOT WORKING CORRECTLY. Die.”)

“Sentiment analysis” is a relatively new idea, but guys like Seth Grimes have already thought through its importance, in articles like this. He argues that such analyses will require a deep dive, and significant technical focus to cover areas such as “the ability to see beyond keywords,” and understand the context in which they’re being used.

Well, that’s going to require far more than bright people capable of going through lots of data. That’s going to require a business intelligence deployment, with ultra-powerful databases and analytic software, capable of analyzing data on the fly, even as it’s incorporating more data. (In storage circles, they refer to it as CDP: Continuous Data Protection. We could easily refer to it as CDI: Continuous Data Infusion.)

The tools to do this are here, and they’re getting more and more powerful. That’s good news, because even though most of the BI professionals who answered our survey weren’t sure as to whether the analysis of Tweets and Facebook posts was important, the means to immediately gather the information is here. And the demand for its analysis won’t be far behind.

Moving beyond beer and diapers

As long as Kognitio has been involved in data warehousing and business intelligence, we have repeatedly heard references to the old “beer and diapers” analogy. The analogy is apocryphal, and goes like this: a supermarket chain did a market basket analysis piece of work and discovered that sales of beer and diapers rose sharply in parallel on Fridays, heading into the weekend. Puzzled, the analysts did some further study, and discovered that new dads going to the store to pick up beer for the weekend were asked by their wives to bring home some diapers as well for their children. So, by locating the two items near each other, the chain was able to drive sales of both by some considerable percentage.

This story was even cited by the Financial Times almost 15 years ago, which gives you an idea of how long it’s been around, and how pervasive the lesson has been: use data mining to uncover patterns that will help you do business more efficiently and profitably.

Of course, the big retailers have known this for years, and have deployed data mining and analytical technologies to help them achieve these goals. Of course, they could: this kind of data analytics has traditionally cost a significant amount, and it’s been well out of the reach of most mid-tier companies.

Which is why the concept of BI delivered as an outsource service, or BI SaaS, makes a huge difference; mid-tier companies that previously would not have been able to access or implement such a piece of analytical work can now analyze the information that makes their companies tick, and discover insights they may not have even thought to ask about, let alone seek out. And as with any SaaS offering, the costs of capital expenditures are slashed, uptime is maximized, and the ability to rapidly quantify a return on the investment is present. Kognitio’s DaaS is one such BI SaaS offering.

Businesses have moved quite a long way past the beer-and-diapers analogy; today, we see companies using data analytics and business intelligence to monitor how effective their TV ads are in a given ZIP code in virtual live-time, based on POS data from supermarkets, or change insurance rate quotes in a fraction of the time previously required. And while companies like these may not be behemoths, they are actively taking advantage of a far more powerful BI infrastructure available at a far lower cost.

We’re proud to deliver the kind of business advantage that allows companies to get more from their data.

When it comes to BI and analytics - plan, plan and plan again

In our line of work, it has become almost a given to suggest that the increased power of business intelligence and data warehousing systems will make it possible for workers throughout the enterprise to access its advantages. No more, we’re told, will a company’s BI capabilities be limited to the “men in white coats”, the power users with answers being sent down to the frontline workers at their whim and whenever they please. Instead, people will have the answers they need, when they need them, so that they can do their jobs more quickly and far better.

Now, granted, the ability to move more quickly is a prerequisite for any successful business these days. And the ability to deliver key information in a timely manner is part of that requirement. Our Data Warehousing as a Service (DaaS) enables precisely that ability; we’re proud that we can deliver answers up to 85 times faster than some of the incumbent systems we’re replacing.

But is that the most important thing? Instead, there’s a basic business issue that should be addressed before you deploy DaaS as a solution: Who needs the information? Why do they need it? And exactly how much better will they be able to do their jobs by getting the information more quickly?

Just because you can do something better doesn’t always mean the older way was necessarily bad. A business intelligence system is not a solution unto itself; if you’re not installing it to answer a clearly defined business challenge, you’re setting yourself up for a massive failure: the kind where CFOs look at the costs and ask, “Exactly what are we getting for this money?”

Even more, you need to balance the ability of systems like DaaS to provide widespread insight to thousands of workers concurrently, with the needs of the information security guys. They’ll tell you about the theory of “least access,” that workers should only have access to the information they need to do their job.

Here’s our suggestion: if you determine that your company has a need for a BI system, or if you find that the one you have isn’t doing the job you need, get your entire team together. Map out an implementation strategy that answers your clearly defined business needs. Do a POC (proof of concept) first. Then, and this is important, do a POV (proof of value) that spells out what you’ll truly get in return for the money and time you’re investing. Then determine whether you’d rather have an on-site appliance or if you can outsource the ability in a DaaS configuration.

Only by answering those questions fully and by planning as a team will you have a comprehensive solution that everyone is happy with.