1 March 2014

The Big Data Revolution: What Happened to Data Quality?

There's a wonderful irony in the world of Big Data Analytics. At a time when interest in Big Data appears to be growing exponentially, it appears that some are forgetting the fundamental challenges of Data Quality. A quick Google Trends analysis highlights the point. The chart below shows two trend lines extracted from Google Trends. The line in blue reflects the popularity of searches for Big Data, while the line in red shows the popularity of searches for Data Quality. It is important to note that the lines show relative popularity, not absolute volumes of search terms. In fact, Google Keywords suggests that in absolute terms searches for Big Data are about 20 times as popular as searches for Data Quality.

This raises an interesting question - what's happened to Data Quality? At a time when organisations are becoming ever more interested in using their data to create performance insights and predictions, interest the Data Quality appears to be declining. Is this because Data Quality is no longer an issue?

I don't think so. On three separate occasions in the last week alone I have been involved in discussions with senior managers from some of the world's leading manufacturing and service businesses. Each time, the issue of Data Quality has come up loud and clear. These firms recognise the potential of Big Data and Analytics, but are realistic enough to know that unless they sort out their data fundamentals - unless the track the right things and make sure the raw data if accessible and of high quality, all of the Big Data Analytics in the world is not going to help them. That's why - in the Cambridge Service Alliance - one of our projects this year is focusing on creating a data diagnostic - a methodology that can be used to check whether the data you have access to is appropriate and can be better used to optimise the delivery of your services and solutions. We're in the process of testing this data diagnostic at the moment and would love to hear from you if you'd be interested in being one of the pilot test sites.

1 comment:

  1. "What's happened to Data Quality" I hear you ask. Having worked for years in academia and now in industry, my answer is simple: nothing HAPPENED to data quality... it never was there! Data quality is an engineering construct where one can design and control data acquisition, collection and storage. In the realm of Big Data, this has been, and will remain, a mere pipe dream. There will always be incompatible and inconsistent data collected by different parties in the same organisation, even if it is simply because different people needs different type of data and therefore collect them accordingly. In the simplest case, data coming in from different sources has different level of fidelity and therefore needs to be reconciled. So, yes, every organisation I have been in touch with realises the importance of data quality, but I believe it will remain an illusive goal for the short to medium term.