Big Data Veracity and the Value of Doubt | NewSci
Big Data Veracity and the Value of Doubt
The fourth V of Big Data – Veracity – is not talked about a lot perhaps because it brings up the uncomfortable reality of the inherent uncertainty of data, especially Big Data. The more data sources you access, and the greater the variety of data types within those sources, the higher the probability for error becomes.
This uncertainty creates doubt which leads some to conclude it is better to work with a smaller set of more reliable data. Others conceive of new ways to ensure the quality and accuracy of all the new data they collect. The first approach gives you the same old answers to the same old questions. The second, while commendable, is a quixotic exercise doomed to fail.
“Doubt is not a pleasant condition, but certainty is absurd.”
Voltaire
The third path is to embrace the doubt, while at the same time taking reasonable steps to improve the quality of the data being collected. Living in Florida we rely on this type of thinking every year during hurricane season. Is it going to be an El Niño or La Niña year? Keep in mind this is the temperature in the Pacific Ocean being used to predict hurricanes in the Atlantic. Talk about a concept filled with potential for uncertainty.
I can only imagine the arguments between the people who wanted accurate data, and those who believed expanding the data universe would help predict one of the most costly weather events on the planet. Why did the weather people finally embrace doubt?
You can start with the catastrophic nature of the problem. People tend to be more creative when lives are at stake. I don’t believe it is a coincidence cancer researchers were early adopters of Big Data. They could make the life and death claim to get executives (and IT) to look past their doubts.
For those of us (which thankfully are most of us) who do not have the life and death leverage, look for problems with potential to dramatically impact the health of your organization. Retention of key relationships (donors, students, members, patrons, volunteers, et al) is a good place to start. Our first NewSci Lab is focused on this very topic in no small part because we know it is of critical importance to our clients.
Data has never been perfect, and certainty is a temporary illusion. Big Data did not create the imperfections or the uncertainty. It has only shined a technological light on it. Rather than run from what we can now see, let’s seek to harness its power to give us the knowledge to make better decisions and take smarter actions.
“Wonder rather than doubt is the root of all knowledge.”
Abraham Joshua Heschel
1 Comment
Source article:
Big Data Veracity and the Value of Doubt | NewSci
