We are drowning in data. By 2025, we will be swimming in at least 175 Zettabytes of data. 90% of the world’s data was generated in the past 2 years!
At the same time, we are also trying to be more data-driven. Organisations are transforming themselves to figure out how best to utilise these treasure troves of data to help them analyse their processes, improve their performance, learn more about their customers, and even predict the future. However, finding the right insights is not easy. Here are some reasons why that is so:
1. There’s simply too much information
While the amount of information available to us has been increasing, our ability to process this information has not changed significantly. Drinking from a firehose is a common saying to describe how vast amount of information can easily drown and overwhelm us. We are living in a super-connected world where we are processing information from so many different sources: emails, Twitter, Facebook, Instagram, Youtube, Discord, Whatsapp, text messages, etc.
Let’s just use email as an example. On average, knowledge workers spend 20 hours a week reading and responding to emails. They check their email apps up to 100 times per day, and yet only 40% is considered crucial. As the internet empowers everyone to have a voice, the amount of content that is generated increases exponentially. As Internet of Things (IoT) devices become more prevalent, we will be generating data at an even higher rate.
2. There’s too much noise
In addition to having to be the recipient of so much information, we are also inundated by noise. These are irrelevant information that slowly whittles away our cognitive energy. As we process and filter these information, our ability to make good decisions depletes too.
In the context of insights generation, that means filtering out irrelevant information and downright incorrect information. The ability to identify the most relevant information to use and assess their credibility still lie on human cognition. As people have more access to generate and distribute data and information, it will become even more difficult to separate the chaff from the wheat.
3. Fragmentation
Data comes in all shapes and sizes, and formats and languages. Having access to a broad range of data helps avoid sample size bias and helps you generate a more objective and comprehensive view of the world. At the same time, data is often collected, processed, and presented differently depending on their use cases, sources, and time during which they are collected. Market size data collected for industry growth analysis may differ from market size data collected for production growth analysis. Unemployment data collected by the government may differ from unemployment data collected by academic institutions. The same type of data that is collected across two different periods may not be 100% comparable due to changes in data collection mechanism, sampling errors, and other external factors.
Often times, to generate truly meaningful insights, one may need to combine fragmented data from different sources. This process takes significant time and effort, requiring various steps to vet the data, standardise them before analysis can start.
4. Context
Once you are ready to generate your insights, you need context. You need to have a clear goal of what you are trying to achieve with the data that you have collected and cleaned. The same set of data can yield different insights depending on what you are using them for.
For instance, mobility data can be used to determine footfall traffic in a shopping mall if you are trying to understand the best location to open a new retail store. The same data can also be used to determine crowd clustering if say you are analysing for hotspots for a virus transmission. Having a clear goal at the beginning will help you generate insights that are relevant and contextualised to your business needs.
