Does the potential of big data depend solely on technological advancements, or does the discernment and skill of the analyst hold the key?

Big data has great potential in the modern world, but technological advancements alone will not be enough to realize its full potential. The discernment and skills of data analysts are paramount, and collaboration from a variety of backgrounds will determine the success of big data utilization.

 

Just five years ago, big data was an unfamiliar concept known only to those in the know. At the time, the amount of data was exploding due to rapid advances in information and technology, but there was little thought given to how to effectively utilize this data. Traditional data analysis methods were unable to handle the sheer volume of data, and as a result, much of the data was left uncollected or used as simple statistics. However, with the emergence of the concept of big data, this situation began to change.
In 2012, the World Economic Forum named big data as one of the top 10 technologies, along with the Internet of Things, and it began to gain traction. This increased interest in big data has led to an increase in attempts to capitalize on it, including by businesses. Big data is not just a technological advancement, but a concept that has the potential to make a huge difference in society as a whole. With the growing realization that data can be used to understand and predict people’s behavior patterns, big data soon became a strategic tool for businesses.
In South Korea, the Seoul Metropolitan Government uses records of late-night taxi rides by citizens and KT call volume data to estimate the number of people on the streets. This is a typical example of the use of big data to operate late-night buses by suggesting the most suitable route map and frequency. In addition to this, various industries in Korea are experimenting and challenging the possibilities of utilizing big data. For example, in the healthcare sector, research is underway to analyze patients’ treatment records and lifestyle data to suggest customized treatment plans. As such, big data is breaking the mold of traditional industries and enabling new innovations.
The smart factory solution proposed by Samsung SDS is able to predict failures in advance by analyzing the factory’s equipment operation information and past equipment defect records. Nevertheless, the question “Has big data changed the world?” cannot be answered in the affirmative. Some argue that big data is mostly used within organizations and is therefore less visible. In addition, some experts point to data security and privacy concerns as reasons why the full potential of big data has yet to be realized. As the amount of data grows, so do privacy concerns, which can be a barrier to companies utilizing big data. Unless policies and technologies evolve to address these ethical concerns, the use of big data will remain limited.
However, according to a survey by the Korea Information Society Agency (KISA), only 4% of Korean companies are using big data, which suggests that big data is not being used in the enterprise. So, why is it that big data is not being utilized to its full potential? Is it because we lack the technology to manage and analyze big data? I believe that the reason for this problem is not the limitation of technology, but the lack of discernment and capability of people who utilize big data. This is connected to the problem of “information overload” that big data presents. Extracting meaningful information from vast amounts of data requires more than just technology, it requires insight. The intimidation of the word “big data” makes many people think that it requires a lot of technology. However, big data is just what it sounds like: tons of data, such as browsing history, social media, and CCTV footage. As long as you have a server to store it and software tools to analyze it, anyone can analyze big data. Of course, various technologies are being developed to store larger data more efficiently and process it faster, but that doesn’t make it better to analyze it. At the end of the day, it’s the role of the analyst who analyzes the data, determines what it means, and turns it into valuable information out of a chunk of data that would otherwise be worthless before processing. In the late-night bus example above, it’s just call volume data to someone else. However, depending on how you look at and analyze that information, you can get information about the flow of people, and through the flow of people, you can determine the optimal bus routes and dispatch periods.
This idea was confirmed when I interned at the Big Data Lab for half a year. The most memorable data I worked with as an intern was the KT free Wi-Fi access data of Seoul National University, which included the IDs of all devices accessing the Wi-Fi and the end time of the access time. Analyzing the data itself was not a difficult task. The problem was the lack of a new way of looking at the data. The biggest challenge in big data analysis is deciding what to analyze rather than how to analyze the data. I succeeded in finding out the population of Seoul National University through this data, but I couldn’t think of anything useful to do with it. In the end, the Wi-Fi access data remained a mineral without becoming a jewel until I stopped interning.
So far, we have described the lack of big data utilization due to the lack of analysts’ capabilities to analyze big data rather than technical limitations. So, how can we analyze big data well? Kevin Lyons, Senior VP of eXelate, a digital marketing data management company, said that diversity is the most important factor in building a big data analytics team. He said he tries to have as many people from different backgrounds as possible on his data analytics team, including accounting, insurance, and more, not just computer experts. This diversity isn’t just about technical knowledge, but also about the problem-solving skills that come from different perspectives and experiences. When people from different fields work together to analyze big data, they can come up with ways to use the data in different places that they might not have thought of on their own. Of course, individual analysts will also need to have a broad knowledge of society at large and a good understanding of their industry.
New kinds of data are still pouring in all the time. I’m looking forward to seeing how data experts, who have gained great insight through experience and collaboration, will change the world through big data. Going forward, big data has the potential to become more than just a trend, but a key driver of innovation. To realize this potential, creative thinking and collaboration among analysts will be essential.