Organisations now have more access to data analytics solutions
Published on : Wednesday 04-05-2022
Abhilash Shukla, Independent Consultant, Digital Transformation Projects.
What exactly is Big Data? What actually is the difference between Big Data and plain Data?
The definition of Big Data may differ significantly from stakeholder to stakeholder. Some people define it as an effective means of representing, integrating, and carrying out high-throughput experiments with very large datasets. Others say that it’s a large collection of small and disparate unstructured datasets that when combined, can provide insight into unusual patterns.
In addition to these two definitions, I would like to elaborate on a third definition. In my view, Big Data refers to the rise of digital enterprises and how organisations can leverage their digital assets to their advantage. These assets are commonly described as large amounts of data and can be seen as a sign that something has been going on that hasn't been articulated and analysed fully.
For me, the significant difference between big data and plain data, a.k.a traditional data, is the size and the way they are stored. Plain data is relative to the size and of a type defined for purpose, wherein the big data is a data that is fast-changing, large in both size and breadth of information, and come from seemingly disparate data sources. The term "Big Data" refers to datasets that are just too large to be analysed using traditional data processing techniques. Traditional data was always viewed in such a way that was easily understood by looking at it or by relating it to one another, but Big Data is something that cannot be understood by a human glance.
What is the relationship between Big Data and Digital Transformation?
In both the public and private sectors, digital technologies are revolutionising how organisations function, from cloud computing to mobile to analytics. Many institutions find it challenging to adopt a digital enterprise model due to the nature of their organisational structure, making widespread data integration and analysis difficult.
With the advent of digital transformation, organisations now have more access to data analytics solutions that help give them access to rapidly analyse a large amount of data and information, allowing them to make better and faster decisions. Having said that, big data sourced from various places like legacy systems, smart devices, IoT transmissions, etc., is melding into an ecosystem, which powers predictive analytics, bringing a whole new perspective to an organisation’s capability.
So digital transformation is allowing data to become available, which in turn with time becomes big data. Just look at the pandemic – governments around the world are quickly adapting to handle the effects of the pandemic by increasing their technological readiness to handle on-ground needs. And to derive decisions, they need data from varied sources like hospitals, vaccination centres, 3rd party testing labs, processed results of the tests, etc. Now all this data is required to be plugged into one place, i.e., ‘Big Data’ so that a 360° view can be achieved, resulting in decision making. Think how powerful the knowledge of data can become if all data can be designed into a piece of meaningful information.
Data has been compared variously to oil (black gold) and also garbage. How to make sense between the two extremes?
To make data logical and valuable it has to be processed in an understandable way. And, the data sources should be free-flowing so that the derivatives it creates are improving with time. The more data you have, the more opportunity you get to become more logically and statistically accurate. The values lie within the quantity of quality data available for us to evaluate. But when you work on a massive dataset it also produces data that has traditionally been looked at or even deemed as trash. Basically, the data in an incomplete form is garbage; similarly, a by-product of structured data results in incomplete data called Garbage. So if you have data being sourced and flowing in properly, it can turn into a meaningful result that can be compared to oil or black gold.
However, data is always seen from perspectives. Each individual sees data differently. What makes sense to me may not make sense to someone else; also if the data is understood well and being used, it can turn out to be Gold. But if the data is unrelatable, or doesn’t make any sense from the individual perspective, it is Garbage. Think like this, as humans we are also gathering data in our minds, of course not effectively as the big data technologies, but we do. So consider a Skin Specialist – more than 80% of people in India who are having a skin disease aren’t aware of it, although we look at our skin on a daily basis. But, when you go to a skin specialist, they tell us about the disease with a kind-of similar look. What I mean to say is that the data on skin was relevant to the Skin Specialist because he could understand it, but not to a lay person. Whereas the data was the same, the perspective made it a source of free flow business like oil or gold, but at the same time for the individual person, it is garbage.
There are various tools and platforms claiming to provide the ideal fit for purpose. How should enterprises evaluate and select the right solution?
The fundamental aspect of evaluating a platform is actually getting to the bottom of defining the ‘fit for purpose’ statement. Most companies simply prefer opt-in platforms without detailing their core requirements. Like a requirement could just be a storage or simply data cleaning, so enterprises should evaluate their real requirements. In most cases, five key areas enterprises should identify are related to data cleaning, storage, processing, orchestration, and visualisation.
Then it is important to answer specific questions in conjunction with the platform selection, like, is the data statistically accurate enough from the quality standpoint to serve the purpose? If yes, then how historical it is, and how latest it is? Finally, the results of the data should be purposeful for users so that actions can be taken. And upon answering these questions, an evaluation of the fit-gap should be drawn as part of the evaluation. No two enterprises are the same, and similarly, no two platforms are the same either.
Whereas manufacturing companies and process industries have well defined benefit statements, how do service industries benefit from Big Data Analytics?
From Banking to Telco, Real Estate to Healthcare, beyond manufacturing and processes, services play a key role in serving data-driven intelligence. Recently I was involved with an FMS (Fashion Management Solution) project, and these are the questions we addressed with the big data available to us: What region is least performing? What SKUs are required to be relocated from region-to-region warehouse-to-warehouse for increasing delivery time and minimizing local dead stocks? What category of product is working out in what season? Now, these questions are not possible if you don’t have big data which is analysed and turned into meaningful information.
Service industries have realised the potential of big data. From optimising internal operations to identifying customers and selling the right products have changed the way traditional businesses have worked. Today big data analytics help us to predict and identify customers’ interests with already available data points. We are able to take action on the basis of behaviour patterns and design sales, marketing, and customer servicing in a much more mature manner. Service industries are able to close deals in less time with better customer acquisition costs and are able to optimise resourcing with better resource management. Big data analytics have changed the approach of how service industries used to function traditionally, and they not only consume data, but now they have become a strong party for generating data, aggregating it, and then supplying it for use.
How does the 5G rollout change the landscape of Digital Transformation?
One of the biggest challenges of data is sourcing, which is affected mainly by bandwidth issues that currently exist. 5G will have a definite impact on IoT when it comes to digital transformation from an overall perspective. Think about the consumer market already demanding 3D gaming and augmented reality which is. Similarly, in industrial communication, typical industrial applications connected to control systems and proprietary protocols would see a dramatic increase in communication, security, and privacy capabilities because of 5G.
5G will enable us to exchange data faster, it will help the healthcare industries with vital tracking, and will help in meeting real-time specifications enabling remote streaming consultations and addressing emergencies with no delay. It will help the agriculture industry to analyse and control the equipment which is sending the massive number of datasets from interconnected sensors collecting precision data of crops, plants, and livestock. It can help the automotive industry with live data processing for improved autonomous driving and can help solar and wind power plants by enhancing their smart capabilities.
5G will revolutionise the coming decade!
Abhilash Shukla works as an independent consultant for digital transformation projects. He has founded and worked in small start-ups, global organisations on multi-team projects, and many organisations in between. Through the innovative use cases of Digital Technologies, AI/ML, and IoT, Abhilash has contributed to many SaaS and PaaS projects in his 13 years of industrial experience.
Abhilash has worked on many private and public sector projects in healthcare, retail, logistics & distribution, consumer packaged goods, real estate, mining, and financial services. Ambitious to lead and build products using high-class technologies and always work with a willingness to create benchmarks for others, his work ranges from building core SOPs to big data businesses. He always tries and adheres to deploying amazing products for his companies and their clients. From conceptualising to the complete business life-cycle, his vision is always to translate business problems with efficient solutions.
Abhilash believes that his work should determine that he is in the business of making people happy by having an uplifting positive impact on the community. He is a storyteller and a creative person!
(The views expressed in interviews are personal, not necessarily of the organisations represented)