The key is data
Big Data is everywhere today, often without us realising it. Whilst self-driving cars, realistic robots and autonomous delivery drones are the more newsworthy and radical forms of this digital transformation, at a more relatable level, the rise of cashier-less supermarkets is imminent and could easily be the new normal for many in the very near future and in a growing trend, groceries and take out food is now being delivered in many cities by tiny robots who even thank you and tell you to have a great day.
None of this would be possible, without data, often referred to as the oil of the fourth industrial revolution, and the technology we’ve built to analyse, interpret and understand it. Big Data is a sophisticated operation and teams of brilliant data scientists make this industrial revolution possible.
The term Big Data itself may not be such an alien concept to us as it would have been just a few years ago, and that’s because many of the technologies it represents have been thoroughly embedded into our everyday lives. Although, still very much in its infancy, some of the key trends that will influence how this type of data and analytics are used for work, play and day to day living, now and in the near future.
Artificial intelligence (AI) has been a gamechanger for analytics and problem solving within companies.The most basic way to think of Artificial intelligence (AI), as it is used today, would be computers and software that are capable of learning for themselves.
Making a comparison, if we only have non-learning computing available to us, we might be able to manually analyse some simple data by creating a database showing us which customers spend the most money. But what if a new customer appears who spends £100 in their first transaction with a business, are they more valuable than a customer who has spent £10 a month for the past year? To understand that we need a lot more data (and the capability to analyse this) such as the customer's lifetime value and even personal data such as their age, spending habits or income level would also be useful. With the huge amount of structured and unstructured data generated by companies and their customers, manual forms of analytics can only scratch at the surface of what’s to be found.
Using a learning computer, AI, to interpret, understand and draw insights from all of those datasets is a far more time effective and efficient way to gather this data. AI can attempt to interpret all of the data together and come up with predictions about what the potential lifetime value of a customer may be based on everything we know, whether or not we understand the connections ourselves. An important element of this is that it doesn't necessarily come up with "right" or "wrong" answers – it provides a range of probabilities and then refines its results depending on how accurate those predictions turn out to be.
Data visualisation is the end of the line in the analytics process before we can take action based on our findings. Traditionally, communication between machines and humans is carried out by visualisation, using graphs, charts and dashboards that highlight key findings and help us to determine what the data is suggesting needs to be done before acting upon it. The problem here is often human error and the inability to spot a potentially valuable insight hidden in a stack of statistics. As it becomes increasingly important that everyone within an organisation is empowered to act on data-driven insight, new ways of communicating these findings are constantly evolving.
A key area where important breakthroughs have been made is the use of human language. This field of technology, known as natural language processing (NLP), can be defined as analytics tools that allow us to question data and to receive answers in clear, human language will greatly increase access to data and improve overall data capabilities in an organisation by allowing humans to better understand the data that’s in front of them.
New technologies that allow us to get a better visual overview and understanding of data by fully immersing ourselves within it. Extended reality (XR) – a term that includes virtual reality (VR) and augmented reality (AR) will clearly be seen to drive innovation here.
VR can be used to create new kinds of visualisations that allow us to impart richer meaning from data, while AR can show us directly how the results of data analytics impact the world in real-time. A perfect example would be, a mechanic trying to diagnose a problem with a car, they may be able to look at the engine wearing AR glasses and be given predictions on what components are likely to be problematic and may need replacing before determining the solution.
In the not too distant future, we expect to see new ways of visualising or communicating data, widening accessibility to analytics and insights and improving the relationship between humans and computers.
Hybrid cloud and the edge
Cloud computing is another technology trend that has had a massive impact on the way Big Data analytics are carried out. The ability to access vast data stores and act on real-time information without needing expensive on-premises infrastructure has fuelled the boom in apps and startups offering data-driven services on-demand. But relying entirely on public cloud providers is not the best model for every business, and when you trust your entire data operations to third parties, there are inevitably concerns around security and governance.
Many companies now find themselves looking towards hybrid cloud systems, where some information is held on these off site clouds such as Amazon Web Service, Microsoft Azure, or Google Cloud servers, while other, perhaps more personal or sensitive data, remains within the proprietary closed platform. Cloud providers are increasingly on-board with this trend, offering "cloud-on-premises'' solutions that potentially provide all of the features and benefits of a public cloud but allowing data owners retain full custody of their data.
Edge computing is another strong trend that will affect the impact that Big Data and analytics have on our lives. Essentially edge computing means devices which are built to process data where it is collected, rather than sending it to a cloud for storage and analysis. Some data needs to be acted on too quickly to risk sending it backwards and forwards such as the data gathered from sensors on autonomous vehicles.
In other situations, consumers can be reassured that they have an additional level of privacy when insights can be obtained and analysed directly from their devices without them having to send data to any third party. A consumer for example would reject the idea of sending a constant stream of their audio environment to a cloud, for instance the Now Playing feature on Google’s new Android phones which continuously scans the environment for music so it can tell us the names of songs playing in the supermarket or movies we’re watching edge computing is the perfect alternative.
DataOps (data operations) is an automated, process-oriented methodology and practice used by data and analytic teams, that borrows from the DevOps framework that is often used in software development. While those in DevOps roles manage ongoing technology processes around service delivery, DataOps is concerned with the end-to-end flow of data through an organisation. In particular, this means removing obstacles that limit the usefulness or accessibility of data and deployment of third-party "as-a-service" data tools.
The evolution of the DataOps role means that even without formal training it makes it a great opportunity for anyone with experience or interest in an IT career that wants to work on the most exciting and innovative projects, which are often data projects. We will also see the growth in popularity of utilising DataOps, offering end-to-end management of data processes. More and more DataOps providers are now making this much more accessible to small and startup organisations with great ideas for new data-driven services but without access to the infrastructure needed to make them a reality.
Overall Big Data is big news and in the fast moving world of technology is only going to increase in importance and relevance as it evolves exponentially, improving our ways of living both within the working world and our daily lives.