Data Trends 2023

The Finnish Innovation Fund Sitra has again published the Megatrends listing. While it focuses on the Finnish perspective, the points are obviously global and relevant to everyone. If you ask us, that is the case with one of the key findings in particular:

“Data has become one of the most important economic resources of our time.”

The report continues to highlight that the use of data creates prosperity and competitiveness while helping to develop societies – and companies we may add. But what about the trends inside the world of data? Together with our band of data experts, we put together a list of 5 trends for the upcoming year.

1) BLENDING MACHINE LEARNING AND DATA ENGINEER ROLES

Machine Learning has typically been limited to people who know which tools and algorithms to use and when. Even if you know a bit of Python, the barrier of setting up necessary tools for solving problems has still been high. When it comes to setting up capabilities around data, projects have required separate roles and hires of ML and Data Engineers.

During 2022, we started to see a rise of multiple vendors making it easy to use the platform to solve both situations with SQL and Python. Databricks, Snowflake, and dbt have all blended the barrier. For example, dbt added Python support and Snowflake released Snowpark with Python support. This helped a lot of experienced Data Engineers to try on Python and ML Engineers to vice versa use datasets with SQL more easily.

We see this trend continuing in 2023 and we predict that the barrier between Machine Learning Engineer and Data Engineers will blend even further helping in the growing demand for both ML and DEs. 

2) THE YEAR OF MLOPS

As stated above, as Machine Learning becomes easier, it will simultaneously speed up development and thus create a bigger demand for deploying and maintaining ML models in production in a reliable and efficient manner. This means that companies need to add MLOps to their toolset similarly as they have done with DevOps and DataOps. At least if they want to keep up with the pace of the growing popularity of Machine Learning.

We see that in 2023, companies need to invest time and resources in building practices and toolsets around MLOps or they'll end up having a multitude of half-baked, unsupervised ML solutions running on top of their data platforms.

3) AI GOING MAINSTREAM

The year 2022 was the year when everybody got hyped about AI. We first saw Dall-E and StableDiffusion showing how well AI can create new computer-generated art and images. At the end of 2022, ChatGPT blew our minds with how far chatbots and large language models (LLMs) have evolved. These solutions have opened people's eyes to what AI can really do and what AI is also capable of in good and bad. 

In the background, we have already witnessed that the computing resources needed for specialized mobile processors and easier optimization (Neural Magic, OpenVino, and Huggingface Optimum) for a multitude of applications have evolved. Already today, it's possible to run ML models in end-user applications such as smartphones creating a better end-user experience. For example, Apple has added support for Stable Diffusion models into macOS and IOS thus opening up a multitude of possibilities for developers wanting to harness the power of Apple devices. 

We predict that in 2023 we will see AI going (even more) mainstream and we will witness a myriad of applications that will leverage AI in some form: Whether around making fashion more sustainable with computer-generated images using InvokeAI, making audiobooks more affordable with AI-generated voices, helping to make 3D assets for games easier and faster, or using AI to help with the creation of faster whole body CT segmentation models for hospital patients. 

4) WISE MONEY AROUND DATA PLATFORMS

Over the past year, we’ve witnessed massive inflation due to the war in Ukraine and the rising costs of electricity. Data solutions do not live in a bubble, they need a working cloud infrastructure. Cloud infrastructure costs will eventually rise and together with the inflation, companies will start looking more closely at which solutions they currently have. Getting approval for a data application that does not yet have a concrete real-life need won’t most likely get the go-ahead.

We predict that in 2023, we will see companies using their money more wisely around data-related solutions, maybe killing solutions that are not actually needed and thus gathering the funds for the projects for their future – such as AI solutions.

5) SLOW DEATH OF OBSOLETE MODERN DATA STACK -TOOLS

Now that Venture Capitalists are not pouring their cash into every new data-related solution, we will see a slow death of obsolete Modern Data Stack -tools. The VCs will simply start demanding results for their investments. This will create a tidal wave of open source that will die and fade away in 2023 unless they manage to get paying customers.

We predict that most companies will re-evaluate their data stack, check which open-source tools they are using, do they have support in the future and do the tools actually help in solving concrete problems. This change will also clear paths for the already proven open-source solutions (like dbt) and give them more support.

BONUS: Chief Data Officer is the new Chief Digital Officer (?)

Not as much of a prediction as a cry for organizations to not make the same mistake twice. Back when digitalization was the talk of the town, multiple companies hired a Chief Digital Officer to just, well, do it. Outsource the whole digitalization thing to one person if you will. We all know how that turned out. So please, don’t think that a Chief Data Officer will magically solve everything and enable everyone else in the company to wash their hands off of data topics. If you want to ensure your company is on board with the data trends and around to read another prediction next year, everyone is needed. As our mantra goes, business and IT need to communicate and cooperate. That’s a good start.

WRAP IT UP 

Predicting the future is easier now that we have all the tools and artificial intelligence to do it for us, right? Think about this moment last year and the things we have witnessed over the past 12 months. Even if we love ML and playing around with predictions, there is no such thing as 100% certainty or knowing for sure what will happen in the future. The events, the aftermath, and the vast consequences of February 24th have brutally reminded us of that.

The goal is not to make you read this far just to conclude that there’s no point in anything. No. Even the fine folks at Sitra point out that surprises do not come out of nowhere. They are driven by longer-term changes. When we come back to the context at hand, the silver lining is that we’re not just some passive creatures at the mercy of the data deities. We ultimately create the trends by making the best out of the situation we find ourselves in. And by correctly harnessing what we already know (aka. data), we may turn something bad into an opportunity.

Just make sure you’re on the right side of data. Fortunately, we can help with that.