Gartner, 5 trends in data science and machine learning

With the increasing importance of data in artificial intelligence as generative AI becomes the world’s buzzword, Gartner has identified five key trends that will shape the future of Data Science & Machine Learning.

Speaking at the Gartner Data & Analytics Summit in Sydney this month, Gartner Director Analyst Peter Krensky said:

“Machine learning adoption is growing rapidly across industries. As a result, DSML, which was previously focused only on predictive models, is now evolving into a more democratic, dynamic, and data-driven discipline. Enthusiasm for generative AI is also fueling these advances. Potential risks are emerging, but at the same time, numerous new features and use cases are emerging.”

According to Gartner, the key trends shaping the future of data science and machine learning are:

Trend 1: Cloud Data Ecosystems

Data ecosystems are moving towards fully cloud-native solutions rather than operating solely on standalone software or hybrid deployments. Gartner predicts that by 2024, 50% of new system deployments in the cloud will be based on a cohesive cloud data ecosystem rather than manually integrated point solutions.

Gartner recommends that when evaluating a data ecosystem, organizations base their assessment on their ability to access and integrate data sources outside their immediate environment, along with their ability to address distributed data challenges.

Trend 2: Edge AI

The demand for edge AI is growing. It supports data processing at the edge as it is created, enabling organizations to gain real-time insights, detect new patterns, and meet stringent data privacy requirements. Edge AI also helps organizations improve the development, orchestration, integration and deployment of AI.

Gartner predicts that by 2025, more than 55% of data analysis by deep neural networks will occur on edge systems at the time of capture, up from less than 10% in 2021. Organizations need to identify the applications, AI training, and inference required to move to edge environments near IoT endpoints.

Trend 3: Responsible AI

Responsible AI is a concept to make AI a positive force rather than a threat to society and itself. It covers the many aspects of making the right business and ethical choices when organizations adopt AI that are addressed independently, including business and societal value, risk, trust, transparency and accountability. Gartner predicts that by 2025, pre-trained AI models will be concentrated in 1% of AI vendors, making responsible AI a societal concern.

Gartner recommends that organizations adopt a risk-proportionate approach to delivering AI value and be cautious when applying solutions and models. You should seek assurances from your suppliers to determine how they are managing your risk and compliance obligations, thereby protecting your organization from potential financial loss, legal action and reputational damage.

Trend 4: Data-Centric AI

Data-centric artificial intelligence is a shift towards building better AI systems by shifting from a model and code-centric approach to a data-centric approach. Solutions such as AI-specific data management, synthetic data, and data labeling technologies are designed to address multiple data challenges such as accessibility, volume, privacy, security, complexity, and scope.

The use of generative AI to generate synthetic data is one of the fast-growing areas, and it removes the burden of acquiring real data so that machine learning models can be effectively trained. Gartner predicts that by 2024, 60% of data for AI will consist of synthetic data to simulate real-world, future scenarios and reduce AI risks, up from 1% in 2021.

Trend 5: Accelerated AI Investment

Investment in AI is expected to continue to be fueled not only by organizations implementing solutions, but also by industries looking to grow through AI technologies and AI-powered businesses. Gartner predicts that by the end of 2026, more than $10 billion will be invested in AI startups that rely on large-scale AI models (foundational models) trained on massive amounts of data.

According to a recent Gartner survey, 45% of more than 2,500 executives have decided to increase their AI investments due to the craze for ChatGPT, with 70% saying their organizations are in research and exploration mode for generative AI, and 19% are in pilot or production. I replied that I was in the mod.