Data chiefs in the Asia-Pacific region are pursuing the rollout of artificial intelligence in earnest, according to an international survey of 600 global data leaders from Informatica. India is racing ahead in the region, with 75% of those surveyed already having adopted generative AI.
However, APAC-based respondents are facing hurdles around the management of data for AI. These include data fragmentation amid an exploding number of data sources, the quality of data available for AI, and embedding data governance that is robust enough for the AI challenge.
Richard Scott, senior vice president of Asia-Pacific and Japan at Informatica, said data literacy is important to support organisational data management. Scott recommended getting cloud data architecture right from the beginning and focusing on people, processes and technology.
AI is driving a parallel focus on data management
APAC data leaders said the ability to deliver reliable and consistent data fit for generative AI (40%) was the leading data strategy priority for 2024, together with improving data governance and processes (40%). This indicates AI is driving a mutual focus on data management.
SEE: The top 10 benefits of improved data quality for your organisation.
The intimate connection between AI and data was also reflected in investment intentions. Three in four (78%) APAC data chiefs predicted their data investments would increase in 2024. Not a single respondent did not plan to invest in data management capabilities in some form.
Regional investment in key data capabilities is rising
A number of data management capabilities are receiving investment in line with data strategy priorities. Data privacy and protection was named number one (45%), reflecting the necessity of keeping data private and secure amid a rise in a fast-changing cybersecurity environment.
This was followed by data quality and observability (42%) and data integration and engineering (40%).
“We’re seeing a surge in data quality as an area of focus, and data governance,” Scott said. “So AI is really going to drive in a kind of a new wave of cleaning up of data estates.”
SEE: How IBM’s Matthew Candy views Australia’s 2024 pursuit of generative AI scale.
AI is bringing many data management challenges
According to Informatica’s global survey results, which were sourced from data leaders in organisations with greater than US $500 million in revenue, almost all (99%) data leaders had encountered roadblocks on their AI journey, including those in APAC.
Data fragmentation and data growth
APAC data leaders expect data fragmentation and complexity to worsen in 2024. Informatica found 56% of data leaders were struggling to balance over 1,000 data sources. In addition, 78% of APAC data leaders expect the number of data sources will increase this calendar year.
“Last year alone, Informatica processed about 86 trillion cloud transactions a month, up 60% from a year prior,” Scott explained. “So while organisations are trying to get their data house in order, the data is still exploding; we are seeing this really explosive growth,” he said.
Data quality and AI model bias
Data quality was named the biggest challenge to generative AI by 42% of global respondents. The potential for bias stood out as a particular concern in APAC due to large language models; 53% of Australian respondents said avoiding bias was their biggest concern (Figure A).
“In the era of analytics, if you had bad data foundations, you would get to the wrong decision quicker,” Scott said. “In the same way, if you have a bad data management environment, you will get an answer from generative AI, but it may take you in the wrong direction.”
Data literacy outside the data estate
Organisational data literacy is holding progress on AI back, according to data leaders surveyed. For example, 98% of global data leaders said they had experienced nontechnical organisational roadblocks to better data management, such as a lack of leadership support.
Improving data-driven culture and data literacy was named by 39% of global data leaders as a top priority for 2024. Improving data literacy was the second most important (42%) measure of data strategy effectiveness, beaten only by getting data ready for AI and analytics initiatives.
“Our CEO at Informatica talks a lot about the fact that, with businesses outsourcing applications, buildings, and so many other aspects of a business, for many companies their only asset is data. So it has to be a really high priority for the executive team and the board,” Scott said.
A growth in data management tools
The number of data management tools is growing. Two thirds (60%) of APAC leaders say they will need five or more data management tools to support priorities and manage data estates — an increase from the number of data chiefs who needed this number of tools in 2023 (55%).
Data governance and democratisation
Improving governance over data and processes was named by 40% of regional data leaders as a top data strategy priority for 2024. APAC data leaders also placed the highest emphasis (67%) on enabling more data democratisation across their organisation when using generative AI.
This is driving vendors to offer governance services and tools. Informatica recently launched an integrated Cloud Data Access Management tool following its acquisition of Privitar, which helps support the compliant management, sharing and use of data in jurisdictions across the globe.
SEE: Data governance to be a renewed focus in IT for Australian organisations in 2024.
Informatica also offers a self-service data marketplace designed to ‘democratise’ data access. Users can request and access data based on permissions. Data is served up with data quality and relevance ratings and is tracked so data stewards understand how it is being used.
Architecture foundational to meeting data challenge
Informatica’s Richard Scott advised regional data leaders to get the right cloud architecture in place to support scale and to focus on people and processes as well as technology.
Start with the right cloud architecture
Organisations should start by ensuring their cloud architecture is sound, Scott said, as getting this right from the start will support future efforts to scale.
“It’s when you are scaling out and you don’t have the right sort of data management architecture that you get into real trouble,” said Scott.
Scott added that getting cloud architecture right from the outset is also cheaper.
“Companies with multiple cloud contracts pay a lot of money in the ingress and egress cost between clouds,” Scott said. “Not only does the wrong cloud architecture result in an environment that maybe can’t support generative AI but it is also very expensive.”
Informatica client NRMA, one of Australia’s oldest member-based organisations, is working successfully with over 3,000 datasets. Organisations that exert the effort to get the architecture right can get on top of data and have a material impact on their data estate, Scott said.
Look at people, processes and technology
The nature of the data challenge means organisations need to look more holistically at people and processes and technology. Scott said for data leaders in organisations that are trying to fix problems as they arise, it can feel like “putting your finger in the dike to stop a flood.”
“What will happen is if you just plug each little hole in the dike by getting a new application or writing some code, you’re going to end up with a very fragmented environment, which is going to be very brittle. You need to look at people, process and technology and have a clear understanding of where you’re headed; then you can bring in technology that’s going to integrate incredibly well and give you that ability to transport data across your environment.”