Written by Denton Cockburn, Senior Partner, Technology Advisory & AI

The past year has reshaped nearly every layer of the data ecosystem - from strategy and platforms to roles, skills, and expectations. Organizations are racing to adopt AI, platforms are consolidating at speed, and long‑standing disciplines like data engineering, analytics, and data science are being redefined in real time. 

As someone who has spent years working at the intersection of data strategy, technology, and delivery, I’ve seen waves of change before, but rarely one that touches so many layers of the ecosystem at once. This article isn’t a manifesto or a polished framework. It’s a set of connected observations from the field. Patterns I’m seeing repeatedly as teams try to move faster, do more with less, and make sense of where humans still matter most in an increasingly AI‑assisted world. 

Across these reflections, a few themes keep surfacing: the tension between ambition and readiness, the shift toward integrated platforms, the acceleration (not replacement) of technical roles, and the enduring importance of curiosity, judgment, and practical thinking. 

The AI-First Paradox in Data Strategy 

Everyone knows what they want, and the answer is AI. But when asked "Why?", "What use cases?", or even "What business problems will this solve?", they struggle to answer.  

This is the paradox we're facing today: organizations are jumping to solutions before understanding their foundations. The state of their data, the systems that contain it, what they do, the governance of it, and the impact on the people within their companies are often expected to be figured out later. It's backwards, but it's the reality. 

In these times, it’s important for companies to have people who can help them challenge their assumptions. With the right approach, AI can really help many companies advance and become more efficient. The Data Strategy planning to get them there is more involved than people expect and can't just be something where a big-4 partner delivers a 60-page report that's just as complex to execute as before that report was written. Practicality is still key. 

The Platform Consolidation Wave 

Data Architecture has really transformed over the last year. What we call "modernization" is often just a push towards platforms that provide far more well-integrated tools. 

Instead of stitching together entirely different systems for 10 steps, we're now often looking at platforms with components that cover 8 of those steps natively and allow integration with external tools for the last 2. This consolidation isn't just about convenience; it's about reducing complexity, improving data quality, and accelerating time to value. This is reflected in the continued rise of platforms like Databricks, Snowflake and Fabric. 

I'm biased, but I've been incredibly impressed with how aggressively Databricks in particular has been extending how much of the data stack they cover. They are also innovating really fast and with a rather open architecture that makes integrations welcome. Platforms that tightly integrate AI while enabling professionals to move faster will deliver greater return on investment, and that will drive its own momentum. 

Data Engineering is not dying. It is thriving! 

This was a space I was genuinely worried for, but I'm now even more excited than ever. 

Many of the daily data problems we struggled with are now significantly simplified. Things like SCD, partitioning, compaction, incremental loads, streaming checkpoints have been made so much easier. Now data engineers can spend more of their time working on the business objectives, and much less time transforming those into low-level data instructions. 

Rather than replacing engineers, AI has made them much more efficient. AI can be used to document code, generate basic tests, repeat patterns. Experts who embrace AI just move at incredible speed here. The productivity gains are real: what used to take days can now take hours, what took hours can now take minutes. 

I do worry about juniors entering this space though, as that grunt work really was the best way to learn. Without hands-on experience with the fundamentals, how will the next generation develop deep understanding? So, my tip for companies that care about the future is to still maintain paths for juniors to one day become seniors. I'm overall really excited for what the future brings. 

The Resilience of Data Analysis 

What a resilient space! This area has really improved with the help of AI. 

The key has always been about curiosity and persistence, asking the right questions and continuing to dig. AI has made the right questions more apparent, and the ability to dig easier. A smart analyst with the right tools can get to important answers quicker. That enables teams to gain the insights they need to make business critical decisions. 

New tooling allows business stakeholders to ask data questions themselves, but data analysts are like police detectives, still essential for deep dives into data. Curiosity is a key human characteristic, and for that reason, this field will continue to be really important. 

Data Science: The Controversial Transformation 

This is probably going to be the most controversial part. This field has changed a lot, and really is nothing like it used to be. 

It used to be a space where a PhD was required to be really good. Now, it's becoming more like a typical software engineering field. That's why we see the rise of roles like the Machine Learning Engineer. That title may have started out as one that supports the work of data scientists, but now they are becoming primary. The democratization of AI tools has shifted the competitive advantage from algorithm expertise to business problem-solving. 

Creating models has become simple. The algorithms are already there and using them takes far less time to learn. Tools are tuning algorithms themselves and the domain knowledge of the data and business analysts are providing faster time to value than teaching it to a standard data scientist. Platforms like Databricks have really simplified much of the voyage between an idea with data, and a production model working on it. 

Looking ahead more optimistically, I would say there’s still a lot of opportunity in mapping business ideas to AI models. And that’s where the next wave of value lies. The expertise to do that thinking will still be in high demand. The stronger scientific background of data scientists will still help here. 

It's normal for fields to transform, and this one is doing so rapidly. I believe data scientists will be able to move quickly to adapt and stay ahead of the curve.