Has the Modern Data Stack Failed the Business?

The modern data stack (MDS) transformed how data teams work, providing more power and efficiency than ever before. Tools like dbt, cloud warehouses, and automated testing have streamlined many processes. But is the business actually realizing the benefits?

What Went Well

From the data team’s perspective, several things improved. Data extraction, transformation, and loading (ETL/ELT) have become much easier. Tools like dbt eliminated the need for complex, custom infrastructure and reduced reliance on data engineers to build a pipeline.

Cloud data warehouses made scalability simple and affordable. You can now handle huge datasets with a few clicks. You can get started instantly and cheaply, plus scale as large as you need to without switching platforms.

Finally, automated testing frameworks like dbt’s also improved quality assurance, catching errors before business users notice. Previously, this type of testing was only possible through complex stored procedures or custom setups, which meant almost no one was able to run tests on their data before dbt.

Where It Fell Apart

However, things aren’t as rosy for business users. The origin of many of the business users’ issues is often the neglect of proper data modeling by the data team. With the ability to create new tables easily, many data teams ended up creating a massive number of tables. This “death of data modeling” led to cluttered, confusing BI platforms where users struggle to find what they need.

Another challenge is the overspecialization of data tools. Companies often buy multiple tools for different tasks—reverse ETL, data quality, observability, orchestration, etc. While each tool may work well individually, the overall system becomes fragmented and hard to navigate for end-users. They have no idea where they need to go to get answers.

The Last Mile Problem

The biggest issue is the “last mile”—the gap between the data team’s setup and the business users experience trying to access the data. Despite all the advances in tool and technology the data team is using, BI platforms have stayed almost exactly the same over the last 10 years.  It still takes 4.5 days to make a dashboard for a business user and 67% of those users feel uncomfortable using their BI platform. Instead of self-serving data requests, they end up asking data teams to do the work.

Data teams become bottlenecks, and business users don’t get the answers they need in a timely manner. This system is full of friction for both data teams and their counterparts in the business.

Solutions

The path forward starts with rethinking data modeling. Just because it’s easy to create new tables doesn’t mean you should. New tables are a liability, not an asset. A simpler, more intentional data model reduces confusion and makes BI platforms easier to navigate for business users.

Consolidating tools is another key step. Instead of using a different tool for every specific task, use your current platforms to cover more functions. This reduces complexity, saves money, and simplifies the user experience.

Improving the “last mile” of analytics is (the most) crucial. BI applications haven’t changed in the past 10 years, while other data tools have proliferated massively.

Data teams need tot demand more from their BI platform. They need a demand solutions that work like their business users do now instead of making them learn new complex software. This is the only way for data teams to truly solve the last mile analytics problem.

tl;dr:

While the modern data stack has delivered major wins for data teams, its benefits for the business are still lagging. Poor data modeling, overcomplicated systems, and lack of innovation in BI have all contributed to a frustrating experience for business users. To really drive value for the business, data teams must simplify the data model, consolidate tools, and bring their BI platform into the modern day. The modern data stack can still fulfill its promise for both data teams and the business, it’s just not there now.

‍