Every day now, we talk to 2 or 3 data teams making the same mistake: waterfall.
The team is planning a new semantic layer (often cube or dbt). They're reviewing the data sources and discussing the important metrics with various teams. And they'll launch it later this year.
Did you spot the mistake? This is the waterfall methodology that got software teams in trouble 10 years ago. Here's how it will play out:
• This data team will spend 9 months building everything out, then release it to the users.
• It will be way different from what the users expected. The ROAS metric doesn't match their google sheet. They don't know which of these conversion rates to use. Finance can't sync this with the ERP. Marketers hate that particular attribution platform.
• So it will need another massive overhaul. By the time that's done, people will have moved on.
• This deployment won't be useful and it won't be used. People will wonder what the data team was working on all year.
Why, why, why are we still waterfalling? Software engineers have moved on to agile development but data engineers haven't.
The right way to handle a new semantic layer (or a new dashboard suite, or new dataset, etc) is one piece at a time. Find a single, specific use case, with a specific team. Build a prototype, work directly with them to get feedback, iterate. Measure usage.
Once you're convinced that particular tool is adding value (and ONLY then), move onto an adjacent use case. Marketing likes their ad tracker; move on to affiliates. MQL conversion dashboard is done? Go speak with rev ops.
But I guarantee you'll mess up if you:
1) Plan, plan, plan.
2) Build, build, build.
Staying agile -- building small then iterating -- will get you better results faster.
