Build the Data Foundation First, Then Let AI In

Every client conversation I have at the moment gets round to AI within about ten minutes. Copilots, agents, chat over their documents, predicting which customers are about to leave. The appetite is there and I think that's a great thing.

The bit that doesn't get talked about enough is what sits underneath. An AI model is only as good as the data you give it, and most of the organizations have their data spread across a CRM, a finance system, a pile of spreadsheets and a data warehouse someone built ten years ago that nobody wants to touch.

If AI is going to inform decisions, automate processes or interact with business data, the data foundation must come first.

For organizations with data spread across multiple systems, I believe Data Vault within a Medallion architecture is one of the strongest ways to build that foundation, delivered incrementally through agile delivery. Some may consider this to be a sledgehammer to crack a nut for small or short-lived use cases, but for complex organizations that need history, traceability and room to change, it should be the default starting point.

Why Data Vault and Medallion work so well together

People sometimes talk about Data Vault and Medallion as if you have to pick one. You don’t. They fit together really nicely.

Medallion gives you the layers. Bronze is where the data lands exactly as it came out of the source system. Silver is where it gets integrated and tidied up. Gold is where it is shaped for whoever is using it, whether that's a Power BI report, a data science team or an AI model. And the key thing here, It’s benefits are more than just AI.

Data Vault is how I build that Silver layer.

The easiest way I can describe it is a family tree. Hubs are the people in it (in data terms your business keys, so a customer, a policy, a product), links are the lines that join them together and satellites are everything you know about each person. When something changes you write the new detail down next to the old one rather than rubbing it out. So nothing gets overwritten and every record can be traced back to where it came from and when it arrived.

That last point matters a lot for AI. When a model gives you an answer, somebody in the business will ask "where did that come from?" and with a Data Vault underneath you can actually tell them. You also have the whole history sat there ready to train on, rather than a snapshot of how things looked last Tuesday.

Swap the parts, keep the model

One of the things I love most about this approach is that the design isn't tied to any one technology.

The patterns are the same whether you run on Snowflake, Databricks or Microsoft Fabric, and whether you load it with dbt, Azure Data Factory or something else entirely.

I think of it a bit like LEGO. The model is the vision and every hub, link and satellite is a brick that follows the same pattern. If a better brick comes along you can take one out and pop the new one in without knocking the whole thing down. Every so often it doesn't go quite right and you have to resolve it, but you're fixing one brick and not rebuilding the house.

This is a big deal right now because the technology is moving so quickly. The AI tools we are using today probably won't be the ones we are using in two years. If your data platform is built on standard patterns you can change your warehouse or your AI tooling as the market moves, and the value you have built up in the data stays exactly where it is.

From a commercial point of view that protects the client's investment, and for me that is doing the right thing.

Get the design and requirements right up front

Being able to swap things out doesn't mean you can skip the thinking. This is where projects go wrong.

Data Vault is built around business keys and the relationships between them, so you need to understand the business before you build anything.

• What is a customer?

• When does a prospect become a customer?

• Does finance mean the same thing by "active" as the sales team does?

If you don't get those answers early you end up building hubs around the wrong keys and paying for it later.

So, I always spend proper time at the start on requirements and design. Talk to the people who use the data every day, map out the core business concepts, agree the business keys and sketch the model before anyone writes a line of code.

It doesn't need to be a six month exercise and it definitely doesn't need to be a 200 page document. It just needs to be good enough that everyone is on the same page before the first sprint starts.

Agile, with a good foundation underneath

I'm a big fan of agile delivery (at Business Thinking we follow the GDS approach) and I don't see any conflict between that and doing the work up front. Change will happen during a project, it always does. A new source system turns up or a stakeholder changes their mind (and sometimes the first sprint shows us something nobody expected).

The difference is that with a good requirements foundation you can take that change in your stride. Because Data Vault is additive, a new source or a new requirement usually means a new satellite or a new link. You don't have to go back and rework what is already live.

That is where the speed comes from. We can deliver a thin slice end to end in the first few sprints, from Bronze all the way through to a Gold output the business can actually use, and then keep adding to it sprint after sprint. The client sees value early and we build their trust as we go.

Getting to AI

Once that foundation is in place, AI stops being a science project. Your Gold layer can serve clean, well described data with its full history to a model, a feature store or a copilot sat over your business data. You know where every number came from and you can explain it, which is exactly what your risk and compliance teams will ask for.

I think too many organisations are trying to jump straight to the AI bit and then wondering why the results don't stack up. Just like building a Lego set, build the foundation one brick at a time and deliver it in small agile slices. The AI use cases will follow a lot quicker than you expect.

If you're thinking about where to start, I'm always happy to have a chat. Let me know your thoughts.