The Truth About Implementing Data Vault

The shocking truth is that Chief Information Officers (CIOs) are being convinced that hand-coding their Data Vault solutions is a good idea. This is a mistake. There are two groups who convince the CIO to make this expensive mistake. I have seen examples of both in recent weeks.

Firstly, at a mid-sized enterprise where the internal data team is led by very clever architects and developers. Their team looked at Data Vault and understood that it is pattern based and therefore can be automated with code. There are expensive tools available, but they believed that they can easily develop something themselves. So, they told their executives that they are going to save money: not by purchasing an automation tool but by building their own. After all, it is straightforward. Isn’t it?!

The second was much more disturbing, we were talking to a large UK retail company, and they had hired one of the big-brand consultancies to deliver their Data Vault project. The consultancy was hand-coding their Data Vault platform. No doubt they would deliver a working solution, but the consultancy was deploying many consultants on the project.

5 reasons to use automation tools on your Data Vault project

This brings me to explaining why implementing Data Vault without an off-the-shelf an automation solution is a mistake. The Data Vault methodology provides an elegant solution for managing complex data integration problems. However, the choice between hand-coding and leveraging automation tools remains pivotal. The reasons why we always recommend automation solutions are:

1. Consistency and standards

It is well known that one of the keys to successful Data Vault 2.0 implementations is to follow the standards. This is particularly true for inexperienced teams who often don’t appreciate the reason for the standards until it is too late.

When you hand-code, there is a temptation to diverge from the standards. That’s where inconsistencies creep in. Automation tools, on the other hand, adhere to standards consistently. Whether it’s loading hubs, satellites, or links, the generated SQL maintains uniformity.

2. Error reduction – it is pattern based

Data Vault inherently lends itself to automation because it is pattern based. Manual coding invites errors. Typos, varying naming conventions, and suboptimal code quality become your unwelcome companions. Automation minimizes them and provides reusable components. No more copy-pasting or reinventing the wheel. Someone else has done the hard-work – why repeat it?

By handling repetitive tasks, tools reduce the risk of mistakes. Fewer errors mean less time spent debugging and more time building.

3. The productivity boost makes it cheaper

Perhaps the most compelling reason is that generating code is simply quicker and more productive. It generates ETL/ELT code, freeing data engineers from mundane SQL writing. A smaller team can focus on the higher value activities including talking to the business, data modeling, business logic, and delivering insights!

To quantify this, one of the tool vendors has a case study of their customer making 70% cost savings using automation against manual development.

4. Continuous maintenance and evolution

Hand-coded solutions are sometimes poorly documented which leads to problems with long-term maintainability. This is frequently intensified by staff turnover. When staff who designed and built the solution leave, the organizational understanding of the solution then declines because it is hard for new people to maintain the system.

The nature of business intelligence is that new reporting and analytics requirements are constantly emerging, so a nimble organization needs its systems to be evolving.

Hand-coded solutions stagnate; automation thrives because new features emerge, bugs get squashed, and enhancements arrive driven by common issues across the market.

5. Scale and complexity

One of the challenges of Business Intelligence in many organizations is that the volume of data grows, the sources and formats of the data change, thus maintaining and scaling custom code becomes more difficult. Automation tools scale effortlessly, adapting to changing requirements without requiring extensive manual adjustments.

Conclusion

It is genuinely shocking with range of automation tools out there (including free ones) that senior executives are sanctioning hand-coding of Data Vault solutions. Data Vault can be implemented by small skilled teams in an Agile manner, but only by making use of the available automation tools.

There are many different Data Vault automation solutions out there. Inevitably they all have their strengths and weaknesses, but the one thing we would always recommend to any client is to use an automation solution. It is faster and cheaper both in the long term and the short term – ask some serious questions if anyone tells you otherwise!