Data Mesh and Data Vault working together – a case study

Data Mesh and Data Vault can be compared to onions and apples. Both look the same from the outside, but the properties of each are far different. We were really interested to hear Paul Rankin – head of data management platforms at Roche Diagnostics talking at the Data Vault User Group answering the question “Data Mesh and Data Vault – can they really work together?”. Let’s take a dive into what Paul presented.

Roche had a centralised IT team providing analytics from a data warehouse. This had several disadvantages:

• Physical and virtual servers hard to maintain and 3 months lead time to scale up

• 3 months average release cycle hindering business agility

• 2-3 major incidents per year (database corruption, etc.)

• 4 days average time to implement a hotfix

• Backlogs of requirements with conflicting business priorities, leaving dissatisfied users

Interestingly the issues Paul highlights are typical of those seen by our customers with legacy, on-premises data warehouse solutions. Indeed, these are common drivers for re-engineering traditional data warehouse solutions to a Data Vault architecture in the cloud.

What is Data Mesh?

Roche believed that shifting to the cloud would not address many of the problems until an innovative approach was suggested. This approach was the then newly published Data Mesh concept.

Data Mesh is a socio-technical paradigm. It’s a decentralised socio-technical approach to sharing access and manage analytical data in complex and large-scale environments within or across organisations. The important take-aways for us is that this is not just about technology is about people and processes also.

At its core, Data Mesh as created by Zhamak Dehghani was founded in decentralisation and distribution of responsibility to people who are closest to the data to support continuous change and scalability. Roche are trailblazers in Data Mesh and indeed were supported by Zhamak personally in their journey.

Principles of Data Mesh

Paul made it clear that Data Mesh is a conceptual mindset shift to be successful and he explored the four principles.

• Domain-orientated ownership – Decentralise the ownership of analytical data to business domains closest to the data

• Data as a product – Domain orientated data is shared as a product directly with users

• Self-serve data platform – Empower domains to build their own pipelines from source to consumption

• Federated computational governance – A data governance operating model based on a federated decision-making structure – Data Mesh proposes a governance operating model that benefits from federated decision making

Success in Data Mesh

We believe that Data Mesh requires a complete mindset shift to work efficiently. Your ways of working need to be rewritten to achieve the desired results from Data Mesh. The primary objectives are to increase value from data at scale, create agility as an organisation and embrace change in a complex and volatile business context. In addition, all aspects that form Data Mesh are useless on their own. Every moving part needs to be working together for it to be successful. Indeed, Paul emphasised that for him the business domain level buy-in was crucial to making their project work.

Data Mesh Architecture

The architecture of Data Mesh was described as a burger by Paul. Federated computational governance on top, data product, business domain, and federated team of domain representatives in the middle (with data sharing APIs linking them all together), and a self-serve data platform at the bottom. The trick with Data Mesh is to start small and get high value success use cases early-on.

Data and analytics platform

• Self service

• Cost effective

• Value driven

• Fully integrated

• Highly scalable

• Built by industry experts

• Ultimate developer experience

How Roche use Data Mesh

Data Mesh has an impressive record at Roche Diagnostics. Paul’s final slide showed Roche’s experience of Data Mesh. It showed that Data Mesh helped Roche achieve:

• 6-8 weeks average MVP time

• 330 Data product teams working

• 50 Data products published following FAIR principles

Summary

Roche proves our belief here at Datavault that Data Mesh and Data Vault are entirely complimentary and can work well together, if applied in the correct manner. Data Mesh is still an emerging approach and is certainly not for everybody, but it is great to hear a success story. Click here to watch the recording of Paul’s presentation to the Data Vault User Group.

The first Data Vault User Group meet-up of 2023 also promises to be filled with lots of learning points. Juna Korpela – a leading Data Modelling expert – is presenting the meet-up titled “Capture your business needs with conceptual data modelling”. Secure your spot by visiting the Meet-Up page here.