Data Mesh; Big Data London 2022

In the data world, the last few months have seen an increasing buzz around the concept of ditching long-held views about data warehouses – and the search for a single truth about data – and utilising a Data Mesh instead.

Indeed, visitors to Big Data London last month could not escape the phrase Data Mesh,. The Data Mesh concept author Zhamak Dehgani first coined the phrase back in 2019, and she was the centre of attention speaking at the show where she signed hundreds of free copies of her book ‘Data Mesh: Delivering Data-Driven Value at Scale‘, which were being given away.

Three short years since she first coined the phrase, it seems as though everyone is talking about Data Mesh. It is has certainly made its way up the hype curve.

You only have to look at the timetable for Big Data London in 2021 – an analysis of the topics discussed shows that just two out of 205 sessions held over two days were dedicated to the subject of Data Mesh. Incredibly that number had risen to 19 in 2022 – with two of the major panel debates on the first day attracting experts in their different fields.

So what are we to make of where the world of data analytics is with Data Mesh right now?

Is it finally that magic silver bullet corporate executives of enterprises of all shapes and sizes have longed for when it comes to maximising value from their data assets and avoid large bottlenecks in companies’ IT?

Data Mesh author Zhamak Dehghani believes it is the pain experienced by many professionals dealing with centralised data warehouses and data lakes that is driving the change.

She points to the way in which microservices were adopted more than a decade ago in the digital world, while software engineering became more and more complex, with centralised teams building bigger and bigger applications, and growing rates of data collection driven by mushrooming volumes produced by growth in the Internet of Things (IoT).

So a way to decentralise data storage and management, focusing on domains and making business teams responsible for the data they consume “feels familiar,” in what also feels like the “right moment in time,” according to Zhamak.

“We had never questioned why we experienced pain-points of scale around centralised data lakes and data warehouses,” she told a packed audience at Big Data London.

But equally for anyone of a nervous disposition when it comes to risking breaking parts or all of a business – trying to migrate to the Cloud, replacing legacy data warehouses and consolidating multiple source systems into something manageable – Zhamak’s own admissions should ensure a long count to 100 before agreeing to take the Data Mesh plunge.

“This is a transformation. We are early in the process, we have not figured out all of the details to do this at scale,“ she admitted. “There will be corners you have to explore and figure out. There is a lot of engineering ingenuity that needs to go into implementation.“

It may pay dividends to stay with what was said in south west London for just a little bit longer, as one of the most fascinating parts of the ‘Data Mesh – What You Need to Know’ discussion.

On the panel, as well as the most respected Data Mesh author, were vendor representatives Starburst Data’s Justin Borgman and Thoughtspot’s Cindi Howson.

Cindi excitedly made one of the boldest predictions you are likely to hear in 2022.

The question from panel chairman Kevin Petrie, Eckerson’s respected Vice-president of Research, was where will Data Mesh be in 2025?… just three short years away.

Cindi claimed: “Those companies that adopt Data Mesh principles will outperform the market by double digits – and achieve costs savings of up to 30 per cent.”

Now that is interesting because as Zhamak herself was at pains to point out – both in her own dedicated presentation about her book earlier that morning and during the debate – at the moment, Data Mesh is still very much a concept.

You certainly can’t buy it out of a box but a desire to do away with 25-year-old technology – and some mindsets that are still focused on approaches from the late 1990s has definitely helped create the demand for a new way.

Cindi Hewson pointed to the fact it takes many large international businesses nine months – and six or seven figures in dollars or euros – before a useable data set can be leveraged into bottom-line return.

She described those experiences as “unacceptable” for the pace of business in a digital economy.

So if Data Mesh is resonating because it offers faster times to data insights, let alone creating more relevant data sets, what are the options?

Again, it is worth remembering some of Zhamak’s thoughts shared in London, and which are echoing around business centres and boardrooms around the world.

Before deciding whether a Data Mesh is right for your business, you should currently be experiencing those data warehousing pain points and the frustrations of managing an ever-deeper data lake.

So if you have the complexity that as Zhamak says, deserves an equally complex solution, are there any case studies of current implementations that can be shown to work.

One of the best-known Data Mesh implementations so far is the experience of Roche, the Swiss-based pharmaceutical giant.

Roche’s Head of Business Intelligence, Omar Khawaja, was part of the panel, and while suggesting the best way to start, was to start with a part of the business that was experiencing the biggest problems with the current data strategy.

Again he stressed, while the technologies – including Snowflake and DataOps.live – are out there to help, you cannot buy a ready-made solution.

But the biggest departure comes in the area of data governance, where Omar warns, anyone looking for a Data Mesh implementation has to accept the biggest changes will have to – and need to – come.

“Governance needs to change to data sharing from controlling,” said Omar. “You need to bring the love towards the data not locked inside some sort of jail. Free the data –that’s my message.”

Underlying the new Brave New World at Roche however, is another tried and tested concept – the Data Vault method.

A divergence on data governance will mean any enterprise or organisation must think about that subject up front – how does it govern and manage data currently, what policies and processes are in place.

The panel – when asked for their views on where Data Mesh will be in the marketplace come 2025 – were all convinced the method will be much more mainstream, the view of its chairman Kevin Petrie were just as revealing as predictions of financial performance.

Kevin, cheekily, but almost cynically, predicted that in less than 36 months’ time: “Most enterprises will say they have a Data Mesh – and a fraction of them will be speaking the truth.

“I am not sure they will have successful Data Meshes, but they will say they have a Data Mesh.“

While other organisations – including both the BBC and Channel 4, and the RAF – spoke about their early experiences with Data Mesh, it will be interesting to see, in particular, how true Kevin’s prediction – based on many years of looking at what the movers and shakers in the data world actually do – stands up to the test of time.