Barry Devlin Interview Part 1

Technology is not the problem or the answer .… but the future of business intelligence is AI

Part 1 of our interview with Dr Barry Devlin author and thought leader in Business Intelligence

Dr Barry Devlin is one of the world’s leading authorities on data warehousing and business intelligence.

He can claim to have defined the architecture for the world’s first data warehouse for IBM in the mid-1980s, and has consulted and lectured on the subject.

Barry has also wrote Data Warehouse: From Architecture to Implementation in 1996 and also Business Unintelligence in 2013.

His more recent work through 9sight Consulting brings a holistic and very personal style of trying to help companies wade-through issues from data strategy, architecture and governance, to using analytics and data science to produce the best outcomes, by concentrating on the human and organisational challenges encountered in helping a business to use IT optimally to manage its data.

With the demand for more real-time analysis and more and more companies grappling with big data, Barry has strong thoughts on how machine learning and AI (artificial intelligence) should be utilised – and controlled – in enterprises’ interests as they seek to meet the demands of digital transformation.

He firmly believes the data world needs new architectures, technologies and methodologies as commerce and economic activity around the world moves more and more towards making decisions to meet instant demands for greater gratification, driven increasingly by AI, and all the implications that has for business intelligence.

Andrew Griffin spoke to Barry about his experiences over the last 30 years, and how he came to the conclusions he now advocates so passionately.

Q) What’s the single biggest and most important lesson you have learned during your consultancy work – and after more than 30 years working in business intelligence?

A) My answer – and you are not the first to ask – is that technology is seldom the problem, or the solution.

Over the years I have met so many people who came to me or came to IBM – or whoever else – and said, “I’ve got a problem. We’ve got the wrong database or we’re using the wrong ETL tool or we’ve got a problem with the business. What’s the technology I need to use to solve it?”

And again and again and again I’ve discovered yes, technology is important. Yes, it’s important to have good tools.

However, almost invariably the problem arises from the people in the organisation, and the solution invariably also has strongly involved the people and the organisation.

And yet we in IT tend not to deal with those aspects too deeply, and so we end up chasing our tails trying to replace technology with better tools, better database, better, whatever – when in fact it is often the case that you need to go back and think about, “What is it you want? What is it you need to be doing? Who is doing things that are getting in your way? Who needs to change their behaviours in order to make this thing work.”

So that’s always where my projects have ended up… looking to see what is the underlying, contextual, organisational, personal issues that need to be solved, and often they don’t want to know.

Q) So what’s the most important piece of advice you would offer a company in devising or revising their data strategy and improving their business intelligence and data science operations?

A) It really has to do with trying to figure out what do people mean when they say something. What does data mean when it’s put there?

What is the intention that people have when they say I want to measure X? Do they want to measure it because they want to make a profit?

Or do they want to measure it because they want to succeed with their customers? And those two intentions give you very different outcomes as to what data you need, how it should be used, where it should be used, and exactly what it means.

So the sort of advice I would give is focus on meaning – focus on intent. Then you begin to see what you need to do with your technology, and it may well be that, the technology that you have is, perhaps, even good enough.

Q) So have there been times when it has been hard to get clients to understand or accept your philosophy?

A) I had one client that I worked with fairly recently, and their key question was “Should we move our data warehouse to the Cloud?”

My answer was: “You need to have a data warehouse first,” – because actually they hadn’t done the underlying work. The underpinning work of figuring out the meaning of the data, making sure it was used in the correct ways … if you haven’t done that – it doesn’t matter whether you do it on the premises or in the Cloud.

In some cases, the client will be doing certain things right, and they will be getting their data in order, they’ll be doing data governance – they’ll be doing good stuff, and will be making progress.

And then the CEO changes or a new CFO comes along and all bets are off. Let’s throw out not only the baby, but also the bathwater and the bath!

So you go ahead and start again, but there is no right answer without understanding what it is that the CEO wants to do, and the CFO is willing to pay for.

In order to get a client to accept the need for change, you have to really understand the organisation, the politics, and the business. Only then can you actually make suggestions as to what they should and hope the take them on board.

Q) From working in different parts of the world, do you find that there are some cultural differences that spill into the business world and which have made you change your message in some cases to reflect that?

A) Well, at the risk of being stereotypical, yes of course. Very broadly speaking, if you talk to a US-based company, for example, they are always interested in the solution.

So it’s ‘deliver me a solution in 30 days’ – and it’s usually 30 days – and “Don’t bother me about the details.”

Some people describe this as “fire, ready, aim!”

Dealing with a US company, you really have to be very aware that deadlines, endpoints are what motivate them.

Europeans on the other hand, and especially Germans – again I’m stereotyping – will architect and design “properly” for as long as it takes.

With the UK, I always feel somewhere in the middle, but it tends more towards following the US route than the European approach.

Both ends of that spectrum have to be dealt with. You have to find a way through.

Q) What has been the biggest change that has affected the way companies work in Business Intelligence over the last 30 years?

A) This is a really important question. There is this belief in the world today, and I think it comes from the USA, that everything must happen now – it’s all about the current moment, the “now” moment.

I have to understand it now. I have to solve it now. I have to make the sale now… I have to do it now.

It’s the old Freddie Mercury refrain, “I want it all and I want it now,” and that has become the driver for everything in business intelligence and artificial intelligence in recent years.

If you don’t follow the big picture, the long term trends, you can’t actually make sense of the small things, and so it comes down to good information governance, which involves understanding which data needs to be now, which is immediately important and which data must be consistent across the business and over time?

How to measure and track things over time becomes really important, and if you don’t do that, having big data arriving instantly from your sensors won’t help you.

Q) So how critical is good information governance and what recommendations do you have for avoiding some of the problems that relying on a data lake rather than a data warehouse can cause>

A) I always think of two terms when someone says data lake – swamps and drowning. And neither is good.

A data lake to me focuses too much on “I want it all and I want it now.” That is valid in some cases, but I don’t think you can do away with your data warehouse. I don’t believe you can do away with your data models. They’re what deliver consistency and historical truth I don’t think you can have everything just being stored as it lands, and then figure out what it is later.

Getting data correct, getting it understood, getting it managed, getting information governance right – it’s hard work. It’s something that requires you to spend time up front. Talking to people and figuring out what it is they really want.

Some folks think there’s too much work involved in that, so let’s just put it on a disk. Let’s store it there in the data lake, and let the users access it when they need it. There’s a nice phrase for it – schema on-read – when we want to do some analysis, we’ll figure out then what the schema for the data should be.

Now there are cases when that will need to be done, absolutely. But mostly you need schema on-write. In other words, you figure out what was the best way to store the data before you start using it.

Every time you try to create a spreadsheet – if you don’t think about the structure of the columns before you start, you are in trouble.

Q) Some would say that data-modelling has fallen out of favour. Do you think that is true?

A) Data modelling has in many ways fallen out of favour. You ask a yes/no question, but I can’t resist thinking about why? And I think it’s because many data-modellers—another generalisation—place too much focus on detail, and too great an insistence on rigour.

Yes, data models do need to be detailed and they need to be rigorous. But there are ways and means of doing it in a way that ordinary people can digest – and many modellers don’t consider that enough. There was a very old joke about data-modellers.

“What’s the difference between a data-modeller and a terrorist? You can negotiate with a terrorist.”

Data modellers, in some cases, consider their modes to be non-negotiable. “This is the model. The model shall be…” They almost have that religious fervour about them, about the model they are building.

Personally, I’m somebody who always wants to follow the middle path, and maybe that’s just my personality.

But when I look at a good data-modeller, they know how to apply the rigour. They know how to focus on detail in a way that can be used – and can be understood.

So, finding that balance between rigour and detail and usefulness to the businesses is the real magic of data modelling.

Q) Is that why you are a fan of the Data Vault?

A) One of the good things about Data Vault is that model, in my opinion, has done well in in bridging some of that gap between perfection and reality.

You can learn more about Barry Devlin’s thoughts on BI and data analytics at his website.