Barry Devlin Interview Part 2

Part 2 of our interview with Dr Barry Devlin data warehousing author and thought leader

Dr. Barry Devlin is one of the founders of the data warehousing industry, a recognised authority on data warehouses, business intelligence (BI), big data and beyond. Barry has authored two ground-breaking books: the classic “Data Warehouse–from Architecture to Implementation” and “Business unIntelligence-Insight and Innovation Beyond Analytics and Big Data”. Available Here.

Barry has over 30 years of experience in the IT industry, previously with IBM, as a consultant, manager and distinguished engineer. As founder and principal of 9sight Consulting in 2008, Barry provides strategic consulting and thought-leadership to buyers and vendors of Business Intelligence and Big Data solutions.

This is the second part of an interview by our Andrew Griffin, who spoke to Barry about his experiences, and his thoughts about the future of the industry.

Q) When seeking Business Intelligence solutions, how do you go about understanding and supporting the needs of different departments/divisions ensuring they all pull in the same direction?

A) That’s a great question because it’s one of the central problems that always comes up in Business Intelligence, and in all sorts of information projects.

If you go back to the early days the question was, always this: Why is it that when the finance people and the salespeople and the whatever people arrive in the CEO’s office, to address the problem with this month’s sales, not only do they have different answers as to what the problem is, but they’ve got different numbers. And that’s always been one of the fundamental questions in Business Intelligence – how do you get everybody talking the same language and pulling in the same direction?

In the early days, and for many years, there was this phrase called a “single version of the truth,” which became very popular. And yet the reality is that it’s only correct at the very highest level of the organisation. Because the truth for finance is a very different truth than it is for sales.

The finance people know they have to satisfy legal requirements etc., so their truth is much more of a global truth than the sales people, who actually have to get commissions paid and get sales counted.

So, you actually don’t want them to be all pulling in exactly the same direction, but they need to have a basis or a core around which they can consolidate whatever they’re doing.

That’s why I think you do need to consolidate meanings and definitions – the idea of meaning and intent, and definitions, is really important. You’ve got to get them aligned only to the extent that it serves the business. If you try to go further than that, you simply cause more chaos.

So yes, you do need to understand and support the needs of different departments. But you do need to be able to start looking for the core pieces of the business, the core pieces of the data model if you like, that have some fundamental common meaning and agreement.

Artificial intelligence (AI) promises to tell you what’s going to happen tomorrow, and that’s why it’s so popular – because people think it’s going to give them certainty. But the reality is that it is based on past data, and a statistical interpretation of that past data. Worryingly, that may be easily biased by the data that has been collected.

So, it is about as reliable as reading the entrails in telling you what it’s going to happen tomorrow if the world changes dramatically, as it did during the pandemic.

Lots of people had to throw away their AI models then, because they were no longer useful. Talking about looking at the future, of course, that’s what most of analytics and AI today is about.

It’s trying to do some prediction of the future and what I have found is that if you understand the personalities of the people and their history, and how their organisations work, then you can actually say something meaningful about the future, because people and organisations often behave in predictable ways based on these factors.

Q) So what do you think will be the biggest change over the next decade in Business Intelligence?

A) AI is where BI is going. There’s only one letter in difference between BI and AI! Without a doubt, business intelligence is moving toward analytics in terms of becoming more driven by statistical methods, becoming more about the underlying data volumes and so on. And then analytics goes to artificial intelligence in the same way, using AI as the basis for driving decision-making.

Of course, that is part of the goal of AI. So I think the next biggest developments in the next decade in this field are going to come from AI – for good or for ill.

And for me, the real challenge is in finding the right balance between automation and augmentation, which is the spectrum of what can be achieved with AI. At one

end of the spectrum you want to automate every human contribution out of existence, and the process just happens. So automation gets rid of the workers.

At the other end of the spectrum, which is the one that the AI companies tend to focus on because it sounds more socially acceptable, augmentation helps people to do better work. So getting that balance right in terms of AI becomes really, really, important because when you start taking people out of the decision-making process as you do with automation, then, of course, you get some very bad decisions being made – simply because the data says you should make them. AI might say you must do X, but on the other hand, if you were thinking about it from a human point of view, you’d probably do the exact opposite.

The problem is that with AI you take humanity out of the loop. So that’s my main worry as we go forward. On the other hand, there are things you can do better with AI, as long as you’ve got the right data underneath.

And that’s the other real problem, which is you have to get your information governance right.

Q) Everyone seems to be in a rush to store and manage their data in the Cloud and technologies like Snowflake have made that option easier and cheaper. What would you recommend?

A) When you’ve been in the business as long as I have, you get to the stage where you see the same things come around again under different names.

Take data bureaux in the 1960s – they were basically the Cloud of the ’60s right? Somebody else was going to run your data centre. Somebody was looking after your data. Different technology, different scale, etc., of course, but the same idea.

If you really look at Cloud versus on-premises, the balance is about who does what – and who does the scaling for you, and who does the management of the technology and so on.

I think there’s an awful lot of hype around the Cloud and I think there’s an awful lot of expectation, which may or may not be met. Cloud is just a new technology. It’s just a new platform, yes, but it has echoes of platforms that we had in the past.

So what do you need to do with it? To me, if I go and talk to a client – and let’s say they are a big bank, or a big retailer – and they’ve got a huge investment in huge databases on Teradata and the machines running it, then what specifically makes technical and financial sense for them?

Should they go to Snowflake? What’s it going to cost? Will they be able to migrate? What are the benefits of going there? Do they really need to have their data available widely through the Cloud? But you have to look at something called data gravity.

Where did the data come from? Where does the data want to be? If it’s all on the ground, putting it up to the Cloud is all very well. But if it needs to be brought down again, you have to look at the balance of costs and benefits.

Legacy organisations and legacy businesses have such an investment in on-premises that although there will be things that make sense to move to the Cloud – and there will be companies that say everything should be on the Cloud, I suspect it’s a bit like IBM mainframes – they never went away even though everybody declared them dead in 1978.

So I think on-premises is going to be the same. I think there will be particular types of problems, particular types of businesses that do benefit hugely from being in the Cloud and others, not so much.

Ultimately, I think that these solutions will all be hybrid. There’s often this urge to try to have a single solution that does everything because you think that’s going to be the most efficient. But in reality there’s another side of the equation, which is that every technology has its strengths and weaknesses. So I think in most cases you will end up having a bit of both.

And let’s be honest, the only significant difference between Cloud and on-premises is that Amazon or Google owns the compute the Cloud runs on. The technological differences underneath that are less important.

Whenever anybody says “Should we go to the Cloud?” The first thought is always financial rather than architectural.

When I talk to my clients about it and talk about data gravity, I think that brings them back to Earth. Then they need to think about, well, what have I got? Where is my stuff? Where are my assets? Where does it really work?

Q) So given the Cloud debate, and what you said about data modelling, what kind of data platform architecture should businesses be thinking about?

A) Selecting a data platform architecture today is about finding trade-offs between key aspects of what you want, and there are two that I always keep coming back to these days. We talked about one of them already, it’s timeliness versus consistency in terms of information.

Do I need timeliness or do I need consistency? What combination of those two things do I need? Clearly, if I’m in the Cloud, timeliness is probably easier. If I am on-premises, to some extent, consistency may be easier.

And the other big consideration is structure versus context. Of course, context leads us into meaning as well. But structure versus context – understanding the context of your information and the usage to which it’s put does give you some indications of what data structure you need to use for it, and now you get technical.

Do I need to put it into a relational database, or am I better off leaving it in its raw form, so the debate about structure and context becomes a very interesting debate in terms of choosing the right data platform.

But it’s horses-for-courses. You must look at the individual company and the goals of the company – and the strategies and the visions of the company – in order to understand what those right answers should be.

Q) What do you think will be the lasting transformation in data science resulting from the pandemic?

A) We all read a lot over the last year about how the pandemic has transformed business, and that’s been a theme I’ve used as well.

And yet what I’m watching going on now is that in government and many companies, management is rushing as fast as possible to try to get back to where things were before. There should be lessons from the pandemic – whether that is in terms of understanding the usefulness of data, or elsewhere.

I wrote an article very early on in the pandemic about how so much of the data being reported was far from ideal, and how the data that we used didn’t really tell us enough to be truly useful. So, we should be learning a lot from the

pandemic in terms of how we collect data, how we use it, how we analyse it etc. But the reality I think is that we haven’t learned a lot and we’re very slow learners.

People thought that we’d all stay working from home, but lots of the big companies are not so happy with that anymore, and I’m not so sure why.

So I don’t think there’s going to be a lasting transformation. I think maybe we’ll pick up a few lessons here and there at best.

But my other big concern is the climate emergency, and as far as I’m concerned, we’re all doomed anyway, so it doesn’t really matter!

Q) Some people describe data scientists’ role in the 21st century as being very “sexy.” What advice would you offer to a young graduate looking for a career in Business Intelligence and data analytics?

A) I would say the most valuable thing they could do is to focus on the meaning and the usage of information and how to discover them, document them, and apply them in systems design.

Probably the best-paid role will be to focus on the technology and become the best Snowflake programmer on the Cloud and you’ll make a lot of money.

But will you really solve the problems the business wants to solve without understanding information meaning and usage? I’m not so sure.

You can learn more about Barry Devlin’s thoughts on BI and data analytics at his website www.9sight.com