9 Hard Problems in Agile Data Warehousing and How to Address Them

What are the “hard problems” that your team is dealing with? First, you may ask “what are hard problems?”. Scott Ambler – agile data coach and consulting methodologist – outlined exactly what he means by this in his excellent talk about “Agile Data Warehousing / Business Intelligence: Addressing the hard problems” during the Data Vault User Group’s recent meetup.

“Strive to become a learning and purposeful organisation”

Agile Data Ways of Thinking

He set out the foundations for tackling hard problems, it’s going to take agile data ways of thinking to be successful. These are:

• Look beyond data – data is important, but so are other things.

• Collaborate closely – both within teams and with stakeholders.

• Be quality infected – data quality problems need to be fixed in the industry.

• Embrace Evolution – the world changes, that’s no different with data.

• Be enterprise aware – do what’s right for the organisation, and the world. Both team and enterprise awareness are key.

• Fit-for-purpose – one size doesn’t fit all. Be flexible and context sensitive.

What is Agile Data Warehousing and Business Intelligence (BI)?

Some people think that building a data warehouse is like building a house. A plan, a schedule, and then following through with the plan. Scott points out that building data warehouses is unpredictable. It doesn’t matter how good your plan is at the start, there will always be unexpected problems arising.

Agile data warehousing/ BI, as outlined by Scott Ambler, is the act of providing quality information in a collaborative and evolutionary manner.. Rather than making data quality tweaks as and when problems arise in the warehouse, they should be resolved right at the source.

Be flexible, though. You can only control what you can control.

Now lets take a look at the “hard problems” that Scott was talking about in the Data Vault User Group meeting:

The “hard problems” with Agile data warehousing and Business Intelligence – and how to address them

1. Upfront data architecture WORK TAKES too long

Data Vault 2.0™ has done a lot of the thinking for you. Identify the main data sources, the goal is to understand the physical landscape. You don’t need to know all the tiny details, just the fundamental information.

Scott speaks about two key diagrams: a data source diagram, and a conceptual diagram. These key models help to reduce the time it takes to produce upfront data architecture. In addition, just barely good enough (JBGE) artifacts are a great way to reduce time investment, but still deliver the same results. Think about it, when something is more than good enough, any further investment in it is a waste.

2. We need to deliver value every sprint

Delivering value every sprint doesn’t need to be so challenging. Adopting ‘vertical slicing’ is a great way of making it easier to deliver immense value every spring.

But what is vertical slicing? It means breaking down tasks into, you guessed it, vertical slices. Meaning smaller wins delivered consistently, rather than the pressure to deliver a larger task in a sprint. Work on one focused task at a time to deliver value every sprint.

3. We need to install required infrastructure

Calling back to a previous problem, leverage vendor solutions to install required infrastructure much more easily. In addition, embrace evolution. Recognise that you don’t need everything correct right away. Your situation will always be changing.

4. We don’t have enough agile data people

Having enough agile data people in your organisation isn’t easy, but Scott highlighted a great way of countering this problem. Look for generalising specialists, rather than a team of specialists. Generalised specialists are people who have a good level skillset in all four of the agile data roles (Data architect, scientist, engineer, and analyst).

The best way to achieve this to invest in your team through training, coaching, and collaborative work.

5. Sprints are too slow we need to go faster

Running your first release as an agile project and evolving via lean continuous delivery are how Scott suggested its best to tackle this problem.

As well as these solutions, shortening your sprints and negotiating stakeholder expectations help. Shorter sprints motivate you to squeeze the waste out of your way of working, and get you closer to a continuous delivery strategy, and maybe the real issue is misalignment. That is where negotiation stakeholder expectations management could help.

6. Data analytics often takes longer than a single sprint

In his presentation, Scott highlighted a scenario to address this issue. The look-ahead data analysis (continuous delivery) means more creator flexibility, and less overhead. Staffing your team with specialists will exacerbate work scheduling challenges that you might face.

7. My product owner doesn’t understand the data

Product owners must collaborate closely with the agile data professionals. Ideally the product owner would have the appropriate skills to understand the data, but that might not be true. Train or coach the product owner to make the process a lot smoother.

You could also adopt inclusive modelling tools and techniques to enable stakeholders to explore their own requirements. This supports a strategy of self-serve data.

8. We have too much data technical debt

Data technical debt generally refers to challenges associated with data sources. It impedes the ability of your organisation to leverage info effectively for informed decision-making – BE QUALITY INFECTED! One of the ways to do this is with database refactoring. Scott recommended this book that he is a co-author of.

9. Our stakeholders don’t want to work with us

Deliver value frequently as opposed to “big release” value delivery. Frequent value delivery delivers value from early on. This opens the door to concrete feedback and to steer. It’s flexible, but requires the team to be skilled and disciplined.

Ultimately, involve your stakeholders in the process, and make it enjoyable for them! Find out more about Scott Ambler’s work at Scott Ambler.com