Essential Skills Checklist for a Successful Data Warehouse Project

A frequent question that clients ask us when planning a Data Warehouse project is “what skills do we need for our project to be successful?”. 

Data Warehouse projects integration many technologies and range from heavy data pipeline engineering to dashboard design, so you will require a mix of different skills. The exact mix will depend on your individual situation and the technology used in the legacy and target platforms.

To help you we have produced a checklist of the skills you might need.

• Data Engineering: Early in the project lifecycle there will be a heavy data engineering workload focused on pipeline building. Later in the project work will focus on orchestration, business rules, data marts and visualization. There will be a spike of activity when new data sources are added.

• Data Visualization: Visualization covers the preparation and display of data for end users. It usually involves the development of dashboards, embedded visuals and tabular data sets. Today, interactive dashboards are the default. Data visualization work will be steady throughout the project and will continue into business as usual. If you need to convert a significant set of legacy dashboards or reports, this may require the set-up of a larger team to work through the backlog.

• Testing: Testing work involves producing test data sets; set up of telemetry; and production of test plans and test code. Test effort is proportional to the effort put into data engineering. Although code generation, the use of modern techniques and automation tools will reduce the testing resources required there is always the need to test the developed system (see solution architecture/methods below).

• Data Modelling: Data modelling is needed for source system analysis, data dictionary, business glossary, data modelling, data flow and mapping. Data modellers/developers will require skills in the chosen data architecture and methods.

• Business Analysis: Business Analyst skills will be needed for gathering and interpreting user requirements, process modelling, conducting workshops and setting User Acceptance Test criteria.

• Solution architecture/method: Our preferred method is Data Vault 2.0.Whichever architecture and method are selected it requires skills and experience to drive the solution design and peer review technical work as it is completed.

• Cloud Engineering: We believe most projects today will be cloud-based. Expertise in exploiting the cloud platform, the platform provider’s tools, platform configuration, testing and managing upgrades (the continuous improvement of platform) will all be required.

• Agile Project Management: We recommend using an agile project management approach. Different organizations are at different stages in adopting agile techniques and there is a plethora of agile techniques available. Data warehousing is sufficiently different to require its own approach, do not reuse Agile approaches that work for other types of software development within your business.

• Coach: A coach is someone from outside the immediate team, who has experience of agile data warehousing, and can advise, challenge and shape the project helping to ensure the team has productive use of their selected agile techniques.

• Change Management / Training: There will be a need to train users and change management skills to ensure user adoption of new services. This will ensure benefits are realised and feedback is obtained.

• Data Analysis / Data Science: Depending upon your organisation’s objectives, leading edge analytics and data science skills may be needed for investigation and analysis. There may then be further skills to move Machine Learning/Artificial Intelligence applications into production for widespread exploitation.

• Security Engineering: Security skills are essential for security policy, design, implementation, verification and monitoring.

• Service Management: Service management skills for data operations such as service planning, design, delivery, monitoring and continuous improvement.

• Tool Specialists: There are increasingly sophisticated tools available to reduce the development effort and improve the long-term maintainability of the solution (such as WhereScape).Specialist skills will be needed to configure the tools and the DataOps function must embrace them to ensure they are used effectively.

These skills are diverse enough that no single member of the team will have mastered all of them. Your project team will be of a size where team members need deep expertise in more than one skill.This will tend to favour the use of more senior engineers. If your team has gaps, you’ll need to plan to cover them through training, coaching, or adding resources from within your organisation or externally.