The Right Team Size for Data Engineering Success

Based on episode 14 of The Business Thinking Podcast with our CEO, Neil Strange, and AutomateDV Product Manager, Alex Higgs.

In a recent podcast episode, we tackled a persistent an interesting question: what is the ideal team size for a data warehousing project? We’ve observed a staggering disparity in team sizes across various projects, ranging from a lean two or three individuals to sprawling teams of 50 or 60. This begs the question: where does the sweet spot lie, and why does such a vast discrepancy exist?

The Illusion of Efficiency

One of the most concerning trends we’ve witnessed is the correlation between large team sizes and manual coding practices. These projects often suffer from significant delays and a noticeable decline in productivity. As communication overhead increases, projects grind to a halt. This phenomenon is not new; it’s a well-documented issue dating back to the 1960s, as outlined in Fred Brooks’ seminal work, "The Mythical Man-Month."

The common misconception is that throwing more people at a problem will accelerate its resolution. However, as we discussed, there’s a point of diminishing returns. Beyond a certain threshold, adding more team members actually hinders progress.

The Skillset Spectrum: Balancing Breadth and Depth

To determine the optimal team size, we must first consider the diverse skill sets required for a successful data warehousing project. These include data engineering, testing, business analysis, solution architecture, platform engineering, dashboard development, programming (e.g., Python), data modeling, and specific tool expertise.

Finding an individual with proficiency in all these areas is akin to discovering a unicorn. Therefore, the challenge lies in distributing these skills across a team without overloading any single member.

The Magic Number: Aiming for a Team of Five

After careful consideration, we concluded that a team size of approximately five individuals strikes the ideal balance. This size allows for a comprehensive skill set coverage while minimizing communication overhead.

As we noted, “If you whittled it down, sort of the minimum skills team size should be around about 5-5 people, I think.”

This approach maximizes productivity, ensuring that team members can focus on their respective areas of expertise without being overwhelmed.

The Power of Automation: Doing More with Less

One of the key factors in optimizing team efficiency is the strategic use of automation tools. By automating tasks such as data landing and data loading, teams can significantly reduce the manual effort required.

We highlighted examples where projects wasted months of effort on manual coding for tasks that could have been automated in a matter of days. As we discussed, “Most of the time we just setup Fivetran. So you provide it with a list of tables and it does it for you.”

Automation not only accelerates project timelines but also reduces the need for large teams, leading to significant cost savings.

Addressing the Challenges of Large Teams

Large teams often grapple with challenges such as lengthy onboarding processes, lack of documentation, and inconsistent standards. These issues can be mitigated through effective documentation and the implementation of standardized patterns, which, in turn, facilitates automation. As we pointed out, “It’s always going to be around on boarding and upskilling spend. Two or three weeks on boarding, learning, learning what they need to do and how their particular environment works.”

The Bottom Line: Efficiency Over Size

In conclusion, the ideal data warehousing team is not defined by its size but by its efficiency. A team of approximately five skilled individuals, leveraging automation tools and adhering to standardized practices, can achieve remarkable results.

As we emphasized, “Good team size about 5 make good use of tooling. And make sure you've got a good mix of skills across your team.”

By prioritizing efficiency over size, organizations can optimize their data warehousing projects, delivering high-quality results in a timely and cost-effective manner.