Getting Data Into Your Warehouse Is Harder Than It Looks. Here's How to Do It Right
- Jul 2
- 5 min read
Most conversations about data warehousing focus on what happens once the data is inside the platform: transformation, modeling, reporting, AI readiness.
What gets far less attention is the step that makes all of that possible in the first place — getting data from your source systems into the warehouse at all. That process, which practitioners call data landing, is where many organizations quietly struggle. The technical complexity is real, but so is the organizational complexity. Both need a strategy.
This Is as Much an Organizational Problem as a Technical One
Before you configure a single connection or select a piece of software, you will likely face a more immediate challenge: politics. The data you need to land in your warehouse does not belong to you. It belongs to another system, managed by another team, often with their own priorities, their own security concerns, and their own instinct to protect what they own.
Getting agreement that data can flow at all requires negotiation. Once you have that agreement, you still need to decide how that flow happens — and the first decision is foundational: does your warehouse go and retrieve the data, or does the source system send it to you?
Push vs. Pull: Getting the Fundamentals Right
These two models define the mechanics of data landing. In a pull model, your data warehouse connects to the source system on its own schedule, queries what it needs, and brings it in. In a push model, something running inside the source system periodically extracts data and delivers it to a designated landing area in your warehouse, on the source system's schedule.
In our experience, pull has become the more commonly adopted approach, and for good reason. It puts control where it belongs: with the team responsible for the warehouse. You decide when each source is queried, in what order, and with what dependencies. If you need source A to load before source B, you can enforce that. With push, you are at the mercy of whatever schedule the source system runs on, and when multiple source systems are pushing data independently, consistency problems in the warehouse become almost inevitable.
That said, pull does not mean unrestricted access to a source database. A well-governed pull implementation typically works through an API — a defined interface that exposes only the data agreed upon, without granting direct access to the underlying system. This protects the source, satisfies security requirements, and makes the arrangement easier to audit and govern. It does require more up-front construction, but the trade-off is worth it.
The Case for Off-the-Shelf Tooling
Once the access model is agreed upon, the next decision is whether to build your own ingestion process or use one of the many tools available in the market. Our recommendation is clear: unless you are working with very simple, small, or rarely-changing datasets — or running a short-term pilot — use an established off-the-shelf tool.
The reason is straightforward. Tools like Fivetran, Airbyte, and others have been built specifically to handle the edge cases, failure modes, and recovery scenarios that any hand-built solution will eventually encounter. They understand how to handle partial loads, connection failures, schema changes, and the dozens of other things that can go wrong when pulling data across organizational and network boundaries. Building that robustness yourself is expensive and time-consuming, and the maintenance burden grows over time.
There is another practical advantage. When you manage multiple source systems through a dedicated ingestion tool, a failure in one source does not cascade across the rest. Each connection is treated independently, which means your warehouse keeps running even when one upstream system has issues.
Data Consistency Is a Recurring Pressure Point
Even with the right tooling in place, timing problems are common. You schedule a pull and the data is not ready yet. A dependency between two source systems means one has to wait for the other. A source system that should be refreshing hourly runs late — and your downstream processes are built around expecting it on time.
When these are recurring issues, the root cause is almost always in the source system, not the warehouse. Which creates a difficult dynamic: your team owns the problem from a business impact perspective, but you do not own the system causing it. This is where the organizational dimension of data landing becomes critical again. Getting source system owners to prioritize data reliability for downstream consumers requires executive alignment and, in many cases, formal data contracts that establish expectations on both sides.
CDC: Capturing Every Change, Not Just the Latest State
One of the most important capabilities in a mature data landing setup is Change Data Capture, or CDC. When you pull data on a scheduled basis — say, every hour — you retrieve a snapshot of how each record looks at that moment. But what happened in between? A record may have been created, updated three times, and then deleted, all within the same hour. A snapshot-based pull will miss that history entirely.
CDC addresses this by reading directly from the database transaction logs, capturing every individual change as it occurs. This means your warehouse can reflect not just the current state of the data, but the full sequence of changes that led to it — which matters enormously for auditing, compliance, and any use case where the history of a record is as important as its current value.
There are two practical considerations to be aware of. First, CDC setup varies significantly by database platform. SQL Server, for example, requires CDC to be activated on a table-by-table basis. Other platforms log changes as a matter of course. Second, CDC can carry additional cost, which is sometimes prohibitive depending on your licensing arrangement. Where that is the case, an alternative approach is to build a Persistent Staging Area — a layer within your own platform that stores historical snapshots and computes the differences over time, effectively replicating the benefit of CDC without relying on source system log access.
The Decisions That Matter Before You Build
Data landing is often treated as a technical implementation problem when it is really a set of architectural decisions that need leadership alignment. Which model — push or pull — fits your organization's governance posture? What level of access is appropriate for each source system? What consistency guarantees does your downstream reporting require? And what is your strategy when a source system is unreliable?
Getting those questions answered before selecting tools and writing configuration files will save significant time and rework later. The infrastructure matters, but the decisions behind it matter more.
Ready to take the next step?
If this has raised questions about how your organization is managing data ingestion, or about the overall maturity of your data platform, there are three ways we can help.
Talk to us.
If you want a direct conversation about your data architecture and where the pressure points are, get in touch. We work with senior data leaders and C-suite executives to cut through complexity and build platforms that perform.
Download our white papers.
We publish in-depth guidance on Data Vault, Medallion architecture, AI readiness, and modern data platform design — written for practitioners and leaders alike.
Take the Data Transformation Readiness Assessment.
Not sure where your platform stands? Our free diagnostic evaluates your organisation across four critical dimensions — platform foundation, reporting and insight speed, operational agility, and strategic positioning — and gives you a personalised report with a clear picture of where to focus. It takes less than ten minutes.