Key Steps to launching a new CDC pipeline.

CDC is different than batch, so has different considerations. In broad terms, the same concepts are relevant: scalability, data freshness, and monitoring. Let's dive in to each one:

Scalability. CDC pipelines put stress on the source database more so than batch pipelines. The source database has to have enough memory for the initial load and enough replication slots to support ongoing runs. It's important to involve the engineering team (or vendor) owning the source database to optimize the configuration here.

Freshness. One reason organizations opt for CDC is to get data into the target warehouse (or other destination) faster, even near real-time. Sometimes, a small lag is tolerable. Getting stakeholder buy-in and understanding for data SLAs surrounding CDC will make sure you provision resources appropriately based on the requirements.

Monitoring. Given more resources and stricter data SLAs, you need to make sure nothing is broken as it will have a more immediate impact. Monitoring a CDC pipeline both preventatively predicting issues as well as alerting when issues happen will help make the pipeline more reliable. A few things to monitor include: resources of the source database, freshness of data in the target, and pipeline errors.