Skip to main content

we’ve observed a number of lost updates when running CDC. The state of initial load seems fine, but later on we came across NO_OP_UPDATES. I’ve ignored the exception for the CDC, but that resulted in some of the updates being lost and records left in stale state.

It seems that it might be connected with updates being in the same batch as the initial insert. How are events ordered within a single batch? Have you observed such scenarios? Source is SQL Server and target is Postgres in case thats relevant.

 

I’ve realised that I’ve set `Parallel Threads: 4` in my CDC app - this can cause the issues, am I right?

Yes as documented, Parallel Threads should be used for Initial Load only. 

You can parallelize CDC workloads using the Router component to write to multiple target components. 

This blog has some examples of that pattern.

Even with `ParallelThreads: 1` we’re observing some missed updates ...

We’re on 4.1.2 and I’ve just noticed that this issue got fixed in 4.2.0:

  • DEV-32632: MS SQL Reader > Database Writer with SQL Server missing events

Was it an issue only with SQL Server target? Our target writer is Postgres. Still talking about CDC

I think we've located the issue within SQL Servers CDC tables, not Striim. Somehow they contained duplicate old entries which reverted the state of some rows to older state.  

Reply


Terms & ConditionsCookie settings