Skip to main content

Parallel Threads causes out of order events during CDC [SOLVED]

  • January 8, 2024
  • 5 replies
  • 142 views

sumek
Forum|alt.badge.img+1

we’ve observed a number of lost updates when running CDC. The state of initial load seems fine, but later on we came across NO_OP_UPDATES. I’ve ignored the exception for the CDC, but that resulted in some of the updates being lost and records left in stale state.

It seems that it might be connected with updates being in the same batch as the initial insert. How are events ordered within a single batch? Have you observed such scenarios? Source is SQL Server and target is Postgres in case thats relevant.

 

5 replies

sumek
Forum|alt.badge.img+1
  • Author
  • Known Participant
  • January 8, 2024

I’ve realised that I’ve set `Parallel Threads: 4` in my CDC app - this can cause the issues, am I right?


john
Forum|alt.badge.img
  • Community Manager
  • January 8, 2024

Yes as documented, Parallel Threads should be used for Initial Load only

You can parallelize CDC workloads using the Router component to write to multiple target components. 

This blog has some examples of that pattern.


sumek
Forum|alt.badge.img+1
  • Author
  • Known Participant
  • January 9, 2024

Even with `ParallelThreads: 1` we’re observing some missed updates ...


sumek
Forum|alt.badge.img+1
  • Author
  • Known Participant
  • January 9, 2024

We’re on 4.1.2 and I’ve just noticed that this issue got fixed in 4.2.0:

  • DEV-32632: MS SQL Reader > Database Writer with SQL Server missing events

Was it an issue only with SQL Server target? Our target writer is Postgres. Still talking about CDC


sumek
Forum|alt.badge.img+1
  • Author
  • Known Participant
  • January 10, 2024

I think we've located the issue within SQL Servers CDC tables, not Striim. Somehow they contained duplicate old entries which reverted the state of some rows to older state.