Striim Hub

Do you have a question? look no further

18 Conversations

Emrah
EmrahParticipating Frequently
asked in Community Q&A

CDC problem with 21C

Dear All,i am trying to create 2 pipelines from Oracle to Postgres.the first one is initial Oracle to Postgres, it worked very well, and schema conversion tool migrated 2 example tables to postgres. but i have an issue with Oracle CDC.i have logminer user on oracle which is striim and i have table owner which is scale_user and i have 2 tables scale_data and scale_data2.i am not using container and PDB on 21C, striim and scale_user are common user.i am receieving an error like below when i insert a data to the tables after commit.i see that reader is looking for tables like CDB$ROOT.TEST.SCALE_DATA but i am not using container so how can resolve the issue.There was no problem with initial load, the problem is related with CD process. Thanks in advance. [{ "_id" : null, "timeStamp" : 1707166962232, "originTimeStamp" : 1707166964000, "key" : null, "sourceUUID" : { "uuidstring" : "01eec465-3101-b301-b854-42010a9c000a" }, "data" : [ "862412", "1", "1", "460" ], "metadata" : { "RbaSqn" : "50

8
Emrah
2 months ago
B
bobbyilievNew Participant
asked in Product Q&A

MySQL Reader + KafkaWriter: How to specify individual topics for the individual MySQL tables?

Hey! I have the following setup:MySQL → Striim → KafkaThis all works as expected, but if I have multiple MySQL tables, all changes are streamed into 1 single Kafka topic.Is there a way to configure the KafkaWriter to stream the messages to dedicated topics for each table?For exmaple:Events from table `users` should go to a topic `users` Events from table `items` should go to a topic `itmes`As far as I can see in the docs, you can only specify one topic per KafkaWriter.

2
B
2 months ago
B
bobbyilievNew Participant
asked in Product Q&A

Suitable Avro type not found for field error when using MySQL -> Striim -> Kafka

I’ve got the following setup:MySQL → Striim → Kafka + CSR and avro encoding The table that I’m replicating is pretty simple: items | CREATE TABLE `items` (  `id` bigint unsigned NOT NULL AUTO_INCREMENT,  `name` varchar(100) DEFAULT NULL,  `category` varchar(100) DEFAULT NULL,  `price` decimal(7,2) DEFAULT NULL,  `inventory` int DEFAULT NULL,  `inventory_updated_at` timestamp NULL DEFAULT CURRENT_TIMESTAMP,  `created_at` timestamp NULL DEFAULT CURRENT_TIMESTAMP,  `updated_at` datetime DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,  PRIMARY KEY (`id`),  UNIQUE KEY `id` (`id`)) ENGINE=InnoDB AUTO_INCREMENT=2001 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci However the bigint seems to be causing the following error when Striim starts ingesting messages:Message: KafkaWriter is unable to produce events to shop. Component Name: Kafka_2_1_0_mysql_kafka_csr_Target. Component Type: TARGET. Cause: KafkaWriter is unable to produce events to shop. Cause: Suitable Avro type not found fo

2
B
2 months ago
mahesh_gupta
mahesh_guptaNew Participant
asked in Product Q&A

Is Postgresql to Oracle CDC pipeline possible ?

Hi Team, Can we create a Postgresql to Oracle CDC pipeline to make sure oracle is sync with postgresql post migration.  Regards, Mahesh

1
john
3 months ago
sumek
sumekKnown Participant
posted in Community Q&A

Parallel Threads causes out of order events during CDC [SOLVED]

we’ve observed a number of lost updates when running CDC. The state of initial load seems fine, but later on we came across NO_OP_UPDATES. I’ve ignored the exception for the CDC, but that resulted in some of the updates being lost and records left in stale state.It seems that it might be connected with updates being in the same batch as the initial insert. How are events ordered within a single batch? Have you observed such scenarios? Source is SQL Server and target is Postgres in case thats relevant. 

5
sumek
3 months ago
sumek
sumekKnown Participant
asked in Community Q&A

adjusting timezone

I’d like to convert a timestamp from location specific one (in areas that have summer time change) to UTC. What would be the best way to do it within Striim? I assume it’s doable using Joda API and I can see in the reference that “Striim supports all date functions natively associated with Joda-Time.” but I didn’t find an example how to call those Joda Time functions

5
sumek
3 months ago
Sweta Garai
Sweta GaraiCommunity Manager
asked in Product Q&A

Error saving CQ with Modify

I am trying to write a CQ with MODIFY statement that replaces all “Not Available” to NULL but I am getting an error “Left expression do not refer to an object”. Please refer to the screenshot for reference. 

4
sumek
3 months ago
mahesh_gupta
mahesh_guptaNew Participant
asked in Community Q&A

Data Validation report post migration

Hi All, Greetings! We need to migrate data from Oracle to PostgresSQL using striim. In this case does striim provides data reconciliation report post loading the data in the target system. Regards, Mahesh

1
john
3 months ago
sploiselle
sploiselleNew Participant
asked in Product Q&A

MySQL to Kafka+CSR

Trying to set up:Source: MySQL replication hosted in RDS, which works on its ownTarget: Kafka (works on its own) + CSR (untested) hosted on Confluent Cloud--using the AvroFormatter.The CSR is untested bc it doesn't seem like CSR is supported w/ the builtin load generators--though maybe I'm wrong.When I deploy it, I get:Error Deploy failed! java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetExceptionI’ve tested the connection to the CSR and it works, so I don’t believe it’s a credential issue.

2
sploiselle
4 months ago
john
johnCommunity Manager
asked in Product Q&A

Striim vault with Google Secrets Manager not working

I created a Vault that integrates with Google Secrets Manager. As the documentation suggests, we use a service account key with ‘Secrets Accessor’ permissions.However we’re getting the below error 

1
swaritjoshipura
4 months ago
sumek
sumekKnown Participant
asked in Community Q&A

CDC Postgres UUID to SQL Servers uniqueidentifier

Hi all, I’m trying to set up a CDC from Postgres(GCP Cloud SQL if it makes any difference) of a table with a UUID column to a SQL Server table with uniqueidentifier column. The wizard says it’s not possible. I’ve tried mapping the tables manually, but all the rows end up being discarded. Is there any way to make it work? Similar migration in the opposite direction works. 

5
sumek
6 months ago
S
sriram.pasamCommunity Manager
asked in Product Q&A

Syntax error in Continuous query(CQ) to inject NULL value for String field value

I am building a CQ to inject NULL values for String data fields that have “Not available” in the source database.CASE WHEN META(o,"TableName").toString()=="<schema-name>.<table-name>" then CASE WHEN TO_STRING(data[12]) ="Not Available" THEN putUserData(o, 'DENOMINATOR', "NULL") WHEN TO_STRING(data[13]) = "Not Available" THEN putUserData(o, 'SCORE', NULL) WHEN TO_STRING(data[14]) = "Not Available" THEN putUserData(o, 'LOWER_ESTIMATE', NULL) WHEN TO_STRING(data[15]) = "Not Available" THEN putUserData(o, 'HIGHER_ESTIMATE', NULL) ELSE OI am receiving the following compile error : Error Saving ComponentSyntax error at: CASENULL)Please fix the errors and click "Save" again. I also tried the following syntaxCASE WHEN TO_STRING(data[12]) ="Not Available" THEN NULLWHEN TO_STRING(data[13]) = "Not Available" THEN NULLWHEN TO_STRING(data[14]) = "Not Available" THEN NULLWHEN TO_STRING(data[15]) = "Not Available" THEN NULLReference docs : https://www.striim.com/docs/en/handling-null

1
S
7 months ago
igg-mario
igg-marioNew Participant
asked in Product Q&A

Forwarding Agent encryption in transit

While it’s great that you offer a Forwarding Agent for sources/destinations that have no direct public IP, I’m having a hard time finding any information as to whether or what sort of encryption in transit is used for the data conveyed by the Agent. And before I go digging into testing what it appears to do, I thought I’d ask here:  Does Striim Forwarding Agent use any encryption for any of its communication?Context here is that it was suggested to me that Striim might be useful in migrating from an antiquated MySQL ring replication cluster to a modern MySQL master-master cluster. Neither source nor destination cluster has, nor will have, a public IP address, and will not have any direct access available to to the greater internet.

3
igg-mario
11 months ago
john
johnCommunity Manager
asked in Product Q&A

Streaming to kafka topic in confluent cloud

Are there any examples of using Striim’s KafkaWriter to stream data to a topic in Confluent Cloud? 

4
Robert Lauriston
1 year ago
john
johnCommunity Manager
asked in Product Q&A

How do we flatten JSON in Striim?

I have a JSON Field in my PostgreSQL database with this format I want to take all the fields in this flat JSON structure and turn each of them into fields in a typed stream. How would I go about doing that? I tried using the ‘makeJSON’ function on the JSON string column but I’m unable to run JSONNode function on it.  Running the above query results in a ‘CRASH, cannot map’ exception.

1
S
1 year ago
D
Dabu720New Participant
asked in Community Q&A

Oracle CDC

We are using Striim to replicate data from Oracle to BigQueryI would like to know how can I find out for UIcurrent archived logs being mined by log miner The current DDL ( LCR , logical count record ) being processed)  Number of DDLs or LCR in the current Transaction or SCNThank You

2
D
1 year ago
Sweta Garai
Sweta GaraiCommunity Manager
asked in Community Q&A

Soft Deletes in Striim

How do I handle soft deletes in Striim? Does Striim performs an update process to mark some data as deleted instead of physically deleting it from a table in the database?

1
john
1 year ago
john.smith
john.smithNew Participant
asked in Community Q&A

Key Steps to launching a new CDC pipeline.

What are the key steps to launching a new CDC pipeline successfully? Wanted to get some ideas from the community on how to best approach that.

1
S
1 year ago

Badge winners

Show all badges
Terms & ConditionsCookie settings