I'm going to have 2 always running CDC parallel jobs for real time ETL. I would like to get some inputs from members on apart from reject links, are there any other error handling functionalities available in the parallel jobs which would accomplish -
1. The job should not crash and should always stay running.
2. Log data related errors somewhere.
3. Log data somewhere which produced errors.
4. Does datastage have something that can handle errors like try..catch block in .net?
5. How to handle Null related errors.
6. How to handle xml parsing errors in xml_input stage.
Error handling in a CDC job
Moderators: chulett, rschirm, roy
Re: Error handling in a CDC job
No reply from anybody? Let me rephrase my question.
What is the best practice for handling errors in CDC parallel job?
What is the best practice for handling errors in CDC parallel job?
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
1. Prevention is the key. Write your jobs and test them so that they cannot crash. That said, don't forget to include some mechanism whereby the job can check whether it has been instructed to shut down (for example for system maintenance).
2. If you can intercept data errors via tests in the ETL flow, you can dispose them however you wish.
3. If you can intercept data errors via tests in the ETL flow, you can create alternate flows for the erroneous data.
4. No. But you can accomplish this with stage variables and If..Then..Else constructions.
5. By using the Null testing functions IsNull() or IsNotNull(). But what is it, exactly, that constitutes a "null related error"? Are you overthinking this?
6. Prevention is the key. Ensure (perhaps by beating up on the provider) that the XML you receive conforms exactly with the XSD with which you were provided.
2. If you can intercept data errors via tests in the ETL flow, you can dispose them however you wish.
3. If you can intercept data errors via tests in the ETL flow, you can create alternate flows for the erroneous data.
4. No. But you can accomplish this with stage variables and If..Then..Else constructions.
5. By using the Null testing functions IsNull() or IsNotNull(). But what is it, exactly, that constitutes a "null related error"? Are you overthinking this?
6. Prevention is the key. Ensure (perhaps by beating up on the provider) that the XML you receive conforms exactly with the XSD with which you were provided.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.