Error handling in a CDC job

[mayurkb]

I'm going to have 2 always running CDC parallel jobs for real time ETL. I would like to get some inputs from members on apart from reject links, are there any other error handling functionalities available in the parallel jobs which would accomplish -
1. The job should not crash and should always stay running.
2. Log data related errors somewhere.
3. Log data somewhere which produced errors.
4. Does datastage have something that can handle errors like try..catch block in .net?
5. How to handle Null related errors.
6. How to handle xml parsing errors in xml_input stage.

[mayurkb]

No reply from anybody? Let me rephrase my question.
What is the best practice for handling errors in CDC parallel job?

[chulett]

I don't think your issue with answers is the "CDC Parallel job" part, it's the "always running job" part that complicates things. I wonder if this would fair better over in the SOA/RTI forum?

[ray.wurlod]

1. Prevention is the key. Write your jobs and test them so that they cannot crash. That said, don't forget to include some mechanism whereby the job can check whether it has been instructed to shut down (for example for system maintenance).

2. If you can intercept data errors via tests in the ETL flow, you can dispose them however you wish.

3. If you can intercept data errors via tests in the ETL flow, you can create alternate flows for the erroneous data.

4. No. But you can accomplish this with stage variables and If..Then..Else constructions.

5. By using the Null testing functions IsNull() or IsNotNull(). But what is it, exactly, that constitutes a "null related error"? Are you overthinking this?

6. Prevention is the key. Ensure (perhaps by beating up on the provider) that the XML you receive conforms exactly with the XSD with which you were provided.