Page 1 of 1

Retry on JDBC connector

Posted: Wed Aug 23, 2017 2:46 am
by clarcombe
Occasionally we get timeout errors when trying to read DB2 on Z/OS using a JDBC stage.

However on the next run of the batch it then works.

For information, this is the error that occurs

Code: Select all

Fatal Error: waitForWriteSignal(): Premature EOF on node vdatastagep.fmsb.be Bad file descriptor
This is one of a number of tables read by a generic routine. Its never the same table that fails twice and the batch can go days without failure


Question

Is there an option in the JDBC stage to allow us to retry x times after a time period has elapsed e.g. 1 minute ?

Are there any other ways to get around this ?

Posted: Wed Aug 23, 2017 10:49 am
by asorrell
We had some jobs that failed intermittently due to bad network latency. Because the jobs were safe to re-run after any failure, we wrapped them in job sequences that re-ran the jobs automatically twice, but aborted on the third failure.

Posted: Thu Nov 16, 2017 3:12 am
by clarcombe
Andy,

Thanks for this.

Can you elaborate on this technique please ?

Our read/write jobs are already being run in Job Sequences and are called from a main Job Sequence.

Are you suggesting that we add a 3rd layer of Job Sequence calls to this ?

Posted: Thu Nov 16, 2017 8:43 am
by qt_ky
That is what we had to do also, because our network is so alarmingly unreliable.

Our highest level of sequence jobs--the ones we place on schedule--perform auto-retry logic by using a loop. The loop uses parameters for number of retries and sleep interval. Doing this has saved us countless manual interventions in the middle of the night!

We may schedule one sequence to retry 6 times and wait 10 minutes between each retry, and schedule another to retry 8 times and wait 1 hour between each retry, etc.

Some pointers: take advantage of the job properties to "add checkpoints so sequence is restartable on failure" and when you call jobs make sure to choose execution action of "Reset if required, then run" which by the way is not the default setting.