Multi-instance job log cleanup

dls · Post by **dls** » Tue Aug 16, 2016 6:56 am

I am creating multi-instance jobs that are intended to run only once. The same instance of the job will never be run again unless there is a failure and need to rerun.

My understanding is that auto-purge criteria is only considered the next time the instance runs.

I'm looking for a technique that will allow me to purge to multi-instance job log after it successfully completes. Otherwise, my Director log fills with instances that I no longer need or want.

chulett · Post by **chulett** » Tue Aug 16, 2016 7:49 am

dls wrote:My understanding is that auto-purge criteria is only considered the next time the instance runs.

No, it happens each time the job runs, regardless of instance.

dls · Post by **dls** » Thu Aug 18, 2016 1:10 pm

Thanks, Craig. I'm setting auto-purge action to 'older than 2 days'.

ray.wurlod · Post by **ray.wurlod** » Fri Aug 19, 2016 3:45 pm

Just to be pedantic, log purge is initiated only if the job completes with no errors. That is, if the job aborts or is stopped, no automatic purge of log events occurs. That means that you can compare the current run's log with the previous.

dls · Post by **dls** » Tue Jan 17, 2017 8:56 am

I've reopened this topic.

Even though the logs are being purged as directed, the multi-instance job details are still appearing in my Director client view.

Is there an automated technique to remove the job details from the Director view when all logs for that (multi-instance) job have been purged?

PaulVL · Post by **PaulVL** » Tue Jan 17, 2017 9:15 am

It has also been my observation that multi instance job logs only purge upon the next execution of that job+invocation id. The purge logic happens at the start of that cycle.

Notice I said observation. I am perfectly willing to be wrong on this one.

Our site also suffers from that type of log buildup as someone coded a date-stamp in the invocation ID of some jobs.

chulett · Post by **chulett** » Tue Jan 17, 2017 10:57 am

From what I recall, if you recompile the base job all of those invocations in the Director will go away.

JRodriguez · Post by **JRodriguez** » Tue Jan 17, 2017 3:07 pm

...There is a bug in IIS8.7 that prevent the multi instances status records from being purged or cleared. The actual DS logs will be cleared as per the setting in the auto purged but not the name of the instances in the DS director view and will become slower and slower to refresh the logs due to the increasing number of empty records...

Fortunately there is patch for this bug patch_JR46141 ..just request the patch for your specific environment

dls · Post by **dls** » Tue Jan 17, 2017 3:21 pm

Thanks, I have submitted a PMR. Hopefully the patch can be applied to v8.5

PaulVL · Post by **PaulVL** » Tue Jan 17, 2017 5:02 pm

FYI:

I'm in 11.3 FP2 and I do not see that my logs going away...

*shrug*

JRodriguez · Post by **JRodriguez** » Tue Jan 17, 2017 5:37 pm

Just in case this could help...
The purge rules apply only for new jobs. If you set the purge rules after deploying your DataStage jobs then logs won't be purge for those old jobs..you would need to redeploy them

ray.wurlod · Post by **ray.wurlod** » Tue Jan 17, 2017 11:43 pm

You can delete all the records for the invocation from RT_STATUSnnn table for job number nnn.

chulett · Post by **chulett** » Wed Jan 18, 2017 7:45 am

JRodriguez wrote:The purge rules apply only for new jobs.

Isn't that only true for older versions? Could swear it was something they changed / fixed in "more recent" releases.

ray.wurlod · Post by **ray.wurlod** » Thu Jan 19, 2017 11:25 pm

I think JRodriguez has in mind newly-created jobs.

Mike · Post by **Mike** » Fri Jan 20, 2017 7:56 am

Just wanted to add a comment to the post by PaulVL...

My current project is on 11.3 FP2 as well. Auto-purge of multi-instance jobs always clears our job log entries as expected. In the vast majority of cases, it also removes the instance so that it no longer appears in the Director client. The strange thing is that there are a small percentage of jobs where the instance does not get removed from Director. It has not been enough of an annoyance at this point for me to try and track down what is different about those jobs.

Mike

DSXchange

Multi-instance job log cleanup

Multi-instance job log cleanup

Re: Multi-instance job log cleanup