in one project I can see the symptoms mentioned by this technote:
http://www-01.ibm.com/support/docview.w ... wg21623578
They are using some server-jobs which just do something like "get a jobid" and so on. They have used server-jobs because this functionality does not need parallel jobs and the overhead of them. I thought that is in general a good idea.Problem(Abstract)
System is overloaded by DataStage "Phantom DSD.RUN..." processes attached to init.
Symptom
PPID of Phantom DSD.RUN is 1 (attached to init) and still consuming CPU.
But now these heavy used, multiple-instance jobs created for quite a time 100% cpu. I can see that in top. In one environment the phantom jobs lasts for 2min. In another I have seen a job running for 12min.
To be sure: The job has ended. Everything is fine in director. The job ran for some seconds. It's just a phantom process, sometimes with a PPID=1.
The technote mentioned to shorten the interval for auto-purge. The original interval was 3 days . We have shortend it to 20 runs and even 1 run. One phantom job is running 100%. If we deactivate auto-purge altogether, it works nicely. But we cannot do that for a long time.
It happens in multiple environments and on multiple projects. We cannot reproduce it on a newly created projects and with newly created jobs. Maybe it takes some weeks.
Workload does not affect the problem. One test was on a machine, that had currently no jobs running. I started on of the server jobs and 100% for 1min. A copy of the job runs nicely
They have also some parallel jobs with quite the same number of instances. But they run nicely. Manually clearing the joblog does not help.
Possible quickfixes
1) disable auto-purge for some days and on Friday set auto-purge to 3 days
2) disable auto-purge and manually purge logs (like CLEAR.FILE RT_LOGnnn)
3) rewrite the jobs to use parallel jobs or maybe shell scripts
Currently were trying 1, but maybe go for 3. But I want to find a real solution.
Do you have any idea?
Enviroment
Datastage 8.7 FP 1
DB2 9.7 for XMETA, but Loggin in XMETA is disabled
Thanks for your help!