Unusually slow DataStage 8.0 engine

[rleishman]

I'm working for a client where DS 8.0 is installed and maintained by a centralised group (not my team); they give us a project to develop and test our jobs. The jobs on this environment are just plain SLOW.

For example, a Server job that reads and writes a single row from one sequential file to another tables 5 seconds in total (i.e. Time between first and last log entry). But the statistics in the Stats log message (2nd last log message) say that it took 0.0sec - about right for one row.

So it looks like there is some inordinate amount of time taken to startup/shutdown the Server job. The same thing happens with Parallel jobs (one parallel thread) - but with them its even worse.

It is not a load issue on the server. Performance is consistent day or night and the server is under very little load. Usually it is completely idle.

I'm sure there is something totally FUBARed with the installation, but I don't know where to start, and I may not be able to anything about it since we do not administer the service.

Our next step is to call in IBM to look at it, but I thought I'd try here to see if there were any suggestions.

[ray.wurlod]

Run it on 415 volts.


I suspect it's the interface to Information Server. Checking for metadata mismatch has to be done through the metadata delivery service. Then a server job has to be started. Are you generating process metadata? If so, that's got to be passed back through the metadata delivery service.

The delay in startup may therefore be unavoidable. But it's a much smaller proportion of your total run time for jobs that process lots of rows.

As you note, server job for one row is still faster than parallel job for one row, even on a single node configuration. That's at least partly because the parallel job has to start the conductor, compose the scores, distribute them, start the section leader, and only then start the job.