I wanted to pass this on to DataStage v6 users. We have just upgraded from v5 to v6 and the most beneficial feature so far has been Row Buffering. Row Buffering can be set at the project level (on the server) or at the job level (job properties).
We are choosing to do it at the job level for tuning reasons. There are two types of row buffering: In Process and Inter Process. In Process should help improve performance on most jobs. Inter Process Row Buffering takes advantage of SMP architecture and spreads the workload across the different CPU's. I am not up on all of the details of how it works but we are seeing at least 50% improvement :D in run time for many different jobs. Some have gone from 2.5hrs to 32 minutes using In Process Row Buffering. Everyone's environment is different so check it out for yourself.
I detected a bug when using Inter Process Row Buffering. It fails when a job has a SORT stage in it.
This information is based on DataStage Server running on AIX.
Welcome to the forum. We are hopeful that many more of our members leverage this forum to spread the word, good or bad. As a matter of practice, you mention a good ( Row Buffering) and a bad (Interprocess Row buffering when SORT is included). If you have reported this as a bug to Ascential, let us know. If you feel like taking the time to share work arounds or feedback, that would also be appreciated.
Otherwise, thanks for your comments. Remember the more you give, the more you get. :D
I have reported the Row Buffering / Sort bug to Ascential. My work around is to use "In Process" row buffering on the job that has a SORT Stage. In Process row buffering does work with the SORT Stage but "Inter Process" does not yet on AIX. I am not sure about NT, Solaris, etc...
The other work around is to not use row buffering at all but it could cost you longer run times.
kevin that's a lot of info for a new user like me , lately i had the opertunity working on version 6.0.1 of Datastage and i was trying to setup Row Buffering and i was able to actually setup row buffering at the project level however i did not find a tab to do that at the job level, the documentation says row buffering at the job level can be set by clicking on the job parameters
when using Inter Process Row Buffering. It fails when a job has a SORT stage in it.