Inter Process Stage

Post questions here relative to DataStage Server Edition for such areas as Server job design, DS Basic, Routines, Job Sequences, etc.

Moderators: chulett, rschirm, roy

Post Reply
reachmexyz
Premium Member
Premium Member
Posts: 296
Joined: Sun Nov 16, 2008 7:41 pm

Inter Process Stage

Post by reachmexyz »

I have a job LinkCollector---->Transformer.
When i compile the warning is
Link Collector does not support inprocess active to active.
In job Properties 'Inter Process' is not selected
So if i put a sequenctial file between link collector and transformer job working fine but the performance of the job is degraded.
Can i use INTER PROCESS stage between Link collector and transformer,
inted of using the flat file.
What will be the consequences of using Inter Process.
Instead what if i directly select Inter Process from Job Properties tab?
In job Properties Inter Process is not selected
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

There must be some other stages in your job - a Link Collector stage must have one or more input links. It is my understanding that the Link Collector stage creates a process for itself and that, as a result, you must have inter-process row buffering enabled if you are using this stage type.

As far as the IPC stage is concerned the same reasoning applies. It should support an active stage on its output. Try it and let us know.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
reachmexyz
Premium Member
Premium Member
Posts: 296
Joined: Sun Nov 16, 2008 7:41 pm

Post by reachmexyz »

ray.wurlod wrote:There must be some other stages in your job - a Link Collector stage must have one or more input links. It is my understanding that the Link Collector stage creates a process for itself and that, as a result, you must have inter-process row buffering enabled if you are using this stage type.

As far as the IPC stage is concerned the same reasoning applies. It should support an active stage on its output. Try it and let us know.
Output from both an Oracle state and a sequential file are collected using Collector Stage and the output from collector is fed to transformer.
When i compile this job i am getting the compilation error "Link Collector does not support inprocess active to active".
Then i between collector stage and transformer i have inserted a sequential file and now job i compiling fine. Later i have replaced sequential file with inter process stage and again compilation is success. Now i deleted the Inter process stage. Now there is no stage b/ collector and xformer. But under job propertie i have selected Inter_Process and again the job compilation is success.
I came to know, active to active direct link will not work. It only works in Inter Process mode.
But i am not sure how the performanc is affected?
eostic
Premium Member
Premium Member
Posts: 3838
Joined: Mon Oct 17, 2005 9:34 am

Post by eostic »

This leads to a whole discussion about parallelism within Server edition...search around...you'll find other threads we've had on this. In short, the interprocess row buffering gives you a single process for each stage, enabling buffered pipeline type parallelism within a Server job. Whether it helps performance or not depends on a lot of other factors, not the least of which is the load on the machine and things like the size of your rows.

Using the inter-process Stage makes it "explicit"....only where you want it (or need it based on a stage where it is required).

I've used it often...but only with care, and on a per job, per situation basis. I would never turn it on for a whole job and certainly not for a whole project. It's a longer subject, but it turns out that it can impact the behavior of the job and where the rows flow in sequence...not a problem with simple straight jobs, but could surprise you in complex multi-path scenarios.

Ernie
Ernie Ostic

blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

If you use Link Partitioner or Link Collector stage in your job, you MUST enable inter-process row buffering for the job.

See Server Job Developers Guide for proof and more information.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Post Reply