Reduce the links to use 9MB APT_DEFAULT_TRANSPORT_BLOCK_SIZE

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
kennyapril
Participant
Posts: 248
Joined: Fri Jul 30, 2010 9:04 am

Reduce the links to use 9MB APT_DEFAULT_TRANSPORT_BLOCK_SIZE

Post by kennyapril »

I used 4 MB of block size and now due to a very big length column it is asking me to give 9MB which could cause the job to use more memory than needed since all the links would use a 9 MB transport block size. Any idea how to use that 9MB for only particular links to reduce the memory usage?


Thank you
Regards,
Kenny
asorrell
Posts: 1707
Joined: Fri Apr 04, 2003 2:00 pm
Location: Colleyville, Texas

Post by asorrell »

I don't have access to DataStage at the moment, but I believe you can set buffering on a per-stage basis on the advanced tab of the stage.

However, unless you have gazillions of stages on the canvas, this is really not worth the effort. If you have 10 stages and 8 nodes, then you'll need about 72 buffers (or less due to combinations). This is roughly 700 mb of memory, which should be negligible on practically any datastage system.

Remember - unless the large column goes away at some point, you'll need to increase the block size on ALL the stages downstream anyway, since they'll need to handle it as well.
Andy Sorrell
Certified DataStage Consultant
IBM Analytics Champion 2009 - 2020
kennyapril
Participant
Posts: 248
Joined: Fri Jul 30, 2010 9:04 am

Post by kennyapril »

Thank you Andy!

Actually the large column will be only upto two stages, so I will give the buffer size to 9mb only to that particular stages in advanced tab and remove the parameter default transport size....Does that work?
Regards,
Kenny
Post Reply