Parallelism- DataStage Processing Nodes

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
devidotcom
Participant
Posts: 247
Joined: Thu Apr 27, 2006 6:38 am
Location: Hyderabad

Parallelism- DataStage Processing Nodes

Post by devidotcom »

Hi All,

I have a doubt here on the logical processing nodes defined in the configuration file need to get it clarified.

Lets consider we have a single CPU and memory and the config file has 2 processing nodes. So if my job runs on 2 node config file there would be 2 process that would run in parallel. How would the speed increase in that case as in a sequential mode. Is that the time would be the same as in a sequential mode.

I have heard of config file with 64 processing nodes would that mean i would have 64/2 = 32 CPUs... Really!!!


How do we decide on the number of processing nodes each CPU can run on...

These some of the basic questions i have.
pLease let me know if anyone has some understanding on this... :(
stefanfrost1
Premium Member
Premium Member
Posts: 99
Joined: Mon Sep 03, 2007 7:49 am
Location: Stockholm, Sweden

Post by stefanfrost1 »

i think the general recommendation is 1 node per core, so if you have 4 dual core cpu you should run with 8 nodes config. However it all depends on your job design and hardware support...
-------------------------------------
http://it.toolbox.com/blogs/bi-aj
my blog on delivering business intelligence using agile principles
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

You monitor your jobs, and determine what percentage of resources is required for each. Note that you assumption "one process per job" is way off - it's more like one process per stage plus one process per node plus one. Based on the percentages you can figure out how many nodes is appropriate. It will probably be different for different jobs.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
devidotcom
Participant
Posts: 247
Joined: Thu Apr 27, 2006 6:38 am
Location: Hyderabad

Post by devidotcom »

Hi Ray,

I am not a premium member so unable to read the complete reply of yours

Thanks
ArndW
Participant
Posts: 16318
Joined: Tue Nov 16, 2004 9:08 am
Location: Germany
Contact:

Post by ArndW »

devidotcom - so it seems that there are two basic choices. You going for the premium membership or Ray posting the answer as non-premium. if I were a betting man I'd know on which one I'd be placing my bets.

The Premium costs is not excessive and I have yet to see a single post in this forum where a premium member complained about the value he/she was getting for the fee.
Post Reply