I seem to be going around in circles with this.
We are migrating to a 9.1.2 environment from 7.5.1 and I am looking for guidance on setting up logical nodes for the configuration file.
We are running DataStage v9.1.2 on AIX 6.1 on a 6-way processor system with (say) 128gig that is connected to fiberSAN disk storage.
Because it is SAN storage, there is no "number of spindles" or "disks" to be concerned with (we get a logical filesystem that spans many drives). The documentation talks about not needing to tie the number of logical nodes of the config file to the number of processors. I don't see anything that gives guidance on how to determine the configuration.
So, how does one determine how many logical nodes should (optimally) be defined in the configuration file?
Second to that, is there a method to monitor and determine if the configuration is helping or hurting?
Determining Config File Nodes on SMP system
Moderators: chulett, rschirm, roy
-
DKostelnik
- Participant
- Posts: 34
- Joined: Tue Jan 30, 2007 6:13 pm
- Location: Central Florida
Determining Config File Nodes on SMP system
Doug
AAA Auto Club Group
Listen to:
Porcupine Tree
Nosound
Days Between Stations
AAA Auto Club Group
Listen to:
Porcupine Tree
Nosound
Days Between Stations
-
DKostelnik
- Participant
- Posts: 34
- Joined: Tue Jan 30, 2007 6:13 pm
- Location: Central Florida
The rule of thumb is a only a starting point.
The configuration file that a job uses is there to optimally allocate the available resources for that job in order for it to either run as quickly as possible, or to run using only a subset of the resources.
Ensuring optimum resources is important in jobs which have lots of stages (i.e. runtime processes) but only has a couple of rows to process. If it is run with one logical node, for example, it might start up 22 processes. With a 2-node configuration it will probably use 43 processes, and scale accordingly. But if it runs in 1 second in a 1-node configuration it will almost certainly run in much more time in a 8-node configuration, since most of the system's resources will be used to spawn processes and establish communication between them.
Similarly, if there are parallel jobs that are run at the same time, you might want to choose their configuration files so that one or more of them don't hog all the resources or that too many processes are running at the same time (potentially causing swapping and paging and thrashing).
This is why it is usually recommended to have several configuration files and parameterize each job's config file; just in case it becomes necessary to fine-tune.
What is important is that, in development and test, a 2-node or higher configuration file is always used. This is because a job that runs correctly with 2-nodes will run the same way with a 1-node or higher configuration, but jobs that run correctly in a 1-node configuration may not necessarily run correctly with multiple nodes.
In your case I might start with a 4-node configuration, just as the rule-of-thumb suggests. During processing, monitor CPU usage and if the CPU load isn't high then increase the number of nodes to 6. Re-run and see if CPU load goes up and/or if the overall runtime of the jobs decreases. If the runtime doesn't decrease, then your speed is limited by some other factor and I'd go back to 4 nodes, or even try a 2-node configuration to see if runtime remains the same.
The configuration file that a job uses is there to optimally allocate the available resources for that job in order for it to either run as quickly as possible, or to run using only a subset of the resources.
Ensuring optimum resources is important in jobs which have lots of stages (i.e. runtime processes) but only has a couple of rows to process. If it is run with one logical node, for example, it might start up 22 processes. With a 2-node configuration it will probably use 43 processes, and scale accordingly. But if it runs in 1 second in a 1-node configuration it will almost certainly run in much more time in a 8-node configuration, since most of the system's resources will be used to spawn processes and establish communication between them.
Similarly, if there are parallel jobs that are run at the same time, you might want to choose their configuration files so that one or more of them don't hog all the resources or that too many processes are running at the same time (potentially causing swapping and paging and thrashing).
This is why it is usually recommended to have several configuration files and parameterize each job's config file; just in case it becomes necessary to fine-tune.
What is important is that, in development and test, a 2-node or higher configuration file is always used. This is because a job that runs correctly with 2-nodes will run the same way with a 1-node or higher configuration, but jobs that run correctly in a 1-node configuration may not necessarily run correctly with multiple nodes.
In your case I might start with a 4-node configuration, just as the rule-of-thumb suggests. During processing, monitor CPU usage and if the CPU load isn't high then increase the number of nodes to 6. Re-run and see if CPU load goes up and/or if the overall runtime of the jobs decreases. If the runtime doesn't decrease, then your speed is limited by some other factor and I'd go back to 4 nodes, or even try a 2-node configuration to see if runtime remains the same.
</a>With today's multi-CPU, hyper-threaded processors, there's no accurate formula I'm aware of to tell you how to set the number of nodes. I usually setup a range of config files from single-node up to 64 nodes. Then I make several test runs with representative job loads to find the "sweet spot" where I'm effectively utilizing resources and maximizing performance.
Finding the best node count for one job that is running in isolation on a system will be very misleading unless that's the way the job normally runs.
On systems where I know it is "safe" (ie: there are no problems with dataset partition differences, etc.) I'll gradually bump the default apt file configuration up or down on daily runs to see what it does to throughput.
Finding the best node count for one job that is running in isolation on a system will be very misleading unless that's the way the job normally runs.
On systems where I know it is "safe" (ie: there are no problems with dataset partition differences, etc.) I'll gradually bump the default apt file configuration up or down on daily runs to see what it does to throughput.
Re: Determining Config File Nodes on SMP system
Method: focus on single a job to tune, or a single set of jobs that represent your typical workload, use a significantly sized set of data, and run repeatedly. Change only one thing at a time and record the measurements.DKostelnik wrote:Second to that, is there a method to monitor and determine if the configuration is helping or hurting?
AIX has the vmstat and lparstat commands, and many, many others. The nmon utility is good too. If you are on an LPAR, then the lparstat -i command can reveal some interesting info. UNIX admins can be a great resource in this arena.
You may also monitor activity and review history using the web-based DataStage and QualityStage Operations Console. If you haven't already set it up in 9.1.2, the configuration steps are detailed on one of the tabs once you login.
Version 9.1 also includes a new policy-driven Workload Management (WLM) feature that lets you setup various queues, priorities, and throttle settings. An IBM Overview and Best Practices PDF is available.
Choose a job you love, and you will never have to work a day in your life. - Confucius
Knowledge of how many other jobs run at the same time is very important. You don't want to turn everything up to 11 if they all run at the same time.
More parallelism is not always equal to a faster job. Throw the 2 cpu per node recommendation out the door. That's nuts.
Fine tuning nodes is an art form that you might not want to waste to much time on. Look at your job performance when running in TEST/UAT. Crank it up by 2/4/8 etc. Your jobs are often IO bound anyhow so you also have to balance what effect your parallelism has on the outside world as well. To many connections out to your target/source database and you may peeve the DBAs off a tad.
Experiment, but do ensure that all jobs get created and developed with a minimum of 2 degrees of parallelism. You want to get your data partitioning logic correct so that you can grow it later.
More parallelism is not always equal to a faster job. Throw the 2 cpu per node recommendation out the door. That's nuts.
Fine tuning nodes is an art form that you might not want to waste to much time on. Look at your job performance when running in TEST/UAT. Crank it up by 2/4/8 etc. Your jobs are often IO bound anyhow so you also have to balance what effect your parallelism has on the outside world as well. To many connections out to your target/source database and you may peeve the DBAs off a tad.
Experiment, but do ensure that all jobs get created and developed with a minimum of 2 degrees of parallelism. You want to get your data partitioning logic correct so that you can grow it later.
