Greetings,
We're a new site with EE. Need advise on best "options" config file for our current & future workload.
System Info:
Red Hat Linux AS 3.x
2x DELL 2850 servers ( symmetrical )
- DataStage programs & projects on NAS share
- ssh used by conductor to start shell session on other player node
Current single apt file:
{
node "node1"
{
fastname "test01"
pools "" "node1" "test01"
resource disk "/datastage/Datasets" {}
resource disk "/export/datastage/Datasets" {}
resource scratchdisk "/Scratch" {}
}
node "node2"
{
fastname "test02"
pools "" "node2" "test02"
resource disk "/export/datastage/Datasets" {}
resource disk "/datastage/Datasets" {}
resource scratchdisk "/Scratch" {}
}
}
Our Current Loading:
multiple concurrent Jobs - all small, mostly single row processing
- hardly any need for parallel engine ( more like penalty currently )
Future load (projects) will make better use of parallel engine - with huge load and fairly partitioned data to process.
Questions:
1) Use the above APT file for future EE loads (& add more physical nodes as required)
2) Would it be wise to define another APT file to define only one logical node to handle existing load? - effectively leaving player node idle.
3) Is there an optimal limit to number of logical nodes I can define for a 2-CPU server - Manual suggests 1 logical node per 2-CPU because EE launches 2 processes per logical node - is that correct? Does our small jobs mean 1 process could be idle at times?
4) Does our existing APT file above means 4 processes are launched each time a job is started? - regardless if our jobs will only use 1 or two?
5) is there a way to ensure only a single process is launched for our jobs - any special parameters?
any advise, enlightenment will be appreciated
thanks
Parallel Engine Config File setup - advise
Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.
Moderators: chulett, rschirm, roy
Return to “IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)”
Jump to
- Moderators' Choice
- ↳ Editor's BLOG Corner
- ↳ Ask the Experts! - Dads and Grads
- ↳ DSXchange Testimonials
- ↳ Cognos (IBM BI)
- FAQs
- ↳ FAQs
- ↳ FAQ Discussion
- DataStage
- ↳ General
- ↳ IBM<sup>®</sup> Infosphere DataStage Server Edition
- ↳ IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)
- ↳ Archive of DataStage Users@Oliver.com
- IBM<sup>®</sup>Infosphere Products<sup></sup>
- ↳ Business Glossary
- Suggestions
- ↳ Site/Forum
- ↳ Enhancement Wish List
- Consulting
- ↳ Talent
- ↳ Looking for Talent
- Support
- ↳ Parameter Manager
- ↳ Compile All Plus
- Usergroup Forums
- ↳ Usergroup Central Forum
- ↳ Heartland Usergroup Forum
- The Written Word
- ↳ Articles, White Papers and Tips and Tricks
- ↳ Product Documentation
- Third Party Applications
- ↳ Third Party Applications
- Product Derivatives
- ↳ Functions
- ↳ Routines
- ↳ Jobs
- ↳ Logs
- Tools
- ↳ Tools Forum
- Category
- ↳ Infosphere Master Data Management
- ↳ Data Quality Best Practices
- ↳ IBM QualityStage
- ↳ Information Analyzer (formerly ProfileStage)
- ↳ IBM<sup>®</sup> SOA Editions (Formerly RTI Services)
- ↳ IBM<sup>®</sup> DataStage TX
- ↳ BI
- ↳ Data Integration