I have sunfire server with 2CPUs and 2GB rams. There is no other application or software running but DS version 6.
I got 2 questions to ask:-
1. We divide job control into 4 streams running parallel, CPU seems to be utilized ok, almost 100% used but memory is not. It seems that CPU is used only 20% used and swap is used quite a lot. How can I configure to use more physical memory, this is set by OS or DS ?
2. How DS utilized memory for hashed file, because from DS Adminstrator I set cache for read and write hashed file stage to 128MB. How is this cache allocated to jobs that have hashed files' cache enabled ?
DS will only use the memory it requires, it will not over-allocate. Hash file caching has upper limit controls, but only allocates what it needs and uses only that which it needs. You have no maintenance to do other than set upper limits, which you probably don't even need to touch.
Your performance is most likely going to be limited by the amounts of parallel job execution you are going to do. For instance, a job that is SEQ -> XFM -> SEQ will probably completely consume a CPU. So, your 2 cpu machine will be able to run less than 2 of these jobs at full speed because of complete CPU utilization (you will have system resources and overhead that consumes some CPU).
Kenneth Bland
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Thank you for your answer but I have additional inquiry.
If you all don't mind, I'd like to response to all at the same time.
Kenneth:
As you said, I am hitting the CPU bound, do you have any suggestion?
Kim:
If you mean Pre-Load file to memory, yes I set the drop-down list to Enable
Amos:
we have set some of UVCONFIG parameters such as
MFILES=50
T30FILES=1000
TXMODE=0
LOGBLSZ=512
LOGBLNUM=8
LOGSYCNT=0
LOGSYINT=0
TXMEM=32
OPTMEM=64
SELBUF=4
DISKCACHE=-1
DCBLOCKSIZE=16
DCMODULUS=256
What else I should set or increase?
Ray:
How can I tune initial memory?? this is the same as Amos mention, UVCONFIG ?
As you said, I am hitting the CPU bound, do you have any suggestion?
Rule of thumb: More slower cpus are better then fewer faster ones.
Ex: 8 X 400mhz cpus ~ 3200 mhz of power
4 X 800mhz cpus ~ 3200 mhz of power
1 X 3200mhz cpu ~ 3200 mhz of power
It looks the same, right? I'm sure mathematically they are equivalent, but practicality and experience suggests otherwise. I'm not trying to start a debate, but my opinion is I'd rather have more cpus than fewer, but the faster the better.
Last edited by kcbland on Wed Oct 15, 2003 11:11 am, edited 1 time in total.
Kenneth Bland
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
That is not true. What bigpoppa was talking about yesterday is there is a certain effectiveness the OS gives to each additional CPU. So if you have 90% effectiveness then the first CPU is
If you re-read what I originally posted, I stated that the math looks the same, but in practical experience they are not equivalent. So if someone throws a theoretical equation at my statement, then it simply is an affirmation of emperical evidence that 1 X 3200mhz is not the same as 8 x 400mhz.
So here's my challenge: Kim, would you prefer a 1 x 3200mhz cpu or a 8 x 400mhz cpus server configuration?
All things considered, Ken Bland would rather have an 8 cpu machine of 400 mhz processors over a 1 machine with 1 cpu of 3200 mhz.
Kenneth Bland
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
I would take the 1 CPU 3200. Normally there are other factors like cache especially in the PC world. Servers in the PC world usually have more cache, faster disk drives and more ram. These are real servers. On Sun UNIX based servers then absolutely I would take the one CPU over 8 CPUs and beat your performance every time. I have done a lot of benchmarks. Usually DataStage is disk or IO bound. This means faster drives and lots of cache can out perform faster CPUs. Network traffic also has a big impact. Higher speed networks could greatly impact DataStage performance. I would take a benchmark challenge any day.
With PX some of that may change because you can run more processes and maybe get more throughput. Without PX then one CPU wins everytime.
Yep, in a single threaded benchmark test of raw cpu performance I would take a high speed cpu over a low speed cpu any day.
An Oracle DBA will talk circles around me regarding Veritas and file handles and header management and gizmos and argue they would rather have more cpus than fewer. My head will spin because they will talk about memory binding to cpus and cross-fire gobbledy-gook. Someone showed me a comparison between HP and Sun's innermost technology dealing with CPUs, memory, and Oracle that not only proved that the earth was round and the sun is hot, but that Oracle runs best on a higher cpu count on Sun machines than a faster but fewer cpu HP machine. IBM has a document that says DB2 runs slow on every machine and configuration. :D
My understanding is that highly parallel processing benefits from more and slower cpus than fewer and faster ones.
Kenneth Bland
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
I agree with that last sentence but using Oracle as your benchmark is not fair nor HP to Sun. Most Sun boxes out perform HPs but compare their downtime. The simpicity of the OS is much greater in a single CPU. They do not have to coordinate memory management or shared memory. Sun does not optimize for a single CPU because they assume they will have multiple CPUs. Suns build IO and memory around each CPU if you know the design of an E10000. I would bet they perform better because of the IO not being spread out more evenly not because there are more CPUs.
All manaufactors publish the effectiveness of adding a new CPU. All agree with the formula I stated above but you are correct in saying actual performance changes because of real world applications.
As long as everyone understands all the factors involved in performance tuning then I think I made my point. It is a very complex problem and ask 3 experts and you will get 3 different answers based on each ones experience. Most do not really understand why something runs faster or slower just what they experienced. I really enjoy these types of discussions. I think most of the forum will benefit from understanding some of this.
Even Intel is realizing that a single fast cpu is not necessarily a winner over 2 slower cpus as witnessed by their hyperthreading technology. I'm an old fart who loves Unix. Unix hardware wins over whippersnapper technology like Wintel because of their "lots of cpu" mentality.
Given the original posters statement of a 2 cpu Unix box, I cringed. They would have been better off with a 2 cpu Wintel box. Now if he stated 16 cpu Unix box, I'd run that up against a 4 cpu Wintel box anyday. On a job by job basis you might win on Wintel. But when I slam all of my jobs at once, that Wintel chassis is going to strain. In my opinion.
Kenneth Bland
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Intel and UNIX, my opinion is they have different architecture, but I am still wondering about one 3200Mhz or four of 800Mhz. There is no black and white because one fast CPU may have overhead of OS to manipulate but four medium CPUs may have overhead of coordinating these CPUs.
However, i have a feelin that no one answer my questions
May I repeat my last question again, any one has any suggestion or any rule of thumb to tune UVCONFIG ???
