Server jobs performance on unix vs windows

azens · Post by **azens** » Wed Sep 21, 2011 4:13 pm

We are migrating DataStage 7.5 server on Windows to 8.5 Enterprise on Solaris. This is a phased project. At phase one, we only do migration from 7.5 server jobs on windows to 8.5 server jobs on solaris, without any change of job design. At phase two, we will do performance tuning and rewrite server jobs into parallel jobs.

Now it's the end of phase one and we are suffering from worse performance of server jobs on solaris compared to windows. The unix server is Sun M4000, 64GB RAM, 1 * CPU Board (2 * 4 core, 2.66GHz SPARC64 VII+ CPUs). Windows server has 8 cores AMD opteron 2.4GHz CPUs on HP Proliant 880, 16GB RAM, a 5-year model. Our unix admin comments that because the jobs we run are mostly single-thread design, hence they are not taking advantage of parallel power from unix server. Furthermore, x64 CPUs perform better than Sun CPUs on single-thread jobs, even it's a 5-year model, regardless of CPU speed. It sounds like CISK vs RISC issue,

According to unix admin's comments, we are making wrong decision of migration, or says, IBM & consultants gave us wrong recommendation of migration before we started this project. We shouldn't do phase one becaues there is no benefit, however, due to limitation of resource, we do phase one as IBM & consultants said new Solaris server must be faster than windows server in terms of running server jobs.

Can we say, DataStage server jobs run faster on windows than solaris, based on similar spec of hardware, because of single-thread job natural?

ray.wurlod · Post by **ray.wurlod** » Wed Sep 21, 2011 6:26 pm

No, because there are just too many other variables. Disk topology (mix of local and remote) and disk device speed, to name just one category.

chulett · Post by **chulett** » Wed Sep 21, 2011 7:21 pm

Have you tried to narrow down exactly why your performance is poor on the Solaris box? What aspect(s) of the jobs you're running have been affected - read speed? Write speed? It is specific to your hashed files, for instance or has access times to remote relational databases taken a hit? You really need to quantify the nature of the 'performance issue' as (as Ray notes) there are ton of factors at play here.

There is also the significant architectural change with the 8.5 release, so realize you're not just simply comparing 'Windows v. UNIX' here.

qt_ky · Post by **qt_ky** » Wed Sep 21, 2011 8:20 pm

Have you compared any settings across servers, like the uvconfig file or the DataStage Administrator settings per project? If someone tweaked settings on Windows and you have default settings on Unix, seems like that is another factor that could slow down server jobs, not because of the OS difference.

azens · Post by **azens** » Wed Sep 21, 2011 8:22 pm

I understand it's not simply windows vs unix scenario. We have done heaps of testing and monitoring. All stats show that disk /IO or performance is great, network performace is great, and other indicators say unix server is in very good condition. However, it just runs slower than windows in terms of server jobs.

Oracle(Sun) also help us to diagnose but there is no encouraging result. Only thing we assure is CPU utilisation is 100% on 1 single CPU while other 7 ones are nearly idle when we are running testing jobs.

The same jobs run on windows show the same CPU utlisation but runing faster. And it's very hard for us to convince management we are making the right decision.

SURA · Post by **SURA** » Wed Sep 21, 2011 8:38 pm

If i am right this is relates to configuration in DS. I heard, while do the installation you can configure (Even after i guess) in this regards.

You can raise it as a PMR. You will get solution.

DS User

azens · Post by **azens** » Wed Sep 21, 2011 9:26 pm

Thanks for all replies. This issue has bothered us for couple weeks. We did any necessary tweak and also raised PMR to IBM, but so far, no much improvement. We did get some performance gain from switching ZFS to UFS for hashed files location. However, the improvement is not good enough to fill the gap.

Now we don't expect server jobs can run faster on solaris than windows but at least, shuoldn't be worse. It's embarrassing that we spend big money on new hardware and software but it results in worse performance.

chulett · Post by **chulett** » Wed Sep 21, 2011 10:31 pm

azens wrote:We did get some performance gain from switching ZFS to UFS for hashed files location.

Ah... I was wondering if this was you but I hadn't checked the older posts yet.

azens · Post by **azens** » Wed Sep 21, 2011 11:56 pm

chulett wrote:
azens wrote:We did get some performance gain from switching ZFS to UFS for hashed files location.
Ah... I was wondering if this was you but I hadn't checked the older posts yet.

Yes, that's me. In the meanwhile, we start to convert some jobs in critical path into parallel and also get IBM to our site to investigate.

ray.wurlod · Post by **ray.wurlod** » Thu Sep 22, 2011 1:04 am

There is no guarantee whatsoever that parallel jobs are faster than server jobs, particularly for small to medium data volumes.

chulett · Post by **chulett** » Thu Sep 22, 2011 6:58 am

Exactly, make sure these 'critical path' Server jobs are actually appropriate for assimilation into the PX collective.

qt_ky · Post by **qt_ky** » Thu Sep 22, 2011 12:50 pm

So you are seeing a CPU bottleneck?

Is that normal for all server jobs to run on a single CPU or core? If you start a second server job while the first server job is running on one CPU at 100%, then would you expect the second server job to land on a different CPU or core?

ray.wurlod · Post by **ray.wurlod** » Thu Sep 22, 2011 3:19 pm

If the operating system behaves conventionally (and since DataStage jobs are processes) that's what I'd expect.

jwiles · Post by **jwiles** » Mon Sep 26, 2011 2:29 pm

Have you analyzed what part of the overall job process is slower in the new environment?

How do you run your jobs? Are there any scripts, before job/after job routines, etc., which may call ds server commands such as dsadmin?

Regards,