Project File system Location

PeteM2 · Post by **PeteM2** » Wed Mar 28, 2012 8:43 am

We are currently experiencing I/O bottlenecks on the unix file system that contains all our datastage projects.

Is there any benefit in creating seperate file systems for each project?

ray.wurlod · Post by **ray.wurlod** » Wed Mar 28, 2012 3:26 pm

Maybe and possibly.

It depends whether or not that I/O is related to DataStage projects or to something else on those file systems.

qt_ky · Post by **qt_ky** » Wed Mar 28, 2012 6:17 pm

It depends on your disk type also. If your server has local physical disks and all projects on one physical disk, then change your configuration. If you're on a SAN, you should not be having this issue, but if you are, then the SAN admin people should be able to fix it for you transparently.

PeteM2 · Post by **PeteM2** » Fri Mar 30, 2012 2:26 am

The datastage projects reside in one file system and nothing else is on this file system.

chulett · Post by **chulett** » Fri Mar 30, 2012 5:23 am

It can also depend on the capabilities of the disks themselves and how their filesystem is configured. Had one client where we found that DataStage had been installed on (their words) "the crappy disks" and performance suffered horribly for it, simply moving it elsewhere helped. Others have needed to "tune" the filesystem, which involved the SAs monitoring the usage and then tweaking parameters related to reads v. writes, caching, etc until we saw performance improvements.

So what O/S and filesystem are we talking about here? If applicable.

qt_ky · Post by **qt_ky** » Fri Mar 30, 2012 5:26 am

Just make sure they're not on "the crappy disks" then!

Our servers don't have any local disks (no crappy disks). All SAN. Works very well...

chulett · Post by **chulett** » Fri Mar 30, 2012 5:28 am

Exactly... forgot they specifically said "crappy internal disks".

PeteM2 · Post by **PeteM2** » Fri Mar 30, 2012 9:28 am

The disks are on SAN and their performance is fine.

The issue is that on the AIX server side we are seeing queuing whilst the disks respond to each request for data.

The issue is caused by batch wanting to access the same disks for data - when you have 40 batch jobs running at the same time all wanting data from the same disks.

qt_ky · Post by **qt_ky** » Fri Mar 30, 2012 11:09 am

I still think that the SAN / disk admin people should be able to fix it for you transparently, ... assuming that those are all read requests.

In any case, ask them how many physical disk spindles that particular file system are sharing.

Also ask for details of any memory cache that's built into the SAN. Ask to see and understand those statistics.

If your 40 jobs are all trying to write to the same file at once and queuing up in line to wait for exclusive write access, then you probably have to address the bottleneck with changing DataStage job designs or schedules.

Your original question was about benefit in create separate file systems per project. It will give you the benefit of a controlled allocation of disk space per project, so if a job or developer has a run-away job log, it will only fill the disk of that one project and limit the risk of corruption. So yes, there is a benefit.

Whether or not creating separate file systems per project will help a performance issue is going to depend upon how each file system is allocated across physical disk spindles (back to your SAN people). One project may land on disk all to yourself. A critical project may be mapped automatically to a few disks that are hammered to begin with.

Please share what you find out next.