Limitation on space usage in a DataStage project

saraswati · Post by **saraswati** » Wed Sep 14, 2011 7:27 am

Do we have a limitation on the space available for a particular project. I used to get xmeta exceptions in a project with a large number of jobs when I had tried to create new jobs or create a copy of a job. When I finally deleted some unused jobs, I am not getting that exception.

arvind_ds · Post by **arvind_ds** » Wed Sep 14, 2011 7:33 am

Make sure to add sufficient amount of disk space into underlying XMETA tablepsace(s).

chulett · Post by **chulett** » Wed Sep 14, 2011 7:49 am

Space? No, not that I'm aware of. Some operating systems (like AIX) have a limit on the number of directories, however.

suse_dk · Post by **suse_dk** » Wed Sep 14, 2011 9:01 am

Your job logs can end up taking a lot of space if no purging options have been set, and this can in my experience result in some the errors you mention.

ray.wurlod · Post by **ray.wurlod** » Wed Sep 14, 2011 3:30 pm

I doubt that this is anything at all to do with XMETA - it's more likely to be the number of subdirectories in the project directory.

qt_ky · Post by **qt_ky** » Tue Sep 20, 2011 8:14 am

I can confirm on AIX 6.1 with jfs2 file system, I can only create 32765 subdirectories within each directory. The error I get when creating one more is:

mkdir: 0653-358 Cannot create tmp32766.
tmp32766: There are too many links to a file.

I'm sure current directory (.) and parent (..) count too, making the limit 32767.

If each DataStage job creates 5 or more subdirectories in the project (more for QualityStage jobs) then you could easily hit a subdirectory limit with having 4000 to 6000+ jobs in a single project.

ray.wurlod · Post by **ray.wurlod** » Tue Sep 20, 2011 2:51 pm

The same limit applies on Solaris operating system.