DSXchange

I recently rediscovered a post about MFILES and the rotating file pool, written by Dave Meeks (now at Ascential Engineering) in 2002 on a forum for UniVerse. I reproduce that post here because it's useful information. For UV read DataStage; the arguments still hold.

>We recently had a situation where a Read Operation error was encountered on
>our system. On the advice of U2 tech support we bumped MFILES from 64 to
>250. After bouncing Universe we began getting these errors all over the
>place along with others like "Unable to initialize distributed file". We
>later found out that we had exceeded our hpux NFILE parameter which was set
>at 13518. We were able to remedy the situation by setting MFILES down to
>128. My questions are these... How do I determine what MFILES should be
>set to? How do I determine what NFILE should be set to? What does Universe
>do with MFILES internally?


Ok... let's see if I can explain this properly:


NFILE - Unix parameter specifying how many actual files can be opened at
any given time across the entire system.


NOFILE - Unix parameter specifying how many actual files can be opened by
any given user process at any given time, such that the
Sum(NOFILE) < NFILE.


MFILES - UniVerse parameter specifying how many actual files can be opened
by any given universe process at any given time, such that
MFILE < NOFILE.


MFILES is part of what is known as our 'Rotating File Pool'. In the
old days, UNIX had a nasty habit of severely limiting the number of files
any given user could have open. Being a database system, this causes
many problems, as any given application could open LOT's of files.
Consider that an actual file 'open' will often open the DICT, the DATA.30,
the OVER.30, and the various INDEX files that make up the virtual hashed file.


So, the Rotating File Pool was created. What this does is map virtual
file channels to physical file channels. Whenever you 'open' a file, it
places an entry into the rotating file pool. Once the pool has reached
MFILE size, any subsequent attempt to open the file channel will result
in one of the other slots being 'closed'. This close is a physical close
at the UNIX level, freeing up a NOFILE/NFILE slot. All state information
is saved before the close (file pointer location, etc...).


Whenever an attempt is made to access a file channel for the purpose of
reading or writing within UniVerse (or other things like status checks),
we will call a routine called 'Mcheck()', which will verify that the
file is rotated back in. If not, it will close one of the open files and
reopen the requested file. This will ensure that any physical operations
we perform are against an actual open file.


Now, as to what to set it. There are a number of actual file channels
that are used that are outside of the Rotating File Pools control. These
include things like STDIN, STDOUT, STDERR, and a number of others. Standard
rule of thumb is to set MFILES no greater than (NOFILE-10).


The advantage of setting it to that maximum value is performance gains, as
you reduce the overhead of rotating files in and out (closing/opening the
physical file).


The downside is that it can be a drag on system resources. Many systems
have done away with NFILE/NOFILE limitations, allowing the system to allocate as
many as they need. But, having all of those files open/managed at one time
has a cost as well.


Balancing the performance gains vs. the OS overhead is the key.


As for NFILE, if you have a system that has N UV users, and they are
maxing out their MFILEs table, you need :


NFILE >= ( N * MFILE ) + ( Buffer room for all other process files)


Phew... hope that helps.


Dave