/TMPDIR - More Information Needed

Post questions here relative to DataStage Server Edition for such areas as Server job design, DS Basic, Routines, Job Sequences, etc.

Moderators: chulett, rschirm, roy

Post Reply
DKostelnik
Participant
Posts: 34
Joined: Tue Jan 30, 2007 6:13 pm
Location: Central Florida

/TMPDIR - More Information Needed

Post by DKostelnik »

Greetings! :D

The managment :twisted: where I work has made an arbitrary decision to consolidate all the various "work" volumes on our AIX system into two file systems. The rules they are about to put in place is this:

1) All files in the production work file system are to be purged after 3 days.
2) All files in the development work file system are purged after 7 days.

This means that all DS work files older than 3 days that are generated by production processes will be purged without any consideration.

What I need to know is: Do I really need to worry about this rule? I can't tell from documentation if there are cases where files sent to the tmpdir file system (from DS) are truely ALWAYS useless after the job completes. What things, if any, should I worry about with these rules from a DataStage point of view?
Doug
AAA Auto Club Group
Listen to:
Porcupine Tree
Nosound
Days Between Stations
Sreedhar
Participant
Posts: 187
Joined: Mon Oct 30, 2006 12:16 am

Post by Sreedhar »

HI

Welcome to Dsxchange...!


We can find the files which are older then certain period of time by using the following command.


1) find . -mtime -n -print

2) find . -ctime -n -print # c indicates the creation time.

where n represent number of days old.

but it give you all the files which have been modified say in less then n number of days.

Ideally the files from the temdir will be of no use once the job has been
completed.

hope this helps you.
Regards,
Shree
785-816-0728
DKostelnik
Participant
Posts: 34
Joined: Tue Jan 30, 2007 6:13 pm
Location: Central Florida

Post by DKostelnik »

I am really looking for any solid ramifications of deleting DataStage files in the filesystem defined to TMPDIR.
Doug
AAA Auto Club Group
Listen to:
Porcupine Tree
Nosound
Days Between Stations
DSguru2B
Charter Member
Charter Member
Posts: 6854
Joined: Wed Feb 09, 2005 3:44 pm
Location: Houston, TX

Post by DSguru2B »

Its not good. Say you have a stream of 50 jobs. All dependent upon each other. The 49th job creates a staging file thats needed by the 50th job. The 50th job fails for some reason on friday. If you are not able to fix the problem in the next 72 hours, the file will be gone. You need to give enough time. 3 days is not long enough
Creativity is allowing yourself to make mistakes. Art is knowing which ones to keep.
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Let's take a step back. Temp is temp. Anything that goes there is fair game for nukage after the process that created it... ends. If you are 'staging' data there then you are asking for trouble.

Here's what the documentation says on the subject:
TMPDIR. This defaults to /tmp. It is used for miscellaneous internal temporary data, including FIFO queues and Transformer temporary storage. As a minor optimization, it can be better to ensure that it is set to a file system separate to the DataStage install directory.
Once the process that creates them completes, anything of a 'miscellaneous internal temporary' nature can be deleted. That's at least my considered opinion, if you are looking for a iron-clad guarantee you'll need to pose this question to IBM.

And if your jobs take days to run and trouble-shoot, you've got other problems. :wink:
-craig

"You can never have too many knives" -- Logan Nine Fingers
DSguru2B
Charter Member
Charter Member
Posts: 6854
Joined: Wed Feb 09, 2005 3:44 pm
Location: Houston, TX

Post by DSguru2B »

We have temp directory and we stage our files there. We have a cleanup process that cleansup these files after 30 days. We can even call it TempStg. Regardless of the name, if you are staging files in that particular directory, dont get rid of them that soon. Its better to move them or archive them in an archive folder and then clearn compressed files after considerable number of days. If its a true /tmp folder as Craig explained, then it shouldnt be a problem to clean it up every few days.
Creativity is allowing yourself to make mistakes. Art is knowing which ones to keep.
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Right - this wasn't a generic 'temporary storage' question, but rather a very specific TMPDIR setting related question.

At least I hope it was... maybe we're both off the mark. Hold on, I'm sure we'll find out soon enough. Perhaps /TMPDIR (missed the slash in the first go around) is something specific to AIX that I'm not aware of. :?
-craig

"You can never have too many knives" -- Logan Nine Fingers
DKostelnik
Participant
Posts: 34
Joined: Tue Jan 30, 2007 6:13 pm
Location: Central Florida

Post by DKostelnik »

Yes, it is specific to the setting of TMPDIR. Sorry for any confusion. In my environment, the variable TMPDIR is set to /worktmp. /worktmp is used buy processes other than datastage.

I am not worried about any files that get staged there because someone chose the directory set for their temporary/work files - that is their fault.

I am only worried about the ramification of deleting files created "internally" by the DataStage product and placed in the TMPDIR. A couple of our jobs are huge and take up to a week to complete the entire run from beginning to end.

I do have a PMR open with IBM and am waiting for a reply. I wanted to get the user community input as well (sometimes IBM misses things).
Doug
AAA Auto Club Group
Listen to:
Porcupine Tree
Nosound
Days Between Stations
DSguru2B
Charter Member
Charter Member
Posts: 6854
Joined: Wed Feb 09, 2005 3:44 pm
Location: Houston, TX

Post by DSguru2B »

I would say any OS level activities would use the tmp directory like sorting etc. by default. Other processes occupy specific directories present in the Project home directory.
Creativity is allowing yourself to make mistakes. Art is knowing which ones to keep.
DKostelnik
Participant
Posts: 34
Joined: Tue Jan 30, 2007 6:13 pm
Location: Central Florida

Post by DKostelnik »

The IBM official response:
Once a job finishes, the files in the /tmp are not required. You should have no problem purging them after 3 days.
Doug
AAA Auto Club Group
Listen to:
Porcupine Tree
Nosound
Days Between Stations
DSguru2B
Charter Member
Charter Member
Posts: 6854
Joined: Wed Feb 09, 2005 3:44 pm
Location: Houston, TX

Post by DSguru2B »

So now you have to make sure your jobs finish within 3 days :?
Creativity is allowing yourself to make mistakes. Art is knowing which ones to keep.
Post Reply