/TMPDIR - More Information Needed
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 34
- Joined: Tue Jan 30, 2007 6:13 pm
- Location: Central Florida
/TMPDIR - More Information Needed
Greetings! :D
The managment where I work has made an arbitrary decision to consolidate all the various "work" volumes on our AIX system into two file systems. The rules they are about to put in place is this:
1) All files in the production work file system are to be purged after 3 days.
2) All files in the development work file system are purged after 7 days.
This means that all DS work files older than 3 days that are generated by production processes will be purged without any consideration.
What I need to know is: Do I really need to worry about this rule? I can't tell from documentation if there are cases where files sent to the tmpdir file system (from DS) are truely ALWAYS useless after the job completes. What things, if any, should I worry about with these rules from a DataStage point of view?
The managment where I work has made an arbitrary decision to consolidate all the various "work" volumes on our AIX system into two file systems. The rules they are about to put in place is this:
1) All files in the production work file system are to be purged after 3 days.
2) All files in the development work file system are purged after 7 days.
This means that all DS work files older than 3 days that are generated by production processes will be purged without any consideration.
What I need to know is: Do I really need to worry about this rule? I can't tell from documentation if there are cases where files sent to the tmpdir file system (from DS) are truely ALWAYS useless after the job completes. What things, if any, should I worry about with these rules from a DataStage point of view?
Doug
AAA Auto Club Group
Listen to:
Porcupine Tree
Nosound
Days Between Stations
AAA Auto Club Group
Listen to:
Porcupine Tree
Nosound
Days Between Stations
HI
Welcome to Dsxchange...!
We can find the files which are older then certain period of time by using the following command.
1) find . -mtime -n -print
2) find . -ctime -n -print # c indicates the creation time.
where n represent number of days old.
but it give you all the files which have been modified say in less then n number of days.
Ideally the files from the temdir will be of no use once the job has been
completed.
hope this helps you.
Welcome to Dsxchange...!
We can find the files which are older then certain period of time by using the following command.
1) find . -mtime -n -print
2) find . -ctime -n -print # c indicates the creation time.
where n represent number of days old.
but it give you all the files which have been modified say in less then n number of days.
Ideally the files from the temdir will be of no use once the job has been
completed.
hope this helps you.
Regards,
Shree
785-816-0728
Shree
785-816-0728
-
- Participant
- Posts: 34
- Joined: Tue Jan 30, 2007 6:13 pm
- Location: Central Florida
Its not good. Say you have a stream of 50 jobs. All dependent upon each other. The 49th job creates a staging file thats needed by the 50th job. The 50th job fails for some reason on friday. If you are not able to fix the problem in the next 72 hours, the file will be gone. You need to give enough time. 3 days is not long enough
Creativity is allowing yourself to make mistakes. Art is knowing which ones to keep.
Let's take a step back. Temp is temp. Anything that goes there is fair game for nukage after the process that created it... ends. If you are 'staging' data there then you are asking for trouble.
Here's what the documentation says on the subject:
And if your jobs take days to run and trouble-shoot, you've got other problems.
Here's what the documentation says on the subject:
Once the process that creates them completes, anything of a 'miscellaneous internal temporary' nature can be deleted. That's at least my considered opinion, if you are looking for a iron-clad guarantee you'll need to pose this question to IBM.TMPDIR. This defaults to /tmp. It is used for miscellaneous internal temporary data, including FIFO queues and Transformer temporary storage. As a minor optimization, it can be better to ensure that it is set to a file system separate to the DataStage install directory.
And if your jobs take days to run and trouble-shoot, you've got other problems.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
We have temp directory and we stage our files there. We have a cleanup process that cleansup these files after 30 days. We can even call it TempStg. Regardless of the name, if you are staging files in that particular directory, dont get rid of them that soon. Its better to move them or archive them in an archive folder and then clearn compressed files after considerable number of days. If its a true /tmp folder as Craig explained, then it shouldnt be a problem to clean it up every few days.
Creativity is allowing yourself to make mistakes. Art is knowing which ones to keep.
Right - this wasn't a generic 'temporary storage' question, but rather a very specific TMPDIR setting related question.
At least I hope it was... maybe we're both off the mark. Hold on, I'm sure we'll find out soon enough. Perhaps /TMPDIR (missed the slash in the first go around) is something specific to AIX that I'm not aware of.
At least I hope it was... maybe we're both off the mark. Hold on, I'm sure we'll find out soon enough. Perhaps /TMPDIR (missed the slash in the first go around) is something specific to AIX that I'm not aware of.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
-
- Participant
- Posts: 34
- Joined: Tue Jan 30, 2007 6:13 pm
- Location: Central Florida
Yes, it is specific to the setting of TMPDIR. Sorry for any confusion. In my environment, the variable TMPDIR is set to /worktmp. /worktmp is used buy processes other than datastage.
I am not worried about any files that get staged there because someone chose the directory set for their temporary/work files - that is their fault.
I am only worried about the ramification of deleting files created "internally" by the DataStage product and placed in the TMPDIR. A couple of our jobs are huge and take up to a week to complete the entire run from beginning to end.
I do have a PMR open with IBM and am waiting for a reply. I wanted to get the user community input as well (sometimes IBM misses things).
I am not worried about any files that get staged there because someone chose the directory set for their temporary/work files - that is their fault.
I am only worried about the ramification of deleting files created "internally" by the DataStage product and placed in the TMPDIR. A couple of our jobs are huge and take up to a week to complete the entire run from beginning to end.
I do have a PMR open with IBM and am waiting for a reply. I wanted to get the user community input as well (sometimes IBM misses things).
Doug
AAA Auto Club Group
Listen to:
Porcupine Tree
Nosound
Days Between Stations
AAA Auto Club Group
Listen to:
Porcupine Tree
Nosound
Days Between Stations
-
- Participant
- Posts: 34
- Joined: Tue Jan 30, 2007 6:13 pm
- Location: Central Florida