Move .XML files to archive directory after processing
Moderators: chulett, rschirm, roy
Move .XML files to archive directory after processing
Hi,
I have a parallel job created that looks in a folder (using 'external source' stage) for any files like ".xml". The job then uses the XML stage to parse the XML and import the data into an SQL database.
That's all working, but after that how can I move the processed XML file out of the folder (maybe into an "archive" sub directory) using infosphere? If it doesn't move then the next time the infosphere job runs it processes the same file again.
I'd prefer that I move the file using the Datastage job as I don't have a lot of control over the server the files are being read from so I can't write scripts on it, and if I did i'm not sure how the script would know that the file has been processed.
Thanks,
I have a parallel job created that looks in a folder (using 'external source' stage) for any files like ".xml". The job then uses the XML stage to parse the XML and import the data into an SQL database.
That's all working, but after that how can I move the processed XML file out of the folder (maybe into an "archive" sub directory) using infosphere? If it doesn't move then the next time the infosphere job runs it processes the same file again.
I'd prefer that I move the file using the Datastage job as I don't have a lot of control over the server the files are being read from so I can't write scripts on it, and if I did i'm not sure how the script would know that the file has been processed.
Thanks,
You don't necessarily need to write a script although that would give you more control over exactly what happens and how you handle errors / issues. Biggest question in my mind is can other files arrive while the job is running or will a search for "*.xml" after the job completes always return the same list that was just processed?
For the latter you can just issue a "mv" of the files in the After Job area as an ExecSH... but mark it so only happens on job success.
For the latter you can just issue a "mv" of the files in the After Job area as an ExecSH... but mark it so only happens on job success.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
-
- Participant
- Posts: 246
- Joined: Mon Jun 30, 2008 3:22 am
- Location: New York
- Contact:
'mv' is the move command. What craig says is, we have two options inside the job properties, before and after commands, which would execute before the job is run or after the job is run as the name suggests.
The unix command to move the .xml files to an archive folder can be given in the after command as an ExecSH (an option in the drop down).
The unix command to move the .xml files to an archive folder can be given in the after command as an ExecSH (an option in the drop down).
Arun
Thanks guys,
I didn't know about the before / after job routines so that helps me a lot.
re the "biggest question" - Could I write a before script to rename any files 'processingFILENAME.xml' then run the DS job, then run the after job routine to move any files like 'processing*.xml' ?
Thanks for your help,
I didn't know about the before / after job routines so that helps me a lot.
re the "biggest question" - Could I write a before script to rename any files 'processingFILENAME.xml' then run the DS job, then run the after job routine to move any files like 'processing*.xml' ?
Thanks for your help,
So you can have files arrive during the processing? If so then yes, you could do something like that. What I used to do in that situation was setup a 'processing' directory / sub-directory and before job move all current files to be processed there so the list was static and it didn't matter what showed up during the run, they would wait for the next run. Then after processing move them from the processing directory to the archive directory. Both could happen via that ExecSH mechanism we discussed. Then lather, rinse, repeat.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers