Move .XML files to archive directory after processing

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
mellidav
Premium Member
Premium Member
Posts: 11
Joined: Mon Oct 13, 2014 4:50 am
Location: manchester

Move .XML files to archive directory after processing

Post by mellidav »

Hi,

I have a parallel job created that looks in a folder (using 'external source' stage) for any files like ".xml". The job then uses the XML stage to parse the XML and import the data into an SQL database.

That's all working, but after that how can I move the processed XML file out of the folder (maybe into an "archive" sub directory) using infosphere? If it doesn't move then the next time the infosphere job runs it processes the same file again.

I'd prefer that I move the file using the Datastage job as I don't have a lot of control over the server the files are being read from so I can't write scripts on it, and if I did i'm not sure how the script would know that the file has been processed.
Thanks,
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

You don't necessarily need to write a script although that would give you more control over exactly what happens and how you handle errors / issues. Biggest question in my mind is can other files arrive while the job is running or will a search for "*.xml" after the job completes always return the same list that was just processed?

For the latter you can just issue a "mv" of the files in the After Job area as an ExecSH... but mark it so only happens on job success.
-craig

"You can never have too many knives" -- Logan Nine Fingers
mellidav
Premium Member
Premium Member
Posts: 11
Joined: Mon Oct 13, 2014 4:50 am
Location: manchester

Post by mellidav »

Thanks for your reply.

Sorry but I'm going to have to ask what you mean by "mv" / after job area / ExecSH? Are these stages within the DS palette?

Thanks,
arunkumarmm
Participant
Posts: 246
Joined: Mon Jun 30, 2008 3:22 am
Location: New York
Contact:

Post by arunkumarmm »

'mv' is the move command. What craig says is, we have two options inside the job properties, before and after commands, which would execute before the job is run or after the job is run as the name suggests.

The unix command to move the .xml files to an archive folder can be given in the after command as an ExecSH (an option in the drop down).
Arun
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Exactly. You would "Execute a Shell command" after the job completes to use an operating system command to move the files elsewhere. Can you answer my "biggest question" please?
-craig

"You can never have too many knives" -- Logan Nine Fingers
mellidav
Premium Member
Premium Member
Posts: 11
Joined: Mon Oct 13, 2014 4:50 am
Location: manchester

Post by mellidav »

Thanks guys,

I didn't know about the before / after job routines so that helps me a lot.

re the "biggest question" - Could I write a before script to rename any files 'processingFILENAME.xml' then run the DS job, then run the after job routine to move any files like 'processing*.xml' ?

Thanks for your help,
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

So you can have files arrive during the processing? If so then yes, you could do something like that. What I used to do in that situation was setup a 'processing' directory / sub-directory and before job move all current files to be processed there so the list was static and it didn't matter what showed up during the run, they would wait for the next run. Then after processing move them from the processing directory to the archive directory. Both could happen via that ExecSH mechanism we discussed. Then lather, rinse, repeat.
-craig

"You can never have too many knives" -- Logan Nine Fingers
mellidav
Premium Member
Premium Member
Posts: 11
Joined: Mon Oct 13, 2014 4:50 am
Location: manchester

Post by mellidav »

Thanks Guys,

All working now,
David
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

8)
-craig

"You can never have too many knives" -- Logan Nine Fingers
Post Reply