DSXchange

Hi,

I have a parallel job created that looks in a folder (using 'external source' stage) for any files like ".xml". The job then uses the XML stage to parse the XML and import the data into an SQL database.

That's all working, but after that how can I move the processed XML file out of the folder (maybe into an "archive" sub directory) using infosphere? If it doesn't move then the next time the infosphere job runs it processes the same file again.

I'd prefer that I move the file using the Datastage job as I don't have a lot of control over the server the files are being read from so I can't write scripts on it, and if I did i'm not sure how the script would know that the file has been processed.
Thanks,

You don't necessarily need to write a script although that would give you more control over exactly what happens and how you handle errors / issues. Biggest question in my mind is can other files arrive while the job is running or will a search for "*.xml" after the job completes always return the same list that was just processed?

For the latter you can just issue a "mv" of the files in the After Job area as an ExecSH... but mark it so only happens on job success.

Thanks for your reply.

Sorry but I'm going to have to ask what you mean by "mv" / after job area / ExecSH? Are these stages within the DS palette?

Thanks,

'mv' is the move command. What craig says is, we have two options inside the job properties, before and after commands, which would execute before the job is run or after the job is run as the name suggests.

The unix command to move the .xml files to an archive folder can be given in the after command as an ExecSH (an option in the drop down).

Exactly. You would "Execute a Shell command" after the job completes to use an operating system command to move the files elsewhere. Can you answer my "biggest question" please?

Thanks guys,

I didn't know about the before / after job routines so that helps me a lot.

re the "biggest question" - Could I write a before script to rename any files 'processingFILENAME.xml' then run the DS job, then run the after job routine to move any files like 'processing*.xml' ?

Thanks for your help,

So you can have files arrive during the processing? If so then yes, you could do something like that. What I used to do in that situation was setup a 'processing' directory / sub-directory and before job move all current files to be processed there so the list was static and it didn't matter what showed up during the run, they would wait for the next run. Then after processing move them from the processing directory to the archive directory. Both could happen via that ExecSH mechanism we discussed. Then lather, rinse, repeat.

Thanks Guys,

All working now,
David