Hello,
I'm using the Folder Stage to merge several files together. Between each file I'm getting a blank line. I'm working on a Unix server. Any suggestions of what to look for?
Mike
Folder Stage + extra lines
Moderators: chulett, rschirm, roy
-
ray.wurlod
- Participant
- Posts: 54595
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
The end of file character (Ctrl-D in UNIX, Ctrl-Z in Windows).
It's probably impossible to check for when using the Folder stage. Alter your design so that empty lines are filtered out (for example, filter through grep -v '' in UNIX)
It's probably impossible to check for when using the Folder stage. Alter your design so that empty lines are filtered out (for example, filter through grep -v '' in UNIX)
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
ray.wurlod wrote:The end of file character (Ctrl-D in UNIX, Ctrl-Z in Windows).
It's probably impossible to check for when using the Folder stage. Alter your design so that empty lines are filtered out (for example, filter through grep -v '' in UNIX)
Hi Ray,
Thanks for the feedback. I'm trying to keep all my file manipulations inside of datastage. But maybe I can clean the resulting file using the After-Job routine. I'll try it out.
Mike
Mike Mette
-
ray.wurlod
- Participant
- Posts: 54595
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
You CAN do this within DataStage! 
The Sequential File stage (version 6 and later) supports the idea of streaming the data through a filter command.
They had zip/unzip in mind when they first created it, but there's no restriction. The stdout of the filter command becomes the stream that is read by the Sequential File stage.
Check it out!
The Sequential File stage (version 6 and later) supports the idea of streaming the data through a filter command.
They had zip/unzip in mind when they first created it, but there's no restriction. The stdout of the filter command becomes the stream that is read by the Sequential File stage.
Check it out!
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
ray.wurlod wrote:You CAN do this within DataStage!
The Sequential File stage (version 6 and later) supports the idea of streaming the data through a filter command.
They had zip/unzip in mind when they first created it, but there's no restriction. The stdout of the filter command becomes the stream that is read by the Sequential File stage.
Check it out!
You know I looked at that originally, but didn't spend enough time with it. I ended up creating another job that strips all blank lines plus others out of the file. It's not pretty but it will be easy to support for others on my team. I will go back and play with that feature, it could have a lot more uses.
Thanks for your help Ray!
Mike
Mike Mette