File Split logic

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
theone
Participant
Posts: 36
Joined: Tue Oct 06, 2009 10:04 am
Location: Michigan

File Split logic

Post by theone »

Can anyone help me in getting this logic implemented using datastage please?

Below is the what the main file looks like

A200198565634
B769348348547
B837563487567
K656895565906
A387562985749
B893745647875
B394857348957
K734564735644
A893745634785
B938457348953
K783456347856
A890345765875
B378945634789
B934785643534
K378945634764

Desired Output:

File1:
A200198565634
B769348348547
B837563487567
K656895565906

File2:
A387562985749
B893745647875
B394857348957
K734564735644

File3:
A893745634785
B938457348953
K783456347856

File4:
A890345765875
B378945634789
B934785643534
K378945634764
hexaware_tmk
Premium Member
Premium Member
Posts: 17
Joined: Wed Mar 19, 2014 3:53 pm

Re: File Split logic

Post by hexaware_tmk »

You can use @INROWNUM system variable in transformer .take 4 output links from transfrmer for each sequential file . In the constarint of each link use @inrownum to distribute the rows in each output link.
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

So is it that you want a new file to start each time a line beginning with "A" occurs in the source file? This might be easier and faster to accomplish using a stream editor such as sed or awk.

If you must use DataStage, set another column to flag whether the "A" appears or not, in the form of an incrementing counter (probably maintained in a stage variable in the Transformer stage). Put this value onto the output link of the Transformer stage then use the distribution capabilities of the Sequential File stage (introduced in version 9.1) to create your files.

This is done according to a standard that you may not like, so you may need an after-job subroutine or similar to rename the files that have been created.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

theone - always best to explain your requirements in words, posting only input and output examples means we have to puzzle out what it is you need and leads to confusion and incorrect solutions. Or far too much back and forth. Start us off on the right foot. Oftentimes the simply act of spelling it out in words leads to a solution.

hexaware_tmk - there's no need to quote everything from a previous post just to reply to it. Do it when necessary but otherwise there's a perfectly lovely Reply to topic available here as well. Less flotsam for me to clean up. :wink:
-craig

"You can never have too many knives" -- Logan Nine Fingers
battaliou
Participant
Posts: 155
Joined: Mon Feb 24, 2003 7:28 am
Location: London
Contact:

Post by battaliou »

This is a classic server job using a type 1 hashed file. It will generate any amount of files you like.
3NF: Every non-key attribute must provide a fact about the key, the whole key, and nothing but the key. So help me Codd.
theone
Participant
Posts: 36
Joined: Tue Oct 06, 2009 10:04 am
Location: Michigan

Post by theone »

ray.wurlod wrote:So is it that you want a new file to start each time a line beginning with "A" occurs in the source file? This might be easier and faster to accomplish using a stream editor such as sed or ...
Thanks, I know awk or nawk works. In my question I asked logic in datastage not in unix.
theone
Participant
Posts: 36
Joined: Tue Oct 06, 2009 10:04 am
Location: Michigan

Post by theone »

battaliou wrote:This is a classic server job using a type 1 hashed file. It will generate any amount of files you like.
Thank you,

Can you please throw little more light on it?
qt_ky
Premium Member
Premium Member
Posts: 2895
Joined: Wed Aug 03, 2011 6:16 am
Location: USA

Post by qt_ky »

theone wrote:In my question I asked logic in datastage not in unix.
Go back and read the replies completely.
Choose a job you love, and you will never have to work a day in your life. - Confucius
theone
Participant
Posts: 36
Joined: Tue Oct 06, 2009 10:04 am
Location: Michigan

Post by theone »

Sorry, I was not able to read complete message as I did not pay. I am going back to Unix command CSPLIT or NAWK

Thank you again Ray
Post Reply