Edit / Combine DSX with same target table?

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
rajalaa
Participant
Posts: 4
Joined: Tue Mar 25, 2014 1:11 am
Location: Chennai

Edit / Combine DSX with same target table?

Post by rajalaa »

Hi,

I have multiple Datastage jobs that need to be created for loading into the same target table, basically, due to parallel development requirement to save time.
So, I would have around 40 column derivations done by developer1 in one Datastage job and another developer would create second Datastage job for around 30 columns. It would require some manual integration at the end of development to combine the 2 jobs - by adding the columns and transformations into a single job.
Please let me know if I can combine both the job dsx instead of opening the Datastage job and editing it?

Thanks
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Okay. First off welcome aboard but secondly welcome to your very own post. I'm guessing in your dsx searching you found the "cutter" discussion but you need to be out here on your own rather than at the tail end of that conversation. Also moving it here, although it seems like a general type question, I suspect it will spin off into more Parallel specific areas.
-craig

"You can never have too many knives" -- Logan Nine Fingers
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

I'm also going to start the voting off with a "no". Or perhaps a highly qualified "yes" with a warning about the work it would take to combine the two jobs via dsx surgery. We'll see what others think.

And to be honest, the idea of splitting a job up between two developers each doing half of the columns sounds like a bad idea, or at least not an ideal solution to me. Seems to me, in an effort to "save time" you are creating more work for your team and would actually take longer doing it that way. But that's just my opinion.

So you've ruled out RCP for this because all of your columns have required transformations that need to be performed on them?
-craig

"You can never have too many knives" -- Logan Nine Fingers
UCDI
Premium Member
Premium Member
Posts: 383
Joined: Mon Mar 21, 2016 2:00 pm

Post by UCDI »

dsx hacking is really only useful for very simple things like copying job parameters from one job to another ( a feature that is sadly lacking), or the occasional overarching search and replace of 1 word or something.

I don't understand enough of your design, but it seems to me you could use a shared container for whatever is shared across the 'multiple jobs' and you could use a join stage (perhaps, this may need to be in another reusable container) and a transformer to put the 30 flow and 40 flow back together and crank out the final product. It may be a little clunky, but live and learn -- if it works and isnt too slow or hard to manage you can salvage this one and think up a better approach next time.
rajalaa
Participant
Posts: 4
Joined: Tue Mar 25, 2014 1:11 am
Location: Chennai

Post by rajalaa »

Thanks for the replies.
We did rule out RCP since the column derivations and lookup tables/columns for the columns are different.
Also we are using BDFS stage and want to reduce the number of read/write across multiple jobs. Hence, we want to develop a single job with all the column calculation in the same job (and not use 3 jobs at the end of development - not to use 2 jobs to develop and 1 to combine).
Thanks
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

So... do you have a resolution here?
-craig

"You can never have too many knives" -- Logan Nine Fingers
rajalaa
Participant
Posts: 4
Joined: Tue Mar 25, 2014 1:11 am
Location: Chennai

Post by rajalaa »

chulett wrote:So... do you have a resolution here?
We are still combining the jobs manually, any further suggestions would help
Thanks
ArndW
Participant
Posts: 16318
Joined: Tue Nov 16, 2004 9:08 am
Location: Germany
Contact:

Post by ArndW »

I like to Play around with editing and modifying things in exported .dsx or .XML files when necessary. It is finicky work and it doesn't take much to corrupt Jobs completely and irrevocably by changing the files.

The internal structure of an Export file doesn't lend itself to merging, the effort in writing some code to do this type of work is going to be far more than getting someone to manually cut-and-paste in the DataStage designer.
UCDI
Premium Member
Premium Member
Posts: 383
Joined: Mon Mar 21, 2016 2:00 pm

Post by UCDI »

if you do anything manually more than 2 or 3 times, its time to automate it. If none of the datastage ideas works for you, write something that will do the surgery for you.

And its not always that much code. The one I did to transfer job params from one job to another took like 2 hours or less from start to finish. Time you have manually hacked on 3 or 4 jobs you could have done that.
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

You can copy/paste between Transformer stages. They don't even have to be in the same project!
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Post Reply