Balanced Optimizer - Transform Stage

rwierdsm · Post by **rwierdsm** » Mon Oct 28, 2013 12:45 pm

Folks,

One of the DS administrators at my current client had suggested that the GCC compiler is needed before any DS job with a transform stage could be pushed down using the balanced optimizer.

Has anyone had experience with pushing down jobs that included a transformer? Was the GCC compiler needed?

Thanks,

Rob

lstsaur · Post by **lstsaur** » Mon Oct 28, 2013 1:11 pm

Sorry, what's "pushing down jobs"?

ray.wurlod · Post by **ray.wurlod** » Mon Oct 28, 2013 1:21 pm

The compiler is needed to compile any parallel job with a Transformer in it. This has nothing at all to do with Balanced Optimization.

I don't know whether pushing any of the job's logic into a database can be done without the compiler (my installations have always had it). It would surely depend on whether you're trying to push some of the Transformer stage logic into a database.

rwierdsm · Post by **rwierdsm** » Mon Oct 28, 2013 1:25 pm

Hi lstsaur,

When I say 'Pushing Down Jobs', I mean using the balanced optimizer to push the logic down to the database. The implications are that the data doesn't leave the database, all processing occurs without data crossing the network. The intent is that our jobs will take less resources and perform significantly faster.

IBM does a wonderful job illustrating this feature in the article at the following link: http://www.ibm.com/developerworks/data/ ... connector/

I've had considerable success using similar features in BODS and Informatica. Now that I'm back in the DataStage world, I'm very interested in making this technology work in my latest projects. At this point, the client is still exploring this feature.

Regards,

Rob

rwierdsm · Post by **rwierdsm** » Wed Oct 30, 2013 1:25 pm

Hi Ray,

While we may find some jobs that we can push down without any logic that would require a transform stage, I don't want to limit myself. My expectation is that I would very quickly get to jobs that have that sort of complexity.

Of course, this will lead to a whole different exercise. We will need to be careful about what functions, data transformations, etc. we use in the transform stage. Each will need to be chosen based on whether there is a database equivalent for us to push down to. Added to that will be the complexity of learning the peculiarities of the tool, i.e. what do we need to do to convince DataStage to push down when it doesn't want to.

Rob

kwwilliams · Post by **kwwilliams** » Wed Oct 30, 2013 5:47 pm

You can't force datastage to push down when it doesn't want to do so. Your only option would be to manually manipulate the job if it didn't push as much logic into the database as you were hoping.

ray.wurlod · Post by **ray.wurlod** » Wed Oct 30, 2013 5:54 pm

Within the Balanced Optimizer dialog you can specify the range of stages to push into the source/target database. Have you investigated this feature which, I believe, is new in version 9.1?

rwierdsm · Post by **rwierdsm** » Wed Oct 30, 2013 6:53 pm

kwwilliams wrote:You can't force datastage to push down when it doesn't want to do so. Your only option would be to manually manipulate the job if it didn't push as much logic into the database as you were hoping.

Hi Keith,

That's exactly what I'm talking about. Determining the tricks needed to manually manipulate the job is going to be where all the fun is.

Rob

rwierdsm · Post by **rwierdsm** » Wed Oct 30, 2013 6:56 pm

ray.wurlod wrote:Within the Balanced Optimizer dialog you can specify the range of stages to push into the source/target database. Have you investigated this feature which, I believe, is new in version 9.1?

Hi Ray,

We're still getting access to 9.1. The admin hasn't yet gotten the GCC compiler installed, this is the main holdup at this point. The initial debate with the admin was whether the compiler was needed to do any push down at all.

At this point it's still all theoretical for me.

Rob