Generate data stage jobs?

dkaranth · Post by **dkaranth** » Wed May 21, 2008 4:31 am

Hi all,
Is there any way I can generate the xml for a datastage job and then import it into my environment? The requirement is that different jobs need to be generated for different user needs at runtime.

For eg, I may not need the transformer stage on some condition provided by the user at runtime. If I had designed the job in designer, then this stage will always be there.

Thanks!
Deepak

chulett · Post by **chulett** » Wed May 21, 2008 6:21 am

Wow. While I have been involved in generating fixed design jobs based on variable metadata, I can't imagine being able to do this based on the whim of 'different users needs at runtime'. And I don't see how 'user needs' play all that much into the ETL world. Who would these users be? What kind of needs are you anticipating? You sure this isn't more the purview of a BI tool in the hands of a 'Power User'?

Well... at some point here there will be Fast Track, which generates jobs based on Excel source-to-target mappings. Or at least as much of a job as it can based on what it was given, which would then be fleshed out by an ETL developer. But trying to roll your own, generating .xml or .dsx formatted 'code' based on user 'needs'... I don't see it.

Curious what others think.

chulett · Post by **chulett** » Wed May 21, 2008 6:25 am

While waiting (and waiting) for this to post, noticed the forum I had ended up in. Since this really doesn't have anything to do with RTI...

Moderator, please do the needful and move this to the PX forum.

Tanks.

ray.wurlod · Post by **ray.wurlod** » Wed May 21, 2008 3:10 pm

There is a vague logic to the choice of forum - the OP requires dynamic design in real time.

Some parts of it can be done (for example by specifying entire expressions as parameters) but, as a general observation, I'd have to agree with Craig that it does not sit well as something one would expect to do with an ETL tool.

vmcburney · Post by **vmcburney** » Wed May 21, 2008 5:19 pm

Someone at IOD last year mentioned they had built a processing engine using Parallel jobs with custom stages or native stages where all the column metadata came from schema files, the database commands were SQL files and the modify commands passed in as job parameters. You need a lot of skill in parallel jobs to set this up and some coding outside of DataStage to generate all the required files and commands.

kduke · Post by **kduke** » Wed May 21, 2008 8:52 pm

I have a routine which generates XML definitions which can be imported into DataStage. It was fairly easy to build. Just export a table in XML format and generate the same format.