Building Generic DS Jobs

[jg4040]

We are attempting to build generic jobs in DS such that they can be called with a set of parameters (metadata) rather than use Templates and create separate jobs for each set of metadata.

Has anyone done this with any success with this on a large scale?

[ray.wurlod]

Welcome aboard.

Yes it is possible, but two pieces of advice.

1. Export these prototype jobs before anyone gets to overwrite (forgets to "Save As") one of them. That way you can reinstate the prototype. Also figure out sanctions against transgressors, such as overwriting their work with the imported original.

2. You will require diligent and vigilant management overview of the (open prototype, choose File > Save As) process. Incorporate the same in your instructions to developers document, along with your design standards and conventions.

[jg4040]

I understand that. We are attempting to do what we thought would be a simple case: Create a generic sort.

Thought this would be easy. Pass filename. schema and sort parameters. But under the covers not so easy to build, for this simple case of sorting a file based on 1 field v.s. 3 fields. We are going down the path of adding a field which would be the concatinated content of all the sort fields and then sorting on that. Are we making this more difficult than it needs to be?

[ArndW]

Building a good solid generic job or shared container in PX isn't as easy as it would seem. But by using rcp and proper programming techniques it is a very powerful and useful thing to do.

[jg4040]

But are we over engineering this. Or is this how we have to do it with Datastage.

Building a good solid generic job or shared container in PX isn't as easy as it would seem. But by using rcp and proper programming techniques it is a very powerful and useful thing to do. ...

[ArndW]

That is hard to tell, but I would think that passing the job the names of the sort columns should be enough and that you wouldn't have to add an extra temporary column to the data in order to sort.