Dynamism in DataStage

pravin1581 · Post by **pravin1581** » Mon Mar 19, 2007 11:19 pm

Hi All,

How can we implement dynamism in DataStage ? My requirement is there is a metadata file which contains the column name , length and thecolumn order and there is another input file which contains the data corresponding to the metadata file . The metadata file is dynamic in nature i.e. the no. of columns can vary in the file. Please suggest me a way to implement the above functionality.

Thanx in advance.

I_Server_Whale · Post by **I_Server_Whale** » Mon Mar 19, 2007 11:40 pm

What is the format of your metadata file? Can you give an example?

pravin1581 · Post by **pravin1581** » Tue Mar 20, 2007 1:01 am

I_Server_Whale wrote:What is the format of your metadata file? Can you give an example?

The metadata file is a csv file. It contains three columns :

ColumnName ColumnWidth ColumnOrder
Name 10 1
EmpId 10 3
Salary 10 2

The Metadata file is more or less like this , but the rows may increase or decrease and that is the dynamism involved. The rows needs to be sorted according to the column order and mapped with the input data file and finally load in a another file.

ray.wurlod · Post by **ray.wurlod** » Tue Mar 20, 2007 6:07 am

I am unaware of any way to achieve this in server jobs.

Doubtless some kind of wizard could be written to read the file and create the table definition in the DataStage Repository - indeed, I've done that, but what becomes of the metadata after that? There's no extra information in the file about what to do with it. And, for text files, whence is the information about the format to be obtained?

pravin1581 · Post by **pravin1581** » Tue Mar 20, 2007 6:17 am

ray.wurlod wrote:I am unaware of any way to achieve this in server jobs.

Doubtless some kind of wizard could be written to read the file and create the table definition in the DataStage Repository - indeed, I've done that, but what becomes of the metadata after that? There's no extra information in the file about what to do with it. And, for text files, whence is the information about the format to be obtained?

Can you go ahead with the wizard part, may be that can be of some help.

ray.wurlod · Post by **ray.wurlod** » Tue Mar 20, 2007 6:28 am

Too busy right now (at least for a week).

chulett · Post by **chulett** » Tue Mar 20, 2007 6:50 am

Invoking a Wizard is generally considered risky... best not to anger them with unreasonable demands.

Even if you do get this 'some help' I don't see how it's really going to help. The functionality you seek isn't available if you are expecting to be able to handle this kind of information 'on the fly' inside one job.

Something could be built to automagically build jobs from a template if you have the skillz, one for each unique combination of file values. By the time you do that however, you could have created a true 'template' job and spun off the jobs yourself by hand I would think.

pravin1581 · Post by **pravin1581** » Fri Mar 23, 2007 7:03 am

chulett wrote:Invoking a Wizard is generally considered risky... best not to anger them with unreasonable demands.

Even if you do get this 'some help' I don't see how it's really going to help. The functionality you seek isn't available if you are expecting to be able to handle this kind of information 'on the fly' inside one job.

Something could be built to automagically build jobs from a template if you have the skillz, one for each unique combination of file values. By the time you do that however, you could have created a true 'template' job and spun off the jobs yourself by hand I would think.

What about RCP property ?

pravin1581 · Post by **pravin1581** » Fri Mar 23, 2007 7:04 am

chulett wrote:Invoking a Wizard is generally considered risky... best not to anger them with unreasonable demands.

Even if you do get this 'some help' I don't see how it's really going to help. The functionality you seek isn't available if you are expecting to be able to handle this kind of information 'on the fly' inside one job.

Something could be built to automagically build jobs from a template if you have the skillz, one for each unique combination of file values. By the time you do that however, you could have created a true 'template' job and spun off the jobs yourself by hand I would think.

What about RCP property ?

chulett · Post by **chulett** » Fri Mar 23, 2007 7:17 am

What about it? Are we moving now from Server to PX jobs?

Don't see how it would help as it can't make metadata up out of whole cloth.

DSguru2B · Post by **DSguru2B** » Fri Mar 23, 2007 7:20 am

RCP is not present in server jobs.

pravin1581 · Post by **pravin1581** » Mon Mar 26, 2007 12:36 am

DSguru2B wrote:RCP is not present in server jobs.

If this requirement can be met in Parallel jobs , then no issues.

DSguru2B · Post by **DSguru2B** » Mon Mar 26, 2007 7:00 am

RCP is a good option that px jobs have but thats just for propogating the records. For any explicit transformations, the columns need to be defined.
And Yes, your requirement can be met in px jobs as long as you understand about the limitations of rcp.

pravin1581 · Post by **pravin1581** » Mon Mar 26, 2007 11:33 pm

DSguru2B wrote:RCP is a good option that px jobs have but thats just for propogating the records. For any explicit transformations, the columns need to be defined.
And Yes, your requirement can be met in px jobs as long as you understand about the limitations of rcp.

I have tried the RCP option and created the schema file that contains the metadata information , but the job is getting aborted with the following error :
Delimiter for field "Store" not found; input: <empty>, at offset: 8

The file that I am currently reading has comma as the filed delimeter and same has been defined in the job also, but its not working. Please provide a solution.