How to auto-generate metadata?
Moderators: chulett, rschirm, roy
-
- Premium Member
- Posts: 34
- Joined: Fri May 16, 2008 6:24 am
How to auto-generate metadata?
Hi, My client has a requirment in which, whenever a source table definition changes (when new columns are added or deleted), it should get updated automatically in the DS job (they are using PX), and should reflect in the target tables. This should happen without making any changes to the job/code, explicitly.
Is there any way to do this? Please suggest.
Is there any way to do this? Please suggest.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
-
- Charter Member
- Posts: 299
- Joined: Wed Nov 13, 2002 5:38 pm
- Location: USA
-
- Premium Member
- Posts: 34
- Joined: Fri May 16, 2008 6:24 am
The source data is on ORACLE & DB2 and loaded into flat files in the target server. What they need is, they do not want the manual intervention to change the Datastage code.throbinson wrote:To quote Walt Disney, "Sometimes dreams do come true".
But for DataStage this dream can quickly turn into a nightmare. It depends on the ease of obtaining the changed metadata. Is it from a DBMS? How is the target metadata maintained and updated?
I just want to know if ther's any possibility to do this using the schema files.
-
- Charter Member
- Posts: 299
- Joined: Wed Nov 13, 2002 5:38 pm
- Location: USA
You could build a DataStage job to query the Oracle system tables for the source table. When a change is detected the job would create the proper schema files. Another DataStage job, RCP enabled, would read the source Oracle table and write out the fields to the just generated schema file in a Sequential stage. Easy to design far more difficult to reliably build and implement.
You have to use some good shell scripting to resolve this issue....OddJob wrote:Combining schema files and Runtime Column Propogation (RCP) could solve your issue.
First you have to read the schema from Oracle and have to generate the Schema files which were then used in the generic DataStage job with RCP enabled but all this will work fine if its just direct loading or extraction.
But if there are some transformations then you may need to maintain the rules for each of the tables and have to try generating the transformation files at runtime which can then be used by DataStage job.