So what exactly is stored in the XMETA repository?

[chulett]

A couple of recent posts by Ray, like this one and another one about Lance's ghost project have bot me thinking... and I'm wondering what exactly is (and is not) stored in the XMETA repository.

For whatever reason, I've been under the impression that it was an extra crispy RDBMS model of the complete original recipe 'Universe' repository for the likes of DS_JOBS, DS_JOBOBJECTS, etc etc. Basically that all of the job design time information, routines, etc were stored there as well as in the old repository... completely duplicated, in other words. From those two posts I'm getting the impression that that isn't the case, that perhaps the job details are still only in the old repository (and why an export of a project may be enough of a backup for recovery purposes) and that only the 'higher level' objects are duplicated in XMETA - pointers or catalog records for Projects, Jobs, objects of that nature but the nitty-gritty is still just down where it has always been.

Can anyone clarify that for me, please?

[ray.wurlod]

Simplistic answer is that things that need to be sharable are in XMETA, things that don't need to be sharable are kept within individual products.

Thus, for example, the data analyzed by Information Analyzer are kept in the IADB but the analysis results are kept in XMETA.

Because nearly all design components from DataStage need to be sharable (at the very least for Metadata Workbench), then most everything from DataStage is also in XMETA. The easiest way to see the table names is to query the table names. For DB2:

Code: Select all

select name from sysibm.systables where name like '%DATASTAGE%'

So your recollection that it's completely duplicated is fairly close to the mark. (Intermediate saves from Designer, such as when you click OK to close stage properties, stored in DS_TEMPnnn, are not saved to XMETA. They are saved to XMETA when the job design is saved.)

Code: Select all

DATASTAGEX_XMETAGEN_DSARGUMENTMAPE2C67B8F                                       
DATASTAGEX_XMETAGEN_DSCANVASANNOTATIONE2C67B8F                                  
DATASTAGEX_XMETAGEN_DSCOLUMNDEFINITIONE2C67B8F                                  
DATASTAGEX_XMETAGEN_DSDATACONNECTIONE2C67B8F                                    
DATASTAGEX_XMETAGEN_DSDATACONNECTION_IMPLEMENTEDBY_DSSTAGETYPE_IMPLEMENTS_DSDATACONNECTIONE2C67B8F
DATASTAGEX_XMETAGEN_DSDATACONNECTION_USES_DSPARAMETERSET_CREATEDFROM_DSDATACONNECTIONE2C67B8F
DATASTAGEX_XMETAGEN_DSDATAELEMENTE2C67B8F                                       
DATASTAGEX_XMETAGEN_DSDATAITEMPROPSE2C67B8F                                     
DATASTAGEX_XMETAGEN_DSDATAQUALITYSPECE2C67B8F                                   
DATASTAGEX_XMETAGEN_DSDERIVATIONE2C67B8F                                        
DATASTAGEX_XMETAGEN_DSDESIGNVIEWE2C67B8F                                        
DATASTAGEX_XMETAGEN_DSEXTERNALDEPENDENCYE2C67B8F                                
DATASTAGEX_XMETAGEN_DSEXTERNALDEPENDENCY_CALLS_DSROUTINE_CALLEDBY_DSEXTERNALDEPENDENCYE2C67B8F
DATASTAGEX_XMETAGEN_DSEXTERNALDEPENDENCY_RUNS_DSJOB_RUNBY_DSEXTERNALDEPENDENCYE2C67B8F
DATASTAGEX_XMETAGEN_DSFILTERCONSTRAINTE2C67B8F                                  
DATASTAGEX_XMETAGEN_DSFLOWVARIABLEE2C67B8F                                      
DATASTAGEX_XMETAGEN_DSFOLDERE2C67B8F                                            
DATASTAGEX_XMETAGEN_DSFOREIGNKEYDEFE2C67B8F                                     
DATASTAGEX_XMETAGEN_DSFOREIGNKEYDEF_REFERENCES_DSTABLEDEFINITION_REFERENCEDBY_DSFOREIGNKEYDEFE2C67B8F
DATASTAGEX_XMETAGEN_DSFUNCTIONCALLE2C67B8F                                      
DATASTAGEX_XMETAGEN_DSIMSAREAE2C67B8F                                           
DATASTAGEX_XMETAGEN_DSIMSDATABASEE2C67B8F                                       
DATASTAGEX_XMETAGEN_DSIMSDATASETE2C67B8F                                        
DATASTAGEX_XMETAGEN_DSIMSFIELDE2C67B8F                                          
DATASTAGEX_XMETAGEN_DSIMSLCHILDE2C67B8F                                         
DATASTAGEX_XMETAGEN_DSIMSSEGMENTE2C67B8F                                        
DATASTAGEX_XMETAGEN_DSIMSSEGMENTSOURCEE2C67B8F                                  
DATASTAGEX_XMETAGEN_DSIMSSEGMENT_REFERENCES_DSTABLEDEFINITION_REFERENCEDBY_DSIMSSEGMENTE2C67B8F
DATASTAGEX_XMETAGEN_DSIMSSENFLDE2C67B8F                                         
DATASTAGEX_XMETAGEN_DSIMSSENSEGE2C67B8F                                         
DATASTAGEX_XMETAGEN_DSIMSSENSEG_REFERENCES_DSTABLEDEFINITION_REFERENCEDBY_DSIMSSENSEGE2C67B8F
DATASTAGEX_XMETAGEN_DSIMSVIEWE2C67B8F                                           
DATASTAGEX_XMETAGEN_DSIMSVIEWSETE2C67B8F                                        
DATASTAGEX_XMETAGEN_DSIMSVIEW_REFERENCES_DSIMSDATABASE_REFERENCEDBY_DSIMSVIEWE2C67B8F
DATASTAGEX_XMETAGEN_DSIMSXDFIELDE2C67B8F                                        
DATASTAGEX_XMETAGEN_DSINPUTPINE2C67B8F                                          
DATASTAGEX_XMETAGEN_DSITEME2C67B8F                                              
DATASTAGEX_XMETAGEN_DSJCLTEMPLATEE2C67B8F                                       
DATASTAGEX_XMETAGEN_DSJOBDEFE2C67B8F                                            
DATASTAGEX_XMETAGEN_DSLINKE2C67B8F                                              
DATASTAGEX_XMETAGEN_DSLOCALCONTAINERDEFE2C67B8F                                 
DATASTAGEX_XMETAGEN_DSMACHINEPROFILEE2C67B8F                                    
DATASTAGEX_XMETAGEN_DSMETABAGE2C67B8F                                           
DATASTAGEX_XMETAGEN_DSMFCOLUMNINFOE2C67B8F                                      
DATASTAGEX_XMETAGEN_DSOUTPUTPINE2C67B8F                                         
DATASTAGEX_XMETAGEN_DSPARAMETERDEFE2C67B8F                                      
DATASTAGEX_XMETAGEN_DSPARAMETERSETE2C67B8F                                      
DATASTAGEX_XMETAGEN_DSPARAMETERVALE2C67B8F                                      
DATASTAGEX_XMETAGEN_DSPROJECTE2C67B8F                                           
DATASTAGEX_XMETAGEN_DSROUTINEE2C67B8F                                           
DATASTAGEX_XMETAGEN_DSSHAREDCONTAINERDEFE2C67B8F                                
DATASTAGEX_XMETAGEN_DSSTAGEE2C67B8F                                             
DATASTAGEX_XMETAGEN_DSSTAGEPARAMPROPSE2C67B8F                                   
DATASTAGEX_XMETAGEN_DSSTAGERECORDE2C67B8F                                       
DATASTAGEX_XMETAGEN_DSSTAGETYPEE2C67B8F                                         
DATASTAGEX_XMETAGEN_DSSTAGETYPE_USES_DSTABLEDEFINITION_USEDBY_DSSTAGETYPEE2C67B8F
DATASTAGEX_XMETAGEN_DSSTAGE_RUNS_DSJOBDEF_RUNBY_DSSTAGEE2C67B8F                 
DATASTAGEX_XMETAGEN_DSSTAGE_USES_DSDATACONNECTION_USEDBY_DSSTAGEE2C67B8F        
DATASTAGEX_XMETAGEN_DSSTAGE_USES_DSIMSVIEWSET_USEDBY_DSSTAGEE2C67B8F            
DATASTAGEX_XMETAGEN_DSSTAGE_USES_DSMACHINEPROFILE_USEDBY_DSSTAGEE2C67B8F        
DATASTAGEX_XMETAGEN_DSTABLEDEFINITIONE2C67B8F                                   
DATASTAGEX_XMETAGEN_DSTABLEDEFINITION_IMPORTEDBY_DSDATACONNECTION_IMPORTS_DSTABLEDEFINITIONE2C67B8F
DATASTAGEX_XMETAGEN_DSTABLEDEFPROPSE2C67B8F                                     
DATASTAGEX_XMETAGEN_DSTRANSFORME2C67B8F                                         
VWDATASTAGEX_XMETAGEN_DSIMSROOTOBJECTE2C67B8F                                   
VWDATASTAGEX_XMETAGEN_DSIMSSUBOBJECTE2C67B8F                                    

  66 record(s) selected.

All you have to do is to train your brain to ignore the DATASTAGEX_XMETAGEN_ prefix and the unique ID suffix on each table name and the data model becomes clear!

Executables do not need to be sharable and hence are not (as far as I have been able to determine) stored in XMETA.

Caveat: reverse engineering from XMETA will violate your licence agreement with IBM. I have only posted what's visible to be seen.

[ray.wurlod]

Logging is more complex, because DataStage logs (where RTLogging/ORLogging direct logging to XMETA) fit into the more generic logging structure of Information Analyzer.

That, in turn, is why the default logging view in Web Console for Information Server only lets you see the entirety of logged events from a project (not from an individual job).

[chulett]

Ah... ok, thanks for the clarification. I guess I just went off on a tangent with the two responses I linked to above, especially the first one where it seemed like just doing work in the Administrator could have resolved the problem. The last paragraph seemed to imply that as well... the Administrator work should remove the job from the Designer but if it is still visible in the Director only then you have to go after the XMETA tables. But

I guess the bottom line there it depended which of the two 'halves' of the information for the job were gone and which still remained after the job creation / deletion failed.