DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
This topic has been marked "Resolved."
Author Message
metadatamen



Group memberships:
Premium Members

Joined: 26 Oct 2016
Posts: 36

Points: 378

Post Posted: Fri Jul 14, 2017 8:09 am Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Parallel
OS: Unix
Is it possible to run lineage at field level when a particular stage in data stage job is actually have Runtime column Propagation enabled?
eostic

Premium Poster



Group memberships:
Premium Members

Joined: 17 Oct 2005
Posts: 3726

Points: 29791

Post Posted: Fri Jul 14, 2017 11:09 am Reply with quote    Back to top    

Yes.....however, because the "design" has no columnar metadata, you need to collect Operational Metadata at run time so that it can deliver the actual column activities that took place in the Job at r ...

_________________
Ernie Ostic

blogit!
Open IGC is Here!
Rate this response:  
metadatamen



Group memberships:
Premium Members

Joined: 26 Oct 2016
Posts: 36

Points: 378

Post Posted: Mon Jul 17, 2017 9:13 am Reply with quote    Back to top    

I Ernie,

I tried executing the job but same issue.
Is there anything else that I have to do apart from executing the jobs?

Am able to select fields in the jobs that are non-RCP, not in in RCP enabled jobs even after executing the jobs.

In fact I tried in few environment and same behavior.

Please advise.
Rate this response:  
eostic

Premium Poster



Group memberships:
Premium Members

Joined: 17 Oct 2005
Posts: 3726

Points: 29791

Post Posted: Mon Jul 17, 2017 10:25 am Reply with quote    Back to top    

Are you checking the box in the run dialog to gemerate operational metadata..?..it is on one of the tabs at the run dialog.

Ernie

_________________
Ernie Ostic

blogit!
Open IGC is Here!
Rate this response:  
metadatamen



Group memberships:
Premium Members

Joined: 26 Oct 2016
Posts: 36

Points: 378

Post Posted: Mon Jul 17, 2017 10:49 am Reply with quote    Back to top    

Thanks Ernie, Yes I was trying this without enabling this in an assumption that it was enabled before. It worked.

You are awesome as always

In fact I tried more than that and it worked cool.
We are not actually executing DS jobs in IGC environment, instead importing only DS Design information for lineage purpose, I figured out that we could import the DSOMD XML files into IGC environment along with design information instead of executing it in IGC environment to see the Parameter values and Field names for RCP enabled jobs. I tested this and working.

Please confirm if my understanding is correct, or advise if any correction needed in this approach.

.
Rate this response:  
eostic

Premium Poster



Group memberships:
Premium Members

Joined: 17 Oct 2005
Posts: 3726

Points: 29791

Post Posted: Mon Jul 17, 2017 9:20 pm Reply with quote    Back to top    

Congrsts!! You are dping exactly as designed. Many sites keep an independent IGC Info Server instance.......and ship metadata from many different DS environments......some of thrm on earlier rrl ...

_________________
Ernie Ostic

blogit!
Open IGC is Here!
Rate this response:  
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42285
Location: Denver, CO
Points: 217115

Post Posted: Mon Jul 17, 2017 9:39 pm Reply with quote    Back to top    

Not quite sure what "thrm on earlier rrleases" or "the most advanced rcp-capanle omd" might be... are you posting from your phone again Ernie? Wink

_________________
-craig

Watch out where the huskies go and don't you eat that yellow snow
Rate this response:  
eostic

Premium Poster



Group memberships:
Premium Members

Joined: 17 Oct 2005
Posts: 3726

Points: 29791

Post Posted: Mon Jul 17, 2017 9:58 pm Reply with quote    Back to top    

Indeed. : )

_________________
Ernie Ostic

blogit!
Open IGC is Here!
Rate this response:  
metadatamen



Group memberships:
Premium Members

Joined: 26 Oct 2016
Posts: 36

Points: 378

Post Posted: Tue Jul 18, 2017 1:34 pm Reply with quote    Back to top    

Thanks Ernie.

And finally am trying to understand this and getting confused with too many documents.

Let me summarize my understanding and please correct me if am wrong.

We have Operational Metadata [OMD] and Operational Database [ODB]. Are they same or different? Is ODB just one another DB in XMETA repository which stores the OMD information and display the content in IGC?

Does the DSOMDMonApp.sh pushes the xml file into ODB? I looked at the xml file content and it pretty much has all the information needed for operational metadata, and I presume this could be the information going into ODB.

Please confirm. If not, where exactly these xml files getting stored? and what is actually the ODB then?
Rate this response:  
eostic

Premium Poster



Group memberships:
Premium Members

Joined: 17 Oct 2005
Posts: 3726

Points: 29791

Post Posted: Tue Jul 18, 2017 10:14 pm Reply with quote    Back to top    

DSODB and OMD xml are conceptually similar, but are not related functionally. OMD xml is a capability that has been a part of DataStage for well over ten years, and is a critical component of run tim ...

_________________
Ernie Ostic

blogit!
Open IGC is Here!
Rate this response:  
Not yet rated
metadatamen



Group memberships:
Premium Members

Joined: 26 Oct 2016
Posts: 36

Points: 378

Post Posted: Wed Aug 02, 2017 12:19 pm Reply with quote    Back to top    

I think I have a better understanding of how things works now, but I have q question still.

Let's say I have a DS Job with one source file, a transformer and one target file.
e.g. Source File -> Transformer -> Target File [one to one Mapping]

I imported the Source File Metadata alone through istool, and then imported the same metadata in Source Sequential file stage but didn't import any metadata for Target file, instead I only mapped the fields that are necessary for Target File.

Next I executed the job and able see the OMD xml file gets created and processed.

Next when I look at these information in IGC, I can see the source metadata information are exactly mapped with the file that I imported and am able to search this asset in IGC and find all relevant information and be able run lineage at any level.

However, am not able to search for Target File, but can see only in Lineage and in the corresponding job in design and operational information section.

So I understood that my Target file is actually a virtual asset hence it can't be searched or governed since I didn't import this Metadata. But Source file can be searched and governed because I imported the metadata.

Is my understanding correct? And is it possible to make virtual assets searchable? Looks like we have to force people to import the source and target metadata separately though they are defined in DS job itself.
Rate this response:  
Not yet rated
eostic

Premium Poster



Group memberships:
Premium Members

Joined: 17 Oct 2005
Posts: 3726

Points: 29791

Post Posted: Wed Aug 16, 2017 3:21 pm Reply with quote    Back to top    

You are correct in your understanding. Virtual assets are valid for lineage but not governed. I agree it would be nice to be able to search for them, but unfortunately that is not possible. ...

_________________
Ernie Ostic

blogit!
Open IGC is Here!
Rate this response:  
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours