DataQuality Notifications for invalid data

A forum for discussing DataStage<sup>®</sup> basics. If you're not sure where your question goes, start here.

Moderators: chulett, rschirm, roy

Post Reply
jackson.eyton
Premium Member
Premium Member
Posts: 145
Joined: Thu Oct 26, 2017 10:43 am

DataQuality Notifications for invalid data

Post by jackson.eyton »

Hi Everyone!

I wasn't sure where to put this so into General it goes.

We are looking to implement some awareness of some kind in regards to Data Quality. One example of our issues: we have several jobs that build unique business keys by concatenating several fields together. An issue can arise when we use fields that can be null (we know we know....).

So far we cannot find a better set of fields to use (thanks fiserv...). So as you can imagine, if we let those keys be generated as null... well that gives issues to say the least.

So, in the meantime we are eliminating tuples with Null in fields we're using to develop keys via the incoming SQL. This works efficiently for the job, BUT makes it a little less visible when there is bad data. Nearly every time we've seen this so far has been user input error in a date field which fiserv seems to have no data verification on.

Just recently I worked with IBM to get our Standardization Rules Designer up and running. I recall some product that had the ability to send notifications to designated managers when bad data was found. I had assumed this was part of the SRD, but alas I think I was mistaken.

All of the data is still present in our staging tables, so our hope was to have something that would review the data in the stage tables and look at fields we specify, then send notifications/reports when the data in those fields is invalid.

So, to the root of my question, does anyone know if there are any components of InfoSphere that are already designed to do this? Otherwise I will need to create some DataQuality jobs that have various constraints and conditions to pull data that's bad and notify us.

Thanks in advance!
-Me
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

You can use the Subscription Center to react to any event raised within Information Server. You can use the Exception stage to raise an exception in a DataStage job. These, together, should give you pretty much what you need.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
jackson.eyton
Premium Member
Premium Member
Posts: 145
Joined: Thu Oct 26, 2017 10:43 am

Post by jackson.eyton »

Thanks Ray! I have been looking into this for some time now. Part of an issue I have is that it appears our IA setup might have some issues. I am working with IBM on that but in the meantime I was hoping you could point me in the right direction for some good use examples/training how to use the exception stage and subscription center, etc.?
-Me
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

While I'm pondering that (not during the weekend) do a Google search for "subscription center Information Server youtube" and check out some of that training material.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Post Reply