Identify records that generate warnings/errors

pdntsap · Post by **pdntsap** » Tue May 08, 2012 2:12 pm

Hello,

What is the best method of identifying records that generate warnings/errors? We have millions of records as input and we implement the StringToDate() function and some records generate the following warning in the log:

Code: Select all

 Conversion error calling conversion routine date_from_string data may have been lost

We know which stage is giving us the warning but would like to know the record that causes it. Thanks for the help.

ray.wurlod · Post by **ray.wurlod** » Tue May 08, 2012 4:18 pm

You'll need to build in your own tests. They don't happen automatically.

pdntsap · Post by **pdntsap** » Wed May 09, 2012 9:52 am

OK. Thanks.

cdp · Post by **cdp** » Wed May 09, 2012 2:41 pm

Have you/do you have access to any data profiling tools, such as Information Server Information Analyzer or InfoSphere Discovery (there are also a whole host of non ibm and open source options). These sorts of tools should be used during the onboarding/analysis of new data sources to identify the unique patterns (e.g. pattern YYYY-MM-DD occurs 70% of the time and DD-MMM-YYYY occurs 20% etc) as opposed to values found in the data so that transformation requirements can be identified upfront.

Else if you want to test in DataStage a precheck something like this post might allow you to trap the data and perform some additional transformations.