FileName in Reject Link of Sequential File Stage

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
nirav.parikh
Participant
Posts: 14
Joined: Thu Dec 13, 2007 2:57 am

FileName in Reject Link of Sequential File Stage

Post by nirav.parikh »

Hi,
We have a job which reads multiple sequential files from a specific folder using FilePattern. Additionally these files are read using a Schema File option. The reject Mode on this Sequential File stage is Set to 'Output' and for now we have attached a peek to the reject link output. We have also added the 'File Name Column' to the output tab as we want to capture the full file names from which each record came. To get these Full File names we have set $APT_IMPORT_PATTERN_USES_FILESET = True. We also have the 'Reject Reason Column' mapped to a column on the output for capturing the reason for any rejects by the Seq File Stage.

Now for the problem:
The above job works fine for us in most cases, we are having Issues when some records are rejected by the Seq file stage. for each reject record we need the Reject Reason and the File Name from which that record originated from, as this information has to be stored in a error table.

The rejected records that are spit out on the reject link and make their way to the Job Log via Peek but for these reject records we are not able to get the corresponding FileNames. That is with each rejected record we need another column, which will indicate that rejected record came from which specific Source File.

To overcome this issue we tried a different design.
We read the entire contents of the Seq File as a single large Varchar Column, and also added the FileName Property to the output Tab. After this Seq File Stage we used the Column Import stage and used the Schema file with this stage. We set the reject mode to Output on Column Import stage, so now for the reject records it shows the entire record plus the filename column but not the reject reason column.

So with either design we are not able to get both the information together which we want for our reject handling process.

Can someone please suggest any alternatives to get the required information for the reject handling part.

Thanks
Thanks & Regards
Nirav
Post Reply