I have designed a job to remove duplicates using the key change column in sort stage. I have written the records with the key change value=1 into target1 file with constraint "keychange value =1" in transformer and other records key change column value=0 with constraint in transformer with "keychange value=0" into other target file.
designed job as below
Code: Select all
seqfile --sort stage--copystage--transformer 1 ---target 1
--transformer 2 ---target 2
Please let me know. I am not able to find attachments option in this message otherwise i would have shown the job design in this message
Thanks
Rakesh