how to remove duplicates in transformer stage..?

[balaya.ds]

Please provide solution for above query?

[keshav0307]

search for "remove Duplicate"

[ray.wurlod]

Use pairs of stage variables (the technique has been discussed here often) to detect change in the sorted keys - here "keys" has the same meaning as in the Remove Duplicates stage - and set a flag that can be used to constrain the output.


Get yourself a premium membership so you can read all of the above.

[nirdesh2]

Partition the Data in Transformer stage and use unique record option. You can remove duplicate record.

[v83rahul]

Partition the Data in Transformer stage using hash partition and use unique record option.

[nvdlrao]

You could remove duplicates using transformer stage 1) with the help of stage variables 2) by doing unique/stable sort.

[asorrell]

No need for stable...