hash partioning problem in remove duplicate stage

[kpavan2004]

Hi
Requirement:I have a requirement to get the latest record based on timestamp field.

Scenario 1: So i have used remove duplicate stage for this.
I have done the hash partitioning on the key column and timestamp field and selected "perform sort" option in the input link of this stage.
and in the remove duplicate stage properties key is the key column[excluding timestamp field]. But the ouput is having duplicate rows for the key.
When i checked i came to know that the records are not in the same partition. So hash partition is not working correctly. Any
thing wrong in the logic?

Scenario 2: If i use sort stage where in the input link i did hashing on key column , and in the stage properties i sorted on key column and timestamp field
and then i used remove duplicate stage on key column the output is without duplicate records.It is working correctly but


Could anyone please tell me why in scenario one the hash partitining is not working? I thought it is due to duplicates but even

[mhester]

I believe that in scenario 1 it is working exactly as you have coded it. You are partitioning on key and timestamp which will not produce the same data distribution as simply partitioning on the key as in your second scenario.

[vinothkumar]

You should not include timestamp for hash partitioning for Scenario1

[kris007]

You need to only partition on the key field but sort on the Timestamp field within the Remove Duplicates stage to achieve the required results. Based on what data you want to retain, you can sort ascending/descending on the TimeStamp field.

Hope that helps.