Logic not working for generating control numbers

Chandra234 · Participant Joined: 12 Nov 2010 Posts: 9 Points: 101

Hello,

This is a 8node parallel job and we are using logic (@PARTITIONNUM+1):@INROWNUM to generate unique numbers in transformer. But this control number is generating same values for different records in the same run. Looks like the issue is happening at partition 4 after row 464. As per the logic looks like 5464 (partition 4 + 1 concatenated with row 464).
This code is to work in 7.5 but when migrated to 8.5 this is happening. Please advise.We have changed the logic to ( (@INROWNUM - 1) * @NUMPARTITIONS + @PARTITIONNUM + 1) and is working. Please help if there is anything wrong with previous code ((@PARTITIONNUM+1):@INROWNUM).

Thank you.
Chandra234

jwiles · Posted: Mon Jul 30, 2012 1:41 pm

What are some examples of the duplicate values you are seeing generated with the original logic, and what do you believe the values <should> have been? Do they come from within the SAME partition or from SEPARATE partitions?

Not accounting for varying length of integer numbers converted to strings (when concatenating them together) is one of the usual reasons one sees duplicates created. As an example using your original logic:

Code:

Partition 0 Record 1464: 0+1 : 1464 = 1 : 1464 = 11464
Partition 10 Record 464: 10+1 : 464 = 11 : 464 = 11464

Regards,

Kryt0n · Participant Joined: 22 Jun 2005 Posts: 584 Points: 3356

(@INROWNUM - 1) * @NUMPARTITIONS + @PARTITIONNUM + 1) would be a much tidier way of generating unique numbers.

While jwiles example is a good reason why not to follow the concatenation route, it only really applies when you have 9+ partitions (or 10+ if you don't use "+1").

No obvious reason as to why row 464 in partition 5 should be a candidate for duplicate, not doing any kind of transformer looping are you? Would assume not if this is a direct port but have to ask... How many rows do you expect in each partition and what is the datatype downstream?

Create a peek stage that includes partition num, inrownumber and your unique row for when it hits 5464 to get a better idea where it is coming from.

Chandra234 · Participant Joined: 12 Nov 2010 Posts: 9 Points: 101

Thank you all for the response . But in my case I have a 8 node configuration. I am trying to understand why exactly duplicates strarted from row 464. Could you please explain with a example for a 8 node configuration? Thank you.

chulett · Posted: Tue Jul 31, 2012 6:22 pm

In order for people to help, you need to answer the questions being asked of you. For example, James asked two in the first part of his reply and Kryton had a couple as well. I'd also be curious how you are partitioning the data, what method you chose.

jwiles · Posted: Tue Jul 31, 2012 11:28 pm

As Kryton stated, there is no obvious reason in the information you've supplied so far that indicates why you're seeing duplicates when using your original logic. This leads to the questions we are asking in an attempt to help you determine what the problem is--understanding what you're expecting, what you're actually seeing and what else your job does helps in doing that.

Another typical and well-used method is to initialize the sequence counter (a stage variable) to @PARTITIONNUM+1 and increment it by @NUMPARTITIONS

Regards,

Chandra234 · Participant Joined: 12 Nov 2010 Posts: 9 Points: 101

Thank you all for support. Marking this topic as resolved.

jwiles · Posted: Mon Aug 06, 2012 2:25 pm

Out of curiosity, was the problem resolved by utilizing one of the alternatives presented, or did you find the cause as to why the original logic was present you with issues?

Regards,

PaulVL · Participant Joined: 17 Dec 2010 Posts: 442 Points: 3192

Chandra234 wrote:

Thank you all for the response . But in my case I have a 8 node configuration. I am trying to understand why exactly duplicates strarted from row 464. Could you please explain with a example for a 8 node configuration? Thank you.

Chandra, much like James orriginally said:

partition 0 row 464 will colide with partition 4 row 64.

Is that California sun getting to you?

Razz

Kryt0n · Participant Joined: 22 Jun 2005 Posts: 584 Points: 3356

PaulVL wrote:

partition 0 row 464 will colide with partition 4 row 64.

Is that California sun getting to you?

Razz

Not quite, partition 0 row 464 will output as 1464 (and partition 4 will be 564)... they are adding 1 to the partition number.

Time to get out the sun Razz