Counter value in Grid Partition

dj · Post by dj » Sat Nov 24, 2012 3:35 am

Hi All,

Iam Currently using Datastage 8.5 Version. I have created a counter stage variable in transformer (whichis set in sequential mode)

My requirment is I need to Process the records when the counter <=50
Only the 50 records for each Accounttype.

Counter: If CurrentAccounttype=PrevAcctype then counter+1 else 1.

I have checked the counter values, it is generated correctly for each accttype.

But the output link from transformer does not consider for the constraint.

even when i tried a filter stage it didnt work.

the grid details are computenodes(2,2) and partition 1. since it runs in 2nodes, not all the records <=50 goes down to the next links.

thanks in advance !

-dj

chulett · Post by **chulett** » Sat Nov 24, 2012 8:18 am

Never seen a situation where a constraint was 'ignored'. Show us your actual constraint expression. Also, have you included the counter on the output link or are you just counting?

ray.wurlod · Post by **ray.wurlod** » Sat Nov 24, 2012 3:31 pm

The fact that it's on a grid is immaterial. Counters are computed per node. Constraint expressions are evaluated per node. So, with four nodes, counter <= 50 should generate approximately 200 rows.

chulett · Post by **chulett** » Sat Nov 24, 2012 6:52 pm

It's actually up to 50 records per account type, so no way to know how many output rows we're talking about here. And yes, the 'grid' aspect plays no role here.

jwiles · Post by **jwiles** » Sat Nov 24, 2012 11:14 pm

You said the transformer performing the counting is running in sequential mode. That means it is active in only one logical node. Is the constraint in that transformer, or in a later transformer which runs in parallel mode (and hence in 2 logical nodes)? Seeing the constraint logic would be a helpful step.

What is the data type for your counter?

Regards,

dj · Post by dj » Sun Nov 25, 2012 7:17 am

The output from the transformer for each account type should restrict for 50rows.

Input to transformer is SOrted on Accttype

stagevar details:

Currval: AcctType
Counter : If CurrVal=PreVal then Counter+1 Else 1
PreVal:Currval

Constraint: Counter<=50

Output From Transfomer;

peekstage:

AcctType Counter
A1 1
A1 2
A1 26
A1 50
B1 3
B1 50

The issue is the for each accttype not all 50rows are outputed from transformer.

How does the transformer process the rows ,does it runs in 2 nodes and skips the other valid rows from node 1 when node 2 counter reaches=50

Thanks,
dj

chulett · Post by **chulett** » Sun Nov 25, 2012 9:21 am

How are you partitioning? Does the job work as expected when you run it on one node? If so, your partitioning is incorrect.

chulett · Post by **chulett** » Sun Nov 25, 2012 9:26 am

I'd also be curious what settings you are using in the Peek stage. By default it does not capture all rows...

dj · Post by dj » Sun Nov 25, 2012 10:57 am

In the transformer i have set it as sequential for generating the counter.

In Peek , it is set by Default (Parallel).

when there are 60rows for accttype ,the peekstage doesn't capture all the 50rows.

AcctType Counter
A1 1
A1 2
A1 26
A1 50
B1 3
B1 50

The input is Hash partitioned and sort on AcctType.
**when i tried mentioning explicitly counter=1 or counter=2 or counter=3 in the constrinat, i was able to get the row3.

Please advice where the gap is..

Thanks,
dj

ray.wurlod · Post by **ray.wurlod** » Sun Nov 25, 2012 12:56 pm

Unless you change the default settings, the Peek stage will capture 10 rows per node.