aggregator performance

[dnat]

I am using an aggregator stage just to count the number of rows from a particular link.

The design is like this

Seq file-->transformer-->aggregator-->seq file

Here i need the aggregator to count the total rows from transformer(the key is same for all the records), so it would pass through only one partition.

I am dealing with millions of records. Now we are doing development, but wanted to know how this would affect the performance. Or is there any other way to do this?

[bkumar103]

Are you getting just count of the record in the output Sequential file.
If yes then you can use wc -l < inputfilename > outputfilename to get the count.

[ray.wurlod]

Do you really need the count as a separate operation? Why not calculate it as you are processing the actual file?

[sjaladurgam]

Even I experienced same issue.But I tried keeping 2 Agg Stages and making first one with hash partitioning and second one with sequential that works fantastic.

Just try this.

Thanks.

[sima79]

One aggregator stage (execution mode parallel) to count the rows in parallel then another aggregator stage (execution mode sequential) to sum up the counts from each partition. No need to use hash partitioning, round robin in this case would be better.

[dnat]

sima and sjaladurgam

So, the two aggregator stages would not hinder the performance while doing for millions of records???. i am just worried since the data is very huge..anyway, thanks for your input.

Ray, i am not sure how we can calculate while actual processing, because anyways i have to calculate withouth the partitioning to get the total count.

[Sainath.Srinivasan]

...and second one with sequential ...

[dnat]

i made the first aggregaor as round robin and next as sequential mode. But the output is not correct.

The first aggregator shows as a collection type.

[dnat]

The first aggregator was showing as collection type because it was in sequential mode. I made it to parallel and partitioned in round robin. The second aggregator is in sequential mode. But it is not giving correct output.

[Sainath.Srinivasan]

What do you mean by "not giving correct data"?

Unless you share the results, it is not even possible to guess what is happening differently.