Abnormal termination of stage AGGREGATOR

bhaskarjha · Post by **bhaskarjha** » Fri Aug 11, 2006 5:34 am

Hi

while i m try to aggregate 923000 rows of data , after 910000 rows its gives the warning as "Abnormal termination of stage AGGREGATOR" and job gets the aborted.

chulett · Post by **chulett** » Fri Aug 11, 2006 6:47 am

Sort your input data to support the aggregation you are doing and mark it as sorted in that order in the Aggregator. If you do it correctly, rows will flow through the stage for each 'sort group' instead of holding on to all rows before returning any.

ray.wurlod · Post by **ray.wurlod** » Fri Aug 11, 2006 10:31 pm

Your Aggregator stage has exhausted its available memory. While there is a set of tunables for addressing how the Aggregator stage allocates memory, there is still a finite upper limit. Sorting the input by the grouping columns means that some memory can be freed as soon as one of those values changes, thereby reducing the total demand for memory. You are not permitted to lie; if you assert that the input data are sorted and an out-of-order row arrives, your job will abort.