Page 1 of 1
Hash Table Warning
Posted: Mon Sep 19, 2011 7:40 am
by Marley777
Thanks for reading.
Getting warnings when using an agg stage.
Hash Table has grown to 32768
I did a forum search and found that sorting on the grouping keys will elimintae the warning. If my 'partitioning type' is set to 'auto', is sorting required...wouldn't partition type set to auto take care of any sorting needed?
fyi - Agg stage is using hash method.
Posted: Mon Sep 19, 2011 7:49 am
by battaliou
If you method is sort then datastage can release the grouped data as soon as it is processed. This is far more efficient.
Posted: Mon Sep 19, 2011 7:53 am
by Marley777
So do I need to do my own sort or let datastage handle by using partition type = auto?
Posted: Mon Sep 19, 2011 7:55 am
by chulett
You need to ensure the data is sorted properly to support the aggregation being done.
Posted: Mon Sep 19, 2011 8:20 am
by Marley777
Auto Partition should should take care of any sorting that needs to be done...correct?
always thought if you have the env variables below set to true DS will sort for you based on the grouping keys used in the agg stage by inserting tsort and partitioning operators?
APT_NO_SORT_INSERTION = TRUE
APT_NO_PART_INSERTION = TRUE
Posted: Mon Sep 19, 2011 9:20 am
by chulett
Setting "NO SORT INSERTION" to true means it is not allowed to insert tsort operators. Same goes for partitioning.
Posted: Mon Sep 19, 2011 10:53 am
by Marley777
Yes..it should be set to false. Which means it should take care of the sorting correct?
Posted: Mon Sep 19, 2011 11:11 am
by chulett
I don't believe so, try it and let us know.
Posted: Mon Sep 19, 2011 2:05 pm
by Marley777
Seems to be working for me. We use the aggregator stage to bring fields from multiple records with the same keys into one record. As of now the data looks good. In the agg stage I'm using the sort method versus hash. Warnings are gone.
Thanks for your help.