Hash Table Warning

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
Marley777
Premium Member
Premium Member
Posts: 316
Joined: Tue Jan 27, 2004 3:26 pm

Hash Table Warning

Post by Marley777 »

Thanks for reading.

Getting warnings when using an agg stage.

Hash Table has grown to 32768

I did a forum search and found that sorting on the grouping keys will elimintae the warning. If my 'partitioning type' is set to 'auto', is sorting required...wouldn't partition type set to auto take care of any sorting needed?

fyi - Agg stage is using hash method.
battaliou
Participant
Posts: 155
Joined: Mon Feb 24, 2003 7:28 am
Location: London
Contact:

Post by battaliou »

If you method is sort then datastage can release the grouped data as soon as it is processed. This is far more efficient.
3NF: Every non-key attribute must provide a fact about the key, the whole key, and nothing but the key. So help me Codd.
Marley777
Premium Member
Premium Member
Posts: 316
Joined: Tue Jan 27, 2004 3:26 pm

Post by Marley777 »

So do I need to do my own sort or let datastage handle by using partition type = auto?
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

You need to ensure the data is sorted properly to support the aggregation being done.
-craig

"You can never have too many knives" -- Logan Nine Fingers
Marley777
Premium Member
Premium Member
Posts: 316
Joined: Tue Jan 27, 2004 3:26 pm

Post by Marley777 »

Auto Partition should should take care of any sorting that needs to be done...correct?

always thought if you have the env variables below set to true DS will sort for you based on the grouping keys used in the agg stage by inserting tsort and partitioning operators?

APT_NO_SORT_INSERTION = TRUE
APT_NO_PART_INSERTION = TRUE
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Setting "NO SORT INSERTION" to true means it is not allowed to insert tsort operators. Same goes for partitioning.
-craig

"You can never have too many knives" -- Logan Nine Fingers
Marley777
Premium Member
Premium Member
Posts: 316
Joined: Tue Jan 27, 2004 3:26 pm

Post by Marley777 »

Yes..it should be set to false. Which means it should take care of the sorting correct?
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

I don't believe so, try it and let us know.
-craig

"You can never have too many knives" -- Logan Nine Fingers
Marley777
Premium Member
Premium Member
Posts: 316
Joined: Tue Jan 27, 2004 3:26 pm

Post by Marley777 »

Seems to be working for me. We use the aggregator stage to bring fields from multiple records with the same keys into one record. As of now the data looks good. In the agg stage I'm using the sort method versus hash. Warnings are gone.

Thanks for your help.
Post Reply