Big lookup: cannot allocate memory error

stefanfrost1 · Post by **stefanfrost1** » Tue Oct 16, 2007 1:02 am

Have you tried to force the lookup to be partitioned? Maybe you will be able to spread your data across nodes and therefore minimize your overall volume....

...btw duplicates are not allowed, if you have a duplicate there will be a warning in the log telling you that a record has been discarded due to duplicates on unique values.

evans036 · Post by **evans036** » Tue Oct 16, 2007 5:40 am

stefanfrost1 wrote:Have you tried to force the lookup to be partitioned? Maybe you will be able to spread your data across nodes and therefore minimize your overall volume....

that is a good idea and will likely work in most cases.

This is an excercise where i am trying to understand what memory constraint is causing the lookup to fail. I need to do this to help us write robust DS code.

thanks for the suggestion,
steve

stefanfrost1 · Post by **stefanfrost1** » Tue Oct 16, 2007 6:24 am

Ok I see..

Not sure if these will help you but in a tes scenario its worth a shot. They are set at project level in administrator and will print some additional information in the log...

APT_LOOKUP_VERBOSE
Causes the lookup operator to print some information when it
has completed loading a lookup table into memory.
APT_PM_PLAYER_MEMORY
Causes each player process to use APT_LOGMSG to report the
process heap memory allocation when returning from
runLocally().

evans036 · Post by **evans036** » Tue Oct 16, 2007 7:08 am

stefanfrost1 wrote: APT_LOOKUP_VERBOSE
Causes the lookup operator to print some information when it
has completed loading a lookup table into memory.
APT_PM_PLAYER_MEMORY
Causes each player process to use APT_LOGMSG to report the
process heap memory allocation when returning from
runLocally().

thankyou very much, i'll look into these

thanks again,
steve