Merge: Memory crash problem

ChristopheA · Post by **ChristopheA** » Fri May 16, 2008 9:40 am

Hi,
I've a list of 10000 ids. I need to retrieve the corresponding name for each id of this list.
Names are stored in a 400000 lignes' table. According to the size of this table, I can't use it as reference in a lookup.

I've triied a merge stage between my entry flux and this table but the job have aborted due to a "memory crash".

Can someone help me? How can I design my job to spend less memory?

Thanks
Chris.

ArndW · Post by **ArndW** » Fri May 16, 2008 9:42 am

What exact message do you get when using it as a lookup or as merge?

ChristopheA · Post by **ChristopheA** » Tue May 20, 2008 3:02 am

Hi,
The exact error message when I'm using a merge stage is:

Code: Select all

asc}}},0: Failure during execution of operator logic.
asc}}},0: Fatal Error: Tsort merger aborting: Scratch space full

Thanks

Chris

ArndW · Post by **ArndW** » Tue May 20, 2008 3:06 am

The message is clear - you have run out of temporary disk space. If you look at your APT_CONFIG file you will see one or more lines for each node reading

Code: Select all

resource scratchdisk "{path}"

. That is the drive that filled up. Either make it larger or add additional scratch disks to your nodes.