non-database sparse lookup alternative options

A forum for discussing DataStage<sup>®</sup> basics. If you're not sure where your question goes, start here.

Moderators: chulett, rschirm, roy

ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

qt_ky wrote:A normal Lookup is doing something more behind the scenes that does not appear to be documented--as it reads data from the reference link, it writes it to some sort of memory mapped file that may or may not have an index built on the key columns (I don't know), and then finally it reads that file into memory.
Your recollection is correct. The reference data are loaded into a hash table in memory and an index to that is built. (That's why the Lookup stage is a composite operator: LUT_CreateTable and LUT_Lookup.) You can do this ahead of time by populating a Lookup File Set.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Aha! Perhaps an option for you... pre-build that before recycling the job?

And kids, don't try this one at home - have the 64bit version of PX installed everywhere EXCEPT production. Large lookups work fine up until that last critical elevation. That was a not fun situation to walk into.
-craig

"You can never have too many knives" -- Logan Nine Fingers
qt_ky
Premium Member
Premium Member
Posts: 2895
Joined: Wed Aug 03, 2011 6:16 am
Location: USA

Post by qt_ky »

Thank you, Ray. That gives us another avenue to explore.

Must the server engine's uvconfig file be modified from the default of:
64BIT_FILES 0
to:
64BIT_FILES 1
and regenerated/restarted to enable 64-bit hashed file capability?

Craig, could you elaborate or post a link to your reference about the 64bit version of PX? I'm not sure what you meant by that.
Choose a job you love, and you will never have to work a day in your life. - Confucius
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

First answer: NO. That forces every hashed file in the project to be 64bit. You just need to pre-create a hashed file as 64bit, I don't recall the syntax for that but it has been mentioned here and I'm sure it's in the docs somewhere. Ah... discussed here.

I just meant that the Production environment there was running the 32bit version of PX but 64bit everywhere else. Bottom line was things like lookups and their "memory maps" that ran fine in all of the lower environments could fail in production unless they handled the partitioning more better.
-craig

"You can never have too many knives" -- Logan Nine Fingers
Post Reply