How the Entire partition lookup data set are read?

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
ray.wurlod
Participant
Posts: 54595
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

What do you mean by "lookup data set"?

A Lookup File Set includes its own index.

A virtual Data Set does not. However, if you look at the record schema you will note that there are four control columns, including partition number, partition count, and row number; these could be used for a "binary chop" style of algorithm. A hint cache could also be used to "remember" recently accessed key values.

A sparse lookup will use any index on the table.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
kumar_s
Charter Member
Charter Member
Posts: 5245
Joined: Thu Jun 16, 2005 11:00 pm

Post by kumar_s »

If it is a lookup file set, you may find it indexed on the key marked.
You can easily find this by passing the main stream with unsorted data. If you get a match for all the input it doesnt read for where it left.
And you can post us your findings.
Impossible doesn't mean 'it is not possible' actually means... 'NOBODY HAS DONE IT SO FAR'
Post Reply