Dynamic lookup in ISD and behaviour of Aggregator stages.

allavivek · Post by **allavivek** » Wed Nov 28, 2012 11:54 am

Hi All,

Iam working on 8.7 version ISD jobs.

When we use lookup stage reference(Normal ), does for every request to service the data gets refreshed and loaded to memory or is it one time load ?

Also , how does the behaviour of aggregator stages work if the data is broke down to packets before sending to ISD input ? dont we get wrong results ?

Thank you.

Vivek

eostic · Post by **eostic** » Wed Nov 28, 2012 7:37 pm

1. Use only "sparse" lookups for ISD based Jobs. Otherwise you will not get updates...there is no refresh, and you wouldn't want one because that could take a long time.

2. The aggregator works fine because it is all based on "end of wave"....meaning that each "request" is like a new job set....the aggregator will release its rows (assuming you are not telling it about sorting, etc.) at the end of each "request's" set of data (which could be a whole array), as though it was a batch job that was finishing.

The only exception I've seen to this behavior is @INROWNUM, which simply continues to increment for the life of the job instance. Perhaps @OUTROWNUM does the same thing, although I haven't tried it.

Ernie

allavivek · Post by **allavivek** » Fri Nov 30, 2012 11:34 am

Thanks Ernie for information.

allavivek · Post by **allavivek** » Sun Dec 02, 2012 11:20 pm

Ernie ,

I have a question , When we are dealing with say 100k of records , does sparse lookup doesn't effect performance ?

Thank you.

ray.wurlod · Post by **ray.wurlod** » Sun Dec 02, 2012 11:28 pm

Probably not, because an ISD job is probably only performing one lookup; the performance difference will be minor.