DSXchange

Load your data into a work table (temp table) and do database join. You dont need px just for handling huge reference data.

Yes.

True. But my comment was w.r.t performance. Memory demands follow simple economics. Demand and supply. I have never had memory leaks with a transformer stage. I have had memory violations while using custom px routine but that was my own faulty memory management in the C code. Other than that, the t...

No need to avoid the transformer stage as its performance has been greatly increased in 7.x versions. No doubt, modify stage still leads the two, but transformer is faster than filter stage and a few other stages. IBM released this information sometime back.

Welcome Aboard
My advice, go for Server To Parallel transition classes. Ray's class is going on in Dallas as we speak.

Create functions like your original bat script has (doPROD, doEND, doDRIV etc). In a korn shell it would be something like ----- ---- ---- ------ doPROD() { ----- ----- ----- } doDRIV() { ----- ----- ----- } ######## # MAIN # ######## doPROD doDRIV other_functions...

You need to design your job to capture rejects. Nothing can be done for that at the unix level.
Warning limit is set while running the job, from command line, it will be dsjob -run -warn x.............where x is the number of warnings encountered before the job aborts.

Welcome Aboard
How are you capturing the rejects? Are you using Oracle OCI stage or the enterprise stage? What is the warning limit set?
The rejections must be greater than the warning limit set which is 50 by default. If you capture the rejects, you should not be getting these warnings.

It is simple. Ask Craig

Going through "How to ask smart questions"?would help.

Then I guess you need to change to either odbc stage or return single row per key.

To be really honest, the script is long and dont have the time to go through it, but a few pointers. Create a variable that holds the path of your log files and then re-direct to the variable instead of the entire path. If your log file path changes, you will have to change it in one place, instead ...

Are you using ODBC stage for lookup. Because only that stage supports multiple row returns. Search on keywords "Reference lookup generated %d rows. Only the last row was used" to get more posts relating to your error.

I concur with Cr.Cezon. Write a small after job sub-routine that has a bunch of DSExecute() commands making all different calls. This way you can have in depth error handling incorporated as well.

The tone is certainly "demanding." A "please" would be nice.