Unbearably slow compile with a funnel

Pavan_Yelugula · Post by **Pavan_Yelugula** » Wed Jun 20, 2007 1:06 pm

Hi All
I am having a simple job with a Couple of Datasets, one lookup and a transformer and a funnel. The job takes around 10 min to compile or more when the funnel type is sequence when i change the funnel to continous or others...it compiles in less than a minute...

Is there any particular reason for this behaviour... I need to use Sequence in funnel...is there some way i can improve the run time as well as the compile time

Thanks
Pavan

ray.wurlod · Post by **ray.wurlod** » Wed Jun 20, 2007 5:46 pm

I doubt it's the Funnel stage - the Funnel stage does not get compiled, it simply generates a funnel operator in the generated OSH. The Funnel type (continuous, sort funnel or sequential) is just a command line option for the funnel operator. Examine the generated OSH.

Sequential funnel is only useful if the data are range partitioned, to preserve the ranges. Otherwise you really don't need it, and continuous funnel will do just as well. Why do you believe you need to "use Sequence in funnel" (which I found to be rather misleading terminology till I thought about it)?

It's only Build and Transformer stages that generate C++ code that has to get compiled, and you do notice a substantially longer compilation time in these cases - particularly for more complex Transformer stages.

Pavan_Yelugula · Post by **Pavan_Yelugula** » Wed Jun 20, 2007 6:19 pm

Ray,
Initially the job started compiling better with the change of funnel option but then it started getting slow again.

I removed the funnel stage altogether and it is compiling much better now. There almost 10 links to the funnel does this have to do with anything....

except for this job rest of the jobs compile and run really fast...

Thanks
Pavan

ArndW · Post by **ArndW** » Wed Jun 20, 2007 6:51 pm

Pavan - don't equate compile time with execution speed; the delay in the compilation phase that I've seen with TRANSformer stages is very lengthy, but the compiled code executes very quickly.

Are you more worried about compile times or run times?

vijayrc · Post by **vijayrc** » Wed Jun 20, 2007 9:11 pm

Pavan_Yelugula wrote:Ray,
Initially the job started compiling better with the change of funnel option but then it started getting slow again.

I removed the funnel stage altogether and it is compiling much better now. There almost 10 links to the funnel does this have to do with anything....

except for this job rest of the jobs compile and run really fast...

Thanks
Pavan

Pavan,
It isn't the Funnel. I guess it's the transformer with those 10 links that's causing the lenghty compile time. the complex derivations in the transformer could be one of the reason.
Hope this helps.
Vijay

Pavan_Yelugula · Post by **Pavan_Yelugula** » Thu Jun 21, 2007 7:09 am

Thanks ArndW and Vijay

ArndW
Though run time is more important than the compile time... It is really frustating to wait for almost 20 min for every minor functionality change in the job.
As Ray Pointed out that the funnel is not compiled but a funnel operator is kept in the generated OSH. I guess It follows the same for the rest of my stages except for the transformer.

Is it right if i generalize that excluding the bottle necks like band width, processor speeds, memory...etc. If a job is getting compiled slowly than the other jobs then the trouble is with the transformer stage in the job...

As Vijay Pointed out will my job compile any faster if i remove the constraints from the links and put them as stage variables. I still wonder if 9 links out of an transformer can hold the compile time for more than 20 minutes.

Thanks
Pavan

chulett · Post by **chulett** » Thu Jun 21, 2007 7:24 am

I'd be surprised if it actually takes all that time to do the physical compile. We've seen situations like this in the past, where the compiler license limits the number of concurrent compiles - sometimes even restricting it to one user at a time. This seemed to translate into long wait times for the compiler to become available rather than to actual compile.

To you the user, however, it just looks like a 'frustratingly long compile time'. Any chance this is going on?

vijayrc · Post by **vijayrc** » Thu Jun 21, 2007 7:20 pm

chulett wrote:I'd be surprised if it actually takes all that time to do the physical compile. We've seen situations like this in the past, where the compiler license limits the number of concurrent compiles - sometimes even restricting it to one user at a time. This seemed to translate into long wait times for the compiler to become available rather than to actual compile.

To you the user, however, it just looks like a 'frustratingly long compile time'. Any chance this is going on?

Well the compile of just this job takes this long to compile. Anyother job compile isn't anywhere near to what this takes, though more complex jobs than this compiles faster. That makes it more interesting.

Here's the history of what I tried......
Ended 8 link out of transformer witha copy, took off the funnel and
continued the link and this didn't help.

Deleted 6 of the 9 links and compiled went in 2mins. Added one more link and it took an extra minute, added one more link and it started taking more and more minutes...!!

Also the transformer with 9 links were replaced with three transformers with three links in each, and this brought down the compile time to 6mins, though this also is in the high-end for a job compile [a not so complex job]
Any other thoughts would be highly appreciated.

Vijay