Startup time is more than Prodcution Run time

vishu19aug · Post by **vishu19aug** » Fri Dec 07, 2012 2:08 pm

Hi,

In one of my job, i can see the following -
Message: main_program: Startup time, 1:39:42; production run time, 34:23

What may be the issue here? I am loading data in 3 tables (1.5 Million, 21 Million, 13 Million)

Thanks,
Vishal Gupta

attu · Post by **attu** » Fri Dec 07, 2012 2:29 pm

Take a look at this technote from IBM:

http://www-01.ibm.com/support/docview.w ... wg21580712

chulett · Post by **chulett** » Fri Dec 07, 2012 2:48 pm

I don't think that is applicable here. We're talking about a "start up" time of over an hour and a half, not a couple of minutes when it should normally be a handful of seconds. I'm guessing this job always has this discrepancy...

I'd start by looking into what all is encompassed in that start up time. For instance, are you creating huge lookup caches?

Mike · Post by **Mike** » Fri Dec 07, 2012 2:54 pm

I've noticed that "start up" time includes the time it takes to do a GENERATE STATISTICS in the case of a Netezza Connector as one example of something not intuitive.

Scan your job log in detail for clues about what may be included in your "start up" time.

Mike

vishu19aug · Post by **vishu19aug** » Fri Dec 07, 2012 3:00 pm

How to check what is included in 'startup time' ?

Mike · Post by **Mike** » Fri Dec 07, 2012 3:07 pm

There's not going to be anything that says "included in start up time".

Look at the timestamps for each log entry. Focus on large time gaps between consecutive log entries.

You may or may not find any clues.

Mike

Mike · Post by **Mike** » Fri Dec 07, 2012 3:09 pm

I think "start up" time really should be called "overhead" time or "time not accounted for anywhere else" time.

Mike

Mike · Post by **Mike** » Fri Dec 07, 2012 4:08 pm

Another thought... since generating stats doesn't seem to be included in production run time, I would guess that rebuilding indexes might be handled the same... and that could certainly account for a long time. Are you rebuilding indexes?

Mike

chulett · Post by **chulett** » Fri Dec 07, 2012 5:17 pm

I'm sure that Ray will be along eventually to school us but as Mike mentioned it seems to me it would be worthwhile to check the log timestamps to see what all of the activities were that took place during that initial 1:39:42 of run time. Unless we're in that "not accounted for elsewhere" zone and start up time also includes wind down time... or the times when rows are not flowing through the job?

ray.wurlod · Post by **ray.wurlod** » Fri Dec 07, 2012 5:52 pm

You can get more information about the startup by setting the environment variable APT_STARTUP_STATUS.
Startup time includes all the work done by the conductor up till the time that player processes begin execution. This includes:

verifying licensing
composing the score from the generated OSH and configuration file
sending requests to start section leader process on each node
waiting for section leaders to respond that they've started
verifying licensing (= existence of parallel engine) on each node
distributing the score to the section leaders

The environment variable setting will log events for most of these.

And, yes, anything more than a few seconds should be cause for concern.

SURA · Post by **SURA** » Sun Dec 09, 2012 4:10 pm

Hi Ray

Thanks for that. I do have the same issue in my project and let me track it.

thanks Vishal Gupta for the question you raised.