Job Sequencer

[RAI ROUSES]

Hi
We design our Job sequencer to have at least 4 jobs running at the same time:
A -> B -> C -> D
E -> F -> G -> H
I -> J -> L -> M
N -> O -> P -> Q

The first 4 jobs start at the same time, and after each jobs has finished it will start the next job, each job have only one dependence it was the previous job, but we verify that the second line of jobs ( B, F, J and O) only start when the first jobs (A, E, I, N) has all finished.

Any idea to this strange problem?
What we can do ?

Rai

[kumar_s]

Hi Rai,

Could you make you question clear.

B, F, J and O) only start when the first jobs (A, E, I, N) has all finished.

Is this not expected? If so, do you expect B, F, J and O to run parallel to A, E, I, N)?

[ray.wurlod]

IF your job sequence (note, not "sequencer") is as you describe, with no dependency between the four streams, then four simultaneous streams of jobs should run.

Please check the start times of the 16 jobs in Director and verify your assertion of consecutive processing.

Is there any constraint on the number of startup processes, such as APT_STARTUP_CONCURRENCY? The default value of this environment variable is 5, and will limit the number of simultaneous processes being started, which might appear to be what you're seeing. But, if A runs for long enough, you should see E start when A is under way.

The obvious other question to ask is whether your server system is overloaded.

[kumar_s]

The first 4 jobs start at the same time, and after each jobs has finished it will start the next job, each job have only one dependence it was the previous job, but we verify that the second line of jobs ( B, F, J and O) only start when the first jobs (A, E, I, N) has all finished.

Not sure how Ray can get the actual issue.
What I could understand is, ( B, F, J and O) only start when the first jobs (A, E, I, N) has all finished. But if the Trigger given as Finished Ok will do the same right.

[ray.wurlod]

Monitor your server machine to see whether its resource consumption, particularly CPU and memory, is up near 100%

[kumar_s]

Now I got what Ray mean to say. Hi point is on max number of concurrent process limit. But what I could guess is, what ever you see is from the log, that job B is waiting for the other to finish. May I know what makes to conclude that its waiting till the others getting finishes?
Perhaps why can you club all A -> B -> C -> D in once sequence and like wise individual 4 sequence for each stream and call it in a sequence.

[ray.wurlod]

That's why I asked you to report all 16 job start times. All 16 finish times would be useful also.

[kumar_s]

Smart way to start the diagnosis.