Problem with execute command activity in sequence job.

A forum for discussing DataStage<sup>®</sup> basics. If you're not sure where your question goes, start here.

Moderators: chulett, rschirm, roy

Post Reply
Pranita
Participant
Posts: 3
Joined: Thu Jul 19, 2007 4:32 am

Problem with execute command activity in sequence job.

Post by Pranita »

Background
There is a limit to the number of jobs that can run on ControlM (ControlM is a scheduling tool that invokes the DS jobs through scripts.) server, Due to this we are trying to reduce the no. of ControlM jobs. There are 220 jobs in the application. Creating 220 scripts (one script per job) is not feasible as these jobs will run for 200+ site in which case we will have approx. 200*220=44000 ControlM jobs on ControlM, which exceeds the ControlM limit.

Problem Description

We have created a generic script, that invokes the job passed to the script as a parameter. The parameter required for the job are identified and assigned values at run time.

The plan is use a sequence job that will call this generic script once for each job in the sequence, for invoking the script we have used the execute command activity, but the problem with this is the execute command activity runs sequentially even though these stages are placed in parallel in sequencer job.

One work around this problem is to call a dummy job from a job activity in the sequence job and inturn this dummy job invokes the generic script. Since the job activity stage runs in parallel it indirectly executes the scripts in parallel. But again here the problem here is the director log for this dummy job would be huge as the same dummy job would be called for 220 jobs in the appliaction multiplied by the number of sites (220*200)

Note:- We cant call the jobs directly from job acctivities since there as some scripts which needs to be called after execution of each job and also we cannot call these scripts in "after job subroutines" as its against the best practices defined for project.

Please let me know if you have any solutions.
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

What is it that makes such restrictions "best" practices? Who said?
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
balajisr
Charter Member
Charter Member
Posts: 785
Joined: Thu Jul 28, 2005 8:58 am

Post by balajisr »

We cant call the jobs directly from job acctivities since there as some scripts which needs to be called after execution of each job and also we cannot call these scripts in "after job subroutines" as its against the best practices defined for project.

If i understand your problem correctly, Can't you call after job script like this in job sequence. :?

Code: Select all

 Job Activity -------OK Trigger ------->Exec Command 
Pranita
Participant
Posts: 3
Joined: Thu Jul 19, 2007 4:32 am

Post by Pranita »

ray.wurlod wrote:What is it that makes such restrictions "best" practices? Who said? ...
Thanks for your comments.

By best practices, I meant that, in my project we are advised against the use of After-job subroutines.


balajisr- We tried using the Execute command activity after the job activity. The problem here again is that, the job activities will run in parallel, but the exec command activities will still run sequentially even though designed for parallel execution
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

WHO advised and, more importantly, for what reason?

After-job subroutines are a very useful part of the product. I put an annotation on the canvas to alert future developers to the use of before/after subroutines, so that they (the developers) are not caught unawares when the subroutine does whatever it does.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Pranita
Participant
Posts: 3
Joined: Thu Jul 19, 2007 4:32 am

Post by Pranita »

ray.wurlod wrote:WHO advised and, more importantly, for what reason?

After-job subroutines are a very useful part of the product. I put an annotation on the canvas to alert future developers to the use of before/a ...
Hi,

I would like to reframe my query to "How can one run EXECUTE COMMAND activities parallely in a sequence job? We've noticed that although we put them in parallel, they actually run sequentially"
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Can you post (or draw) an example where you have put the Execute Command stages 'in parallel'?
-craig

"You can never have too many knives" -- Logan Nine Fingers
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

Job sequences only ever run sequentially, and only ever on the DataStage server machine itself, not on any other node in an MPP or grid environment.

That's because a job sequence is really a special case of a server job.

You can, of course, create parallel streams of processing in a job sequence, but that's designed-in parallelism, not automatic parallelism.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
atulgoel
Participant
Posts: 84
Joined: Tue Feb 03, 2009 1:09 am
Location: Bangalore, India

Post by atulgoel »

Hi Pranita, Do you find the solution for your problem. I am also having the same issue.
Atul
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Rather than ask someone who hasn't posted here in ten years it would be better to start a new topic with the gory details of your version of this issue and we'll go from there.

Thanks.
-craig

"You can never have too many knives" -- Logan Nine Fingers
Post Reply