Restartability mechanisms

A forum for discussing DataStage<sup>®</sup> basics. If you're not sure where your question goes, start here.

Moderators: chulett, rschirm, roy

pandeesh
Premium Member
Premium Member
Posts: 1399
Joined: Sun Oct 24, 2010 5:15 am
Location: CHENNAI, TAMIL NADU

Restartability mechanisms

Post by pandeesh »

Hi,

May i know what are various options available for bringing restartability in a server/parallel job?

Something i know are

1)checkpoint in sequencer

2)using HashJobInfo file and routine.

Which one is the efficient one and whats the difference between those?

Thanks,
pandeeswaran
zulfi123786
Premium Member
Premium Member
Posts: 730
Joined: Tue Nov 04, 2008 10:14 am
Location: Bangalore

Re: Restartability mechanisms

Post by zulfi123786 »

pandeesh wrote:2)using HashJobInfo file and routine.
Could you explain about HashJobInfo file and the routine you are taking about?
pandeesh
Premium Member
Premium Member
Posts: 1399
Joined: Sun Oct 24, 2010 5:15 am
Location: CHENNAI, TAMIL NADU

Post by pandeesh »

Actualy we are using a hsh file called Hash JObInfo in which the status of each job ll be recorded.

Before the next run of the batch job, it will open that hash file find the status.

if it finds the last job is aborted , it llreset and continue form there.

Or if the last run is fine. it ll start the jobs from beginning.

It'll be usually acomplished by BASIC code in the batch jobs.

let me know if you need any more info
pandeeswaran
zulfi123786
Premium Member
Premium Member
Posts: 730
Joined: Tue Nov 04, 2008 10:14 am
Location: Bangalore

Post by zulfi123786 »

The Restart ability you are talking about as what is understand is resetting the job on failure for the next run

If I am right then making the sequencer a checkpointed one alone doesnot make the jobs within it restartable unless you use the option "Reset if required then run". Checkpointing a sequencer makes the seq restartable and if something goes wrong you need to fix that part and compile it then rerun the seq.

And the Hashfile logic you people are using is just one way of doing it where you reset the job if it has failed.

Both ways at the back end the same thing happens "Resetting the job", you do it from the director, unix command, Basic function its up to you.
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

pandeesh wrote:if it finds the last job is aborted , it llreset and continue form there. Or if the last run is fine. it ll start the jobs from beginning.
I'm curious what the difference is between the two other than the Reset... between 'from there' and 'from beginning'.
-craig

"You can never have too many knives" -- Logan Nine Fingers
pandeesh
Premium Member
Premium Member
Posts: 1399
Joined: Sun Oct 24, 2010 5:15 am
Location: CHENNAI, TAMIL NADU

Post by pandeesh »

Actually, if any intermediate job got aborted , the hash file wont be cleared and if we triggered the job, it l run from that aborted job.

If there is no job aborts and evrythingis fininshed fine, finally we will clear that hash file, so that the next run starts from the beginning.

Am i clear?
pandeeswaran
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Sure... in other words, nothing really at the individual job level, which is specifically what I was wondering. You have coded up your own version of the 'Checkpointing' mechanism available in Sequence jobs, it would seem.
-craig

"You can never have too many knives" -- Logan Nine Fingers
pandeesh
Premium Member
Premium Member
Posts: 1399
Joined: Sun Oct 24, 2010 5:15 am
Location: CHENNAI, TAMIL NADU

Post by pandeesh »

So, which one is the efficient method for restartability?
pandeeswaran
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

They sound fairly... equivalent.
-craig

"You can never have too many knives" -- Logan Nine Fingers
pandeesh
Premium Member
Premium Member
Posts: 1399
Joined: Sun Oct 24, 2010 5:15 am
Location: CHENNAI, TAMIL NADU

Post by pandeesh »

if we are using a sequencer and the script which is created using the run model(using ln -s)

in this case, if we trigger the sequencer using that script, it ll reset the sequencer and start the jobs from the beginning..

Is there any way to overcome this?

Is it possible to disable the reset feature in that script(which is created by using DS run model).

Actually i am not able to edit the script. i have seen there is a function inside the script for reset.

One way i can think is ,we have to use our own script using DSJOB..

But if we use our own script ,is it possible to read the parameters from the ini file?(Without giving all the parameters in the script itself?)

Thanks
pandeeswaran
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

pandeesh wrote:Is there any way to overcome this?
Be more selective in your resetting. Make it conditional on the job type and if in a 'Restartable' status.
pandeesh also wrote:But if we use our own script ,is it possible to read the parameters from the ini file?(Without giving all the parameters in the script itself?)
All things are possible if you roll your own. A script can certainly read the file and build the appropriate list of "-param" options for the dsjob command. Parameter Sets with value files in the 8.x release can simplify this.
-craig

"You can never have too many knives" -- Logan Nine Fingers
pandeesh
Premium Member
Premium Member
Posts: 1399
Joined: Sun Oct 24, 2010 5:15 am
Location: CHENNAI, TAMIL NADU

Post by pandeesh »

Parameter Sets with value files in the 8.x release can simplify this.
Can you explain this little more?

Is there any way to tweak the script which is generated using run model?
pandeeswaran
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

pandeesh wrote:Can you explain this little more?
Plenty of information out there. For instance here.
pandeesh also wrote:Is there any way to tweak the script which is generated using run model?
Don't see why not.
-craig

"You can never have too many knives" -- Logan Nine Fingers
pandeesh
Premium Member
Premium Member
Posts: 1399
Joined: Sun Oct 24, 2010 5:15 am
Location: CHENNAI, TAMIL NADU

Post by pandeesh »

Does the below syntax is valid in D.S 7.5.2?

dsjob -run -param paramsetname=valuesfilename projectname jobname

If not whats the way to implement that in 7.5.2?

i am asking about how to pass the parameters from some other file in the

option -param in DSJOB?

Thanks
pandeeswaran
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Parameter sets were introduced in the 8.x release, so no. You would have to code something to read the file and loop to build each -param name/value pair.
-craig

"You can never have too many knives" -- Logan Nine Fingers
Post Reply