DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
Author Message
pandeesh



Group memberships:
Premium Members, Heartland Usergroup

Joined: 24 Oct 2010
Posts: 1399
Location: CHENNAI, TAMIL NADU
Points: 9855

Post Posted: Sun Apr 10, 2011 8:54 am Reply with quote    Back to top    

DataStage® Release: 7x
Job Type: Server
OS: Unix
Additional info: Datastage 7.5.1
Hi,

May i know what are various options available for bringing restartability in a server/parallel job?

Something i know are

1)checkpoint in sequencer

2)using HashJobInfo file and routine.

Which one is the efficient one and whats the difference between those?

Thanks,

_________________
pandeeswaran
zulfi123786



Group memberships:
Premium Members

Joined: 04 Nov 2008
Posts: 730
Location: Bangalore
Points: 6137

Post Posted: Mon Apr 11, 2011 1:07 am Reply with quote    Back to top    

pandeesh wrote:
2)using HashJobInfo file and routine.

Could you explain about HashJobInfo file and the routine you are taking about?
Rate this response:  
Not yet rated
pandeesh



Group memberships:
Premium Members, Heartland Usergroup

Joined: 24 Oct 2010
Posts: 1399
Location: CHENNAI, TAMIL NADU
Points: 9855

Post Posted: Mon Apr 11, 2011 2:32 am Reply with quote    Back to top    

Actualy we are using a hsh file called Hash JObInfo in which the status of each job ll be recorded.

Before the next run of the batch job, it will open that hash file find the status.

if it finds the last job is aborted , it llreset and continue form there.

Or if the last run is fine. it ll start the jobs from beginning.

It'll be usually acomplished by BASIC code in the batch jobs.

let me know if you need any more info

_________________
pandeeswaran
Rate this response:  
Not yet rated
zulfi123786



Group memberships:
Premium Members

Joined: 04 Nov 2008
Posts: 730
Location: Bangalore
Points: 6137

Post Posted: Mon Apr 11, 2011 3:05 am Reply with quote    Back to top    

The Restart ability you are talking about as what is understand is resetting the job on failure for the next run

If I am right then making the sequencer a checkpointed one alone doesnot make the jobs within it restartable unless you use the option "Reset if required then run". Checkpointing a sequencer makes the seq restartable and if something goes wrong you need to fix that part and compile it then rerun the seq.

And the Hashfile logic you people are using is just one way of doing it where you reset the job if it has failed.

Both ways at the back end the same thing happens "Resetting the job", you do it from the director, unix command, Basic function its up to you.
Rate this response:  
Not yet rated
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 43085
Location: Denver, CO
Points: 222463

Post Posted: Mon Apr 11, 2011 6:13 am Reply with quote    Back to top    

pandeesh wrote:
if it finds the last job is aborted , it llreset and continue form there. Or if the last run is fine. it ll start the jobs from beginning.

I'm curious what the difference is between the two other than the Reset... between 'from there' and 'from beginning'.

_________________
-craig

"You can never have too many knives" -- Logan Nine Fingers
Rate this response:  
Not yet rated
pandeesh



Group memberships:
Premium Members, Heartland Usergroup

Joined: 24 Oct 2010
Posts: 1399
Location: CHENNAI, TAMIL NADU
Points: 9855

Post Posted: Mon Apr 11, 2011 6:20 am Reply with quote    Back to top    

Actually, if any intermediate job got aborted , the hash file wont be cleared and if we triggered the job, it l run from that aborted job.

If there is no job aborts and evrythingis fininshed fine, finally we will clear that hash file, so that the next run starts from the beginning.

Am i clear?

_________________
pandeeswaran
Rate this response:  
Not yet rated
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 43085
Location: Denver, CO
Points: 222463

Post Posted: Mon Apr 11, 2011 6:23 am Reply with quote    Back to top    

Sure... in other words, nothing really at the individual job level, which is specifically what I was wondering. You have coded up your own version of the 'Checkpointing' mechanism available in Sequence jobs, it would seem.

_________________
-craig

"You can never have too many knives" -- Logan Nine Fingers
Rate this response:  
Not yet rated
pandeesh



Group memberships:
Premium Members, Heartland Usergroup

Joined: 24 Oct 2010
Posts: 1399
Location: CHENNAI, TAMIL NADU
Points: 9855

Post Posted: Mon Apr 11, 2011 7:24 am Reply with quote    Back to top    

So, which one is the efficient method for restartability?

_________________
pandeeswaran
Rate this response:  
Not yet rated
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 43085
Location: Denver, CO
Points: 222463

Post Posted: Mon Apr 11, 2011 7:28 am Reply with quote    Back to top    

They sound fairly... equivalent.

_________________
-craig

"You can never have too many knives" -- Logan Nine Fingers
Rate this response:  
Not yet rated
pandeesh



Group memberships:
Premium Members, Heartland Usergroup

Joined: 24 Oct 2010
Posts: 1399
Location: CHENNAI, TAMIL NADU
Points: 9855

Post Posted: Mon Apr 11, 2011 7:39 am Reply with quote    Back to top    

if we are using a sequencer and the script which is created using the run model(using ln -s)

in this case, if we trigger the sequencer using that script, it ll reset the sequencer and start the jobs from the beginning..

Is there any way to overcome this?

Is it possible to disable the reset feature in that script(which is created by using DS run model).

Actually i am not able to edit the script. i have seen there is a function inside the script for reset.

One way i can think is ,we have to use our own script using DSJOB..

But if we use our own script ,is it possible to read the parameters from the ini file?(Without giving all the parameters in the script itself?)

Thanks

_________________
pandeeswaran
Rate this response:  
Not yet rated
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 43085
Location: Denver, CO
Points: 222463

Post Posted: Mon Apr 11, 2011 7:54 am Reply with quote    Back to top    

pandeesh wrote:
Is there any way to overcome this?

Be more selective in your resetting. Make it conditional on the job type and if in a 'Restartable' status.

pandeesh also wrote:
But if we use our own script ,is it possible to read the parameters from the ini file?(Without giving all the parameters in the script itself?)

All things are possible if you roll your own. A script can certainly read the file and build the appropriate list of "-param" options for the dsjob command. Parameter Sets with value files in the 8.x release can simplify this.

_________________
-craig

"You can never have too many knives" -- Logan Nine Fingers
Rate this response:  
Not yet rated
pandeesh



Group memberships:
Premium Members, Heartland Usergroup

Joined: 24 Oct 2010
Posts: 1399
Location: CHENNAI, TAMIL NADU
Points: 9855

Post Posted: Mon Apr 11, 2011 8:08 am Reply with quote    Back to top    

Quote:
Parameter Sets with value files in the 8.x release can simplify this.


Can you explain this little more?

Is there any way to tweak the script which is generated using run model?

_________________
pandeeswaran
Rate this response:  
Not yet rated
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 43085
Location: Denver, CO
Points: 222463

Post Posted: Mon Apr 11, 2011 8:25 am Reply with quote    Back to top    

pandeesh wrote:
Can you explain this little more?

Plenty of information out there. For instance here.

pandeesh also wrote:
Is there any way to tweak the script which is generated using run model?

Don't see why not.

_________________
-craig

"You can never have too many knives" -- Logan Nine Fingers
Rate this response:  
Not yet rated
pandeesh



Group memberships:
Premium Members, Heartland Usergroup

Joined: 24 Oct 2010
Posts: 1399
Location: CHENNAI, TAMIL NADU
Points: 9855

Post Posted: Mon Apr 11, 2011 10:05 am Reply with quote    Back to top    

Does the below syntax is valid in D.S 7.5.2?

dsjob -run -param paramsetname=valuesfilename projectname jobname

If not whats the way to implement that in 7.5.2?

i am asking about how to pass the parameters from some other file in the

option -param in DSJOB?

Thanks

_________________
pandeeswaran
Rate this response:  
Not yet rated
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 43085
Location: Denver, CO
Points: 222463

Post Posted: Mon Apr 11, 2011 10:21 am Reply with quote    Back to top    

Parameter sets were introduced in the 8.x release, so no. You would have to code something to read the file and loop to build each -param name/value pair.

_________________
-craig

"You can never have too many knives" -- Logan Nine Fingers
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours