Scratch Space and Resource Disk in Datastage

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
RohanSharma
Participant
Posts: 20
Joined: Sun Jan 28, 2007 10:06 pm
Location: Gurgaon

Scratch Space and Resource Disk in Datastage

Post by RohanSharma »

Hello there...
Well I have this job running at the client end and it processes real heavy gusts of data...
Now some time back the jobs fails saying
"APT_CombinedOperatorController,0: write failed: Output file full, and no more output files
write failed: Output file full, and no more output files"


Now This message says that Output file is full but this does not indicate that whether it is Datasets or Some Temporary Files being full.

Doubt arises when we see the error message in the next run

APT_CombinedOperatorController,0: Fatal Error: Tsort merger aborting: Scratch space full

(there is some sorting logic in the job)

1)Now I need to know what is the Scratch Space
2) Also when I see the Environment settings of my project I see that
something called as resource disk also being used.

Can you tell me what exactly are Scratch Disk and the Resource Disk.


Also there is an all together different temporary directory being used in Datastage in the variable TMPDIR :shock:
This TMPDIR is not specific to our project it is standard to Datastage.
and can bee seen in the administrator's general Settings Tag

So if this TMPDIR is there then what is the need of Scratch Space to be used
[If my assumption that scratch space is a temporary space used while executing a job is correct ] 8)
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

TMPDIR is used after scratch space fills.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
RohanSharma
Participant
Posts: 20
Joined: Sun Jan 28, 2007 10:06 pm
Location: Gurgaon

Post by RohanSharma »

Thanks Ray
For the information,

But then what role DISK SPACE plays??

Also I tried to run the job again...
The job is failing again saying that
"APT_CombinedOperatorController,0: write failed: Output file full, and no more output files"

But when I check the corresponding UNIX directory I see only that 15% of the total disk space in TMPDIR location is used.
So Is there any Settings in datastage which limits the usage of the TMPDIR space.

Please let me know it is very important for me to know this to get out of the quagmire I'm currently trapped in
:P
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

First off, disable operator combination so you can work out which operator is having the problem.

Resource disk is used only for Data Sets and File Sets. If it turns out that the stage having the problem is a Data Set stage, a File Set stage or a Lookup File Set stage, then you're close to a solution.

If it's any other stage then it's probably scratch disk that is filling. You have to monitor this while the job is running, not once it's finished.

TMPDIR is not used until and unless scratch disk fills.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
RohanSharma
Participant
Posts: 20
Joined: Sun Jan 28, 2007 10:06 pm
Location: Gurgaon

Post by RohanSharma »

ray.wurlod wrote:First off, disable operator combination so you can work out which operator is having the problem.

Resource disk is used only for Data Sets and File Sets. If it turns out that the stage having the ...


Ray can you please tell me how to Disable Oerator Combination and what exactly it is ??
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

Operator combination is an optimization whereby multiple adjacent parallel operators are caused to execute in the same process, provided certain conditions are met.

Disable it either using the APT_DISABLE_COMBINATION environment variable or setting the combination setting in individual stages' Advanced tab to "Don't combine".
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Post Reply