Executable file of a DS Job

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

ag_ram wrote:This Job executable file is located at the Project Directory at version DS 7.5

RT_CONFIGnnn - A Run Engine Executable file, nnnn --> Job Number
That is completely incorrect. RT_CONFIGnnn contains only data about the job configuration - data that are used by DSD.RUN and DSD.StageRun.

There is no job executable file per se, unless you count DSD.RUN (the BASIC program used to execute server jobs) and osh (the shell used to execute parallel jobs).
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
VCInDSX
Premium Member
Premium Member
Posts: 223
Joined: Fri Apr 13, 2007 10:02 am
Location: US

Post by VCInDSX »

Good one to learn more about the runtime.
One question, if I may use this thread. ...

So if a job is compiled in a particular configuration and is exported into a DSX file using the "Export Design with Executable" option and gets deployed on another system for execution, it would not really use that systems resources until the job is compiled in that new configuration, correct?

Thanks,
Victor
-V
ArndW
Participant
Posts: 16318
Joined: Tue Nov 16, 2004 9:08 am
Location: Germany
Contact:

Post by ArndW »

V - What do you mean by "not use the system's resources"? Shared containers are loaded into a job's executable at compile time. In server jobs, the code for routines is not loaded in at compile time, links are created and whatever version is locally catalogued at runtime is executed.
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

If you know how to look RT_CONFIGnnn contains far more than "names...and nothing else". It's just that there are no metadata for retrieving the contents of these records. Structure of these records is, however, described in RT_CONFIG.H.

The importer looks after creation of RT_BPnnn.O, RT_CONFIGnnn, RT_LOGnnn, RT_STATUSnnn, DS_TEMPnnn, RT_SCnnn, etc. The "DSBINARY" record in an export file contains only the pcode from RT_BPnnn.O.

The "executable job file" text indicates that RT_CONFIGnnn hashed file is used as the file (of data) for the executable (usually DSD.RUN for server jobs). "Cannot open" does not necessarily refer to "not found".
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

The thread has strayed from its original topic.

DataStage does NOT generate an executable file.

DataStage parallel jobs generate an osh script. When the job is run, a new osh script (called the "score") is written, taking into account the currently selected configuration. The score is executed within an Orchestrate shell - not by the operating system.

The score never gets created as a file, unless you direct DataStage to save it to disk. This is the basis of my contention that there is no executable file. The score is distributed to section leader processes by the conductor process, as a bytestream, not as a file.

The score is a set of instructions interpreted (yes, interpreted) by the osh excutable (osh.exe if on a Windows platform). osh is the only executable pertinent to parallel jobs.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
ag_ram
Premium Member
Premium Member
Posts: 524
Joined: Wed Feb 28, 2007 3:51 am

Post by ag_ram »

ray.wurlod wrote: If you know how to look RT_CONFIGnnn contains far more than "names...and nothing else". It's just that there are no metadata for retrieving the contents of these records. Structure of these records is, however, described in RT_CONFIG.H.
Runtime program header file DSD.RTCONFIG.H.
ray.wurlod wrote:The "DSBINARY" record in an export file contains only the pcode from RT_BPnnn.O.


Of course.

Code: Select all

DSEXECJOB  Record --> Contains below Records
   DSRECORD Record --> Job config file RT_CONFIGnnn entries
   DSBPBINARY Record --> Contents of Compiled Basic Program RT_BPnnn.O
   DSBPSOURCE Record --> Contents of Basic Program RT_BPnnn(if needed)

Unluckily I dont find the DSBINARY record. you might mean the second one.
ray.wurlod wrote:"Cannot open" does not necessarily refer to "not found".
yes it can happen if the file is collapsed also. I manually deleted that file and saw the Error message. I suppose you should have noted what is inside the quote as well.
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

DSBINARY is a structure in a DSX export file.

Please be less hasty in leaping to conclusions.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
devnull
Premium Member
Premium Member
Posts: 37
Joined: Wed Mar 29, 2006 11:17 am
Location: Minneapolis, MN USA

Post by devnull »

ray.wurlod wrote:The thread has strayed from its original topic.

DataStage does NOT generate an executable file.

DataStage parallel jobs generate an osh script. When the job is run, a new osh script (called the "score") is written, taking into account the currently selected configuration. The score is executed within an Orchestrate shell - not by the operating system.

The score never gets created as a file, unless you direct DataStage to save it to disk. This is the basis of my contention that there is no executable file. The score is distributed to section leader processes by the conductor process, as a bytestream, not as a file.

The score is a set of instructions interpreted (yes, interpreted) by the osh excutable (osh.exe if on a Windows platform). osh is the only executable pertinent to parallel jobs.
How do you direct DataStage to do this? If I do a find command in a project directory on '*.osh' , I see an RT_SCNN/OshScript.osh file for every job I've compiled in DataStage. I think the term 'executable file' is confusing here because while these files do not have any execute permissions turned on (allowing you to run them at the command line as-is) I can submit many of them to the osh interpreter myself (with my environment set up properly, of course. This is not all that dissimilar to shell scripts like the configure script use with GNU tarball distributions or Python or Perl scripts that must be executed with the interpreter commands. Is it?
Michael Mabin
Minneapolis, MN USA
d3vvnull@com.gmail
(Reverse com and gmail to send email)
ag_ram
Premium Member
Premium Member
Posts: 524
Joined: Wed Feb 28, 2007 3:51 am

Post by ag_ram »

ray.wurlod wrote:DSBINARY is a structure in a DSX export file.

Please be less hasty in leaping to conclusions.
I have no luck to find the term in the Job Executable DSX file and/or Job Design DSX file.

Could please let me know, which type DSX file has this structure if it is different from DSBPBINARY and what is the content inside?
ag_ram
Premium Member
Premium Member
Posts: 524
Joined: Wed Feb 28, 2007 3:51 am

Post by ag_ram »

ray.wurlod wrote:DSBINARY is a structure in a DSX export file.

Please be less hasty in leaping to conclusions.
I have no luck to find the term in the Job Executable DSX file and/or Job Design DSX file.

Could please let me know, which type DSX file has this structure if it is different from DSBPBINARY and what is the content inside?
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

DSBINARY relates to version 5.2 and earlier. DSBPBINARY was introduced (as a direct replacement) in version 6.0.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
ag_ram
Premium Member
Premium Member
Posts: 524
Joined: Wed Feb 28, 2007 3:51 am

Post by ag_ram »

ray.wurlod wrote:DSBINARY relates to version 5.2 and earlier. DSBPBINARY was introduced (as a direct replacement) in version 6.0.
Thanks ray.wurlod.
Post Reply