exporting job designs with executables

vamsi.4a6 · Post by **vamsi.4a6** » Thu Apr 17, 2014 1:05 am

Export job designs with executables: Both the 'design time' information (what you see in the Designer) and the 'run time code' are exported. I can understand what is meant by design information but no idea about run time code. Could anybody elaborate on this? Why is Export job designs with executables is not suitable for heterogeneous environments?

prasson_ibm · Post by **prasson_ibm** » Thu Apr 17, 2014 2:03 am

Hi,

Export job designs with executables: This option will export your Job design and binary executable files. It is useful for the environments where c++ compiler is not available. Sometimes organizations don't buy the compiler in all environments (like production) due to licensing costs.

vamsi.4a6 · Post by **vamsi.4a6** » Thu Apr 17, 2014 3:04 am

@prasson_ibm

Thanks for the input. Where can I find the binary executable files for a particular job and also what it will contain for a particular job?

For example for a Job Design information includes stage information, target information, link information, transformation details etc.

prasson_ibm · Post by **prasson_ibm** » Thu Apr 17, 2014 6:57 am

Hi,
When you export your job with above option,executable binaries will be included in job.
E.g.

Code: Select all

BEGIN DSBPBINARY
      Identifier "V0S46_Mditof_001_MainJob_TSFM.o"
      B1 "01F70005534BF288000000000001E68A0078300200000911010B0001000000000000000100000240"
      B41 "0000000110000EF00000000110002D10000000010002000200040003000700045245000000000000"
      B81 "000000000000DCB00000000000002560000000000000001CFFFFFFFFFFFFFFFF0000000000000000"
      B121 "0000000000000000000000000000000000000000000000002E746578740000000000000100000240"
      B161 "0000000100000240000000000000DCB0000000000000024000000000000186900000000000000000"
      B201 "000004720000000000000020000000002E646174610000000000000110000EF00000000110000EF0"
      B241 "0000000000002560000000000000DEF0000000000001C4CC00000000000000000000026900000000"
      B281 "00000040000000002E6273730000000000000001100034500000000110003450000000000000001C"

zulfi123786 · Post by **zulfi123786** » Thu Apr 17, 2014 7:33 am

Some components of the job like the transformer, build op's, parallel routines etc are compiled to created object files which contain binaries and if you are migrating your job to an environmnt where you dont have a compiler there is no way to get your job to run unless you have those binaries.

Binaries are not a part of your job design, its a thing which gets created when you compile your designs.

Exporting them with executables embeds those binaries along with the designs.

vamsi.4a6 · Post by **vamsi.4a6** » Thu Apr 17, 2014 11:47 am

Thanks all for the input.

suppose I exported a job with Export job designs with executable in dev environment and imported a job in test environment .will the job will run if i run the job if i Used x compiler in dev environment and y compiler in test environment?

As per my knowledge it will run successfully since run time information is available even though i am using y compiler in test environment but when i google it i came to know it will fail.

any thoughts why it will fail?

zulfi123786 · Post by **zulfi123786** » Fri Apr 18, 2014 12:02 am

It might not as DataStage jobs dont have actual executables (unlike c/c++ which create executables that can be run on any like machines). When you import the job into test along with the output of compilation (with executable option of export) would still have to go through the linking phase which the compiler (linkers come along with compilers) handles.

ray.wurlod · Post by **ray.wurlod** » Fri Apr 18, 2014 6:08 pm

DataStage executable code actually runs in a virtual machine (the DataStage engine) rather than needing to be linked. The codes you see in the export file are pcode instructions for that virtual machine.

zulfi123786 · Post by **zulfi123786** » Sat Apr 19, 2014 2:50 pm

If they are pcode instructions then the executable should be platform independent meaning which if I export with executable a job from unix machine and import onto windows would that work fine ?

Thanks

chulett · Post by **chulett** » Sun Apr 20, 2014 7:43 am

Actually... yes.

IBM Analytics Champion 2009 - 2020 · Post by **asorrell** » Mon Apr 21, 2014 7:21 am

The jobs are only platform independent if the developers are careful to keep them that way.

- Use environment variables for path names (makes it easy to swap C:\windowspath to /unixpath).
- Limit embedded commands to UNIX commands supported by MKS toolkit on Windows platform
- Don't use things like sqlcmd (Windows-only SQL Server command line) that aren't available in all environments

Otherwise it can be a pain...

chulett · Post by **chulett** » Mon Apr 21, 2014 7:38 am

Of course. Figured those details were more applicable in another discussion and here we were fine with a simple 'yes'.

zulfi123786 · Post by **zulfi123786** » Mon Aug 25, 2014 7:59 am

ray.wurlod wrote:DataStage executable code actually runs in a virtual machine (the DataStage engine) rather than needing to be linked. The codes you see in the export file are pcode instructions for that virtual machine.

This seems to be like Java Virutal Machine, One more question comes to mind. When these pcode instructions are converted into native machine code instructions at run time is there any optimisation technique used like the JIT of JVM

Thanks

ray.wurlod · Post by **ray.wurlod** » Mon Aug 25, 2014 7:00 pm

They are not converted to "native machine code instructions". They are dispatched (along with any following arguments) within the DataStage engine to an appropriate function that performs the task associated with that particular opcode.

zulfi123786 · Post by **zulfi123786** » Wed Aug 27, 2014 1:02 am

Thanks for the reply Ray,

If my ignorance isn't overwhelming, Could you please elaborate on how the opcode of DS Engine is understood by the underlying processor. Guess my understanding of the below isn't correct.

Ex-If there is a sort operation in a parallel job on compilation it turns up into an orchestrate operator which is built on C++ and its compilation output would be native to the processor and it gets executed there.

Now that DSEngine works as a virtual machine and all of DataStage code gets executed there how things starting from GUI get converted into the processors opcodes and operands, guess this is different for Parallel and Server jobs.

If there is any documentation related to the above, would highly appreciate if directed there.

Thanks