Datastage job as service

[Nagaraj]

Okay cool will do, thanks a ton

[dlai74]

Hi, I am using v7.5.x, does anybody know if the web services packs is free or not? If it is free, where can I download this pack?

Also, I am asking this not because I MUST use web service but I am actually thinking of turning a batch ETL job to serve real time request from some online application; I do this because I think it is a beauty to reuse the logic/function that is written already for the batch ETL job. In order to do this, I can either modify the interface of this job to receive WS request OR if web service packs is not free, I can implement a FIFO queue logic myself simply receive the online request through database table (the sender writes the request data in the database table, then the DataStage job picks up the input data from the database). However, if I don't want the online response time to span multiple seconds I need to find a way to bypass the DataStage job startup/initiating time (meaning I might not want to go the direction of writing ksh infinite loop to trigger the DataStage job for polling the incoming data in the database); instead I am thinking of implementing some custom stage or some sort to make the job never ending and act like a service demon to be event triggered by the incoming data in the DB. Server job or Parallel job either acceptable for me, any expert has idea/solution to share or recommendation about this event trigger methodology?

[eostic]

Hi dlai74...

Very eloquent description of what RTI does for 7.5 and what ISD does for v8! RTI, which has been in production since 2002, does exactly this --- gives you the opportunity to re-use logic created for batch purposes as a service for online applications...

No sense re-inventing the wheel. RTI/ISD takes care of the start-up issues, etc., using a variety of job patterns for deployment.

Look thru the forum here, in my blog noted below, and/or thru the documentation if you have it.

Web Services Pack is the opposite tooling --- it merely lets DataStage reach out and "invoke" some "other" web service.

Ernie

[dlai74]

Hi dlai74...

Very eloquent description of what RTI does for 7.5 and what ISD does for v8! RTI, which has been in production since 2002, does exactly this --- gives you the opportunity to re-use logic created for batch purposes as a service for online applications...

No sense re-inventing the wheel. RTI/ISD takes care of the start-up issues, etc., using a variety of job patterns for deployment.

Look thru the forum here, in my blog noted below, and/or thru the documentation if you have it.

Web Services Pack is the opposite tooling --- it merely lets DataStage reach out and "invoke" some "other" web service.

Ernie

Thanks for your kind reply. Actually I had trial the RTI back in 2006, I agree and understand your enthusiasm on RTI (I am using v7.5.x). Reading through your reply, I think you are implying that if I am not paying extra money, I might not have a way to enjoy the startup/initiation time elimination, or you think there is still a way but I need to develop it or find it out on my own? Just trying to see anybody has done this before, if so, it is not worth for me to do the technical research/POC on implementing the startup/initiation bypassing mechanism.

[eostic]

Not worth it......too many things to consider for maintaining the "always on" scenario required to bypass the start up time, not to mention all of the underlying work that is done to handle SOAP envelopes, produce WSDL, etc........ An alternative though, would be to simulate some of what ISD provides by using MQ Series. Out of the box, the MQ Series stage as a source is ready for "always on" functionality. Of course you still need MQ independent of DataStage.

You also would have to hand-craft things like the (potential) launching of multiple readers for higher volumes of traffic, and anything beyond pure MQ (SOAP, etc.) would be entirely on your own, where you might wrap your own MQ client (who sends messages into DataStage) with various Web Service host capabilities, and your own WSDL generation, etc.

Ernie