MQ Stage and Real time jobs.

Dedicated to DataStage and DataStage TX editions featuring IBM<sup>®</sup> Service-Oriented Architectures.

Moderators: chulett, rschirm

Post Reply
gsherry1
Charter Member
Charter Member
Posts: 173
Joined: Fri Jun 17, 2005 8:31 am
Location: Canada

MQ Stage and Real time jobs.

Post by gsherry1 »

Hello Forum,

I don't have so much experience with Real time ETL products. My understanding is that MQ Series is usually used in real time processing to consume messages produced a registered source.

So my questions are:

1. Does the MQ Stage perform any real time functionality. Or does it just read messages in batch when I explicitly schedule my job to run?

2. Is the MQ stage suppose/has to be run with RTI/Soa package?

3. If I do use MQ stage within the Soa package, do I get any benefit?

Your input is appreciated.

Greg
eostic
Premium Member
Premium Member
Posts: 3838
Joined: Mon Oct 17, 2005 9:34 am

Post by eostic »

This is a very old entry, but this question comes up all the time, so I figured it was worth addressing.

The MQ Series stage "listens" for new messages on a message queue, which means that the job (by default) will run forever, "waiting" for new messages to arrive. There are, of course, elegant options to terminate the job (after a certain amount of time has elapsed, a number of messages, or a special MQ Message Type is received). It's advised that you always have a way to stop the job from running.

The MQ Stage is entirely separate from RTI..... RTI is more generic, also allowing jobs to be "always on," but with a lot more infrastructure to support message queues, SOAP input, direct Java (EJB) invocation, and things like load balancing and failover.

They are certainly similar, but with MQ it's "just a stage," so you have to be responsible for all the load balancing and environment set-up, if desired.

Additionally, with MQ you have to be careful about using an "always on" job --- and watch out for blocking stages such as the Aggregator. With RTI, DataStage understands when a single "message" is complete, allowing you to utilize the Aggregator and other such stages.

Ernie
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

eostic wrote:With RTI, DataStage understands when a single "message" is complete, allowing you to utilize the Aggregator and other such stages.
Interesting, not something I was really aware of. I was under the impression that stages that do their dirty work when the input stream ends could be... less than functional in an RTI job. Somewhat like the OCI issue you helped us work out when Julio was here. :wink:

Thanks for the info, Ernie! Have to build some test cases and play when I have some spare time.
-craig

"You can never have too many knives" -- Logan Nine Fingers
Post Reply