MQ to MQ - Realtime batch processing

bman · Post by **bman** » Thu Mar 13, 2008 3:06 pm

Hi,

Is it possible to have an MQ to MQ job which runs as an always on job and at the same time process messages in batch ?

What I mean is, run the MQ IN stage with wait time -1, message count -1 and Record count = 100.

and The job flow as,

MQ -> Transformer ->MQ

Job runs fine untill the last batch. If the last batch has less than 100 messages then that batch is never getting processed. Any way to tell MQ to start processing the last batch, keeping the job always on ?

eostic · Post by **eostic** » Thu Mar 13, 2008 7:05 pm

....how do you re-start the job now, after it reaches 100 messages? ...and if the messages come "in batches"....then why would a particular wait time (several minutes, or even longer perhaps) not be sufficient?

Ernie

bman · Post by **bman** » Thu Mar 13, 2008 9:04 pm

Hi Ernie,

Actually messages are not coming in batch but one at a time. And the publisher publishes more than a million records in avery short span. My job was picking up messages one at atime but this is causing perfomance issue and I thought of changing the transaction count to 100. But I am really doubtfull if this can be achieved ? I cant find a way to tell DS, "okay now process available messages, you have waited enough time" when it runs in always on mode.

One other way I am thinking is to keep the job in an infiniteloop using loop sequencer and keep the job wait time for 10 minute ?

eostic · Post by **eostic** » Thu Mar 13, 2008 11:13 pm

I think I'd like to explore with you more why it's having a performance issue......unless there are blocking stages in there...... otherwise, in "always on" mode it should be fine.

Cycling it via job sequencer is one possibility though, for sure. Run it thru 100 or higher number, and/or a particular time-out, and then re-start it. Just be sure you have an ultimate interrupt so the loop is easy to manage.

Ernie

eostic · Post by **eostic** » Thu Mar 13, 2008 11:14 pm

I think I'd like to explore with you more why it's having a performance issue......unless there are blocking stages in there...... otherwise, in "always on" mode it should be fine.

Cycling it via job sequencer is one possibility though, for sure. Run it thru 100 or higher number, and/or a particular time-out, and then re-start it. Just be sure you have an ultimate interrupt so the loop is easy to manage.

Ernie

bman · Post by **bman** » Fri Mar 14, 2008 12:15 am

I said perfomance issue because the MQ connector stage reads messages from queue at very slow rate. The rate is like 10 messages per second and with this rate processing 1 million records take too much of time ? This is why I started thinking of increasing the record count ?

I cant see any other bottleneck in the job as it has only 2 MQ stages and an XML transformer stage which uses an xslt mapping. If i keep record count as one that means each message is read from the queue transfomred and then commited to the destination queue and I think the commititng for each message eats up all the time

ArndW · Post by **ArndW** » Fri Mar 14, 2008 3:11 am

Wouldn't this be a case for "wave processing"?

eostic · Post by **eostic** » Fri Mar 14, 2008 6:19 am

...could be Arnd...lots of things going on. Sounds like end-of-wave may already be utilized...certainly it would be necessary with XMLOutput --- it's likely that XMLTransformer with XSLT needs it too. Plus, running xslt is not exactly a screamer itself --- although I have question --- is the xslt taking one row in and then outputting one row? Curious what motivated the choice of xslt, especially if there is only one row (where row aggregation isn't needed, potentially removing the need for end-of-wave support)..does the row have an embedded set of repeating groups? 100's of columns? A complex hierarchy for the written xml content?

How well does the job run when you replace the MQ stages with Sequential? (what sort of time is the xslt stage consuming)

Are you playing with end-of-wave in the job, to flush the rows through?

Are you using MQClient connection or going to a local MQServer?

If end-of-wave turns out to be not critical, it would be interesting to see the impact of using the older MQStage instead. The MQConnector added some very important logic that interacts with the underlying parallel framework, but may have produced some extra overhead. This is just a guess at this point, and a fairly unlikely one, but it would be worth looking at this on the fringe of this issue if in fact end-of-wave becomes un-necessary.

Ernie

bman · Post by **bman** » Fri Mar 14, 2008 1:01 pm

Thanks Ernie for taking your time for me and getting into the issue ..

I have gone with the xslt transformer as I had issue with the End of wave processing with the xml out stage ( I posted a query on that too some days back

as there is a requirement cretae multiple repeating segments).

One message from input results into one message at out put but I have some logic for the outgoing xml to have multiple repeating segments based on values from input xml. So using xslt transfer I thought of avoiding the end of wave processing by not using an xml out stage, as the xslt transformer takes one message at atime and transform it to output structure.

I am using the MQ connector and have client mode connection here.

And I am not sure if I can have a client connection to MQ using the old MQ stage, I will check it ?

And on perfomance, When I tried with sequential files as out put stage I was able to improve the perfomance to upto 130 rows/sec

eostic · Post by **eostic** » Fri Mar 14, 2008 2:05 pm

...that was the reason I asked....(MQ Client). MQStage is able to use the MQ Client api in v8, but only if you chose it at install time....