MQ to MQ - Realtime batch processing
Moderators: chulett, rschirm, roy
MQ to MQ - Realtime batch processing
Hi,
Is it possible to have an MQ to MQ job which runs as an always on job and at the same time process messages in batch ?
What I mean is, run the MQ IN stage with wait time -1, message count -1 and Record count = 100.
and The job flow as,
MQ -> Transformer ->MQ
Job runs fine untill the last batch. If the last batch has less than 100 messages then that batch is never getting processed. Any way to tell MQ to start processing the last batch, keeping the job always on ?
Is it possible to have an MQ to MQ job which runs as an always on job and at the same time process messages in batch ?
What I mean is, run the MQ IN stage with wait time -1, message count -1 and Record count = 100.
and The job flow as,
MQ -> Transformer ->MQ
Job runs fine untill the last batch. If the last batch has less than 100 messages then that batch is never getting processed. Any way to tell MQ to start processing the last batch, keeping the job always on ?
....how do you re-start the job now, after it reaches 100 messages? ...and if the messages come "in batches"....then why would a particular wait time (several minutes, or even longer perhaps) not be sufficient?
Ernie
Ernie
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
Hi Ernie,
Actually messages are not coming in batch but one at a time. And the publisher publishes more than a million records in avery short span. My job was picking up messages one at atime but this is causing perfomance issue and I thought of changing the transaction count to 100. But I am really doubtfull if this can be achieved ? I cant find a way to tell DS, "okay now process available messages, you have waited enough time" when it runs in always on mode.
One other way I am thinking is to keep the job in an infiniteloop using loop sequencer and keep the job wait time for 10 minute ?
Actually messages are not coming in batch but one at a time. And the publisher publishes more than a million records in avery short span. My job was picking up messages one at atime but this is causing perfomance issue and I thought of changing the transaction count to 100. But I am really doubtfull if this can be achieved ? I cant find a way to tell DS, "okay now process available messages, you have waited enough time" when it runs in always on mode.
One other way I am thinking is to keep the job in an infiniteloop using loop sequencer and keep the job wait time for 10 minute ?
I think I'd like to explore with you more why it's having a performance issue......unless there are blocking stages in there...... otherwise, in "always on" mode it should be fine.
Cycling it via job sequencer is one possibility though, for sure. Run it thru 100 or higher number, and/or a particular time-out, and then re-start it. Just be sure you have an ultimate interrupt so the loop is easy to manage.
Ernie
Cycling it via job sequencer is one possibility though, for sure. Run it thru 100 or higher number, and/or a particular time-out, and then re-start it. Just be sure you have an ultimate interrupt so the loop is easy to manage.
Ernie
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
I think I'd like to explore with you more why it's having a performance issue......unless there are blocking stages in there...... otherwise, in "always on" mode it should be fine.
Cycling it via job sequencer is one possibility though, for sure. Run it thru 100 or higher number, and/or a particular time-out, and then re-start it. Just be sure you have an ultimate interrupt so the loop is easy to manage.
Ernie
Cycling it via job sequencer is one possibility though, for sure. Run it thru 100 or higher number, and/or a particular time-out, and then re-start it. Just be sure you have an ultimate interrupt so the loop is easy to manage.
Ernie
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
I said perfomance issue because the MQ connector stage reads messages from queue at very slow rate. The rate is like 10 messages per second and with this rate processing 1 million records take too much of time ? This is why I started thinking of increasing the record count ?
I cant see any other bottleneck in the job as it has only 2 MQ stages and an XML transformer stage which uses an xslt mapping. If i keep record count as one that means each message is read from the queue transfomred and then commited to the destination queue and I think the commititng for each message eats up all the time
I cant see any other bottleneck in the job as it has only 2 MQ stages and an XML transformer stage which uses an xslt mapping. If i keep record count as one that means each message is read from the queue transfomred and then commited to the destination queue and I think the commititng for each message eats up all the time
Wouldn't this be a case for "wave processing"?
<a href=http://www.worldcommunitygrid.org/team/ ... TZ9H4CGVP1 target="WCGWin">
</a>
</a>
...could be Arnd...lots of things going on. Sounds like end-of-wave may already be utilized...certainly it would be necessary with XMLOutput --- it's likely that XMLTransformer with XSLT needs it too. Plus, running xslt is not exactly a screamer itself --- although I have question --- is the xslt taking one row in and then outputting one row? Curious what motivated the choice of xslt, especially if there is only one row (where row aggregation isn't needed, potentially removing the need for end-of-wave support)..does the row have an embedded set of repeating groups? 100's of columns? A complex hierarchy for the written xml content?
How well does the job run when you replace the MQ stages with Sequential? (what sort of time is the xslt stage consuming)
Are you playing with end-of-wave in the job, to flush the rows through?
Are you using MQClient connection or going to a local MQServer?
If end-of-wave turns out to be not critical, it would be interesting to see the impact of using the older MQStage instead. The MQConnector added some very important logic that interacts with the underlying parallel framework, but may have produced some extra overhead. This is just a guess at this point, and a fairly unlikely one, but it would be worth looking at this on the fringe of this issue if in fact end-of-wave becomes un-necessary.
Ernie
How well does the job run when you replace the MQ stages with Sequential? (what sort of time is the xslt stage consuming)
Are you playing with end-of-wave in the job, to flush the rows through?
Are you using MQClient connection or going to a local MQServer?
If end-of-wave turns out to be not critical, it would be interesting to see the impact of using the older MQStage instead. The MQConnector added some very important logic that interacts with the underlying parallel framework, but may have produced some extra overhead. This is just a guess at this point, and a fairly unlikely one, but it would be worth looking at this on the fringe of this issue if in fact end-of-wave becomes un-necessary.
Ernie
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
Thanks Ernie for taking your time for me and getting into the issue ..
I have gone with the xslt transformer as I had issue with the End of wave processing with the xml out stage ( I posted a query on that too some days back as there is a requirement cretae multiple repeating segments).
One message from input results into one message at out put but I have some logic for the outgoing xml to have multiple repeating segments based on values from input xml. So using xslt transfer I thought of avoiding the end of wave processing by not using an xml out stage, as the xslt transformer takes one message at atime and transform it to output structure.
I am using the MQ connector and have client mode connection here.
And I am not sure if I can have a client connection to MQ using the old MQ stage, I will check it ?
And on perfomance, When I tried with sequential files as out put stage I was able to improve the perfomance to upto 130 rows/sec
I have gone with the xslt transformer as I had issue with the End of wave processing with the xml out stage ( I posted a query on that too some days back as there is a requirement cretae multiple repeating segments).
One message from input results into one message at out put but I have some logic for the outgoing xml to have multiple repeating segments based on values from input xml. So using xslt transfer I thought of avoiding the end of wave processing by not using an xml out stage, as the xslt transformer takes one message at atime and transform it to output structure.
I am using the MQ connector and have client mode connection here.
And I am not sure if I can have a client connection to MQ using the old MQ stage, I will check it ?
And on perfomance, When I tried with sequential files as out put stage I was able to improve the perfomance to upto 130 rows/sec
...that was the reason I asked....(MQ Client). MQStage is able to use the MQ Client api in v8, but only if you chose it at install time....
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>