Page 1 of 1

WAS Connections for ISD and Operation Queue Monitoring

Posted: Mon Mar 07, 2016 5:09 am
by pbttbis
Hi,

Anyone know if there is any limitation on the number of open connections that can be made to an ISD service via the WAS?

Also is there anyways to monitor the number of unserviced requests waiting to be served up to the ISD job instances? From what I understand is that these requests would sit in the operations queue, but I am unsure where to monitor the state of it.

Thanks,

Shaun

Posted: Mon Mar 07, 2016 5:49 am
by eostic
no way to monitor requests active in the queues....there are several places where requests might queue up.......the operations queue and the pipeline buffer and all of the requests that are concurrently inside a single job instance.

better to understand why you are asking......what is the issue you are having?

Ernie

Posted: Mon Mar 07, 2016 7:17 am
by pbttbis
We make use of RGGmanager on Red Hat Linux to control the fail over of our Data Stage tiers.

We had a situation where the

"IBM/InformationServer/HAScripts/InfoSvrServices status" called via RGManager returned a non 0 error code. Causing the DataStage services tier to fail over.

At the point the fail over was triggered we had processed roughly 6000 requests in 2 hours successfully that were served up to two ISD jobs each configured to have 5 instances running. The external services that we call from the ISD job via hierarchical stage was starting to respond slower which means we would most likely have being backing up requests "somewhere". The ISD jobs are all running on a single engine node configuration with the stages set to sequential. The ISD service is scheduled to handle 5 requests per instance (i believe this is the pipelining size).

So my thinking is that we may have possibly hit some sort of threshold when it comes to number of open connection (unserviced requests) which may have caused the non 0 error code from the status script. Possibly timeout or maybe some memory limit for keeping the requests in the queue.

I can see in information server console under session management -> Active Session there is a field called "Current Sessions", but I don't think that has anything to do with unserviced requests.

Looking at the WAS console if I go to Container Services -> Transaction Service I see various timeout fields. Namely:

1) Total transaction life time timeout
2) Async response timeout
3) Client inactivity timeout
4) Maximum transaction timeout

For 1, 3 and 4 these are very large 30min+.

Not sure if the async timeout is applicable here as we may be going over 30 seconds. Either way I would expect the request to be timed out cleanly with no error being thrown by InfoSvrServices status

The ISD jobs do generate 1 warning per request, which is another idea I am investigating. Not sure if this warning being logged indefinitely for each request on an "always on" ISD job may have caused it.

I am working the issue with IBM via a PMR but the person I am dealing with does not seems to know where/how I can monitor the operations queue.

Posted: Mon Mar 07, 2016 8:16 am
by qt_ky
I once tried asking IBM Support similar ISD questions and they kept recommending to purchase Lab Services. I hope you have better luck.