ISD Maximum Running Jobs

pbttbis · Posted: Wed Mar 16, 2016 10:37 am

Hi,

If you look at operations console -> Workload Management -> Queue Management

There are "Maximum Running Jobs" for various queues. The parameter for the ISD queue is set at 100, but it is not configurable (at least in the console) like the others.

Anyone know how to change the value?

Thanks,

Shaun

PaulVL · Posted: Wed Mar 16, 2016 12:58 pm

Google is your friend.

https://www.ibm.com/support/knowledgecenter/SSZJPZ_11.3.0/com.ibm.swg.im.iis.ds.monitor.doc/topics/opcsystempolicies.html

I believe that is what you are looking for.

pbttbis · Posted: Thu Mar 17, 2016 4:20 am

Hi Paul,

Yeah that is for system wide, which I have set to 200.

What I need to be able to do is modify the max limit on ISD jobs under Queue Management.

Thanks,

Shaun

eostic · Posted: Thu Mar 17, 2016 4:53 am

Not sure why you can't edit it, but it's likely because it isn't that meaningful, provided that these are "always on" ISD Jobs (that have an ISDInput)... Carefully consider your ISD Jobs...do tes ...

What's this?

pbttbis · Posted: Thu Mar 17, 2016 9:34 am

Hi Ernie,

The high water mark we estimated to be at 200-300.

The console is a little confusing that it is under the tab "Queue Management", but the parameter there that I want to change is "Maximum Running Jobs". It is currently set to 100 and when trying to change it get the response "Update or delete is not allowed for non batch queues"

Cheers,

Shaun

eostic · Posted: Thu Mar 17, 2016 9:56 am

200 to 300? are these individual services? do they all have isd input? ...or are you talking about 200 to 300 concurrent requests? ......thats a really huge number for isd always ...

What's this?

pbttbis · Posted: Thu Mar 17, 2016 11:12 am

Well for example now we are testing a workload of ~360 request per minute which we are attempting to handle with three applications created in Information Services Director. The applications are configured with 45, 45 and 10 instances as the min and max.

We may be able to reduce the 100 instances down a bit given more load testing but as the ~360 per minute load is only 40% of the expected workload we will need to create more than a 100 instances.

Are you saying ISD is not suited for the above?

eostic · Posted: Thu Mar 17, 2016 1:33 pm

Just saying that seems like a lot. A single instance can often handle 200+ requests per second, even with a slow protocol like SOAP over HTTP. Of course it could depend on your tranformations, but ...

What's this?

pbttbis · Posted: Fri Mar 18, 2016 3:10 am

Well the jobs are very light weight. EE but running on a single node.

The reason for creating so many instances is that even though the job is light weight there is a REST call made by a hierarchical stage in the job that takes anything from 5-12sec to respond. As we need to process a single request in under 30 seconds we decided to create so many instances so that we do not have requests queuing up behind that REST call. In this way we are submitting many parallels REST calls allowing us to come in at under 30 seconds reliably.

The machine is reasonable hefty. 8 cores 32GB of RAM. Saying that while the instances are up and handling requests the CPU idles at 5%. The only signficant resource used is memory. In the case of the 100 instances that is ~20GB.

The client is also looking at upgrading the machine to a size of 48 core and 256GB in the future which may be needed once we start beefing up the jobs with more transformations etc.

So the above is a lot of context and interesting to discuss, but the original post was asking how to change the Maximum Running Jobs for ISD. I have also asked the question to IBM via a PMR.

eostic · Posted: Fri Mar 18, 2016 7:19 am

Ok.. sounds very interesting. Just be careful. Start-up in such situations (with that many osh processes needing to come up) can be very problematic with that few cores, along with that many concur ...

What's this?

pbttbis · Posted: Fri Mar 18, 2016 8:19 am

yeah I set the workload management to start up only 3 jobs every 30 seconds otherwise things do not go well.

pbttbis · Posted: Mon Apr 04, 2016 2:47 pm

Confirmed with IBM that there is hard limit of 100 ISD jobs that can run concurrently per engine.

To work around this we are implementing a multi engine solution and load balancing request across them.

eostic · Posted: Mon Apr 04, 2016 3:18 pm

Ok... let me know if you need any tips on how to deploy your single Operation across multiple DS Engines with automatic round-robin load balancing.

Ernie