Workload Management for ISD

pbttbis · Posted: Fri Feb 05, 2016 5:59 am

Hi,

We have a lightweight ISD job that run for 4-5 seconds. The majority of this time is spent waiting for a response from an application request sent via a REST step from the hierarchical stage.

We are expected to handled +- 300 000 calls to this job a day. I have asked the question around peak workloads.

With regards to workload management I see two areas to configure:

1) Options that can specifically be set in the information server console for the services.

2) DataStage workload management. Queue priority, number of active jobs etc

Are there any best practices around setting up the workload management that I can apply to the above scenario?

Thanks,

Shaun

eostic · Posted: Fri Feb 05, 2016 6:32 am

Does the Job have the Hierarchical Stage as its source? Most of that 4 - 5 seconds is probably in initialization. See if you can write and test it as an intermediate Stage -- one that uses an ISD ...

What's this?

pbttbis · Posted: Fri Feb 05, 2016 10:02 am

hahaha Ernie I believe I am the poster you are referring to w.r.t to the HTTP plumbing issue I had Smile

Currently in our DataStage test environment I have the ISD service to generate 3 instances of a job "ready and waiting" to process requests.

The job is simply put ISD_Input --> Hierachial (REST Step) --> ISD_Output.

The vast majority of the 4-5 seconds is the REST Step waiting for the external application to return a response. So the DataStage portion itself is very light weight. We are currently running it on one node with all stages set to sequential.

So for me one of the questions is how many instances should I have running of the job that we expect to run in 4-5 seconds. The workload is 300 000 request that need to be handled by the job with a peak work load of around 600 a minute. The requests need to be returned by the job in under 30 seconds. I would think we should design for 600 + a percentage to give us some wiggle room above the current max work loads seen.

Financial impact if requests are not returned in under 30 seconds.

eostic · Posted: Fri Feb 05, 2016 12:46 pm

ah. ok... Then you need a test harness. It's the only way you'll figure out the best settings. I like to start with "3" instances, and then make sure that the pipeline and operation buffers ar ...

What's this?

qt_ky · Posted: Sat Feb 06, 2016 9:32 am

Just curious... Would it be possible to load test the service by using a multi-instance job that calls the service, perhaps from a separate Information Server? Any drawbacks to that approach?

eostic · Posted: Sat Feb 06, 2016 5:29 pm

i have done that.....it will work too........but it is very difficult in that pattern to tamp up as high as you might need.....like having 100+ concurrent clients each sending 20 requests per minute.. ...

What's this?

pbttbis · Posted: Sun Feb 14, 2016 6:56 am

Have being making use of CURL and a shell script to run my test harness.

Have settled now on the number of instances that will be needed.

I have being making use of operations console to monitor resources during my testing and notice that currently I am limited by memory and not CPU. This kinda makes sense to me as the JOB is only doing two singleton inserts into a database. Majority of time spent is waiting for the REST step call to return.

Where can I monitor queued requests hitting the ISD service? The job itself logs when it receives the requests, but I would like to monitor requests waiting to be served up to the ISD jobs.

I also noticed that if i try start up all the instances at once that the DS srver hangs. I am getting around this by setting a workload parameter to only start x amount of jobs in x seconds. Is this behaviour normal I am experiencing? Is there a way to set this workload parameter only for ISD jobs?