DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
Author Message
qt_ky



Group memberships:
Premium Members

Joined: 03 Aug 2011
Posts: 2822
Location: USA
Points: 21387

Post Posted: Fri Jun 01, 2018 9:24 am Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Parallel
OS: Unix
Additional info: 11.x on AIX
We are seeing regular warnings in the SystemOut.log about 1-2 times a week:

Code:
00006595 ThreadPool    I   WSVR0652W: The size of thread pool "WebContainer" has reached 80 percent of its maximum.
00006561 ThreadPool    I   WSVR0652W: The size of thread pool "WebContainer" has reached 100 percent of its maximum.


We have worked a Support case and it was confirmed that we are hitting a limit and was suggested that performance tuning is beyond the scope of Support and that we would need to purchase some sort of Services. We are running only 16 ISD applications along with many DataStage and QualityStage jobs (mostly DataStage-only) that are scheduled throughout each day. We are running on IBM POWER8 with 6 cores and plenty of free memory. All server tiers are on the same box. We found guidance from IBM that the thread pool should only be increased up to a limit of 10x the number of cores, or 60 in our case. Has anyone pushed past that ratio?

What is the limit or rule of thumb on the number of ISD applications that can be run simultaneously on a typical Information Server, where all server tiers are installed on the same computer?

Also, does anyone know the limit or rule of thumb on the number of ISD applications that can be run simultaneously on Information Server, when the server tiers are installed on separate computers?

_________________
Choose a job you love, and you will never have to work a day in your life. - Confucius
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42796
Location: Denver, CO
Points: 220596

Post Posted: Fri Jun 01, 2018 11:51 am Reply with quote    Back to top    

So... this limit isn't configurable? Confused

_________________
-craig

"I don't mind you comin' here and wastin' all my time time"
Rate this response:  
Not yet rated
qt_ky



Group memberships:
Premium Members

Joined: 03 Aug 2011
Posts: 2822
Location: USA
Points: 21387

Post Posted: Mon Jun 04, 2018 6:41 am Reply with quote    Back to top    

We already bumped up the max to 60, for our 6 cores, based on the guidance found here (warning: heavy reading):

IBM WebSphere Application Server Performance Cookbook

"Thread pools need to be sized with the total number of hardware processor cores in mind."
...
"Good practice is to use 5 threads per server CPU core for the default thread pool, and 10 threads per server CPU for the ORB and Web container thread pools ."

Has anyone else run into these types of warnings and tried to exceed the suggested ratio of 10 threads per server CPU?

_________________
Choose a job you love, and you will never have to work a day in your life. - Confucius
Rate this response:  
Not yet rated
eostic

Premium Poster



Group memberships:
Premium Members

Joined: 17 Oct 2005
Posts: 3790

Points: 30485

Post Posted: Wed Jun 06, 2018 8:44 am Reply with quote    Back to top    

I've never seen that particular message --- I wonder if it is related to "traffic"....meaning...struggles that WAS might be having as a bottleneck because of the number of requests coming in, as oppos ...

_________________
Ernie Ostic

blogit!
Open IGC is Here!
Rate this response:  
Not yet rated
qt_ky



Group memberships:
Premium Members

Joined: 03 Aug 2011
Posts: 2822
Location: USA
Points: 21387

Post Posted: Wed Jun 06, 2018 12:20 pm Reply with quote    Back to top    

Thanks for responding. The warnings consistenly come in late afternoons, pretty much within minutes of each other, even across days and weeks.

We had initially assumed traffic would be the cause--that there must be an unusually large number of ISD requests hitting our server at once during that time of the afternoon. We have done numerous counts of our ISD audit logs across many dates when we saw thread warnings. We found that the traffic assumption was not supported by the numbers. In fact, the number of overall ISD requests was much greater at other times of the day when thread warnings have never been logged.

We later realized that we had one 30+ minute DataStage sequence scheduled to start about 5 to 10 minutes prior to the time of the thread warnings. We delayed the sequence's start time by 30 minutes to see if the thread warnings would follow. While they did not follow the sequence, they started happening about 20 minutes later than we had ever seen before, which is about 10 minutes before the sequence kicks off. So, we ruled that out also, and I have not yet been able to find a pattern that it does follow.

Thank you for sharing a rule of thumb estimate you knew about. That is new info to me and is what I was looking for as another avenue to investigate. Now we can begin counting something more appropriate and go from there.

_________________
Choose a job you love, and you will never have to work a day in your life. - Confucius
Rate this response:  
Not yet rated
eostic

Premium Poster



Group memberships:
Premium Members

Joined: 17 Oct 2005
Posts: 3790

Points: 30485

Post Posted: Wed Jun 06, 2018 3:25 pm Reply with quote    Back to top    

The "symptom" that led to that rule of thumb is primarily stability...that Jobs that have to "restart" would fall over because there was too much on the system, or that randomly, when ISD would cycle ...

_________________
Ernie Ostic

blogit!
Open IGC is Here!
Rate this response:  
Not yet rated
qt_ky



Group memberships:
Premium Members

Joined: 03 Aug 2011
Posts: 2822
Location: USA
Points: 21387

Post Posted: Fri Jun 08, 2018 11:57 am Reply with quote    Back to top    

Thank you for the additional pointers.

Within the always-on ISD jobs, according to the score of each job we have a total of 224 processes running. As noted earlier, this box has 6 cores, so that averages out to always running at least 37 processes per core. Another surprise is that we found one job set to run on 4 nodes while each was supposed to be set to run on a single node. We are not, however, seeing any churn with ISD jobs stopping and new instances getting spawned. Perhaps we are on the borderline of pushing some undocumented limit.

Our ISD min/max instance settings are mostly set to 1/2 while a small handful are set to 1/5 and one is set to 1/1.

All of our ISD applications have the "Infinite" setting checked under "Idle Time" (if that's the setting you were thinking of).

_________________
Choose a job you love, and you will never have to work a day in your life. - Confucius
Rate this response:  
Not yet rated
eostic

Premium Poster



Group memberships:
Premium Members

Joined: 17 Oct 2005
Posts: 3790

Points: 30485

Post Posted: Fri Jun 08, 2018 3:04 pm Reply with quote    Back to top    

Sounds good and smartly calculated. Probably needs support to find out the true origin of that error.

Ernie

_________________
Ernie Ostic

blogit!
Open IGC is Here!
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours