DSXchange

Looking for advice and tips on how we can monitor usage of WISD DataStage jobs running as web services?

We've recently deployed a set of new services to support a 3rd party application and I'm looking to help report the service usage activity. (ie. peak usage time, low usage time, avg daily usage, etc...). There are several features of this application that use different WISD services and we're curious which ones are getting used more than others by analyzing the frequency of calls to the service and potentially identify any load stress issues.

What tools have others used or know about that could help with this? I would be looking for a capture of every single API call and the timestamp when it occurred or at the very least hourly statistics.

We use the concept of an audit log to log every ISD request. In the beginning of the job design, copy selected input columns and/or generated columns, such as a current system timestamp, into a Sequential File stage.

Include #DSJobInvocationId# in the file name. Set the "File Update Mode" to Append. Set "Cleanup On Failure" to False (important). That last setting defaults to True, and in cases where the job aborts or gets stopped unexpectedly, the True setting results in loss of your current audit log file. Set "First Line is Column Names" to False. Reject Mode = Continue.

Then all your audit logs accumulate over time and can be grep'ed, parsed, or read by another job and loaded somewhere for reporting if you wish.

Thank you for those sequential setup tips. The idea of that had crossed my mind, but concerned about the added service overhead. I'll give that a try and see if there's any noticeable performance difference. That's going to be our best solution if there's not a good overall scraping/combing mechanism that could be plugged in from the outside.

Great! I won't take credit for the idea. I must have run across it in a document at some point. It works well for our environment. I also was unable to locate any out-of-the-box logs that already contained that level of detail. I think you have to create your own. I would be curious if you are able to measure any impact from it.

9.1 can be set up to use the DSODB databse. I believe you can track usage via that and simply query the database for that job name + invocation id run stats.

One possibility is to do it directly withjn WAS. While not exposed in the ISD GUI, ISD services, in the end, are deoloyed as standard J2EE Application Archives.....or EARs. Its been a looong time since I have looked at it, but any added functionality you might want, from complex logging against an endpoint, or sophisticated security, can theoretically be done at the WAS level. I recall downloading ISD EARs, editing their xml and other contents, and then re-importing them. Adding tracking may be far simpler and doable at the WAS admin level or with some additional coding. It probably wont get you to the inner guts of a service, but you may be able to track call activity at the http server level......

Start by researching WebSphere App server logging and customization......no guarantees, but it might surfaxe some under the covers way to track your services.

Ernie