DSXchange

Having a new problem with some 'RTI Enabled' jobs and not sure which 'side' is having the issue. They are using HTTP over SOAP bindings, Oracle as the repository and WebLogic on the RTI Server side. This is an example of what is returned when one of the jobs is called:

Tue Apr 04 15:28:14 MDT 2006 - Exception while calling PropertyUpdateControl.WsApPropertyUpdate: Tue Apr 04 15:28:14 MDT 2006 - EJB Exception: ; nested exception is: com.bea.control.ServiceControlException: <xml-fragment xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <faultcode>RTIException</faultcode> <faultstring>java.rmi.RemoteException: EJB Exception: ; nested exception is: javax.ejb.EJBException: Exception trying to invoke operation WsApPropertyUpdate: No handlers are available to process this request.</faultstring> <detail/> </xml-fragment>

When it says that "No handlers are available to process this request" is this a problem on the ETL / DataStage Server / RTI Agent side or the WebLogic / RTI Server side?

I don't see anything wrong on the DS side - RTI Agent is running, jobs are running, nothing funky is in the logs. I've stopped and restarted them via the RTI Console and the same has allegedly been done on the WebLogic side but it seems to be persisting. Can anyone say for certain what might be the root cause of this error?

Hi,
How many instances of the RTI job do you have running?
When this happens can it be you exceeded the amount of RTI instances that you set to be waiting for requests?

Don't believe that's the case. The volume using it is so low the fact that only one 'invocation' of it is running at any given time really shouldn't be the issue. My BEA folks feel it has something to do with the Connection Pooling and the lack of the ability to 'test' connections automatically with the driver supplied with RTI. Found this in one of their logs:

<Apr 5, 2006 8:33:06 AM MDT> <Error> <JDBC> <BEA-001112> <Test "SELECT 1 FROM DUAL" set up for pool "RTIPool" failed with exception: "java.sql.SQLException: [Ascential][Oracle JDBC Driver]This driver is locked for use with embedded applications.".>
<Apr 5, 2006 8:33:06 AM MDT> <Error> <JDBC> <BEA-001111> <Unable to verify the test "SELECT 1 FROM DUAL" set up for pool "RTIPool". Connections will not be tested.>

Opened a case with Support, we'll see what they say.

Does anyone know the resolution to this problem? We have a job that has been running for months and today it is giving this error.

APT_CombinedOperatorController(0),0: Fatal Error: Fatal: <ns1:Fault xmlns:ns1="http://schemas.xmlsoap.org/soap/envelope/">
<faultcode>RTIException</faultcode>
<faultstring>java.rmi.ServerException: RemoteException occurred in server thread; nested exception is:
java.rmi.RemoteException: ; nested exception is:
javax.ejb.EJBException: Exception trying to invoke operation ds_apl_style_color.


Timed out waiting for an operation handler to become available for Operation &apos;ds_apl_style_color&apos;.</faultstring>
<detail/>

I have searched here and support but have found nothing. We don't think it's the Datastage job and we have restarted all services that are associated with this but no luck.

The NO HANDLER AVAILBLE error is fairly generic, but points to the fact (as indicated in some of the entries of the old thread above) that there is "no instance" of a DS Job available to process the incoming request. This can happen for a multitude of reasons, among them:

The RTI Agent is down.

Something's wrong with the port it is using (be sure 2000 or whatever you chose is "listening" via netstat -a|grep <port>

You've flooded the instances with requests, the pipeline buffer is full, and the queue at the RTI Server is full....

The Job keeps crashing.

I'll think of others...... increase your instances to say, 3, check the Agent (well, you won't be able to increase instances if the agent is down), check the conditions of the error (how many incoming requests, etc.).

Ernie

I forgot about this. We were running our jobs as 'always on' jobs, without allowing it to automagically start new invocations if it needed to. For whatever reason 'they' wanted to stick with that always on approach so we bumped the number of simultaneous invocations up to 2 or 3 depending on the service. Solved the problem for us.

OK, here is what we have done so far but have not resolved the issue.

We have bumped up the simultaneous invocations.
We have restarted all the RTI services and Web Services.
Confirmed the port is listening.

But now we may have the same or a different problem. It appears that the web service is running fine when we look in the log but the calling job is hanging.

Is there anything else interesting that you can tell us about the job? Is it just a standard "batch style" job with no RTIinput or RTIoutput? Are you passing anything into it at all (Job Parameter values?)? Does the job run fine from the Director (without using RTI to kick it off)?

Do you have any other jobs like it that are working?

Just looking for interesting differences in this one particular job.

Ernie

You've marked this topic Resolved. What was the resolution? Help others who may encounter this issue in future by posting the solution here.

*I* marked it as resolved as it was my issue... and my resolution was posted earlier.

Yes, the original issue was resolved. I just had a follow-up question as we were still having problems. But ours has been resolved as well. Our business unit decided to make changes to the source system without informing us, so our RTI was suppose to return around 200 records was getting around 6,000 and was getting hung.

Here is the post with the resolution for this thread, just to make it easier.

I forgot about this. We were running our jobs as 'always on' jobs, without allowing it to automagically start new invocations if it needed to. For whatever reason 'they' wanted to stick with that always on approach so we bumped the number of simultaneous invocations up to 2 or 3 depending on the service. Solved the problem for us.