unable to logon/open DS job to datastage 9.1clients

[zuhairkhan]

Websphere SystemOut.log shows many of the below exceptions.

aused by: com.ascential.xmeta.persistence.PersistenceException: Can't get current session!
at com.ascential.xmeta.persistence.impl.basic.BasicPersistence.getCurrentSession(BasicPersistence.java:5097)
at com.ascential.xmeta.persistence.impl.basic.BasicPersistence.createTransformationContext(BasicPersistence.java:5079)
at com.ascential.xmeta.persistence.impl.basic.BasicPersistence.createTransformationContext(BasicPersistence.java:5065)
at com.ascential.xmeta.persistence.impl.basic.BasicPersistence.createTransformationContext(BasicPersistence.java:5053)
at com.ascential.xmeta.persistence.impl.basic.BasicPersistence.getByRid(BasicPersistence.java:690)
at com.ascential.xmeta.repository.impl.AbstractDefaultRepository.loadByRid(AbstractDefaultRepository.java:688)
... 42 more
Caused by: com.ascential.xmeta.session.SessionException: Session expired. Cannot get session information.
at com.ascential.acs.security.xmeta.XMetaSessionInfoProvider.getCurrentSessionInfo(XMetaSessionInfoProvider.java:79)
at com.ascential.xmeta.persistence.impl.basic.BasicPersistence.getCurrentSession(BasicPersistence.java:5093)
... 47 more
Caused by: com.ascential.asb.util.security.SessionExpiredException: The session [50A85422-833B-49F6-B951-878FE34F5819] has expired or was not found.
at com.ascential.asb.util.security.impl.SessionManagerImpl.getSessionById(SessionManagerImpl.java:218)
at com.ascential.acs.security.xmeta.XMetaSessionInfoProvider.getCurrentSessionInfo(XMetaSessionInfoProvider.java:70)
... 48 more

[10/15/15 2:23:22:317 EDT] 00002c3c ISToolsRESTSe E ISToolsRESTServiceBean updateEObjectCR Error loading objects from list of rids
com.ascential.xmeta.exception.ServiceException: Error loading objects from list of rids
at com.ascential.xmeta.service.repository.RepositoryServiceSupport.unlockEObjects(RepositoryServiceSupport.java:1660)
at com.ascential.xmeta.binding.repository.impl.eobject.AbstractEObjectRepositoryBinding.unlockObjects(AbstractEObjectRepositoryBinding.java:1677)
at com.ascential.xmeta.client.repository.impl.AbstractDefaultRepositoryClient.unlockByRid(AbstractDefaultRepositoryClient.java:577)
at com.ascential.xmeta.client.repository.impl.AbstractBaseRepositoryClient.unlockByRid(AbstractBaseRepositoryClient.java:155)
at com.ibm.istools.rest.server.impl.ISToolsRESTServiceBean.updateEObjectCR(ISToolsRESTServiceBean.java:722)
at com.ibm.istools.rest.server.impl.ISToolsRESTServiceBean.updateEObjectCR(ISToolsRESTServiceBean.java:593)
at com.ibm.istools.rest.server.EJSRemoteStatelessISToolsRESTService2_a41bae5b.updateEObjectCR(Unknown Source)
at com.ibm.istools.rest.server._ISToolsRESTServiceRemote_Stub.updateEObjectCR(_ISToolsRESTServiceRemote_Stub.java:738)
at sun.reflect.GeneratedMethodAccessor201.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at com.ibm.is.binding.rest2.Servlet.callOperation(Servlet.java:447)
at com.ibm.is.binding.rest2.Servlet.processRequest(Servlet.java:370)
at com.ibm.is.binding.rest2.Servlet.doRequest(Servlet.java:275)
at com.ibm.is.binding.rest2.Servlet.doPut(Servlet.java:244)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:598)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1219)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:774)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:456)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1027)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:87)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:895)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1662)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:195)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:458)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:522)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:311)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:282)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1783)
Caused by: com.ascential.xmeta.repository.RepositoryException: Error loading objects from list of rids
at com.ascential.xmeta.repository.impl.AbstractDefaultRepository.loadByRid(AbstractDefaultRepository.java:690)
at com.ascential.xmeta.repository.impl.AbstractDefaultRepository.unlockObjects(AbstractDefaultRepository.java:837)
at com.ascential.xmeta.repository.core.impl.DefaultSandbox.unlockObjects(DefaultSandbox.java:148)
at com.ascential.xmeta.service.repository.RepositoryServiceSupport.unlockEObjects(RepositoryServiceSupport.java:1656)
... 39 more





What could be the problem ?

[zuhairkhan]

even if we login , we get error "Unable to open the job "

when we try top open any job.

[PaulVL]

is you xmeta database up and running?

Can you log in to the IS Console?

Did you bounce your environment to help resolve the issue?

[zuhairkhan]

yes, But i couldnt pinpoint the culprit , i have to avoid this issue from happening again.. any suggestions? restarting the websphere solved the problem, but dont know the issue

[qt_ky]

Strange things can happen if any file system fills up, like /tmp or wherever your projects live, etc.

[zuhairkhan]

increased the active session timeout setting to 7200 in IS console , same as DS Admin . Now i can even access the IS console or weblogic console , he weblogic console is completely hung.

[rkashyap]

Are you working on a sandbox? If I was in your shoes, I will call support.

[zuhairkhan]

SystemErr R 2015-10-16 10:31:26,466 Error [ISF-SECURITY-AUTH-AUTHENTICATION-JAAS] [0003] Invalid user name (cjonnavi) or password.
[10/16/15 10:31:26:488 EDT] 00000092 SystemErr R 2015-10-16 10:31:26,488 Error [ISF-SECURITY-AUTH-AUTHENTICATION-JAAS] [0003] Invalid user name () or password.
[10/16/15 10:31:33:101 EDT] 00000092 SystemErr R 2015-10-16 10:31:33,101 Error [ISF-SECURITY-AUTH-AUTHENTICATION-JAAS] [0003] Invalid user name () or password.
[10/16/15 10:31:33:111 EDT] 00000092 SystemErr R 2015-10-16 10:31:33,111 Error [ISF-SECURITY-AUTH-AUTHENTICATION-JAAS] [0003] Invalid user name () or password.
[10/16/15 10:31:35:732 EDT] 00000092 SystemErr R 2015-10-16 10:31:35,732 Error [ISF-SECURITY-AUTH-AUTHENTICATION-JAAS] [0003] Invalid user name () or password.
[10/16/15 10:31:35:742 EDT] 00000092 SystemErr R 2015-10-16 10:31:35,742 Error [ISF-SECURITY-AUTH-AUTHENTICATION-JAAS] [0003] Invalid user name () or password.
[10/16/15 10:31:37:876 EDT] 00000092 SystemErr R 2015-10-16 10:31:37,876 Error [ISF-SECURITY-AUTH-AUTHENTICATION-JAAS] [0003] Invalid user name () or password.
[10/16/15 10:31:37:886 EDT] 00000092 SystemErr R 2015-10-16 10:31:37,886 Error [ISF-SECURITY-AUTH-AUTHENTICATION-JAAS] [0003] Invalid user name () or password.
[10/16/15 10:31:55:997 EDT] 00000092 SystemErr R 2015-10-16 10:31:55,997 Error [ISF-SECURITY-AUTH-AUTHENTICATION-JAAS] [0003] Invalid user name () or password.
[10/16/15 10:31:56:008 EDT] 00000092 SystemErr R 2015-10-16 10:31:56,008 Error [ISF-SECURITY-AUTH-AUTHENTICATION-JAAS] [0003] Invalid user name () or password.
[10/16/15 10:32:36:165 EDT] 00000091 SystemErr R 2015-10-16 10:32:36,165 Error [ISF-SECURITY-AUTH-AUTHENTICATION-JAAS] [0003] Invalid user name () or password.
[10/16/15 10:32:36:178 EDT] 00000092 SystemErr R 2015-10-16 10:32:36,178 Error [ISF-SECURITY-AUTH-AUTHENTICATION-JAAS] [0003] Invalid user name () or password.


SystemErr.log shows many of the above errors .....

[PaulVL]

I hope that is in your DEV environment.

So basically, are you up and running now that you bounced your websphere?

You may have lost connectivity to your LDAP server if you are LDAP enabled.

[priyadarshikunal]

The details from SystemOut.log is very less, we generally need to see a lot of those lines and mainly a few hundred lines when the problem first occurred. Without that, we are just guessing.

So a few questions
1) Are you on internal registry for authentication or OS/LDAP authentication is used?
2) did you restart all the components or a particular tier to get it working again?
3) what is the default value of ulimit -n for the user you used to restart tiers? Is that overridden in ds.rc or dsenv?
4) search for text "full" or "space" in SystemOut.log, do you find any entry? This may be different from actual disk getting full.
5) did the disk get full at any point in time?

[ray.wurlod]

Check too that the passwords for XMETA, XMETASR and other "system" accounts, as well as those for Engine credential mappings, have not expired.