Page 1 of 1

Disk Space getting filled

Posted: Tue Apr 13, 2010 7:27 am
by kamesh_sk
HI,

i have a 14G space allocated to /opt/IBM folder but this space gets filled by certain log files . I have found certain files under /opt/IBM/InformationServer/ASBNode/bin occupying around 4G of space.
The file names are
-rw-r--r-- 1 root root 1006275756 Apr 7 13:37 orbtrc.05042010.2211.46.txt
-rw-r--r-- 1 root root 465369 Apr 7 14:11 orbtrc.07042010.1402.21.txt
-rw-r--r-- 1 root root 606381173 Apr 8 14:26 orbtrc.07042010.1413.37.txt
-rw-r--r-- 1 root root 1639260160 Apr 13 14:11 orbtrc.08042010.1426.47.txt
-rw-r--r-- 1 root root 4122763264 Apr 13 14:11 logging-agent.err

Can any one help to understand what are these files are and why such huge size files are created and by which Datasatge operation and what should i do to avoid it.
Due to size issue my jobs are getting aborted.
Can anyone help please

Thanks
Kamesh

Posted: Tue Apr 13, 2010 7:52 am
by ArndW
From the names one can see they are log files. Particularly 'logging-agent.err'. Have you looked into this file to see what errors it is logging?

Posted: Tue Apr 13, 2010 8:42 am
by kamesh_sk
The entire Gb space conatains this error.

The file keeps growing

; nested exception is:
javax.ejb.EJBException: nested exception is: java.sql.SQLException: [IBM][Oracle JDBC Driver][Oracle]ORA-01653: unable to extend table XMETA_OWNER.LOGGING_LOGGINGEVENT1466CB5F by 8192 in tablespace INFOS_XMETA_DATA

at com.ascential.acs.logging.server.impl.JDBCLoggingEventDAO.insert(JDBCLoggingEventDAO.java:1031)
at com.ascential.acs.logging.server.impl.JDBCLoggingEventDAO.create(JDBCLoggingEventDAO.java:401)
at com.ascential.acs.logging.server.impl.appender.OperationalRepositoryAppender.append(OperationalRepositoryAppender.java:110)
at com.ascential.acs.logging.server.impl.appender.AppenderController.doAppend(AppenderController.java:73)
at com.ascential.acs.logging.server.impl.appender.OperationalRepositoryAppender.doAppend(OperationalRepositoryAppender.java:84)
at com.ascential.acs.logging.server.impl.config.LoggerImpl.appendLoopOnAppenders(LoggerImpl.java:713)
at com.ascential.acs.logging.server.impl.config.LoggerImpl.callAppenders(LoggerImpl.java:657)
at com.ascential.acs.logging.server.impl.config.LoggerImpl.forcedLog(LoggerImpl.java:615)
at com.ascential.acs.logging.server.impl.config.LoggerImpl.log(LoggerImpl.java:533)
at com.ascential.acs.logging.server.impl.LoggingServiceBean.log(LoggingServiceBean.java:479)
at com.ascential.acs.logging.server.impl.LoggingServiceBean.log(LoggingServiceBean.java:497)
at com.ascential.acs.logging.server.EJSRemoteStatelessLoggingService_5c03a87a.log(Unknown Source)
at com.ascential.acs.logging.server._EJSRemoteStatelessLoggingService_5c03a87a_Tie.log__CORBA_WStringValue__java_util_Collection(_EJSRemoteStatelessLoggingService_5c03a87a_Tie.java:553)
at com.ascential.acs.logging.server._EJSRemoteStatelessLoggingService_5c03a87a_Tie._invoke(_EJSRemoteStatelessLoggingService_5c03a87a_Tie.java:152)
at com.ibm.CORBA.iiop.ServerDelegate.dispatchInvokeHandler(ServerDelegate.java:614)
at com.ibm.CORBA.iiop.ServerDelegate.dispatch(ServerDelegate.java:467)
at com.ibm.rmi.iiop.ORB.process(ORB.java:439)
at com.ibm.CORBA.iiop.ORB.process(ORB.java:1761)
at com.ibm.rmi.iiop.Connection.respondTo(Connection.java:2376)
at com.ibm.rmi.iiop.Connection.doWork(Connection.java:2221)
at com.ibm.rmi.iiop.WorkUnitImpl.doWork(WorkUnitImpl.java:65)
at com.ibm.ejs.oa.pool.PooledThread.run(ThreadPool.java:118)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1475)
Caused by: javax.ejb.EJBException: nested exception is: java.sql.SQLException: [IBM][Oracle JDBC Driver][Oracle]ORA-01653: unable to extend table XMETA_OWNER.LOGGING_LOGGINGEVENT1466CB5F by 8192 in tablespace INFOS_XMETA_DATA

Posted: Tue Apr 13, 2010 9:04 am
by ArndW
kamesh_sk wrote:...unable to extend table XMETA_OWNER.LOGGING_LOGGINGEVENT1466CB5F by 8192 in tablespace INFOS_XMETA_DATA...
So that file seems to be growing because of table extent issues. Fix those and the error log will (probably) stay small. What kind of contents do the other files have?

Posted: Tue Apr 13, 2010 9:08 am
by battaliou
Ok, looks like you got a serious problem here. My first instinct is that your "server side tracing" is on. Check it and turn it off using the datastage administrator app. Also check if any of your load jobs are generating warnings, this fills up log space quickly. You will need to tune your jobs if they do. You can lose the orbtrc.05042010.2211.46.txt files without any impact to your system. I'd ditch the logging-agent.err file too.

Posted: Tue Apr 13, 2010 9:50 am
by kamesh_sk
Thanks very much for the timely information.

We have few quality stage Job which is deployed thru WISD to make it available for MDM application.
One of the job is aborting , (reason not known since RTLogging was not enabled).
Since the job is called by WISD , it tries to run continuously with different Invocation ID.
So eventually what happens is the job aborts every second and restarts again with new invocation ID.
This process generates huge amount of log and they are written into table XMETA_OWNER.LOGGING_LOGGINGEVENT1466CB5F by in tablespace INFOS_XMETA_DATA. The Tablespace gets full after some time and then it starts building up error log under /opt/IBM/InformationServer/ASBNode/bin file name logging-agent.err.

This logging-agent.err file contains nothing but the information saying the tablespace is filled. and this file grows bigger in size and occupies the entire disc space.This file has grown to a size of staggering 5GB
This is building up a huge issue.

All i need to know is whether the table XMETA_OWNER.LOGGING_LOGGINGEVENT1466CB5F in tablespace INFOS_XMETA_DATA will be populated only for errors by qualitystage jobs or any log generated by the Datastage project.
Is it possible to disable this logging? what will be the option to disable logging and what will be its effects.

The RTLogging was kept as 0 and hence no log was generated in the director and it was just changed to 1.
Will this have an effect in the way the log information is handled .

How is logging handled with this version , is it not generated as files , ,or does it populates the Repository tables

Kindly help

Posted: Fri May 07, 2010 8:20 am
by DSShishya
Did you have any success in resolving this issue?

If Yes, please give details for everyone's benifit.

Posted: Fri May 07, 2010 8:24 am
by DSShishya
Did you have any success in resolving this issue?

If Yes, please give details for everyone's benifit.