DataStage is unresponsive, XMETA is 92% full

[sinhasaurabh014]

Hi,
I am using version 8.1 on AIX,XMETA on DB2,running around 50 sequence jobs (each sequence job having three to four server jobs) every 20th and 50th minute of everyhour,everyday.

I have been facing this issue when the moment I bring up DataStage, my UAT box becomes slow. DataStage really hangs and director refresh is too slow.

I googled and DSX-searched and came about this APAR JAR from IBM where they ask you to redirect the logs to hashed files (RT_LOGxxx) as it used to happen in prior versions. The only drawback-as I understood- was that post redirection of logs, we would not be able to see logs in Director. This is actually stopping me to take this action. (Please advise if I have understood it correctly)

When I checked the size of XMETA, it was found to be 92% utilized(call get_dbsize_info(?,?,?,0). So I thought that I must do some data truncation from a few tables here....something that would produce CLEAR.FILE RT_LOGxxx effect. I searched throgh XMETA and came acoss this particular table named: LOGGING_XMETAGEN_LOGGINGEVENT1466CB5F. I noticed that this table has 1124801 rows and average row length is 2694 bytes. I think this is the culprit table.

Can someone please advise if it would be OK to truncate this table?

[sinhasaurabh014]

Hi
I went ahead and truncated that table in DB2. Performance is marvellous!!!! However,I am still to see some logs in director.....just praying that aspect is not screwed up!!

Will post the final observation some time soon.

[chulett]

My understanding is the Director can only see logs when they are in hashed files, just like previous versions. When they're in XMETA, you have to check elsewhere, the web console? You can search for "rtlogging" or "rt_logging" (one of the two) for discussions on this option.

As for the truncation question, you'd be better off asking your official support provider, I would think. I haven't heard of one you can simply truncate in any discussions here as of yet. OK... too late for that, I see. I would still double-check and not make a habit of doing things like that without checking first.

[sinhasaurabh014]

Thanks
I have gone through the various posts here in DSX that says that logs are not viewable in director if they are stored in XMETA. However, just an FYI...i havent found it it to be true. I have two environments-SIT and UAT where the logs were being directed to XMETA. I was able to see all the logs in director with no issue.
Then that problem came---director going slow and XMETA reaching 92% capacity in UAT. So, I truncated the table.Its now that I am not able to view any log in director. When I direct the logs to hashed files, director shows up. However,its till not very predictable and some of my jobs are aborting with this error meessage in &PH& folder:

DataStage Job 55 Phantom 8308
Program "JOB.876970538.DT.1530482431": Line 309, Variable previously undefined. Zero length string used.
Job Aborted after Fatal Error logged.
Program "DSD.WriteLog": Line 315, Abort.
Attempting to Cleanup after ABORT raised in job Seq_Cafeteria_Detail..JobControl

DataStage Phantom Aborting with @ABORT.CODE = 1

I dont know what to say at this time but yes,as Chulett poinetd out,it may not be advisable to truncate the log table in XMETA as it can upset the 'director'.

In SIT, I can see logs through director even as the logs go to XMETA...

I hope the above note helps somebody!!!!

[chulett]

From what I recall, this:

Job Aborted after Fatal Error logged.
Program "DSD.WriteLog": Line 315, Abort.

Is the result of a call to the DSLogFatal function, a conditional user controlled abort and unrelated to the 'zero length string used' informational message or anything else we've been discussing in this thread.

And this thread is probably the best post on the whole RT_LOG versus XMETA change made in 8.1. Have you checked the values of RTLogging and ORLogging in your two environments, made sure they are set the same across the two?

[sinhasaurabh014]

Hi
Yes, we have user controlled aborts in our sequences that may be causing this conditional abort.

I rechecked the settings and verified that RTlogging =0 and ORlogging=1 (which means that the logs be redirected to XMETA.) in both the environment.

Though I have had some luck here---really not 100% sure what worked (because others might have tried too but seemingly it did not produce the same results for them). I am again able to see logs in DS Director with ORLogging=1 and RTLogging =0 and these are fresh logs from XMETA itself. I can see the particular table increasing in size and all the details of my log...everything looks good.

Though I had been bouncing the DS Server and the WAS server to get some resolution, I never bounced the ASB Nodes agent(./NodeAgents.sh stop/start in the ASBNodes/bin directory)
This time, I was logged in as a root user and bounced all the three servers-ASB,DS and WAS.

So now, my XMETA is just 3.4%, DS is running happily and DS Director shows me the logs,my RT_LOG still default size.

[Oritech]

Then that problem came---director going slow and XMETA reaching 92% capacity in UAT. So, I truncated the table!!!!

Can you please provide the table name which is truncated in Xmeta..

[chulett]

It's named in the first post.

[ray.wurlod]

That won't help you, because your table names will be different.

Table names in XMETA all carry an eight hexadecimal digit suffix tied to the individual installation of the Information Server product/environment to which they relate.