Call to input link returned numeric error code: 19

chulett · Post by **chulett** » Wed Mar 07, 2007 10:59 am

Anyone seen this message? It's a new one on me and I can only find one (unresolved) occurance in the forums for it. Got these two messages before it:

Abnormal termination of stage Jobname..transformer detected

Jobname..transformer.link: Run stopped after 2707126 rows

Names have been changed to protect the innocent. Also got this after a Reset:

From previous run
DataStage Job 91 Phantom 29391
Abnormal termination of DataStage.
Fault type is 10. Layer type is BASIC run machine.
Fault occurred in BASIC program DSD.ReplaceParams at addr

Job design is basically:

OCI -> XFRM -> XML Output -> Hashed file

With a couple of hashed lookups. The link reporting the error is from the transformer to the XML stage and the odd thing is it has stopped at the exact same row count three times now. Wondering if it some odd data issue, off to investigate that...

I tried repromoting / recompiling the job in case something odd was corrupted executable-wise, but no joy there.

ray.wurlod · Post by **ray.wurlod** » Wed Mar 07, 2007 1:00 pm

Fault types are documented in a late chapter (appendix?) in the UniVerse Administrators guide which you can download from the IBM U2 website. Don't have time to check right now, and it's one of the ones I have chosen not to memorize.

chulett · Post by **chulett** » Wed Mar 07, 2007 5:32 pm

A fault type of 10 is apparently a "(SIGBUS) bus error". Not sure how much that helps, however...

chulett · Post by **chulett** » Fri Mar 09, 2007 4:30 pm

Definitely environment related. Found the job had IP Row Buffering enabled on the Performance tab, something we don't do as a Rule of Thumb. Turned that off and we've now moved on to a different error:

Code: Select all

Transformer..Link "1": build_record() - Unable to allocate memory
XML Output..Link "2": Cannot write output row.
At row 1, link "2" Call to output link returned numeric error code: 10
At row 1, link "2" Call to output link returned numeric error code: 10
At row 1, link "2" Call to output link returned numeric error code: 10
Job Aborted after 5 errors logged.

From previous run 
DataStage Job 91 Phantom 21413
Job Aborted after 5 errors logged.
Program "DSD.WriteLog": Line 201, Abort.
Attempting to Cleanup after ABORT raised in stage Transformer..XML Output
Program "DSD.OnAbort": Line 164, COMMON size mismatch in subroutine "DSP.Close".
Program "DSD.OnAbort": Line 164, Unable to load file "DSP.Close".
Program "DSD.OnAbort": Line 164, Unable to load subroutine.

Link "1" is going from the Transformer to the XML Output stage while link "2" goes from the XML Output to the Hashed file.

Still working this with Support.

ray.wurlod · Post by **ray.wurlod** » Fri Mar 09, 2007 4:38 pm

DSP.Close is one of theirs.
COMMON size mismatch in subroutine "DSP.Close". means that there's a bug, an incompatibility, between their modules.
"They" need to fix it.

chulett · Post by **chulett** » Sat Apr 07, 2007 6:55 am

The saga continues. After failing to reach any conclusions with IBM Support as to what might be causing 7.5.2 to misbehave on this server and in an effort to isolate the issue, we rolled back to 7.5.1A - a version we use on three other servers with no issues.

Took a full export from another machine and rebuilt the only project it houses. Preliminary tests looked good but longer runs went back to their old tricks of core dumping and generally just falling over dead with no apparent reason until we reset the job. Some of our old friends came back. First visitor:

Code: Select all

Message: 
From previous run 
DataStage Job 127 Phantom 21476 
Program "DSD.StageRun": Line 657, COMMON size mismatch in subroutine "DSP.Close". 
Program "DSD.StageRun": Line 657, Unable to load file "DSP.Close". 
Program "DSD.StageRun": Line 657, Unable to load subroutine. 
Attempting to Cleanup after ABORT raised in stage DODSLDHFullStg..DPC_LDH 
Program "DSD.OnAbort": Line 164, COMMON size mismatch in subroutine "DSP.Close". 
Program "DSD.OnAbort": Line 164, Unable to load file "DSP.Close". 
Program "DSD.OnAbort": Line 164, Unable to load subroutine.

Two differences from last time. One is nothing proceeded this (no memory errors) in the log, only the 'abby normal termination' warning from the active stage, then it dropped dead. Second is the passive stage involved is an OCI stage while before it was an XML Output stage. For whatever that is worth.

Rather than stick with the imported executables from another machine, decided to recompile all of the jobs on this server. Next run of this job yielded a slightly different error:

Code: Select all

Message: 
From previous run 
DataStage Job 127 Phantom 18747 
jobnotify: Unknown error 
DataStage Phantom Finished. 
[18765] DSD.StageRun DODSLDHFullStg. DODSLDHFullStg.xform_1 1 0/5/1 - core dumped.  

Message: 
From previous run 
DataStage Job 127 Phantom 18765 
Abnormal termination of DataStage. 
Fault type is 10.  Layer type is BASIC run machine. 
Fault occurred in BASIC program DSD.Update at address 1d2

This again after just falling over dead during the job run with no other messages in the log.

I then realized that with this new / clean install I hadn't made our 'standard' tweaks to the uvconfig file. While I didn't think it was related to our issue, figured we needed to get that out of the way. Dropped the server, tweaked, regen'd and brought it back up. Next run, right back where we started:

Code: Select all

Message: 
From previous run 
DataStage Job 127 Phantom 2287 
Program "DSD.StageRun": Line 657, COMMON size mismatch in subroutine "DSP.Close". 
Program "DSD.StageRun": Line 657, Unable to load file "DSP.Close". 
Program "DSD.StageRun": Line 657, Unable to load subroutine. 
Attempting to Cleanup after ABORT raised in stage DODSLDHFullStg..DPC_LDH 
Program "DSD.OnAbort": Line 164, COMMON size mismatch in subroutine "DSP.Close". 
Program "DSD.OnAbort": Line 164, Unable to load file "DSP.Close". 
Program "DSD.OnAbort": Line 164, Unable to load subroutine.

I'm really at a loss as to what to do next. Other than report this back to IBM, who still has our original case with 7.5.2 open, and try to get our SAs and HP involved. And I'm not really sure why I'm posting all this here, perhaps just a vain hope for a miracle cure from the group here, perhaps seeking a little guru technobabble...

"The positronic field generated by the interdimensional transformer reflex system is causing a pulsary overload in our aggregator's warp transducer, captain."

Or maybe it's just that misery loves company. :cry:

chulett · Post by **chulett** » Sat Apr 07, 2007 8:44 am

That was all from a job that wasn't causing problems before the 'downgrade'. I've just rerun the job noted earlier and it is still behaving in a very similar manner as before - aborting after producing an amount of XML:

Code: Select all

Message: 
DODSLDHAssetTerms..XML_Output_Ldh_terms: At row 14139141, link "Move_Ldh_Terms" 
Call to input link returned numeric error code: 19 

Message: 
DODSLDHAssetTerms..XML_Output_Ldh_terms: |-100| 

Message: 
Attempting to Cleanup after ABORT raised in stage DODSLDHAssetTerms..XML_Output_Ldh_terms 

Message: 
From previous run 
DataStage Job 116 Phantom 20369 
Abnormal termination of DataStage. 
Fault type is 10.  Layer type is BASIC run machine. 
Fault occurred in BASIC program DSD.Update at address 74.

No memory errors again, but still the odd 'error code: 19' and then the SIGBUS. Got almost all the way done, stopping after a little over 14M rows rather than a measly 2 million. [heavy sigh]

chulett · Post by **chulett** » Sun May 20, 2007 9:07 am

Thought I would revisit this and post the resolutions to all of these issues. The final piece of the puzzle was found in this thread - thanks to ttk1973 for posting that. Both of these seem to be specific to HP-UX 11.11 so keep that in mind if you think you are having a similar problem.

1) Bus Error / Core Dumps. This includes the "Fault Type is 10" messages mentioned earlier. Source? $LD_PRELOAD While it was not supposed to be set, turns out it had been. Removing it from the DataStage Server environment stops that particular dumpage.

2) COMMON size mismatch. As noted in the linked post, the problem was the version of the Oracle Client we were using. We've used 9.2.0.5 in the past and have been using 9.2.0.6 without issue for some time now. This new server was supposed to have been setup 'the same' as all of the other working ones, turns out it had not been. The 9.2.0.1 client the DBAs had installed has a known memory leak issue which results in interesting messages like the ones I posted. Odd thing is I was able to run quite a number of large Oracle processes using the old client but certain jobs would consistently fail. After upgrading the client to 9.2.0.6 this no longer occurs.

So thankfully, after making both changes, the Server is fully operational. :D