Two Source Match Stage, long running job and fails at last
Posted: Wed Feb 07, 2018 2:13 pm
Hi All,
Thanks to you all helping me to resolve this issue.
I have a job using "Two-Source Match" Stage, it has two input link for data match. Now, Job is able to fetch data from two source table and it is getting stuck at "Two-Source Match" Stage. Then job runs for long time and at end it fails with following log details. We have 20 GB Scratch space allocated. Appreciate your help.
Following are log details
---------------------------------------------------
Source1TBL_Data,0: Number of rows fetched on the current node: 1679746.
TwoSourceMatch_StageName,0: Unsupported close in APT_FileBufferOutput::spillToNextFile(): Input/output error.
TwoSourceMatch_StageName,0: write failed: Output file full, and no more output files
TwoSourceMatch_StageName,0: Failure during execution of operator logic.
TwoSourceMatch_StageName,0: Fatal Error: Tsort merger aborting: mergeOneRecord() punted
TwoSourceMatch_StageName,0: Failure during execution of operator logic.
TwoSourceMatch_StageName,0: Fatal Error: Pipe read failed: short read
node_node1: Player 21 terminated unexpectedly.
main_program: APT_PMsectionLeader(1, node1), player 21 - Unexpected exit status 1.
APT_PMsectionLeader(2, node2), player 11 - Unexpected exit status 1.
APT_PMsectionLeader(2, node2), player 12 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 10 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 11 - Unexpected exit status 1.
APT_PMsectionLeader(1, node1), player 12 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 11 - Unexpected exit status 1.
APT_PMsectionLeader(2, node2), player 15 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 13 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 14 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 13 - Unexpected exit status 1.
APT_PMsectionLeader(1, node1), player 14 - Unexpected exit status 1.
APT_PMsectionLeader(1, node1), player 15 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 14 - Unexpected exit status 1.
APT_PMsectionLeader(2, node2), player 17 - Unexpected exit status 1.
APT_PMsectionLeader(2, node2), player 18 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 15 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 16 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 17 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 15 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 16 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 17 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 18 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 19 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 20 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 21 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 22 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 23 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 24 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 25 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 26 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 27 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 28 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 29 - Unexpected exit status 1.
APT_PMsectionLeader(2, node2), player 20 - Unexpected exit status 1.
APT_PMsectionLeader(2, node2), player 21 - Unexpected exit status 1.
APT_PMsectionLeader(2, node2), player 22 - Unexpected exit status 1.
APT_PMsectionLeader(2, node2), player 23 - Unexpected exit status 1.
APT_PMsectionLeader(2, node2), player 24 - Unexpected exit status 1.
APT_PMsectionLeader(2, node2), player 25 - Unexpected exit status 1.
APT_PMsectionLeader(2, node2), player 26 - Unexpected exit status 1.
APT_PMsectionLeader(2, node2), player 27 - Unexpected exit status 1.
APT_PMsectionLeader(2, node2), player 28 - Unexpected exit status 1.
APT_PMsectionLeader(2, node2), player 29 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 18 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 19 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 20 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 21 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 22 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 23 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 24 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 25 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 26 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 27 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 28 - Unexpected exit status 1.
APT_PMsectionLeader(1, node1), player 16 - Unexpected exit status 1.
APT_PMsectionLeader(1, node1), player 17 - Unexpected exit status 1.
APT_PMsectionLeader(1, node1), player 18 - Unexpected exit status 1.
APT_PMsectionLeader(1, node1), player 19 - Unexpected exit status 1.
APT_PMsectionLeader(1, node1), player 20 - Unexpected exit status 1.
APT_PMsectionLeader(1, node1), player 22 - Unexpected exit status 1.
APT_PMsectionLeader(1, node1), player 23 - Unexpected exit status 1.
APT_PMsectionLeader(1, node1), player 24 - Unexpected exit status 1.
APT_PMsectionLeader(1, node1), player 25 - Unexpected exit status 1.
APT_PMsectionLeader(1, node1), player 26 - Unexpected exit status 1.
APT_PMsectionLeader(1, node1), player 27 - Unexpected exit status 1.
APT_PMsectionLeader(1, node1), player 28 - Unexpected exit status 1.
APT_PMsectionLeader(1, node1), player 29 - Unexpected exit status 1.
APT_PMsectionLeader(2, node2), player 30 - Unexpected exit status 1.
APT_PMsectionLeader(2, node2), player 31 - Unexpected exit status 1.
APT_PMsectionLeader(2, node2), player 32 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 30 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 31 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 32 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 29 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 30 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 31 - Unexpected exit status 1.
APT_PMsectionLeader(1, node1), player 30 - Unexpected exit status 1.
APT_PMsectionLeader(1, node1), player 31 - Unexpected exit status 1.
APT_PMsectionLeader(1, node1), player 32 - Unexpected exit status 1.
APT_PMsectionLeader(2, node2), player 35 - Unexpected exit status 1.
APT_PMsectionLeader(2, node2), player 36 - Unexpected exit status 1.
APT_PMsectionLeader(1, node1), player 35 - Unexpected exit status 1.
APT_PMsectionLeader(1, node1), player 36 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 34 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 35 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 35 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 36 - Unexpected exit status 1.
TwoSourceMatch_StageName,0: Input 0 consumed 100714435 records.
Thanks to you all helping me to resolve this issue.
I have a job using "Two-Source Match" Stage, it has two input link for data match. Now, Job is able to fetch data from two source table and it is getting stuck at "Two-Source Match" Stage. Then job runs for long time and at end it fails with following log details. We have 20 GB Scratch space allocated. Appreciate your help.
Following are log details
---------------------------------------------------
Source1TBL_Data,0: Number of rows fetched on the current node: 1679746.
TwoSourceMatch_StageName,0: Unsupported close in APT_FileBufferOutput::spillToNextFile(): Input/output error.
TwoSourceMatch_StageName,0: write failed: Output file full, and no more output files
TwoSourceMatch_StageName,0: Failure during execution of operator logic.
TwoSourceMatch_StageName,0: Fatal Error: Tsort merger aborting: mergeOneRecord() punted
TwoSourceMatch_StageName,0: Failure during execution of operator logic.
TwoSourceMatch_StageName,0: Fatal Error: Pipe read failed: short read
node_node1: Player 21 terminated unexpectedly.
main_program: APT_PMsectionLeader(1, node1), player 21 - Unexpected exit status 1.
APT_PMsectionLeader(2, node2), player 11 - Unexpected exit status 1.
APT_PMsectionLeader(2, node2), player 12 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 10 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 11 - Unexpected exit status 1.
APT_PMsectionLeader(1, node1), player 12 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 11 - Unexpected exit status 1.
APT_PMsectionLeader(2, node2), player 15 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 13 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 14 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 13 - Unexpected exit status 1.
APT_PMsectionLeader(1, node1), player 14 - Unexpected exit status 1.
APT_PMsectionLeader(1, node1), player 15 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 14 - Unexpected exit status 1.
APT_PMsectionLeader(2, node2), player 17 - Unexpected exit status 1.
APT_PMsectionLeader(2, node2), player 18 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 15 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 16 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 17 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 15 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 16 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 17 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 18 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 19 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 20 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 21 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 22 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 23 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 24 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 25 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 26 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 27 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 28 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 29 - Unexpected exit status 1.
APT_PMsectionLeader(2, node2), player 20 - Unexpected exit status 1.
APT_PMsectionLeader(2, node2), player 21 - Unexpected exit status 1.
APT_PMsectionLeader(2, node2), player 22 - Unexpected exit status 1.
APT_PMsectionLeader(2, node2), player 23 - Unexpected exit status 1.
APT_PMsectionLeader(2, node2), player 24 - Unexpected exit status 1.
APT_PMsectionLeader(2, node2), player 25 - Unexpected exit status 1.
APT_PMsectionLeader(2, node2), player 26 - Unexpected exit status 1.
APT_PMsectionLeader(2, node2), player 27 - Unexpected exit status 1.
APT_PMsectionLeader(2, node2), player 28 - Unexpected exit status 1.
APT_PMsectionLeader(2, node2), player 29 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 18 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 19 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 20 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 21 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 22 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 23 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 24 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 25 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 26 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 27 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 28 - Unexpected exit status 1.
APT_PMsectionLeader(1, node1), player 16 - Unexpected exit status 1.
APT_PMsectionLeader(1, node1), player 17 - Unexpected exit status 1.
APT_PMsectionLeader(1, node1), player 18 - Unexpected exit status 1.
APT_PMsectionLeader(1, node1), player 19 - Unexpected exit status 1.
APT_PMsectionLeader(1, node1), player 20 - Unexpected exit status 1.
APT_PMsectionLeader(1, node1), player 22 - Unexpected exit status 1.
APT_PMsectionLeader(1, node1), player 23 - Unexpected exit status 1.
APT_PMsectionLeader(1, node1), player 24 - Unexpected exit status 1.
APT_PMsectionLeader(1, node1), player 25 - Unexpected exit status 1.
APT_PMsectionLeader(1, node1), player 26 - Unexpected exit status 1.
APT_PMsectionLeader(1, node1), player 27 - Unexpected exit status 1.
APT_PMsectionLeader(1, node1), player 28 - Unexpected exit status 1.
APT_PMsectionLeader(1, node1), player 29 - Unexpected exit status 1.
APT_PMsectionLeader(2, node2), player 30 - Unexpected exit status 1.
APT_PMsectionLeader(2, node2), player 31 - Unexpected exit status 1.
APT_PMsectionLeader(2, node2), player 32 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 30 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 31 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 32 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 29 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 30 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 31 - Unexpected exit status 1.
APT_PMsectionLeader(1, node1), player 30 - Unexpected exit status 1.
APT_PMsectionLeader(1, node1), player 31 - Unexpected exit status 1.
APT_PMsectionLeader(1, node1), player 32 - Unexpected exit status 1.
APT_PMsectionLeader(2, node2), player 35 - Unexpected exit status 1.
APT_PMsectionLeader(2, node2), player 36 - Unexpected exit status 1.
APT_PMsectionLeader(1, node1), player 35 - Unexpected exit status 1.
APT_PMsectionLeader(1, node1), player 36 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 34 - Unexpected exit status 1.
APT_PMsectionLeader(4, node4), player 35 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 35 - Unexpected exit status 1.
APT_PMsectionLeader(3, node3), player 36 - Unexpected exit status 1.
TwoSourceMatch_StageName,0: Input 0 consumed 100714435 records.