Lookup Preserve Partitioning Flag Warning

jerome_rajan · Post by **jerome_rajan** » Fri Sep 12, 2014 5:17 am

Hi,

My lookup is throwing the warning

Operator of type "APT_LUTCreateOp": will partition despite the
preserve-partitioning flag on the data set on input port 0.

I have set the preserver partitioning flag on both the stream as well as the reference links to 'clear' but I continue to receive the warning.

What might the cause be?

jerome_rajan · Post by **jerome_rajan** » Fri Sep 12, 2014 5:40 am

Changed lookup to join and receiving a warning on similar lines

Code: Select all

Operator of type "APT_JoinSubOperatorNC": will partition despite the
preserve-partitioning flag on the data set on input port 0.

buffer(3): When checking operator: Operator of type "APT_BufferOperator": will partition despite the
preserve-partitioning flag on the data set on input port 0.

jerome_rajan · Post by **jerome_rajan** » Fri Sep 12, 2014 5:41 am

And here's the output of APT_DUMP_SCORE

Code: Select all

main_program: This step has 30 datasets:
ds0: {op0[1p] (parallel APT_CombinedOperatorController(0):APT_TransformOperatorImplV0S13_NestedTableEDI852_Delimted_Transformer_13 in Transformer_13)
      eOther(APT_HashPartitioner { key={ value=MasterKey, 
        subArgs={ cs }
      }
})<>eCollectAny
      op1[6p] (parallel Join_139.lnkS10Data_Sort)}
ds1: {op0[1p] (parallel APT_CombinedOperatorController(0):APT_TransformOperatorImplV0S13_NestedTableEDI852_Delimted_Transformer_13 in Transformer_13)
      eOther(APT_HashPartitioner { key={ value=MasterKey, 
        subArgs={ cs }
      },
  key={ value=ChildKey, 
        subArgs={ cs }
      }
})<>eCollectAny
      op2[6p] (parallel APT_CombinedOperatorController(1):Column_Import_37)}
ds2: {op0[1p] (parallel APT_CombinedOperatorController(0):APT_TransformOperatorImplV0S13_NestedTableEDI852_Delimted_Transformer_13 in Transformer_13)
      eOther(APT_HashPartitioner { key={ value=MasterKey, 
        subArgs={ cs }
      },
  key={ value=ChildKey, 
        subArgs={ cs }
      }
})<>eCollectAny
      op3[6p] (parallel Column_Import_59)}
ds3: {op0[1p] (parallel APT_CombinedOperatorController(0):APT_TransformOperatorImplV0S13_NestedTableEDI852_Delimted_Transformer_13 in Transformer_13)
      eOther(APT_HashPartitioner { key={ value=MasterKey, 
        subArgs={ cs }
      },
  key={ value=ChildKey, 
        subArgs={ cs }
      }
})<>eCollectAny
      op12[6p] (parallel APT_CombinedOperatorController(2):Column_Import_60)}
ds4: {op1[6p] (parallel Join_139.lnkS10Data_Sort)
      [pp] eSame=>eCollectAny
      op4[6p] (parallel buffer(0))}
ds5: {op2[6p] (parallel APT_CombinedOperatorController(1):APT_TransformOperatorImplV0S71_NestedTableEDI852_Delimted_Transformer_71 in Transformer_71)
      eOther(APT_HashPartitioner { key={ value=MasterKey, 
        subArgs={ cs }
      }
})#>eCollectAny
      op6[6p] (parallel Join_139.DSLink136_Sort)}
ds6: {op2[6p] (parallel APT_CombinedOperatorController(1):APT_TransformOperatorImplV0S71_NestedTableEDI852_Delimted_Transformer_71 in Transformer_71)
      eOther(APT_HashPartitioner { key={ value=MasterKey }
})#>eCollectAny
      op8[6p] (parallel inserted tsort operator {key={value=MasterKey, subArgs={asc, nulls={value=first}, cs}}}(0) in Join_120)}
ds7: {op3[6p] (parallel Column_Import_59)
      eOther(APT_HashPartitioner { key={ value=MasterKey, 
        subArgs={ cs }
      },
  key={ value=ChildKey, 
        subArgs={ cs }
      }
})#>eCollectAny
      op5[6p] (parallel APT_CombinedOperatorController(3):Transformer_72.DSLink64_Sort)}
ds8: {op4[6p] (parallel buffer(0))
      [pp] eSame=>eCollectAny
      op9[6p] (parallel APT_JoinSubOperatorNC(0) in Join_139)}
ds9: {op5[6p] (parallel APT_CombinedOperatorController(3):APT_TransformOperatorImplV0S72_NestedTableEDI852_Delimted_Transformer_72 in Transformer_72)
      eOther(APT_HashPartitioner { key={ value=MasterKey, 
        subArgs={ cs }
      }
})#>eCollectAny
      op10[6p] (parallel Join_139.DSLink155_Sort)}
ds10: {op6[6p] (parallel Join_139.DSLink136_Sort)
      [pp] eSame=>eCollectAny
      op7[6p] (parallel buffer(1))}
ds11: {op7[6p] (parallel buffer(1))
      [pp] eSame=>eCollectAny
      op9[6p] (parallel APT_JoinSubOperatorNC(0) in Join_139)}
ds12: {op8[6p] (parallel inserted tsort operator {key={value=MasterKey, subArgs={asc, nulls={value=first}, cs}}}(0) in Join_120)
      [pp] eSame=>eCollectAny
      op15[6p] (parallel buffer(5))}
ds13: {op9[6p] (parallel APT_JoinSubOperatorNC(0) in Join_139)
      [pp] eOther(APT_HashPartitioner { key={ value=MasterKey }
})#>eCollectAny
      op13[6p] (parallel buffer(3))}
ds14: {op10[6p] (parallel Join_139.DSLink155_Sort)
      [pp] eSame=>eCollectAny
      op11[6p] (parallel buffer(2))}
ds15: {op11[6p] (parallel buffer(2))
      [pp] eSame=>eCollectAny
      op14[6p] (parallel APT_JoinSubOperatorNC(1) in Join_139)}
ds16: {op12[6p] (parallel APT_CombinedOperatorController(2):APT_TransformOperatorImplV0S73_NestedTableEDI852_Delimted_Transformer_73 in Transformer_73)
      eOther(APT_HashPartitioner { key={ value=MasterKey }
})#>eCollectAny
      op20[6p] (parallel inserted tsort operator {key={value=MasterKey, subArgs={asc, nulls={value=first}, cs}}}(0) in Join_127)}
ds17: {op13[6p] (parallel buffer(3))
      [pp] eSame=>eCollectAny
      op14[6p] (parallel APT_JoinSubOperatorNC(1) in Join_139)}
ds18: {op14[6p] (parallel APT_JoinSubOperatorNC(1) in Join_139)
      [pp] eAny=>eCollectAny
      op16[6p] (parallel Oracle_Connector_110)}
ds19: {op15[6p] (parallel buffer(5))
      [pp] eSame=>eCollectAny
      op19[6p] (parallel APT_JoinSubOperatorNC in Join_120)}
ds20: {op16[6p] (parallel Oracle_Connector_110)
      [pp] eSame=>eCollectAny
      op17[6p] (parallel Copy_148)}
ds21: {op17[6p] (parallel Copy_148)
      eOther(APT_HashPartitioner { key={ value=MasterKey }
})#>eCollectAny
      op18[6p] (parallel buffer(4))}
ds22: {op18[6p] (parallel buffer(4))
      [pp] eSame=>eCollectAny
      op19[6p] (parallel APT_JoinSubOperatorNC in Join_120)}
ds23: {op19[6p] (parallel APT_JoinSubOperatorNC in Join_120)
      [pp] eAny=>eCollectAny
      op21[6p] (parallel Oracle_Connector_6)}
ds24: {op20[6p] (parallel inserted tsort operator {key={value=MasterKey, subArgs={asc, nulls={value=first}, cs}}}(0) in Join_127)
      [pp] eSame=>eCollectAny
      op24[6p] (parallel buffer(7))}
ds25: {op21[6p] (parallel Oracle_Connector_6)
      [pp] eSame=>eCollectAny
      op22[6p] (parallel Copy_150)}
ds26: {op22[6p] (parallel Copy_150)
      eOther(APT_HashPartitioner { key={ value=MasterKey }
})#>eCollectAny
      op23[6p] (parallel buffer(6))}
ds27: {op23[6p] (parallel buffer(6))
      [pp] eSame=>eCollectAny
      op25[6p] (parallel APT_JoinSubOperatorNC in Join_127)}
ds28: {op24[6p] (parallel buffer(7))
      [pp] eSame=>eCollectAny
      op25[6p] (parallel APT_JoinSubOperatorNC in Join_127)}
ds29: {op25[6p] (parallel APT_JoinSubOperatorNC in Join_127)
      [pp] eAny=>eCollectAny
      op26[6p] (parallel Copy_of_Oracle_Connector_6)}
It has 27 operators:
op0[1p] {(parallel APT_CombinedOperatorController(0):
      (Sequential_File_0)
      (APT_TransformOperatorImplV0S13_NestedTableEDI852_Delimted_Transformer_13 in Transformer_13)
    ) on nodes (
      node1[op0,p0]
    )}
op1[6p] {(parallel Join_139.lnkS10Data_Sort)
    on nodes (
      node1[op1,p0]
      node2[op1,p1]
      node3[op1,p2]
      node4[op1,p3]
      node5[op1,p4]
      node6[op1,p5]
    )}
op2[6p] {(parallel APT_CombinedOperatorController(1):
      (Column_Import_37)
      (APT_TransformOperatorImplV0S71_NestedTableEDI852_Delimted_Transformer_71 in Transformer_71)
    ) on nodes (
      node1[op2,p0]
      node2[op2,p1]
      node3[op2,p2]
      node4[op2,p3]
      node5[op2,p4]
      node6[op2,p5]
    )}
op3[6p] {(parallel Column_Import_59)
    on nodes (
      node1[op3,p0]
      node2[op3,p1]
      node3[op3,p2]
      node4[op3,p3]
      node5[op3,p4]
      node6[op3,p5]
    )}
op4[6p] {(parallel buffer(0))
    on nodes (
      node1[op4,p0]
      node2[op4,p1]
      node3[op4,p2]
      node4[op4,p3]
      node5[op4,p4]
      node6[op4,p5]
    )}
op5[6p] {(parallel APT_CombinedOperatorController(3):
      (Transformer_72.DSLink64_Sort)
      (APT_TransformOperatorImplV0S72_NestedTableEDI852_Delimted_Transformer_72 in Transformer_72)
    ) on nodes (
      node1[op5,p0]
      node2[op5,p1]
      node3[op5,p2]
      node4[op5,p3]
      node5[op5,p4]
      node6[op5,p5]
    )}
op6[6p] {(parallel Join_139.DSLink136_Sort)
    on nodes (
      node1[op6,p0]
      node2[op6,p1]
      node3[op6,p2]
      node4[op6,p3]
      node5[op6,p4]
      node6[op6,p5]
    )}
op7[6p] {(parallel buffer(1))
    on nodes (
      node1[op7,p0]
      node2[op7,p1]
      node3[op7,p2]
      node4[op7,p3]
      node5[op7,p4]
      node6[op7,p5]
    )}
op8[6p] {(parallel inserted tsort operator {key={value=MasterKey, subArgs={asc, nulls={value=first}, cs}}}(0) in Join_120)
    on nodes (
      node1[op8,p0]
      node2[op8,p1]
      node3[op8,p2]
      node4[op8,p3]
      node5[op8,p4]
      node6[op8,p5]
    )}
op9[6p] {(parallel APT_JoinSubOperatorNC(0) in Join_139)
    on nodes (
      node1[op9,p0]
      node2[op9,p1]
      node3[op9,p2]
      node4[op9,p3]
      node5[op9,p4]
      node6[op9,p5]
    )}
op10[6p] {(parallel Join_139.DSLink155_Sort)
    on nodes (
      node1[op10,p0]
      node2[op10,p1]
      node3[op10,p2]
      node4[op10,p3]
      node5[op10,p4]
      node6[op10,p5]
    )}
op11[6p] {(parallel buffer(2))
    on nodes (
      node1[op11,p0]
      node2[op11,p1]
      node3[op11,p2]
      node4[op11,p3]
      node5[op11,p4]
      node6[op11,p5]
    )}
op12[6p] {(parallel APT_CombinedOperatorController(2):
      (Column_Import_60)
      (APT_TransformOperatorImplV0S73_NestedTableEDI852_Delimted_Transformer_73 in Transformer_73)
    ) on nodes (
      node1[op12,p0]
      node2[op12,p1]
      node3[op12,p2]
      node4[op12,p3]
      node5[op12,p4]
      node6[op12,p5]
    )}
op13[6p] {(parallel buffer(3))
    on nodes (
      node1[op13,p0]
      node2[op13,p1]
      node3[op13,p2]
      node4[op13,p3]
      node5[op13,p4]
      node6[op13,p5]
    )}
op14[6p] {(parallel APT_JoinSubOperatorNC(1) in Join_139)
    on nodes (
      node1[op14,p0]
      node2[op14,p1]
      node3[op14,p2]
      node4[op14,p3]
      node5[op14,p4]
      node6[op14,p5]
    )}
op15[6p] {(parallel buffer(5))
    on nodes (
      node1[op15,p0]
      node2[op15,p1]
      node3[op15,p2]
      node4[op15,p3]
      node5[op15,p4]
      node6[op15,p5]
    )}
op16[6p] {(parallel Oracle_Connector_110)
    on nodes (
      node1[op16,p0]
      node2[op16,p1]
      node3[op16,p2]
      node4[op16,p3]
      node5[op16,p4]
      node6[op16,p5]
    )}
op17[6p] {(parallel Copy_148)
    on nodes (
      node1[op17,p0]
      node2[op17,p1]
      node3[op17,p2]
      node4[op17,p3]
      node5[op17,p4]
      node6[op17,p5]
    )}
op18[6p] {(parallel buffer(4))
    on nodes (
      node1[op18,p0]
      node2[op18,p1]
      node3[op18,p2]
      node4[op18,p3]
      node5[op18,p4]
      node6[op18,p5]
    )}
op19[6p] {(parallel APT_JoinSubOperatorNC in Join_120)
    on nodes (
      node1[op19,p0]
      node2[op19,p1]
      node3[op19,p2]
      node4[op19,p3]
      node5[op19,p4]
      node6[op19,p5]
    )}
op20[6p] {(parallel inserted tsort operator {key={value=MasterKey, subArgs={asc, nulls={value=first}, cs}}}(0) in Join_127)
    on nodes (
      node1[op20,p0]
      node2[op20,p1]
      node3[op20,p2]
      node4[op20,p3]
      node5[op20,p4]
      node6[op20,p5]
    )}
op21[6p] {(parallel Oracle_Connector_6)
    on nodes (
      node1[op21,p0]
      node2[op21,p1]
      node3[op21,p2]
      node4[op21,p3]
      node5[op21,p4]
      node6[op21,p5]
    )}
op22[6p] {(parallel Copy_150)
    on nodes (
      node1[op22,p0]
      node2[op22,p1]
      node3[op22,p2]
      node4[op22,p3]
      node5[op22,p4]
      node6[op22,p5]
    )}
op23[6p] {(parallel buffer(6))
    on nodes (
      node1[op23,p0]
      node2[op23,p1]
      node3[op23,p2]
      node4[op23,p3]
      node5[op23,p4]
      node6[op23,p5]
    )}
op24[6p] {(parallel buffer(7))
    on nodes (
      node1[op24,p0]
      node2[op24,p1]
      node3[op24,p2]
      node4[op24,p3]
      node5[op24,p4]
      node6[op24,p5]
    )}
op25[6p] {(parallel APT_JoinSubOperatorNC in Join_127)
    on nodes (
      node1[op25,p0]
      node2[op25,p1]
      node3[op25,p2]
      node4[op25,p3]
      node5[op25,p4]
      node6[op25,p5]
    )}
op26[6p] {(parallel Copy_of_Oracle_Connector_6)
    on nodes (
      node1[op26,p0]
      node2[op26,p1]
      node3[op26,p2]
      node4[op26,p3]
      node5[op26,p4]
      node6[op26,p5]
    )}
It runs 157 processes on 6 nodes.

vinothkumar · Post by **vinothkumar** » Fri Sep 12, 2014 5:44 am

Did you set preserver partitioning flag as clear in all stages present in your deisgn.

jerome_rajan · Post by **jerome_rajan** » Fri Sep 12, 2014 1:22 pm

All the stages upstream of the lookup, yes.

priyadarshikunal · Post by **priyadarshikunal** » Mon Sep 15, 2014 4:27 am

Try using "set" instead of "propagate" partitioning.

jerome_rajan · Post by **jerome_rajan** » Mon Sep 15, 2014 10:40 pm

Wouldn't 'SET' try to force the next stage to preserve partitioning, thereby producing the same warning again? I've set the flag to 'CLEAR' to ensure that the stage is impervious to the partitioning of downstream stages

priyadarshikunal · Post by **priyadarshikunal** » Wed Sep 17, 2014 8:25 am

look-up expects the data to be in entire partitioning and partitioned if required, which I don't think any stage does for you by default. Hence you see the warning in lookup stage as it will partition even if preserve partitioning is set. Set the partitioning in lookup stage to entire, and it will go away. And if partitioning is set in lookup to any other partitioning, I don't think it will throw that warning. You can always try it.

jerome_rajan · Post by **jerome_rajan** » Wed Sep 17, 2014 9:45 pm

Operator of type "APT_JoinSubOperatorNC": will partition despite the
preserve-partitioning flag on the data set on input port 0.

buffer(3): When checking operator: Operator of type "APT_BufferOperator": will partition despite the
preserve-partitioning flag on the data set on input port 0.

Like I mentioned before, I had removed the lookup stage and replaced it with a JOIN stage and I'm still receiving a warning on similar lines. In my opinion, I have checked off almost every basic troubleshooting step. I have suppressed the warning for now but I really want to know the reason behind this warning.