Hi,
I want to know what's the difference between partition and repartition in parallel jobs.
What I know is that partition split data into separate sets and each set is handled by a separate instance of the job stages processed by a separate node.
Repartitioning allow us to repartition data between stages.
But in the above definition of partitioning do we repartition also data between stages? .. or not?
conceptual difference between partition and repartition
Moderators: chulett, rschirm, roy
conceptual difference between partition and repartition
Last edited by Enzopre on Sat Mar 02, 2013 3:25 pm, edited 1 time in total.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Partitioning occurs when the upstream stage is sequential and the downstream stage executes in parallel mode.
Re-partitioning occurs when the upstream stage is executing in parallel mode and the downstream stage is executing in parallel mode, but specifies a different partitioning algorithm from that used in the upstream stage (implicitly or explicitly).
If the downstream stage does not specify a different partitioning algorithm then no re-partitioning occurs.
Re-partitioning occurs when the upstream stage is executing in parallel mode and the downstream stage is executing in parallel mode, but specifies a different partitioning algorithm from that used in the upstream stage (implicitly or explicitly).
If the downstream stage does not specify a different partitioning algorithm then no re-partitioning occurs.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Download the IBM Redbook pdf on InfoSphere DataStage Parallel Framework Standard Practices, lots of great info there and a whole chapter on partitioning.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers