DSXchange

There's no reason to argue that partitioning loses sorted order. The data on each partition continue to flow in the same order in which they occurred prior to the partitioner.

Remove Duplicates selecting Last (as described in my earlier post in this thread) would be more efficient than Aggregator.

As always, "it depends". Mainly on the size of the reference data.

The number of readers per node determines the size of each "chunk". For example if you have two nodes and two readers per node, then each "chunk" will be, as closely as possible, 25% of the rows in the file.

Yes and no. Yes. The values can certainly be written into Decimal data type. No. Decimal numbers are stored in binary format. The only data types in which you can store leading zeroes are string data types. When View Data displays Decimal data, it adds leading zeroes so that you can be reminded of t...

Welcome aboard. :D Are the output columns from the Lookup stage defined as nullable when you're not using the Transformer stage? Lookup stage does not - of itself - drop null non-key columns, but no stage will write null into a Not Null column unless null-handling properties (such as substitute "in-...

Use a Remove Duplicates stage based on both columns. Ensure that data are partitioned correctly (on both columns).

Build the output rows in a Transformer stage and use an output constraint of emp_id = 1043 or emp_id = 1059. (I've assumed your input is sorted as shown - if not, sort it.) Some of these rows will be partially incomplete. The final row in each group (that is, for each keyID) will be complete. Downst...

What's in the "notes below" regarding long-waiting TCP connections?

The new tool (probably coming out with version 8.1) will - initially at least - provide an interface to Clear Case, according to one of the presentations at IoD 2007.

"Blocking" stage types, like Sort and Aggregator, necessarily "interfere with pipeline partitioning" - it's unavoidable. Basically they can not output their first row as soon as the first input row has arrived - they have to wait until the sorting or grouping has been completed. Partitioning is unre...

Sort in DataStage, using a key-based partitioning algorithm (Hash or Modulus) on the sort key(s). You may be surprised how efficient it is. Make sure you have plenty of scratch disk space.

At DSXchange we prefer a more professional standard of written English, the same as you would use when documenting your work. DSXchange is not a mobile telephone. People whose first language is not English have a hard enough time understanding well-written English - they experience added difficulty...

Try derivations like

Technically the parentheses around the concatenation expression are not required - I use them merely to show that a sub-expression exists.

what is the main difference between Lookup and Join Stage. There are many differences. Who are we to decide which is the main one? Is this an interview question? They are different. That's enough. A spanner and a wrench are different, though they perform similar tasks. What the main difference betw...