DSXchange

That's one way, sure: 1 = new group, 0 = not. Or a simple stage variable holding the concatenated values from all fields that make up the 'group'. First time checking a 'previous' value against a current value?

Of course, no sense evaluating them once per record when once per job run will do.

Always best to fully understand the things you are suggesting and a "-9" kill signal should never be on the top of anyone's list of Things To Try.

Of course, and the "short" version is the first six characters followed by "~1" for the first unique occurance and incrementing from there. So more than likely "format~1" for "formation DS". Plus the extension, if any.

Ah... didn't notice both hiding in there. Couldn't you use CHAR(34) and CHAR(39) to represent them in the expression?

Ummm... nice color choice. Had to highlight that last sentence to be able to read it.

There's no magic answer and many factors, max and average record length versus network packet size being the first that come to mind along with things like the concurrency issue you noted. You'll also need to worry about Transaction Size and how it compares to the Array Size unless you stick with a ...

No, one counter should be fine, alone with a copy of the 'previous' group. Set the counter to one every time the group changes, else increment it. Constrain the output to when the counter is <= 5.

Then don't try using 'the fixed record type'. Can you post the FD or layout of the file you are trying to read? A handful of sample records would help, too.

Use single quotes for the outside string delimiter, then the double just becomes data.

Yup, pretty standard 'group change detection' and you want to make sure you pass no more than 5 records per 'group'.

Hmmm... needing to keep X duplicates would probably mean via stage variables in a Transformer, assuming they're sorted properly first.

First suggestion - please start your own post with all your particular details.

The 'Command' stage was/is Windows only.

Guess not. Seems to be saying that 3M rows or so in, you've got something it doesn't think is a valid timestamp.