DSXchange

Thanks Ray. That's more the kind of information I was fishing for.

Wow... I hope all of that was for the home audience, 'cuz I really do understand how the dreaded thing works. Well, expect for the 'in memory versus temporary file storage' bit which started this all off. I know the whole 'buckets' thing and how it can (in essence) use one bucket when the input data...

Don't use a hash file - it's not needed here (unless it is being used for something other than the date handing we're discussing) and it's just adding unneeded complications! Simply flow your (ordered) data through and use the Stage Variable. Does that sound feasible? Do you really need to do a loo...

Unless I'm really missing something here, which could be the case... One Stage Variable should be fine. It looks like your input data is already sorted, if it is coming in exactly like you showed in your first post. If not, yes you would have to ensure it was sorted properly. Then, your Stage Variab...

Thanks for the clarification. I guess I should have more properly said 'grouping' rather than 'sorting', but the answer seems to be that it all happens in memory rather than using any temporary files like the Sort stage does. So, this leads me to a performance question... When faced with a situation...

This seems like it would be simple to do with a Stage Variable. If 'Initial Entry' is a 0, set the stage variable to the Date field on that line. Use the Stage Variable to populate the flat file output field. Keep using it until you find another 0, then switch it again.

No hash file needed.

In the actual Sort stage, you can specify where it puts the temporary working files (soa*) that it needs. I'm wondering how that works with the Aggregator when it needs to sort data as well as group it. There is no corresponding option in the stage to control where it puts things. I just want to mak...

Don't think so. The OP specified Server jobs and your $APT environment variable is only valid for PX jobs, so I really don't think it will help. Besides, I don't believe they are saying that all of the updates are being rejected (i.e. - failed), but from my reading all update rows, in spite of being...

Wow - something from the Oliver archives. I wasn't sure people actually searched there.

By "full", I'm assuming they mean they only want to process (non-empty) files with records "in subsequent jobs" and have built a methodology to detect / avoid / not process the empty ones they have created.

Ah... the dangers of not disabling Smilies when you post.

Actually you can easily solve this problem with following steps: 1. write a unix script which checks for a file size in bytes 2. Delete it if size is greater than zero 3. Excute this script using after stage subroutine and send filename as argument to subroutine 4. Use ExecSH command to execute scr...

The simple fact of the Sequential stage starting and stopping creates the file as it is opened and then closed, so as Ken said it can't be avoided. People that are bothered by the empty files usually write an After Job script to test and remove them if they are zero bytes. That's a pretty simple thi...

Doh! Should have mentioned that little tid-bit.

I would bet Ken is correct in that the Reject link is the second of the three output links from the Transformer, with Insert being first and Update being third. Make sure you understand that output link ordering is very important in general and especially when using a Reject link. A row qualifies fo...

I agree that Pivot Stage can be used to Pivot one row into many rows. But, this is the otherway round. One column into many columns. I'm not really following what you mean by 'the other way round'. You can turn one column into many columns by splitting the data (using something like FIELD or via su...