DSXchange

Make sure you understand exactly what it is you are turning on before you flip that switch, my friend. An exact search for 64BIT_FILES should turn up all of the relevant conversations on the topic. I personally would suggest handling these beefy hashed files on an individual basis, converting just t...

There was no "not" in anything I said, I suggested you check while they run as well as before/after. A complete picture.

Everything else I'll leave to Zenith.

When you repeat them, I'd suggest a couple of things. One would be to not just count the entries but actually list them out, in date/time order, so you know exactly what is being created there. And do this while they run, not just before and after. A simple trick might be to just look at the most re...

Looks like the tests that Pandeesh ran need to be revisited.

... and was the case up through 7.5.x as well, which is what Pandeesh is running. I doubt it changed in 8.x either.

In either version you can use stage variables... as long as the input is sorted in a usable fashion.

Shared containers are re-usable components, yes. Local containers are just groupings of stages in the job, a mechanism to make the main canvas presentation less "cluttered" and more streamlined with the detail hidden away in those containers. If you need to know the gory details, or do wo...

Well... let's move from belief to proof. Test it. This won't take five days to resolve, simply pick a job and run it five times in a row. Monitor the contents of the phantom directory before, during and after the job runs. Run this test more than once & let us know what you find. For the record,...

From all of the discussions here on what their purpose is and examining the naming convention they use, what do you think the answer is? And I'm curious, why not simply test all these hypothetical scenarios yourself?

Care to share the workaround?

You are completely over-thinking this. There is a long standing Best Practice that encourages you to manage the number of entries in the Phantom directory. I've never worried about how many entries are there or if there is some magical threshold that must be crossed before it becomes an issue. I've ...

I'm assuming this would be complicated on the PX side by the (apparent) need to retain the original input order... which leads me to think the dreaded "fork join" design would be appropriate in that case. Somehow.

Sure, a job could run the O/S command required, same as a script. However, the question as to why you don't see any entries in your Phantom directory while jobs are running isn't being addressed.

Me, I don't see how that's possible...

What kind of a lookup are you doing - sparse?

Server job with a hashed file that just stores the key or record number or whatever uniquely identifies a record. Do a lookup to the hashed file and if the key does not exist, write it to the hashed file and pass it out the output link. If you get a hit on the hashed file, do nothing, as in do not p...