Posted: Sat Oct 28, 2017 4:51 pm
Neeraj,
I am a bit puzzled by your description. In it you state that DataStage is on-site at the customer, as well as the source data. If that is the case, then all data is already present on-site.
However, to answer what I believe to be the underlying question - the DataStage server will dump data to the scratch location(s) whenever the memory buffers between stages (default is 2 MB) exceeds capacity during processing. Other operations, like sorts or aggregations can also dump data to disk as required to handle large datasets.
Now if you use a file from Azure as a source or reference, like during a change capture operation or a lookup - then data from that table is definitely on the server and could be dumped to disk temporarily during buffering operations.
If the Azure data files are true targets, nothing is retrieved from them.
I am a bit puzzled by your description. In it you state that DataStage is on-site at the customer, as well as the source data. If that is the case, then all data is already present on-site.
However, to answer what I believe to be the underlying question - the DataStage server will dump data to the scratch location(s) whenever the memory buffers between stages (default is 2 MB) exceeds capacity during processing. Other operations, like sorts or aggregations can also dump data to disk as required to handle large datasets.
Now if you use a file from Azure as a source or reference, like during a change capture operation or a lookup - then data from that table is definitely on the server and could be dumped to disk temporarily during buffering operations.
If the Azure data files are true targets, nothing is retrieved from them.