DSXchange

We ran into an issue where a query was still running (for awhile) even though the job had been stopped (through the Director), query modified, and the job compiled and re-run. Any advice on how to avoid this in the future? There was an issue with this query - long running and so testing it outside o...

We recently upgraded to v8.5 and for some reason, a job which we imported into v8.5 but was created in v8.0, is no longer working. No record is matching (even though there are records which should match). Just wondering if there is a known issue or change we need to make in stage for v8.5....we have...

chulett wrote:You haven't told us what command you are using in the ESS. Grep supports -i for "case insensitive" so an ls piped to grep could filter out all of the ".xml" files regardless of case. You could ...

Ok, I was only using the ls command...changed it to use grep as well. Thanks

Is there any way to tell the External source stage to grab filenames without being case-sensitive? ie. .xml, .XML, .xMl

I can request that all files are the same case, but figured i'd try to find another solution.

Any advice???

Thanks in advance,

Nicole

Ok, I think I am close to fully understanding - we create a new config file with additional nodes. And also, on the advanced tab of the WSTransformer - should we select 'Parallel'? Is running a configuration with 4 nodes the same as running 4 instances of your job? ....Exactly... you are now on the ...

Sorry for the delayed response - we are still trying to optimize and have made some improvements - but still need some guidance. To answer your questions: - The # of rows varies, but during peak processing the highest # was approx. 653K - I see that we shouldn't use WSTransformer for high volumes. I...

The DB2 stage extracts data - pretty simple join - select with some where criteria on relational columns - and pretty much just passing the data to the web service (web service transformer stage). Trying to optimize as the job is taking a while - so we have been trying to figure out if its in the ex...

Hello everyone - just trying to figure out where the bottleneck is in a Datastage job - when using the DB2 API stage to extract data (cannot use enterprise at this time) - and then passing the data to the web service. As the data is extracted is it passed to the web service immediately or does it ex...

Ok, and just to cover all bases - even if you shut down your machine - it will continue running without a problem. Thanks.

anbu wrote:Yes your job will run even if you close your job in designer

If I run a job but close the Designer or even just the job (so that it is not read-only and someone else can open it) - will the job still run and complete? From what I have seen, yes, you can do this and the job will still complete - the job runs on the server and it does not affect it. I just want...

This is the solution I decided to use - Using the sort stage - sorted by the 2 columns that I need to keep 5 duplicates. I set 'create key change column = true' and in the transformer stage - COUNTER stage variable derivation "If (DSLink20.keyChange = "1") Then COUNTER = 1 Else COUNTE...

Ok, next question - is using the create cluster key change property in sort = true enough to keep track of when to add 1 to the counter? or else how would I create a copy of the previous row's group in order to increment the counter? No, one counter should be fine, alone with a copy of the 'previous...

Hmm...I will try this if its the only way - will I have to create a counter for every group? I have a column named 'conditions' and there are several of these conditions - I need 5 of each of these - I'm assuming yes... Yup, pretty standard 'group change detection' and you want to make sure you pass...

So, sort them and keep a counter - when the counter gets to X - stop processing by adding constraint (ie. counter <= 5) Am I understanding you correctly?

chulett wrote:Hmmm... needing to keep X duplicates would probably mean via stage variables in a Transformer, assuming they're sorted properly first. ...

I was hoping I could use 'Remove Duplicates' to remove dups but I need to keep 5 of them. I didn't see any additional properties besides keeping the first or last dup. Any idea on how I would do this?

Thanks,

Nicole