DSXchange

Ouch, completely overlooked that. Sorry about that. Look into the Remove Duplicate Stage. That will get you what you want.

A simple solution. Use a bulk loader. It drops indexes and rebuilds them after the load finishes as part of its procedure.

Pass it through a hashed file, the last duplicate will be retained. Or if you want the first one, pass it through aggregator and retain the first record, grouped on the column you want distinct on.

As Craig advised, dont pass any value any file. The job will pick it up from DSParams file during run time.

chulett wrote:

DSguru2B wrote:They consider partitioning for billions of records. But maybe thats just my DBA.

It's just yours.

I always have the best lot

Run the above command in your Administrator. Its similar to Oracle Describe.

Put a big, brighty-coloured annotation on it I'm working on this. DO NOT change it! I think easiest, simplest solution is provided by Ray. If others have to work on this in your absence, then the developer should add his/her id and timestamp and a brief description of what was changed, to the annot...

for some strange reason our support team wants all jobs to write to the same log file. Possible, but it has to be sequential. For parallel processing, they have to be seperate files. Perhaps your support team needs to know that simultaneous writes to the same file is not possible, whether its the c...

So is it better to use only Insert New Rows only..in Update Action Moreover downt he line we will be doing incremental load for every two days..Even then using Inser and Update will pose a problem Yes, 110% better. Yes. And if your DBA is anything like mine, he will laugh as the fact that you are a...

You are correct and hence its taking a lot of time. Also it forces your array size to be set to 1 and thats another downfall.

Thats why, always make a complete backup of your project(s).

But not as much as doint UPSERTS.
Alternatively, load your file into a temp table. Do a sql join and get the inserts. Similarly you can get updates as well. Do bulk inserts. For updates, well you can pass it through DRS stage or OCI stage and do updates only. Will be much faster.

Read my reply above.

Now this is a post that belongs in my favourites .
This question has comeup one to many times. I think it should be added to the FAQ forum.

pradkumar wrote:My option int he target DRS is Insert New Rows or Update Existing.

For a start, your problem is right there. Split your inserts and updates. And for millions of records, use the bulk loader.