DSXchange

They have to be checked one way or another so it's not like wasted processing time. Alternately, you could load the file into your database and use SQL to get the non-matches but regardless 100% of the data needs to be checked.

High level would be the consolidated table of customers and a crosswalk / reference table to define the one-to-many relationship between the two.

How were you planning on determining that Customer XYZ in Source 1 was the same entity as Customer 567 in Source 3?

Sure. Set the Lookup's "Failed lookup" rule to Continue and allow only null key columns downstream from there.

Sure, if you can decrypt to standard out the Sequential File stage can read that input stream. Leverage the decryption tool in the Filter property of the stage so that it is not at rest in an unencrypted form.

FTP is notorious for stripping trailing spaces in records. Can you find another mechanism to move the files, say like scp?

i want to maintain Single customer view here as there may be same customer in several sources, in that case i jsut need to have only one customer in customer dimension table. No, you don't. You may need another 'consolidated' view of them, however. Sounds like a MDM layer / system may be in order.

Raftsman wrote:I am also not sure whether the Bulk loader will use up a connection in the job for nothing when not used

It will.

FWIW, in the very small number of times where I've had to differentiate between an initial and subsequent loads, I've created two jobs.

They're not the same. One recent discussion is here:

viewtopic.php?t=151955

Or use a Sort stage to generate a Key Change column on the first field and then all you need is:
And of course, mind your partitioning.

Not really a DataStage question, is it? However...

Use a surrogate for the PK and add a 'source id' column fk'd back to a source system reference table. Track the source customer 'key' in a string.

syedmuhammedmehdi wrote:It is just hanging up without any warning.

Understood. Was just saying that I thought it should log a warning rather than aborting. Or hanging, of course. In your shoes I'd involve your official support provider to help figure this one out.

As noted, post details of your job design.

Odd... "client" should just care that the job works correctly and is efficient. These kind of situations where they dictate stuff like this is just silly to me and not something I would stand for. But maybe that's just me.

Or just log a warning. That is strange. What happens if you add a reject link to the target stage?

Sure... as long as one understands that they all run at the same time, can finish in any order and if you have any issues with one of the "pipelines" you'll need to re-run all of them.