DSXchange

That means it never got created properly. Go ahead and capture the rejects and analyze the data of the rejected records.

Ok. Whatever you business approves. Thought I'd point that out to you.

When you say "wierd data", it does'nt make sense. You sure you are defining all keys properly in both, while creating the hashed file and while reading the hashed file?

I could'nt find mine, and dont have time to search for more, Ray has a similar routine to do the same. Click here.

Are you using a server job (mention of hashed file) or a px job (job type)?
If you make all the seven columns as keys, then from your sample data, you will not lose any data as you composite key (combination of seven keys) is different for each row.

First create a test job that reads your source, changes the datatypes using modify stage and then loads it back to a sequential file stage. If this works out then you can replace the target sequential file stage with the your "rest of the job".

Sure you can. You need to write a routine that checks for file existence and returns an appropriate return value which you can then check inside the transformer and proceed. I wrote a simple one months back. Search for it here.

It doesnt get shipped with the product. Its a user written routine. Ray wrote a IsValid() routine for server jobs. You can search for that.

In comparisons the order of stage variables is very important as they get executed in the order they are defined. I believe the order thats going to work for you is
svFinalDept
svPreviousDept
svDept

If you need unmatching records from both the files then do a FULL OUTER JOIN.

If you search, you will find that folks have tried the same and it has worked for most of them, if not all.

ganesh, your command works fine on command line, I tested it out too. The first awk command that you posted. kashif, why does'nt the command work as a filter for you? Try it out on command line as others requested. Also make sure that the columns that are supposed to be empty are "really empty" and ...

Well then you need to use modify stage. Type conversions are needed in parallel jobs.

Please do not hijack threads. Start your own and give explanation of what the incoming date looks like.

Use that code as a filter command in your sequential file stage. You do not need to redirect the output to a file.