DSXchange

Also make sure that your cut command is working.
run this on your shell make sure there are no leading or trailing spaces

Efficiency is relative. Change data capture will identify the records. Change apply will apply the changes to wherever.

Look into Wait for file stage or external file sniffing capabilities in enterprise schedulers. Read and write to two locations. One to your actual destination and the other to the archive folder. Have a cleanup job run every day check for file create date and anything over 30 days, get rid of it.

You dont have to replace null indefinitely. Just replace it before you feed it to the sort and aggregator stage and then replace it back.

The complete row is lost???
That is a problem. Look at your partitioning method. If the row is there with count > 1, it is getting filtered out and thats ok.

First change it to integer (no decimals) then multiply.

Apply the group by and count and filter out keys for records that have count = 1. Let this be a lookup to your source filtering on matched records.

Convert to numeric and multiply it with 100000000. But be sure that really is the requirement.

Open the file with excel. It accepts tab delimited files. see if it opens up the file properly. If it doesn't, that means the file is not properly delimited. Lets start there.

Is that join going to reduce the data???
If yes, then you can load the data from the "other" database into your current database that houses your source. Pass a sql join and extract.

10 bytes is not a timestamp. Its a date field. Convert your data into date and try inserting.

Yes. You can use the after job subroutine option as well.

Execute Command Stage perhaps !

I would suggest to attach a condition id of some sort with your data and then use some sort of os level command to split your file into many files.

Restrict it in sql. Based on your example, you want everything from T1 which is not there in T2, deduped.
Do a 'not exists' clause and run the output through remove duplicate stage.