DSXchange

Pretty much what you've shown us in your one record example. Use a Transformer stage that remembers the previous row and thereby detects any gap, from which it can build the gap-filling record. It may be easiest to send this gap-filling record onto a separate output link from the Transformer stage, ...

Curious that it failed to create the security file. The dsjob command, if I understand correctly, expects the security file to exist when this option is given.

Some more practical examples are in the training DVD "Programming with DataStage BASIC", which you can obtain through the DSXchange Learning Center

Except, @rkashyap, the OP (@synsog) claims to seem the same result when writing to a Copy stage.

The columns could be generically named for the purposes of this exercise.

chulett wrote:But wouldn't you have to define them so it knows what to drop? Or does it drop any not defined?

Yes, you do have to define them.

Thanks Ray. I tested with Source Sequential File , Target Copy Stage. Even also its reading 80 records per second which will eventually says it will take 30+hrs to process 67Million records. What else I can check ? That is very unusual. Even with one reader I can get tens of thousands of rows per s...

... or the Drop on Import property for columns in the Sequential File stage itself.

To clarify Craig's contribution, the Index() function searches for a substring within another string, returning 0 if it is not found.

And, from memory, you don't get Transformer looping till version 8.5.
Seriously, version 8.1 is really old now. It's out of support. Any reason you can't upgrade?

Are you asking for related tables to be imported?

You need a Transformer with a loop to traverse the columns to detect any invalid or null, and to transfer the row to the output link (possibly with diagnostics) when either is found.

Check it out here.

How long does it take to read into a Copy stage as target? That will advise you of the read speed. Chances are that there's something else amiss with how the connection to SQL Server is being managed.

My sites have more people doing the authoring than are using the tool for authoring. Librarians and taxonomists, most of them. Only a small number actually do the work of researching related terms, synonyms, assignable assets, etc., in the metadata repository. That site is running version 8.7. I hav...