DSXchange

Is your data quoted? If not, then you must have a comma in your data. If the file is quoted, make sure you define the table definition to include quoted data. If the data for a column includes a comma, you MUST have a source file that is constructed with quotes, otherwise, how could anyone know what...

Nothing internal to DS. Consider rollback usage impact with your design.

Okay, but how about the good stuff. Are you moving more data and using more of your server now? Are ya happy?

sheltzer wrote:I actually intend to act on the erroneous data after it is rejected from the database.

As long as you accept the performance penalty with bouncing rows and collecting warning messages in job logs.

Read closer and see that the output sequential file has to be uniquely named, in the filename include either a job parameter that is instance number aware (in my example I use PartitionNumber) or the invocation ID (a macro value available).

Import table definitions for source and target and load into their respective stages. Press VIEW DATA and make sure it works. Now correct your mapping.

Well how are you going to know if a column will even fit into the database? You can map a text value source column into a numeric target column and try all day to insert it. When do you propose to act on that value, before or after the database rejects it? What if your value is null and the target c...

Any chance you're exceeded 32bit limitations?

Consider enforcing each columns datatype, length, and scale using validation functions. Any violators should write to an exception file.

Custom job control that monitors when it starts a job, how long jobs are expected to run, and acts accordingly.

I guess you don't want the 10 page document that SHOWS you how to do this.

Do you see underneath all of my posts there's a "Posters Website" link?

There's no "compression" stage. You'll have to find a compression algorithm and write it as a DS function.

Explain the data, what kind of information you expect to change in it. ETL tools are about text and number manipulation. Binary data manipulation requires you to understand the binary nature of the attribute and how to manipulate it.

To summarize, you tried to "copy" a project. You messed with internal engine files, now things are all messed up. Don't do this kind of stuff!!! I'm loathe to participate in helping out because this is not something you should do. To fix your problem, drop all bad projects, recreate them, and reimpo...

Don't pass binary image, audio, or video data thru any ETL tool. Moving large amounts of binary data on each row thru each active stage is a staggering misuse of memory. Moving text and numbers is no big deal, the tools are written to do that. Rows are measured in bytes, or maybe just a few kilobyte...