DSXchange

Arnd, checkout http://www.datastagexchange.com/etlarticlelist.php , Ray has what you need posted there. This is the companion site, and predecessor to DSXchange.

-Ken

And Ken, since two warnings are logged for each pk constraint error, that's more like 54,000 warnings to the log. 41,000 total - 27,000 good = 14,000 bad 2 warnings messages per issue X 14,000 bad = 28,000 total warning messages My guess was closer, without going over. I win the showcase showdown.

You didn't notice 20,000 yellow error messages in the job log???

We threw a lot of guesses your way.

Add a Reject link to your job in the transformer right before the DB stage going to a reject file. Next time you have this problem, you'll see those rows that fail to load go to this file.

The problem with using an OCI/ODBC lookup is that it will take forever, basically all 70 million rows in the item master table will generate a SQL query against the product table. You do the math, that's 70 million queries. All 70 million rows have to exit the item master instance, traverse the netw...

The item master table has a product foreign key? Are you saying that you wish to select from the item master based on the product keys in the lookup table? Would pseudo-sql look something like: select * from item_master where product key in (select product_key from product_lookup) If this is...

Maybe we can help point you in a direction that better utilizies the features in DataStage. Can we get an idea of row counts in the source and reference tables? There are alternatives, such as high-performance hash files that provide reference capabilities, and maybe just approaching your design dil...

Do not use DataStage transformation jobs to do file renaming. This is a misuse of the tool. It would be far easier, faster, and scalable to write a simple korn, perl, whatever, script to do this. Using DataStage will mean that your job will read each file and write the contents out again. There is n...

What are you trying to do? If you are using DataStage to move files around, you're making a big mistake. This stage is meant to be used for reading a bunch of 1 line data files and turn it into a single dataset where the contents of the files become rows of data. It is NOT to be used to shuffle file...

Any problem using the unix command cat?

Audit stage is the older Quality Manager tool under a new name.

Profile stage is the acquired tool MetaRecon from Metagenix.

Both tools provide you with means to do data profiling.

Yes. Make sure you: 1. Didn't run out of disk space on the project 2. Reset the job to see if any phantom messages show up in the log 3. Look at the system resource utilization at the job the job "mysteriously" died. You may have exceeded an unknown threshold for concurrent files open, user memory l...

There's no direct API as far as I know. You could see if the MakeJobReport API or switch on dsjob has it in its output, and then you'll have to parse for it. It seems a lot of the functionality like this is absent, forcing you to use MetaStage for documented and supported means to gain this informat...

I also don't know what this section: Answer=OConv(Answer,"MCP") Loop I= Index(Answer,"'",1) If I>0 Then Answer=Answer[1,I-1]:Answer[I+1,Len(Answer)] Until I=0 Repeat is doing for you. You should probably not use it. It appears to be removing single quotes,...

This section: StartDate = DSGetJobInfo (DSJ.ME, DSJ.JOBSTARTTIMESTAMP) SummaryArray = DSGetLogSummary(DSJ.ME,DSJ.LOGANY,StartDate,"9999-12-31", 0) Answer="" NLines=0 Loop Str = Field(SummaryArray,@FM,NLines+1) If Str<>"" Then NLines=NLines+1 Even...

The transactions will rollback to the last commit. There's nothing you have to do. Now of course, if you're committing every 10K rows and your job crashes at 50K rows, you'll still have 40K rows inserted/updated. If you set to 0 the transaction count, you have one commit at the end so that a rollbac...