DSXchange

True, but those are correctness issues, not performance issues!

So you are telling us that there are commas in the (quoted) data in a comma-delimited file?

Parallel Sequential File stage is notoriously poor at handling these. Try using a server Sequential File stage, either in a server job or in a server Shared Container in a parallel job.

BASU wrote: I cannot evade these warnings

You're not trying hard enough.

Which will be loaded first? Whichever one you choose to load first.

However, there is a right way and a wrong way. And the wrong way is highly likely to throw errors, for example if referential integrity is enforced in the database (which it should be).

Matrics isn't anything in DataStage.

Sounds like you need one or more of the following:
(a) less load on the machine
(b) less load on the connection(s) to SQL Server
(c) a more powerful server (or two)
(d) faster network
(e) more or dedicated network channels

Define "performance" in an ETL context. Rows/sec is not a good metric, due to the huge number of variables, not least row size, buffering, and the like. NextSurrogateKey() function and Surrogate Key Generator stage are functionally identical - they get the next key value from a set maintai...

Dots are not and never will be accepted, for the same reason that (in most databases anyway) they're not permitted in identifier names such as schema, table and column names.

Your expiry date appears on the home page.

I don't see any IPC stages in the job design you posted. Of course to be able to use Link Partitioner stage you have to have inter-process row buffering enabled, and this will preclude using intra-process buffering. I'd be looking at trying to improve the performance of the Aggregator stages by sort...

How are you partitioning these data? Is each partition processing exactly the same number of rows? If not, expect gaps.

i have a .ds file with 2.6billion records. No you don't. The .ds file contains no records whatsoever. It's a descriptor file that describes locations of physical files that do contain your data. Another way to make a copy of a data set is to use the orchadmin cp command. This does not generate any ...

"Best" in what sense?

What's changed since it "used to run fine"?

Hint: "Nothing" is not the correct answer.

In version 8 it's not controlled by user ID but by membership of the Information Server product role DataStage and QualityStage Administrator. That's what you have to check.