DSXchange

Don't worry about any memory settings. If your data is millions of rows, consider using a Sort stage in front of the Aggregator. Non-sorted data is stored temporarily within the Aggregator until the last row is input, then all the stored rows are output. Sorted data is output as the group changes. T...

Make sure the file exists and you have access to the directory.

Take 5 minutes and search the forum. There's posts going back years.

Each row is about 10 megabytes in content? Sheesh, that's a big row.

Try turning off inter-process and row buffering to see if the problem persists. Watch the size of the output file, see how big it is when the job aborts, that may point to a file system configured for only 2GB (an older 32BIT configuration). Otherwise, release 5.x was quite awhile back and it sounds...

Why not use the sqlplus command line to run the stored procedure? Then you can use a simple Sequence job with a bunch of Command stages configured to run sqlplus.

It's not a factor of servers, but physical cpus. You might find that 2 processing nodes per cpu is adequate, but then again 4 nodes might be better. You can have a single 64 cpu SMP server with 128 processing nodes, or have 16 X 4-cpu SMP servers clustered with 128 processing nodes and have better p...

whoa whoa whoa. The use of the term "parameter" is misleading everyone (except Craig). You simply use cursor tokens in the SQL of the lookup. Change the metadata for the SQL lookup to indicate two columns as KEY. DataStage will prepare a cursor of your SQL statement, and then pass two values to the ...

I tend to avoid custom tweaks on a project level. Those are the things always forgotten during project setup. Then, you corrupt your production project during a simple job import because DS_JOBOBJECTS corrupts. I prefer something more along the lines of the engine install and configuration, probably...

The most obvious issue is with releases 7.5 down. The internal repository stores all job design information in a single hashed file called DS_JOBOBJECTS. When you have too much design information, this file exceeds the default 32BIT limitation of 2.2GB. You will corrupt your repository if this happe...

The Sybase stuff is flaky. You can see this by looking at the generated SQL in the RDBMS stage and see that it is not correct, or, what you see is what is not sent to the database. How are you using the Sybase plug-in to generate the SQL? Did you look at using column derivation-based generated SQL? ...

Rather than reindex, why not create a set of projects, export subsets of the big project and deploy into the new projects. Instead of having 2600 jobs in one project, look to manage this better.

I have such a job on my website Members Area. A job that compiles other jobs. As for a routine, the job control library there as well has compilation built into the job control function. So, you can get the logic from there or use as needed.

For development, it's good to use umask of 002, because you have so many developers. However, in the QA and Prod environments, you may consider only running under a select userid, therefore, consider using umask of 077 to tightly restrict access.

No. Project name and location are irrelevant. When you generate a .dsx file the only thing that carries over is the Folder/Category name. You'll be fine.