DSXchange

You can use DataStage to provide both the primary input and the reference input into a QualityStage plugin that is doing matching. The safest design would be to sort and partition the primary input and set the partitioning of the reference data as entire. I doubt there is any partitioning method oth...

The sort keys on the sort stage need to be the same as the remove duplicate keys. Better yet you can remove duplicates in the sort stage using one of the properties. From memory the only thing the remove duplicates gives you that isn't in the sort stage is a "Last" option.

The Remove Duplicates stage lets you keep the first or last instance based on key fields.

Since you are setting a value in a stage variable in a transformer you are setting it for every row of data. Therefore the easiest way to pass it to another stage is to put it into a new column. If you set an environment variable you are setting it potentially millions of times.

Database stages can also be a pain. Sometimes the enterprise DB2 and Oracle stages do not get installed on the Windows server, or the databases cannot be accessed from a Windows server.

If you dont need the key value elsewhere in your etl process and only care that it is a unique value then you should consider using a trigger on the table into which you are inserting the data... Trigger surrogate key generators are getting the cold shoulder nowadays, too much overhead on bulk inse...

I'll also have a wild stab in the dark, could it be that on line 6, and in the other lines mentioned, you have too many columns in the record?

I agree, ranges are bad, you pass in the start value in one of two ways, either as a job parameter into the job (where it is retrieved via an operating system script) or via a lookup stage to the target table (as a max select). The start value can then be set in the surrogate key stage start value p...

I don't know the answer but it seems to me that if you have to ask the question then you probably have the wrong job design. I can't see any job with more then 1000 lookup stages being buildable, compilable or maintainable.

It's almost structured data if the file always has the same set of fields and you don't need to break down the data inside fields. Read it as a sequential file with a seperator of : and change the end of record character to something different so it reads all rows in a record as one record. You may ...

Any reason why you are not using the surrogate key stage, which generates key much more efficiently, or a parallel transformer counter field as described in the FAQ forum. Both methods can be passed a start value and will concurrently increment across partitions.

It can be hard to find database products other then DB2 on the IBM website, there is a Universe link in the DataStage information FAQ:
http://www.dsxchange.com/viewtopic.php?t=88590

Has anyone tried using DSJOB against the VERSION project? For example dsjob -ljobs VERSION. This might give you a list of all job names without much hassle. You could use any of the DataStage job reporting options listed in the related FAQ such as exporting VERSION to an XML export file and running ...

You really need to read the section on Sequence jobs in the Designer Guide as I'm sure it will answer this question and a lot of other questions that will follow it. Have a look at the page on Job Sequence Properties - General Page. It explains checkpoints and restarts.

This is one crazy job. The transform and funnel look wrong. You should be able to write this job with just a lookup stage and retrieve whatever country values you need from that lookup. Why do you have 105 output links? The lookup stage has a multiple lookup matches option that lets you return multi...