DSXchange

Anyone facing issue with environment variables when promoting jobs from one env to other ? All the user id's and passwords are defined as PROJDEF but still the jobs are failing when moved to prod because of the invalid user id and password error. As a workaround, we delete the user id and password i...

We are having a problem with the bulk load option of Db2 connector stage in a partitioned database environment. If the data type of the primary key is Varchar then it's not loading all the records and just loads records from one of the nodes. FYI...Database is partitioned in 2 nodes. Simplified Job ...

By looking at the job score I believe hash partition happened(please see bolded text) when it went to remove dedup stage which should have sent the records to same partition. Following is the score: main_program: This step has 9 datasets: ds0: {op0[1p] (sequential sf_Case360_PolStubFields) eAny<>eCo...

Thanks Ray...So, I don't have to put a sort stage before join stage and sort the key field If I don't have to worry about allocating memory or anything as DatsStage is going to do that. Correct ?

ArndW - Explicit partitioning is done in Remove Duplicate stage. If nodes and partitions are same then I would not get the expected result as both records goes to different partition. Correct me If I am wrong here. Ray - If partition is subset of data processed in a node then am not sure why both th...

I am trying to understand the correlation between Partitions and Nodes. Job design is two sequentail files feeding to Funnel and Remove Duplicate stage followed by transformer and another sequential file. I hash partitioned on a key in a remove duplicate stage and I run with 8 node config file and t...

Hi I am joining two data sets using join stage and both of them are hash partitioned on the join key but the data sets are not sorted. I believe parallel framework inserts the tsort operator if the data is not sorted. I see in some of the posts that it's better to put the sort stage explicitly but a...

What I don't understand is that the columns are nullable even in the database but it spits the error message as : Converting a nullable source to a non-nullable result These columns are not propagated further down and it's just used for range lookup. Why does it throw entirely opposite warning ? Tha...

I could not figure out the explanation for the behaviour so as a workaround I checked the column from the right side file in stage variable using NullToZero function and then checked the stage variable in the column derivation using IsAlpha function. It worked

I am facing a problem when using Lookup stage. Job design is Seq file, join stage(with seq file), look up stage(with DB2 Connector stage) and few other join stages after that. In the lookup stage, I am doing range lookup on the Version From and To dates in the table with a date field(appl date) come...

Thanks for the reply. I understand the difference between empty string and null but my question is that when we don't find a matching record in the right file what do we get ? As I mentioned in my original post all columns are not nullable so I don't think introducing another column with some defaul...

All my lookup columns are Varchar and funny thing about this is that the field is defined as Not Nullable in Database and the job did not fail but there are no values in the column when you query the table.

Correction in my job design that final stage is DB2 connector not a file.

I noticed something and want to make sure that my understanding is correct. Job design is : SeqFile2 | | SeqFile1---Join Stage----Transformer---Remove Dup----File I join a Seq file with another Seq file using join stage and in transformer I check IsNull(SeqFile2.Col1) to find out if the lookup is fo...

RCP need to be on and I need those columns on the otherside but I believe the reason for the warnings are the multiple paths in the job and was combining them to Parallel shared container output stage. As a workaround, I changed the shared container to a Job and writing to a file and using that as a...

Is there any other way to combine records from two streams instead of using Funnel stage inside the Parallel Shared Container as it throws these warnings and I suspect it also drops those columns ?

Appreciate any input...