Can one remove an end-of-wave from a data stream?

ArndW · Post by **ArndW** » Wed Jan 14, 2015 11:39 am

I have a job consisting of many consecutive stages which processes a very large number of records coming from a database (hundreds of millions of rows). The data is coming into the job correctly sorted and partitioned and the pipeline is streamlined with no stages that would break the pipeline.

I need to add some processing (a lookup, with rejected records that get re-inserted into the data stream using a sorted funnel stage) but if one of the links to the sorted funnel stage has no records the pipeline gets broken as all records get buffered at this point.

I can work around this problem by inserting a end-of-wave marker every n-records to allow processing to proceed in "chunks".

Unfortunately, towards the end of processing I have a transform stage which uses stage-variables to store cumulative values over many records; and this stage gets reset when it sees an end-of-wave marker. Also, in some copies of this job there is also a re-sort; where an end-of-wave marker would result in incorrect sorting. Both of these are unwanted effects in end-of-wave processing and lead to incorrect results.

What I am looking for is a way to remove the end-of-wave marker values prior to reaching these parts of the job. While there's a end-of-wave stage that allows us to insert markers into the data, there's none out there to allow one to remove those markers.

Before throwing this over the fence at IBM support I thought I might ask around here to see if someone might have a solution.

chulett · Post by **chulett** » Wed Jan 14, 2015 1:41 pm

I for one have no idea but will be following this with bated breath.

JRodriguez · Post by **JRodriguez** » Wed Jan 14, 2015 7:57 pm

Hi ArnD,

I believe that is not possible to remove the end-of-wave marker once inserted in the stream, but in your scenario you could use LastRow() function to detect the end-of-wave signal in the transformer and then act on it

Regards

ArndW · Post by **ArndW** » Thu Jan 15, 2015 3:47 am

JDRodriguez - thanks for the hint about the LastRow(); but unfortunately the multiple-row handling necessary in that stage can require records that would have been grouped into another wave. I think I'll open up a call at IBM to see if the powers that be might have a solution.

ArndW · Post by **ArndW** » Thu Jan 15, 2015 6:03 am

I've submitted the question to IBM and will see what they have to say.

I did do some tests with writing to a named pipe and then reading from the same named in the parallel job and that works to eliminate the End-Of-Wave markers, but for other reasons (this happens in a shared-container which is called in parallel and the fifo file names aren't known ahead of time and thus cannot be created with "mkfifo" beforehand) we can't implement this workaround so I'm hoping that IBM might have a suggestion.

ArndW · Post by **ArndW** » Wed Jan 28, 2015 2:58 am

IBM has responded that there is no way to remove end-of-wave markers at present. I've submitted an RFE to add a corresponding counterpart to the Wave-Generator stage, but if they opt to include such a stage in the future it will most likely take quite a while before it is added to the product.

ArndW · Post by **ArndW** » Tue Feb 03, 2015 5:51 am

There now a "Feature request" for this -

https://www.ibm.com/developerworks/rfe/ ... R_ID=65319