DSXchange

Hi,

We have an XML input file size of 500MB to be processed. Can we process it using DS 8.0 ?

What is the XML file size limit for DS 8.0 ?

Can we resolve this by upgrading to DS 8.1 ?

What is the XML file size limit for DS 8.1 ?

Appreciate your prompt response.

Thanks,
Deb

Hi Deb. The methodology in 8.5 changes how DataStage reads xml, and any size issues are effectively eliminated. But in 8.1 and lower, the document needs to be broken up externally first. I've seen some tooling that can do it, namely xmlMax, but I don't think its available in Windows. I've also seen sites use their own java code to break up documents outside of DS before the job gets them.

Ernie

Thanks for the reply Ernie.

I am pretty sure DataStage 8.0 / 8.1 is able to connect up to XML sources (without an external interface) but there is some limitiation to the file size it can process. I am trying to get to the file size limits.

How is it the methodology in 8.5 different from 8.0 / 8.1 ?


Thanks,
Deb

In 8.0/8.1 it uses xalan parser which requires the entire XML document to be loaded into memory, so that you're limited by the amount of available memory. In 8.5 it uses a really clever streaming approach (not certain which parser, probably a purpose-built one). And the new XML Transformer stage is truly a work of art!

Figure that in pre 8.5, you'll start having issues around 200M....your mileage may vary, because it depends on a whole lot of things such as the number and size of the element names and the length and values within the document......but that seems to be about average, even though I've seen bigger work successfully, and smaller fail.....

Ernie