URGENT : XML File Size Limit

DEB_CHOW · Post by **DEB_CHOW** » Tue Nov 23, 2010 2:45 pm

Hi,

We have an XML input file size of 500MB to be processed. Can we process it using DS 8.0 ?

What is the XML file size limit for DS 8.0 ?

Can we resolve this by upgrading to DS 8.1 ?

What is the XML file size limit for DS 8.1 ?

Appreciate your prompt response.

Thanks,
Deb

eostic · Post by **eostic** » Tue Nov 23, 2010 2:49 pm

Hi Deb. The methodology in 8.5 changes how DataStage reads xml, and any size issues are effectively eliminated. But in 8.1 and lower, the document needs to be broken up externally first. I've seen some tooling that can do it, namely xmlMax, but I don't think its available in Windows. I've also seen sites use their own java code to break up documents outside of DS before the job gets them.

Ernie

DEB_CHOW · Post by **DEB_CHOW** » Tue Nov 23, 2010 2:56 pm

Thanks for the reply Ernie.

I am pretty sure DataStage 8.0 / 8.1 is able to connect up to XML sources (without an external interface) but there is some limitiation to the file size it can process. I am trying to get to the file size limits.

How is it the methodology in 8.5 different from 8.0 / 8.1 ?

Thanks,
Deb

ray.wurlod · Post by **ray.wurlod** » Tue Nov 23, 2010 3:23 pm

In 8.0/8.1 it uses xalan parser which requires the entire XML document to be loaded into memory, so that you're limited by the amount of available memory. In 8.5 it uses a really clever streaming approach (not certain which parser, probably a purpose-built one). And the new XML Transformer stage is truly a work of art!

eostic · Post by **eostic** » Tue Nov 23, 2010 3:37 pm

Figure that in pre 8.5, you'll start having issues around 200M....your mileage may vary, because it depends on a whole lot of things such as the number and size of the element names and the length and values within the document......but that seems to be about average, even though I've seen bigger work successfully, and smaller fail.....

Ernie