Parsing large xml files (more than 10 mb)

[MANASHI]

Hi,
We are facing an issue with large xml sizes which are bigger tan 10 MB. We receive a SOAP response, in that response the actual content is embedded. Now we used the xml stage to parse the xml. Now we changed the value of apt_transport_block_size to 10 MB (which I understand is not good practice). Now when the xml files are bigger than 10 mb, we are getting an error saying: "the record is too big to fit in a block".
Files are stored in datastage server, we need to parse the soap and then the actual content and store the data into database table.
Please help me on this. We are facing this issue in production.
Thanks

[eostic]

You should be able to set that variable a whole lot higher than 10 meg. ....that is fairly small.

Another alternative is to use a Server Job......it will have issues above about 100 meg or so, but shoukd work without blinking at sizes up to that level.
Read your soap responses from disk using the Folder stage......unless you are feeding them directly from the ws stages......in that case just writ the job the same as you have now in EE.

Ernie

[MANASHI]

Thanks a lot.
So, we should make it to a size which is more than the size of the size of the maximum size of an xml. But it has a limitation of 276 MB, I guess. Although I feel that should be enough in my case.

[eostic]

I have notes from testing several years ago that say I needed to have the setting be 2x + 4 the size of the xml doc.

Ernie