Read XML File - Size limitation

Post questions here relative to DataStage Server Edition for such areas as Server job design, DS Basic, Routines, Job Sequences, etc.

Moderators: chulett, rschirm, roy

Post Reply
pradkumar
Charter Member
Charter Member
Posts: 393
Joined: Wed Oct 18, 2006 1:09 pm

Read XML File - Size limitation

Post by pradkumar »

Hi Gurus,

I have searched the forum before posting the question and found many anwers to some of the issues iam having in reading the xml files. But i have one outstanding issue with my requirement.

Requirement : Have to read around 40 multiple xml files with the same metadata and convert in to fixed flat file.I have done this by putting the
External Source stage --> XML Input --> T/F-->Seq

Iam using the ls #psFilePath#ABCDE*.xml | sort underSource program(ES Stage), XML Source column as ABCDE, Column Content as URL/File Path in XML Input Stage.

The job is running fine if the source xml file sizes are less than 300 kb and aborted with Heap allocation failure pointing to xml input stage with more than 700 kb .But i may end up having some files larger than 800 kb.

can anyone suggest on this??

Thanks in Advance
eostic
Premium Member
Premium Member
Posts: 3838
Joined: Mon Oct 17, 2005 9:34 am

Post by eostic »

K or M ?

Anything above 300 Meg is getting borderline. That's for "one" of the xml documents in your subdirectory. If you really mean "k" here, then that's small, and shouldn't cause you a problem....

If truly it's megabytes you are talking about, then you will need to break up the documents before hand.

Ernie
Ernie Ostic

blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
pradkumar
Charter Member
Charter Member
Posts: 393
Joined: Wed Oct 18, 2006 1:09 pm

Post by pradkumar »

Hi ,

Thanks for your response.

Yes im talking about Kilobytes's only not Megabytes.Thats weird.

Do i have to do any changes in the job design and properties?
eostic
Premium Member
Premium Member
Posts: 3838
Joined: Mon Oct 17, 2005 9:34 am

Post by eostic »

Strange for such a small document. Right now I'd suspect something else. Strange characters in the data, perhaps? What else are you doing in the Job...

...and now I see that you are in EE...so it's possible it is a column issue, although I don't usually see "heap" type errors, and generally, column size issues occur when you are sending the entire document in a column...but here it looks like you are sending just the filename into the xmlInput Stage...

I would definitely try a Server Job (use Folder Stage to pick up the document), just to debug it.......Server is often more forgiving with large varchar text and also with odd characters...

Ernie
Ernie Ostic

blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
Post Reply