Reading a large array using Web Services

Dedicated to DataStage and DataStage TX editions featuring IBM<sup>®</sup> Service-Oriented Architectures.

Moderators: chulett, rschirm

Post Reply
asorrell
Posts: 1707
Joined: Fri Apr 04, 2003 2:00 pm
Location: Colleyville, Texas

Reading a large array using Web Services

Post by asorrell »

I'm starting a new thread, though this is a continuation of an ongoing issue I'm trying a new tact - so starting a new thread.

Currently I'm calling a WSDL that is returning a very large (500mb) file as a single "chunk" using the definition below.

Code: Select all

    <xs:element name="FileContentData">
        <xs:complexType>
            <xs:sequence>
                <xs:element minOccurs="1" name="FileContentDataLine" type="xs:string"/>
            </xs:sequence>
        </xs:complexType>
    </xs:element>
This very large string is blowing up the job - so I'm trying to get them to modify the WSDL to return the file as a series of rows.

I was originally thinking of reading this as an array, but after reading Ernie's blog and looking at his examples, I'm not certain that it would in fact solve the problem. If I understood those correctly, it would return a SoapBody from the Web Services Transformer and then use an XML stage to decompose it into its elements.

In this case, I'm thinking using an array would still fail because the SoapBody would still be one very large chunk of data (albeit with an internal structure) and would still create a java heap error.

Is that correct? If so, are there any other options?
Andy Sorrell
Certified DataStage Consultant
IBM Analytics Champion 2009 - 2020
Post Reply