Loading Array from WSDL

Dedicated to DataStage and DataStage TX editions featuring IBM<sup>®</sup> Service-Oriented Architectures.

Moderators: chulett, rschirm

Post Reply
asorrell
Posts: 1707
Joined: Fri Apr 04, 2003 2:00 pm
Location: Colleyville, Texas

Loading Array from WSDL

Post by asorrell »

Ok - This is an ongoing problem - trying to load a very large file through a WSDL. In this case the WSDL service has now been modified to send each line as an element as follows:

Code: Select all

    </xs:element>
    <xs:element name="FileContentData">
        <xs:complexType>
            <xs:sequence>
                <xs:element maxOccurs="unbounded" minOccurs="1" name="FileContentDataLine" type="xs:string"/>
            </xs:sequence>
        </xs:complexType>
    </xs:element>
Resulting in a returned array somewhat like this:

Code: Select all

<?xml version="1.0" encoding="UTF-8"?><PBTFileTransWSReply xmlns="urn:SJM_V1.0">
         <TransWSDLOrchReply>
            <OrchJobInfo>
               <jobId>BBC77247C601AA6FA1EACD59E5294B2F</jobId>
               <jobStartTime>2009-08-10T11:44:59.049-05:00</jobStartTime>
               <projectName>FTS</projectName>
               <configurationName>Default</configurationName>
               <orchestrationName>Orchestrations/FTS_WS_HTTP</orchestrationName>
               <routerHostName>ussp-xxxx.com</routerHostName>
            </OrchJobInfo>
            <OrchFaultInfo>
               <name/>
               <message/>
               <activityId/>
               <activityName/>
               <faultTime/>
            </OrchFaultInfo>
            <OrchStatusInfo>
               <status>True</status>
               <errorCode/>
               <string01/>
               <string02/>
               <string03/>
            </OrchStatusInfo>
            <FileContentData>
               <FileContentDataLine>aaa</FileContentDataLine>
               <FileContentDataLine>bbb</FileContentDataLine>
               <FileContentDataLine>ccc</FileContentDataLine>
And I can see this data - however, it is still coming back as one row from the WSDL call. This means that even though the sender has changed their structure to "split up" the data into rows, the DataStage WSDL transformer still returns it as one large row (structure).

I checked, and there doesn't seem to be any way to tell the WSDL to return an array.

If I have to receive a very large (500mb) file is there ANY way to do it from a single WSDL call or am I out of luck? I can request modifications on the WSDL side - I just have to know what they need to do!
Andy Sorrell
Certified DataStage Consultant
IBM Analytics Champion 2009 - 2020
eostic
Premium Member
Premium Member
Posts: 3838
Joined: Mon Oct 17, 2005 9:34 am

Post by eostic »

The "provider" has to dictate that their service sends back an array, and that same "provider" has to generate their WSDL appropriately so that it illustrates that an array is being sent.....

....then, secondly..... you have to deal with it yourself inside of DataStage, because WSPack does not support the ability to directly break out the rows that are in that array.

...then, thirdly, keep in mind that SOAP Web Services like this are stateless, single request reponse functions. You will ALWAYS get back the ENTIRE array in a single SOAP envelope, regardless of the array size. All the array specification does is indicate what the body will look like.

Hopefully, your provider has already done the necessary bits to support an array, although even if they haven't, so what? If you are getting the single response, and it contains what you want, just pass it into an XMLInput Stage and get your individual rows.

What you may be looking for though, is an ability for a provider to similuate a "state" for you. This is done by some providers (SalesForce.com uses this technique). This is where the entire result set is buffered on the server, "for you" and each successive Web Service call that you make can optionally provide a "handle" (a value they return to you in the first call) and a page number (so the provider knows which next chunk of rows to send you).

Ernie
Ernie Ostic

blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
asorrell
Posts: 1707
Joined: Fri Apr 04, 2003 2:00 pm
Location: Colleyville, Texas

Post by asorrell »

Thanks!

So - just to paraphrase your response - to make sure I understand...

1) Regardless of array / non-array architecture, the WSDL call will by default return everything in one "row" as far as DataStage is concerned.

2) The only way to get around this is to build a bit of intelligence into the WSDL interface so it splits the file into "chunks". The calling DataStage job must tell it what "chunk" they want to receive for each particular call.

This pretty much shuts us down from using DataStage & CastIron together to fetch large files. The only way for CastIron to do option #2 would require a ton of overhead on their end to decrypt and parse the entire 500mb file for every call.

We'll research other options. Too bad - I really wanted to build some more WSDL jobs, but apparently we're just stretching the boundaries of what CastIron (and WSDL technology) was really designed to do.

Thanks for the response!
Andy Sorrell
Certified DataStage Consultant
IBM Analytics Champion 2009 - 2020
eostic
Premium Member
Premium Member
Posts: 3838
Joined: Mon Oct 17, 2005 9:34 am

Post by eostic »

I've never tried cut/paste in here, so apologies in advance if this entry gets all screwed up...thoughts below...

(The good thing about being the Content Editor - I went back to your post and inserted a few "Quotes" to make your responses stand out better! - Andy)
1) Regardless of array / non-array architecture, the WSDL call will by default return everything in one "row" as far as DataStage is concerned.
--- Regardless of array, non-array, a normal HTTP based Web Service will return everything in row as far as ANY soap client is concerned.
2) The only way to get around this is to build a bit of intelligence into the WSDL interface so it splits the file into "chunks". The calling DataStage job must tell it what "chunk" they want to receive for each particular call.
--- keep in mind in this question and the prior, that WSDL is a model for design purposes. One would say "build a bit of intelligence into the service interface". WSDL is just a contract; a description of the service.

--- creative web service solutions do exactly this, letting the client (whether it is DataStage or Java or VB dot Net) tell it what chunk it wants. I've seen some that buffer it, and others that just re-retrieve it. How they do it will be dependent on the pressures on the provider for memory, performance, the number of rows, number of clients, etc.
This pretty much shuts us down from using DataStage & CastIron together to fetch large files. The only way for CastIron to do option #2 would require a ton of overhead on their end to decrypt and parse the entire 500mb file for every call.
--- As noted in many prior discussions, Web Services and SOAP are the wrong interface for doing file delivery...DataStage or not. Web Services might kick off a secure and encrypted file transfer method.
Ernie Ostic

blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
Post Reply