https for data retrieval?

Dedicated to DataStage and DataStage TX editions featuring IBM<sup>®</sup> Service-Oriented Architectures.

Moderators: chulett, rschirm

asorrell
Posts: 1707
Joined: Fri Apr 04, 2003 2:00 pm
Location: Colleyville, Texas

https for data retrieval?

Post by asorrell »

I'm at a customer site where a vendor just asked if DataStage can connect and retrieve data through an https connection (from the way it was phrased, I assume without using WSDL).

I've requested more info from the vendor and I'm wading through the manual right now, but also thought I'd throw this up for grabs in case anyone can definitively say either "No" or "look at this particular stage".
Andy Sorrell
Certified DataStage Consultant
IBM Analytics Champion 2009 - 2020
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

The answer's always "yes", since the "how" question wasn't asked. At worst, you could use External Source stage plus a hand-coded application.

UniVerse (but not, alas, DataStage) has its own CallHTTP library, which would have made life very easy. Maybe you could grab that UniVerse manual and reproduce the functions as parallel routines or objects.

No doubt there are easier ways through WISD - we await Ernie's next visit for more on that.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
eostic
Premium Member
Premium Member
Posts: 3838
Joined: Mon Oct 17, 2005 9:34 am

Post by eostic »

I've seen it done via JavaPack and also with the old Perl Stage that we used to have. Should be fairly simple for anyone who has done http inside of a Java class.....

Ernie
Ernie Ostic

blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
eostic
Premium Member
Premium Member
Posts: 3838
Joined: Mon Oct 17, 2005 9:34 am

Post by eostic »

...of course, I forgot to ask what they wanted to get via http...I was assuming they just want a document or something at a url.... are they really asking if they can get to a SOAP based web service, or maybe a REST based web service?

The answer will be the same, unless it's SOAP that WSPack can handle...the Stages in WSPack support https....

Ernie
Ernie Ostic

blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
asorrell
Posts: 1707
Joined: Fri Apr 04, 2003 2:00 pm
Location: Colleyville, Texas

Post by asorrell »

This is still the "Very large files through Cast Iron" problem. I had the server job with row splitter working, then Cast Iron encountered a java heap error caused by a java bug that won't be fixed until the next major release in 2010.

The Engineers from Cast Iron want to get away from using the WSDL and think it can be sent through one of their https interfaces. Now I have to see if we can receive the same. I'm trying to get more info from them now.

-Edit-

Just had a conversation with an engineer from Cast Iron. It turns out they compete with WebLogic and to a certain extent with DataStage. They provide provide prewritten interfaces to dozens of external applications - way more than DataStage. Their strength is their ability to do B-to-B and B-to-Web interfaces for these off-the-shelf apps. They don't do parallel, so they aren't that great for large transfers (where DataStage shines).

In any event, now that we understand a bit more about each product, they are going to try to keep a WSDL interface, but change the driving mechanism for what is actually transferring the files. I'll post if it works!
Andy Sorrell
Certified DataStage Consultant
IBM Analytics Champion 2009 - 2020
eostic
Premium Member
Premium Member
Posts: 3838
Joined: Mon Oct 17, 2005 9:34 am

Post by eostic »

ok.... sounds good. I wonder if they are related to StrikeIron.

Anyway...just in case YOUR side blows up with a heap error, use your right mouse button on the WSTransformer and open it up in Grid Style....if you scroll down you will see a property for java options...there you can increase the heap size used by the WSTransformer....... using -Xms syntax (I can't remember it right now, but it's the syntax where you set the maximum meg for the jvm).

Ernie
Ernie Ostic

blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
asorrell
Posts: 1707
Joined: Fri Apr 04, 2003 2:00 pm
Location: Colleyville, Texas

Post by asorrell »

Grid style looks very cool.

I must say, I'm not impressed with the documentation though. It definitely assumes that you are using a WSDL to load the transformer and doesn't give a great description of the properties if you are looking to manually set or tweak them.

- but back to the original problem -

Ok - the Cast Iron Developer has changed the "guts" of the call and moved it to a new port. He has not changed service names or anything else. Though it is still using the same WSDL-style calling interface, it is using a different mechanism internally to fulfill the request.

When he uses oXygen (www.oxygenxml.com) to test the interface, all he does is point it to the new port, and the call works flawlessly.

When I switch DataStage to point to the new port I'm getting the following error messages:

Code: Select all

wsxfm_SJMFTPGetService,0: Fatal Error: Fatal: 0026`:`com.ascentialsoftware.wsclient.ServiceInvocationException: Service invocation exception: 
 at com.ascentialsoftware.wsclient.WebService.callService(WebService.java:1602)
 at com.ascentialsoftware.wsclient.WebService.callService(WebService.java:1544)
 at com.ascentialsoftware.wspack.Transformer.process(Transformer.java:104)
 at com.ascentialsoftware.jds.StageFactory.runStage(StageFactory.java:544)
 at com.ascentialsoftware.jds.StageFactory.runTransformer(StageFactory.java:361)
Caused by: java.lang.NullPointerException
 at java.io.FilterInputStream.read(FilterInputStream.java:89)
 at org.apache.xerces.impl.XMLEntityManager$RewindableInputStream.read(Unknown Source)
 at org.apache.xerces.impl.XMLEntityManager.setupCurrentEntity(Unknown Source)
 at org.apache.xerces.impl.XMLVersionDetector.determineDocVersion(Unknown Source)
 at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
 at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
 at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
 at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
 at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source)
 at org.apache.axis.encoding.DeserializationContextImpl.parse(DeserializationContextImpl.java:242)
 at org.apache.axis.SOAPPart.getAsSOAPEnvelope(SOAPPart.java:538)
 at org.apache.axis.Message.getSOAPEnvelope(Message.java:376)
 at com.ascentialsoftware.wsclient.OperationCall.handleRemoteException(OperationCall.java:366)
 at com.ascentialsoftware.wsclient.OperationCall.invoke(OperationCall.java:274)
 at com.ascentialsoftware.wsclient.WebService.callServiceInternal(WebService.java:1685)
 at com.ascentialsoftware.wsclient.WebService.callService(WebService.java:1591)
Cast Iron isn't seeing any problems at all - it sees a good incoming call and thinks it is supplying the info that is requested. However, DataStage is obviously having difficulty understanding the message.

Any ideas? I've tried playing with trace options, etc. and I can't get any more / different messages to figure out why it can't understand the reply.

-EDIT-

More Info - called the service from another client and got more info on the error:

Code: Select all

Error: SOAP fault message: <S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/"><S:Body><ns2:Fault xmlns:ns2="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ns3="http://www.w3.org/2003/05/soap-envelope"><faultcode>ns2:VersionMismatch</faultcode><faultstring>Couldn't create SOAP message. Expecting Envelope in namespace http://schemas.xmlsoap.org/soap/envelope/, but got null </faultstring></ns2:Fault></S:Body></S:Envelope>	   

Error: Exception while processing web service activity. Error message: Couldn't create SOAP message. Expecting Envelope in namespace http://schemas.xmlsoap.org/soap/envelope/, but got null . Caused by: Couldn't create SOAP message. Expecting Envelope in namespace http://schemas.xmlsoap.org/soap/envelope/, but got null.

Error: SOAP fault: <?xml version="1.0"?> <ns2:Fault xmlns:ns2="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ns3="http://www.w3.org/2003/05/soap-envelope"><faultcode>ns2:VersionMismatch</faultcode><faultstring>Couldn't create SOAP message. Expecting Envelope in namespace http://schemas.xmlsoap.org/soap/envelope/, but got null </faultstring></ns2:Fault>	   

Error:Encountered fault of type (http://schemas.xmlsoap.org/soap/envelope/, VersionMismatch), please see rest of the log for more information 
So it looks like a null is coming back that is throwing things out of kilter.
Andy Sorrell
Certified DataStage Consultant
IBM Analytics Champion 2009 - 2020
eostic
Premium Member
Premium Member
Posts: 3838
Joined: Mon Oct 17, 2005 9:34 am

Post by eostic »

Couple of things....

a) the Grid style lets you get to the java options....but as you said, the stage relies heavily on your import of WSDL and the "Select" of a particular service once imported. A lot goes on in this Stage.

b) it appears that there is something wrong with the Server side...if another client also blew up, it would seem that the host is not meeting the contract...either not creating the namespaces as required, or doing someting incorrect with the xml structure, etc. Perhaps he needs to produce a new WSDL after his revision.

Ernie
Ernie Ostic

blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
asorrell
Posts: 1707
Joined: Fri Apr 04, 2003 2:00 pm
Location: Colleyville, Texas

Post by asorrell »

It was the a namespace issue. It is cleared up, and the call is working with small files. Put in a 500mb file and it fails with this error:

Code: Select all

sjd_HMS_BigFile_load..wsxfm_SJMFTPGetService: java.lang.OutOfMemoryError
	at java.lang.String.intern(Native Method)
	at java.lang.Class.getName(Class.java:1089)
	at com.ibm.oti.vm.AbstractClassLoader.getPackageName(AbstractClassLoader.java:363)
	at com.ibm.oti.vm.BootstrapClassLoader.loadClass(BootstrapClassLoader.java:71)
	at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
	at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
	at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
	at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
	at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source)
	at org.apache.axis.encoding.DeserializationContextImpl.parse(DeserializationContextImpl.java:242)
	at org.apache.axis.SOAPPart.getAsSOAPEnvelope(SOAPPart.java:538)
	at org.apache.axis.Message.getSOAPEnvelope(Message.java:376)
	at org.apache.axis.client.Call.invokeEngine(Call.java:2583)
	at org.apache.axis.client.Call.invoke(Call.java:2553)
	at org.apache.axis.client.Call.invoke(Call.java:1753)
	at com.ascentialsoftware.wsclient.OperationCall.invoke(OperationCall.java:266)
	at com.ascentialsoftware.wsclient.WebService.callServiceInternal(WebService.java:1685)
	at com.ascentialsoftware.wsclient.WebService.callService(WebService.java:1591)
	at com.ascentialsoftware.wsclient.WebService.callService(WebService.java:1544)
	at com.ascentialsoftware.wspack.Transformer.process(Transformer.java:104)
	at com.ascentialsoftware.jds.StageFactory.runStage(StageFactory.java:544)
	at com.ascentialsoftware.jds.StageFactory.runTransformer(StageFactory.java:361)
Is this the java heap error you were warning me about? I'm trying -Xmx512M in the Java options for input / output. Is there any way to tell if this is occurring on the remote end or the local end?
Andy Sorrell
Certified DataStage Consultant
IBM Analytics Champion 2009 - 2020
eostic
Premium Member
Premium Member
Posts: 3838
Joined: Mon Oct 17, 2005 9:34 am

Post by eostic »

Sometimes its hard to tell, although it's unlikely that you'd even see the error if it was happening at the server --- you would just receive a soap fault that something died.

Go back to the beginning. We talked about this. There are too many limits here --- the WSTransformer for one, and the link/column sizes, especially under EE, as another........No way the WSTransformer will handle things much larger than 300 to 500 Meg...it uses similar techniques as the XMLInput Stage. I know it's not an answer that will help here, but there are better ways than SOAP to send this large a quantity of data.

You can try increasing the heap size further, but you may have reached the limit.

Ernie
Ernie Ostic

blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
asorrell
Posts: 1707
Joined: Fri Apr 04, 2003 2:00 pm
Location: Colleyville, Texas

Post by asorrell »

Yup - I agree, but I'm limited here by either corporate restrictions or resource restrictions. The person in charge of CastIron isn't experienced, and is having difficulty redesigning it into what it should have been in the first place - a row-by-row transfer.

I've got an alternative plan where I just send him a file name to be decrypted and stored on the CastIron server. Once that is done I can use SFTP to load the data.

Of course, that assumes that I can get a patch from IBM customer service to fix the SFTP stage (it isn't working).

FYI - I did have him call the service locally (in CastIron) and it failed - so the out of memory error is on their process, not ours.
Andy Sorrell
Certified DataStage Consultant
IBM Analytics Champion 2009 - 2020
lstsaur
Participant
Posts: 1139
Joined: Thu Oct 21, 2004 9:59 pm

Post by lstsaur »

My SFTP stage (8.0.1 version) works just fine. I know you are working on 8.1 version. What's error message?
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

I'm not seeing an "SFTP" stage, am I missing something? Or do we actually mean the FTP Enterprise stage with the 'Transfer Mode' option set to Secure FTP (SFTP) ? As noted, what issues are you seeing?
-craig

"You can never have too many knives" -- Logan Nine Fingers
lstsaur
Participant
Posts: 1139
Joined: Thu Oct 21, 2004 9:59 pm

Post by lstsaur »

Sorry, I meant the FTP Enterprise stage with the 'Transfer Mode' option set to Secure FTP (SFTP).
asorrell
Posts: 1707
Joined: Fri Apr 04, 2003 2:00 pm
Location: Colleyville, Texas

Post by asorrell »

Yes - PX job, FTP Enterprise stage. Using SFTP, URI starting sftp:// and Port 22. We've played around with all the options, and no matter what we do we get:

SFTP: Error occurred during InitializeFromArgs()

Informative ain't it... :-P

We have ssh installed and configured and it is working and allows seamless connections for both ssh and manual sftp's. IBM has duplicated the issue and is forwarding it to engineering - but if you have any light you can shed please do so!
Andy Sorrell
Certified DataStage Consultant
IBM Analytics Champion 2009 - 2020
Post Reply