DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
Author Message
machudas
Participant



Joined: 16 Jun 2012
Posts: 5
Location: Bangalore
Points: 47

Post Posted: Thu Jan 15, 2015 9:37 am Reply with quote    Back to top    

DataStage® Release: 9x
Job Type: Parallel
OS: Unix
Hi,

Currently we are working on a MDM project.As part of the project we have to perform an address standardization operation which is done through a web service.

Address standardization has to be done for already existing data as well(which will be around 800,000 records).Please let us know whether it is feasible to use real time stage(in our case it is wsdl transformer) for standardization of existing data.

Das

_________________
Known is a drop unknown is an ocean
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42796
Location: Denver, CO
Points: 220596

Post Posted: Thu Jan 15, 2015 10:22 am Reply with quote    Back to top    

Hmmm... does the service accept any kind of 'bulk' processing or is it strictly singleton calls?

_________________
-craig

"I don't mind you comin' here and wastin' all my time time"
Rate this response:  
Not yet rated
machudas
Participant



Joined: 16 Jun 2012
Posts: 5
Location: Bangalore
Points: 47

Post Posted: Thu Jan 15, 2015 11:29 am Reply with quote    Back to top    

For testing purpose we are using singleton calls...but need to check with the web service team whether we can can pass array of i/p and receive the result as array of output.

For address standardization we are passing postal code and building number as input and receive the output.But we are not sure how complex it will be if we pass it as array of input and how the parsing of output array will happen?

More over is there any way in xml stage where we can control the number of rows in an array chunk.

_________________
Known is a drop unknown is an ocean
Rate this response:  
Not yet rated
machudas
Participant



Joined: 16 Jun 2012
Posts: 5
Location: Bangalore
Points: 47

Post Posted: Fri Jan 16, 2015 6:47 am Reply with quote    Back to top    

We checked whether the web service can accept arrays or singleton calls.
It can accept only singleton calls.

_________________
Known is a drop unknown is an ocean
Rate this response:  
Not yet rated
Mike



Group memberships:
Premium Members

Joined: 03 Mar 2002
Posts: 1017
Location: Omaha, NE
Points: 6551

Post Posted: Fri Jan 16, 2015 7:40 am Reply with quote    Back to top    

Singleton calls to a web service is certainly not ideal for bulk data processing.

But since you only have 800,000 records and no stated time constraint, it is certainly feasible.

Depending on the complexity of your web service, I'd guess 800,000 records might go through a single web services transformer in an hour or two.

You can split your data and use multiple web services transformers to reduce overall elapsed time.

Mike
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours