DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
This topic has been marked "Resolved."
Author Message
danddmrs



Group memberships:
Premium Members

Joined: 20 Apr 2007
Posts: 86

Points: 895

Post Posted: Thu Apr 10, 2008 10:25 am Reply with quote    Back to top    

Currently our company extracts 6 billion bytes of data from the mainframe each night, FTP's the data to a server at a remote location, then transforms and loads the data.

Issues are the time it takes to run the cycle and the growing volume of data.

Changed Data Capture seemed a viable alternative so I suggested it. My thought was to determine the Inserts, Updates, and Deletes while the data is still on the mainframe and send only changed data via DStage.

Our DBA's recommended setting up a Staging database on site, dumping the data to the LAN, loading the Staging area with DataStage, and then use DStage for CDC and making the changes at the remote location.

Their recommendation is based on their experience with data warehouses, but can someone explain what advantage is gained or if this will even work? What about deletes?

Server A would hold todays data, Server B holds yesterdays data 500 miles down the road, and 99% of the data is unchanged. Is DataStage the right tool for the job? If so please advise on how to set this up.

Thanks in advance.
gnan_gun
Participant



Joined: 02 Aug 2007
Posts: 30
Location: Mumbai, India
Points: 260

Post Posted: Thu Apr 24, 2008 2:15 am Reply with quote    Back to top    

Hi,
By using Change Capture stage we can do this. But Change Capture is very sensitive. Take care about data is null handled or not & cleansing.
Here the problem is When u extract data from Before & After set data in Change Capture, If your Before set data is huge then your job will be abort if sufficient cache memory is not available. See the default memory and change that as per ur requirement.
Rate this response:  
Not yet rated
ag_ram



Group memberships:
Premium Members

Joined: 28 Feb 2007
Posts: 524

Points: 4344

Post Posted: Thu Apr 24, 2008 6:45 am Reply with quote    Back to top    

danddmrs,

Please post all your queries in the respective forum in order to get the questions answered immediately otherwise your question will not be highly taken into consideration until which is transferred to the respective forum.

This question belongs to forum "IBM® DataStage Enterprise Edition (Formerly Parallel Extender/PX)".

This forum is to give suggestions regarding the DSX site or the Forum.
Rate this response:  
Not yet rated
danddmrs



Group memberships:
Premium Members

Joined: 20 Apr 2007
Posts: 86

Points: 895

Post Posted: Thu Apr 24, 2008 2:32 pm Reply with quote    Back to top    

ag_ram

Thanks for pointing that out. Not sure how I even got there. Probably landed there in a search and didn't realize I hadn't specified the forum.

Sorry!
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours