CDC questions

Make suggestions regarding the DSX site or the Forum

Moderators: chulett, rschirm, roy

Post Reply
danddmrs
Premium Member
Premium Member
Posts: 86
Joined: Fri Apr 20, 2007 12:55 pm

CDC questions

Post by danddmrs »

Currently our company extracts 6 billion bytes of data from the mainframe each night, FTP's the data to a server at a remote location, then transforms and loads the data.

Issues are the time it takes to run the cycle and the growing volume of data.

Changed Data Capture seemed a viable alternative so I suggested it. My thought was to determine the Inserts, Updates, and Deletes while the data is still on the mainframe and send only changed data via DStage.

Our DBA's recommended setting up a Staging database on site, dumping the data to the LAN, loading the Staging area with DataStage, and then use DStage for CDC and making the changes at the remote location.

Their recommendation is based on their experience with data warehouses, but can someone explain what advantage is gained or if this will even work? What about deletes?

Server A would hold todays data, Server B holds yesterdays data 500 miles down the road, and 99% of the data is unchanged. Is DataStage the right tool for the job? If so please advise on how to set this up.

Thanks in advance.
gnan_gun
Participant
Posts: 30
Joined: Thu Aug 02, 2007 5:31 am
Location: Mumbai, India

Re: CDC questions

Post by gnan_gun »

Hi,
By using Change Capture stage we can do this. But Change Capture is very sensitive. Take care about data is null handled or not & cleansing.
Here the problem is When u extract data from Before & After set data in Change Capture, If your Before set data is huge then your job will be abort if sufficient cache memory is not available. See the default memory and change that as per ur requirement.
danddmrs
Premium Member
Premium Member
Posts: 86
Joined: Fri Apr 20, 2007 12:55 pm

Post by danddmrs »

ag_ram

Thanks for pointing that out. Not sure how I even got there. Probably landed there in a search and didn't realize I hadn't specified the forum.

Sorry!
Post Reply