Converting to Real Time

Dedicated to DataStage and DataStage TX editions featuring IBM<sup>®</sup> Service-Oriented Architectures.

Moderators: chulett, rschirm

Post Reply
danddmrs
Premium Member
Premium Member
Posts: 86
Joined: Fri Apr 20, 2007 12:55 pm

Converting to Real Time

Post by danddmrs »

My company currently loads data nightly to Oracle tables that are then accessed via Web applications.

The data flows into the Oracle tables via mainframe processes from VSAM and DataStage Server edition jobs from Datacom.

The processes above run in batch and the data is always a day behind. We also extract, transform, and load every byte of data every night even though only a small percentage has changed (about 7 billion bytes of data are tranferred nightly).

Would it be feasible to convert this to a real time process using the DStage SOA edition?

Thanks in advance for comments.
Richard
eostic
Premium Member
Premium Member
Posts: 3838
Joined: Mon Oct 17, 2005 9:34 am

Post by eostic »

Hi Richard... for the particular use case described (more frequent loading of the target Oracle tables), SOA Edition is not necessarily your solution..... "more frequent" batch loads, or "trickle feeds" (via messaging, or using Changed Data Capture offerings) are more appropriate for that primarily "ETL" function. We can talk about those further........ where SOA Edition (RTI/WISD, depending on the release) could be valuable is in the "web access" that you describe. How are those applications accessing the Oracle tables? Are they writing their own SQL and embedding it in the applications? Who owns the tables? Who understands best what's in them? Who maintains the SQL and the security?
This is where SOA Edition comes into play...to let you design Services for lookups, transforms, data access, etc...... and do that work in DataStage, but deploy those jobs as Services with WSDL that the applications needing data can just consume and have immediate access.

Ernie
Ernie Ostic

blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

Almost certainly but you will need to give some thought as to your model - do you need an "always running" job or one that starts based on some trigger such as the arrival of data. How many instances of the job would be needed to accommodate the maximum expected data throughput?

Have you purchased SOA Edition, which contains the "real time" components that you would need? If not, have you obtained information about these (RTI Input and RTI Output stages, for example) from your vendor or, perhaps, from the IBM web site.

I'm sure that next time Ernie logs in he will provide more information; this is his bailiwick.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
danddmrs
Premium Member
Premium Member
Posts: 86
Joined: Fri Apr 20, 2007 12:55 pm

Post by danddmrs »

Thanks for replies.

My understanding then is that the SOA tool could be used to enhance the Web interface with the Oracle tables once the data is loaded. There are DBA's that own and maintain the tables. The applications that access the Oracle tables are maintained at a different site and I'll get more information if needed but probably outside the scope at this time.

The issue then appears to be how to get the "real time" changes from various systems into the Oracle tables in a timely manner. Since the batch process takes several hours more frequent cycles doesn't appear to be a viable solution. By "Changed Data Capture" do you mean sending adds, deletes, and updates as they occur in the source systems to the Oracle server and using the Insert, Delete, or Update options?
eostic
Premium Member
Premium Member
Posts: 3838
Joined: Mon Oct 17, 2005 9:34 am

Post by eostic »

Indeed! Changed Data Capture implies doing log capture, in a way that is least intrusive to the source system, and then providing everything you need to apply details to a target....if it's just pure "replication" (same target tables, no transformation), then stand-alone replication tools usually do the trick, but if there are transformations (for a data warehousing target or entirely different model) then DataStage comes into play perfectly.....

There are many solutions here, tried and true. We most recently have tied DataStage into DataMirror for doing this style of operation, but have other technologies also, and there are hosts of home-grown solutions that include triggers (yuk ;) ) and other methods....... captured changes are sent via flat files, MQ series, and other mechanisms into DS where they can then be transformed and applied.

What you need depends on a LOT of things, such as volume, speed, complexity, access requirements to the source, etc. etc. etc.....

Ernie
Ernie Ostic

blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
Post Reply