Need help on SOA design

Dedicated to DataStage and DataStage TX editions featuring IBM<sup>®</sup> Service-Oriented Architectures.

Moderators: chulett, rschirm

Post Reply
osuru75
Premium Member
Premium Member
Posts: 13
Joined: Wed Jun 03, 2009 11:03 am

Need help on SOA design

Post by osuru75 »

I am in the process of designing a prototype of SOA(RTI) job. I have a SRC table and TRG table.I just need to populate all new records that come to SRC to TRG table.I am able to do some of the basic steps but kind of lost.
(1)I designed a Datastage job with SRC oracle stage, transformer and TRG oracle stage.I compiled the job and able to run it.
(2)Now i replaced the SRC oracle stage with WSID input and TRG oracle stage with WSID Output stage.
(3)I enabled multiple instances, choose to deploy as a service and compiled the job successfully, i am able to locate this job and able to deploy it using SOAP over http successfully.
(4)I canopen the WSDL document and locate the URL succesfully.
Here are my questions.
(1) How can i use this WSDL URL to the job can run 24/7 i.e how can i cosume my webservice
(2)How can this entire process identify the new records in SRC table//Subra
Subra Osuru
eostic
Premium Member
Premium Member
Posts: 3838
Joined: Mon Oct 17, 2005 9:34 am

Post by eostic »

Hi....

Great job so far. You've created a wISD job and deployed it via SOAP, and have the WSDL. You now have an always on Job for performing Transformations....

Let's look at your questions.

(1). It already is 24 by 7. Having wISDInput automatically means that it will be "always on". Check the Director. You will see a Job instance already there, running, with a long DSInvocationId. Consuming the web service is the role of your Java, .NET or other type of tooling developer. Who are you building this SOA oriented service for? A portal? An online application? A partner application? For your own testing, there are a whole lot of tools out there --- stand alone testing tools or entirely integrated ones. InfoSphere Data Architect (IDA) and most of the Rational Eclipse based tooling (WID, Data Studio, etc.) has the ability to call a web service, as do things like SOAP Box, SOAP UI, and many more. Do a quick search for SOAP testing tools. You absolutely have to have a tool of some type to do strong quality testing of your service. If you are using 8.1 you could also use the REST Binding and test via IE, but I would still recommend that you find a good SOAP testing tool.

(2) This question I'm not sure about.... how can you "identify" new records? Do you mean identify, or just "add"? Are you talking about Changed Data Capture, or just an insert via SOAP based Web Service? If it's the former, there are other options --- tell us more about the Changes you want to identify....wISD may only be part of the solution. If it's the latter, you can do inserts with ISD (best way is to put in a transformer, branch to a target stage, and then still have a final link going to wISDoutput with a return code or something for the waiting SOAP client).

Ernie
Ernie Ostic

blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
osuru75
Premium Member
Premium Member
Posts: 13
Joined: Wed Jun 03, 2009 11:03 am

Post by osuru75 »

Hi Ernie,
Thank you so much for your reply.What i am not able to figureout is, when there are 10 records in my SRC table and job runs, my target will get 10 records. But when 11th record is inserted into SRC table by transactional system, how do i grab that 11th record alone using this service. Also as you said, i can see my job always running once the service is deployed.But i am having the error "..main_program: PATH search failure:
main_program: Could not locate operator definition, wrapper, or Unix command for "rtiinput"; please check that all needed libraries are preloaded, and check the PATH for the wrappers". My last question is once the service is deployed and the job is running 24/7(in my case, just keep dumping new records from SRc to target), then why do i need a client program ..is it just in case we need to pass parameters in and get specific output.Thanks for your support//Subra
Subra Osuru
eostic
Premium Member
Premium Member
Posts: 3838
Joined: Mon Oct 17, 2005 9:34 am

Post by eostic »

It will help if we start higher and understand your goals.

ISD is best used when there is a portal application, with java, .NET or other such clients, and you would like to share a certain Transformation, lookup, QualityStage, or other such DataStage/QualityStage logic with them.

What are you trying to accomplish? It's not clear, since you are referring to loading "10 records" what the use case is that you are looking for? ISD may not even be the right approach.

Ernie
Ernie Ostic

blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
osuru75
Premium Member
Premium Member
Posts: 13
Joined: Wed Jun 03, 2009 11:03 am

Post by osuru75 »

Ernie,
I will put it this way to make it simple.I have a source Oracle table.I need to load this table to TRG oracle warehouse table in real time.There are no updates on source table.There are only insertions.Always new records will appear in source.So the moment i have a new record coming into source table, my job shd run and load it to target.Is my approach coorect in this scenario or what shd be the best way to do it ? I just tried with writing reggular job(without WSID stages) and enabled it as service and deployed it.Is it the right way to do it, i believe it is tropology 1. But even after successful deploy, my job is not running and am i missing something here ?
Thanks//Subra
Subra Osuru
eostic
Premium Member
Premium Member
Posts: 3838
Joined: Mon Oct 17, 2005 9:34 am

Post by eostic »

SOA and ISD is not for this type of requirement.

What you are looking for is some type of Changed Data Capture. Do some searches here in this forum and all around....tools like InfoSphere Changed Data Capture will "watch" your relational logs and move data the instant it is committed to the source.

Alternatively, many people do their own comparisons by doing wholesale extracts and then comparing (although that it generally not as "real time").

Years ago people used triggers for such activity, but that has lots of performance implications.

Ernie
Ernie Ostic

blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
osuru75
Premium Member
Premium Member
Posts: 13
Joined: Wed Jun 03, 2009 11:03 am

Post by osuru75 »

Ernie,
Then what is the use of deploying batch jobs or topology 1 jobs as services. Theoritically i shd be able to deploy any datastage job as a service and in my case i am able to deploy the job.But i got held up in running it.I am kind of lost ..//Subra
Subra Osuru
Sreenivasulu
Premium Member
Premium Member
Posts: 892
Joined: Thu Oct 16, 2003 5:18 am

Post by Sreenivasulu »

Interesting discussion between good old times and the new ways :)
eostic
Premium Member
Premium Member
Posts: 3838
Joined: Mon Oct 17, 2005 9:34 am

Post by eostic »

; ) Yes, indeed.... a valuable discussion.

The use case for a simple topology, or "batch job as a service" [for instance a basic normal job that reads from a flat file and goes to db2] is to provide the simplest way possible for a .NET or java programmer to put a button on a page somewhere that says "start my DataStage Job".

You (if you are that programmer) don't need to attach to a project, you don't need to know the name of the Job, and you don't need to have a userid/password (unless you want one, or want something more SOA oriented like https and ssl). You don't need a trick to run a shell, or understand how to use the C++ Job Control API.

It's a subtle use case for wISD...not the major one, by far (which is to have an always on job that shares critical transformation, lookup, or QualityStage address validation or name cleansing/matching/standardization logic).

Ernie
Ernie Ostic

blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
osuru75
Premium Member
Premium Member
Posts: 13
Joined: Wed Jun 03, 2009 11:03 am

Post by osuru75 »

Hi Ernie,
I have taken a very simple scenario of loading sample data from a src file to target file and this job have WSID input and output.The moment job is deployed, i am getting the gollowing error message when this job with invocation id runs. Any idea what this error is ?
main_program: PATH search failure:
main_program: Could not locate operator definition, wrapper, or Unix command for "rtioutput"; please check that all needed libraries are preloaded, and check the PATH for the wrappers
main_program: Creation of a step finished with status = FAILED.
Note that this job runs well from datastage director.It is giving error only when it is triggered tru WISD.

Thanks//Subra
Subra Osuru
eostic
Premium Member
Premium Member
Posts: 3838
Joined: Mon Oct 17, 2005 9:34 am

Post by eostic »

How does this contrast with the first entry in the thread? (you noted that things deployed succesfully)?

Second, your comment is interesting that it works from the Director....a job with WISDInput makes no sense running directly from the Director, I would expect that one to abort strangely.

Sounds like it may be a config/install issue...but ultimately, if your use case is still the same as above, you probably won't be using ISD.

Ernie
Ernie Ostic

blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
osuru75
Premium Member
Premium Member
Posts: 13
Joined: Wed Jun 03, 2009 11:03 am

Post by osuru75 »

It is in the same context...fully RTI job has been successfully deployed and its instance is always running.But it is failing with the above error message. But similar jobs without Realtime stages are running successfully.So i am not sure if it is related to RTI stages issue or another datastage reated issue.
Thanks//Subra
Subra Osuru
Post Reply