Hi,
Our current production environment has two stand alone installations of engine tier and services tier on each server (supposed to be Active-Passive) while repository tier is being shared from oracle RAC (active-passive) on different servers. No SAN / NAS is used and nothing is shared here. When the repository tier fails over, the primary (E-S) server is able to connect to the repository tier( whichever is active). But the secondary (E-S) server doesn't connect at all to any active instance of the repository tier. Now I am hired to fix this and get the failover working. My suggestion to use SAN is not taken due to budget constraints.
As part of our troubleshooting, a separate repository database from the existing one and point the secondary (E-S) server to use the repos DB by modifying configuration files and then changing values in the WebSphere Application Server administrative console. Now we are able to start all three server tiers. But, the engine credentials in IIS Console has primary server domain instead of the secondary server.
The designer client also shows the projects from primary server.
I am so confused and we have tight deadlines to fix this.
Can you please advise the best way to get the failover setup and to get this engine credentials straight.?
Thanks
Vinod
Registering engine tier with services tier
Moderators: chulett, rschirm, roy
Registering engine tier with services tier
Thanks
Vinod
Vinod
From what I can tell you are on an unsupported configuration. Not only that, but if I understand what is going on, projects will get out of synch and will probably corrupt over time.
It sounds like this setup was an attempt to "cheap-out" on a high-availability architecture.
The two engine tiers should be using the same cross-mounted SAN storage and the services tiers should be in a WAS cluster to provide synchronization.
If you check the Planning, Installation and Configuration Guide you'll see that none of the recommended configurations matches your installation.
DataStage is not static software that can be installed on multiple systems and just swapped out when a failure occurs. Every tier contains multiple interconnected databases for everything from jobs, security settings and operational metadata. That's why the Configuration Guide lists details as to how it has to be setup for correct operation.
Even if you get the engine credentials corrected, this configuration is probably going to have more problems as time goes on. You may even have problems getting Customer Service from IBM as they would probably not provide support on an invalid configuration.
The best way to get this straight is to tell the customer the cross-mounted SAN / NAS isn't optional.
It sounds like this setup was an attempt to "cheap-out" on a high-availability architecture.
The two engine tiers should be using the same cross-mounted SAN storage and the services tiers should be in a WAS cluster to provide synchronization.
If you check the Planning, Installation and Configuration Guide you'll see that none of the recommended configurations matches your installation.
DataStage is not static software that can be installed on multiple systems and just swapped out when a failure occurs. Every tier contains multiple interconnected databases for everything from jobs, security settings and operational metadata. That's why the Configuration Guide lists details as to how it has to be setup for correct operation.
Even if you get the engine credentials corrected, this configuration is probably going to have more problems as time goes on. You may even have problems getting Customer Service from IBM as they would probably not provide support on an invalid configuration.
The best way to get this straight is to tell the customer the cross-mounted SAN / NAS isn't optional.
Hi Sorrell,
Thanks for the response and making valid points. As I already mentioned earlier, the client has concerns on cost and lengthy approval process. My hands are tied now.. and the task in front of me to set up two stand alone (E-S) implementations (Active-Active) to work while the client gets the required approval and resources. Can you please advise me on how to correct the engine tiers register with their own service tiers correctly?
ISALite diagnosis tells the logging agent is unable to establish connection to the engine. "Javan.security.auth.login.LoginException: Could not find the certificate for the client I'd [xxxxxx]. Either the client id is invalid or it has not been properly registered."
Please help..
Thanks
Vinod
Thanks for the response and making valid points. As I already mentioned earlier, the client has concerns on cost and lengthy approval process. My hands are tied now.. and the task in front of me to set up two stand alone (E-S) implementations (Active-Active) to work while the client gets the required approval and resources. Can you please advise me on how to correct the engine tiers register with their own service tiers correctly?
ISALite diagnosis tells the logging agent is unable to establish connection to the engine. "Javan.security.auth.login.LoginException: Could not find the certificate for the client I'd [xxxxxx]. Either the client id is invalid or it has not been properly registered."
Please help..
Thanks
Vinod
Thanks
Vinod
Vinod
Hi,
I am not sure if I detailed the problem clearly.
Current Set up (let's not worry about failover for now)
We have server A and server B with engine tier and services tier installed on both the servers. Metadata repository is clustered on oracle RAC (active-passive). Server A works fine and switches over correctly when Metadata Repository fails over and is currently active in production.
Problem Scenario:
The previous team member tried to configured Server B for the same repository setup. But it wasn't working as expected. Then, the Idea was to create another instance of metadata DB from copy of the existing metadata DB and make it exclusive to Server B and test if it works. Here the point is the new metadata DB contains the server A host name for its engine. We changed the WAS console, ojb-conf.jar, database.properties, and configured data sources correctly. Upon restart the services, logging agent failed to start as metadata still points to the Server A host name for engine tier. We verified the /etc/hosts for short and long host names. It looks good.
Now I need your kind help to correct the setup to make the host names straight and make this engine-services-metadata flow works fine.
Thanks and Regards
Vinod
I am not sure if I detailed the problem clearly.
Current Set up (let's not worry about failover for now)
We have server A and server B with engine tier and services tier installed on both the servers. Metadata repository is clustered on oracle RAC (active-passive). Server A works fine and switches over correctly when Metadata Repository fails over and is currently active in production.
Problem Scenario:
The previous team member tried to configured Server B for the same repository setup. But it wasn't working as expected. Then, the Idea was to create another instance of metadata DB from copy of the existing metadata DB and make it exclusive to Server B and test if it works. Here the point is the new metadata DB contains the server A host name for its engine. We changed the WAS console, ojb-conf.jar, database.properties, and configured data sources correctly. Upon restart the services, logging agent failed to start as metadata still points to the Server A host name for engine tier. We verified the /etc/hosts for short and long host names. It looks good.
Now I need your kind help to correct the setup to make the host names straight and make this engine-services-metadata flow works fine.
Thanks and Regards
Vinod
Thanks
Vinod
Vinod
The metadata repository expects you to use virtual host names (which is part of the correct, supported HA setup). That way the actual host name never changes, it just gets reassigned.
If I read this correctly you are now trying to setup two completely separate DataStage implementations, that happen to share hardware racks.
The customer obviously doesn't care about operational metadata, since it will not be synchronized between the systems. So why not just completely strip off the B system and repository and re-install it? That way it sets up the metadata repository with the System B host name. Then just dump your jobs and config files from system A and load them onto system B. If you have your install procedures documented it should be easy enough and System B would be back up and operational in a day.
... as a side note, I'd be REALLY sure the customer sends you written communication that they want this specific configuration and they have licensing to cover it. Otherwise, you could be held liable since you are doing the installation and configuration.
If I read this correctly you are now trying to setup two completely separate DataStage implementations, that happen to share hardware racks.
The customer obviously doesn't care about operational metadata, since it will not be synchronized between the systems. So why not just completely strip off the B system and repository and re-install it? That way it sets up the metadata repository with the System B host name. Then just dump your jobs and config files from system A and load them onto system B. If you have your install procedures documented it should be easy enough and System B would be back up and operational in a day.
... as a side note, I'd be REALLY sure the customer sends you written communication that they want this specific configuration and they have licensing to cover it. Otherwise, you could be held liable since you are doing the installation and configuration.
Hi Sorrell,
Thanks for the reply.
We are now going in the same direction.. Strip-off B and re- install. I guess fixing the current mess is the hardest part than redoing.
My question here is.. While removing IIS from B.. what precautionary steps should I consider in terms of IIS set up related dependencies (if any) between A and B.?? Or the removal process on Server B will safely remove the current setup without doing any harm to server A ( currently active in production)
Just to avoid any surprises and damages.
Please kindly help..
Thanks
Vinod
Thanks for the reply.
We are now going in the same direction.. Strip-off B and re- install. I guess fixing the current mess is the hardest part than redoing.
My question here is.. While removing IIS from B.. what precautionary steps should I consider in terms of IIS set up related dependencies (if any) between A and B.?? Or the removal process on Server B will safely remove the current setup without doing any harm to server A ( currently active in production)
Just to avoid any surprises and damages.
Please kindly help..
Thanks
Vinod
Thanks
Vinod
Vinod
Well, since the software on B is still referencing components on A, that could be problematic. You could physically shutdown the A system during the uninstall - that would make sure nothing was touched. You could also do a manual uninstall on the B systems.
Instructions for manual uninstallation:
http://pic.dhe.ibm.com/infocenter/iisin ... erver.html
You might want to backup the A Systems first, just to be safe.
Instructions for manual uninstallation:
http://pic.dhe.ibm.com/infocenter/iisin ... erver.html
You might want to backup the A Systems first, just to be safe.
Hi Sorrell,
We have completed the uninstall and re-install on server B. Now we have two stand-alone installations and nothing is shared between them. They are active -active. Now the management is trying to get what is required for SAN based failover. That's a positive sign.
Now the question is..when these two servers are in active-active mode and the switch during failover is completely manual, is there still any need to check the heart beat using scripts and achieve some degree of automation of failover process, if possible??
Thank you for all your answers and support.
Vinod
We have completed the uninstall and re-install on server B. Now we have two stand-alone installations and nothing is shared between them. They are active -active. Now the management is trying to get what is required for SAN based failover. That's a positive sign.
Now the question is..when these two servers are in active-active mode and the switch during failover is completely manual, is there still any need to check the heart beat using scripts and achieve some degree of automation of failover process, if possible??
Thank you for all your answers and support.
Vinod
Thanks
Vinod
Vinod