Environment Isolation

A forum for discussing DataStage<sup>®</sup> basics. If you're not sure where your question goes, start here.

Moderators: chulett, rschirm, roy

Post Reply
thompsonp
Premium Member
Premium Member
Posts: 205
Joined: Tue Mar 01, 2005 8:41 am

Environment Isolation

Post by thompsonp »

On a single large physical server what are the best ways to isolate environments?
For example we might have dev, test and uat environments.
I could have a simple install with all non client tiers on the one large server, then create a dev, a test and a uat project.
Users could have different logins for each project to isolate their access through the client.

Would installing a separate engine for each environment give any better isolation if the engines are still all on the same server or does it just make things more complicated?
With multiple engines is it possible to upgrade one environment e.g. with a fix pack and not the other, or does sharing other tiers mean they all have to be upgraded at the same time?

Are there any licensing implications for installing more than one engine on the same server?

Has anyone come across installation instructions for having multiple engines on the same server? I did find a post that said they had such instructions but it's from years ago and a very old version.

Is there a better solution to isolating environments on a single server?
Has anyone here done something similar and if so what were the pros and cons?

Thanks
qt_ky
Premium Member
Premium Member
Posts: 2895
Joined: Wed Aug 03, 2011 6:16 am
Location: USA

Post by qt_ky »

What is your goal of isolation? Is it purely for security reasons?

You can install multiple instances on the same server or logical partition (LPAR) but it definitely makes everything more complicated to manage and maintain, especially related to using non-default port numbers beyond the first instance. Your metadata repository version, if hosted on the same server, can easily become common across all your instances. I would not recommend this approach.

What are your virtualization options on your hardware and operating system?

The licensing is based on processing capacity--CPU cores.

One thing we have done at the hardware level is to carve out the licensed number of cores into a shared cpu pool, which is shared by multiple LPARs. That allows us to stand up a new LPAR having a newer software version, side by side with the existing version (e.g. 11.3 and 11.5). Then we can stay compliant with licensing while we migrate from old to new, while avoiding the huge headache of multi-engine installations on the same LPAR.

There are many security, firewall, and ongoing maintenance benefits to running each instance on the default ports.

We can even assign each LPAR a different priority. Talk to your local server people to find out your options.
Choose a job you love, and you will never have to work a day in your life. - Confucius
thompsonp
Premium Member
Premium Member
Posts: 205
Joined: Tue Mar 01, 2005 8:41 am

Post by thompsonp »

We have limitations on the size of virtual servers (a la carte from 3rd party).
The physical servers are larger and cheaper.

Two goals for isolation; security and performance impact from other projects / environments.
Security is the prime concern though.

I think the most flexible approach would be virtual servers if we had the level of control over the configuration that allowed us to quickly move resources between them and fire up a new one. Unfortunately that's not something it looks like we will be offered.

It would be RHEL and if virtual ESXi.
PaulVL
Premium Member
Premium Member
Posts: 1315
Joined: Fri Dec 17, 2010 4:36 pm

Post by PaulVL »

As an admin, I would say put each env on a separate host. I look at it from a patching point of view.

If you simply have 3 projects, then you have 1 engine. And a patch should never be applied to PROD without testing in DEV first.

Of course that 3 HW setup will cost you more.

I don't like to install 3 engines on the same host. Port issues will arise as well as /.dshome issues in the ibm scripts.

Super security will be hard if you have all three under the same engine since the engine group credentials will be shared. Lots of the underlying directories rely upon that group RW permission.

So... I vote 3 hosts.
UCDI
Premium Member
Premium Member
Posts: 383
Joined: Mon Mar 21, 2016 2:00 pm

Post by UCDI »

datastage is pretty hacktastic also. You can, for example, write a job that exposes the password in the logs, snagging the your app-id creds for personal use. You can also write a sequence job to use the ap-id creds to issue unix commands. The nature of the app-id is to have a good amount of access into the system, and a troublemaker could do quite a bit with that if so inclined. That is only 2 really basic examples of things you can do... there are quite a few loopholes most of them stemming from the access level that datastage has on its server. I won't even go into what you can do by calling a C program from DS at that access level.
thompsonp
Premium Member
Premium Member
Posts: 205
Joined: Tue Mar 01, 2005 8:41 am

Post by thompsonp »

qt_ky wrote:One thing we have done at the hardware level is to carve out the licensed number of cores into a shared cpu pool, which is shared by multiple LPARs. That allows us to stand up a new LPAR having a newer software version, side by side with the existing version (e.g. 11.3 and 11.5). Then we can stay compliant with licensing while we migrate from old to new, while avoiding the huge headache of multi-engine installations on the same LPAR.
If you have a physical server where all the cpu are licenced and then you create several virtual environments on top of that giving the advantages you have described above, is there much benefit in using Information Server workload management (given that this would be at each virtual server level)?

It could give some control within each virtual environment, for example if there are multiple projects and one takes priority over another, but wouldn't it be the virtualisation software / tools that have to monitor the state of each virtual server and the physical server as a whole to alert if it looks like too much is running across the virtual servers.

One further licence based question. If we have the system as described above and we then want to add QualityStage, can we licence QualityStage with sub capacity licensing on just one virtual server whilst having DataStage still licensed on the whole physical host? Having read the licence agreement I believe this is allowed though it doesn't specifically mention mixing the two methods (full and sub capacity).
qt_ky
Premium Member
Premium Member
Posts: 2895
Joined: Wed Aug 03, 2011 6:16 am
Location: USA

Post by qt_ky »

WLM helps protect the DataStage server from overloading, such that a newly run job on an overloaded system is queued instead of running and potentially crashing due to lack of resources, like memory. I like to run it on each Information Server instance. While I know it is not necessary, I can say the newer servers where we do run WLM have been overall more stable than when we ran old versions without WLM. That of course may also be due to improvements in the server code. The WLM benefit may be real or perceived!

There are also some suggestions that if you have multiple teams that you can configure DataStage WLM to prevent any one team of developers from hogging the system. I am not convinced that is actually possible due to being able to choose any queue you want to at run time.

See this document for more info on WLM: "DataStage Workload Management Server Overview and Best Practice" (good luck with this URL, or go to ibm.com and search on the doc title):
https://www.ibm.com/developerworks/comm ... t+Practice

Within each of your Information Server instances, you are going to have many tools, utilities, and commands available to monitor that instance. If you are doing LPARs then you also have access to various commands to check on LPAR health and utilization. You may likely not have visibility beyond that.

If you own or manage the hardware and have good control over it, you're in good shape. But if some other group manages your systems, then they may likely, without your knowledge or consent, choose to pile on 10, 20, or more additional LPARs onto your physical server until someone starts to complain (depends on your organization). In fact that's one big complaint about all these virtualization technologies--it's very easy to underpower and overload them.

I do not know about mixing license types. Check with an IBM sales person to be sure. If you do subcapacity licensing (LPARs) then IBM requires you to install their IBM License Metric Tool, which carries some overhead. It has an agent that is always running and consuming a fraction of your CPU. You are responsible for licensing the software according to how many cores are available. The license metric tool would detect, for example, if one day your QualityStage had 8 cores available to it and then another day it had 12 cores available.
Choose a job you love, and you will never have to work a day in your life. - Confucius
Post Reply