Gracefully shutting down DataStage.

[rkashyap]

Existing process to shutdown DataStage server involves:

Shutdown DataStage (Steps 1 thru 4)
1. Ensure that no users are connected. ("Disconnect All" on ISConsole Administartion Tab).
2. Ensure that no jobs are running (Sometimes this involves killing jobs).
3. Execute command "ps -ef|grep dsapi" ... to identify and kill all "zombie" sessions.
4. Execute following commands:

Code: Select all

 DSHOME=/opt/IBM/InformationServer/Server/DSEngine; export DSHOME
/opt/IBM/InformationServer/Server/DSODB/bin/DSAppWatcher.sh -stop
/opt/IBM/InformationServer/Server/DSEngine/bin/uv -admin -stop
/usr/local/bin/sudo /opt/IBM/InformationServer/ASBNode/bin/NodeAgents.sh stop
/usr/local/bin/sudo /opt/IBM/InformationServer/ASBServer/bin/MetadataServer.sh stop

5. At this point Unix admin runs a shutdown script that also shuts down Oracle/XMETA.


Steps 1 thru 3 listed above require manual intervention. Admins are requesting a script that will allow them to automate the shutdown (i.e. automate steps 1 thru 3).

Please advise ... Is this process automated in your organization? What is the best way to do this?

[qt_ky]

Here is the sequence I put into an script from IS 8.7 on AIX w/ DB2 (Note: my IS install path is non-default). Normally after the script runs, we still manually check for processes that didn't get stopped as expected, and kill -9 on each of those, unless we reboot the LPAR, then we just reboot.

Code: Select all

# stop ISD applications
...calls another script with commands to undeploy each web service

# stop the AppWatcher process
/opt/IBM/IS01/Server/DSODB/bin/DSAppWatcher.sh -stop

# stop the DataStage engine
# source the dsenv file
. /opt/IBM/IS01/Server/DSEngine/dsenv
/opt/IBM/IS01/Server/DSEngine/bin/uv -admin -stop
sleep 60

# stop the agents
/opt/IBM/IS01/ASBNode/bin/NodeAgents.sh stop
sleep 30

# stop WAS
/opt/IBM/IS01/ASBServer/bin/MetadataServer.sh stop

# stop DB2 (repository)
...command to stop database server, if needed (i.e. db2stop)

# clean up and removing temporary files
...calls another script to clean out various tmp directories, HA_STATUS file directories under each project, etc.

[rkashyap]

Eric, Thanks for the response.

It is my understanding that if any users are connected before the shutdown, then it will result in problem while restarting because of connections stuck in a fin_wait2 status. Please advise, how do you take care of this?

[lstsaur]

You issue a
1. lsof -P
2. find the pid of that fin_waite2
3. kill that pid
4. restart the DS.

[asorrell]

You should try to get rid of the clients that are logged into DataStage using the following methodology:

1) Send email or IM to get active users to logout gracefully
2) Use Disconnect All from the Web Console
3) Use "kill -15 pid" for all dscs or slave processes on the server. Normally if you kill all of one type (example dscs) the other (slave) will exit automatically. This "kill" allows processes to exit gracefully, cleaning up all resources.
4) If kill -15 doesn't work, try kill -4, a slightly different graceful disconnect.
5) If kill -4 doesn't work, use kill -9. This is "bang you are dead - don't cleanup" and always works, but realize that the "kill -9" processes will take an additional 15 minutes to exit the FIN_WAIT state before you can "reboot" DataStage engine. If you reboot the operating system those go away immediately.

[rkashyap]

Andy, Thanks.

Administrators are requesting this script to support Scheduled maintenance. Users would have been notified days in advance of this. At the time of maintenance, we are ok with terminating any active user sessions. Admins wanted to minimize manual steps involved.

In order to automate the termination of user connections, we have tried SessionAdmin command as given below:

Code: Select all

$ cd /opt/IBM/InformationServer/ASBNode/bin
$ ps -ef|grep dscs
   dsadm  9388  8344   0 09:33:13 pts/1       0:00 grep dscs
 evbp637  8181 26002   0 09:28:14 ?           0:00 dscs 5 10800 0
 evbp637  9376 26002   0 09:32:56 ?           0:00 dscs 5 10800 0
$ ./SessionAdmin.sh -user dsadm -password XXXXXXXX -kill-user-sessions
Info All user sessions killed.
$ ps -ef|grep dscs
   dsadm  9572  8344   0 09:33:47 pts/1       0:00 grep dscs
 evbp637  8181 26002   0 09:28:14 ?           0:00 dscs 5 10800 0
 evbp637  9376 26002   0 09:32:56 ?           0:00 dscs 5 10800 0

As you can see this command gave message "Info All user sessions killed", but did not do so. Are we missing something in this command?

[asorrell]

You are using the command correctly, it just doesn't clear everything. I think it only kills real, active sessions, leaving the "zombies" (disconnected sessions). I got so frustrated with it I stopped using it.

[rkashyap]

Andy, Thanks for your response.

I guess my workaround for this issue is ... create a shell script to kill all user sessions.

[lstsaur]

Still that wouldn't solve your fin_wait_2 problem.

[rkashyap]

Pardon my ignorance ... it is my understanding that if all the client sessions are closed/killed prior to shutting down the DataStage engine, then the restart is smooth (dsrpcd deamon is able to bind to the ports). Am I missing something?

[lstsaur]

The problem is that fin_wait_2 has already occurred and stuck on the server side before you even realize there is a problem. You have to clear up this problem first manually.

Of course IF you can get "ALL" the client sessions closed/killed prior to shutting down the DataStage engine, then no problem. But think about if you have hundred thousands users and spread around all over the world.

[rkashyap]

We had asked IBM about "SessionAdmin.sh -kill-user-sessions" not killing any user sessions. IBM's response is

SessionAdmin.sh -kill-user-sessions is basically the command line equivalent to the Web Admin UI Session Management Disconnect All IS sessions function.

SessionAdmin.sh -kill-user-sessions kills the active IS sessions. It does not kill DS job processes. A DS job only has an active IS session when starting a DS job and using domain authentication - ie, the associated IS session is active just long enough to validate an authentication. Thereafter, there is not an active IS session for the job while it is running.

---

Going back to the main topic ... we have created following script to shutdown DataStage

Code: Select all

DSHOME=/opt/IBM/InformationServer/Server/DSEngine; export DSHOME 

# Stop App Watcher 
/opt/IBM/InformationServer/Server/DSODB/bin/DSAppWatcher.sh -stop 

# Kill User Sessions 
/opt/IBM/InformationServer/ASBNode/bin/SessionAdmin.sh -user dsadm -password XXXXXXXX -kill-user-sessions
DS_ALL=`pgrep "dsapi|dscs|dsjob"`; if test -n "$DS_ALL"; then kill -15 $DS_ALL; fi; sleep 30
DS_ALL=`pgrep "dsapi|dscs|dsjob"`; if test -n "$DS_ALL"; then kill -4  $DS_ALL; fi; sleep 30
DS_ALL=`pgrep "dsapi|dscs|dsjob"`; if test -n "$DS_ALL"; then kill -9  $DS_ALL; fi; sleep 30

# If any user session is found then abort shutdown
DS_ALL=`pgrep "dsapi|dscs|dsjob"`; if test -n "$DS_ALL" ; then echo "Cannot shutdown; Active sessions found " $DS_ALL; exit 4; fi


# Stop DSEngine
/opt/IBM/InformationServer/Server/DSEngine/bin/uv -admin -stop  ; sleep 30

# Stop Node Agents
/opt/IBM/InformationServer/ASBNode/bin/NodeAgents.sh stop       ; sleep 30

# Stop WAS
/opt/IBM/InformationServer/ASBServer/bin/MetadataServer.sh stop ; sleep 30

This is followed by DBA script to shutdown Oracle/Xmeta.

I am marking this post as resolved. Appreciate your help.

[ray.wurlod]

Basically SessionAdmin.sh -kill_user_sessions exists to disconnect any actively connected clients.

[mmonahan]

# Put InfoSphere into maintenance mode
/opt/IBM/InformationServer/ASBServer/bin/SessionAdmin -url https://server_hostname:server_port -authfile <path to file with user/password> -set-maint-mode ON

# Stop App Watcher
/opt/IBM/InformationServer/Server/DSODB/bin/DSAppWatcher.sh -stop

# Kill User Sessions
/opt/IBM/InformationServer/ASBServer/bin/SessionAdmin -url https://server_hostname:server_port -authfile <path to file with user/password> -kill-user-sessions

Putting the system into maintenance mode will stop new connections while shutting down. Don't forget to turn it off on restart.

The authfile will allow you to get the credentials out of the script.

And, finally, the v11 syntax uses the -url parm.