Friday, January 11, 2013

Add new node in Veritas cluster

Following are the steps for Adding a node :
1) Stop the cluster while leaving applications up and running.
#/opt/VRTSvcs/bin/hastop -all -force
This will leave all monitored applications (databases, etc) up and running but will stop the cluster.
2) Stop gab and llt on all cluster nodes.
#/etc/rc2.d/S92gab stop
#/etc/rc2.d/S70llt stop
3) Unload the gab and llt modules on all cluster nodes.
#modunload -i `modinfo | grep gab | awk '{print $1}'`
#modunload -i `modinfo | grep llt | awk '{print $1}'`
Make sure that no cluster processes(GAB/LLT/HAD) are running on all the nodes.
4) Modify /etc/llthosts, /etc/gabtab, /etc/llttab and /etc/VRTSvcs/conf/config/main.cf (and any other cf that you may require ie types.cf if you added or modified types and Oracle.cf if you are using the Oracle agent, etc) as required to show the new node. These modifications should be done on all nodes.
Make sure that the "set-node" stanza in /etc/llttab is DIFFERENT for each system, and if you have a /etc/VRTSvcs/conf/sysname file, that whatever you set-node to in the llttab matches what is in that file.
#/etc/llthosts
1 Node1
2 Node2
3 Node3
#/etc/llttab
Node1
set-node Node1
set-cluster 1
link hme1 /dev/hme:1 - ether - -
link qfe0 /dev/qfe:0 - ether - -
Node2
set-node Node2
set-cluster 1
link ce2 /dev/ce:2 - ether - -
link ce4 /dev/ce:4 - ether - -
Node3
set-node Node3
set-cluster 1
link hme0 /dev/hme:0 - ether - -
link hme1 /dev/hme:1 - ether - -
/etc/gabtab
/sbin/gabconfig -c -n 3
5) Install the VCS packages on the new system using pkgadd, no need to run installvcs.
Technically you don't have to copy over main.cf
But if you want, you can copy it using
# rcp /etc/VRTSvcs/conf/config/main.cf Node3:/etc/VRTSvcs/conf/config/
Also, as in 4 above, make sure the 'set-node' is correct in /etc/llttab or you will have problems.
6) Start gab and llt on all cluster nodes.
#/etc/rc.d/S70llt start
#/etc/rc.d/S92gab start
7) Start the cluster on all nodes:
#/opt/VRTSvcs/bin/hastart
Check the status
#/opt/VRTSvcs/bin/hastatus.
8) #haconf -makerw
9) #hasys -add Node3
10) #haconf -dump makero
Assuming all went well, you will probably have to add the node to any service group AutoStartList and SystemList , you may use '/opt/VRTSvcs/bin/hagrp -modify' as required.
Below is the example for adding node to Service group Autostart and System Lists¿
##haconf -makerw
# hagrp -modify SG SystemList -add Node3
# hagrp -modify SG SystemList -add Node3
# hagrp -modify SG SystemList -add Node3
# hagrp -modify SG AutoStartList -add Node3
# hagrp -modify SG AutoStartList -add Node3
# hagrp -modify SG AutoStartList -add Node3
11) Now depending upon the resource in the Service Group, update the resource attribute and ensure the new node has the required mount_point and the correct devices ( Nic) are used, matching the existing configuration.
12) haconf -dump makero