Configure PowerHA cluster 6.1

In this post I will follow the main steps for configuring a two-node PowerHA cluster, I have mainly used this cheatsheet and I have stopped to explain some of the steps that it could cause more confusion.

Configure network interfaces and network files

This is one of the most important subjets to build a cluster and a good configuration is going to determine  that cluster works properly.

A traditional PowerHA network configuration consists of the following:

•  Base ip, also known as boot ip. It is used for heartbeating
• Persistent ip, It is an IP alias, which is permanently connected to one of the base addresses of a node. It is optional but I recommend it.
• Service ip, It is configured in a resource group and PowerHA brings it up and down with the corresponding application.

RSCT only monitors the base addresses on the interfaces to determine the status of the adapter and node and take action if required, so it does not monitor the service IP labels (if using IPAT via alias) or the persistent IP labels. You also have the option to configure heartbeat over IP alias.

It is recommended that each cluster have at least one non-IP network defined for each of the nodes in the cluster to prevent cluster partitioning

Configure boot ip address in both nodes,and then configure the persistent address as an IP alias.

All service and persistent addresses must be on a separate subnet from any of the base subnets and the base addresses and the corresponding persistent and service addresses must have the same subnet mask.

Put all of the IP addresses associated with the cluster (boot, persistent, and service ips) into each /etc/hosts and /usr/es/sbin/cluster/etc/rhosts file on each node of the cluster.

I suggest that you edit the /etc/netsvc.conf file to assure that the /etc/hosts file is read before a DNS lookup is attempted.

hosts = local, bind4

These are some general rules/explanations but I recommend that you go through the resources link and read carefully everything that is related to networking.

Configure cluster

 

Define the cluster

Run smitty cm_config_an_hacmp_cluster_menu_dmn

Define the nodes

Run smitty cm_config_hacmp_nodes_menu_dmn

Define the networks

Run smitty cm_config_hacmp_networks_menu_dmn

This defines one network per Ethernet adapter. Use the Pre-defined option as opposed to the Discovered path, but that is up to you. Check the subnet masks for consistency.

The communication path is first used by PowerHA to confirm that the node can be reached, then used to populate the ODM on each node in the cluster after secure communications have been established between the nodes. However, after the cluster topology has been configured, PowerHA can use any interface

to attempt to communicate between nodes in the cluster.

PowerHA no longer requires the hostname to be a resolvable IP label (for example, an address on one of the IP interfaces). For consistency, it is a better idea to use the hostname which also resolves to the persistent IP address associated with each node. However, this is not mandatory.

In this point you can configure your non-ip network or you can do it later.
I like to use Multi-Node Disk Heartbeat, Select Manage Concurrent Volume Groups for Multi-Node Disk Heartbeat and then configure vg for heartbeat
The disk must belong to a SAN network.
You have to initialize the PV header
chdev -l hdiskX -a pv=yes
To monitor heartbeat network use cltopinfo -m but you'll have to use lssrc -ls topsvcs
for monitoring multi-node disk heartbeat

Define boot ip address

Run smitty cm_config_hacmp_communication_interfaces_devices_menu_dmn
This defines the boot IP addresses on the respective network adapters.
Go to Add Pre-defined Communication Interfaces -Communication Interfaces and Select a network.

Define persistent ip address

Run smitty cm_config_hacmp_persistent_node_ip_label_addresses_menu_dmn

Define service ip address

Run smitty cm_config_hacmp_service_ip_labels_addresses_menu_dmn

Perform discovery and reboot

Run smitty cm_extended_config_menu_dmn
Select the Discover PowerHA-related Information from Configured Nodes option, and check for errors to fix. Check the other node if the information has been replicated, if that's not the case run Extended Verification and Synchronization

Generally, rebooting each node can clear up any minor problems,  anyway it is a good test to restart each server 

Define Resource group

Run smitty cm_hacmp_extended_resource_group_config_menu_dmn
Define the resource group with the desired settings, Set the Fallback Policy to Never Fallback.

Create a shared volume group

Run the smitty cl_vg command, and create a shared volume group. When you create a shared volume group, you only need to select one of the nodes, because the disk is shared. But if you happen not to see the disk, select both nodes.

In the option Enable Fast Disk Takeover or Concurrent Access you can select Fast Disk Takeover, Concurrent or No

By selecting Fast Disk takeover you are choosing to create Enhanced Concurrent volume group.

Enhanced concurrent vg can be used in both concurrent and non-concurrent environments.

An enhanced concurrent vg can be varied on in two different modes

• Active mode is similar to a non-concurrent volume group being varied online with the varyonvg command. 

• Passive mode only allows readability of the VGDA and the first 4 KB of each logical volume. It does not allow read/write access to file systems or logical volumes. It also does not support LVM operations.

When the owning node fails, the fallover node simply changes the volume group state from passive mode to active mode through the LVM.

When configuring enhanced concurrent volume groups in the cluster, ensure that multiple networks (IP and non-IP) exist for communication between the nodes in the cluster, to avoid cluster partitioning. When fast disk takeover is used, the normal SCSI reserve is not set to prevent multiple nodes from accessing the volume group.

You can determine if a volume is enhanced concurrent by running lsvg vgname and checking the Concurrent: field; it should be Enhanced Capable or  by running lqueryvg -p devicename -X command.

If the return output is 0, then it is a regular non-concurrent volume group. If the return output is 32, then it is an enhanced concurrent volume group

There are at least two advantages of using fast disk takeover:

• Faster disk acquisition time

• LVM ODM synchronization

Create lv and Filesystem

Run cl_lv and once you have created the logical volume, create the filesystem cl_fs or go directly to create the filesystem.

The difference is that when you create in the first instance the lv, you can choose the name of the logical volume.

When you create lv or fs, don't forget to leave some space free in the vg if you are not going to use Inline log.

Configure the resource group

Run the command smitty cm_hacmp_extended_resource_group_config_menu_dmn.Select the Change/Show Resources and Attributes for a Resource Group option. Then, perform these steps:

• Select the appropriate service IP addresses.

• Select the appropriate shared volume groups and heartbeat disk.

• Select the appropriate application servers.

Perform a synchronization

Synchronize the cluster configuration. Run the command smitty cm_ver_and_sync

Your cluster is ready, you can start services but before handing over the cluster I highly recommend to perform testing by removing cables, powering off both nodes and seeing how the cluster behaves and resources fail over.

Resources: 

http://www.ibm.com/developerworks/aix/library/au-hacmpcheatsheet/ 

http://www.ibm.com/developerworks/training/kp/au-kp-powerha_cluster/index.html?ca=drs-

http://publib-b.boulder.ibm.com/abstracts/sg247739.html?Open

http://www-03.ibm.com/systems/power/software/availability/aix/index.html

http://publib.boulder.ibm.com/infocenter/aix/v7r1/index.jsp?topic=%2Fcom.ibm.aix.powerha.navigation%2Fpowerha_pdf.htm