https://access.redhat.com/articles/146163

What are the different methods used for communication with RHEL High Availability?

Updated 2016年六月1日01:28 - 

English 

There are 4 types of IP communication used by Red Hat Enterprise Linux(RHEL) High Availability.

• Broadcast: A packet is sent to the subnet's broadcast address and all nodes in the subnet will see that message. All of those cluster nodes need to make a decision whether to take any notice of that packet or not. Broadcast packets are not routed, they are just transmitted all over the network.
• Multicast: A packet is sent to a nominated multicast address. Interested cluster nodes subscribe to multicast addresses and so those that do not subscribe to the cluster multicast do not see the packets. Multicast packets can be routed (though many switches do this badly by default).
• Unicast: A packet is sent to a specific IP address, only that particular cluster node will see it. These messages can be routed.
• Stream: The above messages are UDP packets, streams are a TCP thing and always unicast. Pairs of nodes establish a connection between themselves and exchange a stream of bytes in either/both directions.

NOTE: The 

iba

 transport for corosync is not supported: Is the 

iba

 transport supported in a RHEL 6 or 7 High Availability cluster?

Red Hat Cluster Suite 4+

The service 

cman

 in RHEL 4 used broadcast packets by default. This is mainly because it was the easiest to code and to set up and also the protocol was very simple and so did not generate much traffic. Due to the way the protocol was implemented it wasn't even capable of generating much traffic. Since there was not much traffic generated then broadcast was fine for this sort of application. RHEL 4 does have a multicast option that can be enabled in the 

/etc/cluster/cluster.conf

.

Some packets in RHEL 4 were UDP unicast packets. When 

cman

 knew exactly which cluster node to send data to (eg ACKs) then they were sent directly to that cluster node using a unicast packet.

Red Hat Enterprise Linux Server 5 (with the High Availability Add on) using 

openais

Clustering in RHEL 5 was rewritten from scratch. Red Hat integrated 

openais

 as the major part of the cluster stack. This automatically forced the product to use multicast as that was the only type of communication that 

openais

 could use at the time. 

openais

 also has a much more complex protocol than RHEL 4 

cman

 and thus isolation of traffic from all cluster nodes in a subnet is more important. Without proper isolation using 

IGMP

, cluster traffic could be sent to all nodes, even those not part of the cluster, causing traffic overload.

As in RHEL 4, not all 

openais

 packets are multicast. Quite a lot are actually unicast as 

openais

 is a ring protocol and packets are sent round the ring. The join part of the protocol is one that does use multicast though, so failure of multicast routing can be seen quite quickly in an 

openais

cluster as cluster nodes fail to join.

The broadcast transport was added to 

openais

 and 

corosync

 to help alleviate multicast configuration and functional errors with switches. Broadcast was actually added to openais and corosync to enable deployments where multicast was not functional. Broadcast was originally not supported because it transmits traffic to all nodes in the network resulting in overload and broadcast is incompatible with IPv6.

Red Hat Enterprise Linux Server 6 (with the High Availability Add on) using 

corosync

By default RHEL 6 uses multicast for default transport.

In RHEL 6.2 UDPU option was added to corosync as fully supported transport which already included broadcast and multicast. UDPU helps to resolve the multicast and broadcast issues customers were facing by using UDPU which uses all unicast UDP packets.

All unicast UDP packets are sent directly to targeted cluster nodes. The packets that would have been broadcast or multicasted to all cluster nodes now have to be sent N-1 times (where N is the number of cluster nodes in the cluster) so there is a traffic increase compared to the other systems, but this is often a decent tradeoff, particularly for smaller clusters. One restriction on UDPU clusters is that all cluster nodes of the cluster need to be specified in 

cluster.conf

 (if using only 

corosync

 and not 

cman

 then specified in the 

corosync.conf

) so that the cluster node addresses are known in advance. This means there is an administrative as well as a traffic overhead for 

<cman transport="udpu"/>

 in 

/etc/cluster/cluster.conf

,

Red Hat Enterprise Linux Server 7 (with the High Availability Add on) using 

corosync

By default RHEL 7 uses UDPU as default transport.

What factors can affect cluster communication?

There are various other factors that can affect the different traffic protocols such as what other subsystems are also in use. Below is a description of various services or protocols that can be directly or indirectly involved with cluster communication performance:

DLM

 has always used TCP streams for communication, so it is not directly involved in this. It is indirectly involved when 

plocks

 are used. The 

plocks

 that are used by 

GFS

 and 

GFS2

 file-system use 

openais

 checkpoints(since RHEL 5). These checkpoints exchange information over the 

openais

 or

corosync

 TOTEM protocol and so are now affected by the multicast, broadcast, or UDPU issues. For a cluster with high 

plock

 usage when using a 

GFS

 or

GFS2

 file-system there can be communication issues because of the extra traffic that is generated with broadcast or UPDU which makes these protocolsunsuitable for these configurations. For small scale clusters or clusters not using 

plocks

 on a 

GFS

 or 

GFS2

 file-system then broadcast or UDPU could be used. In RHEL 4 

plocks

 were implemented on top of the 

DLM

. 

plocks

 are range locks with odd semantics and the implementation over 

DLM

 was not an efficient use of the 

DLM

 API. This meant that 

plock

 traffic was sent over the TCP streams of the 

DLM

 and not subject to the issues discussed above.

* Another factor that can affect communication is when network switches put a *cap on the number of multicast packets. This can cause communication issues(and possible fencing of cluster node) when that cap is exceeded.

* 

lvm

 is the other notable user of the cluster protocol. The service 

clvmd

 is a very minimal user actually. It just sends locking notification messages when metadata is altered so there is no serious impact on performance and 

clvmd

 should work quite happily in a multicast, broadcast or UDPU installation.

* The service 

cmirror

 is a different thing altogether. 

cmirror

 use the 

openais

 or 

corosync

 CPG protocol to share information and on a busy system with many mirrors this can be quite a lot of traffic, especially when cluster nodes reboot or resyncs are necessary. For this reason multicast is recommended for 

cmirror

 installations.

* The filesystem 

GFS

 and 

GFS2

 does not have any communication mechanism of its own. 

GFS

 and 

GFS2

 use 

DLM

 which uses TCP streams.

* The service 

rgmanager

 will normally use the the 

DLM

. However in multi-homed clusters it will instead use it's own simplified lock manager called cpglock, which implements basic locking over the 

corosync

 CPG protocol. So this traffic will be subject to the multicast, broadcast, or udpu effects mentioned above. 

rgmanager

 does not generate a lot of lock traffic though, so should be fine with any of the networking systems.

• 产品
• Red Hat Enterprise Linux