Linux includes a bonding driver that allows multiple network interfaces to be merged into a single logical (bonded) interface. Linux bonding is also more generically known as channel bonding, NIC bonding or Ethernet bonding. The type of bonding can be configured at the bonding driver load time. Broadly speaking, the bonded network interfaces can be configured to load balance the traffic, or one of the interfaces can just act as a hot standby, or depending on the switch support (or if you just have direct x-over cables) both interfaces can be aggregated together to form a single channel for greater traffic throughput.
The most authoritative source of documentation on Linux Bonding is at Kernel.org. I have summarized some points from the document and added some of my own comments from my own experiences with examples below. I will add links to detailed worked examples of bonding configuration for the various distributions of Linux in the future.
History Of Linux Bonding Driver
Linux bonding was originally part of the beowulf cluster patches for Linux 2.0 by Donald Becker, but it has changed much since then. Donald was also one of the original developers of the Ethernet drivers for Linux. Most distributions of Linux come with the bonding driver already available as a module.
Is Bonding Supported By My Operating System?
It is a simple check to see if the bonding driver is available in your operating system.
[root@redhat6 ~]# modprobe -l bonding
kernel/drivers/net/bonding/bonding.ko
Does My NIC Support Bonding?
The next question is does my NIC support bonding? Yes all cards can support bonding, but NOT all cards support all of the schemes Linux Bonding has available for monitoring the network link health. However, as we will see this is not a big issue.
MII Monitor (miimon)
Most network cards today implement the Media Independent Interface (MII) standard. Part of the standard defines use of the MII Status Word register which can be used to detect if an Ethernet NIC is connected to the network or not – of particular interest is the “0×0004 Link established” state.
The MII monitor (miimon) simply queries the state of this register (netif_carrier = 1) to determine the health of the network link.
It can also be configured to use ethtool as a last resort in attempting to obtain the same information (netif_carrier=0). So don’t worry if your card does not support MII.
There are two ways to test if your NIC supports MII
[root@redhat6 ~]# mii-tool SIOCGMIIPHY on 'eth0' failed: Operation not supported SIOCGMIIPHY on 'eth1' failed: Operation not supported eth2: negotiated 100baseTx-FD, link ok eth3: negotiated 100baseTx-FD, link ok
In the above example eth0 and eth1 do not support MII but eth2 and eth3 do.
If you operating system does not include mii-tool (SuSE) then you can use ethtool.
NOTE: Interestingly at the writing of this report SuSE does not include mii-tool in the net-tools package and is unlikely to ever do so in the future due to the following statement in the net-tools release note:
*Wed Dec 08 2004 - - Include linux/sockios.h to get the right ioctl [#48873]. This bug is in mii-tool which is not included anymore, but better not risk ever re-adding buggy stuff.
Here is how to use ethtool to determine if your NIC supports MII. Just grep for the string “Current message level:”. If it appears in the ethtool output then MII is supported by your card. In the example below only eth2 and eth3 support MII.
[root@redhat6 ~]# ethtool eth0 | grep "Current message level:" [root@redhat6 ~]# ethtool eth1 | grep "Current message level:" [root@redhat6 ~]# ethtool eth2 | grep "Current message level:" Current message level: 0x00000007 (7) [root@redhat6 ~]# ethtool eth3 | grep "Current message level:" Current message level: 0x00000007 (7)
ARP Monitor
Don’t worry if your NIC cards do not support MII. Most people wrongly assume that miimon is the superior monitor. But remember miimon only checks the local NIC link status. It has no way to establish the health of devices beyond it, such as the health of the switch or of the switch up port. The ARP Monitor on the other hand can do this.
The ARP monitor sends ARP queries to one or more configured IP addresses on the network and uses the response back to indicate if the link is working. A good choice for these configured IP addresses are the routers at the other end of the switches.
NOTE: It’s good practice to define more then one IP address to monitor, especially in high availability environments.
In the example above (from my ESXi hypervisor test environment) for vSwitch1 the uplink vmnic1 is down (I just pulled the cable from my ESXi host) but all the virtual switch ports are still active – and ethtool and mii-tool report the links as still active. I have tested miimod monitoring with this configuration and it just will not work. However, ARP monitoring works perfectly.
NOTE: It should be noted that some switches can fail ports in response to general switch failures in which case miimod monitoring would work fine.
Bonding Manual Failover
You can perform manual failover of your active bonded slave for the active-backup mode of bonding.
You can determine the state of your bonding driver from the /proc pseudo-file system.
[root@redhat6 ~]# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.5.0 (November 4, 2008)
Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth2
MII Status: up
MII Polling Interval (ms): 0
Up Delay (ms): 0
Down Delay (ms): 0
ARP Polling Interval (ms): 1000
ARP IP target/s (n.n.n.n form): 192.168.1.1
Slave Interface: eth2
MII Status: up
Link Failure Count: 5
Permanent HW addr: 00:0c:29:f1:d9:a5
Slave Interface: eth3
MII Status: down
Link Failure Count: 303
Permanent HW addr: 00:0c:29:f1:d9:af
You can also use the ifenslave to change which interface is active.
[root@redhat6 ~]# cat /proc/net/bonding/bond0 | grep eth2 Currently Active Slave: eth2 Slave Interface: eth2 [root@redhat6 ~]# ifenslave --change-active bond0 eth3 Master 'bond0', Slave 'eth3': Error: Change active failed
In the above example eth3 was down so the bonding driver prevented me failing over to it. After I plugged it back in and could see in /var/adm/messages that it’s link status was up again.
Mar 14 14:13:28 redhat6 kernel: bonding: bond0: link status definitely up for interface eth3.
Now the manual failover works!
[root@redhat6 ~]# ifenslave --change-active bond0 eth3 [root@redhat6 ~]# cat /proc/net/bonding/bond0 Ethernet Channel Bonding Driver: v3.5.0 (November 4, 2008) Bonding Mode: fault-tolerance (active-backup) Primary Slave: None Currently Active Slave: eth3 MII Status: up MII Polling Interval (ms): 0 Up Delay (ms): 0 Down Delay (ms): 0 ARP Polling Interval (ms): 1000 ARP IP target/s (n.n.n.n form): 192.168.1.1 Slave Interface: eth2 MII Status: up Link Failure Count: 5 Permanent HW addr: 00:0c:29:f1:d9:a5 Slave Interface: eth3 MII Status: up Link Failure Count: 304 Permanent HW addr: 00:0c:29:f1:d9:af
You may also like -
About Danny W Sheehan
Danny has over 25 years in the IT industry and loves to blog about how to setup computer software, hardware, electronics and gadgets in general.
Comments are closed.