By Jeff Dixon
November 19, 2010
Connecting an ESX server often involves connecting a number of network runs to a switch. Here I am going to presume to have four data connections for ESX that I’m assigning in the vSwitch for the VM’s to use. I’m also going to presume to be using all Cisco Switches.
If I were to plug these into a switch and let them run as is I would have some advantage over a single link but not as much as I could have. The result is any outbound traffic can use all four links, however, inbound will only use one.
So what if you want to step this up a notch and get the best utilization possible by allowing all four links to process traffic both inbound and outbound. The answer is, I need to create a port-channel on the physical switch. This bonds the four links on the Cisco Switch so that they are bonded as one on both sides. This setup allows you to gain additional bandwidth and redundancy.
The first step is that on the vSwitch, the load balancing method must be set to “Route based on ip hash”. If this is not set, you will not communicate back to the server once you enable the port channel.
The rest is done on the Cisco Switch.
Before we get started on the ports, ensure the load balancing method is in the correct state on the switch:
sh etherchannel load balance
You want this to be set to src-dst-ip, if it’s not you can change it with this command:
port-channel load balance src-dst-ip
Once you have verified your load balancing method it’s time to access your ports:
int range gi1/0/1 -2 , gi1/0/1 -2
Join the ports to the port channel:
Channel-group 1 mode on
During this step, mode options allow you to choose pagp, lacp, and on. ESX does not support either protocol pagp (Cisco Proprietary) or lacp (industry standard). Setting the mode to “on” doesn’t use either protocol and simply sets the port-channel to on.
Additionally, any commands entered onto the port after the channel-group command should also show up on the port-channel interface. Entering this command in the beginning allows you to configure that port while you configure these. Otherwise you will need to go back and configure the port-channel interface.
channel-group 1 mode on will create interface port-channel 1 (the group number determines the interface number)
When using pagp or lacp this is also known as an etherchannel
Set trunking:
switchport trunk encapsulation dot1q
switchport mode trunk
switchport nonegotiate
spanning-tree portfast trunk
ESX doesn’t support DTP so using the nonegotiate command disables DTP
Additionally, you will want to ensure that all settings on each port are the same; if they are not they may not all become active on the port channel.
I noted an issue on a 6509 switch where the flowcontrol on the ports wouldn’t match up by default. On all interfaces I manually set the flowcontrol to resolve, for ex: flowcontrol send off
I would also add a description to the ports for future reference before moving on.
This should be all you need on the interfaces; however, you may desire to add QoS or other commands if needed.
Ensure the port-channel interface shows at least these commands: (sh int port-channel 1)
switchport trunk encapsulation dot1q
switchport mode trunk
switchport nonegotiate
At this point everything should be up.
You can check the status with these commands. If something is not active, likely there is some difference between your ports.
sh etherchannel 1 [sum, detail]
This will show you the port status for each interface in the port-channel
sh protocol | include Port-channel
This will show you the port status of the port-channel interface itself. It should be in an up state.
For more information, there is a very good post about this here:
