As I've been building up my lab, I needed to get a backbone switch that would not only support my production/lab network traffic but also iSCSI and to do iSCSI right you need jumbo frames. Lo and behold, I was able to get my hands on a 24 port 3Com 4200G. You might wonder why I would want such a thing (and trust me... it's not your father's Cisco) but I was able to get it for free and it's fully managed (and they're only about $150 on Ebay). HP actually recently released an updated firmware and boot code for it which fixed a few bugs (get it here)- and it was their first one in something like five years. These might be made by the black sheep of the networking world (I don't know many that actually like 3Com switches) but they support many of the functions that I'll need for my lab - namely link aggregation and jumbo frames.
Speaking of link aggregation, let me tell you about how I thought 3Com implemented ether-channel and then actually show you the hidden way to do it. Basically my set-up is as follows: I have an ESXi host with four NICs. Two for production traffic and eventually the other two will be for iSCSI. I'm running 5.1 and my vSwitch is set up as follows:
Pretty standard - nothing extra going on. So I head on over to my 4200G and create, what I thought was, a load sharing link-aggregation group.
[4200G] link-aggregation group 1 mode manual
[4200G] interface GigabitEthernet 1/0/11
[4200G-GigabitEthernet1/0/11] link-aggregation group 1
[4200G-GigabitEthernet1/0/11] interface GigabitEthernet 1/0/12
[4200G-GigabitEthernet1/0/12] link-aggregation group 1
[4200G-GigabitEthernet1/0/12]
Now since ESXi doesn't support LACP without a vDS and I wouldn't want to use a vDS on my management network, etherchannel is my only option and I thought what I was doing created that. Well turns out I was wrong. I went to deploy my vCenter appliance and then went over to see what my networking usage was out of the blue. Suddenly I noticed that my second NIC... hadn't been used at all over the last hour and I know it should've been. I started to investigate between the 3Com's config and the setting on the ESXi server. All seemed correctly set up but just the odd behavior of all the traffic going down one pipe for every VM that I created. I did a "show interface" on the 3Com port that my server's second NIC was plugged into... I then saw something interesting:
[4200G]display interface GigabitEthernet 1/0/12
GigabitEthernet1/0/12 current state : UP
IP Sending Frames' Format is PKTFMT_ETHNT_2, Hardware address is 0024-731c-1d4e
Media type is twisted pair, loopback not set
Port hardware type is 1000_BASE_T
1000Mbps-speed mode, full-duplex mode
Link speed type is autonegotiation, link duplex type is autonegotiation
Flow-control is not enabled
The Maximum Frame Length is 9216
Broadcast MAX-pps: 1488100
Unicast MAX-ratio: 100%
Multicast MAX-ratio: 100%
Unknown Multicast Packet drop: Disable
Unknown Unicast Packet drop: Disable
Allow jumbo frame to pass
PVID: 1
Mdi type: auto
Port link-type: trunk
VLAN passing : 1(default vlan), 2, 4, 20
VLAN permitted: 1(default vlan), 2, 4, 20
Trunk port encapsulation: IEEE 802.1q
It belongs to a link-aggregation
Master port: GigabitEthernet1/0/11
Last 300 seconds input: 2 packets/sec 1957 bytes/sec
Last 300 seconds output: 2 packets/sec 322 bytes/sec
Input(total): 274612 packets, - bytes
- broadcasts, - multicasts, - pauses
Input(normal): 274612 packets, 23868716 bytes
2463 broadcasts, 3080 multicasts, 0 pauses
Input: 0 input errors, 0 runts, 0 giants, - throttles, 0 CRC
0 frame, 0 overruns, 0 aborts, - ignored, - parity errors
Output(total): 540857 packets, - bytes
- broadcasts, - multicasts, - pauses
Output(normal): 540857 packets, 669171676 bytes
11800 broadcasts, 99165 multicasts, 0 pauses
Output: 0 output errors, - underruns, - buffer failures
0 aborts, 0 deferred, 0 collisions, 0 late collisions
- lost carrier, - no carrier
Not quite sure what a "master port" was, I went to ask wise old Google. It came back with links that referenced 3Com link-aggregation ports and the best I could discern was that when a link-aggregation group is created, a master port is chosen by the lowest port number in the group and all traffic is sent through that port and the others are used for "backup".
This bummed me out a little and I spent the next hour playing with LACP and this link-aggregation group setting to see if I could make it work. Nothing... I could not get my ESXi server to send or receive data on both ports at the same time no matter what I did. Last resort I did what any good network engineer would do:
[4200G] ?
I'm not quite sure of what I was looking for but I came across a curious command:
port-group
Huh?
[4200G]port-group ?
INTEGER<1-100> Specify port-group ID
[4200G]port-group 1
With a ? I can see that I can add ports to this port group. I asked Google again what this "port-group" command was but there was a big fat zero hits on it. Eventually I just figured it couldn't hurt to try it so I blew up the link-aggregation group and threw my two ports into this new "port-group" and logged into each of my server to do a transfer from a NAS to test.
[4200G] undo link-aggregation group 1
[4200G] interface GigabitEthernet 1/0/11
[4200G-GigabitEthernet1/0/11] port-group 1
[4200G-GigabitEthernet1/0/11] interface GigabitEthernet 1/0/12
[4200G-GigabitEthernet1/0/12] port-group 1
[4200G-GigabitEthernet1/0/12]
Here's the results... Check it out:
"Houston, we have lift off". I couldn't believe it. Not only is there NO reference to this command anywhere in the manuals that I've read but it seems to be mimicking Cisco proprietary non-LACP etherchannel. I'm still pretty stunned and happy to say the least.