Fix: HP Virtual Connect Flex-10 - ESX 4.0 U1 in an Active/Active Configuration does Not Failover Using SmartLink

I’ve been troubleshooting a ESX implementation on a solution based on HP c7000 blade enclosure, which had two HP Virtual Connect interconnect modules builtin. The blade server used was HP Proliant DL460c G6. I checked the VMware HCL and noticed some requirements to get this to work on vSphere 4.0 Update 1.

Output from VMware HCL:

Notice that ESX 4.0 U1 is supported but there are a couple notes. One is to install a specific driver - esx40-net-bnx2x_400.1.48.107-1.0.4. I downloaded and installed the driver on all the ESX servers.

Now I wanted to test the failover. In the HP Virtual Connect Manager I disabled the Shared Uplink Set for Bay 2. I had already setup a continous ping and verified that I still had connectivity to both service console and to VMs running on the hosts, through interconnect bay 1. Test was succesful.

Then I switched arround, enabled Shared Uplink Set 2 and disabled Shared Uplink Set 1 for Bay 1. This time I lost connectivity to both service console and ESX hosts and even I waited a couple minutes, it never came up. I had one other blade server on that enclosure, that was running Microsoft Windows, which I didnt have any problems connecting to.

So I thought the reason was with the ESX configuration.

After I verified all settings on both the ESX hosts ,HP Virtual Connect and the physical switches, which all were identical configured in regards to both interconnect bays, I decided to call HP Support on this issue. I was referenced to a public advisory stating that HP Virtual Connect Flex-10 - ESX 4.0 U1 in an Active/Active Configuration does Not Failover Using SmartLink.

The solution is the following three action points:

1. Verify that the firmware on HP Virtual Connect was running 2.30 as minimum. This setup was running with 2.32 (newest version)
2. Verify that the NIC driver version was Broadcom NetXtreme II Ethernet Network Controller driver 1.52.12.v40.3 (minimum) for ESX/ESXi 4.0. This was different from what the VMware HCL stated.
3. Verify that the NC532i/m bootcode version 5.0.11 (minimum). The bootcode on the NC532 was NOT up-to-date on each blade.

I updated both the NIC driver in ESX and the NIC bootcode with the HP Firmware Maintenance CD and after a reboot, failover was working just as expected. It is recommended by HP to update the bootcode after the NIC driver is installed on the ESX server.

I have NOT been able to find the public advisory article on the HP website on this in regards to VMware vSphere, hence this article.

Enable VMotion on the vmkernel interface by commandline

To fully automate an installation of ESX server 3.5 certain esxcfg- commands needs to be used. However these commands doesnt allow us to enable VMotion on a vmkernel interface. To do this in a command line use the vimsh command.

Before enabling VMotion create the neccesary virtual switches and vmkernel interfaces using esxcfg-vswitch and esxcfg-vmknic commands

Examples:

To create a new vswitch called vSwitch10 do:

• esxcfg-vswitch -a vSwitch10

To link a vmnic to vSwitch10 do:

• esxcfg-vswitch -L vmnic1 vSwitch10

To create a portgroup called VMotion on vSwitch10 do:

• esxcfg-vswitch -A VMotion vSwitch10

To create the vmkernel interface on the VMotion portgroup do:

• esxcfg-vmknic -a -i 10.0.0.1 -n 255.255.255.0 VMotion

To enable that interface as the VMotion interface. First thing we need to know is the interface name of the vmkernel port. To find that do:

• esxcfg-vmknic -l

This lists all the vmkernel interfaces that is created on the ESX host. Its likely that its called vmk0 because its probably the first vmkernel interface created

Now Enable the vmkernel interface to used by VMotion do this:

• vimsh -n -e “/hostsvc/vmotion/vnic_set vmk0″

All done. Now verify in the VirtualCenter client that your vmkernel interface now is enabled for VMotion.

Note: The vimsh command does only work in ESX 3.5 and not 3i unfortunately as 3i doenst have a service console. To use esxcfg- commands with 3i install the VMware Infrastructure RCLI Tool created by VMware.

Another Security guide for VMware ESX Server

I often get the questions related to ESX server security and often its related to network and how ESX server does things around networking within the ESX system. DISA has released a whitepaper called ESX Server Security Technical Information Guide (STIG).

One of the core missions for the Defense Information Systems Agency (DISA) Field Security Operations (FSO) is to secure Department of Defense (DoD) Computing systems.

Read it here

I highly recommend to read this whitepaper.
 

DMZ virtualization with VMware Infrastructure

Many network engineers is worried about virtualizing DMZ servers on the same physical ESX servers as their production servers. VMware has released a whitepaper on best practices on DMZ virtualization with VMware Infrastructure:

Read it here

VMware Infrastructure 3 in a Cisco Network Environment

Cisco has written up a document regarding how to setup VMware VI3 environments connected with a physical Cisco switched network.

This document is intended for network architects, network engineers, and server administrators interested in understanding and deploying VMware ESX Server 3.x hosts in a Cisco data center environment.

Read it here

It’s pretty good. Enjoy it!