VMware vCenter and SNMP monitoring support with Sysorb

I recently had a case at a customer site, where they wanted to use Sysorb to monitor their VMware vSphere environment. I wrote this article, as I couldn’t find any useful information on Sysorb and VMware vSphere together.

SNMP was configured on the ESX servers and VMware vSphere MIBS was imported into the Sysorb tool.

However trying the same for the vCenter server, didn’t give the expected results. I configured some vCenter configuration alarms to test alarms to verify that SNMP trap where sent out. The Alarms and Event tab in vCenter stated that the SNMP trap was sent, however none were received in Sysorb.
I then installed trap receiver from trapreceiver.com and verified through this tool that the alarms where actually sent out and received from the vCenter server.

I wasn’t familiar with how Sysorb was working and had to read up and investigate how it used SNMP to monitor their server and network devices. I found that Sysorb is NOT supporting SNMP traps, but only works through the SNMP GET function. Basically what this means is, that the Sysorb tool was polling the target for status on preconfigured thresholds/triggers in the Sysorb tool.

Since vCenter itself monitors for configured alarms/thresholds/triggers and therefore sending out an SNMP trap if configured, the 2 systems would not work properly together.

I looked in the VMware vSphere documentation for support for SNMP GET and it was stated that support for SNMP GET on vCenter was not available. VMware vCenter only supports SNMP traps for sending out alarms, which also makes perfect sense to me.’

If any has built any workarrounds to get this to work, please let me know.

Fix: HP Virtual Connect Flex-10 - ESX 4.0 U1 in an Active/Active Configuration does Not Failover Using SmartLink

I’ve been troubleshooting a ESX implementation on a solution based on HP c7000 blade enclosure, which had two HP Virtual Connect interconnect modules builtin. The blade server used was HP Proliant DL460c G6. I checked the VMware HCL and noticed some requirements to get this to work on vSphere 4.0 Update 1.

Output from VMware HCL:

Notice that ESX 4.0 U1 is supported but there are a couple notes. One is to install a specific driver - esx40-net-bnx2x_400.1.48.107-1.0.4. I downloaded and installed the driver on all the ESX servers.

Now I wanted to test the failover. In the HP Virtual Connect Manager I disabled the Shared Uplink Set for Bay 2. I had already setup a continous ping and verified that I still had connectivity to both service console and to VMs running on the hosts, through interconnect bay 1. Test was succesful.

Then I switched arround, enabled Shared Uplink Set 2 and disabled Shared Uplink Set 1 for Bay 1. This time I lost connectivity to both service console and ESX hosts and even I waited a couple minutes, it never came up. I had one other blade server on that enclosure, that was running Microsoft Windows, which I didnt have any problems connecting to.

So I thought the reason was with the ESX configuration.

After I verified all settings on both the ESX hosts ,HP Virtual Connect and the physical switches, which all were identical configured in regards to both interconnect bays, I decided to call HP Support on this issue. I was referenced to a public advisory stating that HP Virtual Connect Flex-10 - ESX 4.0 U1 in an Active/Active Configuration does Not Failover Using SmartLink.

The solution is the following three action points:

1. Verify that the firmware on HP Virtual Connect was running 2.30 as minimum. This setup was running with 2.32 (newest version)
2. Verify that the NIC driver version was Broadcom NetXtreme II Ethernet Network Controller driver 1.52.12.v40.3 (minimum) for ESX/ESXi 4.0. This was different from what the VMware HCL stated.
3. Verify that the NC532i/m bootcode version 5.0.11 (minimum). The bootcode on the NC532 was NOT up-to-date on each blade.

I updated both the NIC driver in ESX and the NIC bootcode with the HP Firmware Maintenance CD and after a reboot, failover was working just as expected. It is recommended by HP to update the bootcode after the NIC driver is installed on the ESX server.

I have NOT been able to find the public advisory article on the HP website on this in regards to VMware vSphere, hence this article.

Eject CD in Lab Manager cause vCenter service to stop.

As i’ve been playing around with VMware Lab Manager recently, I ran into a not so nice problem. If I attached an ISO file to a Lab Manager virtual machine and ejected it again, it caused VMware vCenter service to die every time.

Im running the newest releases: VMware vSphere 4.0 Update 1 and VMware Lab Manager 4.0.1.

As there is no official fix, the below temporary workaround could be implemented. Bare in mind this is an unsupported temporary workaround.

Go to your vCenter server, first stop your vcenter service and make a backup of the vpxd.cfg file located within C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\

Now edit the vpxd.cfg file and add the following:

  <i18nFilter>
    <disable>true</disable>
  </i18nFilter>

Copy/paste the above code in just before the </config> tag.

And start the vCenter service again.

There is a catch-22 by changing this setting. You may observe odd looking tasks in the recent tasks window of VC after this workaround. Hopefully a supported fix is beeing released soon.

Once again, this is not a supported fix and should be changed with care. If you see any problems you should restore the original vpxd.cfg file and restart the vCenter service.

Error while trying to virtualise an ESX server as a virtual machine on a physical ESX

Recently there has been numerious blogposts about how to virtualise an ESX server as a virtual machine on top off a physical ESX server hosts. This is particular useful for mainly testing purposes.

Check some of these blogposts out:
VMware ESX 4 can even virtualize itself by Eric Gray at www.vcritical.com:
http://www.vcritical.com/2009/05/vmware-esx-4-can-even-virtualize-itself/

VMware vSphere in a box by Hany Michael at www.hypervizor.com:
http://www.hypervizor.com/2009/07/vsphere-in-a-box-a-virtual-private-cloud-blueprint/
http://www.hypervizor.com/2009/07/vsphere-in-a-box-part2-putting-the-pieces-all-together/
http://www.hypervizor.com/2009/07/vsphere-in-a-box-part-3-the-lab-manager-40-automation/

The last link above even shows how to utilize VMware Lab Manager to deploy multiple vSphere in a box LABs. That is exactly what I’m trying to achieve in this environment.

So moving forward I ran into problems with the virtualising ESX servers. I could’nt even install ESX or ESXi in the VM. I got the following error ”Kernel panic – not syncing: No supported microcode levels for this stepping of AMD Family….”:

I was quite sure my physical ESX hosts were running a newer processor, that can be used to run a virtual ESX. My setup is like this:
2x HP DL385G5 Servers with AMD Barcelona processors
2x HP DL385G5p Servers with AMD Barcelona processors

I verified the BIOS settings was enabled for AMD-V. I was trying out several of the .vmx tweeks in the past used for VMware Workstation or ESX3.5. Neither of them allowed me to install ESX in the VM.

So I am running different ESX hosts in my cluster and I had enabled EVC on the cluster to allow for VMotion between different CPU models. I tried to disable EVC and bingo, now it worked like a charm.

Installing vSphere 4 Update 1 works without any adjustments in the vmx file. Next step is to see if a VM can be run on top of the virtual ESX server.

Quite a few updates from VMware incl. vSphere 4.0 Update 1 and View 4.0

View 4 available for download among other updates…

The long awaited View 4 is available for download. The earlier version did not support vSphere 4.0 and this new version of View 4 does. It requires VMware vSphere 4.0 Update 1 which also now is available.

So for all that was running View 3.1 in Proof of Concept which required VI 3.5, it would now be possible to upgrade your VMware platform with both vSphere 4.0 Update 1 – which also got released this week.

So there have been quite a few updates from VMware this week:

VMware vSphere 4.0 Update 1 available for both ESX and vCenter
http://www.vmware.com/support/vsphere4/doc/vsp_vc40_u1_rel_notes.html
http://www.vmware.com/support/vsphere4/doc/vsp_esx40_u1_rel_notes.html
http://www.vmware.com/support/vsphere4/doc/vsp_esxi40_u1_rel_notes.html

VMware Data Recovery 1.1 available:
http://www.vmware.com/support/vdr/doc/vdr_110_releasenotes.html

VMware vSphere PowerCLI 4.0 Update 1 available:
http://www.vmware.com/support/developer/windowstoolkit/wintk40u1/windowstoolkit40U1-200911-releasenotes.html

VMware vSphere CLI 4.0 Update 1 available:
http://www.vmware.com/support/developer/vcli/vcli401/vcli_401_relnotes.html

VMware View 4.0 available:
http://www.vmware.com/support/view40/doc/releasenotes_viewmanager40.html

VMware ThinApp 4.0.4 available:
http://www.vmware.com/support/thinapp4/doc/releasenotes_thinapp404.html

I linked to the release notes, because remember that it is important to READ them before using the software.

My Company, BusinessMann A/S won VMware Partner of the Year 2009 award in Denmark

I attended a local party held by Arrow ECS in Copenhagen last night, and several awards were given out during the party.

One award stood out to me, of course, which this blog post is about.

My company was nominated in the category of VMware Partner of the Year in Denmark with 3 other of our competitors.

The happy news was that BusinessMann A/S won the VMware Partner of the Year 2009 award, which makes me really proud.

First of all, thanks to VMware for giving BusinessMann A/S this award and thanks to all my collogues for the hard work in 2009.

Also thanks to Arrow ECS for a great party

VMware announces VMware View 4

VMware has today announced the new VMware View 4 platform for virtualizing desktops on their own VMware vSphere platform. So yes, VMware View 4 has support for VMware vSphere 4.0. It is built and tested on the latest VMware vSphere 4.0 releases

It has been a while since VMware announced the partnership with Teradici to use their PCoIP protocol within VMware View for improved user multimedia experience. This is a huge upgrade from the current RDP protocol, which VMware used in prior versions of View

I have watched a few demo’s of Teradici’s PCoIP protocol and has been pretty impressed with those

An overview of the VMware View Architecture:

VMware View will be GA on 19th of November 2009 and 60 days evaluation will be available for downloads.

Read more or register for an evaluation on VMware View 4here and you will receive an email when VMware View 4 is ready for download

VMs NIC disconnects after a VMotion

I had a customer calling me the other day about a weird problem. Some VMs disconnected their NIC after a VMotion.

It was not possible to connect the NIC back on a again

The customer was in the process of upgrading to vSphere, so used DRS/VMotion to put ESX hosts in maintenance mode, so they could be upgraded.

After looking in log files, we saw the following errors:

————Edited/output ————

6 10:19:21 esx1 vmkernel: 0:20:07:45.197 cpu3:6256)Net: 1317: can’t connect device: LAN: Out of resources
6 10:24:03 esx1 vmkernel: 0:20:12:27.267 cpu3:6256)Net: 1317: can’t connect device: LAN: Out of resources
——————————————-

This led us to check for how many ports the vSwitch was configured to use:
esxcfg-vswitch -l

————Edited/output ————

[root@esx1 ~]# esxcfg-vswitch -l

Switch Name   Num Ports    Used Ports   Configured Ports   MTU    Uplinks
vSwitch0          32                32                32                          1500   vmnic0

PortGroup Name   VLAN ID    Used Ports   Uplinks
LAN                       0               29                vmnic0
Service Console    0               1                  vmnic0
——————————————-

As we can see they have a LAN portgroup connected on vSwitch0 together with the Service Console. The default vSwitch is configured with 32 ports and due to the maintenance mode of one ESX hosts, this customer ran into the problem that the vSwitch didn’t have enough available ports for the VMs that were migrated to that specific host.

It resulted in that the NIC in the VM got disconnected and that we couldn’t reconnect it.

Any additionally created vSwitches is per default created with 64 ports.

So this leads me to the point. First, this is a design issue and should never happen if designed accordingly, however I do think that the DRS/VMotion feature should come up with a warning if a vSwitch is full, instead of migrating (VMotion) the VM and disconnect the VMs NIC interface.

This setup would also provide problems in a VMware HA scenario, if no available ports is free, then it would also leads to disconnected NICs in the VMs restarting.

Make sure that if you use vSwitch0 for VM connectivity to raise the number of ports from default of 32 ports to whatever is needed.

vCalendar Widget added to Virtualtroll.com

Jason Boche over at http://www.boche.net/blog/ came up with an initiative to create a vCalendar.

Its quite nice and if I had a work with a personal desktop, I would certainly try to get one of these. Instead I did the 2nd best option, because Jason also made a vCalendar Widget. I have now added this widget to my blog, so the dailiy tips, tricks, comments etc is shown on virtualtroll.com.

So now you just need to come back here to virtualtroll to read the daily virtualization quote of the day

Read more about it here

Thanks to Jason for this initiative.

Is VMworld Europe coming to Denmark in 2010?

My friend and fellow blogger Eric Sloof over at NTPRO.NL posted a quite interesting and good news today. On the last page of the conference guide, that he received for VMworld in US, reveals that VMworld Europe 2010 will be held in Bella Centre, Copenhagen (Denmark)

That is quite interesting and very surprising for me. I live about 30 minutes from the conference center

Read his post here

Eric thanks for blogging about this.