Updated: VMware ESX/VC 3.5 Update 2 - Bug or Feature within EVC / CPU Masking

I wrote last week on this bug or new feature observation after Update 2. To read my first post about this click here.

I have been in touch with VMware to find out if this is a bug or a new feature. They confirmed there has been a change to VMotion CPU constraints.

Based on my experience, VMware is a little more flexible on CPU constraints than earlier. To sum this up:

If you have two almost identical CPUs, then if you power up VMs on the “oldest” CPUs these VMs would be able to migrate to the new CPU and back. However if you power up your VM on the newer CPU (enabling the new features) then this VM would not be able to VMotion to the older hosts.

However - one thing to keep in mind though.. even though you can avoid the manual work on configuring CPU masking on every VM… DRS and/or VMware HA would over time power on VMs on all hosts - and those powered up - on the new hosts would then have problems doing VMotion.

The solution to this is to use the new feature on EVC (Enhanced VMotion Compatibility). This knowledge base article lists the CPU which is supported for EVC usage: Enhanced VMotion Compatibility (EVC) processor support

VMware ESX/VC 3.5 Update 2 - Bug or Feature within EVC / CPU Masking?

A customer of mine had issues with CPU masking after setting up a new ESX host. They specific ordered the hardware to be alike with the other 3 hosts in that cluster. So it was the same model both Server and CPU wise.

VMotion worked just fine between all hosts as long as the VM was powered on, on one of the three oldest hosts.

But when they powered on a VM on the new host - this VM could not be migrated with VMotion to the other 3 hosts. It came up with the usual Host CPU is incompatible error.

I analyzed the /proc/cpuinfo file in the service console to determine differences between the hosts. The new host was CPU stepping 11 compared to the older which was 6. And the new host had the NX flag enabled. The results on this was:

/proc/cpuinfo file	Older hosts	New hosts
Processor	0	0
Vendor_id	GenuineIntel	GenuineIntel
CPU Family	6	6
Model	15	15
Model Name	Intel(R) Xeon(R) CPU 5150  @ 2.66GHz	Intel(R) Xeon(R) CPU 5150  @ 2.66GHz
Stepping	6	11
CPU MHZ	2660.059	2660.063
Cache Size	4096 KB	4096 KB
Fdiv_bug	No	No
Hlt_bug	No	No
F00f_bug	No	No
Coma_bug	No	No
Fpu	Yes	Yes
Fpu_exeption	Yes	Yes
Cupid level	10	10
Wp	Yes	Yes
Flags	fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss tm lm	fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss tm nx lm
Bogomips	5308.41	5308.41

Then I checked to see if the new Enhanced VMotion Compatibility (EVC) was enabled. But this option was grayed out (disabled) – because the three oldest hosts were not compatible with EVC.

Before Update 2 – and when you had VMotion CPU compatibility issues, VMotions didn’t work either way. So something happened with this in Update 2. In the past you manually had to enter the correct CPU Bits, into each VMs configuration file for VMotion to work.

But in this case I could VMotion all VMs between all hosts forth and back - as long as the VM was started on one of the three older hosts.

I noticed that on the VMs powered on, on the oldest hosts - that CPU masking bits were entered into the vmx file. This was done automatically by the system!

This was added to the vmx file of VMs powered on, on the older hosts.

cpuid.1.eax = “xxxx————xx————–” cpuid.1.ecx = “R—-R–R-RRRR-0———–H-R–” cpuid.1.edx = “—————————T—-” cpuid.80000001.eax.amd = “xxxx————xx————–” cpuid.80000001.ecx.amd = “——————RR-RR-RRR-0—” cpuid.80000001.edx = “—-R—————H———–” cpuid.80000001.edx.amd = “—–R————–H——T—-”   cpuid.1.ecx.amd = “R——-R——————-R—”   hostCPUID.0 = “0000000a756e65476c65746e49656e69″ guestCPUID.0 = “0000000a756e65476c65746e49656e69″ userCPUID.0 = “0000000a756e65476c65746e49656e69″ hostCPUID.1 = “000006f6000208000004e3bdbfebfbff” guestCPUID.1 = “000006f800010800000022110febbbff” userCPUID.1 = “000006f6000208000004e3bdbfebfbff” hostCPUID.80000001 = “00000000000000000000000120000000″ guestCPUID.80000001 = “00000000000000000000000120000000″ userCPUID.80000001 = “00000000000000000000000120000000″ evcCompatibilityMode = “FALSE”

 

 

So - I’m just wondering if this is a new feature in Update 2 VI 3.5. It is not described anywhere (at least I have not found any documentation on it).

After some thoughts I summed up - that indeed this is a nice feature to have (If you dont have the possibility to turn on EVC)… Just make sure to power on your VMs on the old servers and it automatically entered the CPU bits needed to do VMotions against the new host. And still a little weird since EVC was disabled. But it removed all the work on manually entering the CPU bits on every VM.

My concern though is that manually editing the CPU bits hasn’t been supported by VMware… so is this supported?

So is this by design or is it a bug (hence that EVC was disabled)?

Update 3 for VirtualCenter 2.5 released.

This release doesnt contain any new features. It’s only hotfixes in this release, but some very important ones especially in regard of the VMware HA feature (high availability). Also the license server gets an upgrade.

Also this update contains a whole lot of fixes for VMware Update Manager and VMware Converter as well. So make so to upgrade your VI plugins. During the upgrade of VMware Update Manager it will also contain a database upgrade.

Make sure to read the release notes before upgrading: Release notes

To download the VirtualCenter update 3, click here

Remember to do a backup of the SQL database before upgrading. If VirtualCenter run as a virtual environment take a snapshot of it and if possible use VCB (VMware Consolidated Backup) to do an image-level backup as well. That way you can get back to the old one.

Also VirtualCenter needs to be upgraded before VMware Update Manager and Converter.

VMware Check license web feature

 Does your VMware server-based license file cause troubles for you? - Then you can use this cool web feature to validate your license files:

http://www.vmware.com/checklicense/

Just copy and insert the content of your license file and press validate. This will check and correct any errors found and supply you with new content for your license file.

Enabling Windows Single Sign-on support in VirtualCenter 2.5 Update 2

To enable Windows Single Sign-on follow these steps:

1. Log in to a workstation where the VI Client is installed.
2. Right-click the desktop and select New > Shortcut.
3. In the Create Shortcut Wizard, click Browse and navigate to the location of the VpxClient.exe program and click OK. Note: By default it is located in C:\Program Files\VMware\Infrastructure\Virtual Infrastructure Client\Launcher.
4. After the full path is in the Type the location of the item field append -passthroughAuth -s to the end of the line, where is the hostname or IP Address of the VirtualCenter instance you want to connect to.
5. Click Next.
6. Give a Name for the shortcut.
7. Click Finish.
8. After the shortcut has been created, double-click on the new shortcut, and you are logged into the VirtualCenter Server using the currently logged in credentials.

Note: If the currently logged in user does not have appropriate permissions to VirtualCenter log in fails.

ESX 3.5 Update 2 error, incompatible HA networks

HA clusters before ESX 3.5 and ESXi 3.5 Update 2 that configured, despite incompatible networks, now generate faults on the configure tasks.

The error message in the event detail describes the incompatibility (for example, “cluster has network X.X.X.X that does not exist on the host”), along with the recommendation to “Consider using Advanced Cluster Settings das.allowNetwork to control network usage.”

So an attempt to reconfigure clusters might fail now if they do not have identical networks.

Read the resolution in this VMware KB article

Another article regarding VMware HA after Update 2

So what new commands is available in VMware RCLI?

VMware has updated the VMware Remote CLI with a new version on the same time they released Update 2 for VI3.5.

So what new commands has been added? Here’s a list:

• esxcfg-dns - Specifies an ESX Server host’s DNS configuration.
• esxcfg-module - Enables VMkernel options
• esxcfg-user - Creates, modifies, deletes, and lists local direct access users and groups of users
• vicfg-dns - Specifies an ESX Server host’s DNS configuration.
• vicfg-module - Enables VMkernel options
• vicfg-user - Creates, modifies, deletes, and lists local direct access users and groups of users
• vmware-cmd - Manages the virtual machines in your VMware Infrastructure environment

The vi- commands is for use with 3i and the esx- command is for use with 3.5 (and the service console)

For a complete new guide for VMware Remote CLI - check this out.

Also the new RCLI supports the use of SSPI for authentication. This feature allow us to use passthrough authentication when executing scripts.  For this too work - VirtualCenter needs to be updated with the Update 2 package.

VMware releases Update 2 for VI3

The following information provides highlights of some of the enhancements available in this release of VMware Infrastructure 3:

Note: Not all combinations of VirtualCenter and ESX Server versions are supported and not all of these highlighted features are available unless you are using VirtualCenter 2.5 Update 2 with ESX Server 3.5 Update 2. See the document ESX Server, VirtualCenter, and VMware Infrastructure Client Compatibility Matrixes for more information on compatibility.

Features

• Windows Server 2008 support – Windows Server 2008 (Standard, Enterprise, and Datacenter editions) is supported as a guest operating system. With VMware’s memory overcommit technology and the reliability of ESX, virtual machine density can be maximized with this new guest operating system to achieve the highest degree of ROI. Guest operating system customizations and Microsoft Cluster Server (MSCS) are not supported with Windows Server 2008.
• Enhanced VMotion Compatibility – Enhanced VMotion compatibility (EVC) simplifies VMotion compatibility issues across CPU generations by automatically configuring server CPUs with Intel FlexMigration or AMD-V Extended Migration technologies to be compatible with older servers. Once EVC is enabled for a cluster in the VirtualCenter inventory, all hosts in that cluster are configured to ensure CPU compatibility for VMotion. VirtualCenter will not permit the addition of hosts which cannot be automatically configured to be compatible with those already in the EVC cluster.
• Storage VMotion – Storage VMotion from a FC/iSCSI datastore to another FC/iSCSI datastore is supported. This support is extended on ESX/ESXi 3.5 Update 1 as well.
• VSS quiescing support – When creating quiesced snapshot of Windows Server 2003 guests, both filesystem and application quiescing are supported. With Windows Server 2008 guests, only filesystem quiescing is supported. For more information, see the Virtual Machine Backup Guide and the VMware Consolidated Backup 1.5 Release Notes.
• Hot Virtual Extend Support – The ability to extend a virtual disk while virtual machines are running is provided. Hot extend is supported for vmfs flat virtual disks without snapshots opened in persistent mode.
• 192 vCPUs per host – VMware now supports increasing the maximum number of vCPUs per host 192 given that the maximum number of Virtual Machines per host is 170 and that no more than 3 virtual floppy devices or virtual CDROM devices are configured on the host at any given time. This support is extended on ESX 3.5 Update 1 as well.

Hardware Enablement and Management:

• 8Gb Fiber Channel HBAs – Support is available for 8Gb fiber channel HBAs. See the I/O Compatibility Guide for ESX Server 3.5 and ESX Server 3i for details.
  SAS arrays – more configurations are supported. See the Storage/SAN Compatibility Guide for ESX Server 3.5 and ESX Server 3i for details.
• 10 GbE iSCSI initiator – iSCSI over a 10GbE interface is supported. This support is extended on ESX Server 3.5 Update 1, ESX Server version 3.5 Update 1 Embedded and ESX Server version 3.5 Update 1 Installable as well.
• 10 GbE NFS support – NFS over a 10GbE interface is supported.
• IBM System x3950 M2 – x3950 M2 in a 4-chassis configuration is supported, complete with hardware management capabilities through multi-node Intelligent Platform Management Interface (IPMI) driver and provider. Systems with up to 32 cores are fully supported. Systems with more than 32 cores are supported experimentally.
• IPMI OEM extension support – Execution of IPMI OEM extension commands is supported.
• System health monitoring through CIM providers - More Common Information Model (CIM) providers are added for enhanced hardware monitoring, including storage management providers provided by QLogic and Emulex. LSI MegaRAID providers are also included and are supported experimentally.
• CIM SMASH/Server Management API – The VMware CIM SMASH/Server Management API provides an interface for developers building CIM-compliant applications to monitor and manage the health of systems. CIM SMASH is now a fully supported interface on ESX Server 3.5 and VMware ESX Server 3i.
• Display of system health information – More system health information is displayed in VI Client for both ESX Server 3.5 and VMware ESX Server 3i.
• Remote CLI – Remote Command Line Interface (CLI) is now supported on ESX Server 3.5 as well as ESX Server 3i. See the Remote Command-Line Interface Installation and Reference Guide for more information.

Management Agents Support

• HP Management Agents – HP Insight Management Agents provide server management capabilities for ESX Server hosts installed on supported server platforms. A new version, 8.1, is now supported.

Guest Operating System Support

• Solaris10 U5 - Both the 32-bit and 64-bit versions are supported.
• SUSE Linux Enterprise Server 10 SP2 - Both the 32-bit and 64-bit versions are supported. The 32-bit version supports the VMware Virtual Machine Interface (VMI) and is hence performance-optimized for VMware environments.
• Ubuntu 8.04 - Both the 32-bit and 64-bit versions are supported. The 32-bit version supports the VMware Virtual Machine Interface (VMI) and is hence performance-optimized for VMware environments.

Service Console , driver and library updates

• Service Console Operating System Update - The ESX Server Console operating system has been updated to RHEL 3.0 U9. See the document Updated RPMs and Security Fixes for detailed information.
• Driver updates - The tg3 driver has been updated to version 3.81c. The megaraid_sas driver has been updated to version 00.00.03.19.
• Library update - StoreLib has been updated to version 2.51.

VMware High Availability (HA)

VirtualCenter 2.5 update 2 adds full support for monitoring individual virtual machine failures based on VMware tools heartbeats. This release also extends support for clusters containing mixed combinations of ESX and ESXi hosts, and minimizes previous configuration dependencies on DNS.

VirtualCenter Alarms

VirtualCenter 2.5 Update 2 extends support for alarms on the overall health of the server by considering the health of each of the individual system components such as memory and power supplies. Alarms can now be configured to trigger when host health degrades.

Guided Consolidation Enhancements

Guided Consolidation now provides administrators with the ability to filter the list of discovered systems by computer name, IP address, domain name or analyzing status. Administrators can also choose to explicitly add physical hosts for analysis, without waiting for systems to be auto-discovered by the Consolidation wizard. Systems can be manually added for analysis by specifying either a hostname or IP address. Multiple hostnames or IP addresses, separated by comma or semi-colon delimiters, may also be specified for analysis. Systems can also be manually added for analysis by specifying an IP address range or by importing a file containing a list of hostnames or IP addresses that need to be analyzed for consolidation. Guided Consolidation also allows administrators to override the provided recommendations and manually invoke the conversion wizard.

Live Cloning of Virtual Machines

VirtualCenter 2.5 Update 2 provides the ability of creating a clone of a powered-on virtual machine without any downtime to the running virtual machine. Therefore, administrators are no longer required to power off a virtual machine in order to create a clone of it.

Windows Single Sign-on Support

You can now automatically authenticate to VirtualCenter using your current Windows domain login credentials on the local workstation, as long as the credentials are valid on the VirtualCenter server. This capability also supports logging in to Windows using Certificates and Smartcards. It can be used with the VI Client or the VI Remote CLI to ensure that scripts written using the VI Toolkits can take advantage of the Windows credentials of your current session to automatically connect to VirtualCenter.

Plug-in Updates

This release of the VMware Infrastructure 3 software suite also includes:

• An update to VMware Converter Enterprise.For more information, see the VMware Converter Enterprise Update 2 for VirtualCenter 2.5 Release Notes.
• An update to VMware Update Manager. For more information, see the VMware Update Manager 1.0 Update 2 for VirtualCenter 2.5 Release Notes.

Download ESX Server 3.5 Update 2 here here

Download ESX Server 3i Installable Update 2 here here

Download VirtualCenter Update 2 here

Remember to also update VMware Consolidated Backup to version 1.5 which can be downloaded here here

Virtual machine CPU usage spikes and remains abnormally high after VMotion in a VMware DRS enabled cluster

A critical error in VirtualCenter 2.5 resulted in high CPU spikes on virtual machines after VMotion in a DRS enabled Cluster.

This was supposed to be fixed in Update 1 to VC 2.5. Unfortunately in some installations this error still occur and has been reported to VMware support. VMware has released this kb article on the issue with a workaround until a permanent fix is released.

ESX Server 3i Hosts Without Swap Enabled Cannot be Added to an HA Cluster

ESX Server 3i hosts that do not have swap enabled cannot be added to a VMware HA cluster. VirtualCenter will return an error similar to this

“An error occurred during configuration of the HA agent of the host”

This issue is seen in diskless ESX 3i installations.

To adress this issue read the following article