A guide to implement wildcard certificates on vCloud Director 1.5

I know it has been a while since I posted on this blog, but life has been crasy lately. But here we go. This blog post is mostly for my own reference, but I thought others would be in the same situation as I was. I might as well share how I managed to get this to work.

This post is a guide to implement wildcard certificates on vCloud Director 1.5.

I already had a public signed PFX file with our wildcard certificate, which I wanted to import and use with vCloud Director. I was following the following VMware KB Article on how to generate SSL certificates for VMware vCloud Director.

http://kb.vmware.com/kb/1026309

However, when I tried to import the certificate using the keytool –importcert command, I was given the following error:

keytool error: java.lang.Exception: Input not an X.509 certificate

Since I am not an expert on generating java keystore certificates, I contacted our certificate provider Trustzone for a guide on how to do this. They told me the easiest way was to generate a new certificate request and reissue the wildcard certificate based on that request. And it was free of charge.

In the mean time I searched on Google for other articles explaining this process and found this one, which helped me in the right direction.

http://www.digicert.com/csr-creation-java.htm

While this was not vCloud Director specific, I had to change a bit in the commands as follow:

To simplify this blog post I have excluded the path to the keytool, which is located in /opt/vmware/vcloud-director/jre/bin/keytool 

1. Login to your vCloud Director using SSH
2. Stop vCloud Director using the following command
3. Service vmware-vcd stop
4. Create new Keystore:
5. keytool –genkey –alias http –keyalg RSA –keysize 2048 –keystore /etc/yourdomain.jks
6. You will be prompted for DN information and when it asks for first and last name, type in yourdomain and extension and if using wildcard then it’s *.yourdomain.com
7. Confirm that the information is correct and accept by entering Y for Yes.
8. Generate the CSR request with your new keystore
9. keytool -certreq -alias http -keyalg RSA -file yourdomain.csr -keystore yourdomain.jks
10. Enter the keystore password
11. The SSL Certificate CSR file is now created. Open the CSR with a text editor, and copy and paste the text (including the BEGIN and END tags) into your certificate providers order form (to reissue of the wildcard certificate)
12. In return you now have a wildcard certificate as a yourdomain.cer file
13. In order to get the certificate to work and validated, you need to import the root and intermediate certificates as well as the wildcard certificates. The root and intermediate certificates should be provided to you by your certificate provider, when reissuing the wildcard certificate.
14. Import the Certification Authority’s root certificate using the keytool into the keystore using the following command:
15. keytool –storetype JCEKS –storepass password -keystore /etc/yourdomain.jks –import –alias root –file /root/root.cer
16. Import the Certification Authority’s intermediate certificate using the keytool into the keystore using the following command
17. keytool –storetype JCEKS –storepass password –keystore /etc/yourdomain.jks –import –alias intermediate –file /root/intermediate.cer
18. Now Import the http certificate using the keytool into the keystore using the following command
19. keytool –storetype JCEKS –storepass password –keystore /etc/yourdomain.jks –import –alias http –file /root/yourdomain.cer
20. A second SSL certificate is used for the consoloproxy sessions. Again using the same wildcare certificate,
     import the consoleproxy certificate using the keytool into the keystore using the following command
21. keytool –storetype JCEKS –storepass password –keystore /etc/yourdomain.jks –import –alias consoleproxy –file /root/yourdomain.cer
22. Verify that all the certificates have been imported, list the contents of the keystore file with the command
23. keytool -storetype JCEKS -storepass passwd -keystore certificates.ks –list
24. And you should see something like this:
25. 
26. Now you have to reconfigure vCloud Director to use the new keystore using the following command
27. /opt/vmware/cloud-director/bin/configure
28. Run through the interactive wizard and when asked to type a path for certificate store type:
29. /etc/yourdomain.jks
30. Finish the interactive wizard and start vCloud Director again using this command
31. Service vmware-vcd start
32. Eventually do a tail on the cell.log file to verify that everthing started perfectly
33. tail -f /opt/vmware/cloud-director/log/cell.log
34. The final step is to verify that both the vCloud Director portal is protected using the wildcard certificate and a consoleproxy session against a vCloud Director virtual machine.

I hope this blog post will be useful for someone out there, implementing vCloud Director installations.

VMworld Europe 2011 goes to Copenhagen again

Eric Sloof from ntpro.nl posted today that VMware decided to keep VMworld in Copenhagen for their 2011 event.

Thats great news, as I enjoyed getting VMworld to my home city, well almost. I’m living alittle north of Copenhagen.

Read more at vmworld.com

Bella Sky Comwell (Hotel) will open in 106 days. It will the largest designhotel in Scandianvia and its just next door to the conference. So if you want a new first class hotel, go take a look

See you all here in Copenhagen.

How to enable Promiscuous mode in VMware vSphere LAB Manager

Do you want to run VMware vSphere in a box using VMware LAB Manager?. Then this article is for you.

Hany Michael has earlier last year written 3 articles on how to run a VMware vSphere in a box LAB environment and I won’t go into details on how to install and configure VMware vSphere. Hany did a great job documenting his work:

vSphere in a Box: A “Virtual Private Cloud” Blueprint

vSphere In A Box: Part(2): Putting the pieces all together

vSphere In A Box: (Part 3): The Lab Manager 4.0 Automation

One requirement that is needed when virtualizing ESX hosts, (from now on called vESX) is that the portgroup needs to be configured in promiscuous mode. When deploying configurations within LAB Manager it also creates new portgroups specifically for that configuration.

However these portgroups are configured to reject promiscuous mode per default.

The consequence is running vESX hosts with nested VMs won’t work as supposed to. The trick is to configure LAB Manager to enable promiscuous mode. Looking around in the GUI of LAB Manager, this option is not available. Actually it is hidden!

To enable the hidden features, go to the About screen:

On this screen press “CTRL+U” and following is displayed:

Bingo, now enable promiscuous mode. Redeploy your configurations and this solve the hurdle to do this manually.

Thanks to Hany for his blog post series on vSphere in a box.

Update: Just to notice, that changing this setting is unsupported by VMware and that any portgroups afterwards will be configured with this setting.

Transitioning to ESXi ~ ESXi is NOT the free version of vSphere.

One of the most asked questions when I talk to customers about transitioning to ESXi architecture is:

But ESXi is the free edition?

Well, ITS NOT. Actually ESXi is licensed the same way as the classic edition of ESX. It has the same features in each version. VMware has announced that vSphere 4.1 will be the final release of the classic version of ESX. Moving forward all innovation will go into the ESXi platform.

The following versions of vSphere 4.1 can be installed as either the classic service console version ESX or the thin footprint version of ESXi:

• vSphere Essentials Edition
• vSphere Essentials Plus Edition
• vSphere Standard Edition
• vSphere Advanced Edition
• vSphere Enterprise Edition
• vSphere Enterprise Plus Edition

Actually VMware has renamed the free version of ESXi. Now it’s called “VMware vSphere Hypervisor”

So what is my recommendation?

Start planning the transition to ESXi now, why? Because suddenly the next release of vSphere is here. And there might be important features that you want implemented as soon as possible. Therefore transitioning to ESXi will be more complex, not only do you need to learn about ESXi – but also about a whole new release of new features / improvements. Looking in the mirror, vSphere 4.0 and 4.1 had more than 150 new features/improvements each.

So, now is the time to plan, test and transition to ESXi!

So, what are the main benefits for transitioning to ESXi?

ESXi has a much lower footprint (under 100MB, compared to 2GB for the classic version), which means the released patches is about 1/10 of what is released for the classic service console edition. This eases up the administrative duties for patching and maintaining the environment. It is maybe not the biggest issue to patch ESX hosts, but most likely VMware Tools has to be updated, which requires a reboot of each virtual machine. This is a major problem for many companies, because that requires a service window to reboot the virtual machines. ESXi will not eliminate this reboot in case of VMware Tools needs to be updated, but it will most likely not be required as often. So a smaller code base (footprint) means more secure and reliable and still has the same scalability and performance as the classic version.

ESXi also uses a dual-image approach, which allows you to revert to a prior image if needed.

Considerations about transition to ESXi:

Before looking at what to consider before the transition to ESXi, it is important to understand the differences between ESX and ESXi. The following diagram illustrates the main differences:

Basically the service console is gone and in the service console, it was normal to install different agents for management and monitoring. It is NOT possible or supported to install any agents to the ESXi version.

ESXi has its native agents and allow for OEM vendors such as HP, IBM, and DELL to develop CIM Providers (Common Information Model). These CIM providers allow their management platforms to pull out valuable information that is needed to monitor that physical ESXi server host for hardware failures.

Coming back to the considerations: If you have any agents installed within the service console, you have to decide on whether these agents provide any benefits and how that specific agent is supported with an ESXi implementation. Some agents need to be installed within the vSphere Management Assistant (vMA) and use this to monitor the ESX hosts.

Learn about the vSphere Management Assistant (vMA) and deploy this useful tool to manage all of the ESXi hosts from one console. It does work with the classic version as well, so get used to it now. This tool can be used for many features, but one of the most important for troubleshooting VMware performance issues is ESXTOP (resxtop in vMA).

Learn about the remote command line utilities such as vCLI and/or PowerCLI. Understand how to use them in case they are needed.

A few other good resources:

VMworld.com: Transitioning to ESXi
VMware ESXi and ESX Info Center
Getting started with vSphere Management Assistant vMA by virtuallyghetto
Great esxtop page by Duncan at Yellow-bricks.com (my favorite virtualization blog)

HP released a new SAN/iQ version 9.0 for their HP P4000 series storage, which includes VAAI support

HP has released a new version of SAN/iQ for P4000 series StorageWorks boxes, earlier known as Lefthand Networks.

This release has the following new and improved features, however one specific in regards to vSphere is huge reason to upgrade, namely the support for VMware VAAI vStorage offloads:

• New global configurations — Email, SNMP, DNS, remote syslog, and upgrades are done at the management group
• Automated online upgrade management — Identifies, downloads, and applies updates more easily
  Alarms and Events enhancements — Reduced quantity, configured globally, new severity levels
  Storage system alarms
• Network RAID 5 and 6 enhancements — No longer require snapshots, Network RAID 5 on clusters of three or more, Network RAID 6 on clusters of five or more
• VMware VAAI vStorage offloads — Full copy, Block Zeroing, and Hardware Assisted Locking for faster VM deployment and less load on ESX servers
• New Best Practice Analyzer rules — Checks for consistent RAID and network settings
  P4000 Centralized Management Console enhancements — More map views, new volume icons that represent status and RAID, reduced numbers of pop ups, user ability to change CMC font sizes and default names
• Host server cluster management — Automated assignment of volumes to many servers that make up an application cluster
• Server aware sites — Servers and server clusters can be assigned to sites to improve load balancing
• HP Insight Remote Support replaces LeftHand Service Console
• HP SIM monitoring support (6.2)
• Localization for Japan, China, and Korea
• MPIO improvements — Host network load balancing in DSM for MPIO and native windows MPIO support
• iSCSI session management improvements — Sessions are automatically rebalanced after systems come back online and when systems are added. Sessions are automatically disconnected when volumes are unassigned from servers
• Improved PGR / Microsoft Failover Cluster scalability — Up to 256 iSCSI sessions per volume with Microsoft Failover Clusters utilizing PGR
• Cross-Version Remote Copy — Release 9.0 SAN/iQ software can remote copy to and from version 8.5, 8.1, and 8.0 management groups
• Remote Copy bandwidth — Now managed independently of management group bandwidth
• CLI improvements — New commands to manage schedules, SNMP, and servers
• Application-Managed Snapshots support Microsoft Windows CSV

Get the software here (requires login to HP):

https://h20392.www2.hp.com/portal/swdepot/try.do?productNumber=P4000SW&lang=en&cc=dk&hpappid=PDAPI_SWD

Get the FOM firmware and release notes here:

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=dk&prodTypeId=12169&prodSeriesId=3936136&swItem=co-88853-1&prodNameId=3936158&swEnvOID=54&swLang=13&taskId=135&mode=4&idx=0

Time to plan an upgrade of the P4500 SAN at my company.

VMware Cloud Computing (a pizza restaurant?) aka VMware vCloud Director

At VMworld 2010 in Copenhagen, last week, VMware presented a quite nice and very informative video on Cloud Computing, comparing it with a pizza restaurant. Take a look yourself:

Following the VMware keynote I proceed to do a LAB on VMware vCloud Director, to see how VMware had built the interface for their vision on the cloud. I must say that I was impressed.

There have been some very good articles already in the VMware community around VMware vCloud Director, which is worth taking a look upon, if you are interested in this new technology from VMware.

You can find some of the top resources for VMware vCloud Director here.

In the following weeks, I will extend my knowledge around vCloud Director, by building up a demo setup in my environment. Stay tuned on more info

Reach me on twitter: @heinoskov

Welcome to Copenhagen - VMworld 2010

Welcome all VMware fellows to Copenhagen.

I live just a little north of Copenhagen and have been excited about VMworld being held in Copenhagen at the Bella Center. It will be the largest VMworld event outside US, as far as I’ve heard.

So today, I drove out to Bella Center in Copenhagen, to pick up my VMworld Badge and get some first impressions. Tomorrow VMworld 2010 is kicking off with the partner day. As usual at the registration desk, I received a nice VMware backpack. And because I have been at VMworld the last 3 years in a row (5 times in whole), I also received a voucher for an alumni elite gift, which I have to pick up at the VMware Booth during the conference. Might wonder what that gift would be? Furthermore I will be getting priority seating at the general session on Tuesday. Thanks VMware ?

And the latest VMworld news is:
That VMware has decided to open VMworld LABs for everyone tomorrow between 1PM to 6PM. Read more at Yellow-bricks.com by Duncan Epping.

Finally, dont forget the VMUG Party, monday night: VMUG Party at Custom House

Reach me on twitter: @heinoskov

VMware vCenter and SNMP monitoring support with Sysorb

I recently had a case at a customer site, where they wanted to use Sysorb to monitor their VMware vSphere environment. I wrote this article, as I couldn’t find any useful information on Sysorb and VMware vSphere together.

SNMP was configured on the ESX servers and VMware vSphere MIBS was imported into the Sysorb tool.

However trying the same for the vCenter server, didn’t give the expected results. I configured some vCenter configuration alarms to test alarms to verify that SNMP trap where sent out. The Alarms and Event tab in vCenter stated that the SNMP trap was sent, however none were received in Sysorb.
I then installed trap receiver from trapreceiver.com and verified through this tool that the alarms where actually sent out and received from the vCenter server.

I wasn’t familiar with how Sysorb was working and had to read up and investigate how it used SNMP to monitor their server and network devices. I found that Sysorb is NOT supporting SNMP traps, but only works through the SNMP GET function. Basically what this means is, that the Sysorb tool was polling the target for status on preconfigured thresholds/triggers in the Sysorb tool.

Since vCenter itself monitors for configured alarms/thresholds/triggers and therefore sending out an SNMP trap if configured, the 2 systems would not work properly together.

I looked in the VMware vSphere documentation for support for SNMP GET and it was stated that support for SNMP GET on vCenter was not available. VMware vCenter only supports SNMP traps for sending out alarms, which also makes perfect sense to me.’

If any has built any workarrounds to get this to work, please let me know.

Fix: HP Virtual Connect Flex-10 - ESX 4.0 U1 in an Active/Active Configuration does Not Failover Using SmartLink

I’ve been troubleshooting a ESX implementation on a solution based on HP c7000 blade enclosure, which had two HP Virtual Connect interconnect modules builtin. The blade server used was HP Proliant DL460c G6. I checked the VMware HCL and noticed some requirements to get this to work on vSphere 4.0 Update 1.

Output from VMware HCL:

Notice that ESX 4.0 U1 is supported but there are a couple notes. One is to install a specific driver - esx40-net-bnx2x_400.1.48.107-1.0.4. I downloaded and installed the driver on all the ESX servers.

Now I wanted to test the failover. In the HP Virtual Connect Manager I disabled the Shared Uplink Set for Bay 2. I had already setup a continous ping and verified that I still had connectivity to both service console and to VMs running on the hosts, through interconnect bay 1. Test was succesful.

Then I switched arround, enabled Shared Uplink Set 2 and disabled Shared Uplink Set 1 for Bay 1. This time I lost connectivity to both service console and ESX hosts and even I waited a couple minutes, it never came up. I had one other blade server on that enclosure, that was running Microsoft Windows, which I didnt have any problems connecting to.

So I thought the reason was with the ESX configuration.

After I verified all settings on both the ESX hosts ,HP Virtual Connect and the physical switches, which all were identical configured in regards to both interconnect bays, I decided to call HP Support on this issue. I was referenced to a public advisory stating that HP Virtual Connect Flex-10 - ESX 4.0 U1 in an Active/Active Configuration does Not Failover Using SmartLink.

The solution is the following three action points:

1. Verify that the firmware on HP Virtual Connect was running 2.30 as minimum. This setup was running with 2.32 (newest version)
2. Verify that the NIC driver version was Broadcom NetXtreme II Ethernet Network Controller driver 1.52.12.v40.3 (minimum) for ESX/ESXi 4.0. This was different from what the VMware HCL stated.
3. Verify that the NC532i/m bootcode version 5.0.11 (minimum). The bootcode on the NC532 was NOT up-to-date on each blade.

I updated both the NIC driver in ESX and the NIC bootcode with the HP Firmware Maintenance CD and after a reboot, failover was working just as expected. It is recommended by HP to update the bootcode after the NIC driver is installed on the ESX server.

I have NOT been able to find the public advisory article on the HP website on this in regards to VMware vSphere, hence this article.

Eject CD in Lab Manager cause vCenter service to stop.

As i’ve been playing around with VMware Lab Manager recently, I ran into a not so nice problem. If I attached an ISO file to a Lab Manager virtual machine and ejected it again, it caused VMware vCenter service to die every time.

Im running the newest releases: VMware vSphere 4.0 Update 1 and VMware Lab Manager 4.0.1.

As there is no official fix, the below temporary workaround could be implemented. Bare in mind this is an unsupported temporary workaround.

Go to your vCenter server, first stop your vcenter service and make a backup of the vpxd.cfg file located within C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\

Now edit the vpxd.cfg file and add the following:

  <i18nFilter>
    <disable>true</disable>
  </i18nFilter>

Copy/paste the above code in just before the </config> tag.

And start the vCenter service again.

There is a catch-22 by changing this setting. You may observe odd looking tasks in the recent tasks window of VC after this workaround. Hopefully a supported fix is beeing released soon.

Once again, this is not a supported fix and should be changed with care. If you see any problems you should restore the original vpxd.cfg file and restart the vCenter service.