VMs NIC disconnects after a VMotion

I had a customer calling me the other day about a weird problem. Some VMs disconnected their NIC after a VMotion.

It was not possible to connect the NIC back on a again

The customer was in the process of upgrading to vSphere, so used DRS/VMotion to put ESX hosts in maintenance mode, so they could be upgraded.

After looking in log files, we saw the following errors:

————Edited/output ————

6 10:19:21 esx1 vmkernel: 0:20:07:45.197 cpu3:6256)Net: 1317: can’t connect device: LAN: Out of resources
6 10:24:03 esx1 vmkernel: 0:20:12:27.267 cpu3:6256)Net: 1317: can’t connect device: LAN: Out of resources
——————————————-

This led us to check for how many ports the vSwitch was configured to use:
esxcfg-vswitch -l

————Edited/output ————

[root@esx1 ~]# esxcfg-vswitch -l

Switch Name   Num Ports    Used Ports   Configured Ports   MTU    Uplinks
vSwitch0          32                32                32                          1500   vmnic0

PortGroup Name   VLAN ID    Used Ports   Uplinks
LAN                       0               29                vmnic0
Service Console    0               1                  vmnic0
——————————————-

As we can see they have a LAN portgroup connected on vSwitch0 together with the Service Console. The default vSwitch is configured with 32 ports and due to the maintenance mode of one ESX hosts, this customer ran into the problem that the vSwitch didn’t have enough available ports for the VMs that were migrated to that specific host.

It resulted in that the NIC in the VM got disconnected and that we couldn’t reconnect it.

Any additionally created vSwitches is per default created with 64 ports.

So this leads me to the point. First, this is a design issue and should never happen if designed accordingly, however I do think that the DRS/VMotion feature should come up with a warning if a vSwitch is full, instead of migrating (VMotion) the VM and disconnect the VMs NIC interface.

This setup would also provide problems in a VMware HA scenario, if no available ports is free, then it would also leads to disconnected NICs in the VMs restarting.

Make sure that if you use vSwitch0 for VM connectivity to raise the number of ports from default of 32 ports to whatever is needed.