Tag: ESXi 5

Remove VIB – Device or resource busy

Today, I was playing around with some vSphere Installation Bundles (VIB) and ran into an issue when I tried to remove vib:

esxcli software vib remove -n vmware-esx-KoraPlugin 
 [InstallationError]
 Error in running rm /tardisks/:
 Return code: 1
 Output: rm: can't remove '/tardisks/': Device or resource busy

Even adding the –force attribute did not help in this situation.

The following workaround seemed to be working for me:

  1. Stophostd on theESXi host – this will be non-disruptive to yourVMs 
    /etc/init.d/hostd stop
  2. Runlocalcli to uninstallVIB 
    localcli software vib remove -n vmware-esx-KoraPlugin

    Note: We need to run localcli, since esxcli is not available if hostd is stopped

  3. Starthostd on theESXi host 
    /etc/init.d/hostd start
  4. Verify thatvmware-esx-KoraPlugin no longer shows up 
    esxcli software vib list | grep  vmware-esx-KoraPlugin

You should no longer see the VIB installed on your ESXi host.
Localcli is not widely spread within the community and mainly used by VMware’s Technical Support. It provides more troubleshooting capabilities, even if hostd is not running.

 

 

Unmount VMware Datastore – Device Busy

Welcome back, I hope everyone had some time to relax and spend the christmas holidays with their families. I was lucky enough to have some time off and play with my lab.

After playing around with some newly deployed NFS datastores, I tried to unmount them and I got a device busy error and on the CLI I’ve got Sysinfo error on operation returned status : Busy. Please see the VMkernel log for detailed error information.

Let me show you the steps I ran through:

  1. Mounted a newNFSdatastore throughtheESXCLI 
    1. esxcli storage nfs add  --host=nas-ip --share=share-name --volume-name=volume_name
  2. List all NFS shares 
    1. ~ # esxcli storage nfs list
Screen Shot 2015-01-12 at 3.04.01 PM
  3. Verify that all VMs on this datastore are either powered off or have been vMotioned to another datastore
  4. Trytounmountthedatastore 
    1. esxcli storage nfs remove -v b3c1
Sysinfo error on operation returned status : Busy. Please see the VMkernel log for detailed error information
  5. Looking through the vmkernel.log, doesn’thelpmuch either. The only messageprintedthere is 
    1. 2015-01-12T23:10:09.357Z cpu2:49350 opID=abdf676b)WARNING: NFS: 1921: b3c1 has open files, cannot be unmounted
  6. After some searching, I found this article on VMware
  7. Basically, the issue seems to be that vSphere HA Datastore Heartbeats are  enabled on this datastore, which is causing the device to be busy.

The solution for this problem is pretty simple. Open your vSphere Client and edit the vSphere HA settings, after selecting your vSphere HA Cluster. Within the settings, make sure to set the vSphere HA heartbeats to Use Datastores only from the specified list and deselect your datastore, which you try to unmount.

Screen Shot 2015-01-12 at 2.39.42 PM

After changing the setting, I was able to successfully unmount the NFS share with the following command esxcli storage nfs remove -v datastore_name 

Crucial Data In Your VMware ESXi 5 Log Files

As an Escalation Engineer, part of my daily work is reviewing log files of various systems and vendors. In my first blog post, I would like to show which VMware ESXi 5 log files are most relevant for troubleshooting storage and networking related problems.

All current ESXi 5 logs are located under /var/log and as they rotate, they’ll be available under /scratch/logs

 

vmware_esxi_logs

/var/log/vmkernel.log:

  • VMkernel related activities, such as:
    • Rescan and unmount of storage devices and datastores
    • Discovery of new storage like iSCSI and FCP LUNs
    • Networking (vmnic and vmks connectivity)

/var/log/vmkwarning.log:

  • Extracted warning and alert messages from the vmkernel.log

/var/log/hostd.log::

  • Logs related to the host management service
  • SDK connections
  • vCenter tasks and events
  • Connectivity to vpxa service, which is the vCenter agent on the ESXi server

/var/log/vobd.log:

  • VMkernel observations
  • Useful for network and performance issues

Also, if you have a VM which is affected in particular, it might be worth looking into the vmware.log which is stored with the Virtual Machine. You can find the log under /vmfs/volumes/datastore_name/VM_name/vmware.log.

For the location of ESXi 3.5 and 4.x log files, can be found here.