vSphere : PowerCLI Multipath Config

vSphere : PowerCLI Multipath Config

Round-robin configuration is per LUN, per host with vCenter 4.X and ESXi 4.x. When multiple hosts and datastores are in use this is a cumbersome and long-winded configuration that can easily be simplified with PowerCLI.

Single Host Configuration

Open PowerCLI and connect to a/the VCENTER server in your environment, change vcenter_name to match the hostname of your server:

connect-VIServer vcenter_name

To view hosts in a cluster, chage the cluster_name to match that in the vCenter GUI:

get-cluster cluster_name | get-vmhost

Confirm disks attached to each host:

get-cluster cluster_name | Get-VMHost | Get-ScsiLun -LunType disk

In the case of the environment I was working in I only wanted to change the configuration for EVA8400 disks attached to the VM hosts. There were also non-active/active storage devices attached which could not use round-robin.

To view the CanonicalName for all LUNs I used the command:

get-cluster cluster_name | Get-VMHost | Get-ScsiLun -LunType disk | fl -show canonicalname,vendor,capacitymb,model,luntype

From this you can analyse the CanonicalName and find the volumes that share the fist few digits, I ended up with all EVA8400 LUNs sharing ‘naa.6001438005’. An example output is below, you’ll note that two of the three LUNs share CanonicalName string (in red), the other does not (in green):

CanonicalName : naa.600508b1001c3a986c7a228dcad5099b
Vendor : HP
CapacityMB : 57207
LunType : disk
MultipathPolicy : MostRecentlyUsed

CanonicalName : naa.6001438005dee9f70000600003cf0000
Vendor : HP
CapacityMB : 153600
Model : HSV450
LunType : disk
MultipathPolicy : RoundRobin

CanonicalName : naa.6001438005deea3c0000900002a00000
Vendor : HP
CapacityMB : 102400
Model : HSV450
LunType : disk
MultipathPolicy : RoundRobin

Once you have identified which disks you want to change, confirm the string is working – adding a ‘*’ to the end of it, for example in order for me to view the EVA 8400 disks attached to a host:

$vmhost = get-VMhost hostname

Get-VMHost $vmhost | Get-ScsiLun -CanonicalName “naa.6001438005*”

CanonicalN ConsoleDeviceName LunType CapacityMB MultipathPolicy
———- —————– ——- ———- —————
naa.600… /vmfs/devices/disks/naa.600… disk 102400 RoundRobin
naa.600… /vmfs/devices/disks/naa.600… disk 153600 RoundRobin
naa.600… /vmfs/devices/disks/naa.600… disk 153600 RoundRobin
naa.600… /vmfs/devices/disks/naa.600… disk 230400 RoundRobin

I had four LUNs presented from the EVA8400, so I knew this was working, I had also configured these to be RoundRobin already. This enabled quick identification that these were the correct volumes. You can use the following script to map CanonicalName to DatastoreName as per the vCenter console as can be found here:

$h = Get-VMhost hostname

# collect VMFS datastore name and extent cannonical names into a hashtable
$dsLunList = @{}
$dsView = $h | Get-Datastore | ? {$_.Type -eq “VMFS”} | Get-View
foreach ($ds in $dsView) {
foreach ($diskExtent in $ds.Info.Vmfs.Extent) {
$dsLunList[$diskExtent.DiskName] = $ds.Info.Name

#populate datastore name for each lun if available
$lunList = @()
$h | Get-ScsiLun | % {
$lun = “” | select ConsoleDeviceName, CanonicalName, DatastoreName
$lun.ConsoleDeviceName = $_.ConsoleDeviceName
$lun.CanonicalName = $_.CanonicalName

if ($dsLunList.ContainsKey($_.CanonicalName)) {
$lun.DatastoreName = $dsLunList[$_.CanonicalName]

$lunList += $lun

$lunList | ft -show CanonicalName,DatastoreName

Confirm EVA LUN’s that are not configured for RoundRobin multipathing:

$vmhost = get-VMhost hostname

Get-VMHost $vmhost | Get-ScsiLun -CanonicalName “naa.6001438005*” | Where {$_.MultipathPolicy -ne “RoundRobin”}

To modify the LUN multipathing configuration you can use the following command:

$vmhost = get-VMhost hostname

Get-VMHost $vmhost | Get-ScsiLun -CanonicalName “naa.6001438005*” | Where {$_.MultipathPolicy -ne “RoundRobin”} | Set-ScsiLun -MultipathPolicy “roundrobin”

Multiple Host Configuration

To perform this for each node in a VM cluster use the following commands:

foreach ($vmhost in get-cluster cluster_name | get-vmhost) {
Get-VMHost $vmhost | Get-ScsiLun -CanonicalName “naa.6001438005*” | Where {$_.MultipathPolicy -ne “RoundRobin”} | Set-ScsiLun -MultipathPolicy “roundrobin”

Additional Information

naa.6001* is a HP storage device

naa.6006* is an EMC storage device


vSphere : vMA Setup

vSphere : vMA Setup

Import OVF template available form the VM Market Place:

Once completed on first boot a wizard will run, you and set an IP address and vi-admin user account password.

Press Alt-F2 from the VM console then login under vi-admin and execute these commands:

sudo vi /etc/sysconfig/keyboard

Change us to uk and reboot (if applicable!).

Login again via SSH or the console.

sudo vi /etc/ntp.conf

Hash out the serv 0/1/2 lines and enter the lines for your internal NTP servers (domain controllers will work):

Start the NTP service:

sudo service ntpd start

Configure NTP to start at boot:

sudo chkconfig ntpd on

Join the vMA appliance to the domain, modify the domain name, NETBIOSNAME and username:

sudo domainjoin-cli join domain.local NETBIOSNAME\\adminname

Enable connections to each ESXi host from the vMA:

vifp addserver vm01.domain.local
vifp addserver vm02.
vifp addserver vm03.domain.local
vifp addserver vm04.domain.local

Confirm server connections:

vifp listservers

Enable remote logging to the vMA applicance, modify theVM host names as required:

vilogger enable –server vm01.domain.local –numrotation 20 –maxfilesize 10 –collectionperiod 10
vilogger enable –server vm02.
domain.local –numrotation 20 –maxfilesize 10 –collectionperiod 10
vilogger enable –server vm03.
domain.local –numrotation 20 –maxfilesize 10 –collectionperiod 10
vilogger enable –server vm04.
domain.local –numrotation 20 –maxfilesize 10 –collectionperiod 10

Logs are now collected and stored under /var/log/vmware/

Confirm logging setup:

vilogger list

You’ll probably need to resize the /var/log partition (via gParted) or add an additional drive, and modifiy the log location configurationin /etc/vmware/vMA/vMA.conf


VM : High Number of VHD’s – Boot Order Issues

VM : High Number of VHD’s – Boot Order Issues

Encountered an issue with a VM that had around 15 VHDs and multiple SCSI buses. After testing this a few times I discovered that on adding the 8th drive the boot order changed to: 1:0, 1:1, 1:2… 1:7 – this was automatic, I never changed the boot order.

To make matters worse, you can only see 8 VHDs in the VM BIOS, so it is impossible to change the primary boot disk to 0:0 again!

Workaround – I figured, lets move the boot volume to SCSI channel 1 and the data disks to an other controller. I move the data disks to channel 2/3 and the system booted. So, my system disk was on 1:0, additional disks were on channels 2:X and 3:X.

Update 02/02/2012: You can also set the boot drive in the vmx file. Power off the VM and add the line:
bios.hddOrder=scsi0:0 to force the VM to boot from disk 0:0.


HP BL465c G7 : ESXi 4.1 Issues

HP BL465c G7 : ESXi 4.1 Issues

Having just built a new ESXi 4.1 cluster comprised of BL465c G7’s we’ve run into some interesting issues I thought I would share.

Six BL465c G7 servers have been installed in 2 new C7000 G2 enclosures each with 2x Flex-10 Virtual Connect Modules and 2×20-port 8GB Fiber Channel Virtual Connect Modules. Servers are split 3 per-chassis;

In breif the issues are as follows:

  1. PSOD on reboot and random crashes with PSOD
  2. Corrected Memory Error threshold exceeded
  3. Hang on reboot
  4. ILO Duplicate SSL Certificate Serial Number



vCenter VMware VirtualCenter Server WIll Not Stay Started

vCenter VMware VirtualCenter Server WIll Not Stay Started

Installation of vCenter Server succeeds, but services will not stay started. The most recent vpxd log shows the following error:

[2010-10-11 16:51:49.252 03800 error ‘App’] [VpxVmomi] Failed to create WebService socket: class Vmacore::SystemException(An attempt was made to access a socket in a way forbidden by its access permissions. )

[2010-10-11 16:51:49.252 03800 error ‘App’] [VpxdVmomi] Failed to start VMOMI services: An attempt was made to access a socket in a way forbidden by its access permissions.

[2010-10-11 16:51:49.252 03800 info ‘App’] Shutting down VMware VirtualCenter…

In this case McAfee EPO FrameWork Service was using the same port as the vCenter service. This can be verified by stopping the FrameworkService.exe process.

To resolve this issue change the vCenter HTTP/HTTPS Service ports to a port other than 8443/8080.


vSphere: Thin/TBZ disks cannot be opened in multiwriter mode

vSphere: Thin/TBZ disks cannot be opened in multiwriter mode

When using a ‘Virtual’ SCSI Bus Sharing controller on a VM Windows 2008 Cluster I was presented with the following error when creating disks via the vSphere Client:

Thin/TBZ disks cannot be opened in multiwriter mode.

Neither VM would power on with the new disk attached.

To resolve this, create the new vDisk from the shell using the command: vmkfstools -d eagerzeroedthick -c 20G -a lsilogic DISKNAME.vmdk

The important part of this command is eagerzeroedthick without this, or using thin or zeroedthick will result in the the same error being presented at power-on of the VM.


VMWare : Device or Resource Busy

VMWare :  Device or Resource Busy

After removing a VM from an ESX 3.5 update 3 server via the Infrastructure Client, selecting ‘Delete from Disk’ I found that a disused vmdk file was left over on the VMFS datastore.  When deleting the vmdk file I was presented wih the following error:
   cannot remove `TEMP_Data-G_50GB-flat.vmdk’: Device or resource busy

Using the command ‘fuser TEMP_Data-G_50GB-flat.vmdk‘ to identify users/processes which have locked the file showed no active users on the file.

To resolve, run the following command, and wait approx 5 minutes:
   service mgmt-vmware restart

The command will not impact any running VM‘s, it will disconnect any management client sessions (not console access via SSH) and will stop any running VCB’s.

 Now browse the datastore and delete the file.


VMWare : Increasing the HBA / Device Queue Depth

VMWare : Increasing the HBA / Device Queue Depth

ESX 3.5 ships with a standard HBA / LUN queue  depth of 32. For QLogicHBA’s a setting of 64 may improve storage performance. You can identify a storage IO bottleneck using esxtop from the ESX command line. When running esxtop view LUN queue statistics by pressing ‘u‘ – monitor the QUED, ACTV and LOAD stats. If LOAD is above 1.0 constantly, and therefore QUED is greater than 0, increasing the queue depth above 32 may increase performance. As always, apply the ‘if it’s not broke then don’t try to fix it‘ philosophy!

There are two settings that must bechanged to increase the queue depth.

The steps below apply to QLogic HBA’s only.

Adapter Queue Depth

Find the current module name:
   esxcfg-module -l | grep -i ql

Check the current module queue depth setting:
   cat /proc/scsi/qla2300/? | grep -i “queue depth”

This will return the value: Device queue depth = 0x20 (0x20 is HEX this is ‘32’ in decimal)

Change the queue depth using the following command, note there is no output from the command:
   esxcfg-module -s ql2xmaxqdepth=64 qla2300_707

Verify the change has been written to the esx.conf file:
   cat /etc/vmware/esx.conf

Reboot the ESX server
, then check the module configuration:
   cat /proc/scsi/qla2300/? | grep -i “queue depth”

This should return the value: Device queue depth = 0x40

Now, if you stop here you’ll find that the DQLEN will change from 32 to 64 and back again when viewing the LUN statistics in esxtop. It will keep changing randomly unless you perform the step below.


Now we must increase the Disk.SchedNumReqOutstanding to 64, otherwise the setting above will have no effect:
   esxcfg-advcfg -s 64 /Disk/SchedNumReqOutstanding

This setting does NOT require a restart of the ESX environment.

Additional Considerations

Disabling Device Resets (which can cause backup interruptions and flooding of NSR’s) and limiting resets to an individual LUN which will not affect the entire SAN should be considered. ESX 3.X by default enables device resets and disables LUN resets. The following commands can be executed from the ESX console:

   esxcfg-advcfg -s 1 /Disk/UseLunReset
   esxcfg-advcfg -s 0 /Disk/UseDeviceReset


VMWare : Configuring VCB with HP Dataprotector 6

VMWare : Configuring VCB with HP Dataprotector 6

This guide is intended to enable you to setup VCB on ESX 3.5 and HP DataProtector 6.0, it covers all elements from initial configuration of the VCB Proxy Server, to installation of the VCB Framework and both the required steps for backup AND restore of Virtual Machines.

VMWare Consolidatated Backup Concept

In essence a ‘VMWare Consolidatated Backup’ (VCB) is a crash consistant snapshot of a virtual machine which is redirected to a ‘VCB Proxy Server’ either across the LAN or Fibre Channel (SAN) infrastructure. When using HP DataProtector, before the backup is excuted, DataProtector calls a pre-exec script which quiesces the VM and performs the snapshot – redirecting the snapshot to the VCB proxy server, the backup then runs copying the data to tape before running a post-exec script which cleans up the snapshot.

One important thing to note is that VCB is an online process; traditional backup windows do not necessarily apply. One consideration is storage subsystem IOPS; VCB can be resource intensive for your SAN/Storage Infrastructure. I would reccommend running test VCB’s out-of-hours and closely monitoring the storage infrastructure to understand the performance impactof this process before performing backups during business hours.

The VCB Proxy server requires that there is sufficient disk-space equal to the size of the largest VCB (or sum of all concurrent VCBs) snapshot. For example, if you intend to perform a VCB of a VM with a total allocated vmdk size of 500GB, if only 300GB of the vmdk is used then the VCB will require approx 300GB to create a snapshot, if all 500GB has been used then the backup will require approx 500GB.

There are additonal considerations with Windows-based VM’s – it is necessary to ‘zero’ deleted files in order to reduce VCB snapshot sizes. If you were to create two 20GB files and then delete a single 20GB file, Windows will not ‘zero’ out the delete blocks, therefore the backup will still be 40GB. Using sdelete we can zero-out these blocks, reducing the backup by 20GB and therefore reducing the required backup window.

I will attempt to explain all of the above in more detail as we progress through this guide.


1. Windows VCB Proxy Server Configuration

1.1 Disabling Windows Automount

In order to void corruption of SAN attached disk data we must disable windows automounting. We will be presenting the ESX LUN’s to the VCB Proxy Server

Step 1: Open a command prompt and enter the command ‘diskpart

Step 2: Enter the command ‘automount

This will identify whether volume auto mounting is enabled on this system:

Step 3: If auto mounting is enabled we must disable t and flush the mount cache.

Step 4: Enter the command ‘automount disable

Step 5: Enter the command ‘automount scrub

Step 6: Enter the command ‘exit

Step 7: Reboot the VCB Proxy Server

Note: This will not affect existing LUNs that are assigned drive letters on this system.

1.2 Installation of VCB Framework

Step 1: Download the Vmware Consolodated Backup Framework application from and run the setup; accept the licence agreement and follow the default onscreen prompts Please see “Setting up VCB.pdf for full details.

Note: Once the VBC framework is installed, there is no need to configure it, as the VCB scripts downloaded from HP will be used.


1.3 Installation of HP Data protector VCB Scripts

Step 1: Download the VCB scripts from the HP site and extract them.

Step 2: On the VCB proxy server, copy vcbmount.js, vmwarepostexec.cmd and vmwarepreexec.cmd to the omniback\bin share.

Step 3: Create the file “vmware_passwd” in C:\Program Files\VMware\VMware Consolidated Backup Framework\config

The vmware_passwd file should contain the IP or server name of the ESX server, its user name and password in the format below:


1.4 Installation HP Data Protector Media Agent on the VCB Proxy Server

In order to ensure LAN-free backups the VCB Server must be fibre attached and have the Data Protector Media Agent Installed. Install this agent via the Dataprotector GUI.

1.5 Configuration of Table Libraries/Drives for the VCB Proxy Server

If applicable to your environement, the VCB Proxy Server should be zoned to enable the VCB Proxy to access the tape libraries/drives. This is configured by the SAN Fabric Manager or Storage Team.


2. Zoning of SAN Disks for VCB Proxy Server

The VCB Proxy Server must be zoned to allow access to the SAN Volumes used by the ESX host; without this FC backups are not possible.

Step 1: Zone the SAN, VCB Proxy and ESX Server together.

Step 2: Configure host presentation of the VCB Proxy Server on the SAN LUNs

Step 3: Reboot the VCB proxy server.

Please note that if LUNs are presented read-only to the VCB Proxy Server the server will take a long time to start Windows. LUN’s must be presented read-write, hence the atomount setting detailed at the beginning of this guide.


3. ESX Server Configuration

3.1 Creation of User Account and Delegation of Required Permissions

Step 1: Log on to the ESX server using the VMWare Infrastructure Client (VI Client) as a user with Administrator privileges.

Step 2:  From the VI Client, click Administration in the navigation bar. Click the Roles tab. Click Add Role

Step 3: Type a name for the new role, for example, VMware Consolidated Backup User.


Step 4: Select the following privileges for the new role:

  •  VirtualMachine > Configuration > Disk Lease
  • VirtualMachine > State > Create Snapshot
  • VirtualMachine > State > Remove Snapshot
  • VirtualMachine > Provisioning > Allow Virtual Machine Download
  • Virtual Machine > Provisioning > Allow Read?only Disk Access


Click OK to complete the process.  (These are the minimum permissions required)

Step 5: From the VI Client, click Inventory in the navigation bar.

Step 6: Click on Users & Group tab and select the Users button. In the white space, press the right mouse button and select Add.

Step 7: Enter the logon details and click the ok button. Note these logon details should match those in the ‘vmware_passwd’ file on the VCB proxy server. This will create the VCB user account.

Step 8:Select the ESX Server from the system tree on the left-hand-side. Select the Permissions tab from the main window. This will enable us to set server-wide permissions.

Step 9: Right-click and select Add Permission.

Step 10: Select the new user account and click OK.

Step 11: On the Assign Permissions screen, in the drop down menu select the VC Backup Role that was created earlier and press ok.

UPDATE : Windows 2008 R2 Support

It is possible to get this working on Windows 2008 R2; you’ll need to perform the following additinal steps:

  1. Manually create  C:\Program Files\OmniBack\tmp
  2. Set C:\Program Files (x86)\VMware\VMware Consolidated Backup Framework\vcbMounter.exe to Run this program As Administrator under executable compatibility options.

VMWare : Enabling SNMP on ESX 3.5

Enabling SNMP on ESX 3.5

SNMP traps can be used to monitor ESX serverhealth, and individual Virtual Machine status. An example of a free SNMP monitor for ESX is SolarWinds VM Monitor. In order to use these tools it is necessary to configure and enable the SNMPd on your ESX server.

First we must edit the snmpd.conf file:
   vi /etc/snmp/snmpd.conf

Change the rocommunity line to match your community string:
   rocommunity public

Ensure the VMWare MIB are enabled:
   dlmod SNMPESX            /usr/lib/vmware/snmp/

If using the ESX firewall you will need to open the snmp ports:
   esxcfg-firewall -e snmpd

Now start the snmpd service:
   /etc/init.d/snmpd start

Set SNMP to startup automatically on system boot:
   chkconfig snmpd on

You can query the status of the SNMPD service using the command:
   /etc/init.d/snmpd status

These changes can be made online, and there is no requirement to restart your ESX server.