VMware vCenter Backup and Restore

VMware vCenter Backup and Restore

The ADAM data is backed up every 5 minutes into the vCenter Server database. To backup the latest update of ADAM data, ensure that the VMware VirtualCenter Management Webservices service is running for at least 5 minutes before stopping the other vCenter services.

Backup

Stop the VMware VirtualCenter Service, VMwareVCMSDS, and the Database service.

Back-up the vCenter Server database.

This backup includes backing up ADAM information. For more information, see the vSphere Upgrade Guide.

Backup the SSL certificate folder at:

  • On Microsoft Windows 2003: %ALLUSERSPROFILE%\Application Data\VMware\VMware VirtualCenter
  • On Microsoft Windows Vista and Microsoft Windows 2008 Server: %ALLUSERSPROFILE%\VMWare\VMware VirtualCenter

Restore

Restore the database. For more information, see the vSphere Upgrade Guide.

Restore SSL certification.

Copy the backed up SSL certificate folder to the same path on the destination.

Create the folder if it does not already exist.

Prepare a DSN that points to the database.

Install vCenter Server 4 and connect it to the database during installation steps. ADAM information is restored by this process.(If vCenter Server 4 was installed on the target operating system to restore, you need to uninstall it prior to the restoration.)

EsxI Backup Configuration via PowerCLI

EsxI Backup Configuration via PowerCLI The following PowerCLI script will backup the configuration of your ESXi servers: $backpath = “C:\backups\” $eVMHs = Get-View -ViewType HostSystem |?{$_.config.product.ProductLineId -eq “embeddedEsx”} |?{$_.Runtime.ConnectionState -eq “connected”} Foreach ($eVMH in $eVMHs) {Set-VMHostFirmware -VMHost $VMH.name -BackupConfiguration -DestinationPath $backpath} To restore the configuration Step#1, put the ESXi host is maintenance mode, this is a requirement: set-VMHost -vmhost MyESXiHost -state Maintenance Step#2, restore the config bundle: Get-VMHost MyESXiHost | Set-VMHostFirmware -Restore -SourcePath C:\Temp\configBundle-MyESXiHost.tgz You can eventually add the -Force if the bundle is mismatched, that is when you restore to a different ESXi build.

VMWare : Device or Resource Busy

VMWare :  Device or Resource Busy

After removing a VM from an ESX 3.5 update 3 server via the Infrastructure Client, selecting ‘Delete from Disk’ I found that a disused vmdk file was left over on the VMFS datastore.  When deleting the vmdk file I was presented wih the following error:
   cannot remove `TEMP_Data-G_50GB-flat.vmdk’: Device or resource busy

Using the command ‘fuser TEMP_Data-G_50GB-flat.vmdk‘ to identify users/processes which have locked the file showed no active users on the file.

To resolve, run the following command, and wait approx 5 minutes:
   service mgmt-vmware restart

The command will not impact any running VM‘s, it will disconnect any management client sessions (not console access via SSH) and will stop any running VCB’s.

 Now browse the datastore and delete the file.

VMWare : Configuring VCB with HP Dataprotector 6

VMWare : Configuring VCB with HP Dataprotector 6

This guide is intended to enable you to setup VCB on ESX 3.5 and HP DataProtector 6.0, it covers all elements from initial configuration of the VCB Proxy Server, to installation of the VCB Framework and both the required steps for backup AND restore of Virtual Machines.

VMWare Consolidatated Backup Concept

In essence a ‘VMWare Consolidatated Backup’ (VCB) is a crash consistant snapshot of a virtual machine which is redirected to a ‘VCB Proxy Server’ either across the LAN or Fibre Channel (SAN) infrastructure. When using HP DataProtector, before the backup is excuted, DataProtector calls a pre-exec script which quiesces the VM and performs the snapshot – redirecting the snapshot to the VCB proxy server, the backup then runs copying the data to tape before running a post-exec script which cleans up the snapshot.

One important thing to note is that VCB is an online process; traditional backup windows do not necessarily apply. One consideration is storage subsystem IOPS; VCB can be resource intensive for your SAN/Storage Infrastructure. I would reccommend running test VCB’s out-of-hours and closely monitoring the storage infrastructure to understand the performance impactof this process before performing backups during business hours.

The VCB Proxy server requires that there is sufficient disk-space equal to the size of the largest VCB (or sum of all concurrent VCBs) snapshot. For example, if you intend to perform a VCB of a VM with a total allocated vmdk size of 500GB, if only 300GB of the vmdk is used then the VCB will require approx 300GB to create a snapshot, if all 500GB has been used then the backup will require approx 500GB.

There are additonal considerations with Windows-based VM’s – it is necessary to ‘zero’ deleted files in order to reduce VCB snapshot sizes. If you were to create two 20GB files and then delete a single 20GB file, Windows will not ‘zero’ out the delete blocks, therefore the backup will still be 40GB. Using sdelete we can zero-out these blocks, reducing the backup by 20GB and therefore reducing the required backup window.

I will attempt to explain all of the above in more detail as we progress through this guide.

 

1. Windows VCB Proxy Server Configuration

1.1 Disabling Windows Automount

In order to void corruption of SAN attached disk data we must disable windows automounting. We will be presenting the ESX LUN’s to the VCB Proxy Server

Step 1: Open a command prompt and enter the command ‘diskpart

Step 2: Enter the command ‘automount

This will identify whether volume auto mounting is enabled on this system:

Step 3: If auto mounting is enabled we must disable t and flush the mount cache.

Step 4: Enter the command ‘automount disable

Step 5: Enter the command ‘automount scrub

Step 6: Enter the command ‘exit

Step 7: Reboot the VCB Proxy Server

Note: This will not affect existing LUNs that are assigned drive letters on this system.


1.2 Installation of VCB Framework

Step 1: Download the Vmware Consolodated Backup Framework application from http://www.vmware.com and run the setup; accept the licence agreement and follow the default onscreen prompts Please see “Setting up VCB.pdf for full details.

Note: Once the VBC framework is installed, there is no need to configure it, as the VCB scripts downloaded from HP will be used.

 

1.3 Installation of HP Data protector VCB Scripts

Step 1: Download the VCB scripts from the HP site and extract them.

Step 2: On the VCB proxy server, copy vcbmount.js, vmwarepostexec.cmd and vmwarepreexec.cmd to the omniback\bin share.

Step 3: Create the file “vmware_passwd” in C:\Program Files\VMware\VMware Consolidated Backup Framework\config

The vmware_passwd file should contain the IP or server name of the ESX server, its user name and password in the format below:

 

1.4 Installation HP Data Protector Media Agent on the VCB Proxy Server

In order to ensure LAN-free backups the VCB Server must be fibre attached and have the Data Protector Media Agent Installed. Install this agent via the Dataprotector GUI.

1.5 Configuration of Table Libraries/Drives for the VCB Proxy Server

If applicable to your environement, the VCB Proxy Server should be zoned to enable the VCB Proxy to access the tape libraries/drives. This is configured by the SAN Fabric Manager or Storage Team.

 

2. Zoning of SAN Disks for VCB Proxy Server

The VCB Proxy Server must be zoned to allow access to the SAN Volumes used by the ESX host; without this FC backups are not possible.

Step 1: Zone the SAN, VCB Proxy and ESX Server together.

Step 2: Configure host presentation of the VCB Proxy Server on the SAN LUNs

Step 3: Reboot the VCB proxy server.

Please note that if LUNs are presented read-only to the VCB Proxy Server the server will take a long time to start Windows. LUN’s must be presented read-write, hence the atomount setting detailed at the beginning of this guide.

 

3. ESX Server Configuration

3.1 Creation of User Account and Delegation of Required Permissions

Step 1: Log on to the ESX server using the VMWare Infrastructure Client (VI Client) as a user with Administrator privileges.

Step 2:  From the VI Client, click Administration in the navigation bar. Click the Roles tab. Click Add Role

Step 3: Type a name for the new role, for example, VMware Consolidated Backup User.

 

Step 4: Select the following privileges for the new role:

  •  VirtualMachine > Configuration > Disk Lease
  • VirtualMachine > State > Create Snapshot
  • VirtualMachine > State > Remove Snapshot
  • VirtualMachine > Provisioning > Allow Virtual Machine Download
  • Virtual Machine > Provisioning > Allow Read?only Disk Access

 

Click OK to complete the process.  (These are the minimum permissions required)

Step 5: From the VI Client, click Inventory in the navigation bar.

Step 6: Click on Users & Group tab and select the Users button. In the white space, press the right mouse button and select Add.

Step 7: Enter the logon details and click the ok button. Note these logon details should match those in the ‘vmware_passwd’ file on the VCB proxy server. This will create the VCB user account.

Step 8:Select the ESX Server from the system tree on the left-hand-side. Select the Permissions tab from the main window. This will enable us to set server-wide permissions.

Step 9: Right-click and select Add Permission.

Step 10: Select the new user account and click OK.

Step 11: On the Assign Permissions screen, in the drop down menu select the VC Backup Role that was created earlier and press ok.

UPDATE : Windows 2008 R2 Support

It is possible to get this working on Windows 2008 R2; you’ll need to perform the following additinal steps:

  1. Manually create  C:\Program Files\OmniBack\tmp
  2. Set C:\Program Files (x86)\VMware\VMware Consolidated Backup Framework\vcbMounter.exe to Run this program As Administrator under executable compatibility options.

VMWare : Enabling SNMP on ESX 3.5

Enabling SNMP on ESX 3.5

SNMP traps can be used to monitor ESX serverhealth, and individual Virtual Machine status. An example of a free SNMP monitor for ESX is SolarWinds VM Monitor. In order to use these tools it is necessary to configure and enable the SNMPd on your ESX server.

First we must edit the snmpd.conf file:
   vi /etc/snmp/snmpd.conf

Change the rocommunity line to match your community string:
   rocommunity public

Ensure the VMWare MIB are enabled:
   dlmod SNMPESX            /usr/lib/vmware/snmp/libSNMPESX.so

If using the ESX firewall you will need to open the snmp ports:
   esxcfg-firewall -e snmpd

Now start the snmpd service:
   /etc/init.d/snmpd start

Set SNMP to startup automatically on system boot:
   chkconfig snmpd on

You can query the status of the SNMPD service using the command:
   /etc/init.d/snmpd status

These changes can be made online, and there is no requirement to restart your ESX server.

VMWare VCB Troubleshooting ‘ non-zero return code’

VMWare VCB Troubleshooting ‘ non-zero return code’

I recently came across the following error when running a VCB via HP DataProtector 6.0:

 

“Creating a quiesced snapshot failed because the (user-supplied) custom pre-freeze script in the virtual machine exited with a non-zero return code”

The following steps resolved the issue:
  > Perform a ‘repair’ of the VMWare tools installed on the Virtual Machine
  > Restart the Virtual Machine

The error is generated because the VCB fails to execute a script on the VM due to an issue with the VMWare tools on the guest.

VMWare : VCB Cleanup Script

VMWare : VCB Cleanup Script

VCB is a powerful and useful technology for backing up Virtual Machines running on ESX/vSphere, more than once it has rescued an entire Virtual Machin where a standalone server would have had to be rebuilt.

My only fault with the ‘vanilla’ installation is the cleanup process. Whena  VCB backup fails, or even succeeds at times, the snapshot is not cleaned up on the VCB proxy server or the ESX host. This can result in wastedstorage on both systems.

VCB includes a cleanup script which will clean directories in the VCB root folder as specified in the following file: C:\Program Files\VMware\VMware Consolidated Backup Framework\config\config.js

This script works well at cleaning up folders for VCB that are created dynamically in the root folder, however, if you have created subdirectories for per-VCB tasks the script will not automatically cleanup the VCB snapshots. I create a sub-folder on a per-VCB basis, why? This is simple,. because using HP dataprotector to perform VCB backups I have to specify a folder that exists tobackup to tape. If I specify the VCB root folder then if a VCB fails and the data is not cleaned up, the next VCB will be significantly larger as it will contain the data for the failed VCB snapshot and the new VCB snapshot.

I have around 50 VM’s, therefore manually editing the VCB config.js file to reflect the root VCB directory for each backup was a no-no.

You’ll find a link below to a modified version of the vcb-cleanup.wsf file included with the VCB proxy package. Copy this file to the ‘C:\Program Files\VMware\VMware Consolidated Backup Framework\generic’ directory.

Modify the  C:\Program Files\VMware\VMware Consolidated Backup Framework\config\config.js file to reflect the root cotainer for all of your VCB folders.

Follow the steps outlined below to perform VCB cleanup on all subfolders:

  1. Ensure no VCB backups are in progress
  2. Open a command prompt, cd to  “C:\Program Files\VMware\VMware Consolidated Backup Framework\generic”
  3. Execute the following command “cscript.exe vcb-cleanup-mod.wsf “C:\Program Files\VMware\VMware Consolidated Backup Framework”  -y”

Download the script from here. Not, you will need to rename the file from ‘.txt‘ to ‘.wsf

In order for VCB to work under Windows 2008 R2 x64 you must configure vcbMounter.exe to “Run As Administartor”

VMWare VCB : Improving Performance of VCB

VCB Backup Essentials

Having recently introduced VCB backups into Dataprotector 6.0 I thought I would share a few useful tips for ensuring that backup speeds are as fast as possble.

1) Ensure that all VM’s have a scheduled task to zero-out free space prior to VCB running. Windows, when you delete a file does not zero-out the disk space (populate the data blocks with zero’s) – soif you had a 20GB drive that contained 15GB of data, then you delete 10GB of data, unoless you zero out this space the backup will still be 15GB.

I use the free ‘SDELETE‘ tool from sysinternals (now Microsoft) to do this, and simply execute a scheduled task before the backup is due to run. SDELETE can eb found here: http://technet.microsoft.com/en-us/sysinternals/bb897443.aspx

2) When running VCB, check the disk queue performance counter on the VCB Proxy server, the storage to which the VCB snapshot is taken can be a serious bottleneck for VCB performance. Initially I was running VCB over fibre, to a fibre attached SAN disk. I found that after 1.97GB the backup would grind to a halt – 200Kb/sec!!! By changing the VCB snapshot drive to local RAID0 storage this increased to over 2.2GB/min, or 37.5MB/sec. Your hardware may be capable of significantly faster speeds.

3) Disable additional disk paths on the VCB Proxy Server: VCB does not like MPIO/multiple paths to LUNS. This step is probably the biggest potential speed gain you’ll get. Disable the additional disk objects in Windows device manager, test you backups once complete, if they don’t work enable the path you disabled and disable a different one. This can see speed improvements of 100MB/sec.

4) Run multiple VCB snapshots at the same time. Your SAN containing the VM’s will, more than likely, support more than 35MB/sec. Just ensure you change the snapshot directory otherwise your backup application may backup multiple snapshots at once!