Losing Datastores?

Are you losing Datastores and use VMware Consolidated Backup (VCB). Check if you VCB proxy server is setup correctly. Most importantly, check if automount is disabled on the server. Typically Datastores will disappear fron ESX servers that are not running any Virtual Machines from the Datastore, while ESX servers that are running machines can still see the Datastore. Once you power off those machines, the Datastore will disappear from those ESX servers as well.

Over a year ago, I had this happen on our site. The VCB server was installed by a third party and had been running a while. We just replaced the hardware of some of our ESX 3.5 clusters and for some reason, suddenly, we started losing Datastores. Since the VCB server had been running quite a while, I didn’t think of it straight away. But more and more clues let me to believe this must be the reason for the mystery disappearances. So I checked the VCB server to see if automount was disabled and, sure enough, it wasn’t. I still don’t get why it took so long before this happened. Luckilly there is a way to resolve this.

To check if automount is disabled on your VCB server, start a command prompt and run diskpart:
Diskpart
Automount

When it isn’t disabled, disable it with “automount disable” and “automount scrub”

When VCB has claimed a disk and gave it a windows signature, as it did in our case, you can check this with “fdisk –lu” in the CLI of the ESX 3.5 host. Or when you know wich device it is, with “fdisk /dev/sda” (You can find this with esxcfg-vmhbadevs):
# fdisk –lu

Output:
Disk /dev/sdb: 16.1 GB, 16106127360 bytes
255 heads, 63 sectors/track, 1958 cylinders, total 31457280 sectors
Units = sectors of 1 * 512 = 512 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 63 31455269 15727571 42 SFS  (The bold part signals the windows signature)

Missing partition tabel:
Disk /dev/sda: 322.1 GB, 322122547200 bytes
255 heads, 63 sectors/track, 39162 cylinders, total 629145600 sectors
Units = sectors of 1 * 512 = 512 bytes
Disk /dev/sda doesn’t contain a valid partition table

Or it could look something like below on some versions
Disk /dev/sde: 214.7 GB, 214748364800 bytes
255 heads, 63 sectors/track, 26108 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id

This is what it would normally look like:
Disk /dev/sdb: 16.1 GB, 16106127360 bytes
255 heads, 63 sectors/track, 1958 cylinders, total 31457280 sectors
Units = sectors of 1 * 512 = 512 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 128 31455269 15727571 fb Unknown  (This is what a vmfs partition entry should look like)

Before you start with the repair action, make sure all your Virtual machines on the disk are safe. You can sVmotion the machine to a Datastore without this problem or create a snapshot/clone of your SAN vdisk.

sVmotion with PowerCLI:
get-vm <vm name> | move-vm -datastore <dest datastore name>

Be absolutely sure the following is what you need to do, because you risk losing all data on the Datastore.

To restore partitie info:

To start the fdisk partition utility:
# fdisk /dev/sd? (? the letter for your specific volume)
print the partition table:
# p
To delete the previous partition (as I understand it, you may skip this part and jump right to the change partition type part):
# d
to create a new partition:
# n
print the partition table:
# p
choose the first:
# 1
choose the default:
# default
to change the partition type:
# t
set the type to fb (vmfs):
# fb
To change the alignment you have to go to expert mode:
# x
to move the beginning of data on a patition to 128 in stead of the default 63 on the ESX host:
# b
# 1
# 128 (disk alignment, check your SAN manual for the correct value, 128 is correct in most cases…)
To write the changes you have made:
# w

After you have done this for all your devices, you have to rescan the HBA devices, esxcfg-rescan vmhba0, or, better yet, with powerCLI:
Get-Cluster | Get-VMHost | Get-VMHostStorage -RescanAllHBA
Followed by a refresh
Get-Cluster | Get-VMHost | Get-VMHostStorage -Refresh

Here are just a few of the sites/links I found, with information regarding this issue, that helped me resolve this:
vcb proxy – forgot to disable automount…. by Joerg Riether
VMware KB 1002168
The Virtualization Team

About these ads

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s