Tag Archives: Vsan


An interesting question, if VSAN networking can be done/configured on VXLANS backed by NSX?

The answer is No and this is to avoid a circular dependency.

“However, very often, the question of compatibility is asked in the context of being able to place the vSAN network traffic on an NSX managed VxLAN/Geneve overlay. In this case, the answer is no, NSX does not support the configuration of the vSAN data network traffic over an NSX managed VxLAN/Geneve overlay. This is not unique to vSAN. The same restriction applies to any statically defined VMkernel interface traffic such as vMotion, iSCSI, NFS, FCoE, Management, etc.

Part of the reason for not supporting VMkernel traffic over the NSX managed VxLAN overlay is primarily to avoid any circular dependency of having the VMkernel infrastructure networks dependent on the VxLAN overlay that they support. The logical networks that are delivered in conjunction with the NSX managed VxLAN overlay are designed to be used by virtual machines which require network mobility and flexibility.”

Now you know..

vSAN Design Decision

Just a quick read and a refresher on vSAN Design Decision to keep my memory fresh..

Sizing for capacity, maintenance and availability 

1. The minimum configuration required for Virtual SAN is 3 ESXi hosts, or two hosts in conjunction with an external witness node. However, this smallest environment has important restrictions.

2. In Virtual SAN, if there is a failure, an attempt is made to rebuild any virtual machine components from the failed device or host on the remaining cluster. In a 3-node cluster, if one node fails, there is nowhere to rebuild the failed components. The same principle holds for a host that is placed in maintenance mode.

3. One of the maintenance mode options is to evacuate all the data from the host. However, this will only be possible if there are 4 or more nodes in the cluster, and the cluster has enough spare capacity.

4. One additional consideration is the size of the capacity layer. Since virtual machines deployed on Virtual SAN are policy driven, and one of those policy settings (NumberOfFailuresToTolerate) will make a mirror copy of the virtual machine data, one needs to consider how much capacity is required to tolerate one or more failures.

Design decision: 4 nodes or more provide more availability options than 3 node configurations. Ensure there is enough storage capacity to meet the availability requirements and to allow for a rebuild of the components after a failure.

Disks Will Fail – VSAN Doesn’t!

I was building my lab with VSAN backed storage – because I did not have a compatible RAID card I grouped the disks as a RAID-0 and presented it to VSAN. It worked and worked well – VSAN saw one big disk of about 1.6 TB (Had three disks worth 550GB each).

I was advised that it was a bad idea – because if the data got striped by the RAID controller and if one disk goes bad – VSAN now is stuck with missing and corrupted disk.

The proper way to do this was to have a RAID-0 on per disk and present all these individual disk volumes to VSAN to participate in a disk group. I went back and did that and boy am I glad that I did.

Today this happened


The fix was easy – Just highlight the disk and click on actions to remove the disk from the disk group. Depending on the failure you might be able to recover the data vsan-fail2

Thanks to Luke Huckaba – @thephuck for pointing it out in my lab which saved me hours of work later.

Now time to setup SPBM to avoid multiple failures 😀

Running Nested ESXi on VSAN

Was trying to deploy a nested ESXi on VSAN backed storage and kept running into this error during install.

This program has encountered an error:

Error (see log for more info):
Could not format a vmfs volume.
Command ‘/usr/sbin/vmkfstools -C vmfs5 -b 1m -S datastore1
/vmfs/devices/disks/mpx.vmhba1:C0:T0:L0:3′ exited with status 1048320

Turns out the problem is with a SCSI-2 reservation being generated as part of creating a default VMFS datastore. You can read more here.

The fix was this simple hack. = run this in each of your hosts and no system reboot is required.

esxcli system settings advanced set -o /VSAN/FakeSCSIReservations -i 1



An awesome guide is worth writing about. So before you run off to deploy your shiny new VSAN, make sure you have this handy guide beside you.

Corman Hogan wrote up an awesome guide for VSAN Troubleshooting that has pretty much everything you need to get that VSAN fixed up and running.

Here is the link and thanks Corman – awesome stuff!


I did not closely follow the vsan announcement from VMware however this is what I could pick up by the folks who were there to listen about this first hand.

  • 32 node support (up from the 16 node support announced at Partner Exchange last month, and up from the 8 nodes which we supported during the beta)
  • 2 million IOPS (using IOmeter 100% read, 4KB block size. Also 640K IOPS using IOmeter with 70/30 read/write ratio & 4KB block size)
  • 3200 virtual machines (100 per node)
  • 4.4 PB of storage (using 35 disk per host x 32 hosts per cluster)

There is also interoperability with vSphere Data Protection for backups, vMotion, DRS, HA, VMware View and vSphere replication for DR etc. VSAN is set to go GA on March 10th and no pricing or licensing was announced yet. This will probably be after March 10th.

Taken from Source.