RSS .92| RSS 2.0| ATOM 0.3
  • Home

    PowerCLI: Add VM’s in vApp, within vCloud Directory to Security Groups withing vShield App

    October 3rd, 2012

    The title says it all!  The use case:

    You are using vCloud Director, and want to add Virtual Machines from deployed vApps to specific Security Groups within vShield App.  In my case, there were three Security Groups created to make a 3-tier environment.  Web, App and Database.

    Once again Alan Renouf came through by creating a vShield module for PowerCLI.  Follow the directions in his video to install it.  It’s actually quite easy.

    The script I am going to list below requires valid connections to three sources in order to do the work:

    1. The vCenter that manages the compute nodes in your vCloud
    2. The vCloud Director cell.
    3. The vShield Manager for the vCloud stack.

    (You also need to be licensed for vShield App.)

    Prior to connecting to vShield Manager, you will need to instantiate the module Alan created.  That _should_ have been done when watching his video, but if not, do:

    import-module vshield

    within PowerCLI.

    At this point you can connect to your three services:

    • connect-viserver <for vCenter>
    • connect-ciserver <for vCloud Director>
    • connect-vshieldserver <for vShield Manager>

    Ok, so now hopefully our connections are set up.  Let’s describe the script a little more.  As I said before, the use case was to create a 3-tier environment via vShield App: Web, App and DB.  Our VM’s in the vApp are conveniently named “WWW,” “APP” or “DB.”  We are sort of cheating, and keying off that nomenclature to identify the VM’s.

    We have three hardcoded security groups in the script: Web, App and DB.  Their variables are $SGWeb, $SGApp and $SGDb.  I know I am clever.

    We are going to provide the name of a vAPP in vCloud Director from the command line.  This script will then walk the contents of the vApp, which are our three servers.  For those who are heavily involved in vCloud Director, you know that each VM in vCenter is identified by <VMNAME> (vCloud UUID).  In order for us to add a VM to vShield App, which is tied to vCenter, we must actually push that naming nomenclature.  I’m frankly not the best at coding, so I had to cheat and use the trim() function twice in order to pull the UUID out of the urn:vcloud:vm:uuid string.

    At that point, we use PowerShell’s like function to do string comparison, and then run Mr. Renouf’s set-vshieldsecuritygroup in order to place the VM in to appropriate vShield App Security Group.  That command is covered in his movie.  I hope you find it useful!

    Usage: ./<scriptname>.ps1 -vapp <vAPP name in vCD> -datacenter <the datacenter object where your vCD and vShield are attached>

    param (
    # Hardcode Security Groups, for now
    $SGWeb = "Web"
    $SGApp = "App"
    $SGDb = "DB"
     Foreach ($VM in (get-CIVM -vapp $vApp)) {
     $vCloudVM = $
     write-host "VM name: " $vCloudVM
     $vCloudID = $
     write-host "vCloud ID: " $vCloudID
     # for whatever reason the trim() function cuts off too much
     # so I had to trim twice. beats me why...
     $vCloudIDtrim = ($vCloudID).trim("urn:vcloud:")
     $vCloudIDtrim = ($vCloudIDtrim).trim("m:")
     write-host "Trimmed vCloud ID: " $vCloudIDtrim
     if ($vCloudVM -like '*www*'){
     write-host "Adding $vCloudVM to Security Group $SGWeb..."
     # add VM to SecurityGroup
     set-vShieldSecurityGroup -Add -Datacenter (get-Datacenter $dataCenter) -SecurityGroup $SGWeb -VM (Get-VM "$vCloudVM ($vCloudIDtrim)")
     elseif ($vCloudVM -like '*app*') {
     write-host "Adding $vCloudVM to Security Group $SGApp ..."
     # add VM to SecurityGroup
     set-vShieldSecurityGroup -Add -Datacenter (get-Datacenter $dataCenter) -SecurityGroup $SGApp -VM (Get-VM "$vCloudVM ($vCloudIDtrim)")
     elseif ($vCloudVM -like '*db*') {
     write-host "Adding $vCloudVM to Security Group $SGDb ..."
     # add VM to SecurityGroup
     set-vShieldSecurityGroup -Add -Datacenter (get-Datacenter $dataCenter) -SecurityGroup $SGDb -VM (Get-VM "$vCloudVM ($vCloudIDtrim)")

    The output will be of the form:

    VM Name: www001
    vCloudID: urn:vcloud:vm:<UUID>
    Trimmed vCloudID: <UUID>
    Adding www001 to Security Group Web …

    ID : securitygroup-nn
    Datacenter : datacenter
    Member : @{name=www001 (<UUID>); object
    TypeName=VirtualMachine; objectId=<moref>}
    Description :
    Name : Web

    VMware vCloud Networking Options

    May 7th, 2012

    Having worked with VMware vCloud-based technologies for a few months, I’ve come to the conclusion that networking and the automation glue which is required to make the magic happen, are both the most important pieces of the stack.

    To get started, I’ll list out some terms, and then we’ll build from there.

    • VXLAN
    • External Network(s)
    • Organization Network(s)
    • Network Pools
    • VLAN-backed
    • vSphere port group-backed
    • vAPP

    Let’s start from the bottom and work our way up.

    vAPP is not a networking technology, but a way to encapsulate an environment.  With it, we can create a three-tier stack, encapsulate it in a vAPP, and then roll out it out N times, all looking exactly the same.  One can also set start-up precedence (database VM starts first, app second, web third).  It’s great stuff.

    vSphere port group-backed networks are what you would traditionally use in a vSphere environment.  Create a Distributed Virtual Switch, and then create a port group.  vCloud Director can use port group-backed in many scenarios.  It is a simple way to get started by using known methods.

    VLAN-backed networks are a fun little way of defining a pool of VLAN’s (something like VLAN IDs 100-200).  Of course, it is necessary that the network team actually configure the VLAN ID’s on the network, and then assign them to the trunks for your ESXi servers.

    vCloud Director Networking Infrastructure (VCDNI) is a method  of creating private networks backed by a single physical [email protected] on your network.  Once you get more involved in vCloud, it is one way to create vAPP sandboxes in your environment.  In short, VCDNI uses MAC-in-MAC encapsulation.  Basically it works by creating private VLAN’s (you will actually see the port groups attached to your vDS) and then stuffing that data inside a packet that can be used on the physical VLAN.  Is the data private and secure?  From my experience, the answer is: sorta.    If your vAPPs are using VCDNI-backed networking, and attached to the same broadcast domain (the org network), the machines can be hit by any host in that broadcast domain (and then with the use of vShield Edge, you can ACL that).  To be clear, the default rule on a vShield Edge device is deny ingress).  If you have vAPPs in different broadcast domains, they are protected from one another (on layer 2).  One kicker, your virtual Distributed Switch must have MTU set to 1524 (if it was set to default of 1500) to allow for the larger header due to encapsulation.

    Is VCDNI good?  Yes.  Is VCDNI bad?  Probably could be argued by networking folks, since they technically do not control the allocation of networks, other than the physical VLAN VCDNI uses.  Is it the future?  Allegedly that is something else called VXLAN.  (update)My opinion:  It is a path to create private networks in a rapid fashion with minimal interaction by the network team.  It works for now, but hopefully VXLAN will be better.

    Now that we have defined methods to transport the data, we will get in to the nomenclature of vCloud.

    Network Pools can either be defined by VLAN-backed, Network isolation-backed (VCDNI) or Port group-backed.  These pools are consumed by virtual datacenters to create vAPP networks.

    Organization Networks are assigned to an Organization virtual DataCenter.  There are multiple ways to define an OrgNetwork:

    • Direct connection:  This network is akin to a traditional port group-backed network in vSphere.  In short, it provides connectivity to LAN, WAN or Internet traffic.  It is tied to an External network and usually sits on internally routable RFC-1918 address space (most likely for private cloud) or Internet-routable address space for providers.
    • NAT-routed connection:  This connection allows for Network Address Translation (NAT) of External IP space to internal private networks.  The NAT-routed OrgNet is typically in RFC-1918 address space, however there are other cases.
    • Internal Organization network: This is strictly an internal network for the vApps to communicate with each other, but have no external network access.

    External Networks are port group-backed networks (defined in vCenter) that provide ingress and egress to the Cloud environment.  They should be routable networks, either RFC-1918 for private, or Internet routable for providers.

    Place ESXi in to Maintenance Mode from vCloud Director

    February 21st, 2012

    So you have your handy dandy cloud built on top of VMware vSphere and vCloud Director. And then you find out you need to conduct maintenance on the host.  What to do?

    Easy!  Browse to:

    • System-> Manage & Monitor
    • vSphere Resources -> Hosts
    1. Find the host you need to place in to maintenance mode, right click and select Disable Host.
    2.  At that point, the status will turn from a green circle with a check, to a red circle.
    3. Right click on the host again and select Redeploy All VMs.
    4. The ESXi host will go in to maintenance mode in the vCenter server and evacuate all virtual machines as usual.
    5. (Optional!) If you see vsla errors (such as the screenshot), issues with deleting vApps, Unprepare the host which removes the vCloud agent from ESXi
    6. (Optional!) Prepare the host for vCloud by pushing the vCloud agent to ESXi
    7. When maintenance is complete, right click and Enable Host.
    8. And your work is complete!

    Quick and dirty method to mount BCV/Snapshot

    November 25th, 2011

    There are many many blog posts about mounting BCV (Business Continuity Volume) or SAN Snapshots, however here is my method.  It is a quick shell script to run on each ESXi server.  Add it to your business operations manager, and create an ad-hoc method of mounting BCV/snaps for a DR exercise.

    NOTE: Commands are in bold.

    Verify from the storage team that they have assigned the BCV/snap to your hosts

    SSH in to ESXi server (assumes you have all of the buttons pressed and knobs turned)

    Search of the BCV/snap volumes.  Do: esxcfg-volume -l  Note: Be patient, this may take a few minutes.

    The output will be as follows:

    VMFS3 UUID/label: <Datasture UUID>/<Datastore label>
    Can mount: Yes
    Can resignature: Yes
    Extent name: naa.<device ID>     range: <size in MB>

    It is now possible to mount the volume manually via the Datastore UUID or Datastore Label.   Do: esxcfg-volume -m <Datastore UUID -OR- Datastore Label>
    Note: This will conduct a force mount of the volume.
    If there are powered-on VM’s on that datastore, you can unmount it.  Do: esxcfg-volume -u <Datastore UUID -OR- Datastore Label>

    To magically wrap this up to scan for any assigned BCV/snaps and mount them automagically, run the following:

    for volume in `esxcfg-volume -l |grep VMFS3 |awk ‘BEGIN {FS=”/”} ; {print $2}’ |awk ‘{print $2}’` ; do echo “Mounting volume with UUID $volume” ; esxcfg-volume -m $volume ; done

    VAAI and supportability

    June 28th, 2011

    VAAI, or vStorage API for Array Integration, is an industry answer to offload specific storage requests from the ESX(i) servers to the storage array.

    VAAI implements only three functions, as described in VMware KB1021976:

    • Atomic Test & Set
    • Clone Blocks/Full Copy/XCOPY
    • Zero Blocks/Write Same

    As it turns out, only certain arrays support VAAI.  Since VAAI is on by default, at least in 4.1, the hosts will send VAAI commands to the array.  The array will usually say they are not supported, and then ESX will fall back to regular SCSI commands.  However, there seems to be occasions when SAN devices do not support VAAI, but can apparently cause failure states.  I’ve seen it.  It’s true.

    Now, how do we determine if the SAN handles VAAI?  SSH to your servers and run the following command, via KB102197:

    esxcfg-scsidevs -l | egrep “Display Name:|VAAI Status:”

    If the array does not support VAAI, you will most likely see results such as:

    Display Name: <Vendor> Fibre Channel Disk (naa.600*)
    VAAI Status: unknown

    VMware KB1033665 tells us how to disable VAAI both from the ESX(i) command line, as well as the vSphere Remote CLI

    ESX(i) command line:

    esxcfg-advcfg -s 0 /DataMover/HardwareAcceleratedMove

    esxcfg-advcfg -s 0 /DataMover/HardwareAcceleratedInit

    esxcfg-advcfg -s 0 /VMFS3/HardwareAcceleratedLocking

    -s 0 will disable, -s 1 will enable. You can also substitute -s 0 with -g to see the current setting.

    Remote CLI: –server <servername> –username <possibly root> –password <password> -s 0 /DataMover/HardwareAcceleratedMove –server <servername> –username <possibly root> –password <password> -s 0 /DataMover/HardwareAcceleratedInit –server <servername> –username <possibly root> –password <password> -s 0 /DataMover/VMFS3/HardwareAcceleratedLocking

    where items in < > are dependent on your specific configuration.  You can again substitute -s 0 with -g to see the current setting.


    How available is VMware’s Round Robin Path Selection Plugin?

    June 10th, 2011

    VMware wisely introduced Round Robin (RR) as a supported path selection plugin (PSP) as part of their Native Multipathing (NMP) suite.  We originally dealt with Fixed Path (FP) and Most Recently Used (MRU), which led to administrators to directly manage I/O if their servers had multiple host bus adapters (HBA) and/or storage targets.  I’m sure most admins stuck with fixed path, which provided for an active/passive configuration, and went on with their business.  I know I have.

    Along comes Round Robin to many cheers and hoorays, and we merrily go about our business and configure our hosts to use it.  We extolled the virtues to our management saying how available our connection to the SAN fabric and storage has become now that we are using one path per I/O.  I know I did.

    Here is the rub.  Round Robin is not designed to provide High Availability.  As it turns out, VMware has designed and built RR (and the two other PSP for that matter) to only mark a path dead when it receives certain SCSI sense codes.  You can find the full list on the VMware KB.  They have followed the specifications, and I completely understand why they made their choices.

    But what happens when we are having issues with the SAN fabric and do NOT see those specifics sense codes?  If you guessed nothing, you would be very correct.  The implementation of RR will only move to the next path upon a SUCCESSFUL (sense code 0x0 or 0x00) I/O.  If we receive “soft” errors such as command: 0x2 (Host_Bus_Busy) or 0x28 (Task Set Full), RR will not move to the next path, due to the fact it has not received a 0x0 code.  VMware’s definition of SCSI error conditions can be found at VMware KB #1030381.  This means we will be STUCK on a path that is having problems, which results in failed I/Os.  You may notice the ESX(i) server disconnect from vCenter, the ability to run df from the command line diminish, and no way to enumerate the storage.  You will also notice that virtual machines stop responding to pings, and applications failures there-in.  This is a result of failed I/O’s in the virtual machine.  SCSI errors will be clearly visible within the logs of guest OS.

    You can find the error messages on ESX(i) in the system log from the command line:

    • do: cd /var/log
    • do: grep naa messages

    Or look through the old logs:

    • do: zcat messages*.gz |grep naa

    Due to ESXi’s very quick log rotation, you may be out of luck by the time you respond to an event.  You should take the time and export syslog from your ESXi hosts to a central server such as the vMA appliance.  If you need help setting up syslog, see Kanuj Behl’s post blog post on

    vSphere Round Robin MultiPathing

    March 29th, 2011

    There are a number of blog posts describing the configuration of Round Robin (RR) multipathing on vSphere.  *Note: Content on this page has been distilled from the sources referenced below, as well as my colleague  Check those sites for a deeper dive in to the content.  I’ve also removed some identifiers from the output.

    The three commands that are your friends throughout this post:

    esxcli nmp satp list <- Storage Array Type Plugin (SATP)

    esxcli nmp psp list <- Path Selection Plugin (PSP)

    esxcli nmp device list <- List the LUNs from the SAN represented as their device names

    1) SSH in to the server (assuming you enabled remote tech support from the console).

    2) Display the current pathing configuration:

    esxcli nmp device list

    Device Display Name: Fibre Channel Disk (naa.60)
    Storage Array Type: VMW_SATP_DEFAULT_AA
    Storage Array Type Device Config: SATP VMW_SATP_DEFAULT_AA does not support device configuration.
    Path Selection Policy: VMW_PSP_FIXED
    Path Selection Policy Device Config: {preferred=vmhba:C:T:L;current=vmhba:C:T:L}

    3.1) If you have storage from NetApp, do(note, there are two dashes before “psp” and “satp”):

    esxcli nmp satp setdefaultpsp –psp VMW_PSP_RR –satp VMW_SATP_DEFAULT_AA

    3.2) If you have certain storage from an EMC DMX, do:

    esxcli nmp satp setdefaultpsp –psp VMW_PSP_RR –satp VMW_SATP_SYMM

    These commands will change the default pathing to round robin (PSP or Path Selection Plugin) for the specific SATP (Storage Array Type Plugin).

    3.3) At this point, you can reboot the  if LUNs were already presented.  If no SAN storage is attached, scan in the new devices, and they will be automagically set to round robin.  Or, run the following command to set the Path Selection Policy:

    for i in `ls /vmfs/devices/disks/ | grep naa.60` ; do esxcli nmp device setpolicy –device $i -P VMW_PSP_RR ; done

    4) Check the current config, post reboot:

    esxcli nmp device list

    Device Display Name: Fibre Channel Disk (naa.60)
    Storage Array Type: VMW_SATP_DEFAULT_AA
    Storage Array Type Device Config: SATP VMW_SATP_DEFAULT_AA does not support device configuration.
    Path Selection Policy: VMW_PSP_RR
    Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0;lastPathIndex=1: NumIOsPending=0,numBytesPending=0}

    Look at Path Selection Policy.  It now says WVM_PSP_RR instead of VMW_PSP_FIXED.  We are getting closer to our goal.

    5) Now we want to configure the round robin policy to send 1 IO down a path, and then round robin to the next path (note: there are two dashes before “type”).

    for i in `ls /vmfs/devices/disks/ | grep naa.60` ; do echo $i ; esxcli nmp roundrobin setconfig –type “iops” –iops=1 –device $i ;done

    This command will look in the /vmfs/devices/disks/ directory, grab anything that starts with naa.60 (which should pick up SAN storage), and then set the round robin policy to 1 IO per path.

    6) Verify the new configuration:

    esxcli nmp device list

    Device Display Name: Fibre Channel Disk (naa.60)
    Storage Array Type: VMW_SATP_DEFAULT_AA
    Storage Array Type Device Config: SATP VMW_SATP_DEFAULT_AA does not support device configuration.
    Path Selection Policy: VMW_PSP_RR
    Path Selection Policy Device Config: {policy=iops,iops=1,bytes=10485760,useANO=0;lastPathIndex=5: NumIOsPending=0,numBytesPending=0}

    Validating our output, we now have our policy=iops, and iops=1.