RSS .92| RSS 2.0| ATOM 0.3
  • Home

    VMworld 2012 round-up: INF-VSP1196 What’s new with vCloud Director Networking

    August 29th, 2012

    VMware 2012 presentation INF-VSP1196: What’s new with vCloud Director Networking

    This session was discussed the new networking features of vCloud Director 5.1 (VMware decided to sync the version with the release of vSphere 5.1, jumping from 1.5.1 all the way to 5.1).

    From the presentation content, the bulk of changes focus vShield Edge and VxLAN.  vShield now is bundled in two ways: Security and Advanced and sold as Standard or Enterprise.  More will be discussed below about the changes, but in short the actual Edge VM is deployed in two sizes, with different supported features.

    New features of vShield Edge:

    • Multiple interfaces, up to 10, are now supported with the Advanced bundle.  This is an increase of 2.
    • The virtual hardware is now 7.
    • The appliance, as stated before, can be deployed as the compact or full version of edge.  The major difference, according the presentation, is the support for higher throughput and a active/standby edge appliance.  I for one welcome the change since the current instantiation of Edge only allowed for a respawn of a device which required an outage.
    • The Edge appliance can act as a DNS relay for internal clients.
    • External address space can be increased on the fly.
    • Non-contiguous networks can be applied to the external interface of the vShield Edge.
    • Ability to sub-allocate IP addresses to Organization vDCs.

    With vCloud Director version 5.1, a new network object is available for use by Organizations: Organization vDC (virtual datacenter) Networks.  Since an Organization Network (OrgNet) is mapped to a single Organization, the new Org vDC Network can now span multiple org vDC within an Organization.  The fellow glossed over the use-case for this situation, and one does not easily come to mind at the moment.

    VMware is also debuting something they call Service Insertion.  This is basically a new security API for 3rd party vendors to integrate directly in to the networking stack for their products.  Profiles can now be created based on services, and these profiles can then be applied to a Port Group of a Distributed Switch.  I do believe VMware is attempting to allow providers to create billing and a-la carte models to generate income from their clients.  It is an interesting play to see if it is really used only in Public offerings, or if private clouds offer it in a charge-back model.

    Edge can provide a DHCP service, available on isolated networks.  You now can use:

    • Multiple DHCP pools per edge device (necessary with 10 supported interfaces).
    • Single pool per interface.
    • No option for advanced features such as lease times.


    • Rules can be applied to an interface.
    • Rules can be arranged via a drag and drop interface, but they are evaluated from top down.  The first hit causes an exit.
    • Source NAT (SNAT) and Destination NAT (DNAT) supports: TCP, UDP, TCP and UDP, ICMP or any.
    • There are predefined ICMP types.


    • VMware is still trumpeting their Edge firewall as 5 tuple (5 different options for filtering, but it still isn’t all that great).
    • Rules can be arranged via drag and drop.
    • Logging per rule.
    • Support for TCP, UDP, TCP and UDP.
    • Can not filter on ICMP types (ping versus traceroute).  I do believe it is all or nothing.

    Static Routing

    • VMware stated it is useful for routing between Org networks.  I think this use-case would be for far more advanced configurations.
    • Can be used for deep reach in vApp networks.  The current Edge device does support static routing even when using vCDNI, but the MAC in MAC encapsulation adds some serious latency to the connections.  I suspect VxLAN is to thank for this configuration to be better supported.


    • IPsec or SSL site to site configuration, not for user remote access.
    • Compatible with 3rd party software and hardware VPN, since Edge is doing IPsec or SSL.  Nothing proprietary there.

    Load Balancer

    • Load Balance on HTTP, HTTPS or any old TCP port.
    • Can conduct a basic health check of the back-end servers with either a URI (except for HTTPS) or tcp port.
    • Configure pool servers and VIP.
    • Balance on IP Hash, URI or least connections.
    • NOTE:  The current version uses nginx.  I saw it not work even close to correctly with certain network configurations based around VCDNI.  Let’s hope it works better in this version.

    Virtual Service (Load balancing)

    • HTTP persistence can be configured to use cookies with insert feature.
    • HTTPS can use session IDs.
    • There is no persistence option for regular TCP ports.

    And now for the queen mother of all session topics: VXLAN.  Boiling it down, VXLAN allows for a layer 2 network, say, to exist live in two places at once.  Think 2 datacenters, or in this case, the Cloud.

    • Layer 2 overlay on a Layer 3 network .
    • Each overlay network is known as a VXLAN segment.
    • VXLAN identified by 24 bit segment ID, known as a VNI.
    • Traffic carried by VXLAN tunnel endpoints, known as VTEP.
      • ESXi hosts or Cisco Nexus 1000v can act as VTEP.
    • Virtual machines have no idea of the existence of VXLAN transporting their traffic.
    • VM to VM traffic is encapsulated in a VXLAN header.
    • Traffic on same portgroup is not encapsulated.
    • Here is the big kicker: multicast is required
      • Used for VM broadcast and multicast messages
      • In essence, a dedicated virtual Distributed Switch
      • Available vNIC and IP address per switch
      • Mutlicast addresses
      • Multicast configured on the physical network
    • Requires multicast end to end (all networking points between the VTEP).
    • Minimum MTU of 1600 (in the network).

    The technology sounds cool, is hopefully better than VCDNI, but the requirement of multicast may be a show-stopper to some people.

    Fun with VMware vShield Edge

    May 8th, 2012

    As part of VMware’s vCloud implementation, vShield Edge can become a major part of your infrastructure.

    In short, it can be used as a Layer 3 device including a firewall (but nothing outrageously complex).  So once it’s spun out, how does one troubleshoot connectivity errors?  And what sorts of fun things can we do with it?

    The following Q/A between myself and myself are all from the CLI.  So open up the vCenter which your vCD cell uses, and open a console on a vShield Edge device, identified by vse-.  Please authenticate first.  User is: admin and password is: default, but default.

    One more note, question mark (?) and tab completion <tab> are your friends…

    • Question: Where do I start.
    • Answer: list or ?  … No really, hit the Question Mark (?) key.  Help doesn’t help
    • How do I show the system log?
    • Answer: show log follow
    • Question:  How do I tell what ports are currently open?
    • Answer: show system network_connections
    • Question: How do I show NAT rules?
    • Answer: show iptables NAT
    • Question: And how do I do a tcpdump on a vse device?
    • Answer:  You have two options, the outside interface extif, or the inside interface intif.  To monitor the outside: debug packet display interface extif.  To monitor the inside interface, debug packet display interface inif
    You get the idea.
    The vShield Edge Appliance is just a stripped down Linux VM, that can accomplish Layer 3 routing, basic firewalling and IPsec tunneling.  Pretty good stuff.

    VMware vCloud Networking Options

    May 7th, 2012

    Having worked with VMware vCloud-based technologies for a few months, I’ve come to the conclusion that networking and the automation glue which is required to make the magic happen, are both the most important pieces of the stack.

    To get started, I’ll list out some terms, and then we’ll build from there.

    • VXLAN
    • External Network(s)
    • Organization Network(s)
    • Network Pools
    • VLAN-backed
    • vSphere port group-backed
    • vAPP

    Let’s start from the bottom and work our way up.

    vAPP is not a networking technology, but a way to encapsulate an environment.  With it, we can create a three-tier stack, encapsulate it in a vAPP, and then roll out it out N times, all looking exactly the same.  One can also set start-up precedence (database VM starts first, app second, web third).  It’s great stuff.

    vSphere port group-backed networks are what you would traditionally use in a vSphere environment.  Create a Distributed Virtual Switch, and then create a port group.  vCloud Director can use port group-backed in many scenarios.  It is a simple way to get started by using known methods.

    VLAN-backed networks are a fun little way of defining a pool of VLAN’s (something like VLAN IDs 100-200).  Of course, it is necessary that the network team actually configure the VLAN ID’s on the network, and then assign them to the trunks for your ESXi servers.

    vCloud Director Networking Infrastructure (VCDNI) is a method  of creating private networks backed by a single physical [email protected] on your network.  Once you get more involved in vCloud, it is one way to create vAPP sandboxes in your environment.  In short, VCDNI uses MAC-in-MAC encapsulation.  Basically it works by creating private VLAN’s (you will actually see the port groups attached to your vDS) and then stuffing that data inside a packet that can be used on the physical VLAN.  Is the data private and secure?  From my experience, the answer is: sorta.    If your vAPPs are using VCDNI-backed networking, and attached to the same broadcast domain (the org network), the machines can be hit by any host in that broadcast domain (and then with the use of vShield Edge, you can ACL that).  To be clear, the default rule on a vShield Edge device is deny ingress).  If you have vAPPs in different broadcast domains, they are protected from one another (on layer 2).  One kicker, your virtual Distributed Switch must have MTU set to 1524 (if it was set to default of 1500) to allow for the larger header due to encapsulation.

    Is VCDNI good?  Yes.  Is VCDNI bad?  Probably could be argued by networking folks, since they technically do not control the allocation of networks, other than the physical VLAN VCDNI uses.  Is it the future?  Allegedly that is something else called VXLAN.  (update)My opinion:  It is a path to create private networks in a rapid fashion with minimal interaction by the network team.  It works for now, but hopefully VXLAN will be better.

    Now that we have defined methods to transport the data, we will get in to the nomenclature of vCloud.

    Network Pools can either be defined by VLAN-backed, Network isolation-backed (VCDNI) or Port group-backed.  These pools are consumed by virtual datacenters to create vAPP networks.

    Organization Networks are assigned to an Organization virtual DataCenter.  There are multiple ways to define an OrgNetwork:

    • Direct connection:  This network is akin to a traditional port group-backed network in vSphere.  In short, it provides connectivity to LAN, WAN or Internet traffic.  It is tied to an External network and usually sits on internally routable RFC-1918 address space (most likely for private cloud) or Internet-routable address space for providers.
    • NAT-routed connection:  This connection allows for Network Address Translation (NAT) of External IP space to internal private networks.  The NAT-routed OrgNet is typically in RFC-1918 address space, however there are other cases.
    • Internal Organization network: This is strictly an internal network for the vApps to communicate with each other, but have no external network access.

    External Networks are port group-backed networks (defined in vCenter) that provide ingress and egress to the Cloud environment.  They should be routable networks, either RFC-1918 for private, or Internet routable for providers.

    Reporting of baseline compliance with PowerCLI and Update Manager cmdlets

    April 11th, 2012

    The following code will generate a lovely excel spreadsheet to report compliance of ESX(i) hosts against their current baseline as applied from Update Manager.

    You will need Excel, Powershell, PowerCLI and the Update Manager cmdlets installed.  The framework is borrowed from an excellent Powershell script at

    $xlCSV = 6
    $xlXLS = 56
    $csvfile = "compliance.csv"
    $xlsfile = "compliance.xls"
    $Excel = New-Object -ComObject Excel.Application
    $Excel.visible = $True
    $Excel = $Excel.Workbooks.Add()
    $Sheet = $Excel.Worksheets.Item(1)
    $Sheet.Cells.Item(1,1) = "Server"
    $Sheet.Cells.Item(1,2) = "Release"
    $Sheet.Cells.Item(1,3) = "Version"
    $Sheet.Cells.Item(1,4) = "Build"
    $Sheet.Cells.Item(1,5) = "Baseline"
    $Sheet.Cells.Item(1,6) = "Status"
    $intRow = 2
    $WorkBook = $Sheet.UsedRange
    $WorkBook.Interior.ColorIndex = 19
    $WorkBook.Font.ColorIndex = 11
    $WorkBook.Font.Bold = $True
    $compliant = "Compliant"
    $notcompliant = "Not Compliant"
    $unknown = "Unknown Status"
    $vmhosts = get-vmhost
    foreach ($vmhost in $vmhosts) {
      $vmhostview = get-vmhost $vmhost | get-view
      $compliance = get-compliance -entity $vmhost
      $Sheet.Cells.Item($intRow, 1) = [String]$vmhost
      $Sheet.Cells.Item($intRow, 2) = $
      $Sheet.Cells.Item($intRow, 3) = $vmhostview.Config.Product.version
      $Sheet.Cells.Item($intRow, 4) = $
      $Sheet.Cells.Item($intRow, 5) = $
      if($compliance.status -eq 0) {
        $Sheet.Cells.Item($intRow, 6) = [String]$compliant
        $Sheet.Cells.Item($intRow, 6).Interior.ColorIndex = 4
        $Sheet.Cells.Item($intRow, 1).Interior.ColorIndex = 4
      elseif($compliance.status -eq 1) {
        $Sheet.Cells.Item($intRow, 6) = [String]$notcompliant
        $Sheet.Cells.Item($intRow, 6).Interior.ColorIndex = 3
        $Sheet.Cells.Item($intRow, 1).Interior.ColorIndex = 3
      else {
        $Sheet.Cells.Item($intRow, 6) = [String]$unknown
        $Sheet.Cells.Item($intRow, 6).Interior.ColorIndex = 48
        $Sheet.Cells.Item($intRow, 1).Interior.ColorIndex = 48
      $intRow = $intRow + 1
    sleep 5

    The output will show:
    Server | Release (ESX or ESXi) | Version of ESX(i) | Build Number | Attached Baseline | Status (Compliant, Not, or Unknown

    vCloud – vShield Edge Deployment Failure

    February 21st, 2012

    If you get errors during deployment of vApps in vCloud Director, specifically that vShield Edge (vse) devices can not be deployed with any of the following errors:

    The host type is not supported in vCenter, messages regarding not finding port group UUID’s that do not actually exist in the environment, or activity details such as:

    reboot your vShield Manager appliance.  I have found no VMware KB articles about the subject, but it has helped to clear any issues between vCenter, vCloud director and vShield Manager.

    Place ESXi in to Maintenance Mode from vCloud Director

    February 21st, 2012

    So you have your handy dandy cloud built on top of VMware vSphere and vCloud Director. And then you find out you need to conduct maintenance on the host.  What to do?

    Easy!  Browse to:

    • System-> Manage & Monitor
    • vSphere Resources -> Hosts
    1. Find the host you need to place in to maintenance mode, right click and select Disable Host.
    2.  At that point, the status will turn from a green circle with a check, to a red circle.
    3. Right click on the host again and select Redeploy All VMs.
    4. The ESXi host will go in to maintenance mode in the vCenter server and evacuate all virtual machines as usual.
    5. (Optional!) If you see vsla errors (such as the screenshot), issues with deleting vApps, Unprepare the host which removes the vCloud agent from ESXi
    6. (Optional!) Prepare the host for vCloud by pushing the vCloud agent to ESXi
    7. When maintenance is complete, right click and Enable Host.
    8. And your work is complete!

    PowerCLI: Get HP DL server Serial Number via vCenter and iLO

    February 17th, 2012

    Need a quick and easy way to get the serial numbers from you ESXi servers running on HP hardware?

    Fire up this code (snagged and modified from a vmware community post by RvdNieuwendijk.


    • You have an iLO configured on each server, and online
    • You have vCenter access
    # change the variable to whatever you name your iLO
    # example: -iLO or -OA etc
    get-vmhost | where-object {_.Manufacturer -eq "HP" } | \
    sort-object -Property Name | %{
     # Since your ESXi box is attached to vCenter by FQDN,
     # we split the string on "." and take the first
     # element [0] which is the server's short name
     $shortname = ($".")[0])
     $xml = new-object system.xml.xmldocument
     # add together $shortname and $ilo to get "server-ilo"
     new-object psobject -property @{
       "Name" = $shortname
       # Parse the XML and only grab the server serial number
       "SN" = $xml.RIMP.HSI.SBSN

    Quick and dirty method to mount BCV/Snapshot

    November 25th, 2011

    There are many many blog posts about mounting BCV (Business Continuity Volume) or SAN Snapshots, however here is my method.  It is a quick shell script to run on each ESXi server.  Add it to your business operations manager, and create an ad-hoc method of mounting BCV/snaps for a DR exercise.

    NOTE: Commands are in bold.

    Verify from the storage team that they have assigned the BCV/snap to your hosts

    SSH in to ESXi server (assumes you have all of the buttons pressed and knobs turned)

    Search of the BCV/snap volumes.  Do: esxcfg-volume -l  Note: Be patient, this may take a few minutes.

    The output will be as follows:

    VMFS3 UUID/label: <Datasture UUID>/<Datastore label>
    Can mount: Yes
    Can resignature: Yes
    Extent name: naa.<device ID>     range: <size in MB>

    It is now possible to mount the volume manually via the Datastore UUID or Datastore Label.   Do: esxcfg-volume -m <Datastore UUID -OR- Datastore Label>
    Note: This will conduct a force mount of the volume.
    If there are powered-on VM’s on that datastore, you can unmount it.  Do: esxcfg-volume -u <Datastore UUID -OR- Datastore Label>

    To magically wrap this up to scan for any assigned BCV/snaps and mount them automagically, run the following:

    for volume in `esxcfg-volume -l |grep VMFS3 |awk ‘BEGIN {FS=”/”} ; {print $2}’ |awk ‘{print $2}’` ; do echo “Mounting volume with UUID $volume” ; esxcfg-volume -m $volume ; done

    Creating your own HP Smart Update Firmware ISO

    November 25th, 2011

    If you have HP servers, and always wondered how to create one ISO image with all of the latest platform updates, read on!

    Note:  Commands are in bold.

    1. First step is to have a Linux machine available for use (in my case, it is a virtual machine).  Install the package “mkisofs” via the appropriate tool on your platform (yum/apt/install from source).  Also verify that you can log in to the machine via SSH
    2. Create a directory on the machine for the ISO we are going to download in step #3.  Verify you have at least 3G of storage free.  I created a directory structure off of / called /hp/iso/.  Do: mkdir -p /hp/iso/.  Also create a temporary place to mount the ISO.  Do: mkdir /tmp/iso/
    3. Next, grab the latest Smart Update Firmware DVD from the HP support site.
    4. Extract the ISO from the zip file.
    5. Copy the ISO to /hp (in my case) via your SCP tool of choice.
    6. If you are not logged in to the Linux machine, do so now.
    7. We need to mount the HP ISO via the loopback file system, to “explode” the contents of the ISO (and yes, “explode” is a technical term).  Note: The following file name will change for newer versions.  Do: mount -o loop /hp/FW930.2011_0503.09.iso /tmp/iso/
    8. Now the fun begins.  Change directory in to /tmp/iso/ and copy the contents to the direcotry you created in step #2.  Do: cd /tmp/iso/   Do: cp -R . /hp/iso/
    9. Change directory in to /hp/iso.  Do: cd /hp/iso/
    10. Umount the ISO.  Do: umount /tmp/iso/
    11. There are two important directories we will be playing with: hp/swpackages (the location of the new updates) and system/ (the location of files that make the system useable)
    12. At this point you can grab all of the appropriate updates for your servers models to your PC.  You must look for updates for Red Hat Enterprise Linux (grab the packages for RHEL6) since the HP ISO is based off of Linux.  The packages are compatible.  It doesn’t matter if you are running Windows.  I grabbed all firmware, including system BIOS.  This is important since you now have an all-in-one method of updating your servers.  The updates end in either .exe or .scexe.  You may also have .md5 checksum files
    13. Upload all of the .exe, .scexe and .md5 files to /hp/iso/hp/swpackages/
    14. Back on your Linux machine, change directory to /hp/iso/hp/swpackages.  Do: cd /hp/iso/hp/swpackages
    15. Set the executable bit on all of the .exe and .scexe files.  Do: for i in `ls -alt | grep -i cp |grep .*exe | awk ‘{print $9}’ ` ; do echo $i ; chmod ugo+x $i ; done
    16. All of your packages are now ready to be used.
    17. (Optional) If you want to customize the “Automatic or Interactive” splash screen, change in to the system directory.  Do: cd /hp/iso/system/
    18. (Optional) Make a copy of the isolinux.cfg file.  Do: cp isolinux.cfg isolinux.cfg.20111125 (I like to use YYYYMMDD format)
    19. (Optional) Edit the file with VI (If you don’t know VI, use nano/pico/whatever).
    20. (Optional) I deleted the 4 lines in the stanza that started with “label sos.”  This is for the automatic scan of the patches.
    21. (Optional) I edited the stanza for “label vsos” and changed the “MENU LABEL” line.  Place whatever identifier you would like _after_ MENU LABEL.  Save the file
    22. Now we are going to create the updated ISO.  Figure out what you want to name the file.  I will call mine 9.30-20111125.iso.  Do: mkisofs -o /hp/9.30-20111125.iso -N -J -joliet-long -b system/isolinux.bin -c system/ -no-emul-boot -boot-load-size 4 -boot-info-table /hp/iso/
    23. After a minute or two, your brand spanking newly spun ISO will be waiting for you in the /hp/ directory.  Copy that to your workstation, attach it to an iLO.
    24. Boot off of the ISO.  If you updated the isolinux.cfg file, select the label you assigned at step #21.  If you did not, select Interactive.
    25. After the hardware discovery is complete, you will need to select ALLOW NON-BUNDLE PRODUCTS and ALLOW NON-BUNDLE VERSIONS to use the new updates.
    26. Impress your boss and ask for a raise.

    VMworld session – VSP3305 — Upgrading to VMware ESXi 5.0

    August 29th, 2011

    Kyle Gleed presented a session Monday morning to cover tips hints and gotchas for the migration to ESXi 5.0.   5.0 will be the first version of vSphere to only ship with the stripped down ESXi.  No more service console.  At work we conducted the migration from ESX 3.5 to ESXi 4.1 in preparation of the release of 5.0.

    Kyle stressed that script, or command line management of ESXi will be conducted via the esxcli command set, vicfg and PowerCLI.  esxcfg commands have been deprecated.  It is important to note that esxcli commands can be run either local to the ESXi instance, or via the remote CLI suite.

    If you are planning on upgrading your ESXi instances to 5.0 (not including a full install), you must be running ESX 4.x.  NOTE: You can upgrade from ESX 4.x to ESXi 5.0 easily!  You can not upgrade from ESX 3.5 directly to 5.0.

    One important thing to note is that you must have 5G allocated for your boot device, be it SAN or local disk.  The ESXi installer partitions off the first 1G for the OS, and the next 4G for scratch space.

    The usual upgrade path of vCenter to 5.0, followed by ESXi to 5.0, and then virtual machine tool and/or virtual hardware, are steps that were taken from 3.5 to 4.x.  ESXi 5.0 now introduces VMFS-5, which according to VMware is a hot upgrade that does not affect running virtual machines.