Getting ready to upgrade vRA from 6.x to 7.x? If your system contains imported VMs, pay attention.
There’s a known issue with imported VMs when doing an in-place upgrade. VMware has a KB 2150515 on the issue, but you have to do the fix before you upgrade. As it happens, we noticed the KB after the upgrade was done and things were in a bad state. If you can, use the KB. If it is too late, read on.
The symptoms are clear. All the old imported VMs disappear from the Items list. When you see the details of these VMs in Managed Machines, you’ll notice that the machines are in Unmanaged state and the tagged blueprint is “system_blueprint_vsphere”. This is a standard system blueprint and it does not exist in your Design tab. There is a way to fix the situation. There will be some hurdles, but the procedure is clear. First you need to unregister the affected VMs with CloudClient or directly removing them from vRA databases, and then reimport them using Bulk Import. Very simple, but in practice things can go wrong in many ways.
You should start the fix by exporting all the Managed Machines using the Generate CSV File from Bulk Imports functionality. Remember to select the “Include custom properties. This step is crucial to get the VMs back in working order under vRA. Next you need a list of the VMs with the wrong blueprint attached. Unfortunately the bulk export list doesn’t contain the blueprint information, so you will have to get that either with CloudClient or from Managed Machine list in vRA. The list can be exported, but in my experience it is very unreliable. Some items can be missing, some are double, etc.
Compare the two lists and make a final list with just the affected VMs. When done, you’ll need to add and remove some stuff to avoid surprises. First of all, add the proper Deployment name, Blueprint ID and Component Blueprint ID. Deployment name you can decide freely, the Blueprint ID you can find by opening your import blueprint, and Component Blueprint ID is the name of the actual machine inside the blueprint.
Next, you will need to add a custom property for VM OS to each of the imported VM in the csv file as per the vRA Release Notes. Even if you go with vRA 7.3, this needs to be done. Here’s an example, just add the items to the end of the line. You need to have a valid entry for the OS, but funnily enough it actually does not have to match with the real VM OS. The import will fail if the tag is not a valid OS, but that’s all it checks.
You can get a list of suitable OS codes from here. If you don’t have this line in the csv for each machine, the import will fail.
Next thing to modify is the machine lifecycles, or “stubs”. A machine goes through different lifecycles and states when it is being provisioned, imported or deleted, for example. The documentation will walk you through the steps, but for Bulk Import we are interested in the RegisterMachine and MachineProvisioned stubs. Depending on your setup, these stubs have been called when the original VM was imported, so you might need to remove them from the csv file when you import the machines back. For example, in our case the MachineProvisioned called on a backup functionality and added the VM to a DRS group. While getting an extra backup is ok, it’s better not to call the stubs unless they are needed. You should leave other stubs alone such as Disposing, since they are needed when the VM is eventually deleted from vRA.
Same goes with the new Event Broker functionality. Again, depending on your setup, it might be wise to disable all post provisioning tasks that you might have to keep the machines as original as possible. Preprovisioning tasks won’t be called when importing.
Now it’s time to get rid of those VMs in vRA. CloudClient does a good job of doing this, but don’t run too many removals at the same time! We had some issues with “cloudclient forceunregister” command. We looped through over 400 VMs, and while the commands were issued successfully, the unregister operation failed for about 300 VMs. We didn’t spend too much time figuring out why, but it seemed that there was some sort of timeout. We had to clean the unregister mess using the database scripts from VMware KB 2144269. Do them in batches is the lesson here. As for the Bulk Import operation, this seemed to work reliably with 250 VMs at one go without a hiccup.
There were some other smaller issues. Make sure the owner of the imported VM has access to the business group and also that the import blueprint is assigned to at least one entitlement. These will be checked during import. Also vRealize Business can cause problems. In our case vRB Cost Collection failed because vRB found multiple entries for the same VM in vRA database. vRA doesn’t seem to delete anything from the database, it just creates a new ID for a new object. If you delete a VM with CloudClient, the VM is still in vRA DB with DELETED status. vRB picks this up and creates an error. The problem should be fixed in vRB 7.2.1 and 7.3, but we did get this with 7.2.1 as well. Contact VMware support for a fix in case you need it.
Another “fun” problem is ASCII coding. When you export and import a csv file, vRA does some ASCII modification magic and basically fails to recode all the special characters properly. Check your import csv file before you import everything back to vRA, you might need to manually replace some “%40” characters (and plenty of others) with the correct ones.
Before running the Bulk Import, export another list of Unmanaged machines to a csv file. It’s a good idea to compare the Unmanaged list and the list you prepared for Bulk Import. This is a good way to verify all machines were removed from vRA and that your import list is accurate.
Last step is to run the Bulk Import. I would recommend to select the “Ignore managed machines” option to make sure you don’t overrun any existing machines. After the import is done, all the machines should be visible again in the Items tab and ready to roll!