Add or Upgrade Plugins in vCenter Orchestrator Cluster Mode

Configuring vCenter / vRealize Orchestrator in a cluster mode can be tricky. There are several sources of information how to do that, including the official VMware documentation (vCenter Orchestrator 5.5.2 Documentation), so it’s not that big of a problem. Upgrading the plugins in cluster mode, however, can be challenging. There’s a procedure you have to follow if you don’t want to end up in a situation where plugins keep disappearing for no good reason.

The documentation from VMware really doesn’t cover this use case. In a normal operation, you probably have to install new plugins and upgrade the old ones from time to time. If you’ve worked with a single server install of the vCO previously, this was a simple process of uploading and installing the plugin. If you do that with a vCO in a cluster mode, it will go horribly wrong. During a customer implementation, we came up with a procedure that works and keeps the vCO operational. There is a short downtime, because the servers need to be rebooted and kept shutdown for a small period of time. The whole thing takes about 1 hour to complete, but it can be done much faster if there’s only one plugin to implement.

If you can tolerate the whole hour of downtime, I would suggest to disable the load balancer for the duration of the upgrade. If not, it can be partially up during, but obviously there’s a risk that workflows get suspended during server reboots. We kept it up but no one was allowed to use the portal.

The key is to keep only one of the vCO nodes active at a time. Basically you shutdown vco2, upgrade/install vco1, shutdown vco1 and upgrade/install vco2. Follow this procedure to guarantee a successful install:

  1. Snapshot the vCO VMs
  2. Disable the load balancer leg for vCO node vco2
  3. Shutdown vCO node vco2
  4. Open vCO configuration page for vCO node vco1 and install the new/upgraded plugin (reboot if needed)
  5. Shutdown vCO node vco1
  6. Disable load balancer leg for vco1
  7. Restart vCO node vco2
  8. Enable load balancer leg for vco2
  9. Open vCO configuration page for vCO node vco2 and install the new/upgraded plugin (reboot if needed)
  10. Shutdown vCO node vco2
  11. Disable load balancer leg for vco2
  12. Restart vCO node vco1
  13. Enable load balancer leg for vco1
  14. Verify that the new plugin was installed correctly
  15. Restart vCO node vco2
  16. Enable load balancer for leg vco2
  17. Verify that the cluster has both nodes in RUNNING state (Server Availability tab)

If you already managed to destroy the cluster by trying to install a plugin while both of the nodes were up (it happens 😉 ), recovering is not too difficult. You might want to snapshot the VMs before doing anything. Here’s how we did it:

  1. Snapshot the vCO VMs
  2. Disable the load balancer for both vCO nodes
  3. Open vCO configuration page for vCO node vco2 and Disable Cluster Mode and return to Single Server mode
  4. Shutdown vCO node vco2
  5. Open vCO configuration page for vCO node vco1 and Disable Cluster Mode and return to Single Server mode
  6. Install/fix the necessary plugins on vco1
  7. Shutdown vCO node vco1
  8. Restart vCO node vco2
  9. Open vCO configuration page for vCO node vco2 and install/fix the necessary plugins
  10. Restart vCO node vco1
  11. Enable Cluster Mode on vco1 in Server Availability tab and specify 2 active nodes
  12. Export vCO config on vco1. Go to General tab and select Export
  13. Download the config file from vco1 appliance to your desktop
  14. Open vCO configuration page for vCO node vco2, go to General tab and import the config file. Before applying, deselect the check box, we don’t want to modify network settings!
  15. Go to Server Availability tab, and verify that both nodes are visible and RUNNING. We had to restart vco1 for this to happen
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s