Configure Log Insight Forwarder in Enterprise Hybrid Cloud

As part of our Enterprise Hybrid Cloud, we deploy a Log Insight instance to gather the logs from the various components of the solution. Back in the days of EHC 3.5 and older, we used to have a single Log Insight appliance or a cluster, and all the syslog servers were pointed to that. Since EHC 4.0, that design has changed. Now we utilize a separate Log Insight Forwarder instance to collect and forward some of the logs. The reason behind this change is the ability of EHC 4.0 and newer to connect several remote sites (or vCenters) to one main instance of EHC. We want to collect logs from the remote sites as well, but it’s not efficient from networking perspective to collect the logs straight from the components over WAN to the main Log Insight cluster. Log Insight has a nifty built-in feature called Event Forwarding, that can push the local logs to a central location. It’s designed to work over WAN, so it can optimize the network usage and also can encrypt the traffic between sites. Pretty cool! There are plenty of other reasons to use forwarding as well.

LI_Architecture_v3

Getting the Forwarder up and running is a simple process, but it’s not that well documented in the context of an existing Log Insight cluster. The information can be found in VMware documentation, but they don’t really specify the design. First things first, the Log Insight Forwarder is a separate installation of Log Insight. Unlike vRealize Operations, you cannot deploy a “remote collector” instance of Log Insight and add that to the existing cluster. Instead, you have to do a full install of Log Insight. It can be a cluster as well, but since we use it to simply collect and push logs to central location, a single node installation is fine for our purposes. Follow the normal process of deploying the Log Insight OVA, configuring the network and launching the installation UI. Choose “New Deployment” and configure Log Insight just like you did for the main cluster.

In order to get the encrypted connection (not mandatory) to work between the Forwarder and main LI cluster, there needs to be a trust established between the two installations. To make this happen, you will need custom CA-signed certificate on the main cluster, but it should already be there for the cluster to work properly. Using self-signed is not supported when it comes to the distributed components of EHC. For the connection to work, you need to add the root certificate chain of the main Log Insight Cluster to the Forwarder keystore. Official doc for additional information.

  • Copy the trusted root certificate chain with scp or Filezilla into a temporary directory on the Forwarder instance. For example: /home
  • SSH to the forwarder instance and run the following command localhost:
     /usr/java/jre1.8.0_92/bin/keytool -import -alias loginsight -file /home/Root64.cer -keystore cacerts

    The default keystore password is changeit.
    Note: Java versions might vary with time.

  • Restart the vRealize Log Insight Forwarder instance

After the Forwarder instance is up and running, the final step is to add Event Forwarding between the Forwarder and the Cluster. Follow the docs for additional information. Navigate to Administration interface of the Log Insight Forwarder and select Event Forwarding on the left pane. Choose New Destination, fill out the Log Insight Cluster FQDN, check the Use SSL box, make sure you are using Ingestion API and press Test. You can leave the other options to default. Click Save.

LIEventForwarding_test_success

I came across a weird bug with the connection test and SSL. I had a clean Log Insight instance without anything logging to it. I configured all the steps above, and hit Test. It came back with an error “Failed connection with {LI_FQDN}:9543″. Without SSL the connection test was ok. I double checked everything and the certificates seemed fine. I tried an SSL connection by forcing an Log Insight Agent to do an SSL connection with the same root certificate chain with the appliance. This was successful, so the error seemed quite odd. I came back and hit the Test, and it was successful! It seems that if the Log Insight appliance doesn’t have any logs to forward, the Test might fail. It’s also possible that this is a certificate related issue, but I haven’t got to the bottom of it yet.

The last step is to configure the necessary agents and collect information from the local components. In the case of EHC, we divide the components according to the cluster where they are deployed. The Forwarder instance is located in the AMP or Core cluster, so we will use that instance for all the AMP component log collection. This way we can deploy additional sites with the same exact Log Insight setup on all of them.

For EHC, here’s a list of components and the associated Log Insight instance:

Forwarder:

  • VMware vSphere/vCenter
  • VMware Site Recovery Manager
  • VMware ESXi Servers from all the clusters within the site
  • VMware NSX Manager
  • VMware NSX Edges
  • VMware NSX Controllers
  • VMware NSX Distributed Logical Routers
  • VMware vRealize Operations Manager Remote Collector
  • Dell EMC Storage
  • Dell EMC RecoverPoint
  • Dell EMC RecoverPoint for Virtual Machines
  • Dell EMC Avamar
  • Dell EMC SMI-S
  • Core Microsoft SQL Server
  • Core VMware Platform Services Controller 1 & 2
  • VMware vRealize Automation Agents
  • Microsoft Active Directory (if applicable)
  • Cisco UCS

Main Cluster:

  • VMware vRealize Automation (all the components except for Agents)
  • VMware vRealize Orchestrator
  • VMware vRealize Operations Manager (all components except for Remote Collectors)
  • VMware vRealize Business for Cloud
  • Automation Pod Microsoft SQL Server
  • Dell EMC Data Protection Advisor
  • Dell EMC ViPR
  • Automation Pod VMware Platform Services Controller

Done. Time for some serious log inspection!

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s