XenServer

Troubleshoot Workload Balancing

While Workload Balancing usually runs smoothly, this series of sections provides guidance in case you encounter issues.

Notes:

  • Workload Balancing is available for XenServer Premium Edition customers. For more information about XenServer licensing, see Licensing. To upgrade, or to buy a XenServer license, visit the XenServer website.
  • Workload Balancing 8.2.2 and later are compatible with XenServer 8 and Citrix Hypervisor 8.2 Cumulative Update 1.

Determine the status of the Workload Balancing virtual appliance

Run the systemctl status workloadbalancing command. For more information, see Workload Balancing commands.

General troubleshooting tips

  • Start troubleshooting by reviewing the Workload Balancing log files (LogFile.log and wlb_install_log.log). You can find these logs in Workload Balancing virtual appliance in this location (by default):

    /var/log/wlb

    The level of detail in these log files can be configured by using the wlb.conf file. For more information, see Increase the detail in the Workload Balancing log.

  • Check the logs in the XenCenter Logs tab for further information.

  • To check the Workload Balancing virtual appliance build number, run the following command on a host in a pool that the virtual appliance monitors:

     xe pool-retrieve-wlb-diagnostics | more
     <!--NeedCopy-->
    

    The Workload Balancing version number appears at the top of the output.

  • The Workload Balancing virtual appliance is based on the CentOS operating system. If you experience CPU, memory, or disk related issues in the virtual appliance, you can use the standard Linux logs in /var/log/* to analyse the issue.

  • Use standard Linux debugging and performance tuning commands to understand the virtual appliance behavior. For example, top, ps, free, sar, and netstat.

Error messages

Workload Balancing displays errors on screen as dialog boxes and as error messages in the Logs tab in XenCenter.

If an error message appears, review the XenCenter event log for additional information. For more information, see the XenCenter product documentation.

Issues entering Workload Balancing credentials

If you cannot successfully enter the virtual appliance user account and password while configuring the Connect to WLB Server dialog, try the following:

  • Ensure that Workload Balancing virtual appliance imported and was configured correctly and all of its services are running.

  • Check to ensure that you are entering the correct credentials. The Connect to WLB Server dialog asks for two different credentials:

    • WLB Server Credentials: XenServer uses this account to communicate with Workload Balancing. You created this account on the Workload Balancing virtual appliance during Workload Balancing Configuration. By default, the user name for this account is wlbuser.
    • Citrix Hypervisor Credentials: This account is used by the Workload Balancing virtual appliance to connect to the XenServer pool. This account is created on the XenServer pool coordinator and has the pool-admin or pool-operator role.
  • You can enter a host name in the Address box, but it must be the fully qualified domain name (FQDN) of the Workload Balancing virtual appliance. Do not enter the host name of the physical server hosting the appliance. If you are having trouble entering a computer name, try using the Workload Balancing appliance’s IP address instead.

  • Verify that the host is using the correct DNS server and the XenServer host can contact Workload Balancing server using its FQDN. To do this check, ping the Workload Balancing appliance using its FQDN from the XenServer host. For example, enter the following in the XenServer host console:

     ping wlb-vpx-1.mydomain.net
     <!--NeedCopy-->
    

Issues with firewalls

The following error appears if the Workload Balancing virtual appliance is behind a hardware firewall, and you did not configure the appropriate firewall settings: “There was an error connecting to the Workload Balancing server: <pool name> Click Initialize WLB to reinitialize the connection settings.” This error might also appear if the Workload Balancing appliance is otherwise unreachable.

If the Workload Balancing virtual appliance is behind a firewall, open port 8012.

Likewise, the port XenServer uses to contact Workload Balancing (8012 by default), must match the port number specified when you ran the Workload Balancing Configuration wizard.

Workload Balancing connection errors

If you receive a connection error after configuring and connecting to Workload Balancing, the credentials might no longer be valid. To isolate this issue:

  1. Verify that the credentials you entered in the Connect to WLB Server dialog box are correct. For more information, see scenario 1 and 2.

  2. Verify that the IP address or FQDN for the Workload Balancing virtual appliance that you entered in the Connect to WLB Server dialog box is correct.

  3. Verify that the user name you created during Workload Balancing configuration matches the credentials you entered in the Connect to WLB Server dialog box.

  4. If you receive a connection error in the Workload Balancing Status line on the WLB tab, you might need to reconfigure Workload Balancing on that pool. Click the Connect button on the WLB tab and reenter the host credentials.

You may encounter one of the following scenarios when attempting to establish a connection from XenCenter to the Workload Balancing virtual appliance.

Scenario 1

Scenario 1 - Error: WLB could not log into XenServer. It could be due to invalid credentials. Check your settings and try again.

This means that the credentials entered in the Citrix Hypervisor Credentials field in the Connect to WLB Server dialog box are incorrect. To fix this, double-check the credentials or check the Use the current XenCenter credentials box.

Scenario 2

Scenario 2 - Error: WLB rejected our configured authentication details. Check your settings and try again.

This means that there is a problem with the credentials entered in the WLB Server Credentials field in the Connect to WLB Server dialog box when attempting to connect to the Workload Balancing virtual appliance (either the username or the password are incorrect). However, it can also mean that the Workload Balancing service is not running or that there is a problem with the database configuration file.

To fix credential issues, make sure that you are using the correct username and password. The default username for WLB Server Credentials field is wlbuser (not root). Root is the default administrator username. Note that wlbuser is not an actual user with logon privileges in the appliance (it does not exist under /etc/passwd) and thus these credentials are only used to connect to Workload Balancing itself. As such, they can be easily reset by running the wlbconfig command. To change your credentials, see Change the Workload Balancing credentials. To run the wlbconfig command, you must be able to log into the appliance as root. If the root password is unknown, it can be reset using the regular CentOS/RHEL password recovery procedure.

If you have reset your credentials but the error still persists:

  1. Check if the Workload Balancing process is running by using the systemctl status workloadbalancing command.
  2. Make sure the wlb.conf file exists and is in the right directory by running this command: cat /opt/vpx/wlb/wlb.conf

Scenario 3

Scenario 3 - Error: The configured WLB server name failed to resolve in DNS. Check your settings and try again.

This indicates that there is an issue connecting to the port specified under the Server Address options when connecting to Workload Balancing from XenCenter (either the incorrect port was entered or the port is not listening). To troubleshoot this:

  1. Make sure the target appliance is up and running.
  2. Double-check the port entered on the Workload Balancing connection details window (default is 8012).
  3. Make sure this port is enabled in the appliance and listening. Use commands like telnet <port> or iptables -L to help determine if the port is listening or if traffic is being denied on this port.

Scenario 4

Scenario 4 - Error: WLB refused a connection to the server. Check your settings and try again.

This error occurs when there is a problem with stunnel (either it’s not running or the certificate/key pair is incorrect). To troubleshoot this, first verify the certificate and key:

  1. Confirm the certificate has not expired by running the following command:

    openssl x509 -dates -in $(grep cert\ = /etc/stunnel/stunnel.conf |cut -d '=' -f2) -noout
    <!--NeedCopy-->
    
  2. Compare the hex on the output of the following 2 commands. If the output does not match then the wrong key is being used.

    openssl x509 -modulus -in $(grep cert\ = /etc/stunnel/stunnel.conf |cut -d '=' -f2) -noout | openssl md5
    <!--NeedCopy-->
    

    and

    ​openssl rsa -modulus -in $(grep key\ = /etc/stunnel/stunnel.conf | cut -d '=' -f2) -noout | openssl md5
    <!--NeedCopy-->
    

​​If there are no problems with the certificate and key, make sure stunnel is running and is bound to port 8012 (or the configured port):

  1. Run the following command in the WLB appliance CLI:

    netstat -tulpn
    <!--NeedCopy-->
    

    On the output, 8012 (or the custom port) should show status: LISTEN.

  2. If the appliance ran out of space, stunnel won’t run. Use commands like df -h or du -hs /* to see whether you have enough space available on your appliance. To increase the disk space, see Extend the virtual appliance disk.

Scenario 5

Scenario 5 - Error: The connection to the WLB server was reset. Check your settings and try again.

This error can occur because the stunnel process was terminated. If restarting the process yields the same results, restart the Workload Balancing virtual appliance.

Any other errors

If you encounter any other errors when attempting to connect to Workload Balancing or need further assistance performing the steps above, collect the Workload Balancing logs which can be found under the /var/log/wlb directory in the Workload Balancing appliance.

Contact Support for further assistance.

Workload Balancing stops working

If Workload Balancing doesn’t work (for example, it doesn’t let you save changes to settings), check the Workload Balancing log file for the following error message:

dwmdatacolsvc.exe: Don't have a valid pool. Trying again in 10 minutes.
<!--NeedCopy-->

This error typically occurs in pools that have one or more problematic VMs. When VMs are problematic, you might see the following behavior:

  • Windows. The Windows VM crashes due to a stop error (“blue screen”).
  • Linux. The Linux VM might be unresponsive in the console and typically does not shut down.

To work around this issue:

  1. Force the VM to shut down. To do so, you can do one of the following on the host with the problematic VM:

    • In XenCenter, select the VM, and then from the VM menu, click Force Shutdown.
    • Run the vm-shutdown xe command with the force parameter set to true. For example:

       xe vm-shutdown  force=true  uuid=vm_uuid
       <!--NeedCopy-->
      

      You can find the host UUID on the General tab for that host (in XenCenter) or by running the host-list xe command. You can find the VM UUID in the General tab for the VM or by running the vm-list xe command. For more information, see Command line interface.

  2. In the xsconsole of the XenServer serving the crashed VM or in XenCenter, migrate all VMs to another host, then run the xe-toolstack-restart command. (Do not restart the toolstack while HA is enabled. If possible, temporarily disable HA before restarting the toolstack.)

Issues changing Workload Balancing servers

If you connect a pool to a different Workload Balancing server without disconnecting from Workload Balancing, both old and new Workload Balancing servers monitor the pool.

To solve this problem, you can take one of the following actions:

  • Shut down and delete the old Workload Balancing virtual appliance.
  • Manually stop the Workload Balancing services. These services are analysis, data collector, and Web service.

Note:

Do not use the pool-deconfigure-wlb xe command to disconnect a pool from the Workload Balancing virtual appliance or use the pool-initialize-wlb xe command to specify a different appliance.