Troubleshoot Workload Balancing
While Workload Balancing usually runs smoothly, this series of sections provides guidance in case you encounter issues.
Notes:
- Workload Balancing is available for XenServer Premium Edition customers. For more information about XenServer licensing, see Licensing. To upgrade, or to get a XenServer license, visit the XenServer website.
- Workload Balancing 8.3.0 and later are compatible with XenServer 8 and Citrix Hypervisor 8.2 Cumulative Update 1.
Determine the status of the Workload Balancing virtual appliance
Run the systemctl status workloadbalancing command. For more information, see Workload Balancing commands.
General troubleshooting tips
-
Start troubleshooting by reviewing the Workload Balancing log files (
LogFile.logandwlb_install_log.log). You can find these logs in Workload Balancing virtual appliance in this location (by default):/var/log/wlbThe level of detail in these log files can be configured by using the
wlb.conffile. For more information, see Increase the detail in the Workload Balancing log. -
Check the logs in the XenCenter Logs tab for further information.
-
To check the Workload Balancing virtual appliance build number, run the following command on a host in a pool that the virtual appliance monitors:
xe pool-retrieve-wlb-diagnostics | more <!--NeedCopy-->The Workload Balancing version number appears at the top of the output.
-
The Workload Balancing virtual appliance is based on the CentOS operating system. If you experience CPU, memory, or disk related issues in the virtual appliance, you can use the standard Linux logs in
/var/log/*to analyse the issue. -
Use standard Linux debugging and performance tuning commands to understand the virtual appliance behavior. For example,
top,ps,free,sar, andnetstat.
Error messages
Workload Balancing displays errors on screen as dialog boxes and as error messages in the Logs tab in XenCenter.
If an error message appears, review the XenCenter event log for additional information. For more information, see the XenCenter product documentation.
Issues entering Workload Balancing credentials
If you cannot successfully enter the virtual appliance user account and password while configuring the Connect to WLB Server dialog, try the following:
-
Ensure that Workload Balancing virtual appliance imported and was configured correctly and all of its services are running.
-
Check to ensure that you are entering the correct credentials. The Connect to WLB Server dialog asks for two different credentials:
-
WLB Server Credentials: XenServer uses this account to communicate with Workload Balancing. You created this account on the Workload Balancing virtual appliance during Workload Balancing Configuration. By default, the user name for this account is
wlbuser. -
Citrix Hypervisor Credentials: This account is used by the Workload Balancing virtual appliance to connect to the XenServer pool. This account is created on the XenServer pool coordinator and has the
pool-adminorpool-operatorrole.
-
WLB Server Credentials: XenServer uses this account to communicate with Workload Balancing. You created this account on the Workload Balancing virtual appliance during Workload Balancing Configuration. By default, the user name for this account is
-
You can enter a host name in the Address box, but it must be the fully qualified domain name (FQDN) of the Workload Balancing virtual appliance. Do not enter the host name of the physical server hosting the appliance. If you do not have a DNS entry for the FQDN, you must connect using the Workload Balancing appliance’s IP address instead.
-
Verify that the host is using the correct DNS server and the XenServer host can contact Workload Balancing server using its FQDN. To do this check, ping the Workload Balancing appliance using its FQDN from the XenServer host. For example, enter the following in the XenServer host console:
ping wlb-vpx-1.mydomain.net <!--NeedCopy-->
Issues with firewalls
The following error appears if the Workload Balancing virtual appliance is behind a hardware firewall, and you did not configure the appropriate firewall settings: “There was an error connecting to the Workload Balancing server: <pool name> Click Initialize WLB to reinitialize the connection settings.” This error might also appear if the Workload Balancing appliance is otherwise unreachable.
If the Workload Balancing virtual appliance is behind a firewall, open port 8012.
Likewise, the port XenServer uses to contact Workload Balancing (8012 by default), must match the port number specified when you ran the Workload Balancing Configuration wizard.
Workload Balancing connection errors
If you receive a connection error after configuring and connecting to Workload Balancing, the credentials might no longer be valid. To isolate this issue:
-
Verify that the credentials you entered in the Connect to WLB Server dialog box are correct. For more information, see scenario 1 and 2.
-
Verify that the IP address or FQDN for the Workload Balancing virtual appliance that you entered in the Connect to WLB Server dialog box is correct.
-
Verify that the user name you created during Workload Balancing configuration matches the credentials you entered in the Connect to WLB Server dialog box.
-
If you receive a connection error in the Workload Balancing Status line on the WLB tab, you might need to reconfigure Workload Balancing on that pool. Click the Connect button on the WLB tab and reenter the host credentials.
You may encounter one of the following scenarios when attempting to establish a connection from XenCenter to the Workload Balancing virtual appliance.
Scenario 1
This means that the credentials entered in the Citrix Hypervisor Credentials field in the Connect to WLB Server dialog box are incorrect. To fix this, double-check the credentials or check the Use the current XenCenter credentials box.
Scenario 2
This means that there is a problem with the credentials entered in the WLB Server Credentials field in the Connect to WLB Server dialog box when attempting to connect to the Workload Balancing virtual appliance (either the username or the password are incorrect). However, it can also mean that the Workload Balancing service is not running or that there is a problem with the database configuration file.
To fix credential issues, make sure that you are using the correct username and password. The default username for WLB Server Credentials field is wlbuser (not root). Root is the default administrator username. Note that wlbuser is not an actual user with logon privileges in the appliance (it does not exist under /etc/passwd) and thus these credentials are only used to connect to Workload Balancing itself. As such, they can be easily reset by running the wlbconfig command. To change your credentials, see Change the Workload Balancing credentials. To run the wlbconfig command, you must be able to log into the appliance as root. If the root password is unknown, it can be reset using the regular CentOS/RHEL password recovery procedure.
If you have reset your credentials but the error still persists:
- Check if the Workload Balancing process is running by using the
systemctl status workloadbalancingcommand. - Make sure the
wlb.conffile exists and is in the right directory by running this command:cat /opt/vpx/wlb/wlb.conf
Scenario 3
This indicates that there is an issue connecting to the port specified under the Server Address options when connecting to Workload Balancing from XenCenter (either the incorrect port was entered or the port is not listening). To troubleshoot this:
- Make sure the target appliance is up and running.
- Double-check the port entered on the Workload Balancing connection details window (default is 8012).
- Make sure this port is enabled in the appliance and listening. Use commands like
telnet <port>oriptables -Lto help determine if the port is listening or if traffic is being denied on this port.
Scenario 4
This error occurs when there is a problem with stunnel (either it’s not running or the certificate/key pair is incorrect). To troubleshoot this, first verify the certificate and key:
-
Confirm the certificate has not expired by running the following command:
openssl x509 -dates -in $(grep cert\ = /etc/stunnel/stunnel.conf |cut -d '=' -f2) -noout <!--NeedCopy--> -
Compare the hex on the output of the following 2 commands. If the output does not match then the wrong key is being used.
openssl x509 -modulus -in $(grep cert\ = /etc/stunnel/stunnel.conf |cut -d '=' -f2) -noout | openssl md5 <!--NeedCopy-->and
openssl rsa -modulus -in $(grep key\ = /etc/stunnel/stunnel.conf | cut -d '=' -f2) -noout | openssl md5 <!--NeedCopy-->
If there are no problems with the certificate and key, make sure stunnel is running and is bound to port 8012 (or the configured port):
-
Run the following command in the WLB appliance CLI:
netstat -tulpn <!--NeedCopy-->On the output, 8012 (or the custom port) should show
status: LISTEN. -
If the appliance ran out of space, stunnel won’t run. Use commands like
df -hordu -hs /*to see whether you have enough space available on your appliance. To increase the disk space, see Extend the virtual appliance disk.
Scenario 5
This error can occur because the stunnel process was terminated. If restarting the process yields the same results, restart the Workload Balancing virtual appliance.
Any other errors
If you encounter any other errors when attempting to connect to Workload Balancing or need further assistance performing the steps above, collect the Workload Balancing logs which can be found under the /var/log/wlb directory in the Workload Balancing appliance.
Contact Support for further assistance.
Workload Balancing stops working
If Workload Balancing doesn’t work (for example, it doesn’t let you save changes to settings), check the Workload Balancing log file for the following error message:
dwmdatacolsvc.exe: Don't have a valid pool. Trying again in 10 minutes.
<!--NeedCopy-->
This error typically occurs in pools that have one or more problematic VMs. When VMs are problematic, you might see the following behavior:
- Windows. The Windows VM crashes due to a stop error (“blue screen”).
- Linux. The Linux VM might be unresponsive in the console and typically does not shut down.
To work around this issue:
-
Force the VM to shut down. To do so, you can do one of the following on the host with the problematic VM:
- In XenCenter, select the VM, and then from the VM menu, click Force Shutdown.
-
Run the
vm-shutdownxe command with the force parameter set totrue. For example:xe vm-shutdown force=true uuid=vm_uuid <!--NeedCopy-->You can find the host UUID on the General tab for that host (in XenCenter) or by running the
host-listxe command. You can find the VM UUID in the General tab for the VM or by running thevm-listxe command. For more information, see Command line interface.
-
In the
xsconsoleof the XenServer serving the crashed VM or in XenCenter, migrate all VMs to another host, then run thexe-toolstack-restartcommand. (Do not restart the toolstack while HA is enabled. If possible, temporarily disable HA before restarting the toolstack.)
Issues changing Workload Balancing servers
If you connect a pool to a different Workload Balancing server without disconnecting from Workload Balancing, both old and new Workload Balancing servers monitor the pool.
To solve this problem, you can take one of the following actions:
- Shut down and delete the old Workload Balancing virtual appliance.
- Manually stop the Workload Balancing services. These services are analysis, data collector, and Web service.
Note:
Do not use the
pool-deconfigure-wlbxe command to disconnect a pool from the Workload Balancing virtual appliance or use thepool-initialize-wlbxe command to specify a different appliance.