Troubleshoot Changed Block Tracking

This article includes some common error scenarios you might encounter while enabling and using changed block tracking.

Common error scenarios

You can’t enable changed block tracking on a VDI

  • Ensure that the VDI is not a snapshot or a raw VDI. You can’t enable changed block tracking on snapshots or raw VDIs.

You can’t data destroy a VDI

  • Ensure that the VDI you are referring to is a snapshot.
  • Ensure that changed block tracking is enabled. Otherwise, you might see the VDI_NO_CBT_METADATA error.

    • In XenCenter, check the Storage tab to see if changed block tracking is enabled for the VDI you took a snapshot of.
    • Or, use the xe CLI to check the cbt_enabled field of the VDI snapshot: xe vdi-param-list uuid=<snapshot_uuid>
  • Check whether a <snapshot_uuid>.cbtlog file exists in the storage repository:

    • For LVM-based storage repositories, run the command lvs to check whether a <snapshot_uuid>.cbtlog file exists in the storage repository.
    • For file-based storage repositories, look for the files in the location /run/sr-mount/<sr-uuid>/<snapshot-uuid>.cbtlog.

You can’t list changed blocks between two VDI snapshots

  • Ensure that the VDIs are snapshots. If the vdi_to VDI is not a snapshot, ensure it is not attached.
  • Ensure that both VDI snapshots have changed block tracking enabled.
  • Ensure that the VDI snapshots are related and in the right order.

    You can use the cbt-util utility, which helps establish chain relationship. If the VDI snapshots are not linked by changed block metadata, you get errors like “SR_BACKEND_FAILURE_460”, “Failed to calculate changed blocks for given VDIs”, and “Source and target VDI are unrelated”.

    Example usage of cbt-util:

     cbt-util get –c –n <name of cbt log file>
    

    The -c option prints the child log file UUID.

Notable error conditions

Changed block tracking is disabled when certain errors are encountered, for example:

  • The changed block tracking log is found to be inconsistent at the time of attaching a VDI. This situation can occur when a XenServer host or SR crashes.
  • The resize of a changed block tracking log file was unsuccessful on the source VDI resize.
  • Insufficient space remains on disk to create a changed block tracking log file when changed block tracking is activated or a snapshot is created.

However, the primary action (for example, attach, resize, or snapshot) does succeed in those error conditions and a log message is logged in SMlog. Also, an alert is generated in XenCenter to notify you that changed block tracking is disabled.

Troubleshooting the NBD server

XenServer acts as network block device (NBD) server and makes VDI snapshots available over NBD connections. For more information, see Enabling NBD connections on XenServer.

Notes:

  • The NBD server logs are in /var/log/daemon.log.
  • NBD connections work with all network configurations, including VLAN, bond network, and VLAN on bond.
  • NBD connections don’t work when an NBD client is in dom0 on the same host as the NBD server.
  • Networks associated with a XenServer pool that have NBD connections enabled must either all have the purpose nbd or all have the purpose insecure_nbd. You cannot have a mix of normal NBD networks (FORCEDTLS) and insecure NBD networks (NOTLS).
  • The service (xapi-nbd) shows the state “failed” after it stopped. This state does not indicate an error.

Common NBD connection issues

The command get_nbd_info returns an empty list of connection details

  • Ensure that the XenServer host that runs the NBD server has a PIF with an IP address.
  • Ensure that you have at least one network in your pool with the purpose nbd or insecure_nbd.
  • Ensure that the storage repository that the VDI is on is attached to a host that is connected to one of the NBD-enabled networks.

You can’t access the IP address returned by the command get_nbd_info

  • Check whether NBD is enabled on a network that is not reachable by the client.
  • Check whether multiple networks are mixed on the same subnet and NBD is allowed on some of them but blocked on others.
  • Check the NBD network configuration.

The connection refused or closed, or the NBD client hangs

  • Verify whether the session that the client passes to the NBD server is valid. If the session has expired or is not valid, you see the error “SESSION_INVALID”. The session has to be valid for the time of the backup, otherwise, the NBD server refuses the connection or the client might hang.
  • The maximum parallel connection limit might have been exceeded. To work around this, you can restart xapi-nbd.
  • If the NBD server is configured to use TLS, ensure that the client is able to use TLS.
Troubleshoot Changed Block Tracking