Thin-provisioned shared GFS2 block storage
Thin provisioning better utilizes the available storage by allocating disk storage space to VDIs as data is written to the virtual disk, rather than allocating the full virtual size of the VDI in advance. Thin provisioning enables you to significantly reduce the amount of space required on a shared storage array, and with that your Total Cost of Ownership (TCO).
Thin provisioning for shared block storage is of particular interest in the following cases:
- You want increased space efficiency. Images are sparsely and not thickly allocated.
- You want to reduce the number of I/O operations per second on your storage array. The GFS2 SR is the first SR type to support storage read caching on shared block storage.
- You use a common base image for multiple virtual machines. The images of individual VMs will then typically utilize even less space.
- You use snapshots. Each snapshot is an image and each image is now sparse.
- You want to create VDIs that are greater than 2 TiB in size. The GFS2 SR supports VDIs up to 16 TiB in size.
- Your storage does not support NFS or SMB3 and only supports block storage. If your storage supports NFS or SMB3, we recommend you use these SR types instead of GFS2.
- Your storage doesn’t support thin provisioning of LUNs. If your storage does thin provision LUNs, you can encounter problems and run out of space when combining it with GFS2. Combining GFS2 with a thin-provisioned LUN does not provide many additional benefits and is not recommended. XenCenter shows a warning if you attempt to use GFS2 with a thin-provisioned LUN.
Note:
We recommend not to use a GFS2 SR with a VLAN due to a known issue where you cannot add or remove hosts on a clustered pool if the cluster network is on a non-management VLAN.
The shared GFS2 type represents disks as a filesystem created on an iSCSI or HBA LUN. VDIs stored on a GFS2 SR are stored in the QCOW2 image format.
1. Plan your GFS2 environment
To provide the benefits of thin provisioning on shared block storage without risk of data loss, your pool must deliver a good level of reliability and connectivity. It is crucial that the hosts in the resource pool that uses GFS2 can reliably communicate with one another. To ensure this, XenServer requires that you use a clustered pool with your GFS2 SR. We also recommend that you design your environment and configure XenServer features to provide as much resiliency and redundancy as possible.
Before setting up your XenServer pool to work with GFS2 SRs, review the following requirements and recommendations for an ideal GFS2 environment:
-
Recommended: Configure redundant networking infrastructure.
-
Recommended: Create a dedicated bonded network
-
Required: Set up a clustered pool
-
Recommended: Configure storage multipathing
-
Required: Create a GFS2 SR
A clustered pool with GFS2 SRs has some differences in behavior to other types of pool and SR. For more information, see Constraints.
2. Configure redundant networking infrastructure
A bonded network links two or more NICs together to create a single channel for network traffic. We recommend that you use a bonded network for your clustered pool traffic. However, before you set up your bonded network, ensure that your network hardware configuration promotes redundancy in the bonded network. Consider implementing as many of these recommendations as is feasible for your organization and environment.
The following best practices add resiliency against software, hardware, or power failures that can affect your network switches.
- Ensure that you have separate physical network switches available for use in the bonded network, not just ports on the same switch.
- Ensure that the separate switches draw power from different, independent power distribution units (PDUs).
- If possible, in your data center, place the PDUs on different phases of the power feed or even feeds provided by different utility companies.
- Consider using uninterruptible power supply units to ensure that the network switches and servers can continue to function or perform an orderly shutdown in the event of a power failure.
3. Create a dedicated bonded network
It is important to ensure that hosts in a clustered pool can communicate reliably with one another. Creating a bonded network for this pool traffic increases the resiliency of your clustered pool.
A bonded network creates a bond between two or more NICs to create a single, high-performing channel that your clustered pool can use for cluster heartbeat traffic. We strongly recommend that this bonded network is not used for any other traffic. Create a separate network for the pool to use for management traffic.
Note:
If you have a firewall between the hosts in your pool, ensure that hosts can communicate on the cluster network using the following ports:
- TCP: 8892, 8896, 21064
- UDP: 5404, 5405
For more information, see Communication ports used by XenServer.
To create a bonded network to use as the clustering network:
- Open the New Network wizard.
- On the first page of the wizard, select Bonded Network and then click Next.
- On the Bond Members page, select the NICs you want to bond together. To select a NIC, select its check box in the list. Up to four NICs can be selected in this list. Clear the check box to deselect a NIC.
-
Under Bond mode, choose the type of bond:
- Select Active-active to configure an active-active bond. With this bond, traffic is balanced between the bonded NICs. If one NIC within the bond fails, the server’s network traffic automatically routes over the second NIC.
- Select Active-passive to configure an active-passive bond, where traffic passes over only one of the bonded NICs. In this mode, the second NIC only becomes active if the active NIC fails, for example, if it loses network connectivity.
- Select LACP with load balancing based on source MAC address to configure an LACP bond. With this bond, the outgoing NIC is selected based on the MAC address of the VM from which the traffic originated. Use this option to balance traffic in an environment where you have several VMs on the same host. This option is not suitable if there are fewer virtual interfaces (VIFs) than NICs: as load balancing is not optimal because the traffic cannot be split across NICs.
- Select LACP with load balancing based on IP and port of source and destination to configure an LACP bond. This bond uses the source IP address, source port number, destination IP address, and destination port number to allocate the traffic across the NICs. Use this option to balance traffic from VMs in an environment where the number of NICs exceeds the number of VIFs.
Notes:
- To be able to view the LACP bonding options in XenCenter and create an LACP bond, configure vSwitch as the network stack. Also, your switches must support the IEEE 802.3ad standard.
- Active-active and active-passive bond types are available for both the vSwitch and Linux bridge.
- You can bond either two, three, or four NICs when vSwitch is the network stack. However, you can only bond two NICs when the Linux bridge is the network stack.
- To use jumbo frames, set the Maximum Transmission Unit (MTU) to a value between 1500–9216.
- Select the Automatically add this network to new virtual machines check box to have the new network added to any new VMs created using the New VM wizard.
- Click Finish to create the new network and close the wizard.
After you create your bonded network on the pool coordinator, when you join other XenServer hosts to the pool, the network and bond information is automatically replicated to the joining server.
For more information, see Configuring NICs.
Notes:
- Changing the IP address of the cluster network by using XenCenter requires clustering and GFS2 to be temporarily disabled.
- Do not change the bonding of your clustering network while the cluster is live and has running VMs. This action can cause hosts in the cluster to hard restart (fence).
- If you have an IP address conflict (multiple hosts having the same IP address) on your clustering network involving at least one host with clustering enabled, the cluster does not form correctly and the hosts are unable to fence when required. To fix this issue, resolve the IP address conflict.
4. Set up a clustered pool
To use shared GFS2 storage, the XenServer resource pool must be a clustered pool. Enable clustering on your pool before creating a GFS2 SR.
To create a clustered pool:
- Open the New Pool dialog box by clicking New Pool on the toolbar.
- Enter a name for the new pool and an optional description. The name is displayed in the Resources pane.
- Nominate the pool coordinator by selecting a server from the Coordinator list.
- Select more servers to place in the new pool from the Additional members list. All available managed servers are listed. If a server is not listed, you can add it to the list by clicking Add New Server. If a managed server is not listed, it might be because it does not satisfy one or more of the pool join requirements listed in Pool requirements.
- Select Create Pool to create the pool and close the dialog box.
- Select the pool in the Resources panel and in its General tab, select Properties. The Pool Properties window opens.
- In the Clustering tab, select Enable clustering and choose the dedicated bonded network you created to be the cluster network.
- Click OK.
5. Configure storage multipathing
Ensure that storage multipathing is set up between your clustered pool and your GFS2 SR.
Multipathing routes storage traffic to a storage device over multiple paths for redundancy. All routes can have active traffic on them during normal operation, which results in increased throughput.
Before enabling multipathing, verify that the following statements are true:
-
Your ethernet or fibre switch is configured to make multiple targets available on your storage server.
For example, an iSCSI storage back-end queried for
sendtargetson a given portal returns multiple targets, as in the following example:iscsiadm -m discovery --type sendtargets --portal 192.168.0.161 192.168.0.161:3260,1 iqn.strawberry:litchie 192.168.0.204:3260,2 iqn.strawberry:litchieHowever, you can perform additional configuration to enable iSCSI multipath for arrays that only expose a single target. For more information, see iSCSI multipath for arrays that only expose a single target.
-
For iSCSI only, the control domain (dom0) has an IP address on each subnet used by the multipathed storage.
Ensure that for each path to the storage, you have a NIC and that there is an IP address configured on each NIC. For example, if you want four paths to your storage, you must have four NICs that each have an IP address configured.
-
For iSCSI only, every iSCSI target and initiator has a unique IQN.
-
For iSCSI only, the iSCSI target ports are operating in portal mode.
-
For HBA only, multiple HBAs are connected to the switch fabric.
-
If possible, use multiple redundant switches.
To enable multipathing:
Complete the following steps for every server in your pool:
- In the Resources pane, select the server and then put it into maintenance mode. There is a short delay while XenCenter migrates any active virtual machines and unplugs the existing storage. If the server is a pool coordinator, it is disconnected and might disappear from the Resources pane temporarily while a new pool coordinator is assigned. When the server reappears in the Resources pane with the server maintenance mode icon(
), continue to the next step. - On the General tab, select Properties and then select the Multipathing tab.
- To enable multipathing, check the Enable multipathing on this server check box. To disable multipathing, clear the check box.
- Click OK to apply the new setting and close the dialog box. There is a short delay while XenCenter saves the new storage configuration.
- Take the server back out of maintenance mode. Select the server in the Resources pane, right-click, and select Exit Maintenance Mode.
Ensure that you enable multipathing on all hosts in the pool. All cabling and, in the case of iSCSI, subnet configurations must match the corresponding NICs on each host.
6. Create a GFS2 SR
Create your shared GFS2 SR on an iSCSI or an HBA LUN that is visible to all XenServer hosts in your resource pool.
Note:
We do not recommend using a thin-provisioned LUN with GFS2. XenCenter shows a warning if you attempt to use GFS2 with a thin-provisioned LUN.
However, if you do choose this configuration, you must ensure that the LUN always has enough space to allow XenServer to write to it.
You can add up to 62 GFS2 SRs to a clustered pool.
To create a software iSCSI SR
Note:
Before performing the following steps, ensure that the iSCSI Initiator IQN is set appropriately for all hosts in the pool. For more information, see Changing Server Properties.
- Open the New Storage Repository wizard: click New Storage on the toolbar. Alternatively:
- On the Storage tab for the selected pool or server, click New SR.
- On the Storage menu, click New SR.
- In the Resources pane, select a server or pool then right-click and click New SR on the shortcut menu.
- Select Software iSCSI as the physical storage type, then click Next.
- On the Name page, enter the name of the new SR. By default, the wizard generates a description of the SR. This description includes a summary of the configuration options you select as you progress through the wizard. To enter your own description, clear the Auto-generate description check box and type in the Description box. Click Next to continue.
- On the Provisioning page, select Thin provisioning (GFS2).
-
On the Location page, specify the iSCSI target details:
-
Target Host: The IP address or DNS name of the iSCSI target. This can also be a comma-separated list of values.
-
Use CHAP: This is not supported with GFS2 SRs. Leave this option unselected.
-
Target IQN: To specify the iSCSI target IQN, click the Discover IQNs button and then choose an IQN from the Target IQN list.
Important:
The iSCSI target and all servers in the pool must not have the same IQN set. Every iSCSI target and initiator must have a unique IQN. If a non-unique IQN identifier is used, data corruption can occur, access to the target can be denied, or both.
-
Target LUN: To specify the LUN on which to create the storage repository, click the Discover LUNs button. Choose a LUN from the Target LUN list.
Each individual iSCSI storage repository must be contained entirely on a single LUN. The SR cannot span more than one LUN. If the LUN already contains an SR, choose either to use the existing SR or to replace the existing SR with a new one. Replacing the existing SR destroys any data present on the disk.
-
- Click Finish to complete the new SR configuration and close the wizard.
To create a hardware HBA SR
- To open the New Storage Repository wizard, you can do any of the following actions:
- On the toolbar, select New Storage.
- On the Storage tab for the selected pool or server, select New SR.
- On the Storage menu, select New SR.
- In the Resources pane, select a server or pool then right-click and select New SR on the shortcut menu.
- Select Hardware HBA as the physical storage type and then select Next.
- On the Name page, enter the name of the new SR. By default, the wizard generates a description of the SR. This description includes a summary of the configuration options you select as you progress through the wizard. To enter your own description, clear the Auto-generate description check box and type in the Description box. Click Next to continue to the Provisioning page.
- On the Provisioning page, select the Thin provisioning (GFS2).
- Click Next to continue to the Location page.
-
The wizard scans for available LUNs and then displays a page listing all the LUNs found. Select a LUN from the list and click Create.
Note:
A warning message is displayed if there are existing SRs on the LUN you have selected. Review the details and choose one of the following options.
- To use the existing, click Reattach.
- To delete the existing SR and to create an SR, click Format.
- If you prefer to select a different LUN, click Cancel and select a LUN from the list.
- The Summary page displays information about the new SR. Read the information and then click Finish to complete the SR creation process.
Constraints
Shared GFS2 storage currently has the following constraints:
-
As with any thin-provisioned SR, if the GFS2 SR usage grows to 100%, further writes from VMs fail. These failed writes can then lead to failures within the VM, possible data corruption, or both.
-
XenCenter shows an alert when your SR usage grows to 80%. Ensure that you monitor your GFS2 SR for this alert and take the appropriate action if seen. On a GFS2 SR, high usage causes a performance degradation. We recommend that you keep your SR usage below 80%.
-
VM migration with storage migration (live or offline) is not supported for VMs whose VDIs are on a GFS2 SR. You also cannot migrate VDIs from another type of SR to a GFS2 SR.
-
The FCoE transport is not supported with GFS2 SRs.
-
Trim/unmap is not supported on GFS2 SRs.
-
CHAP is not supported on GFS2 SRs.
-
(Citrix Hypervisor 8.2 Cumulative Update 1 only) Changed block tracking is not supported for VDIs stored on GFS2 SRs.
-
You cannot export VDIs that are greater than 2 TiB as VHD or OVA/OVF. However, you can export VMs with VDIs larger than 2 TiB in XVA format.
-
We do not recommend using a thin-provisioned LUN with GFS2. However, if you do choose this configuration, you must ensure that the LUN always has enough space to allow XenServer to write to it.
-
We do not recommend using SAN deduplication with GFS2 SRs. However, if you do choose this configuration, you must use suitable external monitoring of your SAN utilization to ensure that there is always space for XenServer to write to.
-
Your GFS2 file system cannot be larger than 100 TiB.
-
You cannot have more than 62 GFS2 SRs in your pool.
-
Clustered pools only support up to 16 hosts per pool.
-
For cluster traffic, we strongly recommend that you use a bonded network that uses at least two different network switches. Do not use this network for any other purposes.
-
Changing the IP address of the cluster network by using XenCenter requires clustering and GFS2 to be temporarily disabled.
-
Do not change the bonding of your clustering network while the cluster is live and has running VMs. This action can cause hosts in the cluster to hard restart (fence).
-
If you have an IP address conflict (multiple hosts having the same IP address) on your clustering network involving at least one host with clustering enabled, the cluster does not form correctly and the hosts are unable to fence when required. To fix this issue, resolve the IP address conflict.