Using HTTP to interact with XenServer
XenServer exposes an HTTP(S) interface on each host that can be used to perform various operations.
Each operation is performed by constructing an HTTP(S) GET or POST call to a dedicated URL path. Parameters are appended to the URL following a question mark (?) and separated by ampersands (&). The calls require authentication by providing a valid management API session reference as a query parameter. For example:
wget https://<server>/services?session_id=<session_opaque_ref>
<!--NeedCopy-->
Note:
Basic auth using a username and password is also available:
wget https://username:password@<server>/services <!--NeedCopy-->However, this method is not recommended.
The following sections describe in greater detail some of the available operations.
Importing and exporting VMs as XVA packages
Because the import and export of VMs can take some time to complete, an asynchronous HTTP(S) interface is provided for these operations.
VM Export
To export a VM as an XVA package using the XenServer Management API, construct an HTTP(S) GET call. The call accepts the following query parameters:
| Parameter | Description |
|---|---|
session_id |
The opaque reference of the session being used to authenticate. |
uuid |
The uuid of the VM; required only if not using the ref parameter. |
ref |
The opaque reference of the VM; required only if not using the uuid parameter. |
task_id |
(Optional) The opaque reference of a task object with which to monitor the progress of the operation; the task object needs to be created first. |
For example, to export a VM to the XVA package vm.xva using the Linux
command line utility cURL:
curl https://server/export?session_id=<session_opaque_ref>&task_id=<task_opaque_ref>&uuid=<vm_uuid> -o vm.xva
<!--NeedCopy-->
Note that you can perform this operation on the pool coordinator even if the VM runs on a pool supporter. The request might result in a redirect if the VM’s disks are placed on an SR accessible only by a pool member.
If you want to export only the VM metadata without the disks, use the
HTTP(S) handler export_metadata. For example:
curl https://server/export_metadata?session_id=<session_opaque_ref>&ref=<vm_opaque_reference> -o vm_meta.xva
<!--NeedCopy-->
VM Import
To import a VM from an XVA package, construct an HTTP(S) PUT call. The call accepts the following query parameters:
| Parameter | Description |
|---|---|
session_id |
The opaque reference of the session being used to authenticate. |
sr_id |
(Optional) The opaque reference of the SR on which the imported VM’s disks will be placed. If not specified, the disks will be imported to Pool.default_SR. If the latter if not set, the error DEFAULT_SR_NOT_FOUND is returned. |
restore |
(Optional) If true, the import is treated as replacing the original VM from which the XVA package was exported. The implication of this currently is that the MAC addresses on the VIFs are exactly as the export was, which will lead to conflicts if the original VM is still running. |
force |
(Optional) If true, any checksum failures will be ignored (the default is to destroy the VM if a checksum error is detected). |
task_id |
(Optional) The opaque reference of a task object with which to monitor the progress of the operation; the task object needs to be created first |
For example, to import a VM from package vm.xva placing its disks on
a specific SR:
curl -T vm.xva http://server/import?session_id=<session_opaque_ref>&sr_id=<sr_opaque_ref>
<!--NeedCopy-->
Or, to import a VM from package vm.xva placing its disks on the default
SR, while monitoring the progress of the operation:
curl -T vm.xva http://server/import?session_id=<session_opaque_ref>&task_id=<task_opaque_ref>
<!--NeedCopy-->
To import just the metadata, use the HTTP(S) handler import_metadata:
curl -T vm.xva http://server/import_metadata?session_id=<session_opaque_ref>
<!--NeedCopy-->
Xen Virtual Appliance (XVA) VM Import Format
XenServer supports a human-readable VM input format called XVA. This section describes the syntax and structure of the format.
An XVA is essentially a tar archive containing XML metadata and a set
of disk images. A VM represented by an XVA is not intended to be directly
executable. The data within an XVA package is intended
for either archiving on permanent storage or for being transmitted to a
VM server - such as a XenServer host - where it can be extracted and
run.
XVA is a hypervisor-neutral packaging format; it is possible to create simple tools to instantiate an XVA VM on any other platform. XVA does not specify any particular runtime format; for example disks may be instantiated as file images, LVM volumes, QCoW images, VMDK or VHD images. An XVA VM may be instantiated any number of times, each instantiation may have a different runtime format.
Unlike other formats like the Open Virtual Appliance (OVA) format, XVA does not:
- specify any particular serialization or transport format
- provide any mechanism for customizing VMs (or templates) on install
- address how a VM may be upgraded post-install
- define how multiple VMs, acting as an appliance, may communicate.
The extracted content of an XVA file is a directory containing, at a
minimum, a file called ova.xml. This file contains the metadata
describing the VM packaged within the XVA. Each disk image is stored within
a sub-directory and is referenced from the ova.xml. For example, the
XVA describing a VM with two disks would contain the following metadata
file and sub-directories:
$ls -al
total 72
drwx------+ 1 user1 Domain Users 0 Nov 7 10:35 .
drwxrwx---+ 1 user1 Domain Users 0 Nov 7 10:35 ..
-r-x------+ 1 user1 Domain Users 56046 Jan 1 1970 ova.xml
drwx------+ 1 user1 Domain Users 0 Nov 7 10:34 Ref_6
drwx------+ 1 user1 Domain Users 0 Nov 7 10:34 Ref_9
<!--NeedCopy-->
The file ova.xml contains the VM metadata in
XML-RPC format, for example:
<value>
<struct>
<member>
<name>objects</name>
<value>
<array>
<data>
<value>
<struct>
<member><name>class</name><value>VM</value></member>
<member><name>id</name><value>Ref:VM</value></member>
<member>
<name>snapshot</name>
<value>
<struct>
<member><name>power_state</name><value>Halted</value></member>
<member><name>name_label</name><value>Test VM</value></member>
<member><name>memory_static_max</name><value>4294967296</value></member>
<member><name>VCPUs_max</name><value>2</value></member>
<member><name>VIFs</name><value><array><data><value>Ref:VIF</value></data></array></value></member>
<member><name>PV_bootloader</name><value>pygrub</value></member>
<member><name>HVM_boot_policy</name><value>BIOS order</value></member>
<!-- ... -->
</struct>
</value>
</member>
</struct>
</value>
<value>
<struct>
<member><name>class</name><value>VIF</value></member>
<member><name>id</name><value>Ref:VIF</value></member>
<member>
<name>snapshot</name>
<value>
<!--...-->
</value>
</member>
</struct>
</value>
<!--...-->
</data>
</array>
</value>
</member>
</struct>
</value>
<!--NeedCopy-->
A single disk image is represented by a directory containing a sequence of files as follows:
$ls -al Ref_6
total 2054
drwx------+ 1 user1 Domain Users 0 Nov 7 13:40 .
drwx------+ 1 user1 Domain Users 0 Nov 7 10:35 ..
-r-x------+ 1 user1 Domain Users 1048576 Jan 1 1970 00000000
-r-x------+ 1 user1 Domain Users 16 Jan 1 1970 00000000.xxhash
-r-x------+ 1 user1 Domain Users 1048576 Jan 1 1970 00000499
-r-x------+ 1 user1 Domain Users 16 Jan 1 1970 00000499.xxhash
...
<!--NeedCopy-->
Each of the extensionless files contains a block of raw disk image data of size 1MiB. The filename is the block number in decimal. The small size was chosen to be safely under the maximum file size limit of several filesystems. If these files are concatenated, the original image is recovered.
Each of the data block files is accompanied by a file with the same base
name and the extension .xxhash, which contains the checksum of the
corresponding data block file calculated using the
xxHash algorithm.
Note:
In XenServer versions prior to Citrix Hypervisor 8.1, the checksums of the disk image blocks were calculated using the SHA1 algorithm, and the files containing the checksums had the extension
.checksum. For example, a directory representing a disk image would contain the following sequence of files:$ls -al total 3079 drwx------+ 1 user1 Domain Users 0 Nov 7 14:20 . drwx------+ 1 user1 Domain Users 0 Nov 7 10:35 .. -rwx------+ 1 user1 Domain Users 1048576 Aug 4 09:50 00000000000000000000 -rwx------+ 1 user1 Domain Users 40 Aug 4 09:50 00000000000000000000.checksum -rwx------+ 1 user1 Domain Users 1048576 Aug 4 09:50 00000000000000000001 -rwx------+ 1 user1 Domain Users 40 Aug 4 09:50 00000000000000000001.checksum -rwx------+ 1 user1 Domain Users 1048576 Aug 4 09:50 00000000000000000003 -rwx------+ 1 user1 Domain Users 40 Aug 4 09:50 00000000000000000003.checksum ... <!--NeedCopy-->
Getting XenServer Operational Metrics
XenServer records metrics about the operation of various aspects of your XenServer installation. The metrics are stored persistently for long term access and analysis of historical trends.
Metrics available
A wide range of metrics are available for XenServer including: C-State, P-State, IOPS, Latency and many more. Not all of these are turned on by default. The full range of metrics available for both Hosts and VMs are detailed in section Monitor XenServer Performance of XenServer’s documentation. This chapter also details how you can explore these metrics via XenCenter.
The data store for operational metrics
Metrics are stored in Round Robin Databases (RRDs), which are maintained for individual VMs (including the control domain) and the server. Each database consists of multiple Round Robin Archives (RRAs) and is of a fixed size. This is because each RRA is in effect a circular buffer with a predefined maximum capacity.
The VM RRDs are resident on the server on which the VM is running, or the pool coordinator when the VM is not running. This means that the location of the VM must be known in order to retrieve the associated data.
The RRDs are also backed up every day. Metrics are persisted for a maximum of one year, and are stored at different granularities.
RRDs are saved to disk as uncompressed XML. The size of each RRD when written to disk ranges from 200KiB to approximately 1.2MiB when the RRD stores the full year of metrics.
Note:
If metrics cannot be written to disk, for example when a disk is full, metrics will be lost and the last saved version of the RRD will be used.
Data Granularity
Each archive in the database samples its particular metric at a specified granularity. The values are stored at intervals of:
- 5 seconds for the past 10 minutes
- one minute for the past 2 hours
- one hour for the past week
- one day for the past year
The sampling that takes place every 5 seconds records actual data points, however, the following RRAs contain Consolidated Data Points (CDPs) which are produced by applying the Consolidation Functions (CFs) to a number of actual data points, storing historical data in a far more compressed format. The CFs supported by XenServer are:
- AVERAGE
- MIN
- MAX
Downloading RRDs
Metrics can be downloaded over HTTP(s), for example using wget.
Note:
In early versions of the XenServer Management API, instantaneous operational metrics could be obtained using the
VM_metrics,VM_guest_metrics,host_metricsclasses and associated methods. These methods have been deprecated in favor of using the HTTP(S) interface described in this chapter to download the metrics from the RRDs on the VMs and servers. The legacy metrics APIs will not return any values.
Metrics can be downloaded in xport style XML or JSON format. See
rrddump
and rrdxport for
information about the XML format. The JSON format has the same structure
as the XML.
Starting from xapi version 23.17.0, the server decides which format to
return using the HTTP header Accept. To download metrics in XML format
use 'Accept: text/xml', while to download metircs in JSON format use
'Accept: application/json'. When both formats are accepted, for
example, when using 'Accept: */*' (which is the default for the wget
client), the server returns JSON. The content type is provided in the
reponse’s headers.
Note:
When no header is specified, for example, when using
'Accept: ', the server returns XML; to retrieve JSON you need to use the query parameterjson. The header takes precedence over the query parameter and is the recommended method for deciding the returned format.
Downloading the whole RRD
RRDs can be downloaded from the XenServer host on which they reside by
using the HTTP(S) handler registered at /host_rrd or /vm_rrd.
For example, to download a host RRD:
wget https://<server>/host_rrd?session_id=<session_opaque_reference>
<!--NeedCopy-->
To download a VM RRD, you must also specify the VM’s uuid as a query
parameter:
wget https://<server>/vm_rrd?session_id=<session_opaque_reference>&uuid=<vm_uuid>
<!--NeedCopy-->
Getting updates from the RRD
So as to not have to download the whole database when you have most of
the data already, a query can also be made to retrieve just updates.
This is achieved via the HTTP(S) handler /rrd_updates.
Updates are described in relation to a start time which is specified as
epoch time (number of seconds since Jan 1st, 1970) in a query parameter
named start.
To obtain an update of all VM metrics on a server, the URL would be of the form:
wget http://<server>/rrd_updates?session_id=<session_opaque_reference>&start=<start_time>
<!--NeedCopy-->
The response contains data for every VM resident on the particular host
that is being queried. This means that it is not possible to query a
particular VM on its own. To differentiate which column in the export is
associated with which VM, the legend field is prefixed with the VM’s
uuid. The data set also contains a prefix describing the CF used to
collect the data (MAX/MIN/AVERAGE).
To obtain host updates too, add the query parameter host=true:
wget http://<server>/rrd_updates?session_id=<session_opaque_reference>&start=<start_time>&host=true
<!--NeedCopy-->
When using RRD updates, you should be aware of the following:
-
/rrd_updateswill only return data for the metrics that are currently being collected. For example, if you have an unplugged VBD,/vm_rrdwill return the historical data for that VBD from when it was in use, but/rrd_updateswon’t. -
The value of the
startparameter is used to deduce the granularity of the data/rrd_updateswill return. If you specify a shorter time period, you will get more detailed metrics. For example, if you specify astartvalue that is 9 minutes before “now”, you’ll get 108 rows from the 10 minute RRA, but if you specify 11 minutes before “now”, you’ll get 11 rows from the 2 hour RRA. -
Of particular importance is to use the same definition of “now” as the server is using, since the server is likely to be in UTC and your client may be in a different time zone.
-
The data returned by each call to
/rrd_updateswill contain a value (in theendXML tag or JSON field) that can be used as thestartparameter for the next call.
Additional /rrd_updates parameters
| Parameter | Value | Description |
|---|---|---|
cf |
average\|min\|max |
the data consolidation mode |
interval |
<interval> |
the interval between values to be reported |
Note:
By default only
averagemetrics are available. To obtainminandmaxmetrics for a VM, run the following command:xe pool-param-set uuid=<pool_uuid> other-config:create_min_max_in_new_VM_RRDs <!--NeedCopy-->
Analysing RRDs
Once you have obtained the requested RRDs, you can parse and analyze the results.
This might be useful in if you want to perform simple tasks such as
determining what the current Network throughput is, or perhaps
carrying out your own, more complex analysis on the data points.
Some examples written in Python to demonstrate how you can parse and use the metrics can be found at XenServer Operational Metrics.
Example RRDs
Whole RRDs
Below is an example XML for a whole host RRD. The table highlights certain XML tags of interest.
| XML tag | Description |
|---|---|
ds |
A data source field. |
name |
The name of this data source. |
type |
Defines the data source type. Can be GAUGE, COUNTER, DERIVE or ABSOLUTE. |
minimal_heartbeat |
Defines the maximum number of seconds before a ds value is considered unknown. |
min |
The minimum acceptable value. Values less than this number are considered unknown. This is optional. |
max |
The maximum acceptable value. Values exceeding this number are considered unknown. This is optional. |
<?xml version="1.0"?>
<rrd>
<version>0003</version>
<step>5</step>
<lastupdate>1213616574</lastupdate>
<ds>
<name>memory_total_kib</name>
<type>GAUGE</type>
<minimal_heartbeat>300.0000</minimal_heartbeat>
<min>0.0</min>
<max>Infinity</max>
<last_ds>2070172</last_ds>
<value>9631315.6300</value>
<unknown_sec>0</unknown_sec>
</ds>
<ds>
<!-- other data sources - the order of the data sources is important
and defines the ordering of the columns in the archives below -->
</ds>
<rra>
<cf>AVERAGE</cf>
<pdp_per_row>1</pdp_per_row>
<params>
<xff>0.5000</xff>
</params>
<cdp_prep> <!-- This is for internal use -->
<ds>
<primary_value>0.0</primary_value>
<secondary_value>0.0</secondary_value>
<value>0.0</value>
<unknown_datapoints>0</unknown_datapoints>
</ds>
...other data sources - internal use only...
</cdp_prep>
<database>
<row>
<v>2070172.0000</v> <!-- columns correspond to the DSs defined above -->
<v>1756408.0000</v>
<v>0.0</v>
<v>0.0</v>
<v>732.2130</v>
<v>0.0</v>
<v>782.9186</v>
<v>0.0</v>
<v>647.0431</v>
<v>0.0</v>
<v>0.0001</v>
<v>0.0268</v>
<v>0.0100</v>
<v>0.0</v>
<v>615.1072</v>
</row>
...
</rra>
... other archives ...
</rrd>
<!--NeedCopy-->
Here is an example JSON for a whole host RRD:
{
"version": "0003",
"step": "5",
"lastupdate": "1698240426.8141",
"ds": [
{
"name": "memory_total_kib",
"type": "GAUGE",
"minimal_heartbeat": "300.0000",
"min": "0.0",
"max": "Infinity",
"last_ds": "33371836",
"value": "60540111.1548",
"unknown_sec": "0"
}
... other data sources...
],
"rra": [
{
"cf": "AVERAGE",
"pdp_per_row": "1",
"params": {
"xff": "0.5000"
},
"cdp_prep": {
"ds": [
{
"primary_value": "0.00",
"secondary_value": "0.00",
"value": "0.0",
"unknown_datapoints": "0.00"
}
... other data sources - internal use only...
]
},
"database": [
[
"0.0084",
"0.0090",
"0.0113",
"0.0072",
"0.0104",
"0.0099",
"0.0112",
"0.0131",
"0.0101",
"0.0100",
... other values for the first data source...
],
... other data sources...
]
},
... other archives...
]
}
<!--NeedCopy-->
Note that a VM RRD is identically structured, but with different data sources.
RRD updates
Example rrd_updates XML for one VM and no host updates:
<xport>
<meta>
<start>1213578000</start>
<step>3600</step>
<end>1213617600</end>
<rows>12</rows>
<columns>12</columns>
<legend>
<entry>AVERAGE:vm:ecd8d7a0-1be3-4d91-bd0e-4888c0e30ab3:cpu1</entry> <!-- nb - each data source might have multiple entries for different consolidation functions -->
<entry>AVERAGE:vm:ecd8d7a0-1be3-4d91-bd0e-4888c0e30ab3:cpu0</entry>
<entry>AVERAGE:vm:ecd8d7a0-1be3-4d91-bd0e-4888c0e30ab3:memory</entry>
<entry>MIN:vm:ecd8d7a0-1be3-4d91-bd0e-4888c0e30ab3:cpu1</entry>
<entry>MIN:vm:ecd8d7a0-1be3-4d91-bd0e-4888c0e30ab3:cpu0</entry>
<entry>MIN:vm:ecd8d7a0-1be3-4d91-bd0e-4888c0e30ab3:memory</entry>
<entry>MAX:vm:ecd8d7a0-1be3-4d91-bd0e-4888c0e30ab3:cpu1</entry>
<entry>MAX:vm:ecd8d7a0-1be3-4d91-bd0e-4888c0e30ab3:cpu0</entry>
<entry>MAX:vm:ecd8d7a0-1be3-4d91-bd0e-4888c0e30ab3:memory</entry>
<entry>LAST:vm:ecd8d7a0-1be3-4d91-bd0e-4888c0e30ab3:cpu1</entry>
<entry>LAST:vm:ecd8d7a0-1be3-4d91-bd0e-4888c0e30ab3:cpu0</entry>
<entry>LAST:vm:ecd8d7a0-1be3-4d91-bd0e-4888c0e30ab3:memory</entry>
</legend>
</meta>
<data>
<row>
<t>1213617600</t>
<v>0.0</v> <!-- once again, the order or the columns is defined by the legend above -->
<v>0.0282</v>
<v>209715200.0000</v>
<v>0.0</v>
<v>0.0201</v>
<v>209715200.0000</v>
<v>0.0</v>
<v>0.0445</v>
<v>209715200.0000</v>
<v>0.0</v>
<v>0.0243</v>
<v>209715200.0000</v>
</row>
...
</data>
</xport>
<!--NeedCopy-->
Example rrd_updates JSON for one VM and no host updates:
{
"meta": {
"start": "1213660800",
"step": "86400",
"end": "1698192000",
"rows": "366",
"columns": "101",
"legend": [
"AVERAGE:vm:b36c4883-7dca-41c6-9f35-ec91826ec29f:cpu0",
"AVERAGE:vm:b36c4883-7dca-41c6-9f35-ec91826ec29f:cpu1",
"AVERAGE:vm:b36c4883-7dca-41c6-9f35-ec91826ec29f:cpu2",
"AVERAGE:vm:b36c4883-7dca-41c6-9f35-ec91826ec29f:cpu3",
"AVERAGE:vm:b36c4883-7dca-41c6-9f35-ec91826ec29f:cpu4",
"AVERAGE:vm:b36c4883-7dca-41c6-9f35-ec91826ec29f:cpu5",
"AVERAGE:vm:b36c4883-7dca-41c6-9f35-ec91826ec29f:cpu6",
"AVERAGE:vm:b36c4883-7dca-41c6-9f35-ec91826ec29f:cpu7",
"AVERAGE:vm:b36c4883-7dca-41c6-9f35-ec91826ec29f:memory",
...
]
},
"data": [
{
"t": "1698192000",
"values": [
"0.0082",
"0.0080",
"0.0072",
"0.0072",
"0.0069",
"0.0073",
"0.0069",
"0.0076",
"2724740352.0000",
"NaN",
"NaN",
...
]
},
...
]
}
<!--NeedCopy-->