Configure vGPU and GPU pass-through
The following example is a tutorial of a typical vGPU configuration use case using the XenServer PowerShell SDK cmdlets. The example output provided is from a server with both a NVIDIA Grid K1 and K2 card installed.
We start by connecting to the server:
Connect-XenServer -Server server -UserName username -Password password
<!--NeedCopy-->
The first thing we probably want to check is which GPU groups exist (these have been created automatically once the graphics hardware is installed):
PS> Get-XenGPUGroup | select name_label, GPU_types, allocation_algorithm
name_label GPU_types allocation_algorithm
---------- --------- --------------------
Group of NVIDIA Corporation GK104GL [GRID K2] GPUs {10de/11bf} depth_first
Group of NVIDIA Corporation GK107GL [GRID K1] GPUs {10de/0ff2} breadth_first
<!--NeedCopy-->
The allocation_algorithm is the placement policy for assigning VMs to GPUs in order to achieve either maximum density by placing as many VMs as possible on the same GPU (depth_first), or maximum performance by placing VMs on as many GPUs as possible (breadth_first). For example, the following command sets the allocation_algorithm to achieve maximum performance:
PS> Get-XenGPUGroup | Set-XenGPUGroup -AllocationAlgorithm breadth_first
PS> Get-XenGPUGroup | select name_label, GPU_types, allocation_algorithm
name_label GPU_types allocation_algorithm
---------- --------- --------------------
Group of NVIDIA Corporation GK104GL [GRID K2] GPUs {10de/11bf} breadth_first
Group of NVIDIA Corporation GK107GL [GRID K1] GPUs {10de/0ff2} breadth_first
<!--NeedCopy-->
Now we can list some information about the vGPU types. The vGPU types are pre-sets which can be used to create different kinds of vGPUs.
PS> Get-XenVGPUType | ft vendor_name, model_name, framebuffer_size, max_heads, `
max_resolution_x, max_resolution_y
vendor_name model_name framebuffer_size max_heads max_resolution_x max_resolution_y
----------- ---------- ---------------- --------- ---------------- ----------------
NVIDIA Corporation GRID K100 268435456 2 1920 1200
NVIDIA Corporation GRID K140Q 1006632960 2 2560 1600
NVIDIA Corporation GRID K240Q 1006632960 2 2560 1600
NVIDIA Corporation GRID K260Q 2013265920 4 2560 1600
NVIDIA Corporation GRID K200 268435456 2 1920 1200
passthrough 0 0 0 0
<!--NeedCopy-->
The vGPU type passthrough is supported for all PCI display devices, and can be used to create pass-through vGPUs.
We can see for which PCI display devices a vGPU type is supported as follows:
PS> Get-XenVGPUType | Where {$_.model_name -eq "GRID K100"} |`
Get-XenVGPUTypeProperty -XenProperty SupportedOnPGPUs |`
Get-XenPGPU | Get-XenPCI -Ref {$_.PCI} | select device_name, pci_id
device_name pci_id
----------- ------
GK107GL [GRID K1] 0000:0a:00.0
GK107GL [GRID K1] 0000:07:00.0
GK107GL [GRID K1] 0000:08:00.0
GK107GL [GRID K1] 0000:09:00.0
<!--NeedCopy-->
A vGPU type will show up as supported or enabled in a GPU group if it is supported or enabled respectively on at least one of the pGPUs in the group. We can query the GPU groups to find out which vGPU types are supported or enabled. For example:
PS> Get-XenGPUGroup | Where {$_.name_label -match "GRID K2"} |`
Get-XenGPUGroupProperty -XenProperty EnabledVGPUTypes |`
Get-XenVGPUType | select model_name
model_name
----------
GRID K200
passthrough
GRID K260Q
GRID K240Q
<!--NeedCopy-->
We may want to disallow a certain vGPU type on a pGPU. For example, the following commands disable and then re-enable the vGPU type Grid K240Q:
PS> $vgpuType = Get-XenVgpuType | where {$_.model_name -eq "GRID K240Q"}
PS> $vgpuType | Get-XenVGPUTypeProperty -XenProperty EnabledOnPGPUs |`
Get-XenPGPU | select uuid
uuid
----
a913a90d-0be9-aea3-862f-3fbfe1823a3f
8a0d1316-0f09-1ee9-f95b-c778c759ee40
PS> Remove-XenPGPUProperty -Uuid a913a90d-0be9-aea3-862f-3fbfe1823a3f `
-EnabledVGPUTypes $vgputype.opaque_ref
PS> $vgpuType | Get-XenVGPUTypeProperty -XenProperty EnabledOnPGPUs |get-xenpgpu | select uuid
uuid
----
8a0d1316-0f09-1ee9-f95b-c778c759ee40
PS> Add-XenPGPU -Uuid a913a90d-0be9-aea3-862f-3fbfe1823a3f -EnabledVGPUTypes $vgputype.opaque_ref
PS> $vgpuType | Get-XenVGPUTypeProperty -XenProperty EnabledOnPGPUs |get-xenpgpu | select uuid
uuid
----
a913a90d-0be9-aea3-862f-3fbfe1823a3f
8a0d1316-0f09-1ee9-f95b-c778c759ee40
<!--NeedCopy-->
We may want to find out how many vGPUs of a certain type can be started on the pGPUs in a group, in addition to the vGPUs which are already running:
PS> $gpuGroups = Get-XenGPUGroup
PS> Get-XenVGPUtype | % {Get-XenGPUGroupProperty $gpuGroups[0] -XenProperty RemainingCapacity -VgpuType $_}
0
0
8
4
16
2
<!--NeedCopy-->
Now suppose we want to assign a vGPU of type Grid K260Q to a VM which has no vGPU yet. This can be done as follows:
PS> $vm = Get-XenVm -Name "w7-test"
PS> $vgpuTypes = Get-XenVGPUtype
PS> Get-XenVMProperty -VM $vm -XenProperty VGPUs
PS>
PS> New-XenVGPU -VM $vm -GPUGroup $gpuGroups[0] -Device 0 -Type $vgpuTypes[3] -PassThru
uuid : f1122947-3b11-3fd3-0630-779701b37265
VM : XenAPI.XenRef`1[XenAPI.VM]
GPU_group : XenAPI.XenRef`1[XenAPI.GPU_group]
device : 0
currently_attached : False
other_config : {}
type : XenAPI.XenRef`1[XenAPI.VGPU_type]
resident_on : XenAPI.XenRef`1[XenAPI.PGPU]
opaque_ref : OpaqueRef:15b2d1a9-8944-2f28-53df-6b8274d4d6fb
Changed : True
<!--NeedCopy-->
At this stage we can boot the VM, install the in-guest NVIDIA drivers, and enable RDP. Then we may want to disable the VNC console as this has a significant performance overhead. To do this we need to shut down the VM, set the flag vgpu_vnc_enabled to false and then boot the VM.
PS> Invoke-XenVM -VM $vm -XenAction Shutdown –Async –PassThru | Wait-XenTask –ShowProgress
PS> $p = Get-XenVMProperty -VM $vm -XenProperty Platform
PS> $p["vgpu_vnc_enabled"]="false"
PS> Set-XenVM -VM $vm -Platform $p
PS> Invoke-XenVM -VM $vm -XenAction Start –Async –PassThru | Wait-XenTask –ShowProgress
<!--NeedCopy-->
Before finishing, we should remember to disconnect from the server:
Get-XenSession | Disconnect-XenServer
<!--NeedCopy-->