GPU passthrough with Xen 4.2 on Ubuntu 12.04

GPU passthrough with Xen 4.2 on Ubuntu 12.04

[Matthew Dean]

[-- Attachment #1.1: Type: text/plain, Size: 4760 bytes --]

I have been trying to setup GPU passthrough for a couple of days now 
with little luck.  I'm hoping someone can shed some light as to where I 
may be going wrong or at least identify some genuine bugs. Essentially 
pci passthrough works for me but gpu passthrough doesn't.

My system is currently configured as follows (please ask if you need 
further details)

Asrock Z77 e-Itx (stock bios, not sure the version but I can find out.  
vt-d is enabled)
i7 3770S
AMD Radeon HD 7750

Dom0 - Ubuntu server 12.04 with desktop environment and the following 
extra packages installed before the build

build-essential zlib1g-dev python-dev libncurses5-dev libssl-dev 
libx11-dev uuid-dev libyajl-dev libaio-dev libglib2.0-dev pkg-config 
bridge-utils iproute udev bison flex gettext bin86 bcc iasl markdown 
ocaml-nox ocaml-findlib git gcc-multilib texinfo pciutils-dev gawk 
libcurl3 libcurl4-openssl-dev bzip2 module-init-tools transfig tgif 
texlive-latex-base texlive-latex-recommended texlive-fonts-extra 
texlive-fonts-recommended mercurial make gcc libc6-dev python 
python-twisted patch libsdl-dev libjpeg62-dev libbz2-dev e2fslibs-dev 
git-core xz-utils liblzma-dev liblzo2-dev libvncserver-dev

I've used the stock Ubuntu provided kernel which is version 
3.2.0-27-generic.  Dom0 is connected to a display via the intel 
integrated graphics.

xen 4.2 has been build from xen unstable tag 4.2.0-rc1 changeset 25689 
using the following commands

./configure
make world
make install

I had to manually add a line to /etc/fstab to get /proc/xen to mount on 
startup
Modules xen-evtchn, xen-gntdev and xen-pciback were setup to load on 
boot in /etc/modules
I've then setup xencommons to start on boot: "update-rc.d xencommons 
defaults 19 18"

Grub2 has then been setup to automatically boot the xen kernel. I've 
also had to add the option xsave=0 to the xen boot command line to get 
things to boot

On restart everything looks good. "xl list" returns the Dom0 domain 
only.  "cat /proc/xen/capabilities" returns control_d.  I've created a 
windows 7 domU without any pci passthrough and successfully installed 
windows 7 ultimate.  The config file looks like:

builder='hvm'
vcpus = 4
memory = 8192
shadow_memory = 48
name = "xenwin7"
vif = [ 'bridge=br0' ]
acpi = 1
apic = 1
disk = [ 'file:/usr/local/xenImages/xenwin7.img,hda,w']
boot="c"
sdl=0
vnc=1
vncconsole=1
vncpasswd=''
viridian=1
usb=1
serial='pty'
usbdevice='tablet'
on_poweroff="destroy"
on_reboot="restart"
on_crash="destroy"

I would like to be able to pass through the HD 7750 and a USB controller 
to the VM.  I see from lspci there are three devices I need to pass 
through, two for the gpu (the card itseft and the hd audio for hdmi 
device) and one for the USB controller.  The bus/device ids are:

0000:01:00.0
0000:01:00.1
0000:00:14.0

There are other usb controllers but I would like to leave them with 
dom0.  I can successfully configure the devices for passthrough using 
"xl pci-assignable-add".  In the config I then add the line:

  pci=["01:00.0","01:00.1","00:14.0"]

When I boot the VM everything seems to work fine.  Windows picks up all 
three devices and I can install drivers for them.  The USB controller 
works flawlessly and I can use an attached mouse and keyboard.  The 
radeon card requires a restart after which an attached display comes to 
life and I have 3D acceleration. Restarting the VM seems to work OK.  If 
however I shut down the VM I have no end of problems.  On any subsequent 
startup the vm struggles to get past the windows splash screen, waiting 
much longer than usual.  During this period dom0 is sluggish regarding 
mouse and keyboard input even though cpu and memory usage are very low.  
When windows finally loads I have no active display and I have to view 
the VM via VNC.  In device manage I find that windows has disabled the 
GPU saying there are not enough resources to run the card.  From this 
point onwards I can do nothing to get the gpu working again aside from 
removing the device, manually deleting the drivers, and starting again.

Note that at this point I am only trying to do secondary passthrough 
though I would ideally like to get to the point of doing primary 
passthrough.  Adding the line gfx_passthru=1 at any point in all this to 
the machine config however just prevents the VM from booting entirely; 
when I VNC in all I get is a qemu prompt and I never get any ouput to 
the real display.

Any suggestions as to how to get this to work would be greatly 
appreciated as I've hit a bit of a brick wall.  I should also say that I 
have managed to get secondary passthrough working using Debian Wheezy 
and the repository version of xen 4.1.  In that case though Dom0 didn't 
boot reliably.

Sincerely,

Matthew Dean

[-- Attachment #1.2: Type: text/html, Size: 5909 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: GPU passthrough with Xen 4.2 on Ubuntu 12.04

[Pasi Kärkkäinen]

On Wed, Aug 08, 2012 at 02:26:33PM +0100, Matthew Dean wrote:
> 
>    I've used the stock Ubuntu provided kernel which is version
>    3.2.0-27-generic.  Dom0 is connected to a display via the intel integrated
>    graphics.
> 

You might want to try Linux 3.4.x or 3.5.x dom0 kernels aswell.
I think there has been some fixes in xen-pciback.


> 
>    When I boot the VM everything seems to work fine.  Windows picks up all
>    three devices and I can install drivers for them.  The USB controller
>    works flawlessly and I can use an attached mouse and keyboard.  The radeon
>    card requires a restart after which an attached display comes to life and
>    I have 3D acceleration.  Restarting the VM seems to work OK.  If however I
>    shut down the VM I have no end of problems.  On any subsequent startup the
>    vm struggles to get past the windows splash screen, waiting much longer
>    than usual.  During this period dom0 is sluggish regarding mouse and
>    keyboard input even though cpu and memory usage are very low.  When
>    windows finally loads I have no active display and I have to view the VM
>    via VNC.  In device manage I find that windows has disabled the GPU saying
>    there are not enough resources to run the card.  From this point onwards I
>    can do nothing to get the gpu working again aside from removing the
>    device, manually deleting the drivers, and starting again.
> 

Do you get any errors from Xen (xl dmesg), or from dom0 kernel (dmesg) ? 

Do you have a serial console? 


>    Note that at this point I am only trying to do secondary passthrough
>    though I would ideally like to get to the point of doing primary
>    passthrough.  Adding the line gfx_passthru=1 at any point in all this to
>    the machine config however just prevents the VM from booting entirely;
>    when I VNC in all I get is a qemu prompt and I never get any ouput to the
>    real display.
> 

AMD/ATI primary passthru requires extra patches to Xen qemu-dm,
those are not included out-of-the-box yet.

>    Any suggestions as to how to get this to work would be greatly appreciated
>    as I've hit a bit of a brick wall.  I should also say that I have managed
>    to get secondary passthrough working using Debian Wheezy and the
>    repository version of xen 4.1.  In that case though Dom0 didn't boot
>    reliably.
> 

-- Pasi

Re: GPU passthrough with Xen 4.2 on Ubuntu 12.04

[Konrad Rzeszutek Wilk]

On Wed, Aug 08, 2012 at 02:26:33PM +0100, Matthew Dean wrote:
> I have been trying to setup GPU passthrough for a couple of days now
> with little luck.  I'm hoping someone can shed some light as to
> where I may be going wrong or at least identify some genuine bugs.
> Essentially pci passthrough works for me but gpu passthrough
> doesn't.
> 
> My system is currently configured as follows (please ask if you need
> further details)
> 
> Asrock Z77 e-Itx (stock bios, not sure the version but I can find
> out.  vt-d is enabled)
> i7 3770S
> AMD Radeon HD 7750
> 
> Dom0 - Ubuntu server 12.04 with desktop environment and the
> following extra packages installed before the build
> 
> build-essential zlib1g-dev python-dev libncurses5-dev libssl-dev
> libx11-dev uuid-dev libyajl-dev libaio-dev libglib2.0-dev pkg-config
> bridge-utils iproute udev bison flex gettext bin86 bcc iasl markdown
> ocaml-nox ocaml-findlib git gcc-multilib texinfo pciutils-dev gawk
> libcurl3 libcurl4-openssl-dev bzip2 module-init-tools transfig tgif
> texlive-latex-base texlive-latex-recommended texlive-fonts-extra
> texlive-fonts-recommended mercurial make gcc libc6-dev python
> python-twisted patch libsdl-dev libjpeg62-dev libbz2-dev
> e2fslibs-dev git-core xz-utils liblzma-dev liblzo2-dev
> libvncserver-dev
> 
> I've used the stock Ubuntu provided kernel which is version
> 3.2.0-27-generic.  Dom0 is connected to a display via the intel
> integrated graphics.
> 
> xen 4.2 has been build from xen unstable tag 4.2.0-rc1 changeset
> 25689 using the following commands
> 
> ./configure
> make world
> make install
> 
> I had to manually add a line to /etc/fstab to get /proc/xen to mount
> on startup
> Modules xen-evtchn, xen-gntdev and xen-pciback were setup to load on
> boot in /etc/modules
> I've then setup xencommons to start on boot: "update-rc.d xencommons
> defaults 19 18"
> 
> Grub2 has then been setup to automatically boot the xen kernel. I've
> also had to add the option xsave=0 to the xen boot command line to
> get things to boot
> 
> On restart everything looks good. "xl list" returns the Dom0 domain
> only.  "cat /proc/xen/capabilities" returns control_d.  I've created
> a windows 7 domU without any pci passthrough and successfully
> installed windows 7 ultimate.  The config file looks like:
> 
> builder='hvm'
> vcpus = 4
> memory = 8192
> shadow_memory = 48
> name = "xenwin7"
> vif = [ 'bridge=br0' ]
> acpi = 1
> apic = 1
> disk = [ 'file:/usr/local/xenImages/xenwin7.img,hda,w']
> boot="c"
> sdl=0
> vnc=1
> vncconsole=1
> vncpasswd=''
> viridian=1
> usb=1
> serial='pty'
> usbdevice='tablet'
> on_poweroff="destroy"
> on_reboot="restart"
> on_crash="destroy"
> 
> I would like to be able to pass through the HD 7750 and a USB
> controller to the VM.  I see from lspci there are three devices I
> need to pass through, two for the gpu (the card itseft and the hd
> audio for hdmi device) and one for the USB controller.  The
> bus/device ids are:
> 
> 0000:01:00.0
> 0000:01:00.1
> 0000:00:14.0
> 
> There are other usb controllers but I would like to leave them with
> dom0.  I can successfully configure the devices for passthrough
> using "xl pci-assignable-add".  In the config I then add the line:
> 
>  pci=["01:00.0","01:00.1","00:14.0"]
> 
> When I boot the VM everything seems to work fine.  Windows picks up
> all three devices and I can install drivers for them.  The USB
> controller works flawlessly and I can use an attached mouse and
> keyboard.  The radeon card requires a restart after which an
> attached display comes to life and I have 3D acceleration.
> Restarting the VM seems to work OK.  If however I shut down the VM I
> have no end of problems.  On any subsequent startup the vm struggles
> to get past the windows splash screen, waiting much longer than
> usual.  During this period dom0 is sluggish regarding mouse and
> keyboard input even though cpu and memory usage are very low.  When
> windows finally loads I have no active display and I have to view
> the VM via VNC.  In device manage I find that windows has disabled
> the GPU saying there are not enough resources to run the card.  From
> this point onwards I can do nothing to get the gpu working again
> aside from removing the device, manually deleting the drivers, and
> starting again.
> 
> Note that at this point I am only trying to do secondary passthrough
> though I would ideally like to get to the point of doing primary
> passthrough.  Adding the line gfx_passthru=1 at any point in all
> this to the machine config however just prevents the VM from booting
> entirely; when I VNC in all I get is a qemu prompt and I never get


I am not sure if this the same problem I've been seeing is but every time I
restart a Windows VM with GPU passthrough it crashed. I found that if "Eject"
the GPU before shutting down it would work just great the next time I started
it.

Try that and see if that works. This is btw with an unmodified Xen 4.1 and
Fedora's 3.4.something kernel.

> any ouput to the real display.
> 
> Any suggestions as to how to get this to work would be greatly
> appreciated as I've hit a bit of a brick wall.  I should also say
> that I have managed to get secondary passthrough working using
> Debian Wheezy and the repository version of xen 4.1.  In that case
> though Dom0 didn't boot reliably.
> 
> Sincerely,
> 
> Matthew Dean

> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: GPU passthrough with Xen 4.2 on Ubuntu 12.04

[Konrad Rzeszutek Wilk]

On Wed, Aug 08, 2012 at 04:42:08PM +0300, Pasi Kärkkäinen wrote:
> On Wed, Aug 08, 2012 at 02:26:33PM +0100, Matthew Dean wrote:
> > 
> >    I've used the stock Ubuntu provided kernel which is version
> >    3.2.0-27-generic.  Dom0 is connected to a display via the intel integrated
> >    graphics.
> > 
> 
> You might want to try Linux 3.4.x or 3.5.x dom0 kernels aswell.

The v3.5.x has some issues right now with PCI passthrough. Not sure what - but
trying to figure it out.

> I think there has been some fixes in xen-pciback.
> 
> 
> > 
> >    When I boot the VM everything seems to work fine.  Windows picks up all
> >    three devices and I can install drivers for them.  The USB controller
> >    works flawlessly and I can use an attached mouse and keyboard.  The radeon
> >    card requires a restart after which an attached display comes to life and
> >    I have 3D acceleration.  Restarting the VM seems to work OK.  If however I
> >    shut down the VM I have no end of problems.  On any subsequent startup the
> >    vm struggles to get past the windows splash screen, waiting much longer
> >    than usual.  During this period dom0 is sluggish regarding mouse and
> >    keyboard input even though cpu and memory usage are very low.  When
> >    windows finally loads I have no active display and I have to view the VM
> >    via VNC.  In device manage I find that windows has disabled the GPU saying
> >    there are not enough resources to run the card.  From this point onwards I
> >    can do nothing to get the gpu working again aside from removing the
> >    device, manually deleting the drivers, and starting again.
> > 
> 
> Do you get any errors from Xen (xl dmesg), or from dom0 kernel (dmesg) ? 
> 
> Do you have a serial console? 
> 
> 
> >    Note that at this point I am only trying to do secondary passthrough
> >    though I would ideally like to get to the point of doing primary
> >    passthrough.  Adding the line gfx_passthru=1 at any point in all this to
> >    the machine config however just prevents the VM from booting entirely;
> >    when I VNC in all I get is a qemu prompt and I never get any ouput to the
> >    real display.
> > 
> 
> AMD/ATI primary passthru requires extra patches to Xen qemu-dm,
> those are not included out-of-the-box yet.
> 
> >    Any suggestions as to how to get this to work would be greatly appreciated
> >    as I've hit a bit of a brick wall.  I should also say that I have managed
> >    to get secondary passthrough working using Debian Wheezy and the
> >    repository version of xen 4.1.  In that case though Dom0 didn't boot
> >    reliably.
> > 
> 
> -- Pasi
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel