Fwd: Re: Xen 4.3 / 4.4 - concurrent APIs, VGA Passthru

Fwd: Re: Xen 4.3 / 4.4 - concurrent APIs, VGA Passthru

[Georg Bege]

[-- Attachment #1: Type: text/plain, Size: 1748 bytes --]

Im forwading my email, to the mailing list again -
maybe someone else has something to contribute to my problem.

Nobody else would know better, except the developers themselfs. ;)


-------- Original-Nachricht --------
Betreff: 	Re: [Xen-devel] Xen 4.3 / 4.4 - concurrent APIs, VGA Passthru
Datum: 	Mon, 28 Jul 2014 05:14:22 +0200
Von: 	Georg Bege <therion@ninth-art.de>
Antwort an: 	therion@ninth-art.de
An: 	jacek burghardt <jaceksburghardt@gmail.com>



Hi

Thanks for your response, I can imagine hundred of people asked the same
questions already -
but as said it was hard to get an up to date documentation about this.
However I did experiment more with it, my graphics card is an GeForce
GTX470.
I came to the conclusion that passthru works great with XP (both 32bit
and 64bit) but giving me issue's whenever Im trying on Windows 7 64bit.

It seems that vga passthru doesnt work for the newer qemu-xen model
(using Qemu 2.0.0), so Im using
qemu-xen-traditional along with the old bundled qemu-dm which works fine
for XP.

I'll attach my hvm config Im using right now - I'll basically use it for
both XP and Win7.
I've also a different HVM for other HVM's where I rather use qemu 2.0.0
and Spice, but this is a different topic (nothing todo with VGA Passthru).
Basically Im on Gentoo and using Xen 4.4.0-r7, but changed the xen-tools
ebuild slightly as it would not build the traditional qemu by default.

So any idea why this is not working right for Win7?
Win7 also seems to be a lot slower with qemu-xen-traditional, is that a
bug in the old model?

regards,
Georg

Am 28.07.2014 04:47, schrieb jacek burghardt:
> Well we all pass our video cards as secondary display.  *gfx_passtru
> does not work. What video card do you have ?*
>
>




[-- Attachment #2: Win7gaming.hvm --]
[-- Type: text/plain, Size: 648 bytes --]

name = "Win7gaming"
builder = "hvm"
uuid = "9f9a391d-23d0-49c2-bd5e-47c6ad4ca104"
maxmem = 8192
memory = 8192
shadow_memory = 32
pae = 1
apic = 1
acpi = 1
vcpus = 4
nomigrate = 1
xen_platform_pci = 1
pci_msitranslate = 0
pci_power_mgmt = 1
device_model_version = "qemu-xen-traditional"
device_model_override = "/usr/lib/xen/bin/qemu-dm"
vif = [ 'bridge=xenbr0,mac=00:16:3e:06:48:60' ]
disk = ['phy:/dev/zvol/vdisks/Win7gaming,hda,w']
boot = "c"
sdl = 0
vnc = 1
vnclisten = '127.0.0.1'
stdvga = 0
audio = 0
localtime = 1
on_crash = 'destroy'
on_reboot = 'restart'
usb = 1
usbdevice = 'tablet'
keymap = 'de'
gfx_passthru=0
pci=['03:00.0', '03:00.1']

[-- Attachment #3: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Fwd: Re: Xen 4.3 / 4.4 - concurrent APIs, VGA Passthru

[jacek burghardt]

[-- Attachment #1.1: Type: text/plain, Size: 309 bytes --]

I use windows 8.1 under xen 4.5 with qemu git and seabios git. It works
very well. I can shutdown and reboot all day, but i cant reboot it makes
windows 8 try to repair itself. It seems that with win 7 you need to eject
your video card. There is a script for that.
I hope that vga=none will get more patching

[-- Attachment #1.2: Type: text/html, Size: 345 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Fwd: Re: Xen 4.3 / 4.4 - concurrent APIs, VGA Passthru

[Gordan Bobic]

On 07/28/2014 03:10 PM, Georg Bege wrote:

> Thanks for your response, I can imagine hundred of people asked the same
> questions already -
> but as said it was hard to get an up to date documentation about this.
> However I did experiment more with it, my graphics card is an GeForce
> GTX470.
> I came to the conclusion that passthru works great with XP (both 32bit
> and 64bit) but giving me issue's whenever Im trying on Windows 7 64bit.

Unless something major has changed very recently, an unmodified GTX470 
won't work passed through to a domU using unmodified drivers.

I am using Xen 4.3.0, and am using a pair of 780Ti cards at the moment, 
but this is a relatively recent upgrade from 470/480 cards.

GTX470 can be modified into a Quadro 5000 using a small BIOS change 
(google it, I'm sure you'll find information about it), after which it 
will work just fine passed through to a domU.

Having said that - it has recently been reported that various QEMU 
patches have neutered various ways the Nvidia driver uses to check 
whether it is running in a VM. Including things as lame as looking for 
QEMU's CPU model name/number override. Unsurprisingly, once you prevent 
the driver from detecting it is running in a VM, the card works fine.

One thing you might want to try is an older Nvidia driver. There has 
been a lot of cat-and-mouse game going on with QEMU getting patched to 
neuter checks for whether the stack is running virtualized, and Nvidia 
introducing new checks for it to attempt to continue making the GeForce 
cards not work virtualized.

But if you are not afraid of flipping a few bits on your GTX470 to turn 
it into a Quadro 5000, that is probably the easiest option that 
debugging the issue. IIRC was using my modified GTX470 and 480 cards 
with 320.xx and 331.xx drivers without problems.

> It seems that vga passthru doesnt work for the newer qemu-xen model
> (using Qemu 2.0.0), so Im using
> qemu-xen-traditional along with the old bundled qemu-dm which works fine
> for XP.

I am using qemu traditional on my system. I have two VMs running all the 
time, one running XP64, the other running 64-bit Windows. I have 
observed no issues at all with either guest OS.

> So any idea why this is not working right for Win7?
> Win7 also seems to be a lot slower with qemu-xen-traditional, is that a
> bug in the old model?

It could be something in Xen 4.4. I am using 4.3 with no problems in a 
similar setup. The machine runs 24/7 and the VMs are always running as 
they are different "workstations". Everything works exactly as expected, 
including VM rebooting, and performance is close enough to native to not 
impede gaming (Crysis 3 at 2560x1600, Borderlands 2 at 3840x2400).

Gordan

Re: Fwd: Re: Xen 4.3 / 4.4 - concurrent APIs, VGA Passthru

[Georg Bege]

Hi

Am 29.07.2014 12:38, schrieb Gordan Bobic:
> BTW, why off-list?
>
> Replies inline below.
Oh sorry, my mistake.

>
> On 2014-07-29 08:39, Georg Bege wrote:
>> Hi
>>
>> Am 29.07.2014 09:10, schrieb Gordan Bobic:
>>> On 07/29/2014 02:12 AM, Georg Bege wrote:
>>>> Hi Again,
>>>>
>>>> no with Xen 4.3 its not working to me, in Win XP64 I get kinda
>>>> abstract
>>>> distortions in the screen,
>>>
>>> That sounds suspiciously like the IOMEM memory stomp I see with my
>>> motherboard.
>>>
>>>> not much later then I get a kernel panic on Dom0 regarding IRQ16 (like
>>>> sth IRC16: no body cared.. oops) the kernel gets damaged by it and
>>>> then
>>>> the whole machine works like in slow-motion (everything X11, moving of
>>>> the mouse etc.) all I can do then is hard reboot...
>>>
>>> I have seen the interrupt issue before but _only_ when I xl destroy
>>> the domUs. It never happened on a clean shutdown/reboot of domUs. The
>>> interrupt you mention - does it happen to be the IRQ used by your USB
>>> controller?
>> Im not sure but the IRQ 16 is used by:
>>  16:    2065542          0          0          0          0
>> 0          0          0  xen-pirq-ioapic-level  ehci_hcd:usb1,
>> snd_emu10k1, nouveau
>
> ehci_hcd:usb1 sounds like USB.
>
> If you look at lspci -vvv, look for "IRQ 16". It should match.
>
Yes you're right, it is - however Im quite suprised how many devices
share that same Interrupt.

Interrupt: pin A routed to IRQ 16:

00:1a.0 USB controller: Intel Corporation C600/X79 series chipset USB2
Enhanced Host Controller #2 (rev 05) (prog-if 20 [EHCI])
02:00.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host
Controller (rev 04) (prog-if 30 [XHCI])
03:00.0 VGA compatible controller: NVIDIA Corporation GF100GL [Quadro
5000] (rev a3) (prog-if 00 [VGA controller])
04:00.0 VGA compatible controller: NVIDIA Corporation GT218 [GeForce
210] (rev a2) (prog-if 00 [VGA controller])
05:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic
SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
0a:00.0 Multimedia audio controller: Creative Labs SB Live! EMU10k1 (rev 04)

Is this normal?
I've to check out the other stuff, I'll reply again once I did that -
but might take till tomorrow.

Georg

Re: Fwd: Re: Xen 4.3 / 4.4 - concurrent APIs, VGA Passthru

[Gordan Bobic]

On 2014-07-29 12:52, Georg Bege wrote:
> Am 29.07.2014 12:38, schrieb Gordan Bobic:
>> On 2014-07-29 08:39, Georg Bege wrote:
>>> Am 29.07.2014 09:10, schrieb Gordan Bobic:
>>>> On 07/29/2014 02:12 AM, Georg Bege wrote:

>>>>> no with Xen 4.3 its not working to me, in Win XP64 I get kinda
>>>>> abstract
>>>>> distortions in the screen,
>>>> 
>>>> That sounds suspiciously like the IOMEM memory stomp I see with my
>>>> motherboard.
>>>> 
>>>>> not much later then I get a kernel panic on Dom0 regarding IRQ16 
>>>>> (like
>>>>> sth IRC16: no body cared.. oops) the kernel gets damaged by it and
>>>>> then
>>>>> the whole machine works like in slow-motion (everything X11, moving 
>>>>> of
>>>>> the mouse etc.) all I can do then is hard reboot...
>>>> 
>>>> I have seen the interrupt issue before but _only_ when I xl destroy
>>>> the domUs. It never happened on a clean shutdown/reboot of domUs. 
>>>> The
>>>> interrupt you mention - does it happen to be the IRQ used by your 
>>>> USB
>>>> controller?
>>> Im not sure but the IRQ 16 is used by:
>>>  16:    2065542          0          0          0          0
>>> 0          0          0  xen-pirq-ioapic-level  ehci_hcd:usb1,
>>> snd_emu10k1, nouveau
>> 
>> ehci_hcd:usb1 sounds like USB.
>> 
>> If you look at lspci -vvv, look for "IRQ 16". It should match.
>> 
> Yes you're right, it is - however Im quite suprised how many devices
> share that same Interrupt.
> 
> Interrupt: pin A routed to IRQ 16:
> 
> 00:1a.0 USB controller: Intel Corporation C600/X79 series chipset USB2
> Enhanced Host Controller #2 (rev 05) (prog-if 20 [EHCI])
> 02:00.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host
> Controller (rev 04) (prog-if 30 [XHCI])
> 03:00.0 VGA compatible controller: NVIDIA Corporation GF100GL [Quadro
> 5000] (rev a3) (prog-if 00 [VGA controller])
> 04:00.0 VGA compatible controller: NVIDIA Corporation GT218 [GeForce
> 210] (rev a2) (prog-if 00 [VGA controller])
> 05:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic
> SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
> 0a:00.0 Multimedia audio controller: Creative Labs SB Live! EMU10k1 
> (rev 04)
> 
> Is this normal?

It sounds a bit high, but it's not implausible.

On my machine this shows:
for irq in `lspci -vvv | grep IRQ | sed -e 's/.*IRQ //' | sort -un`; do
   echo -n "IRQ $irq: ";
   lspci -vvv | grep IRQ | sed -e 's/.*IRQ //' | grep "^$irq$" | wc -l;
done

IRQ 16: 1
IRQ 18: 3
IRQ 19: 2
IRQ 21: 1
IRQ 22: 1
IRQ 23: 2
IRQ 24: 2
IRQ 29: 1
IRQ 36: 1
IRQ 37: 1
IRQ 38: 1
IRQ 39: 1
IRQ 40: 1
IRQ 212: 1
IRQ 213: 1
IRQ 214: 1
IRQ 215: 1

So IRQ 18 is used by 3 devices, all of which are on the
ICH10 SB chip (2x USB, 1x SMBus).

What surprises me slightly more is the devices that are
sharing IRQs on your system. It almost looks like at least 4
PCIe slots are sharing the same interrupt, which is somewhat
unusual.

> I've to check out the other stuff, I'll reply again once I did that -
> but might take till tomorrow.

Please do. I'd be interested to hear if you are in fact
hitting the same bug as me, as it is rather obscure and
unusual.

Gordan

Re: Fwd: Re: Xen 4.3 / 4.4 - concurrent APIs, VGA Passthru

[Georg Bege]

Hello again

Well so far I can now tell, I tested it on WinXP x64 quite a lot
- most things are working.
Including AAA games like Witcher 2, I didnt try a very memory consuming
game yet.
But I dont really think that I have the same bug you are refering to -
in WinXP x64 the /PATCHTPR boot flag did help greatly, it really boosted
the performance to near native.
I dont think I encountered any memory usage, used like applications
consuming the first 4GB of the DomU.

Im trying to figure what to do for Windows 7, its working good too -
even with VGA Passthrough,
but its almost always consuming 33%-80% CPU usage for no reason...

I also stumpled upon the issue that if you reboot a VGA Passthru DomU
then that the GFX adapter looses performance (which was explained
somewhere in the docs already).
Atm. all this is done with Xen 4.4 - Im not sure why I would go back to
Xen 4.3, Xen is improving greatly over each version I think, that older
versions have more (unfixed) issues is quite logical.

I just wonder what I can do for Win7 performance wise so that I can test
things like Borderlands 2 you referred to... or any other high memory
usage application.

regards,
Georg

Am 29.07.2014 14:22, schrieb Gordan Bobic:
> On 2014-07-29 12:52, Georg Bege wrote:
>> Am 29.07.2014 12:38, schrieb Gordan Bobic:
>>> On 2014-07-29 08:39, Georg Bege wrote:
>>>> Am 29.07.2014 09:10, schrieb Gordan Bobic:
>>>>> On 07/29/2014 02:12 AM, Georg Bege wrote:
>
>>>>>> no with Xen 4.3 its not working to me, in Win XP64 I get kinda
>>>>>> abstract
>>>>>> distortions in the screen,
>>>>>
>>>>> That sounds suspiciously like the IOMEM memory stomp I see with my
>>>>> motherboard.
>>>>>
>>>>>> not much later then I get a kernel panic on Dom0 regarding IRQ16
>>>>>> (like
>>>>>> sth IRC16: no body cared.. oops) the kernel gets damaged by it and
>>>>>> then
>>>>>> the whole machine works like in slow-motion (everything X11,
>>>>>> moving of
>>>>>> the mouse etc.) all I can do then is hard reboot...
>>>>>
>>>>> I have seen the interrupt issue before but _only_ when I xl destroy
>>>>> the domUs. It never happened on a clean shutdown/reboot of domUs. The
>>>>> interrupt you mention - does it happen to be the IRQ used by your USB
>>>>> controller?
>>>> Im not sure but the IRQ 16 is used by:
>>>>  16:    2065542          0          0          0          0
>>>> 0          0          0  xen-pirq-ioapic-level  ehci_hcd:usb1,
>>>> snd_emu10k1, nouveau
>>>
>>> ehci_hcd:usb1 sounds like USB.
>>>
>>> If you look at lspci -vvv, look for "IRQ 16". It should match.
>>>
>> Yes you're right, it is - however Im quite suprised how many devices
>> share that same Interrupt.
>>
>> Interrupt: pin A routed to IRQ 16:
>>
>> 00:1a.0 USB controller: Intel Corporation C600/X79 series chipset USB2
>> Enhanced Host Controller #2 (rev 05) (prog-if 20 [EHCI])
>> 02:00.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host
>> Controller (rev 04) (prog-if 30 [XHCI])
>> 03:00.0 VGA compatible controller: NVIDIA Corporation GF100GL [Quadro
>> 5000] (rev a3) (prog-if 00 [VGA controller])
>> 04:00.0 VGA compatible controller: NVIDIA Corporation GT218 [GeForce
>> 210] (rev a2) (prog-if 00 [VGA controller])
>> 05:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic
>> SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
>> 0a:00.0 Multimedia audio controller: Creative Labs SB Live! EMU10k1
>> (rev 04)
>>
>> Is this normal?
>
> It sounds a bit high, but it's not implausible.
>
> On my machine this shows:
> for irq in `lspci -vvv | grep IRQ | sed -e 's/.*IRQ //' | sort -un`; do
>   echo -n "IRQ $irq: ";
>   lspci -vvv | grep IRQ | sed -e 's/.*IRQ //' | grep "^$irq$" | wc -l;
> done
>
> IRQ 16: 1
> IRQ 18: 3
> IRQ 19: 2
> IRQ 21: 1
> IRQ 22: 1
> IRQ 23: 2
> IRQ 24: 2
> IRQ 29: 1
> IRQ 36: 1
> IRQ 37: 1
> IRQ 38: 1
> IRQ 39: 1
> IRQ 40: 1
> IRQ 212: 1
> IRQ 213: 1
> IRQ 214: 1
> IRQ 215: 1
>
> So IRQ 18 is used by 3 devices, all of which are on the
> ICH10 SB chip (2x USB, 1x SMBus).
>
> What surprises me slightly more is the devices that are
> sharing IRQs on your system. It almost looks like at least 4
> PCIe slots are sharing the same interrupt, which is somewhat
> unusual.
>
>> I've to check out the other stuff, I'll reply again once I did that -
>> but might take till tomorrow.
>
> Please do. I'd be interested to hear if you are in fact
> hitting the same bug as me, as it is rather obscure and
> unusual.
>
> Gordan

Re: Fwd: Re: Xen 4.3 / 4.4 - concurrent APIs, VGA Passthru

[Georg Bege]

Hi

I tried your suggestions now with Xen 4.3.2-r4 (Gentoo).
On Win7 64bit its not working at all, I cannot passthru the GFX -
I rolled back to a snapshot where I installed everything but not the
nvidia package.

So I tried it and it works out, reboots and then I get a message like:
"This device reported an error and had to be stopped, code 43."
within the device manager.
This happens whether Im trying the DomU with 8GB or 1024MB - same results.
On Xen 4.4 I can passthru the GFX, no problem but its very slow (as I've
written already).

On WinXP 64bit its not working either, there I get the ominious error
you think is an memory corruption regarding IOMMU.
Either the machine freezes immediadly or it breaks some memory tables (I
think in nouveau, because I got error messages regarding this in Dom0
dmesg) and everything is extreme slow-motion - I can barly do anything.
This happens no matter if I do 8GB in the DomU or just 1024MB - so I
couldnt really try what you suggested.
Because 1GB memory didnt fix it for me and this time it wasnt only due
xl destroy but immediadly once the system came up (the WinXP DomU).

However on Xen 4.4 at least the WinXP DomU is working fine, you warned
me already that this doesnt mean there wont be any hidden corruption at
all - but so far it has worked for me with numerous games/applications
already including copying a lot of data via samba.

regards,
Georg

Am 29.07.2014 12:38, schrieb Gordan Bobic:
> BTW, why off-list?
>
> Replies inline below.
>
> On 2014-07-29 08:39, Georg Bege wrote:
>> Hi
>>
>> Am 29.07.2014 09:10, schrieb Gordan Bobic:
>>> On 07/29/2014 02:12 AM, Georg Bege wrote:
>>>> Hi Again,
>>>>
>>>> no with Xen 4.3 its not working to me, in Win XP64 I get kinda
>>>> abstract
>>>> distortions in the screen,
>>>
>>> That sounds suspiciously like the IOMEM memory stomp I see with my
>>> motherboard.
>>>
>>>> not much later then I get a kernel panic on Dom0 regarding IRQ16 (like
>>>> sth IRC16: no body cared.. oops) the kernel gets damaged by it and
>>>> then
>>>> the whole machine works like in slow-motion (everything X11, moving of
>>>> the mouse etc.) all I can do then is hard reboot...
>>>
>>> I have seen the interrupt issue before but _only_ when I xl destroy
>>> the domUs. It never happened on a clean shutdown/reboot of domUs. The
>>> interrupt you mention - does it happen to be the IRQ used by your USB
>>> controller?
>> Im not sure but the IRQ 16 is used by:
>>  16:    2065542          0          0          0          0
>> 0          0          0  xen-pirq-ioapic-level  ehci_hcd:usb1,
>> snd_emu10k1, nouveau
>
> ehci_hcd:usb1 sounds like USB.
>
> If you look at lspci -vvv, look for "IRQ 16". It should match.
>
> Unfortunately, I never got to the bottom of what the root cause of
> the fault is. However, I only ever see it when:
>
> 1) I force destroy the domU (xl destroy).
>
> 2) The domU crashes due to the memory stomp when it tries to
> write to a memory address range that is in the PCI IOMEM range
> which due to a bug isn't getting remapped by the IOMMU.
>
> I thus assumed this was all related to the same IOMMU bug,
> but I am not 100% certain.
>
> Try a fresh, clean reboot and start your domU with only 1GB of RAM
> and see if the problem goes away.
>
> If that works OK, look at the output of the following:
> # lspci -vvv | grep "Region .: Memory at" | sed -e 's/.* Memory at //'
> | sort
>
> Look at the first value it returns. Convert that from hex to
> decimal, and set your domU memory to 1MB below that. Assuming
> that is more than 1GB you tried above (if it's less than 1GB,
> try it anyway) and it still works without a crash there is a
> good chance you are up against the same bug as me.
>
> For Windows 7 domU you could potentially work around the
> problem by using bcdedit to mark the IOMEM aperture memory
> blocks as faulty so the OS will never try to use it.
>
> My motherboard only knows how to assign IOMEM below 4GB,
> so I found it easier to just mark everything between 1GB
> and 4GB as "reserved" in the domU e820 map. If your BIOS
> knows how to map things above 4GB, you may need to use the
> mentioned trick of marking the memory areas as faulty.
>
> Caveat - I don't actually know what happens if the domU
> IOMEM mapping for the GPU overlaps an IOMEM mapping in dom0.
> Thankfully, QEMU doesn't map my GPUs to such IOMEM ranges
> so I never had to find out. There has been a proposal to
> introduce an e820_host option to HVM guests (it already
> exists in PV guests), which might help, but with the
> work recently going into PVH domains, this option may become
> available to work around the hardware bug I'm talking about.
> Either way, if your domU GPU apertures aren't overlapping
> any dom0 apertures, it should be fine.
>
>>> I always assumed that this was to do with other IOMEM issues to do
>>> with my motherboard, but since it only happens when I have to
>>> forcefully destroy domUs it doesn't bother me too much.
>>>
>>> What motherboard do you use?
>> Intel DX79SI
>>>
>>> How much RAM are you giving to your domUs? Can you check if it happens
>>> when you only give the domU 1GB of RAM?
>> I could try, maybe later on - right now Im on Xen 4.4 again, just got my
>> xp x64 running fine -
>> maybe I'll later also test Xen 4.5 on that box.
>
> The "running fine" can be deceptive. If this is the bug I thing it
> is, there is a good chance the problem will not manifest until you
> overrun the PCI IOMEM region. Try loading a program that fills up
> all or most of memory and see if that triggers weird artifacting
> and crashing. When I was doing my testing I found that firing up
> Borderlands 2 and waiting for the intro was a very easy way to cause
> the crash to occur. I'm sure there are many other games that are
> similarly resource intensive that will cause it to happen.
>
> One thing to be aware of, though - if you are unlucky and the
> IOMEM stomp clobbers the aperture used by your disk controller
> you could end up with data corrupted on the disk.
>
>> Btw. I read this in advance, just realized you are the same guy
>> http://xen.1045712.n5.nabble.com/GPU-passthrough-on-Xen-4-4-0-FLReset-td5722964.html
>>
>
> Yes.
>
> Gordan

Re: Fwd: Re: Xen 4.3 / 4.4 - concurrent APIs, VGA Passthru

[Gordan Bobic]

On 08/03/2014 02:09 AM, Georg Bege wrote:
> Hi
>
> I tried your suggestions now with Xen 4.3.2-r4 (Gentoo).
> On Win7 64bit its not working at all, I cannot passthru the GFX -
> I rolled back to a snapshot where I installed everything but not the
> nvidia package.

Just to clarify - which suggestion is that?

> So I tried it and it works out, reboots and then I get a message like:
> "This device reported an error and had to be stopped, code 43."
> within the device manager.
> This happens whether Im trying the DomU with 8GB or 1024MB - same results.
> On Xen 4.4 I can passthru the GFX, no problem but its very slow (as I've
> written already).

Have you tried an older Nvidia driver? I'm running the latest 331.x 
driver on both XP and Win7 (both x64).

The slowness might be indicative of an interrupt related problem. The 
boot parameters I'm running with are as follows:

xen.gz: dom0_vcpus_pin iommu=dom0-passthrough unrestricted_guest=1 msi=1
vmlinuz: intel_iommu=on pcie_aspm=off pcie_ports=compat

Note: The pcie_aspm=off pcie_ports=compat are to work around a bug on my 
motherboard that causes a massive interrupt flood due to a device on 
ICH10 flapping between hotplug states and creating an interrupt storm 
that makes the entire machine permanently unusable as soon as the kernel 
starts probing devices. You probably don't need those two options but 
they shouldn't do any harm while you're troubleshooting.

Gordan

Re: Fwd: Re: Xen 4.3 / 4.4 - concurrent APIs, VGA Passthru

[Gordan Bobic]

On 08/02/2014 11:49 AM, Georg Bege wrote:
> Hello again
>
> Well so far I can now tell, I tested it on WinXP x64 quite a lot
> - most things are working.
> Including AAA games like Witcher 2, I didnt try a very memory consuming
> game yet.
> But I dont really think that I have the same bug you are refering to -
> in WinXP x64 the /PATCHTPR boot flag did help greatly, it really boosted
> the performance to near native.

What does this flag do? Googling for it found no results.

> I dont think I encountered any memory usage, used like applications
> consuming the first 4GB of the DomU.

I am inclined to agree, you aren't hitting the same problem.

> Im trying to figure what to do for Windows 7, its working good too -
> even with VGA Passthrough,
> but its almost always consuming 33%-80% CPU usage for no reason...

Can you check from within the domU which process is eating CPU? Or are 
you saying that with Windows 7 inside your domU task manager is showing 
0% CPU usage but xentop is showing 33-80% CPU usage?

> I also stumpled upon the issue that if you reboot a VGA Passthru DomU
> then that the GFX adapter looses performance (which was explained
> somewhere in the docs already).

That is _very_ wrong. That only happens on ATI GPUs, I have never seen 
that happen with an Nvidia GPU.

> Atm. all this is done with Xen 4.4 - Im not sure why I would go back to
> Xen 4.3, Xen is improving greatly over each version I think, that older
> versions have more (unfixed) issues is quite logical.

IMO that doesn't follow for any software. More feature doesn't mean 
fewer bugs, usually the opposite.

> I just wonder what I can do for Win7 performance wise so that I can test
> things like Borderlands 2 you referred to... or any other high memory
> usage application.

Borderlands 2 runs under XP64 just fine. I'm not sure what else to 
suggest, I rebuilt one of my two XP64 VMs to Win7 x64 the other day 
relatively painlessly. Same config, just different disk image. No 
problems so far.

Gordan

Re: Fwd: Re: Xen 4.3 / 4.4 - concurrent APIs, VGA Passthru

[James Harper]

> But I dont really think that I have the same bug you are refering to -
> in WinXP x64 the /PATCHTPR boot flag did help greatly, it really boosted
> the performance to near native.

If you are referring to GPLPV, I don't think PATCHTPR does anything on 64 bit OS. It's only for 32 bit OS, and only XP and 2003sp1 or older.

That said, I've never actually used any of the 64 bit versions of XP, so if you have done conclusive testing I'd like to know. I thought I'd excluded the patch from 64 bit builds but I just checked the code and that's not the case, but I don't know that the opcodes I search for are the same in a 64 bit kernel...

James

Fwd: Re: Fwd: Re: Xen 4.3 / 4.4 - concurrent APIs, VGA Passthru

[Georg Bege]

[-- Attachment #1.1: Type: text/plain, Size: 3103 bytes --]




-------- Original-Nachricht --------
Betreff: 	Re: [Xen-devel] Fwd: Re: Xen 4.3 / 4.4 - concurrent APIs, VGA
Passthru
Datum: 	Sun, 03 Aug 2014 12:09:55 +0200
Von: 	Georg Bege <therion@ninth-art.de>
Antwort an: 	therion@ninth-art.de
An: 	Gordan Bobic <gordan@bobich.net>



Am 03.08.2014 11:49, schrieb Gordan Bobic:
> On 08/02/2014 11:49 AM, Georg Bege wrote:
>> Hello again
>>
>> Well so far I can now tell, I tested it on WinXP x64 quite a lot
>> - most things are working.
>> Including AAA games like Witcher 2, I didnt try a very memory consuming
>> game yet.
>> But I dont really think that I have the same bug you are refering to -
>> in WinXP x64 the /PATCHTPR boot flag did help greatly, it really boosted
>> the performance to near native.
>
> What does this flag do? Googling for it found no results.    
See:
http://lists.xenproject.org/archives/html/xen-users/2013-09/msg00254.html

But now I came to the conclusion that I dont have the same issue with
Win7 at all (see last email).

>
>> I dont think I encountered any memory usage, used like applications
>> consuming the first 4GB of the DomU.
>
> I am inclined to agree, you aren't hitting the same problem.
>
>> Im trying to figure what to do for Windows 7, its working good too -
>> even with VGA Passthrough,
>> but its almost always consuming 33%-80% CPU usage for no reason...
>
> Can you check from within the domU which process is eating CPU? Or are
> you saying that with Windows 7 inside your domU task manager is
> showing 0% CPU usage but xentop is showing 33-80% CPU usage?
I cant check, as states in the last email it must be about interrupts or
power saving states - it only happens when I pass in the GFX.
Without this the CPU is just idling on 0% inside the DomU.

>
>> I also stumpled upon the issue that if you reboot a VGA Passthru DomU
>> then that the GFX adapter looses performance (which was explained
>> somewhere in the docs already).
>
> That is _very_ wrong. That only happens on ATI GPUs, I have never seen
> that happen with an Nvidia GPU.
Interesting information, thank you - if that is so then maybe Im still
hitting some other problems on WinXP as well - but I must say, Im
tinkering a lot so I have to re-verify this.
>
>> Atm. all this is done with Xen 4.4 - Im not sure why I would go back to
>> Xen 4.3, Xen is improving greatly over each version I think, that older
>> versions have more (unfixed) issues is quite logical.
>
> IMO that doesn't follow for any software. More feature doesn't mean
> fewer bugs, usually the opposite.
Ya maybe, as said Xen 4.3 isnt really working at all to me - as Im not
even able to passthru my GFX (err code 43) so I cannot even do the same
tests.

>
>> I just wonder what I can do for Win7 performance wise so that I can test
>> things like Borderlands 2 you referred to... or any other high memory
>> usage application.
>
> Borderlands 2 runs under XP64 just fine. I'm not sure what else to
> suggest, I rebuilt one of my two XP64 VMs to Win7 x64 the other day
> relatively painlessly. Same config, just different disk image. No
> problems so far.
>
> Gordan




[-- Attachment #1.2: Type: text/html, Size: 4750 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Fwd: Re: Xen 4.3 / 4.4 - concurrent APIs, VGA Passthru

[Gordan Bobic]

On 08/03/2014 11:09 AM, Georg Bege wrote:
>
> Am 03.08.2014 11:49, schrieb Gordan Bobic:
>> On 08/02/2014 11:49 AM, Georg Bege wrote:
>>> Hello again
>>>
>>> Well so far I can now tell, I tested it on WinXP x64 quite a lot
>>> - most things are working.
>>> Including AAA games like Witcher 2, I didnt try a very memory consuming
>>> game yet.
>>> But I dont really think that I have the same bug you are refering to -
>>> in WinXP x64 the /PATCHTPR boot flag did help greatly, it really boosted
>>> the performance to near native.
>>
>> What does this flag do? Googling for it found no results.
> See:
> http://lists.xenproject.org/archives/html/xen-users/2013-09/msg00254.html
>
> But now I came to the conclusion that I dont have the same issue with
> Win7 at all (see last email).

This reminds me - I had issues with the latest GPLPV drivers on Windows 
7. The installer would just sit there and never complete, which also, 
IIRC, left the VM OS in a dodgy state. So I nstalled the same version 
(version number, not package, obviously) I've been running on XP64 for 
ages (11.0.372) and that worked fine.

>>> I dont think I encountered any memory usage, used like applications
>>> consuming the first 4GB of the DomU.
>>
>> I am inclined to agree, you aren't hitting the same problem.
>>
>>> Im trying to figure what to do for Windows 7, its working good too -
>>> even with VGA Passthrough,
>>> but its almost always consuming 33%-80% CPU usage for no reason...
>>
>> Can you check from within the domU which process is eating CPU? Or are
>> you saying that with Windows 7 inside your domU task manager is
>> showing 0% CPU usage but xentop is showing 33-80% CPU usage?
> I cant check, as states in the last email it must be about interrupts or
> power saving states - it only happens when I pass in the GFX.
> Without this the CPU is just idling on 0% inside the DomU.

Funnily enough, I'm seeing something similar in dom0 when I use a GT630 
card, but not when I'm using an 8800GT or Q2K. With the GT630, Xorg 
locks up immediately, eating 100% of CPU with ksoftirqd eating another 
100% of CPU. X process is unkillable so the only way to recover is to 
reboot the machine. The same GT630 runs fine in another machine so I'm 
reasonably sure that's not the problem. So it _could_ be a similar 
obscure bug in the Nvidia driver.

Have you tried 331.93? That's the driver I'm using.


Gordan

Re: Fwd: Re: Xen 4.3 / 4.4 - concurrent APIs, VGA Passthru

[Georg Bege]

Hi


Am 04.08.2014 10:18, schrieb Gordan Bobic:
> On 08/03/2014 11:09 AM, Georg Bege wrote:
>>
>> Am 03.08.2014 11:49, schrieb Gordan Bobic:
>>> On 08/02/2014 11:49 AM, Georg Bege wrote:
>>>> Hello again
>>>>
>>>> Well so far I can now tell, I tested it on WinXP x64 quite a lot
>>>> - most things are working.
>>>> Including AAA games like Witcher 2, I didnt try a very memory
>>>> consuming
>>>> game yet.
>>>> But I dont really think that I have the same bug you are refering to -
>>>> in WinXP x64 the /PATCHTPR boot flag did help greatly, it really
>>>> boosted
>>>> the performance to near native.
>>>
>>> What does this flag do? Googling for it found no results.
>> See:
>> http://lists.xenproject.org/archives/html/xen-users/2013-09/msg00254.html
>>
>>
>> But now I came to the conclusion that I dont have the same issue with
>> Win7 at all (see last email).
>
> This reminds me - I had issues with the latest GPLPV drivers on
> Windows 7. The installer would just sit there and never complete,
> which also, IIRC, left the VM OS in a dodgy state. So I nstalled the
> same version (version number, not package, obviously) I've been
> running on XP64 for ages (11.0.372) and that worked fine.
>
>>>> I dont think I encountered any memory usage, used like applications
>>>> consuming the first 4GB of the DomU.
>>>
>>> I am inclined to agree, you aren't hitting the same problem.
>>>
>>>> Im trying to figure what to do for Windows 7, its working good too -
>>>> even with VGA Passthrough,
>>>> but its almost always consuming 33%-80% CPU usage for no reason...
>>>
>>> Can you check from within the domU which process is eating CPU? Or are
>>> you saying that with Windows 7 inside your domU task manager is
>>> showing 0% CPU usage but xentop is showing 33-80% CPU usage?
>> I cant check, as states in the last email it must be about interrupts or
>> power saving states - it only happens when I pass in the GFX.
>> Without this the CPU is just idling on 0% inside the DomU.
>
> Funnily enough, I'm seeing something similar in dom0 when I use a
> GT630 card, but not when I'm using an 8800GT or Q2K. With the GT630,
> Xorg locks up immediately, eating 100% of CPU with ksoftirqd eating
> another 100% of CPU. X process is unkillable so the only way to
> recover is to reboot the machine. The same GT630 runs fine in another
> machine so I'm reasonably sure that's not the problem. So it _could_
> be a similar obscure bug in the Nvidia driver.
>
> Have you tried 331.93? That's the driver I'm using.
I'd like to agree on that point, however that slowdown occurs already if
I just passthru the GFX >without< installing any drivers.
But I'll do some more checks on that...
>
>
> Gordan

Re: Fwd: Re: Xen 4.3 / 4.4 - concurrent APIs, VGA Passthru

[Georg Bege]

Hi

I simply wanted to tell my tale about the outcome of my experiments,
I was occupied with certain other things (including exchanging some
zpool's).
For now I run an Vista x64 and it works pretty well, I never got the
gplpv drivers running on
acceptable levels (they are still quite slow) so I decided to pass
onboard controller, sound and usb3.
This decission seemed to be a wise one and everything works great as
expected, the problem with gplpv
well it might also be a result of my volume scheme since I only run ZFS
- now an raidz1 (on enterprise disks though).
I also replaced the nvidia drivers from 331.65 to 337.88 - this revision
gave me a lot more performance but I can remember I had this issue on
native Windows too.

So after a lot of testing, pain, time consuming days (and nights) its
really working great - not perfect maybe.
My next goal is to dedicate an SSD for that system volume, maybe try
Win7 again as well -
also I'd like to get an GTX690 and hardmod it so I get a lot more speed...

regards,
Georg

Am 04.08.2014 10:18, schrieb Gordan Bobic:
> On 08/03/2014 11:09 AM, Georg Bege wrote:
>>
>> Am 03.08.2014 11:49, schrieb Gordan Bobic:
>>> On 08/02/2014 11:49 AM, Georg Bege wrote:
>>>> Hello again
>>>>
>>>> Well so far I can now tell, I tested it on WinXP x64 quite a lot
>>>> - most things are working.
>>>> Including AAA games like Witcher 2, I didnt try a very memory
>>>> consuming
>>>> game yet.
>>>> But I dont really think that I have the same bug you are refering to -
>>>> in WinXP x64 the /PATCHTPR boot flag did help greatly, it really
>>>> boosted
>>>> the performance to near native.
>>>
>>> What does this flag do? Googling for it found no results.
>> See:
>> http://lists.xenproject.org/archives/html/xen-users/2013-09/msg00254.html
>>
>>
>> But now I came to the conclusion that I dont have the same issue with
>> Win7 at all (see last email).
>
> This reminds me - I had issues with the latest GPLPV drivers on
> Windows 7. The installer would just sit there and never complete,
> which also, IIRC, left the VM OS in a dodgy state. So I nstalled the
> same version (version number, not package, obviously) I've been
> running on XP64 for ages (11.0.372) and that worked fine.
>
>>>> I dont think I encountered any memory usage, used like applications
>>>> consuming the first 4GB of the DomU.
>>>
>>> I am inclined to agree, you aren't hitting the same problem.
>>>
>>>> Im trying to figure what to do for Windows 7, its working good too -
>>>> even with VGA Passthrough,
>>>> but its almost always consuming 33%-80% CPU usage for no reason...
>>>
>>> Can you check from within the domU which process is eating CPU? Or are
>>> you saying that with Windows 7 inside your domU task manager is
>>> showing 0% CPU usage but xentop is showing 33-80% CPU usage?
>> I cant check, as states in the last email it must be about interrupts or
>> power saving states - it only happens when I pass in the GFX.
>> Without this the CPU is just idling on 0% inside the DomU.
>
> Funnily enough, I'm seeing something similar in dom0 when I use a
> GT630 card, but not when I'm using an 8800GT or Q2K. With the GT630,
> Xorg locks up immediately, eating 100% of CPU with ksoftirqd eating
> another 100% of CPU. X process is unkillable so the only way to
> recover is to reboot the machine. The same GT630 runs fine in another
> machine so I'm reasonably sure that's not the problem. So it _could_
> be a similar obscure bug in the Nvidia driver.
>
> Have you tried 331.93? That's the driver I'm using.
>
>
> Gordan

Re: Fwd: Re: Xen 4.3 / 4.4 - concurrent APIs, VGA Passthru

[Gordan Bobic]

On 08/18/2014 04:26 PM, Georg Bege wrote:
> Hi
>
> I simply wanted to tell my tale about the outcome of my experiments,
> I was occupied with certain other things (including exchanging some
> zpool's).
> For now I run an Vista x64 and it works pretty well, I never got the
> gplpv drivers running on
> acceptable levels (they are still quite slow) so I decided to pass
> onboard controller, sound and usb3.
> This decission seemed to be a wise one and everything works great as
> expected, the problem with gplpv
> well it might also be a result of my volume scheme since I only run ZFS
> - now an raidz1 (on enterprise disks though).
> I also replaced the nvidia drivers from 331.65 to 337.88 - this revision
> gave me a lot more performance but I can remember I had this issue on
> native Windows too.
>
> So after a lot of testing, pain, time consuming days (and nights) its
> really working great - not perfect maybe.
> My next goal is to dedicate an SSD for that system volume, maybe try
> Win7 again as well -
> also I'd like to get an GTX690 and hardmod it so I get a lot more speed...

Modifying it won't get you more speed.

I was originally planning to use a GTX690 and pass one GPU to each of my 
VMs, but I eventually replaced it with a pair of 780Tis.

I am running on a SSD backed zvol, with gplpv.

Gordan

Re: Fwd: Re: Xen 4.3 / 4.4 - concurrent APIs, VGA Passthru

[Richie]

On 8/18/2014 11:45 AM, Gordan Bobic wrote:
> On 08/18/2014 04:26 PM, Georg Bege wrote:
>> Hi
>>
>> I simply wanted to tell my tale about the outcome of my experiments,
>> I was occupied with certain other things (including exchanging some
>> zpool's).
>> For now I run an Vista x64 and it works pretty well, I never got the
>> gplpv drivers running on
>> acceptable levels (they are still quite slow) so I decided to pass
>> onboard controller, sound and usb3.
>> This decission seemed to be a wise one and everything works great as
>> expected, the problem with gplpv
>> well it might also be a result of my volume scheme since I only run ZFS
>> - now an raidz1 (on enterprise disks though).
>> I also replaced the nvidia drivers from 331.65 to 337.88 - this revision
>> gave me a lot more performance but I can remember I had this issue on
>> native Windows too.
>>
>> So after a lot of testing, pain, time consuming days (and nights) its
>> really working great - not perfect maybe.
>> My next goal is to dedicate an SSD for that system volume, maybe try
>> Win7 again as well -
>> also I'd like to get an GTX690 and hardmod it so I get a lot more
>> speed...
>
> Modifying it won't get you more speed.
>
> I was originally planning to use a GTX690 and pass one GPU to each of
> my VMs, but I eventually replaced it with a pair of 780Tis.
>
> I am running on a SSD backed zvol, with gplpv.
>
> Gordan
>

Do you mind posting the specifics on your setups?  I read as many 
threads with this topic as I could find in the archives. It sounds like 
you are using Xen 4.3.2, Vista 64, nonmodded GTX as secondary 
passthrough and older nvidia drivers.

Did you have to do any vBAR patching?  Are you loading a copy of the 
VGA's rom bios?   Yes those are things I had to do back in 2010, but 
maybe those things are not applicable anymore.   Thanks for any info.

Re: Fwd: Re: Xen 4.3 / 4.4 - concurrent APIs, VGA Passthru

[Gordan Bobic]

On 2014-08-18 22:13, Richie wrote:
> On 8/18/2014 11:45 AM, Gordan Bobic wrote:
>> On 08/18/2014 04:26 PM, Georg Bege wrote:
>>> Hi
>>> 
>>> I simply wanted to tell my tale about the outcome of my experiments,
>>> I was occupied with certain other things (including exchanging some
>>> zpool's).
>>> For now I run an Vista x64 and it works pretty well, I never got the
>>> gplpv drivers running on
>>> acceptable levels (they are still quite slow) so I decided to pass
>>> onboard controller, sound and usb3.
>>> This decission seemed to be a wise one and everything works great as
>>> expected, the problem with gplpv
>>> well it might also be a result of my volume scheme since I only run 
>>> ZFS
>>> - now an raidz1 (on enterprise disks though).
>>> I also replaced the nvidia drivers from 331.65 to 337.88 - this 
>>> revision
>>> gave me a lot more performance but I can remember I had this issue on
>>> native Windows too.
>>> 
>>> So after a lot of testing, pain, time consuming days (and nights) its
>>> really working great - not perfect maybe.
>>> My next goal is to dedicate an SSD for that system volume, maybe try
>>> Win7 again as well -
>>> also I'd like to get an GTX690 and hardmod it so I get a lot more
>>> speed...
>> 
>> Modifying it won't get you more speed.
>> 
>> I was originally planning to use a GTX690 and pass one GPU to each of
>> my VMs, but I eventually replaced it with a pair of 780Tis.
>> 
>> I am running on a SSD backed zvol, with gplpv.
>> 
>> Gordan
>> 
> 
> Do you mind posting the specifics on your setups?  I read as many
> threads with this topic as I could find in the archives. It sounds
> like you are using Xen 4.3.2, Vista 64, nonmodded GTX as secondary
> passthrough and older nvidia drivers.

Xen 4.3.0 for me - I had to patch my hvmloader to mark most of
the RAM between 1GB and 4GB as reserved due to a bug in my PCIe
multiplexers which break IOMMU operation (Nvidia NF200). I saw
no reason to upgrade since it meant re-patching. I will upgrade
to 4.4.x when the patches that limit RAM below 4GB are incorporated
since I should be able to make do with those instead of my gross
hack job.

I have two VMs, one is XP/64, the other is Win7/64. I am using
_modified_ GTX780Ti cards. Previously I was running a modified
680, and before that a modified 480 (4xx series and older cards
can be modified with just a small BIOS patch).

Using secondary passthrough and the latest 331.xx driver (the
exact version escapes me at the moment).

> Did you have to do any vBAR patching?

No.

> Are you loading a copy of the VGA's rom bios?

No. No BIOS level output from the GPU being passed through
until the driver loads and initializes it.

Gordan