Re: Issues with PCI-Passtrough (VT-d) in HVM with Xen 4.6

["Jürgen Walter" <juwalter@gmail.com>]

On 3 Jun 2016, at 13:52, Jan Beulich wrote:

>>>> On 02.06.16 at 21:59, <syso@datanoise.net> wrote:
>> Temporary Fix
>> -------------
>>
>> After checking the source[1] of the PCI configuration space
>> handling in xen-pciback, I found out that changing line 258
>> to read
>>
>>     if (handled && !err) {
>>
>> instead of:
>>
>>     if (!handled && !err) {
>>
>> solves the issue and I can successfully write to the interface.
>
> This can't possibly a solution (see also the comment right there). But
> without you telling us what register gets written with what value,
> and what's being read back, I don't think there's much we can do or
> suggest. In the end, by changing the conditional you force writes
> that _did_ get emulated to _nevertheless_ also get carried out to
> raw hardware. If that helps for your specific case, then it's quite
> possible that something in the emulation paths invoked from the
> code above that conditional is wrong.

Many thanks Jan for your comment!!

Regarding the flow of data: there is a user level "toolkit" and the 
actual uio driver (uio_netx).

1. The uio_netx driver "only" configures the device and turns the 
interrupts off. This appears to be working, although I find it strange, 
that when it is "write request 4 bytes" conf_space.c seems to do it 
twice, when it is "write request 2 bytes", it is only once (see below) - 
but I am not sure if I overlooked something.

2. the user level driver part (toolkit) first reads out the PCI config, 
send a reset sequence to the device, and then restores (write back the 
previously save PCI config) to the driver.
After that, it should upload the firmware blob to the PCI card, which 
does not happen, because it thinks, the reset was not successful.
But I need to capture this again and compare with the settings of the 
reset sequence in the source of the driver (will send later).

So far I got the following- does that make sense to you?


Dom0:
	lspci -vvv -s 06:00.0
	06:00.0 Unassigned class [ff00]: Hilscher GmbH CIFX 50E-DP(M/S)
		Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
Stepping- SERR+ FastB2B- DisINTx-
		Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- 
<TAbort- <MAbort- >SERR- <PERR- INTx-
		Latency: 64
		Interrupt: pin A routed to IRQ 11
		Region 0: Memory at f7a00000 (32-bit, non-prefetchable) [size=64K]

	modprobe xen-pciback verbose_request=1
	xl pci-assignable-add 06:00.0
	xl pci-attach debian-guest 06:00.0,permissive=1

Driver gets automatically loaded in DomU and the driver tries to disable 
the interrupts
https://github.com/torvalds/linux/blob/master/drivers/uio/uio_netx.c#L99


on Dom0: tail -f /var/log/kern.log

---- snip ----
pciback 0000:06:00.0: enabling permissive mode configuration space 
accesses!
[139971.850691] pciback 0000:06:00.0: permissive mode is potentially 
unsafe!
[139971.904249] xen_pciback: vpci: 0000:06:00.0: assign to virtual slot 
0
[139971.906548] pciback 0000:06:00.0: registering for 17766
[139971.914313] xen-pciback: 0000:06:00.0: read 4 bytes at 0x0
[139971.914327] xen-pciback: 0000:06:00.0: read 4 bytes at 0x0 = 15cf
[139971.914341] xen-pciback: 0000:06:00.0: read 1 bytes at 0xe
[139971.914350] xen-pciback: 0000:06:00.0: read 1 bytes at 0xe = 0
[139971.914356] xen-pciback: 0000:06:00.0: read 2 bytes at 0x6
[139971.914364] xen-pciback: 0000:06:00.0: read 2 bytes at 0x6 = 200
[139971.914378] xen-pciback: 0000:06:00.0: read 4 bytes at 0x8
[139971.914386] xen-pciback: 0000:06:00.0: read 4 bytes at 0x8 = 
ff000000
[139971.914402] xen-pciback: 0000:06:00.0: read 2 bytes at 0x6
[139971.914409] xen-pciback: 0000:06:00.0: read 2 bytes at 0x6 = 200
[139971.914423] xen-pciback: 0000:06:00.0: read 1 bytes at 0x3d
[139971.914437] xen-pciback: 0000:06:00.0: read 1 bytes at 0x3d = 1
[139971.914448] xen-pciback: 0000:06:00.0: read 1 bytes at 0x3c
[139971.914456] xen-pciback: 0000:06:00.0: read 1 bytes at 0x3c = 10
[139971.914466] xen-pciback: 0000:06:00.0: read 2 bytes at 0x4
[139971.914480] xen-pciback: 0000:06:00.0: read 2 bytes at 0x4 = 103
[139971.914490] xen-pciback: 0000:06:00.0: write request 2 bytes at 0x4 
= 100
[139971.914492] xen-pciback: 0000:06:00.0: read 2 bytes at 0x4
[139971.914505] xen-pciback: 0000:06:00.0: read 2 bytes at 0x4 = 103
---- snap ----

which is from here 
https://github.com/torvalds/linux/blob/master/drivers/xen/xen-pciback/conf_space.c#L208

One observation that struck me:
  - if the write request is a word (as opposed to a double word), the 
write always seems to succeed!

	[139971.914490] xen-pciback: 0000:06:00.0: write request 2 bytes at 0x4 
= 100       <---- input value
	[139971.914492] xen-pciback: 0000:06:00.0: read 2 bytes at 0x4
	[139971.914505] xen-pciback: 0000:06:00.0: read 2 bytes at 0x4 = 103    
   <---- before write
	[139971.914527] xen-pciback: 0000:06:00.0: read 2 bytes at 0x4
	[139971.914540] xen-pciback: 0000:06:00.0: read 2 bytes at 0x4 = 100    
   <---- after write


  - if the write request is a double word, the write _sometimes_ seems 
to succeed!
    or is this just some method the overwrite all registers with 
"ffffffff" first and then set the actual value?

   [914572] xbk: 06:00.0: write request 4 bytes at 0x10 = ffffffff
   [914574] xbk: 06:00.0: read 4 bytes at 0x10
   [914582] xbk: 06:00.0: read 4 bytes at 0x10 = f7a00000
   [914591] xbk: 06:00.0: read 4 bytes at 0x10
   [914599] xbk: 06:00.0: read 4 bytes at 0x10 = 10000      <---- fail
   --
   [914636] xbk: 06:00.0: write request 4 bytes at 0x10 = f7a00000
   [914638] xbk: 06:00.0: read 4 bytes at 0x10
   [914646] xbk: 06:00.0: read 4 bytes at 0x10 = 10000
   [914660] xbk: 06:00.0: read 4 bytes at 0x10
   [914667] xbk: 06:00.0: read 4 bytes at 0x10 = f7a00000      <---- 
success
   --
   [919484] xbk: 06:00.0: write request 4 bytes at 0x14 = ffffffff
   [919486] xbk: 06:00.0: read 4 bytes at 0x14
   [919493] xbk: 06:00.0: read 4 bytes at 0x14 = 0
   [919502] xbk: 06:00.0: read 4 bytes at 0x14
   [919508] xbk: 06:00.0: read 4 bytes at 0x14 = 1      <---- fail
   --
   [919542] xbk: 06:00.0: write request 4 bytes at 0x14 = 0
   [919544] xbk: 06:00.0: read 4 bytes at 0x14
   [919551] xbk: 06:00.0: read 4 bytes at 0x14 = 1
   [919564] xbk: 06:00.0: read 4 bytes at 0x14
   [919574] xbk: 06:00.0: read 4 bytes at 0x14 = 0      <---- success
   --
   [922060] xbk: 06:00.0: write request 4 bytes at 0x18 = ffffffff
   [922062] xbk: 06:00.0: read 4 bytes at 0x18
   [922069] xbk: 06:00.0: read 4 bytes at 0x18 = 0
   [922082] xbk: 06:00.0: read 4 bytes at 0x18
   [922089] xbk: 06:00.0: read 4 bytes at 0x18 = 1      <---- fail
   --
   [922136] xbk: 06:00.0: write request 4 bytes at 0x18 = 0
   [922138] xbk: 06:00.0: read 4 bytes at 0x18
   [922146] xbk: 06:00.0: read 4 bytes at 0x18 = 1
   [922158] xbk: 06:00.0: read 4 bytes at 0x18
   [922166] xbk: 06:00.0: read 4 bytes at 0x18 = 0      <---- success



So far, this seems to be fine, actually. The problem starts, once we run 
the diagnostic tool for the PCI card (works fine of course on Dom0), 
which initiates the reset on the card (again: read PCI config, send 
reset sequence, write back config to PCI card, check if some "flag" is 
valid (which is not), and then goes on to upload the firmware to the 
device).


Many thanks!!! Jürgen


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel