* Issues with PCI-Passtrough (VT-d) in HVM with Xen 4.6
@ 2016-06-02 19:59 Sylwester Sosnowski
2016-06-02 20:06 ` Konrad Rzeszutek Wilk
` (2 more replies)
0 siblings, 3 replies; 22+ messages in thread
From: Sylwester Sosnowski @ 2016-06-02 19:59 UTC (permalink / raw)
To: xen-devel; +Cc: Jeremy Fitzhardinge
Issues with PCI-Passtrough (VT-d) in HVM with Xen 4.6
=====================================================
Background
----------
I am trying to passthrough an Industrial Ethernet Interface
(Hilscher GmbH CIFX 50E-DP(M/S)) on a HVM DomU running the
Xen 4.6 Hypervisor. The card is being pass-trough to the HVM
using the PCI permissive mode and VT-d is active on this platform.
The Linux driver of the card (available only with NDA) at first
sight seems to work properly (e.g. no system stability problems,
no call traces in dmesg).
Hilscher provides the libcifx, which is an user-level library
for accessing the card. libcfix uses the generic UIO interface
and some card-specific interfaces to communicate.
Issue
-----
Whenever libcifx tries perform a reset sequence initializing
the card peripherals, we get an empty / invalid result.
I digged deeper into this by adding some debug hooks into the
xen-pciback kernel module.
The card performs writes to offset 0x10 and 0x30 at card
initialization (mostly writing dwords and words).
To verify if the writes were performed successfully, I read
back the values after writing and can see that the read data
differs from the written one.
Temporary Fix
-------------
After checking the source[1] of the PCI configuration space
handling in xen-pciback, I found out that changing line 258
to read
if (handled && !err) {
instead of:
if (!handled && !err) {
solves the issue and I can successfully write to the interface.
I am unsure why this works and if it's the right way to do it
or possibly a Xen bug, so I would like to ask for feedback
for this.
Please let me know whenever I can supply additional logfiles or
info. I'd be happy if we can resolve whenever this is a hardware-
specific, driver-specific or Xen issue.
Thank you,
Sylwester
PS: I tried to CC Ryan Wilson and Konrad Rzeszutek Wilk, but the
E-Mail seems to be no longer active. This is where the changes
originated.
Refereces
---------
1.
https://github.com/google/kasan/blob/master/drivers/xen/xen-pciback/conf_space.c
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Issues with PCI-Passtrough (VT-d) in HVM with Xen 4.6
2016-06-02 19:59 Issues with PCI-Passtrough (VT-d) in HVM with Xen 4.6 Sylwester Sosnowski
@ 2016-06-02 20:06 ` Konrad Rzeszutek Wilk
2016-06-02 20:11 ` Sylwester Sosnowski
2016-06-03 7:24 ` Sylwester Sosnowski
2016-06-03 11:52 ` Jan Beulich
2 siblings, 1 reply; 22+ messages in thread
From: Konrad Rzeszutek Wilk @ 2016-06-02 20:06 UTC (permalink / raw)
To: Sylwester Sosnowski; +Cc: Jeremy Fitzhardinge, xen-devel
On Thu, Jun 02, 2016 at 09:59:29PM +0200, Sylwester Sosnowski wrote:
>
> Issues with PCI-Passtrough (VT-d) in HVM with Xen 4.6
> =====================================================
>
> Background
> ----------
>
> I am trying to passthrough an Industrial Ethernet Interface
> (Hilscher GmbH CIFX 50E-DP(M/S)) on a HVM DomU running the
> Xen 4.6 Hypervisor. The card is being pass-trough to the HVM
> using the PCI permissive mode and VT-d is active on this platform.
>
> The Linux driver of the card (available only with NDA) at first
> sight seems to work properly (e.g. no system stability problems,
> no call traces in dmesg).
>
> Hilscher provides the libcifx, which is an user-level library
> for accessing the card. libcfix uses the generic UIO interface
> and some card-specific interfaces to communicate.
>
> Issue
> -----
>
> Whenever libcifx tries perform a reset sequence initializing
> the card peripherals, we get an empty / invalid result.
>
> I digged deeper into this by adding some debug hooks into the
> xen-pciback kernel module.
>
> The card performs writes to offset 0x10 and 0x30 at card
> initialization (mostly writing dwords and words).
>
> To verify if the writes were performed successfully, I read
> back the values after writing and can see that the read data
> differs from the written one.
>
> Temporary Fix
> -------------
>
> After checking the source[1] of the PCI configuration space
> handling in xen-pciback, I found out that changing line 258
> to read
>
> if (handled && !err) {
>
> instead of:
>
> if (!handled && !err) {
>
> solves the issue and I can successfully write to the interface.
>
> I am unsure why this works and if it's the right way to do it
> or possibly a Xen bug, so I would like to ask for feedback
> for this.
did you try the permissive module parameter?
>
>
> Please let me know whenever I can supply additional logfiles or
> info. I'd be happy if we can resolve whenever this is a hardware-
> specific, driver-specific or Xen issue.
>
> Thank you,
> Sylwester
>
> PS: I tried to CC Ryan Wilson and Konrad Rzeszutek Wilk, but the
> E-Mail seems to be no longer active. This is where the changes
> originated.
>
> Refereces
> ---------
> 1.
> https://github.com/google/kasan/blob/master/drivers/xen/xen-pciback/conf_space.c
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Issues with PCI-Passtrough (VT-d) in HVM with Xen 4.6
2016-06-02 20:06 ` Konrad Rzeszutek Wilk
@ 2016-06-02 20:11 ` Sylwester Sosnowski
0 siblings, 0 replies; 22+ messages in thread
From: Sylwester Sosnowski @ 2016-06-02 20:11 UTC (permalink / raw)
To: xen-devel
On 02.06.16 um 22:06 Konrad Rzeszutek Wilk wrote:
>> did you try the permissive module parameter?
Yes, I have been using the permissive mode when
I was getting these results.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Issues with PCI-Passtrough (VT-d) in HVM with Xen 4.6
2016-06-02 19:59 Issues with PCI-Passtrough (VT-d) in HVM with Xen 4.6 Sylwester Sosnowski
2016-06-02 20:06 ` Konrad Rzeszutek Wilk
@ 2016-06-03 7:24 ` Sylwester Sosnowski
2016-06-03 11:52 ` Jan Beulich
2 siblings, 0 replies; 22+ messages in thread
From: Sylwester Sosnowski @ 2016-06-03 7:24 UTC (permalink / raw)
To: xen-devel; +Cc: Jürgen Walter, Jeremy Fitzhardinge
[-- Attachment #1.1: Type: text/plain, Size: 541 bytes --]
On 02.06.16 at 21:59 Sylwester Sosnowski wrote:
> Issues with PCI-Passtrough (VT-d) in HVM with Xen 4.6
> =====================================================
>
> Background
> ----------
>
> I am trying to passthrough an Industrial Ethernet Interface
> (Hilscher GmbH CIFX 50E-DP(M/S)) on a HVM DomU running the
> Xen 4.6 Hypervisor. The card is being pass-trough to the HVM
> using the PCI permissive mode and VT-d is active on this platform.
Quick correction here: I meant PV and not HVM. I mixed up the
terms.
Sorry for the confusion.
[-- Attachment #1.2: Type: text/html, Size: 917 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Issues with PCI-Passtrough (VT-d) in HVM with Xen 4.6
2016-06-02 19:59 Issues with PCI-Passtrough (VT-d) in HVM with Xen 4.6 Sylwester Sosnowski
2016-06-02 20:06 ` Konrad Rzeszutek Wilk
2016-06-03 7:24 ` Sylwester Sosnowski
@ 2016-06-03 11:52 ` Jan Beulich
2016-06-03 12:02 ` Jürgen Walter
2 siblings, 1 reply; 22+ messages in thread
From: Jan Beulich @ 2016-06-03 11:52 UTC (permalink / raw)
To: Sylwester Sosnowski; +Cc: xen-devel
>>> On 02.06.16 at 21:59, <syso@datanoise.net> wrote:
> Temporary Fix
> -------------
>
> After checking the source[1] of the PCI configuration space
> handling in xen-pciback, I found out that changing line 258
> to read
>
> if (handled && !err) {
>
> instead of:
>
> if (!handled && !err) {
>
> solves the issue and I can successfully write to the interface.
This can't possibly a solution (see also the comment right there). But
without you telling us what register gets written with what value,
and what's being read back, I don't think there's much we can do or
suggest. In the end, by changing the conditional you force writes
that _did_ get emulated to _nevertheless_ also get carried out to
raw hardware. If that helps for your specific case, then it's quite
possible that something in the emulation paths invoked from the
code above that conditional is wrong.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Issues with PCI-Passtrough (VT-d) in HVM with Xen 4.6
2016-06-03 11:52 ` Jan Beulich
@ 2016-06-03 12:02 ` Jürgen Walter
2016-06-03 13:26 ` Jan Beulich
0 siblings, 1 reply; 22+ messages in thread
From: Jürgen Walter @ 2016-06-03 12:02 UTC (permalink / raw)
To: Jan Beulich; +Cc: Sylwester Sosnowski, xen-devel
On 3 Jun 2016, at 13:52, Jan Beulich wrote:
>>>> On 02.06.16 at 21:59, <syso@datanoise.net> wrote:
>> Temporary Fix
>> -------------
>>
>> After checking the source[1] of the PCI configuration space
>> handling in xen-pciback, I found out that changing line 258
>> to read
>>
>> if (handled && !err) {
>>
>> instead of:
>>
>> if (!handled && !err) {
>>
>> solves the issue and I can successfully write to the interface.
>
> This can't possibly a solution (see also the comment right there). But
> without you telling us what register gets written with what value,
> and what's being read back, I don't think there's much we can do or
> suggest. In the end, by changing the conditional you force writes
> that _did_ get emulated to _nevertheless_ also get carried out to
> raw hardware. If that helps for your specific case, then it's quite
> possible that something in the emulation paths invoked from the
> code above that conditional is wrong.
Many thanks Jan for your comment!!
Regarding the flow of data: there is a user level "toolkit" and the
actual uio driver (uio_netx).
1. The uio_netx driver "only" configures the device and turns the
interrupts off. This appears to be working, although I find it strange,
that when it is "write request 4 bytes" conf_space.c seems to do it
twice, when it is "write request 2 bytes", it is only once (see below) -
but I am not sure if I overlooked something.
2. the user level driver part (toolkit) first reads out the PCI config,
send a reset sequence to the device, and then restores (write back the
previously save PCI config) to the driver.
After that, it should upload the firmware blob to the PCI card, which
does not happen, because it thinks, the reset was not successful.
But I need to capture this again and compare with the settings of the
reset sequence in the source of the driver (will send later).
So far I got the following- does that make sense to you?
Dom0:
lspci -vvv -s 06:00.0
06:00.0 Unassigned class [ff00]: Hilscher GmbH CIFX 50E-DP(M/S)
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR+ FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 64
Interrupt: pin A routed to IRQ 11
Region 0: Memory at f7a00000 (32-bit, non-prefetchable) [size=64K]
modprobe xen-pciback verbose_request=1
xl pci-assignable-add 06:00.0
xl pci-attach debian-guest 06:00.0,permissive=1
Driver gets automatically loaded in DomU and the driver tries to disable
the interrupts
https://github.com/torvalds/linux/blob/master/drivers/uio/uio_netx.c#L99
on Dom0: tail -f /var/log/kern.log
---- snip ----
pciback 0000:06:00.0: enabling permissive mode configuration space
accesses!
[139971.850691] pciback 0000:06:00.0: permissive mode is potentially
unsafe!
[139971.904249] xen_pciback: vpci: 0000:06:00.0: assign to virtual slot
0
[139971.906548] pciback 0000:06:00.0: registering for 17766
[139971.914313] xen-pciback: 0000:06:00.0: read 4 bytes at 0x0
[139971.914327] xen-pciback: 0000:06:00.0: read 4 bytes at 0x0 = 15cf
[139971.914341] xen-pciback: 0000:06:00.0: read 1 bytes at 0xe
[139971.914350] xen-pciback: 0000:06:00.0: read 1 bytes at 0xe = 0
[139971.914356] xen-pciback: 0000:06:00.0: read 2 bytes at 0x6
[139971.914364] xen-pciback: 0000:06:00.0: read 2 bytes at 0x6 = 200
[139971.914378] xen-pciback: 0000:06:00.0: read 4 bytes at 0x8
[139971.914386] xen-pciback: 0000:06:00.0: read 4 bytes at 0x8 =
ff000000
[139971.914402] xen-pciback: 0000:06:00.0: read 2 bytes at 0x6
[139971.914409] xen-pciback: 0000:06:00.0: read 2 bytes at 0x6 = 200
[139971.914423] xen-pciback: 0000:06:00.0: read 1 bytes at 0x3d
[139971.914437] xen-pciback: 0000:06:00.0: read 1 bytes at 0x3d = 1
[139971.914448] xen-pciback: 0000:06:00.0: read 1 bytes at 0x3c
[139971.914456] xen-pciback: 0000:06:00.0: read 1 bytes at 0x3c = 10
[139971.914466] xen-pciback: 0000:06:00.0: read 2 bytes at 0x4
[139971.914480] xen-pciback: 0000:06:00.0: read 2 bytes at 0x4 = 103
[139971.914490] xen-pciback: 0000:06:00.0: write request 2 bytes at 0x4
= 100
[139971.914492] xen-pciback: 0000:06:00.0: read 2 bytes at 0x4
[139971.914505] xen-pciback: 0000:06:00.0: read 2 bytes at 0x4 = 103
---- snap ----
which is from here
https://github.com/torvalds/linux/blob/master/drivers/xen/xen-pciback/conf_space.c#L208
One observation that struck me:
- if the write request is a word (as opposed to a double word), the
write always seems to succeed!
[139971.914490] xen-pciback: 0000:06:00.0: write request 2 bytes at 0x4
= 100 <---- input value
[139971.914492] xen-pciback: 0000:06:00.0: read 2 bytes at 0x4
[139971.914505] xen-pciback: 0000:06:00.0: read 2 bytes at 0x4 = 103
<---- before write
[139971.914527] xen-pciback: 0000:06:00.0: read 2 bytes at 0x4
[139971.914540] xen-pciback: 0000:06:00.0: read 2 bytes at 0x4 = 100
<---- after write
- if the write request is a double word, the write _sometimes_ seems
to succeed!
or is this just some method the overwrite all registers with
"ffffffff" first and then set the actual value?
[914572] xbk: 06:00.0: write request 4 bytes at 0x10 = ffffffff
[914574] xbk: 06:00.0: read 4 bytes at 0x10
[914582] xbk: 06:00.0: read 4 bytes at 0x10 = f7a00000
[914591] xbk: 06:00.0: read 4 bytes at 0x10
[914599] xbk: 06:00.0: read 4 bytes at 0x10 = 10000 <---- fail
--
[914636] xbk: 06:00.0: write request 4 bytes at 0x10 = f7a00000
[914638] xbk: 06:00.0: read 4 bytes at 0x10
[914646] xbk: 06:00.0: read 4 bytes at 0x10 = 10000
[914660] xbk: 06:00.0: read 4 bytes at 0x10
[914667] xbk: 06:00.0: read 4 bytes at 0x10 = f7a00000 <----
success
--
[919484] xbk: 06:00.0: write request 4 bytes at 0x14 = ffffffff
[919486] xbk: 06:00.0: read 4 bytes at 0x14
[919493] xbk: 06:00.0: read 4 bytes at 0x14 = 0
[919502] xbk: 06:00.0: read 4 bytes at 0x14
[919508] xbk: 06:00.0: read 4 bytes at 0x14 = 1 <---- fail
--
[919542] xbk: 06:00.0: write request 4 bytes at 0x14 = 0
[919544] xbk: 06:00.0: read 4 bytes at 0x14
[919551] xbk: 06:00.0: read 4 bytes at 0x14 = 1
[919564] xbk: 06:00.0: read 4 bytes at 0x14
[919574] xbk: 06:00.0: read 4 bytes at 0x14 = 0 <---- success
--
[922060] xbk: 06:00.0: write request 4 bytes at 0x18 = ffffffff
[922062] xbk: 06:00.0: read 4 bytes at 0x18
[922069] xbk: 06:00.0: read 4 bytes at 0x18 = 0
[922082] xbk: 06:00.0: read 4 bytes at 0x18
[922089] xbk: 06:00.0: read 4 bytes at 0x18 = 1 <---- fail
--
[922136] xbk: 06:00.0: write request 4 bytes at 0x18 = 0
[922138] xbk: 06:00.0: read 4 bytes at 0x18
[922146] xbk: 06:00.0: read 4 bytes at 0x18 = 1
[922158] xbk: 06:00.0: read 4 bytes at 0x18
[922166] xbk: 06:00.0: read 4 bytes at 0x18 = 0 <---- success
So far, this seems to be fine, actually. The problem starts, once we run
the diagnostic tool for the PCI card (works fine of course on Dom0),
which initiates the reset on the card (again: read PCI config, send
reset sequence, write back config to PCI card, check if some "flag" is
valid (which is not), and then goes on to upload the firmware to the
device).
Many thanks!!! Jürgen
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Issues with PCI-Passtrough (VT-d) in HVM with Xen 4.6
2016-06-03 12:02 ` Jürgen Walter
@ 2016-06-03 13:26 ` Jan Beulich
2016-06-03 14:20 ` Jan Beulich
2016-06-04 15:15 ` Jürgen Walter
0 siblings, 2 replies; 22+ messages in thread
From: Jan Beulich @ 2016-06-03 13:26 UTC (permalink / raw)
To: Jürgen Walter; +Cc: Sylwester Sosnowski, xen-devel
>>> On 03.06.16 at 14:02, <juwalter@gmail.com> wrote:
> One observation that struck me:
> - if the write request is a word (as opposed to a double word), the
> write always seems to succeed!
>
> [139971.914490] xen-pciback: 0000:06:00.0: write request 2 bytes at 0x4
> = 100 <---- input value
> [139971.914492] xen-pciback: 0000:06:00.0: read 2 bytes at 0x4
> [139971.914505] xen-pciback: 0000:06:00.0: read 2 bytes at 0x4 = 103
> <---- before write
> [139971.914527] xen-pciback: 0000:06:00.0: read 2 bytes at 0x4
> [139971.914540] xen-pciback: 0000:06:00.0: read 2 bytes at 0x4 = 100
> <---- after write
>
>
> - if the write request is a double word, the write _sometimes_ seems
> to succeed!
Well, perhaps that depends on which field gets written to, rather
than merely the width of the write.
> or is this just some method the overwrite all registers with
> "ffffffff" first and then set the actual value?
>
> [914572] xbk: 06:00.0: write request 4 bytes at 0x10 = ffffffff
> [914574] xbk: 06:00.0: read 4 bytes at 0x10
> [914582] xbk: 06:00.0: read 4 bytes at 0x10 = f7a00000
> [914591] xbk: 06:00.0: read 4 bytes at 0x10
> [914599] xbk: 06:00.0: read 4 bytes at 0x10 = 10000 <---- fail
That's an unexpected value, indeed (but seems to match up with
the source, so there's definitely something wrong here - this
presumably ought to be 0xffff0000, meaning the size of that
region is 64k). You also can't expect to read back 0xffffffff in this
case: BARs have some read-only bits at the bottom.
> [914636] xbk: 06:00.0: write request 4 bytes at 0x10 = f7a00000
> [914638] xbk: 06:00.0: read 4 bytes at 0x10
> [914646] xbk: 06:00.0: read 4 bytes at 0x10 = 10000
> [914660] xbk: 06:00.0: read 4 bytes at 0x10
> [914667] xbk: 06:00.0: read 4 bytes at 0x10 = f7a00000 <----
> success
But overall this doesn't look to be a trace of the post-reset
restore attempt, but instead I'd guess this is the OS sizing
the BARs. I.e. while apparently pointing to a problem, likely
not directly related to yours.
I'll see to put together a patch for the recognizable problem,
but I'm still unclear about your actual one.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Issues with PCI-Passtrough (VT-d) in HVM with Xen 4.6
2016-06-03 13:26 ` Jan Beulich
@ 2016-06-03 14:20 ` Jan Beulich
2016-06-04 14:36 ` Jürgen Walter
2016-06-04 15:15 ` Jürgen Walter
1 sibling, 1 reply; 22+ messages in thread
From: Jan Beulich @ 2016-06-03 14:20 UTC (permalink / raw)
To: juwalter; +Cc: Sylwester Sosnowski, xen-devel
[-- Attachment #1: Type: text/plain, Size: 812 bytes --]
>>> On 03.06.16 at 15:26, <JBeulich@suse.com> wrote:
>>>> On 03.06.16 at 14:02, <juwalter@gmail.com> wrote:
>> or is this just some method the overwrite all registers with
>> "ffffffff" first and then set the actual value?
>>
>> [914572] xbk: 06:00.0: write request 4 bytes at 0x10 = ffffffff
>> [914574] xbk: 06:00.0: read 4 bytes at 0x10
>> [914582] xbk: 06:00.0: read 4 bytes at 0x10 = f7a00000
>> [914591] xbk: 06:00.0: read 4 bytes at 0x10
>> [914599] xbk: 06:00.0: read 4 bytes at 0x10 = 10000 <---- fail
>
> That's an unexpected value, indeed (but seems to match up with
> the source, so there's definitely something wrong here - this
> presumably ought to be 0xffff0000, meaning the size of that
> region is 64k).
Mind trying out the attached patch?
Jan
[-- Attachment #2: xen-pciback-BAR-lengths.patch --]
[-- Type: text/plain, Size: 2627 bytes --]
xen-pciback: return proper values during BAR sizing
Reads following writes with all address bits set to 1 should return all
changeable address bits as one, not the BAR size (nor, as was the case
for the upper half of 64-bit BARs, the high half of the region's end
address). Presumably this didn't cause any problems so far because
consumers use the value to calculate the size (usually via val & -val),
and do nothing else with it.
But also consider the exception here: Unimplemented BARs should always
return all zeroes.
And finally, the check for whether to return the sizing address on read
for the ROM BAR should ignore all non-address bits, not just the ROM
Enable one.
Signed-off-by: Jan Beulich <jbeulich@suse.com>
--- head.orig/drivers/xen/xen-pciback/conf_space_header.c 2015-03-16 15:38:59.000000000 +0100
+++ head/drivers/xen/xen-pciback/conf_space_header.c 2016-06-03 15:55:27.629520779 +0200
@@ -149,7 +149,7 @@ static int rom_write(struct pci_dev *dev
/* A write to obtain the length must happen as a 32-bit write.
* This does not (yet) support writing individual bytes
*/
- if (value == ~PCI_ROM_ADDRESS_ENABLE)
+ if ((value | ~PCI_ROM_ADDRESS_MASK) == ~0)
bar->which = 1;
else {
u32 tmpval;
@@ -229,38 +229,42 @@ static inline void read_dev_bar(struct p
(PCI_BASE_ADDRESS_SPACE_MEMORY |
PCI_BASE_ADDRESS_MEM_TYPE_64))) {
bar_info->val = res[pos - 1].start >> 32;
- bar_info->len_val = res[pos - 1].end >> 32;
+ bar_info->len_val = -resource_size(&res[pos - 1]) >> 32;
return;
}
}
+ if (!res[pos].flags ||
+ (res[pos].flags & (IORESOURCE_DISABLED | IORESOURCE_UNSET |
+ IORESOURCE_BUSY)))
+ return;
+
bar_info->val = res[pos].start |
(res[pos].flags & PCI_REGION_FLAG_MASK);
- bar_info->len_val = resource_size(&res[pos]);
+ bar_info->len_val = -resource_size(&res[pos]) |
+ (res[pos].flags & PCI_REGION_FLAG_MASK);
}
static void *bar_init(struct pci_dev *dev, int offset)
{
- struct pci_bar_info *bar = kmalloc(sizeof(*bar), GFP_KERNEL);
+ struct pci_bar_info *bar = kzalloc(sizeof(*bar), GFP_KERNEL);
if (!bar)
return ERR_PTR(-ENOMEM);
read_dev_bar(dev, bar, offset, ~0);
- bar->which = 0;
return bar;
}
static void *rom_init(struct pci_dev *dev, int offset)
{
- struct pci_bar_info *bar = kmalloc(sizeof(*bar), GFP_KERNEL);
+ struct pci_bar_info *bar = kzalloc(sizeof(*bar), GFP_KERNEL);
if (!bar)
return ERR_PTR(-ENOMEM);
read_dev_bar(dev, bar, offset, ~PCI_ROM_ADDRESS_ENABLE);
- bar->which = 0;
return bar;
}
[-- Attachment #3: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Issues with PCI-Passtrough (VT-d) in HVM with Xen 4.6
2016-06-03 14:20 ` Jan Beulich
@ 2016-06-04 14:36 ` Jürgen Walter
2016-06-06 7:59 ` Jan Beulich
0 siblings, 1 reply; 22+ messages in thread
From: Jürgen Walter @ 2016-06-04 14:36 UTC (permalink / raw)
To: Jan Beulich; +Cc: Sylwester Sosnowski, xen-devel
[-- Attachment #1: Type: text/plain, Size: 1656 bytes --]
Hi Jan,
On 3 Jun 2016, at 16:20, Jan Beulich wrote:
>>>> On 03.06.16 at 15:26, <JBeulich@suse.com> wrote:
>>>>> On 03.06.16 at 14:02, <juwalter@gmail.com> wrote:
>>> or is this just some method the overwrite all registers with
>>> "ffffffff" first and then set the actual value?
>>>
>>> [914572] xbk: 06:00.0: write request 4 bytes at 0x10 = ffffffff
>>> [914574] xbk: 06:00.0: read 4 bytes at 0x10
>>> [914582] xbk: 06:00.0: read 4 bytes at 0x10 = f7a00000
>>> [914591] xbk: 06:00.0: read 4 bytes at 0x10
>>> [914599] xbk: 06:00.0: read 4 bytes at 0x10 = 10000 <----
>>> fail
>>
>> That's an unexpected value, indeed (but seems to match up with
>> the source, so there's definitely something wrong here - this
>> presumably ought to be 0xffff0000, meaning the size of that
>> region is 64k).
>
> Mind trying out the attached patch?
Many thanks for your patch!! I did try it, and it is different now
(details in files attached)
"best of conf_space_header" see below.
I cannot really make sense of it, though. Can you?
Thank you, Jürgen
original:
xen-pciback: 0000:06:00.0: write request 4 bytes at 0x10 = ffffffff
xen-pciback: 0000:06:00.0: read 4 bytes at 0x10
xen-pciback: 0000:06:00.0: read 4 bytes at 0x10 = f7a00000
xen-pciback: 0000:06:00.0: read 4 bytes at 0x10
xen-pciback: 0000:06:00.0: read 4 bytes at 0x10 = 10000
patched:
xen-pciback: 0000:06:00.0: write request 4 bytes at 0x10 = ffffffff
xen-pciback: 0000:06:00.0: read 4 bytes at 0x10
xen-pciback: 0000:06:00.0: read 4 bytes at 0x10 = f7a00000
xen-pciback: 0000:06:00.0: read 4 bytes at 0x10
xen-pciback: 0000:06:00.0: read 4 bytes at 0x10 = ffff0000
[-- Attachment #2: cmp-orig-patch.txt --]
[-- Type: text/plain, Size: 5873 bytes --]
==> pci-attach-orig-grep <==
3906.673306] xen-pciback: 0000:06:00.0: write request 4 bytes at 0x10 = ffffffff
3906.673307] xen-pciback: 0000:06:00.0: read 4 bytes at 0x10
3906.673313] xen-pciback: 0000:06:00.0: read 4 bytes at 0x10 = f7a00000
--
3906.673338] xen-pciback: 0000:06:00.0: write request 4 bytes at 0x10 = f7a00000
3906.673339] xen-pciback: 0000:06:00.0: read 4 bytes at 0x10
3906.673345] xen-pciback: 0000:06:00.0: read 4 bytes at 0x10 = 10000
--
3906.673405] xen-pciback: 0000:06:00.0: write request 4 bytes at 0x14 = ffffffff
3906.673406] xen-pciback: 0000:06:00.0: read 4 bytes at 0x14
3906.673413] xen-pciback: 0000:06:00.0: read 4 bytes at 0x14 = 0
--
3906.673434] xen-pciback: 0000:06:00.0: write request 4 bytes at 0x14 = 0
3906.673436] xen-pciback: 0000:06:00.0: read 4 bytes at 0x14
3906.673442] xen-pciback: 0000:06:00.0: read 4 bytes at 0x14 = 1
--
3906.673496] xen-pciback: 0000:06:00.0: write request 4 bytes at 0x18 = ffffffff
3906.673498] xen-pciback: 0000:06:00.0: read 4 bytes at 0x18
3906.673504] xen-pciback: 0000:06:00.0: read 4 bytes at 0x18 = 0
--
3906.673526] xen-pciback: 0000:06:00.0: write request 4 bytes at 0x18 = 0
3906.673528] xen-pciback: 0000:06:00.0: read 4 bytes at 0x18
3906.673533] xen-pciback: 0000:06:00.0: read 4 bytes at 0x18 = 1
--
3906.673590] xen-pciback: 0000:06:00.0: write request 4 bytes at 0x1c = ffffffff
3906.673598] xen-pciback: 0000:06:00.0: read 4 bytes at 0x1c
3906.673612] xen-pciback: 0000:06:00.0: read 4 bytes at 0x1c = 0
--
3906.673660] xen-pciback: 0000:06:00.0: write request 4 bytes at 0x1c = 0
3906.673670] xen-pciback: 0000:06:00.0: read 4 bytes at 0x1c
3906.673684] xen-pciback: 0000:06:00.0: read 4 bytes at 0x1c = 1
--
3906.673747] xen-pciback: 0000:06:00.0: write request 4 bytes at 0x20 = ffffffff
3906.673749] xen-pciback: 0000:06:00.0: read 4 bytes at 0x20
3906.673756] xen-pciback: 0000:06:00.0: read 4 bytes at 0x20 = 0
--
3906.673801] xen-pciback: 0000:06:00.0: write request 4 bytes at 0x20 = 0
3906.673804] xen-pciback: 0000:06:00.0: read 4 bytes at 0x20
3906.673812] xen-pciback: 0000:06:00.0: read 4 bytes at 0x20 = 1
--
3906.673988] xen-pciback: 0000:06:00.0: write request 4 bytes at 0x24 = ffffffff
3906.673992] xen-pciback: 0000:06:00.0: read 4 bytes at 0x24
3906.674002] xen-pciback: 0000:06:00.0: read 4 bytes at 0x24 = 0
--
3906.674067] xen-pciback: 0000:06:00.0: write request 4 bytes at 0x24 = 0
3906.674069] xen-pciback: 0000:06:00.0: read 4 bytes at 0x24
3906.674076] xen-pciback: 0000:06:00.0: read 4 bytes at 0x24 = 1
--
3906.674146] xen-pciback: 0000:06:00.0: write request 4 bytes at 0x30 = fffff800
3906.674149] xen-pciback: 0000:06:00.0: read 4 bytes at 0x30
3906.674156] xen-pciback: 0000:06:00.0: read 4 bytes at 0x30 = 0
--
3906.674220] xen-pciback: 0000:06:00.0: write request 4 bytes at 0x30 = 0
3906.674223] xen-pciback: 0000:06:00.0: read 4 bytes at 0x30
3906.674232] xen-pciback: 0000:06:00.0: read 4 bytes at 0x30 = 0
==> pci-attach-jan-grep <==
4743.120637] xen-pciback: 0000:06:00.0: write request 4 bytes at 0x10 = ffffffff
4743.120638] xen-pciback: 0000:06:00.0: read 4 bytes at 0x10
4743.120645] xen-pciback: 0000:06:00.0: read 4 bytes at 0x10 = f7a00000
--
4743.120675] xen-pciback: 0000:06:00.0: write request 4 bytes at 0x10 = f7a00000
4743.120677] xen-pciback: 0000:06:00.0: read 4 bytes at 0x10
4743.120685] xen-pciback: 0000:06:00.0: read 4 bytes at 0x10 = ffff0000
--
4743.120741] xen-pciback: 0000:06:00.0: write request 4 bytes at 0x14 = ffffffff
4743.120742] xen-pciback: 0000:06:00.0: read 4 bytes at 0x14
4743.120749] xen-pciback: 0000:06:00.0: read 4 bytes at 0x14 = 0
--
4743.120779] xen-pciback: 0000:06:00.0: write request 4 bytes at 0x14 = 0
4743.120781] xen-pciback: 0000:06:00.0: read 4 bytes at 0x14
4743.120788] xen-pciback: 0000:06:00.0: read 4 bytes at 0x14 = 0
--
4743.120845] xen-pciback: 0000:06:00.0: write request 4 bytes at 0x18 = ffffffff
4743.120847] xen-pciback: 0000:06:00.0: read 4 bytes at 0x18
4743.120854] xen-pciback: 0000:06:00.0: read 4 bytes at 0x18 = 0
--
4743.120884] xen-pciback: 0000:06:00.0: write request 4 bytes at 0x18 = 0
4743.120886] xen-pciback: 0000:06:00.0: read 4 bytes at 0x18
4743.120891] xen-pciback: 0000:06:00.0: read 4 bytes at 0x18 = 0
--
4743.120954] xen-pciback: 0000:06:00.0: write request 4 bytes at 0x1c = ffffffff
4743.120955] xen-pciback: 0000:06:00.0: read 4 bytes at 0x1c
4743.120961] xen-pciback: 0000:06:00.0: read 4 bytes at 0x1c = 0
--
4743.120993] xen-pciback: 0000:06:00.0: write request 4 bytes at 0x1c = 0
4743.120994] xen-pciback: 0000:06:00.0: read 4 bytes at 0x1c
4743.121000] xen-pciback: 0000:06:00.0: read 4 bytes at 0x1c = 0
--
4743.121055] xen-pciback: 0000:06:00.0: write request 4 bytes at 0x20 = ffffffff
4743.121058] xen-pciback: 0000:06:00.0: read 4 bytes at 0x20
4743.121064] xen-pciback: 0000:06:00.0: read 4 bytes at 0x20 = 0
--
4743.121098] xen-pciback: 0000:06:00.0: write request 4 bytes at 0x20 = 0
4743.121099] xen-pciback: 0000:06:00.0: read 4 bytes at 0x20
4743.121107] xen-pciback: 0000:06:00.0: read 4 bytes at 0x20 = 0
--
4743.121171] xen-pciback: 0000:06:00.0: write request 4 bytes at 0x24 = ffffffff
4743.121173] xen-pciback: 0000:06:00.0: read 4 bytes at 0x24
4743.121180] xen-pciback: 0000:06:00.0: read 4 bytes at 0x24 = 0
--
4743.121205] xen-pciback: 0000:06:00.0: write request 4 bytes at 0x24 = 0
4743.121207] xen-pciback: 0000:06:00.0: read 4 bytes at 0x24
4743.121214] xen-pciback: 0000:06:00.0: read 4 bytes at 0x24 = 0
--
4743.121278] xen-pciback: 0000:06:00.0: write request 4 bytes at 0x30 = fffff800
4743.121280] xen-pciback: 0000:06:00.0: read 4 bytes at 0x30
4743.121287] xen-pciback: 0000:06:00.0: read 4 bytes at 0x30 = 0
--
4743.121326] xen-pciback: 0000:06:00.0: write request 4 bytes at 0x30 = 0
4743.121328] xen-pciback: 0000:06:00.0: read 4 bytes at 0x30
4743.121333] xen-pciback: 0000:06:00.0: read 4 bytes at 0x30 = 0
[-- Attachment #3: pci-attach-orig.txt --]
[-- Type: text/plain, Size: 7076 bytes --]
pciback 0000:06:00.0: enabling permissive mode configuration space accesses!
pciback 0000:06:00.0: permissive mode is potentially unsafe!
xen_pciback: vpci: 0000:06:00.0: assign to virtual slot 0
pciback 0000:06:00.0: registering for 33
xen-pciback: 0000:06:00.0: read 4 bytes at 0x0
xen-pciback: 0000:06:00.0: read 4 bytes at 0x0 = 15cf
xen-pciback: 0000:06:00.0: read 1 bytes at 0xe
xen-pciback: 0000:06:00.0: read 1 bytes at 0xe = 0
xen-pciback: 0000:06:00.0: read 2 bytes at 0x6
xen-pciback: 0000:06:00.0: read 2 bytes at 0x6 = 200
xen-pciback: 0000:06:00.0: read 4 bytes at 0x8
xen-pciback: 0000:06:00.0: read 4 bytes at 0x8 = ff000000
xen-pciback: 0000:06:00.0: read 2 bytes at 0x6
xen-pciback: 0000:06:00.0: read 2 bytes at 0x6 = 200
xen-pciback: 0000:06:00.0: read 1 bytes at 0x3d
xen-pciback: 0000:06:00.0: read 1 bytes at 0x3d = 1
xen-pciback: 0000:06:00.0: read 1 bytes at 0x3c
xen-pciback: 0000:06:00.0: read 1 bytes at 0x3c = 10
xen-pciback: 0000:06:00.0: read 2 bytes at 0x4
xen-pciback: 0000:06:00.0: read 2 bytes at 0x4 = 0
xen-pciback: 0000:06:00.0: read 4 bytes at 0x10
xen-pciback: 0000:06:00.0: read 4 bytes at 0x10 = f7a00000
xen-pciback: 0000:06:00.0: write request 4 bytes at 0x10 = ffffffff
xen-pciback: 0000:06:00.0: read 4 bytes at 0x10
xen-pciback: 0000:06:00.0: read 4 bytes at 0x10 = f7a00000
xen-pciback: 0000:06:00.0: read 4 bytes at 0x10
xen-pciback: 0000:06:00.0: read 4 bytes at 0x10 = 10000
xen-pciback: 0000:06:00.0: write request 4 bytes at 0x10 = f7a00000
xen-pciback: 0000:06:00.0: read 4 bytes at 0x10
xen-pciback: 0000:06:00.0: read 4 bytes at 0x10 = 10000
xen-pciback: 0000:06:00.0: read 2 bytes at 0x4
xen-pciback: 0000:06:00.0: read 2 bytes at 0x4 = 0
xen-pciback: 0000:06:00.0: read 4 bytes at 0x14
xen-pciback: 0000:06:00.0: read 4 bytes at 0x14 = 0
xen-pciback: 0000:06:00.0: write request 4 bytes at 0x14 = ffffffff
xen-pciback: 0000:06:00.0: read 4 bytes at 0x14
xen-pciback: 0000:06:00.0: read 4 bytes at 0x14 = 0
xen-pciback: 0000:06:00.0: read 4 bytes at 0x14
xen-pciback: 0000:06:00.0: read 4 bytes at 0x14 = 1
xen-pciback: 0000:06:00.0: write request 4 bytes at 0x14 = 0
xen-pciback: 0000:06:00.0: read 4 bytes at 0x14
xen-pciback: 0000:06:00.0: read 4 bytes at 0x14 = 1
xen-pciback: 0000:06:00.0: read 2 bytes at 0x4
xen-pciback: 0000:06:00.0: read 2 bytes at 0x4 = 0
xen-pciback: 0000:06:00.0: read 4 bytes at 0x18
xen-pciback: 0000:06:00.0: read 4 bytes at 0x18 = 0
xen-pciback: 0000:06:00.0: write request 4 bytes at 0x18 = ffffffff
xen-pciback: 0000:06:00.0: read 4 bytes at 0x18
xen-pciback: 0000:06:00.0: read 4 bytes at 0x18 = 0
xen-pciback: 0000:06:00.0: read 4 bytes at 0x18
xen-pciback: 0000:06:00.0: read 4 bytes at 0x18 = 1
xen-pciback: 0000:06:00.0: write request 4 bytes at 0x18 = 0
xen-pciback: 0000:06:00.0: read 4 bytes at 0x18
xen-pciback: 0000:06:00.0: read 4 bytes at 0x18 = 1
xen-pciback: 0000:06:00.0: read 2 bytes at 0x4
xen-pciback: 0000:06:00.0: read 2 bytes at 0x4 = 0
xen-pciback: 0000:06:00.0: read 4 bytes at 0x1c
xen-pciback: 0000:06:00.0: read 4 bytes at 0x1c = 0
xen-pciback: 0000:06:00.0: write request 4 bytes at 0x1c = ffffffff
xen-pciback: 0000:06:00.0: read 4 bytes at 0x1c
xen-pciback: 0000:06:00.0: read 4 bytes at 0x1c = 0
xen-pciback: 0000:06:00.0: read 4 bytes at 0x1c
xen-pciback: 0000:06:00.0: read 4 bytes at 0x1c = 1
xen-pciback: 0000:06:00.0: write request 4 bytes at 0x1c = 0
xen-pciback: 0000:06:00.0: read 4 bytes at 0x1c
xen-pciback: 0000:06:00.0: read 4 bytes at 0x1c = 1
xen-pciback: 0000:06:00.0: read 2 bytes at 0x4
xen-pciback: 0000:06:00.0: read 2 bytes at 0x4 = 0
xen-pciback: 0000:06:00.0: read 4 bytes at 0x20
xen-pciback: 0000:06:00.0: read 4 bytes at 0x20 = 0
xen-pciback: 0000:06:00.0: write request 4 bytes at 0x20 = ffffffff
xen-pciback: 0000:06:00.0: read 4 bytes at 0x20
xen-pciback: 0000:06:00.0: read 4 bytes at 0x20 = 0
xen-pciback: 0000:06:00.0: read 4 bytes at 0x20
xen-pciback: 0000:06:00.0: read 4 bytes at 0x20 = 1
xen-pciback: 0000:06:00.0: write request 4 bytes at 0x20 = 0
xen-pciback: 0000:06:00.0: read 4 bytes at 0x20
xen-pciback: 0000:06:00.0: read 4 bytes at 0x20 = 1
xen-pciback: 0000:06:00.0: read 2 bytes at 0x4
xen-pciback: 0000:06:00.0: read 2 bytes at 0x4 = 0
xen-pciback: 0000:06:00.0: read 4 bytes at 0x24
xen-pciback: 0000:06:00.0: read 4 bytes at 0x24 = 0
xen-pciback: 0000:06:00.0: write request 4 bytes at 0x24 = ffffffff
xen-pciback: 0000:06:00.0: read 4 bytes at 0x24
xen-pciback: 0000:06:00.0: read 4 bytes at 0x24 = 0
xen-pciback: 0000:06:00.0: read 4 bytes at 0x24
xen-pciback: 0000:06:00.0: read 4 bytes at 0x24 = 1
xen-pciback: 0000:06:00.0: write request 4 bytes at 0x24 = 0
xen-pciback: 0000:06:00.0: read 4 bytes at 0x24
xen-pciback: 0000:06:00.0: read 4 bytes at 0x24 = 1
xen-pciback: 0000:06:00.0: read 2 bytes at 0x4
xen-pciback: 0000:06:00.0: read 2 bytes at 0x4 = 0
xen-pciback: 0000:06:00.0: read 4 bytes at 0x30
xen-pciback: 0000:06:00.0: read 4 bytes at 0x30 = 0
xen-pciback: 0000:06:00.0: write request 4 bytes at 0x30 = fffff800
xen-pciback: 0000:06:00.0: read 4 bytes at 0x30
xen-pciback: 0000:06:00.0: read 4 bytes at 0x30 = 0
xen-pciback: 0000:06:00.0: read 4 bytes at 0x30
xen-pciback: 0000:06:00.0: read 4 bytes at 0x30 = 0
xen-pciback: 0000:06:00.0: write request 4 bytes at 0x30 = 0
xen-pciback: 0000:06:00.0: read 4 bytes at 0x30
xen-pciback: 0000:06:00.0: read 4 bytes at 0x30 = 0
xen-pciback: 0000:06:00.0: read 2 bytes at 0x2c
xen-pciback: 0000:06:00.0: read 2 bytes at 0x2c = 0
xen-pciback: 0000:06:00.0: read 2 bytes at 0x2e
xen-pciback: 0000:06:00.0: read 2 bytes at 0x2e = 0
xen-pciback: 0000:06:00.0: read 2 bytes at 0x6
xen-pciback: 0000:06:00.0: read 2 bytes at 0x6 = 200
xen-pciback: 0000:06:00.0: read 2 bytes at 0x6
xen-pciback: 0000:06:00.0: read 2 bytes at 0x6 = 200
xen-pciback: 0000:06:00.0: read 2 bytes at 0x6
xen-pciback: 0000:06:00.0: read 2 bytes at 0x6 = 200
xen-pciback: 0000:06:00.0: read 2 bytes at 0x6
xen-pciback: 0000:06:00.0: read 2 bytes at 0x6 = 200
xen-pciback: 0000:06:00.0: read 2 bytes at 0x6
xen-pciback: 0000:06:00.0: read 2 bytes at 0x6 = 200
xen-pciback: 0000:06:00.0: read 2 bytes at 0x6
xen-pciback: 0000:06:00.0: read 2 bytes at 0x6 = 200
xen-pciback: 0000:06:00.0: read 2 bytes at 0x6
xen-pciback: 0000:06:00.0: read 2 bytes at 0x6 = 200
xen-pciback: 0000:06:00.0: read 2 bytes at 0x6
xen-pciback: 0000:06:00.0: read 2 bytes at 0x6 = 200
xen-pciback: 0000:06:00.0: read 2 bytes at 0x4
xen-pciback: 0000:06:00.0: read 2 bytes at 0x4 = 0
xen-pciback: 0000:06:00.0: write request 2 bytes at 0x4 = 2
xen-pciback: 0000:06:00.0: read 2 bytes at 0x4
xen-pciback: 0000:06:00.0: read 2 bytes at 0x4 = 0
xen-pciback: 0000:06:00.0: enable
pciback 0000:06:00.0: enabling device (0000 -> 0002)
xen: registering gsi 16 triggering 0 polarity 1
Already setup the GSI :16
xen-pciback: 0000:06:00.0: read 1 bytes at 0x3c
xen-pciback: 0000:06:00.0: read 1 bytes at 0x3c = 10
xen-pciback: 0000:06:00.0: read 1 bytes at 0x3d
xen-pciback: 0000:06:00.0: read 1 bytes at 0x3d = 1
xen-pciback: 0000:06:00.0: read 2 bytes at 0x4
xen-pciback: 0000:06:00.0: read 2 bytes at 0x4 = 2
[-- Attachment #4: pci-attach-patch.txt --]
[-- Type: text/plain, Size: 7082 bytes --]
pciback 0000:06:00.0: enabling permissive mode configuration space accesses!
pciback 0000:06:00.0: permissive mode is potentially unsafe!
xen_pciback: vpci: 0000:06:00.0: assign to virtual slot 0
pciback 0000:06:00.0: registering for 33
xen-pciback: 0000:06:00.0: read 4 bytes at 0x0
xen-pciback: 0000:06:00.0: read 4 bytes at 0x0 = 15cf
xen-pciback: 0000:06:00.0: read 1 bytes at 0xe
xen-pciback: 0000:06:00.0: read 1 bytes at 0xe = 0
xen-pciback: 0000:06:00.0: read 2 bytes at 0x6
xen-pciback: 0000:06:00.0: read 2 bytes at 0x6 = 200
xen-pciback: 0000:06:00.0: read 4 bytes at 0x8
xen-pciback: 0000:06:00.0: read 4 bytes at 0x8 = ff000000
xen-pciback: 0000:06:00.0: read 2 bytes at 0x6
xen-pciback: 0000:06:00.0: read 2 bytes at 0x6 = 200
xen-pciback: 0000:06:00.0: read 1 bytes at 0x3d
xen-pciback: 0000:06:00.0: read 1 bytes at 0x3d = 1
xen-pciback: 0000:06:00.0: read 1 bytes at 0x3c
xen-pciback: 0000:06:00.0: read 1 bytes at 0x3c = 10
xen-pciback: 0000:06:00.0: read 2 bytes at 0x4
xen-pciback: 0000:06:00.0: read 2 bytes at 0x4 = 0
xen-pciback: 0000:06:00.0: read 4 bytes at 0x10
xen-pciback: 0000:06:00.0: read 4 bytes at 0x10 = f7a00000
xen-pciback: 0000:06:00.0: write request 4 bytes at 0x10 = ffffffff
xen-pciback: 0000:06:00.0: read 4 bytes at 0x10
xen-pciback: 0000:06:00.0: read 4 bytes at 0x10 = f7a00000
xen-pciback: 0000:06:00.0: read 4 bytes at 0x10
xen-pciback: 0000:06:00.0: read 4 bytes at 0x10 = ffff0000
xen-pciback: 0000:06:00.0: write request 4 bytes at 0x10 = f7a00000
xen-pciback: 0000:06:00.0: read 4 bytes at 0x10
xen-pciback: 0000:06:00.0: read 4 bytes at 0x10 = ffff0000
xen-pciback: 0000:06:00.0: read 2 bytes at 0x4
xen-pciback: 0000:06:00.0: read 2 bytes at 0x4 = 0
xen-pciback: 0000:06:00.0: read 4 bytes at 0x14
xen-pciback: 0000:06:00.0: read 4 bytes at 0x14 = 0
xen-pciback: 0000:06:00.0: write request 4 bytes at 0x14 = ffffffff
xen-pciback: 0000:06:00.0: read 4 bytes at 0x14
xen-pciback: 0000:06:00.0: read 4 bytes at 0x14 = 0
xen-pciback: 0000:06:00.0: read 4 bytes at 0x14
xen-pciback: 0000:06:00.0: read 4 bytes at 0x14 = 0
xen-pciback: 0000:06:00.0: write request 4 bytes at 0x14 = 0
xen-pciback: 0000:06:00.0: read 4 bytes at 0x14
xen-pciback: 0000:06:00.0: read 4 bytes at 0x14 = 0
xen-pciback: 0000:06:00.0: read 2 bytes at 0x4
xen-pciback: 0000:06:00.0: read 2 bytes at 0x4 = 0
xen-pciback: 0000:06:00.0: read 4 bytes at 0x18
xen-pciback: 0000:06:00.0: read 4 bytes at 0x18 = 0
xen-pciback: 0000:06:00.0: write request 4 bytes at 0x18 = ffffffff
xen-pciback: 0000:06:00.0: read 4 bytes at 0x18
xen-pciback: 0000:06:00.0: read 4 bytes at 0x18 = 0
xen-pciback: 0000:06:00.0: read 4 bytes at 0x18
xen-pciback: 0000:06:00.0: read 4 bytes at 0x18 = 0
xen-pciback: 0000:06:00.0: write request 4 bytes at 0x18 = 0
xen-pciback: 0000:06:00.0: read 4 bytes at 0x18
xen-pciback: 0000:06:00.0: read 4 bytes at 0x18 = 0
xen-pciback: 0000:06:00.0: read 2 bytes at 0x4
xen-pciback: 0000:06:00.0: read 2 bytes at 0x4 = 0
xen-pciback: 0000:06:00.0: read 4 bytes at 0x1c
xen-pciback: 0000:06:00.0: read 4 bytes at 0x1c = 0
xen-pciback: 0000:06:00.0: write request 4 bytes at 0x1c = ffffffff
xen-pciback: 0000:06:00.0: read 4 bytes at 0x1c
xen-pciback: 0000:06:00.0: read 4 bytes at 0x1c = 0
xen-pciback: 0000:06:00.0: read 4 bytes at 0x1c
xen-pciback: 0000:06:00.0: read 4 bytes at 0x1c = 0
xen-pciback: 0000:06:00.0: write request 4 bytes at 0x1c = 0
xen-pciback: 0000:06:00.0: read 4 bytes at 0x1c
xen-pciback: 0000:06:00.0: read 4 bytes at 0x1c = 0
xen-pciback: 0000:06:00.0: read 2 bytes at 0x4
xen-pciback: 0000:06:00.0: read 2 bytes at 0x4 = 0
xen-pciback: 0000:06:00.0: read 4 bytes at 0x20
xen-pciback: 0000:06:00.0: read 4 bytes at 0x20 = 0
xen-pciback: 0000:06:00.0: write request 4 bytes at 0x20 = ffffffff
xen-pciback: 0000:06:00.0: read 4 bytes at 0x20
xen-pciback: 0000:06:00.0: read 4 bytes at 0x20 = 0
xen-pciback: 0000:06:00.0: read 4 bytes at 0x20
xen-pciback: 0000:06:00.0: read 4 bytes at 0x20 = 0
xen-pciback: 0000:06:00.0: write request 4 bytes at 0x20 = 0
xen-pciback: 0000:06:00.0: read 4 bytes at 0x20
xen-pciback: 0000:06:00.0: read 4 bytes at 0x20 = 0
xen-pciback: 0000:06:00.0: read 2 bytes at 0x4
xen-pciback: 0000:06:00.0: read 2 bytes at 0x4 = 0
xen-pciback: 0000:06:00.0: read 4 bytes at 0x24
xen-pciback: 0000:06:00.0: read 4 bytes at 0x24 = 0
xen-pciback: 0000:06:00.0: write request 4 bytes at 0x24 = ffffffff
xen-pciback: 0000:06:00.0: read 4 bytes at 0x24
xen-pciback: 0000:06:00.0: read 4 bytes at 0x24 = 0
xen-pciback: 0000:06:00.0: read 4 bytes at 0x24
xen-pciback: 0000:06:00.0: read 4 bytes at 0x24 = 0
xen-pciback: 0000:06:00.0: write request 4 bytes at 0x24 = 0
xen-pciback: 0000:06:00.0: read 4 bytes at 0x24
xen-pciback: 0000:06:00.0: read 4 bytes at 0x24 = 0
xen-pciback: 0000:06:00.0: read 2 bytes at 0x4
xen-pciback: 0000:06:00.0: read 2 bytes at 0x4 = 0
xen-pciback: 0000:06:00.0: read 4 bytes at 0x30
xen-pciback: 0000:06:00.0: read 4 bytes at 0x30 = 0
xen-pciback: 0000:06:00.0: write request 4 bytes at 0x30 = fffff800
xen-pciback: 0000:06:00.0: read 4 bytes at 0x30
xen-pciback: 0000:06:00.0: read 4 bytes at 0x30 = 0
xen-pciback: 0000:06:00.0: read 4 bytes at 0x30
xen-pciback: 0000:06:00.0: read 4 bytes at 0x30 = 0
xen-pciback: 0000:06:00.0: write request 4 bytes at 0x30 = 0
xen-pciback: 0000:06:00.0: read 4 bytes at 0x30
xen-pciback: 0000:06:00.0: read 4 bytes at 0x30 = 0
xen-pciback: 0000:06:00.0: read 2 bytes at 0x2c
xen-pciback: 0000:06:00.0: read 2 bytes at 0x2c = 0
xen-pciback: 0000:06:00.0: read 2 bytes at 0x2e
xen-pciback: 0000:06:00.0: read 2 bytes at 0x2e = 0
xen-pciback: 0000:06:00.0: read 2 bytes at 0x6
xen-pciback: 0000:06:00.0: read 2 bytes at 0x6 = 200
xen-pciback: 0000:06:00.0: read 2 bytes at 0x6
xen-pciback: 0000:06:00.0: read 2 bytes at 0x6 = 200
xen-pciback: 0000:06:00.0: read 2 bytes at 0x6
xen-pciback: 0000:06:00.0: read 2 bytes at 0x6 = 200
xen-pciback: 0000:06:00.0: read 2 bytes at 0x6
xen-pciback: 0000:06:00.0: read 2 bytes at 0x6 = 200
xen-pciback: 0000:06:00.0: read 2 bytes at 0x6
xen-pciback: 0000:06:00.0: read 2 bytes at 0x6 = 200
xen-pciback: 0000:06:00.0: read 2 bytes at 0x6
xen-pciback: 0000:06:00.0: read 2 bytes at 0x6 = 200
xen-pciback: 0000:06:00.0: read 2 bytes at 0x6
xen-pciback: 0000:06:00.0: read 2 bytes at 0x6 = 200
xen-pciback: 0000:06:00.0: read 2 bytes at 0x6
xen-pciback: 0000:06:00.0: read 2 bytes at 0x6 = 200
xen-pciback: 0000:06:00.0: read 2 bytes at 0x4
xen-pciback: 0000:06:00.0: read 2 bytes at 0x4 = 0
xen-pciback: 0000:06:00.0: write request 2 bytes at 0x4 = 2
xen-pciback: 0000:06:00.0: read 2 bytes at 0x4
xen-pciback: 0000:06:00.0: read 2 bytes at 0x4 = 0
xen-pciback: 0000:06:00.0: enable
pciback 0000:06:00.0: enabling device (0000 -> 0002)
xen: registering gsi 16 triggering 0 polarity 1
Already setup the GSI :16
xen-pciback: 0000:06:00.0: read 1 bytes at 0x3c
xen-pciback: 0000:06:00.0: read 1 bytes at 0x3c = 10
xen-pciback: 0000:06:00.0: read 1 bytes at 0x3d
xen-pciback: 0000:06:00.0: read 1 bytes at 0x3d = 1
xen-pciback: 0000:06:00.0: read 2 bytes at 0x4
xen-pciback: 0000:06:00.0: read 2 bytes at 0x4 = 2
[-- Attachment #5: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Issues with PCI-Passtrough (VT-d) in HVM with Xen 4.6
2016-06-03 13:26 ` Jan Beulich
2016-06-03 14:20 ` Jan Beulich
@ 2016-06-04 15:15 ` Jürgen Walter
2016-06-06 8:41 ` Jan Beulich
1 sibling, 1 reply; 22+ messages in thread
From: Jürgen Walter @ 2016-06-04 15:15 UTC (permalink / raw)
To: Jan Beulich; +Cc: Sylwester Sosnowski, xen-devel
On 3 Jun 2016, at 15:26, Jan Beulich wrote:
> I'll see to put together a patch for the recognizable problem,
> but I'm still unclear about your actual one.
regarding the actual one: while it is still not working, I managed to
dig deeper and found a few observations
background: there is a driver/kernel module, and a user space library,
and a "test console"
the "test console" uses the library, and invokes "cifXInit" (initialise
the PCI card)
during init, it does
a) read pci config (space)
b) send reset sequence to the PCI device
c) writes back the pci config saved from step "a)" (restore)
I used "lspci -xxx -s BDF" to read that config space, now
- on the Dom0 - when I run that "test console", everything works as
expected, and the "lspci -xxx" output is the same before and after the
reset sequence (there is a delay built-in to allow rebooting of the ARM
chip on the device)
before the reset:
06:00.0 Unassigned class [ff00]: Hilscher GmbH CIFX 50E-DP(M/S)
00: cf 15 00 00 02 01 00 02 00 00 00 ff 00 40 00 00
10: 00 00 a0 f7 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00
30: 00 00 00 00 00 00 00 00 00 00 00 00 0b 01 00 00
40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
diff before-reset after-reset:
2,3c2,3
< 00: cf 15 00 00 02 01 00 02 00 00 00 ff 00 40 00 00
< 10: 00 00 a0 f7 00 00 00 00 00 00 00 00 00 00 00 00
---
> 00: cf 15 00 00 00 00 00 02 00 00 00 ff 00 00 00 00
> 10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5c5
< 30: 00 00 00 00 00 00 00 00 00 00 00 00 0b 01 00 00
---
> 30: 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00
and it is 100% identical to "before reset" after the configuration
restore operation
- on the DomU - when I run that "test console" tool, the "lspci -xxx"
output is different from before/after
not much, though, only one register(?)
diff lspci.before-testconsole lspci.after-testconsole
2c2
< 00: cf 15 00 00 02 01 00 02 00 00 00 ff 00 40 00 00
---
> 00: cf 15 00 00 02 01 00 02 00 00 00 ff 00 00 00 00
Now, I my desperation I found out about "setpci" and thought I give it a
shot, alas no dice, it does not seem to write
root@DomU: setpci -s 00:00.0 D.B # the device is at 00.00.0
log from Dom0 # the device is at 06.00.0
xen-pciback: 0000:06:00.0: read 1 bytes at 0xd
xen-pciback: 0000:06:00.0: read 1 bytes at 0xd = 0
root@DomU: setpci -s 00:00.0 D.B=40 # the device is at 00.00.0
log from Dom0 # the device is at 06.00.0
xen-pciback: 0000:06:00.0: write request 1 bytes at 0xd = 40
xen-pciback: 0000:06:00.0: read 1 bytes at 0xd
xen-pciback: 0000:06:00.0: read 1 bytes at 0xd = 0
Even though, when I do "cat /sys/bus/pci/drivers/pciback/quirks" it duly
lists this address as writeable:
06:00.0
15cf:0000:0000:0000
00000000:2:00000000
00000002:2:00000000
00000004:2:00000000
0000003c:1:00000000
0000003d:1:00000000
0000000c:1:00000000
0000000d:1:00000000 <------- ok to write 1 byte
0000000f:1:00000000
00000010:4:00000000
00000014:4:00000000
00000018:4:00000000
0000001c:4:00000000
00000020:4:00000000
00000024:4:00000000
00000030:4:00000000
Lastly, this is the only difference of "lspci -xxx" (after a _PCI_
device reset) between Dom0 and DomU
diff lspci.before-domU lspci.before-dom0
< 30: 00 00 00 00 00 00 00 00 00 00 00 00 10 01 00 00
---
> 30: 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00
Long story short, not sure where to look anymore :(
It just seems like this thing is not "coming back" properly after the
driver send the device reset.
The comment in the driver source say, if this address has 0xFFFFFFFF
"This happens if memory is not available."
(but I doubt, this is necessarily the correct error message)
Very lastly, when I do "xl dmesg | grep VT-d" - there is some msg re: no
Dom0 DMA passtrough
(XEN) Intel VT-d iommu 0 supported page sizes: 4kB.
(XEN) Intel VT-d iommu 1 supported page sizes: 4kB.
(XEN) Intel VT-d Snoop Control not enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled. <------ ??
important ???
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping enabled.
(XEN) Intel VT-d Shared EPT tables not enabled.
Thanks again for reading that far! Jürgen
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Issues with PCI-Passtrough (VT-d) in HVM with Xen 4.6
2016-06-04 14:36 ` Jürgen Walter
@ 2016-06-06 7:59 ` Jan Beulich
0 siblings, 0 replies; 22+ messages in thread
From: Jan Beulich @ 2016-06-06 7:59 UTC (permalink / raw)
To: Jürgen Walter; +Cc: Sylwester Sosnowski, xen-devel
>>> On 04.06.16 at 16:36, <juwalter@gmail.com> wrote:
> On 3 Jun 2016, at 16:20, Jan Beulich wrote:
>> Mind trying out the attached patch?
>
> Many thanks for your patch!! I did try it, and it is different now
> (details in files attached)
>
> "best of conf_space_header" see below.
>
> I cannot really make sense of it, though. Can you?
Sure - it's now as expected. As said - there are read-only bits at the
bottom of all of the BAR registers.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Issues with PCI-Passtrough (VT-d) in HVM with Xen 4.6
2016-06-04 15:15 ` Jürgen Walter
@ 2016-06-06 8:41 ` Jan Beulich
2016-06-06 9:09 ` Jürgen Walter
2016-06-06 14:01 ` Boris Ostrovsky
0 siblings, 2 replies; 22+ messages in thread
From: Jan Beulich @ 2016-06-06 8:41 UTC (permalink / raw)
To: Jürgen Walter
Cc: Juergen Gross, xen-devel, David Vrabel, Boris Ostrovsky,
Sylwester Sosnowski
>>> On 04.06.16 at 17:15, <juwalter@gmail.com> wrote:
> On 3 Jun 2016, at 15:26, Jan Beulich wrote:
>> I'll see to put together a patch for the recognizable problem,
>> but I'm still unclear about your actual one.
>
> regarding the actual one: while it is still not working, I managed to
> dig deeper and found a few observations
>
> background: there is a driver/kernel module, and a user space library,
> and a "test console"
> the "test console" uses the library, and invokes "cifXInit" (initialise
> the PCI card)
>
> during init, it does
> a) read pci config (space)
> b) send reset sequence to the PCI device
> c) writes back the pci config saved from step "a)" (restore)
I'm sorry to mention that, but so far you still didn't provide a
matching pciback log for that portion of the operation, at least
afaict.
> - on the DomU - when I run that "test console" tool, the "lspci -xxx"
> output is different from before/after
> not much, though, only one register(?)
>
> diff lspci.before-testconsole lspci.after-testconsole
> 2c2
> < 00: cf 15 00 00 02 01 00 02 00 00 00 ff 00 40 00 00
> ---
>> 00: cf 15 00 00 02 01 00 02 00 00 00 ff 00 00 00 00
Okay, that's the Latency Timer field, and I think just like BARs we
may need to permit restoring this field. However, please note that
the reset being done behind the back of pciback really is the
problem here: pciback assumes (for a reason, as you can certainly
understand) that it has full control over config space of a passed
through device.
And actually the latency timer would, as a side effect of enabling
bus mastering on the device (via the pci_set_master() call from
command_write()) set the Latency Timer field properly, just that
again pciback (and the rest of Dom0's PCI subsystem) thinks that
bus mastering is already enabled on the device. So perhaps in
permissive mode we should simply allow the latency timer field to
be written, just like we allow writing various of the Command
Register bits in that mode. Maintainers, what do you think?
If we decide to go that route, I would then wonder whether
Cache Line Size being unconditionally writable right now would
also better be restricted to permissive mode.
In any event, Jürgen, it would be helpful if you could confirm
that instead of your gross hack, only permitting the actual write
of that one field would also make the device usable by the guest.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Issues with PCI-Passtrough (VT-d) in HVM with Xen 4.6
2016-06-06 8:41 ` Jan Beulich
@ 2016-06-06 9:09 ` Jürgen Walter
2016-06-06 9:43 ` Jan Beulich
2016-06-06 14:01 ` Boris Ostrovsky
1 sibling, 1 reply; 22+ messages in thread
From: Jürgen Walter @ 2016-06-06 9:09 UTC (permalink / raw)
To: Jan Beulich
Cc: Juergen Gross, xen-devel, David Vrabel, Boris Ostrovsky,
Sylwester Sosnowski
Hi Jan,
On 6 Jun 2016, at 10:41, Jan Beulich wrote:
>>>> On 04.06.16 at 17:15, <juwalter@gmail.com> wrote:
>> On 3 Jun 2016, at 15:26, Jan Beulich wrote:
>>> I'll see to put together a patch for the recognizable problem,
>>> but I'm still unclear about your actual one.
>>
>> regarding the actual one: while it is still not working, I managed to
>> dig deeper and found a few observations
>>
>> background: there is a driver/kernel module, and a user space
>> library,
>> and a "test console"
>> the "test console" uses the library, and invokes "cifXInit"
>> (initialise
>> the PCI card)
>>
>> during init, it does
>> a) read pci config (space)
>> b) send reset sequence to the PCI device
>> c) writes back the pci config saved from step "a)" (restore)
>
> I'm sorry to mention that, but so far you still didn't provide a
> matching pciback log for that portion of the operation, at least
> afaict.
sorry about that- we managed to pull this together. note: we changed the
user space library to perform another read after the write/restore
we did on Dom0 and DomU, "inside permissive mode" printk statement we
added if it goes into "if (!handled ...)"
DOM0
# read PCI config and save for later restore
(gdb) x/64wx 0x622190
0x622190: 0x000015cf 0x02000107 0xff000000 0x00004000
0x6221a0: 0xf7a00000 0x00000000 0x00000000 0x00000000
0x6221b0: 0x00000000 0x00000000 0x00000001 0x00000000
0x6221c0: 0x00000000 0x00000000 0x00000000 0x0000010b
0x6221d0: 0x00000000 0x00000000 0x00000000 0x00000000
0x6221e0: 0x00000000 0x00000000 0x00000000 0x00000000
0x6221f0: 0x00000000 0x00000000 0x00000000 0x00000000
0x622200: 0x00000000 0x00000000 0x00000000 0x00000000
0x622210: 0x00000000 0x00000000 0x00000000 0x00000000
0x622220: 0x00000000 0x00000000 0x00000000 0x00000000
0x622230: 0x00000000 0x00000000 0x00000000 0x00000000
0x622240: 0x00000000 0x00000000 0x00000000 0x00000000
0x622250: 0x00000000 0x00000000 0x00000000 0x00000000
0x622260: 0x00000000 0x00000000 0x00000000 0x00000000
0x622270: 0x00000000 0x00000000 0x00000000 0x00000000
0x622280: 0x00000000 0x00000000 0x00000000 0x00000000
# write/restore PCI config
# ...
# read PCI config again and see what is there now
(gdb) x/64w 0x622190
0x622190: 0x000015cf 0x02000107 0xff000000 0x00004000 <--- worked
0x6221a0: 0xf7a00000 0x00000000 0x00000000 0x00000000
0x6221b0: 0x00000000 0x00000000 0x00000001 0x00000000
0x6221c0: 0x00000000 0x00000000 0x00000000 0x0000010b
0x6221d0: 0x00000000 0x00000000 0x00000000 0x00000000
0x6221e0: 0x00000000 0x00000000 0x00000000 0x00000000
0x6221f0: 0x00000000 0x00000000 0x00000000 0x00000000
0x622200: 0x00000000 0x00000000 0x00000000 0x00000000
0x622210: 0x00000000 0x00000000 0x00000000 0x00000000
0x622220: 0x00000000 0x00000000 0x00000000 0x00000000
0x622230: 0x00000000 0x00000000 0x00000000 0x00000000
0x622240: 0x00000000 0x00000000 0x00000000 0x00000000
0x622250: 0x00000000 0x00000000 0x00000000 0x00000000
0x622260: 0x00000000 0x00000000 0x00000000 0x00000000
0x622270: 0x00000000 0x00000000 0x00000000 0x00000000
0x622280: 0x00000000 0x00000000 0x00000000 0x00000000
(gdb)
DOMU
# read PCI config and save for later restore
(gdb) p pvPCIConfig
$2 = (void *) 0x61ef90
(gdb) x/64wx 0x61ef90
0x61ef90: 0x000015cf 0x02000102 0xff000000 0x00004000
0x61efa0: 0xf7a00000 0x00000000 0x00000000 0x00000000
0x61efb0: 0x00000000 0x00000000 0x00000001 0x00000000
0x61efc0: 0x00000000 0x00000000 0x00000000 0x00000110
0x61efd0: 0x00000000 0x00000000 0x00000000 0x00000000
0x61efe0: 0x00000000 0x00000000 0x00000000 0x00000000
0x61eff0: 0x00000000 0x00000000 0x00000000 0x00000000
0x61f000: 0x00000000 0x00000000 0x00000000 0x00000000
0x61f010: 0x00000000 0x00000000 0x00000000 0x00000000
0x61f020: 0x00000000 0x00000000 0x00000000 0x00000000
0x61f030: 0x00000000 0x00000000 0x00000000 0x00000000
0x61f040: 0x00000000 0x00000000 0x00000000 0x00000000
0x61f050: 0x00000000 0x00000000 0x00000000 0x00000000
0x61f060: 0x00000000 0x00000000 0x00000000 0x00000000
0x61f070: 0x00000000 0x00000000 0x00000000 0x00000000
0x61f080: 0x00000000 0x00000000 0x00000000 0x00000000
# write/restore PCI config
# ...
# read PCI config again and see what is there now
(gdb) p OS_ReadPCIConfig(ptDevInstance->pvOSDependent)
$3 = (void *) 0x61ef90
(gdb) x/64wx 0x61ef90
0x61ef90: 0x000015cf 0x02000102 0xff000000 0x00000000 <--- failed
0x61efa0: 0xf7a00000 0x00000000 0x00000000 0x00000000
0x61efb0: 0x00000000 0x00000000 0x00000001 0x00000000
0x61efc0: 0x00000000 0x00000000 0x00000000 0x00000110
0x61efd0: 0x00000000 0x00000000 0x00000000 0x00000000
0x61efe0: 0x00000000 0x00000000 0x00000000 0x00000000
0x61eff0: 0x00000000 0x00000000 0x00000000 0x00000000
0x61f000: 0x00000000 0x00000000 0x00000000 0x00000000
0x61f010: 0x00000000 0x00000000 0x00000000 0x00000000
0x61f020: 0x00000000 0x00000000 0x00000000 0x00000000
0x61f030: 0x00000000 0x00000000 0x00000000 0x00000000
0x61f040: 0x00000000 0x00000000 0x00000000 0x00000000
0x61f050: 0x00000000 0x00000000 0x00000000 0x00000000
0x61f060: 0x00000000 0x00000000 0x00000000 0x00000000
0x61f070: 0x00000000 0x00000000 0x00000000 0x00000000
0x61f080: 0x00000000 0x00000000 0x00000000 0x00000000
[ 1466.964782 < 139.522689>] xen-pciback: 0000:06:00.0: write request 4
bytes at 0x0 = 15cf
[ 1466.964786 < 0.000004>] xen-pciback: 0000:06:00.0: read 2 bytes at
0x0
[ 1466.964799 < 0.000013>] xen-pciback: 0000:06:00.0: read 2 bytes at
0x0 = 15cf
[ 1466.964801 < 0.000002>] xen-pciback: 0000:06:00.0: read 2 bytes at
0x2
[ 1466.964807 < 0.000006>] xen-pciback: 0000:06:00.0: read 2 bytes at
0x2 = 0
[ 1466.964828 < 0.000021>] xen-pciback: 0000:06:00.0: write request 4
bytes at 0x4 = 2000102
[ 1466.964829 < 0.000001>] xen-pciback: 0000:06:00.0: read 2 bytes at
0x4
[ 1466.964841 < 0.000012>] xen-pciback: 0000:06:00.0: read 2 bytes at
0x4 = 102
[ 1466.964870 < 0.000029>] xen-pciback: 0000:06:00.0: write request 4
bytes at 0x8 = ff000000
[ 1466.964872 < 0.000002>] xen-pciback: 0000:06:00.0: inside
permissive mode - write request 4 bytes at 0x8 = ff000000
[ 1466.964895 < 0.000023>] xen-pciback: 0000:06:00.0: write request 4
bytes at 0xc = 4000
[ 1466.964897 < 0.000002>] xen-pciback: 0000:06:00.0: read 1 bytes at
0xc
[ 1466.964907 < 0.000010>] xen-pciback: 0000:06:00.0: read 1 bytes at
0xc = 0
[ 1466.964914 < 0.000007>] xen-pciback: 0000:06:00.0: read 1 bytes at
0xf
[ 1466.964925 < 0.000011>] xen-pciback: 0000:06:00.0: read 1 bytes at
0xf = 0
[ 1466.964947 < 0.000022>] xen-pciback: 0000:06:00.0: write request 4
bytes at 0x10 = f7a00000
[ 1466.964948 < 0.000001>] xen-pciback: 0000:06:00.0: read 4 bytes at
0x10
[ 1466.964955 < 0.000007>] xen-pciback: 0000:06:00.0: read 4 bytes at
0x10 = f7a00000
[ 1466.964972 < 0.000017>] xen-pciback: 0000:06:00.0: write request 4
bytes at 0x14 = 0
[ 1466.964973 < 0.000001>] xen-pciback: 0000:06:00.0: read 4 bytes at
0x14
[ 1466.964980 < 0.000007>] xen-pciback: 0000:06:00.0: read 4 bytes at
0x14 = 0
[ 1466.964998 < 0.000018>] xen-pciback: 0000:06:00.0: write request 4
bytes at 0x18 = 0
[ 1466.964999 < 0.000001>] xen-pciback: 0000:06:00.0: read 4 bytes at
0x18
[ 1466.965006 < 0.000007>] xen-pciback: 0000:06:00.0: read 4 bytes at
0x18 = 0
>> - on the DomU - when I run that "test console" tool, the "lspci -xxx"
>> output is different from before/after
>> not much, though, only one register(?)
>>
>> diff lspci.before-testconsole lspci.after-testconsole
>> 2c2
>> < 00: cf 15 00 00 02 01 00 02 00 00 00 ff 00 40 00 00
>> ---
>>> 00: cf 15 00 00 02 01 00 02 00 00 00 ff 00 00 00 00
>
> Okay, that's the Latency Timer field, and I think just like BARs we
> may need to permit restoring this field. However, please note that
> the reset being done behind the back of pciback really is the
> problem here: pciback assumes (for a reason, as you can certainly
> understand) that it has full control over config space of a passed
> through device.
I agree that it is very ugly to reset the device behind the back of
pciback.
However, this is how the vendor decided to do it, and have no better
idea how
to accomplish this.
While I cannot image that there is, but can you think think of
any "correct" way to reset a pci device from
a) inside a DomU
b) from the user space library?
What I cannot do: modify the firmware or the HW of the PCI device
(which would be good, because then it could simply do
all memory cleaning etc. w/o affecting its PCI config known to the host)
What I can do: rewrite all parts of the driver (uio_netx) and user space
library.
> And actually the latency timer would, as a side effect of enabling
> bus mastering on the device (via the pci_set_master() call from
> command_write()) set the Latency Timer field properly, just that
> again pciback (and the rest of Dom0's PCI subsystem) thinks that
> bus mastering is already enabled on the device. So perhaps in
> permissive mode we should simply allow the latency timer field to
> be written, just like we allow writing various of the Command
> Register bits in that mode. Maintainers, what do you think?
(is that something that /sys/bus/pci/drivers/pciback/quirks could
help with? according to docs, this is only used/in effect when
in permissive mode)
> If we decide to go that route, I would then wonder whether
> Cache Line Size being unconditionally writable right now would
> also better be restricted to permissive mode.
> In any event, Jürgen, it would be helpful if you could confirm
If I understood you correctly, I modify xen_pcibk_config_write
in "drivers/xen/xen-pciback/conf_space.c" to specifically allow only
this field (latency timer) to be written and log everything.
Will do and report back!
Thanks for your time!!!
Jürgen
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Issues with PCI-Passtrough (VT-d) in HVM with Xen 4.6
2016-06-06 9:09 ` Jürgen Walter
@ 2016-06-06 9:43 ` Jan Beulich
0 siblings, 0 replies; 22+ messages in thread
From: Jan Beulich @ 2016-06-06 9:43 UTC (permalink / raw)
To: Jürgen Walter
Cc: Juergen Gross, xen-devel, David Vrabel, Boris Ostrovsky,
Sylwester Sosnowski
>>> On 06.06.16 at 11:09, <juwalter@gmail.com> wrote:
> [ 1466.964895 < 0.000023>] xen-pciback: 0000:06:00.0: write request 4
> bytes at 0xc = 4000
> [ 1466.964897 < 0.000002>] xen-pciback: 0000:06:00.0: read 1 bytes at
> 0xc
> [ 1466.964907 < 0.000010>] xen-pciback: 0000:06:00.0: read 1 bytes at
> 0xc = 0
> [ 1466.964914 < 0.000007>] xen-pciback: 0000:06:00.0: read 1 bytes at
> 0xf
> [ 1466.964925 < 0.000011>] xen-pciback: 0000:06:00.0: read 1 bytes at
> 0xf = 0
No read of 0xd and 0xe?
> I agree that it is very ugly to reset the device behind the back of
> pciback.
> However, this is how the vendor decided to do it, and have no better
> idea how
> to accomplish this.
> While I cannot image that there is, but can you think think of
> any "correct" way to reset a pci device from
> a) inside a DomU
> b) from the user space library?
No, resetting a device shouldn't be needed at all _after_ assignment
to a guest. Do you have any insight why that's needed here in the
first place?
>> And actually the latency timer would, as a side effect of enabling
>> bus mastering on the device (via the pci_set_master() call from
>> command_write()) set the Latency Timer field properly, just that
>> again pciback (and the rest of Dom0's PCI subsystem) thinks that
>> bus mastering is already enabled on the device. So perhaps in
>> permissive mode we should simply allow the latency timer field to
>> be written, just like we allow writing various of the Command
>> Register bits in that mode. Maintainers, what do you think?
> (is that something that /sys/bus/pci/drivers/pciback/quirks could
> help with? according to docs, this is only used/in effect when
> in permissive mode)
I don't think so - quirks can, afaict, only be registered for fields that
don't have explicit handling associated to them (see the
xen_pcibk_field_is_dup() call in xen_pcibk_config_add_field_offset()).
>> If we decide to go that route, I would then wonder whether
>> Cache Line Size being unconditionally writable right now would
>> also better be restricted to permissive mode.
>
>> In any event, Jürgen, it would be helpful if you could confirm
> If I understood you correctly, I modify xen_pcibk_config_write
> in "drivers/xen/xen-pciback/conf_space.c" to specifically allow only
> this field (latency timer) to be written and log everything.
Right - just limit your existing workaround to just this one field.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Issues with PCI-Passtrough (VT-d) in HVM with Xen 4.6
2016-06-06 8:41 ` Jan Beulich
2016-06-06 9:09 ` Jürgen Walter
@ 2016-06-06 14:01 ` Boris Ostrovsky
2016-06-06 14:21 ` Jan Beulich
1 sibling, 1 reply; 22+ messages in thread
From: Boris Ostrovsky @ 2016-06-06 14:01 UTC (permalink / raw)
To: Jan Beulich, Jürgen Walter
Cc: Juergen Gross, xen-devel, David Vrabel, Sylwester Sosnowski
On 06/06/2016 04:41 AM, Jan Beulich wrote:
>
>> - on the DomU - when I run that "test console" tool, the "lspci -xxx"
>> output is different from before/after
>> not much, though, only one register(?)
>>
>> diff lspci.before-testconsole lspci.after-testconsole
>> 2c2
>> < 00: cf 15 00 00 02 01 00 02 00 00 00 ff 00 40 00 00
>> ---
>>> 00: cf 15 00 00 02 01 00 02 00 00 00 ff 00 00 00 00
> Okay, that's the Latency Timer field, and I think just like BARs we
> may need to permit restoring this field. However, please note that
> the reset being done behind the back of pciback really is the
> problem here: pciback assumes (for a reason, as you can certainly
> understand) that it has full control over config space of a passed
> through device.
>
> And actually the latency timer would, as a side effect of enabling
> bus mastering on the device (via the pci_set_master() call from
> command_write()) set the Latency Timer field properly, just that
> again pciback (and the rest of Dom0's PCI subsystem) thinks that
> bus mastering is already enabled on the device. So perhaps in
> permissive mode we should simply allow the latency timer field to
> be written, just like we allow writing various of the Command
> Register bits in that mode. Maintainers, what do you think?
Don't we currently allow it to be written unconditionally? It is no
different from accessing Cache Line Size.
Or are you suggesting to make it stricter?
>
> If we decide to go that route, I would then wonder whether
> Cache Line Size being unconditionally writable right now would
> also better be restricted to permissive mode.
-boris
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Issues with PCI-Passtrough (VT-d) in HVM with Xen 4.6
2016-06-06 14:01 ` Boris Ostrovsky
@ 2016-06-06 14:21 ` Jan Beulich
2016-06-06 14:45 ` Boris Ostrovsky
0 siblings, 1 reply; 22+ messages in thread
From: Jan Beulich @ 2016-06-06 14:21 UTC (permalink / raw)
To: Boris Ostrovsky
Cc: Juergen Gross, juwalter, xen-devel, David Vrabel, Sylwester Sosnowski
>>> On 06.06.16 at 16:01, <boris.ostrovsky@oracle.com> wrote:
> On 06/06/2016 04:41 AM, Jan Beulich wrote:
>>
>>> - on the DomU - when I run that "test console" tool, the "lspci -xxx"
>>> output is different from before/after
>>> not much, though, only one register(?)
>>>
>>> diff lspci.before-testconsole lspci.after-testconsole
>>> 2c2
>>> < 00: cf 15 00 00 02 01 00 02 00 00 00 ff 00 40 00 00
>>> ---
>>>> 00: cf 15 00 00 02 01 00 02 00 00 00 ff 00 00 00 00
>> Okay, that's the Latency Timer field, and I think just like BARs we
>> may need to permit restoring this field. However, please note that
>> the reset being done behind the back of pciback really is the
>> problem here: pciback assumes (for a reason, as you can certainly
>> understand) that it has full control over config space of a passed
>> through device.
>>
>> And actually the latency timer would, as a side effect of enabling
>> bus mastering on the device (via the pci_set_master() call from
>> command_write()) set the Latency Timer field properly, just that
>> again pciback (and the rest of Dom0's PCI subsystem) thinks that
>> bus mastering is already enabled on the device. So perhaps in
>> permissive mode we should simply allow the latency timer field to
>> be written, just like we allow writing various of the Command
>> Register bits in that mode. Maintainers, what do you think?
>
> Don't we currently allow it to be written unconditionally? It is no
> different from accessing Cache Line Size.
No, we don't. And no, someone must have thought differently.
This is what we have right now:
{
/* Any side effects of letting driver domain control cache line? */
.offset = PCI_CACHE_LINE_SIZE,
.size = 1,
.u.b.read = xen_pcibk_read_config_byte,
.u.b.write = xen_pcibk_write_config_byte,
},
{
.offset = PCI_LATENCY_TIMER,
.size = 1,
.u.b.read = xen_pcibk_read_config_byte,
},
I.e. whoever wrote this originally thought that writes to Latency
Timer should be suppressed altogether, and there may be reasons
to also suppress writes to Cache Line Size.
> Or are you suggesting to make it stricter?
Well, following the comment I indeed thought about making the
Cache Line Size one more strict. Latency Timer can't possibly be
made more strict (unless we also wanted to disallow reads). And
it is, iirc, irrelevant for PCIe anyway, so allowing writes there
might be an option, but as they ought to have no effect, it would
seem kind of pointless.
In general I think we should be rather conservative with allowing
writes to any register.
Jan
>> If we decide to go that route, I would then wonder whether
>> Cache Line Size being unconditionally writable right now would
>> also better be restricted to permissive mode.
>
>
> -boris
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Issues with PCI-Passtrough (VT-d) in HVM with Xen 4.6
2016-06-06 14:21 ` Jan Beulich
@ 2016-06-06 14:45 ` Boris Ostrovsky
0 siblings, 0 replies; 22+ messages in thread
From: Boris Ostrovsky @ 2016-06-06 14:45 UTC (permalink / raw)
To: Jan Beulich
Cc: Juergen Gross, juwalter, xen-devel, David Vrabel, Sylwester Sosnowski
On 06/06/2016 10:21 AM, Jan Beulich wrote:
>>>> On 06.06.16 at 16:01, <boris.ostrovsky@oracle.com> wrote:
>> On 06/06/2016 04:41 AM, Jan Beulich wrote:
>>>> - on the DomU - when I run that "test console" tool, the "lspci -xxx"
>>>> output is different from before/after
>>>> not much, though, only one register(?)
>>>>
>>>> diff lspci.before-testconsole lspci.after-testconsole
>>>> 2c2
>>>> < 00: cf 15 00 00 02 01 00 02 00 00 00 ff 00 40 00 00
>>>> ---
>>>>> 00: cf 15 00 00 02 01 00 02 00 00 00 ff 00 00 00 00
>>> Okay, that's the Latency Timer field, and I think just like BARs we
>>> may need to permit restoring this field. However, please note that
>>> the reset being done behind the back of pciback really is the
>>> problem here: pciback assumes (for a reason, as you can certainly
>>> understand) that it has full control over config space of a passed
>>> through device.
>>>
>>> And actually the latency timer would, as a side effect of enabling
>>> bus mastering on the device (via the pci_set_master() call from
>>> command_write()) set the Latency Timer field properly, just that
>>> again pciback (and the rest of Dom0's PCI subsystem) thinks that
>>> bus mastering is already enabled on the device. So perhaps in
>>> permissive mode we should simply allow the latency timer field to
>>> be written, just like we allow writing various of the Command
>>> Register bits in that mode. Maintainers, what do you think?
>> Don't we currently allow it to be written unconditionally? It is no
>> different from accessing Cache Line Size.
> No, we don't. And no, someone must have thought differently.
> This is what we have right now:
>
> {
> /* Any side effects of letting driver domain control cache line? */
> .offset = PCI_CACHE_LINE_SIZE,
> .size = 1,
> .u.b.read = xen_pcibk_read_config_byte,
> .u.b.write = xen_pcibk_write_config_byte,
> },
> {
> .offset = PCI_LATENCY_TIMER,
> .size = 1,
> .u.b.read = xen_pcibk_read_config_byte,
> },
>
> I.e. whoever wrote this originally thought that writes to Latency
> Timer should be suppressed altogether, and there may be reasons
> to also suppress writes to Cache Line Size.
>
>> Or are you suggesting to make it stricter?
> Well, following the comment I indeed thought about making the
> Cache Line Size one more strict. Latency Timer can't possibly be
> made more strict (unless we also wanted to disallow reads).
Oh, yes, I completely mis-read the chunk that you quoted above. Didn't
notice that timer doesn't have a write op.
> And
> it is, iirc, irrelevant for PCIe anyway, so allowing writes there
> might be an option, but as they ought to have no effect, it would
> seem kind of pointless.
>
> In general I think we should be rather conservative with allowing
> writes to any register.
Yes.
-boris
>
> Jan
>
>>> If we decide to go that route, I would then wonder whether
>>> Cache Line Size being unconditionally writable right now would
>>> also better be restricted to permissive mode.
>>
>> -boris
>
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Issues with PCI-Passtrough (VT-d) in HVM with Xen 4.6
2016-06-18 3:24 ` Andrey Grodzovsky
@ 2016-06-20 6:57 ` Jan Beulich
0 siblings, 0 replies; 22+ messages in thread
From: Jan Beulich @ 2016-06-20 6:57 UTC (permalink / raw)
To: Andrey Grodzovsky; +Cc: jw, xen-devel
>>> On 18.06.16 at 05:24, <andrey2805@gmail.com> wrote:
> On Wed, Jun 15, 2016 at 9:14 AM, Jan Beulich <JBeulich@suse.com> wrote:
>> >>> On 15.06.16 at 12:45, <andrey2805@gmail.com> wrote:
>> > In reply to -
>> > http://lists.xen.org/archives/html/xen-devel/2016-06/msg00622.html
>> >
>> > HI, I am working with Jurgen on the issue, as per Jan's request I tried
>> to
>> > write explicitly only latency timer to be written -
>> > bool force_write = false;
>> > if ((dev_data->permissive || xen_pcibk_permissive) &&
>> > offset == PCI_CACHE_LINE_SIZE && size == 4)
>> > force_write = true;
>> > ...
>> >
>> > if ((force_write || !handled) && !err) {...}
>> >
>> > But then it exposed another issue, the command register field seems not
>> to
>> > be restored also
>> > because I think the bits which are to be restored are not
>> > in PCI_COMMAND_GUEST mask.
>>
>> Please be more precise: Which bits in particular are not getting
>> set back to the needed values (I would guess the memory
>> and/or I/O decode ones, which we specifically must not allow
>> the guest to control, but I'd like to be certain)?
>
> Indeed, 0x103 which would be MEMORY,IO and SERR.
>
>> If my guess is correct, then I think rather than adding some
>> hackish workaround to pciback you'd better see whether there's
>> a way to cause a pci_disable_device() through your driver before
>> the reset, and a pci_enable_device() after. Or actually, I think
>> you can do this via plain config space writes from your driver: Try
>> writing the Command Register with the two decode bits clear
>> right before restoring the intended value (see the logic close to
>> the top of command_write()).
>>
>> So i tried the first option, as you suggested by writing
> to /sys/bus/pci/devices/0000\:00\:00.0/enable
> '0' == disable before reset and '1' == enable after from the test app, but
> no writes get through to pic-back on DOM0 , only the reads.
> But this basically still tries to write the MEMORY and IO bits to command
> register down the call stack in pci_enable_resources
> <http://lxr.free-electrons.com/ident?i=pci_enable_resources>
> which would again be blocked in xen-pciback xen_pcibk_config_write or am I
> missing something ?
I don't think it would get entirely blocked: What you really need to
look at (and perhaps add some logging to) is command_write()'s logic
involving pci_is_enabled() and is_enable_cmd(). (And I'm not, btw,
convinced that you writing the enable sysfs node you can achieve
the desired effects. As said - I think you should do plain config space
writes.)
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Issues with PCI-Passtrough (VT-d) in HVM with Xen 4.6
2016-06-15 13:14 ` Jan Beulich
@ 2016-06-18 3:24 ` Andrey Grodzovsky
2016-06-20 6:57 ` Jan Beulich
0 siblings, 1 reply; 22+ messages in thread
From: Andrey Grodzovsky @ 2016-06-18 3:24 UTC (permalink / raw)
To: Jan Beulich; +Cc: Jürgen Walter • Quattru, xen-devel
[-- Attachment #1.1: Type: text/plain, Size: 2133 bytes --]
On Wed, Jun 15, 2016 at 9:14 AM, Jan Beulich <JBeulich@suse.com> wrote:
> >>> On 15.06.16 at 12:45, <andrey2805@gmail.com> wrote:
> > In reply to -
> > http://lists.xen.org/archives/html/xen-devel/2016-06/msg00622.html
> >
> > HI, I am working with Jurgen on the issue, as per Jan's request I tried
> to
> > write explicitly only latency timer to be written -
> > bool force_write = false;
> > if ((dev_data->permissive || xen_pcibk_permissive) &&
> > offset == PCI_CACHE_LINE_SIZE && size == 4)
> > force_write = true;
> > ...
> >
> > if ((force_write || !handled) && !err) {...}
> >
> > But then it exposed another issue, the command register field seems not
> to
> > be restored also
> > because I think the bits which are to be restored are not
> > in PCI_COMMAND_GUEST mask.
>
> Please be more precise: Which bits in particular are not getting
> set back to the needed values (I would guess the memory
> and/or I/O decode ones, which we specifically must not allow
> the guest to control, but I'd like to be certain)?
>
Indeed, 0x103 which would be MEMORY,IO and SERR.
>
> If my guess is correct, then I think rather than adding some
> hackish workaround to pciback you'd better see whether there's
> a way to cause a pci_disable_device() through your driver before
> the reset, and a pci_enable_device() after. Or actually, I think
> you can do this via plain config space writes from your driver: Try
> writing the Command Register with the two decode bits clear
> right before restoring the intended value (see the logic close to
> the top of command_write()).
>
> Jan
>
> So i tried the first option, as you suggested by writing
to /sys/bus/pci/devices/0000\:00\:00.0/enable
'0' == disable before reset and '1' == enable after from the test app, but
no writes get through to pic-back on DOM0 , only the reads.
But this basically still tries to write the MEMORY and IO bits to command
register down the call stack in pci_enable_resources
<http://lxr.free-electrons.com/ident?i=pci_enable_resources>
which would again be blocked in xen-pciback xen_pcibk_config_write or am I
missing something ?
[-- Attachment #1.2: Type: text/html, Size: 3464 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Issues with PCI-Passtrough (VT-d) in HVM with Xen 4.6
2016-06-15 10:45 Andrey Grodzovsky
@ 2016-06-15 13:14 ` Jan Beulich
2016-06-18 3:24 ` Andrey Grodzovsky
0 siblings, 1 reply; 22+ messages in thread
From: Jan Beulich @ 2016-06-15 13:14 UTC (permalink / raw)
To: Andrey Grodzovsky; +Cc: jw, xen-devel
>>> On 15.06.16 at 12:45, <andrey2805@gmail.com> wrote:
> In reply to -
> http://lists.xen.org/archives/html/xen-devel/2016-06/msg00622.html
>
> HI, I am working with Jurgen on the issue, as per Jan's request I tried to
> write explicitly only latency timer to be written -
> bool force_write = false;
> if ((dev_data->permissive || xen_pcibk_permissive) &&
> offset == PCI_CACHE_LINE_SIZE && size == 4)
> force_write = true;
> ...
>
> if ((force_write || !handled) && !err) {...}
>
> But then it exposed another issue, the command register field seems not to
> be restored also
> because I think the bits which are to be restored are not
> in PCI_COMMAND_GUEST mask.
Please be more precise: Which bits in particular are not getting
set back to the needed values (I would guess the memory
and/or I/O decode ones, which we specifically must not allow
the guest to control, but I'd like to be certain)?
If my guess is correct, then I think rather than adding some
hackish workaround to pciback you'd better see whether there's
a way to cause a pci_disable_device() through your driver before
the reset, and a pci_enable_device() after. Or actually, I think
you can do this via plain config space writes from your driver: Try
writing the Command Register with the two decode bits clear
right before restoring the intended value (see the logic close to
the top of command_write()).
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Issues with PCI-Passtrough (VT-d) in HVM with Xen 4.6
@ 2016-06-15 10:45 Andrey Grodzovsky
2016-06-15 13:14 ` Jan Beulich
0 siblings, 1 reply; 22+ messages in thread
From: Andrey Grodzovsky @ 2016-06-15 10:45 UTC (permalink / raw)
To: xen-devel; +Cc: Jürgen Walter • Quattru
[-- Attachment #1.1: Type: text/plain, Size: 1321 bytes --]
In reply to -
http://lists.xen.org/archives/html/xen-devel/2016-06/msg00622.html
HI, I am working with Jurgen on the issue, as per Jan's request I tried to
write explicitly only latency timer to be written -
bool force_write = false;
if ((dev_data->permissive || xen_pcibk_permissive) &&
offset == PCI_CACHE_LINE_SIZE && size == 4)
force_write = true;
...
if ((force_write || !handled) && !err) {...}
But then it exposed another issue, the command register field seems not to
be restored also
because I think the bits which are to be restored are not
in PCI_COMMAND_GUEST mask.
The only hack which always works for me is force write even if the
confpsace fields filter doens't allow it -
+ int remainder = size;
+
if (unlikely(verbose_request))
printk(KERN_DEBUG
DRV_NAME ": %s: write request %d bytes at 0x%x =
%x\n",
@@ -252,10 +254,11 @@ int xen_pcibk_config_write(struct pci_dev *dev, int
offset, int size, u32 value)
* special helpers to work correctly.
*/
handled = 1;
+ remainder -= field->size
}
}
- if (!handled && !err) {
+ if ((remainder || !handled) && !err) {
Thanks.
Andrey Grodzovsky.
[-- Attachment #1.2: Type: text/html, Size: 2029 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Issues with PCI-Passtrough (VT-d) in HVM with Xen 4.6
@ 2016-06-02 19:49 Sylwester Sosnowski
0 siblings, 0 replies; 22+ messages in thread
From: Sylwester Sosnowski @ 2016-06-02 19:49 UTC (permalink / raw)
To: xen-devel; +Cc: Jeremy Fitzhardinge
Issues with PCI-Passtrough (VT-d) in HVM with Xen 4.6
=====================================================
Background
----------
I am trying to passthrough an Industrial Ethernet Interface
(Hilscher GmbH CIFX 50E-DP(M/S)) on a HVM DomU running the
Xen 4.6 Hypervisor. The card is being pass-trough to the HVM
using the PCI permissive mode and VT-d is active on this platform.
The Linux driver of the card (available only with NDA) at first
sight seems to work properly (e.g. no system stability problems,
no call traces in dmesg).
Hilscher provides the libcifx, which is an user-level library
for accessing the card. libcfix uses the generic UIO interface
and some card-specific interfaces to communicate.
Issue
-----
Whenever libcifx tries perform a reset sequence initializing
the card peripherals, we get an empty / invalid result.
I digged deeper into this by adding some debug hooks into the
xen-pciback kernel module.
The card performs writes to offset 0x10 and 0x30 at card
initialization (mostly writing dwords and words).
To verify if the writes were performed successfully, I read
back the values after writing and can see that the read data
differs from the written one.
Temporary Fix
-------------
After checking the source[1] of the PCI configuration space
handling in xen-pciback, I found out that changing line 258
to read
if (handled && !err) {
instead of:
if (!handled && !err) {
solves the issue and I can successfully write to the interface.
I am unsure why this works and if it's the right way to do it
or possibly a Xen bug, so I would like to ask for feedback
for this.
Please let me know whenever I can supply additional logfiles or
info. I'd be happy if we can resolve whenever this is a hardware-
specific, driver-specific or Xen issue.
Thank you,
Sylwester
PS: I tried to CC Ryan Wilson and Konrad Rzeszutek Wilk, but the
E-Mail seems to be no longer active. This is where the changes
originated.
Refereces
---------
1.
https://github.com/google/kasan/blob/master/drivers/xen/xen-pciback/conf_space.c
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
end of thread, other threads:[~2016-06-20 6:57 UTC | newest]
Thread overview: 22+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-06-02 19:59 Issues with PCI-Passtrough (VT-d) in HVM with Xen 4.6 Sylwester Sosnowski
2016-06-02 20:06 ` Konrad Rzeszutek Wilk
2016-06-02 20:11 ` Sylwester Sosnowski
2016-06-03 7:24 ` Sylwester Sosnowski
2016-06-03 11:52 ` Jan Beulich
2016-06-03 12:02 ` Jürgen Walter
2016-06-03 13:26 ` Jan Beulich
2016-06-03 14:20 ` Jan Beulich
2016-06-04 14:36 ` Jürgen Walter
2016-06-06 7:59 ` Jan Beulich
2016-06-04 15:15 ` Jürgen Walter
2016-06-06 8:41 ` Jan Beulich
2016-06-06 9:09 ` Jürgen Walter
2016-06-06 9:43 ` Jan Beulich
2016-06-06 14:01 ` Boris Ostrovsky
2016-06-06 14:21 ` Jan Beulich
2016-06-06 14:45 ` Boris Ostrovsky
-- strict thread matches above, loose matches on Subject: below --
2016-06-15 10:45 Andrey Grodzovsky
2016-06-15 13:14 ` Jan Beulich
2016-06-18 3:24 ` Andrey Grodzovsky
2016-06-20 6:57 ` Jan Beulich
2016-06-02 19:49 Sylwester Sosnowski
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).