From: "Jürgen Walter" <juwalter@gmail.com>
To: Jan Beulich <JBeulich@suse.com>
Cc: Juergen Gross <JGross@suse.com>,
xen-devel@lists.xen.org, David Vrabel <david.vrabel@citrix.com>,
Boris Ostrovsky <boris.ostrovsky@oracle.com>,
Sylwester Sosnowski <syso@datanoise.net>
Subject: Re: Issues with PCI-Passtrough (VT-d) in HVM with Xen 4.6
Date: Mon, 06 Jun 2016 11:09:55 +0200 [thread overview]
Message-ID: <D2095C89-D1EC-4739-BF68-418825C0BADE@gmail.com> (raw)
In-Reply-To: <5755536202000078000F1E23@prv-mh.provo.novell.com>
Hi Jan,
On 6 Jun 2016, at 10:41, Jan Beulich wrote:
>>>> On 04.06.16 at 17:15, <juwalter@gmail.com> wrote:
>> On 3 Jun 2016, at 15:26, Jan Beulich wrote:
>>> I'll see to put together a patch for the recognizable problem,
>>> but I'm still unclear about your actual one.
>>
>> regarding the actual one: while it is still not working, I managed to
>> dig deeper and found a few observations
>>
>> background: there is a driver/kernel module, and a user space
>> library,
>> and a "test console"
>> the "test console" uses the library, and invokes "cifXInit"
>> (initialise
>> the PCI card)
>>
>> during init, it does
>> a) read pci config (space)
>> b) send reset sequence to the PCI device
>> c) writes back the pci config saved from step "a)" (restore)
>
> I'm sorry to mention that, but so far you still didn't provide a
> matching pciback log for that portion of the operation, at least
> afaict.
sorry about that- we managed to pull this together. note: we changed the
user space library to perform another read after the write/restore
we did on Dom0 and DomU, "inside permissive mode" printk statement we
added if it goes into "if (!handled ...)"
DOM0
# read PCI config and save for later restore
(gdb) x/64wx 0x622190
0x622190: 0x000015cf 0x02000107 0xff000000 0x00004000
0x6221a0: 0xf7a00000 0x00000000 0x00000000 0x00000000
0x6221b0: 0x00000000 0x00000000 0x00000001 0x00000000
0x6221c0: 0x00000000 0x00000000 0x00000000 0x0000010b
0x6221d0: 0x00000000 0x00000000 0x00000000 0x00000000
0x6221e0: 0x00000000 0x00000000 0x00000000 0x00000000
0x6221f0: 0x00000000 0x00000000 0x00000000 0x00000000
0x622200: 0x00000000 0x00000000 0x00000000 0x00000000
0x622210: 0x00000000 0x00000000 0x00000000 0x00000000
0x622220: 0x00000000 0x00000000 0x00000000 0x00000000
0x622230: 0x00000000 0x00000000 0x00000000 0x00000000
0x622240: 0x00000000 0x00000000 0x00000000 0x00000000
0x622250: 0x00000000 0x00000000 0x00000000 0x00000000
0x622260: 0x00000000 0x00000000 0x00000000 0x00000000
0x622270: 0x00000000 0x00000000 0x00000000 0x00000000
0x622280: 0x00000000 0x00000000 0x00000000 0x00000000
# write/restore PCI config
# ...
# read PCI config again and see what is there now
(gdb) x/64w 0x622190
0x622190: 0x000015cf 0x02000107 0xff000000 0x00004000 <--- worked
0x6221a0: 0xf7a00000 0x00000000 0x00000000 0x00000000
0x6221b0: 0x00000000 0x00000000 0x00000001 0x00000000
0x6221c0: 0x00000000 0x00000000 0x00000000 0x0000010b
0x6221d0: 0x00000000 0x00000000 0x00000000 0x00000000
0x6221e0: 0x00000000 0x00000000 0x00000000 0x00000000
0x6221f0: 0x00000000 0x00000000 0x00000000 0x00000000
0x622200: 0x00000000 0x00000000 0x00000000 0x00000000
0x622210: 0x00000000 0x00000000 0x00000000 0x00000000
0x622220: 0x00000000 0x00000000 0x00000000 0x00000000
0x622230: 0x00000000 0x00000000 0x00000000 0x00000000
0x622240: 0x00000000 0x00000000 0x00000000 0x00000000
0x622250: 0x00000000 0x00000000 0x00000000 0x00000000
0x622260: 0x00000000 0x00000000 0x00000000 0x00000000
0x622270: 0x00000000 0x00000000 0x00000000 0x00000000
0x622280: 0x00000000 0x00000000 0x00000000 0x00000000
(gdb)
DOMU
# read PCI config and save for later restore
(gdb) p pvPCIConfig
$2 = (void *) 0x61ef90
(gdb) x/64wx 0x61ef90
0x61ef90: 0x000015cf 0x02000102 0xff000000 0x00004000
0x61efa0: 0xf7a00000 0x00000000 0x00000000 0x00000000
0x61efb0: 0x00000000 0x00000000 0x00000001 0x00000000
0x61efc0: 0x00000000 0x00000000 0x00000000 0x00000110
0x61efd0: 0x00000000 0x00000000 0x00000000 0x00000000
0x61efe0: 0x00000000 0x00000000 0x00000000 0x00000000
0x61eff0: 0x00000000 0x00000000 0x00000000 0x00000000
0x61f000: 0x00000000 0x00000000 0x00000000 0x00000000
0x61f010: 0x00000000 0x00000000 0x00000000 0x00000000
0x61f020: 0x00000000 0x00000000 0x00000000 0x00000000
0x61f030: 0x00000000 0x00000000 0x00000000 0x00000000
0x61f040: 0x00000000 0x00000000 0x00000000 0x00000000
0x61f050: 0x00000000 0x00000000 0x00000000 0x00000000
0x61f060: 0x00000000 0x00000000 0x00000000 0x00000000
0x61f070: 0x00000000 0x00000000 0x00000000 0x00000000
0x61f080: 0x00000000 0x00000000 0x00000000 0x00000000
# write/restore PCI config
# ...
# read PCI config again and see what is there now
(gdb) p OS_ReadPCIConfig(ptDevInstance->pvOSDependent)
$3 = (void *) 0x61ef90
(gdb) x/64wx 0x61ef90
0x61ef90: 0x000015cf 0x02000102 0xff000000 0x00000000 <--- failed
0x61efa0: 0xf7a00000 0x00000000 0x00000000 0x00000000
0x61efb0: 0x00000000 0x00000000 0x00000001 0x00000000
0x61efc0: 0x00000000 0x00000000 0x00000000 0x00000110
0x61efd0: 0x00000000 0x00000000 0x00000000 0x00000000
0x61efe0: 0x00000000 0x00000000 0x00000000 0x00000000
0x61eff0: 0x00000000 0x00000000 0x00000000 0x00000000
0x61f000: 0x00000000 0x00000000 0x00000000 0x00000000
0x61f010: 0x00000000 0x00000000 0x00000000 0x00000000
0x61f020: 0x00000000 0x00000000 0x00000000 0x00000000
0x61f030: 0x00000000 0x00000000 0x00000000 0x00000000
0x61f040: 0x00000000 0x00000000 0x00000000 0x00000000
0x61f050: 0x00000000 0x00000000 0x00000000 0x00000000
0x61f060: 0x00000000 0x00000000 0x00000000 0x00000000
0x61f070: 0x00000000 0x00000000 0x00000000 0x00000000
0x61f080: 0x00000000 0x00000000 0x00000000 0x00000000
[ 1466.964782 < 139.522689>] xen-pciback: 0000:06:00.0: write request 4
bytes at 0x0 = 15cf
[ 1466.964786 < 0.000004>] xen-pciback: 0000:06:00.0: read 2 bytes at
0x0
[ 1466.964799 < 0.000013>] xen-pciback: 0000:06:00.0: read 2 bytes at
0x0 = 15cf
[ 1466.964801 < 0.000002>] xen-pciback: 0000:06:00.0: read 2 bytes at
0x2
[ 1466.964807 < 0.000006>] xen-pciback: 0000:06:00.0: read 2 bytes at
0x2 = 0
[ 1466.964828 < 0.000021>] xen-pciback: 0000:06:00.0: write request 4
bytes at 0x4 = 2000102
[ 1466.964829 < 0.000001>] xen-pciback: 0000:06:00.0: read 2 bytes at
0x4
[ 1466.964841 < 0.000012>] xen-pciback: 0000:06:00.0: read 2 bytes at
0x4 = 102
[ 1466.964870 < 0.000029>] xen-pciback: 0000:06:00.0: write request 4
bytes at 0x8 = ff000000
[ 1466.964872 < 0.000002>] xen-pciback: 0000:06:00.0: inside
permissive mode - write request 4 bytes at 0x8 = ff000000
[ 1466.964895 < 0.000023>] xen-pciback: 0000:06:00.0: write request 4
bytes at 0xc = 4000
[ 1466.964897 < 0.000002>] xen-pciback: 0000:06:00.0: read 1 bytes at
0xc
[ 1466.964907 < 0.000010>] xen-pciback: 0000:06:00.0: read 1 bytes at
0xc = 0
[ 1466.964914 < 0.000007>] xen-pciback: 0000:06:00.0: read 1 bytes at
0xf
[ 1466.964925 < 0.000011>] xen-pciback: 0000:06:00.0: read 1 bytes at
0xf = 0
[ 1466.964947 < 0.000022>] xen-pciback: 0000:06:00.0: write request 4
bytes at 0x10 = f7a00000
[ 1466.964948 < 0.000001>] xen-pciback: 0000:06:00.0: read 4 bytes at
0x10
[ 1466.964955 < 0.000007>] xen-pciback: 0000:06:00.0: read 4 bytes at
0x10 = f7a00000
[ 1466.964972 < 0.000017>] xen-pciback: 0000:06:00.0: write request 4
bytes at 0x14 = 0
[ 1466.964973 < 0.000001>] xen-pciback: 0000:06:00.0: read 4 bytes at
0x14
[ 1466.964980 < 0.000007>] xen-pciback: 0000:06:00.0: read 4 bytes at
0x14 = 0
[ 1466.964998 < 0.000018>] xen-pciback: 0000:06:00.0: write request 4
bytes at 0x18 = 0
[ 1466.964999 < 0.000001>] xen-pciback: 0000:06:00.0: read 4 bytes at
0x18
[ 1466.965006 < 0.000007>] xen-pciback: 0000:06:00.0: read 4 bytes at
0x18 = 0
>> - on the DomU - when I run that "test console" tool, the "lspci -xxx"
>> output is different from before/after
>> not much, though, only one register(?)
>>
>> diff lspci.before-testconsole lspci.after-testconsole
>> 2c2
>> < 00: cf 15 00 00 02 01 00 02 00 00 00 ff 00 40 00 00
>> ---
>>> 00: cf 15 00 00 02 01 00 02 00 00 00 ff 00 00 00 00
>
> Okay, that's the Latency Timer field, and I think just like BARs we
> may need to permit restoring this field. However, please note that
> the reset being done behind the back of pciback really is the
> problem here: pciback assumes (for a reason, as you can certainly
> understand) that it has full control over config space of a passed
> through device.
I agree that it is very ugly to reset the device behind the back of
pciback.
However, this is how the vendor decided to do it, and have no better
idea how
to accomplish this.
While I cannot image that there is, but can you think think of
any "correct" way to reset a pci device from
a) inside a DomU
b) from the user space library?
What I cannot do: modify the firmware or the HW of the PCI device
(which would be good, because then it could simply do
all memory cleaning etc. w/o affecting its PCI config known to the host)
What I can do: rewrite all parts of the driver (uio_netx) and user space
library.
> And actually the latency timer would, as a side effect of enabling
> bus mastering on the device (via the pci_set_master() call from
> command_write()) set the Latency Timer field properly, just that
> again pciback (and the rest of Dom0's PCI subsystem) thinks that
> bus mastering is already enabled on the device. So perhaps in
> permissive mode we should simply allow the latency timer field to
> be written, just like we allow writing various of the Command
> Register bits in that mode. Maintainers, what do you think?
(is that something that /sys/bus/pci/drivers/pciback/quirks could
help with? according to docs, this is only used/in effect when
in permissive mode)
> If we decide to go that route, I would then wonder whether
> Cache Line Size being unconditionally writable right now would
> also better be restricted to permissive mode.
> In any event, Jürgen, it would be helpful if you could confirm
If I understood you correctly, I modify xen_pcibk_config_write
in "drivers/xen/xen-pciback/conf_space.c" to specifically allow only
this field (latency timer) to be written and log everything.
Will do and report back!
Thanks for your time!!!
Jürgen
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
next prev parent reply other threads:[~2016-06-06 9:09 UTC|newest]
Thread overview: 22+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-06-02 19:59 Issues with PCI-Passtrough (VT-d) in HVM with Xen 4.6 Sylwester Sosnowski
2016-06-02 20:06 ` Konrad Rzeszutek Wilk
2016-06-02 20:11 ` Sylwester Sosnowski
2016-06-03 7:24 ` Sylwester Sosnowski
2016-06-03 11:52 ` Jan Beulich
2016-06-03 12:02 ` Jürgen Walter
2016-06-03 13:26 ` Jan Beulich
2016-06-03 14:20 ` Jan Beulich
2016-06-04 14:36 ` Jürgen Walter
2016-06-06 7:59 ` Jan Beulich
2016-06-04 15:15 ` Jürgen Walter
2016-06-06 8:41 ` Jan Beulich
2016-06-06 9:09 ` Jürgen Walter [this message]
2016-06-06 9:43 ` Jan Beulich
2016-06-06 14:01 ` Boris Ostrovsky
2016-06-06 14:21 ` Jan Beulich
2016-06-06 14:45 ` Boris Ostrovsky
-- strict thread matches above, loose matches on Subject: below --
2016-06-15 10:45 Andrey Grodzovsky
2016-06-15 13:14 ` Jan Beulich
2016-06-18 3:24 ` Andrey Grodzovsky
2016-06-20 6:57 ` Jan Beulich
2016-06-02 19:49 Sylwester Sosnowski
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=D2095C89-D1EC-4739-BF68-418825C0BADE@gmail.com \
--to=juwalter@gmail.com \
--cc=JBeulich@suse.com \
--cc=JGross@suse.com \
--cc=boris.ostrovsky@oracle.com \
--cc=david.vrabel@citrix.com \
--cc=syso@datanoise.net \
--cc=xen-devel@lists.xen.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).