From: "Roger Pau Monné" <roger.pau@citrix.com>
To: Jan Beulich <jbeulich@suse.com>
Cc: Huang Rui <ray.huang@amd.com>,
"Deucher, Alexander" <Alexander.Deucher@amd.com>,
"Koenig, Christian" <Christian.Koenig@amd.com>,
"Hildebrand, Stewart" <Stewart.Hildebrand@amd.com>,
Xenia Ragiadakou <burzalodowa@gmail.com>,
"Huang, Honglei1" <Honglei1.Huang@amd.com>,
"Zhang, Julia" <Julia.Zhang@amd.com>,
"Chen, Jiqian" <Jiqian.Chen@amd.com>,
Stefano Stabellini <sstabellini@kernel.org>,
Anthony PERARD <anthony.perard@citrix.com>,
"xen-devel@lists.xenproject.org" <xen-devel@lists.xenproject.org>
Subject: Re: [RFC XEN PATCH 2/6] vpci: accept BAR writes if dom0 is PVH
Date: Thu, 23 Mar 2023 11:43:53 +0100 [thread overview]
Message-ID: <ZBwtaceTNvCYksmR@Air-de-Roger> (raw)
In-Reply-To: <85aa668d-9614-a80d-8f44-174ecbdcf1f7@suse.com>
On Wed, Mar 22, 2023 at 01:48:30PM +0100, Jan Beulich wrote:
> On 22.03.2023 13:33, Huang Rui wrote:
> > I traced that while we do pci-assignable-add, we will follow below trace to
> > bind the passthrough device.
> >
> > pciassignable_add()->libxl_device_pci_assignable_add()->libxl__device_pci_assignable_add()->pciback_dev_assign()
> >
> > Then kernel xen-pciback driver want to add virtual configuration spaces. In
> > this phase, the bar_write() in xen hypervisor will be called. I still need
> > a bit more time to figure the exact reason. May I know where the
> > xen-pciback driver would trigger a hvm_io_intercept to xen hypervisor?
>
> Any config space access would. And I might guess ...
>
> > [ 309.719049] xen_pciback: wants to seize 0000:03:00.0
> > [ 462.911251] pciback 0000:03:00.0: xen_pciback: probing...
> > [ 462.911256] pciback 0000:03:00.0: xen_pciback: seizing device
> > [ 462.911257] pciback 0000:03:00.0: xen_pciback: pcistub_device_alloc
> > [ 462.911261] pciback 0000:03:00.0: xen_pciback: initializing...
> > [ 462.911263] pciback 0000:03:00.0: xen_pciback: initializing config
> > [ 462.911265] pciback 0000:03:00.0: xen-pciback: initializing virtual configuration space
> > [ 462.911268] pciback 0000:03:00.0: xen-pciback: added config field at offset 0x00
> > [ 462.911271] pciback 0000:03:00.0: xen-pciback: added config field at offset 0x02
> > [ 462.911284] pciback 0000:03:00.0: xen-pciback: added config field at offset 0x04
> > [ 462.911286] pciback 0000:03:00.0: xen-pciback: added config field at offset 0x3c
> > [ 462.911289] pciback 0000:03:00.0: xen-pciback: added config field at offset 0x3d
> > [ 462.911291] pciback 0000:03:00.0: xen-pciback: added config field at offset 0x0c
> > [ 462.911294] pciback 0000:03:00.0: xen-pciback: added config field at offset 0x0d
> > [ 462.911296] pciback 0000:03:00.0: xen-pciback: added config field at offset 0x0f
> > [ 462.911301] pciback 0000:03:00.0: xen-pciback: added config field at offset 0x10
> > [ 462.911306] pciback 0000:03:00.0: xen-pciback: added config field at offset 0x14
> > [ 462.911309] pciback 0000:03:00.0: xen-pciback: added config field at offset 0x18
> > [ 462.911313] pciback 0000:03:00.0: xen-pciback: added config field at offset 0x1c
> > [ 462.911317] pciback 0000:03:00.0: xen-pciback: added config field at offset 0x20
> > [ 462.911321] pciback 0000:03:00.0: xen-pciback: added config field at offset 0x24
> > [ 462.911325] pciback 0000:03:00.0: xen-pciback: added config field at offset 0x30
> > [ 462.911358] pciback 0000:03:00.0: Found capability 0x1 at 0x50
> > [ 462.911361] pciback 0000:03:00.0: xen-pciback: added config field at offset 0x50
> > [ 462.911363] pciback 0000:03:00.0: xen-pciback: added config field at offset 0x52
> > [ 462.911368] pciback 0000:03:00.0: xen-pciback: added config field at offset 0x54
> > [ 462.911371] pciback 0000:03:00.0: xen-pciback: added config field at offset 0x56
> > [ 462.911373] pciback 0000:03:00.0: xen-pciback: added config field at offset 0x57
> > [ 462.911386] pciback 0000:03:00.0: Found capability 0x5 at 0xa0
> > [ 462.911388] pciback 0000:03:00.0: xen-pciback: added config field at offset 0xa0
> > [ 462.911391] pciback 0000:03:00.0: xen-pciback: added config field at offset 0xa2
> > [ 462.911405] pciback 0000:03:00.0: xen_pciback: enabling device
> > [ 462.911412] pciback 0000:03:00.0: enabling device (0006 -> 0007)
> > [ 462.911658] Already setup the GSI :28
> > [ 462.911668] Already map the GSI :28 and IRQ: 115
> > [ 462.911684] pciback 0000:03:00.0: xen_pciback: save state of device
> > [ 462.912154] pciback 0000:03:00.0: xen_pciback: resetting (FLR, D3, etc) the device
> > [ 463.954998] pciback 0000:03:00.0: xen_pciback: reset device
>
> ... it is actually the reset here, saving and then restoring config space.
> If e.g. that restore was done "blindly" (i.e. simply writing fields low to
> high), then memory decode would be re-enabled before the BARs are written.
The problem is also that we don't tell vPCI that the device has been
reset, so the current cached state in pdev->vpci is all out of date
with the real device state.
I didn't hit this on my test because the device I was using had no
reset support.
I don't think it's feasible for Xen to detect all the possible reset
methods dom0 might use, as some of those are device specific for
example.
We would have to introduce a new hypercall that clears all vPCI device
state, PHYSDEVOP_pci_device_reset for example. This will involve
adding proper cleanup functions, as the current code in
vpci_remove_device() only deals with allocated memory (because so far
devices where not deassigned) but we now also need to make sure
MSI(-X) interrupts are torn down and freed, and will also require
removing any mappings of BARs into the dom0 physmap.
Thanks, Roger.
next prev parent reply other threads:[~2023-03-23 10:44 UTC|newest]
Thread overview: 75+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-03-12 7:54 [RFC XEN PATCH 0/6] Introduce VirtIO GPU and Passthrough GPU support on Xen PVH dom0 Huang Rui
2023-03-12 7:54 ` [RFC XEN PATCH 1/6] x86/pvh: report ACPI VFCT table to dom0 if present Huang Rui
2023-03-13 11:55 ` Andrew Cooper
2023-03-13 12:21 ` Roger Pau Monné
2023-03-13 12:27 ` Andrew Cooper
2023-03-21 6:26 ` Huang Rui
2023-03-12 7:54 ` [RFC XEN PATCH 2/6] vpci: accept BAR writes if dom0 is PVH Huang Rui
2023-03-13 7:23 ` Christian König
2023-03-13 7:26 ` Christian König
2023-03-13 8:46 ` Jan Beulich
2023-03-13 8:55 ` Huang Rui
2023-03-14 23:42 ` Stefano Stabellini
2023-03-14 16:02 ` Jan Beulich
2023-03-21 9:36 ` Huang Rui
2023-03-21 9:41 ` Jan Beulich
2023-03-21 10:14 ` Huang Rui
2023-03-21 10:20 ` Jan Beulich
2023-03-21 11:49 ` Huang Rui
2023-03-21 12:20 ` Roger Pau Monné
2023-03-21 12:25 ` Jan Beulich
2023-03-21 12:59 ` Huang Rui
2023-03-21 12:27 ` Jan Beulich
2023-03-21 13:03 ` Huang Rui
2023-03-22 7:28 ` Huang Rui
2023-03-22 7:45 ` Jan Beulich
2023-03-22 9:34 ` Roger Pau Monné
2023-03-22 12:33 ` Huang Rui
2023-03-22 12:48 ` Jan Beulich
2023-03-23 10:26 ` Huang Rui
2023-03-23 14:16 ` Jan Beulich
2023-03-23 10:43 ` Roger Pau Monné [this message]
2023-03-23 13:34 ` Huang Rui
2023-03-23 16:23 ` Roger Pau Monné
2023-03-24 4:37 ` Huang Rui
2023-03-12 7:54 ` [RFC XEN PATCH 3/6] x86/pvh: shouldn't check pirq flag when map pirq in PVH Huang Rui
2023-03-14 16:27 ` Jan Beulich
2023-03-15 15:57 ` Roger Pau Monné
2023-03-16 0:22 ` Stefano Stabellini
2023-03-21 10:09 ` Huang Rui
2023-03-21 10:17 ` Jan Beulich
2023-03-12 7:54 ` [RFC XEN PATCH 4/6] x86/pvh: PVH dom0 also need PHYSDEVOP_setup_gsi call Huang Rui
2023-03-14 16:30 ` Jan Beulich
2023-03-15 17:01 ` Andrew Cooper
2023-03-16 0:26 ` Stefano Stabellini
2023-03-16 0:39 ` Stefano Stabellini
2023-03-16 8:51 ` Jan Beulich
2023-03-16 9:18 ` Roger Pau Monné
2023-03-16 7:05 ` Jan Beulich
2023-03-21 12:42 ` Huang Rui
2023-03-21 12:22 ` Huang Rui
2023-03-12 7:54 ` [RFC XEN PATCH 5/6] tools/libs/call: add linux os call to get gsi from irq Huang Rui
2023-03-14 16:36 ` Jan Beulich
2023-03-12 7:54 ` [RFC XEN PATCH 6/6] tools/libs/light: pci: translate irq to gsi Huang Rui
2023-03-14 16:39 ` Jan Beulich
2023-03-15 16:35 ` Roger Pau Monné
2023-03-16 0:44 ` Stefano Stabellini
2023-03-16 8:54 ` Roger Pau Monné
2023-03-16 8:55 ` Jan Beulich
2023-03-16 9:27 ` Roger Pau Monné
2023-03-16 9:42 ` Jan Beulich
2023-03-16 23:19 ` Stefano Stabellini
2023-03-17 8:39 ` Jan Beulich
2023-03-17 9:51 ` Roger Pau Monné
2023-03-17 18:15 ` Stefano Stabellini
2023-03-17 19:48 ` Roger Pau Monné
2023-03-17 20:55 ` Stefano Stabellini
2023-03-20 15:16 ` Roger Pau Monné
2023-03-20 15:29 ` Jan Beulich
2023-03-20 16:50 ` Roger Pau Monné
2023-07-31 16:40 ` Chen, Jiqian
2023-08-23 8:57 ` Roger Pau Monné
2023-08-31 8:56 ` Chen, Jiqian
2023-03-13 7:24 ` [RFC XEN PATCH 0/6] Introduce VirtIO GPU and Passthrough GPU support on Xen PVH dom0 Christian König
2023-03-21 10:26 ` Huang Rui
2023-03-20 16:22 ` Huang Rui
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=ZBwtaceTNvCYksmR@Air-de-Roger \
--to=roger.pau@citrix.com \
--cc=Alexander.Deucher@amd.com \
--cc=Christian.Koenig@amd.com \
--cc=Honglei1.Huang@amd.com \
--cc=Jiqian.Chen@amd.com \
--cc=Julia.Zhang@amd.com \
--cc=Stewart.Hildebrand@amd.com \
--cc=anthony.perard@citrix.com \
--cc=burzalodowa@gmail.com \
--cc=jbeulich@suse.com \
--cc=ray.huang@amd.com \
--cc=sstabellini@kernel.org \
--cc=xen-devel@lists.xenproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).