From: Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com>
To: "Roger Pau Monné" <roger.pau@citrix.com>
Cc: "xen-devel@lists.xenproject.org" <xen-devel@lists.xenproject.org>,
"julien@xen.org" <julien@xen.org>,
"sstabellini@kernel.org" <sstabellini@kernel.org>,
Oleksandr Tyshchenko <Oleksandr_Tyshchenko@epam.com>,
Volodymyr Babchuk <Volodymyr_Babchuk@epam.com>,
Artem Mygaiev <Artem_Mygaiev@epam.com>,
"jbeulich@suse.com" <jbeulich@suse.com>,
"andrew.cooper3@citrix.com" <andrew.cooper3@citrix.com>,
"george.dunlap@citrix.com" <george.dunlap@citrix.com>,
"paul@xen.org" <paul@xen.org>,
Bertrand Marquis <bertrand.marquis@arm.com>,
Rahul Singh <rahul.singh@arm.com>,
Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com>
Subject: Re: [PATCH v6 06/13] vpci/header: implement guest BAR register handlers
Date: Tue, 8 Feb 2022 09:31:37 +0000 [thread overview]
Message-ID: <498ca523-2642-e35f-b316-0a089f98ca91@epam.com> (raw)
In-Reply-To: <YgI3HpFffudiEmNN@Air-de-Roger>
On 08.02.22 11:25, Roger Pau Monné wrote:
> On Fri, Feb 04, 2022 at 08:34:52AM +0200, Oleksandr Andrushchenko wrote:
>> From: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>
>>
>> Add relevant vpci register handlers when assigning PCI device to a domain
>> and remove those when de-assigning. This allows having different
>> handlers for different domains, e.g. hwdom and other guests.
>>
>> Emulate guest BAR register values: this allows creating a guest view
>> of the registers and emulates size and properties probe as it is done
>> during PCI device enumeration by the guest.
>>
>> All empty, IO and ROM BARs for guests are emulated by returning 0 on
>> reads and ignoring writes: this BARs are special with this respect as
>> their lower bits have special meaning, so returning default ~0 on read
>> may confuse guest OS.
>>
>> Memory decoding is initially disabled when used by guests in order to
>> prevent the BAR being placed on top of a RAM region.
>>
>> Signed-off-by: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>
>> ---
>> Since v5:
>> - make sure that the guest set address has the same page offset
>> as the physical address on the host
>> - remove guest_rom_{read|write} as those just implement the default
>> behaviour of the registers not being handled
>> - adjusted comment for struct vpci.addr field
>> - add guest handlers for BARs which are not handled and will otherwise
>> return ~0 on read and ignore writes. The BARs are special with this
>> respect as their lower bits have special meaning, so returning ~0
>> doesn't seem to be right
>> Since v4:
>> - updated commit message
>> - s/guest_addr/guest_reg
>> Since v3:
>> - squashed two patches: dynamic add/remove handlers and guest BAR
>> handler implementation
>> - fix guest BAR read of the high part of a 64bit BAR (Roger)
>> - add error handling to vpci_assign_device
>> - s/dom%pd/%pd
>> - blank line before return
>> Since v2:
>> - remove unneeded ifdefs for CONFIG_HAS_VPCI_GUEST_SUPPORT as more code
>> has been eliminated from being built on x86
>> Since v1:
>> - constify struct pci_dev where possible
>> - do not open code is_system_domain()
>> - simplify some code3. simplify
>> - use gdprintk + error code instead of gprintk
>> - gate vpci_bar_{add|remove}_handlers with CONFIG_HAS_VPCI_GUEST_SUPPORT,
>> so these do not get compiled for x86
>> - removed unneeded is_system_domain check
>> - re-work guest read/write to be much simpler and do more work on write
>> than read which is expected to be called more frequently
>> - removed one too obvious comment
>> ---
>> xen/drivers/vpci/header.c | 131 +++++++++++++++++++++++++++++++++-----
>> xen/include/xen/vpci.h | 3 +
>> 2 files changed, 118 insertions(+), 16 deletions(-)
>>
>> diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c
>> index bd23c0274d48..2620a95ff35b 100644
>> --- a/xen/drivers/vpci/header.c
>> +++ b/xen/drivers/vpci/header.c
>> @@ -406,6 +406,81 @@ static void bar_write(const struct pci_dev *pdev, unsigned int reg,
>> pci_conf_write32(pdev->sbdf, reg, val);
>> }
>>
>> +static void guest_bar_write(const struct pci_dev *pdev, unsigned int reg,
>> + uint32_t val, void *data)
>> +{
>> + struct vpci_bar *bar = data;
>> + bool hi = false;
>> + uint64_t guest_reg = bar->guest_reg;
>> +
>> + if ( bar->type == VPCI_BAR_MEM64_HI )
>> + {
>> + ASSERT(reg > PCI_BASE_ADDRESS_0);
>> + bar--;
>> + hi = true;
>> + }
>> + else
>> + {
>> + val &= PCI_BASE_ADDRESS_MEM_MASK;
>> + val |= bar->type == VPCI_BAR_MEM32 ? PCI_BASE_ADDRESS_MEM_TYPE_32
>> + : PCI_BASE_ADDRESS_MEM_TYPE_64;
>> + val |= bar->prefetchable ? PCI_BASE_ADDRESS_MEM_PREFETCH : 0;
>> + }
>> +
>> + guest_reg &= ~(0xffffffffull << (hi ? 32 : 0));
>> + guest_reg |= (uint64_t)val << (hi ? 32 : 0);
>> +
>> + guest_reg &= ~(bar->size - 1) | ~PCI_BASE_ADDRESS_MEM_MASK;
>> +
>> + /*
>> + * Make sure that the guest set address has the same page offset
>> + * as the physical address on the host or otherwise things won't work as
>> + * expected.
>> + */
>> + if ( (guest_reg & (~PAGE_MASK & PCI_BASE_ADDRESS_MEM_MASK)) !=
>> + (bar->addr & ~PAGE_MASK) )
> This is only required when !hi, but I'm fine with doing it
> unconditionally as it's clearer.
This is correct wrt hi
>
>> + {
>> + gprintk(XENLOG_WARNING,
>> + "%pp: ignored BAR %zu write with wrong page offset\n",
> "%pp: ignored BAR %zu write attempting to change page offset\n"
Ok
>
>> + &pdev->sbdf, bar - pdev->vpci->header.bars + hi);
>> + return;
>> + }
>> +
>> + bar->guest_reg = guest_reg;
>> +}
>> +
>> +static uint32_t guest_bar_read(const struct pci_dev *pdev, unsigned int reg,
>> + void *data)
>> +{
>> + const struct vpci_bar *bar = data;
>> + bool hi = false;
>> +
>> + if ( bar->type == VPCI_BAR_MEM64_HI )
>> + {
>> + ASSERT(reg > PCI_BASE_ADDRESS_0);
>> + bar--;
>> + hi = true;
>> + }
>> +
>> + return bar->guest_reg >> (hi ? 32 : 0);
>> +}
>> +
>> +static uint32_t guest_bar_ignore_read(const struct pci_dev *pdev,
>> + unsigned int reg, void *data)
>> +{
>> + return 0;
>> +}
>> +
>> +static int bar_ignore_access(const struct pci_dev *pdev, unsigned int reg,
>> + struct vpci_bar *bar)
>> +{
>> + if ( is_hardware_domain(pdev->domain) )
>> + return 0;
>> +
>> + return vpci_add_register(pdev->vpci, guest_bar_ignore_read, NULL,
>> + reg, 4, bar);
>> +}
>> +
>> static void rom_write(const struct pci_dev *pdev, unsigned int reg,
>> uint32_t val, void *data)
>> {
>> @@ -462,6 +537,7 @@ static int init_bars(struct pci_dev *pdev)
>> struct vpci_header *header = &pdev->vpci->header;
>> struct vpci_bar *bars = header->bars;
>> int rc;
>> + bool is_hwdom = is_hardware_domain(pdev->domain);
>>
>> switch ( pci_conf_read8(pdev->sbdf, PCI_HEADER_TYPE) & 0x7f )
>> {
>> @@ -501,8 +577,10 @@ static int init_bars(struct pci_dev *pdev)
>> if ( i && bars[i - 1].type == VPCI_BAR_MEM64_LO )
>> {
>> bars[i].type = VPCI_BAR_MEM64_HI;
>> - rc = vpci_add_register(pdev->vpci, vpci_hw_read32, bar_write, reg,
>> - 4, &bars[i]);
>> + rc = vpci_add_register(pdev->vpci,
>> + is_hwdom ? vpci_hw_read32 : guest_bar_read,
>> + is_hwdom ? bar_write : guest_bar_write,
>> + reg, 4, &bars[i]);
>> if ( rc )
>> {
>> pci_conf_write16(pdev->sbdf, PCI_COMMAND, cmd);
>> @@ -516,6 +594,11 @@ static int init_bars(struct pci_dev *pdev)
>> if ( (val & PCI_BASE_ADDRESS_SPACE) == PCI_BASE_ADDRESS_SPACE_IO )
>> {
>> bars[i].type = VPCI_BAR_IO;
>> +
>> + rc = bar_ignore_access(pdev, reg, &bars[i]);
> This is wrong: you only want to ignore access to IO BARs for Arm, for
> x86 we should keep the previous behavior. Even more if you go with
> Jan's suggestions to make bar_ignore_access also applicable to dom0.
How do we want this?
#ifdef CONFIG_ARM?
>
>> + if ( rc )
>> + return rc;
>> +
>> continue;
>> }
>> if ( (val & PCI_BASE_ADDRESS_MEM_TYPE_MASK) ==
>> @@ -535,6 +618,11 @@ static int init_bars(struct pci_dev *pdev)
>> if ( size == 0 )
>> {
>> bars[i].type = VPCI_BAR_EMPTY;
>> +
>> + rc = bar_ignore_access(pdev, reg, &bars[i]);
>> + if ( rc )
>> + return rc;
> I would be fine to just call vpci_add_register here, ie;
>
> if ( !is_hwdom )
> {
> rc = vpci_add_register(pdev->vpci, guest_bar_ignore_read, NULL,
> reg, 4, &bars[i]);
> if ( rc )
> {
> ...
> }
> }
But we have 3 places where we do the same and also handle errors
the same way. I was thinking having a helper will make the code
clearer. Do you want to open code all the uses?
> Feel free to unify the writing of the PCI_COMMAND register on the
> error path into a label, as then the error case would simply be a
> `goto error;`
I was thinking about it. Will it be ok to make this change in this patch
or you want a dedicated one for that?
> Thanks, Roger.
Thank you,
Oleksandr
next prev parent reply other threads:[~2022-02-08 9:32 UTC|newest]
Thread overview: 138+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-02-04 6:34 [PATCH v6 00/13] PCI devices passthrough on Arm, part 3 Oleksandr Andrushchenko
2022-02-04 6:34 ` [PATCH v6 01/13] xen/pci: arm: add stub for is_memory_hole Oleksandr Andrushchenko
2022-02-04 8:51 ` Julien Grall
2022-02-04 9:01 ` Oleksandr Andrushchenko
2022-02-04 9:41 ` Julien Grall
2022-02-04 9:47 ` Oleksandr Andrushchenko
2022-02-04 9:57 ` Julien Grall
2022-02-04 10:35 ` Oleksandr Andrushchenko
2022-02-04 11:00 ` Julien Grall
2022-02-04 11:25 ` Oleksandr Andrushchenko
2022-02-04 6:34 ` [PATCH v6 02/13] rangeset: add RANGESETF_no_print flag Oleksandr Andrushchenko
2022-02-04 6:34 ` [PATCH v6 03/13] vpci: move lock outside of struct vpci Oleksandr Andrushchenko
2022-02-04 7:52 ` Jan Beulich
2022-02-04 8:13 ` Oleksandr Andrushchenko
2022-02-04 8:36 ` Jan Beulich
2022-02-04 8:58 ` Oleksandr Andrushchenko
2022-02-04 9:15 ` Jan Beulich
2022-02-04 10:12 ` Oleksandr Andrushchenko
2022-02-04 10:49 ` Jan Beulich
2022-02-04 11:13 ` Roger Pau Monné
2022-02-04 11:37 ` Jan Beulich
2022-02-04 12:37 ` Oleksandr Andrushchenko
2022-02-04 12:47 ` Jan Beulich
2022-02-04 12:53 ` Oleksandr Andrushchenko
2022-02-04 13:03 ` Jan Beulich
2022-02-04 13:06 ` Roger Pau Monné
2022-02-04 14:43 ` Oleksandr Andrushchenko
2022-02-04 14:57 ` Roger Pau Monné
2022-02-07 11:08 ` Oleksandr Andrushchenko
2022-02-07 12:34 ` Jan Beulich
2022-02-07 12:57 ` Oleksandr Andrushchenko
2022-02-07 13:02 ` Jan Beulich
2022-02-07 12:46 ` Roger Pau Monné
2022-02-07 13:53 ` Oleksandr Andrushchenko
2022-02-07 14:11 ` Jan Beulich
2022-02-07 14:27 ` Roger Pau Monné
2022-02-07 14:33 ` Jan Beulich
2022-02-07 14:35 ` Oleksandr Andrushchenko
2022-02-07 15:11 ` Oleksandr Andrushchenko
2022-02-07 15:26 ` Jan Beulich
2022-02-07 16:07 ` Oleksandr Andrushchenko
2022-02-07 16:15 ` Jan Beulich
2022-02-07 16:21 ` Oleksandr Andrushchenko
2022-02-07 16:37 ` Jan Beulich
2022-02-07 16:44 ` Oleksandr Andrushchenko
2022-02-08 7:35 ` Oleksandr Andrushchenko
2022-02-08 8:57 ` Jan Beulich
2022-02-08 9:03 ` Oleksandr Andrushchenko
2022-02-08 10:50 ` Roger Pau Monné
2022-02-08 11:13 ` Oleksandr Andrushchenko
2022-02-08 13:38 ` Roger Pau Monné
2022-02-08 13:52 ` Oleksandr Andrushchenko
2022-02-08 8:53 ` Jan Beulich
2022-02-08 9:00 ` Oleksandr Andrushchenko
2022-02-08 10:11 ` Roger Pau Monné
2022-02-08 10:32 ` Oleksandr Andrushchenko
2022-02-07 16:08 ` Roger Pau Monné
2022-02-07 16:12 ` Jan Beulich
2022-02-07 14:28 ` Oleksandr Andrushchenko
2022-02-07 14:19 ` Roger Pau Monné
2022-02-07 14:27 ` Oleksandr Andrushchenko
2022-02-04 11:37 ` Oleksandr Andrushchenko
2022-02-04 12:15 ` Roger Pau Monné
2022-02-04 10:57 ` Roger Pau Monné
2022-02-04 6:34 ` [PATCH v6 04/13] vpci: restrict unhandled read/write operations for guests Oleksandr Andrushchenko
2022-02-04 14:11 ` Jan Beulich
2022-02-04 14:24 ` Oleksandr Andrushchenko
2022-02-08 8:00 ` Oleksandr Andrushchenko
2022-02-08 9:04 ` Jan Beulich
2022-02-08 9:09 ` Oleksandr Andrushchenko
2022-02-08 9:05 ` Roger Pau Monné
2022-02-08 9:10 ` Oleksandr Andrushchenko
2022-02-04 6:34 ` [PATCH v6 05/13] vpci: add hooks for PCI device assign/de-assign Oleksandr Andrushchenko
2022-02-07 16:28 ` Jan Beulich
2022-02-08 8:32 ` Oleksandr Andrushchenko
2022-02-08 9:13 ` Jan Beulich
2022-02-08 9:27 ` Oleksandr Andrushchenko
2022-02-08 9:44 ` Jan Beulich
2022-02-08 9:55 ` Oleksandr Andrushchenko
2022-02-08 10:09 ` Jan Beulich
2022-02-08 10:22 ` Oleksandr Andrushchenko
2022-02-08 10:29 ` Jan Beulich
2022-02-08 10:52 ` Oleksandr Andrushchenko
2022-02-08 11:00 ` Jan Beulich
2022-02-08 11:25 ` Oleksandr Andrushchenko
2022-02-10 8:21 ` Oleksandr Andrushchenko
2022-02-10 9:22 ` Jan Beulich
2022-02-10 9:33 ` Oleksandr Andrushchenko
2022-02-04 6:34 ` [PATCH v6 06/13] vpci/header: implement guest BAR register handlers Oleksandr Andrushchenko
2022-02-07 17:06 ` Jan Beulich
2022-02-08 8:06 ` Oleksandr Andrushchenko
2022-02-08 9:16 ` Jan Beulich
2022-02-08 9:29 ` Roger Pau Monné
2022-02-08 9:25 ` Roger Pau Monné
2022-02-08 9:31 ` Oleksandr Andrushchenko [this message]
2022-02-08 9:48 ` Jan Beulich
2022-02-08 9:57 ` Oleksandr Andrushchenko
2022-02-08 10:15 ` Jan Beulich
2022-02-08 10:29 ` Oleksandr Andrushchenko
2022-02-08 13:58 ` Roger Pau Monné
2022-02-04 6:34 ` [PATCH v6 07/13] vpci/header: handle p2m range sets per BAR Oleksandr Andrushchenko
2022-02-04 6:34 ` [PATCH v6 08/13] vpci/header: program p2m with guest BAR view Oleksandr Andrushchenko
2022-02-04 6:34 ` [PATCH v6 09/13] vpci/header: emulate PCI_COMMAND register for guests Oleksandr Andrushchenko
2022-02-04 14:25 ` Jan Beulich
2022-02-08 8:13 ` Oleksandr Andrushchenko
2022-02-08 9:33 ` Jan Beulich
2022-02-08 9:38 ` Oleksandr Andrushchenko
2022-02-08 9:52 ` Jan Beulich
2022-02-08 9:58 ` Oleksandr Andrushchenko
2022-02-08 11:11 ` Roger Pau Monné
2022-02-08 11:29 ` Oleksandr Andrushchenko
2022-02-08 14:09 ` Roger Pau Monné
2022-02-08 14:13 ` Oleksandr Andrushchenko
2022-02-04 6:34 ` [PATCH v6 10/13] vpci/header: reset the command register when adding devices Oleksandr Andrushchenko
2022-02-04 14:30 ` Jan Beulich
2022-02-04 14:37 ` Oleksandr Andrushchenko
2022-02-07 7:29 ` Jan Beulich
2022-02-07 11:27 ` Oleksandr Andrushchenko
2022-02-07 12:38 ` Jan Beulich
2022-02-07 12:51 ` Oleksandr Andrushchenko
2022-02-07 12:54 ` Jan Beulich
2022-02-07 14:17 ` Oleksandr Andrushchenko
2022-02-07 14:31 ` Jan Beulich
2022-02-07 14:46 ` Oleksandr Andrushchenko
2022-02-07 15:05 ` Jan Beulich
2022-02-07 15:14 ` Oleksandr Andrushchenko
2022-02-07 15:28 ` Jan Beulich
2022-02-07 15:59 ` Oleksandr Andrushchenko
2022-02-10 12:54 ` Oleksandr Andrushchenko
2022-02-10 13:36 ` Jan Beulich
2022-02-10 13:56 ` Oleksandr Andrushchenko
2022-02-10 12:59 ` Oleksandr Andrushchenko
2022-02-04 6:34 ` [PATCH v6 11/13] vpci: add initial support for virtual PCI bus topology Oleksandr Andrushchenko
2022-02-04 6:34 ` [PATCH v6 12/13] xen/arm: translate virtual PCI bus topology for guests Oleksandr Andrushchenko
2022-02-04 7:56 ` Jan Beulich
2022-02-04 8:18 ` Oleksandr Andrushchenko
2022-02-04 6:34 ` [PATCH v6 13/13] xen/arm: account IO handlers for emulated PCI MSI-X Oleksandr Andrushchenko
2022-02-11 15:28 ` Julien Grall
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=498ca523-2642-e35f-b316-0a089f98ca91@epam.com \
--to=oleksandr_andrushchenko@epam.com \
--cc=Artem_Mygaiev@epam.com \
--cc=Oleksandr_Tyshchenko@epam.com \
--cc=Volodymyr_Babchuk@epam.com \
--cc=andrew.cooper3@citrix.com \
--cc=bertrand.marquis@arm.com \
--cc=george.dunlap@citrix.com \
--cc=jbeulich@suse.com \
--cc=julien@xen.org \
--cc=paul@xen.org \
--cc=rahul.singh@arm.com \
--cc=roger.pau@citrix.com \
--cc=sstabellini@kernel.org \
--cc=xen-devel@lists.xenproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).