From: Alex Williamson <alex.williamson@redhat.com>
To: Matthew Rosato <mjrosato@linux.ibm.com>
Cc: Bjorn Helgaas <helgaas@kernel.org>,
bhelgaas@google.com, schnelle@linux.ibm.com,
pmorel@linux.ibm.com, mpe@ellerman.id.au, oohall@gmail.com,
cohuck@redhat.com, kevin.tian@intel.com,
linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org,
kvm@vger.kernel.org, linux-pci@vger.kernel.org
Subject: Re: [PATCH v4 1/3] PCI/IOV: Mark VFs as not implementing MSE bit
Date: Wed, 9 Sep 2020 17:07:46 -0600 [thread overview]
Message-ID: <20200909170746.2286b83a@w520.home> (raw)
In-Reply-To: <38f95349-237e-34e2-66ef-e626cd4aec25@linux.ibm.com>
On Thu, 3 Sep 2020 13:10:02 -0400
Matthew Rosato <mjrosato@linux.ibm.com> wrote:
> On 9/3/20 12:41 PM, Bjorn Helgaas wrote:
> > On Wed, Sep 02, 2020 at 03:46:34PM -0400, Matthew Rosato wrote:
> >> Per the PCIe spec, VFs cannot implement the MSE bit
> >> AKA PCI_COMMAND_MEMORY, and it must be hard-wired to 0.
> >> Use a dev_flags bit to signify this requirement.
> >
> > This approach seems sensible to me, but
> >
> > - This is confusing because while the spec does not use "MSE" to
> > refer to the Command Register "Memory Space Enable" bit
> > (PCI_COMMAND_MEMORY), it *does* use "MSE" in the context of the
> > "VF MSE" bit, which is in the PF SR-IOV Capability. But of
> > course, you're not talking about that here. Maybe something like
> > this?
> >
> > For VFs, the Memory Space Enable bit in the Command Register is
> > hard-wired to 0.
> >
> > Add a dev_flags bit to signify devices where the Command
> > Register Memory Space Enable bit does not control the device's
> > response to MMIO accesses.
>
> Will do. I'll change the usage of the MSE acronym in the other patches
> as well.
>
> >
> > - "PCI_DEV_FLAGS_FORCE_COMMAND_MEM" says something about how you
> > plan to *use* this, but I'd rather use a term that describes the
> > hardware, e.g., "PCI_DEV_FLAGS_NO_COMMAND_MEMORY".
>
> Sure, I will change.
>
> >
> > - How do we decide whether to use dev_flags vs a bitfield like
> > dev->is_virtfn? The latter seems simpler unless there's a reason
> > to use dev_flags. If there's a reason, maybe we could add a
> > comment at pci_dev_flags for future reference.
> >
>
> Something like:
>
> /*
> * Device does not implement PCI_COMMAND_MEMORY - this is true for any
> * device marked is_virtfn, but is also true for any VF passed-through
> * a lower-level hypervisor where emulation of the Memory Space Enable
> * bit was not provided.
> */
> PCI_DEV_FLAGS_NO_COMMAND_MEMORY = (__force pci_dev_flags_t) (1 << 12),
>
> ?
>
> > - Wrap the commit log to fill a 75-char line. It's arbitrary, but
> > that's what I use for consistency.
>
> Sure, will do. I'll roll up a new version once I have feedback from
> Alex on the vfio changes.
The usage of MSE threw me a bit too, as Bjorn notes that's specific to
the SR-IOV capability. I think this also uncovers a latent bug in our
calling of vfio_bar_restore(), it really doesn't do a good job of
determining whether an enable bit is implemented, regardless of whether
it's a VF or the device simply doesn't use that address space. For
example I imagine you could reproduce triggering a reset recovery on
s390 by trying to write the VF command register to 1 with setpci from a
guest (since you won't have is_virtfn to bail out of the recovery
function). I think we'll still need this dev_flag to differentiate
unimplmented and enabled versus simply unimplemented to resolve that
though, so the change looks ok to me. Thanks,
Alex
> >> Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
> >> ---
> >> drivers/pci/iov.c | 1 +
> >> include/linux/pci.h | 2 ++
> >> 2 files changed, 3 insertions(+)
> >>
> >> diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
> >> index b37e08c..2bec77c 100644
> >> --- a/drivers/pci/iov.c
> >> +++ b/drivers/pci/iov.c
> >> @@ -180,6 +180,7 @@ int pci_iov_add_virtfn(struct pci_dev *dev, int id)
> >> virtfn->device = iov->vf_device;
> >> virtfn->is_virtfn = 1;
> >> virtfn->physfn = pci_dev_get(dev);
> >> + virtfn->dev_flags |= PCI_DEV_FLAGS_FORCE_COMMAND_MEM;
> >>
> >> if (id == 0)
> >> pci_read_vf_config_common(virtfn);
> >> diff --git a/include/linux/pci.h b/include/linux/pci.h
> >> index 8355306..9316cce 100644
> >> --- a/include/linux/pci.h
> >> +++ b/include/linux/pci.h
> >> @@ -227,6 +227,8 @@ enum pci_dev_flags {
> >> PCI_DEV_FLAGS_NO_FLR_RESET = (__force pci_dev_flags_t) (1 << 10),
> >> /* Don't use Relaxed Ordering for TLPs directed at this device */
> >> PCI_DEV_FLAGS_NO_RELAXED_ORDERING = (__force pci_dev_flags_t) (1 << 11),
> >> + /* Device does not implement PCI_COMMAND_MEMORY (e.g. a VF) */
> >> + PCI_DEV_FLAGS_FORCE_COMMAND_MEM = (__force pci_dev_flags_t) (1 << 12),
> >> };
> >>
> >> enum pci_irq_reroute_variant {
> >> --
> >> 1.8.3.1
> >>
>
next prev parent reply other threads:[~2020-09-10 2:24 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-09-02 19:46 [PATCH v4 0/3] vfio/pci: Restore MMIO access for s390 detached VFs Matthew Rosato
2020-09-02 19:46 ` [PATCH v4 1/3] PCI/IOV: Mark VFs as not implementing MSE bit Matthew Rosato
2020-09-03 16:41 ` Bjorn Helgaas
2020-09-03 17:10 ` Matthew Rosato
2020-09-09 23:07 ` Alex Williamson [this message]
2020-09-10 0:39 ` Bjorn Helgaas
2020-09-02 19:46 ` [PATCH v4 2/3] s390/pci: Mark all " Matthew Rosato
2020-09-02 19:46 ` [PATCH v4 3/3] vfio/pci: Decouple MSE bit checks from is_virtfn Matthew Rosato
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200909170746.2286b83a@w520.home \
--to=alex.williamson@redhat.com \
--cc=bhelgaas@google.com \
--cc=cohuck@redhat.com \
--cc=helgaas@kernel.org \
--cc=kevin.tian@intel.com \
--cc=kvm@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-pci@vger.kernel.org \
--cc=linux-s390@vger.kernel.org \
--cc=mjrosato@linux.ibm.com \
--cc=mpe@ellerman.id.au \
--cc=oohall@gmail.com \
--cc=pmorel@linux.ibm.com \
--cc=schnelle@linux.ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).