* [PATCH] vfio/pci: Support 8-byte PCI loads and stores
@ 2024-04-19 13:53 Gerd Bayer
2024-04-19 13:58 ` Jason Gunthorpe
0 siblings, 1 reply; 6+ messages in thread
From: Gerd Bayer @ 2024-04-19 13:53 UTC (permalink / raw)
To: Alex Williamson, Jason Gunthorpe, Ankit Agrawal, Yishai Hadas
Cc: kvm, linux-s390, Halil Pasic, Niklas Schnelle, Ben Segal, Gerd Bayer
From: Ben Segal <bpsegal@us.ibm.com>
Many PCI adapters can benefit or even require full 64bit read
and write access to their registers. In order to enable work on
user-space drivers for these devices add two new variations
vfio_pci_core_io{read|write}64 of the existing access methods
when the architecture supports 64-bit ioreads and iowrites.
Signed-off-by: Ben Segal <bpsegal@us.ibm.com>
Co-developed-by: Gerd Bayer <gbayer@linux.ibm.com>
Signed-off-by: Gerd Bayer <gbayer@linux.ibm.com>
---
Hi all,
we've successfully used this patch with a user-mode driver for a PCI
device that requires 64bit register read/writes on s390. A quick grep
showed that there are several other drivers for PCI devices in the kernel
that use readq/writeq and eventually could use this too.
So we decided to propose this for general inclusion.
We've added conditional compiles for non-64bit architectures that
produce graceful run-time errors. However, that path is just
compile-tested.
Thank you,
Gerd Bayer
drivers/vfio/pci/vfio_pci_rdwr.c | 39 +++++++++++++++++++++++++++++++-
include/linux/vfio_pci_core.h | 3 +++
2 files changed, 41 insertions(+), 1 deletion(-)
diff --git a/drivers/vfio/pci/vfio_pci_rdwr.c b/drivers/vfio/pci/vfio_pci_rdwr.c
index 03b8f7ada1ac..3f91945ea3ff 100644
--- a/drivers/vfio/pci/vfio_pci_rdwr.c
+++ b/drivers/vfio/pci/vfio_pci_rdwr.c
@@ -89,6 +89,9 @@ EXPORT_SYMBOL_GPL(vfio_pci_core_ioread##size);
VFIO_IOREAD(8)
VFIO_IOREAD(16)
VFIO_IOREAD(32)
+#ifdef ioread64
+VFIO_IOREAD(64)
+#endif
/*
* Read or write from an __iomem region (MMIO or I/O port) with an excluded
@@ -114,7 +117,41 @@ ssize_t vfio_pci_core_do_io_rw(struct vfio_pci_core_device *vdev, bool test_mem,
else
fillable = 0;
- if (fillable >= 4 && !(off % 4)) {
+ if (fillable >= 8 && !(off % 8)) {
+#if defined(ioread64) || defined(iowrite64)
+ u64 val;
+#endif
+
+ if (iswrite) {
+#ifndef iowrite64
+ pr_err_once("vfio does not support iowrite64 on this arch");
+ return -EIO;
+#else
+ if (copy_from_user(&val, buf, 8))
+ return -EFAULT;
+
+ ret = vfio_pci_core_iowrite64(vdev, test_mem,
+ val, io + off);
+ if (ret)
+ return ret;
+#endif
+ } else {
+#ifndef ioread64
+ pr_err_once("vfio does not support ioread64 on this arch");
+ return -EIO;
+#else
+ ret = vfio_pci_core_ioread64(vdev, test_mem,
+ &val, io + off);
+ if (ret)
+ return ret;
+
+ if (copy_to_user(buf, &val, 8))
+ return -EFAULT;
+#endif
+ }
+
+ filled = 8;
+ } else if (fillable >= 4 && !(off % 4)) {
u32 val;
if (iswrite) {
diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h
index a2c8b8bba711..f4cf5fd2350c 100644
--- a/include/linux/vfio_pci_core.h
+++ b/include/linux/vfio_pci_core.h
@@ -157,5 +157,8 @@ int vfio_pci_core_ioread##size(struct vfio_pci_core_device *vdev, \
VFIO_IOREAD_DECLATION(8)
VFIO_IOREAD_DECLATION(16)
VFIO_IOREAD_DECLATION(32)
+#ifdef ioread64
+VFIO_IOREAD_DECLATION(64)
+#endif
#endif /* VFIO_PCI_CORE_H */
--
2.44.0
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH] vfio/pci: Support 8-byte PCI loads and stores
2024-04-19 13:53 [PATCH] vfio/pci: Support 8-byte PCI loads and stores Gerd Bayer
@ 2024-04-19 13:58 ` Jason Gunthorpe
2024-04-19 15:57 ` Niklas Schnelle
0 siblings, 1 reply; 6+ messages in thread
From: Jason Gunthorpe @ 2024-04-19 13:58 UTC (permalink / raw)
To: Gerd Bayer
Cc: Alex Williamson, Ankit Agrawal, Yishai Hadas, kvm, linux-s390,
Halil Pasic, Niklas Schnelle, Ben Segal
On Fri, Apr 19, 2024 at 03:53:23PM +0200, Gerd Bayer wrote:
> From: Ben Segal <bpsegal@us.ibm.com>
>
> Many PCI adapters can benefit or even require full 64bit read
> and write access to their registers. In order to enable work on
> user-space drivers for these devices add two new variations
> vfio_pci_core_io{read|write}64 of the existing access methods
> when the architecture supports 64-bit ioreads and iowrites.
>
> Signed-off-by: Ben Segal <bpsegal@us.ibm.com>
> Co-developed-by: Gerd Bayer <gbayer@linux.ibm.com>
> Signed-off-by: Gerd Bayer <gbayer@linux.ibm.com>
> ---
>
> Hi all,
>
> we've successfully used this patch with a user-mode driver for a PCI
> device that requires 64bit register read/writes on s390.
But why? S390 already has a system call for userspace to do the 64 bit
write, and newer S390 has a userspace instruction to do it.
Why would you want to use a VFIO system call on the mmio emulation
path?
mmap the registers and access them normally?
> * Read or write from an __iomem region (MMIO or I/O port) with an excluded
> @@ -114,7 +117,41 @@ ssize_t vfio_pci_core_do_io_rw(struct vfio_pci_core_device *vdev, bool test_mem,
> else
> fillable = 0;
>
> - if (fillable >= 4 && !(off % 4)) {
> + if (fillable >= 8 && !(off % 8)) {
> +#if defined(ioread64) || defined(iowrite64)
> + u64 val;
> +#endif
> +
> + if (iswrite) {
> +#ifndef iowrite64
> + pr_err_once("vfio does not support iowrite64 on this arch");
> + return -EIO;
can't do that you have to go back to what the old stuff did and do the
4 byte copy.
Jason
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] vfio/pci: Support 8-byte PCI loads and stores
2024-04-19 13:58 ` Jason Gunthorpe
@ 2024-04-19 15:57 ` Niklas Schnelle
2024-04-19 16:11 ` Jason Gunthorpe
0 siblings, 1 reply; 6+ messages in thread
From: Niklas Schnelle @ 2024-04-19 15:57 UTC (permalink / raw)
To: Jason Gunthorpe, Gerd Bayer
Cc: Alex Williamson, Ankit Agrawal, Yishai Hadas, kvm, linux-s390,
Halil Pasic, Ben Segal
On Fri, 2024-04-19 at 10:58 -0300, Jason Gunthorpe wrote:
> On Fri, Apr 19, 2024 at 03:53:23PM +0200, Gerd Bayer wrote:
> > From: Ben Segal <bpsegal@us.ibm.com>
> >
> > Many PCI adapters can benefit or even require full 64bit read
> > and write access to their registers. In order to enable work on
> > user-space drivers for these devices add two new variations
> > vfio_pci_core_io{read|write}64 of the existing access methods
> > when the architecture supports 64-bit ioreads and iowrites.
> >
> > Signed-off-by: Ben Segal <bpsegal@us.ibm.com>
> > Co-developed-by: Gerd Bayer <gbayer@linux.ibm.com>
> > Signed-off-by: Gerd Bayer <gbayer@linux.ibm.com>
> > ---
> >
> > Hi all,
> >
> > we've successfully used this patch with a user-mode driver for a PCI
> > device that requires 64bit register read/writes on s390.
>
> But why? S390 already has a system call for userspace to do the 64 bit
> write, and newer S390 has a userspace instruction to do it.
>
> Why would you want to use a VFIO system call on the mmio emulation
> path?
>
> mmap the registers and access them normally?
It's a very good point and digging into why this wasn't used by
Benjamin. It turns out VFIO_PCI_MMAP is disabled for S390 which it
really shouldn't be especially now that we have the user-space
instructions. Before that though Benjamin turned to this interface
which then lead him to this limitation. So yeah we'll definitely verify
that it also works via VFIO_PCI_MMAP and send a patch to enable that.
That said I still think it's odd not to have the 8 byte case working
here even if it isn't the right approach. Could still be useful for
debug/testing without having to add the MIO instructions or the our
special syscall.
Thanks,
Niklas
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] vfio/pci: Support 8-byte PCI loads and stores
2024-04-19 15:57 ` Niklas Schnelle
@ 2024-04-19 16:11 ` Jason Gunthorpe
2024-04-19 16:47 ` Alex Williamson
0 siblings, 1 reply; 6+ messages in thread
From: Jason Gunthorpe @ 2024-04-19 16:11 UTC (permalink / raw)
To: Niklas Schnelle
Cc: Gerd Bayer, Alex Williamson, Ankit Agrawal, Yishai Hadas, kvm,
linux-s390, Halil Pasic, Ben Segal
On Fri, Apr 19, 2024 at 05:57:52PM +0200, Niklas Schnelle wrote:
> On Fri, 2024-04-19 at 10:58 -0300, Jason Gunthorpe wrote:
> > On Fri, Apr 19, 2024 at 03:53:23PM +0200, Gerd Bayer wrote:
> > > From: Ben Segal <bpsegal@us.ibm.com>
> > >
> > > Many PCI adapters can benefit or even require full 64bit read
> > > and write access to their registers. In order to enable work on
> > > user-space drivers for these devices add two new variations
> > > vfio_pci_core_io{read|write}64 of the existing access methods
> > > when the architecture supports 64-bit ioreads and iowrites.
> > >
> > > Signed-off-by: Ben Segal <bpsegal@us.ibm.com>
> > > Co-developed-by: Gerd Bayer <gbayer@linux.ibm.com>
> > > Signed-off-by: Gerd Bayer <gbayer@linux.ibm.com>
> > > ---
> > >
> > > Hi all,
> > >
> > > we've successfully used this patch with a user-mode driver for a PCI
> > > device that requires 64bit register read/writes on s390.
> >
> > But why? S390 already has a system call for userspace to do the 64 bit
> > write, and newer S390 has a userspace instruction to do it.
> >
> > Why would you want to use a VFIO system call on the mmio emulation
> > path?
> >
> > mmap the registers and access them normally?
>
> It's a very good point and digging into why this wasn't used by
> Benjamin. It turns out VFIO_PCI_MMAP is disabled for S390 which it
> really shouldn't be especially now that we have the user-space
> instructions. Before that though Benjamin turned to this interface
> which then lead him to this limitation. So yeah we'll definitely verify
> that it also works via VFIO_PCI_MMAP and send a patch to enable that.
Make sense to me!
> That said I still think it's odd not to have the 8 byte case working
> here even if it isn't the right approach. Could still be useful for
> debug/testing without having to add the MIO instructions or the our
> special syscall.
Yes, this also makes sense, but this patch needs some adjusting
Jason
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] vfio/pci: Support 8-byte PCI loads and stores
2024-04-19 16:11 ` Jason Gunthorpe
@ 2024-04-19 16:47 ` Alex Williamson
2024-04-22 11:08 ` Gerd Bayer
0 siblings, 1 reply; 6+ messages in thread
From: Alex Williamson @ 2024-04-19 16:47 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Niklas Schnelle, Gerd Bayer, Ankit Agrawal, Yishai Hadas, kvm,
linux-s390, Halil Pasic, Ben Segal
On Fri, 19 Apr 2024 13:11:35 -0300
Jason Gunthorpe <jgg@ziepe.ca> wrote:
> On Fri, Apr 19, 2024 at 05:57:52PM +0200, Niklas Schnelle wrote:
> > On Fri, 2024-04-19 at 10:58 -0300, Jason Gunthorpe wrote:
> > > On Fri, Apr 19, 2024 at 03:53:23PM +0200, Gerd Bayer wrote:
> > > > From: Ben Segal <bpsegal@us.ibm.com>
> > > >
> > > > Many PCI adapters can benefit or even require full 64bit read
> > > > and write access to their registers. In order to enable work on
> > > > user-space drivers for these devices add two new variations
> > > > vfio_pci_core_io{read|write}64 of the existing access methods
> > > > when the architecture supports 64-bit ioreads and iowrites.
> > > >
> > > > Signed-off-by: Ben Segal <bpsegal@us.ibm.com>
> > > > Co-developed-by: Gerd Bayer <gbayer@linux.ibm.com>
> > > > Signed-off-by: Gerd Bayer <gbayer@linux.ibm.com>
> > > > ---
> > > >
> > > > Hi all,
> > > >
> > > > we've successfully used this patch with a user-mode driver for a PCI
> > > > device that requires 64bit register read/writes on s390.
> > >
> > > But why? S390 already has a system call for userspace to do the 64 bit
> > > write, and newer S390 has a userspace instruction to do it.
> > >
> > > Why would you want to use a VFIO system call on the mmio emulation
> > > path?
> > >
> > > mmap the registers and access them normally?
> >
> > It's a very good point and digging into why this wasn't used by
> > Benjamin. It turns out VFIO_PCI_MMAP is disabled for S390 which it
> > really shouldn't be especially now that we have the user-space
> > instructions. Before that though Benjamin turned to this interface
> > which then lead him to this limitation. So yeah we'll definitely verify
> > that it also works via VFIO_PCI_MMAP and send a patch to enable that.
>
> Make sense to me!
>
> > That said I still think it's odd not to have the 8 byte case working
> > here even if it isn't the right approach. Could still be useful for
> > debug/testing without having to add the MIO instructions or the our
> > special syscall.
>
> Yes, this also makes sense, but this patch needs some adjusting
Yes, I think so too, falling back to 4-byte accesses of course if
8-byte is not available. Thanks,
Alex
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] vfio/pci: Support 8-byte PCI loads and stores
2024-04-19 16:47 ` Alex Williamson
@ 2024-04-22 11:08 ` Gerd Bayer
0 siblings, 0 replies; 6+ messages in thread
From: Gerd Bayer @ 2024-04-22 11:08 UTC (permalink / raw)
To: Alex Williamson, Jason Gunthorpe
Cc: Niklas Schnelle, Ankit Agrawal, Yishai Hadas, kvm, linux-s390,
Halil Pasic, Ben Segal
On Fri, 2024-04-19 at 10:47 -0600, Alex Williamson wrote:
> On Fri, 19 Apr 2024 13:11:35 -0300
> Jason Gunthorpe <jgg@ziepe.ca> wrote:
>
> > On Fri, Apr 19, 2024 at 05:57:52PM +0200, Niklas Schnelle wrote:
> > > On Fri, 2024-04-19 at 10:58 -0300, Jason Gunthorpe wrote:
> > > > On Fri, Apr 19, 2024 at 03:53:23PM +0200, Gerd Bayer wrote:
> > > > > From: Ben Segal <bpsegal@us.ibm.com>
> > > > >
> > > > > Many PCI adapters can benefit or even require full 64bit read
> > > > > and write access to their registers. In order to enable work
> > > > > on
> > > > > user-space drivers for these devices add two new variations
> > > > > vfio_pci_core_io{read|write}64 of the existing access methods
> > > > > when the architecture supports 64-bit ioreads and iowrites.
> > > > >
> > > > > Signed-off-by: Ben Segal <bpsegal@us.ibm.com>
> > > > > Co-developed-by: Gerd Bayer <gbayer@linux.ibm.com>
> > > > > Signed-off-by: Gerd Bayer <gbayer@linux.ibm.com>
> > > > > ---
> > > > >
> > > > > Hi all,
> > > > >
> > > > > we've successfully used this patch with a user-mode driver
> > > > > for a PCI device that requires 64bit register read/writes on
> > > > > s390.
> > > >
> > > > But why? S390 already has a system call for userspace to do the
> > > > 64 bit write, and newer S390 has a userspace instruction to do
> > > > it.
> > > >
> > > > Why would you want to use a VFIO system call on the mmio
> > > > emulation path?
> > > >
> > > > mmap the registers and access them normally?
> > >
> > > It's a very good point and digging into why this wasn't used by
> > > Benjamin. It turns out VFIO_PCI_MMAP is disabled for S390 which
> > > it really shouldn't be especially now that we have the user-space
> > > instructions. Before that though Benjamin turned to this
> > > interface which then lead him to this limitation. So yeah we'll
> > > definitely verify that it also works via VFIO_PCI_MMAP and send a
> > > patch to enable that.
> >
> > Make sense to me!
> >
> > > That said I still think it's odd not to have the 8 byte case
> > > working here even if it isn't the right approach. Could still be
> > > useful for debug/testing without having to add the MIO
> > > instructions or the our special syscall.
> >
> > Yes, this also makes sense, but this patch needs some adjusting
>
> Yes, I think so too, falling back to 4-byte accesses of course if
> 8-byte is not available. Thanks,
So I'll rework this to simply fall back to 32-bit if 64-bit is not
available in a v2. And we'll investigate the VFIO_PCI_MMAP case
separately.
Thank you,
Gerd
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2024-04-22 11:08 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-04-19 13:53 [PATCH] vfio/pci: Support 8-byte PCI loads and stores Gerd Bayer
2024-04-19 13:58 ` Jason Gunthorpe
2024-04-19 15:57 ` Niklas Schnelle
2024-04-19 16:11 ` Jason Gunthorpe
2024-04-19 16:47 ` Alex Williamson
2024-04-22 11:08 ` Gerd Bayer
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.