From: Alex Williamson <alex.williamson@redhat.com>
To: Don Dutile <ddutile@redhat.com>
Cc: "Jens Axboe" <axboe@kernel.dk>,
"Keith Busch" <keith.busch@intel.com>,
linux-nvdimm@lists.01.org, linux-rdma@vger.kernel.org,
linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org,
linux-nvme@lists.infradead.org, linux-block@vger.kernel.org,
"Jérôme Glisse" <jglisse@redhat.com>,
"Jason Gunthorpe" <jgg@mellanox.com>,
"Bjorn Helgaas" <helgaas@kernel.org>,
"Christian König" <christian.koenig@amd.com>,
"Benjamin Herrenschmidt" <benh@kernel.crashing.org>,
"Bjorn Helgaas" <bhelgaas@google.com>,
"Max Gurtovoy" <maxg@mellanox.com>,
"Christoph Hellwig" <hch@lst.de>
Subject: Re: [PATCH v4 00/14] Copy Offload in NVMe Fabrics with P2P PCI Memory
Date: Tue, 8 May 2018 15:40:39 -0600 [thread overview]
Message-ID: <20180508154039.5c85a8f8@w520.home> (raw)
In-Reply-To: <c16ecc49-a0f3-152d-785b-a646a3dab60d@redhat.com>
On Tue, 8 May 2018 17:25:24 -0400
Don Dutile <ddutile@redhat.com> wrote:
> On 05/08/2018 12:57 PM, Alex Williamson wrote:
> > On Mon, 7 May 2018 18:23:46 -0500
> > Bjorn Helgaas <helgaas@kernel.org> wrote:
> >
> >> On Mon, Apr 23, 2018 at 05:30:32PM -0600, Logan Gunthorpe wrote:
> >>> Hi Everyone,
> >>>
> >>> Here's v4 of our series to introduce P2P based copy offload to NVMe
> >>> fabrics. This version has been rebased onto v4.17-rc2. A git repo
> >>> is here:
> >>>
> >>> https://github.com/sbates130272/linux-p2pmem pci-p2p-v4
> >>> ...
> >>
> >>> Logan Gunthorpe (14):
> >>> PCI/P2PDMA: Support peer-to-peer memory
> >>> PCI/P2PDMA: Add sysfs group to display p2pmem stats
> >>> PCI/P2PDMA: Add PCI p2pmem dma mappings to adjust the bus offset
> >>> PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches
> >>> docs-rst: Add a new directory for PCI documentation
> >>> PCI/P2PDMA: Add P2P DMA driver writer's documentation
> >>> block: Introduce PCI P2P flags for request and request queue
> >>> IB/core: Ensure we map P2P memory correctly in
> >>> rdma_rw_ctx_[init|destroy]()
> >>> nvme-pci: Use PCI p2pmem subsystem to manage the CMB
> >>> nvme-pci: Add support for P2P memory in requests
> >>> nvme-pci: Add a quirk for a pseudo CMB
> >>> nvmet: Introduce helper functions to allocate and free request SGLs
> >>> nvmet-rdma: Use new SGL alloc/free helper for requests
> >>> nvmet: Optionally use PCI P2P memory
> >>>
> >>> Documentation/ABI/testing/sysfs-bus-pci | 25 +
> >>> Documentation/PCI/index.rst | 14 +
> >>> Documentation/driver-api/index.rst | 2 +-
> >>> Documentation/driver-api/pci/index.rst | 20 +
> >>> Documentation/driver-api/pci/p2pdma.rst | 166 ++++++
> >>> Documentation/driver-api/{ => pci}/pci.rst | 0
> >>> Documentation/index.rst | 3 +-
> >>> block/blk-core.c | 3 +
> >>> drivers/infiniband/core/rw.c | 13 +-
> >>> drivers/nvme/host/core.c | 4 +
> >>> drivers/nvme/host/nvme.h | 8 +
> >>> drivers/nvme/host/pci.c | 118 +++--
> >>> drivers/nvme/target/configfs.c | 67 +++
> >>> drivers/nvme/target/core.c | 143 ++++-
> >>> drivers/nvme/target/io-cmd.c | 3 +
> >>> drivers/nvme/target/nvmet.h | 15 +
> >>> drivers/nvme/target/rdma.c | 22 +-
> >>> drivers/pci/Kconfig | 26 +
> >>> drivers/pci/Makefile | 1 +
> >>> drivers/pci/p2pdma.c | 814 +++++++++++++++++++++++++++++
> >>> drivers/pci/pci.c | 6 +
> >>> include/linux/blk_types.h | 18 +-
> >>> include/linux/blkdev.h | 3 +
> >>> include/linux/memremap.h | 19 +
> >>> include/linux/pci-p2pdma.h | 118 +++++
> >>> include/linux/pci.h | 4 +
> >>> 26 files changed, 1579 insertions(+), 56 deletions(-)
> >>> create mode 100644 Documentation/PCI/index.rst
> >>> create mode 100644 Documentation/driver-api/pci/index.rst
> >>> create mode 100644 Documentation/driver-api/pci/p2pdma.rst
> >>> rename Documentation/driver-api/{ => pci}/pci.rst (100%)
> >>> create mode 100644 drivers/pci/p2pdma.c
> >>> create mode 100644 include/linux/pci-p2pdma.h
> >>
> >> How do you envison merging this? There's a big chunk in drivers/pci, but
> >> really no opportunity for conflicts there, and there's significant stuff in
> >> block and nvme that I don't really want to merge.
> >>
> >> If Alex is OK with the ACS situation, I can ack the PCI parts and you could
> >> merge it elsewhere?
> >
> > AIUI from previously questioning this, the change is hidden behind a
> > build-time config option and only custom kernels or distros optimized
> > for this sort of support would enable that build option. I'm more than
> > a little dubious though that we're not going to have a wave of distros
> > enabling this only to get user complaints that they can no longer make
> > effective use of their devices for assignment due to the resulting span
> > of the IOMMU groups, nor is there any sort of compromise, configure
> > the kernel for p2p or device assignment, not both. Is this really such
> > a unique feature that distro users aren't going to be asking for both
> > features? Thanks,
> >
> > Alex
> At least 1/2 the cases presented to me by existing customers want it in a tunable kernel,
> and tunable btwn two points, if the hw allows it to be 'contained' in that manner, which
> a (layer of) switch(ing) provides.
> To me, that means a kernel cmdline parameter to _enable_, and another sysfs (configfs? ... i'm not a configfs afficionato to say which is best),
> method to make two points p2p dma capable.
That's not what's done here AIUI. There are also some complications to
making IOMMU groups dynamic, for instance could a downstream endpoint
already be in use by a userspace tool as ACS is being twiddled in
sysfs? Probably the easiest solution would be that all devices
affected by the ACS change are soft unplugged before and re-added after
the ACS change. Note that "affected" is not necessarily only the
downstream devices if the downstream port at which we're playing with
ACS is part of a multifunction device. Thanks,
Alex
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
prev parent reply other threads:[~2018-05-08 21:40 UTC|newest]
Thread overview: 103+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-04-23 23:30 [PATCH v4 00/14] Copy Offload in NVMe Fabrics with P2P PCI Memory Logan Gunthorpe
2018-04-23 23:30 ` [PATCH v4 01/14] PCI/P2PDMA: Support peer-to-peer memory Logan Gunthorpe
2018-05-07 23:00 ` Bjorn Helgaas
2018-05-07 23:09 ` Logan Gunthorpe
2018-04-23 23:30 ` [PATCH v4 02/14] PCI/P2PDMA: Add sysfs group to display p2pmem stats Logan Gunthorpe
2018-04-23 23:30 ` [PATCH v4 03/14] PCI/P2PDMA: Add PCI p2pmem dma mappings to adjust the bus offset Logan Gunthorpe
2018-05-07 23:02 ` Bjorn Helgaas
2018-04-23 23:30 ` [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches Logan Gunthorpe
2018-04-24 3:33 ` Randy Dunlap
2018-05-07 23:13 ` Bjorn Helgaas
2018-05-08 7:17 ` Christian König
2018-05-08 14:25 ` Stephen Bates
2018-05-08 16:37 ` Christian König
2018-05-08 16:27 ` Logan Gunthorpe
2018-05-08 16:50 ` Christian König
2018-05-08 19:13 ` Logan Gunthorpe
2018-05-08 19:34 ` Alex Williamson
2018-05-08 19:45 ` Logan Gunthorpe
2018-05-08 20:13 ` Alex Williamson
2018-05-08 20:19 ` Logan Gunthorpe
2018-05-08 20:43 ` Alex Williamson
2018-05-08 20:49 ` Logan Gunthorpe
2018-05-08 21:26 ` Alex Williamson
2018-05-08 21:42 ` Stephen Bates
2018-05-08 22:03 ` Alex Williamson
2018-05-08 22:10 ` Logan Gunthorpe
2018-05-08 22:25 ` Stephen Bates
2018-05-08 23:11 ` Alex Williamson
2018-05-08 23:31 ` Logan Gunthorpe
2018-05-09 0:17 ` Alex Williamson
2018-05-08 22:32 ` Alex Williamson
2018-05-08 23:00 ` Dan Williams
2018-05-08 23:15 ` Logan Gunthorpe
2018-05-09 12:38 ` Stephen Bates
2018-05-08 22:21 ` Don Dutile
2018-05-09 12:44 ` Stephen Bates
2018-05-09 15:58 ` Don Dutile
2018-05-08 20:50 ` Jerome Glisse
2018-05-08 21:35 ` Stephen Bates
2018-05-09 13:12 ` Stephen Bates
2018-05-09 13:40 ` Christian König
2018-05-09 15:41 ` Stephen Bates
2018-05-09 16:07 ` Jerome Glisse
2018-05-09 16:30 ` Stephen Bates
2018-05-09 17:49 ` Jerome Glisse
2018-05-10 14:20 ` Stephen Bates
2018-05-10 14:29 ` Christian König
2018-05-10 14:59 ` Jerome Glisse
2018-05-10 18:44 ` Stephen Bates
2018-05-09 16:45 ` Logan Gunthorpe
2018-05-10 12:52 ` Christian König
2018-05-10 14:16 ` Stephen Bates
2018-05-10 14:41 ` Jerome Glisse
2018-05-10 18:41 ` Stephen Bates
2018-05-10 18:59 ` Logan Gunthorpe
2018-05-10 19:10 ` Alex Williamson
2018-05-10 19:24 ` Jerome Glisse
2018-05-10 16:32 ` Logan Gunthorpe
2018-05-10 17:11 ` Stephen Bates
2018-05-10 17:15 ` Logan Gunthorpe
2018-05-11 8:52 ` Christian König
2018-05-11 15:48 ` Logan Gunthorpe
2018-05-11 21:50 ` Stephen Bates
2018-05-11 22:24 ` Stephen Bates
2018-05-11 22:55 ` Logan Gunthorpe
2018-05-08 14:31 ` Dan Williams
2018-05-08 14:44 ` Stephen Bates
2018-05-08 21:04 ` Don Dutile
2018-05-08 21:27 ` Stephen Bates
2018-05-08 23:06 ` Don Dutile
2018-05-09 0:01 ` Alex Williamson
2018-05-09 12:35 ` Stephen Bates
2018-05-09 14:44 ` Alex Williamson
2018-05-09 15:52 ` Don Dutile
2018-05-09 15:47 ` Don Dutile
2018-05-09 15:53 ` Don Dutile
2018-04-23 23:30 ` [PATCH v4 05/14] docs-rst: Add a new directory for PCI documentation Logan Gunthorpe
2018-04-23 23:30 ` [PATCH v4 06/14] PCI/P2PDMA: Add P2P DMA driver writer's documentation Logan Gunthorpe
2018-05-07 23:20 ` Bjorn Helgaas
2018-05-22 21:24 ` Randy Dunlap
2018-05-22 21:28 ` Logan Gunthorpe
2018-04-23 23:30 ` [PATCH v4 07/14] block: Introduce PCI P2P flags for request and request queue Logan Gunthorpe
2018-04-23 23:30 ` [PATCH v4 08/14] IB/core: Ensure we map P2P memory correctly in rdma_rw_ctx_[init|destroy]() Logan Gunthorpe
2018-04-23 23:30 ` [PATCH v4 09/14] nvme-pci: Use PCI p2pmem subsystem to manage the CMB Logan Gunthorpe
2018-04-23 23:30 ` [PATCH v4 10/14] nvme-pci: Add support for P2P memory in requests Logan Gunthorpe
2018-04-23 23:30 ` [PATCH v4 11/14] nvme-pci: Add a quirk for a pseudo CMB Logan Gunthorpe
2018-04-23 23:30 ` [PATCH v4 12/14] nvmet: Introduce helper functions to allocate and free request SGLs Logan Gunthorpe
2018-04-23 23:30 ` [PATCH v4 13/14] nvmet-rdma: Use new SGL alloc/free helper for requests Logan Gunthorpe
2018-04-23 23:30 ` [PATCH v4 14/14] nvmet: Optionally use PCI P2P memory Logan Gunthorpe
2018-05-02 11:51 ` [PATCH v4 00/14] Copy Offload in NVMe Fabrics with P2P PCI Memory Christian König
2018-05-02 15:56 ` Logan Gunthorpe
2018-05-03 9:05 ` Christian König
2018-05-03 15:59 ` Logan Gunthorpe
2018-05-03 17:29 ` Christian König
2018-05-03 18:43 ` Logan Gunthorpe
2018-05-04 14:27 ` Christian König
2018-05-04 15:52 ` Logan Gunthorpe
2018-05-07 23:23 ` Bjorn Helgaas
2018-05-07 23:34 ` Logan Gunthorpe
2018-05-08 16:57 ` Alex Williamson
2018-05-08 19:14 ` Logan Gunthorpe
2018-05-08 21:25 ` Don Dutile
2018-05-08 21:40 ` Alex Williamson [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20180508154039.5c85a8f8@w520.home \
--to=alex.williamson@redhat.com \
--cc=axboe@kernel.dk \
--cc=benh@kernel.crashing.org \
--cc=bhelgaas@google.com \
--cc=christian.koenig@amd.com \
--cc=ddutile@redhat.com \
--cc=hch@lst.de \
--cc=helgaas@kernel.org \
--cc=jgg@mellanox.com \
--cc=jglisse@redhat.com \
--cc=keith.busch@intel.com \
--cc=linux-block@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nvdimm@lists.01.org \
--cc=linux-nvme@lists.infradead.org \
--cc=linux-pci@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=maxg@mellanox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).