linux-nvme.lists.infradead.org archive mirror
 help / color / mirror / Atom feed
From: Logan Gunthorpe <logang@deltatee.com>
To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org,
	linux-block@vger.kernel.org, linux-pci@vger.kernel.org,
	linux-mm@kvack.org, iommu@lists.linux-foundation.org
Cc: "Stephen Bates" <sbates@raithlin.com>,
	"Christoph Hellwig" <hch@lst.de>,
	"Dan Williams" <dan.j.williams@intel.com>,
	"Jason Gunthorpe" <jgg@ziepe.ca>,
	"Christian König" <christian.koenig@amd.com>,
	"Ira Weiny" <iweiny@intel.com>,
	"John Hubbard" <jhubbard@nvidia.com>,
	"Don Dutile" <ddutile@redhat.com>,
	"Matthew Wilcox" <willy@infradead.org>,
	"Daniel Vetter" <daniel.vetter@ffwll.ch>,
	"Jakowski Andrzej" <andrzej.jakowski@intel.com>,
	"Minturn Dave B" <dave.b.minturn@intel.com>,
	"Jason Ekstrand" <jason@jlekstrand.net>,
	"Dave Hansen" <dave.hansen@linux.intel.com>,
	"Xiong Jianxin" <jianxin.xiong@intel.com>,
	"Logan Gunthorpe" <logang@deltatee.com>
Subject: [RFC PATCH v2 00/11] Add support to dma_map_sg for P2PDMA
Date: Thu, 11 Mar 2021 16:31:30 -0700	[thread overview]
Message-ID: <20210311233142.7900-1-logang@deltatee.com> (raw)

Hi,

This is a rework of the first half of my RFC for doing P2PDMA in userspace
with O_DIRECT[1].

The largest issue with that series was the gross way of flagging P2PDMA
SGL segments. This RFC proposes a different approach, (suggested by
Dan Williams[2]) which uses the third bit in the page_link field of the
SGL.

This approach is a lot less hacky but comes at the cost of adding a
CONFIG_64BIT dependency to CONFIG_PCI_P2PDMA and using up the last
scarce bit in the page_link. For our purposes, a 64BIT restriction is
acceptable but it's not clear if this is ok for all usecases hoping
to make use of P2PDMA.

Matthew Wilcox has already suggested (off-list) that this is the wrong
approach, preferring a new dma mapping operation and an SGL replacement. I
don't disagree that something along those lines would be a better long
term solution, but it involves overcoming a lot of challenges to get
there. Creating a new mapping operation still means adding support to more
than 25 dma_map_ops implementations (many of which are on obscure
architectures) or creating a redundant path to fallback with dma_map_sg()
for every driver that uses the new operation. This RFC is an approach
that doesn't require overcoming these blocks.

Any alternative ideas or feedback is welcome.

These patches are based on v5.12-rc2 and a git branch is available here:

  https://github.com/sbates130272/linux-p2pmem/  p2pdma_dma_map_ops_rfc

A branch with the patches from the previous RFC that add userspace
O_DIRECT support is available at the same URL with the name
"p2pdma_dma_map_ops_rfc+user" (however, none of the issues with those
extra patches from the feedback of the last posting have been fixed).

Thanks,

Logan

[1] https://lore.kernel.org/linux-block/20201106170036.18713-1-logang@deltatee.com/
[2] https://lore.kernel.org/linux-block/CAPcyv4ifGcrdOtUt8qr7pmFhmecGHqGVre9G0RorGczCGVECQQ@mail.gmail.com/

--

Logan Gunthorpe (11):
  PCI/P2PDMA: Pass gfp_mask flags to upstream_bridge_distance_warn()
  PCI/P2PDMA: Avoid pci_get_slot() which sleeps
  PCI/P2PDMA: Attempt to set map_type if it has not been set
  PCI/P2PDMA: Introduce pci_p2pdma_should_map_bus() and
    pci_p2pdma_bus_offset()
  lib/scatterlist: Add flag for indicating P2PDMA segments in an SGL
  dma-direct: Support PCI P2PDMA pages in dma-direct map_sg
  dma-mapping: Add flags to dma_map_ops to indicate PCI P2PDMA support
  iommu/dma: Support PCI P2PDMA pages in dma-iommu map_sg
  block: Add BLK_STS_P2PDMA
  nvme-pci: Check DMA ops when indicating support for PCI P2PDMA
  nvme-pci: Convert to using dma_map_sg for p2pdma pages

 block/blk-core.c            |  2 +
 drivers/iommu/dma-iommu.c   | 63 +++++++++++++++++++++-----
 drivers/nvme/host/core.c    |  3 +-
 drivers/nvme/host/nvme.h    |  2 +-
 drivers/nvme/host/pci.c     | 38 +++++++---------
 drivers/pci/Kconfig         |  2 +-
 drivers/pci/p2pdma.c        | 89 +++++++++++++++++++++++++++++++------
 include/linux/blk_types.h   |  7 +++
 include/linux/dma-map-ops.h |  3 ++
 include/linux/dma-mapping.h |  5 +++
 include/linux/pci-p2pdma.h  | 11 +++++
 include/linux/scatterlist.h | 49 ++++++++++++++++++--
 kernel/dma/direct.c         | 35 +++++++++++++--
 kernel/dma/mapping.c        | 21 +++++++--
 14 files changed, 271 insertions(+), 59 deletions(-)


base-commit: a38fd8748464831584a19438cbb3082b5a2dab15
--
2.20.1

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

             reply	other threads:[~2021-03-11 23:37 UTC|newest]

Thread overview: 47+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-03-11 23:31 Logan Gunthorpe [this message]
2021-03-11 23:31 ` [RFC PATCH v2 01/11] PCI/P2PDMA: Pass gfp_mask flags to upstream_bridge_distance_warn() Logan Gunthorpe
2021-03-12 20:39   ` Bjorn Helgaas
2021-03-12 20:53     ` Logan Gunthorpe
2021-03-11 23:31 ` [RFC PATCH v2 02/11] PCI/P2PDMA: Avoid pci_get_slot() which sleeps Logan Gunthorpe
2021-03-12 20:57   ` Bjorn Helgaas
2021-03-12 21:37     ` Logan Gunthorpe
2021-03-11 23:31 ` [RFC PATCH v2 03/11] PCI/P2PDMA: Attempt to set map_type if it has not been set Logan Gunthorpe
2021-03-11 23:31 ` [RFC PATCH v2 04/11] PCI/P2PDMA: Introduce pci_p2pdma_should_map_bus() and pci_p2pdma_bus_offset() Logan Gunthorpe
2021-03-13  1:38   ` Ira Weiny
2021-03-15 16:27     ` Logan Gunthorpe
2021-03-24 17:21       ` Jason Gunthorpe
2021-03-24 18:34         ` Christian König
2021-03-13  2:32   ` Ira Weiny
2021-03-15 16:30     ` Logan Gunthorpe
2021-03-16  8:14   ` Christoph Hellwig
2021-03-11 23:31 ` [RFC PATCH v2 05/11] lib/scatterlist: Add flag for indicating P2PDMA segments in an SGL Logan Gunthorpe
2021-03-11 23:31 ` [RFC PATCH v2 06/11] dma-direct: Support PCI P2PDMA pages in dma-direct map_sg Logan Gunthorpe
2021-03-12 15:52   ` Robin Murphy
2021-03-12 16:24     ` Logan Gunthorpe
2021-03-12 18:11       ` Robin Murphy
2021-03-12 18:27         ` Logan Gunthorpe
2021-03-16  7:58           ` Christoph Hellwig
2021-03-16 15:54             ` Logan Gunthorpe
2021-03-16  7:56         ` Christoph Hellwig
2021-03-16  8:11   ` Christoph Hellwig
2021-03-11 23:31 ` [RFC PATCH v2 07/11] dma-mapping: Add flags to dma_map_ops to indicate PCI P2PDMA support Logan Gunthorpe
2021-03-13  2:36   ` Ira Weiny
2021-03-15 16:33     ` Logan Gunthorpe
2021-03-16  8:00       ` Christoph Hellwig
2021-03-16  8:15   ` Christoph Hellwig
2021-03-11 23:31 ` [RFC PATCH v2 08/11] iommu/dma: Support PCI P2PDMA pages in dma-iommu map_sg Logan Gunthorpe
2021-03-12 15:52   ` Robin Murphy
2021-03-12 17:03     ` Logan Gunthorpe
2021-03-12 19:47       ` Robin Murphy
2021-03-12 20:06         ` Logan Gunthorpe
2021-03-11 23:31 ` [RFC PATCH v2 09/11] block: Add BLK_STS_P2PDMA Logan Gunthorpe
2021-03-16  8:00   ` Christoph Hellwig
2021-03-16 16:02     ` Logan Gunthorpe
2021-03-11 23:31 ` [RFC PATCH v2 10/11] nvme-pci: Check DMA ops when indicating support for PCI P2PDMA Logan Gunthorpe
2021-03-11 23:31 ` [RFC PATCH v2 11/11] nvme-pci: Convert to using dma_map_sg for p2pdma pages Logan Gunthorpe
2021-03-11 23:59   ` Jason Gunthorpe
2021-03-12  1:37     ` Logan Gunthorpe
2021-03-12 15:51 ` [RFC PATCH v2 00/11] Add support to dma_map_sg for P2PDMA Robin Murphy
2021-03-12 16:18   ` Logan Gunthorpe
2021-03-12 17:46     ` Robin Murphy
2021-03-12 18:24       ` Logan Gunthorpe

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210311233142.7900-1-logang@deltatee.com \
    --to=logang@deltatee.com \
    --cc=andrzej.jakowski@intel.com \
    --cc=christian.koenig@amd.com \
    --cc=dan.j.williams@intel.com \
    --cc=daniel.vetter@ffwll.ch \
    --cc=dave.b.minturn@intel.com \
    --cc=dave.hansen@linux.intel.com \
    --cc=ddutile@redhat.com \
    --cc=hch@lst.de \
    --cc=iommu@lists.linux-foundation.org \
    --cc=iweiny@intel.com \
    --cc=jason@jlekstrand.net \
    --cc=jgg@ziepe.ca \
    --cc=jhubbard@nvidia.com \
    --cc=jianxin.xiong@intel.com \
    --cc=linux-block@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=linux-nvme@lists.infradead.org \
    --cc=linux-pci@vger.kernel.org \
    --cc=sbates@raithlin.com \
    --cc=willy@infradead.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).