From: Chaitanya Kulkarni <chaitanyak@nvidia.com>
To: Logan Gunthorpe <logang@deltatee.com>
Cc: "linux-mm@kvack.org" <linux-mm@kvack.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
"linux-nvme@lists.infradead.org" <linux-nvme@lists.infradead.org>,
"linux-block@vger.kernel.org" <linux-block@vger.kernel.org>,
"iommu@lists.linux-foundation.org"
<iommu@lists.linux-foundation.org>
Subject: Re: [PATCH v6 00/21] Userspace P2PDMA with O_DIRECT NVMe devices
Date: Mon, 16 May 2022 22:31:59 +0000 [thread overview]
Message-ID: <32d5901d-f7d3-0701-0b72-1493897b025e@nvidia.com> (raw)
In-Reply-To: <20220407154717.7695-1-logang@deltatee.com>
On 4/7/22 08:46, Logan Gunthorpe wrote:
> Hi,
>
> This patchset continues my work to add userspace P2PDMA access using
> O_DIRECT NVMe devices. This posting contains some minor fixes and a
> rebase onto v5.18-rc1 which contains cleanup from Christoph around
> free_zone_device_page() that helps to enable this patchset. The
> previous posting was here[1].
>
> The patchset enables userspace P2PDMA by allowing userspace to mmap()
> allocated chunks of the CMB. The resulting VMA can be passed only
> to O_DIRECT IO on NVMe backed files or block devices. A flag is added
> to GUP() in Patch <>, then Patches <> through <> wire this flag up based
> on whether the block queue indicates P2PDMA support. Patches <>
> through <> enable the CMB to be mapped into userspace by mmaping
> the nvme char device.
>
> This is relatively straightforward, however the one significant
> problem is that, presently, pci_p2pdma_map_sg() requires a homogeneous
> SGL with all P2PDMA pages or all regular pages. Enhancing GUP to
> support enforcing this rule would require a huge hack that I don't
> expect would be all that pallatable. So the first 13 patches add
> support for P2PDMA pages to dma_map_sg[table]() to the dma-direct
> and dma-iommu implementations. Thus systems without an IOMMU plus
> Intel and AMD IOMMUs are supported. (Other IOMMU implementations would
> then be unsupported, notably ARM and PowerPC but support would be added
> when they convert to dma-iommu).
>
> dma_map_sgtable() is preferred when dealing with P2PDMA memory as it
> will return -EREMOTEIO when the DMA device cannot map specific P2PDMA
> pages based on the existing rules in calc_map_type_and_dist().
>
> The other issue is dma_unmap_sg() needs a flag to determine whether a
> given dma_addr_t was mapped regularly or as a PCI bus address. To allow
> this, a third flag is added to the page_link field in struct
> scatterlist. This effectively means support for P2PDMA will now depend
> on CONFIG_64BIT.
>
> Feedback welcome.
>
> This series is based on v5.18-rc1. A git branch is available here:
>
> https://github.com/sbates130272/linux-p2pmem/ p2pdma_user_cmb_v6
>
> Thanks,
>
> Logan
>
> [1] lkml.kernel.org/r/20220128002614.6136-1-logang@deltatee.com
>
> --
Do you have any plans to re-spin this ?
-ck
next prev parent reply other threads:[~2022-05-16 22:32 UTC|newest]
Thread overview: 36+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-04-07 15:46 [PATCH v6 00/21] Userspace P2PDMA with O_DIRECT NVMe devices Logan Gunthorpe
2022-04-07 15:46 ` [PATCH v6 01/21] lib/scatterlist: add flag for indicating P2PDMA segments in an SGL Logan Gunthorpe
2022-04-07 15:46 ` [PATCH v6 02/21] PCI/P2PDMA: Attempt to set map_type if it has not been set Logan Gunthorpe
2022-04-07 15:46 ` [PATCH v6 03/21] PCI/P2PDMA: Expose pci_p2pdma_map_type() Logan Gunthorpe
2022-04-07 15:47 ` [PATCH v6 04/21] PCI/P2PDMA: Introduce helpers for dma_map_sg implementations Logan Gunthorpe
2022-04-07 15:47 ` [PATCH v6 05/21] dma-mapping: allow EREMOTEIO return code for P2PDMA transfers Logan Gunthorpe
2022-04-07 15:47 ` [PATCH v6 06/21] dma-direct: support PCI P2PDMA pages in dma-direct map_sg Logan Gunthorpe
2022-04-07 15:47 ` [PATCH v6 07/21] dma-mapping: add flags to dma_map_ops to indicate PCI P2PDMA support Logan Gunthorpe
2022-04-07 15:47 ` [PATCH v6 08/21] iommu/dma: support PCI P2PDMA pages in dma-iommu map_sg Logan Gunthorpe
2022-04-07 15:47 ` [PATCH v6 09/21] nvme-pci: check DMA ops when indicating support for PCI P2PDMA Logan Gunthorpe
2022-04-07 15:47 ` [PATCH v6 10/21] nvme-pci: convert to using dma_map_sgtable() Logan Gunthorpe
2022-04-07 15:47 ` [PATCH v6 11/21] RDMA/core: introduce ib_dma_pci_p2p_dma_supported() Logan Gunthorpe
2022-04-07 15:47 ` [PATCH v6 12/21] RDMA/rw: drop pci_p2pdma_[un]map_sg() Logan Gunthorpe
2022-04-07 15:47 ` [PATCH v6 13/21] PCI/P2PDMA: Remove pci_p2pdma_[un]map_sg() Logan Gunthorpe
2022-04-07 15:47 ` [PATCH v6 14/21] mm: introduce FOLL_PCI_P2PDMA to gate getting PCI P2PDMA pages Logan Gunthorpe
2022-04-07 15:47 ` [PATCH v6 15/21] iov_iter: introduce iov_iter_get_pages_[alloc_]flags() Logan Gunthorpe
2022-04-07 15:47 ` [PATCH v6 16/21] block: add check when merging zone device pages Logan Gunthorpe
2022-04-07 15:47 ` [PATCH v6 17/21] lib/scatterlist: " Logan Gunthorpe
2022-04-07 15:47 ` [PATCH v6 18/21] block: set FOLL_PCI_P2PDMA in __bio_iov_iter_get_pages() Logan Gunthorpe
2022-04-07 15:47 ` [PATCH v6 19/21] block: set FOLL_PCI_P2PDMA in bio_map_user_iov() Logan Gunthorpe
2022-04-07 15:47 ` [PATCH v6 20/21] PCI/P2PDMA: Introduce pci_mmap_p2pmem() Logan Gunthorpe
2022-05-27 12:55 ` Jason Gunthorpe
2022-05-27 15:35 ` Logan Gunthorpe
2022-05-27 19:03 ` Jason Gunthorpe
2022-05-27 22:41 ` Logan Gunthorpe
2022-06-02 0:00 ` Jason Gunthorpe
2022-06-02 16:16 ` Logan Gunthorpe
2022-06-02 16:30 ` Jason Gunthorpe
2022-06-02 16:45 ` Logan Gunthorpe
2022-06-02 17:18 ` Jason Gunthorpe
2022-06-02 16:49 ` Logan Gunthorpe
2022-06-02 17:28 ` Jason Gunthorpe
2022-06-02 17:49 ` Logan Gunthorpe
2022-04-07 15:47 ` [PATCH v6 21/21] nvme-pci: allow mmaping the CMB in userspace Logan Gunthorpe
2022-05-16 22:31 ` Chaitanya Kulkarni [this message]
2022-05-16 22:35 ` [PATCH v6 00/21] Userspace P2PDMA with O_DIRECT NVMe devices Logan Gunthorpe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=32d5901d-f7d3-0701-0b72-1493897b025e@nvidia.com \
--to=chaitanyak@nvidia.com \
--cc=iommu@lists.linux-foundation.org \
--cc=linux-block@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=linux-nvme@lists.infradead.org \
--cc=linux-pci@vger.kernel.org \
--cc=logang@deltatee.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).