From: Jerome Glisse <jglisse@redhat.com>
To: Leon Romanovsky <leonro@mellanox.com>
Cc: "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
Jason Gunthorpe <jgg@mellanox.com>,
Doug Ledford <dledford@redhat.com>,
Artemy Kovalyov <artemyko@mellanox.com>,
Moni Shoua <monis@mellanox.com>,
Mike Marciniszyn <mike.marciniszyn@intel.com>,
Kaike Wan <kaike.wan@intel.com>,
Dennis Dalessandro <dennis.dalessandro@intel.com>
Subject: Re: [PATCH v3 0/1] Use HMM for ODP v3
Date: Thu, 11 Apr 2019 13:25:15 -0400 [thread overview]
Message-ID: <20190411172515.GC4266@redhat.com> (raw)
In-Reply-To: <20190411122940.GZ3201@mtr-leonro.mtl.com>
On Thu, Apr 11, 2019 at 12:29:43PM +0000, Leon Romanovsky wrote:
> On Wed, Apr 10, 2019 at 11:41:24AM -0400, jglisse@redhat.com wrote:
> > From: Jérôme Glisse <jglisse@redhat.com>
> >
> > Changes since v1/v2 are about rebase and better comments in the code.
> > Previous cover letter slightly updated.
> >
> >
> > This patchset convert RDMA ODP to use HMM underneath this is motivated
> > by stronger code sharing for same feature (share virtual memory SVM or
> > Share Virtual Address SVA) and also stronger integration with mm code to
> > achieve that. It depends on HMM patchset posted for inclusion in 5.2 [2]
> > and [3].
> >
> > It has been tested with pingpong test with -o and others flags to test
> > different size/features associated with ODP.
> >
> > Moreover they are some features of HMM in the works like peer to peer
> > support, fast CPU page table snapshot, fast IOMMU mapping update ...
> > It will be easier for RDMA devices with ODP to leverage those if they
> > use HMM underneath.
> >
> > Quick summary of what HMM is:
> > HMM is a toolbox for device driver to implement software support for
> > Share Virtual Memory (SVM). Not only it provides helpers to mirror a
> > process address space on a device (hmm_mirror). It also provides
> > helper to allow to use device memory to back regular valid virtual
> > address of a process (any valid mmap that is not an mmap of a device
> > or a DAX mapping). They are two kinds of device memory. Private memory
> > that is not accessible to CPU because it does not have all the expected
> > properties (this is for all PCIE devices) or public memory which can
> > also be access by CPU without restriction (with OpenCAPI or CCIX or
> > similar cache-coherent and atomic inter-connect).
> >
> > Device driver can use each of HMM tools separatly. You do not have to
> > use all the tools it provides.
> >
> > For RDMA device i do not expect a need to use the device memory support
> > of HMM. This device memory support is geared toward accelerator like GPU.
> >
> >
> > You can find a branch [1] with all the prerequisite in. This patch is on
> > top of rdma-next with the HMM patchset [2] and mmu notifier patchset [3]
> > applied on top of it.
> >
> > [1] https://cgit.freedesktop.org/~glisse/linux/log/?h=rdma-5.2
>
> Hi Jerome,
>
> I took this branch and merged with our latest rdma-next, but it doesn't
> compile.
>
> In file included from drivers/infiniband/hw/mlx5/mem.c:35:
> ./include/rdma/ib_umem_odp.h:110:20: error: field _mirror_ has
> incomplete type
> struct hmm_mirror mirror;
> ^~~~~~
> ./include/rdma/ib_umem_odp.h:132:18: warning: _struct hmm_range_ declared inside parameter list will not be visible outside of this definition or declaration
> struct hmm_range *range);
> ^~~~~~~~~
> make[4]: *** [scripts/Makefile.build:276: drivers/infiniband/hw/mlx5/mem.o] Error 1
>
> The reason to it that in my .config, ZONE_DEVICE, MEMORY_HOTPLUG and HMM options were disabled.
Silly my i forgot to update kconfig so i pushed a branch with
proper kconfig changes in the ODP patch but it depends on changes
to the HMM kconfig so that HMM_MIRROR can be enabled on arch that
do not have everything for HMM_DEVICE.
https://cgit.freedesktop.org/~glisse/linux/log/?h=rdma-odp-hmm-v4
I doing build of various kconfig variation before posting to make
sure it is all good.
Cheers,
Jérôme
prev parent reply other threads:[~2019-04-11 17:25 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-04-10 15:41 [PATCH v3 0/1] Use HMM for ODP v3 jglisse
2019-04-10 15:41 ` [PATCH v3 1/1] RDMA/odp: convert to use " jglisse
2019-04-11 12:29 ` [PATCH v3 0/1] Use " Leon Romanovsky
2019-04-11 17:25 ` Jerome Glisse [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190411172515.GC4266@redhat.com \
--to=jglisse@redhat.com \
--cc=artemyko@mellanox.com \
--cc=dennis.dalessandro@intel.com \
--cc=dledford@redhat.com \
--cc=jgg@mellanox.com \
--cc=kaike.wan@intel.com \
--cc=leonro@mellanox.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=mike.marciniszyn@intel.com \
--cc=monis@mellanox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).