Re: [PATCH v4 0/1] Use HMM for ODP v4

From: Jerome Glisse <jglisse@redhat.com>
To: Jason Gunthorpe <jgg@ziepe.ca>
Cc: linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org,
	Leon Romanovsky <leonro@mellanox.com>,
	Doug Ledford <dledford@redhat.com>,
	Artemy Kovalyov <artemyko@mellanox.com>,
	Moni Shoua <monis@mellanox.com>,
	Mike Marciniszyn <mike.marciniszyn@intel.com>,
	Kaike Wan <kaike.wan@intel.com>,
	Dennis Dalessandro <dennis.dalessandro@intel.com>
Subject: Re: [PATCH v4 0/1] Use HMM for ODP v4
Date: Thu, 23 May 2019 11:52:08 -0400	[thread overview]
Message-ID: <20190523155207.GC5104@redhat.com> (raw)
In-Reply-To: <20190523154149.GB12159@ziepe.ca>

On Thu, May 23, 2019 at 12:41:49PM -0300, Jason Gunthorpe wrote:
> On Thu, May 23, 2019 at 11:04:32AM -0400, Jerome Glisse wrote:
> > On Wed, May 22, 2019 at 08:57:37PM -0300, Jason Gunthorpe wrote:
> > > On Wed, May 22, 2019 at 01:48:52PM -0400, Jerome Glisse wrote:
> > > 
> > > > > > So attached is a rebase on top of 5.2-rc1, i have tested with pingpong
> > > > > > (prefetch and not and different sizes). Seems to work ok.
> > > > > 
> > > > > Urk, it already doesn't apply to the rdma tree :(
> > > > > 
> > > > > The conflicts are a little more extensive than I'd prefer to handle..
> > > > > Can I ask you to rebase it on top of this branch please:
> > > > > 
> > > > > https://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma.git/log/?h=wip/jgg-for-next
> > > > > 
> > > > > Specifically it conflicts with this patch:
> > > > > 
> > > > > https://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma.git/commit/?h=wip/jgg-for-next&id=d2183c6f1958e6b6dfdde279f4cee04280710e34
> > > 
> > > There is at least one more serious blocker here:
> > > 
> > > config ARCH_HAS_HMM_MIRROR
> > >         bool
> > >         default y
> > >         depends on (X86_64 || PPC64)
> > >         depends on MMU && 64BIT
> > > 
> > > I can't loose ARM64 support for ODP by merging this, that is too
> > > serious of a regression.
> > > 
> > > Can you fix it?
> > 
> > 5.2 already has patch to fix the Kconfig (ARCH_HAS_HMM_MIRROR and
> > ARCH_HAS_HMM_DEVICE replacing ARCH_HAS_HMM) I need to update nouveau
> 
> Newer than 5.2-rc1? Is this why ARCH_HAS_HMM_MIRROR is not used anywhere?

Yes this is multi-step update, first add the new Kconfig release n,
update driver in release n+1, update core Kconfig in release n+2

So we are in release n (5.2), in 5.3 i will update nouveau and amdgpu
so that in 5.4 in ca remove the old ARCH_HAS_HMM

> > in 5.3 so that i can drop the old ARCH_HAS_HMM and then convert
> > core mm in 5.4 to use ARCH_HAS_HMM_MIRROR and ARCH_HAS_HMM_DEVICE
> > instead of ARCH_HAS_HMM
> 
> My problem is that ODP needs HMM_MIRROR which needs HMM & ARCH_HAS_HMM
> - and then even if fixed we still have the ARCH_HAS_HMM_MIRROR
> restricted to ARM64..
> 
> Can we broaden HMM_MIRROR to all arches? I would very much prefer
> that.

Ignore ARCH_HAS_HMM it will be remove in 5.4, all that will matter
for ODP is ARCH_HAS_HMM_MIRROR which should be enabled for ARM64 as
ARM64 has everything needed for that. I just did not add ARM64 to
ARCH_HAS_HMM_MIRROR because i did not had hardware to test it on.

So in 5.3 i will update nouveau and amdgpu to use ARCH_HAS_HMM_DEVICE
and ARCH_HAS_HMM_MIRROR. In 5.4 i will update mm/Kconig to remove
ARCH_HAS_HMM

> 
> > So it seems it will have to wait 5.4 for ODP. I will re-spin the
> > patch for ODP once i am done reviewing Ralph changes and yours
> > for 5.3.
> 
> I think we are still OK for 5.3.

I can not update mm/Kconfig in 5.3 so any Kconfig update will be
5.4

> 
> If mm takes the fixup patches so hmm mirror is as reliable as ODP's
> existing stuff, and patch from you to enable ARM64, then we can
> continue to merge into 5.3
> 
> So, let us try to get acks on those other threads..

I will be merging your patchset and Ralph and repost, they are only
minor change mostly that you can not update the driver API in just
one release. First add the new API in release n, then replace old
API usage in release n+1, then remove old API in n+2.

Cheers,
Jérôme