Re: [PATCH 1/6] mmput: use notifier chain to call subsystem exit handler.

["Gabbay, Oded" <Oded.Gabbay@amd.com>]

On Tue, 2014-07-01 at 11:15 +0200, Joerg Roedel wrote:
> On Mon, Jun 30, 2014 at 02:35:57PM -0400, Jerome Glisse wrote:
> > We do intend to tear down all secondary mapping inside the relase
> > callback but still we can not cleanup all the resources associated
> > with it.
> >
> 
> And why can't you cleanup the other resources in the file close path?
> Tearing down the mappings is all you need to do in the release function
> anyway.
> 
> > As said from the release call back you can not call
> > mmu_notifier_unregister and thus you can not fully cleanup things.
> 
> You don't need to call mmu_notifier_unregister when the release function
> is already running from exit_mmap because this is equivalent to calling
> mmu_notifier_unregister.
> 
> > Only way to achieve so is to do it ouside mmu_notifier callback.
> 
> The resources that can't be handled there can be cleaned up in the
> file-close path. No need for a new notifier in mm code.
> 
> In the end all you need to do in the release function is to tear down
> the secondary mapping and make sure the device can no longer access the
> address space when the release function returns. Everything else, like
> freeing any resources can be done later when the file descriptors are
> teared down.

I will answer from the KFD perpective, as I'm AMD's maintainer of this
driver.

Little background: AMD's HSA Linux kernel driver (called radeon_kfd or
KFD in short), has been developed for the past year by AMD, to support
running Linux compute applications on AMD's HSA-enabled APUs, i.e Kaveri
(A10-7850K/7700K). The driver will be up for kernel community review in
about 2-3 weeks so we could push it during the 3.17 merge window. Prior
discussions were made with gpu/drm subsystem maintainers about this
driver.

In the KFD, we need to maintain a notion of each compute process.
Therefore, we have an object called "kfd_process" that is created for
each process that uses the KFD. Naturally, we need to be able to track
the process's shutdown in order to perform cleanup of the resources it
uses (compute queues, virtual address space, gpu local memory
allocations, etc.).

To enable this tracking mechanism, we decided to associate the
kfd_process with mm_struct to ensure that a kfd_process object has
exactly the same lifespan as the process it represents. We preferred to
use the mm_struct and not a file description because using a file
descriptor to track “process” shutdown is wrong in two ways:

* Technical: file descriptors can be passed to unrelated processes using
AF_UNIX sockets. This means that a process can exit while the file stays
open. Even if we implement this “correctly” i.e. holding the address
space & page tables alive until the file is finally released, it’s
really dodgy.

* Philosophical: our ioctls are actually system calls in disguise. They
operate on the process, not on a device.

Moreover, because the GPU interacts with the process only through
virtual memory (and not e.g. file descriptors), and because virtual
address space is fundamental to an intuitive notion of what a process
is, the decision to associate the kfd_process with mm_struct seems like
a natural choice.

Then arrived the issue of how the KFD is notified about an mm_struct
destruction. Because the mmu_notifier release callback is called from an
RCU read lock, it can't destory the mmu_notifier object, which is the
kfd_process object itself. Therefore, I talked to Jerome and Andrew
Morton on a way to implement this and after the discussion (which was in
private emails), Jerome was kind enough to write a patch, which is the
patch we are now discussing.

You are more than welcomed to take a look at the entire driver, at
http://cgit.freedesktop.org/~gabbayo/linux/?h=kfd-0.6.x      although
the driver will undergo some changes before sending the pull request to
Dave Airle.

I believe that converting amd_iommu_v2 driver to use this patch as well,
will benefit all parties. AFAIK, KFD is the _only_ client of the
amd_iommu_v2 driver, so it is imperative that we will work together on
this.

	Oded
> > If you know any other way to call mmu_notifier_unregister before the
> > end of mmput function than i am all ear. I am not adding this call
> > back just for the fun of it i spend serious time trying to find a
> > way to do thing without it. I might have miss a way so if i did please
> > show it to me.
> 
> Why do you need to call mmu_notifier_unregister manually when it is done
> implicitly in exit_mmap already? 
> 
> 
> 	Joerg
> 
>