From: Felix Kuehling <felix.kuehling@amd.com>
To: "Michal Hocko" <mhocko@kernel.org>,
"Christian König" <christian.koenig@amd.com>
Cc: LKML <linux-kernel@vger.kernel.org>,
"David (ChunMing) Zhou" <David1.Zhou@amd.com>,
"Paolo Bonzini" <pbonzini@redhat.com>,
"Radim Krčmář" <rkrcmar@redhat.com>,
"Alex Deucher" <alexander.deucher@amd.com>,
"David Airlie" <airlied@linux.ie>,
"Jani Nikula" <jani.nikula@linux.intel.com>,
"Joonas Lahtinen" <joonas.lahtinen@linux.intel.com>,
"Rodrigo Vivi" <rodrigo.vivi@intel.com>,
"Doug Ledford" <dledford@redhat.com>,
"Jason Gunthorpe" <jgg@ziepe.ca>,
"Mike Marciniszyn" <mike.marciniszyn@intel.com>,
"Dennis Dalessandro" <dennis.dalessandro@intel.com>,
"Sudeep Dutt" <sudeep.dutt@intel.com>,
"Ashutosh Dixit" <ashutosh.dixit@intel.com>,
"Dimitri Sivanich" <sivanich@sgi.com>,
"Boris Ostrovsky" <boris.ostrovsky@oracle.com>,
"Juergen Gross" <jgross@suse.com>,
"Jérôme Glisse" <jglisse@redhat.com>,
"Andrea Arcangeli" <aarcange@redhat.com>,
kvm@vger.kernel.org, amd-gfx@lists.freedesktop.org,
dri-devel@lists.freedesktop.org, intel-gfx@lists.freedesktop.org,
linux-rdma@vger.kernel.org, xen-devel@lists.xenproject.org,
linux-mm@kvack.org, "David Rientjes" <rientjes@google.com>
Subject: Re: [RFC PATCH] mm, oom: distinguish blockable mode for mmu notifiers
Date: Fri, 22 Jun 2018 16:09:06 -0400 [thread overview]
Message-ID: <dd260800-6457-f3ff-47df-b65ef258f4b7@amd.com> (raw)
In-Reply-To: <20180622152444.GC10465@dhcp22.suse.cz>
On 2018-06-22 11:24 AM, Michal Hocko wrote:
> On Fri 22-06-18 17:13:02, Christian König wrote:
>> Hi Michal,
>>
>> [Adding Felix as well]
>>
>> Well first of all you have a misconception why at least the AMD graphics
>> driver need to be able to sleep in an MMU notifier: We need to sleep because
>> we need to wait for hardware operations to finish and *NOT* because we need
>> to wait for locks.
>>
>> I'm not sure if your flag now means that you generally can't sleep in MMU
>> notifiers any more, but if that's the case at least AMD hardware will break
>> badly. In our case the approach of waiting for a short time for the process
>> to be reaped and then select another victim actually sounds like the right
>> thing to do.
> Well, I do not need to make the notifier code non blocking all the time.
> All I need is to ensure that it won't sleep if the flag says so and
> return -EAGAIN instead.
>
> So here is what I do for amdgpu:
In the case of KFD we also need to take the DQM lock:
amdgpu_mn_invalidate_range_start_hsa -> amdgpu_amdkfd_evict_userptr ->
kgd2kfd_quiesce_mm -> kfd_process_evict_queues -> evict_process_queues_cpsch
So we'd need to pass the blockable parameter all the way through that
call chain.
Regards,
Felix
>
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c
>>> index 83e344fbb50a..d138a526feff 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c
>>> @@ -136,12 +136,18 @@ void amdgpu_mn_unlock(struct amdgpu_mn *mn)
>>> *
>>> * Take the rmn read side lock.
>>> */
>>> -static void amdgpu_mn_read_lock(struct amdgpu_mn *rmn)
>>> +static int amdgpu_mn_read_lock(struct amdgpu_mn *rmn, bool blockable)
>>> {
>>> - mutex_lock(&rmn->read_lock);
>>> + if (blockable)
>>> + mutex_lock(&rmn->read_lock);
>>> + else if (!mutex_trylock(&rmn->read_lock))
>>> + return -EAGAIN;
>>> +
>>> if (atomic_inc_return(&rmn->recursion) == 1)
>>> down_read_non_owner(&rmn->lock);
>>> mutex_unlock(&rmn->read_lock);
>>> +
>>> + return 0;
>>> }
>>> /**
>>> @@ -197,10 +203,11 @@ static void amdgpu_mn_invalidate_node(struct amdgpu_mn_node *node,
>>> * We block for all BOs between start and end to be idle and
>>> * unmap them by move them into system domain again.
>>> */
>>> -static void amdgpu_mn_invalidate_range_start_gfx(struct mmu_notifier *mn,
>>> +static int amdgpu_mn_invalidate_range_start_gfx(struct mmu_notifier *mn,
>>> struct mm_struct *mm,
>>> unsigned long start,
>>> - unsigned long end)
>>> + unsigned long end,
>>> + bool blockable)
>>> {
>>> struct amdgpu_mn *rmn = container_of(mn, struct amdgpu_mn, mn);
>>> struct interval_tree_node *it;
>>> @@ -208,7 +215,11 @@ static void amdgpu_mn_invalidate_range_start_gfx(struct mmu_notifier *mn,
>>> /* notification is exclusive, but interval is inclusive */
>>> end -= 1;
>>> - amdgpu_mn_read_lock(rmn);
>>> + /* TODO we should be able to split locking for interval tree and
>>> + * amdgpu_mn_invalidate_node
>>> + */
>>> + if (amdgpu_mn_read_lock(rmn, blockable))
>>> + return -EAGAIN;
>>> it = interval_tree_iter_first(&rmn->objects, start, end);
>>> while (it) {
>>> @@ -219,6 +230,8 @@ static void amdgpu_mn_invalidate_range_start_gfx(struct mmu_notifier *mn,
>>> amdgpu_mn_invalidate_node(node, start, end);
>>> }
>>> +
>>> + return 0;
>>> }
>>> /**
>>> @@ -233,10 +246,11 @@ static void amdgpu_mn_invalidate_range_start_gfx(struct mmu_notifier *mn,
>>> * necessitates evicting all user-mode queues of the process. The BOs
>>> * are restorted in amdgpu_mn_invalidate_range_end_hsa.
>>> */
>>> -static void amdgpu_mn_invalidate_range_start_hsa(struct mmu_notifier *mn,
>>> +static int amdgpu_mn_invalidate_range_start_hsa(struct mmu_notifier *mn,
>>> struct mm_struct *mm,
>>> unsigned long start,
>>> - unsigned long end)
>>> + unsigned long end,
>>> + bool blockable)
>>> {
>>> struct amdgpu_mn *rmn = container_of(mn, struct amdgpu_mn, mn);
>>> struct interval_tree_node *it;
>>> @@ -244,7 +258,8 @@ static void amdgpu_mn_invalidate_range_start_hsa(struct mmu_notifier *mn,
>>> /* notification is exclusive, but interval is inclusive */
>>> end -= 1;
>>> - amdgpu_mn_read_lock(rmn);
>>> + if (amdgpu_mn_read_lock(rmn, blockable))
>>> + return -EAGAIN;
>>> it = interval_tree_iter_first(&rmn->objects, start, end);
>>> while (it) {
>>> @@ -262,6 +277,8 @@ static void amdgpu_mn_invalidate_range_start_hsa(struct mmu_notifier *mn,
>>> amdgpu_amdkfd_evict_userptr(mem, mm);
>>> }
>>> }
>>> +
>>> + return 0;
>>> }
>>> /**
next prev parent reply other threads:[~2018-06-22 20:09 UTC|newest]
Thread overview: 33+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-06-22 15:02 [RFC PATCH] mm, oom: distinguish blockable mode for mmu notifiers Michal Hocko
2018-06-22 15:13 ` Christian König
2018-06-22 15:24 ` Michal Hocko
2018-06-22 20:09 ` Felix Kuehling [this message]
2018-06-25 8:01 ` Michal Hocko
2018-06-25 13:31 ` Michal Hocko
[not found] ` <152968180950.11773.3374981930722769733@mail.alporthouse.com>
2018-06-22 15:57 ` [Intel-gfx] " Michal Hocko
2018-06-22 16:18 ` Jerome Glisse
2018-06-22 16:25 ` Jerome Glisse
2018-06-24 8:11 ` Paolo Bonzini
2018-06-25 7:57 ` Michal Hocko
2018-06-25 8:10 ` Paolo Bonzini
2018-06-25 8:45 ` Michal Hocko
2018-06-25 10:34 ` Paolo Bonzini
2018-06-25 11:08 ` Michal Hocko
2018-06-27 7:44 ` Michal Hocko
2018-07-02 9:14 ` Christian König
2018-07-02 11:54 ` Michal Hocko
2018-07-02 12:13 ` Christian König
2018-07-02 12:20 ` Michal Hocko
2018-07-02 12:24 ` Christian König
2018-07-02 12:35 ` Michal Hocko
2018-07-02 12:39 ` Christian König
2018-07-02 12:56 ` Michal Hocko
2018-07-09 12:29 ` Michal Hocko
2018-07-10 13:40 ` Leon Romanovsky
2018-07-10 14:14 ` Michal Hocko
2018-07-10 16:20 ` Leon Romanovsky
2018-07-11 9:03 ` Michal Hocko
2018-07-11 10:14 ` Leon Romanovsky
2018-07-11 11:13 ` Michal Hocko
2018-07-11 12:08 ` Leon Romanovsky
2018-07-16 7:59 ` Leon Romanovsky
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=dd260800-6457-f3ff-47df-b65ef258f4b7@amd.com \
--to=felix.kuehling@amd.com \
--cc=David1.Zhou@amd.com \
--cc=aarcange@redhat.com \
--cc=airlied@linux.ie \
--cc=alexander.deucher@amd.com \
--cc=amd-gfx@lists.freedesktop.org \
--cc=ashutosh.dixit@intel.com \
--cc=boris.ostrovsky@oracle.com \
--cc=christian.koenig@amd.com \
--cc=dennis.dalessandro@intel.com \
--cc=dledford@redhat.com \
--cc=dri-devel@lists.freedesktop.org \
--cc=intel-gfx@lists.freedesktop.org \
--cc=jani.nikula@linux.intel.com \
--cc=jgg@ziepe.ca \
--cc=jglisse@redhat.com \
--cc=jgross@suse.com \
--cc=joonas.lahtinen@linux.intel.com \
--cc=kvm@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=linux-rdma@vger.kernel.org \
--cc=mhocko@kernel.org \
--cc=mike.marciniszyn@intel.com \
--cc=pbonzini@redhat.com \
--cc=rientjes@google.com \
--cc=rkrcmar@redhat.com \
--cc=rodrigo.vivi@intel.com \
--cc=sivanich@sgi.com \
--cc=sudeep.dutt@intel.com \
--cc=xen-devel@lists.xenproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).