From: Dima Zavin <dmitriyz@waymo.com>
To: Vlastimil Babka <vbabka@suse.cz>
Cc: Peter Zijlstra <peterz@infradead.org>,
Christopher Lameter <cl@linux.com>, Li Zefan <lizefan@huawei.com>,
Pekka Enberg <penberg@kernel.org>,
David Rientjes <rientjes@google.com>,
Joonsoo Kim <iamjoonsoo.kim@lge.com>,
Andrew Morton <akpm@linux-foundation.org>,
cgroups@vger.kernel.org, LKML <linux-kernel@vger.kernel.org>,
linux-mm@kvack.org, Cliff Spradlin <cspradlin@waymo.com>,
Mel Gorman <mgorman@techsingularity.net>
Subject: Re: [PATCH v2] cpuset: fix a deadlock due to incomplete patching of cpusets_enabled()
Date: Fri, 28 Jul 2017 09:52:21 -0700 [thread overview]
Message-ID: <CAPz4a6ALRdkPZd7QBOMrC7t4_SqPPYm3gdhxnuBiVZOmPqs73g@mail.gmail.com> (raw)
In-Reply-To: <9e14ff85-1680-e76d-1b71-22301c16c286@suse.cz>
On Fri, Jul 28, 2017 at 7:05 AM, Vlastimil Babka <vbabka@suse.cz> wrote:
> On 07/28/2017 11:30 AM, Peter Zijlstra wrote:
>> On Fri, Jul 28, 2017 at 09:45:16AM +0200, Vlastimil Babka wrote:
>>> [+CC PeterZ]
>>>
>>> On 07/27/2017 06:46 PM, Dima Zavin wrote:
>>>> In codepaths that use the begin/retry interface for reading
>>>> mems_allowed_seq with irqs disabled, there exists a race condition that
>>>> stalls the patch process after only modifying a subset of the
>>>> static_branch call sites.
>>>>
>>>> This problem manifested itself as a dead lock in the slub
>>>> allocator, inside get_any_partial. The loop reads
>>>> mems_allowed_seq value (via read_mems_allowed_begin),
>>>> performs the defrag operation, and then verifies the consistency
>>>> of mem_allowed via the read_mems_allowed_retry and the cookie
>>>> returned by xxx_begin. The issue here is that both begin and retry
>>>> first check if cpusets are enabled via cpusets_enabled() static branch.
>>>> This branch can be rewritted dynamically (via cpuset_inc) if a new
>>>> cpuset is created. The x86 jump label code fully synchronizes across
>>>> all CPUs for every entry it rewrites. If it rewrites only one of the
>>>> callsites (specifically the one in read_mems_allowed_retry) and then
>>>> waits for the smp_call_function(do_sync_core) to complete while a CPU is
>>>> inside the begin/retry section with IRQs off and the mems_allowed value
>>>> is changed, we can hang. This is because begin() will always return 0
>>>> (since it wasn't patched yet) while retry() will test the 0 against
>>>> the actual value of the seq counter.
>>>
>>> Hm I wonder if there are other static branch users potentially having
>>> similar problem. Then it would be best to fix this at static branch
>>> level. Any idea, Peter? An inelegant solution would be to have indicate
>>> static_branch_(un)likely() callsites ordering for the patching. I.e.
>>> here we would make sure that read_mems_allowed_begin() callsites are
>>> patched before read_mems_allowed_retry() when enabling the static key,
>>> and the opposite order when disabling the static key.
>>
>> I'm not aware of any other sure ordering requirements. But you can
>> manually create this order by using 2 static keys. Then flip them in the
>> desired order.
>
> Right, thanks for the suggestion. I think that would be preferable to
> complicating the cookie handling. Add a new key next to
> cpusets_enabled_key, let's say "cpusets_enabled_pre_key". Make
> read_mems_allowed_begin() check this key instead of cpusets_enabled().
> Change cpuset_inc/dec to inc/dec also this new key in the right order
> and that should be it. Dima, can you try that or should I?
Yeah, I like that approach much better. I'll re-spin a new version in a bit.
--Dima
>
> Thanks,
> Vlastimil
>
>> --
>> To unsubscribe, send a message with 'unsubscribe linux-mm' in
>> the body to majordomo@kvack.org. For more info on Linux MM,
>> see: http://www.linux-mm.org/ .
>> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
>>
>
next prev parent reply other threads:[~2017-07-28 16:52 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-07-26 16:50 [RFC PATCH] mm/slub: fix a deadlock due to incomplete patching of cpusets_enabled() Dima Zavin
2017-07-26 17:02 ` Christopher Lameter
2017-07-26 19:54 ` Dima Zavin
2017-07-27 16:46 ` [PATCH v2] cpuset: " Dima Zavin
2017-07-27 19:48 ` Andrew Morton
2017-07-27 21:41 ` Dima Zavin
2017-07-27 19:51 ` Andrew Morton
2017-07-27 21:41 ` Dima Zavin
2017-07-28 7:45 ` Vlastimil Babka
2017-07-28 8:48 ` Dima Zavin
2017-07-28 9:30 ` Peter Zijlstra
2017-07-28 14:05 ` Vlastimil Babka
2017-07-28 16:52 ` Dima Zavin [this message]
2017-07-31 4:01 ` [PATCH v3] " Dima Zavin
2017-07-31 4:04 ` Dima Zavin
2017-07-31 8:02 ` Vlastimil Babka
2017-07-31 9:05 ` Dima Zavin
2017-07-29 4:56 ` [PATCH v2] " kbuild test robot
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAPz4a6ALRdkPZd7QBOMrC7t4_SqPPYm3gdhxnuBiVZOmPqs73g@mail.gmail.com \
--to=dmitriyz@waymo.com \
--cc=akpm@linux-foundation.org \
--cc=cgroups@vger.kernel.org \
--cc=cl@linux.com \
--cc=cspradlin@waymo.com \
--cc=iamjoonsoo.kim@lge.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=lizefan@huawei.com \
--cc=mgorman@techsingularity.net \
--cc=penberg@kernel.org \
--cc=peterz@infradead.org \
--cc=rientjes@google.com \
--cc=vbabka@suse.cz \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).