* [PATCH] x86/mm: Don't reenter flush_tlb_func_common()
@ 2017-06-19 4:48 Andy Lutomirski
2017-06-19 13:33 ` zhong jiang
0 siblings, 1 reply; 4+ messages in thread
From: Andy Lutomirski @ 2017-06-19 4:48 UTC (permalink / raw)
To: x86
Cc: linux-kernel, Borislav Petkov, Linus Torvalds, Andrew Morton,
Mel Gorman, linux-mm, Nadav Amit, Rik van Riel, Dave Hansen,
Arjan van de Ven, Peter Zijlstra, Andy Lutomirski
It was historically possible to have two concurrent TLB flushes
targeting the same CPU: one initiated locally and one initiated
remotely. This can now cause an OOPS in leave_mm() at
arch/x86/mm/tlb.c:47:
if (this_cpu_read(cpu_tlbstate.state) == TLBSTATE_OK)
BUG();
with this call trace:
flush_tlb_func_local arch/x86/mm/tlb.c:239 [inline]
flush_tlb_mm_range+0x26d/0x370 arch/x86/mm/tlb.c:317
Without reentrancy, this OOPS is impossible: leave_mm() is only
called if we're not in TLBSTATE_OK, but then we're unexpectedly
in TLBSTATE_OK in leave_mm().
This can be caused by flush_tlb_func_remote() happening between
the two checks and calling leave_mm(), resulting in two consecutive
leave_mm() calls on the same CPU with no intervening switch_mm()
calls.
We never saw this OOPS before because the old leave_mm()
implementation didn't put us back in TLBSTATE_OK, so the assertion
didn't fire.
Nadav noticed the reentrancy issue in a different context, but
neither of us realized that it caused a problem yet.
Cc: Nadav Amit <nadav.amit@gmail.com>
Cc: Dave Hansen <dave.hansen@intel.com>
Reported-by: "Levin, Alexander (Sasha Levin)" <alexander.levin@verizon.com>
Fixes: 3d28ebceaffa ("x86/mm: Rework lazy TLB to track the actual loaded mm")
Signed-off-by: Andy Lutomirski <luto@kernel.org>
---
arch/x86/mm/tlb.c | 15 +++++++++++++--
1 file changed, 13 insertions(+), 2 deletions(-)
diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c
index 2a5e851f2035..f06239c6919f 100644
--- a/arch/x86/mm/tlb.c
+++ b/arch/x86/mm/tlb.c
@@ -208,6 +208,9 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next,
static void flush_tlb_func_common(const struct flush_tlb_info *f,
bool local, enum tlb_flush_reason reason)
{
+ /* This code cannot presently handle being reentered. */
+ VM_WARN_ON(!irqs_disabled());
+
if (this_cpu_read(cpu_tlbstate.state) != TLBSTATE_OK) {
leave_mm(smp_processor_id());
return;
@@ -313,8 +316,12 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start,
info.end = TLB_FLUSH_ALL;
}
- if (mm == this_cpu_read(cpu_tlbstate.loaded_mm))
+ if (mm == this_cpu_read(cpu_tlbstate.loaded_mm)) {
+ local_irq_disable();
flush_tlb_func_local(&info, TLB_LOCAL_MM_SHOOTDOWN);
+ local_irq_enable();
+ }
+
if (cpumask_any_but(mm_cpumask(mm), cpu) < nr_cpu_ids)
flush_tlb_others(mm_cpumask(mm), &info);
put_cpu();
@@ -370,8 +377,12 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)
int cpu = get_cpu();
- if (cpumask_test_cpu(cpu, &batch->cpumask))
+ if (cpumask_test_cpu(cpu, &batch->cpumask)) {
+ local_irq_disable();
flush_tlb_func_local(&info, TLB_LOCAL_SHOOTDOWN);
+ local_irq_enable();
+ }
+
if (cpumask_any_but(&batch->cpumask, cpu) < nr_cpu_ids)
flush_tlb_others(&batch->cpumask, &info);
cpumask_clear(&batch->cpumask);
--
2.9.4
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH] x86/mm: Don't reenter flush_tlb_func_common()
2017-06-19 4:48 [PATCH] x86/mm: Don't reenter flush_tlb_func_common() Andy Lutomirski
@ 2017-06-19 13:33 ` zhong jiang
2017-06-19 15:05 ` Andy Lutomirski
0 siblings, 1 reply; 4+ messages in thread
From: zhong jiang @ 2017-06-19 13:33 UTC (permalink / raw)
To: Andy Lutomirski
Cc: x86, linux-kernel, Borislav Petkov, Linus Torvalds,
Andrew Morton, Mel Gorman, linux-mm, Nadav Amit, Rik van Riel,
Dave Hansen, Arjan van de Ven, Peter Zijlstra
On 2017/6/19 12:48, Andy Lutomirski wrote:
> It was historically possible to have two concurrent TLB flushes
> targeting the same CPU: one initiated locally and one initiated
> remotely. This can now cause an OOPS in leave_mm() at
> arch/x86/mm/tlb.c:47:
>
> if (this_cpu_read(cpu_tlbstate.state) == TLBSTATE_OK)
> BUG();
>
> with this call trace:
> flush_tlb_func_local arch/x86/mm/tlb.c:239 [inline]
> flush_tlb_mm_range+0x26d/0x370 arch/x86/mm/tlb.c:317
>
> Without reentrancy, this OOPS is impossible: leave_mm() is only
> called if we're not in TLBSTATE_OK, but then we're unexpectedly
> in TLBSTATE_OK in leave_mm().
>
> This can be caused by flush_tlb_func_remote() happening between
> the two checks and calling leave_mm(), resulting in two consecutive
> leave_mm() calls on the same CPU with no intervening switch_mm()
> calls.
>
> We never saw this OOPS before because the old leave_mm()
> implementation didn't put us back in TLBSTATE_OK, so the assertion
> didn't fire.
HI, Andy
Today, I see same OOPS in linux 3.4 stable. It prove that it indeed has fired.
but It is rarely to appear. I review the code. I found the a issue.
when current->mm is NULL, leave_mm will be called. but it maybe in
TLBSTATE_OK, eg: unuse_mm call after task->mm = NULL , but before enter_lazy_tlb.
therefore, it will fire. is it right?
Thanks
zhongjiang
> Nadav noticed the reentrancy issue in a different context, but
> neither of us realized that it caused a problem yet.
>
> Cc: Nadav Amit <nadav.amit@gmail.com>
> Cc: Dave Hansen <dave.hansen@intel.com>
> Reported-by: "Levin, Alexander (Sasha Levin)" <alexander.levin@verizon.com>
> Fixes: 3d28ebceaffa ("x86/mm: Rework lazy TLB to track the actual loaded mm")
> Signed-off-by: Andy Lutomirski <luto@kernel.org>
> ---
> arch/x86/mm/tlb.c | 15 +++++++++++++--
> 1 file changed, 13 insertions(+), 2 deletions(-)
>
> diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c
> index 2a5e851f2035..f06239c6919f 100644
> --- a/arch/x86/mm/tlb.c
> +++ b/arch/x86/mm/tlb.c
> @@ -208,6 +208,9 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next,
> static void flush_tlb_func_common(const struct flush_tlb_info *f,
> bool local, enum tlb_flush_reason reason)
> {
> + /* This code cannot presently handle being reentered. */
> + VM_WARN_ON(!irqs_disabled());
> +
> if (this_cpu_read(cpu_tlbstate.state) != TLBSTATE_OK) {
> leave_mm(smp_processor_id());
> return;
> @@ -313,8 +316,12 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start,
> info.end = TLB_FLUSH_ALL;
> }
>
> - if (mm == this_cpu_read(cpu_tlbstate.loaded_mm))
> + if (mm == this_cpu_read(cpu_tlbstate.loaded_mm)) {
> + local_irq_disable();
> flush_tlb_func_local(&info, TLB_LOCAL_MM_SHOOTDOWN);
> + local_irq_enable();
> + }
> +
> if (cpumask_any_but(mm_cpumask(mm), cpu) < nr_cpu_ids)
> flush_tlb_others(mm_cpumask(mm), &info);
> put_cpu();
> @@ -370,8 +377,12 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)
>
> int cpu = get_cpu();
>
> - if (cpumask_test_cpu(cpu, &batch->cpumask))
> + if (cpumask_test_cpu(cpu, &batch->cpumask)) {
> + local_irq_disable();
> flush_tlb_func_local(&info, TLB_LOCAL_SHOOTDOWN);
> + local_irq_enable();
> + }
> +
> if (cpumask_any_but(&batch->cpumask, cpu) < nr_cpu_ids)
> flush_tlb_others(&batch->cpumask, &info);
> cpumask_clear(&batch->cpumask);
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH] x86/mm: Don't reenter flush_tlb_func_common()
2017-06-19 13:33 ` zhong jiang
@ 2017-06-19 15:05 ` Andy Lutomirski
2017-06-20 2:56 ` zhong jiang
0 siblings, 1 reply; 4+ messages in thread
From: Andy Lutomirski @ 2017-06-19 15:05 UTC (permalink / raw)
To: zhong jiang
Cc: Andy Lutomirski, X86 ML, linux-kernel, Borislav Petkov,
Linus Torvalds, Andrew Morton, Mel Gorman, linux-mm, Nadav Amit,
Rik van Riel, Dave Hansen, Arjan van de Ven, Peter Zijlstra
On Mon, Jun 19, 2017 at 6:33 AM, zhong jiang <zhongjiang@huawei.com> wrote:
> On 2017/6/19 12:48, Andy Lutomirski wrote:
>> It was historically possible to have two concurrent TLB flushes
>> targeting the same CPU: one initiated locally and one initiated
>> remotely. This can now cause an OOPS in leave_mm() at
>> arch/x86/mm/tlb.c:47:
>>
>> if (this_cpu_read(cpu_tlbstate.state) == TLBSTATE_OK)
>> BUG();
>>
>> with this call trace:
>> flush_tlb_func_local arch/x86/mm/tlb.c:239 [inline]
>> flush_tlb_mm_range+0x26d/0x370 arch/x86/mm/tlb.c:317
>>
>> Without reentrancy, this OOPS is impossible: leave_mm() is only
>> called if we're not in TLBSTATE_OK, but then we're unexpectedly
>> in TLBSTATE_OK in leave_mm().
>>
>> This can be caused by flush_tlb_func_remote() happening between
>> the two checks and calling leave_mm(), resulting in two consecutive
>> leave_mm() calls on the same CPU with no intervening switch_mm()
>> calls.
>>
>> We never saw this OOPS before because the old leave_mm()
>> implementation didn't put us back in TLBSTATE_OK, so the assertion
>> didn't fire.
> HI, Andy
>
> Today, I see same OOPS in linux 3.4 stable. It prove that it indeed has fired.
> but It is rarely to appear. I review the code. I found the a issue.
> when current->mm is NULL, leave_mm will be called. but it maybe in
> TLBSTATE_OK, eg: unuse_mm call after task->mm = NULL , but before enter_lazy_tlb.
>
> therefore, it will fire. is it right?
Is there a code path that does this?
Also, the IPI handler on 3.4 looks like this:
if (f->flush_mm == percpu_read(cpu_tlbstate.active_mm)) {
if (percpu_read(cpu_tlbstate.state) == TLBSTATE_OK) {
if (f->flush_va == TLB_FLUSH_ALL)
local_flush_tlb();
else
__flush_tlb_one(f->flush_va);
} else
leave_mm(cpu);
}
but leave_mm() checks the same condition (cpu_tlbstate.state, not
current->mm). How is the BUG triggering?
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH] x86/mm: Don't reenter flush_tlb_func_common()
2017-06-19 15:05 ` Andy Lutomirski
@ 2017-06-20 2:56 ` zhong jiang
0 siblings, 0 replies; 4+ messages in thread
From: zhong jiang @ 2017-06-20 2:56 UTC (permalink / raw)
To: Andy Lutomirski
Cc: X86 ML, linux-kernel, Borislav Petkov, Linus Torvalds,
Andrew Morton, Mel Gorman, linux-mm, Nadav Amit, Rik van Riel,
Dave Hansen, Arjan van de Ven, Peter Zijlstra
On 2017/6/19 23:05, Andy Lutomirski wrote:
> On Mon, Jun 19, 2017 at 6:33 AM, zhong jiang <zhongjiang@huawei.com> wrote:
>> On 2017/6/19 12:48, Andy Lutomirski wrote:
>>> It was historically possible to have two concurrent TLB flushes
>>> targeting the same CPU: one initiated locally and one initiated
>>> remotely. This can now cause an OOPS in leave_mm() at
>>> arch/x86/mm/tlb.c:47:
>>>
>>> if (this_cpu_read(cpu_tlbstate.state) == TLBSTATE_OK)
>>> BUG();
>>>
>>> with this call trace:
>>> flush_tlb_func_local arch/x86/mm/tlb.c:239 [inline]
>>> flush_tlb_mm_range+0x26d/0x370 arch/x86/mm/tlb.c:317
>>>
>>> Without reentrancy, this OOPS is impossible: leave_mm() is only
>>> called if we're not in TLBSTATE_OK, but then we're unexpectedly
>>> in TLBSTATE_OK in leave_mm().
>>>
>>> This can be caused by flush_tlb_func_remote() happening between
>>> the two checks and calling leave_mm(), resulting in two consecutive
>>> leave_mm() calls on the same CPU with no intervening switch_mm()
>>> calls.
>>>
>>> We never saw this OOPS before because the old leave_mm()
>>> implementation didn't put us back in TLBSTATE_OK, so the assertion
>>> didn't fire.
>> HI, Andy
>>
>> Today, I see same OOPS in linux 3.4 stable. It prove that it indeed has fired.
>> but It is rarely to appear. I review the code. I found the a issue.
>> when current->mm is NULL, leave_mm will be called. but it maybe in
>> TLBSTATE_OK, eg: unuse_mm call after task->mm = NULL , but before enter_lazy_tlb.
>>
>> therefore, it will fire. is it right?
> Is there a code path that does this?
eg:
cpu1 cpu2
flush_tlb_page unuse_mm
current->mm = NULL
current->mm == NULL
leave_mm (cpu_tlbstate.state is TLBSATATE_OK)
enter_lazy_tlb
I am not sure the above race whether exist or not. Do you point out the problem if it is not existence? please
Thanks
zhongjiang
>
> Also, the IPI handler on 3.4 looks like this:
>
> if (f->flush_mm == percpu_read(cpu_tlbstate.active_mm)) {
> if (percpu_read(cpu_tlbstate.state) == TLBSTATE_OK) {
> if (f->flush_va == TLB_FLUSH_ALL)
> local_flush_tlb();
> else
> __flush_tlb_one(f->flush_va);
> } else
> leave_mm(cpu);
> }
>
> but leave_mm() checks the same condition (cpu_tlbstate.state, not
> current->mm). How is the BUG triggering?
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org. For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
>
>
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2017-06-20 3:01 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-06-19 4:48 [PATCH] x86/mm: Don't reenter flush_tlb_func_common() Andy Lutomirski
2017-06-19 13:33 ` zhong jiang
2017-06-19 15:05 ` Andy Lutomirski
2017-06-20 2:56 ` zhong jiang
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).