From: Michael Ellerman <mpe@ellerman.id.au>
To: Athira Rajeev <atrajeev@linux.vnet.ibm.com>
Cc: "Aneesh Kumar K . V" <aneesh.kumar@linux.ibm.com>,
linuxppc-dev <linuxppc-dev@lists.ozlabs.org>,
Nicholas Piggin <npiggin@gmail.com>
Subject: Re: [PATCH] powerpc/64s/hash: Fix hash_preload running with interrupts enabled
Date: Tue, 28 Jul 2020 10:44:56 +1000 [thread overview]
Message-ID: <87h7ts79iv.fsf@mpe.ellerman.id.au> (raw)
In-Reply-To: <9A09C920-D61C-41DC-BCCC-702F9D4BB055@linux.vnet.ibm.com>
Athira Rajeev <atrajeev@linux.vnet.ibm.com> writes:
>> On 27-Jul-2020, at 6:05 PM, Michael Ellerman <mpe@ellerman.id.au> wrote:
>>
>> Athira Rajeev <atrajeev@linux.vnet.ibm.com> writes:
>>>> On 27-Jul-2020, at 11:39 AM, Nicholas Piggin <npiggin@gmail.com> wrote:
>>>>
>>>> Commit 2f92447f9f96 ("powerpc/book3s64/hash: Use the pte_t address from the
>>>> caller") removed the local_irq_disable from hash_preload, but it was
>>>> required for more than just the page table walk: the hash pte busy bit is
>>>> effectively a lock which may be taken in interrupt context, and the local
>>>> update flag test must not be preempted before it's used.
>>>>
>>>> This solves apparent lockups with perf interrupting __hash_page_64K. If
>>>> get_perf_callchain then also takes a hash fault on the same page while it
>>>> is already locked, it will loop forever taking hash faults, which looks like
>>>> this:
>>>>
>>>> cpu 0x49e: Vector: 100 (System Reset) at [c00000001a4f7d70]
>>>> pc: c000000000072dc8: hash_page_mm+0x8/0x800
>>>> lr: c00000000000c5a4: do_hash_page+0x24/0x38
>>>> sp: c0002ac1cc69ac70
>>>> msr: 8000000000081033
>>>> current = 0xc0002ac1cc602e00
>>>> paca = 0xc00000001de1f280 irqmask: 0x03 irq_happened: 0x01
>>>> pid = 20118, comm = pread2_processe
>>>> Linux version 5.8.0-rc6-00345-g1fad14f18bc6
>>>> 49e:mon> t
>>>> [c0002ac1cc69ac70] c00000000000c5a4 do_hash_page+0x24/0x38 (unreliable)
>>>> --- Exception: 300 (Data Access) at c00000000008fa60 __copy_tofrom_user_power7+0x20c/0x7ac
>>>> [link register ] c000000000335d10 copy_from_user_nofault+0xf0/0x150
>>>> [c0002ac1cc69af70] c00032bf9fa3c880 (unreliable)
>>>> [c0002ac1cc69afa0] c000000000109df0 read_user_stack_64+0x70/0xf0
>>>> [c0002ac1cc69afd0] c000000000109fcc perf_callchain_user_64+0x15c/0x410
>>>> [c0002ac1cc69b060] c000000000109c00 perf_callchain_user+0x20/0x40
>>>> [c0002ac1cc69b080] c00000000031c6cc get_perf_callchain+0x25c/0x360
>>>> [c0002ac1cc69b120] c000000000316b50 perf_callchain+0x70/0xa0
>>>> [c0002ac1cc69b140] c000000000316ddc perf_prepare_sample+0x25c/0x790
>>>> [c0002ac1cc69b1a0] c000000000317350 perf_event_output_forward+0x40/0xb0
>>>> [c0002ac1cc69b220] c000000000306138 __perf_event_overflow+0x88/0x1a0
>>>> [c0002ac1cc69b270] c00000000010cf70 record_and_restart+0x230/0x750
>>>> [c0002ac1cc69b620] c00000000010d69c perf_event_interrupt+0x20c/0x510
>>>> [c0002ac1cc69b730] c000000000027d9c performance_monitor_exception+0x4c/0x60
>>>> [c0002ac1cc69b750] c00000000000b2f8 performance_monitor_common_virt+0x1b8/0x1c0
>>>> --- Exception: f00 (Performance Monitor) at c0000000000cb5b0 pSeries_lpar_hpte_insert+0x0/0x160
>>>> [link register ] c0000000000846f0 __hash_page_64K+0x210/0x540
>>>> [c0002ac1cc69ba50] 0000000000000000 (unreliable)
>>>> [c0002ac1cc69bb00] c000000000073ae0 update_mmu_cache+0x390/0x3a0
>>>> [c0002ac1cc69bb70] c00000000037f024 wp_page_copy+0x364/0xce0
>>>> [c0002ac1cc69bc20] c00000000038272c do_wp_page+0xdc/0xa60
>>>> [c0002ac1cc69bc70] c0000000003857bc handle_mm_fault+0xb9c/0x1b60
>>>> [c0002ac1cc69bd50] c00000000006c434 __do_page_fault+0x314/0xc90
>>>> [c0002ac1cc69be20] c00000000000c5c8 handle_page_fault+0x10/0x2c
>>>> --- Exception: 300 (Data Access) at 00007fff8c861fe8
>>>> SP (7ffff6b19660) is in userspace
>>>>
>>>> Reported-by: Athira Rajeev <atrajeev@linux.vnet.ibm.com>
>>>> Reported-by: Anton Blanchard <anton@ozlabs.org>
>>>> Reviewed-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
>>>> Fixes: 2f92447f9f96 ("powerpc/book3s64/hash: Use the pte_t address from the
>>>> caller")
>>>> Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
>>>
>>>
>>> Hi,
>>>
>>> Tested with the patch and it fixes the lockups I was seeing with my test run.
>>> Thanks for the fix.
>>>
>>> Tested-by: Athira Rajeev <atrajeev@linux.vnet.ibm.com>
>>
>> Thanks for testing.
>>
>> What test are you running?
>
> Hi Michael
>
> I was running “perf record” and Unixbench tests ( https://github.com/kdlucas/byte-unixbench ) in parallel where we were getting soft lockups
>
> 1. Perf command run:
> # perf record -a -g -c 10000000 -o <data_file> sleep 60
>
> 2. Unixbench tests
> # Run -q -c <nr_threads> spawn
Thanks, I can reproduce it with that.
cheers
next prev parent reply other threads:[~2020-07-28 0:46 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-07-27 6:09 [PATCH] powerpc/64s/hash: Fix hash_preload running with interrupts enabled Nicholas Piggin
2020-07-27 9:32 ` Athira Rajeev
2020-07-27 12:35 ` Michael Ellerman
2020-07-27 17:21 ` Athira Rajeev
2020-07-28 0:44 ` Michael Ellerman [this message]
2020-07-29 4:18 ` Athira Rajeev
2020-08-02 13:24 ` Michael Ellerman
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87h7ts79iv.fsf@mpe.ellerman.id.au \
--to=mpe@ellerman.id.au \
--cc=aneesh.kumar@linux.ibm.com \
--cc=atrajeev@linux.vnet.ibm.com \
--cc=linuxppc-dev@lists.ozlabs.org \
--cc=npiggin@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).