From: john.p.donnelly@oracle.com
To: Waiman Long <longman@redhat.com>,
chenguanyou <chenguanyou9338@gmail.com>,
gregkh@linuxfoundation.org
Cc: dave@stgolabs.net, hdanton@sina.com,
linux-kernel@vger.kernel.org, mazhenhua@xiaomi.com,
mingo@redhat.com, peterz@infradead.org, quic_aiquny@quicinc.com,
will@kernel.org, sashal@kernel.org,
John Donnelly <john.p.donnelly@oracle.com>
Subject: Re: [PATCH v5] locking/rwsem: Make handoff bit handling more consistent
Date: Mon, 11 Apr 2022 16:03:40 -0500 [thread overview]
Message-ID: <31178c33-e25c-c3e8-35e2-776b5211200c@oracle.com> (raw)
In-Reply-To: <e873727e-22db-3330-015d-bd6581a2937a@redhat.com>
On 4/11/22 1:40 PM, Waiman Long wrote:
> On 4/11/22 14:26, john.p.donnelly@oracle.com wrote:
>> On 2/14/22 9:47 AM, chenguanyou wrote:
>>> Hi Waiman, Greg,
>>> This patch has been merged in branch linux-5.16.y.
>>> Can we take it to the linux-5.10.y LTS version?
>>>
>>> thanks,
>>
>> Hi,
>>
>> As a FYI:
>>
>> We have observed that following lockup with this commit added to
>> 5.15.LTS:
>>
>> d257cc8cb8d5 - locking/rwsem: Make handoff bit handling more
>> consistent (4 months ago) <Waiman Long>
>>
>> The "fio" test suit fails with LVM devices composed of four NVME
>> devices with these observed lockup, panic.
>>
>>
>>
>> ext4:
>>
>> PID: 3682 TASK: ffff8f489ae34bc0 CPU: 2 COMMAND: "dio/dm-0"
>> #0 [fffffe0000083e50] crash_nmi_callback at ffffffff828772b3
>> #1 [fffffe0000083e58] nmi_handle at ffffffff82840778
>> #2 [fffffe0000083ea0] default_do_nmi at ffffffff8337a1e2
>> #3 [fffffe0000083ec8] exc_nmi at ffffffff8337a48d
>> #4 [fffffe0000083ef0] end_repeat_nmi at ffffffff8340153b
>> [exception RIP: _raw_spin_lock_irq+23]
>> RIP: ffffffff8338b2e7 RSP: ffff9c4409b47c78 RFLAGS: 00000046
>> RAX: 0000000000000000 RBX: ffff8f489ae34bc0 RCX: 0000000000000000
>> RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8f47f7b90104
>> RBP: ffff9c4409b47d20 R8: 0000000000000000 R9: 0000000000000000
>> R10: 0000000000000000 R11: 0000000000000000 R12: ffff8f47f7b90104
>> R13: ffff9c4409b47cb0 R14: ffff8f47f7b900f0 R15: 0000000000000000
>> ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
>> <NMI exception stack> ---
>> #5 [ffff9c4409b47c78] _raw_spin_lock_irq at ffffffff8338b2e7
>> #6 [ffff9c4409b47c78] rwsem_down_write_slowpath at ffffffff82925be9
>> #7 [ffff9c4409b47d28] ext4_map_blocks at ffffffffc11c26dc [ext4]
>> #8 [ffff9c4409b47d98] ext4_convert_unwritten_extents at
>> ffffffffc11ad9e0 [ext4]
>> #9 [ffff9c4409b47df0] ext4_dio_write_end_io at ffffffffc11b22aa [ext4]
>>
>> xfs:
>>
>> PID: 3719 TASK: ffff9f81d2d74bc0 CPU: 37 COMMAND: "dio/dm-0"
>> #0 [fffffe0000894e50] crash_nmi_callback at ffffffffad6772b3
>> #1 [fffffe0000894e58] nmi_handle at ffffffffad640778
>> #2 [fffffe0000894ea0] default_do_nmi at ffffffffae17a1e2
>> #3 [fffffe0000894ec8] exc_nmi at ffffffffae17a48d
>> #4 [fffffe0000894ef0] end_repeat_nmi at ffffffffae20153b
>> [exception RIP: _raw_spin_lock_irq+23]
>> RIP: ffffffffae18b2e7 RSP: ffffbb7ec9637c48 RFLAGS: 00000046
>> RAX: 0000000000000000 RBX: ffff9f81d2d74bc0 RCX: 0000000000000000
>> RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff9f81c04a918c
>> RBP: ffffbb7ec9637ce8 R8: 0000000000000000 R9: 0000000000000000
>> R10: 0000000000000000 R11: 0000000000000000 R12: ffff9f81c04a918c
>> R13: ffffbb7ec9637c80 R14: ffff9f81c04a9178 R15: 0000000000000000
>> ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
>> <NMI exception stack> ---
>> #5 [ffffbb7ec9637c48] _raw_spin_lock_irq at ffffffffae18b2e7
>> #6 [ffffbb7ec9637c48] rwsem_down_write_slowpath at ffffffffad725be9
>> #7 [ffffbb7ec9637cf0] xfs_trans_alloc_inode at ffffffffc074f2bd [xfs]
>> #8 [ffffbb7ec9637d50] xfs_iomap_write_unwritten at ffffffffc073ad15
>> [xfs]
>> #9 [ffffbb7ec9637dd0] xfs_dio_write_end_io at ffffffffc072db62 [xfs]
>>
>>
>> I have reached out to Waiman and he suggested this for our next test
>> pass:
>>
>>
>> 1ee326196c6658 locking/rwsem: Always try to wake waiters in out_nolock
>> path
>
> Does this commit help to avoid the lockup problem?
>
> Commit 1ee326196c6658 fixes a potential missed wakeup problem when a
> reader first in the wait queue is interrupted out without acquiring the
> lock. It is actually not a fix for commit d257cc8cb8d5. However, this
> commit changes the out_nolock path behavior of writers by leaving the
> handoff bit set when the wait queue isn't empty. That likely makes the
> missed wakeup problem easier to reproduce.
>
> Cheers,
> Longman
>
Hi,
We are testing now
ETA for fio soak test completion is ~15hr from now.
I wanted to share the stack traces for future reference + occurrences.
Cheers.
JD
...
next prev parent reply other threads:[~2022-04-11 21:04 UTC|newest]
Thread overview: 36+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-11-16 1:29 [PATCH v5] locking/rwsem: Make handoff bit handling more consistent Waiman Long
2021-11-16 2:52 ` Aiqun(Maria) Yu
2021-11-16 9:14 ` Peter Zijlstra
2021-11-16 9:24 ` Peter Zijlstra
2021-11-16 14:52 ` Waiman Long
2021-11-17 13:36 ` Peter Zijlstra
2021-11-23 8:53 ` [tip: locking/urgent] " tip-bot2 for Waiman Long
2022-02-14 15:47 ` Re:[PATCH v5] " chenguanyou
2022-02-14 16:01 ` [PATCH " Greg KH
2022-04-11 18:26 ` john.p.donnelly
2022-04-11 18:40 ` Waiman Long
2022-04-11 21:03 ` john.p.donnelly [this message]
2022-04-11 21:07 ` Waiman Long
2022-04-12 16:28 ` john.p.donnelly
2022-04-12 17:04 ` Waiman Long
2022-04-14 10:48 ` Greg KH
2022-04-14 15:18 ` Waiman Long
2022-04-14 15:42 ` Greg KH
2022-04-14 15:44 ` Waiman Long
2022-04-20 13:55 ` john.p.donnelly
2022-04-26 20:21 ` Waiman Long
2022-04-26 21:22 ` john.p.donnelly
2022-02-14 16:22 ` chenguanyou
2022-02-15 7:41 ` [PATCH " Greg KH
2022-02-16 16:30 ` Waiman Long
2022-02-17 15:41 ` chenguanyou
2022-03-14 8:07 ` [PATCH " Greg KH
2022-03-22 2:49 ` chenguanyou
2022-03-24 12:51 ` [PATCH " Greg KH
2022-07-19 0:27 ` Doug Anderson
[not found] ` <20220719104104.1634-1-hdanton@sina.com>
2022-07-19 15:30 ` Doug Anderson
[not found] ` <20220722115510.2101-1-hdanton@sina.com>
2022-07-22 14:02 ` Doug Anderson
[not found] ` <20220723001713.2156-1-hdanton@sina.com>
2022-08-05 17:14 ` Doug Anderson
2022-08-05 19:02 ` Waiman Long
2022-08-05 19:16 ` Doug Anderson
2022-08-30 16:18 ` Doug Anderson
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=31178c33-e25c-c3e8-35e2-776b5211200c@oracle.com \
--to=john.p.donnelly@oracle.com \
--cc=chenguanyou9338@gmail.com \
--cc=dave@stgolabs.net \
--cc=gregkh@linuxfoundation.org \
--cc=hdanton@sina.com \
--cc=linux-kernel@vger.kernel.org \
--cc=longman@redhat.com \
--cc=mazhenhua@xiaomi.com \
--cc=mingo@redhat.com \
--cc=peterz@infradead.org \
--cc=quic_aiquny@quicinc.com \
--cc=sashal@kernel.org \
--cc=will@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).