* [Ocfs2-devel] [PATCH] fix kernel crash after mounting when dlm doesn't ready
@ 2022-03-09 13:30 Heming Zhao via Ocfs2-devel
2022-03-10 3:59 ` Joseph Qi via Ocfs2-devel
0 siblings, 1 reply; 3+ messages in thread
From: Heming Zhao via Ocfs2-devel @ 2022-03-09 13:30 UTC (permalink / raw)
To: ocfs2-devel, joseph.qi
How to trigger:
tb-ocfs1 # dd if=/dev/zero of=/dev/vdb bs=1M count=20 oflag=direct
tb-ocfs1 # mkfs.ocfs2 --cluster-stack=pcmk -N 4 /dev/vdb \
--cluster-name=ocfstst
== there is no dlm running ==
tb-ocfs1 # mount -t ocfs2 /dev/vdb /mnt
== kernel crash ==
Crash log
```
kernel: DLM installed
kernel: ocfs2: Registered cluster interface user
kernel: dlm: no local IP address has been set
kernel: dlm: cannot start dlm lowcomms -107
kernel: (mount.ocfs2,2225,0):ocfs2_dlm_init:3355 ERROR: status = -107
kernel: (mount.ocfs2,2225,0):ocfs2_mount_volume:1817 ERROR: status = -107
kernel: (mount.ocfs2,2225,0):ocfs2_fill_super:1186 ERROR: status = -107
kernel: BUG: kernel NULL pointer dereference, address: 0000000000000030
kernel: #PF: supervisor read access in kernel mode
kernel: #PF: error_code(0x0000) - not-present page
kernel: PGD 0 P4D 0
kernel: Oops: 0000 [#1] PREEMPT SMP NOPTI
kernel: CPU: 0 PID: 2225 Comm: mount.ocfs2 Not tainted 5.16.2-1-default #1 openSUSE Tumbleweed b40a195b7ff0f3399a616c3290f963c4ad189e84
kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.14.0-0-g155821a-rebuilt.opensuse.org 04/01/2014
kernel: RIP: 0010:ocfs2_clear_inode+0x2e9/0x720 [ocfs2]
kernel: Code: 24 e8 9b 68 04 00 48 c7 c7 70 88 db c0 48 8b 80 98 03 00
00 48 8b 80 70 01 00 00 48 89 44 24 08 e8 3c 50 d8 d8 48 8b 44 24 08
<48> 8b 48 30 49 39 4f c8 0f 88 ff 00 00 00 48 c>
kernel: RSP: 0018:ffffbbf000847bf0 EFLAGS: 00010246
kernel: RAX: 0000000000000000 RBX: ffff95f6834b8000 RCX: 0000000000000000
kernel: RDX: 0000000000000001 RSI: ffff95f6a1fbcbe0 RDI: ffffffffc0db8870
kernel: RBP: ffff95f6a1fbc6b8 R08: 00000ab5a9371b7a R09: 0000000000000230
kernel: R10: ffffbbf000847bc0 R11: ffffffffc0d53ea0 R12: ffff95f6a1fbc560
kernel: R13: ffff95f6a1fbc408 R14: ffff95f6834b8000 R15: ffff95f6a1fbc908
kernel: FS: 00007f366f151740(0000) GS:ffff95f6fdc00000(0000) knlGS:0000000000000000
kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kernel: CR2: 0000000000000030 CR3: 0000000003fcc004 CR4: 0000000000370ef0
kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
kernel: Call Trace:
kernel: <TASK>
kernel: ? ocfs2_evict_inode+0x1fe/0x630 [ocfs2 411bc..281]
kernel: evict+0xc0/0x1c0
kernel: ocfs2_release_system_inodes+0x21/0xc0 [ocfs2 411bc..281]
kernel: ocfs2_dismount_volume+0x10b/0x2d0 [ocfs2 411bc..281]
kernel: ocfs2_fill_super+0xaf/0x19e0 [ocfs2 411bc..281]
kernel: mount_bdev+0x182/0x1b0
kernel: ? ocfs2_initialize_super.isra.0+0xf50/0xf50 [ocfs2 411bc..281]
kernel: legacy_get_tree+0x24/0x40
kernel: vfs_get_tree+0x22/0xb0
kernel: path_mount+0x465/0xac0
kernel: __x64_sys_mount+0x103/0x140
kernel: do_syscall_64+0x59/0x80
kernel: ? syscall_exit_to_user_mode+0x18/0x40
kernel: ? do_syscall_64+0x69/0x80
kernel: ? syscall_exit_to_user_mode+0x18/0x40
kernel: ? do_syscall_64+0x69/0x80
kernel: ? exc_page_fault+0x68/0x150
kernel: entry_SYSCALL_64_after_hwframe+0x44/0xae
```
Analysis:
ocfs2_fill_super
ocfs2_mount_volume
ocfs2_dlm_init //failed, journal still doesn't be initized.
goto read_super_error
ocfs2_dismount_volume
ocfs2_release_system_inodes
...
evict
...
ocfs2_clear_inode
ocfs2_checkpoint_inode
ocfs2_ci_fully_checkpointed
time_after(journal->j_trans_id, ci->ci_last_trans)
+ journal is empty, crash!
Signed-off-by: Heming Zhao <heming.zhao@suse.com>
---
fs/ocfs2/inode.c | 3 ++-
fs/ocfs2/journal.h | 2 +-
2 files changed, 3 insertions(+), 2 deletions(-)
diff --git a/fs/ocfs2/inode.c b/fs/ocfs2/inode.c
index 6c2411c2afcf..3826ab7eab3e 100644
--- a/fs/ocfs2/inode.c
+++ b/fs/ocfs2/inode.c
@@ -1205,7 +1205,8 @@ static void ocfs2_clear_inode(struct inode *inode)
* the journal is flushed before journal shutdown. Thus it is safe to
* have inodes get cleaned up after journal shutdown.
*/
- jbd2_journal_release_jbd_inode(osb->journal->j_journal,
+ if (osb->journal)
+ jbd2_journal_release_jbd_inode(osb->journal->j_journal,
&oi->ip_jinode);
}
diff --git a/fs/ocfs2/journal.h b/fs/ocfs2/journal.h
index 8dcb2f2cadbc..1610d49b4112 100644
--- a/fs/ocfs2/journal.h
+++ b/fs/ocfs2/journal.h
@@ -189,7 +189,7 @@ static inline void ocfs2_checkpoint_inode(struct inode *inode)
{
struct ocfs2_super *osb = OCFS2_SB(inode->i_sb);
- if (ocfs2_mount_local(osb))
+ if (!osb->journal || ocfs2_mount_local(osb))
return;
if (!ocfs2_ci_fully_checkpointed(INODE_CACHE(inode))) {
--
2.34.1
_______________________________________________
Ocfs2-devel mailing list
Ocfs2-devel@oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel
^ permalink raw reply related [flat|nested] 3+ messages in thread* Re: [Ocfs2-devel] [PATCH] fix kernel crash after mounting when dlm doesn't ready
2022-03-09 13:30 [Ocfs2-devel] [PATCH] fix kernel crash after mounting when dlm doesn't ready Heming Zhao via Ocfs2-devel
@ 2022-03-10 3:59 ` Joseph Qi via Ocfs2-devel
2022-03-10 8:16 ` heming.zhao--- via Ocfs2-devel
0 siblings, 1 reply; 3+ messages in thread
From: Joseph Qi via Ocfs2-devel @ 2022-03-10 3:59 UTC (permalink / raw)
To: Heming Zhao, ocfs2-devel
It looks this issue is introduced by the following commit:
da5e7c87827e ocfs2: cleanup journal init and shutdown
Before that, journal was initialized in ocfs2_initialize_super().
On 3/9/22 9:30 PM, Heming Zhao wrote:
> How to trigger:
>
> tb-ocfs1 # dd if=/dev/zero of=/dev/vdb bs=1M count=20 oflag=direct
> tb-ocfs1 # mkfs.ocfs2 --cluster-stack=pcmk -N 4 /dev/vdb \
> --cluster-name=ocfstst
> == there is no dlm running ==
> tb-ocfs1 # mount -t ocfs2 /dev/vdb /mnt
> == kernel crash ==
>
> Crash log
>
> ```
> kernel: DLM installed
> kernel: ocfs2: Registered cluster interface user
> kernel: dlm: no local IP address has been set
> kernel: dlm: cannot start dlm lowcomms -107
> kernel: (mount.ocfs2,2225,0):ocfs2_dlm_init:3355 ERROR: status = -107
> kernel: (mount.ocfs2,2225,0):ocfs2_mount_volume:1817 ERROR: status = -107
> kernel: (mount.ocfs2,2225,0):ocfs2_fill_super:1186 ERROR: status = -107
> kernel: BUG: kernel NULL pointer dereference, address: 0000000000000030
> kernel: #PF: supervisor read access in kernel mode
> kernel: #PF: error_code(0x0000) - not-present page
> kernel: PGD 0 P4D 0
> kernel: Oops: 0000 [#1] PREEMPT SMP NOPTI
> kernel: CPU: 0 PID: 2225 Comm: mount.ocfs2 Not tainted 5.16.2-1-default #1 openSUSE Tumbleweed b40a195b7ff0f3399a616c3290f963c4ad189e84
> kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.14.0-0-g155821a-rebuilt.opensuse.org 04/01/2014
> kernel: RIP: 0010:ocfs2_clear_inode+0x2e9/0x720 [ocfs2]
> kernel: Code: 24 e8 9b 68 04 00 48 c7 c7 70 88 db c0 48 8b 80 98 03 00
> 00 48 8b 80 70 01 00 00 48 89 44 24 08 e8 3c 50 d8 d8 48 8b 44 24 08
> <48> 8b 48 30 49 39 4f c8 0f 88 ff 00 00 00 48 c>
> kernel: RSP: 0018:ffffbbf000847bf0 EFLAGS: 00010246
> kernel: RAX: 0000000000000000 RBX: ffff95f6834b8000 RCX: 0000000000000000
> kernel: RDX: 0000000000000001 RSI: ffff95f6a1fbcbe0 RDI: ffffffffc0db8870
> kernel: RBP: ffff95f6a1fbc6b8 R08: 00000ab5a9371b7a R09: 0000000000000230
> kernel: R10: ffffbbf000847bc0 R11: ffffffffc0d53ea0 R12: ffff95f6a1fbc560
> kernel: R13: ffff95f6a1fbc408 R14: ffff95f6834b8000 R15: ffff95f6a1fbc908
> kernel: FS: 00007f366f151740(0000) GS:ffff95f6fdc00000(0000) knlGS:0000000000000000
> kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> kernel: CR2: 0000000000000030 CR3: 0000000003fcc004 CR4: 0000000000370ef0
> kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> kernel: Call Trace:
> kernel: <TASK>
> kernel: ? ocfs2_evict_inode+0x1fe/0x630 [ocfs2 411bc..281]
> kernel: evict+0xc0/0x1c0
> kernel: ocfs2_release_system_inodes+0x21/0xc0 [ocfs2 411bc..281]
> kernel: ocfs2_dismount_volume+0x10b/0x2d0 [ocfs2 411bc..281]
> kernel: ocfs2_fill_super+0xaf/0x19e0 [ocfs2 411bc..281]
> kernel: mount_bdev+0x182/0x1b0
> kernel: ? ocfs2_initialize_super.isra.0+0xf50/0xf50 [ocfs2 411bc..281]
> kernel: legacy_get_tree+0x24/0x40
> kernel: vfs_get_tree+0x22/0xb0
> kernel: path_mount+0x465/0xac0
> kernel: __x64_sys_mount+0x103/0x140
> kernel: do_syscall_64+0x59/0x80
> kernel: ? syscall_exit_to_user_mode+0x18/0x40
> kernel: ? do_syscall_64+0x69/0x80
> kernel: ? syscall_exit_to_user_mode+0x18/0x40
> kernel: ? do_syscall_64+0x69/0x80
> kernel: ? exc_page_fault+0x68/0x150
> kernel: entry_SYSCALL_64_after_hwframe+0x44/0xae
> ```
>
> Analysis:
>
> ocfs2_fill_super
> ocfs2_mount_volume
> ocfs2_dlm_init //failed, journal still doesn't be initized.
> goto read_super_error
This is not a part of call trace, I don't think it is needed here.
> ocfs2_dismount_volume
> ocfs2_release_system_inodes
> ...
> evict
> ...
> ocfs2_clear_inode
> ocfs2_checkpoint_inode
> ocfs2_ci_fully_checkpointed
> time_after(journal->j_trans_id, ci->ci_last_trans)
> + journal is empty, crash!
>
Suggest we use the following way to description this commit:
<call trace>
<reproducer>
<your analysis and how to fix>
BTW, as I mentioned at the first, a fixes tag should be mentioned here.
Could you also please check all possible use of journal during
ocfs2_dismount_volume()?
Thanks,
Joseph
> Signed-off-by: Heming Zhao <heming.zhao@suse.com>
> ---
> fs/ocfs2/inode.c | 3 ++-
> fs/ocfs2/journal.h | 2 +-
> 2 files changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/fs/ocfs2/inode.c b/fs/ocfs2/inode.c
> index 6c2411c2afcf..3826ab7eab3e 100644
> --- a/fs/ocfs2/inode.c
> +++ b/fs/ocfs2/inode.c
> @@ -1205,7 +1205,8 @@ static void ocfs2_clear_inode(struct inode *inode)
> * the journal is flushed before journal shutdown. Thus it is safe to
> * have inodes get cleaned up after journal shutdown.
> */
> - jbd2_journal_release_jbd_inode(osb->journal->j_journal,
> + if (osb->journal)
> + jbd2_journal_release_jbd_inode(osb->journal->j_journal,
> &oi->ip_jinode);
> }
>
> diff --git a/fs/ocfs2/journal.h b/fs/ocfs2/journal.h
> index 8dcb2f2cadbc..1610d49b4112 100644
> --- a/fs/ocfs2/journal.h
> +++ b/fs/ocfs2/journal.h
> @@ -189,7 +189,7 @@ static inline void ocfs2_checkpoint_inode(struct inode *inode)
> {
> struct ocfs2_super *osb = OCFS2_SB(inode->i_sb);
>
> - if (ocfs2_mount_local(osb))
> + if (!osb->journal || ocfs2_mount_local(osb))
> return;
>
> if (!ocfs2_ci_fully_checkpointed(INODE_CACHE(inode))) {
_______________________________________________
Ocfs2-devel mailing list
Ocfs2-devel@oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel
^ permalink raw reply [flat|nested] 3+ messages in thread* Re: [Ocfs2-devel] [PATCH] fix kernel crash after mounting when dlm doesn't ready
2022-03-10 3:59 ` Joseph Qi via Ocfs2-devel
@ 2022-03-10 8:16 ` heming.zhao--- via Ocfs2-devel
0 siblings, 0 replies; 3+ messages in thread
From: heming.zhao--- via Ocfs2-devel @ 2022-03-10 8:16 UTC (permalink / raw)
To: Joseph Qi, ocfs2-devel
Hello Joseph,
Thank you for your review.
On 3/10/22 11:59, Joseph Qi wrote:
> It looks this issue is introduced by the following commit:
> da5e7c87827e ocfs2: cleanup journal init and shutdown
>
> Before that, journal was initialized in ocfs2_initialize_super().
I agree, and will add Fixes in V2.
>
> On 3/9/22 9:30 PM, Heming Zhao wrote:
>> How to trigger:
>>
>> tb-ocfs1 # dd if=/dev/zero of=/dev/vdb bs=1M count=20 oflag=direct
>> tb-ocfs1 # mkfs.ocfs2 --cluster-stack=pcmk -N 4 /dev/vdb \
>> --cluster-name=ocfstst
>> == there is no dlm running ==
>> tb-ocfs1 # mount -t ocfs2 /dev/vdb /mnt
>> == kernel crash ==
>>
>> Crash log
>>
>> ```
>> kernel: DLM installed
>> kernel: ocfs2: Registered cluster interface user
>> kernel: dlm: no local IP address has been set
>> kernel: dlm: cannot start dlm lowcomms -107
>> kernel: (mount.ocfs2,2225,0):ocfs2_dlm_init:3355 ERROR: status = -107
>> kernel: (mount.ocfs2,2225,0):ocfs2_mount_volume:1817 ERROR: status = -107
>> kernel: (mount.ocfs2,2225,0):ocfs2_fill_super:1186 ERROR: status = -107
>> kernel: BUG: kernel NULL pointer dereference, address: 0000000000000030
>> kernel: #PF: supervisor read access in kernel mode
>> kernel: #PF: error_code(0x0000) - not-present page
>> kernel: PGD 0 P4D 0
>> kernel: Oops: 0000 [#1] PREEMPT SMP NOPTI
>> kernel: CPU: 0 PID: 2225 Comm: mount.ocfs2 Not tainted 5.16.2-1-default #1 openSUSE Tumbleweed b40a195b7ff0f3399a616c3290f963c4ad189e84
>> kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.14.0-0-g155821a-rebuilt.opensuse.org 04/01/2014
>> kernel: RIP: 0010:ocfs2_clear_inode+0x2e9/0x720 [ocfs2]
>> kernel: Code: 24 e8 9b 68 04 00 48 c7 c7 70 88 db c0 48 8b 80 98 03 00
>> 00 48 8b 80 70 01 00 00 48 89 44 24 08 e8 3c 50 d8 d8 48 8b 44 24 08
>> <48> 8b 48 30 49 39 4f c8 0f 88 ff 00 00 00 48 c>
>> kernel: RSP: 0018:ffffbbf000847bf0 EFLAGS: 00010246
>> kernel: RAX: 0000000000000000 RBX: ffff95f6834b8000 RCX: 0000000000000000
>> kernel: RDX: 0000000000000001 RSI: ffff95f6a1fbcbe0 RDI: ffffffffc0db8870
>> kernel: RBP: ffff95f6a1fbc6b8 R08: 00000ab5a9371b7a R09: 0000000000000230
>> kernel: R10: ffffbbf000847bc0 R11: ffffffffc0d53ea0 R12: ffff95f6a1fbc560
>> kernel: R13: ffff95f6a1fbc408 R14: ffff95f6834b8000 R15: ffff95f6a1fbc908
>> kernel: FS: 00007f366f151740(0000) GS:ffff95f6fdc00000(0000) knlGS:0000000000000000
>> kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> kernel: CR2: 0000000000000030 CR3: 0000000003fcc004 CR4: 0000000000370ef0
>> kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
>> kernel: Call Trace:
>> kernel: <TASK>
>> kernel: ? ocfs2_evict_inode+0x1fe/0x630 [ocfs2 411bc..281]
>> kernel: evict+0xc0/0x1c0
>> kernel: ocfs2_release_system_inodes+0x21/0xc0 [ocfs2 411bc..281]
>> kernel: ocfs2_dismount_volume+0x10b/0x2d0 [ocfs2 411bc..281]
>> kernel: ocfs2_fill_super+0xaf/0x19e0 [ocfs2 411bc..281]
>> kernel: mount_bdev+0x182/0x1b0
>> kernel: ? ocfs2_initialize_super.isra.0+0xf50/0xf50 [ocfs2 411bc..281]
>> kernel: legacy_get_tree+0x24/0x40
>> kernel: vfs_get_tree+0x22/0xb0
>> kernel: path_mount+0x465/0xac0
>> kernel: __x64_sys_mount+0x103/0x140
>> kernel: do_syscall_64+0x59/0x80
>> kernel: ? syscall_exit_to_user_mode+0x18/0x40
>> kernel: ? do_syscall_64+0x69/0x80
>> kernel: ? syscall_exit_to_user_mode+0x18/0x40
>> kernel: ? do_syscall_64+0x69/0x80
>> kernel: ? exc_page_fault+0x68/0x150
>> kernel: entry_SYSCALL_64_after_hwframe+0x44/0xae
>> ```
>>
>> Analysis:
>>
>> ocfs2_fill_super
>> ocfs2_mount_volume
>> ocfs2_dlm_init //failed, journal still doesn't be initized.
>> goto read_super_error
> This is not a part of call trace, I don't think it is needed here.
OK, will drop in v2.
>
>> ocfs2_dismount_volume
>> ocfs2_release_system_inodes
>> ...
>> evict
>> ...
>> ocfs2_clear_inode
>> ocfs2_checkpoint_inode
>> ocfs2_ci_fully_checkpointed
>> time_after(journal->j_trans_id, ci->ci_last_trans)
>> + journal is empty, crash!
>>
>
> Suggest we use the following way to description this commit:
> <call trace>
> <reproducer>
> <your analysis and how to fix>
OK, will follow this style.
>
> BTW, as I mentioned at the first, a fixes tag should be mentioned here.
> Could you also please check all possible use of journal during
> ocfs2_dismount_volume()?
I verified my patch before file V1, there was no crash anymore with no dlm case.
Under your comment, there may have other places need to fix in ocfs2_dismount_volume().
The reverse operations in ocfs2_dismount_volume() for range [ocfs2_initialize_super, \
ocfs2_journal_init) should be checked.
>
> Thanks,
> Joseph
>
Thanks,
Heming
_______________________________________________
Ocfs2-devel mailing list
Ocfs2-devel@oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2022-03-10 8:16 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-03-09 13:30 [Ocfs2-devel] [PATCH] fix kernel crash after mounting when dlm doesn't ready Heming Zhao via Ocfs2-devel
2022-03-10 3:59 ` Joseph Qi via Ocfs2-devel
2022-03-10 8:16 ` heming.zhao--- via Ocfs2-devel
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).