* linux 4.1-rc7 deadlock @ 2015-06-09 11:54 Morten Stevens 2015-06-09 14:10 ` Daniel Wagner 0 siblings, 1 reply; 6+ messages in thread From: Morten Stevens @ 2015-06-09 11:54 UTC (permalink / raw) To: linux-kernel Hi, After each restart appears directly this deadlock: [ 28.177939] ====================================================== [ 28.177959] [ INFO: possible circular locking dependency detected ] [ 28.177980] 4.1.0-0.rc7.git0.1.fc23.x86_64+debug #1 Tainted: G W [ 28.178002] ------------------------------------------------------- [ 28.178022] sshd/1764 is trying to acquire lock: [ 28.178037] (&isec->lock){+.+.+.}, at: [<ffffffff813b52c5>] inode_doinit_with_dentry+0xc5/0x6a0 [ 28.178078] but task is already holding lock: [ 28.178097] (&mm->mmap_sem){++++++}, at: [<ffffffff81216a0f>] vm_mmap_pgoff+0x8f/0xf0 [ 28.178131] which lock already depends on the new lock. [ 28.178157] the existing dependency chain (in reverse order) is: [ 28.178180] -> #2 (&mm->mmap_sem){++++++}: [ 28.178201] [<ffffffff81114017>] lock_acquire+0xc7/0x2a0 [ 28.178225] [<ffffffff8122853c>] might_fault+0x8c/0xb0 [ 28.178248] [<ffffffff8129af3a>] filldir+0x9a/0x130 [ 28.178269] [<ffffffffa019cfd6>] xfs_dir2_block_getdents.isra.12+0x1a6/0x1d0 [xfs] [ 28.178330] [<ffffffffa019dae4>] xfs_readdir+0x1c4/0x360 [xfs] [ 28.178368] [<ffffffffa01a0a5b>] xfs_file_readdir+0x2b/0x30 [xfs] [ 28.178404] [<ffffffff8129ad0a>] iterate_dir+0x9a/0x140 [ 28.178425] [<ffffffff8129b241>] SyS_getdents+0x91/0x120 [ 28.178447] [<ffffffff818a016e>] system_call_fastpath+0x12/0x76 [ 28.178471] -> #1 (&xfs_dir_ilock_class){++++.+}: [ 28.178494] [<ffffffff81114017>] lock_acquire+0xc7/0x2a0 [ 28.178515] [<ffffffff8110bee7>] down_read_nested+0x57/0xa0 [ 28.178538] [<ffffffffa01b2ed1>] xfs_ilock+0x171/0x390 [xfs] [ 28.178579] [<ffffffffa01b3168>] xfs_ilock_attr_map_shared+0x38/0x50 [xfs] [ 28.178618] [<ffffffffa0145d8d>] xfs_attr_get+0xbd/0x1b0 [xfs] [ 28.178651] [<ffffffffa01c44ad>] xfs_xattr_get+0x3d/0x80 [xfs] [ 28.178688] [<ffffffff812b022f>] generic_getxattr+0x4f/0x70 [ 28.178711] [<ffffffff813b5372>] inode_doinit_with_dentry+0x172/0x6a0 [ 28.178737] [<ffffffff813b68db>] sb_finish_set_opts+0xdb/0x260 [ 28.178759] [<ffffffff813b6ff1>] selinux_set_mnt_opts+0x331/0x670 [ 28.178783] [<ffffffff813b9b47>] superblock_doinit+0x77/0xf0 [ 28.178804] [<ffffffff813b9bd0>] delayed_superblock_init+0x10/0x20 [ 28.178849] [<ffffffff8128691a>] iterate_supers+0xba/0x120 [ 28.178872] [<ffffffff813bef23>] selinux_complete_init+0x33/0x40 [ 28.178897] [<ffffffff813cf313>] security_load_policy+0x103/0x640 [ 28.178920] [<ffffffff813c0a76>] sel_write_load+0xb6/0x790 [ 28.179482] [<ffffffff812821f7>] __vfs_write+0x37/0x110 [ 28.180047] [<ffffffff81282c89>] vfs_write+0xa9/0x1c0 [ 28.180630] [<ffffffff81283a1c>] SyS_write+0x5c/0xd0 [ 28.181168] [<ffffffff818a016e>] system_call_fastpath+0x12/0x76 [ 28.181740] -> #0 (&isec->lock){+.+.+.}: [ 28.182808] [<ffffffff81113331>] __lock_acquire+0x1b31/0x1e40 [ 28.183347] [<ffffffff81114017>] lock_acquire+0xc7/0x2a0 [ 28.183897] [<ffffffff8189c10d>] mutex_lock_nested+0x7d/0x460 [ 28.184427] [<ffffffff813b52c5>] inode_doinit_with_dentry+0xc5/0x6a0 [ 28.184944] [<ffffffff813b58bc>] selinux_d_instantiate+0x1c/0x20 [ 28.185470] [<ffffffff813b07ab>] security_d_instantiate+0x1b/0x30 [ 28.185980] [<ffffffff8129e8c4>] d_instantiate+0x54/0x80 [ 28.186495] [<ffffffff81211edc>] __shmem_file_setup+0xdc/0x250 [ 28.186990] [<ffffffff812164a8>] shmem_zero_setup+0x28/0x70 [ 28.187500] [<ffffffff8123471c>] mmap_region+0x66c/0x680 [ 28.188006] [<ffffffff81234a53>] do_mmap_pgoff+0x323/0x410 [ 28.188500] [<ffffffff81216a30>] vm_mmap_pgoff+0xb0/0xf0 [ 28.189005] [<ffffffff81232bf6>] SyS_mmap_pgoff+0x116/0x2b0 [ 28.189490] [<ffffffff810232bb>] SyS_mmap+0x1b/0x30 [ 28.189975] [<ffffffff818a016e>] system_call_fastpath+0x12/0x76 [ 28.190474] other info that might help us debug this: [ 28.191901] Chain exists of: &isec->lock --> &xfs_dir_ilock_class --> &mm->mmap_sem [ 28.193327] Possible unsafe locking scenario: [ 28.194297] CPU0 CPU1 [ 28.194774] ---- ---- [ 28.195254] lock(&mm->mmap_sem); [ 28.195709] lock(&xfs_dir_ilock_class); [ 28.196174] lock(&mm->mmap_sem); [ 28.196654] lock(&isec->lock); [ 28.197108] *** DEADLOCK *** [ 28.198451] 1 lock held by sshd/1764: [ 28.198900] #0: (&mm->mmap_sem){++++++}, at: [<ffffffff81216a0f>] vm_mmap_pgoff+0x8f/0xf0 [ 28.199370] stack backtrace: [ 28.200276] CPU: 2 PID: 1764 Comm: sshd Tainted: G W 4.1.0-0.rc7.git0.1.fc23.x86_64+debug #1 [ 28.200753] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/20/2014 [ 28.201246] 0000000000000000 00000000eda89a94 ffff8800a86a39c8 ffffffff81896375 [ 28.201771] 0000000000000000 ffffffff82a910d0 ffff8800a86a3a18 ffffffff8110fbd6 [ 28.202275] 0000000000000002 ffff8800a86a3a78 0000000000000001 ffff8800a897b008 [ 28.203099] Call Trace: [ 28.204237] [<ffffffff81896375>] dump_stack+0x4c/0x65 [ 28.205362] [<ffffffff8110fbd6>] print_circular_bug+0x206/0x280 [ 28.206502] [<ffffffff81113331>] __lock_acquire+0x1b31/0x1e40 [ 28.207650] [<ffffffff81114017>] lock_acquire+0xc7/0x2a0 [ 28.208758] [<ffffffff813b52c5>] ? inode_doinit_with_dentry+0xc5/0x6a0 [ 28.209902] [<ffffffff8189c10d>] mutex_lock_nested+0x7d/0x460 [ 28.211023] [<ffffffff813b52c5>] ? inode_doinit_with_dentry+0xc5/0x6a0 [ 28.212162] [<ffffffff813b52c5>] ? inode_doinit_with_dentry+0xc5/0x6a0 [ 28.213283] [<ffffffff81027e7d>] ? native_sched_clock+0x2d/0xa0 [ 28.214403] [<ffffffff81027ef9>] ? sched_clock+0x9/0x10 [ 28.215514] [<ffffffff813b52c5>] inode_doinit_with_dentry+0xc5/0x6a0 [ 28.216656] [<ffffffff813b58bc>] selinux_d_instantiate+0x1c/0x20 [ 28.217776] [<ffffffff813b07ab>] security_d_instantiate+0x1b/0x30 [ 28.218902] [<ffffffff8129e8c4>] d_instantiate+0x54/0x80 [ 28.219992] [<ffffffff81211edc>] __shmem_file_setup+0xdc/0x250 [ 28.221112] [<ffffffff812164a8>] shmem_zero_setup+0x28/0x70 [ 28.222234] [<ffffffff8123471c>] mmap_region+0x66c/0x680 [ 28.223362] [<ffffffff81234a53>] do_mmap_pgoff+0x323/0x410 [ 28.224493] [<ffffffff81216a0f>] ? vm_mmap_pgoff+0x8f/0xf0 [ 28.225643] [<ffffffff81216a30>] vm_mmap_pgoff+0xb0/0xf0 [ 28.226771] [<ffffffff81232bf6>] SyS_mmap_pgoff+0x116/0x2b0 [ 28.227900] [<ffffffff812996ce>] ? SyS_fcntl+0x5de/0x760 [ 28.229042] [<ffffffff810232bb>] SyS_mmap+0x1b/0x30 [ 28.230156] [<ffffffff818a016e>] system_call_fastpath+0x12/0x76 [ 46.520367] Adjusting tsc more than 11% (5419175 vs 7179037) Any ideas? Best regards, Morten ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: linux 4.1-rc7 deadlock 2015-06-09 11:54 linux 4.1-rc7 deadlock Morten Stevens @ 2015-06-09 14:10 ` Daniel Wagner 2015-06-09 15:35 ` Morten Stevens 0 siblings, 1 reply; 6+ messages in thread From: Daniel Wagner @ 2015-06-09 14:10 UTC (permalink / raw) To: Morten Stevens, linux-kernel Hi Morten, On 06/09/2015 01:54 PM, Morten Stevens wrote: > [ 28.193327] Possible unsafe locking scenario: > > [ 28.194297] CPU0 CPU1 > [ 28.194774] ---- ---- > [ 28.195254] lock(&mm->mmap_sem); > [ 28.195709] lock(&xfs_dir_ilock_class); > [ 28.196174] lock(&mm->mmap_sem); > [ 28.196654] lock(&isec->lock); > [ 28.197108] [...] > Any ideas? I think you hit the same problem many have already reported: https://lkml.org/lkml/2015/3/30/594 cheers, daniel ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: linux 4.1-rc7 deadlock 2015-06-09 14:10 ` Daniel Wagner @ 2015-06-09 15:35 ` Morten Stevens 2015-06-11 20:06 ` Hugh Dickins 0 siblings, 1 reply; 6+ messages in thread From: Morten Stevens @ 2015-06-09 15:35 UTC (permalink / raw) To: Daniel Wagner; +Cc: linux-kernel 2015-06-09 16:10 GMT+02:00 Daniel Wagner <wagi@monom.org>: > Hi Morten, Hi Daniel, > On 06/09/2015 01:54 PM, Morten Stevens wrote: >> [ 28.193327] Possible unsafe locking scenario: >> >> [ 28.194297] CPU0 CPU1 >> [ 28.194774] ---- ---- >> [ 28.195254] lock(&mm->mmap_sem); >> [ 28.195709] lock(&xfs_dir_ilock_class); >> [ 28.196174] lock(&mm->mmap_sem); >> [ 28.196654] lock(&isec->lock); >> [ 28.197108] > > [...] > >> Any ideas? > > I think you hit the same problem many have already reported: > > https://lkml.org/lkml/2015/3/30/594 Yes, that sounds very likely. But that was about 1 month ago, so I thought that it has been fixed in the last weeks? Best regards, Morten ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: linux 4.1-rc7 deadlock 2015-06-09 15:35 ` Morten Stevens @ 2015-06-11 20:06 ` Hugh Dickins 2015-06-11 22:15 ` Prarit Bhargava 0 siblings, 1 reply; 6+ messages in thread From: Hugh Dickins @ 2015-06-11 20:06 UTC (permalink / raw) To: Morten Stevens Cc: Daniel Wagner, Prarit Bhargava, Dave Chinner, Eric Paris, Eric Sandeen, Andrew Morton, linux-mm, linux-kernel On Tue, 9 Jun 2015, Morten Stevens wrote: > 2015-06-09 16:10 GMT+02:00 Daniel Wagner <wagi@monom.org>: > > On 06/09/2015 01:54 PM, Morten Stevens wrote: > >> [ 28.193327] Possible unsafe locking scenario: > >> > >> [ 28.194297] CPU0 CPU1 > >> [ 28.194774] ---- ---- > >> [ 28.195254] lock(&mm->mmap_sem); > >> [ 28.195709] lock(&xfs_dir_ilock_class); > >> [ 28.196174] lock(&mm->mmap_sem); > >> [ 28.196654] lock(&isec->lock); > >> [ 28.197108] > > > > [...] > > > >> Any ideas? > > > > I think you hit the same problem many have already reported: > > > > https://lkml.org/lkml/2015/3/30/594 > > Yes, that sounds very likely. But that was about 1 month ago, so I > thought that it has been fixed in the last weeks? It's not likely to get fixed without Cc'ing the right people. This appears to be the same as Prarit reported to linux-mm on 2014/09/10. Dave Chinner thinks it's a shmem bug, I disagree, but I am hopeful that it can be easily fixed at the shmem end. Here's the patch I suggested nine months ago: but got no feedback, so it remains Not-Yet-Signed-off-by. Please, if you find this works (and does not just delay the lockdep conflict until a little later), do let me know, then I can add some Tested-bys and send it to Linus. mm: shmem_zero_setup skip security check and lockdep conflict with XFS It appears that, at some point last year, XFS made directory handling changes which bring it into lockdep conflict with shmem_zero_setup(): it is surprising that mmap() can clone an inode while holding mmap_sem, but that has been so for many years. Since those few lockdep traces that I've seen all implicated selinux, I'm hoping that we can use the __shmem_file_setup(,,,S_PRIVATE) which v3.13's commit c7277090927a ("security: shmem: implement kernel private shmem inodes") introduced to avoid LSM checks on kernel-internal inodes: the mmap("/dev/zero") cloned inode is indeed a kernel-internal detail. This also covers the !CONFIG_SHMEM use of ramfs to support /dev/zero (and MAP_SHARED|MAP_ANONYMOUS). I thought there were also drivers which cloned inode in mmap(), but if so, I cannot locate them now. Reported-by: Prarit Bhargava <prarit@redhat.com> Reported-by: Daniel Wagner <wagi@monom.org> Reported-by: Morten Stevens <mstevens@fedoraproject.org> Not-Yet-Signed-off-by: Hugh Dickins <hughd@google.com> --- mm/shmem.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) --- 4.1-rc7/mm/shmem.c 2015-04-26 19:16:31.352191298 -0700 +++ linux/mm/shmem.c 2015-06-11 11:08:21.042745594 -0700 @@ -3401,7 +3401,7 @@ int shmem_zero_setup(struct vm_area_stru struct file *file; loff_t size = vma->vm_end - vma->vm_start; - file = shmem_file_setup("dev/zero", size, vma->vm_flags); + file = __shmem_file_setup("dev/zero", size, vma->vm_flags, S_PRIVATE); if (IS_ERR(file)) return PTR_ERR(file); ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: linux 4.1-rc7 deadlock 2015-06-11 20:06 ` Hugh Dickins @ 2015-06-11 22:15 ` Prarit Bhargava 2015-06-14 16:44 ` Hugh Dickins 0 siblings, 1 reply; 6+ messages in thread From: Prarit Bhargava @ 2015-06-11 22:15 UTC (permalink / raw) To: Hugh Dickins Cc: Morten Stevens, Daniel Wagner, Dave Chinner, Eric Paris, Eric Sandeen, Andrew Morton, linux-mm, linux-kernel On 06/11/2015 04:06 PM, Hugh Dickins wrote: > On Tue, 9 Jun 2015, Morten Stevens wrote: >> 2015-06-09 16:10 GMT+02:00 Daniel Wagner <wagi@monom.org>: >>> On 06/09/2015 01:54 PM, Morten Stevens wrote: >>>> [ 28.193327] Possible unsafe locking scenario: >>>> >>>> [ 28.194297] CPU0 CPU1 >>>> [ 28.194774] ---- ---- >>>> [ 28.195254] lock(&mm->mmap_sem); >>>> [ 28.195709] lock(&xfs_dir_ilock_class); >>>> [ 28.196174] lock(&mm->mmap_sem); >>>> [ 28.196654] lock(&isec->lock); >>>> [ 28.197108] >>> >>> [...] >>> >>>> Any ideas? >>> >>> I think you hit the same problem many have already reported: >>> >>> https://lkml.org/lkml/2015/3/30/594 >> >> Yes, that sounds very likely. But that was about 1 month ago, so I >> thought that it has been fixed in the last weeks? > > It's not likely to get fixed without Cc'ing the right people. > > This appears to be the same as Prarit reported to linux-mm on > 2014/09/10. Dave Chinner thinks it's a shmem bug, I disagree, > but I am hopeful that it can be easily fixed at the shmem end. > > Here's the patch I suggested nine months ago: but got no feedback, > so it remains Not-Yet-Signed-off-by. Please, if you find this works > (and does not just delay the lockdep conflict until a little later), > do let me know, then I can add some Tested-bys and send it to Linus. > > mm: shmem_zero_setup skip security check and lockdep conflict with XFS > > It appears that, at some point last year, XFS made directory handling > changes which bring it into lockdep conflict with shmem_zero_setup(): > it is surprising that mmap() can clone an inode while holding mmap_sem, > but that has been so for many years. > > Since those few lockdep traces that I've seen all implicated selinux, > I'm hoping that we can use the __shmem_file_setup(,,,S_PRIVATE) which > v3.13's commit c7277090927a ("security: shmem: implement kernel private > shmem inodes") introduced to avoid LSM checks on kernel-internal inodes: > the mmap("/dev/zero") cloned inode is indeed a kernel-internal detail. > > This also covers the !CONFIG_SHMEM use of ramfs to support /dev/zero > (and MAP_SHARED|MAP_ANONYMOUS). I thought there were also drivers > which cloned inode in mmap(), but if so, I cannot locate them now. > > Reported-by: Prarit Bhargava <prarit@redhat.com> > Reported-by: Daniel Wagner <wagi@monom.org> > Reported-by: Morten Stevens <mstevens@fedoraproject.org> > Not-Yet-Signed-off-by: Hugh Dickins <hughd@google.com> > --- > > mm/shmem.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > --- 4.1-rc7/mm/shmem.c 2015-04-26 19:16:31.352191298 -0700 > +++ linux/mm/shmem.c 2015-06-11 11:08:21.042745594 -0700 > @@ -3401,7 +3401,7 @@ int shmem_zero_setup(struct vm_area_stru > struct file *file; > loff_t size = vma->vm_end - vma->vm_start; > > - file = shmem_file_setup("dev/zero", size, vma->vm_flags); > + file = __shmem_file_setup("dev/zero", size, vma->vm_flags, S_PRIVATE); Perhaps, file = shmem_kernel_file_setup("dev/zero", size, vma->vm_flags) ? Tested-by: Prarit Bhargava <prarit@redhat.com> P. > if (IS_ERR(file)) > return PTR_ERR(file); > > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: linux 4.1-rc7 deadlock 2015-06-11 22:15 ` Prarit Bhargava @ 2015-06-14 16:44 ` Hugh Dickins 0 siblings, 0 replies; 6+ messages in thread From: Hugh Dickins @ 2015-06-14 16:44 UTC (permalink / raw) To: Prarit Bhargava Cc: Hugh Dickins, Morten Stevens, Daniel Wagner, Dave Chinner, Eric Paris, Eric Sandeen, Andrew Morton, linux-mm, linux-kernel On Thu, 11 Jun 2015, Prarit Bhargava wrote: > On 06/11/2015 04:06 PM, Hugh Dickins wrote: > > On Tue, 9 Jun 2015, Morten Stevens wrote: > >> 2015-06-09 16:10 GMT+02:00 Daniel Wagner <wagi@monom.org>: > >>> On 06/09/2015 01:54 PM, Morten Stevens wrote: > > > > Reported-by: Prarit Bhargava <prarit@redhat.com> > > Reported-by: Daniel Wagner <wagi@monom.org> > > Reported-by: Morten Stevens <mstevens@fedoraproject.org> > > Not-Yet-Signed-off-by: Hugh Dickins <hughd@google.com> > > --- > > > > mm/shmem.c | 2 +- > > 1 file changed, 1 insertion(+), 1 deletion(-) > > > > --- 4.1-rc7/mm/shmem.c 2015-04-26 19:16:31.352191298 -0700 > > +++ linux/mm/shmem.c 2015-06-11 11:08:21.042745594 -0700 > > @@ -3401,7 +3401,7 @@ int shmem_zero_setup(struct vm_area_stru > > struct file *file; > > loff_t size = vma->vm_end - vma->vm_start; > > > > - file = shmem_file_setup("dev/zero", size, vma->vm_flags); > > + file = __shmem_file_setup("dev/zero", size, vma->vm_flags, S_PRIVATE); > > Perhaps, > > file = shmem_kernel_file_setup("dev/zero", size, vma->vm_flags) ? Perhaps. I couldn't decide whether this is a proper intended use of shmem_kernel_file_setup(), or a handy reuse of its flag. Andrew asked for a comment, so in the end I left that line as is, but refer to shmem_kernel_file_setup() in the comment. And that forced me to look a little closer at the security implications: but we do seem to be safe. > > Tested-by: Prarit Bhargava <prarit@redhat.com> Thank you: I had been hoping for some corroboration from one of the other guys (no offence to you, but 33% looks a bit weak!), but now it's Sunday so I think I'd better send this off in the hope that it makes -rc8. Hugh > > P. > > > if (IS_ERR(file)) > > return PTR_ERR(file); ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2015-06-14 16:44 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2015-06-09 11:54 linux 4.1-rc7 deadlock Morten Stevens 2015-06-09 14:10 ` Daniel Wagner 2015-06-09 15:35 ` Morten Stevens 2015-06-11 20:06 ` Hugh Dickins 2015-06-11 22:15 ` Prarit Bhargava 2015-06-14 16:44 ` Hugh Dickins
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).