From: Tuomas Tynkkynen <tuomas@tuxera.com>
To: <v9fs-developer@lists.sourceforge.net>,
<linux-fsdevel@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
Al Viro <viro@zeniv.linux.org.uk>
Subject: Re: 9pfs hangs since 4.7
Date: Mon, 2 Jan 2017 10:20:35 +0200 [thread overview]
Message-ID: <20170102102035.7d1cf903@duuni> (raw)
In-Reply-To: <20161124215023.02deb03c@duuni>
Hi fsdevel,
I tracked this problem down a bit and it seems that it started happening
after the VFS merge in 4.7-rc1: 7f427d3a6029331304f91ef4d7cf646f054216d2:
Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs
Pull parallel filesystem directory handling update from Al Viro.
Al, do you have any ideas about this problem with 9pfs I've been observing
(quoted below in full)? Many thanks in advance!
On Thu, 24 Nov 2016 21:50:23 +0200
Tuomas Tynkkynen <tuomas@tuxera.com> wrote:
> Hi fsdevel,
>
> I have been observing hangs when running xfstests generic/224. Curiously
> enough, the test is *not* causing problems on the FS under test (I've
> tried both ext4 and f2fs) but instead it's causing the 9pfs that I'm
> using as the root filesystem to crap out.
>
> How it shows up is that the test doesn't finish in time (usually
> takes ~50 sec) but the hung task detector triggers for some task in
> d_alloc_parallel():
>
> [ 660.701646] INFO: task 224:7800 blocked for more than 300 seconds.
> [ 660.702756] Not tainted 4.9.0-rc5 #1-NixOS
> [ 660.703232] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [ 660.703927] 224 D 0 7800 549 0x00000000
> [ 660.704501] ffff8a82ec022800 0000000000000000 ffff8a82fc03c800 ffff8a82ff217dc0
> [ 660.705302] ffff8a82d0f88c00 ffffa94a41a27b88 ffffffffaeb4ad1d ffffa94a41a27b78
> [ 660.706125] ffffffffae800fc6 ffff8a82fbd90f08 ffff8a82d0f88c00 ffff8a82fbfd5418
> [ 660.706924] Call Trace:
> [ 660.707185] [<ffffffffaeb4ad1d>] ? __schedule+0x18d/0x640
> [ 660.707751] [<ffffffffae800fc6>] ? __d_alloc+0x126/0x1e0
> [ 660.708304] [<ffffffffaeb4b206>] schedule+0x36/0x80
> [ 660.708841] [<ffffffffae801937>] d_alloc_parallel+0x3a7/0x480
> [ 660.709454] [<ffffffffae697800>] ? wake_up_q+0x70/0x70
> [ 660.710007] [<ffffffffae7f37e3>] lookup_slow+0x73/0x140
> [ 660.710572] [<ffffffffae7f3eea>] walk_component+0x1ca/0x2f0
> [ 660.711167] [<ffffffffae7f2409>] ? path_init+0x1d9/0x330
> [ 660.711747] [<ffffffffae808ca4>] ? mntput+0x24/0x40
> [ 660.716962] [<ffffffffae7f4fbd>] path_lookupat+0x5d/0x110
> [ 660.717581] [<ffffffffae7f784e>] filename_lookup+0x9e/0x150
> [ 660.718194] [<ffffffffae7ca036>] ? kmem_cache_alloc+0x156/0x1b0
> [ 660.719037] [<ffffffffae7f7496>] ? getname_flags+0x56/0x1f0
> [ 660.719801] [<ffffffffae7f74b2>] ? getname_flags+0x72/0x1f0
> [ 660.720492] [<ffffffffae7f79b6>] user_path_at_empty+0x36/0x40
> [ 660.721206] [<ffffffffae7ec943>] vfs_fstatat+0x53/0xa0
> [ 660.721980] [<ffffffffae7ecdbf>] SYSC_newstat+0x1f/0x40
> [ 660.722732] [<ffffffffae7ecebe>] SyS_newstat+0xe/0x10
> [ 660.723702] [<ffffffffaeb4fb77>] entry_SYSCALL_64_fastpath+0x1a/0xa9
>
> SysRq-T is full of things stuck inside p9_client_rpc like:
>
> [ 271.703598] bash S 0 100 96 0x00000000
> [ 271.703968] ffff8a82ff824800 0000000000000000 ffff8a82faee4800 ffff8a82ff217dc0
> [ 271.704486] ffff8a82fb946c00 ffffa94a404ebae8 ffffffffaeb4ad1d ffff8a82fb9fc058
> [ 271.705024] ffffa94a404ebb10 ffffffffae8f21f9 ffff8a82fb946c00 ffff8a82fbbba000
> [ 271.705542] Call Trace:
> [ 271.705715] [<ffffffffaeb4ad1d>] ? __schedule+0x18d/0x640
> [ 271.706079] [<ffffffffae8f21f9>] ? idr_get_empty_slot+0x199/0x3b0
> [ 271.706489] [<ffffffffaeb4b206>] schedule+0x36/0x80
> [ 271.706825] [<ffffffffc01b02ba>] p9_client_rpc+0x12a/0x460 [9pnet]
> [ 271.707239] [<ffffffffae8f2497>] ? idr_alloc+0x87/0x100
> [ 271.707596] [<ffffffffae6abf90>] ? wake_atomic_t_function+0x60/0x60
> [ 271.708043] [<ffffffffc01b2247>] p9_client_walk+0x77/0x200 [9pnet]
> [ 271.708459] [<ffffffffc01ccae9>] v9fs_vfs_lookup.part.16+0x59/0x120 [9p]
> [ 271.708912] [<ffffffffc01ccbcf>] v9fs_vfs_lookup+0x1f/0x30 [9p]
> [ 271.709308] [<ffffffffae7f3806>] lookup_slow+0x96/0x140
> [ 271.709664] [<ffffffffae7f3eea>] walk_component+0x1ca/0x2f0
> [ 271.710036] [<ffffffffae7f2409>] ? path_init+0x1d9/0x330
> [ 271.710390] [<ffffffffae7f4fbd>] path_lookupat+0x5d/0x110
> [ 271.710763] [<ffffffffae7f784e>] filename_lookup+0x9e/0x150
> [ 271.711136] [<ffffffffae7e091e>] ? mem_cgroup_commit_charge+0x7e/0x4a0
> [ 271.711581] [<ffffffffae7ca036>] ? kmem_cache_alloc+0x156/0x1b0
> [ 271.711977] [<ffffffffae7f7496>] ? getname_flags+0x56/0x1f0
> [ 271.712349] [<ffffffffae7f74b2>] ? getname_flags+0x72/0x1f0
> [ 271.712726] [<ffffffffae7f79b6>] user_path_at_empty+0x36/0x40
> [ 271.713110] [<ffffffffae7ec943>] vfs_fstatat+0x53/0xa0
> [ 271.713454] [<ffffffffae7ecdbf>] SYSC_newstat+0x1f/0x40
> [ 271.713810] [<ffffffffae7ecebe>] SyS_newstat+0xe/0x10
> [ 271.714150] [<ffffffffaeb4fb77>] entry_SYSCALL_64_fastpath+0x1a/0xa9
>
> [ 271.729022] sleep S 0 218 216 0x00000002
> [ 271.729391] ffff8a82fb990800 0000000000000000 ffff8a82fc0d8000 ffff8a82ff317dc0
> [ 271.729915] ffff8a82fbbec800 ffffa94a404f3cf8 ffffffffaeb4ad1d ffff8a82fb9fc058
> [ 271.730426] ffffec95c1ee08c0 0000000000000001 ffff8a82fbbec800 ffff8a82fbbba000
> [ 271.730950] Call Trace:
> [ 271.731115] [<ffffffffaeb4ad1d>] ? __schedule+0x18d/0x640
> [ 271.731479] [<ffffffffaeb4b206>] schedule+0x36/0x80
> [ 271.731814] [<ffffffffc01b02ba>] p9_client_rpc+0x12a/0x460 [9pnet]
> [ 271.732226] [<ffffffffae6abf90>] ? wake_atomic_t_function+0x60/0x60
> [ 271.732649] [<ffffffffc01b2158>] p9_client_clunk+0x38/0xb0 [9pnet]
> [ 271.733061] [<ffffffffc01cf38a>] v9fs_dir_release+0x1a/0x30 [9p]
> [ 271.733494] [<ffffffffae7e964f>] __fput+0xdf/0x1f0
> [ 271.733844] [<ffffffffae7e979e>] ____fput+0xe/0x10
> [ 271.734176] [<ffffffffae68b5be>] task_work_run+0x7e/0xa0
> [ 271.734532] [<ffffffffae672559>] do_exit+0x2b9/0xad0
> [ 271.734888] [<ffffffffae65cc67>] ? __do_page_fault+0x287/0x4b0
> [ 271.735276] [<ffffffffae672df3>] do_group_exit+0x43/0xb0
> [ 271.735639] [<ffffffffae672e74>] SyS_exit_group+0x14/0x20
> [ 271.736002] [<ffffffffaeb4fb77>] entry_SYSCALL_64_fastpath+0x1a/0xa9
>
> Full dmesgs is available from:
> https://gist.githubusercontent.com/dezgeg/31c2a50a1ce82e4284f6c9e617e7eba8/raw/e5ed6e62c7a1a5234d9316563154e530a2e95586/dmesg (shorter)
> https://gist.githubusercontent.com/dezgeg/6989a0746ba8c000324455f473dc58e9/raw/4d7c6c58de88ef9d0367147c4ef1d990cfb267ce/dmesg (longer)
>
> This typically reproduces quite fast, maybe half an hour or so. 4.7,
> 4.8.10 and 4.9-rc5 all are affected.
>
> This happens in a 2-core QEMU+KVM vm with 2GB RAM using its
> internal 9p server
> (-virtfs local,path=MOUNTPOINT,security_model=none,mount_tag=store).
>
> Any ideas for further debugging?
next prev parent reply other threads:[~2017-01-02 8:20 UTC|newest]
Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-11-24 19:50 9pfs hangs since 4.7 Tuomas Tynkkynen
2016-11-29 16:39 ` Eric Van Hensbergen
2016-12-02 20:11 ` Tuomas Tynkkynen
2017-01-02 8:20 ` Tuomas Tynkkynen [this message]
2017-01-02 16:23 ` Al Viro
2017-01-03 23:34 ` Tuomas Tynkkynen
2017-01-04 1:47 ` Al Viro
2017-01-04 20:04 ` Tuomas Tynkkynen
2017-01-04 23:01 ` Al Viro
2017-01-06 13:52 ` [V9fs-developer] " Greg Kurz
2017-01-07 6:26 ` Al Viro
2017-01-07 15:10 ` Greg Kurz
2017-01-07 17:19 ` Al Viro
2017-01-07 18:15 ` Al Viro
2017-01-08 5:46 ` Al Viro
2017-01-10 8:16 ` Greg Kurz
2017-01-09 18:29 ` Greg Kurz
2017-01-09 18:39 ` Tuomas Tynkkynen
2017-01-09 20:05 ` Al Viro
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20170102102035.7d1cf903@duuni \
--to=tuomas@tuxera.com \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=v9fs-developer@lists.sourceforge.net \
--cc=viro@zeniv.linux.org.uk \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).