From: Sargun Dhillon <sargun@sargun.me>
To: linux-fsdevel@vger.kernel.org, fuse-devel@lists.sourceforge.net
Subject: Deadlock when using FUSE with PID namespaces
Date: Tue, 26 Mar 2019 09:51:21 -0700 [thread overview]
Message-ID: <CAMp4zn8PuWwZm3d_Mjf+AMw8Z6Ghbt_5nTFKCJxcYr1ASiH8cQ@mail.gmail.com> (raw)
We have a FUSE daemon, and users that access them that run in a pid
namespace. When the pid namespace is torn down, we end up in a
situation where in userspace, a user gets stuck. Specifically, they
get stuck in request_wait_answer.
PID: 3655410 TASK: ffff8bc9f87ada00 CPU: 55 COMMAND: "mount.objective"
#0 [ffffb12820137ca8] __schedule at ffffffffb5aecf8d
#1 [ffffb12820137d40] schedule at ffffffffb5aed492
#2 [ffffb12820137d50] request_wait_answer at ffffffffb558ec1a
#3 [ffffb12820137da8] __fuse_request_send at ffffffffb558ff1f
#4 [ffffb12820137dc8] fuse_readdir at ffffffffb5594022
#5 [ffffb12820137e78] iterate_dir at ffffffffb548a1ae
#6 [ffffb12820137eb8] __x64_sys_getdents at ffffffffb548ad29
#7 [ffffb12820137f38] do_syscall_64 at ffffffffb5204185
#8 [ffffb12820137f50] entry_SYSCALL_64_after_hwframe at ffffffffb5c00088
RIP: 00007fdae2d888eb RSP: 00007fdadcc9fcf0 RFLAGS: 00000206
RAX: ffffffffffffffda RBX: 00007fda840008f0 RCX: 00007fdae2d888eb
RDX: 0000000000020000 RSI: 00007fda840008f0 RDI: 0000000000000040
RBP: 00007fda840008f0 R8: 00007fda84000000 R9: 0000000000000001
R10: 0000000000000000 R11: 0000000000000206 R12: 00007fdadcc9fe20
R13: 00007fda840008c0 R14: 0000000000000000 R15: 00007fdae3ba7250
ORIG_RAX: 000000000000004e CS: 0033 SS: 002b
I have a reproduction here:
https://github.com/sargun/fuse-example
Basically, what’s happening is that the same process is the fuse
daemon, and the filesystem client. There is a thread in that FUSE
daemon. let’s call the FUSE daemon P10, and the thread P11.
P10 wires up a FUSE filesystem on /tmp, it opens /dev/fuse with FD5.
P11 enacts an uninterruptible (blocking operation) on /tmp/foo, called
OP1, and /tmp/foo is FD6.
P10 reads the operation (OP1) from /dev/fuse, so now OP1 is in userspace
----The Pid 1 of the namespace is killed--
P10 just terminated, and cannot make progress. It will never respond
to OP1. FD5 and FD6 remain open, because P11 is in uninterruptible
sleep. P11 is in an uninterruptible disk sleep waiting for OP1 to
respond, and the fuse connection never aborts.
FUSE abortion doesn’t kick in because the FD of `/dev/fuse` that P10
originally opened as the FUSE daemon under FD5 will not be closed
until all threads as part of the process are terminated. The mount
namespace wont be torn down until P11 is torn down. P11 will never be
torn down because it’s waiting for someone to do something.
reply other threads:[~2019-03-26 16:52 UTC|newest]
Thread overview: [no followups] expand[flat|nested] mbox.gz Atom feed
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAMp4zn8PuWwZm3d_Mjf+AMw8Z6Ghbt_5nTFKCJxcYr1ASiH8cQ@mail.gmail.com \
--to=sargun@sargun.me \
--cc=fuse-devel@lists.sourceforge.net \
--cc=linux-fsdevel@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).