Stable Archive on lore.kernel.org
 help / color / Atom feed
From: Matthew Ruffell <matthew.ruffell@canonical.com>
To: stable@vger.kernel.org
Subject: [PATCH 4.9 0/3] fanotify: Fix notification subsystem hang
Date: Thu, 11 Apr 2019 15:24:27 +1200
Message-ID: <20190411032430.17353-1-matthew.ruffell@canonical.com> (raw)

BugLink: https://bugs.launchpad.net/bugs/1775165

[Note to upstream]
I understand that this patch is a little long for -stable, but this patch series
fixes a real issue, seen by real users, is testable, and is made up from
upstream commits. Please consider it.

[Impact]

When userspace tasks which are processing fanotify permission events act 
incorrectly, the fsnotify_mark_srcu SRCU is held indefinitely which causes
the whole notification subsystem to hang. 

This has been seen in production, and it can also be seen when running the 
Linux Test Project testsuite, specifically fanotify07. 

[Fix]

Instead of holding the SRCU lock while waiting for userspace to respond, 
which may never happen, or not in the order we are expecting, we drop the 
fsnotify_mark_srcu SRCU lock before waiting for userspace response, and then 
reacquire the lock again when userspace responds.

The fixes are from a series of upstream commits:

05f0e38724e8449184acd8fbf0473ee5a07adc6c (cherry-pick)
9385a84d7e1f658bb2d96ab798393e4b16268aaa (backport)
abc77577a669f424c5d0c185b9994f2621c52aa4 (backport)

[Testcase]

You can reproduce the problem pretty quickly with the Linux Test Project:

Steps (with root):
  1. sudo apt-get install git xfsprogs -y
  2. git clone --depth=1 https://github.com/linux-test-project/ltp.git
  3. cd ltp
  4. make autotools
  5. ./configure
  6. make; make install
  7. cd /opt/ltp
  8. echo -e "fanotify07 fanotify07 \nfanotify08 fanotify08" > /tmp/jobs
  9. ./runltp -f /tmp/jobs
  
On a stock 4.9.168 kernel, the system will hang, and the testcase will look like:

<<<test_start>>>
tag=fanotify07 stime=1554326200
cmdline="fanotify07 "
contacts=""
analysis=exit
<<<test_output>>>
tst_test.c:1096: INFO: Timeout per run is 0h 05m 00s
Test timeouted, sending SIGKILL!
Test timeouted, sending SIGKILL!
Test timeouted, sending SIGKILL!
Test timeouted, sending SIGKILL!
Test timeouted, sending SIGKILL!
Test timeouted, sending SIGKILL!
Test timeouted, sending SIGKILL!
Test timeouted, sending SIGKILL!
Test timeouted, sending SIGKILL!
Test timeouted, sending SIGKILL!
Test timeouted, sending SIGKILL!
Cannot kill test processes!
Congratulation, likely test hit a kernel bug.
Exitting uncleanly...
<<<execution_status>>>
initiation_status="ok"
duration=350 termination_type=exited termination_id=1 corefile=no
cutime=0 cstime=0
<<<test_end>>>

Looking at dmesg, we see the following call stack

[   41.648244] LTP: starting fanotify07 (fanotify07 )
[  242.729211] INFO: task fanotify07:1511 blocked for more than 120 seconds.
[  242.729257]       Not tainted 4.9.168vanilla #1
[  242.729281] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  242.729320] fanotify07      D    0  1511   1510 0x00000000
[  242.729325]  ffff92faf98a64c0 ffff92faf50f8000 ffff92faf19f2d00 ffff92faf989ad00
[  242.729329]  ffff92faffc19900 ffffb2f5c0cbbc70 ffffffffae8cf2c2 ffffb2f5c0cbbd60
[  242.729333]  00e590a200000010 ffff92faffc19900 0000000000000046 7fffffffffffffff
[  242.729336] Call Trace:
[  242.729345]  [<ffffffffae8cf2c2>] ? __schedule+0x242/0x700
[  242.729348]  [<ffffffffae8cf7ac>] schedule+0x2c/0x80
[  242.729351]  [<ffffffffae8d2e0b>] schedule_timeout+0x1fb/0x370
[  242.729355]  [<ffffffffae0fcaae>] ? add_timer+0x11e/0x290
[  242.729358]  [<ffffffffae8d021a>] wait_for_completion+0xba/0x140
[  242.729361]  [<ffffffffae0b4ac0>] ? wake_up_q+0x80/0x80
[  242.729364]  [<ffffffffae0f1a44>] __synchronize_srcu+0xf4/0x140
[  242.729367]  [<ffffffffae0f08a0>] ? trace_raw_output_rcu_utilization+0x60/0x60
[  242.729370]  [<ffffffffae0f1ab3>] synchronize_srcu+0x23/0x40
[  242.729374]  [<ffffffffae283bab>] fsnotify_mark_destroy_list+0x7b/0xe0
[  242.729377]  [<ffffffffae282d7f>] fsnotify_destroy_group+0x1f/0x50
[  242.729380]  [<ffffffffae285e66>] fanotify_release+0xd6/0x140
[  242.729384]  [<ffffffffae23e70a>] __fput+0xea/0x230
[  242.729386]  [<ffffffffae23e88e>] ____fput+0xe/0x10
[  242.729390]  [<ffffffffae0a6f2c>] task_work_run+0x7c/0xa0
[  242.729394]  [<ffffffffae003753>] exit_to_usermode_loop+0x93/0xa0
[  242.729397]  [<ffffffffae003bc7>] do_syscall_64+0xc7/0xe0
[  242.729400]  [<ffffffffae8d448e>] entry_SYSCALL_64_after_swapgs+0x58/0xc6

Note this call stack is the same as the one for 4.4, but some function names 
have changed due to the two commits already included in 4.9.

On a patched kernel, the test will pass successfully, and there will be no
messages in dmesg. 

[Regression Potential]

This makes modifications to how locking is performed in fsnotify / fanotify and 
there may be some cause for regression. Running all fanotify Linux Test Project
tests shows that there are no extra failures caused by the patches, and instead
fewer failures are seen due to the bugfix. 

Running the entire Linux Test Project testsuite actually works and runs to 
completion, something which doesn't happen in a unpatched kernel since it will 
hang on the fanotify07 test.

The patches are taken from upstream, and all necessary commits have been taken
into account, so I am happy with the potential risks and that testing has been
completed.

Jan Kara (3):
  fsnotify: Provide framework for dropping SRCU lock in ->handle_event
  fsnotify: Pass fsnotify_iter_info into handle_event handler
  fanotify: Release SRCU lock when waiting for userspace response

 fs/notify/dnotify/dnotify.c          |  3 +-
 fs/notify/fanotify/fanotify.c        | 20 ++++++-
 fs/notify/fsnotify.c                 | 19 +++++--
 fs/notify/fsnotify.h                 |  6 ++
 fs/notify/group.c                    |  1 +
 fs/notify/inotify/inotify.h          |  3 +-
 fs/notify/inotify/inotify_fsnotify.c |  3 +-
 fs/notify/inotify/inotify_user.c     |  2 +-
 fs/notify/mark.c                     | 83 +++++++++++++++++++++++++++-
 include/linux/fsnotify_backend.h     |  8 ++-
 kernel/audit_fsnotify.c              |  3 +-
 kernel/audit_tree.c                  |  3 +-
 kernel/audit_watch.c                 |  3 +-
 13 files changed, 139 insertions(+), 18 deletions(-)

-- 
2.19.1


             reply index

Thread overview: 5+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-04-11  3:24 Matthew Ruffell [this message]
2019-04-11  3:24 ` [PATCH 4.9 1/3] fsnotify: Provide framework for dropping SRCU lock in ->handle_event Matthew Ruffell
2019-04-11  3:24 ` [PATCH 4.9 2/3] fsnotify: Pass fsnotify_iter_info into handle_event handler Matthew Ruffell
2019-04-11  3:24 ` [PATCH 4.9 3/3] fanotify: Release SRCU lock when waiting for userspace response Matthew Ruffell
2019-04-11 14:43 ` [PATCH 4.9 0/3] fanotify: Fix notification subsystem hang Sasha Levin

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20190411032430.17353-1-matthew.ruffell@canonical.com \
    --to=matthew.ruffell@canonical.com \
    --cc=stable@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Stable Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/stable/0 stable/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 stable stable/ https://lore.kernel.org/stable \
		stable@vger.kernel.org stable@archiver.kernel.org
	public-inbox-index stable

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.stable


AGPL code for this site: git clone https://public-inbox.org/ public-inbox