From: tj <tj@kernel.org>
To: CAI Qian <caiqian@redhat.com>
Cc: Al Viro <viro@ZenIV.linux.org.uk>,
Linus Torvalds <torvalds@linux-foundation.org>,
Dave Chinner <david@fromorbit.com>,
linux-xfs <linux-xfs@vger.kernel.org>,
Jens Axboe <axboe@kernel.dk>, Nick Piggin <npiggin@gmail.com>,
linux-fsdevel@vger.kernel.org
Subject: Re: local DoS - systemd hang or timeout (WAS: Re: [RFC][CFT] splice_read reworked)
Date: Tue, 4 Oct 2016 17:42:19 -0400 [thread overview]
Message-ID: <20161004214219.GN4205@htj.duckdns.org> (raw)
In-Reply-To: <1812816839.401734.1475602751170.JavaMail.zimbra@redhat.com>
Hello, CAI.
On Tue, Oct 04, 2016 at 01:39:11PM -0400, CAI Qian wrote:
...
> Not sure if related, but right after this lockdep happened and trinity running by a
> non-privileged user finished inside the container. The host's systemctl command just
> hang or timeout which renders the whole system unusable.
>
> # systemctl status docker
> Failed to get properties: Connection timed out
>
> # systemctl reboot (hang)
>
...
> [ 5535.893675] INFO: lockdep is turned off.
> [ 5535.898085] INFO: task kworker/45:4:146035 blocked for more than 120 seconds.
> [ 5535.906059] Tainted: G W 4.8.0-rc8-fornext+ #1
> [ 5535.912865] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [ 5535.921613] kworker/45:4 D ffff880853e9b950 14048 146035 2 0x00000080
> [ 5535.929630] Workqueue: cgroup_destroy css_killed_work_fn
> [ 5535.935582] ffff880853e9b950 0000000000000000 0000000000000000 ffff88086c6da000
> [ 5535.943882] ffff88086c9e2000 ffff880853e9c000 ffff880853e9baa0 ffff88086c9e2000
> [ 5535.952205] ffff880853e9ba98 0000000000000001 ffff880853e9b968 ffffffff817cdaaf
> [ 5535.960522] Call Trace:
> [ 5535.963265] [<ffffffff817cdaaf>] schedule+0x3f/0xa0
> [ 5535.968817] [<ffffffff817d33fb>] schedule_timeout+0x3db/0x6f0
> [ 5535.975346] [<ffffffff817cf055>] ? wait_for_completion+0x45/0x130
> [ 5535.982256] [<ffffffff817cf0d3>] wait_for_completion+0xc3/0x130
> [ 5535.988972] [<ffffffff810d1fd0>] ? wake_up_q+0x80/0x80
> [ 5535.994804] [<ffffffff8130de64>] drop_sysctl_table+0xc4/0xe0
> [ 5536.001227] [<ffffffff8130de17>] drop_sysctl_table+0x77/0xe0
> [ 5536.007648] [<ffffffff8130decd>] unregister_sysctl_table+0x4d/0xa0
> [ 5536.014654] [<ffffffff8130deff>] unregister_sysctl_table+0x7f/0xa0
> [ 5536.021657] [<ffffffff810f57f5>] unregister_sched_domain_sysctl+0x15/0x40
> [ 5536.029344] [<ffffffff810d7704>] partition_sched_domains+0x44/0x450
> [ 5536.036447] [<ffffffff817d0761>] ? __mutex_unlock_slowpath+0x111/0x1f0
> [ 5536.043844] [<ffffffff81167684>] rebuild_sched_domains_locked+0x64/0xb0
> [ 5536.051336] [<ffffffff8116789d>] update_flag+0x11d/0x210
> [ 5536.057373] [<ffffffff817cf61f>] ? mutex_lock_nested+0x2df/0x450
> [ 5536.064186] [<ffffffff81167acb>] ? cpuset_css_offline+0x1b/0x60
> [ 5536.070899] [<ffffffff810fce3d>] ? trace_hardirqs_on+0xd/0x10
> [ 5536.077420] [<ffffffff817cf61f>] ? mutex_lock_nested+0x2df/0x450
> [ 5536.084234] [<ffffffff8115a9f5>] ? css_killed_work_fn+0x25/0x220
> [ 5536.091049] [<ffffffff81167ae5>] cpuset_css_offline+0x35/0x60
> [ 5536.097571] [<ffffffff8115aa2c>] css_killed_work_fn+0x5c/0x220
> [ 5536.104207] [<ffffffff810bc83f>] process_one_work+0x1df/0x710
> [ 5536.110736] [<ffffffff810bc7c0>] ? process_one_work+0x160/0x710
> [ 5536.117461] [<ffffffff810bce9b>] worker_thread+0x12b/0x4a0
> [ 5536.123697] [<ffffffff810bcd70>] ? process_one_work+0x710/0x710
> [ 5536.130426] [<ffffffff810c3f7e>] kthread+0xfe/0x120
> [ 5536.135991] [<ffffffff817d4baf>] ret_from_fork+0x1f/0x40
> [ 5536.142041] [<ffffffff810c3e80>] ? kthread_create_on_node+0x230/0x230
This one seems to be the offender. cgroup is trying to offline a
cpuset css, which takes place under cgroup_mutex. The offlining ends
up trying to drain active usages of a sysctl table which apprently is
not happening. Did something hang or crash while trying to generate
sysctl content?
Thanks.
--
tejun
next prev parent reply other threads:[~2016-10-04 21:42 UTC|newest]
Thread overview: 152+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <723420070.1340881.1472835555274.JavaMail.zimbra@redhat.com>
[not found] ` <1832555471.1341372.1472835736236.JavaMail.zimbra@redhat.com>
2016-09-03 0:39 ` xfs_file_splice_read: possible circular locking dependency detected Dave Chinner
2016-09-03 0:57 ` Linus Torvalds
2016-09-03 1:45 ` Al Viro
2016-09-06 23:59 ` Dave Chinner
2016-09-08 20:35 ` Al Viro
2016-09-06 21:53 ` CAI Qian
2016-09-06 23:34 ` Dave Chinner
2016-09-08 15:29 ` CAI Qian
2016-09-08 17:56 ` Al Viro
2016-09-08 18:12 ` Linus Torvalds
2016-09-08 18:18 ` Linus Torvalds
2016-09-08 20:44 ` Al Viro
2016-09-08 20:57 ` Al Viro
2016-09-08 21:23 ` Al Viro
2016-09-08 21:38 ` Dave Chinner
2016-09-08 23:55 ` Al Viro
2016-09-09 1:53 ` Dave Chinner
2016-09-09 2:22 ` Linus Torvalds
2016-09-09 2:26 ` Linus Torvalds
2016-09-09 2:34 ` Al Viro
2016-09-09 2:50 ` Linus Torvalds
2016-09-09 22:19 ` Al Viro
2016-09-10 2:06 ` Linus Torvalds
2016-09-14 3:16 ` Al Viro
2016-09-14 3:39 ` Nicholas Piggin
2016-09-14 4:01 ` Linus Torvalds
2016-09-18 5:33 ` Al Viro
2016-09-19 3:08 ` Nicholas Piggin
2016-09-19 6:11 ` Al Viro
2016-09-19 7:26 ` Nicholas Piggin
2016-09-14 3:49 ` Linus Torvalds
2016-09-14 4:26 ` Al Viro
2016-09-17 8:20 ` Al Viro
2016-09-17 19:00 ` Al Viro
2016-09-17 20:15 ` Linus Torvalds
2016-09-18 19:31 ` skb_splice_bits() and large chunks in pipe (was " Al Viro
2016-09-18 20:12 ` Linus Torvalds
2016-09-18 22:31 ` Al Viro
2016-09-19 0:18 ` Linus Torvalds
2016-09-19 0:22 ` Al Viro
2016-09-19 0:22 ` Al Viro
2016-09-20 9:51 ` Herbert Xu
2016-09-23 19:00 ` [RFC][CFT] splice_read reworked Al Viro
2016-09-23 19:01 ` [PATCH 01/11] fix memory leaks in tracing_buffers_splice_read() Al Viro
2016-09-23 19:02 ` [PATCH 02/11] splice_to_pipe(): don't open-code wakeup_pipe_readers() Al Viro
2016-09-23 19:02 ` [PATCH 03/11] splice: switch get_iovec_page_array() to iov_iter Al Viro
2016-09-23 19:02 ` Al Viro
2016-09-23 19:03 ` [PATCH 04/11] splice: lift pipe_lock out of splice_to_pipe() Al Viro
2016-09-23 19:45 ` Linus Torvalds
2016-09-23 20:10 ` Al Viro
2016-09-23 20:36 ` Linus Torvalds
2016-09-24 3:59 ` Al Viro
2016-09-24 17:29 ` Al Viro
2016-09-27 15:38 ` Nicholas Piggin
2016-09-27 15:53 ` Chuck Lever
2016-09-27 15:53 ` Chuck Lever
2016-09-24 3:59 ` [PATCH 04/12] " Al Viro
2016-09-26 13:35 ` Miklos Szeredi
2016-09-26 13:35 ` Miklos Szeredi
2016-09-27 4:14 ` Al Viro
2016-09-27 4:14 ` Al Viro
2016-12-17 19:54 ` Andreas Schwab
2016-12-18 19:28 ` Linus Torvalds
2016-12-18 19:57 ` Andreas Schwab
2016-12-18 20:12 ` Al Viro
2016-12-18 20:30 ` Al Viro
2016-12-18 22:10 ` Linus Torvalds
2016-12-18 22:18 ` Al Viro
2016-12-18 22:22 ` Linus Torvalds
2016-12-18 22:49 ` Andreas Schwab
2016-12-21 18:56 ` Andreas Schwab
2016-12-21 19:12 ` Linus Torvalds
2016-09-24 4:00 ` [PATCH 06/12] new helper: add_to_pipe() Al Viro
2016-09-26 13:49 ` Miklos Szeredi
2016-09-24 4:01 ` [PATCH 10/12] new iov_iter flavour: pipe-backed Al Viro
2016-09-29 20:53 ` Miklos Szeredi
2016-09-29 22:50 ` Al Viro
2016-09-29 22:50 ` Al Viro
2016-09-30 7:30 ` Miklos Szeredi
2016-10-03 3:34 ` [RFC] O_DIRECT vs EFAULT (was Re: [PATCH 10/12] new iov_iter flavour: pipe-backed) Al Viro
2016-10-03 17:07 ` Linus Torvalds
2016-10-03 18:54 ` Al Viro
2016-09-24 4:01 ` [PATCH 11/12] switch generic_file_splice_read() to use of ->read_iter() Al Viro
2016-09-24 4:02 ` [PATCH 12/12] switch default_file_splice_read() to use of pipe-backed iov_iter Al Viro
2016-09-23 19:03 ` [PATCH 05/11] skb_splice_bits(): get rid of callback Al Viro
2016-09-23 19:03 ` Al Viro
2016-09-23 19:04 ` [PATCH 06/11] new helper: add_to_pipe() Al Viro
2016-09-23 19:04 ` [PATCH 07/11] fuse_dev_splice_read(): switch to add_to_pipe() Al Viro
2016-09-23 19:06 ` [PATCH 08/11] cifs: don't use memcpy() to copy struct iov_iter Al Viro
2016-09-23 19:08 ` [PATCH 09/11] fuse_ioctl_copy_user(): don't open-code copy_page_{to,from}_iter() Al Viro
2016-09-26 9:31 ` Miklos Szeredi
2016-09-23 19:09 ` [PATCH 10/11] new iov_iter flavour: pipe-backed Al Viro
2016-09-23 19:10 ` [PATCH 11/11] switch generic_file_splice_read() to use of ->read_iter() Al Viro
2016-09-30 13:32 ` [RFC][CFT] splice_read reworked CAI Qian
2016-09-30 17:42 ` CAI Qian
2016-09-30 18:33 ` CAI Qian
2016-09-30 18:33 ` CAI Qian
2016-10-03 1:37 ` Al Viro
2016-10-03 17:49 ` CAI Qian
2016-10-04 17:39 ` local DoS - systemd hang or timeout (WAS: Re: [RFC][CFT] splice_read reworked) CAI Qian
2016-10-04 21:42 ` tj [this message]
2016-10-05 14:09 ` CAI Qian
2016-10-05 15:30 ` tj
2016-10-05 15:54 ` CAI Qian
2016-10-05 18:57 ` CAI Qian
2016-10-05 20:05 ` Al Viro
2016-10-06 12:20 ` CAI Qian
2016-10-06 12:25 ` CAI Qian
2016-10-06 16:11 ` CAI Qian
2016-10-06 17:00 ` Linus Torvalds
2016-10-06 18:12 ` CAI Qian
2016-10-07 9:57 ` Dave Chinner
2016-10-07 15:25 ` Linus Torvalds
2016-10-07 7:08 ` Jan Kara
2016-10-07 14:43 ` CAI Qian
2016-10-07 15:27 ` CAI Qian
2016-10-07 18:56 ` CAI Qian
2016-10-09 21:54 ` Dave Chinner
2016-10-10 14:10 ` CAI Qian
2016-10-10 20:14 ` CAI Qian
2016-10-10 21:57 ` Dave Chinner
2016-10-12 19:50 ` [bisected] " CAI Qian
2016-10-12 20:59 ` Dave Chinner
2016-10-13 16:25 ` CAI Qian
2016-10-13 20:49 ` Dave Chinner
2016-10-13 20:56 ` CAI Qian
2016-10-09 21:51 ` Dave Chinner
2016-10-21 15:38 ` [4.9-rc1+] overlayfs lockdep CAI Qian
2016-10-24 12:57 ` Miklos Szeredi
2016-10-07 9:27 ` local DoS - systemd hang or timeout (WAS: Re: [RFC][CFT] splice_read reworked) Dave Chinner
2016-10-27 12:52 ` local DoS - systemd hang or timeout with cgroup traces CAI Qian
2016-10-03 1:42 ` [RFC][CFT] splice_read reworked Al Viro
2016-10-03 14:06 ` CAI Qian
2016-10-03 15:20 ` CAI Qian
2016-10-03 21:12 ` Dave Chinner
2016-10-04 13:57 ` CAI Qian
2016-10-03 20:32 ` CAI Qian
2016-10-03 20:35 ` Al Viro
2016-10-04 13:29 ` CAI Qian
2016-10-04 14:28 ` Al Viro
2016-10-04 16:21 ` CAI Qian
2016-10-04 20:12 ` Al Viro
2016-10-05 14:30 ` CAI Qian
2016-10-05 16:07 ` Al Viro
2016-09-09 2:31 ` xfs_file_splice_read: possible circular locking dependency detected Al Viro
2016-09-09 2:39 ` Linus Torvalds
2016-09-09 2:26 ` Al Viro
2016-09-09 2:19 ` Al Viro
2016-09-08 18:01 ` Linus Torvalds
2016-09-08 20:39 ` CAI Qian
2016-09-08 21:19 ` Dave Chinner
2016-09-08 21:30 ` Al Viro
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20161004214219.GN4205@htj.duckdns.org \
--to=tj@kernel.org \
--cc=axboe@kernel.dk \
--cc=caiqian@redhat.com \
--cc=david@fromorbit.com \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-xfs@vger.kernel.org \
--cc=npiggin@gmail.com \
--cc=torvalds@linux-foundation.org \
--cc=viro@ZenIV.linux.org.uk \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.