From: "Paul E. McKenney" <paulmck@linux.ibm.com>
To: Dmitry Vyukov <dvyukov@google.com>
Cc: "Theodore Ts'o" <tytso@mit.edu>,
syzbot <syzbot+4bfbbf28a2e50ab07368@syzkaller.appspotmail.com>,
Andreas Dilger <adilger.kernel@dilger.ca>,
David Miller <davem@davemloft.net>,
eladr@mellanox.com, Ido Schimmel <idosch@mellanox.com>,
Jiri Pirko <jiri@mellanox.com>,
John Stultz <john.stultz@linaro.org>,
linux-ext4@vger.kernel.org, LKML <linux-kernel@vger.kernel.org>,
netdev <netdev@vger.kernel.org>,
syzkaller-bugs <syzkaller-bugs@googlegroups.com>,
Thomas Gleixner <tglx@linutronix.de>,
Peter Zijlstra <peterz@infradead.org>,
Ingo Molnar <mingo@kernel.org>
Subject: Re: INFO: rcu detected stall in ext4_write_checks
Date: Fri, 5 Jul 2019 08:16:58 -0700 [thread overview]
Message-ID: <20190705151658.GP26519@linux.ibm.com> (raw)
In-Reply-To: <CACT4Y+YTpUErjEmjrqki-tJ0Lyx0c53MQDGVS4CixfmcAnuY=A@mail.gmail.com>
On Fri, Jul 05, 2019 at 03:24:26PM +0200, Dmitry Vyukov wrote:
> On Thu, Jun 27, 2019 at 12:47 AM Theodore Ts'o <tytso@mit.edu> wrote:
> >
> > More details about what is going on. First, it requires root, because
> > one of that is required is using sched_setattr (which is enough to
> > shoot yourself in the foot):
> >
> > sched_setattr(0, {size=0, sched_policy=0x6 /* SCHED_??? */, sched_flags=0, sched_nice=0, sched_priority=0, sched_runtime=2251799813724439, sched_deadline=4611686018427453437, sched_period=0}, 0) = 0
> >
> > This is setting the scheduler policy to be SCHED_DEADLINE, with a
> > runtime parameter of 2251799.813724439 seconds (or 26 days) and a
> > deadline of 4611686018.427453437 seconds (or 146 *years*). This means
> > a particular kernel thread can run for up to 26 **days** before it is
> > scheduled away, and if a kernel reads gets woken up or sent a signal,
> > no worries, it will wake up roughly seven times the interval that Rip
> > Van Winkle spent snoozing in a cave in the Catskill Mountains (in
> > Washington Irving's short story).
> >
> > We then kick off a half-dozen threads all running:
> >
> > sendfile(fd, fd, &pos, 0x8080fffffffe);
> >
> > (and since count is a ridiculously large number, this gets cut down to):
> >
> > sendfile(fd, fd, &pos, 2147479552);
> >
> > Is it any wonder that we are seeing RCU stalls? :-)
>
> +Peter, Ingo for sched_setattr and +Paul for rcu
>
> First of all: is it a semi-intended result of a root (CAP_SYS_NICE)
> doing local DoS abusing sched_setattr? It would perfectly reasonable
> to starve other processes, but I am not sure about rcu. In the end the
> high prio process can use rcu itself, and then it will simply blow
> system memory by stalling rcu. So it seems that rcu stalls should not
> happen as a result of weird sched_setattr values. If that is the case,
> what needs to be fixed? sched_setattr? rcu? sendfile?
Does the (untested, probably does not even build) patch shown below help?
This patch assumes that the kernel was built with CONFIG_PREEMPT=n.
And that I found all the tight loops on the do_sendfile() code path.
> If this is semi-intended, the only option I see is to disable
> something in syzkaller: sched_setattr entirely, or drop CAP_SYS_NICE,
> or ...? Any preference either way?
Long-running tight loops in the kernel really should contain
cond_resched() or better.
Thanx, Paul
------------------------------------------------------------------------
diff --git a/fs/splice.c b/fs/splice.c
index 25212dcca2df..50aa3286764a 100644
--- a/fs/splice.c
+++ b/fs/splice.c
@@ -985,6 +985,7 @@ ssize_t splice_direct_to_actor(struct file *in, struct splice_desc *sd,
sd->pos = prev_pos + ret;
goto out_release;
}
+ cond_resched();
}
done:
next prev parent reply other threads:[~2019-07-05 15:17 UTC|newest]
Thread overview: 31+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-06-26 17:27 INFO: rcu detected stall in ext4_write_checks syzbot
2019-06-26 18:42 ` Theodore Ts'o
2019-06-26 21:03 ` Theodore Ts'o
2019-06-26 22:47 ` Theodore Ts'o
2019-07-05 13:24 ` Dmitry Vyukov
2019-07-05 15:16 ` Paul E. McKenney [this message]
2019-07-05 15:47 ` Amir Goldstein
2019-07-05 15:48 ` Dmitry Vyukov
2019-07-05 19:10 ` Paul E. McKenney
2019-07-06 4:28 ` Theodore Ts'o
2019-07-06 6:16 ` Paul E. McKenney
2019-07-06 15:02 ` Theodore Ts'o
2019-07-06 18:03 ` Paul E. McKenney
2019-07-07 1:16 ` Paul E. McKenney
2019-07-14 14:48 ` Dmitry Vyukov
2019-07-14 18:49 ` Paul E. McKenney
2019-07-15 13:29 ` Peter Zijlstra
2019-07-15 13:33 ` Dmitry Vyukov
2019-07-15 13:46 ` Peter Zijlstra
2019-07-15 14:02 ` Paul E. McKenney
2019-07-22 10:03 ` Dmitry Vyukov
2019-07-23 8:51 ` Dmitry Vyukov
2019-07-14 19:05 ` Theodore Ts'o
2019-07-14 19:29 ` Paul E. McKenney
2019-07-15 3:10 ` Paul E. McKenney
2019-07-15 13:01 ` Paul E. McKenney
2019-07-15 13:29 ` Dmitry Vyukov
2019-07-15 13:39 ` Peter Zijlstra
2019-07-15 14:03 ` Paul E. McKenney
2019-07-15 13:22 ` Peter Zijlstra
2019-07-05 13:18 ` Dmitry Vyukov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190705151658.GP26519@linux.ibm.com \
--to=paulmck@linux.ibm.com \
--cc=adilger.kernel@dilger.ca \
--cc=davem@davemloft.net \
--cc=dvyukov@google.com \
--cc=eladr@mellanox.com \
--cc=idosch@mellanox.com \
--cc=jiri@mellanox.com \
--cc=john.stultz@linaro.org \
--cc=linux-ext4@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@kernel.org \
--cc=netdev@vger.kernel.org \
--cc=peterz@infradead.org \
--cc=syzbot+4bfbbf28a2e50ab07368@syzkaller.appspotmail.com \
--cc=syzkaller-bugs@googlegroups.com \
--cc=tglx@linutronix.de \
--cc=tytso@mit.edu \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).