Re: Selftest failures related to kern_sync_rcu()

["Paul E. McKenney" <paulmck@kernel.org>]

On Wed, Apr 14, 2021 at 09:18:09PM +0200, Toke Høiland-Jørgensen wrote:
> "Paul E. McKenney" <paulmck@kernel.org> writes:
> 
> > On Wed, Apr 14, 2021 at 08:39:04PM +0200, Toke Høiland-Jørgensen wrote:
> >> "Paul E. McKenney" <paulmck@kernel.org> writes:
> >> 
> >> > On Wed, Apr 14, 2021 at 10:59:23AM -0700, Alexei Starovoitov wrote:
> >> >> On Wed, Apr 14, 2021 at 10:52 AM Paul E. McKenney <paulmck@kernel.org> wrote:
> >> >> >
> >> >> > > > > >                 if (num_online_cpus() > 1)
> >> >> > > > > >                         synchronize_rcu();
> >> >> >
> >> >> > In CONFIG_PREEMPT_NONE=y and CONFIG_PREEMPT_VOLUNTARY=y kernels, this
> >> >> > synchronize_rcu() will be a no-op anyway due to there only being the
> >> >> > one CPU.  Or are these failures all happening in CONFIG_PREEMPT=y kernels,
> >> >> > and in tests where preemption could result in the observed failures?
> >> >> >
> >> >> > Could you please send your .config file, or at least the relevant portions
> >> >> > of it?
> >> >> 
> >> >> That's my understanding as well. I assumed Toke has preempt=y.
> >> >> Otherwise the whole thing needs to be root caused properly.
> >> >
> >> > Given that there is only a single CPU, I am still confused about what
> >> > the tests are expecting the membarrier() system call to do for them.
> >> 
> >> It's basically a proxy for waiting until the objects are freed on the
> >> kernel side, as far as I understand...
> >
> > There are in-kernel objects that are freed via call_rcu(), and the idea
> > is to wait until these objects really are freed?  Or am I still missing
> > out on what is going on?
> 
> Something like that? Although I'm not actually sure these are using
> call_rcu()? One of them needs __put_task_struct() to run, and the other
> waits for map freeing, with this comment:
> 
> 
> 	/* we need to either wait for or force synchronize_rcu(), before
> 	 * checking for "still exists" condition, otherwise map could still be
> 	 * resolvable by ID, causing false positives.
> 	 *
> 	 * Older kernels (5.8 and earlier) freed map only after two
> 	 * synchronize_rcu()s, so trigger two, to be entirely sure.
> 	 */
> 	CHECK(kern_sync_rcu(), "sync_rcu", "failed\n");
> 	CHECK(kern_sync_rcu(), "sync_rcu", "failed\n");

OK, so the issue is that the membarrier() system call is designed to force
ordering only within a user process, and you need it in the kernel.

Give or take my being puzzled as to why the membarrier() system call
doesn't do it for you on a CONFIG_PREEMPT_NONE=y system, this brings
us back to the question Alexei asked me in the first place, what is the
best way to invoke an in-kernel synchronize_rcu() from userspace?

You guys gave some reasonable examples.  Here are a few others:

o	Bring a CPU online, then force it offline, or vice versa.
	But in this case, sys_membarrier() would do what you need
	given more than one CPU.

o	Use the membarrier() system call, but require that the tests
	run on systems with at least two CPUs.

o	Create a kernel module whose init function does a
	synchronize_rcu() and then returns failure.  This will
	avoid the overhead of removing that kernel module.

o	Create a sysfs or debugfs interface that does a
	synchronize_rcu().

But I am still concerned that you are needing more than synchronize_rcu()
can do.  Otherwise, the membarrier() system call would work just fine
on a single CPU on your CONFIG_PREEMPT_VOLUNTARY=y kernel.

							Thanx, Paul