Re: [PATCH v3 07/14] sched: Introduce restrict_cpus_allowed_ptr() to limit task CPU affinity

From: Will Deacon <will@kernel.org>
To: Quentin Perret <qperret@google.com>
Cc: linux-arm-kernel@lists.infradead.org, linux-arch@vger.kernel.org,
	linux-kernel@vger.kernel.org,
	Catalin Marinas <catalin.marinas@arm.com>,
	Marc Zyngier <maz@kernel.org>,
	Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	Peter Zijlstra <peterz@infradead.org>,
	Morten Rasmussen <morten.rasmussen@arm.com>,
	Qais Yousef <qais.yousef@arm.com>,
	Suren Baghdasaryan <surenb@google.com>, Tejun Heo <tj@kernel.org>,
	Li Zefan <lizefan@huawei.com>,
	Johannes Weiner <hannes@cmpxchg.org>,
	Ingo Molnar <mingo@redhat.com>,
	Juri Lelli <juri.lelli@redhat.com>,
	Vincent Guittot <vincent.guittot@linaro.org>,
	kernel-team@android.com
Subject: Re: [PATCH v3 07/14] sched: Introduce restrict_cpus_allowed_ptr() to limit task CPU affinity
Date: Thu, 19 Nov 2020 11:05:50 +0000	[thread overview]
Message-ID: <20201119110549.GA3946@willie-the-truck> (raw)
In-Reply-To: <20201119091820.GA2416649@google.com>

Hi Quentin,

Thanks for having a look.

On Thu, Nov 19, 2020 at 09:18:20AM +0000, Quentin Perret wrote:
> On Friday 13 Nov 2020 at 09:37:12 (+0000), Will Deacon wrote:
> > -static int __set_cpus_allowed_ptr(struct task_struct *p,
> > -				  const struct cpumask *new_mask, bool check)
> > +static int __set_cpus_allowed_ptr_locked(struct task_struct *p,
> > +					 const struct cpumask *new_mask,
> > +					 bool check,
> > +					 struct rq *rq,
> > +					 struct rq_flags *rf)
> >  {
> >  	const struct cpumask *cpu_valid_mask = cpu_active_mask;
> >  	unsigned int dest_cpu;
> > -	struct rq_flags rf;
> > -	struct rq *rq;
> >  	int ret = 0;
> 
> Should we have a lockdep assertion here?

I pondered that, but I don't think it's necessary because we already have
one in do_set_cpus_allowed() so adding an extra one here doesn't really add
anything.

> > -	rq = task_rq_lock(p, &rf);
> >  	update_rq_clock(rq);
> >  
> >  	if (p->flags & PF_KTHREAD) {
> > @@ -1929,7 +1923,7 @@ static int __set_cpus_allowed_ptr(struct task_struct *p,
> >  	if (task_running(rq, p) || p->state == TASK_WAKING) {
> >  		struct migration_arg arg = { p, dest_cpu };
> >  		/* Need help from migration thread: drop lock and wait. */
> > -		task_rq_unlock(rq, p, &rf);
> > +		task_rq_unlock(rq, p, rf);
> >  		stop_one_cpu(cpu_of(rq), migration_cpu_stop, &arg);
> >  		return 0;
> >  	} else if (task_on_rq_queued(p)) {
> > @@ -1937,20 +1931,69 @@ static int __set_cpus_allowed_ptr(struct task_struct *p,
> >  		 * OK, since we're going to drop the lock immediately
> >  		 * afterwards anyway.
> >  		 */
> > -		rq = move_queued_task(rq, &rf, p, dest_cpu);
> > +		rq = move_queued_task(rq, rf, p, dest_cpu);
> >  	}
> >  out:
> > -	task_rq_unlock(rq, p, &rf);
> > +	task_rq_unlock(rq, p, rf);
> 
> And that's a little odd to have here no? Can we move it back on the
> caller's side?

I don't think so, unfortunately. __set_cpus_allowed_ptr_locked() can trigger
migration, so it can drop the rq lock as part of that and end up relocking a
new rq, which it also unlocks before returning. Doing the unlock in the
caller is therfore even weirder, because you'd have to return the lock
pointer or something horrible like that.

I did add a comment about this right before the function and it's an
internal function to the scheduler so I think it's ok.

Will