[PATCH 0/2] A couple of sched_getattr() fixes for DL

[PATCH 0/2] A couple of sched_getattr() fixes for DL

[Quentin Perret]

Hi all,

In the context of [1] it became apparent that the way __getparam_dl()
copies the sched_flags into the dl_entity struct for deadline tasks can
have undesirable side effects. This series fixes two issues in that area
causing sched_getattr() to report stale values or things it shouldn't
report to userspace.

Thanks,
Quentin

[1] https://lore.kernel.org/lkml/ad30be79-8fb2-023d-9936-01f7173164e4@arm.com/

Quentin Perret (2):
  sched/deadline: Fix reset_on_fork reporting of DL tasks
  sched: Don't report SCHED_FLAG_SUGOV in sched_getattr()

 kernel/sched/core.c     | 1 +
 kernel/sched/deadline.c | 7 ++++---
 kernel/sched/sched.h    | 2 ++
 3 files changed, 7 insertions(+), 3 deletions(-)

-- 
2.32.0.432.gabb21c7263-goog

[PATCH 1/2] sched/deadline: Fix reset_on_fork reporting of DL tasks

[Quentin Perret]

It is possible for sched_getattr() to incorrectly report the state of
the reset_on_fork flag when called on a deadline task.

Indeed, if the flag was set on a deadline task using sched_setattr()
with flags (SCHED_FLAG_RESET_ON_FORK | SCHED_FLAG_KEEP_PARAMS), then
p->sched_reset_on_fork will be set, but __setscheduler() will bail out
early, which means that the dl_se->flags will not get updated by
__setscheduler_params()->__setparam_dl(). Consequently, if
sched_getattr() is then called on the task, __getparam_dl() will
override kattr.sched_flags with the now out-of-date copy in dl_se->flags
and report the stale value to userspace.

To fix this, make sure to only copy the flags that are relevant to
sched_deadline to and from the dl_se->flags field.

Signed-off-by: Quentin Perret <qperret@google.com>
---
 kernel/sched/deadline.c | 7 ++++---
 kernel/sched/sched.h    | 2 ++
 2 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index aaacd6cfd42f..5cafc642e647 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -2741,7 +2741,7 @@ void __setparam_dl(struct task_struct *p, const struct sched_attr *attr)
 	dl_se->dl_runtime = attr->sched_runtime;
 	dl_se->dl_deadline = attr->sched_deadline;
 	dl_se->dl_period = attr->sched_period ?: dl_se->dl_deadline;
-	dl_se->flags = attr->sched_flags;
+	dl_se->flags = attr->sched_flags & SCHED_DL_FLAGS;
 	dl_se->dl_bw = to_ratio(dl_se->dl_period, dl_se->dl_runtime);
 	dl_se->dl_density = to_ratio(dl_se->dl_deadline, dl_se->dl_runtime);
 }
@@ -2754,7 +2754,8 @@ void __getparam_dl(struct task_struct *p, struct sched_attr *attr)
 	attr->sched_runtime = dl_se->dl_runtime;
 	attr->sched_deadline = dl_se->dl_deadline;
 	attr->sched_period = dl_se->dl_period;
-	attr->sched_flags = dl_se->flags;
+	attr->sched_flags &= ~SCHED_DL_FLAGS;
+	attr->sched_flags |= dl_se->flags;
 }
 
 /*
@@ -2851,7 +2852,7 @@ bool dl_param_changed(struct task_struct *p, const struct sched_attr *attr)
 	if (dl_se->dl_runtime != attr->sched_runtime ||
 	    dl_se->dl_deadline != attr->sched_deadline ||
 	    dl_se->dl_period != attr->sched_period ||
-	    dl_se->flags != attr->sched_flags)
+	    dl_se->flags != (attr->sched_flags & SCHED_DL_FLAGS))
 		return true;
 
 	return false;
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 14a41a243f7b..ad3aee63db26 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -227,6 +227,8 @@ static inline void update_avg(u64 *avg, u64 sample)
  */
 #define SCHED_FLAG_SUGOV	0x10000000
 
+#define SCHED_DL_FLAGS (SCHED_FLAG_RECLAIM | SCHED_FLAG_DL_OVERRUN | SCHED_FLAG_SUGOV)
+
 static inline bool dl_entity_is_special(struct sched_dl_entity *dl_se)
 {
 #ifdef CONFIG_CPU_FREQ_GOV_SCHEDUTIL
-- 
2.32.0.432.gabb21c7263-goog

[PATCH 2/2] sched: Don't report SCHED_FLAG_SUGOV in sched_getattr()

[Quentin Perret]

SCHED_FLAG_SUGOV is supposed to be a kernel-only flag that userspace
cannot interact with. However, sched_getattr() currently reports it
in sched_flags if called on a sugov worker even though it is not
actually defined in a UAPI header. To avoid this, make sure to
clean-up the sched_flags field in sched_getattr() before returning to
userspace.

Signed-off-by: Quentin Perret <qperret@google.com>
---
 kernel/sched/core.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 2d9ff40f4661..d8f489dcc383 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -7535,6 +7535,7 @@ SYSCALL_DEFINE4(sched_getattr, pid_t, pid, struct sched_attr __user *, uattr,
 		kattr.sched_priority = p->rt_priority;
 	else
 		kattr.sched_nice = task_nice(p);
+	kattr.sched_flags &= SCHED_FLAG_ALL;
 
 #ifdef CONFIG_UCLAMP_TASK
 	/*
-- 
2.32.0.432.gabb21c7263-goog

Re: [PATCH 2/2] sched: Don't report SCHED_FLAG_SUGOV in sched_getattr()

[Juri Lelli]

Hi Quentin,

On 27/07/21 11:11, Quentin Perret wrote:
> SCHED_FLAG_SUGOV is supposed to be a kernel-only flag that userspace
> cannot interact with. However, sched_getattr() currently reports it
> in sched_flags if called on a sugov worker even though it is not
> actually defined in a UAPI header. To avoid this, make sure to
> clean-up the sched_flags field in sched_getattr() before returning to
> userspace.
> 
> Signed-off-by: Quentin Perret <qperret@google.com>
> ---
>  kernel/sched/core.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index 2d9ff40f4661..d8f489dcc383 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -7535,6 +7535,7 @@ SYSCALL_DEFINE4(sched_getattr, pid_t, pid, struct sched_attr __user *, uattr,
>  		kattr.sched_priority = p->rt_priority;
>  	else
>  		kattr.sched_nice = task_nice(p);
> +	kattr.sched_flags &= SCHED_FLAG_ALL;

Maybe we can do this in the previous patch so that it's kept confined to
deadline bits?

Thanks,
Juri

Re: [PATCH 2/2] sched: Don't report SCHED_FLAG_SUGOV in sched_getattr()

[Quentin Perret]

On Wednesday 28 Jul 2021 at 11:12:03 (+0200), Juri Lelli wrote:
> Hi Quentin,
> 
> On 27/07/21 11:11, Quentin Perret wrote:
> > SCHED_FLAG_SUGOV is supposed to be a kernel-only flag that userspace
> > cannot interact with. However, sched_getattr() currently reports it
> > in sched_flags if called on a sugov worker even though it is not
> > actually defined in a UAPI header. To avoid this, make sure to
> > clean-up the sched_flags field in sched_getattr() before returning to
> > userspace.
> > 
> > Signed-off-by: Quentin Perret <qperret@google.com>
> > ---
> >  kernel/sched/core.c | 1 +
> >  1 file changed, 1 insertion(+)
> > 
> > diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> > index 2d9ff40f4661..d8f489dcc383 100644
> > --- a/kernel/sched/core.c
> > +++ b/kernel/sched/core.c
> > @@ -7535,6 +7535,7 @@ SYSCALL_DEFINE4(sched_getattr, pid_t, pid, struct sched_attr __user *, uattr,
> >  		kattr.sched_priority = p->rt_priority;
> >  	else
> >  		kattr.sched_nice = task_nice(p);
> > +	kattr.sched_flags &= SCHED_FLAG_ALL;
> 
> Maybe we can do this in the previous patch so that it's kept confined to
> deadline bits?

That works too, it just felt like this could happen again if we start
using non-standard flags outside of deadline for any reason at some
point in the future. But no strong opinion really.

Cheers,
Quentin

Re: [PATCH 2/2] sched: Don't report SCHED_FLAG_SUGOV in sched_getattr()

[Juri Lelli]

On 28/07/21 10:39, Quentin Perret wrote:
> On Wednesday 28 Jul 2021 at 11:12:03 (+0200), Juri Lelli wrote:
> > Hi Quentin,
> > 
> > On 27/07/21 11:11, Quentin Perret wrote:
> > > SCHED_FLAG_SUGOV is supposed to be a kernel-only flag that userspace
> > > cannot interact with. However, sched_getattr() currently reports it
> > > in sched_flags if called on a sugov worker even though it is not
> > > actually defined in a UAPI header. To avoid this, make sure to
> > > clean-up the sched_flags field in sched_getattr() before returning to
> > > userspace.
> > > 
> > > Signed-off-by: Quentin Perret <qperret@google.com>
> > > ---
> > >  kernel/sched/core.c | 1 +
> > >  1 file changed, 1 insertion(+)
> > > 
> > > diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> > > index 2d9ff40f4661..d8f489dcc383 100644
> > > --- a/kernel/sched/core.c
> > > +++ b/kernel/sched/core.c
> > > @@ -7535,6 +7535,7 @@ SYSCALL_DEFINE4(sched_getattr, pid_t, pid, struct sched_attr __user *, uattr,
> > >  		kattr.sched_priority = p->rt_priority;
> > >  	else
> > >  		kattr.sched_nice = task_nice(p);
> > > +	kattr.sched_flags &= SCHED_FLAG_ALL;
> > 
> > Maybe we can do this in the previous patch so that it's kept confined to
> > deadline bits?
> 
> That works too, it just felt like this could happen again if we start
> using non-standard flags outside of deadline for any reason at some
> point in the future. But no strong opinion really.

Yeah, I also see this point. :)

So no prob with me to keep it in core.c as you do here.

Best,
Juri

Re: [PATCH 1/2] sched/deadline: Fix reset_on_fork reporting of DL tasks

[Dietmar Eggemann]

On 27/07/2021 12:11, Quentin Perret wrote:
> It is possible for sched_getattr() to incorrectly report the state of
> the reset_on_fork flag when called on a deadline task.
> 
> Indeed, if the flag was set on a deadline task using sched_setattr()
> with flags (SCHED_FLAG_RESET_ON_FORK | SCHED_FLAG_KEEP_PARAMS), then
> p->sched_reset_on_fork will be set, but __setscheduler() will bail out
> early, which means that the dl_se->flags will not get updated by
> __setscheduler_params()->__setparam_dl(). Consequently, if

True, but it would also be awkward if non-DL related flags would have to
be stored in dl_se->flags.

> sched_getattr() is then called on the task, __getparam_dl() will
> override kattr.sched_flags with the now out-of-date copy in dl_se->flags
> and report the stale value to userspace.
> 
> To fix this, make sure to only copy the flags that are relevant to
> sched_deadline to and from the dl_se->flags field.

It also fixes the 'hidden' issue that a

    uclampset -mX -MY -p dl_task

would end up at 'change:' label because of

    dl_se->flags != attr->sched_flags

and not because of

    attr->sched_flags & SCHED_FLAG_UTIL_CLAMP


And it also unblocks the uclamp-dl issue raised in
https://lkml.kernel.org/r/ad30be79-8fb2-023d-9936-01f7173164e4@arm.com
which surfaced when using `get_params()->__getparam_dl()` in
SYSCALL_DEFINE3(sched_setattr,...).
Just for reference, IIRC, you mentioned this already on irc.

Reviewed-by: Dietmar Eggemann <dietmar.eggemann@arm.com>

[...]

Re: [PATCH 2/2] sched: Don't report SCHED_FLAG_SUGOV in sched_getattr()

[Dietmar Eggemann]

On 28/07/2021 14:36, Juri Lelli wrote:
> On 28/07/21 10:39, Quentin Perret wrote:
>> On Wednesday 28 Jul 2021 at 11:12:03 (+0200), Juri Lelli wrote:

[...]

>>> Maybe we can do this in the previous patch so that it's kept confined to
>>> deadline bits?
>>
>> That works too, it just felt like this could happen again if we start
>> using non-standard flags outside of deadline for any reason at some
>> point in the future. But no strong opinion really.
> 
> Yeah, I also see this point. :)
> 
> So no prob with me to keep it in core.c as you do here.
> 
> Best,
> Juri
> 

I would vote for not exporting SCHED_FLAG_SUGOV from __getparam_dl() in
patch 1/2 to underpin the idea that this flag is a hack.

@ -2759,7 +2759,7 @@ void __getparam_dl(struct task_struct *p, struct
sched_attr *attr)
        attr->sched_deadline = dl_se->dl_deadline;
        attr->sched_period = dl_se->dl_period;
        attr->sched_flags &= ~SCHED_DL_FLAGS;
-       attr->sched_flags |= dl_se->flags;
+       attr->sched_flags |= dl_se->flags & ~SCHED_FLAG_SUGOV;

Re: [PATCH 2/2] sched: Don't report SCHED_FLAG_SUGOV in sched_getattr()

[Quentin Perret]

On Thursday 29 Jul 2021 at 19:21:03 (+0200), Dietmar Eggemann wrote:
> On 28/07/2021 14:36, Juri Lelli wrote:
> > On 28/07/21 10:39, Quentin Perret wrote:
> >> On Wednesday 28 Jul 2021 at 11:12:03 (+0200), Juri Lelli wrote:
> 
> [...]
> 
> >>> Maybe we can do this in the previous patch so that it's kept confined to
> >>> deadline bits?
> >>
> >> That works too, it just felt like this could happen again if we start
> >> using non-standard flags outside of deadline for any reason at some
> >> point in the future. But no strong opinion really.
> > 
> > Yeah, I also see this point. :)
> > 
> > So no prob with me to keep it in core.c as you do here.
> > 
> > Best,
> > Juri
> > 
> 
> I would vote for not exporting SCHED_FLAG_SUGOV from __getparam_dl() in
> patch 1/2 to underpin the idea that this flag is a hack.
> 
> @ -2759,7 +2759,7 @@ void __getparam_dl(struct task_struct *p, struct
> sched_attr *attr)
>         attr->sched_deadline = dl_se->dl_deadline;
>         attr->sched_period = dl_se->dl_period;
>         attr->sched_flags &= ~SCHED_DL_FLAGS;
> -       attr->sched_flags |= dl_se->flags;
> +       attr->sched_flags |= dl_se->flags & ~SCHED_FLAG_SUGOV;

Alright, that's 2 votes against 1, you win!
I'll post a v2 shortly.

Cheers,
Quentin

[tip: sched/core] sched: Don't report SCHED_FLAG_SUGOV in sched_getattr()

[tip-bot2 for Quentin Perret]

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     7ad721bf10718a4e480a27ded8bb16b8f6feb2d1
Gitweb:        https://git.kernel.org/tip/7ad721bf10718a4e480a27ded8bb16b8f6feb2d1
Author:        Quentin Perret <qperret@google.com>
AuthorDate:    Tue, 27 Jul 2021 11:11:02 +01:00
Committer:     Peter Zijlstra <peterz@infradead.org>
CommitterDate: Wed, 04 Aug 2021 15:16:44 +02:00

sched: Don't report SCHED_FLAG_SUGOV in sched_getattr()

SCHED_FLAG_SUGOV is supposed to be a kernel-only flag that userspace
cannot interact with. However, sched_getattr() currently reports it
in sched_flags if called on a sugov worker even though it is not
actually defined in a UAPI header. To avoid this, make sure to
clean-up the sched_flags field in sched_getattr() before returning to
userspace.

Signed-off-by: Quentin Perret <qperret@google.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://lore.kernel.org/r/20210727101103.2729607-3-qperret@google.com
---
 kernel/sched/core.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 6c562ad..314f70d 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -7530,6 +7530,7 @@ SYSCALL_DEFINE4(sched_getattr, pid_t, pid, struct sched_attr __user *, uattr,
 		kattr.sched_priority = p->rt_priority;
 	else
 		kattr.sched_nice = task_nice(p);
+	kattr.sched_flags &= SCHED_FLAG_ALL;
 
 #ifdef CONFIG_UCLAMP_TASK
 	/*

[tip: sched/core] sched/deadline: Fix reset_on_fork reporting of DL tasks

[tip-bot2 for Quentin Perret]

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     f95091536f78971b269ec321b057b8d630b0ad8a
Gitweb:        https://git.kernel.org/tip/f95091536f78971b269ec321b057b8d630b0ad8a
Author:        Quentin Perret <qperret@google.com>
AuthorDate:    Tue, 27 Jul 2021 11:11:01 +01:00
Committer:     Peter Zijlstra <peterz@infradead.org>
CommitterDate: Wed, 04 Aug 2021 15:16:43 +02:00

sched/deadline: Fix reset_on_fork reporting of DL tasks

It is possible for sched_getattr() to incorrectly report the state of
the reset_on_fork flag when called on a deadline task.

Indeed, if the flag was set on a deadline task using sched_setattr()
with flags (SCHED_FLAG_RESET_ON_FORK | SCHED_FLAG_KEEP_PARAMS), then
p->sched_reset_on_fork will be set, but __setscheduler() will bail out
early, which means that the dl_se->flags will not get updated by
__setscheduler_params()->__setparam_dl(). Consequently, if
sched_getattr() is then called on the task, __getparam_dl() will
override kattr.sched_flags with the now out-of-date copy in dl_se->flags
and report the stale value to userspace.

To fix this, make sure to only copy the flags that are relevant to
sched_deadline to and from the dl_se->flags field.

Signed-off-by: Quentin Perret <qperret@google.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://lore.kernel.org/r/20210727101103.2729607-2-qperret@google.com
---
 kernel/sched/deadline.c | 7 ++++---
 kernel/sched/sched.h    | 2 ++
 2 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index aaacd6c..5cafc64 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -2741,7 +2741,7 @@ void __setparam_dl(struct task_struct *p, const struct sched_attr *attr)
 	dl_se->dl_runtime = attr->sched_runtime;
 	dl_se->dl_deadline = attr->sched_deadline;
 	dl_se->dl_period = attr->sched_period ?: dl_se->dl_deadline;
-	dl_se->flags = attr->sched_flags;
+	dl_se->flags = attr->sched_flags & SCHED_DL_FLAGS;
 	dl_se->dl_bw = to_ratio(dl_se->dl_period, dl_se->dl_runtime);
 	dl_se->dl_density = to_ratio(dl_se->dl_deadline, dl_se->dl_runtime);
 }
@@ -2754,7 +2754,8 @@ void __getparam_dl(struct task_struct *p, struct sched_attr *attr)
 	attr->sched_runtime = dl_se->dl_runtime;
 	attr->sched_deadline = dl_se->dl_deadline;
 	attr->sched_period = dl_se->dl_period;
-	attr->sched_flags = dl_se->flags;
+	attr->sched_flags &= ~SCHED_DL_FLAGS;
+	attr->sched_flags |= dl_se->flags;
 }
 
 /*
@@ -2851,7 +2852,7 @@ bool dl_param_changed(struct task_struct *p, const struct sched_attr *attr)
 	if (dl_se->dl_runtime != attr->sched_runtime ||
 	    dl_se->dl_deadline != attr->sched_deadline ||
 	    dl_se->dl_period != attr->sched_period ||
-	    dl_se->flags != attr->sched_flags)
+	    dl_se->flags != (attr->sched_flags & SCHED_DL_FLAGS))
 		return true;
 
 	return false;
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 9a1c6ae..75a5c12 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -227,6 +227,8 @@ static inline void update_avg(u64 *avg, u64 sample)
  */
 #define SCHED_FLAG_SUGOV	0x10000000
 
+#define SCHED_DL_FLAGS (SCHED_FLAG_RECLAIM | SCHED_FLAG_DL_OVERRUN | SCHED_FLAG_SUGOV)
+
 static inline bool dl_entity_is_special(struct sched_dl_entity *dl_se)
 {
 #ifdef CONFIG_CPU_FREQ_GOV_SCHEDUTIL