sched_deadline sched_setattr fails after setting cpuset

sched_deadline sched_setattr fails after setting cpuset

[Ming Yang]

Hi,

sched_setattr for SCHED_DEADLINE fails if it is called after setting
cpuset that doesn't include all CPUs. The sched_deadline document
states that cgroup/cpuset should be used for task affinity.  But I got
errno EPERM when calling sched_setattr, after setting cpuset.

Section 5.1 example in sched_deadline document is broken. Not only
rt-app doesn't run in the way shown in the example anymore, but also
other sched_deadline program fails.

rt-tests deadline_test.c illustrated a working way to use cpuset for
sched_deadline tasks.  It sets cpuset *after* sched_setattr
(https://git.kernel.org/pub/scm/utils/rt-tests/rt-tests.git/tree/src/sched_deadline/deadline_test.c?h=unstable/devel/latest).
I suppose this is the only way to assign CPUs for sched_dealine tasks
now.

Can anyone please explain why sched_setattr fails after setting
cpuset? Does the sched_deadline scheduler rejects both affinity mask
and cpuset now?
Does this mean admission test always assume all CPUs and no admission
test would be examined when setting the cpuset of the task?

Thanks,
Ming

Re: sched_deadline sched_setattr fails after setting cpuset

[Luca Abeni]

Hi,

On Thu, 20 Jun 2019 at 22:25, Ming Yang <minos.future@gmail.com> wrote:
>
> Hi,
>
> sched_setattr for SCHED_DEADLINE fails if it is called after setting
> cpuset that doesn't include all CPUs. The sched_deadline document
> states that cgroup/cpuset should be used for task affinity.  But I got
> errno EPERM when calling sched_setattr, after setting cpuset.

Which kernel are you using? I just tried on 5.0.0 (actually, it is
Ubuntu's 5.0.0-17-lowlatency) and it worked.
I followed the cpuset instructions from the documentation (except for
not mounting the cpuset fs: Ubuntu already mounted it for me), and the
"Minimal main()" from Appendix B of the document.
The SCHED_DEADLINE thread runs correctly, with the specified runtime
and period, and is affine to CPU 0 only.

What did you exactly do, to get the failure?

> Section 5.1 example in sched_deadline document is broken. Not only
> rt-app doesn't run in the way shown in the example anymore, but also
> other sched_deadline program fails.

Yes, rt-app behaviour changed... But other SCHED_DEADLINE programs
(for example, the program from Appendix B) seem to work correctly.


>
> rt-tests deadline_test.c illustrated a working way to use cpuset for
> sched_deadline tasks.  It sets cpuset *after* sched_setattr
> (https://git.kernel.org/pub/scm/utils/rt-tests/rt-tests.git/tree/src/sched_deadline/deadline_test.c?h=unstable/devel/latest).
> I suppose this is the only way to assign CPUs for sched_dealine tasks
> now.

I suspect this is actually a bug... Juri had a patchset to fix it, if
I remember correctly.


> Can anyone please explain why sched_setattr fails after setting
> cpuset?

It does not fail here... Maybe this is a bug introduced after 5.0.0,
or maybe there is some other difference between your setup an my one.


Luca

>  Does the sched_deadline scheduler rejects both affinity mask
> and cpuset now?
> Does this mean admission test always assume all CPUs and no admission
> test would be examined when setting the cpuset of the task?
>
> Thanks,
> Ming

Re: sched_deadline sched_setattr fails after setting cpuset

[Ming Yang]

Hi Luca,

Thanks for the reply.  I couldn't get the errno anymore... Sorry for
the confusion!  The minimal main in appendix B runs well for me too.

Btw, rt-tests released today still have the bug.

I have a question about the admission test with cpuset.  Is the
sched_deadline admission test examined with respect to the CPUs
allowed in the cpuset?
I get deterministic admission test result without cpuset.  m-1 cores
can be utilized (not m?).
But with cpuset, the total utilization of accepted sched_deadline
tasks seems not respecting the cpuset and changes every run.  For
example, if cpuset contains 4 cores, I can have task sets (bound to
this cpuset) utilizing 400%, 600%, or 300% before failing to add
another task. It could be my program's issue. I'm still looking into
it.

Thanks,
Ming

On Thu, Jun 20, 2019 at 2:02 PM Luca Abeni <lucabe72@gmail.com> wrote:
>
> Hi,
>
> On Thu, 20 Jun 2019 at 22:25, Ming Yang <minos.future@gmail.com> wrote:
> >
> > Hi,
> >
> > sched_setattr for SCHED_DEADLINE fails if it is called after setting
> > cpuset that doesn't include all CPUs. The sched_deadline document
> > states that cgroup/cpuset should be used for task affinity.  But I got
> > errno EPERM when calling sched_setattr, after setting cpuset.
>
> Which kernel are you using? I just tried on 5.0.0 (actually, it is
> Ubuntu's 5.0.0-17-lowlatency) and it worked.
> I followed the cpuset instructions from the documentation (except for
> not mounting the cpuset fs: Ubuntu already mounted it for me), and the
> "Minimal main()" from Appendix B of the document.
> The SCHED_DEADLINE thread runs correctly, with the specified runtime
> and period, and is affine to CPU 0 only.
>
> What did you exactly do, to get the failure?
>
> > Section 5.1 example in sched_deadline document is broken. Not only
> > rt-app doesn't run in the way shown in the example anymore, but also
> > other sched_deadline program fails.
>
> Yes, rt-app behaviour changed... But other SCHED_DEADLINE programs
> (for example, the program from Appendix B) seem to work correctly.
>
>
> >
> > rt-tests deadline_test.c illustrated a working way to use cpuset for
> > sched_deadline tasks.  It sets cpuset *after* sched_setattr
> > (https://git.kernel.org/pub/scm/utils/rt-tests/rt-tests.git/tree/src/sched_deadline/deadline_test.c?h=unstable/devel/latest).
> > I suppose this is the only way to assign CPUs for sched_dealine tasks
> > now.
>
> I suspect this is actually a bug... Juri had a patchset to fix it, if
> I remember correctly.
>
>
> > Can anyone please explain why sched_setattr fails after setting
> > cpuset?
>
> It does not fail here... Maybe this is a bug introduced after 5.0.0,
> or maybe there is some other difference between your setup an my one.
>
>
> Luca
>
> >  Does the sched_deadline scheduler rejects both affinity mask
> > and cpuset now?
> > Does this mean admission test always assume all CPUs and no admission
> > test would be examined when setting the cpuset of the task?
> >
> > Thanks,
> > Ming

Re: sched_deadline sched_setattr fails after setting cpuset

[Luca Abeni]

Hi,

On Fri, 21 Jun 2019 at 02:54, Ming Yang <minos.future@gmail.com> wrote:
[...]
> I have a question about the admission test with cpuset.  Is the
> sched_deadline admission test examined with respect to the CPUs
> allowed in the cpuset?

Yes, in theory you should be able to use up to 95% (or whatever you
configured in /proc/sys/kernel/sched_rt*) of the CPUs contained in the
cpuset.

I am pretty sure it worked correctly when I tested this, but it was a
lot of time ago...

> I get deterministic admission test result without cpuset.  m-1 cores
> can be utilized (not m?).

The maximum SCHED_DEADLINE utilization should be 95% of m (unless you
changed the values in /proc/sys/kernel/sched_rt_{runtime,period}_us).


> But with cpuset, the total utilization of accepted sched_deadline
> tasks seems not respecting the cpuset and changes every run.  For
> example, if cpuset contains 4 cores, I can have task sets (bound to
> this cpuset) utilizing 400%, 600%, or 300% before failing to add
> another task. It could be my program's issue. I'm still looking into
> it.

If you get this result, I suspect you found a bug... Can you provide a
simple testcase for reproducing int?


Luca

Re: sched_deadline sched_setattr fails after setting cpuset

[Ming Yang]

Hi,

On Fri, Jun 21, 2019 at 3:34 AM Luca Abeni <lucabe72@gmail.com> wrote:
>
> Hi,
>
> On Fri, 21 Jun 2019 at 02:54, Ming Yang <minos.future@gmail.com> wrote:
> [...]
> > I have a question about the admission test with cpuset.  Is the
> > sched_deadline admission test examined with respect to the CPUs
> > allowed in the cpuset?
>
> Yes, in theory you should be able to use up to 95% (or whatever you
> configured in /proc/sys/kernel/sched_rt*) of the CPUs contained in the
> cpuset.
>
> I am pretty sure it worked correctly when I tested this, but it was a
> lot of time ago...
>

Thanks!
I suppose acceptance test on cpusets that share CPUs doesn't work? For
example, the following three scenarios:

Scenario 1, sub-cpusets:
cpuset/A/cpuset.cpu_exclusive: 1
cpuset/A/cpuset.sched_load_balance: 0
cpuset/A/cpuset.cpus: 0,1,2,3
cpuset/A/B/cpuset.cpu_exclusive: 1
cpuset/A/B/cpuset.cpus: 0,1
cpuset/A/C/cpuset.cpu_exclusive: 1
cpuset/A/C/cpuset.cpus: 2,3
Is any acceptance test examined when a task is added to A or B (or C)?
My experiment results:
For A, yes, but it fails to utilize more than *two* cores when tasks
are added to cpuset/A/tasks. Even if all cores are added to
A/cpuset.cpus, it's still examined against two cores.
For B (or C), yes, the acceptance test is examined for total
utilization of two cores when tasks are added to cpuset/A/B(or
C)/tasks.

Scenario 2, identical cpusets:
cpuset/A/cpuset.cpu_exclusive: 0
cpuset/A/cpuset.cpus: 0,1
cpuset/B/cpuset.cpu_exclusive: 0
cpuset/B/cpuset.cpus: 0,1
Is any acceptance test examined when a task is added to A (or B)?
My experiment result:
Yes, the acceptance test is examined with respect to cpuset {0,1},
regardless of A or B.

Scenario 3, overlapping cpusets:
cpuset/A/cpuset.cpu_exclusive: 0
cpuset/A/cpuset.cpus: 0,1
cpuset/B/cpuset.cpu_exclusive: 0
cpuset/B/cpuset.cpus: 1,2
Is any acceptance test examined when a task is added to A (or B)?
My experiment result:
Operation not permitted for adding deadline tasks to cpusets that
overlaps with others.

> > I get deterministic admission test result without cpuset.  m-1 cores
> > can be utilized (not m?).
>
> The maximum SCHED_DEADLINE utilization should be 95% of m (unless you
> changed the values in /proc/sys/kernel/sched_rt_{runtime,period}_us).
>
>
> > But with cpuset, the total utilization of accepted sched_deadline
> > tasks seems not respecting the cpuset and changes every run.  For
> > example, if cpuset contains 4 cores, I can have task sets (bound to
> > this cpuset) utilizing 400%, 600%, or 300% before failing to add
> > another task. It could be my program's issue. I'm still looking into
> > it.
>
> If you get this result, I suspect you found a bug... Can you provide a
> simple testcase for reproducing int?
>

I'll try to send you a test case soon.
In the meantime, I figured out why I got the EPERM I mentioned in my
first email. It's because I didn't disable cpuset.sched_load_balance
at the root domain. It makes sense as allowing tasks to move in and
out the cpuset violates the acceptance test assumption. But it might
worth emphasizing this flag in the documentation though.

Thanks,
Ming

>
> Luca

Re: sched_deadline sched_setattr fails after setting cpuset

[Ming Yang]

I looked into Linux source code and here make an attempt to answer my question:

Linux uses `root_domain` to manage bandwidth and uses `cpuset` to
manage cpu mask.  Overlapping cpusets are unioned into one root
domain.  So each CPU belongs to only one `root_domain` but may belong
to multiple `cpuset`.  When the cpuset is a subset of the
`root_domain`, it rejects new tasks because of the
not-containing-all-cpu rule. It makes sense because the acceptance
test is performed against the bandwidth, which is defined per root
domain.  This is shown by scenario 3, where the root domain contains
cpus 0,1,2 but either cpuset contains only two of the three cpus, so
adding tasks fails.

For scenario 1, when a task is added to A, it's added to the root
domain of {0,1} or {2,3}, depending on where the task executes on.  So
allowed total utilization or bandwidth is two.

Please correct me if any of this is wrong.

Thanks,
Ming

On Mon, Jun 24, 2019 at 4:17 PM Ming Yang <minos.future@gmail.com> wrote:
>
> Hi,
>
> On Fri, Jun 21, 2019 at 3:34 AM Luca Abeni <lucabe72@gmail.com> wrote:
> >
> > Hi,
> >
> > On Fri, 21 Jun 2019 at 02:54, Ming Yang <minos.future@gmail.com> wrote:
> > [...]
> > > I have a question about the admission test with cpuset.  Is the
> > > sched_deadline admission test examined with respect to the CPUs
> > > allowed in the cpuset?
> >
> > Yes, in theory you should be able to use up to 95% (or whatever you
> > configured in /proc/sys/kernel/sched_rt*) of the CPUs contained in the
> > cpuset.
> >
> > I am pretty sure it worked correctly when I tested this, but it was a
> > lot of time ago...
> >
>
> Thanks!
> I suppose acceptance test on cpusets that share CPUs doesn't work? For
> example, the following three scenarios:
>
> Scenario 1, sub-cpusets:
> cpuset/A/cpuset.cpu_exclusive: 1
> cpuset/A/cpuset.sched_load_balance: 0
> cpuset/A/cpuset.cpus: 0,1,2,3
> cpuset/A/B/cpuset.cpu_exclusive: 1
> cpuset/A/B/cpuset.cpus: 0,1
> cpuset/A/C/cpuset.cpu_exclusive: 1
> cpuset/A/C/cpuset.cpus: 2,3
> Is any acceptance test examined when a task is added to A or B (or C)?
> My experiment results:
> For A, yes, but it fails to utilize more than *two* cores when tasks
> are added to cpuset/A/tasks. Even if all cores are added to
> A/cpuset.cpus, it's still examined against two cores.
> For B (or C), yes, the acceptance test is examined for total
> utilization of two cores when tasks are added to cpuset/A/B(or
> C)/tasks.
>
> Scenario 2, identical cpusets:
> cpuset/A/cpuset.cpu_exclusive: 0
> cpuset/A/cpuset.cpus: 0,1
> cpuset/B/cpuset.cpu_exclusive: 0
> cpuset/B/cpuset.cpus: 0,1
> Is any acceptance test examined when a task is added to A (or B)?
> My experiment result:
> Yes, the acceptance test is examined with respect to cpuset {0,1},
> regardless of A or B.
>
> Scenario 3, overlapping cpusets:
> cpuset/A/cpuset.cpu_exclusive: 0
> cpuset/A/cpuset.cpus: 0,1
> cpuset/B/cpuset.cpu_exclusive: 0
> cpuset/B/cpuset.cpus: 1,2
> Is any acceptance test examined when a task is added to A (or B)?
> My experiment result:
> Operation not permitted for adding deadline tasks to cpusets that
> overlaps with others.
>
> > > I get deterministic admission test result without cpuset.  m-1 cores
> > > can be utilized (not m?).
> >
> > The maximum SCHED_DEADLINE utilization should be 95% of m (unless you
> > changed the values in /proc/sys/kernel/sched_rt_{runtime,period}_us).
> >
> >
> > > But with cpuset, the total utilization of accepted sched_deadline
> > > tasks seems not respecting the cpuset and changes every run.  For
> > > example, if cpuset contains 4 cores, I can have task sets (bound to
> > > this cpuset) utilizing 400%, 600%, or 300% before failing to add
> > > another task. It could be my program's issue. I'm still looking into
> > > it.
> >
> > If you get this result, I suspect you found a bug... Can you provide a
> > simple testcase for reproducing int?
> >
>
> I'll try to send you a test case soon.
> In the meantime, I figured out why I got the EPERM I mentioned in my
> first email. It's because I didn't disable cpuset.sched_load_balance
> at the root domain. It makes sense as allowing tasks to move in and
> out the cpuset violates the acceptance test assumption. But it might
> worth emphasizing this flag in the documentation though.
>
> Thanks,
> Ming
>
> >
> > Luca