Re: sched_deadline sched_setattr fails after setting cpuset

[Ming Yang <minos.future@gmail.com>]

Hi,

On Fri, Jun 21, 2019 at 3:34 AM Luca Abeni <lucabe72@gmail.com> wrote:
>
> Hi,
>
> On Fri, 21 Jun 2019 at 02:54, Ming Yang <minos.future@gmail.com> wrote:
> [...]
> > I have a question about the admission test with cpuset.  Is the
> > sched_deadline admission test examined with respect to the CPUs
> > allowed in the cpuset?
>
> Yes, in theory you should be able to use up to 95% (or whatever you
> configured in /proc/sys/kernel/sched_rt*) of the CPUs contained in the
> cpuset.
>
> I am pretty sure it worked correctly when I tested this, but it was a
> lot of time ago...
>

Thanks!
I suppose acceptance test on cpusets that share CPUs doesn't work? For
example, the following three scenarios:

Scenario 1, sub-cpusets:
cpuset/A/cpuset.cpu_exclusive: 1
cpuset/A/cpuset.sched_load_balance: 0
cpuset/A/cpuset.cpus: 0,1,2,3
cpuset/A/B/cpuset.cpu_exclusive: 1
cpuset/A/B/cpuset.cpus: 0,1
cpuset/A/C/cpuset.cpu_exclusive: 1
cpuset/A/C/cpuset.cpus: 2,3
Is any acceptance test examined when a task is added to A or B (or C)?
My experiment results:
For A, yes, but it fails to utilize more than *two* cores when tasks
are added to cpuset/A/tasks. Even if all cores are added to
A/cpuset.cpus, it's still examined against two cores.
For B (or C), yes, the acceptance test is examined for total
utilization of two cores when tasks are added to cpuset/A/B(or
C)/tasks.

Scenario 2, identical cpusets:
cpuset/A/cpuset.cpu_exclusive: 0
cpuset/A/cpuset.cpus: 0,1
cpuset/B/cpuset.cpu_exclusive: 0
cpuset/B/cpuset.cpus: 0,1
Is any acceptance test examined when a task is added to A (or B)?
My experiment result:
Yes, the acceptance test is examined with respect to cpuset {0,1},
regardless of A or B.

Scenario 3, overlapping cpusets:
cpuset/A/cpuset.cpu_exclusive: 0
cpuset/A/cpuset.cpus: 0,1
cpuset/B/cpuset.cpu_exclusive: 0
cpuset/B/cpuset.cpus: 1,2
Is any acceptance test examined when a task is added to A (or B)?
My experiment result:
Operation not permitted for adding deadline tasks to cpusets that
overlaps with others.

> > I get deterministic admission test result without cpuset.  m-1 cores
> > can be utilized (not m?).
>
> The maximum SCHED_DEADLINE utilization should be 95% of m (unless you
> changed the values in /proc/sys/kernel/sched_rt_{runtime,period}_us).
>
>
> > But with cpuset, the total utilization of accepted sched_deadline
> > tasks seems not respecting the cpuset and changes every run.  For
> > example, if cpuset contains 4 cores, I can have task sets (bound to
> > this cpuset) utilizing 400%, 600%, or 300% before failing to add
> > another task. It could be my program's issue. I'm still looking into
> > it.
>
> If you get this result, I suspect you found a bug... Can you provide a
> simple testcase for reproducing int?
>

I'll try to send you a test case soon.
In the meantime, I figured out why I got the EPERM I mentioned in my
first email. It's because I didn't disable cpuset.sched_load_balance
at the root domain. It makes sense as allowing tasks to move in and
out the cpuset violates the acceptance test assumption. But it might
worth emphasizing this flag in the documentation though.

Thanks,
Ming

>
> Luca