Issue with cyclictest, RT_GROUP_SCHED, isolcpus and NOHZ_FULL

Issue with cyclictest, RT_GROUP_SCHED, isolcpus and NOHZ_FULL

[Jonathan Schwender]

Hi everyone,

I've been trying to test the real-time `performance` possible with
containers, by running cyclictest in a container on an RT-Kernel.
The issue I've been having does not require containers or an
RT kernel though.

Issue: cyclictest freezes after running for a few seconds
to minutes. After that only the loadavg section is updated,
while the count line does not change anymore.
cyclictest can't be killed after that point
other than by restarting the machine, and
this also takes a few minutes until the kernel kills
cyclictest.

This behaviour only occurs when the following conditions are
met:

- RT_GROUP_SCHED is used
- cyclictest is bound to an isolated cpu core with
   nohz_full=<core>, and isolcpus=nohz,domain,<core>

I've tested this on a machine with Fedora 33 and vanilla
stable 5.10.3 kernel with RT_GROUP_SCHED.
The same behaviour also exists on 5.10.1-rt20 with
PREEMPT_RT and RT_GROUP_SCHED configured.

After booting I configure the rt_runtime_us like this:
`echo "700000" > /sys/fs/cgroup/cpu,cpuacct/user.slice/cpu.rt_runtime_us`
`echo "100000" > /sys/fs/cgroup/cpu,cpuacct/system.slice/cpu.rt_runtime_us`

Then I start cyclictest via:
`taskset -c 14 cgexec -g cpu,cpuacct:user.slice cyclictest --mlockall \
   --priority=96 --interval=200 --affinity=14 --duration=15m`

These are the cmdline options I tried out to narrow the problem down:
working: `isolcpus=14 irqaffinity=0-3 maxcpus=15 
systemd.unified_cgroup_hierarchy=0`
working: `isolcpus=nohz,14 nohz_full=14 irqaffinity=0-3 maxcpus=15 
systemd.unified_cgroup_hierarchy=0`
working: `isolcpus=nohz,domain,14 irqaffinity=0-3 maxcpus=15 
systemd.unified_cgroup_hierarchy=0`
broken:  `isolcpus=nohz,domain,14 nohz_full=14 irqaffinity=0-3 
maxcpus=15 systemd.unified_cgroup_hierarchy=0`

unified_cgroup_hierarchy is needed to get cgroups v1, which
seems to be needed for RT_GROUP_SCHED (at least I couldn't
find any options similar to cpu.rt_runtime_us with the default
cgroup v2).
Basically it boils down to that the combination of the
domain parameter to isolcpus and nohz_full together with
RT_GROUP_SCHED cause the problem I'm observing.

Does anyone have any idea what could be causing this?
Am I doing something wrong, or is there an issue with cyclictest or
even the kernel that's causing this?

My motivation is running (testing) a real-time container on isolated
cores, so I think I do need all the kernel parameters I used above to
get good latencies.

Regards,

Jonathan Schwender

Re: Issue with cyclictest, RT_GROUP_SCHED, isolcpus and NOHZ_FULL

[Sebastian Andrzej Siewior]

On 2020-12-30 14:09:19 [+0100], Jonathan Schwender wrote:
> Hi everyone,
> 
> I've been trying to test the real-time `performance` possible with
> containers, by running cyclictest in a container on an RT-Kernel.
> The issue I've been having does not require containers or an
> RT kernel though.
> 
> Issue: cyclictest freezes after running for a few seconds
> to minutes. After that only the loadavg section is updated,
> while the count line does not change anymore.
> cyclictest can't be killed after that point
> other than by restarting the machine, and
> this also takes a few minutes until the kernel kills
> cyclictest.
> 
> This behaviour only occurs when the following conditions are
> met:
> 
> - RT_GROUP_SCHED is used
> - cyclictest is bound to an isolated cpu core with
>   nohz_full=<core>, and isolcpus=nohz,domain,<core>

So if you remove RT_GROUP_SCHED and use cyclictest on the nohz_full
cores then everything is fine?

> I've tested this on a machine with Fedora 33 and vanilla
> stable 5.10.3 kernel with RT_GROUP_SCHED.
> The same behaviour also exists on 5.10.1-rt20 with
> PREEMPT_RT and RT_GROUP_SCHED configured.
> 
> After booting I configure the rt_runtime_us like this:
> `echo "700000" > /sys/fs/cgroup/cpu,cpuacct/user.slice/cpu.rt_runtime_us`
> `echo "100000" > /sys/fs/cgroup/cpu,cpuacct/system.slice/cpu.rt_runtime_us`
> 
> Then I start cyclictest via:
> `taskset -c 14 cgexec -g cpu,cpuacct:user.slice cyclictest --mlockall \
>   --priority=96 --interval=200 --affinity=14 --duration=15m`
> 
> These are the cmdline options I tried out to narrow the problem down:
> working: `isolcpus=14 irqaffinity=0-3 maxcpus=15
> systemd.unified_cgroup_hierarchy=0`
> working: `isolcpus=nohz,14 nohz_full=14 irqaffinity=0-3 maxcpus=15
> systemd.unified_cgroup_hierarchy=0`
> working: `isolcpus=nohz,domain,14 irqaffinity=0-3 maxcpus=15
> systemd.unified_cgroup_hierarchy=0`
> broken:  `isolcpus=nohz,domain,14 nohz_full=14 irqaffinity=0-3 maxcpus=15
> systemd.unified_cgroup_hierarchy=0`
> 
> unified_cgroup_hierarchy is needed to get cgroups v1, which
> seems to be needed for RT_GROUP_SCHED (at least I couldn't
> find any options similar to cpu.rt_runtime_us with the default
> cgroup v2).
> Basically it boils down to that the combination of the
> domain parameter to isolcpus and nohz_full together with
> RT_GROUP_SCHED cause the problem I'm observing.
> 
> Does anyone have any idea what could be causing this?
> Am I doing something wrong, or is there an issue with cyclictest or
> even the kernel that's causing this?
> 
> My motivation is running (testing) a real-time container on isolated
> cores, so I think I do need all the kernel parameters I used above to
> get good latencies.

You might want to try without nohz_full. My understanding is that this
used if your application remains mostly in userland (and uses no
syscalls, etc.).

Let me this on my list of things to try out.

> Regards,
> 
> Jonathan Schwender

Sebastian

Re: Issue with cyclictest, RT_GROUP_SCHED, isolcpus and NOHZ_FULL

[Jonathan Schwender]

On 2/18/21 4:12 PM, Sebastian Andrzej Siewior wrote:
> On 2020-12-30 14:09:19 [+0100], Jonathan Schwender wrote:
>> Hi everyone,
>>
>> I've been trying to test the real-time `performance` possible with
>> containers, by running cyclictest in a container on an RT-Kernel.
>> The issue I've been having does not require containers or an
>> RT kernel though.
>>
>> Issue: cyclictest freezes after running for a few seconds
>> to minutes. After that only the loadavg section is updated,
>> while the count line does not change anymore.
>> cyclictest can't be killed after that point
>> other than by restarting the machine, and
>> this also takes a few minutes until the kernel kills
>> cyclictest.
>>
>> This behaviour only occurs when the following conditions are
>> met:
>>
>> - RT_GROUP_SCHED is used
>> - cyclictest is bound to an isolated cpu core with
>>    nohz_full=<core>, and isolcpus=nohz,domain,<core>
> So if you remove RT_GROUP_SCHED and use cyclictest on the nohz_full
> cores then everything is fine?

Yes, I traced it down to RT throttling breaking if nohz_full is set and 
a non-root cgroup is used.
The timer tick which resets the runtime, gets moved to a housekeeping 
CPU, which
doesn't service the isolated CPUs. This leads to the task being RT 
throttled indefinitely.

I proposed a patch [1] which solved my issue however, I haven't received 
any feedback on it yet.

The patch basically just extends the exception for the root cgroup to 
also apply to all housekeeping CPUs,
but I assume there are better ways to fix this that I don't know about.


[1] 
https://lore.kernel.org/lkml/20210202090010.5350-1-schwenderjonathan@gmail.com/

> Sebastian

Jonathan