Re: Issue with cyclictest, RT_GROUP_SCHED, isolcpus and NOHZ_FULL

[Jonathan Schwender <schwenderjonathan@gmail.com>]

On 2/18/21 4:12 PM, Sebastian Andrzej Siewior wrote:
> On 2020-12-30 14:09:19 [+0100], Jonathan Schwender wrote:
>> Hi everyone,
>>
>> I've been trying to test the real-time `performance` possible with
>> containers, by running cyclictest in a container on an RT-Kernel.
>> The issue I've been having does not require containers or an
>> RT kernel though.
>>
>> Issue: cyclictest freezes after running for a few seconds
>> to minutes. After that only the loadavg section is updated,
>> while the count line does not change anymore.
>> cyclictest can't be killed after that point
>> other than by restarting the machine, and
>> this also takes a few minutes until the kernel kills
>> cyclictest.
>>
>> This behaviour only occurs when the following conditions are
>> met:
>>
>> - RT_GROUP_SCHED is used
>> - cyclictest is bound to an isolated cpu core with
>>    nohz_full=<core>, and isolcpus=nohz,domain,<core>
> So if you remove RT_GROUP_SCHED and use cyclictest on the nohz_full
> cores then everything is fine?

Yes, I traced it down to RT throttling breaking if nohz_full is set and 
a non-root cgroup is used.
The timer tick which resets the runtime, gets moved to a housekeeping 
CPU, which
doesn't service the isolated CPUs. This leads to the task being RT 
throttled indefinitely.

I proposed a patch [1] which solved my issue however, I haven't received 
any feedback on it yet.

The patch basically just extends the exception for the root cgroup to 
also apply to all housekeeping CPUs,
but I assume there are better ways to fix this that I don't know about.


[1] 
https://lore.kernel.org/lkml/20210202090010.5350-1-schwenderjonathan@gmail.com/

> Sebastian

Jonathan