Re: [External] Re: [PATCH] sched/fair: fix broken bandwidth control with nohz_full

[Chengming Zhou <zhouchengming@bytedance.com>]

On 2022/3/28 23:17, Peter Zijlstra wrote:
> On Mon, Mar 28, 2022 at 09:50:05PM +0800, Chengming Zhou wrote:
>> On 2022/3/28 21:20, Peter Zijlstra wrote:
>>> On Mon, Mar 28, 2022 at 07:07:51PM +0800, Chengming Zhou wrote:
>>>> With nohz_full enabled on cpu, the scheduler_tick() will be stopped
>>>> when only one CFS task left on rq.
>>>>
>>>> scheduler_tick()
>>>>   task_tick_fair()
>>>>     entity_tick()
>>>>       update_curr()
>>>>         account_cfs_rq_runtime(cfs_rq, delta_exec) --> stopped
>>>>
>>>> So that running task can't account its runtime periodically, but
>>>> the cfs_bandwidth hrtimer still __refill_cfs_bandwidth_runtime()
>>>> periodically. Later in one period, the task would account very
>>>> big delta_exec, which cause the cfs_rq to be throttled for a
>>>> long time.
>>>>
>>>> There are two solutions for the problem, the first is that we
>>>> can check in sched_can_stop_tick() if current task's cfs_rq
>>>> have runtime_enabled, in which case we don't stop tick. But
>>>> it will make nohz_full almost useless in cloud environment
>>>> that every container has the cpu bandwidth control setting.
>>>
>>> How is NOHZ_FULL useful in that environment to begin with? If you set
>>> bandwidth crap, the expectation is that there is overcommit, which more
>>> or less assumes lots of scheduling, presumably VMs or somesuch crud.
>>>
>>> So how does NOHZ_FULL make sense?
>>
>> Yes, we have scheduled some VMs in cgroups on the host, which
>> enabled NOHZ_FULL to reduce the interference of tick to vcpu task
>> if it's the only task running on cpu.
>>
>> This problem will however throttle it wrongly, even if it hasn't
>> used up its quota.
>>
>> Do you suggest that we shouldn't stop tick when the current task's
>> cfs_rq has runtime_enabled ?
> 
> I'm not suggesting anything just yet as I'm not sure I understand things
> well enough. I'm just wondering if NOHZ_FULL makes sense for you since
> NOHZ_FULL makes system entry/exit so much more expensive.

Ok, I see. It seems a normal use-case that run VMs on system with NOHZ_FULL
enabled, and set bandwidth for overcommit. At some times, the cpu will
stop tick when low load.

> 
> NOHZ_FULL is for use-cases that 'never' intend to go into the kernel,
> your use-case actively relies on going into the kernel. Hence the
> confusion.

In fact, I put a testcase at the end of git message, in which only run
a userspace loop workload:

cd /sys/fs/cgroup
echo "+cpu" > cgroup.subtree_control

mkdir test
echo "105000 100000" > test/cpu.max

echo $$ > test/cgroup.procs
taskset -c 1 bash -c "while true; do let i++; done"  --> will be throttled

Thanks.