[PATCH] cgroup/cpuset: Fix a race between cpuset_attach() and cpu hotplug

[PATCH] cgroup/cpuset: Fix a race between cpuset_attach() and cpu hotplug

[Zhang Qiao]

As previously discussed(https://lkml.org/lkml/2022/1/20/51),
cpuset_attach() is affected with similar cpu hotplug race,
as follow scenario:

     cpuset_attach()				cpu hotplug
    ---------------------------            ----------------------
    down_write(cpuset_rwsem)
    guarantee_online_cpus() // (load cpus_attach)
					sched_cpu_deactivate
					  set_cpu_active()
					  // will change cpu_active_mask
    set_cpus_allowed_ptr(cpus_attach)
      __set_cpus_allowed_ptr_locked()
       // (if the intersection of cpus_attach and
         cpu_active_mask is empty, will return -EINVAL)
    up_write(cpuset_rwsem)

To avoid races such as described above, protect cpuset_attach() call
with cpu_hotplug_lock.

Fixes: be367d099270 ("cgroups: let ss->can_attach and ss->attach do whole threadgroups at a time")
Reported-by: Zhao Gongyi <zhaogongyi@huawei.com>
Signed-off-by: Zhang Qiao <zhangqiao22@huawei.com>
---
 kernel/cgroup/cpuset.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index dc653ab26e50..0af5725cc1df 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -2252,6 +2252,7 @@ static void cpuset_attach(struct cgroup_taskset *tset)
 	cgroup_taskset_first(tset, &css);
 	cs = css_cs(css);

+	cpus_read_lock();
 	percpu_down_write(&cpuset_rwsem);

 	guarantee_online_mems(cs, &cpuset_attach_nodemask_to);
@@ -2305,6 +2306,7 @@ static void cpuset_attach(struct cgroup_taskset *tset)
 		wake_up(&cpuset_attach_wq);

 	percpu_up_write(&cpuset_rwsem);
+	cpus_read_unlock();
 }

 /* The various types of files and directories in a cpuset file system */
--
2.18.0

Re: [PATCH] cgroup/cpuset: Fix a race between cpuset_attach() and cpu hotplug

[Waiman Long]

On 1/21/22 05:12, Zhang Qiao wrote:
> As previously discussed(https://lkml.org/lkml/2022/1/20/51),
> cpuset_attach() is affected with similar cpu hotplug race,
> as follow scenario:
>
>       cpuset_attach()				cpu hotplug
>      ---------------------------            ----------------------
>      down_write(cpuset_rwsem)
>      guarantee_online_cpus() // (load cpus_attach)
> 					sched_cpu_deactivate
> 					  set_cpu_active()
> 					  // will change cpu_active_mask
>      set_cpus_allowed_ptr(cpus_attach)
>        __set_cpus_allowed_ptr_locked()
>         // (if the intersection of cpus_attach and
>           cpu_active_mask is empty, will return -EINVAL)
>      up_write(cpuset_rwsem)
>
> To avoid races such as described above, protect cpuset_attach() call
> with cpu_hotplug_lock.
>
> Fixes: be367d099270 ("cgroups: let ss->can_attach and ss->attach do whole threadgroups at a time")
> Reported-by: Zhao Gongyi <zhaogongyi@huawei.com>
> Signed-off-by: Zhang Qiao <zhangqiao22@huawei.com>
> ---
>   kernel/cgroup/cpuset.c | 2 ++
>   1 file changed, 2 insertions(+)
>
> diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
> index dc653ab26e50..0af5725cc1df 100644
> --- a/kernel/cgroup/cpuset.c
> +++ b/kernel/cgroup/cpuset.c
> @@ -2252,6 +2252,7 @@ static void cpuset_attach(struct cgroup_taskset *tset)
>   	cgroup_taskset_first(tset, &css);
>   	cs = css_cs(css);
>
> +	cpus_read_lock();
>   	percpu_down_write(&cpuset_rwsem);
>
>   	guarantee_online_mems(cs, &cpuset_attach_nodemask_to);
> @@ -2305,6 +2306,7 @@ static void cpuset_attach(struct cgroup_taskset *tset)
>   		wake_up(&cpuset_attach_wq);
>
>   	percpu_up_write(&cpuset_rwsem);
> +	cpus_read_unlock();
>   }
>
>   /* The various types of files and directories in a cpuset file system */
> --
> 2.18.0

The locking sequence looks right.

Acked-by: Waiman Long <longman@redhat.com>

Re: [PATCH] cgroup/cpuset: Fix a race between cpuset_attach() and cpu hotplug

[Michal Koutný]

On Fri, Jan 21, 2022 at 06:12:10PM +0800, Zhang Qiao <zhangqiao22@huawei.com> wrote:
> Fixes: be367d099270 ("cgroups: let ss->can_attach and ss->attach do whole threadgroups at a time")

What a deep stratigraphy (not sure if it's this one but anything else
would likely come from eras ago too).

> Reported-by: Zhao Gongyi <zhaogongyi@huawei.com>
> Signed-off-by: Zhang Qiao <zhangqiao22@huawei.com>
> ---
>  kernel/cgroup/cpuset.c | 2 ++
>  1 file changed, 2 insertions(+)

Feel free to include
Reviewed-by: Michal Koutný <mkoutny@suse.com>

Re: [PATCH] cgroup/cpuset: Fix a race between cpuset_attach() and cpu hotplug

[Zhang Qiao]

在 2022/1/21 23:57, Michal Koutný 写道:
> On Fri, Jan 21, 2022 at 06:12:10PM +0800, Zhang Qiao <zhangqiao22@huawei.com> wrote:
>> Fixes: be367d099270 ("cgroups: let ss->can_attach and ss->attach do whole threadgroups at a time")
> 
> What a deep stratigraphy (not sure if it's this one but anything else
> would likely come from eras ago too).

Yes, I checked again and this bug may have existed since the tag Linux-2.6.12-rc2.
>> Reported-by: Zhao Gongyi <zhaogongyi@huawei.com>
>> Signed-off-by: Zhang Qiao <zhangqiao22@huawei.com>
>> ---
>>  kernel/cgroup/cpuset.c | 2 ++
>>  1 file changed, 2 insertions(+)
> 
> Feel free to include
> Reviewed-by: Michal Koutný <mkoutny@suse.com>
> 
Thanks for your review!

Regards,
Qiao
.

> .
> 

Re: [PATCH] cgroup/cpuset: Fix a race between cpuset_attach() and cpu hotplug

[Zhang Qiao]

在 2022/1/21 20:43, Waiman Long 写道:
> On 1/21/22 05:12, Zhang Qiao wrote:
>> As previously discussed(https://lkml.org/lkml/2022/1/20/51),
>> cpuset_attach() is affected with similar cpu hotplug race,
>> as follow scenario:
>>
>>       cpuset_attach()                cpu hotplug
>>      ---------------------------            ----------------------
>>      down_write(cpuset_rwsem)
>>      guarantee_online_cpus() // (load cpus_attach)
>>                     sched_cpu_deactivate
>>                       set_cpu_active()
>>                       // will change cpu_active_mask
>>      set_cpus_allowed_ptr(cpus_attach)
>>        __set_cpus_allowed_ptr_locked()
>>         // (if the intersection of cpus_attach and
>>           cpu_active_mask is empty, will return -EINVAL)
>>      up_write(cpuset_rwsem)
>>
>> To avoid races such as described above, protect cpuset_attach() call
>> with cpu_hotplug_lock.
>>
>> Fixes: be367d099270 ("cgroups: let ss->can_attach and ss->attach do whole threadgroups at a time")
>> Reported-by: Zhao Gongyi <zhaogongyi@huawei.com>
>> Signed-off-by: Zhang Qiao <zhangqiao22@huawei.com>
>> ---
>>   kernel/cgroup/cpuset.c | 2 ++
>>   1 file changed, 2 insertions(+)
>>
>> diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
>> index dc653ab26e50..0af5725cc1df 100644
>> --- a/kernel/cgroup/cpuset.c
>> +++ b/kernel/cgroup/cpuset.c
>> @@ -2252,6 +2252,7 @@ static void cpuset_attach(struct cgroup_taskset *tset)
>>       cgroup_taskset_first(tset, &css);
>>       cs = css_cs(css);
>>
>> +    cpus_read_lock();
>>       percpu_down_write(&cpuset_rwsem);
>>
>>       guarantee_online_mems(cs, &cpuset_attach_nodemask_to);
>> @@ -2305,6 +2306,7 @@ static void cpuset_attach(struct cgroup_taskset *tset)
>>           wake_up(&cpuset_attach_wq);
>>
>>       percpu_up_write(&cpuset_rwsem);
>> +    cpus_read_unlock();
>>   }
>>
>>   /* The various types of files and directories in a cpuset file system */
>> -- 
>> 2.18.0
> 
> The locking sequence looks right.
> 
> Acked-by: Waiman Long <longman@redhat.com>
> 
Thanks for your review!

Regards,
Qiao
.

> .

Re: [PATCH] cgroup/cpuset: Fix a race between cpuset_attach() and cpu hotplug

[Zhang Qiao]

Gentle ping on this patch.

在 2022/1/21 18:12, Zhang Qiao 写道:
> As previously discussed(https://lkml.org/lkml/2022/1/20/51),
> cpuset_attach() is affected with similar cpu hotplug race,
> as follow scenario:
> 
>      cpuset_attach()				cpu hotplug
>     ---------------------------            ----------------------
>     down_write(cpuset_rwsem)
>     guarantee_online_cpus() // (load cpus_attach)
> 					sched_cpu_deactivate
> 					  set_cpu_active()
> 					  // will change cpu_active_mask
>     set_cpus_allowed_ptr(cpus_attach)
>       __set_cpus_allowed_ptr_locked()
>        // (if the intersection of cpus_attach and
>          cpu_active_mask is empty, will return -EINVAL)
>     up_write(cpuset_rwsem)
> 
> To avoid races such as described above, protect cpuset_attach() call
> with cpu_hotplug_lock.
> 
> Fixes: be367d099270 ("cgroups: let ss->can_attach and ss->attach do whole threadgroups at a time")
> Reported-by: Zhao Gongyi <zhaogongyi@huawei.com>
> Signed-off-by: Zhang Qiao <zhangqiao22@huawei.com>
> ---
>  kernel/cgroup/cpuset.c | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
> index dc653ab26e50..0af5725cc1df 100644
> --- a/kernel/cgroup/cpuset.c
> +++ b/kernel/cgroup/cpuset.c
> @@ -2252,6 +2252,7 @@ static void cpuset_attach(struct cgroup_taskset *tset)
>  	cgroup_taskset_first(tset, &css);
>  	cs = css_cs(css);
> 
> +	cpus_read_lock();
>  	percpu_down_write(&cpuset_rwsem);
> 
>  	guarantee_online_mems(cs, &cpuset_attach_nodemask_to);
> @@ -2305,6 +2306,7 @@ static void cpuset_attach(struct cgroup_taskset *tset)
>  		wake_up(&cpuset_attach_wq);
> 
>  	percpu_up_write(&cpuset_rwsem);
> +	cpus_read_unlock();
>  }
> 
>  /* The various types of files and directories in a cpuset file system */
> --
> 2.18.0
> 
> .
> 

Re: [PATCH] cgroup/cpuset: Fix a race between cpuset_attach() and cpu hotplug

[Tejun Heo]

On Fri, Jan 21, 2022 at 06:12:10PM +0800, Zhang Qiao wrote:
> As previously discussed(https://lkml.org/lkml/2022/1/20/51),
> cpuset_attach() is affected with similar cpu hotplug race,
> as follow scenario:
> 
>      cpuset_attach()				cpu hotplug
>     ---------------------------            ----------------------
>     down_write(cpuset_rwsem)
>     guarantee_online_cpus() // (load cpus_attach)
> 					sched_cpu_deactivate
> 					  set_cpu_active()
> 					  // will change cpu_active_mask
>     set_cpus_allowed_ptr(cpus_attach)
>       __set_cpus_allowed_ptr_locked()
>        // (if the intersection of cpus_attach and
>          cpu_active_mask is empty, will return -EINVAL)
>     up_write(cpuset_rwsem)
> 
> To avoid races such as described above, protect cpuset_attach() call
> with cpu_hotplug_lock.
> 
> Fixes: be367d099270 ("cgroups: let ss->can_attach and ss->attach do whole threadgroups at a time")
> Reported-by: Zhao Gongyi <zhaogongyi@huawei.com>
> Signed-off-by: Zhang Qiao <zhangqiao22@huawei.com>

Applied to cgroup/for-5.17-fixes w/ stable cc'd.

Thanks and sorry about the delay.

-- 
tejun