/sys/devices/system/cpu/possible is immutable?

/sys/devices/system/cpu/possible is immutable?

[Junyeong Jeong]

Hello everyone :)

I hope that kernelnewbies mailing list is a suitable place for asking my
question.

I wonder that possible-CPU-mask(/sys/devices/system/cpu/possible) can be
changed after boot in some way or other. I read that it is fixed at boot
time (https://elixir.bootlin.com/linux/v5.8/source/include/linux/cpumask.h#L50).
But I am not convinced that it is really immutable even if some cgroup
or virtualization magic is used.

Let me account for why I am curious about it.
Nowadays I am developing BPF library written in rust language.
In order to call `bpf_lookup_elem()` to get values from
BPF_MAP_TYPE_PERCPU_ARRAY in userspace, we need to know the correct
number of per-cpu areas before calling it. Because an out-buffer for
multiple per-cpu values should be allocated and passed to the
`bpf_lookup_elem()`. But this process is strongly based on the
assumption that the number of per-cpu area is always immutable.

I am referring to /sys/devices/system/cpu/possible file to get to know
the number of per-cpu areas. I don't know the better way for figuring
out the number. What I am anxious about is that the number of per-cpu
areas varies from time to time under some circumstances with cgroup or
virtualization magic.

So I checked some cgroup and virtualization ordinary use-cases which did
not affect the possible-CPU-mask.

--
1.
$ docker run --cpuset-cpus=0-3 -it ubuntu:20.10 bash

This does not affect /sys/devices/system/cpu/possible at all. The value
it contains is the same with the value of the host machine.

2.
$ virsh setvcpus --current ubuntu20.10 5

Before starting guest OS, the number of maximum vCPU was set to 8 and
current vCPU was set to 4. While guest OS is running, I changed the
number of vCPU to 5. And _inside guest OS_, I enabled the new CPU by
setting /sys/devices/system/cpu/cpu4/online to 1. But
/sys/devices/system/cpu/possible of guest OS did not change as expected.
--

While I was conducting some tests, I realized that it's not possible to
prove the immutability of possible-CPU-mask using inductive
method. Because there must be some corner cases that I can never
imagine.


Can anyone explain that possible-CPU-mask and the number of per-cpu
areas never change after boot-time even by cgroup magic or some tricks
from outside of hypervisors?

Thanks,
    Junyeong

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: /sys/devices/system/cpu/possible is immutable?

[Greg KH]

On Thu, Mar 18, 2021 at 11:10:18PM +0900, Junyeong Jeong wrote:
> Hello everyone :)
> 
> I hope that kernelnewbies mailing list is a suitable place for asking my
> question.
> 
> I wonder that possible-CPU-mask(/sys/devices/system/cpu/possible) can be
> changed after boot in some way or other. I read that it is fixed at boot
> time (https://elixir.bootlin.com/linux/v5.8/source/include/linux/cpumask.h#L50).
> But I am not convinced that it is really immutable even if some cgroup
> or virtualization magic is used.
> 
> Let me account for why I am curious about it.
> Nowadays I am developing BPF library written in rust language.
> In order to call `bpf_lookup_elem()` to get values from
> BPF_MAP_TYPE_PERCPU_ARRAY in userspace, we need to know the correct
> number of per-cpu areas before calling it. Because an out-buffer for
> multiple per-cpu values should be allocated and passed to the
> `bpf_lookup_elem()`. But this process is strongly based on the
> assumption that the number of per-cpu area is always immutable.
> 
> I am referring to /sys/devices/system/cpu/possible file to get to know
> the number of per-cpu areas. I don't know the better way for figuring
> out the number. What I am anxious about is that the number of per-cpu
> areas varies from time to time under some circumstances with cgroup or
> virtualization magic.
> 
> So I checked some cgroup and virtualization ordinary use-cases which did
> not affect the possible-CPU-mask.
> 
> --
> 1.
> $ docker run --cpuset-cpus=0-3 -it ubuntu:20.10 bash
> 
> This does not affect /sys/devices/system/cpu/possible at all. The value
> it contains is the same with the value of the host machine.
> 
> 2.
> $ virsh setvcpus --current ubuntu20.10 5
> 
> Before starting guest OS, the number of maximum vCPU was set to 8 and
> current vCPU was set to 4. While guest OS is running, I changed the
> number of vCPU to 5. And _inside guest OS_, I enabled the new CPU by
> setting /sys/devices/system/cpu/cpu4/online to 1. But
> /sys/devices/system/cpu/possible of guest OS did not change as expected.
> --
> 
> While I was conducting some tests, I realized that it's not possible to
> prove the immutability of possible-CPU-mask using inductive
> method. Because there must be some corner cases that I can never
> imagine.
> 
> 
> Can anyone explain that possible-CPU-mask and the number of per-cpu
> areas never change after boot-time even by cgroup magic or some tricks
> from outside of hypervisors?

That sysfs value itself will not change for the single system while the
kernel is running, but your program could be moved from a system with
one value for that file, to another system with a different value while
it is running without knowing that you were migrated.

But that being said, that file does not show you how many cpus you
actually have access to at that moment in time, it's just a max-value
for that specific kernel build.  CPUs can come and go while your program
is running, so be aware of that.  Your program can also be forbidden to
run on specific cpus if the admin decides to do so.

So be careful here, and use the "normal" api to deal with cpu values and
numbers, never assume that once you start, that the number of cpus your
program has access to will not change.

good luck!

greg k-h

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: /sys/devices/system/cpu/possible is immutable?

[Junyeong Jeong]

>
> That sysfs value itself will not change for the single system while the
> kernel is running, but your program could be moved from a system with
> one value for that file, to another system with a different value while
> it is running without knowing that you were migrated.
>

Running process migration to another system? Wow, I never knew that.
If migration happens between the time of reading
/sys/devices/system/cpu/possible and the time of calling
`bpf_lookup_elem`, TOCTOU(Time of check to Time of Use) problem would
arise. HaHa.

> So be careful here, and use the "normal" api to deal with cpu values and
> numbers, never assume that once you start, that the number of cpus your
> program has access to will not change.

Thank you Kroah-Hartman for your advice.

regards,
Junyeong

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies