[PATCH v2] KVM: kvmclock: Fix vCPUs > 64 can't be online/hotpluged

[PATCH v2] KVM: kvmclock: Fix vCPUs > 64 can't be online/hotpluged

[Wanpeng Li]

From: Wanpeng Li <wanpengli@tencent.com>

The per-cpu vsyscall pvclock data pointer assigns either an element of the 
static array hv_clock_boot (#vCPU <= 64) or dynamically allocated memory 
hvclock_mem (vCPU > 64), the dynamically memory will not be allocated if 
kvmclock vsyscall is disabled, this can result in cpu hotpluged fails in 
kvmclock_setup_percpu() which returns -ENOMEM. This patch fixes it by not 
assigning vsyscall pvclock data pointer if kvmclock vdso_clock_mode is not 
VDSO_CLOCKMODE_PVCLOCK.

Fixes: 6a1cac56f4 ("x86/kvm: Use __bss_decrypted attribute in shared variables")
Reported-by: Zelin Deng <zelin.deng@linux.alibaba.com>
Tested-by: Haiwei Li <lihaiwei@tencent.com>
Cc: Brijesh Singh <brijesh.singh@amd.com>
Cc: stable@vger.kernel.org#v4.19-rc5+
Signed-off-by: Wanpeng Li <wanpengli@tencent.com>
---
v1 -> v2:
 * add code comments

 arch/x86/kernel/kvmclock.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kernel/kvmclock.c b/arch/x86/kernel/kvmclock.c
index aa59374..01d4e55c 100644
--- a/arch/x86/kernel/kvmclock.c
+++ b/arch/x86/kernel/kvmclock.c
@@ -294,9 +294,11 @@ static int kvmclock_setup_percpu(unsigned int cpu)
 	/*
 	 * The per cpu area setup replicates CPU0 data to all cpu
 	 * pointers. So carefully check. CPU0 has been set up in init
-	 * already.
+	 * already. Assign vsyscall pvclock data pointer iff kvmclock
+	 * vsyscall is enabled.
 	 */
-	if (!cpu || (p && p != per_cpu(hv_clock_per_cpu, 0)))
+	if (!cpu || (p && p != per_cpu(hv_clock_per_cpu, 0)) ||
+	    (kvm_clock.vdso_clock_mode != VDSO_CLOCKMODE_PVCLOCK))
 		return 0;
 
 	/* Use the static page for the first CPUs, allocate otherwise */
-- 
2.7.4

Re: [PATCH v2] KVM: kvmclock: Fix vCPUs > 64 can't be online/hotpluged

[Wanpeng Li]

ping，
On Mon, 18 Jan 2021 at 17:08, Wanpeng Li <kernellwp@gmail.com> wrote:
>
> From: Wanpeng Li <wanpengli@tencent.com>
>
> The per-cpu vsyscall pvclock data pointer assigns either an element of the
> static array hv_clock_boot (#vCPU <= 64) or dynamically allocated memory
> hvclock_mem (vCPU > 64), the dynamically memory will not be allocated if
> kvmclock vsyscall is disabled, this can result in cpu hotpluged fails in
> kvmclock_setup_percpu() which returns -ENOMEM. This patch fixes it by not
> assigning vsyscall pvclock data pointer if kvmclock vdso_clock_mode is not
> VDSO_CLOCKMODE_PVCLOCK.
>
> Fixes: 6a1cac56f4 ("x86/kvm: Use __bss_decrypted attribute in shared variables")
> Reported-by: Zelin Deng <zelin.deng@linux.alibaba.com>
> Tested-by: Haiwei Li <lihaiwei@tencent.com>
> Cc: Brijesh Singh <brijesh.singh@amd.com>
> Cc: stable@vger.kernel.org#v4.19-rc5+
> Signed-off-by: Wanpeng Li <wanpengli@tencent.com>
> ---
> v1 -> v2:
>  * add code comments
>
>  arch/x86/kernel/kvmclock.c | 6 ++++--
>  1 file changed, 4 insertions(+), 2 deletions(-)
>
> diff --git a/arch/x86/kernel/kvmclock.c b/arch/x86/kernel/kvmclock.c
> index aa59374..01d4e55c 100644
> --- a/arch/x86/kernel/kvmclock.c
> +++ b/arch/x86/kernel/kvmclock.c
> @@ -294,9 +294,11 @@ static int kvmclock_setup_percpu(unsigned int cpu)
>         /*
>          * The per cpu area setup replicates CPU0 data to all cpu
>          * pointers. So carefully check. CPU0 has been set up in init
> -        * already.
> +        * already. Assign vsyscall pvclock data pointer iff kvmclock
> +        * vsyscall is enabled.
>          */
> -       if (!cpu || (p && p != per_cpu(hv_clock_per_cpu, 0)))
> +       if (!cpu || (p && p != per_cpu(hv_clock_per_cpu, 0)) ||
> +           (kvm_clock.vdso_clock_mode != VDSO_CLOCKMODE_PVCLOCK))
>                 return 0;
>
>         /* Use the static page for the first CPUs, allocate otherwise */
> --
> 2.7.4
>

Re: [PATCH v2] KVM: kvmclock: Fix vCPUs > 64 can't be online/hotpluged

[Paolo Bonzini]

On 26/01/21 02:28, Wanpeng Li wrote:
> ping，
> On Mon, 18 Jan 2021 at 17:08, Wanpeng Li <kernellwp@gmail.com> wrote:
>>
>> From: Wanpeng Li <wanpengli@tencent.com>
>>
>> The per-cpu vsyscall pvclock data pointer assigns either an element of the
>> static array hv_clock_boot (#vCPU <= 64) or dynamically allocated memory
>> hvclock_mem (vCPU > 64), the dynamically memory will not be allocated if
>> kvmclock vsyscall is disabled, this can result in cpu hotpluged fails in
>> kvmclock_setup_percpu() which returns -ENOMEM. This patch fixes it by not
>> assigning vsyscall pvclock data pointer if kvmclock vdso_clock_mode is not
>> VDSO_CLOCKMODE_PVCLOCK.

I am sorry, I still cannot figure out this patch.

Is hotplug still broken if kvm vsyscall is enabled?

Paolo

>> Fixes: 6a1cac56f4 ("x86/kvm: Use __bss_decrypted attribute in shared variables")
>> Reported-by: Zelin Deng <zelin.deng@linux.alibaba.com>
>> Tested-by: Haiwei Li <lihaiwei@tencent.com>
>> Cc: Brijesh Singh <brijesh.singh@amd.com>
>> Cc: stable@vger.kernel.org#v4.19-rc5+
>> Signed-off-by: Wanpeng Li <wanpengli@tencent.com>
>> ---
>> v1 -> v2:
>>   * add code comments
>>
>>   arch/x86/kernel/kvmclock.c | 6 ++++--
>>   1 file changed, 4 insertions(+), 2 deletions(-)
>>
>> diff --git a/arch/x86/kernel/kvmclock.c b/arch/x86/kernel/kvmclock.c
>> index aa59374..01d4e55c 100644
>> --- a/arch/x86/kernel/kvmclock.c
>> +++ b/arch/x86/kernel/kvmclock.c
>> @@ -294,9 +294,11 @@ static int kvmclock_setup_percpu(unsigned int cpu)
>>          /*
>>           * The per cpu area setup replicates CPU0 data to all cpu
>>           * pointers. So carefully check. CPU0 has been set up in init
>> -        * already.
>> +        * already. Assign vsyscall pvclock data pointer iff kvmclock
>> +        * vsyscall is enabled.
>>           */
>> -       if (!cpu || (p && p != per_cpu(hv_clock_per_cpu, 0)))
>> +       if (!cpu || (p && p != per_cpu(hv_clock_per_cpu, 0)) ||
>> +           (kvm_clock.vdso_clock_mode != VDSO_CLOCKMODE_PVCLOCK))
>>                  return 0;
>>
>>          /* Use the static page for the first CPUs, allocate otherwise */
>> --
>> 2.7.4
>>
> 

Re: [PATCH v2] KVM: kvmclock: Fix vCPUs > 64 can't be online/hotpluged

[Wanpeng Li]

On Wed, 27 Jan 2021 at 01:26, Paolo Bonzini <pbonzini@redhat.com> wrote:
>
> On 26/01/21 02:28, Wanpeng Li wrote:
> > ping，
> > On Mon, 18 Jan 2021 at 17:08, Wanpeng Li <kernellwp@gmail.com> wrote:
> >>
> >> From: Wanpeng Li <wanpengli@tencent.com>
> >>
> >> The per-cpu vsyscall pvclock data pointer assigns either an element of the
> >> static array hv_clock_boot (#vCPU <= 64) or dynamically allocated memory
> >> hvclock_mem (vCPU > 64), the dynamically memory will not be allocated if
> >> kvmclock vsyscall is disabled, this can result in cpu hotpluged fails in
> >> kvmclock_setup_percpu() which returns -ENOMEM. This patch fixes it by not
> >> assigning vsyscall pvclock data pointer if kvmclock vdso_clock_mode is not
> >> VDSO_CLOCKMODE_PVCLOCK.
>
> I am sorry, I still cannot figure out this patch.
>
> Is hotplug still broken if kvm vsyscall is enabled?

Just when kvm vsyscall is disabled. :)

# lscpu
Architecture:           x86_64
CPU op-mode(s):    32-bit, 64-bit
Byte Order:             Little Endian
CPU(s):                   88
On-line CPU(s) list:   0-63
Off-line CPU(s) list:  64-87

# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-5.10.0-rc3-tlinux2-0050+ root=/dev/mapper/cl-root
ro rd.lvm.lv=cl/root rhgb quiet console=ttyS0 LANG=en_US
.UTF-8 no-kvmclock-vsyscall

# echo 1 > /sys/devices/system/cpu/cpu76/online
-bash: echo: write error: Cannot allocate memory

    Wanpeng

Re: [PATCH v2] KVM: kvmclock: Fix vCPUs > 64 can't be online/hotpluged

[Wanpeng Li]

On Wed, 27 Jan 2021 at 08:28, Wanpeng Li <kernellwp@gmail.com> wrote:
>
> On Wed, 27 Jan 2021 at 01:26, Paolo Bonzini <pbonzini@redhat.com> wrote:
> >
> > On 26/01/21 02:28, Wanpeng Li wrote:
> > > ping，
> > > On Mon, 18 Jan 2021 at 17:08, Wanpeng Li <kernellwp@gmail.com> wrote:
> > >>
> > >> From: Wanpeng Li <wanpengli@tencent.com>
> > >>
> > >> The per-cpu vsyscall pvclock data pointer assigns either an element of the
> > >> static array hv_clock_boot (#vCPU <= 64) or dynamically allocated memory
> > >> hvclock_mem (vCPU > 64), the dynamically memory will not be allocated if
> > >> kvmclock vsyscall is disabled, this can result in cpu hotpluged fails in
> > >> kvmclock_setup_percpu() which returns -ENOMEM. This patch fixes it by not
> > >> assigning vsyscall pvclock data pointer if kvmclock vdso_clock_mode is not
> > >> VDSO_CLOCKMODE_PVCLOCK.
> >
> > I am sorry, I still cannot figure out this patch.
> >
> > Is hotplug still broken if kvm vsyscall is enabled?
>
> Just when kvm vsyscall is disabled. :)
>
> # lscpu
> Architecture:           x86_64
> CPU op-mode(s):    32-bit, 64-bit
> Byte Order:             Little Endian
> CPU(s):                   88
> On-line CPU(s) list:   0-63
> Off-line CPU(s) list:  64-87
>
> # cat /proc/cmdline
> BOOT_IMAGE=/vmlinuz-5.10.0-rc3-tlinux2-0050+ root=/dev/mapper/cl-root
> ro rd.lvm.lv=cl/root rhgb quiet console=ttyS0 LANG=en_US
> .UTF-8 no-kvmclock-vsyscall
>
> # echo 1 > /sys/devices/system/cpu/cpu76/online
> -bash: echo: write error: Cannot allocate memory

The original bug report is here.
https://bugzilla.kernel.org/show_bug.cgi?id=210213

    Wanpeng