[PATCH v2] xen: avoid crash in disable_hotplug_cpu

[PATCH v2] xen: avoid crash in disable_hotplug_cpu

[Olaf Hering]

The command 'xl vcpu-set 0 0', issued in dom0, will crash dom0:

BUG: unable to handle kernel NULL pointer dereference at 00000000000002d8
PGD 0 P4D 0
Oops: 0000 [#1] PREEMPT SMP NOPTI
CPU: 7 PID: 65 Comm: xenwatch Not tainted 4.19.0-rc2-1.ga9462db-default #1 openSUSE Tumbleweed (unreleased)
Hardware name: Intel Corporation S5520UR/S5520UR, BIOS S5500.86B.01.00.0050.050620101605 05/06/2010
RIP: e030:device_offline+0x9/0xb0
Code: 77 24 00 e9 ce fe ff ff 48 8b 13 e9 68 ff ff ff 48 8b 13 e9 29 ff ff ff 48 8b 13 e9 ea fe ff ff 90 66 66 66 66 90 41 54 55 53 <f6> 87 d8 02 00 00 01 0f 85 88 00 00 00 48 c7 c2 20 09 60 81 31 f6
RSP: e02b:ffffc90040f27e80 EFLAGS: 00010203
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
RDX: ffff8801f3800000 RSI: ffffc90040f27e70 RDI: 0000000000000000
RBP: 0000000000000000 R08: ffffffff820e47b3 R09: 0000000000000000
R10: 0000000000007ff0 R11: 0000000000000000 R12: ffffffff822e6d30
R13: dead000000000200 R14: dead000000000100 R15: ffffffff8158b4e0
FS:  00007ffa595158c0(0000) GS:ffff8801f39c0000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000002d8 CR3: 00000001d9602000 CR4: 0000000000002660
Call Trace:
 handle_vcpu_hotplug_event+0xb5/0xc0
 xenwatch_thread+0x80/0x140
 ? wait_woken+0x80/0x80
 kthread+0x112/0x130
 ? kthread_create_worker_on_cpu+0x40/0x40
 ret_from_fork+0x3a/0x50

This happens because handle_vcpu_hotplug_event is called twice. In the
first iteration cpu_present is still true, in the second iteration
cpu_present is false which causes get_cpu_device to return NULL.
In case of cpu#0, cpu_online is apparently always true.

Fix this crash by checking if the cpu can be hotplugged, which is false
for a cpu that was just removed.

Signed-off-by: Olaf Hering <olaf@aepfle.de>
---
 drivers/xen/cpu_hotplug.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/xen/cpu_hotplug.c b/drivers/xen/cpu_hotplug.c
index d4265c8ebb22..bf1e41ed9d41 100644
--- a/drivers/xen/cpu_hotplug.c
+++ b/drivers/xen/cpu_hotplug.c
@@ -19,6 +19,8 @@ static void enable_hotplug_cpu(int cpu)
 
 static void disable_hotplug_cpu(int cpu)
 {
+	if (!cpu_is_hotpluggable(cpu))
+		return;
 	if (cpu_online(cpu)) {
 		lock_device_hotplug();
 		device_offline(get_cpu_device(cpu));

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH v2] xen: avoid crash in disable_hotplug_cpu

[Olaf Hering]

[-- Attachment #1: Type: text/plain, Size: 319 bytes --]

Am Thu, 6 Sep 2018 22:31:45 +0200
schrieb Olaf Hering <olaf@aepfle.de>:

> IF cpu0_hotpluggable is broken, then only "if (!cpu) return;" can help.

Another option is to do cpu_online() twice to check if device_offline did anything.
Not sure if the compiler would fold the two checks into a single check.

Olaf

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [PATCH v2] xen: avoid crash in disable_hotplug_cpu

[Olaf Hering]

[-- Attachment #1.1: Type: text/plain, Size: 319 bytes --]

Am Thu, 6 Sep 2018 22:31:45 +0200
schrieb Olaf Hering <olaf@aepfle.de>:

> IF cpu0_hotpluggable is broken, then only "if (!cpu) return;" can help.

Another option is to do cpu_online() twice to check if device_offline did anything.
Not sure if the compiler would fold the two checks into a single check.

Olaf

[-- Attachment #1.2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH v2] xen: avoid crash in disable_hotplug_cpu

[Juergen Gross]

On 06/09/18 22:31, Olaf Hering wrote:
> Am Thu, 6 Sep 2018 14:45:57 -0400
> schrieb Boris Ostrovsky <boris.ostrovsky@oracle.com>:
> 
>> On 09/06/2018 02:37 AM, Olaf Hering wrote:
>>> The command 'xl vcpu-set 0 0', issued in dom0, will crash dom0:
> 
>>> This happens because handle_vcpu_hotplug_event is called twice. In the
>>> first iteration cpu_present is still true, in the second iteration
>>> cpu_present is false which causes get_cpu_device to return NULL.
>>> In case of cpu#0, cpu_online is apparently always true.
> 
>> I think we should check both this and num_online_cpus() != 0.
> 
> This can not possibly help. cpu#0 is the first one that goes offline.
> IF cpu0_hotpluggable is broken, then only "if (!cpu) return;" can help.

We should add the Xen PV guest test to arch_register_cpu() and switch
cpu0_hotpluggable off.


Juergen

Re: [PATCH v2] xen: avoid crash in disable_hotplug_cpu

[Juergen Gross]

On 06/09/18 22:31, Olaf Hering wrote:
> Am Thu, 6 Sep 2018 14:45:57 -0400
> schrieb Boris Ostrovsky <boris.ostrovsky@oracle.com>:
> 
>> On 09/06/2018 02:37 AM, Olaf Hering wrote:
>>> The command 'xl vcpu-set 0 0', issued in dom0, will crash dom0:
> 
>>> This happens because handle_vcpu_hotplug_event is called twice. In the
>>> first iteration cpu_present is still true, in the second iteration
>>> cpu_present is false which causes get_cpu_device to return NULL.
>>> In case of cpu#0, cpu_online is apparently always true.
> 
>> I think we should check both this and num_online_cpus() != 0.
> 
> This can not possibly help. cpu#0 is the first one that goes offline.
> IF cpu0_hotpluggable is broken, then only "if (!cpu) return;" can help.

We should add the Xen PV guest test to arch_register_cpu() and switch
cpu0_hotpluggable off.


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH v2] xen: avoid crash in disable_hotplug_cpu

[Boris Ostrovsky]

[-- Attachment #1.1: Type: text/plain, Size: 938 bytes --]

On 09/06/2018 04:31 PM, Olaf Hering wrote:
> Am Thu, 6 Sep 2018 14:45:57 -0400
> schrieb Boris Ostrovsky <boris.ostrovsky@oracle.com>:
>
>> On 09/06/2018 02:37 AM, Olaf Hering wrote:
>>> The command 'xl vcpu-set 0 0', issued in dom0, will crash dom0:
>>> This happens because handle_vcpu_hotplug_event is called twice. In the
>>> first iteration cpu_present is still true, in the second iteration
>>> cpu_present is false which causes get_cpu_device to return NULL.
>>> In case of cpu#0, cpu_online is apparently always true.
>> I think we should check both this and num_online_cpus() != 0.
> This can not possibly help. cpu#0 is the first one that goes offline.
> IF cpu0_hotpluggable is broken, then only "if (!cpu) return;" can help.


And maybe that needs to be part of the check, in addition to
cpu_is_hotpluggable() test.

Offlining CPU0 is problematic. For example, look at xen_pv_cpu_disable().



-boris


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v2] xen: avoid crash in disable_hotplug_cpu

[Boris Ostrovsky]

[-- Attachment #1.1.1: Type: text/plain, Size: 938 bytes --]

On 09/06/2018 04:31 PM, Olaf Hering wrote:
> Am Thu, 6 Sep 2018 14:45:57 -0400
> schrieb Boris Ostrovsky <boris.ostrovsky@oracle.com>:
>
>> On 09/06/2018 02:37 AM, Olaf Hering wrote:
>>> The command 'xl vcpu-set 0 0', issued in dom0, will crash dom0:
>>> This happens because handle_vcpu_hotplug_event is called twice. In the
>>> first iteration cpu_present is still true, in the second iteration
>>> cpu_present is false which causes get_cpu_device to return NULL.
>>> In case of cpu#0, cpu_online is apparently always true.
>> I think we should check both this and num_online_cpus() != 0.
> This can not possibly help. cpu#0 is the first one that goes offline.
> IF cpu0_hotpluggable is broken, then only "if (!cpu) return;" can help.


And maybe that needs to be part of the check, in addition to
cpu_is_hotpluggable() test.

Offlining CPU0 is problematic. For example, look at xen_pv_cpu_disable().



-boris


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH v2] xen: avoid crash in disable_hotplug_cpu

[Olaf Hering]

[-- Attachment #1: Type: text/plain, Size: 709 bytes --]

Am Thu, 6 Sep 2018 14:45:57 -0400
schrieb Boris Ostrovsky <boris.ostrovsky@oracle.com>:

> On 09/06/2018 02:37 AM, Olaf Hering wrote:
> > The command 'xl vcpu-set 0 0', issued in dom0, will crash dom0:

> > This happens because handle_vcpu_hotplug_event is called twice. In the
> > first iteration cpu_present is still true, in the second iteration
> > cpu_present is false which causes get_cpu_device to return NULL.
> > In case of cpu#0, cpu_online is apparently always true.

> I think we should check both this and num_online_cpus() != 0.

This can not possibly help. cpu#0 is the first one that goes offline.
IF cpu0_hotpluggable is broken, then only "if (!cpu) return;" can help.

Olaf

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [PATCH v2] xen: avoid crash in disable_hotplug_cpu

[Olaf Hering]

[-- Attachment #1.1: Type: text/plain, Size: 709 bytes --]

Am Thu, 6 Sep 2018 14:45:57 -0400
schrieb Boris Ostrovsky <boris.ostrovsky@oracle.com>:

> On 09/06/2018 02:37 AM, Olaf Hering wrote:
> > The command 'xl vcpu-set 0 0', issued in dom0, will crash dom0:

> > This happens because handle_vcpu_hotplug_event is called twice. In the
> > first iteration cpu_present is still true, in the second iteration
> > cpu_present is false which causes get_cpu_device to return NULL.
> > In case of cpu#0, cpu_online is apparently always true.

> I think we should check both this and num_online_cpus() != 0.

This can not possibly help. cpu#0 is the first one that goes offline.
IF cpu0_hotpluggable is broken, then only "if (!cpu) return;" can help.

Olaf

[-- Attachment #1.2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH v2] xen: avoid crash in disable_hotplug_cpu

[Boris Ostrovsky]

On 09/06/2018 02:37 AM, Olaf Hering wrote:
> The command 'xl vcpu-set 0 0', issued in dom0, will crash dom0:
>
> BUG: unable to handle kernel NULL pointer dereference at 00000000000002d8
> PGD 0 P4D 0
> Oops: 0000 [#1] PREEMPT SMP NOPTI
> CPU: 7 PID: 65 Comm: xenwatch Not tainted 4.19.0-rc2-1.ga9462db-default #1 openSUSE Tumbleweed (unreleased)
> Hardware name: Intel Corporation S5520UR/S5520UR, BIOS S5500.86B.01.00.0050.050620101605 05/06/2010
> RIP: e030:device_offline+0x9/0xb0
> Code: 77 24 00 e9 ce fe ff ff 48 8b 13 e9 68 ff ff ff 48 8b 13 e9 29 ff ff ff 48 8b 13 e9 ea fe ff ff 90 66 66 66 66 90 41 54 55 53 <f6> 87 d8 02 00 00 01 0f 85 88 00 00 00 48 c7 c2 20 09 60 81 31 f6
> RSP: e02b:ffffc90040f27e80 EFLAGS: 00010203
> RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
> RDX: ffff8801f3800000 RSI: ffffc90040f27e70 RDI: 0000000000000000
> RBP: 0000000000000000 R08: ffffffff820e47b3 R09: 0000000000000000
> R10: 0000000000007ff0 R11: 0000000000000000 R12: ffffffff822e6d30
> R13: dead000000000200 R14: dead000000000100 R15: ffffffff8158b4e0
> FS:  00007ffa595158c0(0000) GS:ffff8801f39c0000(0000) knlGS:0000000000000000
> CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00000000000002d8 CR3: 00000001d9602000 CR4: 0000000000002660
> Call Trace:
>  handle_vcpu_hotplug_event+0xb5/0xc0
>  xenwatch_thread+0x80/0x140
>  ? wait_woken+0x80/0x80
>  kthread+0x112/0x130
>  ? kthread_create_worker_on_cpu+0x40/0x40
>  ret_from_fork+0x3a/0x50
>
> This happens because handle_vcpu_hotplug_event is called twice. In the
> first iteration cpu_present is still true, in the second iteration
> cpu_present is false which causes get_cpu_device to return NULL.
> In case of cpu#0, cpu_online is apparently always true.
>
> Fix this crash by checking if the cpu can be hotplugged, which is false
> for a cpu that was just removed.
>
> Signed-off-by: Olaf Hering <olaf@aepfle.de>
> ---
>  drivers/xen/cpu_hotplug.c | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/drivers/xen/cpu_hotplug.c b/drivers/xen/cpu_hotplug.c
> index d4265c8ebb22..bf1e41ed9d41 100644
> --- a/drivers/xen/cpu_hotplug.c
> +++ b/drivers/xen/cpu_hotplug.c
> @@ -19,6 +19,8 @@ static void enable_hotplug_cpu(int cpu)
>  
>  static void disable_hotplug_cpu(int cpu)
>  {
> +	if (!cpu_is_hotpluggable(cpu))


I think we should check both this and num_online_cpus() != 0.

Even though I don't believe cpu0_hotpluggable currently works, at some
point it might.

-boris


> +		return;
>  	if (cpu_online(cpu)) {
>  		lock_device_hotplug();
>  		device_offline(get_cpu_device(cpu));

Re: [PATCH v2] xen: avoid crash in disable_hotplug_cpu

[Boris Ostrovsky]

On 09/06/2018 02:37 AM, Olaf Hering wrote:
> The command 'xl vcpu-set 0 0', issued in dom0, will crash dom0:
>
> BUG: unable to handle kernel NULL pointer dereference at 00000000000002d8
> PGD 0 P4D 0
> Oops: 0000 [#1] PREEMPT SMP NOPTI
> CPU: 7 PID: 65 Comm: xenwatch Not tainted 4.19.0-rc2-1.ga9462db-default #1 openSUSE Tumbleweed (unreleased)
> Hardware name: Intel Corporation S5520UR/S5520UR, BIOS S5500.86B.01.00.0050.050620101605 05/06/2010
> RIP: e030:device_offline+0x9/0xb0
> Code: 77 24 00 e9 ce fe ff ff 48 8b 13 e9 68 ff ff ff 48 8b 13 e9 29 ff ff ff 48 8b 13 e9 ea fe ff ff 90 66 66 66 66 90 41 54 55 53 <f6> 87 d8 02 00 00 01 0f 85 88 00 00 00 48 c7 c2 20 09 60 81 31 f6
> RSP: e02b:ffffc90040f27e80 EFLAGS: 00010203
> RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
> RDX: ffff8801f3800000 RSI: ffffc90040f27e70 RDI: 0000000000000000
> RBP: 0000000000000000 R08: ffffffff820e47b3 R09: 0000000000000000
> R10: 0000000000007ff0 R11: 0000000000000000 R12: ffffffff822e6d30
> R13: dead000000000200 R14: dead000000000100 R15: ffffffff8158b4e0
> FS:  00007ffa595158c0(0000) GS:ffff8801f39c0000(0000) knlGS:0000000000000000
> CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00000000000002d8 CR3: 00000001d9602000 CR4: 0000000000002660
> Call Trace:
>  handle_vcpu_hotplug_event+0xb5/0xc0
>  xenwatch_thread+0x80/0x140
>  ? wait_woken+0x80/0x80
>  kthread+0x112/0x130
>  ? kthread_create_worker_on_cpu+0x40/0x40
>  ret_from_fork+0x3a/0x50
>
> This happens because handle_vcpu_hotplug_event is called twice. In the
> first iteration cpu_present is still true, in the second iteration
> cpu_present is false which causes get_cpu_device to return NULL.
> In case of cpu#0, cpu_online is apparently always true.
>
> Fix this crash by checking if the cpu can be hotplugged, which is false
> for a cpu that was just removed.
>
> Signed-off-by: Olaf Hering <olaf@aepfle.de>
> ---
>  drivers/xen/cpu_hotplug.c | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/drivers/xen/cpu_hotplug.c b/drivers/xen/cpu_hotplug.c
> index d4265c8ebb22..bf1e41ed9d41 100644
> --- a/drivers/xen/cpu_hotplug.c
> +++ b/drivers/xen/cpu_hotplug.c
> @@ -19,6 +19,8 @@ static void enable_hotplug_cpu(int cpu)
>  
>  static void disable_hotplug_cpu(int cpu)
>  {
> +	if (!cpu_is_hotpluggable(cpu))


I think we should check both this and num_online_cpus() != 0.

Even though I don't believe cpu0_hotpluggable currently works, at some
point it might.

-boris


> +		return;
>  	if (cpu_online(cpu)) {
>  		lock_device_hotplug();
>  		device_offline(get_cpu_device(cpu));


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH v2] xen: avoid crash in disable_hotplug_cpu

[Olaf Hering]

The command 'xl vcpu-set 0 0', issued in dom0, will crash dom0:

BUG: unable to handle kernel NULL pointer dereference at 00000000000002d8
PGD 0 P4D 0
Oops: 0000 [#1] PREEMPT SMP NOPTI
CPU: 7 PID: 65 Comm: xenwatch Not tainted 4.19.0-rc2-1.ga9462db-default #1 openSUSE Tumbleweed (unreleased)
Hardware name: Intel Corporation S5520UR/S5520UR, BIOS S5500.86B.01.00.0050.050620101605 05/06/2010
RIP: e030:device_offline+0x9/0xb0
Code: 77 24 00 e9 ce fe ff ff 48 8b 13 e9 68 ff ff ff 48 8b 13 e9 29 ff ff ff 48 8b 13 e9 ea fe ff ff 90 66 66 66 66 90 41 54 55 53 <f6> 87 d8 02 00 00 01 0f 85 88 00 00 00 48 c7 c2 20 09 60 81 31 f6
RSP: e02b:ffffc90040f27e80 EFLAGS: 00010203
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
RDX: ffff8801f3800000 RSI: ffffc90040f27e70 RDI: 0000000000000000
RBP: 0000000000000000 R08: ffffffff820e47b3 R09: 0000000000000000
R10: 0000000000007ff0 R11: 0000000000000000 R12: ffffffff822e6d30
R13: dead000000000200 R14: dead000000000100 R15: ffffffff8158b4e0
FS:  00007ffa595158c0(0000) GS:ffff8801f39c0000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000002d8 CR3: 00000001d9602000 CR4: 0000000000002660
Call Trace:
 handle_vcpu_hotplug_event+0xb5/0xc0
 xenwatch_thread+0x80/0x140
 ? wait_woken+0x80/0x80
 kthread+0x112/0x130
 ? kthread_create_worker_on_cpu+0x40/0x40
 ret_from_fork+0x3a/0x50

This happens because handle_vcpu_hotplug_event is called twice. In the
first iteration cpu_present is still true, in the second iteration
cpu_present is false which causes get_cpu_device to return NULL.
In case of cpu#0, cpu_online is apparently always true.

Fix this crash by checking if the cpu can be hotplugged, which is false
for a cpu that was just removed.

Signed-off-by: Olaf Hering <olaf@aepfle.de>
---
 drivers/xen/cpu_hotplug.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/xen/cpu_hotplug.c b/drivers/xen/cpu_hotplug.c
index d4265c8ebb22..bf1e41ed9d41 100644
--- a/drivers/xen/cpu_hotplug.c
+++ b/drivers/xen/cpu_hotplug.c
@@ -19,6 +19,8 @@ static void enable_hotplug_cpu(int cpu)
 
 static void disable_hotplug_cpu(int cpu)
 {
+	if (!cpu_is_hotpluggable(cpu))
+		return;
 	if (cpu_online(cpu)) {
 		lock_device_hotplug();
 		device_offline(get_cpu_device(cpu));