* RE: [PATCH v2] Drivers: hv: vmbus: prevent cpu offlining on newer hypervisors
2014-12-01 10:52 ` [PATCH v2] " Vitaly Kuznetsov
@ 2014-12-01 11:12 ` Dexuan Cui
2015-01-09 20:57 ` Greg Kroah-Hartman
2015-01-12 16:50 ` [PATCH v3] " Vitaly Kuznetsov
2 siblings, 0 replies; 14+ messages in thread
From: Dexuan Cui @ 2014-12-01 11:12 UTC (permalink / raw)
To: Vitaly Kuznetsov, KY Srinivasan, Haiyang Zhang
Cc: devel, Greg Kroah-Hartman, linux-kernel
> -----Original Message-----
> From: Vitaly Kuznetsov [mailto:vkuznets@redhat.com]
> Sent: Monday, December 1, 2014 18:53 PM
> To: KY Srinivasan; Haiyang Zhang
> Cc: devel@linuxdriverproject.org; Greg Kroah-Hartman; linux-
> kernel@vger.kernel.org; Dexuan Cui
> Subject: [PATCH v2] Drivers: hv: vmbus: prevent cpu offlining on newer
> hypervisors
>
> When an SMP Hyper-V guest is running on top of 2012R2 Server and
> secondary
> cpus are sent offline (with echo 0 > /sys/devices/system/cpu/cpu$cpu/online)
> the system freeze is observed. This happens due to the fact that on newer
> hypervisors (Win8, WS2012R2, ...) vmbus channel handlers are distributed
> across all cpus (see init_vp_index() function in drivers/hv/channel_mgmt.c)
> and on cpu offlining nobody reassigns them to CPU0. Prevent cpu offlining
> when vmbus is loaded until the issue is fixed host-side.
>
> This patch also disables hibernation but it is OK as it is also broken (MCE
> error is hit on resume). Suspend still works.
>
> Tested with WS2008R2 and WS2012R2.
>
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
>
> ---
> Changes since v1:
> - introduce hv_cpu_hotplug_quirk() function to not spread #ifdefs [Greg KH]
> - add pr_notice() message "hv_vmbus: CPU offlining is not supported by
> hypervisor"
> ---
> drivers/hv/vmbus_drv.c | 33 +++++++++++++++++++++++++++++++++
> 1 file changed, 33 insertions(+)
>
> diff --git a/drivers/hv/vmbus_drv.c b/drivers/hv/vmbus_drv.c
> index 4d6b269..2e6b38e 100644
> --- a/drivers/hv/vmbus_drv.c
> +++ b/drivers/hv/vmbus_drv.c
> @@ -32,6 +32,7 @@
> #include <linux/completion.h>
> #include <linux/hyperv.h>
> #include <linux/kernel_stat.h>
> +#include <linux/cpu.h>
> #include <asm/hyperv.h>
> #include <asm/hypervisor.h>
> #include <asm/mshyperv.h>
> @@ -671,6 +672,36 @@ static void vmbus_isr(void)
> tasklet_schedule(&msg_dpc);
> }
>
> +#ifdef CONFIG_HOTPLUG_CPU
> +static int hyperv_cpu_disable(void)
> +{
> + return -1;
> +}
> +
> +static void hv_cpu_hotplug_quirk(bool vmbus_loaded)
> +{
> + /*
> + * Offlining a CPU when running on newer hypervisors (WS2012R2,
> Win8,
> + * ...) is not supported at this moment as channel interrupts are
> + * distributed across all of them.
> + */
> +
> + if ((vmbus_proto_version == VERSION_WS2008) ||
> + (vmbus_proto_version == VERSION_WIN7))
> + return;
> +
> + if (vmbus_loaded) {
> + smp_ops.cpu_disable = hyperv_cpu_disable;
> + pr_notice("CPU offlining is not supported by hypervisor");
> + } else
> + smp_ops.cpu_disable = native_cpu_disable;
> +}
> +#else
> +static void hv_cpu_hotplug_quirk(bool vmbus_loaded)
> +{
> +}
> +#endif
> +
> /*
> * vmbus_bus_init -Main vmbus driver initialization routine.
> *
> @@ -711,6 +742,7 @@ static int vmbus_bus_init(int irq)
> if (ret)
> goto err_alloc;
>
> + hv_cpu_hotplug_quirk(true);
> vmbus_request_offers();
>
> return 0;
> @@ -964,6 +996,7 @@ static void __exit vmbus_exit(void)
> bus_unregister(&hv_bus);
> hv_cleanup();
> acpi_bus_unregister_driver(&vmbus_acpi_driver);
> + hv_cpu_hotplug_quirk(false);
> }
Acked-by: Dexuan Cui <decui@microsoft.com>
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: [PATCH v2] Drivers: hv: vmbus: prevent cpu offlining on newer hypervisors
2014-12-01 10:52 ` [PATCH v2] " Vitaly Kuznetsov
2014-12-01 11:12 ` Dexuan Cui
@ 2015-01-09 20:57 ` Greg Kroah-Hartman
2015-01-12 16:50 ` [PATCH v3] " Vitaly Kuznetsov
2 siblings, 0 replies; 14+ messages in thread
From: Greg Kroah-Hartman @ 2015-01-09 20:57 UTC (permalink / raw)
To: Vitaly Kuznetsov
Cc: K. Y. Srinivasan, Haiyang Zhang, devel, linux-kernel, Dexuan Cui
On Mon, Dec 01, 2014 at 11:52:41AM +0100, Vitaly Kuznetsov wrote:
> When an SMP Hyper-V guest is running on top of 2012R2 Server and secondary
> cpus are sent offline (with echo 0 > /sys/devices/system/cpu/cpu$cpu/online)
> the system freeze is observed. This happens due to the fact that on newer
> hypervisors (Win8, WS2012R2, ...) vmbus channel handlers are distributed
> across all cpus (see init_vp_index() function in drivers/hv/channel_mgmt.c)
> and on cpu offlining nobody reassigns them to CPU0. Prevent cpu offlining
> when vmbus is loaded until the issue is fixed host-side.
>
> This patch also disables hibernation but it is OK as it is also broken (MCE
> error is hit on resume). Suspend still works.
>
> Tested with WS2008R2 and WS2012R2.
>
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> Acked-by: Dexuan Cui <decui@microsoft.com>
>
> ---
> Changes since v1:
> - introduce hv_cpu_hotplug_quirk() function to not spread #ifdefs [Greg KH]
> - add pr_notice() message "hv_vmbus: CPU offlining is not supported by hypervisor"
> ---
> drivers/hv/vmbus_drv.c | 33 +++++++++++++++++++++++++++++++++
> 1 file changed, 33 insertions(+)
>
> diff --git a/drivers/hv/vmbus_drv.c b/drivers/hv/vmbus_drv.c
> index 4d6b269..2e6b38e 100644
> --- a/drivers/hv/vmbus_drv.c
> +++ b/drivers/hv/vmbus_drv.c
> @@ -32,6 +32,7 @@
> #include <linux/completion.h>
> #include <linux/hyperv.h>
> #include <linux/kernel_stat.h>
> +#include <linux/cpu.h>
> #include <asm/hyperv.h>
> #include <asm/hypervisor.h>
> #include <asm/mshyperv.h>
> @@ -671,6 +672,36 @@ static void vmbus_isr(void)
> tasklet_schedule(&msg_dpc);
> }
>
> +#ifdef CONFIG_HOTPLUG_CPU
> +static int hyperv_cpu_disable(void)
> +{
> + return -1;
> +}
> +
> +static void hv_cpu_hotplug_quirk(bool vmbus_loaded)
> +{
> + /*
> + * Offlining a CPU when running on newer hypervisors (WS2012R2, Win8,
> + * ...) is not supported at this moment as channel interrupts are
> + * distributed across all of them.
> + */
> +
> + if ((vmbus_proto_version == VERSION_WS2008) ||
> + (vmbus_proto_version == VERSION_WIN7))
> + return;
> +
> + if (vmbus_loaded) {
> + smp_ops.cpu_disable = hyperv_cpu_disable;
> + pr_notice("CPU offlining is not supported by hypervisor");
> + } else
> + smp_ops.cpu_disable = native_cpu_disable;
> +}
> +#else
> +static void hv_cpu_hotplug_quirk(bool vmbus_loaded)
> +{
> +}
> +#endif
> +
> /*
> * vmbus_bus_init -Main vmbus driver initialization routine.
> *
> @@ -711,6 +742,7 @@ static int vmbus_bus_init(int irq)
> if (ret)
> goto err_alloc;
>
> + hv_cpu_hotplug_quirk(true);
> vmbus_request_offers();
>
> return 0;
> @@ -964,6 +996,7 @@ static void __exit vmbus_exit(void)
> bus_unregister(&hv_bus);
> hv_cleanup();
> acpi_bus_unregister_driver(&vmbus_acpi_driver);
> + hv_cpu_hotplug_quirk(false);
> }
>
>
> --
> 1.9.3
This breaks the build:
ERROR: "native_cpu_disable" [drivers/hv/hv_vmbus.ko] undefined!
Please test better.
greg k-h
^ permalink raw reply [flat|nested] 14+ messages in thread* [PATCH v3] Drivers: hv: vmbus: prevent cpu offlining on newer hypervisors
2014-12-01 10:52 ` [PATCH v2] " Vitaly Kuznetsov
2014-12-01 11:12 ` Dexuan Cui
2015-01-09 20:57 ` Greg Kroah-Hartman
@ 2015-01-12 16:50 ` Vitaly Kuznetsov
2015-01-12 18:54 ` KY Srinivasan
2015-01-25 13:36 ` Greg Kroah-Hartman
2 siblings, 2 replies; 14+ messages in thread
From: Vitaly Kuznetsov @ 2015-01-12 16:50 UTC (permalink / raw)
To: K. Y. Srinivasan, devel
Cc: Haiyang Zhang, Greg Kroah-Hartman, linux-kernel, Dexuan Cui
When an SMP Hyper-V guest is running on top of 2012R2 Server and secondary
cpus are sent offline (with echo 0 > /sys/devices/system/cpu/cpu$cpu/online)
the system freeze is observed. This happens due to the fact that on newer
hypervisors (Win8, WS2012R2, ...) vmbus channel handlers are distributed
across all cpus (see init_vp_index() function in drivers/hv/channel_mgmt.c)
and on cpu offlining nobody reassigns them to CPU0. Prevent cpu offlining
when vmbus is loaded until the issue is fixed host-side.
This patch also disables hibernation but it is OK as it is also broken (MCE
error is hit on resume). Suspend still works.
Tested with WS2008R2 and WS2012R2.
Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
Changes since v2:
- repair the build when vmbus is builded as a module [Greg KH] by saving
current cpu_disable pointer to previous_cpu_disable and restoring it on
unload;
- return -ENOSYS (same as native_cpu_disable when !CONFIG_HOTPLUG_CPU) instead
of -1 in hyperv_cpu_disable().
Changes since v1:
- introduce hv_cpu_hotplug_quirk() function to not spread #ifdefs [Greg KH];
- add pr_notice() message "hv_vmbus: CPU offlining is not supported by
hypervisor".
---
drivers/hv/vmbus_drv.c | 36 ++++++++++++++++++++++++++++++++++++
1 file changed, 36 insertions(+)
diff --git a/drivers/hv/vmbus_drv.c b/drivers/hv/vmbus_drv.c
index 4d6b269..233da0b 100644
--- a/drivers/hv/vmbus_drv.c
+++ b/drivers/hv/vmbus_drv.c
@@ -32,6 +32,7 @@
#include <linux/completion.h>
#include <linux/hyperv.h>
#include <linux/kernel_stat.h>
+#include <linux/cpu.h>
#include <asm/hyperv.h>
#include <asm/hypervisor.h>
#include <asm/mshyperv.h>
@@ -671,6 +672,39 @@ static void vmbus_isr(void)
tasklet_schedule(&msg_dpc);
}
+#ifdef CONFIG_HOTPLUG_CPU
+static int hyperv_cpu_disable(void)
+{
+ return -ENOSYS;
+}
+
+static void hv_cpu_hotplug_quirk(bool vmbus_loaded)
+{
+ static void *previous_cpu_disable;
+
+ /*
+ * Offlining a CPU when running on newer hypervisors (WS2012R2, Win8,
+ * ...) is not supported at this moment as channel interrupts are
+ * distributed across all of them.
+ */
+
+ if ((vmbus_proto_version == VERSION_WS2008) ||
+ (vmbus_proto_version == VERSION_WIN7))
+ return;
+
+ if (vmbus_loaded) {
+ previous_cpu_disable = smp_ops.cpu_disable;
+ smp_ops.cpu_disable = hyperv_cpu_disable;
+ pr_notice("CPU offlining is not supported by hypervisor\n");
+ } else if (previous_cpu_disable)
+ smp_ops.cpu_disable = previous_cpu_disable;
+}
+#else
+static void hv_cpu_hotplug_quirk(bool vmbus_loaded)
+{
+}
+#endif
+
/*
* vmbus_bus_init -Main vmbus driver initialization routine.
*
@@ -711,6 +745,7 @@ static int vmbus_bus_init(int irq)
if (ret)
goto err_alloc;
+ hv_cpu_hotplug_quirk(true);
vmbus_request_offers();
return 0;
@@ -964,6 +999,7 @@ static void __exit vmbus_exit(void)
bus_unregister(&hv_bus);
hv_cleanup();
acpi_bus_unregister_driver(&vmbus_acpi_driver);
+ hv_cpu_hotplug_quirk(false);
}
--
1.9.3
^ permalink raw reply related [flat|nested] 14+ messages in thread* RE: [PATCH v3] Drivers: hv: vmbus: prevent cpu offlining on newer hypervisors
2015-01-12 16:50 ` [PATCH v3] " Vitaly Kuznetsov
@ 2015-01-12 18:54 ` KY Srinivasan
2015-01-25 13:36 ` Greg Kroah-Hartman
1 sibling, 0 replies; 14+ messages in thread
From: KY Srinivasan @ 2015-01-12 18:54 UTC (permalink / raw)
To: Vitaly Kuznetsov, devel
Cc: Haiyang Zhang, Greg Kroah-Hartman, linux-kernel, Dexuan Cui
> -----Original Message-----
> From: Vitaly Kuznetsov [mailto:vkuznets@redhat.com]
> Sent: Monday, January 12, 2015 8:50 AM
> To: KY Srinivasan; devel@linuxdriverproject.org
> Cc: Haiyang Zhang; Greg Kroah-Hartman; linux-kernel@vger.kernel.org;
> Dexuan Cui
> Subject: [PATCH v3] Drivers: hv: vmbus: prevent cpu offlining on newer
> hypervisors
>
> When an SMP Hyper-V guest is running on top of 2012R2 Server and
> secondary
> cpus are sent offline (with echo 0 >
> /sys/devices/system/cpu/cpu$cpu/online)
> the system freeze is observed. This happens due to the fact that on newer
> hypervisors (Win8, WS2012R2, ...) vmbus channel handlers are distributed
> across all cpus (see init_vp_index() function in drivers/hv/channel_mgmt.c)
> and on cpu offlining nobody reassigns them to CPU0. Prevent cpu offlining
> when vmbus is loaded until the issue is fixed host-side.
>
> This patch also disables hibernation but it is OK as it is also broken (MCE
> error is hit on resume). Suspend still works.
>
> Tested with WS2008R2 and WS2012R2.
>
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Thank you.
Signed-off-by: K. Y. Srinivasan <kys@microsoft.com>
> ---
> Changes since v2:
> - repair the build when vmbus is builded as a module [Greg KH] by saving
> current cpu_disable pointer to previous_cpu_disable and restoring it on
> unload;
> - return -ENOSYS (same as native_cpu_disable when
> !CONFIG_HOTPLUG_CPU) instead
> of -1 in hyperv_cpu_disable().
>
> Changes since v1:
> - introduce hv_cpu_hotplug_quirk() function to not spread #ifdefs [Greg
> KH];
> - add pr_notice() message "hv_vmbus: CPU offlining is not supported by
> hypervisor".
> ---
> drivers/hv/vmbus_drv.c | 36 ++++++++++++++++++++++++++++++++++++
> 1 file changed, 36 insertions(+)
>
> diff --git a/drivers/hv/vmbus_drv.c b/drivers/hv/vmbus_drv.c
> index 4d6b269..233da0b 100644
> --- a/drivers/hv/vmbus_drv.c
> +++ b/drivers/hv/vmbus_drv.c
> @@ -32,6 +32,7 @@
> #include <linux/completion.h>
> #include <linux/hyperv.h>
> #include <linux/kernel_stat.h>
> +#include <linux/cpu.h>
> #include <asm/hyperv.h>
> #include <asm/hypervisor.h>
> #include <asm/mshyperv.h>
> @@ -671,6 +672,39 @@ static void vmbus_isr(void)
> tasklet_schedule(&msg_dpc);
> }
>
> +#ifdef CONFIG_HOTPLUG_CPU
> +static int hyperv_cpu_disable(void)
> +{
> + return -ENOSYS;
> +}
> +
> +static void hv_cpu_hotplug_quirk(bool vmbus_loaded)
> +{
> + static void *previous_cpu_disable;
> +
> + /*
> + * Offlining a CPU when running on newer hypervisors (WS2012R2,
> Win8,
> + * ...) is not supported at this moment as channel interrupts are
> + * distributed across all of them.
> + */
> +
> + if ((vmbus_proto_version == VERSION_WS2008) ||
> + (vmbus_proto_version == VERSION_WIN7))
> + return;
> +
> + if (vmbus_loaded) {
> + previous_cpu_disable = smp_ops.cpu_disable;
> + smp_ops.cpu_disable = hyperv_cpu_disable;
> + pr_notice("CPU offlining is not supported by hypervisor\n");
> + } else if (previous_cpu_disable)
> + smp_ops.cpu_disable = previous_cpu_disable;
> +}
> +#else
> +static void hv_cpu_hotplug_quirk(bool vmbus_loaded)
> +{
> +}
> +#endif
> +
> /*
> * vmbus_bus_init -Main vmbus driver initialization routine.
> *
> @@ -711,6 +745,7 @@ static int vmbus_bus_init(int irq)
> if (ret)
> goto err_alloc;
>
> + hv_cpu_hotplug_quirk(true);
> vmbus_request_offers();
>
> return 0;
> @@ -964,6 +999,7 @@ static void __exit vmbus_exit(void)
> bus_unregister(&hv_bus);
> hv_cleanup();
> acpi_bus_unregister_driver(&vmbus_acpi_driver);
> + hv_cpu_hotplug_quirk(false);
> }
>
>
> --
> 1.9.3
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: [PATCH v3] Drivers: hv: vmbus: prevent cpu offlining on newer hypervisors
2015-01-12 16:50 ` [PATCH v3] " Vitaly Kuznetsov
2015-01-12 18:54 ` KY Srinivasan
@ 2015-01-25 13:36 ` Greg Kroah-Hartman
2015-01-26 10:38 ` Vitaly Kuznetsov
1 sibling, 1 reply; 14+ messages in thread
From: Greg Kroah-Hartman @ 2015-01-25 13:36 UTC (permalink / raw)
To: Vitaly Kuznetsov
Cc: K. Y. Srinivasan, devel, Haiyang Zhang, linux-kernel, Dexuan Cui
On Mon, Jan 12, 2015 at 05:50:11PM +0100, Vitaly Kuznetsov wrote:
> When an SMP Hyper-V guest is running on top of 2012R2 Server and secondary
> cpus are sent offline (with echo 0 > /sys/devices/system/cpu/cpu$cpu/online)
> the system freeze is observed. This happens due to the fact that on newer
> hypervisors (Win8, WS2012R2, ...) vmbus channel handlers are distributed
> across all cpus (see init_vp_index() function in drivers/hv/channel_mgmt.c)
> and on cpu offlining nobody reassigns them to CPU0. Prevent cpu offlining
> when vmbus is loaded until the issue is fixed host-side.
>
> This patch also disables hibernation but it is OK as it is also broken (MCE
> error is hit on resume). Suspend still works.
>
> Tested with WS2008R2 and WS2012R2.
>
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> Signed-off-by: K. Y. Srinivasan <kys@microsoft.com>
> ---
> Changes since v2:
> - repair the build when vmbus is builded as a module [Greg KH] by saving
> current cpu_disable pointer to previous_cpu_disable and restoring it on
> unload;
> - return -ENOSYS (same as native_cpu_disable when !CONFIG_HOTPLUG_CPU) instead
> of -1 in hyperv_cpu_disable().
>
> Changes since v1:
> - introduce hv_cpu_hotplug_quirk() function to not spread #ifdefs [Greg KH];
> - add pr_notice() message "hv_vmbus: CPU offlining is not supported by
> hypervisor".
> ---
> drivers/hv/vmbus_drv.c | 36 ++++++++++++++++++++++++++++++++++++
> 1 file changed, 36 insertions(+)
Doesn't apply to my char-misc-test branch at all :(
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH v3] Drivers: hv: vmbus: prevent cpu offlining on newer hypervisors
2015-01-25 13:36 ` Greg Kroah-Hartman
@ 2015-01-26 10:38 ` Vitaly Kuznetsov
2015-01-26 16:11 ` KY Srinivasan
2015-01-26 22:54 ` Greg Kroah-Hartman
0 siblings, 2 replies; 14+ messages in thread
From: Vitaly Kuznetsov @ 2015-01-26 10:38 UTC (permalink / raw)
To: Greg Kroah-Hartman
Cc: K. Y. Srinivasan, devel, Haiyang Zhang, linux-kernel, Dexuan Cui
[-- Attachment #1: Type: text/plain, Size: 2056 bytes --]
Greg Kroah-Hartman <gregkh@linuxfoundation.org> writes:
> On Mon, Jan 12, 2015 at 05:50:11PM +0100, Vitaly Kuznetsov wrote:
>> When an SMP Hyper-V guest is running on top of 2012R2 Server and secondary
>> cpus are sent offline (with echo 0 > /sys/devices/system/cpu/cpu$cpu/online)
>> the system freeze is observed. This happens due to the fact that on newer
>> hypervisors (Win8, WS2012R2, ...) vmbus channel handlers are distributed
>> across all cpus (see init_vp_index() function in drivers/hv/channel_mgmt.c)
>> and on cpu offlining nobody reassigns them to CPU0. Prevent cpu offlining
>> when vmbus is loaded until the issue is fixed host-side.
>>
>> This patch also disables hibernation but it is OK as it is also broken (MCE
>> error is hit on resume). Suspend still works.
>>
>> Tested with WS2008R2 and WS2012R2.
>>
>> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
>> Signed-off-by: K. Y. Srinivasan <kys@microsoft.com>
>> ---
>> Changes since v2:
>> - repair the build when vmbus is builded as a module [Greg KH] by saving
>> current cpu_disable pointer to previous_cpu_disable and restoring it on
>> unload;
>> - return -ENOSYS (same as native_cpu_disable when !CONFIG_HOTPLUG_CPU) instead
>> of -1 in hyperv_cpu_disable().
>>
>> Changes since v1:
>> - introduce hv_cpu_hotplug_quirk() function to not spread #ifdefs [Greg KH];
>> - add pr_notice() message "hv_vmbus: CPU offlining is not supported by
>> hypervisor".
>> ---
>> drivers/hv/vmbus_drv.c | 36 ++++++++++++++++++++++++++++++++++++
>> 1 file changed, 36 insertions(+)
>
> Doesn't apply to my char-misc-test branch at all :(
Another mid-air collision with K.Y's "Drivers: hv: vmbus: Implement a
clockevent device", please use the attached version. No functional
changes are required, I just fixed the merge conflict (includes).
Othere than that (and sorry for meddling), would it it be better if
you switch to 'pull requests' workflow with K.Y? There is a lot of
ongoing work in hyperv nowdays and such collisions seem otherwise
inevitable ...
--
Vitaly
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Drivers-hv-vmbus-prevent-cpu-offlining-on-newer-hype.patch --]
[-- Type: text/x-patch, Size: 2781 bytes --]
>From 95f9b0ff3f73c3bbe8aa9c525414097c2c26a3ef Mon Sep 17 00:00:00 2001
From: Vitaly Kuznetsov <vkuznets@redhat.com>
Date: Mon, 12 Jan 2015 17:50:11 +0100
Subject: [PATCH] Drivers: hv: vmbus: prevent cpu offlining on newer
hypervisors
When an SMP Hyper-V guest is running on top of 2012R2 Server and secondary
cpus are sent offline (with echo 0 > /sys/devices/system/cpu/cpu$cpu/online)
the system freeze is observed. This happens due to the fact that on newer
hypervisors (Win8, WS2012R2, ...) vmbus channel handlers are distributed
across all cpus (see init_vp_index() function in drivers/hv/channel_mgmt.c)
and on cpu offlining nobody reassigns them to CPU0. Prevent cpu offlining
when vmbus is loaded until the issue is fixed host-side.
This patch also disables hibernation but it is OK as it is also broken (MCE
error is hit on resume). Suspend still works.
Tested with WS2008R2 and WS2012R2.
Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Signed-off-by: K. Y. Srinivasan <kys@microsoft.com>
---
drivers/hv/vmbus_drv.c | 36 ++++++++++++++++++++++++++++++++++++
1 file changed, 36 insertions(+)
diff --git a/drivers/hv/vmbus_drv.c b/drivers/hv/vmbus_drv.c
index 35e3f42..90c3400 100644
--- a/drivers/hv/vmbus_drv.c
+++ b/drivers/hv/vmbus_drv.c
@@ -33,6 +33,7 @@
#include <linux/hyperv.h>
#include <linux/kernel_stat.h>
#include <linux/clockchips.h>
+#include <linux/cpu.h>
#include <asm/hyperv.h>
#include <asm/hypervisor.h>
#include <asm/mshyperv.h>
@@ -704,6 +705,39 @@ static void vmbus_isr(void)
}
}
+#ifdef CONFIG_HOTPLUG_CPU
+static int hyperv_cpu_disable(void)
+{
+ return -ENOSYS;
+}
+
+static void hv_cpu_hotplug_quirk(bool vmbus_loaded)
+{
+ static void *previous_cpu_disable;
+
+ /*
+ * Offlining a CPU when running on newer hypervisors (WS2012R2, Win8,
+ * ...) is not supported at this moment as channel interrupts are
+ * distributed across all of them.
+ */
+
+ if ((vmbus_proto_version == VERSION_WS2008) ||
+ (vmbus_proto_version == VERSION_WIN7))
+ return;
+
+ if (vmbus_loaded) {
+ previous_cpu_disable = smp_ops.cpu_disable;
+ smp_ops.cpu_disable = hyperv_cpu_disable;
+ pr_notice("CPU offlining is not supported by hypervisor\n");
+ } else if (previous_cpu_disable)
+ smp_ops.cpu_disable = previous_cpu_disable;
+}
+#else
+static void hv_cpu_hotplug_quirk(bool vmbus_loaded)
+{
+}
+#endif
+
/*
* vmbus_bus_init -Main vmbus driver initialization routine.
*
@@ -744,6 +778,7 @@ static int vmbus_bus_init(int irq)
if (ret)
goto err_alloc;
+ hv_cpu_hotplug_quirk(true);
vmbus_request_offers();
return 0;
@@ -997,6 +1032,7 @@ static void __exit vmbus_exit(void)
bus_unregister(&hv_bus);
hv_cleanup();
acpi_bus_unregister_driver(&vmbus_acpi_driver);
+ hv_cpu_hotplug_quirk(false);
}
--
1.9.3
^ permalink raw reply related [flat|nested] 14+ messages in thread* RE: [PATCH v3] Drivers: hv: vmbus: prevent cpu offlining on newer hypervisors
2015-01-26 10:38 ` Vitaly Kuznetsov
@ 2015-01-26 16:11 ` KY Srinivasan
2015-01-26 22:54 ` Greg Kroah-Hartman
1 sibling, 0 replies; 14+ messages in thread
From: KY Srinivasan @ 2015-01-26 16:11 UTC (permalink / raw)
To: Vitaly Kuznetsov, Greg Kroah-Hartman
Cc: devel, Haiyang Zhang, linux-kernel, Dexuan Cui
> -----Original Message-----
> From: Vitaly Kuznetsov [mailto:vkuznets@redhat.com]
> Sent: Monday, January 26, 2015 2:39 AM
> To: Greg Kroah-Hartman
> Cc: KY Srinivasan; devel@linuxdriverproject.org; Haiyang Zhang; linux-
> kernel@vger.kernel.org; Dexuan Cui
> Subject: Re: [PATCH v3] Drivers: hv: vmbus: prevent cpu offlining on newer
> hypervisors
>
> Greg Kroah-Hartman <gregkh@linuxfoundation.org> writes:
>
> > On Mon, Jan 12, 2015 at 05:50:11PM +0100, Vitaly Kuznetsov wrote:
> >> When an SMP Hyper-V guest is running on top of 2012R2 Server and
> >> secondary cpus are sent offline (with echo 0 >
> >> /sys/devices/system/cpu/cpu$cpu/online)
> >> the system freeze is observed. This happens due to the fact that on
> >> newer hypervisors (Win8, WS2012R2, ...) vmbus channel handlers are
> >> distributed across all cpus (see init_vp_index() function in
> >> drivers/hv/channel_mgmt.c) and on cpu offlining nobody reassigns them
> >> to CPU0. Prevent cpu offlining when vmbus is loaded until the issue is
> fixed host-side.
> >>
> >> This patch also disables hibernation but it is OK as it is also
> >> broken (MCE error is hit on resume). Suspend still works.
> >>
> >> Tested with WS2008R2 and WS2012R2.
> >>
> >> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> >> Signed-off-by: K. Y. Srinivasan <kys@microsoft.com>
> >> ---
> >> Changes since v2:
> >> - repair the build when vmbus is builded as a module [Greg KH] by saving
> >> current cpu_disable pointer to previous_cpu_disable and restoring it on
> >> unload;
> >> - return -ENOSYS (same as native_cpu_disable when
> !CONFIG_HOTPLUG_CPU) instead
> >> of -1 in hyperv_cpu_disable().
> >>
> >> Changes since v1:
> >> - introduce hv_cpu_hotplug_quirk() function to not spread #ifdefs
> >> [Greg KH];
> >> - add pr_notice() message "hv_vmbus: CPU offlining is not supported by
> >> hypervisor".
> >> ---
> >> drivers/hv/vmbus_drv.c | 36
> ++++++++++++++++++++++++++++++++++++
> >> 1 file changed, 36 insertions(+)
> >
> > Doesn't apply to my char-misc-test branch at all :(
>
> Another mid-air collision with K.Y's "Drivers: hv: vmbus: Implement a
> clockevent device", please use the attached version. No functional changes
> are required, I just fixed the merge conflict (includes).
>
> Othere than that (and sorry for meddling), would it it be better if you switch
> to 'pull requests' workflow with K.Y? There is a lot of ongoing work in hyperv
> nowdays and such collisions seem otherwise inevitable ...
I will co-ordinate and forward Hyper-V patches to Greg.
K. Y
>
> --
> Vitaly
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH v3] Drivers: hv: vmbus: prevent cpu offlining on newer hypervisors
2015-01-26 10:38 ` Vitaly Kuznetsov
2015-01-26 16:11 ` KY Srinivasan
@ 2015-01-26 22:54 ` Greg Kroah-Hartman
2015-01-26 23:53 ` KY Srinivasan
1 sibling, 1 reply; 14+ messages in thread
From: Greg Kroah-Hartman @ 2015-01-26 22:54 UTC (permalink / raw)
To: Vitaly Kuznetsov
Cc: K. Y. Srinivasan, devel, Haiyang Zhang, linux-kernel, Dexuan Cui
On Mon, Jan 26, 2015 at 11:38:54AM +0100, Vitaly Kuznetsov wrote:
> Greg Kroah-Hartman <gregkh@linuxfoundation.org> writes:
>
> > On Mon, Jan 12, 2015 at 05:50:11PM +0100, Vitaly Kuznetsov wrote:
> >> When an SMP Hyper-V guest is running on top of 2012R2 Server and secondary
> >> cpus are sent offline (with echo 0 > /sys/devices/system/cpu/cpu$cpu/online)
> >> the system freeze is observed. This happens due to the fact that on newer
> >> hypervisors (Win8, WS2012R2, ...) vmbus channel handlers are distributed
> >> across all cpus (see init_vp_index() function in drivers/hv/channel_mgmt.c)
> >> and on cpu offlining nobody reassigns them to CPU0. Prevent cpu offlining
> >> when vmbus is loaded until the issue is fixed host-side.
> >>
> >> This patch also disables hibernation but it is OK as it is also broken (MCE
> >> error is hit on resume). Suspend still works.
> >>
> >> Tested with WS2008R2 and WS2012R2.
> >>
> >> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> >> Signed-off-by: K. Y. Srinivasan <kys@microsoft.com>
> >> ---
> >> Changes since v2:
> >> - repair the build when vmbus is builded as a module [Greg KH] by saving
> >> current cpu_disable pointer to previous_cpu_disable and restoring it on
> >> unload;
> >> - return -ENOSYS (same as native_cpu_disable when !CONFIG_HOTPLUG_CPU) instead
> >> of -1 in hyperv_cpu_disable().
> >>
> >> Changes since v1:
> >> - introduce hv_cpu_hotplug_quirk() function to not spread #ifdefs [Greg KH];
> >> - add pr_notice() message "hv_vmbus: CPU offlining is not supported by
> >> hypervisor".
> >> ---
> >> drivers/hv/vmbus_drv.c | 36 ++++++++++++++++++++++++++++++++++++
> >> 1 file changed, 36 insertions(+)
> >
> > Doesn't apply to my char-misc-test branch at all :(
>
> Another mid-air collision with K.Y's "Drivers: hv: vmbus: Implement a
> clockevent device", please use the attached version. No functional
> changes are required, I just fixed the merge conflict (includes).
>
> Othere than that (and sorry for meddling), would it it be better if
> you switch to 'pull requests' workflow with K.Y? There is a lot of
> ongoing work in hyperv nowdays and such collisions seem otherwise
> inevitable ...
This is a trivial number of patches for this subsystem, so no, a git
workflow isn't needed. But if KY would collect the patches up and send
them to me for all hyperv patches from now on, that would make these
types of issues go away, KY, can you do that from now on?
thanks,
greg k-h
^ permalink raw reply [flat|nested] 14+ messages in thread
* RE: [PATCH v3] Drivers: hv: vmbus: prevent cpu offlining on newer hypervisors
2015-01-26 22:54 ` Greg Kroah-Hartman
@ 2015-01-26 23:53 ` KY Srinivasan
0 siblings, 0 replies; 14+ messages in thread
From: KY Srinivasan @ 2015-01-26 23:53 UTC (permalink / raw)
To: Greg Kroah-Hartman, Vitaly Kuznetsov
Cc: devel, Haiyang Zhang, linux-kernel, Dexuan Cui
> -----Original Message-----
> From: Greg Kroah-Hartman [mailto:gregkh@linuxfoundation.org]
> Sent: Monday, January 26, 2015 2:54 PM
> To: Vitaly Kuznetsov
> Cc: KY Srinivasan; devel@linuxdriverproject.org; Haiyang Zhang; linux-
> kernel@vger.kernel.org; Dexuan Cui
> Subject: Re: [PATCH v3] Drivers: hv: vmbus: prevent cpu offlining on newer
> hypervisors
>
> On Mon, Jan 26, 2015 at 11:38:54AM +0100, Vitaly Kuznetsov wrote:
> > Greg Kroah-Hartman <gregkh@linuxfoundation.org> writes:
> >
> > > On Mon, Jan 12, 2015 at 05:50:11PM +0100, Vitaly Kuznetsov wrote:
> > >> When an SMP Hyper-V guest is running on top of 2012R2 Server and
> > >> secondary cpus are sent offline (with echo 0 >
> > >> /sys/devices/system/cpu/cpu$cpu/online)
> > >> the system freeze is observed. This happens due to the fact that on
> > >> newer hypervisors (Win8, WS2012R2, ...) vmbus channel handlers are
> > >> distributed across all cpus (see init_vp_index() function in
> > >> drivers/hv/channel_mgmt.c) and on cpu offlining nobody reassigns
> > >> them to CPU0. Prevent cpu offlining when vmbus is loaded until the
> issue is fixed host-side.
> > >>
> > >> This patch also disables hibernation but it is OK as it is also
> > >> broken (MCE error is hit on resume). Suspend still works.
> > >>
> > >> Tested with WS2008R2 and WS2012R2.
> > >>
> > >> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> > >> Signed-off-by: K. Y. Srinivasan <kys@microsoft.com>
> > >> ---
> > >> Changes since v2:
> > >> - repair the build when vmbus is builded as a module [Greg KH] by
> saving
> > >> current cpu_disable pointer to previous_cpu_disable and restoring it
> on
> > >> unload;
> > >> - return -ENOSYS (same as native_cpu_disable when
> !CONFIG_HOTPLUG_CPU) instead
> > >> of -1 in hyperv_cpu_disable().
> > >>
> > >> Changes since v1:
> > >> - introduce hv_cpu_hotplug_quirk() function to not spread #ifdefs
> > >> [Greg KH];
> > >> - add pr_notice() message "hv_vmbus: CPU offlining is not supported by
> > >> hypervisor".
> > >> ---
> > >> drivers/hv/vmbus_drv.c | 36
> ++++++++++++++++++++++++++++++++++++
> > >> 1 file changed, 36 insertions(+)
> > >
> > > Doesn't apply to my char-misc-test branch at all :(
> >
> > Another mid-air collision with K.Y's "Drivers: hv: vmbus: Implement a
> > clockevent device", please use the attached version. No functional
> > changes are required, I just fixed the merge conflict (includes).
> >
> > Othere than that (and sorry for meddling), would it it be better if
> > you switch to 'pull requests' workflow with K.Y? There is a lot of
> > ongoing work in hyperv nowdays and such collisions seem otherwise
> > inevitable ...
>
> This is a trivial number of patches for this subsystem, so no, a git workflow
> isn't needed. But if KY would collect the patches up and send them to me for
> all hyperv patches from now on, that would make these types of issues go
> away, KY, can you do that from now on?
Will do.
Thanks,
K. Y
>
> thanks,
>
> greg k-h
^ permalink raw reply [flat|nested] 14+ messages in thread