* [RFC 0/2] Drivers: hv: balloon: Temporary fixes for ARM64
@ 2022-02-23 13:15 Boqun Feng
2022-02-23 13:15 ` [RFC 1/2] Drivers: hv: balloon: Support status report for larger page sizes Boqun Feng
2022-02-23 13:15 ` [RFC 2/2] Drivers: hv: balloon: Disable balloon and hot-add accordingly Boqun Feng
0 siblings, 2 replies; 10+ messages in thread
From: Boqun Feng @ 2022-02-23 13:15 UTC (permalink / raw)
To: Wei Liu
Cc: Vitaly Kuznetsov, linux-hyperv, K. Y. Srinivasan, Haiyang Zhang,
Stephen Hemminger, Dexuan Cui, Michael Kelley, David Hildenbrand,
linux-kernel, Boqun Feng
Since Hyper-V always uses 4k pages, hv_balloon has some difficulties
working on ARM64 with larger pages[1]. Besides the memory hot add
messages of Hyper-V doesn't have the information of NUMA node id of the
added memory range, and ARM64 currently doesn't provide the conversion
from a physical address to a node id, as a result the hv_balloon driver
couldn't handle hot add properly when there are more than one NUMA node.
Among these issues, post_status() is easy to fix, while the unballoon
issue and the hot-add issue requires more discussion. To make the
hv_balloon driver work at the best effort, this patchset fixes the
post_status() and temporarily disable the balloon and hot-add
accordingly.
Looking forwards to comments and suggestions.
Regards,
Boqun
[1]: https://lore.kernel.org/lkml/20220105165028.1343706-1-vkuznets@redhat.com/
Boqun Feng (2):
Drivers: hv: balloon: Support status report for larger page sizes
Drivers: hv: balloon: Disable balloon and hot-add accordingly
drivers/hv/hv_balloon.c | 27 ++++++++++++++++++++++-----
1 file changed, 22 insertions(+), 5 deletions(-)
--
2.35.1
^ permalink raw reply [flat|nested] 10+ messages in thread
* [RFC 1/2] Drivers: hv: balloon: Support status report for larger page sizes
2022-02-23 13:15 [RFC 0/2] Drivers: hv: balloon: Temporary fixes for ARM64 Boqun Feng
@ 2022-02-23 13:15 ` Boqun Feng
2022-02-23 16:45 ` Michael Kelley (LINUX)
2022-02-23 13:15 ` [RFC 2/2] Drivers: hv: balloon: Disable balloon and hot-add accordingly Boqun Feng
1 sibling, 1 reply; 10+ messages in thread
From: Boqun Feng @ 2022-02-23 13:15 UTC (permalink / raw)
To: Wei Liu
Cc: Vitaly Kuznetsov, linux-hyperv, K. Y. Srinivasan, Haiyang Zhang,
Stephen Hemminger, Dexuan Cui, Michael Kelley, David Hildenbrand,
linux-kernel, Boqun Feng
DM_STATUS_REPORT expects the numbers of pages in the unit of 4k pages
(HV_HYP_PAGE) instead of guest pages, so to make it work when guest page
sizes are larger than 4k, convert the numbers of guest pages into the
numbers of HV_HYP_PAGEs.
Note that the numbers of guest pages are still used for tracing because
tracing is internal to the guest kernel.
Reported-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
---
drivers/hv/hv_balloon.c | 13 ++++++++++---
1 file changed, 10 insertions(+), 3 deletions(-)
diff --git a/drivers/hv/hv_balloon.c b/drivers/hv/hv_balloon.c
index f2d05bff4245..062156b88a87 100644
--- a/drivers/hv/hv_balloon.c
+++ b/drivers/hv/hv_balloon.c
@@ -17,6 +17,7 @@
#include <linux/slab.h>
#include <linux/kthread.h>
#include <linux/completion.h>
+#include <linux/count_zeros.h>
#include <linux/memory_hotplug.h>
#include <linux/memory.h>
#include <linux/notifier.h>
@@ -1130,6 +1131,7 @@ static void post_status(struct hv_dynmem_device *dm)
struct dm_status status;
unsigned long now = jiffies;
unsigned long last_post = last_post_time;
+ unsigned long num_pages_avail, num_pages_committed;
if (pressure_report_delay > 0) {
--pressure_report_delay;
@@ -1154,16 +1156,21 @@ static void post_status(struct hv_dynmem_device *dm)
* num_pages_onlined) as committed to the host, otherwise it can try
* asking us to balloon them out.
*/
- status.num_avail = si_mem_available();
- status.num_committed = vm_memory_committed() +
+ num_pages_avail = si_mem_available();
+ num_pages_committed = vm_memory_committed() +
dm->num_pages_ballooned +
(dm->num_pages_added > dm->num_pages_onlined ?
dm->num_pages_added - dm->num_pages_onlined : 0) +
compute_balloon_floor();
- trace_balloon_status(status.num_avail, status.num_committed,
+ trace_balloon_status(num_pages_avail, num_pages_committed,
vm_memory_committed(), dm->num_pages_ballooned,
dm->num_pages_added, dm->num_pages_onlined);
+
+ /* Convert numbers of pages into numbers of HV_HYP_PAGEs. */
+ status.num_avail = num_pages_avail * NR_HV_HYP_PAGES_IN_PAGE;
+ status.num_committed = num_pages_committed * NR_HV_HYP_PAGES_IN_PAGE;
+
/*
* If our transaction ID is no longer current, just don't
* send the status. This can happen if we were interrupted
--
2.35.1
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [RFC 2/2] Drivers: hv: balloon: Disable balloon and hot-add accordingly
2022-02-23 13:15 [RFC 0/2] Drivers: hv: balloon: Temporary fixes for ARM64 Boqun Feng
2022-02-23 13:15 ` [RFC 1/2] Drivers: hv: balloon: Support status report for larger page sizes Boqun Feng
@ 2022-02-23 13:15 ` Boqun Feng
2022-02-23 16:55 ` Michael Kelley (LINUX)
2022-02-25 2:17 ` [RFC v1.1] " Boqun Feng
1 sibling, 2 replies; 10+ messages in thread
From: Boqun Feng @ 2022-02-23 13:15 UTC (permalink / raw)
To: Wei Liu
Cc: Vitaly Kuznetsov, linux-hyperv, K. Y. Srinivasan, Haiyang Zhang,
Stephen Hemminger, Dexuan Cui, Michael Kelley, David Hildenbrand,
linux-kernel, Boqun Feng
Currently there are known potential issues for balloon and hot-add on
ARM64:
* Unballoon requests from Hyper-V should only unballoon ranges
that are guest page size aligned, otherwise guests cannot handle
because it's impossible to partially free a page.
* Memory hot-add requests from Hyper-V should provide the NUMA
node id of the added ranges or ARM64 should have a functional
memory_add_physaddr_to_nid(), otherwise the node id is missing
for add_memory().
These issues require discussions on design and implementation. In the
meanwhile, post_status() is working and essiential to guest monitoring.
Therefore instead of the entire hv_balloon driver, the balloon and
hot-add are disabled accordingly for now. Once the issues are fixed,
they can be re-enable in these cases.
Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
---
drivers/hv/hv_balloon.c | 14 ++++++++++++--
1 file changed, 12 insertions(+), 2 deletions(-)
diff --git a/drivers/hv/hv_balloon.c b/drivers/hv/hv_balloon.c
index 062156b88a87..35dcda20be85 100644
--- a/drivers/hv/hv_balloon.c
+++ b/drivers/hv/hv_balloon.c
@@ -1730,9 +1730,19 @@ static int balloon_connect_vsp(struct hv_device *dev)
* When hibernation (i.e. virtual ACPI S4 state) is enabled, the host
* currently still requires the bits to be set, so we have to add code
* to fail the host's hot-add and balloon up/down requests, if any.
+ *
+ * We disable balloon if the page size is larger than 4k, since
+ * currently it's unclear to us whether an unballoon request can make
+ * sure all page ranges are guest page size aligned.
+ *
+ * We also disable hot add on ARM64, because we currently rely on
+ * memory_add_physaddr_to_nid() to get a node id of a hot add range,
+ * however ARM64's memory_add_physaddr_to_nid() always return 0 and
+ * DM_MEM_HOT_ADD_REQUEST doesn't have the NUMA node information for
+ * add_memory().
*/
- cap_msg.caps.cap_bits.balloon = 1;
- cap_msg.caps.cap_bits.hot_add = 1;
+ cap_msg.caps.cap_bits.balloon = !(PAGE_SIZE > 4096UL);
+ cap_msg.caps.cap_bits.hot_add = !IS_ENABLED(CONFIG_ARM64);
/*
* Specify our alignment requirements as it relates
--
2.35.1
^ permalink raw reply related [flat|nested] 10+ messages in thread
* RE: [RFC 1/2] Drivers: hv: balloon: Support status report for larger page sizes
2022-02-23 13:15 ` [RFC 1/2] Drivers: hv: balloon: Support status report for larger page sizes Boqun Feng
@ 2022-02-23 16:45 ` Michael Kelley (LINUX)
0 siblings, 0 replies; 10+ messages in thread
From: Michael Kelley (LINUX) @ 2022-02-23 16:45 UTC (permalink / raw)
To: Boqun Feng, Wei Liu
Cc: vkuznets, linux-hyperv, KY Srinivasan, Haiyang Zhang,
Stephen Hemminger, Dexuan Cui, David Hildenbrand, linux-kernel
From: Boqun Feng <boqun.feng@gmail.com> Sent: Wednesday, February 23, 2022 5:16 AM
>
> DM_STATUS_REPORT expects the numbers of pages in the unit of 4k pages
> (HV_HYP_PAGE) instead of guest pages, so to make it work when guest page
> sizes are larger than 4k, convert the numbers of guest pages into the
> numbers of HV_HYP_PAGEs.
>
> Note that the numbers of guest pages are still used for tracing because
> tracing is internal to the guest kernel.
>
> Reported-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
> ---
> drivers/hv/hv_balloon.c | 13 ++++++++++---
> 1 file changed, 10 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/hv/hv_balloon.c b/drivers/hv/hv_balloon.c
> index f2d05bff4245..062156b88a87 100644
> --- a/drivers/hv/hv_balloon.c
> +++ b/drivers/hv/hv_balloon.c
> @@ -17,6 +17,7 @@
> #include <linux/slab.h>
> #include <linux/kthread.h>
> #include <linux/completion.h>
> +#include <linux/count_zeros.h>
> #include <linux/memory_hotplug.h>
> #include <linux/memory.h>
> #include <linux/notifier.h>
> @@ -1130,6 +1131,7 @@ static void post_status(struct hv_dynmem_device *dm)
> struct dm_status status;
> unsigned long now = jiffies;
> unsigned long last_post = last_post_time;
> + unsigned long num_pages_avail, num_pages_committed;
>
> if (pressure_report_delay > 0) {
> --pressure_report_delay;
> @@ -1154,16 +1156,21 @@ static void post_status(struct hv_dynmem_device *dm)
> * num_pages_onlined) as committed to the host, otherwise it can try
> * asking us to balloon them out.
> */
> - status.num_avail = si_mem_available();
> - status.num_committed = vm_memory_committed() +
> + num_pages_avail = si_mem_available();
> + num_pages_committed = vm_memory_committed() +
> dm->num_pages_ballooned +
> (dm->num_pages_added > dm->num_pages_onlined ?
> dm->num_pages_added - dm->num_pages_onlined : 0) +
> compute_balloon_floor();
>
> - trace_balloon_status(status.num_avail, status.num_committed,
> + trace_balloon_status(num_pages_avail, num_pages_committed,
> vm_memory_committed(), dm->num_pages_ballooned,
> dm->num_pages_added, dm->num_pages_onlined);
> +
> + /* Convert numbers of pages into numbers of HV_HYP_PAGEs. */
> + status.num_avail = num_pages_avail * NR_HV_HYP_PAGES_IN_PAGE;
> + status.num_committed = num_pages_committed * NR_HV_HYP_PAGES_IN_PAGE;
> +
> /*
> * If our transaction ID is no longer current, just don't
> * send the status. This can happen if we were interrupted
> --
> 2.35.1
Reviewed-by: Michael Kelley <mikelley@microsoft.com>
^ permalink raw reply [flat|nested] 10+ messages in thread
* RE: [RFC 2/2] Drivers: hv: balloon: Disable balloon and hot-add accordingly
2022-02-23 13:15 ` [RFC 2/2] Drivers: hv: balloon: Disable balloon and hot-add accordingly Boqun Feng
@ 2022-02-23 16:55 ` Michael Kelley (LINUX)
2022-02-24 2:44 ` Boqun Feng
2022-02-25 2:17 ` [RFC v1.1] " Boqun Feng
1 sibling, 1 reply; 10+ messages in thread
From: Michael Kelley (LINUX) @ 2022-02-23 16:55 UTC (permalink / raw)
To: Boqun Feng, Wei Liu
Cc: vkuznets, linux-hyperv, KY Srinivasan, Haiyang Zhang,
Stephen Hemminger, Dexuan Cui, David Hildenbrand, linux-kernel
From: Boqun Feng <boqun.feng@gmail.com> Sent: Wednesday, February 23, 2022 5:16 AM
>
> Currently there are known potential issues for balloon and hot-add on
> ARM64:
>
> * Unballoon requests from Hyper-V should only unballoon ranges
> that are guest page size aligned, otherwise guests cannot handle
> because it's impossible to partially free a page.
>
> * Memory hot-add requests from Hyper-V should provide the NUMA
> node id of the added ranges or ARM64 should have a functional
> memory_add_physaddr_to_nid(), otherwise the node id is missing
> for add_memory().
>
> These issues require discussions on design and implementation. In the
> meanwhile, post_status() is working and essiential to guest monitoring.
> Therefore instead of the entire hv_balloon driver, the balloon and
> hot-add are disabled accordingly for now. Once the issues are fixed,
> they can be re-enable in these cases.
>
> Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
> ---
> drivers/hv/hv_balloon.c | 14 ++++++++++++--
> 1 file changed, 12 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/hv/hv_balloon.c b/drivers/hv/hv_balloon.c
> index 062156b88a87..35dcda20be85 100644
> --- a/drivers/hv/hv_balloon.c
> +++ b/drivers/hv/hv_balloon.c
> @@ -1730,9 +1730,19 @@ static int balloon_connect_vsp(struct hv_device *dev)
> * When hibernation (i.e. virtual ACPI S4 state) is enabled, the host
> * currently still requires the bits to be set, so we have to add code
> * to fail the host's hot-add and balloon up/down requests, if any.
> + *
> + * We disable balloon if the page size is larger than 4k, since
> + * currently it's unclear to us whether an unballoon request can make
> + * sure all page ranges are guest page size aligned.
> + *
> + * We also disable hot add on ARM64, because we currently rely on
> + * memory_add_physaddr_to_nid() to get a node id of a hot add range,
> + * however ARM64's memory_add_physaddr_to_nid() always return 0 and
> + * DM_MEM_HOT_ADD_REQUEST doesn't have the NUMA node information for
> + * add_memory().
> */
> - cap_msg.caps.cap_bits.balloon = 1;
> - cap_msg.caps.cap_bits.hot_add = 1;
> + cap_msg.caps.cap_bits.balloon = !(PAGE_SIZE > 4096UL);
Any reasons not to use HV_HYP_PAGE_SIZE vs. open coding "4096"? So
cap_msg.caps.cap_bits.balloon = (PAGE_SIZE == HV_HYP_PAGE_SIZE);
> + cap_msg.caps.cap_bits.hot_add = !IS_ENABLED(CONFIG_ARM64);
I think we should output a message so that there's no mystery as to
whether ballooning and/or hot_add are disabled, and why. Each setting
should have its own message. Maybe something like:
if (!cap_msg.caps.cap_bits.balloon)
pr_info("Ballooning disabled because page size is not 4096 bytes\n");
if (!cap_msg.cap_bits.hot_add)
pr_info("Memory hot add disabled on ARM64\n");
>
> /*
> * Specify our alignment requirements as it relates
> --
> 2.35.1
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC 2/2] Drivers: hv: balloon: Disable balloon and hot-add accordingly
2022-02-23 16:55 ` Michael Kelley (LINUX)
@ 2022-02-24 2:44 ` Boqun Feng
2022-02-24 4:44 ` Michael Kelley (LINUX)
0 siblings, 1 reply; 10+ messages in thread
From: Boqun Feng @ 2022-02-24 2:44 UTC (permalink / raw)
To: Michael Kelley (LINUX)
Cc: Wei Liu, vkuznets, linux-hyperv, KY Srinivasan, Haiyang Zhang,
Stephen Hemminger, Dexuan Cui, David Hildenbrand, linux-kernel
On Wed, Feb 23, 2022 at 04:55:25PM +0000, Michael Kelley (LINUX) wrote:
> From: Boqun Feng <boqun.feng@gmail.com> Sent: Wednesday, February 23, 2022 5:16 AM
> >
> > Currently there are known potential issues for balloon and hot-add on
> > ARM64:
> >
> > * Unballoon requests from Hyper-V should only unballoon ranges
> > that are guest page size aligned, otherwise guests cannot handle
> > because it's impossible to partially free a page.
> >
> > * Memory hot-add requests from Hyper-V should provide the NUMA
> > node id of the added ranges or ARM64 should have a functional
> > memory_add_physaddr_to_nid(), otherwise the node id is missing
> > for add_memory().
> >
> > These issues require discussions on design and implementation. In the
> > meanwhile, post_status() is working and essiential to guest monitoring.
> > Therefore instead of the entire hv_balloon driver, the balloon and
> > hot-add are disabled accordingly for now. Once the issues are fixed,
> > they can be re-enable in these cases.
> >
> > Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
> > ---
> > drivers/hv/hv_balloon.c | 14 ++++++++++++--
> > 1 file changed, 12 insertions(+), 2 deletions(-)
> >
> > diff --git a/drivers/hv/hv_balloon.c b/drivers/hv/hv_balloon.c
> > index 062156b88a87..35dcda20be85 100644
> > --- a/drivers/hv/hv_balloon.c
> > +++ b/drivers/hv/hv_balloon.c
> > @@ -1730,9 +1730,19 @@ static int balloon_connect_vsp(struct hv_device *dev)
> > * When hibernation (i.e. virtual ACPI S4 state) is enabled, the host
> > * currently still requires the bits to be set, so we have to add code
> > * to fail the host's hot-add and balloon up/down requests, if any.
> > + *
> > + * We disable balloon if the page size is larger than 4k, since
> > + * currently it's unclear to us whether an unballoon request can make
> > + * sure all page ranges are guest page size aligned.
> > + *
> > + * We also disable hot add on ARM64, because we currently rely on
> > + * memory_add_physaddr_to_nid() to get a node id of a hot add range,
> > + * however ARM64's memory_add_physaddr_to_nid() always return 0 and
> > + * DM_MEM_HOT_ADD_REQUEST doesn't have the NUMA node information for
> > + * add_memory().
> > */
> > - cap_msg.caps.cap_bits.balloon = 1;
> > - cap_msg.caps.cap_bits.hot_add = 1;
> > + cap_msg.caps.cap_bits.balloon = !(PAGE_SIZE > 4096UL);
>
> Any reasons not to use HV_HYP_PAGE_SIZE vs. open coding "4096"? So
>
> cap_msg.caps.cap_bits.balloon = (PAGE_SIZE == HV_HYP_PAGE_SIZE);
>
You're right. I will change that to it in the next version.
> > + cap_msg.caps.cap_bits.hot_add = !IS_ENABLED(CONFIG_ARM64);
>
> I think we should output a message so that there's no mystery as to
> whether ballooning and/or hot_add are disabled, and why. Each setting
> should have its own message. Maybe something like:
>
> if (!cap_msg.caps.cap_bits.balloon)
> pr_info("Ballooning disabled because page size is not 4096 bytes\n");
>
> if (!cap_msg.cap_bits.hot_add)
> pr_info("Memory hot add disabled on ARM64\n");
>
I agree with your suggestion, however, while I'm at it, I think it's
better that we have functions that check and print, and .balloon and
.hot_add can rely on the return value, for example:
static int balloon_enabled(void)
{
if (PAGE_SIZE != HV_HYP_PAGE_SIZE) {
pr_info("Ballooning disabled because page size is not 4096 bytes\n");
return 0;
}
return 1;
}
// in balloon_vsp_connect()
cap_msg.caps.cap_bits.balloon = balloon_enabled();
In this way, we keep the checking and reason printing in the same
function and it's easier to maintain the consistency.
Thoughts?
Regards,
Boqun
> >
> > /*
> > * Specify our alignment requirements as it relates
> > --
> > 2.35.1
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* RE: [RFC 2/2] Drivers: hv: balloon: Disable balloon and hot-add accordingly
2022-02-24 2:44 ` Boqun Feng
@ 2022-02-24 4:44 ` Michael Kelley (LINUX)
0 siblings, 0 replies; 10+ messages in thread
From: Michael Kelley (LINUX) @ 2022-02-24 4:44 UTC (permalink / raw)
To: Boqun Feng
Cc: Wei Liu, vkuznets, linux-hyperv, KY Srinivasan, Haiyang Zhang,
Stephen Hemminger, Dexuan Cui, David Hildenbrand, linux-kernel
From: Boqun Feng <boqun.feng@gmail.com> Sent: Wednesday, February 23, 2022 6:44 PM
>
> On Wed, Feb 23, 2022 at 04:55:25PM +0000, Michael Kelley (LINUX) wrote:
> > From: Boqun Feng <boqun.feng@gmail.com> Sent: Wednesday, February 23, 2022
> 5:16 AM
> > >
> > > Currently there are known potential issues for balloon and hot-add on
> > > ARM64:
> > >
> > > * Unballoon requests from Hyper-V should only unballoon ranges
> > > that are guest page size aligned, otherwise guests cannot handle
> > > because it's impossible to partially free a page.
> > >
> > > * Memory hot-add requests from Hyper-V should provide the NUMA
> > > node id of the added ranges or ARM64 should have a functional
> > > memory_add_physaddr_to_nid(), otherwise the node id is missing
> > > for add_memory().
> > >
> > > These issues require discussions on design and implementation. In the
> > > meanwhile, post_status() is working and essiential to guest monitoring.
> > > Therefore instead of the entire hv_balloon driver, the balloon and
> > > hot-add are disabled accordingly for now. Once the issues are fixed,
> > > they can be re-enable in these cases.
> > >
> > > Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
> > > ---
> > > drivers/hv/hv_balloon.c | 14 ++++++++++++--
> > > 1 file changed, 12 insertions(+), 2 deletions(-)
> > >
> > > diff --git a/drivers/hv/hv_balloon.c b/drivers/hv/hv_balloon.c
> > > index 062156b88a87..35dcda20be85 100644
> > > --- a/drivers/hv/hv_balloon.c
> > > +++ b/drivers/hv/hv_balloon.c
> > > @@ -1730,9 +1730,19 @@ static int balloon_connect_vsp(struct hv_device *dev)
> > > * When hibernation (i.e. virtual ACPI S4 state) is enabled, the host
> > > * currently still requires the bits to be set, so we have to add code
> > > * to fail the host's hot-add and balloon up/down requests, if any.
> > > + *
> > > + * We disable balloon if the page size is larger than 4k, since
> > > + * currently it's unclear to us whether an unballoon request can make
> > > + * sure all page ranges are guest page size aligned.
> > > + *
> > > + * We also disable hot add on ARM64, because we currently rely on
> > > + * memory_add_physaddr_to_nid() to get a node id of a hot add range,
> > > + * however ARM64's memory_add_physaddr_to_nid() always return 0 and
> > > + * DM_MEM_HOT_ADD_REQUEST doesn't have the NUMA node information
> for
> > > + * add_memory().
> > > */
> > > - cap_msg.caps.cap_bits.balloon = 1;
> > > - cap_msg.caps.cap_bits.hot_add = 1;
> > > + cap_msg.caps.cap_bits.balloon = !(PAGE_SIZE > 4096UL);
> >
> > Any reasons not to use HV_HYP_PAGE_SIZE vs. open coding "4096"? So
> >
> > cap_msg.caps.cap_bits.balloon = (PAGE_SIZE == HV_HYP_PAGE_SIZE);
> >
>
> You're right. I will change that to it in the next version.
>
> > > + cap_msg.caps.cap_bits.hot_add = !IS_ENABLED(CONFIG_ARM64);
> >
> > I think we should output a message so that there's no mystery as to
> > whether ballooning and/or hot_add are disabled, and why. Each setting
> > should have its own message. Maybe something like:
> >
> > if (!cap_msg.caps.cap_bits.balloon)
> > pr_info("Ballooning disabled because page size is not 4096 bytes\n");
> >
> > if (!cap_msg.cap_bits.hot_add)
> > pr_info("Memory hot add disabled on ARM64\n");
> >
>
> I agree with your suggestion, however, while I'm at it, I think it's
> better that we have functions that check and print, and .balloon and
> .hot_add can rely on the return value, for example:
>
> static int balloon_enabled(void)
> {
> if (PAGE_SIZE != HV_HYP_PAGE_SIZE) {
> pr_info("Ballooning disabled because page size is not 4096 bytes\n");
> return 0;
> }
>
> return 1;
> }
>
> // in balloon_vsp_connect()
>
> cap_msg.caps.cap_bits.balloon = balloon_enabled();
>
> In this way, we keep the checking and reason printing in the same
> function and it's easier to maintain the consistency.
>
> Thoughts?
Yes, that approach looks good to me.
Michael
^ permalink raw reply [flat|nested] 10+ messages in thread
* [RFC v1.1] Drivers: hv: balloon: Disable balloon and hot-add accordingly
2022-02-23 13:15 ` [RFC 2/2] Drivers: hv: balloon: Disable balloon and hot-add accordingly Boqun Feng
2022-02-23 16:55 ` Michael Kelley (LINUX)
@ 2022-02-25 2:17 ` Boqun Feng
2022-02-25 17:06 ` Michael Kelley (LINUX)
1 sibling, 1 reply; 10+ messages in thread
From: Boqun Feng @ 2022-02-25 2:17 UTC (permalink / raw)
To: Wei Liu
Cc: Vitaly Kuznetsov, linux-hyperv, K. Y. Srinivasan, Haiyang Zhang,
Stephen Hemminger, Dexuan Cui, Michael Kelley, David Hildenbrand,
linux-kernel, Boqun Feng
Currently there are known potential issues for balloon and hot-add on
ARM64:
* Unballoon requests from Hyper-V should only unballoon ranges
that are guest page size aligned, otherwise guests cannot handle
because it's impossible to partially free a page.
* Memory hot-add requests from Hyper-V should provide the NUMA
node id of the added ranges or ARM64 should have a functional
memory_add_physaddr_to_nid(), otherwise the node id is missing
for add_memory().
These issues require discussions on design and implementation. In the
meanwhile, post_status() is working and essiential to guest monitoring.
Therefore instead of the entire hv_balloon driver, the balloon and
hot-add are disabled accordingly for now. Once the issues are fixed,
they can be re-enable in these cases.
Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
---
v1 --> v1.1:
* Use HV_HYP_PAGE_SIZE instead of hard coding 4096 as suggested by
Michael.
* Explicitly print out the disable message if a function is
disabled as suggested by Michael.
drivers/hv/hv_balloon.c | 36 ++++++++++++++++++++++++++++++++++--
1 file changed, 34 insertions(+), 2 deletions(-)
diff --git a/drivers/hv/hv_balloon.c b/drivers/hv/hv_balloon.c
index 062156b88a87..eee7402cfc02 100644
--- a/drivers/hv/hv_balloon.c
+++ b/drivers/hv/hv_balloon.c
@@ -1660,6 +1660,38 @@ static void disable_page_reporting(void)
}
}
+static int ballooning_enabled(void)
+{
+ /*
+ * Disable ballooning if the page size is not 4k (HV_HYP_PAGE_SIZE),
+ * since currently it's unclear to us whether an unballoon request can
+ * make sure all page ranges are guest page size aligned.
+ */
+ if (PAGE_SIZE != HV_HYP_PAGE_SIZE) {
+ pr_info("Ballooning disabled because page size is not 4096 bytes\n");
+ return 0;
+ }
+
+ return 1;
+}
+
+static int hot_add_enabled(void)
+{
+ /*
+ * Disable hot add on ARM64, because we currently rely on
+ * memory_add_physaddr_to_nid() to get a node id of a hot add range,
+ * however ARM64's memory_add_physaddr_to_nid() always return 0 and
+ * DM_MEM_HOT_ADD_REQUEST doesn't have the NUMA node information for
+ * add_memory().
+ */
+ if (IS_ENABLED(CONFIG_ARM64)) {
+ pr_info("Memory hot add disabled on ARM64\n");
+ return 0;
+ }
+
+ return 1;
+}
+
static int balloon_connect_vsp(struct hv_device *dev)
{
struct dm_version_request version_req;
@@ -1731,8 +1763,8 @@ static int balloon_connect_vsp(struct hv_device *dev)
* currently still requires the bits to be set, so we have to add code
* to fail the host's hot-add and balloon up/down requests, if any.
*/
- cap_msg.caps.cap_bits.balloon = 1;
- cap_msg.caps.cap_bits.hot_add = 1;
+ cap_msg.caps.cap_bits.balloon = ballooning_enabled();
+ cap_msg.caps.cap_bits.hot_add = hot_add_enabled();
/*
* Specify our alignment requirements as it relates
--
2.33.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* RE: [RFC v1.1] Drivers: hv: balloon: Disable balloon and hot-add accordingly
2022-02-25 2:17 ` [RFC v1.1] " Boqun Feng
@ 2022-02-25 17:06 ` Michael Kelley (LINUX)
2022-02-26 1:30 ` Boqun Feng
0 siblings, 1 reply; 10+ messages in thread
From: Michael Kelley (LINUX) @ 2022-02-25 17:06 UTC (permalink / raw)
To: Boqun Feng, Wei Liu
Cc: vkuznets, linux-hyperv, KY Srinivasan, Haiyang Zhang,
Stephen Hemminger, Dexuan Cui, David Hildenbrand, linux-kernel
From: Boqun Feng <boqun.feng@gmail.com> Sent: Thursday, February 24, 2022 6:17 PM
>
> Currently there are known potential issues for balloon and hot-add on
> ARM64:
>
> * Unballoon requests from Hyper-V should only unballoon ranges
> that are guest page size aligned, otherwise guests cannot handle
> because it's impossible to partially free a page.
The above problem occurs only when the guest page size is > 4 Kbytes.
>
> * Memory hot-add requests from Hyper-V should provide the NUMA
> node id of the added ranges or ARM64 should have a functional
> memory_add_physaddr_to_nid(), otherwise the node id is missing
> for add_memory().
>
> These issues require discussions on design and implementation. In the
> meanwhile, post_status() is working and essiential to guest monitoring.
s/essiential/essential/
> Therefore instead of the entire hv_balloon driver, the balloon and
> hot-add are disabled accordingly for now. Once the issues are fixed,
> they can be re-enable in these cases.
Missing the word "disabling" in the first line? Also the balloon
function is disabled only if the page size is > 4 Kbytes.
>
> Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
> ---
> v1 --> v1.1:
>
> * Use HV_HYP_PAGE_SIZE instead of hard coding 4096 as suggested by
> Michael.
>
> * Explicitly print out the disable message if a function is
> disabled as suggested by Michael.
>
> drivers/hv/hv_balloon.c | 36 ++++++++++++++++++++++++++++++++++--
> 1 file changed, 34 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/hv/hv_balloon.c b/drivers/hv/hv_balloon.c
> index 062156b88a87..eee7402cfc02 100644
> --- a/drivers/hv/hv_balloon.c
> +++ b/drivers/hv/hv_balloon.c
> @@ -1660,6 +1660,38 @@ static void disable_page_reporting(void)
> }
> }
>
> +static int ballooning_enabled(void)
> +{
> + /*
> + * Disable ballooning if the page size is not 4k (HV_HYP_PAGE_SIZE),
> + * since currently it's unclear to us whether an unballoon request can
> + * make sure all page ranges are guest page size aligned.
My interpretation of the conversations with Hyper-V is that that they clearly
don't guarantee page ranges are guest page aligned.
> + */
> + if (PAGE_SIZE != HV_HYP_PAGE_SIZE) {
> + pr_info("Ballooning disabled because page size is not 4096 bytes\n");
> + return 0;
> + }
> +
> + return 1;
> +}
> +
> +static int hot_add_enabled(void)
> +{
> + /*
> + * Disable hot add on ARM64, because we currently rely on
> + * memory_add_physaddr_to_nid() to get a node id of a hot add range,
> + * however ARM64's memory_add_physaddr_to_nid() always return 0 and
> + * DM_MEM_HOT_ADD_REQUEST doesn't have the NUMA node information for
> + * add_memory().
> + */
> + if (IS_ENABLED(CONFIG_ARM64)) {
> + pr_info("Memory hot add disabled on ARM64\n");
> + return 0;
> + }
> +
> + return 1;
> +}
> +
> static int balloon_connect_vsp(struct hv_device *dev)
> {
> struct dm_version_request version_req;
> @@ -1731,8 +1763,8 @@ static int balloon_connect_vsp(struct hv_device *dev)
> * currently still requires the bits to be set, so we have to add code
> * to fail the host's hot-add and balloon up/down requests, if any.
> */
> - cap_msg.caps.cap_bits.balloon = 1;
> - cap_msg.caps.cap_bits.hot_add = 1;
> + cap_msg.caps.cap_bits.balloon = ballooning_enabled();
> + cap_msg.caps.cap_bits.hot_add = hot_add_enabled();
>
> /*
> * Specify our alignment requirements as it relates
> --
> 2.33.0
The code looks good to me.
Michael
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC v1.1] Drivers: hv: balloon: Disable balloon and hot-add accordingly
2022-02-25 17:06 ` Michael Kelley (LINUX)
@ 2022-02-26 1:30 ` Boqun Feng
0 siblings, 0 replies; 10+ messages in thread
From: Boqun Feng @ 2022-02-26 1:30 UTC (permalink / raw)
To: Michael Kelley (LINUX)
Cc: Wei Liu, vkuznets, linux-hyperv, KY Srinivasan, Haiyang Zhang,
Stephen Hemminger, Dexuan Cui, David Hildenbrand, linux-kernel
On Fri, Feb 25, 2022 at 05:06:45PM +0000, Michael Kelley (LINUX) wrote:
> From: Boqun Feng <boqun.feng@gmail.com> Sent: Thursday, February 24, 2022 6:17 PM
> >
> > Currently there are known potential issues for balloon and hot-add on
> > ARM64:
> >
> > * Unballoon requests from Hyper-V should only unballoon ranges
> > that are guest page size aligned, otherwise guests cannot handle
> > because it's impossible to partially free a page.
>
> The above problem occurs only when the guest page size is > 4 Kbytes.
>
Ok, I wil call it out in next version.
> >
> > * Memory hot-add requests from Hyper-V should provide the NUMA
> > node id of the added ranges or ARM64 should have a functional
> > memory_add_physaddr_to_nid(), otherwise the node id is missing
> > for add_memory().
> >
> > These issues require discussions on design and implementation. In the
> > meanwhile, post_status() is working and essiential to guest monitoring.
>
> s/essiential/essential/
>
> > Therefore instead of the entire hv_balloon driver, the balloon and
> > hot-add are disabled accordingly for now. Once the issues are fixed,
> > they can be re-enable in these cases.
>
> Missing the word "disabling" in the first line? Also the balloon
The phrasing that I was trying to use here is "Instead of A, B and C are
disabled" or "B and C are disabled instead of A". Looks like I'm
inventing my own English? Any I will add the "disabling" in the next
version ;-)
Regards,
Boqun
> function is disabled only if the page size is > 4 Kbytes.
>
> >
> > Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
> > ---
> > v1 --> v1.1:
> >
> > * Use HV_HYP_PAGE_SIZE instead of hard coding 4096 as suggested by
> > Michael.
> >
> > * Explicitly print out the disable message if a function is
> > disabled as suggested by Michael.
> >
> > drivers/hv/hv_balloon.c | 36 ++++++++++++++++++++++++++++++++++--
> > 1 file changed, 34 insertions(+), 2 deletions(-)
> >
> > diff --git a/drivers/hv/hv_balloon.c b/drivers/hv/hv_balloon.c
> > index 062156b88a87..eee7402cfc02 100644
> > --- a/drivers/hv/hv_balloon.c
> > +++ b/drivers/hv/hv_balloon.c
> > @@ -1660,6 +1660,38 @@ static void disable_page_reporting(void)
> > }
> > }
> >
> > +static int ballooning_enabled(void)
> > +{
> > + /*
> > + * Disable ballooning if the page size is not 4k (HV_HYP_PAGE_SIZE),
> > + * since currently it's unclear to us whether an unballoon request can
> > + * make sure all page ranges are guest page size aligned.
>
> My interpretation of the conversations with Hyper-V is that that they clearly
> don't guarantee page ranges are guest page aligned.
>
> > + */
> > + if (PAGE_SIZE != HV_HYP_PAGE_SIZE) {
> > + pr_info("Ballooning disabled because page size is not 4096 bytes\n");
> > + return 0;
> > + }
> > +
> > + return 1;
> > +}
> > +
> > +static int hot_add_enabled(void)
> > +{
> > + /*
> > + * Disable hot add on ARM64, because we currently rely on
> > + * memory_add_physaddr_to_nid() to get a node id of a hot add range,
> > + * however ARM64's memory_add_physaddr_to_nid() always return 0 and
> > + * DM_MEM_HOT_ADD_REQUEST doesn't have the NUMA node information for
> > + * add_memory().
> > + */
> > + if (IS_ENABLED(CONFIG_ARM64)) {
> > + pr_info("Memory hot add disabled on ARM64\n");
> > + return 0;
> > + }
> > +
> > + return 1;
> > +}
> > +
> > static int balloon_connect_vsp(struct hv_device *dev)
> > {
> > struct dm_version_request version_req;
> > @@ -1731,8 +1763,8 @@ static int balloon_connect_vsp(struct hv_device *dev)
> > * currently still requires the bits to be set, so we have to add code
> > * to fail the host's hot-add and balloon up/down requests, if any.
> > */
> > - cap_msg.caps.cap_bits.balloon = 1;
> > - cap_msg.caps.cap_bits.hot_add = 1;
> > + cap_msg.caps.cap_bits.balloon = ballooning_enabled();
> > + cap_msg.caps.cap_bits.hot_add = hot_add_enabled();
> >
> > /*
> > * Specify our alignment requirements as it relates
> > --
> > 2.33.0
>
> The code looks good to me.
>
> Michael
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2022-02-26 1:31 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-02-23 13:15 [RFC 0/2] Drivers: hv: balloon: Temporary fixes for ARM64 Boqun Feng
2022-02-23 13:15 ` [RFC 1/2] Drivers: hv: balloon: Support status report for larger page sizes Boqun Feng
2022-02-23 16:45 ` Michael Kelley (LINUX)
2022-02-23 13:15 ` [RFC 2/2] Drivers: hv: balloon: Disable balloon and hot-add accordingly Boqun Feng
2022-02-23 16:55 ` Michael Kelley (LINUX)
2022-02-24 2:44 ` Boqun Feng
2022-02-24 4:44 ` Michael Kelley (LINUX)
2022-02-25 2:17 ` [RFC v1.1] " Boqun Feng
2022-02-25 17:06 ` Michael Kelley (LINUX)
2022-02-26 1:30 ` Boqun Feng
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).