hv_balloon: Only works in ubuntu

hv_balloon: Only works in ubuntu

[Florian Müller]

Hello there,

I'm trying to debug several, probably related issues to your hv_balloon kernel module. 

All tests are done on Win11 21H2 (22000.675) pro, on a Ryzen 3950x with active AMD-V, and 64GB of memory.
An updated Ubuntu Server 20.04 Guest is used for comparison. It's currently running kernel 5.13.0-1023-azure, configured with 1GB memory to start, and active ballooning between 512MB and 32GB of memory. Memory hot-plugging and -unplugging works.

Issues showed up when I set up a Kali Linux Guest. I missed the memory configuration before booting up the instance, so it started with 1GB of memory, and ballooning active between 512MB and several TB of memory. Hyper-V started to allocate more and more memory to this guest since the reported memory requirements also increased. The guest kernel didn't see any of that allocated memory, as far as I can tell. 

This is clearly reproduceable on at least 2 Hyper-V hosts, with current live images of Debian (Bullseye) and Kali Linux, but not with Ubuntu 20.04 or 22.04.
(Get the Kali live image, create a new guest (version 10), turn off secure boot and boot from that image. It takes 3-5 minutes until the issue is visible in the hyper-v console.) 

After running more tests with different memory settings for ballooning, I am pretty sure:
- Hyper-V respects the maximum setting for the memory balloon. Although it doesn't care if there's enough memory.
- Guests can't use/see more memory than they had while booting up.
- Guests can unplug memory.
- Guests can hotplug previously unplugged memory up to their starting amount.
- The reported values seem to be off: After compiling a kernel on Kali (and cooling down), the guest kernel shows a total of 3207MB memory, with 294MB used, 137MB free, 2775MB buffers/caches and 2611MB available. Hyper-V reports 4905MB required and 5840MB allocated. 
- As of kernel 5.17.11, the issue is not solved.

To sum up: I could use memory ballooning if I set the initial memory size to the maximum size and wait until it got freed up. 

There are several reports out there about what looks like a memory leak, without a solution. 

I'm currently comparing the kernel built by canonical with the kali kernel, but as I am not really a developer, I'm not sure if I could even find the difference if there is one. So, I'm calling (and hoping) for help, and offering any support when it comes to testing. I can apply patches, build kernel packages and read logs if that helps.

Thx-a-lot!
Flo

RE: hv_balloon: Only works in ubuntu

[Michael Kelley (LINUX)]

From: Florian Müller <max06.net@outlook.com> Sent: Saturday, June 11, 2022 8:40 AM
> 
> Hello there,
> 
> I'm trying to debug several, probably related issues to your hv_balloon kernel module.
> 
> All tests are done on Win11 21H2 (22000.675) pro, on a Ryzen 3950x with active AMD-
> V, and 64GB of memory.
> An updated Ubuntu Server 20.04 Guest is used for comparison. It's currently running
> kernel 5.13.0-1023-azure, configured with 1GB memory to start, and active ballooning
> between 512MB and 32GB of memory. Memory hot-plugging and -unplugging works.
> 
> Issues showed up when I set up a Kali Linux Guest. I missed the memory configuration
> before booting up the instance, so it started with 1GB of memory, and ballooning active
> between 512MB and several TB of memory. Hyper-V started to allocate more and more
> memory to this guest since the reported memory requirements also increased. The
> guest kernel didn't see any of that allocated memory, as far as I can tell.

Hot-adding new memory is done partially by the hv_balloon driver, but it also
requires user space action.  The user space action is provided by udev rules.  In
your Ubuntu Server 20.04 guest, there's a file /lib/udev/rules.d/40-vm-hotadd.rules.
This file contains udev rules for hot-adding memory and CPUs.  You should be able to
copy this file with the udev rules onto your Kali system, and then the memory
hot-add should work correctly.

I'm not sure why Kali doesn't already have such udev rules.  You might grep through
all the files in /lib/udev/rules.d and if there are any rules for SUBSYSTEM==memory.
Sometimes there are rules present, but commented out.

> 
> This is clearly reproduceable on at least 2 Hyper-V hosts, with current live images of
> Debian (Bullseye) and Kali Linux, but not with Ubuntu 20.04 or 22.04.
> (Get the Kali live image, create a new guest (version 10), turn off secure boot and boot
> from that image. It takes 3-5 minutes until the issue is visible in the hyper-v console.)
> 
> After running more tests with different memory settings for ballooning, I am pretty sure:
> - Hyper-V respects the maximum setting for the memory balloon. Although it doesn't
> care if there's enough memory.
> - Guests can't use/see more memory than they had while booting up.
> - Guests can unplug memory.
> - Guests can hotplug previously unplugged memory up to their starting amount.

Yes, adding this memory is done via the ballooning mechanism, and does not
require user space participation.   The hot-add mechanism is different from
ballooning, even though both are packaging in the hv_balloon driver.  Hot-add
is required only when adding memory that was not present when the VM first
booted, and that's when user space participation is needed via the udev rule.

> - The reported values seem to be off: After compiling a kernel on Kali (and cooling
> down), the guest kernel shows a total of 3207MB memory, with 294MB used, 137MB
> free, 2775MB buffers/caches and 2611MB available. Hyper-V reports 4905MB required
> and 5840MB allocated.
> - As of kernel 5.17.11, the issue is not solved.
> 
> To sum up: I could use memory ballooning if I set the initial memory size to the
> maximum size and wait until it got freed up.
> 
> There are several reports out there about what looks like a memory leak, without a
> solution.

Could you point me at those reports?  I'm not familiar with them.

Michael

> 
> I'm currently comparing the kernel built by canonical with the kali kernel, but as I am not
> really a developer, I'm not sure if I could even find the difference if there is one. So, I'm
> calling (and hoping) for help, and offering any support when it comes to testing. I can
> apply patches, build kernel packages and read logs if that helps.
> 
> Thx-a-lot!
> Flo

AW: hv_balloon: Only works in ubuntu

[Florian Müller]

> Von: Michael Kelley (LINUX) <mikelley@microsoft.com>
> From: Florian Müller <max06.net@outlook.com> Sent: Saturday, June 11,
> 2022 8:40 AM
> >
> > Hello there,
> >
> > I'm trying to debug several, probably related issues to your hv_balloon kernel module.
> >
> > All tests are done on Win11 21H2 (22000.675) pro, on a Ryzen 3950x
> > with active AMD- V, and 64GB of memory.
> > An updated Ubuntu Server 20.04 Guest is used for comparison. It's
> > currently running kernel 5.13.0-1023-azure, configured with 1GB memory
> > to start, and active ballooning between 512MB and 32GB of memory.
> > Memory hot-plugging and -unplugging works.
> >
> > Issues showed up when I set up a Kali Linux Guest. I missed the memory
> > configuration before booting up the instance, so it started with 1GB
> > of memory, and ballooning active between 512MB and several TB of
> > memory. Hyper-V started to allocate more and more memory to this guest
> > since the reported memory requirements also increased. The guest kernel
> > didn't see any of that allocated memory, as far as I can tell.
> 
> Hot-adding new memory is done partially by the hv_balloon driver, but it also
> requires user space action.  The user space action is provided by udev rules.
> In your Ubuntu Server 20.04 guest, there's a file /lib/udev/rules.d/40-vm-
> hotadd.rules.
> This file contains udev rules for hot-adding memory and CPUs.  You should be
> able to copy this file with the udev rules onto your Kali system, and then the
> memory hot-add should work correctly.
> 
> I'm not sure why Kali doesn't already have such udev rules.  You might grep
> through all the files in /lib/udev/rules.d and if there are any rules for
> SUBSYSTEM==memory.
> Sometimes there are rules present, but commented out.
> 
Thanks, I'll check these ones out!

In the meantime, I was able to get it working, by compiling a kernel with 
CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y
Which was previously unset. It's enabled in ubuntu and it seems to make hv_balloon work properly. 
This seems to be the same case with Debian.

> >
> > This is clearly reproduceable on at least 2 Hyper-V hosts, with
> > current live images of Debian (Bullseye) and Kali Linux, but not with Ubuntu
> 20.04 or 22.04.
> > (Get the Kali live image, create a new guest (version 10), turn off
> > secure boot and boot from that image. It takes 3-5 minutes until the
> > issue is visible in the hyper-v console.)
> >
> > After running more tests with different memory settings for ballooning, I
> > am pretty sure:
> > - Hyper-V respects the maximum setting for the memory balloon.
> > Although it doesn't care if there's enough memory.
> > - Guests can't use/see more memory than they had while booting up.
> > - Guests can unplug memory.
> > - Guests can hotplug previously unplugged memory up to their starting
> > amount.
> 
> Yes, adding this memory is done via the ballooning mechanism, and does not
> require user space participation.   The hot-add mechanism is different from
> ballooning, even though both are packaging in the hv_balloon driver.  Hot-
> add is required only when adding memory that was not present when the
> VM first booted, and that's when user space participation is needed via the
> udev rule.
> 
Gotcha, thanks for the clarification. I wasn't aware of that.

> > - The reported values seem to be off: After compiling a kernel on Kali
> > (and cooling down), the guest kernel shows a total of 3207MB memory,
> > with 294MB used, 137MB free, 2775MB buffers/caches and 2611MB
> > available. Hyper-V reports 4905MB required and 5840MB allocated.
> > - As of kernel 5.17.11, the issue is not solved.
> >
> > To sum up: I could use memory ballooning if I set the initial memory
> > size to the maximum size and wait until it got freed up.
> >
> > There are several reports out there about what looks like a memory
> > leak, without a solution.
> 
> Could you point me at those reports?  I'm not familiar with them.
Yes, as much as I'm able to find them again.
- https://community.clearlinux.org/t/dynamic-memory-broken-on-hyper-v/3891
... and the rest is gone. I'll post them when I find them again.

> 
> Michael
> 
> >
> > I'm currently comparing the kernel built by canonical with the kali
> > kernel, but as I am not really a developer, I'm not sure if I could
> > even find the difference if there is one. So, I'm calling (and hoping)
> > for help, and offering any support when it comes to testing. I can apply
> patches, build kernel packages and read logs if that helps.
> >
> > Thx-a-lot!
> > Flo
I hope I'm doing this mailing list thing right... 

RE: hv_balloon: Only works in ubuntu

[Michael Kelley (LINUX)]

From: Florian Müller <max06.net@outlook.com> Sent: Saturday, June 11, 2022 10:58 AM
> 
> > Von: Michael Kelley (LINUX) <mikelley@microsoft.com>
> > From: Florian Müller <max06.net@outlook.com> Sent: Saturday, June 11,
> > 2022 8:40 AM
> > >
> > > Hello there,
> > >
> > > I'm trying to debug several, probably related issues to your hv_balloon kernel module.
> > >
> > > All tests are done on Win11 21H2 (22000.675) pro, on a Ryzen 3950x
> > > with active AMD- V, and 64GB of memory.
> > > An updated Ubuntu Server 20.04 Guest is used for comparison. It's
> > > currently running kernel 5.13.0-1023-azure, configured with 1GB memory
> > > to start, and active ballooning between 512MB and 32GB of memory.
> > > Memory hot-plugging and -unplugging works.
> > >
> > > Issues showed up when I set up a Kali Linux Guest. I missed the memory
> > > configuration before booting up the instance, so it started with 1GB
> > > of memory, and ballooning active between 512MB and several TB of
> > > memory. Hyper-V started to allocate more and more memory to this guest
> > > since the reported memory requirements also increased. The guest kernel
> > > didn't see any of that allocated memory, as far as I can tell.
> >
> > Hot-adding new memory is done partially by the hv_balloon driver, but it also
> > requires user space action.  The user space action is provided by udev rules.
> > In your Ubuntu Server 20.04 guest, there's a file /lib/udev/rules.d/40-vm-
> > hotadd.rules.
> > This file contains udev rules for hot-adding memory and CPUs.  You should be
> > able to copy this file with the udev rules onto your Kali system, and then the
> > memory hot-add should work correctly.
> >
> > I'm not sure why Kali doesn't already have such udev rules.  You might grep
> > through all the files in /lib/udev/rules.d and if there are any rules for
> > SUBSYSTEM==memory.
> > Sometimes there are rules present, but commented out.
> >
> Thanks, I'll check these ones out!
> 
> In the meantime, I was able to get it working, by compiling a kernel with
> CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y
> Which was previously unset. It's enabled in ubuntu and it seems to make hv_balloon
> work properly.
> This seems to be the same case with Debian.
> 

Yes, that looks like a good solution.  I didn't remember that there is a
kernel config option to automatically do the onlining.  With this kernel
option enabled, using a udev rule obviously isn't needed.  The kernel
option was added in Linux kernel version 4.7, which might be
after the last time I looked at Hyper-V memory hot-add in detail.

Michael

> > >
> > > This is clearly reproduceable on at least 2 Hyper-V hosts, with
> > > current live images of Debian (Bullseye) and Kali Linux, but not with Ubuntu 20.04 or 22.04.
> > > (Get the Kali live image, create a new guest (version 10), turn off
> > > secure boot and boot from that image. It takes 3-5 minutes until the
> > > issue is visible in the hyper-v console.)
> > >
> > > After running more tests with different memory settings for ballooning, I
> > > am pretty sure:
> > > - Hyper-V respects the maximum setting for the memory balloon.
> > > Although it doesn't care if there's enough memory.
> > > - Guests can't use/see more memory than they had while booting up.
> > > - Guests can unplug memory.
> > > - Guests can hotplug previously unplugged memory up to their starting
> > > amount.
> >
> > Yes, adding this memory is done via the ballooning mechanism, and does not
> > require user space participation.   The hot-add mechanism is different from
> > ballooning, even though both are packaging in the hv_balloon driver.  Hot-
> > add is required only when adding memory that was not present when the
> > VM first booted, and that's when user space participation is needed via the
> > udev rule.
> >
> Gotcha, thanks for the clarification. I wasn't aware of that.
> 
> > > - The reported values seem to be off: After compiling a kernel on Kali
> > > (and cooling down), the guest kernel shows a total of 3207MB memory,
> > > with 294MB used, 137MB free, 2775MB buffers/caches and 2611MB
> > > available. Hyper-V reports 4905MB required and 5840MB allocated.
> > > - As of kernel 5.17.11, the issue is not solved.
> > >
> > > To sum up: I could use memory ballooning if I set the initial memory
> > > size to the maximum size and wait until it got freed up.
> > >
> > > There are several reports out there about what looks like a memory
> > > leak, without a solution.
> >
> > Could you point me at those reports?  I'm not familiar with them.
> Yes, as much as I'm able to find them again.
> - https://community.clearlinux.org/t/dynamic-memory-broken-on-hyper-v/3891
> ... and the rest is gone. I'll post them when I find them again.
> 

Thanks.

> >
> > Michael
> >
> > >
> > > I'm currently comparing the kernel built by canonical with the kali
> > > kernel, but as I am not really a developer, I'm not sure if I could
> > > even find the difference if there is one. So, I'm calling (and hoping)
> > > for help, and offering any support when it comes to testing. I can apply
> > patches, build kernel packages and read logs if that helps.
> > >
> > > Thx-a-lot!
> > > Flo
> I hope I'm doing this mailing list thing right...

Looks OK to me. :-)

Michael

AW: hv_balloon: Only works in ubuntu

[Florian Müller]

> Von: Michael Kelley (LINUX) <mikelley@microsoft.com>
> From: Florian Müller <max06.net@outlook.com> Sent: Saturday, June 11,
> 2022 10:58 AM
> >
> > > Von: Michael Kelley (LINUX) <mikelley@microsoft.com>
> > > From: Florian Müller <max06.net@outlook.com> Sent: Saturday, June
> > > 11,
> > > 2022 8:40 AM
> > > >
> > > > Hello there,
> > > >
> > > > I'm trying to debug several, probably related issues to your hv_balloon
> kernel module.
> > > >
> > > > All tests are done on Win11 21H2 (22000.675) pro, on a Ryzen 3950x
> > > > with active AMD- V, and 64GB of memory.
> > > > An updated Ubuntu Server 20.04 Guest is used for comparison. It's
> > > > currently running kernel 5.13.0-1023-azure, configured with 1GB
> > > > memory to start, and active ballooning between 512MB and 32GB of
> memory.
> > > > Memory hot-plugging and -unplugging works.
> > > >
> > > > Issues showed up when I set up a Kali Linux Guest. I missed the
> > > > memory configuration before booting up the instance, so it started
> > > > with 1GB of memory, and ballooning active between 512MB and
> > > > several TB of memory. Hyper-V started to allocate more and more
> > > > memory to this guest since the reported memory requirements also
> > > > increased. The guest kernel didn't see any of that allocated memory, as
> far as I can tell.
> > >
> > > Hot-adding new memory is done partially by the hv_balloon driver,
> > > but it also requires user space action.  The user space action is provided
> by udev rules.
> > > In your Ubuntu Server 20.04 guest, there's a file
> > > /lib/udev/rules.d/40-vm- hotadd.rules.
> > > This file contains udev rules for hot-adding memory and CPUs.  You
> > > should be able to copy this file with the udev rules onto your Kali
> > > system, and then the memory hot-add should work correctly.
> > >
> > > I'm not sure why Kali doesn't already have such udev rules.  You
> > > might grep through all the files in /lib/udev/rules.d and if there
> > > are any rules for SUBSYSTEM==memory.
> > > Sometimes there are rules present, but commented out.
> > >
> > Thanks, I'll check these ones out!
> >
> > In the meantime, I was able to get it working, by compiling a kernel
> > with CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y
> > Which was previously unset. It's enabled in ubuntu and it seems to
> > make hv_balloon work properly.
> > This seems to be the same case with Debian.
> >
> 
> Yes, that looks like a good solution.  I didn't remember that there is a kernel
> config option to automatically do the onlining.  With this kernel option
> enabled, using a udev rule obviously isn't needed.  The kernel option was
> added in Linux kernel version 4.7, which might be after the last time I looked
> at Hyper-V memory hot-add in detail.
> 
> Michael
> 

Awesome!

Last question: Since not having the kernel option by default and also not having the udev rule in some distributions causes the Hyper-V host to eat up all the memory up to the defined limit (and to die eventually), should this be considered as a bug? And if the answer is no, how can I (or anyone) forward the requirement to the publishers to be solved at the source?

Thank you!

> > > >
> > > > This is clearly reproduceable on at least 2 Hyper-V hosts, with
> > > > current live images of Debian (Bullseye) and Kali Linux, but not with
> Ubuntu 20.04 or 22.04.
> > > > (Get the Kali live image, create a new guest (version 10), turn
> > > > off secure boot and boot from that image. It takes 3-5 minutes
> > > > until the issue is visible in the hyper-v console.)
> > > >
> > > > After running more tests with different memory settings for
> > > > ballooning, I am pretty sure:
> > > > - Hyper-V respects the maximum setting for the memory balloon.
> > > > Although it doesn't care if there's enough memory.
> > > > - Guests can't use/see more memory than they had while booting up.
> > > > - Guests can unplug memory.
> > > > - Guests can hotplug previously unplugged memory up to their
> > > > starting amount.
> > >
> > > Yes, adding this memory is done via the ballooning mechanism, and does
> not
> > > require user space participation.   The hot-add mechanism is different
> from
> > > ballooning, even though both are packaging in the hv_balloon driver.
> > > Hot- add is required only when adding memory that was not present
> > > when the VM first booted, and that's when user space participation
> > > is needed via the udev rule.
> > >
> > Gotcha, thanks for the clarification. I wasn't aware of that.
> >
> > > > - The reported values seem to be off: After compiling a kernel on
> > > > Kali (and cooling down), the guest kernel shows a total of 3207MB
> > > > memory, with 294MB used, 137MB free, 2775MB buffers/caches and
> > > > 2611MB available. Hyper-V reports 4905MB required and 5840MB
> allocated.
> > > > - As of kernel 5.17.11, the issue is not solved.
> > > >
> > > > To sum up: I could use memory ballooning if I set the initial
> > > > memory size to the maximum size and wait until it got freed up.
> > > >
> > > > There are several reports out there about what looks like a memory
> > > > leak, without a solution.
> > >
> > > Could you point me at those reports?  I'm not familiar with them.
> > Yes, as much as I'm able to find them again.
> > -
> >
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcom
> m
> > unity.clearlinux.org%2Ft%2Fdynamic-memory-broken-on-hyper-
> v%2F3891&amp
> >
> ;data=05%7C01%7C%7Cf4d2032b2a48483e8be508da4be0c95c%7C84df9e7fe9
> f640af
> >
> b435aaaaaaaaaaaa%7C1%7C0%7C637905726063903963%7CUnknown%7CTWF
> pbGZsb3d8
> >
> eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3
> D%7C3
> >
> 000%7C%7C%7C&amp;sdata=Q4HXxFY2UHIvBFZv1RyEfdX21BNzo8Bd6%2BDE
> 6NhMDZM%3
> > D&amp;reserved=0 ... and the rest is gone. I'll post them when I find
> > them again.
> >
> 
> Thanks.
> 
> > >
> > > Michael
> > >
> > > >
> > > > I'm currently comparing the kernel built by canonical with the
> > > > kali kernel, but as I am not really a developer, I'm not sure if I
> > > > could even find the difference if there is one. So, I'm calling
> > > > (and hoping) for help, and offering any support when it comes to
> > > > testing. I can apply
> > > patches, build kernel packages and read logs if that helps.
> > > >
> > > > Thx-a-lot!
> > > > Flo
> > I hope I'm doing this mailing list thing right...
> 
> Looks OK to me. :-)
> 
> Michael

RE: hv_balloon: Only works in ubuntu

[Michael Kelley (LINUX)]

From: Florian Müller <max06.net@outlook.com> Sent: Saturday, June 11, 2022 12:45 PM
> 
> > > > >
> > > > > Issues showed up when I set up a Kali Linux Guest. I missed the
> > > > > memory configuration before booting up the instance, so it started
> > > > > with 1GB of memory, and ballooning active between 512MB and
> > > > > several TB of memory. Hyper-V started to allocate more and more
> > > > > memory to this guest since the reported memory requirements also
> > > > > increased. The guest kernel didn't see any of that allocated memory, as 
> > > > > far as I can tell.
> > > >
> > > > Hot-adding new memory is done partially by the hv_balloon driver,
> > > > but it also requires user space action.  The user space action is provided
> > > > by udev rules.
> > > > In your Ubuntu Server 20.04 guest, there's a file
> > > > /lib/udev/rules.d/40-vm- hotadd.rules.
> > > > This file contains udev rules for hot-adding memory and CPUs.  You
> > > > should be able to copy this file with the udev rules onto your Kali
> > > > system, and then the memory hot-add should work correctly.
> > > >
> > > > I'm not sure why Kali doesn't already have such udev rules.  You
> > > > might grep through all the files in /lib/udev/rules.d and if there
> > > > are any rules for SUBSYSTEM==memory.
> > > > Sometimes there are rules present, but commented out.
> > > >
> > > Thanks, I'll check these ones out!
> > >
> > > In the meantime, I was able to get it working, by compiling a kernel
> > > with CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y
> > > Which was previously unset. It's enabled in ubuntu and it seems to
> > > make hv_balloon work properly.
> > > This seems to be the same case with Debian.
> > >
> >
> > Yes, that looks like a good solution.  I didn't remember that there is a kernel
> > config option to automatically do the onlining.  With this kernel option
> > enabled, using a udev rule obviously isn't needed.  The kernel option was
> > added in Linux kernel version 4.7, which might be after the last time I looked
> > at Hyper-V memory hot-add in detail.
> >
> > Michael
> >
> 
> Awesome!
> 
> Last question: Since not having the kernel option by default and also not having the
> udev rule in some distributions causes the Hyper-V host to eat up all the memory up to
> the defined limit (and to die eventually), should this be considered as a bug? And if the
> answer is no, how can I (or anyone) forward the requirement to the publishers to be
> solved at the source?
> 
> Thank you!
> 

It's unclear whether this should be treated as a bug.  We certainly want the
"right" thing to happen as seamlessly as possible, but there are tradeoffs.  Back
when Vitaly Kuznetsov added CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE,
I can see there was some debate about whether this option should be enabled
by default. There was reluctance to do so because of potential backwards
compatibility problems with other environments.  When hot-adding real
physical memory to a bare-metal server, apparently you don't want to 
automatically online the added memory.

The Ubuntu image you were testing is specifically an image configured for use
in an Azure VM, so it makes sense to have memory automatically onlined by
the kernel.  So I looked at a generic Ubuntu 18.04 system, and it also has
this kernel option set by default.  But as another data point, RHEL/CentOS 8.4
does *not* have the kernel option set.  So each distro evidently makes their
own decision about this.  But both generic Ubuntu 18.04 and RHEL/CentOS 8.4
have the udev rules.  The RHEL/CentOS 8.4 udev rule is more sophisticated
in that it does not online the added memory when running on System/390
and PowerPC.

I could envision changing the Linux kernel config rules to automatically set
CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE whenever 
CONFIG_HYPERV_BALLOON is selected.  Alternatively, the Kali Linux folks
could consider adding the appropriate udev rule.  My team at Microsoft
has direct engineering-to-engineer contacts with the most prevalent
distros for production server usage, such as in Azure.  But Kali is not one of
those, so I don't necessarily have a better way to contact them than you
do.

I've included Vitaly on the "To:" line of this email thread in case he has
any further thoughts.

Michael

RE: hv_balloon: Only works in ubuntu

[Vitaly Kuznetsov]

"Michael Kelley (LINUX)" <mikelley@microsoft.com> writes:

> From: Florian Müller <max06.net@outlook.com> Sent: Saturday, June 11, 2022 12:45 PM
>> 
>> > > > >
>> > > > > Issues showed up when I set up a Kali Linux Guest. I missed the
>> > > > > memory configuration before booting up the instance, so it started
>> > > > > with 1GB of memory, and ballooning active between 512MB and
>> > > > > several TB of memory. Hyper-V started to allocate more and more
>> > > > > memory to this guest since the reported memory requirements also
>> > > > > increased. The guest kernel didn't see any of that allocated memory, as 
>> > > > > far as I can tell.
>> > > >
>> > > > Hot-adding new memory is done partially by the hv_balloon driver,
>> > > > but it also requires user space action.  The user space action is provided
>> > > > by udev rules.
>> > > > In your Ubuntu Server 20.04 guest, there's a file
>> > > > /lib/udev/rules.d/40-vm- hotadd.rules.
>> > > > This file contains udev rules for hot-adding memory and CPUs.  You
>> > > > should be able to copy this file with the udev rules onto your Kali
>> > > > system, and then the memory hot-add should work correctly.
>> > > >
>> > > > I'm not sure why Kali doesn't already have such udev rules.  You
>> > > > might grep through all the files in /lib/udev/rules.d and if there
>> > > > are any rules for SUBSYSTEM==memory.
>> > > > Sometimes there are rules present, but commented out.
>> > > >
>> > > Thanks, I'll check these ones out!
>> > >
>> > > In the meantime, I was able to get it working, by compiling a kernel
>> > > with CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y
>> > > Which was previously unset. It's enabled in ubuntu and it seems to
>> > > make hv_balloon work properly.
>> > > This seems to be the same case with Debian.
>> > >
>> >
>> > Yes, that looks like a good solution.  I didn't remember that there is a kernel
>> > config option to automatically do the onlining.  With this kernel option
>> > enabled, using a udev rule obviously isn't needed.  The kernel option was
>> > added in Linux kernel version 4.7, which might be after the last time I looked
>> > at Hyper-V memory hot-add in detail.
>> >
>> > Michael
>> >
>> 
>> Awesome!
>> 
>> Last question: Since not having the kernel option by default and also not having the
>> udev rule in some distributions causes the Hyper-V host to eat up all the memory up to
>> the defined limit (and to die eventually), should this be considered as a bug? And if the
>> answer is no, how can I (or anyone) forward the requirement to the publishers to be
>> solved at the source?
>> 
>> Thank you!
>> 
>
> It's unclear whether this should be treated as a bug.  We certainly want the
> "right" thing to happen as seamlessly as possible, but there are tradeoffs.  Back
> when Vitaly Kuznetsov added CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE,
> I can see there was some debate about whether this option should be enabled
> by default. There was reluctance to do so because of potential backwards
> compatibility problems with other environments.  When hot-adding real
> physical memory to a bare-metal server, apparently you don't want to 
> automatically online the added memory.

On x86/ARM you most likely do (as why plugging in memory in the first
place?) but there are not many bare metal systems which allow that.

Note, there were some developments after I've added
CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE, namely

commit 5f47adf762b78cae97de58d9ff01d2d44db09467
Author: David Hildenbrand <david@redhat.com>
Date:   Mon Apr 6 20:07:44 2020 -0700

    mm/memory_hotplug: allow to specify a default online_type

>
> The Ubuntu image you were testing is specifically an image configured for use
> in an Azure VM, so it makes sense to have memory automatically onlined by
> the kernel.  So I looked at a generic Ubuntu 18.04 system, and it also has
> this kernel option set by default.  But as another data point, RHEL/CentOS 8.4
> does *not* have the kernel option set.  So each distro evidently makes their
> own decision about this.  

Note: Fedora kernels come with CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y as
well.

> But both generic Ubuntu 18.04 and RHEL/CentOS 8.4
> have the udev rules.  The RHEL/CentOS 8.4 udev rule is more sophisticated
> in that it does not online the added memory when running on System/390
> and PowerPC.

I remember that with s390 we certainly don't want all memory to go
online automatically because but I don't remember the reason :-)

>
> I could envision changing the Linux kernel config rules to automatically set
> CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE whenever 
> CONFIG_HYPERV_BALLOON is selected. 

That could work for custom kernels but not for those shipped with
distributions as these will always have CONFIG_HYPERV_BALLOON if
Hyper-V is one of the supported targets (and it likely is).

> Alternatively, the Kali Linux folks could consider adding the
> appropriate udev rule.

Or just enable CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE. We've enabled
CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE in Fedora in 2016 and haven't heard
many (any?) complaints (besides ppc64 where it was disabled) since.

Cc: David who mostly picked up the auto onlining work since.

-- 
Vitaly

Re: hv_balloon: Only works in ubuntu

[David Hildenbrand]

On 13.06.22 09:58, Vitaly Kuznetsov wrote:
> "Michael Kelley (LINUX)" <mikelley@microsoft.com> writes:
> 
>> From: Florian Müller <max06.net@outlook.com> Sent: Saturday, June 11, 2022 12:45 PM
>>>
>>>>>>>
>>>>>>> Issues showed up when I set up a Kali Linux Guest. I missed the
>>>>>>> memory configuration before booting up the instance, so it started
>>>>>>> with 1GB of memory, and ballooning active between 512MB and
>>>>>>> several TB of memory. Hyper-V started to allocate more and more
>>>>>>> memory to this guest since the reported memory requirements also
>>>>>>> increased. The guest kernel didn't see any of that allocated memory, as 
>>>>>>> far as I can tell.
>>>>>>
>>>>>> Hot-adding new memory is done partially by the hv_balloon driver,
>>>>>> but it also requires user space action.  The user space action is provided
>>>>>> by udev rules.
>>>>>> In your Ubuntu Server 20.04 guest, there's a file
>>>>>> /lib/udev/rules.d/40-vm- hotadd.rules.
>>>>>> This file contains udev rules for hot-adding memory and CPUs.  You
>>>>>> should be able to copy this file with the udev rules onto your Kali
>>>>>> system, and then the memory hot-add should work correctly.
>>>>>>
>>>>>> I'm not sure why Kali doesn't already have such udev rules.  You
>>>>>> might grep through all the files in /lib/udev/rules.d and if there
>>>>>> are any rules for SUBSYSTEM==memory.
>>>>>> Sometimes there are rules present, but commented out.
>>>>>>
>>>>> Thanks, I'll check these ones out!
>>>>>
>>>>> In the meantime, I was able to get it working, by compiling a kernel
>>>>> with CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y
>>>>> Which was previously unset. It's enabled in ubuntu and it seems to
>>>>> make hv_balloon work properly.
>>>>> This seems to be the same case with Debian.
>>>>>
>>>>
>>>> Yes, that looks like a good solution.  I didn't remember that there is a kernel
>>>> config option to automatically do the onlining.  With this kernel option
>>>> enabled, using a udev rule obviously isn't needed.  The kernel option was
>>>> added in Linux kernel version 4.7, which might be after the last time I looked
>>>> at Hyper-V memory hot-add in detail.
>>>>
>>>> Michael
>>>>
>>>
>>> Awesome!
>>>
>>> Last question: Since not having the kernel option by default and also not having the
>>> udev rule in some distributions causes the Hyper-V host to eat up all the memory up to
>>> the defined limit (and to die eventually), should this be considered as a bug? And if the
>>> answer is no, how can I (or anyone) forward the requirement to the publishers to be
>>> solved at the source?
>>>
>>> Thank you!
>>>
>>
>> It's unclear whether this should be treated as a bug.  We certainly want the
>> "right" thing to happen as seamlessly as possible, but there are tradeoffs.  Back
>> when Vitaly Kuznetsov added CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE,
>> I can see there was some debate about whether this option should be enabled
>> by default. There was reluctance to do so because of potential backwards
>> compatibility problems with other environments.  When hot-adding real
>> physical memory to a bare-metal server, apparently you don't want to 
>> automatically online the added memory.
> 
> On x86/ARM you most likely do (as why plugging in memory in the first
> place?) but there are not many bare metal systems which allow that.

You actually do want to online in most cases, except, for example,
s390x. There might be weird cases with PFN-mapping offline memory into
VMs, but that is really a corner case when it comes to hotplugging memory.

However, "how" you want to online memory (normal/movable) depends on the
use case, the hotplug mechanism, and other factors (e.g., how much
memory will we hotplug, what are the system demands regarding
normal/movable memory, ...).

Hyperv-balloon just wants to online memory "normal", so not to
ZONE_MOVABLE. CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE should get the job done.

> 
> Note, there were some developments after I've added
> CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE, namely
> 
> commit 5f47adf762b78cae97de58d9ff01d2d44db09467
> Author: David Hildenbrand <david@redhat.com>
> Date:   Mon Apr 6 20:07:44 2020 -0700
> 
>     mm/memory_hotplug: allow to specify a default online_type
> 

I was working on a systemd unit to just configure memory onlining
semi-automatically, avoiding udev rules and to just configure
auto-onlining in the kernel during system boot.

Delaying memory onlining due to udev rules is a known issue when it
comes to hotplugging a lot of memory via hyperv balloon. We want kernel
to be onlined as fast as possible, directly when adding it in the kernel.

I currently don't have the capacity to work on the systemd unit, but the
latest version resides at:

https://github.com/davidhildenbrand/systemd/tree/memoryhotplugd

The idea is to let the unit figure out what might be best to do in the
detected environment. In addition, the admin can adjust the
configure/fine-tune handling without messing with udev rules, kernel
parameters or the sysfs.

>>
>> The Ubuntu image you were testing is specifically an image configured for use
>> in an Azure VM, so it makes sense to have memory automatically onlined by
>> the kernel.  So I looked at a generic Ubuntu 18.04 system, and it also has
>> this kernel option set by default.  But as another data point, RHEL/CentOS 8.4
>> does *not* have the kernel option set.  So each distro evidently makes their
>> own decision about this.  
> 
> Note: Fedora kernels come with CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y as
> well.

IIRC, even on s390x. It's different on RHEL, where we leave standby
memory under s390x offline via udev rules right now.

> 
>> But both generic Ubuntu 18.04 and RHEL/CentOS 8.4
>> have the udev rules.  The RHEL/CentOS 8.4 udev rule is more sophisticated
>> in that it does not online the added memory when running on System/390
>> and PowerPC.
> 
> I remember that with s390 we certainly don't want all memory to go
> online automatically because but I don't remember the reason :-)

The semantics of standby memory as you would find under z/VM and LPAR is
different: the admin is supposed to online it on demand; until then,
it's supposed to stay offline. Onlining will trigger actual backing of
the VM with additional memory, so some weird form of on-demand memory
hotplug.

> 
>>
>> I could envision changing the Linux kernel config rules to automatically set
>> CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE whenever 
>> CONFIG_HYPERV_BALLOON is selected. 
> 
> That could work for custom kernels but not for those shipped with
> distributions as these will always have CONFIG_HYPERV_BALLOON if
> Hyper-V is one of the supported targets (and it likely is).
> 

Yeah, that would be broken and nacked by me.

>> Alternatively, the Kali Linux folks could consider adding the
>> appropriate udev rule.
> 
> Or just enable CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE. We've enabled
> CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE in Fedora in 2016 and haven't heard
> many (any?) complaints (besides ppc64 where it was disabled) since.

The would work, most probably also for s390x. I assume most s390x users
that run these distros (fedora, kali,...) most probably don't care about
standby memory at all, meaning, not even having the option available in
their LPARs/VMs.


-- 
Thanks,

David / dhildenb

AW: hv_balloon: Only works in ubuntu

[Florian M?ller]

> >> > > > >
> >> > > > > Issues showed up when I set up a Kali Linux Guest. I missed
> >> > > > > the memory configuration before booting up the instance, so
> >> > > > > it started with 1GB of memory, and ballooning active between
> >> > > > > 512MB and several TB of memory. Hyper-V started to allocate
> >> > > > > more and more memory to this guest since the reported memory
> >> > > > > requirements also increased. The guest kernel didn't see any
> >> > > > > of that allocated memory, as far as I can tell.

Please do not forget about this: (emoji-pointing-up)

> >> >
> >> > Yes, that looks like a good solution.  I didn't remember that there
> >> > is a kernel config option to automatically do the onlining.  With
> >> > this kernel option enabled, using a udev rule obviously isn't
> >> > needed.  The kernel option was added in Linux kernel version 4.7,
> >> > which might be after the last time I looked at Hyper-V memory hot-add
> in detail.
> >> >
> >> > Michael
> >> >
> >>
> >> Awesome!
> >>
> >> Last question: Since not having the kernel option by default and also
> >> not having the udev rule in some distributions causes the Hyper-V
> >> host to eat up all the memory up to the defined limit (and to die
> >> eventually), should this be considered as a bug? And if the answer is
> >> no, how can I (or anyone) forward the requirement to the publishers to
> be solved at the source?
> >>
> >> Thank you!
> >>
> >
> > It's unclear whether this should be treated as a bug.  We certainly
> > want the "right" thing to happen as seamlessly as possible, but there
> > are tradeoffs.  Back when Vitaly Kuznetsov added
> > CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE,
> > I can see there was some debate about whether this option should be
> > enabled by default. There was reluctance to do so because of potential
> > backwards compatibility problems with other environments.  When
> > hot-adding real physical memory to a bare-metal server, apparently you
> > don't want to automatically online the added memory.

By bug I meant the effects on the hypervisor (see above). A guest without proper onlining of newly added memory is currently able to choke the host to standstill.

> 
> On x86/ARM you most likely do (as why plugging in memory in the first
> place?) but there are not many bare metal systems which allow that.
> 
> Note, there were some developments after I've added
> CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE, namely
> 
> commit 5f47adf762b78cae97de58d9ff01d2d44db09467
> Author: David Hildenbrand <david@redhat.com>
> Date:   Mon Apr 6 20:07:44 2020 -0700
> 
>     mm/memory_hotplug: allow to specify a default online_type
> 
> >
> > The Ubuntu image you were testing is specifically an image configured
> > for use in an Azure VM, so it makes sense to have memory automatically
> > onlined by the kernel.  So I looked at a generic Ubuntu 18.04 system,
> > and it also has this kernel option set by default.  But as another
> > data point, RHEL/CentOS 8.4 does *not* have the kernel option set.  So
> > each distro evidently makes their own decision about this.
> 
> Note: Fedora kernels come with
> CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y as well.
> 
> > But both generic Ubuntu 18.04 and RHEL/CentOS 8.4 have the udev rules.
> > The RHEL/CentOS 8.4 udev rule is more sophisticated in that it does
> > not online the added memory when running on System/390 and PowerPC.
> 
> I remember that with s390 we certainly don't want all memory to go online
> automatically because but I don't remember the reason :-)
> 

So, using the udev rule is the easier solution, it doesn't require a kernel configuration change - but it needs to be added (and managed) by every vendor.
Changing the kernel default is not the easiest thing to do due to possible regressions, and it can still be overwritten by vendors.
Having CONFIG_HYPERV_BALLOON require CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE seems like a good solution to me, but I'm not the expert here.

> >
> > I could envision changing the Linux kernel config rules to
> > automatically set CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE
> whenever
> > CONFIG_HYPERV_BALLOON is selected.
> 
> That could work for custom kernels but not for those shipped with
> distributions as these will always have CONFIG_HYPERV_BALLOON if Hyper-V
> is one of the supported targets (and it likely is).
> 
> > Alternatively, the Kali Linux folks could consider adding the
> > appropriate udev rule.
> 
I am in contact with the kali maintainers and forwarded this mail thread to them. 

> Or just enable CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE. We've
> enabled CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE in Fedora in 2016
> and haven't heard many (any?) complaints (besides ppc64 where it was
> disabled) since.
> 
> Cc: David who mostly picked up the auto onlining work since.
> 
> --
> Vitaly

Re: AW: hv_balloon: Only works in ubuntu

[David Hildenbrand]

On 13.06.22 10:53, Florian M?ller wrote:
>>>>>>>>
>>>>>>>> Issues showed up when I set up a Kali Linux Guest. I missed
>>>>>>>> the memory configuration before booting up the instance, so
>>>>>>>> it started with 1GB of memory, and ballooning active between
>>>>>>>> 512MB and several TB of memory. Hyper-V started to allocate
>>>>>>>> more and more memory to this guest since the reported memory
>>>>>>>> requirements also increased. The guest kernel didn't see any
>>>>>>>> of that allocated memory, as far as I can tell.
> 
> Please do not forget about this: (emoji-pointing-up)
> 
>>>>>
>>>>> Yes, that looks like a good solution.  I didn't remember that there
>>>>> is a kernel config option to automatically do the onlining.  With
>>>>> this kernel option enabled, using a udev rule obviously isn't
>>>>> needed.  The kernel option was added in Linux kernel version 4.7,
>>>>> which might be after the last time I looked at Hyper-V memory hot-add
>> in detail.
>>>>>
>>>>> Michael
>>>>>
>>>>
>>>> Awesome!
>>>>
>>>> Last question: Since not having the kernel option by default and also
>>>> not having the udev rule in some distributions causes the Hyper-V
>>>> host to eat up all the memory up to the defined limit (and to die
>>>> eventually), should this be considered as a bug? And if the answer is
>>>> no, how can I (or anyone) forward the requirement to the publishers to
>> be solved at the source?
>>>>
>>>> Thank you!
>>>>
>>>
>>> It's unclear whether this should be treated as a bug.  We certainly
>>> want the "right" thing to happen as seamlessly as possible, but there
>>> are tradeoffs.  Back when Vitaly Kuznetsov added
>>> CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE,
>>> I can see there was some debate about whether this option should be
>>> enabled by default. There was reluctance to do so because of potential
>>> backwards compatibility problems with other environments.  When
>>> hot-adding real physical memory to a bare-metal server, apparently you
>>> don't want to automatically online the added memory.
> 
> By bug I meant the effects on the hypervisor (see above). A guest without proper onlining of newly added memory is currently able to choke the host to standstill.

That's a hypervisor bug.


-- 
Thanks,

David / dhildenb

RE: hv_balloon: Only works in ubuntu

[Michael Kelley (LINUX)]

From: Florian M?ller <max06.net@outlook.com> Sent: Monday, June 13, 2022 1:53 AM
> 
> > >> > > > >
> > >> > > > > Issues showed up when I set up a Kali Linux Guest. I missed
> > >> > > > > the memory configuration before booting up the instance, so
> > >> > > > > it started with 1GB of memory, and ballooning active between
> > >> > > > > 512MB and several TB of memory. Hyper-V started to allocate
> > >> > > > > more and more memory to this guest since the reported memory
> > >> > > > > requirements also increased. The guest kernel didn't see any
> > >> > > > > of that allocated memory, as far as I can tell.
> 
> Please do not forget about this: (emoji-pointing-up)
> 

Hmmm.  Right off the bat, I don't know how to fix this.  Hyper-V tells the
guest "Here is more memory".  The hv_balloon driver adds the memory
(but doesn't mark it "online"), and sends a positive ACK to Hyper-V. 
From Hyper-V's standpoint, it has successfully given the memory to the
guest. But if the guest hasn't onlined the memory and isn't using it, the
guest continues to report high memory pressure.  Hyper-V assigns yet
more memory to the guest, still to no effect.  Having the hv_balloon
driver delay the ACK until the memory comes online is fraught with
problems, and of course Hyper-V has no visibility into whether the guest
has onlined the memory.

This may be one where the guest configuration really must be
correct.   But I'm open to other suggestions for a possible solution.

Michael

AW: hv_balloon: Only works in ubuntu

[Florian M?ller]

> Von: Michael Kelley (LINUX) <mikelley@microsoft.com>
> > > >> > > > >
> > > >> > > > > Issues showed up when I set up a Kali Linux Guest. I
> > > >> > > > > missed the memory configuration before booting up the
> > > >> > > > > instance, so it started with 1GB of memory, and
> > > >> > > > > ballooning active between 512MB and several TB of memory.
> > > >> > > > > Hyper-V started to allocate more and more memory to this
> > > >> > > > > guest since the reported memory requirements also
> > > >> > > > > increased. The guest kernel didn't see any of that allocated
> memory, as far as I can tell.
> >
> > Please do not forget about this: (emoji-pointing-up)
> >
> 
> Hmmm.  Right off the bat, I don't know how to fix this.  Hyper-V tells the
> guest "Here is more memory".  The hv_balloon driver adds the memory (but
> doesn't mark it "online"), and sends a positive ACK to Hyper-V.
> From Hyper-V's standpoint, it has successfully given the memory to the
> guest. But if the guest hasn't onlined the memory and isn't using it, the guest
> continues to report high memory pressure.  Hyper-V assigns yet more
> memory to the guest, still to no effect.  Having the hv_balloon driver delay
> the ACK until the memory comes online is fraught with problems, and of
> course Hyper-V has no visibility into whether the guest has onlined the
> memory.
> 
> This may be one where the guest configuration really must be
> correct.   But I'm open to other suggestions for a possible solution.
> 
> Michael

From checking the drivers code, it looks like the guest tells only the free and committed memory, not the total. I can also see considerations about num_pages_onlined in the committed-calculation.

I see 2 possible options at the moment: Adding num_total to the message (changing the protocol), or stop reporting if the guest fails to online memory after the first increase. A third, more complicated option would be checking for not onlined pages (I've seen functions for that in the code) and adding them to the free value in the report.

I'd love to write a patch for this if I had a clue how to test and debug it without rebuilding my kernel all the time. 

Flo

RE: hv_balloon: Only works in ubuntu

[Michael Kelley (LINUX)]

From: Florian M?ller <max06.net@outlook.com>
> 
> > Von: Michael Kelley (LINUX) <mikelley@microsoft.com>
> > > > >> > > > >
> > > > >> > > > > Issues showed up when I set up a Kali Linux Guest. I
> > > > >> > > > > missed the memory configuration before booting up the
> > > > >> > > > > instance, so it started with 1GB of memory, and
> > > > >> > > > > ballooning active between 512MB and several TB of memory.
> > > > >> > > > > Hyper-V started to allocate more and more memory to this
> > > > >> > > > > guest since the reported memory requirements also
> > > > >> > > > > increased. The guest kernel didn't see any of that allocated
> > memory, as far as I can tell.
> > >
> > > Please do not forget about this: (emoji-pointing-up)
> > >
> >
> > Hmmm.  Right off the bat, I don't know how to fix this.  Hyper-V tells the
> > guest "Here is more memory".  The hv_balloon driver adds the memory (but
> > doesn't mark it "online"), and sends a positive ACK to Hyper-V.
> > From Hyper-V's standpoint, it has successfully given the memory to the
> > guest. But if the guest hasn't onlined the memory and isn't using it, the guest
> > continues to report high memory pressure.  Hyper-V assigns yet more
> > memory to the guest, still to no effect.  Having the hv_balloon driver delay
> > the ACK until the memory comes online is fraught with problems, and of
> > course Hyper-V has no visibility into whether the guest has onlined the
> > memory.
> >
> > This may be one where the guest configuration really must be
> > correct.   But I'm open to other suggestions for a possible solution.
> >
> > Michael
> 
> From checking the drivers code, it looks like the guest tells only the free and committed
> memory, not the total. I can also see considerations about num_pages_onlined in the
> committed-calculation.
> 
> I see 2 possible options at the moment: Adding num_total to the message (changing the
> protocol), or stop reporting if the guest fails to online memory after the first increase. A
> third, more complicated option would be checking for not onlined pages (I've seen
> functions for that in the code) and adding them to the free value in the report.
> 

Changing the protocol with the Hyper-V host probably isn't practical.  This
wouldn't be a high enough priority issue for the Hyper-V team to make the
needed changes on their side.  The second option also has problems.  We really
don't want to just stop reporting (and I'm not sure what Hyper-V does if the
guest stops reporting), and it's always hard to know how long to wait for user
space to do something.  The third option sounds feasible, though I haven't
looked at the details.

> I'd love to write a patch for this if I had a clue how to test and debug it without
> rebuilding my kernel all the time.
> 

Feel free to have at it. :-)   I can also put the third option on our list of things to
look at, but it will be in the "when we can get to it" category.

Michael