AW: hv_balloon: Only works in ubuntu

["Florian Müller" <max06.net@outlook.com>]

> Von: Michael Kelley (LINUX) <mikelley@microsoft.com>
> From: Florian Müller <max06.net@outlook.com> Sent: Saturday, June 11,
> 2022 8:40 AM
> >
> > Hello there,
> >
> > I'm trying to debug several, probably related issues to your hv_balloon kernel module.
> >
> > All tests are done on Win11 21H2 (22000.675) pro, on a Ryzen 3950x
> > with active AMD- V, and 64GB of memory.
> > An updated Ubuntu Server 20.04 Guest is used for comparison. It's
> > currently running kernel 5.13.0-1023-azure, configured with 1GB memory
> > to start, and active ballooning between 512MB and 32GB of memory.
> > Memory hot-plugging and -unplugging works.
> >
> > Issues showed up when I set up a Kali Linux Guest. I missed the memory
> > configuration before booting up the instance, so it started with 1GB
> > of memory, and ballooning active between 512MB and several TB of
> > memory. Hyper-V started to allocate more and more memory to this guest
> > since the reported memory requirements also increased. The guest kernel
> > didn't see any of that allocated memory, as far as I can tell.
> 
> Hot-adding new memory is done partially by the hv_balloon driver, but it also
> requires user space action.  The user space action is provided by udev rules.
> In your Ubuntu Server 20.04 guest, there's a file /lib/udev/rules.d/40-vm-
> hotadd.rules.
> This file contains udev rules for hot-adding memory and CPUs.  You should be
> able to copy this file with the udev rules onto your Kali system, and then the
> memory hot-add should work correctly.
> 
> I'm not sure why Kali doesn't already have such udev rules.  You might grep
> through all the files in /lib/udev/rules.d and if there are any rules for
> SUBSYSTEM==memory.
> Sometimes there are rules present, but commented out.
> 
Thanks, I'll check these ones out!

In the meantime, I was able to get it working, by compiling a kernel with 
CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y
Which was previously unset. It's enabled in ubuntu and it seems to make hv_balloon work properly. 
This seems to be the same case with Debian.

> >
> > This is clearly reproduceable on at least 2 Hyper-V hosts, with
> > current live images of Debian (Bullseye) and Kali Linux, but not with Ubuntu
> 20.04 or 22.04.
> > (Get the Kali live image, create a new guest (version 10), turn off
> > secure boot and boot from that image. It takes 3-5 minutes until the
> > issue is visible in the hyper-v console.)
> >
> > After running more tests with different memory settings for ballooning, I
> > am pretty sure:
> > - Hyper-V respects the maximum setting for the memory balloon.
> > Although it doesn't care if there's enough memory.
> > - Guests can't use/see more memory than they had while booting up.
> > - Guests can unplug memory.
> > - Guests can hotplug previously unplugged memory up to their starting
> > amount.
> 
> Yes, adding this memory is done via the ballooning mechanism, and does not
> require user space participation.   The hot-add mechanism is different from
> ballooning, even though both are packaging in the hv_balloon driver.  Hot-
> add is required only when adding memory that was not present when the
> VM first booted, and that's when user space participation is needed via the
> udev rule.
> 
Gotcha, thanks for the clarification. I wasn't aware of that.

> > - The reported values seem to be off: After compiling a kernel on Kali
> > (and cooling down), the guest kernel shows a total of 3207MB memory,
> > with 294MB used, 137MB free, 2775MB buffers/caches and 2611MB
> > available. Hyper-V reports 4905MB required and 5840MB allocated.
> > - As of kernel 5.17.11, the issue is not solved.
> >
> > To sum up: I could use memory ballooning if I set the initial memory
> > size to the maximum size and wait until it got freed up.
> >
> > There are several reports out there about what looks like a memory
> > leak, without a solution.
> 
> Could you point me at those reports?  I'm not familiar with them.
Yes, as much as I'm able to find them again.
- https://community.clearlinux.org/t/dynamic-memory-broken-on-hyper-v/3891
... and the rest is gone. I'll post them when I find them again.

> 
> Michael
> 
> >
> > I'm currently comparing the kernel built by canonical with the kali
> > kernel, but as I am not really a developer, I'm not sure if I could
> > even find the difference if there is one. So, I'm calling (and hoping)
> > for help, and offering any support when it comes to testing. I can apply
> patches, build kernel packages and read logs if that helps.
> >
> > Thx-a-lot!
> > Flo
I hope I'm doing this mailing list thing right...