AW: hv_balloon: Only works in ubuntu

["Florian Müller" <max06.net@outlook.com>]

> Von: Michael Kelley (LINUX) <mikelley@microsoft.com>
> From: Florian Müller <max06.net@outlook.com> Sent: Saturday, June 11,
> 2022 10:58 AM
> >
> > > Von: Michael Kelley (LINUX) <mikelley@microsoft.com>
> > > From: Florian Müller <max06.net@outlook.com> Sent: Saturday, June
> > > 11,
> > > 2022 8:40 AM
> > > >
> > > > Hello there,
> > > >
> > > > I'm trying to debug several, probably related issues to your hv_balloon
> kernel module.
> > > >
> > > > All tests are done on Win11 21H2 (22000.675) pro, on a Ryzen 3950x
> > > > with active AMD- V, and 64GB of memory.
> > > > An updated Ubuntu Server 20.04 Guest is used for comparison. It's
> > > > currently running kernel 5.13.0-1023-azure, configured with 1GB
> > > > memory to start, and active ballooning between 512MB and 32GB of
> memory.
> > > > Memory hot-plugging and -unplugging works.
> > > >
> > > > Issues showed up when I set up a Kali Linux Guest. I missed the
> > > > memory configuration before booting up the instance, so it started
> > > > with 1GB of memory, and ballooning active between 512MB and
> > > > several TB of memory. Hyper-V started to allocate more and more
> > > > memory to this guest since the reported memory requirements also
> > > > increased. The guest kernel didn't see any of that allocated memory, as
> far as I can tell.
> > >
> > > Hot-adding new memory is done partially by the hv_balloon driver,
> > > but it also requires user space action.  The user space action is provided
> by udev rules.
> > > In your Ubuntu Server 20.04 guest, there's a file
> > > /lib/udev/rules.d/40-vm- hotadd.rules.
> > > This file contains udev rules for hot-adding memory and CPUs.  You
> > > should be able to copy this file with the udev rules onto your Kali
> > > system, and then the memory hot-add should work correctly.
> > >
> > > I'm not sure why Kali doesn't already have such udev rules.  You
> > > might grep through all the files in /lib/udev/rules.d and if there
> > > are any rules for SUBSYSTEM==memory.
> > > Sometimes there are rules present, but commented out.
> > >
> > Thanks, I'll check these ones out!
> >
> > In the meantime, I was able to get it working, by compiling a kernel
> > with CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y
> > Which was previously unset. It's enabled in ubuntu and it seems to
> > make hv_balloon work properly.
> > This seems to be the same case with Debian.
> >
> 
> Yes, that looks like a good solution.  I didn't remember that there is a kernel
> config option to automatically do the onlining.  With this kernel option
> enabled, using a udev rule obviously isn't needed.  The kernel option was
> added in Linux kernel version 4.7, which might be after the last time I looked
> at Hyper-V memory hot-add in detail.
> 
> Michael
> 

Awesome!

Last question: Since not having the kernel option by default and also not having the udev rule in some distributions causes the Hyper-V host to eat up all the memory up to the defined limit (and to die eventually), should this be considered as a bug? And if the answer is no, how can I (or anyone) forward the requirement to the publishers to be solved at the source?

Thank you!

> > > >
> > > > This is clearly reproduceable on at least 2 Hyper-V hosts, with
> > > > current live images of Debian (Bullseye) and Kali Linux, but not with
> Ubuntu 20.04 or 22.04.
> > > > (Get the Kali live image, create a new guest (version 10), turn
> > > > off secure boot and boot from that image. It takes 3-5 minutes
> > > > until the issue is visible in the hyper-v console.)
> > > >
> > > > After running more tests with different memory settings for
> > > > ballooning, I am pretty sure:
> > > > - Hyper-V respects the maximum setting for the memory balloon.
> > > > Although it doesn't care if there's enough memory.
> > > > - Guests can't use/see more memory than they had while booting up.
> > > > - Guests can unplug memory.
> > > > - Guests can hotplug previously unplugged memory up to their
> > > > starting amount.
> > >
> > > Yes, adding this memory is done via the ballooning mechanism, and does
> not
> > > require user space participation.   The hot-add mechanism is different
> from
> > > ballooning, even though both are packaging in the hv_balloon driver.
> > > Hot- add is required only when adding memory that was not present
> > > when the VM first booted, and that's when user space participation
> > > is needed via the udev rule.
> > >
> > Gotcha, thanks for the clarification. I wasn't aware of that.
> >
> > > > - The reported values seem to be off: After compiling a kernel on
> > > > Kali (and cooling down), the guest kernel shows a total of 3207MB
> > > > memory, with 294MB used, 137MB free, 2775MB buffers/caches and
> > > > 2611MB available. Hyper-V reports 4905MB required and 5840MB
> allocated.
> > > > - As of kernel 5.17.11, the issue is not solved.
> > > >
> > > > To sum up: I could use memory ballooning if I set the initial
> > > > memory size to the maximum size and wait until it got freed up.
> > > >
> > > > There are several reports out there about what looks like a memory
> > > > leak, without a solution.
> > >
> > > Could you point me at those reports?  I'm not familiar with them.
> > Yes, as much as I'm able to find them again.
> > -
> >
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcom
> m
> > unity.clearlinux.org%2Ft%2Fdynamic-memory-broken-on-hyper-
> v%2F3891&amp
> >
> ;data=05%7C01%7C%7Cf4d2032b2a48483e8be508da4be0c95c%7C84df9e7fe9
> f640af
> >
> b435aaaaaaaaaaaa%7C1%7C0%7C637905726063903963%7CUnknown%7CTWF
> pbGZsb3d8
> >
> eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3
> D%7C3
> >
> 000%7C%7C%7C&amp;sdata=Q4HXxFY2UHIvBFZv1RyEfdX21BNzo8Bd6%2BDE
> 6NhMDZM%3
> > D&amp;reserved=0 ... and the rest is gone. I'll post them when I find
> > them again.
> >
> 
> Thanks.
> 
> > >
> > > Michael
> > >
> > > >
> > > > I'm currently comparing the kernel built by canonical with the
> > > > kali kernel, but as I am not really a developer, I'm not sure if I
> > > > could even find the difference if there is one. So, I'm calling
> > > > (and hoping) for help, and offering any support when it comes to
> > > > testing. I can apply
> > > patches, build kernel packages and read logs if that helps.
> > > >
> > > > Thx-a-lot!
> > > > Flo
> > I hope I'm doing this mailing list thing right...
> 
> Looks OK to me. :-)
> 
> Michael