RE: hv_balloon: Only works in ubuntu

["Michael Kelley (LINUX)" <mikelley@microsoft.com>]

From: Florian Müller <max06.net@outlook.com> Sent: Saturday, June 11, 2022 12:45 PM
> 
> > > > >
> > > > > Issues showed up when I set up a Kali Linux Guest. I missed the
> > > > > memory configuration before booting up the instance, so it started
> > > > > with 1GB of memory, and ballooning active between 512MB and
> > > > > several TB of memory. Hyper-V started to allocate more and more
> > > > > memory to this guest since the reported memory requirements also
> > > > > increased. The guest kernel didn't see any of that allocated memory, as 
> > > > > far as I can tell.
> > > >
> > > > Hot-adding new memory is done partially by the hv_balloon driver,
> > > > but it also requires user space action.  The user space action is provided
> > > > by udev rules.
> > > > In your Ubuntu Server 20.04 guest, there's a file
> > > > /lib/udev/rules.d/40-vm- hotadd.rules.
> > > > This file contains udev rules for hot-adding memory and CPUs.  You
> > > > should be able to copy this file with the udev rules onto your Kali
> > > > system, and then the memory hot-add should work correctly.
> > > >
> > > > I'm not sure why Kali doesn't already have such udev rules.  You
> > > > might grep through all the files in /lib/udev/rules.d and if there
> > > > are any rules for SUBSYSTEM==memory.
> > > > Sometimes there are rules present, but commented out.
> > > >
> > > Thanks, I'll check these ones out!
> > >
> > > In the meantime, I was able to get it working, by compiling a kernel
> > > with CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y
> > > Which was previously unset. It's enabled in ubuntu and it seems to
> > > make hv_balloon work properly.
> > > This seems to be the same case with Debian.
> > >
> >
> > Yes, that looks like a good solution.  I didn't remember that there is a kernel
> > config option to automatically do the onlining.  With this kernel option
> > enabled, using a udev rule obviously isn't needed.  The kernel option was
> > added in Linux kernel version 4.7, which might be after the last time I looked
> > at Hyper-V memory hot-add in detail.
> >
> > Michael
> >
> 
> Awesome!
> 
> Last question: Since not having the kernel option by default and also not having the
> udev rule in some distributions causes the Hyper-V host to eat up all the memory up to
> the defined limit (and to die eventually), should this be considered as a bug? And if the
> answer is no, how can I (or anyone) forward the requirement to the publishers to be
> solved at the source?
> 
> Thank you!
> 

It's unclear whether this should be treated as a bug.  We certainly want the
"right" thing to happen as seamlessly as possible, but there are tradeoffs.  Back
when Vitaly Kuznetsov added CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE,
I can see there was some debate about whether this option should be enabled
by default. There was reluctance to do so because of potential backwards
compatibility problems with other environments.  When hot-adding real
physical memory to a bare-metal server, apparently you don't want to 
automatically online the added memory.

The Ubuntu image you were testing is specifically an image configured for use
in an Azure VM, so it makes sense to have memory automatically onlined by
the kernel.  So I looked at a generic Ubuntu 18.04 system, and it also has
this kernel option set by default.  But as another data point, RHEL/CentOS 8.4
does *not* have the kernel option set.  So each distro evidently makes their
own decision about this.  But both generic Ubuntu 18.04 and RHEL/CentOS 8.4
have the udev rules.  The RHEL/CentOS 8.4 udev rule is more sophisticated
in that it does not online the added memory when running on System/390
and PowerPC.

I could envision changing the Linux kernel config rules to automatically set
CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE whenever 
CONFIG_HYPERV_BALLOON is selected.  Alternatively, the Kali Linux folks
could consider adding the appropriate udev rule.  My team at Microsoft
has direct engineering-to-engineer contacts with the most prevalent
distros for production server usage, such as in Azure.  But Kali is not one of
those, so I don't necessarily have a better way to contact them than you
do.

I've included Vitaly on the "To:" line of this email thread in case he has
any further thoughts.

Michael