From: David Hildenbrand <david@redhat.com> To: "Roger Pau Monné" <roger.pau@citrix.com>, "Jürgen Groß" <jgross@suse.com> Cc: linux-kernel@vger.kernel.org, Boris Ostrovsky <boris.ostrovsky@oracle.com>, Stefano Stabellini <sstabellini@kernel.org>, Andrew Morton <akpm@linux-foundation.org>, xen-devel@lists.xenproject.org, linux-mm@kvack.org Subject: Re: [PATCH 3/3] memory: introduce an option to force onlining of hotplug memory Date: Thu, 23 Jul 2020 15:14:31 +0200 [thread overview] Message-ID: <0e04b526-924d-aa51-cc2e-2c7561ce3df2@redhat.com> (raw) In-Reply-To: <20200723130831.GE7191@Air-de-Roger> On 23.07.20 15:08, Roger Pau Monné wrote: > On Thu, Jul 23, 2020 at 02:28:13PM +0200, Jürgen Groß wrote: >> On 23.07.20 14:23, Roger Pau Monné wrote: >>> On Thu, Jul 23, 2020 at 01:37:03PM +0200, David Hildenbrand wrote: >>>> On 23.07.20 10:45, Roger Pau Monne wrote: >>>>> Add an extra option to add_memory_resource that overrides the memory >>>>> hotplug online behavior in order to force onlining of memory from >>>>> add_memory_resource unconditionally. >>>>> >>>>> This is required for the Xen balloon driver, that must run the >>>>> online page callback in order to correctly process the newly added >>>>> memory region, note this is an unpopulated region that is used by Linux >>>>> to either hotplug RAM or to map foreign pages from other domains, and >>>>> hence memory hotplug when running on Xen can be used even without the >>>>> user explicitly requesting it, as part of the normal operations of the >>>>> OS when attempting to map memory from a different domain. >>>>> >>>>> Setting a different default value of memhp_default_online_type when >>>>> attaching the balloon driver is not a robust solution, as the user (or >>>>> distro init scripts) could still change it and thus break the Xen >>>>> balloon driver. >>>> >>>> I think we discussed this a couple of times before (even triggered by my >>>> request), and this is responsibility of user space to configure. Usually >>>> distros have udev rules to online memory automatically. Especially, user >>>> space should eb able to configure *how* to online memory. >>> >>> Note (as per the commit message) that in the specific case I'm >>> referring to the memory hotplugged by the Xen balloon driver will be >>> an unpopulated range to be used internally by certain Xen subsystems, >>> like the xen-blkback or the privcmd drivers. The addition of such >>> blocks of (unpopulated) memory can happen without the user explicitly >>> requesting it, and hence not even aware such hotplug process is taking >>> place. To be clear: no actual RAM will be added to the system. >>> >>> Failure to online such blocks using the Xen specific online handler >>> (which does not handle back the memory to the allocator in any way) >>> will result in the system getting stuck and malfunctioning. >>> >>>> It's the admin/distro responsibility to configure this properly. In case >>>> this doesn't happen (or as you say, users change it), bad luck. >>>> >>>> E.g., virtio-mem takes care to not add more memory in case it is not >>>> getting onlined. I remember hyper-v has similar code to at least wait a >>>> bit for memory to get onlined. >>> >>> I don't think VirtIO or Hyper-V use the hotplug system in the same way >>> as Xen, as said this is done to add unpopulated memory regions that >>> will be used to map foreign memory (from other domains) by Xen drivers >>> on the system. >>> >>> Maybe this should somehow use a different mechanism to hotplug such >>> empty memory blocks? I don't mind doing this differently, but I would >>> need some pointers. Allowing user-space to change a (seemingly >>> unrelated) parameter and as a result produce failures on Xen drivers >>> is not an acceptable solution IMO. >> >> Maybe we can use the same approach as Xen PV-domains: pre-allocate a >> region in the memory map to be used for mapping foreign pages. For the >> kernel it will look like pre-ballooned memory, so it will create struct >> page for the region (which is what we are after), but it won't give the >> memory to the allocator. > > IMO using something similar to memory hotplug would give us more > flexibility, and TBH the logic is already there in the balloon driver. > It seems quite wasteful to allocate such region(s) beforehand for all > domains, even when most of them won't end up using foreign mappings at > all. I do wonder why these issues you describe start to pop up now, literally years after this stuff has been implemented - or am I missing something important? > > Anyway, I'm going to take a look at how to do that, I guess it's going > to involve playing with the memory map and reserving some space. > > I suggest we should remove the Xen balloon hotplug logic, as it's not > working properly and we don't have a plan to fix it. Which exact hotplug logic are you referring to? -- Thanks, David / dhildenb
WARNING: multiple messages have this Message-ID (diff)
From: David Hildenbrand <david@redhat.com> To: "Roger Pau Monné" <roger.pau@citrix.com>, "Jürgen Groß" <jgross@suse.com> Cc: Stefano Stabellini <sstabellini@kernel.org>, linux-kernel@vger.kernel.org, linux-mm@kvack.org, xen-devel@lists.xenproject.org, Boris Ostrovsky <boris.ostrovsky@oracle.com>, Andrew Morton <akpm@linux-foundation.org> Subject: Re: [PATCH 3/3] memory: introduce an option to force onlining of hotplug memory Date: Thu, 23 Jul 2020 15:14:31 +0200 [thread overview] Message-ID: <0e04b526-924d-aa51-cc2e-2c7561ce3df2@redhat.com> (raw) In-Reply-To: <20200723130831.GE7191@Air-de-Roger> On 23.07.20 15:08, Roger Pau Monné wrote: > On Thu, Jul 23, 2020 at 02:28:13PM +0200, Jürgen Groß wrote: >> On 23.07.20 14:23, Roger Pau Monné wrote: >>> On Thu, Jul 23, 2020 at 01:37:03PM +0200, David Hildenbrand wrote: >>>> On 23.07.20 10:45, Roger Pau Monne wrote: >>>>> Add an extra option to add_memory_resource that overrides the memory >>>>> hotplug online behavior in order to force onlining of memory from >>>>> add_memory_resource unconditionally. >>>>> >>>>> This is required for the Xen balloon driver, that must run the >>>>> online page callback in order to correctly process the newly added >>>>> memory region, note this is an unpopulated region that is used by Linux >>>>> to either hotplug RAM or to map foreign pages from other domains, and >>>>> hence memory hotplug when running on Xen can be used even without the >>>>> user explicitly requesting it, as part of the normal operations of the >>>>> OS when attempting to map memory from a different domain. >>>>> >>>>> Setting a different default value of memhp_default_online_type when >>>>> attaching the balloon driver is not a robust solution, as the user (or >>>>> distro init scripts) could still change it and thus break the Xen >>>>> balloon driver. >>>> >>>> I think we discussed this a couple of times before (even triggered by my >>>> request), and this is responsibility of user space to configure. Usually >>>> distros have udev rules to online memory automatically. Especially, user >>>> space should eb able to configure *how* to online memory. >>> >>> Note (as per the commit message) that in the specific case I'm >>> referring to the memory hotplugged by the Xen balloon driver will be >>> an unpopulated range to be used internally by certain Xen subsystems, >>> like the xen-blkback or the privcmd drivers. The addition of such >>> blocks of (unpopulated) memory can happen without the user explicitly >>> requesting it, and hence not even aware such hotplug process is taking >>> place. To be clear: no actual RAM will be added to the system. >>> >>> Failure to online such blocks using the Xen specific online handler >>> (which does not handle back the memory to the allocator in any way) >>> will result in the system getting stuck and malfunctioning. >>> >>>> It's the admin/distro responsibility to configure this properly. In case >>>> this doesn't happen (or as you say, users change it), bad luck. >>>> >>>> E.g., virtio-mem takes care to not add more memory in case it is not >>>> getting onlined. I remember hyper-v has similar code to at least wait a >>>> bit for memory to get onlined. >>> >>> I don't think VirtIO or Hyper-V use the hotplug system in the same way >>> as Xen, as said this is done to add unpopulated memory regions that >>> will be used to map foreign memory (from other domains) by Xen drivers >>> on the system. >>> >>> Maybe this should somehow use a different mechanism to hotplug such >>> empty memory blocks? I don't mind doing this differently, but I would >>> need some pointers. Allowing user-space to change a (seemingly >>> unrelated) parameter and as a result produce failures on Xen drivers >>> is not an acceptable solution IMO. >> >> Maybe we can use the same approach as Xen PV-domains: pre-allocate a >> region in the memory map to be used for mapping foreign pages. For the >> kernel it will look like pre-ballooned memory, so it will create struct >> page for the region (which is what we are after), but it won't give the >> memory to the allocator. > > IMO using something similar to memory hotplug would give us more > flexibility, and TBH the logic is already there in the balloon driver. > It seems quite wasteful to allocate such region(s) beforehand for all > domains, even when most of them won't end up using foreign mappings at > all. I do wonder why these issues you describe start to pop up now, literally years after this stuff has been implemented - or am I missing something important? > > Anyway, I'm going to take a look at how to do that, I guess it's going > to involve playing with the memory map and reserving some space. > > I suggest we should remove the Xen balloon hotplug logic, as it's not > working properly and we don't have a plan to fix it. Which exact hotplug logic are you referring to? -- Thanks, David / dhildenb
next prev parent reply other threads:[~2020-07-23 13:14 UTC|newest] Thread overview: 47+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-07-23 8:45 [PATCH 0/3] xen/balloon: fixes for memory hotplug Roger Pau Monne 2020-07-23 8:45 ` [PATCH 1/3] xen/balloon: fix accounting in alloc_xenballooned_pages error path Roger Pau Monne 2020-07-23 8:45 ` Roger Pau Monne 2020-07-23 8:45 ` [PATCH 2/3] xen/balloon: make the balloon wait interruptible Roger Pau Monne 2020-07-23 8:45 ` Roger Pau Monne 2020-07-23 8:45 ` [PATCH 3/3] memory: introduce an option to force onlining of hotplug memory Roger Pau Monne 2020-07-23 8:45 ` Roger Pau Monne 2020-07-23 11:37 ` David Hildenbrand 2020-07-23 11:37 ` David Hildenbrand 2020-07-23 11:52 ` David Hildenbrand 2020-07-23 11:52 ` David Hildenbrand 2020-07-23 12:23 ` Roger Pau Monné 2020-07-23 12:23 ` Roger Pau Monné 2020-07-23 12:28 ` Jürgen Groß 2020-07-23 12:28 ` Jürgen Groß 2020-07-23 12:31 ` David Hildenbrand 2020-07-23 12:31 ` David Hildenbrand 2020-07-23 13:08 ` Roger Pau Monné 2020-07-23 13:08 ` Roger Pau Monné 2020-07-23 13:14 ` David Hildenbrand [this message] 2020-07-23 13:14 ` David Hildenbrand 2020-07-23 13:25 ` Roger Pau Monné 2020-07-23 13:25 ` Roger Pau Monné 2020-07-23 13:20 ` Jürgen Groß 2020-07-23 13:20 ` Jürgen Groß 2020-07-23 13:39 ` Roger Pau Monné 2020-07-23 13:39 ` Roger Pau Monné 2020-07-23 13:49 ` Jürgen Groß 2020-07-23 13:49 ` Jürgen Groß 2020-07-23 13:22 ` David Hildenbrand 2020-07-23 13:22 ` David Hildenbrand 2020-07-23 13:47 ` David Hildenbrand 2020-07-23 13:47 ` David Hildenbrand 2020-07-23 13:53 ` Jürgen Groß 2020-07-23 13:53 ` Jürgen Groß 2020-07-23 13:59 ` Roger Pau Monné 2020-07-23 13:59 ` Roger Pau Monné 2020-07-23 15:10 ` Jürgen Groß 2020-07-23 15:10 ` Jürgen Groß 2020-07-23 16:03 ` Andrew Cooper 2020-07-23 16:03 ` Andrew Cooper 2020-07-23 16:22 ` Roger Pau Monné 2020-07-23 16:22 ` Roger Pau Monné 2020-07-23 17:39 ` David Hildenbrand 2020-07-23 17:39 ` David Hildenbrand 2020-07-24 7:28 ` Michal Hocko 2020-07-24 7:28 ` Michal Hocko
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=0e04b526-924d-aa51-cc2e-2c7561ce3df2@redhat.com \ --to=david@redhat.com \ --cc=akpm@linux-foundation.org \ --cc=boris.ostrovsky@oracle.com \ --cc=jgross@suse.com \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=roger.pau@citrix.com \ --cc=sstabellini@kernel.org \ --cc=xen-devel@lists.xenproject.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.