From: Wei Yang <richard.weiyang@linux.alibaba.com>
To: David Hildenbrand <david@redhat.com>
Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org,
virtualization@lists.linux-foundation.org,
Andrew Morton <akpm@linux-foundation.org>,
"Michael S . Tsirkin" <mst@redhat.com>,
Jason Wang <jasowang@redhat.com>,
Pankaj Gupta <pankaj.gupta.linux@gmail.com>,
Michal Hocko <mhocko@kernel.org>,
Oscar Salvador <osalvador@suse.de>,
Wei Yang <richard.weiyang@linux.alibaba.com>
Subject: Re: [PATCH v1 27/29] mm/memory_hotplug: extend offline_and_remove_memory() to handle more than one memory block
Date: Mon, 19 Oct 2020 11:22:10 +0800 [thread overview]
Message-ID: <20201019032210.GC54484@L-31X9LVDL-1304.local> (raw)
In-Reply-To: <20201012125323.17509-28-david@redhat.com>
On Mon, Oct 12, 2020 at 02:53:21PM +0200, David Hildenbrand wrote:
>virtio-mem soon wants to use offline_and_remove_memory() memory that
>exceeds a single Linux memory block (memory_block_size_bytes()). Let's
>remove that restriction.
>
>Let's remember the old state and try to restore that if anything goes
>wrong. While re-onlining can, in general, fail, it's highly unlikely to
>happen (usually only when a notifier fails to allocate memory, and these
>are rather rare).
>
>This will be used by virtio-mem to offline+remove memory ranges that are
>bigger than a single memory block - for example, with a device block
>size of 1 GiB (e.g., gigantic pages in the hypervisor) and a Linux memory
>block size of 128MB.
>
>While we could compress the state into 2 bit, using 8 bit is much
>easier.
>
>This handling is similar, but different to acpi_scan_try_to_offline():
>
>a) We don't try to offline twice. I am not sure if this CONFIG_MEMCG
>optimization is still relevant - it should only apply to ZONE_NORMAL
>(where we have no guarantees). If relevant, we can always add it.
>
>b) acpi_scan_try_to_offline() simply onlines all memory in case
>something goes wrong. It doesn't restore previous online type. Let's do
>that, so we won't overwrite what e.g., user space configured.
>
>Cc: "Michael S. Tsirkin" <mst@redhat.com>
>Cc: Jason Wang <jasowang@redhat.com>
>Cc: Pankaj Gupta <pankaj.gupta.linux@gmail.com>
>Cc: Michal Hocko <mhocko@kernel.org>
>Cc: Oscar Salvador <osalvador@suse.de>
>Cc: Wei Yang <richard.weiyang@linux.alibaba.com>
>Cc: Andrew Morton <akpm@linux-foundation.org>
>Signed-off-by: David Hildenbrand <david@redhat.com>
Looks good to me.
Reviewed-by: Wei Yang <richard.weiyang@linux.alibaba.com>
>---
> mm/memory_hotplug.c | 105 +++++++++++++++++++++++++++++++++++++-------
> 1 file changed, 89 insertions(+), 16 deletions(-)
>
>diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
>index b44d4c7ba73b..217080ca93e5 100644
>--- a/mm/memory_hotplug.c
>+++ b/mm/memory_hotplug.c
>@@ -1806,39 +1806,112 @@ int remove_memory(int nid, u64 start, u64 size)
> }
> EXPORT_SYMBOL_GPL(remove_memory);
>
>+static int try_offline_memory_block(struct memory_block *mem, void *arg)
>+{
>+ uint8_t online_type = MMOP_ONLINE_KERNEL;
>+ uint8_t **online_types = arg;
>+ struct page *page;
>+ int rc;
>+
>+ /*
>+ * Sense the online_type via the zone of the memory block. Offlining
>+ * with multiple zones within one memory block will be rejected
>+ * by offlining code ... so we don't care about that.
>+ */
>+ page = pfn_to_online_page(section_nr_to_pfn(mem->start_section_nr));
>+ if (page && zone_idx(page_zone(page)) == ZONE_MOVABLE)
>+ online_type = MMOP_ONLINE_MOVABLE;
>+
>+ rc = device_offline(&mem->dev);
>+ /*
>+ * Default is MMOP_OFFLINE - change it only if offlining succeeded,
>+ * so try_reonline_memory_block() can do the right thing.
>+ */
>+ if (!rc)
>+ **online_types = online_type;
>+
>+ (*online_types)++;
>+ /* Ignore if already offline. */
>+ return rc < 0 ? rc : 0;
>+}
>+
>+static int try_reonline_memory_block(struct memory_block *mem, void *arg)
>+{
>+ uint8_t **online_types = arg;
>+ int rc;
>+
>+ if (**online_types != MMOP_OFFLINE) {
>+ mem->online_type = **online_types;
>+ rc = device_online(&mem->dev);
>+ if (rc < 0)
>+ pr_warn("%s: Failed to re-online memory: %d",
>+ __func__, rc);
>+ }
>+
>+ /* Continue processing all remaining memory blocks. */
>+ (*online_types)++;
>+ return 0;
>+}
>+
> /*
>- * Try to offline and remove a memory block. Might take a long time to
>- * finish in case memory is still in use. Primarily useful for memory devices
>- * that logically unplugged all memory (so it's no longer in use) and want to
>- * offline + remove the memory block.
>+ * Try to offline and remove memory. Might take a long time to finish in case
>+ * memory is still in use. Primarily useful for memory devices that logically
>+ * unplugged all memory (so it's no longer in use) and want to offline + remove
>+ * that memory.
> */
> int offline_and_remove_memory(int nid, u64 start, u64 size)
> {
>- struct memory_block *mem;
>- int rc = -EINVAL;
>+ const unsigned long mb_count = size / memory_block_size_bytes();
>+ uint8_t *online_types, *tmp;
>+ int rc;
>
> if (!IS_ALIGNED(start, memory_block_size_bytes()) ||
>- size != memory_block_size_bytes())
>- return rc;
>+ !IS_ALIGNED(size, memory_block_size_bytes()) || !size)
>+ return -EINVAL;
>+
>+ /*
>+ * We'll remember the old online type of each memory block, so we can
>+ * try to revert whatever we did when offlining one memory block fails
>+ * after offlining some others succeeded.
>+ */
>+ online_types = kmalloc_array(mb_count, sizeof(*online_types),
>+ GFP_KERNEL);
>+ if (!online_types)
>+ return -ENOMEM;
>+ /*
>+ * Initialize all states to MMOP_OFFLINE, so when we abort processing in
>+ * try_offline_memory_block(), we'll skip all unprocessed blocks in
>+ * try_reonline_memory_block().
>+ */
>+ memset(online_types, MMOP_OFFLINE, mb_count);
>
> lock_device_hotplug();
>- mem = find_memory_block(__pfn_to_section(PFN_DOWN(start)));
>- if (mem)
>- rc = device_offline(&mem->dev);
>- /* Ignore if the device is already offline. */
>- if (rc > 0)
>- rc = 0;
>+
>+ tmp = online_types;
>+ rc = walk_memory_blocks(start, size, &tmp, try_offline_memory_block);
>
> /*
>- * In case we succeeded to offline the memory block, remove it.
>+ * In case we succeeded to offline all memory, remove it.
> * This cannot fail as it cannot get onlined in the meantime.
> */
> if (!rc) {
> rc = try_remove_memory(nid, start, size);
>- WARN_ON_ONCE(rc);
>+ if (rc)
>+ pr_err("%s: Failed to remove memory: %d", __func__, rc);
>+ }
>+
>+ /*
>+ * Rollback what we did. While memory onlining might theoretically fail
>+ * (nacked by a notifier), it barely ever happens.
>+ */
>+ if (rc) {
>+ tmp = online_types;
>+ walk_memory_blocks(start, size, &tmp,
>+ try_reonline_memory_block);
> }
> unlock_device_hotplug();
>
>+ kfree(online_types);
> return rc;
> }
> EXPORT_SYMBOL_GPL(offline_and_remove_memory);
>--
>2.26.2
--
Wei Yang
Help you, Help me
next prev parent reply other threads:[~2020-10-19 3:22 UTC|newest]
Thread overview: 109+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-10-12 12:52 [PATCH v1 00/29] virtio-mem: Big Block Mode (BBM) David Hildenbrand
2020-10-12 12:52 ` [PATCH v1 01/29] virtio-mem: determine nid only once using memory_add_physaddr_to_nid() David Hildenbrand
2020-10-15 3:56 ` Wei Yang
2020-10-15 19:26 ` Pankaj Gupta
2020-10-12 12:52 ` [PATCH v1 02/29] virtio-mem: simplify calculation in virtio_mem_mb_state_prepare_next_mb() David Hildenbrand
2020-10-15 4:02 ` Wei Yang
2020-10-15 8:00 ` David Hildenbrand
2020-10-15 10:00 ` Wei Yang
2020-10-15 10:01 ` David Hildenbrand
2020-10-15 20:24 ` Pankaj Gupta
2020-10-16 9:00 ` David Hildenbrand
2020-10-12 12:52 ` [PATCH v1 03/29] virtio-mem: simplify MAX_ORDER - 1 / pageblock_order handling David Hildenbrand
2020-10-15 7:06 ` Wei Yang
2020-10-12 12:52 ` [PATCH v1 04/29] virtio-mem: drop rc2 in virtio_mem_mb_plug_and_add() David Hildenbrand
2020-10-12 13:09 ` Pankaj Gupta
2020-10-15 7:14 ` Wei Yang
2020-10-12 12:52 ` [PATCH v1 05/29] virtio-mem: generalize check for added memory David Hildenbrand
2020-10-15 8:28 ` Wei Yang
2020-10-15 8:50 ` David Hildenbrand
2020-10-16 2:16 ` Wei Yang
2020-10-16 9:11 ` David Hildenbrand
2020-10-16 10:02 ` Wei Yang
2020-10-16 10:32 ` David Hildenbrand
2020-10-16 22:38 ` Wei Yang
2020-10-17 7:39 ` David Hildenbrand
2020-10-18 12:27 ` Wei Yang
2020-10-16 22:39 ` Wei Yang
2020-10-12 12:53 ` [PATCH v1 06/29] virtio-mem: generalize virtio_mem_owned_mb() David Hildenbrand
2020-10-15 8:32 ` Wei Yang
2020-10-15 8:37 ` David Hildenbrand
2020-10-15 20:30 ` Pankaj Gupta
2020-10-12 12:53 ` [PATCH v1 07/29] virtio-mem: generalize virtio_mem_overlaps_range() David Hildenbrand
2020-10-20 9:22 ` Pankaj Gupta
2020-10-12 12:53 ` [PATCH v1 08/29] virtio-mem: drop last_mb_id David Hildenbrand
2020-10-15 8:35 ` Wei Yang
2020-10-15 20:32 ` Pankaj Gupta
2020-10-12 12:53 ` [PATCH v1 09/29] virtio-mem: don't always trigger the workqueue when offlining memory David Hildenbrand
2020-10-16 4:03 ` Wei Yang
2020-10-16 9:18 ` David Hildenbrand
2020-10-18 3:57 ` Wei Yang
2020-10-19 9:04 ` David Hildenbrand
2020-10-20 0:41 ` Wei Yang
2020-10-20 9:09 ` David Hildenbrand
2020-10-12 12:53 ` [PATCH v1 10/29] virtio-mem: generalize handling when memory is getting onlined deferred David Hildenbrand
2020-10-12 12:53 ` [PATCH v1 11/29] virtio-mem: use "unsigned long" for nr_pages when fake onlining/offlining David Hildenbrand
2020-10-15 20:31 ` Pankaj Gupta
2020-10-16 6:11 ` Wei Yang
2020-10-12 12:53 ` [PATCH v1 12/29] virtio-mem: factor out fake-offlining into virtio_mem_fake_offline() David Hildenbrand
2020-10-16 6:24 ` Wei Yang
2020-10-20 9:31 ` Pankaj Gupta
2020-10-12 12:53 ` [PATCH v1 13/29] virtio-mem: factor out handling of fake-offline pages in memory notifier David Hildenbrand
2020-10-16 7:15 ` Wei Yang
2020-10-16 8:00 ` Wei Yang
2020-10-16 8:57 ` David Hildenbrand
2020-10-18 12:37 ` Wei Yang
2020-10-18 12:38 ` Wei Yang
2020-10-12 12:53 ` [PATCH v1 14/29] virtio-mem: retry fake-offlining via alloc_contig_range() on ZONE_MOVABLE David Hildenbrand
2020-10-12 12:53 ` [PATCH v1 15/29] virito-mem: document Sub Block Mode (SBM) David Hildenbrand
2020-10-15 9:33 ` David Hildenbrand
2020-10-20 9:38 ` Pankaj Gupta
2020-10-16 8:03 ` Wei Yang
2020-10-12 12:53 ` [PATCH v1 16/29] virtio-mem: memory block states are specific to " David Hildenbrand
2020-10-16 8:40 ` Wei Yang
2020-10-16 8:43 ` Wei Yang
2020-10-20 9:48 ` Pankaj Gupta
2020-10-12 12:53 ` [PATCH v1 17/29] virito-mem: subblock " David Hildenbrand
2020-10-16 8:43 ` Wei Yang
2020-10-20 9:54 ` Pankaj Gupta
2020-10-12 12:53 ` [PATCH v1 18/29] virtio-mem: factor out calculation of the bit number within the sb_states bitmap David Hildenbrand
2020-10-16 8:46 ` Wei Yang
2020-10-20 9:58 ` Pankaj Gupta
2020-10-12 12:53 ` [PATCH v1 19/29] virito-mem: existing (un)plug functions are specific to Sub Block Mode (SBM) David Hildenbrand
2020-10-16 8:49 ` Wei Yang
2020-10-12 12:53 ` [PATCH v1 20/29] virtio-mem: nb_sb_per_mb and subblock_size " David Hildenbrand
2020-10-16 8:51 ` Wei Yang
2020-10-16 8:53 ` Wei Yang
2020-10-16 13:17 ` David Hildenbrand
2020-10-18 12:41 ` Wei Yang
2020-10-19 11:57 ` David Hildenbrand
2020-10-12 12:53 ` [PATCH v1 21/29] virtio-mem: memory notifier callbacks " David Hildenbrand
2020-10-19 1:57 ` Wei Yang
2020-10-19 10:22 ` David Hildenbrand
2020-10-12 12:53 ` [PATCH v1 22/29] virtio-mem: memory block ids " David Hildenbrand
2020-10-16 8:54 ` Wei Yang
2020-10-12 12:53 ` [PATCH v1 23/29] virtio-mem: factor out adding/removing memory from Linux David Hildenbrand
2020-10-16 8:59 ` Wei Yang
2020-10-12 12:53 ` [PATCH v1 24/29] virtio-mem: print debug messages from virtio_mem_send_*_request() David Hildenbrand
2020-10-16 9:07 ` Wei Yang
2020-10-12 12:53 ` [PATCH v1 25/29] virtio-mem: Big Block Mode (BBM) memory hotplug David Hildenbrand
2020-10-16 9:38 ` Wei Yang
2020-10-16 13:13 ` David Hildenbrand
2020-10-19 2:26 ` Wei Yang
2020-10-19 9:15 ` David Hildenbrand
2020-10-12 12:53 ` [PATCH v1 26/29] virtio-mem: allow to force Big Block Mode (BBM) and set the big block size David Hildenbrand
2020-10-12 12:53 ` [PATCH v1 27/29] mm/memory_hotplug: extend offline_and_remove_memory() to handle more than one memory block David Hildenbrand
2020-10-15 13:08 ` Michael S. Tsirkin
2020-10-19 3:22 ` Wei Yang [this message]
2020-10-12 12:53 ` [PATCH v1 28/29] virtio-mem: Big Block Mode (BBM) - basic memory hotunplug David Hildenbrand
2020-10-19 3:48 ` Wei Yang
2020-10-19 9:12 ` David Hildenbrand
2020-10-12 12:53 ` [PATCH v1 29/29] virtio-mem: Big Block Mode (BBM) - safe " David Hildenbrand
2020-10-19 7:54 ` Wei Yang
2020-10-19 8:50 ` David Hildenbrand
2020-10-20 0:23 ` Wei Yang
2020-10-20 0:24 ` Wei Yang
2020-10-18 12:49 ` [PATCH v1 00/29] virtio-mem: Big Block Mode (BBM) Wei Yang
2020-10-18 16:35 ` David Hildenbrand
2020-10-18 15:29 ` Michael S. Tsirkin
2020-10-18 16:34 ` David Hildenbrand
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20201019032210.GC54484@L-31X9LVDL-1304.local \
--to=richard.weiyang@linux.alibaba.com \
--cc=akpm@linux-foundation.org \
--cc=david@redhat.com \
--cc=jasowang@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mhocko@kernel.org \
--cc=mst@redhat.com \
--cc=osalvador@suse.de \
--cc=pankaj.gupta.linux@gmail.com \
--cc=virtualization@lists.linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).