From: David Hildenbrand <david@redhat.com> To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, virtio-dev@lists.oasis-open.org, virtualization@lists.linux-foundation.org, kvm@vger.kernel.org, Michal Hocko <mhocko@kernel.org>, Andrew Morton <akpm@linux-foundation.org>, "Michael S . Tsirkin" <mst@redhat.com>, David Hildenbrand <david@redhat.com>, Jason Wang <jasowang@redhat.com>, Oscar Salvador <osalvador@suse.de>, Igor Mammedov <imammedo@redhat.com>, Dave Young <dyoung@redhat.com>, Dan Williams <dan.j.williams@intel.com>, Pavel Tatashin <pasha.tatashin@soleen.com>, Stefan Hajnoczi <stefanha@redhat.com>, Vlastimil Babka <vbabka@suse.cz> Subject: [PATCH v2 08/10] virtio-mem: Offline and remove completely unplugged memory blocks Date: Wed, 11 Mar 2020 18:14:20 +0100 [thread overview] Message-ID: <20200311171422.10484-9-david@redhat.com> (raw) In-Reply-To: <20200311171422.10484-1-david@redhat.com> Let's offline+remove memory blocks once all subblocks are unplugged. We can use the new Linux MM interface for that. As no memory is in use anymore, this shouldn't take a long time and shouldn't fail. There might be corner cases where the offlining could still fail (especially, if another notifier NACKs the offlining request). Cc: "Michael S. Tsirkin" <mst@redhat.com> Cc: Jason Wang <jasowang@redhat.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Michal Hocko <mhocko@kernel.org> Cc: Igor Mammedov <imammedo@redhat.com> Cc: Dave Young <dyoung@redhat.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Dan Williams <dan.j.williams@intel.com> Cc: Pavel Tatashin <pasha.tatashin@soleen.com> Cc: Stefan Hajnoczi <stefanha@redhat.com> Cc: Vlastimil Babka <vbabka@suse.cz> Signed-off-by: David Hildenbrand <david@redhat.com> --- drivers/virtio/virtio_mem.c | 47 +++++++++++++++++++++++++++++++++---- 1 file changed, 43 insertions(+), 4 deletions(-) diff --git a/drivers/virtio/virtio_mem.c b/drivers/virtio/virtio_mem.c index 35f20232770c..aa322e7732a4 100644 --- a/drivers/virtio/virtio_mem.c +++ b/drivers/virtio/virtio_mem.c @@ -443,6 +443,28 @@ static int virtio_mem_mb_remove(struct virtio_mem *vm, unsigned long mb_id) return remove_memory(nid, addr, memory_block_size_bytes()); } +/* + * Try to offline and remove a memory block from Linux. + * + * Must not be called with the vm->hotplug_mutex held (possible deadlock with + * onlining code). + * + * Will not modify the state of the memory block. + */ +static int virtio_mem_mb_offline_and_remove(struct virtio_mem *vm, + unsigned long mb_id) +{ + const uint64_t addr = virtio_mem_mb_id_to_phys(mb_id); + int nid = vm->nid; + + if (nid == NUMA_NO_NODE) + nid = memory_add_physaddr_to_nid(addr); + + dev_dbg(&vm->vdev->dev, "offlining and removing memory block: %lu\n", + mb_id); + return offline_and_remove_memory(nid, addr, memory_block_size_bytes()); +} + /* * Trigger the workqueue so the device can perform its magic. */ @@ -535,7 +557,13 @@ static void virtio_mem_notify_offline(struct virtio_mem *vm, break; } - /* trigger the workqueue, maybe we can now unplug memory. */ + /* + * Trigger the workqueue, maybe we can now unplug memory. Also, + * when we offline and remove a memory block, this will re-trigger + * us immediately - which is often nice because the removal of + * the memory block (e.g., memmap) might have freed up memory + * on other memory blocks we manage. + */ virtio_mem_retry(vm); } @@ -1282,7 +1310,8 @@ static int virtio_mem_mb_unplug_any_sb_offline(struct virtio_mem *vm, * Unplug the desired number of plugged subblocks of an online memory block. * Will skip subblock that are busy. * - * Will modify the state of the memory block. + * Will modify the state of the memory block. Might temporarily drop the + * hotplug_mutex. * * Note: Can fail after some subblocks were successfully unplugged. Can * return 0 even if subblocks were busy and could not get unplugged. @@ -1338,9 +1367,19 @@ static int virtio_mem_mb_unplug_any_sb_online(struct virtio_mem *vm, } /* - * TODO: Once all subblocks of a memory block were unplugged, we want - * to offline the memory block and remove it. + * Once all subblocks of a memory block were unplugged, offline and + * remove it. This will usually not fail, as no memory is in use + * anymore - however some other notifiers might NACK the request. */ + if (virtio_mem_mb_test_sb_unplugged(vm, mb_id, 0, vm->nb_sb_per_mb)) { + mutex_unlock(&vm->hotplug_mutex); + rc = virtio_mem_mb_offline_and_remove(vm, mb_id); + mutex_lock(&vm->hotplug_mutex); + if (!rc) + virtio_mem_mb_set_state(vm, mb_id, + VIRTIO_MEM_MB_STATE_UNUSED); + } + return 0; } -- 2.24.1
WARNING: multiple messages have this Message-ID (diff)
From: David Hildenbrand <david@redhat.com> To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, virtio-dev@lists.oasis-open.org, virtualization@lists.linux-foundation.org, kvm@vger.kernel.org, Michal Hocko <mhocko@kernel.org>, Andrew Morton <akpm@linux-foundation.org>, "Michael S . Tsirkin" <mst@redhat.com>, David Hildenbrand <david@redhat.com>, Jason Wang <jasowang@redhat.com>, Oscar Salvador <osalvador@suse.de>, Igor Mammedov <imammedo@redhat.com>, Dave Young <dyoung@redhat.com>, Dan Williams <dan.j.williams@intel.com>, Pavel Tatashin <pasha.tatashin@soleen.com>, Stefan Hajnoczi <stefanha@redhat.com>, Vlastimil Babka <vbabka@suse.cz> Subject: [virtio-dev] [PATCH v2 08/10] virtio-mem: Offline and remove completely unplugged memory blocks Date: Wed, 11 Mar 2020 18:14:20 +0100 [thread overview] Message-ID: <20200311171422.10484-9-david@redhat.com> (raw) In-Reply-To: <20200311171422.10484-1-david@redhat.com> Let's offline+remove memory blocks once all subblocks are unplugged. We can use the new Linux MM interface for that. As no memory is in use anymore, this shouldn't take a long time and shouldn't fail. There might be corner cases where the offlining could still fail (especially, if another notifier NACKs the offlining request). Cc: "Michael S. Tsirkin" <mst@redhat.com> Cc: Jason Wang <jasowang@redhat.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Michal Hocko <mhocko@kernel.org> Cc: Igor Mammedov <imammedo@redhat.com> Cc: Dave Young <dyoung@redhat.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Dan Williams <dan.j.williams@intel.com> Cc: Pavel Tatashin <pasha.tatashin@soleen.com> Cc: Stefan Hajnoczi <stefanha@redhat.com> Cc: Vlastimil Babka <vbabka@suse.cz> Signed-off-by: David Hildenbrand <david@redhat.com> --- drivers/virtio/virtio_mem.c | 47 +++++++++++++++++++++++++++++++++---- 1 file changed, 43 insertions(+), 4 deletions(-) diff --git a/drivers/virtio/virtio_mem.c b/drivers/virtio/virtio_mem.c index 35f20232770c..aa322e7732a4 100644 --- a/drivers/virtio/virtio_mem.c +++ b/drivers/virtio/virtio_mem.c @@ -443,6 +443,28 @@ static int virtio_mem_mb_remove(struct virtio_mem *vm, unsigned long mb_id) return remove_memory(nid, addr, memory_block_size_bytes()); } +/* + * Try to offline and remove a memory block from Linux. + * + * Must not be called with the vm->hotplug_mutex held (possible deadlock with + * onlining code). + * + * Will not modify the state of the memory block. + */ +static int virtio_mem_mb_offline_and_remove(struct virtio_mem *vm, + unsigned long mb_id) +{ + const uint64_t addr = virtio_mem_mb_id_to_phys(mb_id); + int nid = vm->nid; + + if (nid == NUMA_NO_NODE) + nid = memory_add_physaddr_to_nid(addr); + + dev_dbg(&vm->vdev->dev, "offlining and removing memory block: %lu\n", + mb_id); + return offline_and_remove_memory(nid, addr, memory_block_size_bytes()); +} + /* * Trigger the workqueue so the device can perform its magic. */ @@ -535,7 +557,13 @@ static void virtio_mem_notify_offline(struct virtio_mem *vm, break; } - /* trigger the workqueue, maybe we can now unplug memory. */ + /* + * Trigger the workqueue, maybe we can now unplug memory. Also, + * when we offline and remove a memory block, this will re-trigger + * us immediately - which is often nice because the removal of + * the memory block (e.g., memmap) might have freed up memory + * on other memory blocks we manage. + */ virtio_mem_retry(vm); } @@ -1282,7 +1310,8 @@ static int virtio_mem_mb_unplug_any_sb_offline(struct virtio_mem *vm, * Unplug the desired number of plugged subblocks of an online memory block. * Will skip subblock that are busy. * - * Will modify the state of the memory block. + * Will modify the state of the memory block. Might temporarily drop the + * hotplug_mutex. * * Note: Can fail after some subblocks were successfully unplugged. Can * return 0 even if subblocks were busy and could not get unplugged. @@ -1338,9 +1367,19 @@ static int virtio_mem_mb_unplug_any_sb_online(struct virtio_mem *vm, } /* - * TODO: Once all subblocks of a memory block were unplugged, we want - * to offline the memory block and remove it. + * Once all subblocks of a memory block were unplugged, offline and + * remove it. This will usually not fail, as no memory is in use + * anymore - however some other notifiers might NACK the request. */ + if (virtio_mem_mb_test_sb_unplugged(vm, mb_id, 0, vm->nb_sb_per_mb)) { + mutex_unlock(&vm->hotplug_mutex); + rc = virtio_mem_mb_offline_and_remove(vm, mb_id); + mutex_lock(&vm->hotplug_mutex); + if (!rc) + virtio_mem_mb_set_state(vm, mb_id, + VIRTIO_MEM_MB_STATE_UNUSED); + } + return 0; } -- 2.24.1 --------------------------------------------------------------------- To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
next prev parent reply other threads:[~2020-03-11 17:16 UTC|newest] Thread overview: 71+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-03-11 17:14 [PATCH v2 00/10] virtio-mem: paravirtualized memory David Hildenbrand 2020-03-11 17:14 ` [virtio-dev] " David Hildenbrand 2020-03-11 17:14 ` David Hildenbrand 2020-03-11 17:14 ` [PATCH v2 01/10] virtio-mem: Paravirtualized memory hotplug David Hildenbrand 2020-03-11 17:14 ` [virtio-dev] " David Hildenbrand 2020-04-15 5:59 ` Pankaj Gupta 2020-04-15 5:59 ` Pankaj Gupta 2020-03-11 17:14 ` [PATCH v2 02/10] virtio-mem: Allow to specify an ACPI PXM as nid David Hildenbrand 2020-03-11 17:14 ` [virtio-dev] " David Hildenbrand 2020-03-30 11:04 ` Pankaj Gupta 2020-03-30 11:04 ` Pankaj Gupta 2020-03-11 17:14 ` [PATCH v2 03/10] virtio-mem: Paravirtualized memory hotunplug part 1 David Hildenbrand 2020-03-11 17:14 ` [virtio-dev] " David Hildenbrand 2020-03-11 17:14 ` [PATCH v2 04/10] virtio-mem: Paravirtualized memory hotunplug part 2 David Hildenbrand 2020-03-11 17:14 ` [virtio-dev] " David Hildenbrand 2020-03-11 17:14 ` [PATCH v2 05/10] mm: Allow to offline unmovable PageOffline() pages via MEM_GOING_OFFLINE David Hildenbrand 2020-03-11 17:14 ` [virtio-dev] " David Hildenbrand 2020-04-14 16:34 ` Michael S. Tsirkin 2020-04-14 16:34 ` [virtio-dev] " Michael S. Tsirkin 2020-04-14 16:34 ` Michael S. Tsirkin 2020-04-15 0:32 ` Andrew Morton 2020-04-15 0:32 ` Andrew Morton 2020-03-11 17:14 ` [PATCH v2 06/10] virtio-mem: Allow to offline partially unplugged memory blocks David Hildenbrand 2020-03-11 17:14 ` [virtio-dev] " David Hildenbrand 2020-03-11 17:14 ` [PATCH v2 07/10] mm/memory_hotplug: Introduce offline_and_remove_memory() David Hildenbrand 2020-03-11 17:14 ` [virtio-dev] " David Hildenbrand 2020-03-11 17:19 ` David Hildenbrand 2020-03-11 17:19 ` [virtio-dev] " David Hildenbrand 2020-04-14 16:35 ` Michael S. Tsirkin 2020-04-14 16:35 ` [virtio-dev] " Michael S. Tsirkin 2020-04-15 0:30 ` Andrew Morton 2020-04-15 7:35 ` Pankaj Gupta 2020-04-15 7:35 ` Pankaj Gupta 2020-03-11 17:14 ` David Hildenbrand [this message] 2020-03-11 17:14 ` [virtio-dev] [PATCH v2 08/10] virtio-mem: Offline and remove completely unplugged memory blocks David Hildenbrand 2020-04-15 7:42 ` Pankaj Gupta 2020-04-15 7:42 ` Pankaj Gupta 2020-03-11 17:14 ` [PATCH v2 09/10] virtio-mem: Better retry handling David Hildenbrand 2020-03-11 17:14 ` [virtio-dev] " David Hildenbrand 2020-03-11 17:14 ` [PATCH v2 10/10] MAINTAINERS: Add myself as virtio-mem maintainer David Hildenbrand 2020-03-11 17:14 ` [virtio-dev] " David Hildenbrand 2020-03-27 16:58 ` [PATCH v2 00/10] virtio-mem: paravirtualized memory Pankaj Gupta 2020-03-27 16:58 ` Pankaj Gupta 2020-03-27 16:58 ` Pankaj Gupta 2020-03-27 17:03 ` David Hildenbrand 2020-03-27 17:03 ` [virtio-dev] " David Hildenbrand 2020-03-27 17:03 ` David Hildenbrand 2020-03-27 17:07 ` David Hildenbrand 2020-03-27 17:07 ` [virtio-dev] " David Hildenbrand 2020-03-27 17:07 ` David Hildenbrand 2020-03-29 15:41 ` Pankaj Gupta 2020-03-29 15:41 ` Pankaj Gupta 2020-03-29 15:41 ` Pankaj Gupta 2020-03-30 8:42 ` David Hildenbrand 2020-03-30 8:42 ` [virtio-dev] " David Hildenbrand 2020-03-30 8:42 ` David Hildenbrand 2020-03-29 12:42 ` Michael S. Tsirkin 2020-03-29 12:42 ` [virtio-dev] " Michael S. Tsirkin 2020-03-29 12:42 ` Michael S. Tsirkin 2020-03-30 8:16 ` David Hildenbrand 2020-03-30 8:16 ` [virtio-dev] " David Hildenbrand 2020-03-30 8:16 ` David Hildenbrand 2020-04-14 9:15 ` David Hildenbrand 2020-04-14 9:15 ` [virtio-dev] " David Hildenbrand 2020-04-14 9:15 ` David Hildenbrand 2020-04-14 16:28 ` Michael S. Tsirkin 2020-04-14 16:28 ` [virtio-dev] " Michael S. Tsirkin 2020-04-14 16:28 ` Michael S. Tsirkin 2020-04-14 18:39 ` David Hildenbrand 2020-04-14 18:39 ` [virtio-dev] " David Hildenbrand 2020-04-14 18:39 ` David Hildenbrand
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20200311171422.10484-9-david@redhat.com \ --to=david@redhat.com \ --cc=akpm@linux-foundation.org \ --cc=dan.j.williams@intel.com \ --cc=dyoung@redhat.com \ --cc=imammedo@redhat.com \ --cc=jasowang@redhat.com \ --cc=kvm@vger.kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=mhocko@kernel.org \ --cc=mst@redhat.com \ --cc=osalvador@suse.de \ --cc=pasha.tatashin@soleen.com \ --cc=stefanha@redhat.com \ --cc=vbabka@suse.cz \ --cc=virtio-dev@lists.oasis-open.org \ --cc=virtualization@lists.linux-foundation.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.