From: David Hildenbrand <david@redhat.com> To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, virtio-dev@lists.oasis-open.org, virtualization@lists.linux-foundation.org, kvm@vger.kernel.org, Michal Hocko <mhocko@kernel.org>, Andrew Morton <akpm@linux-foundation.org>, "Michael S . Tsirkin" <mst@redhat.com>, David Hildenbrand <david@redhat.com>, Pankaj Gupta <pankaj.gupta.linux@gmail.com>, Jason Wang <jasowang@redhat.com>, Oscar Salvador <osalvador@suse.de>, Igor Mammedov <imammedo@redhat.com>, Dave Young <dyoung@redhat.com>, Dan Williams <dan.j.williams@intel.com>, Pavel Tatashin <pasha.tatashin@soleen.com>, Stefan Hajnoczi <stefanha@redhat.com>, Vlastimil Babka <vbabka@suse.cz> Subject: [PATCH v4 09/15] virtio-mem: Offline and remove completely unplugged memory blocks Date: Thu, 7 May 2020 16:01:33 +0200 [thread overview] Message-ID: <20200507140139.17083-10-david@redhat.com> (raw) In-Reply-To: <20200507140139.17083-1-david@redhat.com> Let's offline+remove memory blocks once all subblocks are unplugged. We can use the new Linux MM interface for that. As no memory is in use anymore, this shouldn't take a long time and shouldn't fail. There might be corner cases where the offlining could still fail (especially, if another notifier NACKs the offlining request). Acked-by: Pankaj Gupta <pankaj.gupta.linux@gmail.com> Tested-by: Pankaj Gupta <pankaj.gupta.linux@gmail.com> Cc: "Michael S. Tsirkin" <mst@redhat.com> Cc: Jason Wang <jasowang@redhat.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Michal Hocko <mhocko@kernel.org> Cc: Igor Mammedov <imammedo@redhat.com> Cc: Dave Young <dyoung@redhat.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Dan Williams <dan.j.williams@intel.com> Cc: Pavel Tatashin <pasha.tatashin@soleen.com> Cc: Stefan Hajnoczi <stefanha@redhat.com> Cc: Vlastimil Babka <vbabka@suse.cz> Signed-off-by: David Hildenbrand <david@redhat.com> --- drivers/virtio/virtio_mem.c | 47 +++++++++++++++++++++++++++++++++---- 1 file changed, 43 insertions(+), 4 deletions(-) diff --git a/drivers/virtio/virtio_mem.c b/drivers/virtio/virtio_mem.c index b0b41c73ce89..a2edb87e5ed8 100644 --- a/drivers/virtio/virtio_mem.c +++ b/drivers/virtio/virtio_mem.c @@ -446,6 +446,28 @@ static int virtio_mem_mb_remove(struct virtio_mem *vm, unsigned long mb_id) return remove_memory(nid, addr, memory_block_size_bytes()); } +/* + * Try to offline and remove a memory block from Linux. + * + * Must not be called with the vm->hotplug_mutex held (possible deadlock with + * onlining code). + * + * Will not modify the state of the memory block. + */ +static int virtio_mem_mb_offline_and_remove(struct virtio_mem *vm, + unsigned long mb_id) +{ + const uint64_t addr = virtio_mem_mb_id_to_phys(mb_id); + int nid = vm->nid; + + if (nid == NUMA_NO_NODE) + nid = memory_add_physaddr_to_nid(addr); + + dev_dbg(&vm->vdev->dev, "offlining and removing memory block: %lu\n", + mb_id); + return offline_and_remove_memory(nid, addr, memory_block_size_bytes()); +} + /* * Trigger the workqueue so the device can perform its magic. */ @@ -537,7 +559,13 @@ static void virtio_mem_notify_offline(struct virtio_mem *vm, break; } - /* trigger the workqueue, maybe we can now unplug memory. */ + /* + * Trigger the workqueue, maybe we can now unplug memory. Also, + * when we offline and remove a memory block, this will re-trigger + * us immediately - which is often nice because the removal of + * the memory block (e.g., memmap) might have freed up memory + * on other memory blocks we manage. + */ virtio_mem_retry(vm); } @@ -1284,7 +1312,8 @@ static int virtio_mem_mb_unplug_any_sb_offline(struct virtio_mem *vm, * Unplug the desired number of plugged subblocks of an online memory block. * Will skip subblock that are busy. * - * Will modify the state of the memory block. + * Will modify the state of the memory block. Might temporarily drop the + * hotplug_mutex. * * Note: Can fail after some subblocks were successfully unplugged. Can * return 0 even if subblocks were busy and could not get unplugged. @@ -1340,9 +1369,19 @@ static int virtio_mem_mb_unplug_any_sb_online(struct virtio_mem *vm, } /* - * TODO: Once all subblocks of a memory block were unplugged, we want - * to offline the memory block and remove it. + * Once all subblocks of a memory block were unplugged, offline and + * remove it. This will usually not fail, as no memory is in use + * anymore - however some other notifiers might NACK the request. */ + if (virtio_mem_mb_test_sb_unplugged(vm, mb_id, 0, vm->nb_sb_per_mb)) { + mutex_unlock(&vm->hotplug_mutex); + rc = virtio_mem_mb_offline_and_remove(vm, mb_id); + mutex_lock(&vm->hotplug_mutex); + if (!rc) + virtio_mem_mb_set_state(vm, mb_id, + VIRTIO_MEM_MB_STATE_UNUSED); + } + return 0; } -- 2.25.3
WARNING: multiple messages have this Message-ID (diff)
From: David Hildenbrand <david@redhat.com> To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, virtio-dev@lists.oasis-open.org, virtualization@lists.linux-foundation.org, kvm@vger.kernel.org, Michal Hocko <mhocko@kernel.org>, Andrew Morton <akpm@linux-foundation.org>, "Michael S . Tsirkin" <mst@redhat.com>, David Hildenbrand <david@redhat.com>, Pankaj Gupta <pankaj.gupta.linux@gmail.com>, Jason Wang <jasowang@redhat.com>, Oscar Salvador <osalvador@suse.de>, Igor Mammedov <imammedo@redhat.com>, Dave Young <dyoung@redhat.com>, Dan Williams <dan.j.williams@intel.com>, Pavel Tatashin <pasha.tatashin@soleen.com>, Stefan Hajnoczi <stefanha@redhat.com>, Vlastimil Babka <vbabka@suse.cz> Subject: [virtio-dev] [PATCH v4 09/15] virtio-mem: Offline and remove completely unplugged memory blocks Date: Thu, 7 May 2020 16:01:33 +0200 [thread overview] Message-ID: <20200507140139.17083-10-david@redhat.com> (raw) In-Reply-To: <20200507140139.17083-1-david@redhat.com> Let's offline+remove memory blocks once all subblocks are unplugged. We can use the new Linux MM interface for that. As no memory is in use anymore, this shouldn't take a long time and shouldn't fail. There might be corner cases where the offlining could still fail (especially, if another notifier NACKs the offlining request). Acked-by: Pankaj Gupta <pankaj.gupta.linux@gmail.com> Tested-by: Pankaj Gupta <pankaj.gupta.linux@gmail.com> Cc: "Michael S. Tsirkin" <mst@redhat.com> Cc: Jason Wang <jasowang@redhat.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Michal Hocko <mhocko@kernel.org> Cc: Igor Mammedov <imammedo@redhat.com> Cc: Dave Young <dyoung@redhat.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Dan Williams <dan.j.williams@intel.com> Cc: Pavel Tatashin <pasha.tatashin@soleen.com> Cc: Stefan Hajnoczi <stefanha@redhat.com> Cc: Vlastimil Babka <vbabka@suse.cz> Signed-off-by: David Hildenbrand <david@redhat.com> --- drivers/virtio/virtio_mem.c | 47 +++++++++++++++++++++++++++++++++---- 1 file changed, 43 insertions(+), 4 deletions(-) diff --git a/drivers/virtio/virtio_mem.c b/drivers/virtio/virtio_mem.c index b0b41c73ce89..a2edb87e5ed8 100644 --- a/drivers/virtio/virtio_mem.c +++ b/drivers/virtio/virtio_mem.c @@ -446,6 +446,28 @@ static int virtio_mem_mb_remove(struct virtio_mem *vm, unsigned long mb_id) return remove_memory(nid, addr, memory_block_size_bytes()); } +/* + * Try to offline and remove a memory block from Linux. + * + * Must not be called with the vm->hotplug_mutex held (possible deadlock with + * onlining code). + * + * Will not modify the state of the memory block. + */ +static int virtio_mem_mb_offline_and_remove(struct virtio_mem *vm, + unsigned long mb_id) +{ + const uint64_t addr = virtio_mem_mb_id_to_phys(mb_id); + int nid = vm->nid; + + if (nid == NUMA_NO_NODE) + nid = memory_add_physaddr_to_nid(addr); + + dev_dbg(&vm->vdev->dev, "offlining and removing memory block: %lu\n", + mb_id); + return offline_and_remove_memory(nid, addr, memory_block_size_bytes()); +} + /* * Trigger the workqueue so the device can perform its magic. */ @@ -537,7 +559,13 @@ static void virtio_mem_notify_offline(struct virtio_mem *vm, break; } - /* trigger the workqueue, maybe we can now unplug memory. */ + /* + * Trigger the workqueue, maybe we can now unplug memory. Also, + * when we offline and remove a memory block, this will re-trigger + * us immediately - which is often nice because the removal of + * the memory block (e.g., memmap) might have freed up memory + * on other memory blocks we manage. + */ virtio_mem_retry(vm); } @@ -1284,7 +1312,8 @@ static int virtio_mem_mb_unplug_any_sb_offline(struct virtio_mem *vm, * Unplug the desired number of plugged subblocks of an online memory block. * Will skip subblock that are busy. * - * Will modify the state of the memory block. + * Will modify the state of the memory block. Might temporarily drop the + * hotplug_mutex. * * Note: Can fail after some subblocks were successfully unplugged. Can * return 0 even if subblocks were busy and could not get unplugged. @@ -1340,9 +1369,19 @@ static int virtio_mem_mb_unplug_any_sb_online(struct virtio_mem *vm, } /* - * TODO: Once all subblocks of a memory block were unplugged, we want - * to offline the memory block and remove it. + * Once all subblocks of a memory block were unplugged, offline and + * remove it. This will usually not fail, as no memory is in use + * anymore - however some other notifiers might NACK the request. */ + if (virtio_mem_mb_test_sb_unplugged(vm, mb_id, 0, vm->nb_sb_per_mb)) { + mutex_unlock(&vm->hotplug_mutex); + rc = virtio_mem_mb_offline_and_remove(vm, mb_id); + mutex_lock(&vm->hotplug_mutex); + if (!rc) + virtio_mem_mb_set_state(vm, mb_id, + VIRTIO_MEM_MB_STATE_UNUSED); + } + return 0; } -- 2.25.3 --------------------------------------------------------------------- To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
next prev parent reply other threads:[~2020-05-07 14:03 UTC|newest] Thread overview: 50+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-05-07 14:01 [PATCH v4 00/15] virtio-mem: paravirtualized memory David Hildenbrand 2020-05-07 14:01 ` [virtio-dev] " David Hildenbrand 2020-05-07 14:01 ` David Hildenbrand 2020-05-07 14:01 ` [PATCH v4 01/15] virtio-mem: Paravirtualized memory hotplug David Hildenbrand 2020-05-07 14:01 ` [virtio-dev] " David Hildenbrand 2020-05-07 14:01 ` [PATCH v4 02/15] MAINTAINERS: Add myself as virtio-mem maintainer David Hildenbrand 2020-05-07 14:01 ` [virtio-dev] " David Hildenbrand 2020-05-07 14:01 ` [PATCH v4 03/15] virtio-mem: Allow to specify an ACPI PXM as nid David Hildenbrand 2020-05-07 14:01 ` [virtio-dev] " David Hildenbrand 2020-05-07 14:01 ` [PATCH v4 04/15] virtio-mem: Paravirtualized memory hotunplug part 1 David Hildenbrand 2020-05-07 14:01 ` [virtio-dev] " David Hildenbrand 2020-05-07 14:01 ` David Hildenbrand 2020-05-07 14:01 ` [PATCH v4 05/15] virtio-mem: Paravirtualized memory hotunplug part 2 David Hildenbrand 2020-05-07 14:01 ` [virtio-dev] " David Hildenbrand 2020-05-07 14:01 ` David Hildenbrand 2020-05-07 14:01 ` [PATCH v4 06/15] mm: Allow to offline unmovable PageOffline() pages via MEM_GOING_OFFLINE David Hildenbrand 2020-05-07 14:01 ` [virtio-dev] " David Hildenbrand 2020-05-07 14:01 ` David Hildenbrand 2020-05-07 14:01 ` [PATCH v4 07/15] virtio-mem: Allow to offline partially unplugged memory blocks David Hildenbrand 2020-05-07 14:01 ` [virtio-dev] " David Hildenbrand 2020-05-07 14:01 ` [PATCH v4 08/15] mm/memory_hotplug: Introduce offline_and_remove_memory() David Hildenbrand 2020-05-07 14:01 ` [virtio-dev] " David Hildenbrand 2020-05-07 14:01 ` David Hildenbrand [this message] 2020-05-07 14:01 ` [virtio-dev] [PATCH v4 09/15] virtio-mem: Offline and remove completely unplugged memory blocks David Hildenbrand 2020-05-07 14:01 ` [PATCH v4 10/15] virtio-mem: Better retry handling David Hildenbrand 2020-05-07 14:01 ` [virtio-dev] " David Hildenbrand 2020-05-07 14:01 ` [PATCH v4 11/15] virtio-mem: Add parent resource for all added "System RAM" David Hildenbrand 2020-05-07 14:01 ` [virtio-dev] " David Hildenbrand 2020-05-07 14:06 ` Pankaj Gupta 2020-05-07 14:06 ` Pankaj Gupta 2020-05-07 14:01 ` [PATCH v4 12/15] virtio-mem: Drop manual check for already present memory David Hildenbrand 2020-05-07 14:01 ` [virtio-dev] " David Hildenbrand 2020-05-07 14:01 ` [PATCH v4 13/15] virtio-mem: Unplug subblocks right-to-left David Hildenbrand 2020-05-07 14:01 ` [virtio-dev] " David Hildenbrand 2020-05-07 14:01 ` [PATCH v4 14/15] virtio-mem: Use -ETXTBSY as error code if the device is busy David Hildenbrand 2020-05-07 14:01 ` [virtio-dev] " David Hildenbrand 2020-05-07 14:01 ` David Hildenbrand 2020-05-07 14:01 ` [PATCH v4 15/15] virtio-mem: Try to unplug the complete online memory block first David Hildenbrand 2020-05-07 14:01 ` [virtio-dev] " David Hildenbrand 2020-05-15 10:14 ` [PATCH v4 16/15] virtio-mem: Don't rely on implicit compiler padding for requests David Hildenbrand 2020-05-15 10:14 ` [virtio-dev] " David Hildenbrand 2020-05-20 5:25 ` [PATCH v4 00/15] virtio-mem: paravirtualized memory teawater 2020-05-20 5:25 ` [virtio-dev] " teawater 2020-05-20 5:25 ` teawater 2020-05-20 7:56 ` David Hildenbrand 2020-05-20 7:56 ` [virtio-dev] " David Hildenbrand 2020-05-20 7:56 ` David Hildenbrand 2020-06-02 7:09 ` David Hildenbrand 2020-06-02 7:09 ` [virtio-dev] " David Hildenbrand 2020-06-02 7:09 ` David Hildenbrand
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20200507140139.17083-10-david@redhat.com \ --to=david@redhat.com \ --cc=akpm@linux-foundation.org \ --cc=dan.j.williams@intel.com \ --cc=dyoung@redhat.com \ --cc=imammedo@redhat.com \ --cc=jasowang@redhat.com \ --cc=kvm@vger.kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=mhocko@kernel.org \ --cc=mst@redhat.com \ --cc=osalvador@suse.de \ --cc=pankaj.gupta.linux@gmail.com \ --cc=pasha.tatashin@soleen.com \ --cc=stefanha@redhat.com \ --cc=vbabka@suse.cz \ --cc=virtio-dev@lists.oasis-open.org \ --cc=virtualization@lists.linux-foundation.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.