On Wed, Feb 5, 2020 at 2:52 PM David Hildenbrand wrote: > > > Am 05.02.2020 um 23:37 schrieb Tyler Sanderson : > >  > > > On Wed, Feb 5, 2020 at 8:34 AM David Hildenbrand wrote: > >> Commit 71994620bb25 ("virtio_balloon: replace oom notifier with shrinker") >> changed the behavior when deflation happens automatically. Instead of >> deflating when called by the OOM handler, the shrinker is used. >> >> However, the balloon is not simply some slab cache that should be >> shrunk when under memory pressure. The shrinker does not have a concept of >> priorities, so this behavior cannot be configured. >> >> There was a report that this results in undesired side effects when >> inflating the balloon to shrink the page cache. [1] >> "When inflating the balloon against page cache (i.e. no free >> memory >> remains) vmscan.c will both shrink page cache, but also invoke >> the >> shrinkers -- including the balloon's shrinker. So the balloon >> driver allocates memory which requires reclaim, vmscan gets this >> memory by shrinking the balloon, and then the driver adds the >> memory back to the balloon. Basically a busy no-op." >> >> The name "deflate on OOM" makes it pretty clear when deflation should >> happen - after other approaches to reclaim memory failed, not while >> reclaiming. This allows to minimize the footprint of a guest - memory >> will only be taken out of the balloon when really needed. >> >> Especially, a drop_slab() will result in the whole balloon getting >> deflated - undesired. While handling it via the OOM handler might not be >> perfect, it keeps existing behavior. If we want a different behavior, then >> we need a new feature bit and document it properly (although, there should >> be a clear use case and the intended effects should be well described). >> >> Keep using the shrinker for VIRTIO_BALLOON_F_FREE_PAGE_HINT, because >> this has no such side effects. Always register the shrinker with >> VIRTIO_BALLOON_F_FREE_PAGE_HINT now. We are always allowed to reuse free >> pages that are still to be processed by the guest. The hypervisor takes >> care of identifying and resolving possible races between processing a >> hinting request and the guest reusing a page. >> >> In contrast to pre commit 71994620bb25 ("virtio_balloon: replace oom >> notifier with shrinker"), don't add a moodule parameter to configure the >> number of pages to deflate on OOM. Can be re-added if really needed. >> Also, pay attention that leak_balloon() returns the number of 4k pages - >> convert it properly in virtio_balloon_oom_notify(). >> >> Note1: using the OOM handler is frowned upon, but it really is what we >> need for this feature. >> >> Note2: without VIRTIO_BALLOON_F_MUST_TELL_HOST (iow, always with QEMU) we >> could actually skip sending deflation requests to our hypervisor, >> making the OOM path *very* simple. Besically freeing pages and >> updating the balloon. If the communication with the host ever >> becomes a problem on this call path. >> >> [1] https://www.spinics.net/lists/linux-virtualization/msg40863.html >> >> Reported-by: Tyler Sanderson >> Cc: Michael S. Tsirkin >> Cc: Wei Wang >> Cc: Alexander Duyck >> Cc: David Rientjes >> Cc: Nadav Amit >> Cc: Michal Hocko >> Signed-off-by: David Hildenbrand >> --- >> drivers/virtio/virtio_balloon.c | 107 +++++++++++++------------------- >> 1 file changed, 44 insertions(+), 63 deletions(-) >> >> diff --git a/drivers/virtio/virtio_balloon.c >> b/drivers/virtio/virtio_balloon.c >> index 7e5d84caeb94..e7b18f556c5e 100644 >> --- a/drivers/virtio/virtio_balloon.c >> +++ b/drivers/virtio/virtio_balloon.c >> @@ -14,6 +14,7 @@ >> #include >> #include >> #include >> +#include >> #include >> #include >> #include >> @@ -27,7 +28,9 @@ >> */ >> #define VIRTIO_BALLOON_PAGES_PER_PAGE (unsigned)(PAGE_SIZE >> >> VIRTIO_BALLOON_PFN_SHIFT) >> #define VIRTIO_BALLOON_ARRAY_PFNS_MAX 256 >> -#define VIRTBALLOON_OOM_NOTIFY_PRIORITY 80 >> +/* Maximum number of (4k) pages to deflate on OOM notifications. */ >> +#define VIRTIO_BALLOON_OOM_NR_PAGES 256 >> +#define VIRTIO_BALLOON_OOM_NOTIFY_PRIORITY 80 >> >> #define VIRTIO_BALLOON_FREE_PAGE_ALLOC_FLAG (__GFP_NORETRY | >> __GFP_NOWARN | \ >> __GFP_NOMEMALLOC) >> @@ -112,8 +115,11 @@ struct virtio_balloon { >> /* Memory statistics */ >> struct virtio_balloon_stat stats[VIRTIO_BALLOON_S_NR]; >> >> - /* To register a shrinker to shrink memory upon memory pressure */ >> + /* Shrinker to return free pages - >> VIRTIO_BALLOON_F_FREE_PAGE_HINT */ >> struct shrinker shrinker; >> + >> + /* OOM notifier to deflate on OOM - >> VIRTIO_BALLOON_F_DEFLATE_ON_OOM */ >> + struct notifier_block oom_nb; >> }; >> >> static struct virtio_device_id id_table[] = { >> @@ -786,50 +792,13 @@ static unsigned long shrink_free_pages(struct >> virtio_balloon *vb, >> return blocks_freed * VIRTIO_BALLOON_HINT_BLOCK_PAGES; >> } >> >> -static unsigned long leak_balloon_pages(struct virtio_balloon *vb, >> - unsigned long pages_to_free) >> -{ >> - return leak_balloon(vb, pages_to_free * >> VIRTIO_BALLOON_PAGES_PER_PAGE) / >> - VIRTIO_BALLOON_PAGES_PER_PAGE; >> -} >> - >> -static unsigned long shrink_balloon_pages(struct virtio_balloon *vb, >> - unsigned long pages_to_free) >> -{ >> - unsigned long pages_freed = 0; >> - >> - /* >> - * One invocation of leak_balloon can deflate at most >> - * VIRTIO_BALLOON_ARRAY_PFNS_MAX balloon pages, so we call it >> - * multiple times to deflate pages till reaching pages_to_free. >> - */ >> - while (vb->num_pages && pages_freed < pages_to_free) >> - pages_freed += leak_balloon_pages(vb, >> - pages_to_free - >> pages_freed); >> - >> - update_balloon_size(vb); >> - >> - return pages_freed; >> -} >> - >> static unsigned long virtio_balloon_shrinker_scan(struct shrinker >> *shrinker, >> struct shrink_control >> *sc) >> { >> - unsigned long pages_to_free, pages_freed = 0; >> struct virtio_balloon *vb = container_of(shrinker, >> struct virtio_balloon, shrinker); >> >> - pages_to_free = sc->nr_to_scan; >> - >> - if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_FREE_PAGE_HINT)) >> - pages_freed = shrink_free_pages(vb, pages_to_free); >> - >> - if (pages_freed >= pages_to_free) >> - return pages_freed; >> - >> - pages_freed += shrink_balloon_pages(vb, pages_to_free - >> pages_freed); >> - >> - return pages_freed; >> + return shrink_free_pages(vb, sc->nr_to_scan); >> } >> >> static unsigned long virtio_balloon_shrinker_count(struct shrinker >> *shrinker, >> @@ -837,26 +806,22 @@ static unsigned long >> virtio_balloon_shrinker_count(struct shrinker *shrinker, >> { >> struct virtio_balloon *vb = container_of(shrinker, >> struct virtio_balloon, shrinker); >> - unsigned long count; >> - >> - count = vb->num_pages / VIRTIO_BALLOON_PAGES_PER_PAGE; >> - count += vb->num_free_page_blocks * >> VIRTIO_BALLOON_HINT_BLOCK_PAGES; >> >> - return count; >> + return vb->num_free_page_blocks * VIRTIO_BALLOON_HINT_BLOCK_PAGES; >> } >> >> -static void virtio_balloon_unregister_shrinker(struct virtio_balloon *vb) >> +static int virtio_balloon_oom_notify(struct notifier_block *nb, >> + unsigned long dummy, void *parm) >> { >> - unregister_shrinker(&vb->shrinker); >> -} >> + struct virtio_balloon *vb = container_of(nb, >> + struct virtio_balloon, >> oom_nb); >> + unsigned long *freed = parm; >> >> -static int virtio_balloon_register_shrinker(struct virtio_balloon *vb) >> -{ >> - vb->shrinker.scan_objects = virtio_balloon_shrinker_scan; >> - vb->shrinker.count_objects = virtio_balloon_shrinker_count; >> - vb->shrinker.seeks = DEFAULT_SEEKS; >> + *freed += leak_balloon(vb, VIRTIO_BALLOON_OOM_NR_PAGES) / >> + VIRTIO_BALLOON_PAGES_PER_PAGE; >> + update_balloon_size(vb); >> >> - return register_shrinker(&vb->shrinker); >> + return NOTIFY_OK; >> } >> >> static int virtballoon_probe(struct virtio_device *vdev) >> @@ -933,22 +898,35 @@ static int virtballoon_probe(struct virtio_device >> *vdev) >> virtio_cwrite(vb->vdev, struct >> virtio_balloon_config, >> poison_val, &poison_val); >> } >> - } >> - /* >> - * We continue to use VIRTIO_BALLOON_F_DEFLATE_ON_OOM to decide >> if a >> - * shrinker needs to be registered to relieve memory pressure. >> - */ >> - if (virtio_has_feature(vb->vdev, >> VIRTIO_BALLOON_F_DEFLATE_ON_OOM)) { >> - err = virtio_balloon_register_shrinker(vb); >> + >> + /* >> + * We're allowed to reuse any free pages, even if they are >> + * still to be processed by the host. >> > It is important to clarify that pages that are on the inflate queue but > not ACKed by the host (the queue entry has not been returned) are _not_ > okay to reuse. > If the host is going to do something destructive to the page (like deback > it) then that needs to happen before the entry is returned. > > > While you are correct, this comment is in the „free page hinting“ > section/if (not obvious by looking at the diff only), so it does not apply > to inflate/deflate queues - but only free pages that are getting hinted. Or > am I misreading your suggestion/missing something? > Ah you are right. Thanks! > > Thanks! > > >