From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C3688C2D0B1 for ; Thu, 6 Feb 2020 09:12:21 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 69BDF206CC for ; Thu, 6 Feb 2020 09:12:21 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="SalRD9kT" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 69BDF206CC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0A8906B0006; Thu, 6 Feb 2020 04:12:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 082516B0007; Thu, 6 Feb 2020 04:12:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ED99E6B0008; Thu, 6 Feb 2020 04:12:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0038.hostedemail.com [216.40.44.38]) by kanga.kvack.org (Postfix) with ESMTP id D02576B0006 for ; Thu, 6 Feb 2020 04:12:20 -0500 (EST) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 66FA58248D52 for ; Thu, 6 Feb 2020 09:12:20 +0000 (UTC) X-FDA: 76459136040.22.spot24_317a218b5f413 X-HE-Tag: spot24_317a218b5f413 X-Filterd-Recvd-Size: 14712 Received: from us-smtp-1.mimecast.com (us-smtp-delivery-1.mimecast.com [207.211.31.120]) by imf40.hostedemail.com (Postfix) with ESMTP for ; Thu, 6 Feb 2020 09:12:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1580980339; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hXGHWkN4g9yAvrMcPhR2p/U9EJCf2qksIXBdMKfovl4=; b=SalRD9kTyU1YGyOOadEZOcO5uF6zPeXtLHCuuaSrVJF4nKaalhEyAmCyUl/oesN5aydIBk ZLITiiMtOtbbAGa6P3J4Uk3UlSQzdTqMKB0HD1XuGP2a2QZZi/UfOmhTZ+cIrbIM2moC0M S5cgOmCZf/sKoSBvLzs7q60cJZP+Krs= Received: from mail-qv1-f72.google.com (mail-qv1-f72.google.com [209.85.219.72]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-5-TW8YVV8pOjS9vbJNuDYOgA-1; Thu, 06 Feb 2020 04:12:17 -0500 Received: by mail-qv1-f72.google.com with SMTP id dr18so3282112qvb.14 for ; Thu, 06 Feb 2020 01:12:17 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=E+oh5Pmn7fq2TiHt+yzyj7uEE9/6W9iJHrYGL0+d2SQ=; b=BHjaK+DMq3Fc7Ux0fzJragnC4KJPHwxfhnbXj0pp1IG88TstyGa3iKRUyjpyvLJL7C Kgor7dwwr8djff7Tlvs/QcRNKkSIyle7s2a+/DaHdUTA+gp7/Bq6t+48B4oA/bjdYS5t 3sTMxPjgxvLz8tnN5k3Z+o2spK7BzV0LDOMArSsj7SmseqzpziuIclh+ir4OgtYd2LGa QA1oINAdQjViFkLyVT+tzZa7p2NZ5F7I0FJFbva1t8K95z8zbuWH3IUk2zNn0GTFxt2/ LFjoyKFqYIwkm4UWweQ8cWHDuz1gDoUP5RstXQ7RjZObbbQBfZXSIMJ9tu1wdz8l2Ztv 2XEg== X-Gm-Message-State: APjAAAVdCQZRuOhYiWB5TQOh1DyA2yn53QJs5pLsrVOBa2n5f4PA2CAO 9PRHEh6T5WI8um/r9VOtzsEwtJi6AMQzx92svHrGJlXr8aa2nuQlJVlWnNTom1BRJd4nKJ7/L0n IP0i3QLkHDPM= X-Received: by 2002:ae9:ee11:: with SMTP id i17mr1597822qkg.333.1580980336772; Thu, 06 Feb 2020 01:12:16 -0800 (PST) X-Google-Smtp-Source: APXvYqyPiuidl9Qwdt7QNWhWnc/fuiWxC40B0nSebTITJk+DZOpvXjL4dAG3wJ07AI2Q3eVLI6fr0w== X-Received: by 2002:ae9:ee11:: with SMTP id i17mr1597801qkg.333.1580980336498; Thu, 06 Feb 2020 01:12:16 -0800 (PST) Received: from redhat.com (bzq-79-176-41-183.red.bezeqint.net. [79.176.41.183]) by smtp.gmail.com with ESMTPSA id d71sm1114923qkg.4.2020.02.06.01.12.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 06 Feb 2020 01:12:15 -0800 (PST) Date: Thu, 6 Feb 2020 04:12:11 -0500 From: "Michael S. Tsirkin" To: David Hildenbrand Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, virtualization@lists.linux-foundation.org, Tyler Sanderson , Wei Wang , Alexander Duyck , David Rientjes , Nadav Amit , Michal Hocko Subject: Re: [PATCH v1 3/3] virtio-balloon: Switch back to OOM handler for VIRTIO_BALLOON_F_DEFLATE_ON_OOM Message-ID: <20200206041126-mutt-send-email-mst@kernel.org> References: <20200205163402.42627-1-david@redhat.com> <20200205163402.42627-4-david@redhat.com> MIME-Version: 1.0 In-Reply-To: <20200205163402.42627-4-david@redhat.com> X-MC-Unique: TW8YVV8pOjS9vbJNuDYOgA-1 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Content-Disposition: inline X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Feb 05, 2020 at 05:34:02PM +0100, David Hildenbrand wrote: > Commit 71994620bb25 ("virtio_balloon: replace oom notifier with shrinker"= ) > changed the behavior when deflation happens automatically. Instead of > deflating when called by the OOM handler, the shrinker is used. >=20 > However, the balloon is not simply some slab cache that should be > shrunk when under memory pressure. The shrinker does not have a concept o= f > priorities, so this behavior cannot be configured. >=20 > There was a report that this results in undesired side effects when > inflating the balloon to shrink the page cache. [1] > =09"When inflating the balloon against page cache (i.e. no free memory > =09 remains) vmscan.c will both shrink page cache, but also invoke the > =09 shrinkers -- including the balloon's shrinker. So the balloon > =09 driver allocates memory which requires reclaim, vmscan gets this > =09 memory by shrinking the balloon, and then the driver adds the > =09 memory back to the balloon. Basically a busy no-op." >=20 > The name "deflate on OOM" makes it pretty clear when deflation should > happen - after other approaches to reclaim memory failed, not while > reclaiming. This allows to minimize the footprint of a guest - memory > will only be taken out of the balloon when really needed. >=20 > Especially, a drop_slab() will result in the whole balloon getting > deflated - undesired. While handling it via the OOM handler might not be > perfect, it keeps existing behavior. If we want a different behavior, the= n > we need a new feature bit and document it properly (although, there shoul= d > be a clear use case and the intended effects should be well described). >=20 > Keep using the shrinker for VIRTIO_BALLOON_F_FREE_PAGE_HINT, because > this has no such side effects. Always register the shrinker with > VIRTIO_BALLOON_F_FREE_PAGE_HINT now. We are always allowed to reuse free > pages that are still to be processed by the guest. The hypervisor takes > care of identifying and resolving possible races between processing a > hinting request and the guest reusing a page. >=20 > In contrast to pre commit 71994620bb25 ("virtio_balloon: replace oom > notifier with shrinker"), don't add a moodule parameter to configure the > number of pages to deflate on OOM. Can be re-added if really needed. > Also, pay attention that leak_balloon() returns the number of 4k pages - > convert it properly in virtio_balloon_oom_notify(). >=20 > Note1: using the OOM handler is frowned upon, but it really is what we > need for this feature. >=20 > Note2: without VIRTIO_BALLOON_F_MUST_TELL_HOST (iow, always with QEMU) we > could actually skip sending deflation requests to our hypervisor, > making the OOM path *very* simple. Besically freeing pages and > updating the balloon. If the communication with the host ever > becomes a problem on this call path. >=20 > [1] https://www.spinics.net/lists/linux-virtualization/msg40863.html >=20 > Reported-by: Tyler Sanderson > Cc: Michael S. Tsirkin > Cc: Wei Wang > Cc: Alexander Duyck > Cc: David Rientjes > Cc: Nadav Amit > Cc: Michal Hocko > Signed-off-by: David Hildenbrand I guess we should add a Fixes tag to the patch it's reverting, this way it's backported and hypervisors will be able to rely on OOM behaviour. > --- > drivers/virtio/virtio_balloon.c | 107 +++++++++++++------------------- > 1 file changed, 44 insertions(+), 63 deletions(-) >=20 > diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_ball= oon.c > index 7e5d84caeb94..e7b18f556c5e 100644 > --- a/drivers/virtio/virtio_balloon.c > +++ b/drivers/virtio/virtio_balloon.c > @@ -14,6 +14,7 @@ > #include > #include > #include > +#include > #include > #include > #include > @@ -27,7 +28,9 @@ > */ > #define VIRTIO_BALLOON_PAGES_PER_PAGE (unsigned)(PAGE_SIZE >> VIRTIO_BAL= LOON_PFN_SHIFT) > #define VIRTIO_BALLOON_ARRAY_PFNS_MAX 256 > -#define VIRTBALLOON_OOM_NOTIFY_PRIORITY 80 > +/* Maximum number of (4k) pages to deflate on OOM notifications. */ > +#define VIRTIO_BALLOON_OOM_NR_PAGES 256 > +#define VIRTIO_BALLOON_OOM_NOTIFY_PRIORITY 80 > =20 > #define VIRTIO_BALLOON_FREE_PAGE_ALLOC_FLAG (__GFP_NORETRY | __GFP_NOWAR= N | \ > =09=09=09=09=09 __GFP_NOMEMALLOC) > @@ -112,8 +115,11 @@ struct virtio_balloon { > =09/* Memory statistics */ > =09struct virtio_balloon_stat stats[VIRTIO_BALLOON_S_NR]; > =20 > -=09/* To register a shrinker to shrink memory upon memory pressure */ > +=09/* Shrinker to return free pages - VIRTIO_BALLOON_F_FREE_PAGE_HINT */ > =09struct shrinker shrinker; > + > +=09/* OOM notifier to deflate on OOM - VIRTIO_BALLOON_F_DEFLATE_ON_OOM *= / > +=09struct notifier_block oom_nb; > }; > =20 > static struct virtio_device_id id_table[] =3D { > @@ -786,50 +792,13 @@ static unsigned long shrink_free_pages(struct virti= o_balloon *vb, > =09return blocks_freed * VIRTIO_BALLOON_HINT_BLOCK_PAGES; > } > =20 > -static unsigned long leak_balloon_pages(struct virtio_balloon *vb, > - unsigned long pages_to_free) > -{ > -=09return leak_balloon(vb, pages_to_free * VIRTIO_BALLOON_PAGES_PER_PAGE= ) / > -=09=09VIRTIO_BALLOON_PAGES_PER_PAGE; > -} > - > -static unsigned long shrink_balloon_pages(struct virtio_balloon *vb, > -=09=09=09=09=09 unsigned long pages_to_free) > -{ > -=09unsigned long pages_freed =3D 0; > - > -=09/* > -=09 * One invocation of leak_balloon can deflate at most > -=09 * VIRTIO_BALLOON_ARRAY_PFNS_MAX balloon pages, so we call it > -=09 * multiple times to deflate pages till reaching pages_to_free. > -=09 */ > -=09while (vb->num_pages && pages_freed < pages_to_free) > -=09=09pages_freed +=3D leak_balloon_pages(vb, > -=09=09=09=09=09=09 pages_to_free - pages_freed); > - > -=09update_balloon_size(vb); > - > -=09return pages_freed; > -} > - > static unsigned long virtio_balloon_shrinker_scan(struct shrinker *shrin= ker, > =09=09=09=09=09=09 struct shrink_control *sc) > { > -=09unsigned long pages_to_free, pages_freed =3D 0; > =09struct virtio_balloon *vb =3D container_of(shrinker, > =09=09=09=09=09struct virtio_balloon, shrinker); > =20 > -=09pages_to_free =3D sc->nr_to_scan; > - > -=09if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_FREE_PAGE_HINT)) > -=09=09pages_freed =3D shrink_free_pages(vb, pages_to_free); > - > -=09if (pages_freed >=3D pages_to_free) > -=09=09return pages_freed; > - > -=09pages_freed +=3D shrink_balloon_pages(vb, pages_to_free - pages_freed= ); > - > -=09return pages_freed; > +=09return shrink_free_pages(vb, sc->nr_to_scan); > } > =20 > static unsigned long virtio_balloon_shrinker_count(struct shrinker *shri= nker, > @@ -837,26 +806,22 @@ static unsigned long virtio_balloon_shrinker_count(= struct shrinker *shrinker, > { > =09struct virtio_balloon *vb =3D container_of(shrinker, > =09=09=09=09=09struct virtio_balloon, shrinker); > -=09unsigned long count; > - > -=09count =3D vb->num_pages / VIRTIO_BALLOON_PAGES_PER_PAGE; > -=09count +=3D vb->num_free_page_blocks * VIRTIO_BALLOON_HINT_BLOCK_PAGES= ; > =20 > -=09return count; > +=09return vb->num_free_page_blocks * VIRTIO_BALLOON_HINT_BLOCK_PAGES; > } > =20 > -static void virtio_balloon_unregister_shrinker(struct virtio_balloon *vb= ) > +static int virtio_balloon_oom_notify(struct notifier_block *nb, > +=09=09=09=09 unsigned long dummy, void *parm) > { > -=09unregister_shrinker(&vb->shrinker); > -} > +=09struct virtio_balloon *vb =3D container_of(nb, > +=09=09=09=09=09=09 struct virtio_balloon, oom_nb); > +=09unsigned long *freed =3D parm; > =20 > -static int virtio_balloon_register_shrinker(struct virtio_balloon *vb) > -{ > -=09vb->shrinker.scan_objects =3D virtio_balloon_shrinker_scan; > -=09vb->shrinker.count_objects =3D virtio_balloon_shrinker_count; > -=09vb->shrinker.seeks =3D DEFAULT_SEEKS; > +=09*freed +=3D leak_balloon(vb, VIRTIO_BALLOON_OOM_NR_PAGES) / > +=09=09 VIRTIO_BALLOON_PAGES_PER_PAGE; > +=09update_balloon_size(vb); > =20 > -=09return register_shrinker(&vb->shrinker); > +=09return NOTIFY_OK; > } > =20 > static int virtballoon_probe(struct virtio_device *vdev) > @@ -933,22 +898,35 @@ static int virtballoon_probe(struct virtio_device *= vdev) > =09=09=09virtio_cwrite(vb->vdev, struct virtio_balloon_config, > =09=09=09=09 poison_val, &poison_val); > =09=09} > -=09} > -=09/* > -=09 * We continue to use VIRTIO_BALLOON_F_DEFLATE_ON_OOM to decide if a > -=09 * shrinker needs to be registered to relieve memory pressure. > -=09 */ > -=09if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_DEFLATE_ON_OOM)) { > -=09=09err =3D virtio_balloon_register_shrinker(vb); > + > +=09=09/* > +=09=09 * We're allowed to reuse any free pages, even if they are > +=09=09 * still to be processed by the host. > +=09=09 */ > +=09=09vb->shrinker.scan_objects =3D virtio_balloon_shrinker_scan; > +=09=09vb->shrinker.count_objects =3D virtio_balloon_shrinker_count; > +=09=09vb->shrinker.seeks =3D DEFAULT_SEEKS; > +=09=09err =3D register_shrinker(&vb->shrinker); > =09=09if (err) > =09=09=09goto out_del_balloon_wq; > =09} > +=09if (virtio_has_feature(vdev, VIRTIO_BALLOON_F_DEFLATE_ON_OOM)) { > +=09=09vb->oom_nb.notifier_call =3D virtio_balloon_oom_notify; > +=09=09vb->oom_nb.priority =3D VIRTIO_BALLOON_OOM_NOTIFY_PRIORITY; > +=09=09err =3D register_oom_notifier(&vb->oom_nb); > +=09=09if (err < 0) > +=09=09=09goto out_unregister_shrinker; > +=09} > + > =09virtio_device_ready(vdev); > =20 > =09if (towards_target(vb)) > =09=09virtballoon_changed(vdev); > =09return 0; > =20 > +out_unregister_shrinker: > +=09if (virtio_has_feature(vdev, VIRTIO_BALLOON_F_FREE_PAGE_HINT)) > +=09=09unregister_shrinker(&vb->shrinker); > out_del_balloon_wq: > =09if (virtio_has_feature(vdev, VIRTIO_BALLOON_F_FREE_PAGE_HINT)) > =09=09destroy_workqueue(vb->balloon_wq); > @@ -987,8 +965,11 @@ static void virtballoon_remove(struct virtio_device = *vdev) > { > =09struct virtio_balloon *vb =3D vdev->priv; > =20 > -=09if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_DEFLATE_ON_OOM)) > -=09=09virtio_balloon_unregister_shrinker(vb); > +=09if (virtio_has_feature(vdev, VIRTIO_BALLOON_F_DEFLATE_ON_OOM)) > +=09=09unregister_oom_notifier(&vb->oom_nb); > +=09if (virtio_has_feature(vdev, VIRTIO_BALLOON_F_FREE_PAGE_HINT)) > +=09=09unregister_shrinker(&vb->shrinker); > + > =09spin_lock_irq(&vb->stop_update_lock); > =09vb->stop_update =3D true; > =09spin_unlock_irq(&vb->stop_update_lock); > --=20 > 2.24.1