From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.7 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,UNWANTED_LANGUAGE_BODY,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E4801C433E7 for ; Mon, 19 Oct 2020 14:57:10 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 50E79222D9 for ; Mon, 19 Oct 2020 14:57:10 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=bytedance-com.20150623.gappssmtp.com header.i=@bytedance-com.20150623.gappssmtp.com header.b="UY2vLLiq" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 50E79222D9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id CC7916B0073; Mon, 19 Oct 2020 10:57:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C7A506B0074; Mon, 19 Oct 2020 10:57:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B68126B0075; Mon, 19 Oct 2020 10:57:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0221.hostedemail.com [216.40.44.221]) by kanga.kvack.org (Postfix) with ESMTP id 8B9426B0073 for ; Mon, 19 Oct 2020 10:57:09 -0400 (EDT) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 1E71C1EF1 for ; Mon, 19 Oct 2020 14:57:09 +0000 (UTC) X-FDA: 77388977778.23.goat86_3e0994f27237 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin23.hostedemail.com (Postfix) with ESMTP id F380F37604 for ; Mon, 19 Oct 2020 14:57:08 +0000 (UTC) X-HE-Tag: goat86_3e0994f27237 X-Filterd-Recvd-Size: 11212 Received: from mail-pj1-f68.google.com (mail-pj1-f68.google.com [209.85.216.68]) by imf01.hostedemail.com (Postfix) with ESMTP for ; Mon, 19 Oct 2020 14:57:08 +0000 (UTC) Received: by mail-pj1-f68.google.com with SMTP id h4so5878891pjk.0 for ; Mon, 19 Oct 2020 07:57:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=xlwaBLktv2hjXeUdi+NrDvpVzYyjXC6N+oAeciA1bdM=; b=UY2vLLiqUa4Mo5Nkd9qQWe1h85EsA2WVuRjs2vWDfS+TksR3mSnE45CPjLKGRX0iJm 1sjYKlNmjtrfSBtTdGy6lOURfssad+9GpD3UwcKGp2DuWOP0RUhqzTH8aIu8nQNYNWNY /xXL1co2/0EGjADaSX9VJz8FnDRaLf+kAvxckIem4Hv4LRr3rhupQl+Ml9jeakhpz7UR tS0f2+wbNIrMFqjmEF8mU6n7/Vs4cvXpdv6BCv1l2KFBxqM7b2uIJl+6wvA2M8ORowIS iZCH9WtspLEro/wz0igyrWlKzmFBpCb+2vZxnYBxC52yZzQpjhc42PwGESi/Tz4BoF1P PQ+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=xlwaBLktv2hjXeUdi+NrDvpVzYyjXC6N+oAeciA1bdM=; b=aNho5CXjA6J5IQARpOO1BnYwp3Y+8vTFfHoLfzUYmxVA0JkUKOF8HOe+PphE/zRaFb mpao/4tfym8+sVhGwShYMOLUbugk8joR00Q56urWIOP6lAXKgE3oda0MGe/q7slD/d7C jVriberIhs2HEmoH+rEAUgb2h0esC5cVSEzRzT8X4zK4Nk+GXRqXoag5FmtHT2an/fnz nrv8D1XqF7zQXVtD2+G3Fg/kb1DBe2f3Pf0zwXy14Y8FrYszISsmhfFcL9WcoVJBskNS nr9xxeBPfhpz0PyhrOViMr+6ybuiBvMViiqZws+GrcKc130ntHHykXXrs/wN9W2W4jAr c40w== X-Gm-Message-State: AOAM533qYuvyOmB4PHzDF+pBntKZsi4w3EgMUrdDu8Z4BtU0Yi4ry+SK LXMxtWeF2Kv2iqvxb+gsQMcC X-Google-Smtp-Source: ABdhPJwjEZ/0jd0mu3JQLABrL2ZdiZLMr0q2aggM7CAAdA1fagjTEEwPe4tYGTlzKEkPaY8GTut9WQ== X-Received: by 2002:a17:90a:d317:: with SMTP id p23mr205662pju.52.1603119427291; Mon, 19 Oct 2020 07:57:07 -0700 (PDT) Received: from localhost ([103.136.220.106]) by smtp.gmail.com with ESMTPSA id 14sm81231pjn.48.2020.10.19.07.57.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 19 Oct 2020 07:57:06 -0700 (PDT) From: Xie Yongji To: mst@redhat.com, jasowang@redhat.com, akpm@linux-foundation.org Cc: linux-mm@kvack.org, virtualization@lists.linux-foundation.org Subject: [RFC 4/4] vduse: Add memory shrinker to reclaim bounce pages Date: Mon, 19 Oct 2020 22:56:23 +0800 Message-Id: <20201019145623.671-5-xieyongji@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201019145623.671-1-xieyongji@bytedance.com> References: <20201019145623.671-1-xieyongji@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a shrinker to reclaim several pages used by bounce buffer in order to avoid memory pressures. We will do reclaiming chunk by chunk. And only reclaim the iova chunk that no one used. Signed-off-by: Xie Yongji --- drivers/vdpa/vdpa_user/iova_domain.c | 83 ++++++++++++++++++++++++++-- drivers/vdpa/vdpa_user/iova_domain.h | 10 ++++ drivers/vdpa/vdpa_user/vduse_dev.c | 51 +++++++++++++++++ 3 files changed, 140 insertions(+), 4 deletions(-) diff --git a/drivers/vdpa/vdpa_user/iova_domain.c b/drivers/vdpa/vdpa_use= r/iova_domain.c index a274f78f00d2..9e3d4686de4f 100644 --- a/drivers/vdpa/vdpa_user/iova_domain.c +++ b/drivers/vdpa/vdpa_user/iova_domain.c @@ -30,6 +30,8 @@ struct vduse_mmap_vma { struct list_head list; }; =20 +struct percpu_counter vduse_total_bounce_pages; + static inline struct page * vduse_domain_get_bounce_page(struct vduse_iova_domain *domain, unsigned long iova) @@ -49,6 +51,13 @@ vduse_domain_set_bounce_page(struct vduse_iova_domain = *domain, unsigned long chunkoff =3D iova & ~IOVA_CHUNK_MASK; unsigned long pgindex =3D chunkoff >> PAGE_SHIFT; =20 + if (page) { + domain->chunks[index].used_bounce_pages++; + percpu_counter_inc(&vduse_total_bounce_pages); + } else { + domain->chunks[index].used_bounce_pages--; + percpu_counter_dec(&vduse_total_bounce_pages); + } domain->chunks[index].bounce_pages[pgindex] =3D page; } =20 @@ -159,6 +168,29 @@ void vduse_domain_remove_mapping(struct vduse_iova_d= omain *domain, spin_unlock(&domain->map_lock); } =20 +static bool vduse_domain_try_unmap(struct vduse_iova_domain *domain, + unsigned long iova, size_t size) +{ + struct vduse_mmap_vma *mmap_vma; + unsigned long uaddr; + bool unmap =3D true; + + mutex_lock(&domain->vma_lock); + list_for_each_entry(mmap_vma, &domain->vma_list, list) { + if (!mmap_read_trylock(mmap_vma->vma->vm_mm)) { + unmap =3D false; + break; + } + + uaddr =3D iova + mmap_vma->vma->vm_start; + zap_page_range(mmap_vma->vma, uaddr, size); + mmap_read_unlock(mmap_vma->vma->vm_mm); + } + mutex_unlock(&domain->vma_lock); + + return unmap; +} + void vduse_domain_unmap(struct vduse_iova_domain *domain, unsigned long iova, size_t size) { @@ -284,6 +316,32 @@ bool vduse_domain_is_direct_map(struct vduse_iova_do= main *domain, return atomic_read(&chunk->map_type) =3D=3D TYPE_DIRECT_MAP; } =20 +int vduse_domain_reclaim(struct vduse_iova_domain *domain) +{ + struct vduse_iova_chunk *chunk; + int i, freed =3D 0; + + for (i =3D domain->chunk_num - 1; i >=3D 0; i--) { + chunk =3D &domain->chunks[i]; + if (!chunk->used_bounce_pages) + continue; + + if (atomic_cmpxchg(&chunk->state, 0, INT_MIN) !=3D 0) + continue; + + if (!vduse_domain_try_unmap(domain, + chunk->start, IOVA_CHUNK_SIZE)) { + atomic_sub(INT_MIN, &chunk->state); + break; + } + freed +=3D vduse_domain_free_bounce_pages(domain, + chunk->start, IOVA_CHUNK_SIZE); + atomic_sub(INT_MIN, &chunk->state); + } + + return freed; +} + unsigned long vduse_domain_alloc_iova(struct vduse_iova_domain *domain, size_t size, enum iova_map_type type) { @@ -301,10 +359,13 @@ unsigned long vduse_domain_alloc_iova(struct vduse_= iova_domain *domain, if (atomic_read(&chunk->map_type) !=3D type) continue; =20 - iova =3D gen_pool_alloc_algo(chunk->pool, size, + if (atomic_fetch_inc(&chunk->state) >=3D 0) { + iova =3D gen_pool_alloc_algo(chunk->pool, size, gen_pool_first_fit_align, &data); - if (iova) - break; + if (iova) + break; + } + atomic_dec(&chunk->state); } =20 return iova; @@ -317,6 +378,7 @@ void vduse_domain_free_iova(struct vduse_iova_domain = *domain, struct vduse_iova_chunk *chunk =3D &domain->chunks[index]; =20 gen_pool_free(chunk->pool, iova, size); + atomic_dec(&chunk->state); } =20 static void vduse_iova_chunk_cleanup(struct vduse_iova_chunk *chunk) @@ -332,7 +394,8 @@ void vduse_iova_domain_destroy(struct vduse_iova_doma= in *domain) =20 for (i =3D 0; i < domain->chunk_num; i++) { chunk =3D &domain->chunks[i]; - vduse_domain_free_bounce_pages(domain, + if (chunk->used_bounce_pages) + vduse_domain_free_bounce_pages(domain, chunk->start, IOVA_CHUNK_SIZE); vduse_iova_chunk_cleanup(chunk); } @@ -365,8 +428,10 @@ static int vduse_iova_chunk_init(struct vduse_iova_c= hunk *chunk, if (!chunk->bounce_pages) goto err; =20 + chunk->used_bounce_pages =3D 0; chunk->start =3D addr; atomic_set(&chunk->map_type, TYPE_NONE); + atomic_set(&chunk->state, 0); =20 return 0; err: @@ -411,3 +476,13 @@ struct vduse_iova_domain *vduse_iova_domain_create(s= ize_t size) =20 return NULL; } + +int vduse_domain_init(void) +{ + return percpu_counter_init(&vduse_total_bounce_pages, 0, GFP_KERNEL); +} + +void vduse_domain_exit(void) +{ + percpu_counter_destroy(&vduse_total_bounce_pages); +} diff --git a/drivers/vdpa/vdpa_user/iova_domain.h b/drivers/vdpa/vdpa_use= r/iova_domain.h index 7ae60c0e50ec..016f84d4bef2 100644 --- a/drivers/vdpa/vdpa_user/iova_domain.h +++ b/drivers/vdpa/vdpa_user/iova_domain.h @@ -24,8 +24,10 @@ enum iova_map_type { struct vduse_iova_chunk { struct gen_pool *pool; struct page **bounce_pages; + int used_bounce_pages; unsigned long start; atomic_t map_type; + atomic_t state; }; =20 struct vduse_iova_domain { @@ -45,6 +47,8 @@ struct vduse_iova_map { enum dma_data_direction dir; }; =20 +extern struct percpu_counter vduse_total_bounce_pages; + int vduse_domain_add_vma(struct vduse_iova_domain *domain, struct vm_area_struct *vma); =20 @@ -78,6 +82,8 @@ int vduse_domain_bounce_map(struct vduse_iova_domain *d= omain, bool vduse_domain_is_direct_map(struct vduse_iova_domain *domain, unsigned long iova); =20 +int vduse_domain_reclaim(struct vduse_iova_domain *domain); + unsigned long vduse_domain_alloc_iova(struct vduse_iova_domain *domain, size_t size, enum iova_map_type type); =20 @@ -91,4 +97,8 @@ void vduse_iova_domain_destroy(struct vduse_iova_domain= *domain); =20 struct vduse_iova_domain *vduse_iova_domain_create(size_t size); =20 +int vduse_domain_init(void); + +void vduse_domain_exit(void); + #endif /* _VDUSE_IOVA_DOMAIN_H */ diff --git a/drivers/vdpa/vdpa_user/vduse_dev.c b/drivers/vdpa/vdpa_user/= vduse_dev.c index f04aa02de8c1..1163209ffff3 100644 --- a/drivers/vdpa/vdpa_user/vduse_dev.c +++ b/drivers/vdpa/vdpa_user/vduse_dev.c @@ -977,6 +977,43 @@ static long vduse_ioctl(struct file *file, unsigned = int cmd, return ret; } =20 +static unsigned long vduse_shrink_scan(struct shrinker *shrinker, + struct shrink_control *sc) +{ + unsigned long freed =3D 0; + struct vduse_dev *dev; + + if (!mutex_trylock(&vduse_lock)) + return SHRINK_STOP; + + list_for_each_entry(dev, &vduse_devs, list) { + if (!dev->domain) + continue; + + freed =3D vduse_domain_reclaim(dev->domain); + if (!freed) + continue; + + list_move_tail(&dev->list, &vduse_devs); + break; + } + mutex_unlock(&vduse_lock); + + return freed ? freed : SHRINK_STOP; +} + +static unsigned long vduse_shrink_count(struct shrinker *shrink, + struct shrink_control *sc) +{ + return percpu_counter_read_positive(&vduse_total_bounce_pages); +} + +static struct shrinker vduse_bounce_pages_shrinker =3D { + .count_objects =3D vduse_shrink_count, + .scan_objects =3D vduse_shrink_scan, + .seeks =3D DEFAULT_SEEKS, +}; + static const struct file_operations vduse_fops =3D { .owner =3D THIS_MODULE, .unlocked_ioctl =3D vduse_ioctl, @@ -1007,7 +1044,19 @@ static int vduse_init(void) if (ret) goto err_irqfd; =20 + ret =3D vduse_domain_init(); + if (ret) + goto err_domain; + + ret =3D register_shrinker(&vduse_bounce_pages_shrinker); + if (ret) + goto err_shrinker; + return 0; +err_shrinker: + vduse_domain_exit(); +err_domain: + vduse_virqfd_exit(); err_irqfd: destroy_workqueue(vduse_vdpa_wq); err_vdpa_wq: @@ -1018,8 +1067,10 @@ module_init(vduse_init); =20 static void vduse_exit(void) { + unregister_shrinker(&vduse_bounce_pages_shrinker); misc_deregister(&vduse_misc); destroy_workqueue(vduse_vdpa_wq); + vduse_domain_exit(); vduse_virqfd_exit(); } module_exit(vduse_exit); --=20 2.25.1