From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6094EC432BE for ; Wed, 28 Jul 2021 19:56:07 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id DF50E61038 for ; Wed, 28 Jul 2021 19:56:06 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org DF50E61038 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 66E118D0001; Wed, 28 Jul 2021 15:56:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 61E526B0036; Wed, 28 Jul 2021 15:56:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4E5788D0001; Wed, 28 Jul 2021 15:56:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0179.hostedemail.com [216.40.44.179]) by kanga.kvack.org (Postfix) with ESMTP id 320686B0033 for ; Wed, 28 Jul 2021 15:56:06 -0400 (EDT) Received: from smtpin32.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id CEF9D18571 for ; Wed, 28 Jul 2021 19:56:05 +0000 (UTC) X-FDA: 78413052690.32.9853DAE Received: from mail-pj1-f41.google.com (mail-pj1-f41.google.com [209.85.216.41]) by imf15.hostedemail.com (Postfix) with ESMTP id 20360D0042E0 for ; Wed, 28 Jul 2021 19:56:05 +0000 (UTC) Received: by mail-pj1-f41.google.com with SMTP id a4-20020a17090aa504b0290176a0d2b67aso11854728pjq.2 for ; Wed, 28 Jul 2021 12:56:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=hjw/5hPfcSAnUDtkGO7dLZoRiawjIU1v57wTHZKdYCc=; b=t3PfB1msBLB28JyzEIMcKXwRsHMPWdoEN3TENtYZlt2tYrmfA0vJeqRHiG6LEJgYXc 6U1MlRMdS5sumPb5IGtGvzY9zqiqrCwE88JT7X7eACtneUkCx3IXPgwzLRp20tWezGbV 3tfofj2uFWoi/9fyyMoVjsrz4WkZtCRozGny3Fs1G98xiCHbsQbCPGzAoEAzhTwFkQdc eG4K/g/2vBogT6hHN99sA0UBpTckeriuQ2G3l8nzGbdDZ1a4c+xWHpvbJWhCI2SPFXak mb1SI5+oF4QcRdj8tbtXnLi7Q6f1k3GDMiLzEqiIgmM1Eo+lUEeuSbV4S9IZlON/3CZN 1FRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=hjw/5hPfcSAnUDtkGO7dLZoRiawjIU1v57wTHZKdYCc=; b=S1ueD4l8SbCKzYIJs4WoKvlT6VschN6nxzinZHOmiVhoUWvQeNhRhlf3nPTefsvxWj sPxl5kioRES3qc202/gZSipfTsBhzs42fkO4hsE6sgyibiZpG2cxlZol7ZM5jzWiICBy TySuCAq9qZijfWaGD9FmcjDQWVDe92rZSHP81FlStDyO4ShrY1s1IKxlMA0xBE4KBIJQ s+hnVcHdyj9gp8sak+Pq+QBNf25WXD7Ja7wh/IU0NHjyt7Rm6MlDm4IBI3y7JwSmmG7a c283elbkTJYIvoEJHTRQWNW26XX6Q2H9zofzt26dTgKkmxofIrlW6iMlqLZsC0CrnMMn ra4w== X-Gm-Message-State: AOAM531fxDmq5I/rGXUqdo3A0BMRoaZ4LKFOIu/UhH5vLBD8sYajrCbi HsdJxxCz/r+rFh7oGA+qIE4lhXkizqMf6HTshtMpGw== X-Google-Smtp-Source: ABdhPJwr07MbmYenDrsCXLzA/k3E+4LlKYbMRwTmY0e+U3JwUORF0A3zdz8QcxmndAVjTfm9JaG1BaDYu23LMCTIpkc= X-Received: by 2002:a17:902:7d91:b029:12b:45b0:736b with SMTP id a17-20020a1709027d91b029012b45b0736bmr1123726plm.79.1627502163850; Wed, 28 Jul 2021 12:56:03 -0700 (PDT) MIME-Version: 1.0 References: <20210714193542.21857-1-joao.m.martins@oracle.com> <20210714193542.21857-14-joao.m.martins@oracle.com> In-Reply-To: <20210714193542.21857-14-joao.m.martins@oracle.com> From: Dan Williams Date: Wed, 28 Jul 2021 12:55:53 -0700 Message-ID: Subject: Re: [PATCH v3 13/14] mm/gup: grab head page refcount once for group of subpages To: Joao Martins Cc: Linux MM , Vishal Verma , Dave Jiang , Naoya Horiguchi , Matthew Wilcox , Jason Gunthorpe , John Hubbard , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Jonathan Corbet , Linux NVDIMM , Linux Doc Mailing List Content-Type: text/plain; charset="UTF-8" Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=intel-com.20150623.gappssmtp.com header.s=20150623 header.b=t3PfB1ms; spf=none (imf15.hostedemail.com: domain of dan.j.williams@intel.com has no SPF policy when checking 209.85.216.41) smtp.mailfrom=dan.j.williams@intel.com; dmarc=fail reason="No valid SPF, DKIM not aligned (relaxed)" header.from=intel.com (policy=none) X-Rspamd-Server: rspam02 X-Stat-Signature: 8uffxoooentu3fdz5kmhw8nwgqy3y4u5 X-Rspamd-Queue-Id: 20360D0042E0 X-HE-Tag: 1627502165-178662 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Jul 14, 2021 at 12:36 PM Joao Martins wrote: > > Use try_grab_compound_head() for device-dax GUP when configured with a > compound pagemap. > > Rather than incrementing the refcount for each page, do one atomic > addition for all the pages to be pinned. > > Performance measured by gup_benchmark improves considerably > get_user_pages_fast() and pin_user_pages_fast() with NVDIMMs: > > $ gup_test -f /dev/dax1.0 -m 16384 -r 10 -S [-u,-a] -n 512 -w > (get_user_pages_fast 2M pages) ~59 ms -> ~6.1 ms > (pin_user_pages_fast 2M pages) ~87 ms -> ~6.2 ms > [altmap] > (get_user_pages_fast 2M pages) ~494 ms -> ~9 ms > (pin_user_pages_fast 2M pages) ~494 ms -> ~10 ms > > $ gup_test -f /dev/dax1.0 -m 129022 -r 10 -S [-u,-a] -n 512 -w > (get_user_pages_fast 2M pages) ~492 ms -> ~49 ms > (pin_user_pages_fast 2M pages) ~493 ms -> ~50 ms > [altmap with -m 127004] > (get_user_pages_fast 2M pages) ~3.91 sec -> ~70 ms > (pin_user_pages_fast 2M pages) ~3.97 sec -> ~74 ms > > Signed-off-by: Joao Martins > --- > mm/gup.c | 53 +++++++++++++++++++++++++++++++++-------------------- > 1 file changed, 33 insertions(+), 20 deletions(-) > > diff --git a/mm/gup.c b/mm/gup.c > index 42b8b1fa6521..9baaa1c0b7f3 100644 > --- a/mm/gup.c > +++ b/mm/gup.c > @@ -2234,31 +2234,55 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, > } > #endif /* CONFIG_ARCH_HAS_PTE_SPECIAL */ > > + > +static int record_subpages(struct page *page, unsigned long addr, > + unsigned long end, struct page **pages) > +{ > + int nr; > + > + for (nr = 0; addr != end; addr += PAGE_SIZE) > + pages[nr++] = page++; > + > + return nr; > +} > + > #if defined(CONFIG_ARCH_HAS_PTE_DEVMAP) && defined(CONFIG_TRANSPARENT_HUGEPAGE) > static int __gup_device_huge(unsigned long pfn, unsigned long addr, > unsigned long end, unsigned int flags, > struct page **pages, int *nr) > { > - int nr_start = *nr; > + int refs, nr_start = *nr; > struct dev_pagemap *pgmap = NULL; > > do { > - struct page *page = pfn_to_page(pfn); > + struct page *pinned_head, *head, *page = pfn_to_page(pfn); > + unsigned long next; > > pgmap = get_dev_pagemap(pfn, pgmap); > if (unlikely(!pgmap)) { > undo_dev_pagemap(nr, nr_start, flags, pages); > return 0; > } > - SetPageReferenced(page); > - pages[*nr] = page; > - if (unlikely(!try_grab_page(page, flags))) { > - undo_dev_pagemap(nr, nr_start, flags, pages); > + > + head = compound_head(page); > + /* @end is assumed to be limited at most one compound page */ > + next = PageCompound(head) ? end : addr + PAGE_SIZE; Please no ternary operator for this check, but otherwise this patch looks good to me. Reviewed-by: Dan Williams > + refs = record_subpages(page, addr, next, pages + *nr); > + > + SetPageReferenced(head); > + pinned_head = try_grab_compound_head(head, refs, flags); > + if (!pinned_head) { > + if (PageCompound(head)) { > + ClearPageReferenced(head); > + put_dev_pagemap(pgmap); > + } else { > + undo_dev_pagemap(nr, nr_start, flags, pages); > + } > return 0; > } > - (*nr)++; > - pfn++; > - } while (addr += PAGE_SIZE, addr != end); > + *nr += refs; > + pfn += refs; > + } while (addr += (refs << PAGE_SHIFT), addr != end); > > if (pgmap) > put_dev_pagemap(pgmap); > @@ -2318,17 +2342,6 @@ static int __gup_device_huge_pud(pud_t pud, pud_t *pudp, unsigned long addr, > } > #endif > > -static int record_subpages(struct page *page, unsigned long addr, > - unsigned long end, struct page **pages) > -{ > - int nr; > - > - for (nr = 0; addr != end; addr += PAGE_SIZE) > - pages[nr++] = page++; > - > - return nr; > -} > - > #ifdef CONFIG_ARCH_HAS_HUGEPD > static unsigned long hugepte_addr_end(unsigned long addr, unsigned long end, > unsigned long sz) > -- > 2.17.1 >