From: John Hubbard <jhubbard@nvidia.com> To: Andrew Morton <akpm@linux-foundation.org> Cc: "Al Viro" <viro@zeniv.linux.org.uk>, "Alex Williamson" <alex.williamson@redhat.com>, "Benjamin Herrenschmidt" <benh@kernel.crashing.org>, "Björn Töpel" <bjorn.topel@intel.com>, "Christoph Hellwig" <hch@infradead.org>, "Dan Williams" <dan.j.williams@intel.com>, "Daniel Vetter" <daniel@ffwll.ch>, "Dave Chinner" <david@fromorbit.com>, "David Airlie" <airlied@linux.ie>, "David S . Miller" <davem@davemloft.net>, "Ira Weiny" <ira.weiny@intel.com>, "Jan Kara" <jack@suse.cz>, "Jason Gunthorpe" <jgg@ziepe.ca>, "Jens Axboe" <axboe@kernel.dk>, "Jonathan Corbet" <corbet@lwn.net>, "Jérôme Glisse" <jglisse@redhat.com>, "Magnus Karlsson" <magnus.karlsson@intel.com>, "Mauro Carvalho Chehab" <mchehab@kernel.org>, "Michael Ellerman" <mpe@ellerman.id.au>, "Michal Hocko" <mhocko@suse.com>, "Mike Kravetz" <mike.kravetz@oracle.com>, "Paul Mackerras" <paulus@samba.org>, "Shuah Khan" <shuah@kernel.org>, "Vlastimil Babka" <vbabka@suse.cz>, bpf@vger.kernel.org, dri-devel@lists.freedesktop.org, kvm@vger.kernel.org, linux-block@vger.kernel.org, linux-doc@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-media@vger.kernel.org, linux-rdma@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, netdev@vger.kernel.org, linux-mm@kvack.org, LKML <linux-kernel@vger.kernel.org>, "John Hubbard" <jhubbard@nvidia.com>, "Christoph Hellwig" <hch@lst.de>, "Aneesh Kumar K . V" <aneesh.kumar@linux.ibm.com> Subject: [PATCH v2 02/18] mm/gup: factor out duplicate code from four routines Date: Sun, 3 Nov 2019 13:17:57 -0800 Message-ID: <20191103211813.213227-3-jhubbard@nvidia.com> (raw) In-Reply-To: <20191103211813.213227-1-jhubbard@nvidia.com> There are four locations in gup.c that have a fair amount of code duplication. This means that changing one requires making the same changes in four places, not to mention reading the same code four times, and wondering if there are subtle differences. Factor out the common code into static functions, thus reducing the overall line count and the code's complexity. Also, take the opportunity to slightly improve the efficiency of the error cases, by doing a mass subtraction of the refcount, surrounded by get_page()/put_page(). Also, further simplify (slightly), by waiting until the the successful end of each routine, to increment *nr. Cc: Ira Weiny <ira.weiny@intel.com> Cc: Christoph Hellwig <hch@lst.de> Cc: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com> Signed-off-by: John Hubbard <jhubbard@nvidia.com> --- mm/gup.c | 104 ++++++++++++++++++++++++------------------------------- 1 file changed, 45 insertions(+), 59 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 85caf76b3012..199da99e8ffc 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1969,6 +1969,34 @@ static int __gup_device_huge_pud(pud_t pud, pud_t *pudp, unsigned long addr, } #endif +static int __record_subpages(struct page *page, unsigned long addr, + unsigned long end, struct page **pages, int nr) +{ + int nr_recorded_pages = 0; + + do { + pages[nr] = page; + nr++; + page++; + nr_recorded_pages++; + } while (addr += PAGE_SIZE, addr != end); + return nr_recorded_pages; +} + +static void put_compound_head(struct page *page, int refs) +{ + /* Do a get_page() first, in case refs == page->_refcount */ + get_page(page); + page_ref_sub(page, refs); + put_page(page); +} + +static void __huge_pt_done(struct page *head, int nr_recorded_pages, int *nr) +{ + *nr += nr_recorded_pages; + SetPageReferenced(head); +} + #ifdef CONFIG_ARCH_HAS_HUGEPD static unsigned long hugepte_addr_end(unsigned long addr, unsigned long end, unsigned long sz) @@ -1998,33 +2026,20 @@ static int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr, /* hugepages are never "special" */ VM_BUG_ON(!pfn_valid(pte_pfn(pte))); - refs = 0; head = pte_page(pte); - page = head + ((addr & (sz-1)) >> PAGE_SHIFT); - do { - VM_BUG_ON(compound_head(page) != head); - pages[*nr] = page; - (*nr)++; - page++; - refs++; - } while (addr += PAGE_SIZE, addr != end); + refs = __record_subpages(page, addr, end, pages, *nr); head = try_get_compound_head(head, refs); - if (!head) { - *nr -= refs; + if (!head) return 0; - } if (unlikely(pte_val(pte) != pte_val(*ptep))) { - /* Could be optimized better */ - *nr -= refs; - while (refs--) - put_page(head); + put_compound_head(head, refs); return 0; } - SetPageReferenced(head); + __huge_pt_done(head, refs, nr); return 1; } @@ -2071,29 +2086,19 @@ static int gup_huge_pmd(pmd_t orig, pmd_t *pmdp, unsigned long addr, pages, nr); } - refs = 0; page = pmd_page(orig) + ((addr & ~PMD_MASK) >> PAGE_SHIFT); - do { - pages[*nr] = page; - (*nr)++; - page++; - refs++; - } while (addr += PAGE_SIZE, addr != end); + refs = __record_subpages(page, addr, end, pages, *nr); head = try_get_compound_head(pmd_page(orig), refs); - if (!head) { - *nr -= refs; + if (!head) return 0; - } if (unlikely(pmd_val(orig) != pmd_val(*pmdp))) { - *nr -= refs; - while (refs--) - put_page(head); + put_compound_head(head, refs); return 0; } - SetPageReferenced(head); + __huge_pt_done(head, refs, nr); return 1; } @@ -2114,29 +2119,19 @@ static int gup_huge_pud(pud_t orig, pud_t *pudp, unsigned long addr, pages, nr); } - refs = 0; page = pud_page(orig) + ((addr & ~PUD_MASK) >> PAGE_SHIFT); - do { - pages[*nr] = page; - (*nr)++; - page++; - refs++; - } while (addr += PAGE_SIZE, addr != end); + refs = __record_subpages(page, addr, end, pages, *nr); head = try_get_compound_head(pud_page(orig), refs); - if (!head) { - *nr -= refs; + if (!head) return 0; - } if (unlikely(pud_val(orig) != pud_val(*pudp))) { - *nr -= refs; - while (refs--) - put_page(head); + put_compound_head(head, refs); return 0; } - SetPageReferenced(head); + __huge_pt_done(head, refs, nr); return 1; } @@ -2151,29 +2146,20 @@ static int gup_huge_pgd(pgd_t orig, pgd_t *pgdp, unsigned long addr, return 0; BUILD_BUG_ON(pgd_devmap(orig)); - refs = 0; + page = pgd_page(orig) + ((addr & ~PGDIR_MASK) >> PAGE_SHIFT); - do { - pages[*nr] = page; - (*nr)++; - page++; - refs++; - } while (addr += PAGE_SIZE, addr != end); + refs = __record_subpages(page, addr, end, pages, *nr); head = try_get_compound_head(pgd_page(orig), refs); - if (!head) { - *nr -= refs; + if (!head) return 0; - } if (unlikely(pgd_val(orig) != pgd_val(*pgdp))) { - *nr -= refs; - while (refs--) - put_page(head); + put_compound_head(head, refs); return 0; } - SetPageReferenced(head); + __huge_pt_done(head, refs, nr); return 1; } -- 2.23.0
next prev parent reply index Thread overview: 57+ messages / expand[flat|nested] mbox.gz Atom feed top 2019-11-03 21:17 [PATCH v2 00/18] mm/gup: track dma-pinned pages: FOLL_PIN, FOLL_LONGTERM John Hubbard 2019-11-03 21:17 ` [PATCH v2 01/18] mm/gup: pass flags arg to __gup_device_* functions John Hubbard 2019-11-04 16:39 ` Jerome Glisse 2019-11-03 21:17 ` John Hubbard [this message] 2019-11-04 16:51 ` [PATCH v2 02/18] mm/gup: factor out duplicate code from four routines Jerome Glisse 2019-11-03 21:17 ` [PATCH v2 03/18] goldish_pipe: rename local pin_user_pages() routine John Hubbard 2019-11-04 16:52 ` Jerome Glisse 2019-11-03 21:17 ` [PATCH v2 04/18] media/v4l2-core: set pages dirty upon releasing DMA buffers John Hubbard 2019-11-10 10:10 ` Hans Verkuil 2019-11-11 21:46 ` John Hubbard 2019-11-03 21:18 ` [PATCH v2 05/18] mm/gup: introduce pin_user_pages*() and FOLL_PIN John Hubbard 2019-11-04 17:33 ` Jerome Glisse 2019-11-04 19:04 ` John Hubbard 2019-11-04 19:18 ` Jerome Glisse 2019-11-04 19:30 ` John Hubbard 2019-11-04 19:52 ` Jerome Glisse 2019-11-04 20:09 ` John Hubbard 2019-11-04 20:31 ` Jason Gunthorpe 2019-11-04 20:40 ` John Hubbard 2019-11-04 20:31 ` Jerome Glisse 2019-11-04 20:37 ` Jason Gunthorpe 2019-11-04 20:57 ` John Hubbard 2019-11-04 21:15 ` Jason Gunthorpe 2019-11-04 21:34 ` John Hubbard 2019-11-04 20:33 ` David Rientjes 2019-11-04 20:48 ` Jerome Glisse 2019-11-05 13:10 ` Mike Rapoport 2019-11-05 19:00 ` John Hubbard 2019-11-07 2:25 ` Ira Weiny 2019-11-07 8:07 ` Mike Rapoport 2019-11-03 21:18 ` [PATCH v2 06/18] goldish_pipe: convert to pin_user_pages() and put_user_page() John Hubbard 2019-11-03 21:18 ` [PATCH v2 07/18] infiniband: set FOLL_PIN, FOLL_LONGTERM via pin_longterm_pages*() John Hubbard 2019-11-04 20:33 ` Jason Gunthorpe 2019-11-04 20:48 ` John Hubbard 2019-11-04 20:57 ` Jason Gunthorpe 2019-11-04 22:03 ` John Hubbard 2019-11-05 2:32 ` Jason Gunthorpe 2019-11-07 2:26 ` Ira Weiny 2019-11-03 21:18 ` [PATCH v2 08/18] mm/process_vm_access: set FOLL_PIN via pin_user_pages_remote() John Hubbard 2019-11-04 17:41 ` Jerome Glisse 2019-11-03 21:18 ` [PATCH v2 09/18] drm/via: set FOLL_PIN via pin_user_pages_fast() John Hubbard 2019-11-04 17:44 ` Jerome Glisse 2019-11-04 18:22 ` John Hubbard 2019-11-03 21:18 ` [PATCH v2 10/18] fs/io_uring: set FOLL_PIN via pin_user_pages() John Hubbard 2019-11-03 21:18 ` [PATCH v2 11/18] net/xdp: " John Hubbard 2019-11-03 21:18 ` [PATCH v2 12/18] mm/gup: track FOLL_PIN pages John Hubbard 2019-11-04 18:52 ` Jerome Glisse 2019-11-04 22:49 ` John Hubbard 2019-11-04 23:49 ` Jerome Glisse 2019-11-05 0:18 ` John Hubbard 2019-11-03 21:18 ` [PATCH v2 13/18] media/v4l2-core: pin_longterm_pages (FOLL_PIN) and put_user_page() conversion John Hubbard 2019-11-10 10:11 ` Hans Verkuil 2019-11-03 21:18 ` [PATCH v2 14/18] vfio, mm: " John Hubbard 2019-11-03 21:18 ` [PATCH v2 15/18] powerpc: book3s64: convert to pin_longterm_pages() and put_user_page() John Hubbard 2019-11-03 21:18 ` [PATCH v2 16/18] mm/gup_benchmark: support pin_user_pages() and related calls John Hubbard 2019-11-03 21:18 ` [PATCH v2 17/18] selftests/vm: run_vmtests: invoke gup_benchmark with basic FOLL_PIN coverage John Hubbard 2019-11-03 21:18 ` [PATCH v2 18/18] mm/gup: remove support for gup(FOLL_LONGTERM) John Hubbard
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20191103211813.213227-3-jhubbard@nvidia.com \ --to=jhubbard@nvidia.com \ --cc=airlied@linux.ie \ --cc=akpm@linux-foundation.org \ --cc=alex.williamson@redhat.com \ --cc=aneesh.kumar@linux.ibm.com \ --cc=axboe@kernel.dk \ --cc=benh@kernel.crashing.org \ --cc=bjorn.topel@intel.com \ --cc=bpf@vger.kernel.org \ --cc=corbet@lwn.net \ --cc=dan.j.williams@intel.com \ --cc=daniel@ffwll.ch \ --cc=davem@davemloft.net \ --cc=david@fromorbit.com \ --cc=dri-devel@lists.freedesktop.org \ --cc=hch@infradead.org \ --cc=hch@lst.de \ --cc=ira.weiny@intel.com \ --cc=jack@suse.cz \ --cc=jgg@ziepe.ca \ --cc=jglisse@redhat.com \ --cc=kvm@vger.kernel.org \ --cc=linux-block@vger.kernel.org \ --cc=linux-doc@vger.kernel.org \ --cc=linux-fsdevel@vger.kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-kselftest@vger.kernel.org \ --cc=linux-media@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=linux-rdma@vger.kernel.org \ --cc=linuxppc-dev@lists.ozlabs.org \ --cc=magnus.karlsson@intel.com \ --cc=mchehab@kernel.org \ --cc=mhocko@suse.com \ --cc=mike.kravetz@oracle.com \ --cc=mpe@ellerman.id.au \ --cc=netdev@vger.kernel.org \ --cc=paulus@samba.org \ --cc=shuah@kernel.org \ --cc=vbabka@suse.cz \ --cc=viro@zeniv.linux.org.uk \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
BPF Archive on lore.kernel.org Archives are clonable: git clone --mirror https://lore.kernel.org/bpf/0 bpf/git/0.git # If you have public-inbox 1.1+ installed, you may # initialize and index your mirror using the following commands: public-inbox-init -V2 bpf bpf/ https://lore.kernel.org/bpf \ bpf@vger.kernel.org public-inbox-index bpf Example config snippet for mirrors Newsgroup available over NNTP: nntp://nntp.lore.kernel.org/org.kernel.vger.bpf AGPL code for this site: git clone https://public-inbox.org/public-inbox.git