From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE520C43461 for ; Tue, 15 Sep 2020 17:42:29 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 583D120936 for ; Tue, 15 Sep 2020 17:42:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1600191749; bh=Wut+/LKuurJjV6yWdQ0LkAiD7T7c8wwiKIpWcqWDTyQ=; h=Date:From:To:Cc:Subject:References:In-Reply-To:List-ID:From; b=TujMjvYkjfq13GV6Jq/PlwPD3b851f/GsRG7jCU0TCVH1w73zM7hB4UHsvNmrH+t7 cxgxeMw6V/JRmjsxf+qkLZARrAbK2Q8amTPyWOVIPmcCvE0IV6oqLjqqjesJOtP+WC A5Y947zV5lG9RrTaKknzrNCPxBA9X6aTyG5R0ePs= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727895AbgIORmZ (ORCPT ); Tue, 15 Sep 2020 13:42:25 -0400 Received: from mail.kernel.org ([198.145.29.99]:51006 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727753AbgIORSj (ORCPT ); Tue, 15 Sep 2020 13:18:39 -0400 Received: from kernel.org (unknown [87.71.73.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 33F3520872; Tue, 15 Sep 2020 17:18:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1600190318; bh=Wut+/LKuurJjV6yWdQ0LkAiD7T7c8wwiKIpWcqWDTyQ=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=YeUxScuVHsoRcQepFJGfs1FFAf2e1LA1+mSzjWkJRc8NB4Y0dZ0Yq79Hxs/lrBnxk pVbp2fKt9JCX5uVPX6sH97oaOJOt5ijiHPqC0TWMYPs7q/pJCrcD7UjG64D4R6O2M1 qX+5OQ6Q4SltpWyluJLoWn8FICTrQN6EcS6Jp4sg= Date: Tue, 15 Sep 2020 20:18:23 +0300 From: Mike Rapoport To: Vasily Gorbik Cc: Jason Gunthorpe , John Hubbard , Linus Torvalds , Gerald Schaefer , Alexander Gordeev , Peter Zijlstra , Dave Hansen , LKML , linux-mm , linux-arch , Andrew Morton , Russell King , Catalin Marinas , Will Deacon , Michael Ellerman , Benjamin Herrenschmidt , Paul Mackerras , Jeff Dike , Richard Weinberger , Dave Hansen , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Arnd Bergmann , Andrey Ryabinin , linux-x86 , linux-arm , linux-power , linux-sparc , linux-um , linux-s390 , Heiko Carstens , Christian Borntraeger , Claudio Imbrenda Subject: Re: [PATCH v2] mm/gup: fix gup_fast with dynamic page table folding Message-ID: <20200915171823.GJ2142832@kernel.org> References: <20200911200511.GC1221970@ziepe.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, Sep 11, 2020 at 10:36:43PM +0200, Vasily Gorbik wrote: > Currently to make sure that every page table entry is read just once > gup_fast walks perform READ_ONCE and pass pXd value down to the next > gup_pXd_range function by value e.g.: > > static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end, > unsigned int flags, struct page **pages, int *nr) > ... > pudp = pud_offset(&p4d, addr); > > This function passes a reference on that local value copy to pXd_offset, > and might get the very same pointer in return. This happens when the > level is folded (on most arches), and that pointer should not be iterated. > > On s390 due to the fact that each task might have different 5,4 or > 3-level address translation and hence different levels folded the logic > is more complex and non-iteratable pointer to a local copy leads to > severe problems. > > Here is an example of what happens with gup_fast on s390, for a task > with 3-levels paging, crossing a 2 GB pud boundary: > > // addr = 0x1007ffff000, end = 0x10080001000 > static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end, > unsigned int flags, struct page **pages, int *nr) > { > unsigned long next; > pud_t *pudp; > > // pud_offset returns &p4d itself (a pointer to a value on stack) > pudp = pud_offset(&p4d, addr); > do { > // on second iteratation reading "random" stack value > pud_t pud = READ_ONCE(*pudp); > > // next = 0x10080000000, due to PUD_SIZE/MASK != PGDIR_SIZE/MASK on s390 > next = pud_addr_end(addr, end); > ... > } while (pudp++, addr = next, addr != end); // pudp++ iterating over stack > > return 1; > } > > This happens since s390 moved to common gup code with > commit d1874a0c2805 ("s390/mm: make the pxd_offset functions more robust") > and commit 1a42010cdc26 ("s390/mm: convert to the generic > get_user_pages_fast code"). s390 tried to mimic static level folding by > changing pXd_offset primitives to always calculate top level page table > offset in pgd_offset and just return the value passed when pXd_offset > has to act as folded. > > What is crucial for gup_fast and what has been overlooked is > that PxD_SIZE/MASK and thus pXd_addr_end should also change > correspondingly. And the latter is not possible with dynamic folding. > > To fix the issue in addition to pXd values pass original > pXdp pointers down to gup_pXd_range functions. And introduce > pXd_offset_lockless helpers, which take an additional pXd > entry value parameter. This has already been discussed in > https://lkml.kernel.org/r/20190418100218.0a4afd51@mschwideX1 > > Cc: # 5.2+ > Fixes: 1a42010cdc26 ("s390/mm: convert to the generic get_user_pages_fast code") > Reviewed-by: Gerald Schaefer > Reviewed-by: Alexander Gordeev > Signed-off-by: Vasily Gorbik Reviewed-by: Mike Rapoport > --- > v2: added brackets &pgd -> &(pgd) > > arch/s390/include/asm/pgtable.h | 42 +++++++++++++++++++++++---------- > include/linux/pgtable.h | 10 ++++++++ > mm/gup.c | 18 +++++++------- > 3 files changed, 49 insertions(+), 21 deletions(-) > > diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h > index 7eb01a5459cd..b55561cc8786 100644 > --- a/arch/s390/include/asm/pgtable.h > +++ b/arch/s390/include/asm/pgtable.h > @@ -1260,26 +1260,44 @@ static inline pgd_t *pgd_offset_raw(pgd_t *pgd, unsigned long address) > > #define pgd_offset(mm, address) pgd_offset_raw(READ_ONCE((mm)->pgd), address) > > -static inline p4d_t *p4d_offset(pgd_t *pgd, unsigned long address) > +static inline p4d_t *p4d_offset_lockless(pgd_t *pgdp, pgd_t pgd, unsigned long address) > { > - if ((pgd_val(*pgd) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R1) > - return (p4d_t *) pgd_deref(*pgd) + p4d_index(address); > - return (p4d_t *) pgd; > + if ((pgd_val(pgd) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R1) > + return (p4d_t *) pgd_deref(pgd) + p4d_index(address); > + return (p4d_t *) pgdp; > } > +#define p4d_offset_lockless p4d_offset_lockless > > -static inline pud_t *pud_offset(p4d_t *p4d, unsigned long address) > +static inline p4d_t *p4d_offset(pgd_t *pgdp, unsigned long address) > { > - if ((p4d_val(*p4d) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R2) > - return (pud_t *) p4d_deref(*p4d) + pud_index(address); > - return (pud_t *) p4d; > + return p4d_offset_lockless(pgdp, *pgdp, address); > +} > + > +static inline pud_t *pud_offset_lockless(p4d_t *p4dp, p4d_t p4d, unsigned long address) > +{ > + if ((p4d_val(p4d) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R2) > + return (pud_t *) p4d_deref(p4d) + pud_index(address); > + return (pud_t *) p4dp; > +} > +#define pud_offset_lockless pud_offset_lockless > + > +static inline pud_t *pud_offset(p4d_t *p4dp, unsigned long address) > +{ > + return pud_offset_lockless(p4dp, *p4dp, address); > } > #define pud_offset pud_offset > > -static inline pmd_t *pmd_offset(pud_t *pud, unsigned long address) > +static inline pmd_t *pmd_offset_lockless(pud_t *pudp, pud_t pud, unsigned long address) > +{ > + if ((pud_val(pud) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R3) > + return (pmd_t *) pud_deref(pud) + pmd_index(address); > + return (pmd_t *) pudp; > +} > +#define pmd_offset_lockless pmd_offset_lockless > + > +static inline pmd_t *pmd_offset(pud_t *pudp, unsigned long address) > { > - if ((pud_val(*pud) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R3) > - return (pmd_t *) pud_deref(*pud) + pmd_index(address); > - return (pmd_t *) pud; > + return pmd_offset_lockless(pudp, *pudp, address); > } > #define pmd_offset pmd_offset > > diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h > index e8cbc2e795d5..90654cb63e9e 100644 > --- a/include/linux/pgtable.h > +++ b/include/linux/pgtable.h > @@ -1427,6 +1427,16 @@ typedef unsigned int pgtbl_mod_mask; > #define mm_pmd_folded(mm) __is_defined(__PAGETABLE_PMD_FOLDED) > #endif > > +#ifndef p4d_offset_lockless > +#define p4d_offset_lockless(pgdp, pgd, address) p4d_offset(&(pgd), address) > +#endif > +#ifndef pud_offset_lockless > +#define pud_offset_lockless(p4dp, p4d, address) pud_offset(&(p4d), address) > +#endif > +#ifndef pmd_offset_lockless > +#define pmd_offset_lockless(pudp, pud, address) pmd_offset(&(pud), address) > +#endif > + > /* > * p?d_leaf() - true if this entry is a final mapping to a physical address. > * This differs from p?d_huge() by the fact that they are always available (if > diff --git a/mm/gup.c b/mm/gup.c > index e5739a1974d5..578bf5bd8bf8 100644 > --- a/mm/gup.c > +++ b/mm/gup.c > @@ -2485,13 +2485,13 @@ static int gup_huge_pgd(pgd_t orig, pgd_t *pgdp, unsigned long addr, > return 1; > } > > -static int gup_pmd_range(pud_t pud, unsigned long addr, unsigned long end, > +static int gup_pmd_range(pud_t *pudp, pud_t pud, unsigned long addr, unsigned long end, > unsigned int flags, struct page **pages, int *nr) > { > unsigned long next; > pmd_t *pmdp; > > - pmdp = pmd_offset(&pud, addr); > + pmdp = pmd_offset_lockless(pudp, pud, addr); > do { > pmd_t pmd = READ_ONCE(*pmdp); > > @@ -2528,13 +2528,13 @@ static int gup_pmd_range(pud_t pud, unsigned long addr, unsigned long end, > return 1; > } > > -static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end, > +static int gup_pud_range(p4d_t *p4dp, p4d_t p4d, unsigned long addr, unsigned long end, > unsigned int flags, struct page **pages, int *nr) > { > unsigned long next; > pud_t *pudp; > > - pudp = pud_offset(&p4d, addr); > + pudp = pud_offset_lockless(p4dp, p4d, addr); > do { > pud_t pud = READ_ONCE(*pudp); > > @@ -2549,20 +2549,20 @@ static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end, > if (!gup_huge_pd(__hugepd(pud_val(pud)), addr, > PUD_SHIFT, next, flags, pages, nr)) > return 0; > - } else if (!gup_pmd_range(pud, addr, next, flags, pages, nr)) > + } else if (!gup_pmd_range(pudp, pud, addr, next, flags, pages, nr)) > return 0; > } while (pudp++, addr = next, addr != end); > > return 1; > } > > -static int gup_p4d_range(pgd_t pgd, unsigned long addr, unsigned long end, > +static int gup_p4d_range(pgd_t *pgdp, pgd_t pgd, unsigned long addr, unsigned long end, > unsigned int flags, struct page **pages, int *nr) > { > unsigned long next; > p4d_t *p4dp; > > - p4dp = p4d_offset(&pgd, addr); > + p4dp = p4d_offset_lockless(pgdp, pgd, addr); > do { > p4d_t p4d = READ_ONCE(*p4dp); > > @@ -2574,7 +2574,7 @@ static int gup_p4d_range(pgd_t pgd, unsigned long addr, unsigned long end, > if (!gup_huge_pd(__hugepd(p4d_val(p4d)), addr, > P4D_SHIFT, next, flags, pages, nr)) > return 0; > - } else if (!gup_pud_range(p4d, addr, next, flags, pages, nr)) > + } else if (!gup_pud_range(p4dp, p4d, addr, next, flags, pages, nr)) > return 0; > } while (p4dp++, addr = next, addr != end); > > @@ -2602,7 +2602,7 @@ static void gup_pgd_range(unsigned long addr, unsigned long end, > if (!gup_huge_pd(__hugepd(pgd_val(pgd)), addr, > PGDIR_SHIFT, next, flags, pages, nr)) > return; > - } else if (!gup_p4d_range(pgd, addr, next, flags, pages, nr)) > + } else if (!gup_p4d_range(pgdp, pgd, addr, next, flags, pages, nr)) > return; > } while (pgdp++, addr = next, addr != end); > } > -- > ⣿⣿⣿⣿⢋⡀⣀⠹⣿⣿⣿⣿ > ⣿⣿⣿⣿⠠⣶⡦⠀⣿⣿⣿⣿ > ⣿⣿⣿⠏⣴⣮⣴⣧⠈⢿⣿⣿ > ⣿⣿⡏⢰⣿⠖⣠⣿⡆⠈⣿⣿ > ⣿⢛⣵⣄⠙⣶⣶⡟⣅⣠⠹⣿ > ⣿⣜⣛⠻⢎⣉⣉⣀⠿⣫⣵⣿ -- Sincerely yours, Mike. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Mike Rapoport Date: Tue, 15 Sep 2020 17:18:23 +0000 Subject: Re: [PATCH v2] mm/gup: fix gup_fast with dynamic page table folding Message-Id: <20200915171823.GJ2142832@kernel.org> List-Id: References: <20200911200511.GC1221970@ziepe.ca> In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit To: Vasily Gorbik Cc: Jason Gunthorpe , John Hubbard , Linus Torvalds , Gerald Schaefer , Alexander Gordeev , Peter Zijlstra , Dave Hansen , LKML , linux-mm , linux-arch , Andrew Morton , Russell King , Catalin Marinas , Will Deacon , Michael Ellerman , Benjamin Herrenschmidt , Paul Mackerras , Jeff Dike , Richard Weinberger , Dave Hansen , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Arnd Bergmann , Andrey Ryabinin , linux-x86 , linux-arm , linux-power , linux-sparc , linux-um , linux-s390 , Heiko Carstens , Christian Borntraeger , Claudio Imbrenda On Fri, Sep 11, 2020 at 10:36:43PM +0200, Vasily Gorbik wrote: > Currently to make sure that every page table entry is read just once > gup_fast walks perform READ_ONCE and pass pXd value down to the next > gup_pXd_range function by value e.g.: > > static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end, > unsigned int flags, struct page **pages, int *nr) > ... > pudp = pud_offset(&p4d, addr); > > This function passes a reference on that local value copy to pXd_offset, > and might get the very same pointer in return. This happens when the > level is folded (on most arches), and that pointer should not be iterated. > > On s390 due to the fact that each task might have different 5,4 or > 3-level address translation and hence different levels folded the logic > is more complex and non-iteratable pointer to a local copy leads to > severe problems. > > Here is an example of what happens with gup_fast on s390, for a task > with 3-levels paging, crossing a 2 GB pud boundary: > > // addr = 0x1007ffff000, end = 0x10080001000 > static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end, > unsigned int flags, struct page **pages, int *nr) > { > unsigned long next; > pud_t *pudp; > > // pud_offset returns &p4d itself (a pointer to a value on stack) > pudp = pud_offset(&p4d, addr); > do { > // on second iteratation reading "random" stack value > pud_t pud = READ_ONCE(*pudp); > > // next = 0x10080000000, due to PUD_SIZE/MASK != PGDIR_SIZE/MASK on s390 > next = pud_addr_end(addr, end); > ... > } while (pudp++, addr = next, addr != end); // pudp++ iterating over stack > > return 1; > } > > This happens since s390 moved to common gup code with > commit d1874a0c2805 ("s390/mm: make the pxd_offset functions more robust") > and commit 1a42010cdc26 ("s390/mm: convert to the generic > get_user_pages_fast code"). s390 tried to mimic static level folding by > changing pXd_offset primitives to always calculate top level page table > offset in pgd_offset and just return the value passed when pXd_offset > has to act as folded. > > What is crucial for gup_fast and what has been overlooked is > that PxD_SIZE/MASK and thus pXd_addr_end should also change > correspondingly. And the latter is not possible with dynamic folding. > > To fix the issue in addition to pXd values pass original > pXdp pointers down to gup_pXd_range functions. And introduce > pXd_offset_lockless helpers, which take an additional pXd > entry value parameter. This has already been discussed in > https://lkml.kernel.org/r/20190418100218.0a4afd51@mschwideX1 > > Cc: # 5.2+ > Fixes: 1a42010cdc26 ("s390/mm: convert to the generic get_user_pages_fast code") > Reviewed-by: Gerald Schaefer > Reviewed-by: Alexander Gordeev > Signed-off-by: Vasily Gorbik Reviewed-by: Mike Rapoport > --- > v2: added brackets &pgd -> &(pgd) > > arch/s390/include/asm/pgtable.h | 42 +++++++++++++++++++++++---------- > include/linux/pgtable.h | 10 ++++++++ > mm/gup.c | 18 +++++++------- > 3 files changed, 49 insertions(+), 21 deletions(-) > > diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h > index 7eb01a5459cd..b55561cc8786 100644 > --- a/arch/s390/include/asm/pgtable.h > +++ b/arch/s390/include/asm/pgtable.h > @@ -1260,26 +1260,44 @@ static inline pgd_t *pgd_offset_raw(pgd_t *pgd, unsigned long address) > > #define pgd_offset(mm, address) pgd_offset_raw(READ_ONCE((mm)->pgd), address) > > -static inline p4d_t *p4d_offset(pgd_t *pgd, unsigned long address) > +static inline p4d_t *p4d_offset_lockless(pgd_t *pgdp, pgd_t pgd, unsigned long address) > { > - if ((pgd_val(*pgd) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R1) > - return (p4d_t *) pgd_deref(*pgd) + p4d_index(address); > - return (p4d_t *) pgd; > + if ((pgd_val(pgd) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R1) > + return (p4d_t *) pgd_deref(pgd) + p4d_index(address); > + return (p4d_t *) pgdp; > } > +#define p4d_offset_lockless p4d_offset_lockless > > -static inline pud_t *pud_offset(p4d_t *p4d, unsigned long address) > +static inline p4d_t *p4d_offset(pgd_t *pgdp, unsigned long address) > { > - if ((p4d_val(*p4d) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R2) > - return (pud_t *) p4d_deref(*p4d) + pud_index(address); > - return (pud_t *) p4d; > + return p4d_offset_lockless(pgdp, *pgdp, address); > +} > + > +static inline pud_t *pud_offset_lockless(p4d_t *p4dp, p4d_t p4d, unsigned long address) > +{ > + if ((p4d_val(p4d) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R2) > + return (pud_t *) p4d_deref(p4d) + pud_index(address); > + return (pud_t *) p4dp; > +} > +#define pud_offset_lockless pud_offset_lockless > + > +static inline pud_t *pud_offset(p4d_t *p4dp, unsigned long address) > +{ > + return pud_offset_lockless(p4dp, *p4dp, address); > } > #define pud_offset pud_offset > > -static inline pmd_t *pmd_offset(pud_t *pud, unsigned long address) > +static inline pmd_t *pmd_offset_lockless(pud_t *pudp, pud_t pud, unsigned long address) > +{ > + if ((pud_val(pud) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R3) > + return (pmd_t *) pud_deref(pud) + pmd_index(address); > + return (pmd_t *) pudp; > +} > +#define pmd_offset_lockless pmd_offset_lockless > + > +static inline pmd_t *pmd_offset(pud_t *pudp, unsigned long address) > { > - if ((pud_val(*pud) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R3) > - return (pmd_t *) pud_deref(*pud) + pmd_index(address); > - return (pmd_t *) pud; > + return pmd_offset_lockless(pudp, *pudp, address); > } > #define pmd_offset pmd_offset > > diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h > index e8cbc2e795d5..90654cb63e9e 100644 > --- a/include/linux/pgtable.h > +++ b/include/linux/pgtable.h > @@ -1427,6 +1427,16 @@ typedef unsigned int pgtbl_mod_mask; > #define mm_pmd_folded(mm) __is_defined(__PAGETABLE_PMD_FOLDED) > #endif > > +#ifndef p4d_offset_lockless > +#define p4d_offset_lockless(pgdp, pgd, address) p4d_offset(&(pgd), address) > +#endif > +#ifndef pud_offset_lockless > +#define pud_offset_lockless(p4dp, p4d, address) pud_offset(&(p4d), address) > +#endif > +#ifndef pmd_offset_lockless > +#define pmd_offset_lockless(pudp, pud, address) pmd_offset(&(pud), address) > +#endif > + > /* > * p?d_leaf() - true if this entry is a final mapping to a physical address. > * This differs from p?d_huge() by the fact that they are always available (if > diff --git a/mm/gup.c b/mm/gup.c > index e5739a1974d5..578bf5bd8bf8 100644 > --- a/mm/gup.c > +++ b/mm/gup.c > @@ -2485,13 +2485,13 @@ static int gup_huge_pgd(pgd_t orig, pgd_t *pgdp, unsigned long addr, > return 1; > } > > -static int gup_pmd_range(pud_t pud, unsigned long addr, unsigned long end, > +static int gup_pmd_range(pud_t *pudp, pud_t pud, unsigned long addr, unsigned long end, > unsigned int flags, struct page **pages, int *nr) > { > unsigned long next; > pmd_t *pmdp; > > - pmdp = pmd_offset(&pud, addr); > + pmdp = pmd_offset_lockless(pudp, pud, addr); > do { > pmd_t pmd = READ_ONCE(*pmdp); > > @@ -2528,13 +2528,13 @@ static int gup_pmd_range(pud_t pud, unsigned long addr, unsigned long end, > return 1; > } > > -static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end, > +static int gup_pud_range(p4d_t *p4dp, p4d_t p4d, unsigned long addr, unsigned long end, > unsigned int flags, struct page **pages, int *nr) > { > unsigned long next; > pud_t *pudp; > > - pudp = pud_offset(&p4d, addr); > + pudp = pud_offset_lockless(p4dp, p4d, addr); > do { > pud_t pud = READ_ONCE(*pudp); > > @@ -2549,20 +2549,20 @@ static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end, > if (!gup_huge_pd(__hugepd(pud_val(pud)), addr, > PUD_SHIFT, next, flags, pages, nr)) > return 0; > - } else if (!gup_pmd_range(pud, addr, next, flags, pages, nr)) > + } else if (!gup_pmd_range(pudp, pud, addr, next, flags, pages, nr)) > return 0; > } while (pudp++, addr = next, addr != end); > > return 1; > } > > -static int gup_p4d_range(pgd_t pgd, unsigned long addr, unsigned long end, > +static int gup_p4d_range(pgd_t *pgdp, pgd_t pgd, unsigned long addr, unsigned long end, > unsigned int flags, struct page **pages, int *nr) > { > unsigned long next; > p4d_t *p4dp; > > - p4dp = p4d_offset(&pgd, addr); > + p4dp = p4d_offset_lockless(pgdp, pgd, addr); > do { > p4d_t p4d = READ_ONCE(*p4dp); > > @@ -2574,7 +2574,7 @@ static int gup_p4d_range(pgd_t pgd, unsigned long addr, unsigned long end, > if (!gup_huge_pd(__hugepd(p4d_val(p4d)), addr, > P4D_SHIFT, next, flags, pages, nr)) > return 0; > - } else if (!gup_pud_range(p4d, addr, next, flags, pages, nr)) > + } else if (!gup_pud_range(p4dp, p4d, addr, next, flags, pages, nr)) > return 0; > } while (p4dp++, addr = next, addr != end); > > @@ -2602,7 +2602,7 @@ static void gup_pgd_range(unsigned long addr, unsigned long end, > if (!gup_huge_pd(__hugepd(pgd_val(pgd)), addr, > PGDIR_SHIFT, next, flags, pages, nr)) > return; > - } else if (!gup_p4d_range(pgd, addr, next, flags, pages, nr)) > + } else if (!gup_p4d_range(pgdp, pgd, addr, next, flags, pages, nr)) > return; > } while (pgdp++, addr = next, addr != end); > } > -- > ⣿⣿⣿⣿⢋⡀⣀⠹⣿⣿⣿⣿ > ⣿⣿⣿⣿⠠⣶⡦⠀⣿⣿⣿⣿ > ⣿⣿⣿⠏⣴⣮⣴⣧⠈⢿⣿⣿ > ⣿⣿⡏⢰⣿⠖⣠⣿⡆⠈⣿⣿ > ⣿⢛⣵⣄⠙⣶⣶⡟⣅⣠⠹⣿ > ⣿⣜⣛⠻⢎⣉⣉⣀⠿⣫⣵⣿ -- Sincerely yours, Mike. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.8 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 69029C433E2 for ; Tue, 15 Sep 2020 17:20:49 +0000 (UTC) Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 87C25205CB for ; Tue, 15 Sep 2020 17:20:48 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="YeUxScuV" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 87C25205CB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4BrVP91ZyhzDqNJ for ; Wed, 16 Sep 2020 03:20:45 +1000 (AEST) Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=kernel.org (client-ip=198.145.29.99; helo=mail.kernel.org; envelope-from=rppt@kernel.org; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=kernel.org Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=default header.b=YeUxScuV; dkim-atps=neutral Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4BrVLn3fSJzDqNJ for ; Wed, 16 Sep 2020 03:18:41 +1000 (AEST) Received: from kernel.org (unknown [87.71.73.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 33F3520872; Tue, 15 Sep 2020 17:18:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1600190318; bh=Wut+/LKuurJjV6yWdQ0LkAiD7T7c8wwiKIpWcqWDTyQ=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=YeUxScuVHsoRcQepFJGfs1FFAf2e1LA1+mSzjWkJRc8NB4Y0dZ0Yq79Hxs/lrBnxk pVbp2fKt9JCX5uVPX6sH97oaOJOt5ijiHPqC0TWMYPs7q/pJCrcD7UjG64D4R6O2M1 qX+5OQ6Q4SltpWyluJLoWn8FICTrQN6EcS6Jp4sg= Date: Tue, 15 Sep 2020 20:18:23 +0300 From: Mike Rapoport To: Vasily Gorbik Subject: Re: [PATCH v2] mm/gup: fix gup_fast with dynamic page table folding Message-ID: <20200915171823.GJ2142832@kernel.org> References: <20200911200511.GC1221970@ziepe.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Zijlstra , Dave Hansen , Dave Hansen , Paul Mackerras , linux-sparc , Alexander Gordeev , Claudio Imbrenda , Will Deacon , linux-arch , linux-s390 , Christian Borntraeger , Richard Weinberger , linux-x86 , Russell King , Jason Gunthorpe , Ingo Molnar , Catalin Marinas , Andrey Ryabinin , Gerald Schaefer , Heiko Carstens , Arnd Bergmann , John Hubbard , Jeff Dike , linux-um , Borislav Petkov , Andy Lutomirski , Thomas Gleixner , linux-arm , linux-mm , linux-power , LKML , Andrew Morton , Linus Torvalds Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" On Fri, Sep 11, 2020 at 10:36:43PM +0200, Vasily Gorbik wrote: > Currently to make sure that every page table entry is read just once > gup_fast walks perform READ_ONCE and pass pXd value down to the next > gup_pXd_range function by value e.g.: > > static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end, > unsigned int flags, struct page **pages, int *nr) > ... > pudp = pud_offset(&p4d, addr); > > This function passes a reference on that local value copy to pXd_offset, > and might get the very same pointer in return. This happens when the > level is folded (on most arches), and that pointer should not be iterated. > > On s390 due to the fact that each task might have different 5,4 or > 3-level address translation and hence different levels folded the logic > is more complex and non-iteratable pointer to a local copy leads to > severe problems. > > Here is an example of what happens with gup_fast on s390, for a task > with 3-levels paging, crossing a 2 GB pud boundary: > > // addr = 0x1007ffff000, end = 0x10080001000 > static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end, > unsigned int flags, struct page **pages, int *nr) > { > unsigned long next; > pud_t *pudp; > > // pud_offset returns &p4d itself (a pointer to a value on stack) > pudp = pud_offset(&p4d, addr); > do { > // on second iteratation reading "random" stack value > pud_t pud = READ_ONCE(*pudp); > > // next = 0x10080000000, due to PUD_SIZE/MASK != PGDIR_SIZE/MASK on s390 > next = pud_addr_end(addr, end); > ... > } while (pudp++, addr = next, addr != end); // pudp++ iterating over stack > > return 1; > } > > This happens since s390 moved to common gup code with > commit d1874a0c2805 ("s390/mm: make the pxd_offset functions more robust") > and commit 1a42010cdc26 ("s390/mm: convert to the generic > get_user_pages_fast code"). s390 tried to mimic static level folding by > changing pXd_offset primitives to always calculate top level page table > offset in pgd_offset and just return the value passed when pXd_offset > has to act as folded. > > What is crucial for gup_fast and what has been overlooked is > that PxD_SIZE/MASK and thus pXd_addr_end should also change > correspondingly. And the latter is not possible with dynamic folding. > > To fix the issue in addition to pXd values pass original > pXdp pointers down to gup_pXd_range functions. And introduce > pXd_offset_lockless helpers, which take an additional pXd > entry value parameter. This has already been discussed in > https://lkml.kernel.org/r/20190418100218.0a4afd51@mschwideX1 > > Cc: # 5.2+ > Fixes: 1a42010cdc26 ("s390/mm: convert to the generic get_user_pages_fast code") > Reviewed-by: Gerald Schaefer > Reviewed-by: Alexander Gordeev > Signed-off-by: Vasily Gorbik Reviewed-by: Mike Rapoport > --- > v2: added brackets &pgd -> &(pgd) > > arch/s390/include/asm/pgtable.h | 42 +++++++++++++++++++++++---------- > include/linux/pgtable.h | 10 ++++++++ > mm/gup.c | 18 +++++++------- > 3 files changed, 49 insertions(+), 21 deletions(-) > > diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h > index 7eb01a5459cd..b55561cc8786 100644 > --- a/arch/s390/include/asm/pgtable.h > +++ b/arch/s390/include/asm/pgtable.h > @@ -1260,26 +1260,44 @@ static inline pgd_t *pgd_offset_raw(pgd_t *pgd, unsigned long address) > > #define pgd_offset(mm, address) pgd_offset_raw(READ_ONCE((mm)->pgd), address) > > -static inline p4d_t *p4d_offset(pgd_t *pgd, unsigned long address) > +static inline p4d_t *p4d_offset_lockless(pgd_t *pgdp, pgd_t pgd, unsigned long address) > { > - if ((pgd_val(*pgd) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R1) > - return (p4d_t *) pgd_deref(*pgd) + p4d_index(address); > - return (p4d_t *) pgd; > + if ((pgd_val(pgd) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R1) > + return (p4d_t *) pgd_deref(pgd) + p4d_index(address); > + return (p4d_t *) pgdp; > } > +#define p4d_offset_lockless p4d_offset_lockless > > -static inline pud_t *pud_offset(p4d_t *p4d, unsigned long address) > +static inline p4d_t *p4d_offset(pgd_t *pgdp, unsigned long address) > { > - if ((p4d_val(*p4d) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R2) > - return (pud_t *) p4d_deref(*p4d) + pud_index(address); > - return (pud_t *) p4d; > + return p4d_offset_lockless(pgdp, *pgdp, address); > +} > + > +static inline pud_t *pud_offset_lockless(p4d_t *p4dp, p4d_t p4d, unsigned long address) > +{ > + if ((p4d_val(p4d) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R2) > + return (pud_t *) p4d_deref(p4d) + pud_index(address); > + return (pud_t *) p4dp; > +} > +#define pud_offset_lockless pud_offset_lockless > + > +static inline pud_t *pud_offset(p4d_t *p4dp, unsigned long address) > +{ > + return pud_offset_lockless(p4dp, *p4dp, address); > } > #define pud_offset pud_offset > > -static inline pmd_t *pmd_offset(pud_t *pud, unsigned long address) > +static inline pmd_t *pmd_offset_lockless(pud_t *pudp, pud_t pud, unsigned long address) > +{ > + if ((pud_val(pud) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R3) > + return (pmd_t *) pud_deref(pud) + pmd_index(address); > + return (pmd_t *) pudp; > +} > +#define pmd_offset_lockless pmd_offset_lockless > + > +static inline pmd_t *pmd_offset(pud_t *pudp, unsigned long address) > { > - if ((pud_val(*pud) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R3) > - return (pmd_t *) pud_deref(*pud) + pmd_index(address); > - return (pmd_t *) pud; > + return pmd_offset_lockless(pudp, *pudp, address); > } > #define pmd_offset pmd_offset > > diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h > index e8cbc2e795d5..90654cb63e9e 100644 > --- a/include/linux/pgtable.h > +++ b/include/linux/pgtable.h > @@ -1427,6 +1427,16 @@ typedef unsigned int pgtbl_mod_mask; > #define mm_pmd_folded(mm) __is_defined(__PAGETABLE_PMD_FOLDED) > #endif > > +#ifndef p4d_offset_lockless > +#define p4d_offset_lockless(pgdp, pgd, address) p4d_offset(&(pgd), address) > +#endif > +#ifndef pud_offset_lockless > +#define pud_offset_lockless(p4dp, p4d, address) pud_offset(&(p4d), address) > +#endif > +#ifndef pmd_offset_lockless > +#define pmd_offset_lockless(pudp, pud, address) pmd_offset(&(pud), address) > +#endif > + > /* > * p?d_leaf() - true if this entry is a final mapping to a physical address. > * This differs from p?d_huge() by the fact that they are always available (if > diff --git a/mm/gup.c b/mm/gup.c > index e5739a1974d5..578bf5bd8bf8 100644 > --- a/mm/gup.c > +++ b/mm/gup.c > @@ -2485,13 +2485,13 @@ static int gup_huge_pgd(pgd_t orig, pgd_t *pgdp, unsigned long addr, > return 1; > } > > -static int gup_pmd_range(pud_t pud, unsigned long addr, unsigned long end, > +static int gup_pmd_range(pud_t *pudp, pud_t pud, unsigned long addr, unsigned long end, > unsigned int flags, struct page **pages, int *nr) > { > unsigned long next; > pmd_t *pmdp; > > - pmdp = pmd_offset(&pud, addr); > + pmdp = pmd_offset_lockless(pudp, pud, addr); > do { > pmd_t pmd = READ_ONCE(*pmdp); > > @@ -2528,13 +2528,13 @@ static int gup_pmd_range(pud_t pud, unsigned long addr, unsigned long end, > return 1; > } > > -static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end, > +static int gup_pud_range(p4d_t *p4dp, p4d_t p4d, unsigned long addr, unsigned long end, > unsigned int flags, struct page **pages, int *nr) > { > unsigned long next; > pud_t *pudp; > > - pudp = pud_offset(&p4d, addr); > + pudp = pud_offset_lockless(p4dp, p4d, addr); > do { > pud_t pud = READ_ONCE(*pudp); > > @@ -2549,20 +2549,20 @@ static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end, > if (!gup_huge_pd(__hugepd(pud_val(pud)), addr, > PUD_SHIFT, next, flags, pages, nr)) > return 0; > - } else if (!gup_pmd_range(pud, addr, next, flags, pages, nr)) > + } else if (!gup_pmd_range(pudp, pud, addr, next, flags, pages, nr)) > return 0; > } while (pudp++, addr = next, addr != end); > > return 1; > } > > -static int gup_p4d_range(pgd_t pgd, unsigned long addr, unsigned long end, > +static int gup_p4d_range(pgd_t *pgdp, pgd_t pgd, unsigned long addr, unsigned long end, > unsigned int flags, struct page **pages, int *nr) > { > unsigned long next; > p4d_t *p4dp; > > - p4dp = p4d_offset(&pgd, addr); > + p4dp = p4d_offset_lockless(pgdp, pgd, addr); > do { > p4d_t p4d = READ_ONCE(*p4dp); > > @@ -2574,7 +2574,7 @@ static int gup_p4d_range(pgd_t pgd, unsigned long addr, unsigned long end, > if (!gup_huge_pd(__hugepd(p4d_val(p4d)), addr, > P4D_SHIFT, next, flags, pages, nr)) > return 0; > - } else if (!gup_pud_range(p4d, addr, next, flags, pages, nr)) > + } else if (!gup_pud_range(p4dp, p4d, addr, next, flags, pages, nr)) > return 0; > } while (p4dp++, addr = next, addr != end); > > @@ -2602,7 +2602,7 @@ static void gup_pgd_range(unsigned long addr, unsigned long end, > if (!gup_huge_pd(__hugepd(pgd_val(pgd)), addr, > PGDIR_SHIFT, next, flags, pages, nr)) > return; > - } else if (!gup_p4d_range(pgd, addr, next, flags, pages, nr)) > + } else if (!gup_p4d_range(pgdp, pgd, addr, next, flags, pages, nr)) > return; > } while (pgdp++, addr = next, addr != end); > } > -- > ⣿⣿⣿⣿⢋⡀⣀⠹⣿⣿⣿⣿ > ⣿⣿⣿⣿⠠⣶⡦⠀⣿⣿⣿⣿ > ⣿⣿⣿⠏⣴⣮⣴⣧⠈⢿⣿⣿ > ⣿⣿⡏⢰⣿⠖⣠⣿⡆⠈⣿⣿ > ⣿⢛⣵⣄⠙⣶⣶⡟⣅⣠⠹⣿ > ⣿⣜⣛⠻⢎⣉⣉⣀⠿⣫⣵⣿ -- Sincerely yours, Mike. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8A960C433E2 for ; Tue, 15 Sep 2020 17:20:33 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 049D121655 for ; Tue, 15 Sep 2020 17:20:32 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="BHRac4Cs"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="YeUxScuV" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 049D121655 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References:Message-ID: Subject:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=Iw0OHpoXn6nQv5+HdFzbSTNTU2ULR0Qhn14NsZU/y3Y=; b=BHRac4CsGhNwu37UmP0GqlkW5 E6juyduhB4XkQiMRVfIN8UH5WWadsVTW0cjK1r1lgRqPSwKL7vAdjwdMwRFrcJHB0IiltCHhsDGfL NV+6HsU3U4jvmSE0hmQhGM9maxFeGlelNOhptZS3m4D60gZoTYiwMqFPT9dlOVulif1Nk4kBo+viZ IXVQoAvJv+bXfhnPPBXHimh7+PswLrucJrwM5aPHoUtOfrSWm4ZPZPNDg9a+Re75hFQAteRWhjKr4 jY1Sk3bk5+I2KHibpaQ8ScaK8GIb5TvLUyhTJzq+EksoZl2NH+/gYW65sdo4kVVH7QgmjeUarWKfs 5pE8V/zyg==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kIEbO-0000Xv-25; Tue, 15 Sep 2020 17:18:46 +0000 Received: from mail.kernel.org ([198.145.29.99]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kIEbH-0000W5-OR; Tue, 15 Sep 2020 17:18:40 +0000 Received: from kernel.org (unknown [87.71.73.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 33F3520872; Tue, 15 Sep 2020 17:18:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1600190318; bh=Wut+/LKuurJjV6yWdQ0LkAiD7T7c8wwiKIpWcqWDTyQ=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=YeUxScuVHsoRcQepFJGfs1FFAf2e1LA1+mSzjWkJRc8NB4Y0dZ0Yq79Hxs/lrBnxk pVbp2fKt9JCX5uVPX6sH97oaOJOt5ijiHPqC0TWMYPs7q/pJCrcD7UjG64D4R6O2M1 qX+5OQ6Q4SltpWyluJLoWn8FICTrQN6EcS6Jp4sg= Date: Tue, 15 Sep 2020 20:18:23 +0300 From: Mike Rapoport To: Vasily Gorbik Subject: Re: [PATCH v2] mm/gup: fix gup_fast with dynamic page table folding Message-ID: <20200915171823.GJ2142832@kernel.org> References: <20200911200511.GC1221970@ziepe.ca> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200915_131839_987232_408C7DB9 X-CRM114-Status: GOOD ( 36.82 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Zijlstra , Benjamin Herrenschmidt , Dave Hansen , Dave Hansen , Paul Mackerras , linux-sparc , Alexander Gordeev , Claudio Imbrenda , Will Deacon , linux-arch , linux-s390 , Christian Borntraeger , Richard Weinberger , linux-x86 , Russell King , Jason Gunthorpe , Ingo Molnar , Catalin Marinas , Andrey Ryabinin , Gerald Schaefer , Heiko Carstens , Arnd Bergmann , John Hubbard , Jeff Dike , linux-um , Borislav Petkov , Andy Lutomirski , Thomas Gleixner , linux-arm , linux-mm , linux-power , LKML , Michael Ellerman , Andrew Morton , Linus Torvalds Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gRnJpLCBTZXAgMTEsIDIwMjAgYXQgMTA6MzY6NDNQTSArMDIwMCwgVmFzaWx5IEdvcmJpayB3 cm90ZToKPiBDdXJyZW50bHkgdG8gbWFrZSBzdXJlIHRoYXQgZXZlcnkgcGFnZSB0YWJsZSBlbnRy eSBpcyByZWFkIGp1c3Qgb25jZQo+IGd1cF9mYXN0IHdhbGtzIHBlcmZvcm0gUkVBRF9PTkNFIGFu ZCBwYXNzIHBYZCB2YWx1ZSBkb3duIHRvIHRoZSBuZXh0Cj4gZ3VwX3BYZF9yYW5nZSBmdW5jdGlv biBieSB2YWx1ZSBlLmcuOgo+IAo+IHN0YXRpYyBpbnQgZ3VwX3B1ZF9yYW5nZShwNGRfdCBwNGQs IHVuc2lnbmVkIGxvbmcgYWRkciwgdW5zaWduZWQgbG9uZyBlbmQsCj4gICAgICAgICAgICAgICAg ICAgICAgICAgIHVuc2lnbmVkIGludCBmbGFncywgc3RydWN0IHBhZ2UgKipwYWdlcywgaW50ICpu cikKPiAuLi4KPiAgICAgICAgIHB1ZHAgPSBwdWRfb2Zmc2V0KCZwNGQsIGFkZHIpOwo+IAo+IFRo aXMgZnVuY3Rpb24gcGFzc2VzIGEgcmVmZXJlbmNlIG9uIHRoYXQgbG9jYWwgdmFsdWUgY29weSB0 byBwWGRfb2Zmc2V0LAo+IGFuZCBtaWdodCBnZXQgdGhlIHZlcnkgc2FtZSBwb2ludGVyIGluIHJl dHVybi4gVGhpcyBoYXBwZW5zIHdoZW4gdGhlCj4gbGV2ZWwgaXMgZm9sZGVkIChvbiBtb3N0IGFy Y2hlcyksIGFuZCB0aGF0IHBvaW50ZXIgc2hvdWxkIG5vdCBiZSBpdGVyYXRlZC4KPiAKPiBPbiBz MzkwIGR1ZSB0byB0aGUgZmFjdCB0aGF0IGVhY2ggdGFzayBtaWdodCBoYXZlIGRpZmZlcmVudCA1 LDQgb3IKPiAzLWxldmVsIGFkZHJlc3MgdHJhbnNsYXRpb24gYW5kIGhlbmNlIGRpZmZlcmVudCBs ZXZlbHMgZm9sZGVkIHRoZSBsb2dpYwo+IGlzIG1vcmUgY29tcGxleCBhbmQgbm9uLWl0ZXJhdGFi bGUgcG9pbnRlciB0byBhIGxvY2FsIGNvcHkgbGVhZHMgdG8KPiBzZXZlcmUgcHJvYmxlbXMuCj4g Cj4gSGVyZSBpcyBhbiBleGFtcGxlIG9mIHdoYXQgaGFwcGVucyB3aXRoIGd1cF9mYXN0IG9uIHMz OTAsIGZvciBhIHRhc2sKPiB3aXRoIDMtbGV2ZWxzIHBhZ2luZywgY3Jvc3NpbmcgYSAyIEdCIHB1 ZCBib3VuZGFyeToKPiAKPiAvLyBhZGRyID0gMHgxMDA3ZmZmZjAwMCwgZW5kID0gMHgxMDA4MDAw MTAwMAo+IHN0YXRpYyBpbnQgZ3VwX3B1ZF9yYW5nZShwNGRfdCBwNGQsIHVuc2lnbmVkIGxvbmcg YWRkciwgdW5zaWduZWQgbG9uZyBlbmQsCj4gICAgICAgICAgICAgICAgICAgICAgICAgIHVuc2ln bmVkIGludCBmbGFncywgc3RydWN0IHBhZ2UgKipwYWdlcywgaW50ICpucikKPiB7Cj4gICAgICAg ICB1bnNpZ25lZCBsb25nIG5leHQ7Cj4gICAgICAgICBwdWRfdCAqcHVkcDsKPiAKPiAgICAgICAg IC8vIHB1ZF9vZmZzZXQgcmV0dXJucyAmcDRkIGl0c2VsZiAoYSBwb2ludGVyIHRvIGEgdmFsdWUg b24gc3RhY2spCj4gICAgICAgICBwdWRwID0gcHVkX29mZnNldCgmcDRkLCBhZGRyKTsKPiAgICAg ICAgIGRvIHsKPiAgICAgICAgICAgICAgICAgLy8gb24gc2Vjb25kIGl0ZXJhdGF0aW9uIHJlYWRp bmcgInJhbmRvbSIgc3RhY2sgdmFsdWUKPiAgICAgICAgICAgICAgICAgcHVkX3QgcHVkID0gUkVB RF9PTkNFKCpwdWRwKTsKPiAKPiAgICAgICAgICAgICAgICAgLy8gbmV4dCA9IDB4MTAwODAwMDAw MDAsIGR1ZSB0byBQVURfU0laRS9NQVNLICE9IFBHRElSX1NJWkUvTUFTSyBvbiBzMzkwCj4gICAg ICAgICAgICAgICAgIG5leHQgPSBwdWRfYWRkcl9lbmQoYWRkciwgZW5kKTsKPiAgICAgICAgICAg ICAgICAgLi4uCj4gICAgICAgICB9IHdoaWxlIChwdWRwKyssIGFkZHIgPSBuZXh0LCBhZGRyICE9 IGVuZCk7IC8vIHB1ZHArKyBpdGVyYXRpbmcgb3ZlciBzdGFjawo+IAo+ICAgICAgICAgcmV0dXJu IDE7Cj4gfQo+IAo+IFRoaXMgaGFwcGVucyBzaW5jZSBzMzkwIG1vdmVkIHRvIGNvbW1vbiBndXAg Y29kZSB3aXRoCj4gY29tbWl0IGQxODc0YTBjMjgwNSAoInMzOTAvbW06IG1ha2UgdGhlIHB4ZF9v ZmZzZXQgZnVuY3Rpb25zIG1vcmUgcm9idXN0IikKPiBhbmQgY29tbWl0IDFhNDIwMTBjZGMyNiAo InMzOTAvbW06IGNvbnZlcnQgdG8gdGhlIGdlbmVyaWMKPiBnZXRfdXNlcl9wYWdlc19mYXN0IGNv ZGUiKS4gczM5MCB0cmllZCB0byBtaW1pYyBzdGF0aWMgbGV2ZWwgZm9sZGluZyBieQo+IGNoYW5n aW5nIHBYZF9vZmZzZXQgcHJpbWl0aXZlcyB0byBhbHdheXMgY2FsY3VsYXRlIHRvcCBsZXZlbCBw YWdlIHRhYmxlCj4gb2Zmc2V0IGluIHBnZF9vZmZzZXQgYW5kIGp1c3QgcmV0dXJuIHRoZSB2YWx1 ZSBwYXNzZWQgd2hlbiBwWGRfb2Zmc2V0Cj4gaGFzIHRvIGFjdCBhcyBmb2xkZWQuCj4gCj4gV2hh dCBpcyBjcnVjaWFsIGZvciBndXBfZmFzdCBhbmQgd2hhdCBoYXMgYmVlbiBvdmVybG9va2VkIGlz Cj4gdGhhdCBQeERfU0laRS9NQVNLIGFuZCB0aHVzIHBYZF9hZGRyX2VuZCBzaG91bGQgYWxzbyBj aGFuZ2UKPiBjb3JyZXNwb25kaW5nbHkuIEFuZCB0aGUgbGF0dGVyIGlzIG5vdCBwb3NzaWJsZSB3 aXRoIGR5bmFtaWMgZm9sZGluZy4KPiAKPiBUbyBmaXggdGhlIGlzc3VlIGluIGFkZGl0aW9uIHRv IHBYZCB2YWx1ZXMgcGFzcyBvcmlnaW5hbAo+IHBYZHAgcG9pbnRlcnMgZG93biB0byBndXBfcFhk X3JhbmdlIGZ1bmN0aW9ucy4gQW5kIGludHJvZHVjZQo+IHBYZF9vZmZzZXRfbG9ja2xlc3MgaGVs cGVycywgd2hpY2ggdGFrZSBhbiBhZGRpdGlvbmFsIHBYZAo+IGVudHJ5IHZhbHVlIHBhcmFtZXRl ci4gVGhpcyBoYXMgYWxyZWFkeSBiZWVuIGRpc2N1c3NlZCBpbgo+IGh0dHBzOi8vbGttbC5rZXJu ZWwub3JnL3IvMjAxOTA0MTgxMDAyMTguMGE0YWZkNTFAbXNjaHdpZGVYMQo+IAo+IENjOiA8c3Rh YmxlQHZnZXIua2VybmVsLm9yZz4gIyA1LjIrCj4gRml4ZXM6IDFhNDIwMTBjZGMyNiAoInMzOTAv bW06IGNvbnZlcnQgdG8gdGhlIGdlbmVyaWMgZ2V0X3VzZXJfcGFnZXNfZmFzdCBjb2RlIikKPiBS ZXZpZXdlZC1ieTogR2VyYWxkIFNjaGFlZmVyIDxnZXJhbGQuc2NoYWVmZXJAbGludXguaWJtLmNv bT4KPiBSZXZpZXdlZC1ieTogQWxleGFuZGVyIEdvcmRlZXYgPGFnb3JkZWV2QGxpbnV4LmlibS5j b20+Cj4gU2lnbmVkLW9mZi1ieTogVmFzaWx5IEdvcmJpayA8Z29yQGxpbnV4LmlibS5jb20+CgpS ZXZpZXdlZC1ieTogTWlrZSBSYXBvcG9ydCA8cnBwdEBsaW51eC5pYm0uY29tPgoKPiAtLS0KPiB2 MjogYWRkZWQgYnJhY2tldHMgJnBnZCAtPiAmKHBnZCkKPiAKPiAgYXJjaC9zMzkwL2luY2x1ZGUv YXNtL3BndGFibGUuaCB8IDQyICsrKysrKysrKysrKysrKysrKysrKysrLS0tLS0tLS0tLQo+ICBp bmNsdWRlL2xpbnV4L3BndGFibGUuaCAgICAgICAgIHwgMTAgKysrKysrKysKPiAgbW0vZ3VwLmMg ICAgICAgICAgICAgICAgICAgICAgICB8IDE4ICsrKysrKystLS0tLS0tCj4gIDMgZmlsZXMgY2hh bmdlZCwgNDkgaW5zZXJ0aW9ucygrKSwgMjEgZGVsZXRpb25zKC0pCj4gCj4gZGlmZiAtLWdpdCBh L2FyY2gvczM5MC9pbmNsdWRlL2FzbS9wZ3RhYmxlLmggYi9hcmNoL3MzOTAvaW5jbHVkZS9hc20v cGd0YWJsZS5oCj4gaW5kZXggN2ViMDFhNTQ1OWNkLi5iNTU1NjFjYzg3ODYgMTAwNjQ0Cj4gLS0t IGEvYXJjaC9zMzkwL2luY2x1ZGUvYXNtL3BndGFibGUuaAo+ICsrKyBiL2FyY2gvczM5MC9pbmNs dWRlL2FzbS9wZ3RhYmxlLmgKPiBAQCAtMTI2MCwyNiArMTI2MCw0NCBAQCBzdGF0aWMgaW5saW5l IHBnZF90ICpwZ2Rfb2Zmc2V0X3JhdyhwZ2RfdCAqcGdkLCB1bnNpZ25lZCBsb25nIGFkZHJlc3Mp Cj4gIAo+ICAjZGVmaW5lIHBnZF9vZmZzZXQobW0sIGFkZHJlc3MpIHBnZF9vZmZzZXRfcmF3KFJF QURfT05DRSgobW0pLT5wZ2QpLCBhZGRyZXNzKQo+ICAKPiAtc3RhdGljIGlubGluZSBwNGRfdCAq cDRkX29mZnNldChwZ2RfdCAqcGdkLCB1bnNpZ25lZCBsb25nIGFkZHJlc3MpCj4gK3N0YXRpYyBp bmxpbmUgcDRkX3QgKnA0ZF9vZmZzZXRfbG9ja2xlc3MocGdkX3QgKnBnZHAsIHBnZF90IHBnZCwg dW5zaWduZWQgbG9uZyBhZGRyZXNzKQo+ICB7Cj4gLQlpZiAoKHBnZF92YWwoKnBnZCkgJiBfUkVH SU9OX0VOVFJZX1RZUEVfTUFTSykgPj0gX1JFR0lPTl9FTlRSWV9UWVBFX1IxKQo+IC0JCXJldHVy biAocDRkX3QgKikgcGdkX2RlcmVmKCpwZ2QpICsgcDRkX2luZGV4KGFkZHJlc3MpOwo+IC0JcmV0 dXJuIChwNGRfdCAqKSBwZ2Q7Cj4gKwlpZiAoKHBnZF92YWwocGdkKSAmIF9SRUdJT05fRU5UUllf VFlQRV9NQVNLKSA+PSBfUkVHSU9OX0VOVFJZX1RZUEVfUjEpCj4gKwkJcmV0dXJuIChwNGRfdCAq KSBwZ2RfZGVyZWYocGdkKSArIHA0ZF9pbmRleChhZGRyZXNzKTsKPiArCXJldHVybiAocDRkX3Qg KikgcGdkcDsKPiAgfQo+ICsjZGVmaW5lIHA0ZF9vZmZzZXRfbG9ja2xlc3MgcDRkX29mZnNldF9s b2NrbGVzcwo+ICAKPiAtc3RhdGljIGlubGluZSBwdWRfdCAqcHVkX29mZnNldChwNGRfdCAqcDRk LCB1bnNpZ25lZCBsb25nIGFkZHJlc3MpCj4gK3N0YXRpYyBpbmxpbmUgcDRkX3QgKnA0ZF9vZmZz ZXQocGdkX3QgKnBnZHAsIHVuc2lnbmVkIGxvbmcgYWRkcmVzcykKPiAgewo+IC0JaWYgKChwNGRf dmFsKCpwNGQpICYgX1JFR0lPTl9FTlRSWV9UWVBFX01BU0spID49IF9SRUdJT05fRU5UUllfVFlQ RV9SMikKPiAtCQlyZXR1cm4gKHB1ZF90ICopIHA0ZF9kZXJlZigqcDRkKSArIHB1ZF9pbmRleChh ZGRyZXNzKTsKPiAtCXJldHVybiAocHVkX3QgKikgcDRkOwo+ICsJcmV0dXJuIHA0ZF9vZmZzZXRf bG9ja2xlc3MocGdkcCwgKnBnZHAsIGFkZHJlc3MpOwo+ICt9Cj4gKwo+ICtzdGF0aWMgaW5saW5l IHB1ZF90ICpwdWRfb2Zmc2V0X2xvY2tsZXNzKHA0ZF90ICpwNGRwLCBwNGRfdCBwNGQsIHVuc2ln bmVkIGxvbmcgYWRkcmVzcykKPiArewo+ICsJaWYgKChwNGRfdmFsKHA0ZCkgJiBfUkVHSU9OX0VO VFJZX1RZUEVfTUFTSykgPj0gX1JFR0lPTl9FTlRSWV9UWVBFX1IyKQo+ICsJCXJldHVybiAocHVk X3QgKikgcDRkX2RlcmVmKHA0ZCkgKyBwdWRfaW5kZXgoYWRkcmVzcyk7Cj4gKwlyZXR1cm4gKHB1 ZF90ICopIHA0ZHA7Cj4gK30KPiArI2RlZmluZSBwdWRfb2Zmc2V0X2xvY2tsZXNzIHB1ZF9vZmZz ZXRfbG9ja2xlc3MKPiArCj4gK3N0YXRpYyBpbmxpbmUgcHVkX3QgKnB1ZF9vZmZzZXQocDRkX3Qg KnA0ZHAsIHVuc2lnbmVkIGxvbmcgYWRkcmVzcykKPiArewo+ICsJcmV0dXJuIHB1ZF9vZmZzZXRf bG9ja2xlc3MocDRkcCwgKnA0ZHAsIGFkZHJlc3MpOwo+ICB9Cj4gICNkZWZpbmUgcHVkX29mZnNl dCBwdWRfb2Zmc2V0Cj4gIAo+IC1zdGF0aWMgaW5saW5lIHBtZF90ICpwbWRfb2Zmc2V0KHB1ZF90 ICpwdWQsIHVuc2lnbmVkIGxvbmcgYWRkcmVzcykKPiArc3RhdGljIGlubGluZSBwbWRfdCAqcG1k X29mZnNldF9sb2NrbGVzcyhwdWRfdCAqcHVkcCwgcHVkX3QgcHVkLCB1bnNpZ25lZCBsb25nIGFk ZHJlc3MpCj4gK3sKPiArCWlmICgocHVkX3ZhbChwdWQpICYgX1JFR0lPTl9FTlRSWV9UWVBFX01B U0spID49IF9SRUdJT05fRU5UUllfVFlQRV9SMykKPiArCQlyZXR1cm4gKHBtZF90ICopIHB1ZF9k ZXJlZihwdWQpICsgcG1kX2luZGV4KGFkZHJlc3MpOwo+ICsJcmV0dXJuIChwbWRfdCAqKSBwdWRw Owo+ICt9Cj4gKyNkZWZpbmUgcG1kX29mZnNldF9sb2NrbGVzcyBwbWRfb2Zmc2V0X2xvY2tsZXNz Cj4gKwo+ICtzdGF0aWMgaW5saW5lIHBtZF90ICpwbWRfb2Zmc2V0KHB1ZF90ICpwdWRwLCB1bnNp Z25lZCBsb25nIGFkZHJlc3MpCj4gIHsKPiAtCWlmICgocHVkX3ZhbCgqcHVkKSAmIF9SRUdJT05f RU5UUllfVFlQRV9NQVNLKSA+PSBfUkVHSU9OX0VOVFJZX1RZUEVfUjMpCj4gLQkJcmV0dXJuIChw bWRfdCAqKSBwdWRfZGVyZWYoKnB1ZCkgKyBwbWRfaW5kZXgoYWRkcmVzcyk7Cj4gLQlyZXR1cm4g KHBtZF90ICopIHB1ZDsKPiArCXJldHVybiBwbWRfb2Zmc2V0X2xvY2tsZXNzKHB1ZHAsICpwdWRw LCBhZGRyZXNzKTsKPiAgfQo+ICAjZGVmaW5lIHBtZF9vZmZzZXQgcG1kX29mZnNldAo+ICAKPiBk aWZmIC0tZ2l0IGEvaW5jbHVkZS9saW51eC9wZ3RhYmxlLmggYi9pbmNsdWRlL2xpbnV4L3BndGFi bGUuaAo+IGluZGV4IGU4Y2JjMmU3OTVkNS4uOTA2NTRjYjYzZTllIDEwMDY0NAo+IC0tLSBhL2lu Y2x1ZGUvbGludXgvcGd0YWJsZS5oCj4gKysrIGIvaW5jbHVkZS9saW51eC9wZ3RhYmxlLmgKPiBA QCAtMTQyNyw2ICsxNDI3LDE2IEBAIHR5cGVkZWYgdW5zaWduZWQgaW50IHBndGJsX21vZF9tYXNr Owo+ICAjZGVmaW5lIG1tX3BtZF9mb2xkZWQobW0pCV9faXNfZGVmaW5lZChfX1BBR0VUQUJMRV9Q TURfRk9MREVEKQo+ICAjZW5kaWYKPiAgCj4gKyNpZm5kZWYgcDRkX29mZnNldF9sb2NrbGVzcwo+ ICsjZGVmaW5lIHA0ZF9vZmZzZXRfbG9ja2xlc3MocGdkcCwgcGdkLCBhZGRyZXNzKSBwNGRfb2Zm c2V0KCYocGdkKSwgYWRkcmVzcykKPiArI2VuZGlmCj4gKyNpZm5kZWYgcHVkX29mZnNldF9sb2Nr bGVzcwo+ICsjZGVmaW5lIHB1ZF9vZmZzZXRfbG9ja2xlc3MocDRkcCwgcDRkLCBhZGRyZXNzKSBw dWRfb2Zmc2V0KCYocDRkKSwgYWRkcmVzcykKPiArI2VuZGlmCj4gKyNpZm5kZWYgcG1kX29mZnNl dF9sb2NrbGVzcwo+ICsjZGVmaW5lIHBtZF9vZmZzZXRfbG9ja2xlc3MocHVkcCwgcHVkLCBhZGRy ZXNzKSBwbWRfb2Zmc2V0KCYocHVkKSwgYWRkcmVzcykKPiArI2VuZGlmCj4gKwo+ICAvKgo+ICAg KiBwP2RfbGVhZigpIC0gdHJ1ZSBpZiB0aGlzIGVudHJ5IGlzIGEgZmluYWwgbWFwcGluZyB0byBh IHBoeXNpY2FsIGFkZHJlc3MuCj4gICAqIFRoaXMgZGlmZmVycyBmcm9tIHA/ZF9odWdlKCkgYnkg dGhlIGZhY3QgdGhhdCB0aGV5IGFyZSBhbHdheXMgYXZhaWxhYmxlIChpZgo+IGRpZmYgLS1naXQg YS9tbS9ndXAuYyBiL21tL2d1cC5jCj4gaW5kZXggZTU3MzlhMTk3NGQ1Li41NzhiZjViZDhiZjgg MTAwNjQ0Cj4gLS0tIGEvbW0vZ3VwLmMKPiArKysgYi9tbS9ndXAuYwo+IEBAIC0yNDg1LDEzICsy NDg1LDEzIEBAIHN0YXRpYyBpbnQgZ3VwX2h1Z2VfcGdkKHBnZF90IG9yaWcsIHBnZF90ICpwZ2Rw LCB1bnNpZ25lZCBsb25nIGFkZHIsCj4gIAlyZXR1cm4gMTsKPiAgfQo+ICAKPiAtc3RhdGljIGlu dCBndXBfcG1kX3JhbmdlKHB1ZF90IHB1ZCwgdW5zaWduZWQgbG9uZyBhZGRyLCB1bnNpZ25lZCBs b25nIGVuZCwKPiArc3RhdGljIGludCBndXBfcG1kX3JhbmdlKHB1ZF90ICpwdWRwLCBwdWRfdCBw dWQsIHVuc2lnbmVkIGxvbmcgYWRkciwgdW5zaWduZWQgbG9uZyBlbmQsCj4gIAkJdW5zaWduZWQg aW50IGZsYWdzLCBzdHJ1Y3QgcGFnZSAqKnBhZ2VzLCBpbnQgKm5yKQo+ICB7Cj4gIAl1bnNpZ25l ZCBsb25nIG5leHQ7Cj4gIAlwbWRfdCAqcG1kcDsKPiAgCj4gLQlwbWRwID0gcG1kX29mZnNldCgm cHVkLCBhZGRyKTsKPiArCXBtZHAgPSBwbWRfb2Zmc2V0X2xvY2tsZXNzKHB1ZHAsIHB1ZCwgYWRk cik7Cj4gIAlkbyB7Cj4gIAkJcG1kX3QgcG1kID0gUkVBRF9PTkNFKCpwbWRwKTsKPiAgCj4gQEAg LTI1MjgsMTMgKzI1MjgsMTMgQEAgc3RhdGljIGludCBndXBfcG1kX3JhbmdlKHB1ZF90IHB1ZCwg dW5zaWduZWQgbG9uZyBhZGRyLCB1bnNpZ25lZCBsb25nIGVuZCwKPiAgCXJldHVybiAxOwo+ICB9 Cj4gIAo+IC1zdGF0aWMgaW50IGd1cF9wdWRfcmFuZ2UocDRkX3QgcDRkLCB1bnNpZ25lZCBsb25n IGFkZHIsIHVuc2lnbmVkIGxvbmcgZW5kLAo+ICtzdGF0aWMgaW50IGd1cF9wdWRfcmFuZ2UocDRk X3QgKnA0ZHAsIHA0ZF90IHA0ZCwgdW5zaWduZWQgbG9uZyBhZGRyLCB1bnNpZ25lZCBsb25nIGVu ZCwKPiAgCQkJIHVuc2lnbmVkIGludCBmbGFncywgc3RydWN0IHBhZ2UgKipwYWdlcywgaW50ICpu cikKPiAgewo+ICAJdW5zaWduZWQgbG9uZyBuZXh0Owo+ICAJcHVkX3QgKnB1ZHA7Cj4gIAo+IC0J cHVkcCA9IHB1ZF9vZmZzZXQoJnA0ZCwgYWRkcik7Cj4gKwlwdWRwID0gcHVkX29mZnNldF9sb2Nr bGVzcyhwNGRwLCBwNGQsIGFkZHIpOwo+ICAJZG8gewo+ICAJCXB1ZF90IHB1ZCA9IFJFQURfT05D RSgqcHVkcCk7Cj4gIAo+IEBAIC0yNTQ5LDIwICsyNTQ5LDIwIEBAIHN0YXRpYyBpbnQgZ3VwX3B1 ZF9yYW5nZShwNGRfdCBwNGQsIHVuc2lnbmVkIGxvbmcgYWRkciwgdW5zaWduZWQgbG9uZyBlbmQs Cj4gIAkJCWlmICghZ3VwX2h1Z2VfcGQoX19odWdlcGQocHVkX3ZhbChwdWQpKSwgYWRkciwKPiAg CQkJCQkgUFVEX1NISUZULCBuZXh0LCBmbGFncywgcGFnZXMsIG5yKSkKPiAgCQkJCXJldHVybiAw Owo+IC0JCX0gZWxzZSBpZiAoIWd1cF9wbWRfcmFuZ2UocHVkLCBhZGRyLCBuZXh0LCBmbGFncywg cGFnZXMsIG5yKSkKPiArCQl9IGVsc2UgaWYgKCFndXBfcG1kX3JhbmdlKHB1ZHAsIHB1ZCwgYWRk ciwgbmV4dCwgZmxhZ3MsIHBhZ2VzLCBucikpCj4gIAkJCXJldHVybiAwOwo+ICAJfSB3aGlsZSAo cHVkcCsrLCBhZGRyID0gbmV4dCwgYWRkciAhPSBlbmQpOwo+ICAKPiAgCXJldHVybiAxOwo+ICB9 Cj4gIAo+IC1zdGF0aWMgaW50IGd1cF9wNGRfcmFuZ2UocGdkX3QgcGdkLCB1bnNpZ25lZCBsb25n IGFkZHIsIHVuc2lnbmVkIGxvbmcgZW5kLAo+ICtzdGF0aWMgaW50IGd1cF9wNGRfcmFuZ2UocGdk X3QgKnBnZHAsIHBnZF90IHBnZCwgdW5zaWduZWQgbG9uZyBhZGRyLCB1bnNpZ25lZCBsb25nIGVu ZCwKPiAgCQkJIHVuc2lnbmVkIGludCBmbGFncywgc3RydWN0IHBhZ2UgKipwYWdlcywgaW50ICpu cikKPiAgewo+ICAJdW5zaWduZWQgbG9uZyBuZXh0Owo+ICAJcDRkX3QgKnA0ZHA7Cj4gIAo+IC0J cDRkcCA9IHA0ZF9vZmZzZXQoJnBnZCwgYWRkcik7Cj4gKwlwNGRwID0gcDRkX29mZnNldF9sb2Nr bGVzcyhwZ2RwLCBwZ2QsIGFkZHIpOwo+ICAJZG8gewo+ICAJCXA0ZF90IHA0ZCA9IFJFQURfT05D RSgqcDRkcCk7Cj4gIAo+IEBAIC0yNTc0LDcgKzI1NzQsNyBAQCBzdGF0aWMgaW50IGd1cF9wNGRf cmFuZ2UocGdkX3QgcGdkLCB1bnNpZ25lZCBsb25nIGFkZHIsIHVuc2lnbmVkIGxvbmcgZW5kLAo+ ICAJCQlpZiAoIWd1cF9odWdlX3BkKF9faHVnZXBkKHA0ZF92YWwocDRkKSksIGFkZHIsCj4gIAkJ CQkJIFA0RF9TSElGVCwgbmV4dCwgZmxhZ3MsIHBhZ2VzLCBucikpCj4gIAkJCQlyZXR1cm4gMDsK PiAtCQl9IGVsc2UgaWYgKCFndXBfcHVkX3JhbmdlKHA0ZCwgYWRkciwgbmV4dCwgZmxhZ3MsIHBh Z2VzLCBucikpCj4gKwkJfSBlbHNlIGlmICghZ3VwX3B1ZF9yYW5nZShwNGRwLCBwNGQsIGFkZHIs IG5leHQsIGZsYWdzLCBwYWdlcywgbnIpKQo+ICAJCQlyZXR1cm4gMDsKPiAgCX0gd2hpbGUgKHA0 ZHArKywgYWRkciA9IG5leHQsIGFkZHIgIT0gZW5kKTsKPiAgCj4gQEAgLTI2MDIsNyArMjYwMiw3 IEBAIHN0YXRpYyB2b2lkIGd1cF9wZ2RfcmFuZ2UodW5zaWduZWQgbG9uZyBhZGRyLCB1bnNpZ25l ZCBsb25nIGVuZCwKPiAgCQkJaWYgKCFndXBfaHVnZV9wZChfX2h1Z2VwZChwZ2RfdmFsKHBnZCkp LCBhZGRyLAo+ICAJCQkJCSBQR0RJUl9TSElGVCwgbmV4dCwgZmxhZ3MsIHBhZ2VzLCBucikpCj4g IAkJCQlyZXR1cm47Cj4gLQkJfSBlbHNlIGlmICghZ3VwX3A0ZF9yYW5nZShwZ2QsIGFkZHIsIG5l eHQsIGZsYWdzLCBwYWdlcywgbnIpKQo+ICsJCX0gZWxzZSBpZiAoIWd1cF9wNGRfcmFuZ2UocGdk cCwgcGdkLCBhZGRyLCBuZXh0LCBmbGFncywgcGFnZXMsIG5yKSkKPiAgCQkJcmV0dXJuOwo+ICAJ fSB3aGlsZSAocGdkcCsrLCBhZGRyID0gbmV4dCwgYWRkciAhPSBlbmQpOwo+ICB9Cj4gLS0gCj4g 4qO/4qO/4qO/4qO/4qKL4qGA4qOA4qC54qO/4qO/4qO/4qO/Cj4g4qO/4qO/4qO/4qO/4qCg4qO2 4qGm4qCA4qO/4qO/4qO/4qO/Cj4g4qO/4qO/4qO/4qCP4qO04qOu4qO04qOn4qCI4qK/4qO/4qO/ Cj4g4qO/4qO/4qGP4qKw4qO/4qCW4qOg4qO/4qGG4qCI4qO/4qO/Cj4g4qO/4qKb4qO14qOE4qCZ 4qO24qO24qGf4qOF4qOg4qC54qO/Cj4g4qO/4qOc4qOb4qC74qKO4qOJ4qOJ4qOA4qC/4qOr4qO1 4qO/CgotLSAKU2luY2VyZWx5IHlvdXJzLApNaWtlLgoKX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX18KbGludXgtYXJtLWtlcm5lbCBtYWlsaW5nIGxpc3QKbGlu dXgtYXJtLWtlcm5lbEBsaXN0cy5pbmZyYWRlYWQub3JnCmh0dHA6Ly9saXN0cy5pbmZyYWRlYWQu b3JnL21haWxtYW4vbGlzdGluZm8vbGludXgtYXJtLWtlcm5lbAo= From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Date: Tue, 15 Sep 2020 20:18:23 +0300 From: Mike Rapoport Subject: Re: [PATCH v2] mm/gup: fix gup_fast with dynamic page table folding Message-ID: <20200915171823.GJ2142832@kernel.org> References: <20200911200511.GC1221970@ziepe.ca> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-um" Errors-To: linux-um-bounces+geert=linux-m68k.org@lists.infradead.org To: Vasily Gorbik Cc: Peter Zijlstra , Benjamin Herrenschmidt , Dave Hansen , Dave Hansen , Paul Mackerras , linux-sparc , Alexander Gordeev , Claudio Imbrenda , Will Deacon , linux-arch , linux-s390 , Christian Borntraeger , Richard Weinberger , linux-x86 , Russell King , Jason Gunthorpe , Ingo Molnar , Catalin Marinas , Andrey Ryabinin , Gerald Schaefer , Heiko Carstens , Arnd Bergmann , John Hubbard , Jeff Dike , linux-um , Borislav Petkov , Andy Lutomirski , Thomas Gleixner , linux-arm , linux-mm , linux-power , LKML , Michael Ellerman , Andrew Morton , Linus Torvalds T24gRnJpLCBTZXAgMTEsIDIwMjAgYXQgMTA6MzY6NDNQTSArMDIwMCwgVmFzaWx5IEdvcmJpayB3 cm90ZToKPiBDdXJyZW50bHkgdG8gbWFrZSBzdXJlIHRoYXQgZXZlcnkgcGFnZSB0YWJsZSBlbnRy eSBpcyByZWFkIGp1c3Qgb25jZQo+IGd1cF9mYXN0IHdhbGtzIHBlcmZvcm0gUkVBRF9PTkNFIGFu ZCBwYXNzIHBYZCB2YWx1ZSBkb3duIHRvIHRoZSBuZXh0Cj4gZ3VwX3BYZF9yYW5nZSBmdW5jdGlv biBieSB2YWx1ZSBlLmcuOgo+IAo+IHN0YXRpYyBpbnQgZ3VwX3B1ZF9yYW5nZShwNGRfdCBwNGQs IHVuc2lnbmVkIGxvbmcgYWRkciwgdW5zaWduZWQgbG9uZyBlbmQsCj4gICAgICAgICAgICAgICAg ICAgICAgICAgIHVuc2lnbmVkIGludCBmbGFncywgc3RydWN0IHBhZ2UgKipwYWdlcywgaW50ICpu cikKPiAuLi4KPiAgICAgICAgIHB1ZHAgPSBwdWRfb2Zmc2V0KCZwNGQsIGFkZHIpOwo+IAo+IFRo aXMgZnVuY3Rpb24gcGFzc2VzIGEgcmVmZXJlbmNlIG9uIHRoYXQgbG9jYWwgdmFsdWUgY29weSB0 byBwWGRfb2Zmc2V0LAo+IGFuZCBtaWdodCBnZXQgdGhlIHZlcnkgc2FtZSBwb2ludGVyIGluIHJl dHVybi4gVGhpcyBoYXBwZW5zIHdoZW4gdGhlCj4gbGV2ZWwgaXMgZm9sZGVkIChvbiBtb3N0IGFy Y2hlcyksIGFuZCB0aGF0IHBvaW50ZXIgc2hvdWxkIG5vdCBiZSBpdGVyYXRlZC4KPiAKPiBPbiBz MzkwIGR1ZSB0byB0aGUgZmFjdCB0aGF0IGVhY2ggdGFzayBtaWdodCBoYXZlIGRpZmZlcmVudCA1 LDQgb3IKPiAzLWxldmVsIGFkZHJlc3MgdHJhbnNsYXRpb24gYW5kIGhlbmNlIGRpZmZlcmVudCBs ZXZlbHMgZm9sZGVkIHRoZSBsb2dpYwo+IGlzIG1vcmUgY29tcGxleCBhbmQgbm9uLWl0ZXJhdGFi bGUgcG9pbnRlciB0byBhIGxvY2FsIGNvcHkgbGVhZHMgdG8KPiBzZXZlcmUgcHJvYmxlbXMuCj4g Cj4gSGVyZSBpcyBhbiBleGFtcGxlIG9mIHdoYXQgaGFwcGVucyB3aXRoIGd1cF9mYXN0IG9uIHMz OTAsIGZvciBhIHRhc2sKPiB3aXRoIDMtbGV2ZWxzIHBhZ2luZywgY3Jvc3NpbmcgYSAyIEdCIHB1 ZCBib3VuZGFyeToKPiAKPiAvLyBhZGRyID0gMHgxMDA3ZmZmZjAwMCwgZW5kID0gMHgxMDA4MDAw MTAwMAo+IHN0YXRpYyBpbnQgZ3VwX3B1ZF9yYW5nZShwNGRfdCBwNGQsIHVuc2lnbmVkIGxvbmcg YWRkciwgdW5zaWduZWQgbG9uZyBlbmQsCj4gICAgICAgICAgICAgICAgICAgICAgICAgIHVuc2ln bmVkIGludCBmbGFncywgc3RydWN0IHBhZ2UgKipwYWdlcywgaW50ICpucikKPiB7Cj4gICAgICAg ICB1bnNpZ25lZCBsb25nIG5leHQ7Cj4gICAgICAgICBwdWRfdCAqcHVkcDsKPiAKPiAgICAgICAg IC8vIHB1ZF9vZmZzZXQgcmV0dXJucyAmcDRkIGl0c2VsZiAoYSBwb2ludGVyIHRvIGEgdmFsdWUg b24gc3RhY2spCj4gICAgICAgICBwdWRwID0gcHVkX29mZnNldCgmcDRkLCBhZGRyKTsKPiAgICAg ICAgIGRvIHsKPiAgICAgICAgICAgICAgICAgLy8gb24gc2Vjb25kIGl0ZXJhdGF0aW9uIHJlYWRp bmcgInJhbmRvbSIgc3RhY2sgdmFsdWUKPiAgICAgICAgICAgICAgICAgcHVkX3QgcHVkID0gUkVB RF9PTkNFKCpwdWRwKTsKPiAKPiAgICAgICAgICAgICAgICAgLy8gbmV4dCA9IDB4MTAwODAwMDAw MDAsIGR1ZSB0byBQVURfU0laRS9NQVNLICE9IFBHRElSX1NJWkUvTUFTSyBvbiBzMzkwCj4gICAg ICAgICAgICAgICAgIG5leHQgPSBwdWRfYWRkcl9lbmQoYWRkciwgZW5kKTsKPiAgICAgICAgICAg ICAgICAgLi4uCj4gICAgICAgICB9IHdoaWxlIChwdWRwKyssIGFkZHIgPSBuZXh0LCBhZGRyICE9 IGVuZCk7IC8vIHB1ZHArKyBpdGVyYXRpbmcgb3ZlciBzdGFjawo+IAo+ICAgICAgICAgcmV0dXJu IDE7Cj4gfQo+IAo+IFRoaXMgaGFwcGVucyBzaW5jZSBzMzkwIG1vdmVkIHRvIGNvbW1vbiBndXAg Y29kZSB3aXRoCj4gY29tbWl0IGQxODc0YTBjMjgwNSAoInMzOTAvbW06IG1ha2UgdGhlIHB4ZF9v ZmZzZXQgZnVuY3Rpb25zIG1vcmUgcm9idXN0IikKPiBhbmQgY29tbWl0IDFhNDIwMTBjZGMyNiAo InMzOTAvbW06IGNvbnZlcnQgdG8gdGhlIGdlbmVyaWMKPiBnZXRfdXNlcl9wYWdlc19mYXN0IGNv ZGUiKS4gczM5MCB0cmllZCB0byBtaW1pYyBzdGF0aWMgbGV2ZWwgZm9sZGluZyBieQo+IGNoYW5n aW5nIHBYZF9vZmZzZXQgcHJpbWl0aXZlcyB0byBhbHdheXMgY2FsY3VsYXRlIHRvcCBsZXZlbCBw YWdlIHRhYmxlCj4gb2Zmc2V0IGluIHBnZF9vZmZzZXQgYW5kIGp1c3QgcmV0dXJuIHRoZSB2YWx1 ZSBwYXNzZWQgd2hlbiBwWGRfb2Zmc2V0Cj4gaGFzIHRvIGFjdCBhcyBmb2xkZWQuCj4gCj4gV2hh dCBpcyBjcnVjaWFsIGZvciBndXBfZmFzdCBhbmQgd2hhdCBoYXMgYmVlbiBvdmVybG9va2VkIGlz Cj4gdGhhdCBQeERfU0laRS9NQVNLIGFuZCB0aHVzIHBYZF9hZGRyX2VuZCBzaG91bGQgYWxzbyBj aGFuZ2UKPiBjb3JyZXNwb25kaW5nbHkuIEFuZCB0aGUgbGF0dGVyIGlzIG5vdCBwb3NzaWJsZSB3 aXRoIGR5bmFtaWMgZm9sZGluZy4KPiAKPiBUbyBmaXggdGhlIGlzc3VlIGluIGFkZGl0aW9uIHRv IHBYZCB2YWx1ZXMgcGFzcyBvcmlnaW5hbAo+IHBYZHAgcG9pbnRlcnMgZG93biB0byBndXBfcFhk X3JhbmdlIGZ1bmN0aW9ucy4gQW5kIGludHJvZHVjZQo+IHBYZF9vZmZzZXRfbG9ja2xlc3MgaGVs cGVycywgd2hpY2ggdGFrZSBhbiBhZGRpdGlvbmFsIHBYZAo+IGVudHJ5IHZhbHVlIHBhcmFtZXRl ci4gVGhpcyBoYXMgYWxyZWFkeSBiZWVuIGRpc2N1c3NlZCBpbgo+IGh0dHBzOi8vbGttbC5rZXJu ZWwub3JnL3IvMjAxOTA0MTgxMDAyMTguMGE0YWZkNTFAbXNjaHdpZGVYMQo+IAo+IENjOiA8c3Rh YmxlQHZnZXIua2VybmVsLm9yZz4gIyA1LjIrCj4gRml4ZXM6IDFhNDIwMTBjZGMyNiAoInMzOTAv bW06IGNvbnZlcnQgdG8gdGhlIGdlbmVyaWMgZ2V0X3VzZXJfcGFnZXNfZmFzdCBjb2RlIikKPiBS ZXZpZXdlZC1ieTogR2VyYWxkIFNjaGFlZmVyIDxnZXJhbGQuc2NoYWVmZXJAbGludXguaWJtLmNv bT4KPiBSZXZpZXdlZC1ieTogQWxleGFuZGVyIEdvcmRlZXYgPGFnb3JkZWV2QGxpbnV4LmlibS5j b20+Cj4gU2lnbmVkLW9mZi1ieTogVmFzaWx5IEdvcmJpayA8Z29yQGxpbnV4LmlibS5jb20+CgpS ZXZpZXdlZC1ieTogTWlrZSBSYXBvcG9ydCA8cnBwdEBsaW51eC5pYm0uY29tPgoKPiAtLS0KPiB2 MjogYWRkZWQgYnJhY2tldHMgJnBnZCAtPiAmKHBnZCkKPiAKPiAgYXJjaC9zMzkwL2luY2x1ZGUv YXNtL3BndGFibGUuaCB8IDQyICsrKysrKysrKysrKysrKysrKysrKysrLS0tLS0tLS0tLQo+ICBp bmNsdWRlL2xpbnV4L3BndGFibGUuaCAgICAgICAgIHwgMTAgKysrKysrKysKPiAgbW0vZ3VwLmMg ICAgICAgICAgICAgICAgICAgICAgICB8IDE4ICsrKysrKystLS0tLS0tCj4gIDMgZmlsZXMgY2hh bmdlZCwgNDkgaW5zZXJ0aW9ucygrKSwgMjEgZGVsZXRpb25zKC0pCj4gCj4gZGlmZiAtLWdpdCBh L2FyY2gvczM5MC9pbmNsdWRlL2FzbS9wZ3RhYmxlLmggYi9hcmNoL3MzOTAvaW5jbHVkZS9hc20v cGd0YWJsZS5oCj4gaW5kZXggN2ViMDFhNTQ1OWNkLi5iNTU1NjFjYzg3ODYgMTAwNjQ0Cj4gLS0t IGEvYXJjaC9zMzkwL2luY2x1ZGUvYXNtL3BndGFibGUuaAo+ICsrKyBiL2FyY2gvczM5MC9pbmNs dWRlL2FzbS9wZ3RhYmxlLmgKPiBAQCAtMTI2MCwyNiArMTI2MCw0NCBAQCBzdGF0aWMgaW5saW5l IHBnZF90ICpwZ2Rfb2Zmc2V0X3JhdyhwZ2RfdCAqcGdkLCB1bnNpZ25lZCBsb25nIGFkZHJlc3Mp Cj4gIAo+ICAjZGVmaW5lIHBnZF9vZmZzZXQobW0sIGFkZHJlc3MpIHBnZF9vZmZzZXRfcmF3KFJF QURfT05DRSgobW0pLT5wZ2QpLCBhZGRyZXNzKQo+ICAKPiAtc3RhdGljIGlubGluZSBwNGRfdCAq cDRkX29mZnNldChwZ2RfdCAqcGdkLCB1bnNpZ25lZCBsb25nIGFkZHJlc3MpCj4gK3N0YXRpYyBp bmxpbmUgcDRkX3QgKnA0ZF9vZmZzZXRfbG9ja2xlc3MocGdkX3QgKnBnZHAsIHBnZF90IHBnZCwg dW5zaWduZWQgbG9uZyBhZGRyZXNzKQo+ICB7Cj4gLQlpZiAoKHBnZF92YWwoKnBnZCkgJiBfUkVH SU9OX0VOVFJZX1RZUEVfTUFTSykgPj0gX1JFR0lPTl9FTlRSWV9UWVBFX1IxKQo+IC0JCXJldHVy biAocDRkX3QgKikgcGdkX2RlcmVmKCpwZ2QpICsgcDRkX2luZGV4KGFkZHJlc3MpOwo+IC0JcmV0 dXJuIChwNGRfdCAqKSBwZ2Q7Cj4gKwlpZiAoKHBnZF92YWwocGdkKSAmIF9SRUdJT05fRU5UUllf VFlQRV9NQVNLKSA+PSBfUkVHSU9OX0VOVFJZX1RZUEVfUjEpCj4gKwkJcmV0dXJuIChwNGRfdCAq KSBwZ2RfZGVyZWYocGdkKSArIHA0ZF9pbmRleChhZGRyZXNzKTsKPiArCXJldHVybiAocDRkX3Qg KikgcGdkcDsKPiAgfQo+ICsjZGVmaW5lIHA0ZF9vZmZzZXRfbG9ja2xlc3MgcDRkX29mZnNldF9s b2NrbGVzcwo+ICAKPiAtc3RhdGljIGlubGluZSBwdWRfdCAqcHVkX29mZnNldChwNGRfdCAqcDRk LCB1bnNpZ25lZCBsb25nIGFkZHJlc3MpCj4gK3N0YXRpYyBpbmxpbmUgcDRkX3QgKnA0ZF9vZmZz ZXQocGdkX3QgKnBnZHAsIHVuc2lnbmVkIGxvbmcgYWRkcmVzcykKPiAgewo+IC0JaWYgKChwNGRf dmFsKCpwNGQpICYgX1JFR0lPTl9FTlRSWV9UWVBFX01BU0spID49IF9SRUdJT05fRU5UUllfVFlQ RV9SMikKPiAtCQlyZXR1cm4gKHB1ZF90ICopIHA0ZF9kZXJlZigqcDRkKSArIHB1ZF9pbmRleChh ZGRyZXNzKTsKPiAtCXJldHVybiAocHVkX3QgKikgcDRkOwo+ICsJcmV0dXJuIHA0ZF9vZmZzZXRf bG9ja2xlc3MocGdkcCwgKnBnZHAsIGFkZHJlc3MpOwo+ICt9Cj4gKwo+ICtzdGF0aWMgaW5saW5l IHB1ZF90ICpwdWRfb2Zmc2V0X2xvY2tsZXNzKHA0ZF90ICpwNGRwLCBwNGRfdCBwNGQsIHVuc2ln bmVkIGxvbmcgYWRkcmVzcykKPiArewo+ICsJaWYgKChwNGRfdmFsKHA0ZCkgJiBfUkVHSU9OX0VO VFJZX1RZUEVfTUFTSykgPj0gX1JFR0lPTl9FTlRSWV9UWVBFX1IyKQo+ICsJCXJldHVybiAocHVk X3QgKikgcDRkX2RlcmVmKHA0ZCkgKyBwdWRfaW5kZXgoYWRkcmVzcyk7Cj4gKwlyZXR1cm4gKHB1 ZF90ICopIHA0ZHA7Cj4gK30KPiArI2RlZmluZSBwdWRfb2Zmc2V0X2xvY2tsZXNzIHB1ZF9vZmZz ZXRfbG9ja2xlc3MKPiArCj4gK3N0YXRpYyBpbmxpbmUgcHVkX3QgKnB1ZF9vZmZzZXQocDRkX3Qg KnA0ZHAsIHVuc2lnbmVkIGxvbmcgYWRkcmVzcykKPiArewo+ICsJcmV0dXJuIHB1ZF9vZmZzZXRf bG9ja2xlc3MocDRkcCwgKnA0ZHAsIGFkZHJlc3MpOwo+ICB9Cj4gICNkZWZpbmUgcHVkX29mZnNl dCBwdWRfb2Zmc2V0Cj4gIAo+IC1zdGF0aWMgaW5saW5lIHBtZF90ICpwbWRfb2Zmc2V0KHB1ZF90 ICpwdWQsIHVuc2lnbmVkIGxvbmcgYWRkcmVzcykKPiArc3RhdGljIGlubGluZSBwbWRfdCAqcG1k X29mZnNldF9sb2NrbGVzcyhwdWRfdCAqcHVkcCwgcHVkX3QgcHVkLCB1bnNpZ25lZCBsb25nIGFk ZHJlc3MpCj4gK3sKPiArCWlmICgocHVkX3ZhbChwdWQpICYgX1JFR0lPTl9FTlRSWV9UWVBFX01B U0spID49IF9SRUdJT05fRU5UUllfVFlQRV9SMykKPiArCQlyZXR1cm4gKHBtZF90ICopIHB1ZF9k ZXJlZihwdWQpICsgcG1kX2luZGV4KGFkZHJlc3MpOwo+ICsJcmV0dXJuIChwbWRfdCAqKSBwdWRw Owo+ICt9Cj4gKyNkZWZpbmUgcG1kX29mZnNldF9sb2NrbGVzcyBwbWRfb2Zmc2V0X2xvY2tsZXNz Cj4gKwo+ICtzdGF0aWMgaW5saW5lIHBtZF90ICpwbWRfb2Zmc2V0KHB1ZF90ICpwdWRwLCB1bnNp Z25lZCBsb25nIGFkZHJlc3MpCj4gIHsKPiAtCWlmICgocHVkX3ZhbCgqcHVkKSAmIF9SRUdJT05f RU5UUllfVFlQRV9NQVNLKSA+PSBfUkVHSU9OX0VOVFJZX1RZUEVfUjMpCj4gLQkJcmV0dXJuIChw bWRfdCAqKSBwdWRfZGVyZWYoKnB1ZCkgKyBwbWRfaW5kZXgoYWRkcmVzcyk7Cj4gLQlyZXR1cm4g KHBtZF90ICopIHB1ZDsKPiArCXJldHVybiBwbWRfb2Zmc2V0X2xvY2tsZXNzKHB1ZHAsICpwdWRw LCBhZGRyZXNzKTsKPiAgfQo+ICAjZGVmaW5lIHBtZF9vZmZzZXQgcG1kX29mZnNldAo+ICAKPiBk aWZmIC0tZ2l0IGEvaW5jbHVkZS9saW51eC9wZ3RhYmxlLmggYi9pbmNsdWRlL2xpbnV4L3BndGFi bGUuaAo+IGluZGV4IGU4Y2JjMmU3OTVkNS4uOTA2NTRjYjYzZTllIDEwMDY0NAo+IC0tLSBhL2lu Y2x1ZGUvbGludXgvcGd0YWJsZS5oCj4gKysrIGIvaW5jbHVkZS9saW51eC9wZ3RhYmxlLmgKPiBA QCAtMTQyNyw2ICsxNDI3LDE2IEBAIHR5cGVkZWYgdW5zaWduZWQgaW50IHBndGJsX21vZF9tYXNr Owo+ICAjZGVmaW5lIG1tX3BtZF9mb2xkZWQobW0pCV9faXNfZGVmaW5lZChfX1BBR0VUQUJMRV9Q TURfRk9MREVEKQo+ICAjZW5kaWYKPiAgCj4gKyNpZm5kZWYgcDRkX29mZnNldF9sb2NrbGVzcwo+ ICsjZGVmaW5lIHA0ZF9vZmZzZXRfbG9ja2xlc3MocGdkcCwgcGdkLCBhZGRyZXNzKSBwNGRfb2Zm c2V0KCYocGdkKSwgYWRkcmVzcykKPiArI2VuZGlmCj4gKyNpZm5kZWYgcHVkX29mZnNldF9sb2Nr bGVzcwo+ICsjZGVmaW5lIHB1ZF9vZmZzZXRfbG9ja2xlc3MocDRkcCwgcDRkLCBhZGRyZXNzKSBw dWRfb2Zmc2V0KCYocDRkKSwgYWRkcmVzcykKPiArI2VuZGlmCj4gKyNpZm5kZWYgcG1kX29mZnNl dF9sb2NrbGVzcwo+ICsjZGVmaW5lIHBtZF9vZmZzZXRfbG9ja2xlc3MocHVkcCwgcHVkLCBhZGRy ZXNzKSBwbWRfb2Zmc2V0KCYocHVkKSwgYWRkcmVzcykKPiArI2VuZGlmCj4gKwo+ICAvKgo+ICAg KiBwP2RfbGVhZigpIC0gdHJ1ZSBpZiB0aGlzIGVudHJ5IGlzIGEgZmluYWwgbWFwcGluZyB0byBh IHBoeXNpY2FsIGFkZHJlc3MuCj4gICAqIFRoaXMgZGlmZmVycyBmcm9tIHA/ZF9odWdlKCkgYnkg dGhlIGZhY3QgdGhhdCB0aGV5IGFyZSBhbHdheXMgYXZhaWxhYmxlIChpZgo+IGRpZmYgLS1naXQg YS9tbS9ndXAuYyBiL21tL2d1cC5jCj4gaW5kZXggZTU3MzlhMTk3NGQ1Li41NzhiZjViZDhiZjgg MTAwNjQ0Cj4gLS0tIGEvbW0vZ3VwLmMKPiArKysgYi9tbS9ndXAuYwo+IEBAIC0yNDg1LDEzICsy NDg1LDEzIEBAIHN0YXRpYyBpbnQgZ3VwX2h1Z2VfcGdkKHBnZF90IG9yaWcsIHBnZF90ICpwZ2Rw LCB1bnNpZ25lZCBsb25nIGFkZHIsCj4gIAlyZXR1cm4gMTsKPiAgfQo+ICAKPiAtc3RhdGljIGlu dCBndXBfcG1kX3JhbmdlKHB1ZF90IHB1ZCwgdW5zaWduZWQgbG9uZyBhZGRyLCB1bnNpZ25lZCBs b25nIGVuZCwKPiArc3RhdGljIGludCBndXBfcG1kX3JhbmdlKHB1ZF90ICpwdWRwLCBwdWRfdCBw dWQsIHVuc2lnbmVkIGxvbmcgYWRkciwgdW5zaWduZWQgbG9uZyBlbmQsCj4gIAkJdW5zaWduZWQg aW50IGZsYWdzLCBzdHJ1Y3QgcGFnZSAqKnBhZ2VzLCBpbnQgKm5yKQo+ICB7Cj4gIAl1bnNpZ25l ZCBsb25nIG5leHQ7Cj4gIAlwbWRfdCAqcG1kcDsKPiAgCj4gLQlwbWRwID0gcG1kX29mZnNldCgm cHVkLCBhZGRyKTsKPiArCXBtZHAgPSBwbWRfb2Zmc2V0X2xvY2tsZXNzKHB1ZHAsIHB1ZCwgYWRk cik7Cj4gIAlkbyB7Cj4gIAkJcG1kX3QgcG1kID0gUkVBRF9PTkNFKCpwbWRwKTsKPiAgCj4gQEAg LTI1MjgsMTMgKzI1MjgsMTMgQEAgc3RhdGljIGludCBndXBfcG1kX3JhbmdlKHB1ZF90IHB1ZCwg dW5zaWduZWQgbG9uZyBhZGRyLCB1bnNpZ25lZCBsb25nIGVuZCwKPiAgCXJldHVybiAxOwo+ICB9 Cj4gIAo+IC1zdGF0aWMgaW50IGd1cF9wdWRfcmFuZ2UocDRkX3QgcDRkLCB1bnNpZ25lZCBsb25n IGFkZHIsIHVuc2lnbmVkIGxvbmcgZW5kLAo+ICtzdGF0aWMgaW50IGd1cF9wdWRfcmFuZ2UocDRk X3QgKnA0ZHAsIHA0ZF90IHA0ZCwgdW5zaWduZWQgbG9uZyBhZGRyLCB1bnNpZ25lZCBsb25nIGVu ZCwKPiAgCQkJIHVuc2lnbmVkIGludCBmbGFncywgc3RydWN0IHBhZ2UgKipwYWdlcywgaW50ICpu cikKPiAgewo+ICAJdW5zaWduZWQgbG9uZyBuZXh0Owo+ICAJcHVkX3QgKnB1ZHA7Cj4gIAo+IC0J cHVkcCA9IHB1ZF9vZmZzZXQoJnA0ZCwgYWRkcik7Cj4gKwlwdWRwID0gcHVkX29mZnNldF9sb2Nr bGVzcyhwNGRwLCBwNGQsIGFkZHIpOwo+ICAJZG8gewo+ICAJCXB1ZF90IHB1ZCA9IFJFQURfT05D RSgqcHVkcCk7Cj4gIAo+IEBAIC0yNTQ5LDIwICsyNTQ5LDIwIEBAIHN0YXRpYyBpbnQgZ3VwX3B1 ZF9yYW5nZShwNGRfdCBwNGQsIHVuc2lnbmVkIGxvbmcgYWRkciwgdW5zaWduZWQgbG9uZyBlbmQs Cj4gIAkJCWlmICghZ3VwX2h1Z2VfcGQoX19odWdlcGQocHVkX3ZhbChwdWQpKSwgYWRkciwKPiAg CQkJCQkgUFVEX1NISUZULCBuZXh0LCBmbGFncywgcGFnZXMsIG5yKSkKPiAgCQkJCXJldHVybiAw Owo+IC0JCX0gZWxzZSBpZiAoIWd1cF9wbWRfcmFuZ2UocHVkLCBhZGRyLCBuZXh0LCBmbGFncywg cGFnZXMsIG5yKSkKPiArCQl9IGVsc2UgaWYgKCFndXBfcG1kX3JhbmdlKHB1ZHAsIHB1ZCwgYWRk ciwgbmV4dCwgZmxhZ3MsIHBhZ2VzLCBucikpCj4gIAkJCXJldHVybiAwOwo+ICAJfSB3aGlsZSAo cHVkcCsrLCBhZGRyID0gbmV4dCwgYWRkciAhPSBlbmQpOwo+ICAKPiAgCXJldHVybiAxOwo+ICB9 Cj4gIAo+IC1zdGF0aWMgaW50IGd1cF9wNGRfcmFuZ2UocGdkX3QgcGdkLCB1bnNpZ25lZCBsb25n IGFkZHIsIHVuc2lnbmVkIGxvbmcgZW5kLAo+ICtzdGF0aWMgaW50IGd1cF9wNGRfcmFuZ2UocGdk X3QgKnBnZHAsIHBnZF90IHBnZCwgdW5zaWduZWQgbG9uZyBhZGRyLCB1bnNpZ25lZCBsb25nIGVu ZCwKPiAgCQkJIHVuc2lnbmVkIGludCBmbGFncywgc3RydWN0IHBhZ2UgKipwYWdlcywgaW50ICpu cikKPiAgewo+ICAJdW5zaWduZWQgbG9uZyBuZXh0Owo+ICAJcDRkX3QgKnA0ZHA7Cj4gIAo+IC0J cDRkcCA9IHA0ZF9vZmZzZXQoJnBnZCwgYWRkcik7Cj4gKwlwNGRwID0gcDRkX29mZnNldF9sb2Nr bGVzcyhwZ2RwLCBwZ2QsIGFkZHIpOwo+ICAJZG8gewo+ICAJCXA0ZF90IHA0ZCA9IFJFQURfT05D RSgqcDRkcCk7Cj4gIAo+IEBAIC0yNTc0LDcgKzI1NzQsNyBAQCBzdGF0aWMgaW50IGd1cF9wNGRf cmFuZ2UocGdkX3QgcGdkLCB1bnNpZ25lZCBsb25nIGFkZHIsIHVuc2lnbmVkIGxvbmcgZW5kLAo+ ICAJCQlpZiAoIWd1cF9odWdlX3BkKF9faHVnZXBkKHA0ZF92YWwocDRkKSksIGFkZHIsCj4gIAkJ CQkJIFA0RF9TSElGVCwgbmV4dCwgZmxhZ3MsIHBhZ2VzLCBucikpCj4gIAkJCQlyZXR1cm4gMDsK PiAtCQl9IGVsc2UgaWYgKCFndXBfcHVkX3JhbmdlKHA0ZCwgYWRkciwgbmV4dCwgZmxhZ3MsIHBh Z2VzLCBucikpCj4gKwkJfSBlbHNlIGlmICghZ3VwX3B1ZF9yYW5nZShwNGRwLCBwNGQsIGFkZHIs IG5leHQsIGZsYWdzLCBwYWdlcywgbnIpKQo+ICAJCQlyZXR1cm4gMDsKPiAgCX0gd2hpbGUgKHA0 ZHArKywgYWRkciA9IG5leHQsIGFkZHIgIT0gZW5kKTsKPiAgCj4gQEAgLTI2MDIsNyArMjYwMiw3 IEBAIHN0YXRpYyB2b2lkIGd1cF9wZ2RfcmFuZ2UodW5zaWduZWQgbG9uZyBhZGRyLCB1bnNpZ25l ZCBsb25nIGVuZCwKPiAgCQkJaWYgKCFndXBfaHVnZV9wZChfX2h1Z2VwZChwZ2RfdmFsKHBnZCkp LCBhZGRyLAo+ICAJCQkJCSBQR0RJUl9TSElGVCwgbmV4dCwgZmxhZ3MsIHBhZ2VzLCBucikpCj4g IAkJCQlyZXR1cm47Cj4gLQkJfSBlbHNlIGlmICghZ3VwX3A0ZF9yYW5nZShwZ2QsIGFkZHIsIG5l eHQsIGZsYWdzLCBwYWdlcywgbnIpKQo+ICsJCX0gZWxzZSBpZiAoIWd1cF9wNGRfcmFuZ2UocGdk cCwgcGdkLCBhZGRyLCBuZXh0LCBmbGFncywgcGFnZXMsIG5yKSkKPiAgCQkJcmV0dXJuOwo+ICAJ fSB3aGlsZSAocGdkcCsrLCBhZGRyID0gbmV4dCwgYWRkciAhPSBlbmQpOwo+ICB9Cj4gLS0gCj4g 4qO/4qO/4qO/4qO/4qKL4qGA4qOA4qC54qO/4qO/4qO/4qO/Cj4g4qO/4qO/4qO/4qO/4qCg4qO2 4qGm4qCA4qO/4qO/4qO/4qO/Cj4g4qO/4qO/4qO/4qCP4qO04qOu4qO04qOn4qCI4qK/4qO/4qO/ Cj4g4qO/4qO/4qGP4qKw4qO/4qCW4qOg4qO/4qGG4qCI4qO/4qO/Cj4g4qO/4qKb4qO14qOE4qCZ 4qO24qO24qGf4qOF4qOg4qC54qO/Cj4g4qO/4qOc4qOb4qC74qKO4qOJ4qOJ4qOA4qC/4qOr4qO1 4qO/CgotLSAKU2luY2VyZWx5IHlvdXJzLApNaWtlLgoKX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX18KbGludXgtdW0gbWFpbGluZyBsaXN0CmxpbnV4LXVtQGxp c3RzLmluZnJhZGVhZC5vcmcKaHR0cDovL2xpc3RzLmluZnJhZGVhZC5vcmcvbWFpbG1hbi9saXN0 aW5mby9saW51eC11bQo=