From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC025C43461 for ; Fri, 11 Sep 2020 20:37:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3E7E32075A for ; Fri, 11 Sep 2020 20:37:50 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="cz3YsExB" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1725906AbgIKUhp (ORCPT ); Fri, 11 Sep 2020 16:37:45 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:32996 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725816AbgIKUhn (ORCPT ); Fri, 11 Sep 2020 16:37:43 -0400 Received: from pps.filterd (m0098396.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 08BKWdvP170701; Fri, 11 Sep 2020 16:36:55 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=date : from : to : cc : subject : message-id : references : mime-version : content-type : content-transfer-encoding : in-reply-to; s=pp1; bh=hOcVC6Va1COznQ0oD3ZHmavI0zSfYZBaHJyzu45rYTc=; b=cz3YsExB/lDwLPfrpKWbTdWb5Qzbr5rHXXf/uwggBSWxDSVGHw4DszUvmlUKbaJ1o5Ko TzxhoMu/K5Gf0zonK+k5Y8vnz5rB9fu90H1Nobz5XHKeaPkNmM3vBNK8py3MV/KXKyiD LPBS5/2S4QWDUBed8/p5N9fWoVqnK6U0OjxWZe6GI5Fd9pkYNvx1+FAu3vrbG1i2q76k +zPztI43rnQoxqfO+w71ZlU27pQ/sVlrWge7XwBoWSVLxemoml91gQzG1tf/sz7mtQ9B CuAisIeupIx84L/7BmCPhhWqeYo0efEyMvf2+rbGzFA4QFuIUzqlC3oPwLWNeapC3Ngh Dw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 33gdycv10x-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 11 Sep 2020 16:36:55 -0400 Received: from m0098396.ppops.net (m0098396.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 08BKXDuQ172749; Fri, 11 Sep 2020 16:36:54 -0400 Received: from ppma04ams.nl.ibm.com (63.31.33a9.ip4.static.sl-reverse.com [169.51.49.99]) by mx0a-001b2d01.pphosted.com with ESMTP id 33gdycv0yy-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 11 Sep 2020 16:36:54 -0400 Received: from pps.filterd (ppma04ams.nl.ibm.com [127.0.0.1]) by ppma04ams.nl.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 08BKapqm003573; Fri, 11 Sep 2020 20:36:51 GMT Received: from b06cxnps4075.portsmouth.uk.ibm.com (d06relay12.portsmouth.uk.ibm.com [9.149.109.197]) by ppma04ams.nl.ibm.com with ESMTP id 33c2a8fn1f-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 11 Sep 2020 20:36:51 +0000 Received: from d06av26.portsmouth.uk.ibm.com (d06av26.portsmouth.uk.ibm.com [9.149.105.62]) by b06cxnps4075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 08BKamlD33358238 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 11 Sep 2020 20:36:48 GMT Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 536EAAE051; Fri, 11 Sep 2020 20:36:48 +0000 (GMT) Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 70537AE04D; Fri, 11 Sep 2020 20:36:46 +0000 (GMT) Received: from localhost (unknown [9.145.43.16]) by d06av26.portsmouth.uk.ibm.com (Postfix) with ESMTPS; Fri, 11 Sep 2020 20:36:46 +0000 (GMT) Date: Fri, 11 Sep 2020 22:36:43 +0200 From: Vasily Gorbik To: Jason Gunthorpe , John Hubbard Cc: Linus Torvalds , Gerald Schaefer , Alexander Gordeev , Peter Zijlstra , Dave Hansen , LKML , linux-mm , linux-arch , Andrew Morton , Russell King , Mike Rapoport , Catalin Marinas , Will Deacon , Michael Ellerman , Benjamin Herrenschmidt , Paul Mackerras , Jeff Dike , Richard Weinberger , Dave Hansen , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Arnd Bergmann , Andrey Ryabinin , linux-x86 , linux-arm , linux-power , linux-sparc , linux-um , linux-s390 , Heiko Carstens , Christian Borntraeger , Claudio Imbrenda Subject: [PATCH v2] mm/gup: fix gup_fast with dynamic page table folding Message-ID: References: <20200911200511.GC1221970@ziepe.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20200911200511.GC1221970@ziepe.ca> X-Patchwork-Bot: notify X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.687 definitions=2020-09-11_10:2020-09-10,2020-09-11 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 mlxlogscore=999 adultscore=0 impostorscore=0 mlxscore=0 malwarescore=0 suspectscore=0 lowpriorityscore=0 spamscore=0 phishscore=0 priorityscore=1501 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2006250000 definitions=main-2009110162 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Currently to make sure that every page table entry is read just once gup_fast walks perform READ_ONCE and pass pXd value down to the next gup_pXd_range function by value e.g.: static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end, unsigned int flags, struct page **pages, int *nr) ... pudp = pud_offset(&p4d, addr); This function passes a reference on that local value copy to pXd_offset, and might get the very same pointer in return. This happens when the level is folded (on most arches), and that pointer should not be iterated. On s390 due to the fact that each task might have different 5,4 or 3-level address translation and hence different levels folded the logic is more complex and non-iteratable pointer to a local copy leads to severe problems. Here is an example of what happens with gup_fast on s390, for a task with 3-levels paging, crossing a 2 GB pud boundary: // addr = 0x1007ffff000, end = 0x10080001000 static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end, unsigned int flags, struct page **pages, int *nr) { unsigned long next; pud_t *pudp; // pud_offset returns &p4d itself (a pointer to a value on stack) pudp = pud_offset(&p4d, addr); do { // on second iteratation reading "random" stack value pud_t pud = READ_ONCE(*pudp); // next = 0x10080000000, due to PUD_SIZE/MASK != PGDIR_SIZE/MASK on s390 next = pud_addr_end(addr, end); ... } while (pudp++, addr = next, addr != end); // pudp++ iterating over stack return 1; } This happens since s390 moved to common gup code with commit d1874a0c2805 ("s390/mm: make the pxd_offset functions more robust") and commit 1a42010cdc26 ("s390/mm: convert to the generic get_user_pages_fast code"). s390 tried to mimic static level folding by changing pXd_offset primitives to always calculate top level page table offset in pgd_offset and just return the value passed when pXd_offset has to act as folded. What is crucial for gup_fast and what has been overlooked is that PxD_SIZE/MASK and thus pXd_addr_end should also change correspondingly. And the latter is not possible with dynamic folding. To fix the issue in addition to pXd values pass original pXdp pointers down to gup_pXd_range functions. And introduce pXd_offset_lockless helpers, which take an additional pXd entry value parameter. This has already been discussed in https://lkml.kernel.org/r/20190418100218.0a4afd51@mschwideX1 Cc: # 5.2+ Fixes: 1a42010cdc26 ("s390/mm: convert to the generic get_user_pages_fast code") Reviewed-by: Gerald Schaefer Reviewed-by: Alexander Gordeev Signed-off-by: Vasily Gorbik --- v2: added brackets &pgd -> &(pgd) arch/s390/include/asm/pgtable.h | 42 +++++++++++++++++++++++---------- include/linux/pgtable.h | 10 ++++++++ mm/gup.c | 18 +++++++------- 3 files changed, 49 insertions(+), 21 deletions(-) diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h index 7eb01a5459cd..b55561cc8786 100644 --- a/arch/s390/include/asm/pgtable.h +++ b/arch/s390/include/asm/pgtable.h @@ -1260,26 +1260,44 @@ static inline pgd_t *pgd_offset_raw(pgd_t *pgd, unsigned long address) #define pgd_offset(mm, address) pgd_offset_raw(READ_ONCE((mm)->pgd), address) -static inline p4d_t *p4d_offset(pgd_t *pgd, unsigned long address) +static inline p4d_t *p4d_offset_lockless(pgd_t *pgdp, pgd_t pgd, unsigned long address) { - if ((pgd_val(*pgd) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R1) - return (p4d_t *) pgd_deref(*pgd) + p4d_index(address); - return (p4d_t *) pgd; + if ((pgd_val(pgd) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R1) + return (p4d_t *) pgd_deref(pgd) + p4d_index(address); + return (p4d_t *) pgdp; } +#define p4d_offset_lockless p4d_offset_lockless -static inline pud_t *pud_offset(p4d_t *p4d, unsigned long address) +static inline p4d_t *p4d_offset(pgd_t *pgdp, unsigned long address) { - if ((p4d_val(*p4d) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R2) - return (pud_t *) p4d_deref(*p4d) + pud_index(address); - return (pud_t *) p4d; + return p4d_offset_lockless(pgdp, *pgdp, address); +} + +static inline pud_t *pud_offset_lockless(p4d_t *p4dp, p4d_t p4d, unsigned long address) +{ + if ((p4d_val(p4d) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R2) + return (pud_t *) p4d_deref(p4d) + pud_index(address); + return (pud_t *) p4dp; +} +#define pud_offset_lockless pud_offset_lockless + +static inline pud_t *pud_offset(p4d_t *p4dp, unsigned long address) +{ + return pud_offset_lockless(p4dp, *p4dp, address); } #define pud_offset pud_offset -static inline pmd_t *pmd_offset(pud_t *pud, unsigned long address) +static inline pmd_t *pmd_offset_lockless(pud_t *pudp, pud_t pud, unsigned long address) +{ + if ((pud_val(pud) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R3) + return (pmd_t *) pud_deref(pud) + pmd_index(address); + return (pmd_t *) pudp; +} +#define pmd_offset_lockless pmd_offset_lockless + +static inline pmd_t *pmd_offset(pud_t *pudp, unsigned long address) { - if ((pud_val(*pud) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R3) - return (pmd_t *) pud_deref(*pud) + pmd_index(address); - return (pmd_t *) pud; + return pmd_offset_lockless(pudp, *pudp, address); } #define pmd_offset pmd_offset diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index e8cbc2e795d5..90654cb63e9e 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1427,6 +1427,16 @@ typedef unsigned int pgtbl_mod_mask; #define mm_pmd_folded(mm) __is_defined(__PAGETABLE_PMD_FOLDED) #endif +#ifndef p4d_offset_lockless +#define p4d_offset_lockless(pgdp, pgd, address) p4d_offset(&(pgd), address) +#endif +#ifndef pud_offset_lockless +#define pud_offset_lockless(p4dp, p4d, address) pud_offset(&(p4d), address) +#endif +#ifndef pmd_offset_lockless +#define pmd_offset_lockless(pudp, pud, address) pmd_offset(&(pud), address) +#endif + /* * p?d_leaf() - true if this entry is a final mapping to a physical address. * This differs from p?d_huge() by the fact that they are always available (if diff --git a/mm/gup.c b/mm/gup.c index e5739a1974d5..578bf5bd8bf8 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2485,13 +2485,13 @@ static int gup_huge_pgd(pgd_t orig, pgd_t *pgdp, unsigned long addr, return 1; } -static int gup_pmd_range(pud_t pud, unsigned long addr, unsigned long end, +static int gup_pmd_range(pud_t *pudp, pud_t pud, unsigned long addr, unsigned long end, unsigned int flags, struct page **pages, int *nr) { unsigned long next; pmd_t *pmdp; - pmdp = pmd_offset(&pud, addr); + pmdp = pmd_offset_lockless(pudp, pud, addr); do { pmd_t pmd = READ_ONCE(*pmdp); @@ -2528,13 +2528,13 @@ static int gup_pmd_range(pud_t pud, unsigned long addr, unsigned long end, return 1; } -static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end, +static int gup_pud_range(p4d_t *p4dp, p4d_t p4d, unsigned long addr, unsigned long end, unsigned int flags, struct page **pages, int *nr) { unsigned long next; pud_t *pudp; - pudp = pud_offset(&p4d, addr); + pudp = pud_offset_lockless(p4dp, p4d, addr); do { pud_t pud = READ_ONCE(*pudp); @@ -2549,20 +2549,20 @@ static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end, if (!gup_huge_pd(__hugepd(pud_val(pud)), addr, PUD_SHIFT, next, flags, pages, nr)) return 0; - } else if (!gup_pmd_range(pud, addr, next, flags, pages, nr)) + } else if (!gup_pmd_range(pudp, pud, addr, next, flags, pages, nr)) return 0; } while (pudp++, addr = next, addr != end); return 1; } -static int gup_p4d_range(pgd_t pgd, unsigned long addr, unsigned long end, +static int gup_p4d_range(pgd_t *pgdp, pgd_t pgd, unsigned long addr, unsigned long end, unsigned int flags, struct page **pages, int *nr) { unsigned long next; p4d_t *p4dp; - p4dp = p4d_offset(&pgd, addr); + p4dp = p4d_offset_lockless(pgdp, pgd, addr); do { p4d_t p4d = READ_ONCE(*p4dp); @@ -2574,7 +2574,7 @@ static int gup_p4d_range(pgd_t pgd, unsigned long addr, unsigned long end, if (!gup_huge_pd(__hugepd(p4d_val(p4d)), addr, P4D_SHIFT, next, flags, pages, nr)) return 0; - } else if (!gup_pud_range(p4d, addr, next, flags, pages, nr)) + } else if (!gup_pud_range(p4dp, p4d, addr, next, flags, pages, nr)) return 0; } while (p4dp++, addr = next, addr != end); @@ -2602,7 +2602,7 @@ static void gup_pgd_range(unsigned long addr, unsigned long end, if (!gup_huge_pd(__hugepd(pgd_val(pgd)), addr, PGDIR_SHIFT, next, flags, pages, nr)) return; - } else if (!gup_p4d_range(pgd, addr, next, flags, pages, nr)) + } else if (!gup_p4d_range(pgdp, pgd, addr, next, flags, pages, nr)) return; } while (pgdp++, addr = next, addr != end); } -- ⣿⣿⣿⣿⢋⡀⣀⠹⣿⣿⣿⣿ ⣿⣿⣿⣿⠠⣶⡦⠀⣿⣿⣿⣿ ⣿⣿⣿⠏⣴⣮⣴⣧⠈⢿⣿⣿ ⣿⣿⡏⢰⣿⠖⣠⣿⡆⠈⣿⣿ ⣿⢛⣵⣄⠙⣶⣶⡟⣅⣠⠹⣿ ⣿⣜⣛⠻⢎⣉⣉⣀⠿⣫⣵⣿ From mboxrd@z Thu Jan 1 00:00:00 1970 From: Vasily Gorbik Date: Fri, 11 Sep 2020 20:36:43 +0000 Subject: [PATCH v2] mm/gup: fix gup_fast with dynamic page table folding Message-Id: List-Id: References: <20200911200511.GC1221970@ziepe.ca> In-Reply-To: <20200911200511.GC1221970@ziepe.ca> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit To: Jason Gunthorpe , John Hubbard Cc: Linus Torvalds , Gerald Schaefer , Alexander Gordeev , Peter Zijlstra , Dave Hansen , LKML , linux-mm , linux-arch , Andrew Morton , Russell King , Mike Rapoport , Catalin Marinas , Will Deacon , Michael Ellerman , Benjamin Herrenschmidt , Paul Mackerras , Jeff Dike , Richard Weinberger , Dave Hansen , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Arnd Bergmann , Andrey Ryabinin , linux-x86 , linux-arm , linux-power , linux-sparc , linux-um , linux-s390 , Heiko Carstens , Christian Borntraeger , Claudio Imbrenda Currently to make sure that every page table entry is read just once gup_fast walks perform READ_ONCE and pass pXd value down to the next gup_pXd_range function by value e.g.: static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end, unsigned int flags, struct page **pages, int *nr) ... pudp = pud_offset(&p4d, addr); This function passes a reference on that local value copy to pXd_offset, and might get the very same pointer in return. This happens when the level is folded (on most arches), and that pointer should not be iterated. On s390 due to the fact that each task might have different 5,4 or 3-level address translation and hence different levels folded the logic is more complex and non-iteratable pointer to a local copy leads to severe problems. Here is an example of what happens with gup_fast on s390, for a task with 3-levels paging, crossing a 2 GB pud boundary: // addr = 0x1007ffff000, end = 0x10080001000 static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end, unsigned int flags, struct page **pages, int *nr) { unsigned long next; pud_t *pudp; // pud_offset returns &p4d itself (a pointer to a value on stack) pudp = pud_offset(&p4d, addr); do { // on second iteratation reading "random" stack value pud_t pud = READ_ONCE(*pudp); // next = 0x10080000000, due to PUD_SIZE/MASK != PGDIR_SIZE/MASK on s390 next = pud_addr_end(addr, end); ... } while (pudp++, addr = next, addr != end); // pudp++ iterating over stack return 1; } This happens since s390 moved to common gup code with commit d1874a0c2805 ("s390/mm: make the pxd_offset functions more robust") and commit 1a42010cdc26 ("s390/mm: convert to the generic get_user_pages_fast code"). s390 tried to mimic static level folding by changing pXd_offset primitives to always calculate top level page table offset in pgd_offset and just return the value passed when pXd_offset has to act as folded. What is crucial for gup_fast and what has been overlooked is that PxD_SIZE/MASK and thus pXd_addr_end should also change correspondingly. And the latter is not possible with dynamic folding. To fix the issue in addition to pXd values pass original pXdp pointers down to gup_pXd_range functions. And introduce pXd_offset_lockless helpers, which take an additional pXd entry value parameter. This has already been discussed in https://lkml.kernel.org/r/20190418100218.0a4afd51@mschwideX1 Cc: # 5.2+ Fixes: 1a42010cdc26 ("s390/mm: convert to the generic get_user_pages_fast code") Reviewed-by: Gerald Schaefer Reviewed-by: Alexander Gordeev Signed-off-by: Vasily Gorbik --- v2: added brackets &pgd -> &(pgd) arch/s390/include/asm/pgtable.h | 42 +++++++++++++++++++++++---------- include/linux/pgtable.h | 10 ++++++++ mm/gup.c | 18 +++++++------- 3 files changed, 49 insertions(+), 21 deletions(-) diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h index 7eb01a5459cd..b55561cc8786 100644 --- a/arch/s390/include/asm/pgtable.h +++ b/arch/s390/include/asm/pgtable.h @@ -1260,26 +1260,44 @@ static inline pgd_t *pgd_offset_raw(pgd_t *pgd, unsigned long address) #define pgd_offset(mm, address) pgd_offset_raw(READ_ONCE((mm)->pgd), address) -static inline p4d_t *p4d_offset(pgd_t *pgd, unsigned long address) +static inline p4d_t *p4d_offset_lockless(pgd_t *pgdp, pgd_t pgd, unsigned long address) { - if ((pgd_val(*pgd) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R1) - return (p4d_t *) pgd_deref(*pgd) + p4d_index(address); - return (p4d_t *) pgd; + if ((pgd_val(pgd) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R1) + return (p4d_t *) pgd_deref(pgd) + p4d_index(address); + return (p4d_t *) pgdp; } +#define p4d_offset_lockless p4d_offset_lockless -static inline pud_t *pud_offset(p4d_t *p4d, unsigned long address) +static inline p4d_t *p4d_offset(pgd_t *pgdp, unsigned long address) { - if ((p4d_val(*p4d) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R2) - return (pud_t *) p4d_deref(*p4d) + pud_index(address); - return (pud_t *) p4d; + return p4d_offset_lockless(pgdp, *pgdp, address); +} + +static inline pud_t *pud_offset_lockless(p4d_t *p4dp, p4d_t p4d, unsigned long address) +{ + if ((p4d_val(p4d) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R2) + return (pud_t *) p4d_deref(p4d) + pud_index(address); + return (pud_t *) p4dp; +} +#define pud_offset_lockless pud_offset_lockless + +static inline pud_t *pud_offset(p4d_t *p4dp, unsigned long address) +{ + return pud_offset_lockless(p4dp, *p4dp, address); } #define pud_offset pud_offset -static inline pmd_t *pmd_offset(pud_t *pud, unsigned long address) +static inline pmd_t *pmd_offset_lockless(pud_t *pudp, pud_t pud, unsigned long address) +{ + if ((pud_val(pud) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R3) + return (pmd_t *) pud_deref(pud) + pmd_index(address); + return (pmd_t *) pudp; +} +#define pmd_offset_lockless pmd_offset_lockless + +static inline pmd_t *pmd_offset(pud_t *pudp, unsigned long address) { - if ((pud_val(*pud) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R3) - return (pmd_t *) pud_deref(*pud) + pmd_index(address); - return (pmd_t *) pud; + return pmd_offset_lockless(pudp, *pudp, address); } #define pmd_offset pmd_offset diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index e8cbc2e795d5..90654cb63e9e 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1427,6 +1427,16 @@ typedef unsigned int pgtbl_mod_mask; #define mm_pmd_folded(mm) __is_defined(__PAGETABLE_PMD_FOLDED) #endif +#ifndef p4d_offset_lockless +#define p4d_offset_lockless(pgdp, pgd, address) p4d_offset(&(pgd), address) +#endif +#ifndef pud_offset_lockless +#define pud_offset_lockless(p4dp, p4d, address) pud_offset(&(p4d), address) +#endif +#ifndef pmd_offset_lockless +#define pmd_offset_lockless(pudp, pud, address) pmd_offset(&(pud), address) +#endif + /* * p?d_leaf() - true if this entry is a final mapping to a physical address. * This differs from p?d_huge() by the fact that they are always available (if diff --git a/mm/gup.c b/mm/gup.c index e5739a1974d5..578bf5bd8bf8 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2485,13 +2485,13 @@ static int gup_huge_pgd(pgd_t orig, pgd_t *pgdp, unsigned long addr, return 1; } -static int gup_pmd_range(pud_t pud, unsigned long addr, unsigned long end, +static int gup_pmd_range(pud_t *pudp, pud_t pud, unsigned long addr, unsigned long end, unsigned int flags, struct page **pages, int *nr) { unsigned long next; pmd_t *pmdp; - pmdp = pmd_offset(&pud, addr); + pmdp = pmd_offset_lockless(pudp, pud, addr); do { pmd_t pmd = READ_ONCE(*pmdp); @@ -2528,13 +2528,13 @@ static int gup_pmd_range(pud_t pud, unsigned long addr, unsigned long end, return 1; } -static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end, +static int gup_pud_range(p4d_t *p4dp, p4d_t p4d, unsigned long addr, unsigned long end, unsigned int flags, struct page **pages, int *nr) { unsigned long next; pud_t *pudp; - pudp = pud_offset(&p4d, addr); + pudp = pud_offset_lockless(p4dp, p4d, addr); do { pud_t pud = READ_ONCE(*pudp); @@ -2549,20 +2549,20 @@ static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end, if (!gup_huge_pd(__hugepd(pud_val(pud)), addr, PUD_SHIFT, next, flags, pages, nr)) return 0; - } else if (!gup_pmd_range(pud, addr, next, flags, pages, nr)) + } else if (!gup_pmd_range(pudp, pud, addr, next, flags, pages, nr)) return 0; } while (pudp++, addr = next, addr != end); return 1; } -static int gup_p4d_range(pgd_t pgd, unsigned long addr, unsigned long end, +static int gup_p4d_range(pgd_t *pgdp, pgd_t pgd, unsigned long addr, unsigned long end, unsigned int flags, struct page **pages, int *nr) { unsigned long next; p4d_t *p4dp; - p4dp = p4d_offset(&pgd, addr); + p4dp = p4d_offset_lockless(pgdp, pgd, addr); do { p4d_t p4d = READ_ONCE(*p4dp); @@ -2574,7 +2574,7 @@ static int gup_p4d_range(pgd_t pgd, unsigned long addr, unsigned long end, if (!gup_huge_pd(__hugepd(p4d_val(p4d)), addr, P4D_SHIFT, next, flags, pages, nr)) return 0; - } else if (!gup_pud_range(p4d, addr, next, flags, pages, nr)) + } else if (!gup_pud_range(p4dp, p4d, addr, next, flags, pages, nr)) return 0; } while (p4dp++, addr = next, addr != end); @@ -2602,7 +2602,7 @@ static void gup_pgd_range(unsigned long addr, unsigned long end, if (!gup_huge_pd(__hugepd(pgd_val(pgd)), addr, PGDIR_SHIFT, next, flags, pages, nr)) return; - } else if (!gup_p4d_range(pgd, addr, next, flags, pages, nr)) + } else if (!gup_p4d_range(pgdp, pgd, addr, next, flags, pages, nr)) return; } while (pgdp++, addr = next, addr != end); } -- ⣿⣿⣿⣿⢋⡀⣀⠹⣿⣿⣿⣿ ⣿⣿⣿⣿⠠⣶⡦⠀⣿⣿⣿⣿ ⣿⣿⣿⠏⣴⣮⣴⣧⠈⢿⣿⣿ ⣿⣿⡏⢰⣿⠖⣠⣿⡆⠈⣿⣿ ⣿⢛⣵⣄⠙⣶⣶⡟⣅⣠⠹⣿ ⣿⣜⣛⠻⢎⣉⣉⣀⠿⣫⣵⣿ From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C2D73C433E2 for ; Fri, 11 Sep 2020 23:13:32 +0000 (UTC) Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id ABE35221F0 for ; Fri, 11 Sep 2020 23:13:31 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="cz3YsExB" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org ABE35221F0 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4BpBQ123nlzDqyb for ; Sat, 12 Sep 2020 09:13:29 +1000 (AEST) Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=linux.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=gor@linux.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=cz3YsExB; dkim-atps=neutral Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Bp6y00kR8zDqvR for ; Sat, 12 Sep 2020 06:37:27 +1000 (AEST) Received: from pps.filterd (m0098396.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 08BKWdvP170701; Fri, 11 Sep 2020 16:36:55 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=date : from : to : cc : subject : message-id : references : mime-version : content-type : content-transfer-encoding : in-reply-to; s=pp1; bh=hOcVC6Va1COznQ0oD3ZHmavI0zSfYZBaHJyzu45rYTc=; b=cz3YsExB/lDwLPfrpKWbTdWb5Qzbr5rHXXf/uwggBSWxDSVGHw4DszUvmlUKbaJ1o5Ko TzxhoMu/K5Gf0zonK+k5Y8vnz5rB9fu90H1Nobz5XHKeaPkNmM3vBNK8py3MV/KXKyiD LPBS5/2S4QWDUBed8/p5N9fWoVqnK6U0OjxWZe6GI5Fd9pkYNvx1+FAu3vrbG1i2q76k +zPztI43rnQoxqfO+w71ZlU27pQ/sVlrWge7XwBoWSVLxemoml91gQzG1tf/sz7mtQ9B CuAisIeupIx84L/7BmCPhhWqeYo0efEyMvf2+rbGzFA4QFuIUzqlC3oPwLWNeapC3Ngh Dw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 33gdycv10x-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 11 Sep 2020 16:36:55 -0400 Received: from m0098396.ppops.net (m0098396.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 08BKXDuQ172749; Fri, 11 Sep 2020 16:36:54 -0400 Received: from ppma04ams.nl.ibm.com (63.31.33a9.ip4.static.sl-reverse.com [169.51.49.99]) by mx0a-001b2d01.pphosted.com with ESMTP id 33gdycv0yy-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 11 Sep 2020 16:36:54 -0400 Received: from pps.filterd (ppma04ams.nl.ibm.com [127.0.0.1]) by ppma04ams.nl.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 08BKapqm003573; Fri, 11 Sep 2020 20:36:51 GMT Received: from b06cxnps4075.portsmouth.uk.ibm.com (d06relay12.portsmouth.uk.ibm.com [9.149.109.197]) by ppma04ams.nl.ibm.com with ESMTP id 33c2a8fn1f-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 11 Sep 2020 20:36:51 +0000 Received: from d06av26.portsmouth.uk.ibm.com (d06av26.portsmouth.uk.ibm.com [9.149.105.62]) by b06cxnps4075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 08BKamlD33358238 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 11 Sep 2020 20:36:48 GMT Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 536EAAE051; Fri, 11 Sep 2020 20:36:48 +0000 (GMT) Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 70537AE04D; Fri, 11 Sep 2020 20:36:46 +0000 (GMT) Received: from localhost (unknown [9.145.43.16]) by d06av26.portsmouth.uk.ibm.com (Postfix) with ESMTPS; Fri, 11 Sep 2020 20:36:46 +0000 (GMT) Date: Fri, 11 Sep 2020 22:36:43 +0200 From: Vasily Gorbik To: Jason Gunthorpe , John Hubbard Subject: [PATCH v2] mm/gup: fix gup_fast with dynamic page table folding Message-ID: References: <20200911200511.GC1221970@ziepe.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20200911200511.GC1221970@ziepe.ca> X-Patchwork-Bot: notify X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235, 18.0.687 definitions=2020-09-11_10:2020-09-10, 2020-09-11 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 mlxlogscore=999 adultscore=0 impostorscore=0 mlxscore=0 malwarescore=0 suspectscore=0 lowpriorityscore=0 spamscore=0 phishscore=0 priorityscore=1501 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2006250000 definitions=main-2009110162 X-Mailman-Approved-At: Sat, 12 Sep 2020 09:10:03 +1000 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Zijlstra , Dave Hansen , linux-mm , Paul Mackerras , linux-sparc , Alexander Gordeev , Claudio Imbrenda , Will Deacon , linux-arch , linux-s390 , Richard Weinberger , linux-x86 , Russell King , Christian Borntraeger , Ingo Molnar , Catalin Marinas , Andrey Ryabinin , Gerald Schaefer , Heiko Carstens , Arnd Bergmann , Jeff Dike , linux-um , Borislav Petkov , Andy Lutomirski , Thomas Gleixner , linux-arm , Dave Hansen , linux-power , LKML , Andrew Morton , Linus Torvalds , Mike Rapoport Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" Currently to make sure that every page table entry is read just once gup_fast walks perform READ_ONCE and pass pXd value down to the next gup_pXd_range function by value e.g.: static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end, unsigned int flags, struct page **pages, int *nr) ... pudp = pud_offset(&p4d, addr); This function passes a reference on that local value copy to pXd_offset, and might get the very same pointer in return. This happens when the level is folded (on most arches), and that pointer should not be iterated. On s390 due to the fact that each task might have different 5,4 or 3-level address translation and hence different levels folded the logic is more complex and non-iteratable pointer to a local copy leads to severe problems. Here is an example of what happens with gup_fast on s390, for a task with 3-levels paging, crossing a 2 GB pud boundary: // addr = 0x1007ffff000, end = 0x10080001000 static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end, unsigned int flags, struct page **pages, int *nr) { unsigned long next; pud_t *pudp; // pud_offset returns &p4d itself (a pointer to a value on stack) pudp = pud_offset(&p4d, addr); do { // on second iteratation reading "random" stack value pud_t pud = READ_ONCE(*pudp); // next = 0x10080000000, due to PUD_SIZE/MASK != PGDIR_SIZE/MASK on s390 next = pud_addr_end(addr, end); ... } while (pudp++, addr = next, addr != end); // pudp++ iterating over stack return 1; } This happens since s390 moved to common gup code with commit d1874a0c2805 ("s390/mm: make the pxd_offset functions more robust") and commit 1a42010cdc26 ("s390/mm: convert to the generic get_user_pages_fast code"). s390 tried to mimic static level folding by changing pXd_offset primitives to always calculate top level page table offset in pgd_offset and just return the value passed when pXd_offset has to act as folded. What is crucial for gup_fast and what has been overlooked is that PxD_SIZE/MASK and thus pXd_addr_end should also change correspondingly. And the latter is not possible with dynamic folding. To fix the issue in addition to pXd values pass original pXdp pointers down to gup_pXd_range functions. And introduce pXd_offset_lockless helpers, which take an additional pXd entry value parameter. This has already been discussed in https://lkml.kernel.org/r/20190418100218.0a4afd51@mschwideX1 Cc: # 5.2+ Fixes: 1a42010cdc26 ("s390/mm: convert to the generic get_user_pages_fast code") Reviewed-by: Gerald Schaefer Reviewed-by: Alexander Gordeev Signed-off-by: Vasily Gorbik --- v2: added brackets &pgd -> &(pgd) arch/s390/include/asm/pgtable.h | 42 +++++++++++++++++++++++---------- include/linux/pgtable.h | 10 ++++++++ mm/gup.c | 18 +++++++------- 3 files changed, 49 insertions(+), 21 deletions(-) diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h index 7eb01a5459cd..b55561cc8786 100644 --- a/arch/s390/include/asm/pgtable.h +++ b/arch/s390/include/asm/pgtable.h @@ -1260,26 +1260,44 @@ static inline pgd_t *pgd_offset_raw(pgd_t *pgd, unsigned long address) #define pgd_offset(mm, address) pgd_offset_raw(READ_ONCE((mm)->pgd), address) -static inline p4d_t *p4d_offset(pgd_t *pgd, unsigned long address) +static inline p4d_t *p4d_offset_lockless(pgd_t *pgdp, pgd_t pgd, unsigned long address) { - if ((pgd_val(*pgd) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R1) - return (p4d_t *) pgd_deref(*pgd) + p4d_index(address); - return (p4d_t *) pgd; + if ((pgd_val(pgd) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R1) + return (p4d_t *) pgd_deref(pgd) + p4d_index(address); + return (p4d_t *) pgdp; } +#define p4d_offset_lockless p4d_offset_lockless -static inline pud_t *pud_offset(p4d_t *p4d, unsigned long address) +static inline p4d_t *p4d_offset(pgd_t *pgdp, unsigned long address) { - if ((p4d_val(*p4d) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R2) - return (pud_t *) p4d_deref(*p4d) + pud_index(address); - return (pud_t *) p4d; + return p4d_offset_lockless(pgdp, *pgdp, address); +} + +static inline pud_t *pud_offset_lockless(p4d_t *p4dp, p4d_t p4d, unsigned long address) +{ + if ((p4d_val(p4d) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R2) + return (pud_t *) p4d_deref(p4d) + pud_index(address); + return (pud_t *) p4dp; +} +#define pud_offset_lockless pud_offset_lockless + +static inline pud_t *pud_offset(p4d_t *p4dp, unsigned long address) +{ + return pud_offset_lockless(p4dp, *p4dp, address); } #define pud_offset pud_offset -static inline pmd_t *pmd_offset(pud_t *pud, unsigned long address) +static inline pmd_t *pmd_offset_lockless(pud_t *pudp, pud_t pud, unsigned long address) +{ + if ((pud_val(pud) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R3) + return (pmd_t *) pud_deref(pud) + pmd_index(address); + return (pmd_t *) pudp; +} +#define pmd_offset_lockless pmd_offset_lockless + +static inline pmd_t *pmd_offset(pud_t *pudp, unsigned long address) { - if ((pud_val(*pud) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R3) - return (pmd_t *) pud_deref(*pud) + pmd_index(address); - return (pmd_t *) pud; + return pmd_offset_lockless(pudp, *pudp, address); } #define pmd_offset pmd_offset diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index e8cbc2e795d5..90654cb63e9e 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1427,6 +1427,16 @@ typedef unsigned int pgtbl_mod_mask; #define mm_pmd_folded(mm) __is_defined(__PAGETABLE_PMD_FOLDED) #endif +#ifndef p4d_offset_lockless +#define p4d_offset_lockless(pgdp, pgd, address) p4d_offset(&(pgd), address) +#endif +#ifndef pud_offset_lockless +#define pud_offset_lockless(p4dp, p4d, address) pud_offset(&(p4d), address) +#endif +#ifndef pmd_offset_lockless +#define pmd_offset_lockless(pudp, pud, address) pmd_offset(&(pud), address) +#endif + /* * p?d_leaf() - true if this entry is a final mapping to a physical address. * This differs from p?d_huge() by the fact that they are always available (if diff --git a/mm/gup.c b/mm/gup.c index e5739a1974d5..578bf5bd8bf8 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2485,13 +2485,13 @@ static int gup_huge_pgd(pgd_t orig, pgd_t *pgdp, unsigned long addr, return 1; } -static int gup_pmd_range(pud_t pud, unsigned long addr, unsigned long end, +static int gup_pmd_range(pud_t *pudp, pud_t pud, unsigned long addr, unsigned long end, unsigned int flags, struct page **pages, int *nr) { unsigned long next; pmd_t *pmdp; - pmdp = pmd_offset(&pud, addr); + pmdp = pmd_offset_lockless(pudp, pud, addr); do { pmd_t pmd = READ_ONCE(*pmdp); @@ -2528,13 +2528,13 @@ static int gup_pmd_range(pud_t pud, unsigned long addr, unsigned long end, return 1; } -static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end, +static int gup_pud_range(p4d_t *p4dp, p4d_t p4d, unsigned long addr, unsigned long end, unsigned int flags, struct page **pages, int *nr) { unsigned long next; pud_t *pudp; - pudp = pud_offset(&p4d, addr); + pudp = pud_offset_lockless(p4dp, p4d, addr); do { pud_t pud = READ_ONCE(*pudp); @@ -2549,20 +2549,20 @@ static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end, if (!gup_huge_pd(__hugepd(pud_val(pud)), addr, PUD_SHIFT, next, flags, pages, nr)) return 0; - } else if (!gup_pmd_range(pud, addr, next, flags, pages, nr)) + } else if (!gup_pmd_range(pudp, pud, addr, next, flags, pages, nr)) return 0; } while (pudp++, addr = next, addr != end); return 1; } -static int gup_p4d_range(pgd_t pgd, unsigned long addr, unsigned long end, +static int gup_p4d_range(pgd_t *pgdp, pgd_t pgd, unsigned long addr, unsigned long end, unsigned int flags, struct page **pages, int *nr) { unsigned long next; p4d_t *p4dp; - p4dp = p4d_offset(&pgd, addr); + p4dp = p4d_offset_lockless(pgdp, pgd, addr); do { p4d_t p4d = READ_ONCE(*p4dp); @@ -2574,7 +2574,7 @@ static int gup_p4d_range(pgd_t pgd, unsigned long addr, unsigned long end, if (!gup_huge_pd(__hugepd(p4d_val(p4d)), addr, P4D_SHIFT, next, flags, pages, nr)) return 0; - } else if (!gup_pud_range(p4d, addr, next, flags, pages, nr)) + } else if (!gup_pud_range(p4dp, p4d, addr, next, flags, pages, nr)) return 0; } while (p4dp++, addr = next, addr != end); @@ -2602,7 +2602,7 @@ static void gup_pgd_range(unsigned long addr, unsigned long end, if (!gup_huge_pd(__hugepd(pgd_val(pgd)), addr, PGDIR_SHIFT, next, flags, pages, nr)) return; - } else if (!gup_p4d_range(pgd, addr, next, flags, pages, nr)) + } else if (!gup_p4d_range(pgdp, pgd, addr, next, flags, pages, nr)) return; } while (pgdp++, addr = next, addr != end); } -- ⣿⣿⣿⣿⢋⡀⣀⠹⣿⣿⣿⣿ ⣿⣿⣿⣿⠠⣶⡦⠀⣿⣿⣿⣿ ⣿⣿⣿⠏⣴⣮⣴⣧⠈⢿⣿⣿ ⣿⣿⡏⢰⣿⠖⣠⣿⡆⠈⣿⣿ ⣿⢛⣵⣄⠙⣶⣶⡟⣅⣠⠹⣿ ⣿⣜⣛⠻⢎⣉⣉⣀⠿⣫⣵⣿ From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D46F1C433E2 for ; Fri, 11 Sep 2020 20:39:04 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 036B62075A for ; Fri, 11 Sep 2020 20:39:03 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="22RT9p59"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="cz3YsExB" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 036B62075A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.ibm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References:Message-ID: Subject:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=C3wjr8n3LC5vkQMr0tISKAA6TjY+mr6N7tn+0LmQauU=; b=22RT9p59FOc0LqFE9KiPJ5+ng rUt8kMD8WBPuqqHsO26gcryFG1ek8i+CxbB1R/TzJVhjG00dew81oW7DePyTGahjkcTPfaBrrnw/c bSWYXNx3pIlGnVD2i6OWqh2hW5oBDYedT7Pv4M4P9eg2IirTXXD1+tsAWo8L5Wxxv+bt2AN5G626W cJtj63i0XJ49le3kPQ8sgvWKnbnKzpxqVL5p/b4HQBCC4e0Wz7fAjlCBmLQHxDpCXXHyOLbjP1iEQ b0jrfuUMXpgleV/hOB6yoknc3/Ciqz3Rm8qUhOjjLAUDrXzzmRwRpB6tikLE4IsJZWToW62wfuxMj RaT4KJbSw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kGpnU-0007j9-C0; Fri, 11 Sep 2020 20:37:28 +0000 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kGpnQ-0007in-Ib; Fri, 11 Sep 2020 20:37:25 +0000 Received: from pps.filterd (m0098396.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 08BKWdvP170701; Fri, 11 Sep 2020 16:36:55 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=date : from : to : cc : subject : message-id : references : mime-version : content-type : content-transfer-encoding : in-reply-to; s=pp1; bh=hOcVC6Va1COznQ0oD3ZHmavI0zSfYZBaHJyzu45rYTc=; b=cz3YsExB/lDwLPfrpKWbTdWb5Qzbr5rHXXf/uwggBSWxDSVGHw4DszUvmlUKbaJ1o5Ko TzxhoMu/K5Gf0zonK+k5Y8vnz5rB9fu90H1Nobz5XHKeaPkNmM3vBNK8py3MV/KXKyiD LPBS5/2S4QWDUBed8/p5N9fWoVqnK6U0OjxWZe6GI5Fd9pkYNvx1+FAu3vrbG1i2q76k +zPztI43rnQoxqfO+w71ZlU27pQ/sVlrWge7XwBoWSVLxemoml91gQzG1tf/sz7mtQ9B CuAisIeupIx84L/7BmCPhhWqeYo0efEyMvf2+rbGzFA4QFuIUzqlC3oPwLWNeapC3Ngh Dw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 33gdycv10x-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 11 Sep 2020 16:36:55 -0400 Received: from m0098396.ppops.net (m0098396.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 08BKXDuQ172749; Fri, 11 Sep 2020 16:36:54 -0400 Received: from ppma04ams.nl.ibm.com (63.31.33a9.ip4.static.sl-reverse.com [169.51.49.99]) by mx0a-001b2d01.pphosted.com with ESMTP id 33gdycv0yy-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 11 Sep 2020 16:36:54 -0400 Received: from pps.filterd (ppma04ams.nl.ibm.com [127.0.0.1]) by ppma04ams.nl.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 08BKapqm003573; Fri, 11 Sep 2020 20:36:51 GMT Received: from b06cxnps4075.portsmouth.uk.ibm.com (d06relay12.portsmouth.uk.ibm.com [9.149.109.197]) by ppma04ams.nl.ibm.com with ESMTP id 33c2a8fn1f-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 11 Sep 2020 20:36:51 +0000 Received: from d06av26.portsmouth.uk.ibm.com (d06av26.portsmouth.uk.ibm.com [9.149.105.62]) by b06cxnps4075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 08BKamlD33358238 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 11 Sep 2020 20:36:48 GMT Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 536EAAE051; Fri, 11 Sep 2020 20:36:48 +0000 (GMT) Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 70537AE04D; Fri, 11 Sep 2020 20:36:46 +0000 (GMT) Received: from localhost (unknown [9.145.43.16]) by d06av26.portsmouth.uk.ibm.com (Postfix) with ESMTPS; Fri, 11 Sep 2020 20:36:46 +0000 (GMT) Date: Fri, 11 Sep 2020 22:36:43 +0200 From: Vasily Gorbik To: Jason Gunthorpe , John Hubbard Subject: [PATCH v2] mm/gup: fix gup_fast with dynamic page table folding Message-ID: References: <20200911200511.GC1221970@ziepe.ca> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20200911200511.GC1221970@ziepe.ca> X-Patchwork-Bot: notify X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235, 18.0.687 definitions=2020-09-11_10:2020-09-10, 2020-09-11 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 mlxlogscore=999 adultscore=0 impostorscore=0 mlxscore=0 malwarescore=0 suspectscore=0 lowpriorityscore=0 spamscore=0 phishscore=0 priorityscore=1501 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2006250000 definitions=main-2009110162 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200911_163724_747323_8A2239C9 X-CRM114-Status: GOOD ( 30.15 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Zijlstra , Benjamin Herrenschmidt , Dave Hansen , linux-mm , Paul Mackerras , linux-sparc , Alexander Gordeev , Claudio Imbrenda , Will Deacon , linux-arch , linux-s390 , Richard Weinberger , linux-x86 , Russell King , Christian Borntraeger , Ingo Molnar , Catalin Marinas , Andrey Ryabinin , Gerald Schaefer , Heiko Carstens , Arnd Bergmann , Jeff Dike , linux-um , Borislav Petkov , Andy Lutomirski , Thomas Gleixner , linux-arm , Dave Hansen , linux-power , LKML , Michael Ellerman , Andrew Morton , Linus Torvalds , Mike Rapoport Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Q3VycmVudGx5IHRvIG1ha2Ugc3VyZSB0aGF0IGV2ZXJ5IHBhZ2UgdGFibGUgZW50cnkgaXMgcmVh ZCBqdXN0IG9uY2UKZ3VwX2Zhc3Qgd2Fsa3MgcGVyZm9ybSBSRUFEX09OQ0UgYW5kIHBhc3MgcFhk IHZhbHVlIGRvd24gdG8gdGhlIG5leHQKZ3VwX3BYZF9yYW5nZSBmdW5jdGlvbiBieSB2YWx1ZSBl LmcuOgoKc3RhdGljIGludCBndXBfcHVkX3JhbmdlKHA0ZF90IHA0ZCwgdW5zaWduZWQgbG9uZyBh ZGRyLCB1bnNpZ25lZCBsb25nIGVuZCwKICAgICAgICAgICAgICAgICAgICAgICAgIHVuc2lnbmVk IGludCBmbGFncywgc3RydWN0IHBhZ2UgKipwYWdlcywgaW50ICpucikKLi4uCiAgICAgICAgcHVk cCA9IHB1ZF9vZmZzZXQoJnA0ZCwgYWRkcik7CgpUaGlzIGZ1bmN0aW9uIHBhc3NlcyBhIHJlZmVy ZW5jZSBvbiB0aGF0IGxvY2FsIHZhbHVlIGNvcHkgdG8gcFhkX29mZnNldCwKYW5kIG1pZ2h0IGdl dCB0aGUgdmVyeSBzYW1lIHBvaW50ZXIgaW4gcmV0dXJuLiBUaGlzIGhhcHBlbnMgd2hlbiB0aGUK bGV2ZWwgaXMgZm9sZGVkIChvbiBtb3N0IGFyY2hlcyksIGFuZCB0aGF0IHBvaW50ZXIgc2hvdWxk IG5vdCBiZSBpdGVyYXRlZC4KCk9uIHMzOTAgZHVlIHRvIHRoZSBmYWN0IHRoYXQgZWFjaCB0YXNr IG1pZ2h0IGhhdmUgZGlmZmVyZW50IDUsNCBvcgozLWxldmVsIGFkZHJlc3MgdHJhbnNsYXRpb24g YW5kIGhlbmNlIGRpZmZlcmVudCBsZXZlbHMgZm9sZGVkIHRoZSBsb2dpYwppcyBtb3JlIGNvbXBs ZXggYW5kIG5vbi1pdGVyYXRhYmxlIHBvaW50ZXIgdG8gYSBsb2NhbCBjb3B5IGxlYWRzIHRvCnNl dmVyZSBwcm9ibGVtcy4KCkhlcmUgaXMgYW4gZXhhbXBsZSBvZiB3aGF0IGhhcHBlbnMgd2l0aCBn dXBfZmFzdCBvbiBzMzkwLCBmb3IgYSB0YXNrCndpdGggMy1sZXZlbHMgcGFnaW5nLCBjcm9zc2lu ZyBhIDIgR0IgcHVkIGJvdW5kYXJ5OgoKLy8gYWRkciA9IDB4MTAwN2ZmZmYwMDAsIGVuZCA9IDB4 MTAwODAwMDEwMDAKc3RhdGljIGludCBndXBfcHVkX3JhbmdlKHA0ZF90IHA0ZCwgdW5zaWduZWQg bG9uZyBhZGRyLCB1bnNpZ25lZCBsb25nIGVuZCwKICAgICAgICAgICAgICAgICAgICAgICAgIHVu c2lnbmVkIGludCBmbGFncywgc3RydWN0IHBhZ2UgKipwYWdlcywgaW50ICpucikKewogICAgICAg IHVuc2lnbmVkIGxvbmcgbmV4dDsKICAgICAgICBwdWRfdCAqcHVkcDsKCiAgICAgICAgLy8gcHVk X29mZnNldCByZXR1cm5zICZwNGQgaXRzZWxmIChhIHBvaW50ZXIgdG8gYSB2YWx1ZSBvbiBzdGFj aykKICAgICAgICBwdWRwID0gcHVkX29mZnNldCgmcDRkLCBhZGRyKTsKICAgICAgICBkbyB7CiAg ICAgICAgICAgICAgICAvLyBvbiBzZWNvbmQgaXRlcmF0YXRpb24gcmVhZGluZyAicmFuZG9tIiBz dGFjayB2YWx1ZQogICAgICAgICAgICAgICAgcHVkX3QgcHVkID0gUkVBRF9PTkNFKCpwdWRwKTsK CiAgICAgICAgICAgICAgICAvLyBuZXh0ID0gMHgxMDA4MDAwMDAwMCwgZHVlIHRvIFBVRF9TSVpF L01BU0sgIT0gUEdESVJfU0laRS9NQVNLIG9uIHMzOTAKICAgICAgICAgICAgICAgIG5leHQgPSBw dWRfYWRkcl9lbmQoYWRkciwgZW5kKTsKICAgICAgICAgICAgICAgIC4uLgogICAgICAgIH0gd2hp bGUgKHB1ZHArKywgYWRkciA9IG5leHQsIGFkZHIgIT0gZW5kKTsgLy8gcHVkcCsrIGl0ZXJhdGlu ZyBvdmVyIHN0YWNrCgogICAgICAgIHJldHVybiAxOwp9CgpUaGlzIGhhcHBlbnMgc2luY2UgczM5 MCBtb3ZlZCB0byBjb21tb24gZ3VwIGNvZGUgd2l0aApjb21taXQgZDE4NzRhMGMyODA1ICgiczM5 MC9tbTogbWFrZSB0aGUgcHhkX29mZnNldCBmdW5jdGlvbnMgbW9yZSByb2J1c3QiKQphbmQgY29t bWl0IDFhNDIwMTBjZGMyNiAoInMzOTAvbW06IGNvbnZlcnQgdG8gdGhlIGdlbmVyaWMKZ2V0X3Vz ZXJfcGFnZXNfZmFzdCBjb2RlIikuIHMzOTAgdHJpZWQgdG8gbWltaWMgc3RhdGljIGxldmVsIGZv bGRpbmcgYnkKY2hhbmdpbmcgcFhkX29mZnNldCBwcmltaXRpdmVzIHRvIGFsd2F5cyBjYWxjdWxh dGUgdG9wIGxldmVsIHBhZ2UgdGFibGUKb2Zmc2V0IGluIHBnZF9vZmZzZXQgYW5kIGp1c3QgcmV0 dXJuIHRoZSB2YWx1ZSBwYXNzZWQgd2hlbiBwWGRfb2Zmc2V0CmhhcyB0byBhY3QgYXMgZm9sZGVk LgoKV2hhdCBpcyBjcnVjaWFsIGZvciBndXBfZmFzdCBhbmQgd2hhdCBoYXMgYmVlbiBvdmVybG9v a2VkIGlzCnRoYXQgUHhEX1NJWkUvTUFTSyBhbmQgdGh1cyBwWGRfYWRkcl9lbmQgc2hvdWxkIGFs c28gY2hhbmdlCmNvcnJlc3BvbmRpbmdseS4gQW5kIHRoZSBsYXR0ZXIgaXMgbm90IHBvc3NpYmxl IHdpdGggZHluYW1pYyBmb2xkaW5nLgoKVG8gZml4IHRoZSBpc3N1ZSBpbiBhZGRpdGlvbiB0byBw WGQgdmFsdWVzIHBhc3Mgb3JpZ2luYWwKcFhkcCBwb2ludGVycyBkb3duIHRvIGd1cF9wWGRfcmFu Z2UgZnVuY3Rpb25zLiBBbmQgaW50cm9kdWNlCnBYZF9vZmZzZXRfbG9ja2xlc3MgaGVscGVycywg d2hpY2ggdGFrZSBhbiBhZGRpdGlvbmFsIHBYZAplbnRyeSB2YWx1ZSBwYXJhbWV0ZXIuIFRoaXMg aGFzIGFscmVhZHkgYmVlbiBkaXNjdXNzZWQgaW4KaHR0cHM6Ly9sa21sLmtlcm5lbC5vcmcvci8y MDE5MDQxODEwMDIxOC4wYTRhZmQ1MUBtc2Nod2lkZVgxCgpDYzogPHN0YWJsZUB2Z2VyLmtlcm5l bC5vcmc+ICMgNS4yKwpGaXhlczogMWE0MjAxMGNkYzI2ICgiczM5MC9tbTogY29udmVydCB0byB0 aGUgZ2VuZXJpYyBnZXRfdXNlcl9wYWdlc19mYXN0IGNvZGUiKQpSZXZpZXdlZC1ieTogR2VyYWxk IFNjaGFlZmVyIDxnZXJhbGQuc2NoYWVmZXJAbGludXguaWJtLmNvbT4KUmV2aWV3ZWQtYnk6IEFs ZXhhbmRlciBHb3JkZWV2IDxhZ29yZGVldkBsaW51eC5pYm0uY29tPgpTaWduZWQtb2ZmLWJ5OiBW YXNpbHkgR29yYmlrIDxnb3JAbGludXguaWJtLmNvbT4KLS0tCnYyOiBhZGRlZCBicmFja2V0cyAm cGdkIC0+ICYocGdkKQoKIGFyY2gvczM5MC9pbmNsdWRlL2FzbS9wZ3RhYmxlLmggfCA0MiArKysr KysrKysrKysrKysrKysrKysrKy0tLS0tLS0tLS0KIGluY2x1ZGUvbGludXgvcGd0YWJsZS5oICAg ICAgICAgfCAxMCArKysrKysrKwogbW0vZ3VwLmMgICAgICAgICAgICAgICAgICAgICAgICB8IDE4 ICsrKysrKystLS0tLS0tCiAzIGZpbGVzIGNoYW5nZWQsIDQ5IGluc2VydGlvbnMoKyksIDIxIGRl bGV0aW9ucygtKQoKZGlmZiAtLWdpdCBhL2FyY2gvczM5MC9pbmNsdWRlL2FzbS9wZ3RhYmxlLmgg Yi9hcmNoL3MzOTAvaW5jbHVkZS9hc20vcGd0YWJsZS5oCmluZGV4IDdlYjAxYTU0NTljZC4uYjU1 NTYxY2M4Nzg2IDEwMDY0NAotLS0gYS9hcmNoL3MzOTAvaW5jbHVkZS9hc20vcGd0YWJsZS5oCisr KyBiL2FyY2gvczM5MC9pbmNsdWRlL2FzbS9wZ3RhYmxlLmgKQEAgLTEyNjAsMjYgKzEyNjAsNDQg QEAgc3RhdGljIGlubGluZSBwZ2RfdCAqcGdkX29mZnNldF9yYXcocGdkX3QgKnBnZCwgdW5zaWdu ZWQgbG9uZyBhZGRyZXNzKQogCiAjZGVmaW5lIHBnZF9vZmZzZXQobW0sIGFkZHJlc3MpIHBnZF9v ZmZzZXRfcmF3KFJFQURfT05DRSgobW0pLT5wZ2QpLCBhZGRyZXNzKQogCi1zdGF0aWMgaW5saW5l IHA0ZF90ICpwNGRfb2Zmc2V0KHBnZF90ICpwZ2QsIHVuc2lnbmVkIGxvbmcgYWRkcmVzcykKK3N0 YXRpYyBpbmxpbmUgcDRkX3QgKnA0ZF9vZmZzZXRfbG9ja2xlc3MocGdkX3QgKnBnZHAsIHBnZF90 IHBnZCwgdW5zaWduZWQgbG9uZyBhZGRyZXNzKQogewotCWlmICgocGdkX3ZhbCgqcGdkKSAmIF9S RUdJT05fRU5UUllfVFlQRV9NQVNLKSA+PSBfUkVHSU9OX0VOVFJZX1RZUEVfUjEpCi0JCXJldHVy biAocDRkX3QgKikgcGdkX2RlcmVmKCpwZ2QpICsgcDRkX2luZGV4KGFkZHJlc3MpOwotCXJldHVy biAocDRkX3QgKikgcGdkOworCWlmICgocGdkX3ZhbChwZ2QpICYgX1JFR0lPTl9FTlRSWV9UWVBF X01BU0spID49IF9SRUdJT05fRU5UUllfVFlQRV9SMSkKKwkJcmV0dXJuIChwNGRfdCAqKSBwZ2Rf ZGVyZWYocGdkKSArIHA0ZF9pbmRleChhZGRyZXNzKTsKKwlyZXR1cm4gKHA0ZF90ICopIHBnZHA7 CiB9CisjZGVmaW5lIHA0ZF9vZmZzZXRfbG9ja2xlc3MgcDRkX29mZnNldF9sb2NrbGVzcwogCi1z dGF0aWMgaW5saW5lIHB1ZF90ICpwdWRfb2Zmc2V0KHA0ZF90ICpwNGQsIHVuc2lnbmVkIGxvbmcg YWRkcmVzcykKK3N0YXRpYyBpbmxpbmUgcDRkX3QgKnA0ZF9vZmZzZXQocGdkX3QgKnBnZHAsIHVu c2lnbmVkIGxvbmcgYWRkcmVzcykKIHsKLQlpZiAoKHA0ZF92YWwoKnA0ZCkgJiBfUkVHSU9OX0VO VFJZX1RZUEVfTUFTSykgPj0gX1JFR0lPTl9FTlRSWV9UWVBFX1IyKQotCQlyZXR1cm4gKHB1ZF90 ICopIHA0ZF9kZXJlZigqcDRkKSArIHB1ZF9pbmRleChhZGRyZXNzKTsKLQlyZXR1cm4gKHB1ZF90 ICopIHA0ZDsKKwlyZXR1cm4gcDRkX29mZnNldF9sb2NrbGVzcyhwZ2RwLCAqcGdkcCwgYWRkcmVz cyk7Cit9CisKK3N0YXRpYyBpbmxpbmUgcHVkX3QgKnB1ZF9vZmZzZXRfbG9ja2xlc3MocDRkX3Qg KnA0ZHAsIHA0ZF90IHA0ZCwgdW5zaWduZWQgbG9uZyBhZGRyZXNzKQoreworCWlmICgocDRkX3Zh bChwNGQpICYgX1JFR0lPTl9FTlRSWV9UWVBFX01BU0spID49IF9SRUdJT05fRU5UUllfVFlQRV9S MikKKwkJcmV0dXJuIChwdWRfdCAqKSBwNGRfZGVyZWYocDRkKSArIHB1ZF9pbmRleChhZGRyZXNz KTsKKwlyZXR1cm4gKHB1ZF90ICopIHA0ZHA7Cit9CisjZGVmaW5lIHB1ZF9vZmZzZXRfbG9ja2xl c3MgcHVkX29mZnNldF9sb2NrbGVzcworCitzdGF0aWMgaW5saW5lIHB1ZF90ICpwdWRfb2Zmc2V0 KHA0ZF90ICpwNGRwLCB1bnNpZ25lZCBsb25nIGFkZHJlc3MpCit7CisJcmV0dXJuIHB1ZF9vZmZz ZXRfbG9ja2xlc3MocDRkcCwgKnA0ZHAsIGFkZHJlc3MpOwogfQogI2RlZmluZSBwdWRfb2Zmc2V0 IHB1ZF9vZmZzZXQKIAotc3RhdGljIGlubGluZSBwbWRfdCAqcG1kX29mZnNldChwdWRfdCAqcHVk LCB1bnNpZ25lZCBsb25nIGFkZHJlc3MpCitzdGF0aWMgaW5saW5lIHBtZF90ICpwbWRfb2Zmc2V0 X2xvY2tsZXNzKHB1ZF90ICpwdWRwLCBwdWRfdCBwdWQsIHVuc2lnbmVkIGxvbmcgYWRkcmVzcykK K3sKKwlpZiAoKHB1ZF92YWwocHVkKSAmIF9SRUdJT05fRU5UUllfVFlQRV9NQVNLKSA+PSBfUkVH SU9OX0VOVFJZX1RZUEVfUjMpCisJCXJldHVybiAocG1kX3QgKikgcHVkX2RlcmVmKHB1ZCkgKyBw bWRfaW5kZXgoYWRkcmVzcyk7CisJcmV0dXJuIChwbWRfdCAqKSBwdWRwOworfQorI2RlZmluZSBw bWRfb2Zmc2V0X2xvY2tsZXNzIHBtZF9vZmZzZXRfbG9ja2xlc3MKKworc3RhdGljIGlubGluZSBw bWRfdCAqcG1kX29mZnNldChwdWRfdCAqcHVkcCwgdW5zaWduZWQgbG9uZyBhZGRyZXNzKQogewot CWlmICgocHVkX3ZhbCgqcHVkKSAmIF9SRUdJT05fRU5UUllfVFlQRV9NQVNLKSA+PSBfUkVHSU9O X0VOVFJZX1RZUEVfUjMpCi0JCXJldHVybiAocG1kX3QgKikgcHVkX2RlcmVmKCpwdWQpICsgcG1k X2luZGV4KGFkZHJlc3MpOwotCXJldHVybiAocG1kX3QgKikgcHVkOworCXJldHVybiBwbWRfb2Zm c2V0X2xvY2tsZXNzKHB1ZHAsICpwdWRwLCBhZGRyZXNzKTsKIH0KICNkZWZpbmUgcG1kX29mZnNl dCBwbWRfb2Zmc2V0CiAKZGlmZiAtLWdpdCBhL2luY2x1ZGUvbGludXgvcGd0YWJsZS5oIGIvaW5j bHVkZS9saW51eC9wZ3RhYmxlLmgKaW5kZXggZThjYmMyZTc5NWQ1Li45MDY1NGNiNjNlOWUgMTAw NjQ0Ci0tLSBhL2luY2x1ZGUvbGludXgvcGd0YWJsZS5oCisrKyBiL2luY2x1ZGUvbGludXgvcGd0 YWJsZS5oCkBAIC0xNDI3LDYgKzE0MjcsMTYgQEAgdHlwZWRlZiB1bnNpZ25lZCBpbnQgcGd0Ymxf bW9kX21hc2s7CiAjZGVmaW5lIG1tX3BtZF9mb2xkZWQobW0pCV9faXNfZGVmaW5lZChfX1BBR0VU QUJMRV9QTURfRk9MREVEKQogI2VuZGlmCiAKKyNpZm5kZWYgcDRkX29mZnNldF9sb2NrbGVzcwor I2RlZmluZSBwNGRfb2Zmc2V0X2xvY2tsZXNzKHBnZHAsIHBnZCwgYWRkcmVzcykgcDRkX29mZnNl dCgmKHBnZCksIGFkZHJlc3MpCisjZW5kaWYKKyNpZm5kZWYgcHVkX29mZnNldF9sb2NrbGVzcwor I2RlZmluZSBwdWRfb2Zmc2V0X2xvY2tsZXNzKHA0ZHAsIHA0ZCwgYWRkcmVzcykgcHVkX29mZnNl dCgmKHA0ZCksIGFkZHJlc3MpCisjZW5kaWYKKyNpZm5kZWYgcG1kX29mZnNldF9sb2NrbGVzcwor I2RlZmluZSBwbWRfb2Zmc2V0X2xvY2tsZXNzKHB1ZHAsIHB1ZCwgYWRkcmVzcykgcG1kX29mZnNl dCgmKHB1ZCksIGFkZHJlc3MpCisjZW5kaWYKKwogLyoKICAqIHA/ZF9sZWFmKCkgLSB0cnVlIGlm IHRoaXMgZW50cnkgaXMgYSBmaW5hbCBtYXBwaW5nIHRvIGEgcGh5c2ljYWwgYWRkcmVzcy4KICAq IFRoaXMgZGlmZmVycyBmcm9tIHA/ZF9odWdlKCkgYnkgdGhlIGZhY3QgdGhhdCB0aGV5IGFyZSBh bHdheXMgYXZhaWxhYmxlIChpZgpkaWZmIC0tZ2l0IGEvbW0vZ3VwLmMgYi9tbS9ndXAuYwppbmRl eCBlNTczOWExOTc0ZDUuLjU3OGJmNWJkOGJmOCAxMDA2NDQKLS0tIGEvbW0vZ3VwLmMKKysrIGIv bW0vZ3VwLmMKQEAgLTI0ODUsMTMgKzI0ODUsMTMgQEAgc3RhdGljIGludCBndXBfaHVnZV9wZ2Qo cGdkX3Qgb3JpZywgcGdkX3QgKnBnZHAsIHVuc2lnbmVkIGxvbmcgYWRkciwKIAlyZXR1cm4gMTsK IH0KIAotc3RhdGljIGludCBndXBfcG1kX3JhbmdlKHB1ZF90IHB1ZCwgdW5zaWduZWQgbG9uZyBh ZGRyLCB1bnNpZ25lZCBsb25nIGVuZCwKK3N0YXRpYyBpbnQgZ3VwX3BtZF9yYW5nZShwdWRfdCAq cHVkcCwgcHVkX3QgcHVkLCB1bnNpZ25lZCBsb25nIGFkZHIsIHVuc2lnbmVkIGxvbmcgZW5kLAog CQl1bnNpZ25lZCBpbnQgZmxhZ3MsIHN0cnVjdCBwYWdlICoqcGFnZXMsIGludCAqbnIpCiB7CiAJ dW5zaWduZWQgbG9uZyBuZXh0OwogCXBtZF90ICpwbWRwOwogCi0JcG1kcCA9IHBtZF9vZmZzZXQo JnB1ZCwgYWRkcik7CisJcG1kcCA9IHBtZF9vZmZzZXRfbG9ja2xlc3MocHVkcCwgcHVkLCBhZGRy KTsKIAlkbyB7CiAJCXBtZF90IHBtZCA9IFJFQURfT05DRSgqcG1kcCk7CiAKQEAgLTI1MjgsMTMg KzI1MjgsMTMgQEAgc3RhdGljIGludCBndXBfcG1kX3JhbmdlKHB1ZF90IHB1ZCwgdW5zaWduZWQg bG9uZyBhZGRyLCB1bnNpZ25lZCBsb25nIGVuZCwKIAlyZXR1cm4gMTsKIH0KIAotc3RhdGljIGlu dCBndXBfcHVkX3JhbmdlKHA0ZF90IHA0ZCwgdW5zaWduZWQgbG9uZyBhZGRyLCB1bnNpZ25lZCBs b25nIGVuZCwKK3N0YXRpYyBpbnQgZ3VwX3B1ZF9yYW5nZShwNGRfdCAqcDRkcCwgcDRkX3QgcDRk LCB1bnNpZ25lZCBsb25nIGFkZHIsIHVuc2lnbmVkIGxvbmcgZW5kLAogCQkJIHVuc2lnbmVkIGlu dCBmbGFncywgc3RydWN0IHBhZ2UgKipwYWdlcywgaW50ICpucikKIHsKIAl1bnNpZ25lZCBsb25n IG5leHQ7CiAJcHVkX3QgKnB1ZHA7CiAKLQlwdWRwID0gcHVkX29mZnNldCgmcDRkLCBhZGRyKTsK KwlwdWRwID0gcHVkX29mZnNldF9sb2NrbGVzcyhwNGRwLCBwNGQsIGFkZHIpOwogCWRvIHsKIAkJ cHVkX3QgcHVkID0gUkVBRF9PTkNFKCpwdWRwKTsKIApAQCAtMjU0OSwyMCArMjU0OSwyMCBAQCBz dGF0aWMgaW50IGd1cF9wdWRfcmFuZ2UocDRkX3QgcDRkLCB1bnNpZ25lZCBsb25nIGFkZHIsIHVu c2lnbmVkIGxvbmcgZW5kLAogCQkJaWYgKCFndXBfaHVnZV9wZChfX2h1Z2VwZChwdWRfdmFsKHB1 ZCkpLCBhZGRyLAogCQkJCQkgUFVEX1NISUZULCBuZXh0LCBmbGFncywgcGFnZXMsIG5yKSkKIAkJ CQlyZXR1cm4gMDsKLQkJfSBlbHNlIGlmICghZ3VwX3BtZF9yYW5nZShwdWQsIGFkZHIsIG5leHQs IGZsYWdzLCBwYWdlcywgbnIpKQorCQl9IGVsc2UgaWYgKCFndXBfcG1kX3JhbmdlKHB1ZHAsIHB1 ZCwgYWRkciwgbmV4dCwgZmxhZ3MsIHBhZ2VzLCBucikpCiAJCQlyZXR1cm4gMDsKIAl9IHdoaWxl IChwdWRwKyssIGFkZHIgPSBuZXh0LCBhZGRyICE9IGVuZCk7CiAKIAlyZXR1cm4gMTsKIH0KIAot c3RhdGljIGludCBndXBfcDRkX3JhbmdlKHBnZF90IHBnZCwgdW5zaWduZWQgbG9uZyBhZGRyLCB1 bnNpZ25lZCBsb25nIGVuZCwKK3N0YXRpYyBpbnQgZ3VwX3A0ZF9yYW5nZShwZ2RfdCAqcGdkcCwg cGdkX3QgcGdkLCB1bnNpZ25lZCBsb25nIGFkZHIsIHVuc2lnbmVkIGxvbmcgZW5kLAogCQkJIHVu c2lnbmVkIGludCBmbGFncywgc3RydWN0IHBhZ2UgKipwYWdlcywgaW50ICpucikKIHsKIAl1bnNp Z25lZCBsb25nIG5leHQ7CiAJcDRkX3QgKnA0ZHA7CiAKLQlwNGRwID0gcDRkX29mZnNldCgmcGdk LCBhZGRyKTsKKwlwNGRwID0gcDRkX29mZnNldF9sb2NrbGVzcyhwZ2RwLCBwZ2QsIGFkZHIpOwog CWRvIHsKIAkJcDRkX3QgcDRkID0gUkVBRF9PTkNFKCpwNGRwKTsKIApAQCAtMjU3NCw3ICsyNTc0 LDcgQEAgc3RhdGljIGludCBndXBfcDRkX3JhbmdlKHBnZF90IHBnZCwgdW5zaWduZWQgbG9uZyBh ZGRyLCB1bnNpZ25lZCBsb25nIGVuZCwKIAkJCWlmICghZ3VwX2h1Z2VfcGQoX19odWdlcGQocDRk X3ZhbChwNGQpKSwgYWRkciwKIAkJCQkJIFA0RF9TSElGVCwgbmV4dCwgZmxhZ3MsIHBhZ2VzLCBu cikpCiAJCQkJcmV0dXJuIDA7Ci0JCX0gZWxzZSBpZiAoIWd1cF9wdWRfcmFuZ2UocDRkLCBhZGRy LCBuZXh0LCBmbGFncywgcGFnZXMsIG5yKSkKKwkJfSBlbHNlIGlmICghZ3VwX3B1ZF9yYW5nZShw NGRwLCBwNGQsIGFkZHIsIG5leHQsIGZsYWdzLCBwYWdlcywgbnIpKQogCQkJcmV0dXJuIDA7CiAJ fSB3aGlsZSAocDRkcCsrLCBhZGRyID0gbmV4dCwgYWRkciAhPSBlbmQpOwogCkBAIC0yNjAyLDcg KzI2MDIsNyBAQCBzdGF0aWMgdm9pZCBndXBfcGdkX3JhbmdlKHVuc2lnbmVkIGxvbmcgYWRkciwg dW5zaWduZWQgbG9uZyBlbmQsCiAJCQlpZiAoIWd1cF9odWdlX3BkKF9faHVnZXBkKHBnZF92YWwo cGdkKSksIGFkZHIsCiAJCQkJCSBQR0RJUl9TSElGVCwgbmV4dCwgZmxhZ3MsIHBhZ2VzLCBucikp CiAJCQkJcmV0dXJuOwotCQl9IGVsc2UgaWYgKCFndXBfcDRkX3JhbmdlKHBnZCwgYWRkciwgbmV4 dCwgZmxhZ3MsIHBhZ2VzLCBucikpCisJCX0gZWxzZSBpZiAoIWd1cF9wNGRfcmFuZ2UocGdkcCwg cGdkLCBhZGRyLCBuZXh0LCBmbGFncywgcGFnZXMsIG5yKSkKIAkJCXJldHVybjsKIAl9IHdoaWxl IChwZ2RwKyssIGFkZHIgPSBuZXh0LCBhZGRyICE9IGVuZCk7CiB9Ci0tIArio7/io7/io7/io7/i oovioYDio4DioLnio7/io7/io7/io78K4qO/4qO/4qO/4qO/4qCg4qO24qGm4qCA4qO/4qO/4qO/ 4qO/CuKjv+Kjv+Kjv+Kgj+KjtOKjruKjtOKjp+KgiOKiv+Kjv+Kjvwrio7/io7/ioY/iorDio7/i oJbio6Dio7/ioYbioIjio7/io78K4qO/4qKb4qO14qOE4qCZ4qO24qO24qGf4qOF4qOg4qC54qO/ CuKjv+KjnOKjm+Kgu+KijuKjieKjieKjgOKgv+Kjq+KjteKjvwoKX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX18KbGludXgtYXJtLWtlcm5lbCBtYWlsaW5nIGxp c3QKbGludXgtYXJtLWtlcm5lbEBsaXN0cy5pbmZyYWRlYWQub3JnCmh0dHA6Ly9saXN0cy5pbmZy YWRlYWQub3JnL21haWxtYW4vbGlzdGluZm8vbGludXgtYXJtLWtlcm5lbAo= From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Date: Fri, 11 Sep 2020 22:36:43 +0200 From: Vasily Gorbik Subject: [PATCH v2] mm/gup: fix gup_fast with dynamic page table folding Message-ID: References: <20200911200511.GC1221970@ziepe.ca> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20200911200511.GC1221970@ziepe.ca> List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-um" Errors-To: linux-um-bounces+geert=linux-m68k.org@lists.infradead.org To: Jason Gunthorpe , John Hubbard Cc: Peter Zijlstra , Benjamin Herrenschmidt , Dave Hansen , linux-mm , Paul Mackerras , linux-sparc , Alexander Gordeev , Claudio Imbrenda , Will Deacon , linux-arch , linux-s390 , Richard Weinberger , linux-x86 , Russell King , Christian Borntraeger , Ingo Molnar , Catalin Marinas , Andrey Ryabinin , Gerald Schaefer , Heiko Carstens , Arnd Bergmann , Jeff Dike , linux-um , Borislav Petkov , Andy Lutomirski , Thomas Gleixner , linux-arm , Dave Hansen , linux-power , LKML , Michael Ellerman , Andrew Morton , Linus Torvalds , Mike Rapoport Q3VycmVudGx5IHRvIG1ha2Ugc3VyZSB0aGF0IGV2ZXJ5IHBhZ2UgdGFibGUgZW50cnkgaXMgcmVh ZCBqdXN0IG9uY2UKZ3VwX2Zhc3Qgd2Fsa3MgcGVyZm9ybSBSRUFEX09OQ0UgYW5kIHBhc3MgcFhk IHZhbHVlIGRvd24gdG8gdGhlIG5leHQKZ3VwX3BYZF9yYW5nZSBmdW5jdGlvbiBieSB2YWx1ZSBl LmcuOgoKc3RhdGljIGludCBndXBfcHVkX3JhbmdlKHA0ZF90IHA0ZCwgdW5zaWduZWQgbG9uZyBh ZGRyLCB1bnNpZ25lZCBsb25nIGVuZCwKICAgICAgICAgICAgICAgICAgICAgICAgIHVuc2lnbmVk IGludCBmbGFncywgc3RydWN0IHBhZ2UgKipwYWdlcywgaW50ICpucikKLi4uCiAgICAgICAgcHVk cCA9IHB1ZF9vZmZzZXQoJnA0ZCwgYWRkcik7CgpUaGlzIGZ1bmN0aW9uIHBhc3NlcyBhIHJlZmVy ZW5jZSBvbiB0aGF0IGxvY2FsIHZhbHVlIGNvcHkgdG8gcFhkX29mZnNldCwKYW5kIG1pZ2h0IGdl dCB0aGUgdmVyeSBzYW1lIHBvaW50ZXIgaW4gcmV0dXJuLiBUaGlzIGhhcHBlbnMgd2hlbiB0aGUK bGV2ZWwgaXMgZm9sZGVkIChvbiBtb3N0IGFyY2hlcyksIGFuZCB0aGF0IHBvaW50ZXIgc2hvdWxk IG5vdCBiZSBpdGVyYXRlZC4KCk9uIHMzOTAgZHVlIHRvIHRoZSBmYWN0IHRoYXQgZWFjaCB0YXNr IG1pZ2h0IGhhdmUgZGlmZmVyZW50IDUsNCBvcgozLWxldmVsIGFkZHJlc3MgdHJhbnNsYXRpb24g YW5kIGhlbmNlIGRpZmZlcmVudCBsZXZlbHMgZm9sZGVkIHRoZSBsb2dpYwppcyBtb3JlIGNvbXBs ZXggYW5kIG5vbi1pdGVyYXRhYmxlIHBvaW50ZXIgdG8gYSBsb2NhbCBjb3B5IGxlYWRzIHRvCnNl dmVyZSBwcm9ibGVtcy4KCkhlcmUgaXMgYW4gZXhhbXBsZSBvZiB3aGF0IGhhcHBlbnMgd2l0aCBn dXBfZmFzdCBvbiBzMzkwLCBmb3IgYSB0YXNrCndpdGggMy1sZXZlbHMgcGFnaW5nLCBjcm9zc2lu ZyBhIDIgR0IgcHVkIGJvdW5kYXJ5OgoKLy8gYWRkciA9IDB4MTAwN2ZmZmYwMDAsIGVuZCA9IDB4 MTAwODAwMDEwMDAKc3RhdGljIGludCBndXBfcHVkX3JhbmdlKHA0ZF90IHA0ZCwgdW5zaWduZWQg bG9uZyBhZGRyLCB1bnNpZ25lZCBsb25nIGVuZCwKICAgICAgICAgICAgICAgICAgICAgICAgIHVu c2lnbmVkIGludCBmbGFncywgc3RydWN0IHBhZ2UgKipwYWdlcywgaW50ICpucikKewogICAgICAg IHVuc2lnbmVkIGxvbmcgbmV4dDsKICAgICAgICBwdWRfdCAqcHVkcDsKCiAgICAgICAgLy8gcHVk X29mZnNldCByZXR1cm5zICZwNGQgaXRzZWxmIChhIHBvaW50ZXIgdG8gYSB2YWx1ZSBvbiBzdGFj aykKICAgICAgICBwdWRwID0gcHVkX29mZnNldCgmcDRkLCBhZGRyKTsKICAgICAgICBkbyB7CiAg ICAgICAgICAgICAgICAvLyBvbiBzZWNvbmQgaXRlcmF0YXRpb24gcmVhZGluZyAicmFuZG9tIiBz dGFjayB2YWx1ZQogICAgICAgICAgICAgICAgcHVkX3QgcHVkID0gUkVBRF9PTkNFKCpwdWRwKTsK CiAgICAgICAgICAgICAgICAvLyBuZXh0ID0gMHgxMDA4MDAwMDAwMCwgZHVlIHRvIFBVRF9TSVpF L01BU0sgIT0gUEdESVJfU0laRS9NQVNLIG9uIHMzOTAKICAgICAgICAgICAgICAgIG5leHQgPSBw dWRfYWRkcl9lbmQoYWRkciwgZW5kKTsKICAgICAgICAgICAgICAgIC4uLgogICAgICAgIH0gd2hp bGUgKHB1ZHArKywgYWRkciA9IG5leHQsIGFkZHIgIT0gZW5kKTsgLy8gcHVkcCsrIGl0ZXJhdGlu ZyBvdmVyIHN0YWNrCgogICAgICAgIHJldHVybiAxOwp9CgpUaGlzIGhhcHBlbnMgc2luY2UgczM5 MCBtb3ZlZCB0byBjb21tb24gZ3VwIGNvZGUgd2l0aApjb21taXQgZDE4NzRhMGMyODA1ICgiczM5 MC9tbTogbWFrZSB0aGUgcHhkX29mZnNldCBmdW5jdGlvbnMgbW9yZSByb2J1c3QiKQphbmQgY29t bWl0IDFhNDIwMTBjZGMyNiAoInMzOTAvbW06IGNvbnZlcnQgdG8gdGhlIGdlbmVyaWMKZ2V0X3Vz ZXJfcGFnZXNfZmFzdCBjb2RlIikuIHMzOTAgdHJpZWQgdG8gbWltaWMgc3RhdGljIGxldmVsIGZv bGRpbmcgYnkKY2hhbmdpbmcgcFhkX29mZnNldCBwcmltaXRpdmVzIHRvIGFsd2F5cyBjYWxjdWxh dGUgdG9wIGxldmVsIHBhZ2UgdGFibGUKb2Zmc2V0IGluIHBnZF9vZmZzZXQgYW5kIGp1c3QgcmV0 dXJuIHRoZSB2YWx1ZSBwYXNzZWQgd2hlbiBwWGRfb2Zmc2V0CmhhcyB0byBhY3QgYXMgZm9sZGVk LgoKV2hhdCBpcyBjcnVjaWFsIGZvciBndXBfZmFzdCBhbmQgd2hhdCBoYXMgYmVlbiBvdmVybG9v a2VkIGlzCnRoYXQgUHhEX1NJWkUvTUFTSyBhbmQgdGh1cyBwWGRfYWRkcl9lbmQgc2hvdWxkIGFs c28gY2hhbmdlCmNvcnJlc3BvbmRpbmdseS4gQW5kIHRoZSBsYXR0ZXIgaXMgbm90IHBvc3NpYmxl IHdpdGggZHluYW1pYyBmb2xkaW5nLgoKVG8gZml4IHRoZSBpc3N1ZSBpbiBhZGRpdGlvbiB0byBw WGQgdmFsdWVzIHBhc3Mgb3JpZ2luYWwKcFhkcCBwb2ludGVycyBkb3duIHRvIGd1cF9wWGRfcmFu Z2UgZnVuY3Rpb25zLiBBbmQgaW50cm9kdWNlCnBYZF9vZmZzZXRfbG9ja2xlc3MgaGVscGVycywg d2hpY2ggdGFrZSBhbiBhZGRpdGlvbmFsIHBYZAplbnRyeSB2YWx1ZSBwYXJhbWV0ZXIuIFRoaXMg aGFzIGFscmVhZHkgYmVlbiBkaXNjdXNzZWQgaW4KaHR0cHM6Ly9sa21sLmtlcm5lbC5vcmcvci8y MDE5MDQxODEwMDIxOC4wYTRhZmQ1MUBtc2Nod2lkZVgxCgpDYzogPHN0YWJsZUB2Z2VyLmtlcm5l bC5vcmc+ICMgNS4yKwpGaXhlczogMWE0MjAxMGNkYzI2ICgiczM5MC9tbTogY29udmVydCB0byB0 aGUgZ2VuZXJpYyBnZXRfdXNlcl9wYWdlc19mYXN0IGNvZGUiKQpSZXZpZXdlZC1ieTogR2VyYWxk IFNjaGFlZmVyIDxnZXJhbGQuc2NoYWVmZXJAbGludXguaWJtLmNvbT4KUmV2aWV3ZWQtYnk6IEFs ZXhhbmRlciBHb3JkZWV2IDxhZ29yZGVldkBsaW51eC5pYm0uY29tPgpTaWduZWQtb2ZmLWJ5OiBW YXNpbHkgR29yYmlrIDxnb3JAbGludXguaWJtLmNvbT4KLS0tCnYyOiBhZGRlZCBicmFja2V0cyAm cGdkIC0+ICYocGdkKQoKIGFyY2gvczM5MC9pbmNsdWRlL2FzbS9wZ3RhYmxlLmggfCA0MiArKysr KysrKysrKysrKysrKysrKysrKy0tLS0tLS0tLS0KIGluY2x1ZGUvbGludXgvcGd0YWJsZS5oICAg ICAgICAgfCAxMCArKysrKysrKwogbW0vZ3VwLmMgICAgICAgICAgICAgICAgICAgICAgICB8IDE4 ICsrKysrKystLS0tLS0tCiAzIGZpbGVzIGNoYW5nZWQsIDQ5IGluc2VydGlvbnMoKyksIDIxIGRl bGV0aW9ucygtKQoKZGlmZiAtLWdpdCBhL2FyY2gvczM5MC9pbmNsdWRlL2FzbS9wZ3RhYmxlLmgg Yi9hcmNoL3MzOTAvaW5jbHVkZS9hc20vcGd0YWJsZS5oCmluZGV4IDdlYjAxYTU0NTljZC4uYjU1 NTYxY2M4Nzg2IDEwMDY0NAotLS0gYS9hcmNoL3MzOTAvaW5jbHVkZS9hc20vcGd0YWJsZS5oCisr KyBiL2FyY2gvczM5MC9pbmNsdWRlL2FzbS9wZ3RhYmxlLmgKQEAgLTEyNjAsMjYgKzEyNjAsNDQg QEAgc3RhdGljIGlubGluZSBwZ2RfdCAqcGdkX29mZnNldF9yYXcocGdkX3QgKnBnZCwgdW5zaWdu ZWQgbG9uZyBhZGRyZXNzKQogCiAjZGVmaW5lIHBnZF9vZmZzZXQobW0sIGFkZHJlc3MpIHBnZF9v ZmZzZXRfcmF3KFJFQURfT05DRSgobW0pLT5wZ2QpLCBhZGRyZXNzKQogCi1zdGF0aWMgaW5saW5l IHA0ZF90ICpwNGRfb2Zmc2V0KHBnZF90ICpwZ2QsIHVuc2lnbmVkIGxvbmcgYWRkcmVzcykKK3N0 YXRpYyBpbmxpbmUgcDRkX3QgKnA0ZF9vZmZzZXRfbG9ja2xlc3MocGdkX3QgKnBnZHAsIHBnZF90 IHBnZCwgdW5zaWduZWQgbG9uZyBhZGRyZXNzKQogewotCWlmICgocGdkX3ZhbCgqcGdkKSAmIF9S RUdJT05fRU5UUllfVFlQRV9NQVNLKSA+PSBfUkVHSU9OX0VOVFJZX1RZUEVfUjEpCi0JCXJldHVy biAocDRkX3QgKikgcGdkX2RlcmVmKCpwZ2QpICsgcDRkX2luZGV4KGFkZHJlc3MpOwotCXJldHVy biAocDRkX3QgKikgcGdkOworCWlmICgocGdkX3ZhbChwZ2QpICYgX1JFR0lPTl9FTlRSWV9UWVBF X01BU0spID49IF9SRUdJT05fRU5UUllfVFlQRV9SMSkKKwkJcmV0dXJuIChwNGRfdCAqKSBwZ2Rf ZGVyZWYocGdkKSArIHA0ZF9pbmRleChhZGRyZXNzKTsKKwlyZXR1cm4gKHA0ZF90ICopIHBnZHA7 CiB9CisjZGVmaW5lIHA0ZF9vZmZzZXRfbG9ja2xlc3MgcDRkX29mZnNldF9sb2NrbGVzcwogCi1z dGF0aWMgaW5saW5lIHB1ZF90ICpwdWRfb2Zmc2V0KHA0ZF90ICpwNGQsIHVuc2lnbmVkIGxvbmcg YWRkcmVzcykKK3N0YXRpYyBpbmxpbmUgcDRkX3QgKnA0ZF9vZmZzZXQocGdkX3QgKnBnZHAsIHVu c2lnbmVkIGxvbmcgYWRkcmVzcykKIHsKLQlpZiAoKHA0ZF92YWwoKnA0ZCkgJiBfUkVHSU9OX0VO VFJZX1RZUEVfTUFTSykgPj0gX1JFR0lPTl9FTlRSWV9UWVBFX1IyKQotCQlyZXR1cm4gKHB1ZF90 ICopIHA0ZF9kZXJlZigqcDRkKSArIHB1ZF9pbmRleChhZGRyZXNzKTsKLQlyZXR1cm4gKHB1ZF90 ICopIHA0ZDsKKwlyZXR1cm4gcDRkX29mZnNldF9sb2NrbGVzcyhwZ2RwLCAqcGdkcCwgYWRkcmVz cyk7Cit9CisKK3N0YXRpYyBpbmxpbmUgcHVkX3QgKnB1ZF9vZmZzZXRfbG9ja2xlc3MocDRkX3Qg KnA0ZHAsIHA0ZF90IHA0ZCwgdW5zaWduZWQgbG9uZyBhZGRyZXNzKQoreworCWlmICgocDRkX3Zh bChwNGQpICYgX1JFR0lPTl9FTlRSWV9UWVBFX01BU0spID49IF9SRUdJT05fRU5UUllfVFlQRV9S MikKKwkJcmV0dXJuIChwdWRfdCAqKSBwNGRfZGVyZWYocDRkKSArIHB1ZF9pbmRleChhZGRyZXNz KTsKKwlyZXR1cm4gKHB1ZF90ICopIHA0ZHA7Cit9CisjZGVmaW5lIHB1ZF9vZmZzZXRfbG9ja2xl c3MgcHVkX29mZnNldF9sb2NrbGVzcworCitzdGF0aWMgaW5saW5lIHB1ZF90ICpwdWRfb2Zmc2V0 KHA0ZF90ICpwNGRwLCB1bnNpZ25lZCBsb25nIGFkZHJlc3MpCit7CisJcmV0dXJuIHB1ZF9vZmZz ZXRfbG9ja2xlc3MocDRkcCwgKnA0ZHAsIGFkZHJlc3MpOwogfQogI2RlZmluZSBwdWRfb2Zmc2V0 IHB1ZF9vZmZzZXQKIAotc3RhdGljIGlubGluZSBwbWRfdCAqcG1kX29mZnNldChwdWRfdCAqcHVk LCB1bnNpZ25lZCBsb25nIGFkZHJlc3MpCitzdGF0aWMgaW5saW5lIHBtZF90ICpwbWRfb2Zmc2V0 X2xvY2tsZXNzKHB1ZF90ICpwdWRwLCBwdWRfdCBwdWQsIHVuc2lnbmVkIGxvbmcgYWRkcmVzcykK K3sKKwlpZiAoKHB1ZF92YWwocHVkKSAmIF9SRUdJT05fRU5UUllfVFlQRV9NQVNLKSA+PSBfUkVH SU9OX0VOVFJZX1RZUEVfUjMpCisJCXJldHVybiAocG1kX3QgKikgcHVkX2RlcmVmKHB1ZCkgKyBw bWRfaW5kZXgoYWRkcmVzcyk7CisJcmV0dXJuIChwbWRfdCAqKSBwdWRwOworfQorI2RlZmluZSBw bWRfb2Zmc2V0X2xvY2tsZXNzIHBtZF9vZmZzZXRfbG9ja2xlc3MKKworc3RhdGljIGlubGluZSBw bWRfdCAqcG1kX29mZnNldChwdWRfdCAqcHVkcCwgdW5zaWduZWQgbG9uZyBhZGRyZXNzKQogewot CWlmICgocHVkX3ZhbCgqcHVkKSAmIF9SRUdJT05fRU5UUllfVFlQRV9NQVNLKSA+PSBfUkVHSU9O X0VOVFJZX1RZUEVfUjMpCi0JCXJldHVybiAocG1kX3QgKikgcHVkX2RlcmVmKCpwdWQpICsgcG1k X2luZGV4KGFkZHJlc3MpOwotCXJldHVybiAocG1kX3QgKikgcHVkOworCXJldHVybiBwbWRfb2Zm c2V0X2xvY2tsZXNzKHB1ZHAsICpwdWRwLCBhZGRyZXNzKTsKIH0KICNkZWZpbmUgcG1kX29mZnNl dCBwbWRfb2Zmc2V0CiAKZGlmZiAtLWdpdCBhL2luY2x1ZGUvbGludXgvcGd0YWJsZS5oIGIvaW5j bHVkZS9saW51eC9wZ3RhYmxlLmgKaW5kZXggZThjYmMyZTc5NWQ1Li45MDY1NGNiNjNlOWUgMTAw NjQ0Ci0tLSBhL2luY2x1ZGUvbGludXgvcGd0YWJsZS5oCisrKyBiL2luY2x1ZGUvbGludXgvcGd0 YWJsZS5oCkBAIC0xNDI3LDYgKzE0MjcsMTYgQEAgdHlwZWRlZiB1bnNpZ25lZCBpbnQgcGd0Ymxf bW9kX21hc2s7CiAjZGVmaW5lIG1tX3BtZF9mb2xkZWQobW0pCV9faXNfZGVmaW5lZChfX1BBR0VU QUJMRV9QTURfRk9MREVEKQogI2VuZGlmCiAKKyNpZm5kZWYgcDRkX29mZnNldF9sb2NrbGVzcwor I2RlZmluZSBwNGRfb2Zmc2V0X2xvY2tsZXNzKHBnZHAsIHBnZCwgYWRkcmVzcykgcDRkX29mZnNl dCgmKHBnZCksIGFkZHJlc3MpCisjZW5kaWYKKyNpZm5kZWYgcHVkX29mZnNldF9sb2NrbGVzcwor I2RlZmluZSBwdWRfb2Zmc2V0X2xvY2tsZXNzKHA0ZHAsIHA0ZCwgYWRkcmVzcykgcHVkX29mZnNl dCgmKHA0ZCksIGFkZHJlc3MpCisjZW5kaWYKKyNpZm5kZWYgcG1kX29mZnNldF9sb2NrbGVzcwor I2RlZmluZSBwbWRfb2Zmc2V0X2xvY2tsZXNzKHB1ZHAsIHB1ZCwgYWRkcmVzcykgcG1kX29mZnNl dCgmKHB1ZCksIGFkZHJlc3MpCisjZW5kaWYKKwogLyoKICAqIHA/ZF9sZWFmKCkgLSB0cnVlIGlm IHRoaXMgZW50cnkgaXMgYSBmaW5hbCBtYXBwaW5nIHRvIGEgcGh5c2ljYWwgYWRkcmVzcy4KICAq IFRoaXMgZGlmZmVycyBmcm9tIHA/ZF9odWdlKCkgYnkgdGhlIGZhY3QgdGhhdCB0aGV5IGFyZSBh bHdheXMgYXZhaWxhYmxlIChpZgpkaWZmIC0tZ2l0IGEvbW0vZ3VwLmMgYi9tbS9ndXAuYwppbmRl eCBlNTczOWExOTc0ZDUuLjU3OGJmNWJkOGJmOCAxMDA2NDQKLS0tIGEvbW0vZ3VwLmMKKysrIGIv bW0vZ3VwLmMKQEAgLTI0ODUsMTMgKzI0ODUsMTMgQEAgc3RhdGljIGludCBndXBfaHVnZV9wZ2Qo cGdkX3Qgb3JpZywgcGdkX3QgKnBnZHAsIHVuc2lnbmVkIGxvbmcgYWRkciwKIAlyZXR1cm4gMTsK IH0KIAotc3RhdGljIGludCBndXBfcG1kX3JhbmdlKHB1ZF90IHB1ZCwgdW5zaWduZWQgbG9uZyBh ZGRyLCB1bnNpZ25lZCBsb25nIGVuZCwKK3N0YXRpYyBpbnQgZ3VwX3BtZF9yYW5nZShwdWRfdCAq cHVkcCwgcHVkX3QgcHVkLCB1bnNpZ25lZCBsb25nIGFkZHIsIHVuc2lnbmVkIGxvbmcgZW5kLAog CQl1bnNpZ25lZCBpbnQgZmxhZ3MsIHN0cnVjdCBwYWdlICoqcGFnZXMsIGludCAqbnIpCiB7CiAJ dW5zaWduZWQgbG9uZyBuZXh0OwogCXBtZF90ICpwbWRwOwogCi0JcG1kcCA9IHBtZF9vZmZzZXQo JnB1ZCwgYWRkcik7CisJcG1kcCA9IHBtZF9vZmZzZXRfbG9ja2xlc3MocHVkcCwgcHVkLCBhZGRy KTsKIAlkbyB7CiAJCXBtZF90IHBtZCA9IFJFQURfT05DRSgqcG1kcCk7CiAKQEAgLTI1MjgsMTMg KzI1MjgsMTMgQEAgc3RhdGljIGludCBndXBfcG1kX3JhbmdlKHB1ZF90IHB1ZCwgdW5zaWduZWQg bG9uZyBhZGRyLCB1bnNpZ25lZCBsb25nIGVuZCwKIAlyZXR1cm4gMTsKIH0KIAotc3RhdGljIGlu dCBndXBfcHVkX3JhbmdlKHA0ZF90IHA0ZCwgdW5zaWduZWQgbG9uZyBhZGRyLCB1bnNpZ25lZCBs b25nIGVuZCwKK3N0YXRpYyBpbnQgZ3VwX3B1ZF9yYW5nZShwNGRfdCAqcDRkcCwgcDRkX3QgcDRk LCB1bnNpZ25lZCBsb25nIGFkZHIsIHVuc2lnbmVkIGxvbmcgZW5kLAogCQkJIHVuc2lnbmVkIGlu dCBmbGFncywgc3RydWN0IHBhZ2UgKipwYWdlcywgaW50ICpucikKIHsKIAl1bnNpZ25lZCBsb25n IG5leHQ7CiAJcHVkX3QgKnB1ZHA7CiAKLQlwdWRwID0gcHVkX29mZnNldCgmcDRkLCBhZGRyKTsK KwlwdWRwID0gcHVkX29mZnNldF9sb2NrbGVzcyhwNGRwLCBwNGQsIGFkZHIpOwogCWRvIHsKIAkJ cHVkX3QgcHVkID0gUkVBRF9PTkNFKCpwdWRwKTsKIApAQCAtMjU0OSwyMCArMjU0OSwyMCBAQCBz dGF0aWMgaW50IGd1cF9wdWRfcmFuZ2UocDRkX3QgcDRkLCB1bnNpZ25lZCBsb25nIGFkZHIsIHVu c2lnbmVkIGxvbmcgZW5kLAogCQkJaWYgKCFndXBfaHVnZV9wZChfX2h1Z2VwZChwdWRfdmFsKHB1 ZCkpLCBhZGRyLAogCQkJCQkgUFVEX1NISUZULCBuZXh0LCBmbGFncywgcGFnZXMsIG5yKSkKIAkJ CQlyZXR1cm4gMDsKLQkJfSBlbHNlIGlmICghZ3VwX3BtZF9yYW5nZShwdWQsIGFkZHIsIG5leHQs IGZsYWdzLCBwYWdlcywgbnIpKQorCQl9IGVsc2UgaWYgKCFndXBfcG1kX3JhbmdlKHB1ZHAsIHB1 ZCwgYWRkciwgbmV4dCwgZmxhZ3MsIHBhZ2VzLCBucikpCiAJCQlyZXR1cm4gMDsKIAl9IHdoaWxl IChwdWRwKyssIGFkZHIgPSBuZXh0LCBhZGRyICE9IGVuZCk7CiAKIAlyZXR1cm4gMTsKIH0KIAot c3RhdGljIGludCBndXBfcDRkX3JhbmdlKHBnZF90IHBnZCwgdW5zaWduZWQgbG9uZyBhZGRyLCB1 bnNpZ25lZCBsb25nIGVuZCwKK3N0YXRpYyBpbnQgZ3VwX3A0ZF9yYW5nZShwZ2RfdCAqcGdkcCwg cGdkX3QgcGdkLCB1bnNpZ25lZCBsb25nIGFkZHIsIHVuc2lnbmVkIGxvbmcgZW5kLAogCQkJIHVu c2lnbmVkIGludCBmbGFncywgc3RydWN0IHBhZ2UgKipwYWdlcywgaW50ICpucikKIHsKIAl1bnNp Z25lZCBsb25nIG5leHQ7CiAJcDRkX3QgKnA0ZHA7CiAKLQlwNGRwID0gcDRkX29mZnNldCgmcGdk LCBhZGRyKTsKKwlwNGRwID0gcDRkX29mZnNldF9sb2NrbGVzcyhwZ2RwLCBwZ2QsIGFkZHIpOwog CWRvIHsKIAkJcDRkX3QgcDRkID0gUkVBRF9PTkNFKCpwNGRwKTsKIApAQCAtMjU3NCw3ICsyNTc0 LDcgQEAgc3RhdGljIGludCBndXBfcDRkX3JhbmdlKHBnZF90IHBnZCwgdW5zaWduZWQgbG9uZyBh ZGRyLCB1bnNpZ25lZCBsb25nIGVuZCwKIAkJCWlmICghZ3VwX2h1Z2VfcGQoX19odWdlcGQocDRk X3ZhbChwNGQpKSwgYWRkciwKIAkJCQkJIFA0RF9TSElGVCwgbmV4dCwgZmxhZ3MsIHBhZ2VzLCBu cikpCiAJCQkJcmV0dXJuIDA7Ci0JCX0gZWxzZSBpZiAoIWd1cF9wdWRfcmFuZ2UocDRkLCBhZGRy LCBuZXh0LCBmbGFncywgcGFnZXMsIG5yKSkKKwkJfSBlbHNlIGlmICghZ3VwX3B1ZF9yYW5nZShw NGRwLCBwNGQsIGFkZHIsIG5leHQsIGZsYWdzLCBwYWdlcywgbnIpKQogCQkJcmV0dXJuIDA7CiAJ fSB3aGlsZSAocDRkcCsrLCBhZGRyID0gbmV4dCwgYWRkciAhPSBlbmQpOwogCkBAIC0yNjAyLDcg KzI2MDIsNyBAQCBzdGF0aWMgdm9pZCBndXBfcGdkX3JhbmdlKHVuc2lnbmVkIGxvbmcgYWRkciwg dW5zaWduZWQgbG9uZyBlbmQsCiAJCQlpZiAoIWd1cF9odWdlX3BkKF9faHVnZXBkKHBnZF92YWwo cGdkKSksIGFkZHIsCiAJCQkJCSBQR0RJUl9TSElGVCwgbmV4dCwgZmxhZ3MsIHBhZ2VzLCBucikp CiAJCQkJcmV0dXJuOwotCQl9IGVsc2UgaWYgKCFndXBfcDRkX3JhbmdlKHBnZCwgYWRkciwgbmV4 dCwgZmxhZ3MsIHBhZ2VzLCBucikpCisJCX0gZWxzZSBpZiAoIWd1cF9wNGRfcmFuZ2UocGdkcCwg cGdkLCBhZGRyLCBuZXh0LCBmbGFncywgcGFnZXMsIG5yKSkKIAkJCXJldHVybjsKIAl9IHdoaWxl IChwZ2RwKyssIGFkZHIgPSBuZXh0LCBhZGRyICE9IGVuZCk7CiB9Ci0tIArio7/io7/io7/io7/i oovioYDio4DioLnio7/io7/io7/io78K4qO/4qO/4qO/4qO/4qCg4qO24qGm4qCA4qO/4qO/4qO/ 4qO/CuKjv+Kjv+Kjv+Kgj+KjtOKjruKjtOKjp+KgiOKiv+Kjv+Kjvwrio7/io7/ioY/iorDio7/i oJbio6Dio7/ioYbioIjio7/io78K4qO/4qKb4qO14qOE4qCZ4qO24qO24qGf4qOF4qOg4qC54qO/ CuKjv+KjnOKjm+Kgu+KijuKjieKjieKjgOKgv+Kjq+KjteKjvwoKX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX18KbGludXgtdW0gbWFpbGluZyBsaXN0CmxpbnV4 LXVtQGxpc3RzLmluZnJhZGVhZC5vcmcKaHR0cDovL2xpc3RzLmluZnJhZGVhZC5vcmcvbWFpbG1h bi9saXN0aW5mby9saW51eC11bQo=