From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.9 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40CABCA9ECB for ; Wed, 30 Oct 2019 22:52:08 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 092062067D for ; Wed, 30 Oct 2019 22:52:08 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=nvidia.com header.i=@nvidia.com header.b="q6Q/A8TB" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728142AbfJ3Wv5 (ORCPT ); Wed, 30 Oct 2019 18:51:57 -0400 Received: from hqemgate14.nvidia.com ([216.228.121.143]:1382 "EHLO hqemgate14.nvidia.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727671AbfJ3Wts (ORCPT ); Wed, 30 Oct 2019 18:49:48 -0400 Received: from hqpgpgate101.nvidia.com (Not Verified[216.228.121.13]) by hqemgate14.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Wed, 30 Oct 2019 15:49:43 -0700 Received: from hqmail.nvidia.com ([172.20.161.6]) by hqpgpgate101.nvidia.com (PGP Universal service); Wed, 30 Oct 2019 15:49:37 -0700 X-PGP-Universal: processed; by hqpgpgate101.nvidia.com on Wed, 30 Oct 2019 15:49:37 -0700 Received: from HQMAIL109.nvidia.com (172.20.187.15) by HQMAIL105.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Wed, 30 Oct 2019 22:49:37 +0000 Received: from HQMAIL109.nvidia.com (172.20.187.15) by HQMAIL109.nvidia.com (172.20.187.15) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Wed, 30 Oct 2019 22:49:37 +0000 Received: from rnnvemgw01.nvidia.com (10.128.109.123) by HQMAIL109.nvidia.com (172.20.187.15) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Wed, 30 Oct 2019 22:49:35 +0000 Received: from blueforge.nvidia.com (Not Verified[10.110.48.28]) by rnnvemgw01.nvidia.com with Trustwave SEG (v7,5,8,10121) id ; Wed, 30 Oct 2019 15:49:35 -0700 From: John Hubbard To: Andrew Morton CC: Al Viro , Alex Williamson , Benjamin Herrenschmidt , =?UTF-8?q?Bj=C3=B6rn=20T=C3=B6pel?= , Christoph Hellwig , Dan Williams , Daniel Vetter , Dave Chinner , David Airlie , "David S . Miller" , Ira Weiny , Jan Kara , Jason Gunthorpe , Jens Axboe , Jonathan Corbet , =?UTF-8?q?J=C3=A9r=C3=B4me=20Glisse?= , Magnus Karlsson , "Mauro Carvalho Chehab" , Michael Ellerman , Michal Hocko , Mike Kravetz , "Paul Mackerras" , Shuah Khan , Vlastimil Babka , , , , , , , , , , , , , LKML , John Hubbard , "Christoph Hellwig" , "Aneesh Kumar K . V" Subject: [PATCH 02/19] mm/gup: factor out duplicate code from four routines Date: Wed, 30 Oct 2019 15:49:13 -0700 Message-ID: <20191030224930.3990755-3-jhubbard@nvidia.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191030224930.3990755-1-jhubbard@nvidia.com> References: <20191030224930.3990755-1-jhubbard@nvidia.com> MIME-Version: 1.0 X-NVConfidentiality: public Content-Transfer-Encoding: quoted-printable Content-Type: text/plain DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1572475783; bh=tBh63Up4GmYWMpZfv2SoHkQER3l3/AwmpTx4CmGyUOY=; h=X-PGP-Universal:From:To:CC:Subject:Date:Message-ID:X-Mailer: In-Reply-To:References:MIME-Version:X-NVConfidentiality: Content-Transfer-Encoding:Content-Type; b=q6Q/A8TBBZ55PyF/FZZqYnJ+lgBDwYNDtC48DaucDzCF0TjR8x89MUFmfJrXfVIy8 YhLIOpQBeIReIooXA8TPZibffV6cpNr4lEyyTxNLOCl4fSlAiNzImWEmpAHpBQAgnm 2VJfy72jnhXV4Na9artoMHPQqpC3lkGe4ymiRJWOT1WeuiuvX5ZsQGDmnAJG4LGImR 9ty1Onsdh/I+zLky4Hr6hnCHYagTmLcM/PUFXTvUJ2RHGzNqzx9q0kjJ96JHXHJJWk 2oGM89HTfGbtaCai1JlzNt3fE78ewH7qAVLMEPdwaEG9DhYOb9MPVg/HVxjVhX4dEl BY9wQk9uQlmQw== Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org There are four locations in gup.c that have a fair amount of code duplication. This means that changing one requires making the same changes in four places, not to mention reading the same code four times, and wondering if there are subtle differences. Factor out the common code into static functions, thus reducing the overall line count and the code's complexity. Also, take the opportunity to slightly improve the efficiency of the error cases, by doing a mass subtraction of the refcount, surrounded by get_page()/put_page(). Also, further simplify (slightly), by waiting until the the successful end of each routine, to increment *nr. Cc: Christoph Hellwig Cc: Aneesh Kumar K.V Signed-off-by: John Hubbard --- mm/gup.c | 113 ++++++++++++++++++++++--------------------------------- 1 file changed, 46 insertions(+), 67 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 85caf76b3012..8fb0d9cdfaf5 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1969,6 +1969,35 @@ static int __gup_device_huge_pud(pud_t pud, pud_t *p= udp, unsigned long addr, } #endif =20 +static int __record_subpages(struct page *page, unsigned long addr, + unsigned long end, struct page **pages, int nr) +{ + int nr_recorded_pages =3D 0; + + do { + pages[nr] =3D page; + nr++; + page++; + nr_recorded_pages++; + } while (addr +=3D PAGE_SIZE, addr !=3D end); + return nr_recorded_pages; +} + +static void __remove_refs_from_head(struct page *page, int refs) +{ + /* Do a get_page() first, in case refs =3D=3D page->_refcount */ + get_page(page); + page_ref_sub(page, refs); + put_page(page); +} + +static int __huge_pt_done(struct page *head, int nr_recorded_pages, int *n= r) +{ + *nr +=3D nr_recorded_pages; + SetPageReferenced(head); + return 1; +} + #ifdef CONFIG_ARCH_HAS_HUGEPD static unsigned long hugepte_addr_end(unsigned long addr, unsigned long en= d, unsigned long sz) @@ -1998,34 +2027,19 @@ static int gup_hugepte(pte_t *ptep, unsigned long s= z, unsigned long addr, /* hugepages are never "special" */ VM_BUG_ON(!pfn_valid(pte_pfn(pte))); =20 - refs =3D 0; head =3D pte_page(pte); - page =3D head + ((addr & (sz-1)) >> PAGE_SHIFT); - do { - VM_BUG_ON(compound_head(page) !=3D head); - pages[*nr] =3D page; - (*nr)++; - page++; - refs++; - } while (addr +=3D PAGE_SIZE, addr !=3D end); + refs =3D __record_subpages(page, addr, end, pages, *nr); =20 head =3D try_get_compound_head(head, refs); - if (!head) { - *nr -=3D refs; + if (!head) return 0; - } =20 if (unlikely(pte_val(pte) !=3D pte_val(*ptep))) { - /* Could be optimized better */ - *nr -=3D refs; - while (refs--) - put_page(head); + __remove_refs_from_head(head, refs); return 0; } - - SetPageReferenced(head); - return 1; + return __huge_pt_done(head, refs, nr); } =20 static int gup_huge_pd(hugepd_t hugepd, unsigned long addr, @@ -2071,30 +2085,18 @@ static int gup_huge_pmd(pmd_t orig, pmd_t *pmdp, un= signed long addr, pages, nr); } =20 - refs =3D 0; page =3D pmd_page(orig) + ((addr & ~PMD_MASK) >> PAGE_SHIFT); - do { - pages[*nr] =3D page; - (*nr)++; - page++; - refs++; - } while (addr +=3D PAGE_SIZE, addr !=3D end); + refs =3D __record_subpages(page, addr, end, pages, *nr); =20 head =3D try_get_compound_head(pmd_page(orig), refs); - if (!head) { - *nr -=3D refs; + if (!head) return 0; - } =20 if (unlikely(pmd_val(orig) !=3D pmd_val(*pmdp))) { - *nr -=3D refs; - while (refs--) - put_page(head); + __remove_refs_from_head(head, refs); return 0; } - - SetPageReferenced(head); - return 1; + return __huge_pt_done(head, refs, nr); } =20 static int gup_huge_pud(pud_t orig, pud_t *pudp, unsigned long addr, @@ -2114,30 +2116,18 @@ static int gup_huge_pud(pud_t orig, pud_t *pudp, un= signed long addr, pages, nr); } =20 - refs =3D 0; page =3D pud_page(orig) + ((addr & ~PUD_MASK) >> PAGE_SHIFT); - do { - pages[*nr] =3D page; - (*nr)++; - page++; - refs++; - } while (addr +=3D PAGE_SIZE, addr !=3D end); + refs =3D __record_subpages(page, addr, end, pages, *nr); =20 head =3D try_get_compound_head(pud_page(orig), refs); - if (!head) { - *nr -=3D refs; + if (!head) return 0; - } =20 if (unlikely(pud_val(orig) !=3D pud_val(*pudp))) { - *nr -=3D refs; - while (refs--) - put_page(head); + __remove_refs_from_head(head, refs); return 0; } - - SetPageReferenced(head); - return 1; + return __huge_pt_done(head, refs, nr); } =20 static int gup_huge_pgd(pgd_t orig, pgd_t *pgdp, unsigned long addr, @@ -2151,30 +2141,19 @@ static int gup_huge_pgd(pgd_t orig, pgd_t *pgdp, un= signed long addr, return 0; =20 BUILD_BUG_ON(pgd_devmap(orig)); - refs =3D 0; + page =3D pgd_page(orig) + ((addr & ~PGDIR_MASK) >> PAGE_SHIFT); - do { - pages[*nr] =3D page; - (*nr)++; - page++; - refs++; - } while (addr +=3D PAGE_SIZE, addr !=3D end); + refs =3D __record_subpages(page, addr, end, pages, *nr); =20 head =3D try_get_compound_head(pgd_page(orig), refs); - if (!head) { - *nr -=3D refs; + if (!head) return 0; - } =20 if (unlikely(pgd_val(orig) !=3D pgd_val(*pgdp))) { - *nr -=3D refs; - while (refs--) - put_page(head); + __remove_refs_from_head(head, refs); return 0; } - - SetPageReferenced(head); - return 1; + return __huge_pt_done(head, refs, nr); } =20 static int gup_pmd_range(pud_t pud, unsigned long addr, unsigned long end, --=20 2.23.0 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.6 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1AA62CA9EC7 for ; Wed, 30 Oct 2019 22:58:48 +0000 (UTC) Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id BB16220874 for ; Wed, 30 Oct 2019 22:58:47 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=nvidia.com header.i=@nvidia.com header.b="q6Q/A8TB" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BB16220874 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=nvidia.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 473P5K53Q0zF3nF for ; Thu, 31 Oct 2019 09:58:45 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nvidia.com (client-ip=216.228.121.143; helo=hqemgate14.nvidia.com; envelope-from=jhubbard@nvidia.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=nvidia.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=nvidia.com header.i=@nvidia.com header.b="q6Q/A8TB"; dkim-atps=neutral Received: from hqemgate14.nvidia.com (hqemgate14.nvidia.com [216.228.121.143]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 473Ntv3R0nzF4lh for ; Thu, 31 Oct 2019 09:49:43 +1100 (AEDT) Received: from hqpgpgate101.nvidia.com (Not Verified[216.228.121.13]) by hqemgate14.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Wed, 30 Oct 2019 15:49:43 -0700 Received: from hqmail.nvidia.com ([172.20.161.6]) by hqpgpgate101.nvidia.com (PGP Universal service); Wed, 30 Oct 2019 15:49:37 -0700 X-PGP-Universal: processed; by hqpgpgate101.nvidia.com on Wed, 30 Oct 2019 15:49:37 -0700 Received: from HQMAIL109.nvidia.com (172.20.187.15) by HQMAIL105.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Wed, 30 Oct 2019 22:49:37 +0000 Received: from HQMAIL109.nvidia.com (172.20.187.15) by HQMAIL109.nvidia.com (172.20.187.15) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Wed, 30 Oct 2019 22:49:37 +0000 Received: from rnnvemgw01.nvidia.com (10.128.109.123) by HQMAIL109.nvidia.com (172.20.187.15) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Wed, 30 Oct 2019 22:49:35 +0000 Received: from blueforge.nvidia.com (Not Verified[10.110.48.28]) by rnnvemgw01.nvidia.com with Trustwave SEG (v7, 5, 8, 10121) id ; Wed, 30 Oct 2019 15:49:35 -0700 From: John Hubbard To: Andrew Morton Subject: [PATCH 02/19] mm/gup: factor out duplicate code from four routines Date: Wed, 30 Oct 2019 15:49:13 -0700 Message-ID: <20191030224930.3990755-3-jhubbard@nvidia.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191030224930.3990755-1-jhubbard@nvidia.com> References: <20191030224930.3990755-1-jhubbard@nvidia.com> MIME-Version: 1.0 X-NVConfidentiality: public Content-Transfer-Encoding: quoted-printable Content-Type: text/plain DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1572475783; bh=tBh63Up4GmYWMpZfv2SoHkQER3l3/AwmpTx4CmGyUOY=; h=X-PGP-Universal:From:To:CC:Subject:Date:Message-ID:X-Mailer: In-Reply-To:References:MIME-Version:X-NVConfidentiality: Content-Transfer-Encoding:Content-Type; b=q6Q/A8TBBZ55PyF/FZZqYnJ+lgBDwYNDtC48DaucDzCF0TjR8x89MUFmfJrXfVIy8 YhLIOpQBeIReIooXA8TPZibffV6cpNr4lEyyTxNLOCl4fSlAiNzImWEmpAHpBQAgnm 2VJfy72jnhXV4Na9artoMHPQqpC3lkGe4ymiRJWOT1WeuiuvX5ZsQGDmnAJG4LGImR 9ty1Onsdh/I+zLky4Hr6hnCHYagTmLcM/PUFXTvUJ2RHGzNqzx9q0kjJ96JHXHJJWk 2oGM89HTfGbtaCai1JlzNt3fE78ewH7qAVLMEPdwaEG9DhYOb9MPVg/HVxjVhX4dEl BY9wQk9uQlmQw== X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Michal Hocko , Jan Kara , kvm@vger.kernel.org, linux-doc@vger.kernel.org, David Airlie , Dave Chinner , dri-devel@lists.freedesktop.org, LKML , linux-mm@kvack.org, Paul Mackerras , linux-kselftest@vger.kernel.org, Ira Weiny , Christoph Hellwig , Jonathan Corbet , linux-rdma@vger.kernel.org, Christoph Hellwig , Jason Gunthorpe , Vlastimil Babka , =?UTF-8?q?Bj=C3=B6rn=20T=C3=B6pel?= , linux-media@vger.kernel.org, Shuah Khan , John Hubbard , linux-block@vger.kernel.org, =?UTF-8?q?J=C3=A9r=C3=B4me=20Glisse?= , Al Viro , Dan Williams , Mauro Carvalho Chehab , Magnus Karlsson , Jens Axboe , netdev@vger.kernel.org, Alex Williamson , Daniel Vetter , "Aneesh Kumar K . V" , linux-fsdevel@vger.kernel.org, bpf@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, "David S . Miller" , Mike Kravetz Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" There are four locations in gup.c that have a fair amount of code duplication. This means that changing one requires making the same changes in four places, not to mention reading the same code four times, and wondering if there are subtle differences. Factor out the common code into static functions, thus reducing the overall line count and the code's complexity. Also, take the opportunity to slightly improve the efficiency of the error cases, by doing a mass subtraction of the refcount, surrounded by get_page()/put_page(). Also, further simplify (slightly), by waiting until the the successful end of each routine, to increment *nr. Cc: Christoph Hellwig Cc: Aneesh Kumar K.V Signed-off-by: John Hubbard --- mm/gup.c | 113 ++++++++++++++++++++++--------------------------------- 1 file changed, 46 insertions(+), 67 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 85caf76b3012..8fb0d9cdfaf5 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1969,6 +1969,35 @@ static int __gup_device_huge_pud(pud_t pud, pud_t *p= udp, unsigned long addr, } #endif =20 +static int __record_subpages(struct page *page, unsigned long addr, + unsigned long end, struct page **pages, int nr) +{ + int nr_recorded_pages =3D 0; + + do { + pages[nr] =3D page; + nr++; + page++; + nr_recorded_pages++; + } while (addr +=3D PAGE_SIZE, addr !=3D end); + return nr_recorded_pages; +} + +static void __remove_refs_from_head(struct page *page, int refs) +{ + /* Do a get_page() first, in case refs =3D=3D page->_refcount */ + get_page(page); + page_ref_sub(page, refs); + put_page(page); +} + +static int __huge_pt_done(struct page *head, int nr_recorded_pages, int *n= r) +{ + *nr +=3D nr_recorded_pages; + SetPageReferenced(head); + return 1; +} + #ifdef CONFIG_ARCH_HAS_HUGEPD static unsigned long hugepte_addr_end(unsigned long addr, unsigned long en= d, unsigned long sz) @@ -1998,34 +2027,19 @@ static int gup_hugepte(pte_t *ptep, unsigned long s= z, unsigned long addr, /* hugepages are never "special" */ VM_BUG_ON(!pfn_valid(pte_pfn(pte))); =20 - refs =3D 0; head =3D pte_page(pte); - page =3D head + ((addr & (sz-1)) >> PAGE_SHIFT); - do { - VM_BUG_ON(compound_head(page) !=3D head); - pages[*nr] =3D page; - (*nr)++; - page++; - refs++; - } while (addr +=3D PAGE_SIZE, addr !=3D end); + refs =3D __record_subpages(page, addr, end, pages, *nr); =20 head =3D try_get_compound_head(head, refs); - if (!head) { - *nr -=3D refs; + if (!head) return 0; - } =20 if (unlikely(pte_val(pte) !=3D pte_val(*ptep))) { - /* Could be optimized better */ - *nr -=3D refs; - while (refs--) - put_page(head); + __remove_refs_from_head(head, refs); return 0; } - - SetPageReferenced(head); - return 1; + return __huge_pt_done(head, refs, nr); } =20 static int gup_huge_pd(hugepd_t hugepd, unsigned long addr, @@ -2071,30 +2085,18 @@ static int gup_huge_pmd(pmd_t orig, pmd_t *pmdp, un= signed long addr, pages, nr); } =20 - refs =3D 0; page =3D pmd_page(orig) + ((addr & ~PMD_MASK) >> PAGE_SHIFT); - do { - pages[*nr] =3D page; - (*nr)++; - page++; - refs++; - } while (addr +=3D PAGE_SIZE, addr !=3D end); + refs =3D __record_subpages(page, addr, end, pages, *nr); =20 head =3D try_get_compound_head(pmd_page(orig), refs); - if (!head) { - *nr -=3D refs; + if (!head) return 0; - } =20 if (unlikely(pmd_val(orig) !=3D pmd_val(*pmdp))) { - *nr -=3D refs; - while (refs--) - put_page(head); + __remove_refs_from_head(head, refs); return 0; } - - SetPageReferenced(head); - return 1; + return __huge_pt_done(head, refs, nr); } =20 static int gup_huge_pud(pud_t orig, pud_t *pudp, unsigned long addr, @@ -2114,30 +2116,18 @@ static int gup_huge_pud(pud_t orig, pud_t *pudp, un= signed long addr, pages, nr); } =20 - refs =3D 0; page =3D pud_page(orig) + ((addr & ~PUD_MASK) >> PAGE_SHIFT); - do { - pages[*nr] =3D page; - (*nr)++; - page++; - refs++; - } while (addr +=3D PAGE_SIZE, addr !=3D end); + refs =3D __record_subpages(page, addr, end, pages, *nr); =20 head =3D try_get_compound_head(pud_page(orig), refs); - if (!head) { - *nr -=3D refs; + if (!head) return 0; - } =20 if (unlikely(pud_val(orig) !=3D pud_val(*pudp))) { - *nr -=3D refs; - while (refs--) - put_page(head); + __remove_refs_from_head(head, refs); return 0; } - - SetPageReferenced(head); - return 1; + return __huge_pt_done(head, refs, nr); } =20 static int gup_huge_pgd(pgd_t orig, pgd_t *pgdp, unsigned long addr, @@ -2151,30 +2141,19 @@ static int gup_huge_pgd(pgd_t orig, pgd_t *pgdp, un= signed long addr, return 0; =20 BUILD_BUG_ON(pgd_devmap(orig)); - refs =3D 0; + page =3D pgd_page(orig) + ((addr & ~PGDIR_MASK) >> PAGE_SHIFT); - do { - pages[*nr] =3D page; - (*nr)++; - page++; - refs++; - } while (addr +=3D PAGE_SIZE, addr !=3D end); + refs =3D __record_subpages(page, addr, end, pages, *nr); =20 head =3D try_get_compound_head(pgd_page(orig), refs); - if (!head) { - *nr -=3D refs; + if (!head) return 0; - } =20 if (unlikely(pgd_val(orig) !=3D pgd_val(*pgdp))) { - *nr -=3D refs; - while (refs--) - put_page(head); + __remove_refs_from_head(head, refs); return 0; } - - SetPageReferenced(head); - return 1; + return __huge_pt_done(head, refs, nr); } =20 static int gup_pmd_range(pud_t pud, unsigned long addr, unsigned long end, --=20 2.23.0 From mboxrd@z Thu Jan 1 00:00:00 1970 From: John Hubbard Subject: [PATCH 02/19] mm/gup: factor out duplicate code from four routines Date: Wed, 30 Oct 2019 15:49:13 -0700 Message-ID: <20191030224930.3990755-3-jhubbard@nvidia.com> References: <20191030224930.3990755-1-jhubbard@nvidia.com> Mime-Version: 1.0 Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: <20191030224930.3990755-1-jhubbard@nvidia.com> Sender: linux-kernel-owner@vger.kernel.org To: Andrew Morton Cc: Al Viro , Alex Williamson , Benjamin Herrenschmidt , =?UTF-8?q?Bj=C3=B6rn=20T=C3=B6pel?= , Christoph Hellwig , Dan Williams , Daniel Vetter , Dave Chinner , David Airlie , "David S . Miller" , Ira Weiny , Jan Kara , Jason Gunthorpe , Jens Axboe , Jonathan Corbet , =?UTF-8?q?J=C3=A9r=C3=B4me=20Glisse?= , Magnus Karlsson , Mauro Carvalho Chehab , Michael Ellerman List-Id: dri-devel@lists.freedesktop.org There are four locations in gup.c that have a fair amount of code duplication. This means that changing one requires making the same changes in four places, not to mention reading the same code four times, and wondering if there are subtle differences. Factor out the common code into static functions, thus reducing the overall line count and the code's complexity. Also, take the opportunity to slightly improve the efficiency of the error cases, by doing a mass subtraction of the refcount, surrounded by get_page()/put_page(). Also, further simplify (slightly), by waiting until the the successful end of each routine, to increment *nr. Cc: Christoph Hellwig Cc: Aneesh Kumar K.V Signed-off-by: John Hubbard --- mm/gup.c | 113 ++++++++++++++++++++++--------------------------------- 1 file changed, 46 insertions(+), 67 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 85caf76b3012..8fb0d9cdfaf5 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1969,6 +1969,35 @@ static int __gup_device_huge_pud(pud_t pud, pud_t *p= udp, unsigned long addr, } #endif =20 +static int __record_subpages(struct page *page, unsigned long addr, + unsigned long end, struct page **pages, int nr) +{ + int nr_recorded_pages =3D 0; + + do { + pages[nr] =3D page; + nr++; + page++; + nr_recorded_pages++; + } while (addr +=3D PAGE_SIZE, addr !=3D end); + return nr_recorded_pages; +} + +static void __remove_refs_from_head(struct page *page, int refs) +{ + /* Do a get_page() first, in case refs =3D=3D page->_refcount */ + get_page(page); + page_ref_sub(page, refs); + put_page(page); +} + +static int __huge_pt_done(struct page *head, int nr_recorded_pages, int *n= r) +{ + *nr +=3D nr_recorded_pages; + SetPageReferenced(head); + return 1; +} + #ifdef CONFIG_ARCH_HAS_HUGEPD static unsigned long hugepte_addr_end(unsigned long addr, unsigned long en= d, unsigned long sz) @@ -1998,34 +2027,19 @@ static int gup_hugepte(pte_t *ptep, unsigned long s= z, unsigned long addr, /* hugepages are never "special" */ VM_BUG_ON(!pfn_valid(pte_pfn(pte))); =20 - refs =3D 0; head =3D pte_page(pte); - page =3D head + ((addr & (sz-1)) >> PAGE_SHIFT); - do { - VM_BUG_ON(compound_head(page) !=3D head); - pages[*nr] =3D page; - (*nr)++; - page++; - refs++; - } while (addr +=3D PAGE_SIZE, addr !=3D end); + refs =3D __record_subpages(page, addr, end, pages, *nr); =20 head =3D try_get_compound_head(head, refs); - if (!head) { - *nr -=3D refs; + if (!head) return 0; - } =20 if (unlikely(pte_val(pte) !=3D pte_val(*ptep))) { - /* Could be optimized better */ - *nr -=3D refs; - while (refs--) - put_page(head); + __remove_refs_from_head(head, refs); return 0; } - - SetPageReferenced(head); - return 1; + return __huge_pt_done(head, refs, nr); } =20 static int gup_huge_pd(hugepd_t hugepd, unsigned long addr, @@ -2071,30 +2085,18 @@ static int gup_huge_pmd(pmd_t orig, pmd_t *pmdp, un= signed long addr, pages, nr); } =20 - refs =3D 0; page =3D pmd_page(orig) + ((addr & ~PMD_MASK) >> PAGE_SHIFT); - do { - pages[*nr] =3D page; - (*nr)++; - page++; - refs++; - } while (addr +=3D PAGE_SIZE, addr !=3D end); + refs =3D __record_subpages(page, addr, end, pages, *nr); =20 head =3D try_get_compound_head(pmd_page(orig), refs); - if (!head) { - *nr -=3D refs; + if (!head) return 0; - } =20 if (unlikely(pmd_val(orig) !=3D pmd_val(*pmdp))) { - *nr -=3D refs; - while (refs--) - put_page(head); + __remove_refs_from_head(head, refs); return 0; } - - SetPageReferenced(head); - return 1; + return __huge_pt_done(head, refs, nr); } =20 static int gup_huge_pud(pud_t orig, pud_t *pudp, unsigned long addr, @@ -2114,30 +2116,18 @@ static int gup_huge_pud(pud_t orig, pud_t *pudp, un= signed long addr, pages, nr); } =20 - refs =3D 0; page =3D pud_page(orig) + ((addr & ~PUD_MASK) >> PAGE_SHIFT); - do { - pages[*nr] =3D page; - (*nr)++; - page++; - refs++; - } while (addr +=3D PAGE_SIZE, addr !=3D end); + refs =3D __record_subpages(page, addr, end, pages, *nr); =20 head =3D try_get_compound_head(pud_page(orig), refs); - if (!head) { - *nr -=3D refs; + if (!head) return 0; - } =20 if (unlikely(pud_val(orig) !=3D pud_val(*pudp))) { - *nr -=3D refs; - while (refs--) - put_page(head); + __remove_refs_from_head(head, refs); return 0; } - - SetPageReferenced(head); - return 1; + return __huge_pt_done(head, refs, nr); } =20 static int gup_huge_pgd(pgd_t orig, pgd_t *pgdp, unsigned long addr, @@ -2151,30 +2141,19 @@ static int gup_huge_pgd(pgd_t orig, pgd_t *pgdp, un= signed long addr, return 0; =20 BUILD_BUG_ON(pgd_devmap(orig)); - refs =3D 0; + page =3D pgd_page(orig) + ((addr & ~PGDIR_MASK) >> PAGE_SHIFT); - do { - pages[*nr] =3D page; - (*nr)++; - page++; - refs++; - } while (addr +=3D PAGE_SIZE, addr !=3D end); + refs =3D __record_subpages(page, addr, end, pages, *nr); =20 head =3D try_get_compound_head(pgd_page(orig), refs); - if (!head) { - *nr -=3D refs; + if (!head) return 0; - } =20 if (unlikely(pgd_val(orig) !=3D pgd_val(*pgdp))) { - *nr -=3D refs; - while (refs--) - put_page(head); + __remove_refs_from_head(head, refs); return 0; } - - SetPageReferenced(head); - return 1; + return __huge_pt_done(head, refs, nr); } =20 static int gup_pmd_range(pud_t pud, unsigned long addr, unsigned long end, --=20 2.23.0 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.8 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BFD9CCA9EC3 for ; Thu, 31 Oct 2019 09:05:32 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9A9902083E for ; Thu, 31 Oct 2019 09:05:32 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9A9902083E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=nvidia.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 4EA826EDEC; Thu, 31 Oct 2019 09:04:48 +0000 (UTC) Received: from hqemgate14.nvidia.com (hqemgate14.nvidia.com [216.228.121.143]) by gabe.freedesktop.org (Postfix) with ESMTPS id 60C396EBB5 for ; Wed, 30 Oct 2019 22:49:38 +0000 (UTC) Received: from hqpgpgate101.nvidia.com (Not Verified[216.228.121.13]) by hqemgate14.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Wed, 30 Oct 2019 15:49:43 -0700 Received: from hqmail.nvidia.com ([172.20.161.6]) by hqpgpgate101.nvidia.com (PGP Universal service); Wed, 30 Oct 2019 15:49:37 -0700 X-PGP-Universal: processed; by hqpgpgate101.nvidia.com on Wed, 30 Oct 2019 15:49:37 -0700 Received: from HQMAIL109.nvidia.com (172.20.187.15) by HQMAIL105.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Wed, 30 Oct 2019 22:49:37 +0000 Received: from HQMAIL109.nvidia.com (172.20.187.15) by HQMAIL109.nvidia.com (172.20.187.15) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Wed, 30 Oct 2019 22:49:37 +0000 Received: from rnnvemgw01.nvidia.com (10.128.109.123) by HQMAIL109.nvidia.com (172.20.187.15) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Wed, 30 Oct 2019 22:49:35 +0000 Received: from blueforge.nvidia.com (Not Verified[10.110.48.28]) by rnnvemgw01.nvidia.com with Trustwave SEG (v7, 5, 8, 10121) id ; Wed, 30 Oct 2019 15:49:35 -0700 From: John Hubbard To: Andrew Morton Subject: [PATCH 02/19] mm/gup: factor out duplicate code from four routines Date: Wed, 30 Oct 2019 15:49:13 -0700 Message-ID: <20191030224930.3990755-3-jhubbard@nvidia.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191030224930.3990755-1-jhubbard@nvidia.com> References: <20191030224930.3990755-1-jhubbard@nvidia.com> MIME-Version: 1.0 X-NVConfidentiality: public X-Mailman-Approved-At: Thu, 31 Oct 2019 09:04:21 +0000 X-Mailman-Original-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1572475784; bh=tBh63Up4GmYWMpZfv2SoHkQER3l3/AwmpTx4CmGyUOY=; h=X-PGP-Universal:From:To:CC:Subject:Date:Message-ID:X-Mailer: In-Reply-To:References:MIME-Version:X-NVConfidentiality: Content-Transfer-Encoding:Content-Type; b=HfYnE1A7nNkNpazJA+iLbvozSO9YbiH5kFIM0F6u1AK11c0lDyfxJZvAXrwIUDkv5 Vi1RgSE+zFxIrfTIb+uivdz+6fy0MYuaxW2g0l5bgHHvYabolD6OOfarmWn9segALX 8JoVoT/njsBqs073lipSciCZeLvL3FcZ63n2zzY2AyGVYc0U2PFJ7DVW15EbUZzRaC nr9qEHpp4TQE8CN0n9dZIQZcOAFydDiH5gU9U2HcVPlAQ1/9WyMmCuL+SLoSd1EOQF /Ez6RQKc30HZLoGVdYe0xDSBYplfU9zfQz9KjxQQQDaUpbasR5wg6ipvfGvkp4XC3s GKCvLZ2BAnk4g== X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Michal Hocko , Jan Kara , kvm@vger.kernel.org, linux-doc@vger.kernel.org, David Airlie , Dave Chinner , dri-devel@lists.freedesktop.org, LKML , linux-mm@kvack.org, Paul Mackerras , linux-kselftest@vger.kernel.org, Ira Weiny , Christoph Hellwig , Jonathan Corbet , linux-rdma@vger.kernel.org, Michael Ellerman , Christoph Hellwig , Jason Gunthorpe , Vlastimil Babka , =?UTF-8?q?Bj=C3=B6rn=20T=C3=B6pel?= , linux-media@vger.kernel.org, Shuah Khan , John Hubbard , linux-block@vger.kernel.org, =?UTF-8?q?J=C3=A9r=C3=B4me=20Glisse?= , Al Viro , Dan Williams , Mauro Carvalho Chehab , Magnus Karlsson , Jens Axboe , netdev@vger.kernel.org, Alex Williamson , "Aneesh Kumar K . V" , linux-fsdevel@vger.kernel.org, bpf@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, "David S . Miller" , Mike Kravetz Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Message-ID: <20191030224913.lYYHWSDQShCfnfy1AS8GVmlXaDEvIvWnwX69kC4LGK8@z> VGhlcmUgYXJlIGZvdXIgbG9jYXRpb25zIGluIGd1cC5jIHRoYXQgaGF2ZSBhIGZhaXIgYW1vdW50 IG9mIGNvZGUKZHVwbGljYXRpb24uIFRoaXMgbWVhbnMgdGhhdCBjaGFuZ2luZyBvbmUgcmVxdWly ZXMgbWFraW5nIHRoZSBzYW1lCmNoYW5nZXMgaW4gZm91ciBwbGFjZXMsIG5vdCB0byBtZW50aW9u IHJlYWRpbmcgdGhlIHNhbWUgY29kZSBmb3VyCnRpbWVzLCBhbmQgd29uZGVyaW5nIGlmIHRoZXJl IGFyZSBzdWJ0bGUgZGlmZmVyZW5jZXMuCgpGYWN0b3Igb3V0IHRoZSBjb21tb24gY29kZSBpbnRv IHN0YXRpYyBmdW5jdGlvbnMsIHRodXMgcmVkdWNpbmcgdGhlCm92ZXJhbGwgbGluZSBjb3VudCBh bmQgdGhlIGNvZGUncyBjb21wbGV4aXR5LgoKQWxzbywgdGFrZSB0aGUgb3Bwb3J0dW5pdHkgdG8g c2xpZ2h0bHkgaW1wcm92ZSB0aGUgZWZmaWNpZW5jeSBvZiB0aGUKZXJyb3IgY2FzZXMsIGJ5IGRv aW5nIGEgbWFzcyBzdWJ0cmFjdGlvbiBvZiB0aGUgcmVmY291bnQsIHN1cnJvdW5kZWQKYnkgZ2V0 X3BhZ2UoKS9wdXRfcGFnZSgpLgoKQWxzbywgZnVydGhlciBzaW1wbGlmeSAoc2xpZ2h0bHkpLCBi eSB3YWl0aW5nIHVudGlsIHRoZSB0aGUgc3VjY2Vzc2Z1bAplbmQgb2YgZWFjaCByb3V0aW5lLCB0 byBpbmNyZW1lbnQgKm5yLgoKQ2M6IENocmlzdG9waCBIZWxsd2lnIDxoY2hAbHN0LmRlPgpDYzog QW5lZXNoIEt1bWFyIEsuViA8YW5lZXNoLmt1bWFyQGxpbnV4LmlibS5jb20+ClNpZ25lZC1vZmYt Ynk6IEpvaG4gSHViYmFyZCA8amh1YmJhcmRAbnZpZGlhLmNvbT4KLS0tCiBtbS9ndXAuYyB8IDEx MyArKysrKysrKysrKysrKysrKysrKysrLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t CiAxIGZpbGUgY2hhbmdlZCwgNDYgaW5zZXJ0aW9ucygrKSwgNjcgZGVsZXRpb25zKC0pCgpkaWZm IC0tZ2l0IGEvbW0vZ3VwLmMgYi9tbS9ndXAuYwppbmRleCA4NWNhZjc2YjMwMTIuLjhmYjBkOWNk ZmFmNSAxMDA2NDQKLS0tIGEvbW0vZ3VwLmMKKysrIGIvbW0vZ3VwLmMKQEAgLTE5NjksNiArMTk2 OSwzNSBAQCBzdGF0aWMgaW50IF9fZ3VwX2RldmljZV9odWdlX3B1ZChwdWRfdCBwdWQsIHB1ZF90 ICpwdWRwLCB1bnNpZ25lZCBsb25nIGFkZHIsCiB9CiAjZW5kaWYKIAorc3RhdGljIGludCBfX3Jl Y29yZF9zdWJwYWdlcyhzdHJ1Y3QgcGFnZSAqcGFnZSwgdW5zaWduZWQgbG9uZyBhZGRyLAorCQkJ ICAgICB1bnNpZ25lZCBsb25nIGVuZCwgc3RydWN0IHBhZ2UgKipwYWdlcywgaW50IG5yKQorewor CWludCBucl9yZWNvcmRlZF9wYWdlcyA9IDA7CisKKwlkbyB7CisJCXBhZ2VzW25yXSA9IHBhZ2U7 CisJCW5yKys7CisJCXBhZ2UrKzsKKwkJbnJfcmVjb3JkZWRfcGFnZXMrKzsKKwl9IHdoaWxlIChh ZGRyICs9IFBBR0VfU0laRSwgYWRkciAhPSBlbmQpOworCXJldHVybiBucl9yZWNvcmRlZF9wYWdl czsKK30KKworc3RhdGljIHZvaWQgX19yZW1vdmVfcmVmc19mcm9tX2hlYWQoc3RydWN0IHBhZ2Ug KnBhZ2UsIGludCByZWZzKQoreworCS8qIERvIGEgZ2V0X3BhZ2UoKSBmaXJzdCwgaW4gY2FzZSBy ZWZzID09IHBhZ2UtPl9yZWZjb3VudCAqLworCWdldF9wYWdlKHBhZ2UpOworCXBhZ2VfcmVmX3N1 YihwYWdlLCByZWZzKTsKKwlwdXRfcGFnZShwYWdlKTsKK30KKworc3RhdGljIGludCBfX2h1Z2Vf cHRfZG9uZShzdHJ1Y3QgcGFnZSAqaGVhZCwgaW50IG5yX3JlY29yZGVkX3BhZ2VzLCBpbnQgKm5y KQoreworCSpuciArPSBucl9yZWNvcmRlZF9wYWdlczsKKwlTZXRQYWdlUmVmZXJlbmNlZChoZWFk KTsKKwlyZXR1cm4gMTsKK30KKwogI2lmZGVmIENPTkZJR19BUkNIX0hBU19IVUdFUEQKIHN0YXRp YyB1bnNpZ25lZCBsb25nIGh1Z2VwdGVfYWRkcl9lbmQodW5zaWduZWQgbG9uZyBhZGRyLCB1bnNp Z25lZCBsb25nIGVuZCwKIAkJCQkgICAgICB1bnNpZ25lZCBsb25nIHN6KQpAQCAtMTk5OCwzNCAr MjAyNywxOSBAQCBzdGF0aWMgaW50IGd1cF9odWdlcHRlKHB0ZV90ICpwdGVwLCB1bnNpZ25lZCBs b25nIHN6LCB1bnNpZ25lZCBsb25nIGFkZHIsCiAJLyogaHVnZXBhZ2VzIGFyZSBuZXZlciAic3Bl Y2lhbCIgKi8KIAlWTV9CVUdfT04oIXBmbl92YWxpZChwdGVfcGZuKHB0ZSkpKTsKIAotCXJlZnMg PSAwOwogCWhlYWQgPSBwdGVfcGFnZShwdGUpOwotCiAJcGFnZSA9IGhlYWQgKyAoKGFkZHIgJiAo c3otMSkpID4+IFBBR0VfU0hJRlQpOwotCWRvIHsKLQkJVk1fQlVHX09OKGNvbXBvdW5kX2hlYWQo cGFnZSkgIT0gaGVhZCk7Ci0JCXBhZ2VzWypucl0gPSBwYWdlOwotCQkoKm5yKSsrOwotCQlwYWdl Kys7Ci0JCXJlZnMrKzsKLQl9IHdoaWxlIChhZGRyICs9IFBBR0VfU0laRSwgYWRkciAhPSBlbmQp OworCXJlZnMgPSBfX3JlY29yZF9zdWJwYWdlcyhwYWdlLCBhZGRyLCBlbmQsIHBhZ2VzLCAqbnIp OwogCiAJaGVhZCA9IHRyeV9nZXRfY29tcG91bmRfaGVhZChoZWFkLCByZWZzKTsKLQlpZiAoIWhl YWQpIHsKLQkJKm5yIC09IHJlZnM7CisJaWYgKCFoZWFkKQogCQlyZXR1cm4gMDsKLQl9CiAKIAlp ZiAodW5saWtlbHkocHRlX3ZhbChwdGUpICE9IHB0ZV92YWwoKnB0ZXApKSkgewotCQkvKiBDb3Vs ZCBiZSBvcHRpbWl6ZWQgYmV0dGVyICovCi0JCSpuciAtPSByZWZzOwotCQl3aGlsZSAocmVmcy0t KQotCQkJcHV0X3BhZ2UoaGVhZCk7CisJCV9fcmVtb3ZlX3JlZnNfZnJvbV9oZWFkKGhlYWQsIHJl ZnMpOwogCQlyZXR1cm4gMDsKIAl9Ci0KLQlTZXRQYWdlUmVmZXJlbmNlZChoZWFkKTsKLQlyZXR1 cm4gMTsKKwlyZXR1cm4gX19odWdlX3B0X2RvbmUoaGVhZCwgcmVmcywgbnIpOwogfQogCiBzdGF0 aWMgaW50IGd1cF9odWdlX3BkKGh1Z2VwZF90IGh1Z2VwZCwgdW5zaWduZWQgbG9uZyBhZGRyLApA QCAtMjA3MSwzMCArMjA4NSwxOCBAQCBzdGF0aWMgaW50IGd1cF9odWdlX3BtZChwbWRfdCBvcmln LCBwbWRfdCAqcG1kcCwgdW5zaWduZWQgbG9uZyBhZGRyLAogCQkJCQkgICAgIHBhZ2VzLCBucik7 CiAJfQogCi0JcmVmcyA9IDA7CiAJcGFnZSA9IHBtZF9wYWdlKG9yaWcpICsgKChhZGRyICYgflBN RF9NQVNLKSA+PiBQQUdFX1NISUZUKTsKLQlkbyB7Ci0JCXBhZ2VzWypucl0gPSBwYWdlOwotCQko Km5yKSsrOwotCQlwYWdlKys7Ci0JCXJlZnMrKzsKLQl9IHdoaWxlIChhZGRyICs9IFBBR0VfU0la RSwgYWRkciAhPSBlbmQpOworCXJlZnMgPSBfX3JlY29yZF9zdWJwYWdlcyhwYWdlLCBhZGRyLCBl bmQsIHBhZ2VzLCAqbnIpOwogCiAJaGVhZCA9IHRyeV9nZXRfY29tcG91bmRfaGVhZChwbWRfcGFn ZShvcmlnKSwgcmVmcyk7Ci0JaWYgKCFoZWFkKSB7Ci0JCSpuciAtPSByZWZzOworCWlmICghaGVh ZCkKIAkJcmV0dXJuIDA7Ci0JfQogCiAJaWYgKHVubGlrZWx5KHBtZF92YWwob3JpZykgIT0gcG1k X3ZhbCgqcG1kcCkpKSB7Ci0JCSpuciAtPSByZWZzOwotCQl3aGlsZSAocmVmcy0tKQotCQkJcHV0 X3BhZ2UoaGVhZCk7CisJCV9fcmVtb3ZlX3JlZnNfZnJvbV9oZWFkKGhlYWQsIHJlZnMpOwogCQly ZXR1cm4gMDsKIAl9Ci0KLQlTZXRQYWdlUmVmZXJlbmNlZChoZWFkKTsKLQlyZXR1cm4gMTsKKwly ZXR1cm4gX19odWdlX3B0X2RvbmUoaGVhZCwgcmVmcywgbnIpOwogfQogCiBzdGF0aWMgaW50IGd1 cF9odWdlX3B1ZChwdWRfdCBvcmlnLCBwdWRfdCAqcHVkcCwgdW5zaWduZWQgbG9uZyBhZGRyLApA QCAtMjExNCwzMCArMjExNiwxOCBAQCBzdGF0aWMgaW50IGd1cF9odWdlX3B1ZChwdWRfdCBvcmln LCBwdWRfdCAqcHVkcCwgdW5zaWduZWQgbG9uZyBhZGRyLAogCQkJCQkgICAgIHBhZ2VzLCBucik7 CiAJfQogCi0JcmVmcyA9IDA7CiAJcGFnZSA9IHB1ZF9wYWdlKG9yaWcpICsgKChhZGRyICYgflBV RF9NQVNLKSA+PiBQQUdFX1NISUZUKTsKLQlkbyB7Ci0JCXBhZ2VzWypucl0gPSBwYWdlOwotCQko Km5yKSsrOwotCQlwYWdlKys7Ci0JCXJlZnMrKzsKLQl9IHdoaWxlIChhZGRyICs9IFBBR0VfU0la RSwgYWRkciAhPSBlbmQpOworCXJlZnMgPSBfX3JlY29yZF9zdWJwYWdlcyhwYWdlLCBhZGRyLCBl bmQsIHBhZ2VzLCAqbnIpOwogCiAJaGVhZCA9IHRyeV9nZXRfY29tcG91bmRfaGVhZChwdWRfcGFn ZShvcmlnKSwgcmVmcyk7Ci0JaWYgKCFoZWFkKSB7Ci0JCSpuciAtPSByZWZzOworCWlmICghaGVh ZCkKIAkJcmV0dXJuIDA7Ci0JfQogCiAJaWYgKHVubGlrZWx5KHB1ZF92YWwob3JpZykgIT0gcHVk X3ZhbCgqcHVkcCkpKSB7Ci0JCSpuciAtPSByZWZzOwotCQl3aGlsZSAocmVmcy0tKQotCQkJcHV0 X3BhZ2UoaGVhZCk7CisJCV9fcmVtb3ZlX3JlZnNfZnJvbV9oZWFkKGhlYWQsIHJlZnMpOwogCQly ZXR1cm4gMDsKIAl9Ci0KLQlTZXRQYWdlUmVmZXJlbmNlZChoZWFkKTsKLQlyZXR1cm4gMTsKKwly ZXR1cm4gX19odWdlX3B0X2RvbmUoaGVhZCwgcmVmcywgbnIpOwogfQogCiBzdGF0aWMgaW50IGd1 cF9odWdlX3BnZChwZ2RfdCBvcmlnLCBwZ2RfdCAqcGdkcCwgdW5zaWduZWQgbG9uZyBhZGRyLApA QCAtMjE1MSwzMCArMjE0MSwxOSBAQCBzdGF0aWMgaW50IGd1cF9odWdlX3BnZChwZ2RfdCBvcmln LCBwZ2RfdCAqcGdkcCwgdW5zaWduZWQgbG9uZyBhZGRyLAogCQlyZXR1cm4gMDsKIAogCUJVSUxE X0JVR19PTihwZ2RfZGV2bWFwKG9yaWcpKTsKLQlyZWZzID0gMDsKKwogCXBhZ2UgPSBwZ2RfcGFn ZShvcmlnKSArICgoYWRkciAmIH5QR0RJUl9NQVNLKSA+PiBQQUdFX1NISUZUKTsKLQlkbyB7Ci0J CXBhZ2VzWypucl0gPSBwYWdlOwotCQkoKm5yKSsrOwotCQlwYWdlKys7Ci0JCXJlZnMrKzsKLQl9 IHdoaWxlIChhZGRyICs9IFBBR0VfU0laRSwgYWRkciAhPSBlbmQpOworCXJlZnMgPSBfX3JlY29y ZF9zdWJwYWdlcyhwYWdlLCBhZGRyLCBlbmQsIHBhZ2VzLCAqbnIpOwogCiAJaGVhZCA9IHRyeV9n ZXRfY29tcG91bmRfaGVhZChwZ2RfcGFnZShvcmlnKSwgcmVmcyk7Ci0JaWYgKCFoZWFkKSB7Ci0J CSpuciAtPSByZWZzOworCWlmICghaGVhZCkKIAkJcmV0dXJuIDA7Ci0JfQogCiAJaWYgKHVubGlr ZWx5KHBnZF92YWwob3JpZykgIT0gcGdkX3ZhbCgqcGdkcCkpKSB7Ci0JCSpuciAtPSByZWZzOwot CQl3aGlsZSAocmVmcy0tKQotCQkJcHV0X3BhZ2UoaGVhZCk7CisJCV9fcmVtb3ZlX3JlZnNfZnJv bV9oZWFkKGhlYWQsIHJlZnMpOwogCQlyZXR1cm4gMDsKIAl9Ci0KLQlTZXRQYWdlUmVmZXJlbmNl ZChoZWFkKTsKLQlyZXR1cm4gMTsKKwlyZXR1cm4gX19odWdlX3B0X2RvbmUoaGVhZCwgcmVmcywg bnIpOwogfQogCiBzdGF0aWMgaW50IGd1cF9wbWRfcmFuZ2UocHVkX3QgcHVkLCB1bnNpZ25lZCBs b25nIGFkZHIsIHVuc2lnbmVkIGxvbmcgZW5kLAotLSAKMi4yMy4wCgpfX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpkcmktZGV2ZWwgbWFpbGluZyBsaXN0CmRy aS1kZXZlbEBsaXN0cy5mcmVlZGVza3RvcC5vcmcKaHR0cHM6Ly9saXN0cy5mcmVlZGVza3RvcC5v cmcvbWFpbG1hbi9saXN0aW5mby9kcmktZGV2ZWw=