From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.3 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS, USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 235C8C47253 for ; Fri, 1 May 2020 08:59:10 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A23F9208C3 for ; Fri, 1 May 2020 08:59:09 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=oneplus.com header.i=@oneplus.com header.b="keeCkRIx" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A23F9208C3 Authentication-Results: mail.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=oneplus.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1E6588E000E; Fri, 1 May 2020 04:59:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 196648E0001; Fri, 1 May 2020 04:59:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 038B38E000E; Fri, 1 May 2020 04:59:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0224.hostedemail.com [216.40.44.224]) by kanga.kvack.org (Postfix) with ESMTP id DCEA78E0001 for ; Fri, 1 May 2020 04:59:07 -0400 (EDT) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 8A7C7181AC9B6 for ; Fri, 1 May 2020 08:59:07 +0000 (UTC) X-FDA: 76767550734.14.net58_4f82f964d8e0e X-HE-Tag: net58_4f82f964d8e0e X-Filterd-Recvd-Size: 24537 Received: from APC01-SG2-obe.outbound.protection.outlook.com (mail-eopbgr1310099.outbound.protection.outlook.com [40.107.131.99]) by imf11.hostedemail.com (Postfix) with ESMTP for ; Fri, 1 May 2020 08:59:06 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=KyAFkMZHZkpowQsdShEevqFw97NO9M435+I6G7yRj3pgHx56x4gbLVwX/pJWRwmP3vwexI10k4hEC86x+iVDlZz2tg//q5U4A8nW6VXhgd/kCIVoLUdPnYrG9pNX3ioHFlq976/f0zHMoyiA4gDTOBrCYuhEBM1aq5lWvhrF+hwB5OL+7ScFPmQcNtlNhhkkfnRPPNmVKjb3zR+E+LC0OxiiBK2kc/X19LxWh6J5lxv7D8vqU/JlxEEGj0HDzxS4O78jCC0hucs3mP49c4e+4+A8qQqt94W5FNuSumsvtG0v3nEu7k6ZOTOdD1Zo7mO8XcWxwK0XWxERGMt5X/CwOw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vzIIokh2412PoE6C9FZTxno4lBbsLBaaWgMMopiB5k8=; b=b0pL0wQnU1a9usyBZrbzD8WJkti0RccCTDstHPivZBlB66miFmLNTze8U4DCln3YLmMqaOcDTZm9qKGBMgNQOX3nxn9QNIWwkPDtdehgjZFEgvIYhSpV2pPQXvB82zHxY40eCGhuK9cnUrX6fL8oJdgqXLzrm/ZRUK6g+L96jQmf8RDZPeiLAUTGkZ5fHGkrs7vMS0UZLsX+F4WafPV1WkuUVyBxnEBdWnOI2zfZzs/fLnaGjqJuJQF33Ic1Pjw3aG15SQ11Y9izCgqCG3eTUI8Fm1LN+TbflgjMR5M7tnpcrl4pBCZi7RGA+/jfZvxonkHsHsn8WNMk7GeCWPxeSQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oneplus.com; dmarc=pass action=none header.from=oneplus.com; dkim=pass header.d=oneplus.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oneplus.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vzIIokh2412PoE6C9FZTxno4lBbsLBaaWgMMopiB5k8=; b=keeCkRIxI1cY5Frje6HxKFvWFrqj9YqHUwP37zPna3hnIwrBg1AFpO+9Dd0mkdJKDQ5HFBgjuibVQIGDDZ+FR7JrwDl7x/LlCgbSViw6+MXbCiwgox8SDAHWTgYqyKb0yXS+37tZQgLzlpuVBMjPdXWGUUZ8UFmbX1VKhOVJ1OA= Authentication-Results: kernel.org; dkim=none (message not signed) header.d=none;kernel.org; dmarc=none action=none header.from=oneplus.com; Received: from SG2PR04MB3093.apcprd04.prod.outlook.com (2603:1096:4:6d::11) by SG2PR04MB3446.apcprd04.prod.outlook.com (2603:1096:4:6d::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2958.19; Fri, 1 May 2020 08:59:01 +0000 Received: from SG2PR04MB3093.apcprd04.prod.outlook.com ([fe80::8c32:9fdd:f7e:163b]) by SG2PR04MB3093.apcprd04.prod.outlook.com ([fe80::8c32:9fdd:f7e:163b%7]) with mapi id 15.20.2937.028; Fri, 1 May 2020 08:59:01 +0000 Date: Fri, 1 May 2020 14:28:55 +0530 From: Prathu Baronia To: Will Deacon Cc: Vlastimil Babka , catalin.marinas@arm.com, alexander.duyck@gmail.com, chintan.pandya@oneplus.com, mhocko@suse.com, akpm@linux-foundation.org, linux-mm@kvack.org, gregkh@linuxfoundation.com, gthelen@google.com, jack@suse.cz, ken.lin@oneplus.com, gasine.xu@oneplus.com, ying.huang@intel.com, mark.rutland@arm.com Subject: Re: [PATCH v2] mm: Optimized hugepage zeroing & copying from user Message-ID: <20200501085855.c5dzk5hfrdzunqdl@oneplus.com> References: <87k12bt3ff.fsf@yhuang-dev.intel.com> <20200421093621.3fuptvf2qbyfzwfz@oneplus.com> <20200421100932.GC17256@willie-the-truck> <02d5daa8-ee7b-7d2d-6753-5191a7d761b9@suse.cz> <20200421133935.GC17875@willie-the-truck> <5e334947-22e9-e59d-f7bb-63e04cc8caf0@suse.cz> <20200422081852.GB29541@willie-the-truck> <20200422111928.GA32051@willie-the-truck> <20200422143841.ozuow4jkltzymvgs@oneplus.com> Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20200422143841.ozuow4jkltzymvgs@oneplus.com> User-Agent: NeoMutt/20171215 X-ClientProxiedBy: HK0PR03CA0113.apcprd03.prod.outlook.com (2603:1096:203:b0::29) To SG2PR04MB3093.apcprd04.prod.outlook.com (2603:1096:4:6d::11) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from oneplus.com (183.83.136.195) by HK0PR03CA0113.apcprd03.prod.outlook.com (2603:1096:203:b0::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2958.20 via Frontend Transport; Fri, 1 May 2020 08:58:58 +0000 X-Originating-IP: [183.83.136.195] X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: d798f3bc-28f9-482d-d418-08d7edade411 X-MS-TrafficTypeDiagnostic: SG2PR04MB3446:|SG2PR04MB3446: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:3044; X-Forefront-PRVS: 0390DB4BDA X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: BWeKhVQEtntya/8IJ65tL8W7mwBLfPZ+qTXmXqCX6JjwETUMDPbdM4jxCaJSQv1ktdToSsZ92f7yG4y8OO4ykSaLc2McUF5Lzz8KeRogur6g+IyX9E1jHJUt7L6/BIT6gn2ZnyHI3E0Ck3O7RiHPG89s2s9SnQLVixLv2fIoYNTVEOnMK/6BZHuCDt4IwCRe5jERwq2/Z/ROyYB2WrVC7w6QG2oM/pyfZJFLX9vfThrGNK9GHKPVQT189MQKX7KU+WqDDFMFHkKqZlICOWQY5aQrKRFNZSBoqJdKYo+uEXT3JY20gsicHB5WxKdHb6UUvk0GCKG6uIs3EH2Gmubs4DoZoK2emBXW9tFRDh/vKebOulylWBsSG6559bzCnX+Rl5740ZAWjWPa5D1zRARsWkzIB11V4v3ngnkBdvv2zCQcE/ILD5Vo8qVo+56CVWav X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SG2PR04MB3093.apcprd04.prod.outlook.com;PTR:;CAT:NONE;SFTY:;SFS:(4636009)(346002)(376002)(366004)(136003)(39860400002)(396003)(7696005)(6916009)(26005)(30864003)(1076003)(6666004)(4326008)(478600001)(55016002)(316002)(7416002)(956004)(44832011)(2616005)(53546011)(66556008)(66946007)(186003)(52116002)(16526019)(66476007)(5660300002)(36756003)(86362001)(8936002)(8676002)(8886007)(2906002);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData: KueOsWnO9xme1YLi4RT3mBNiwSLboPIsIInR7Okome6OMxJ0biiOhAm62Pn6l3YrqBYYrik/2WYVuH7lZfnsR/EPqg113hVlGExKL9F8s/YSWygrBZ4fzDHsSocrQKdWiY09h341qtuGYHrZz2fLUskNiOVo9eukvrovbC1uaJHrxdSw9186wOs8CLH7IpCS3cx5GQitmc4PrkB8YVbgsJ3mfGX9ABei4oOZ867LHpGD+5e1a8kUauLJdukQ+L8eNbrjPl2uKnYHTvYEraQdv2a0Nv104NpPKcCh83Z791y9mxMYwGOFof99B3Ft2noR1McFGEVr8Es8IcMLDCC2CfcIEMx/kkl9Ew75JOq0ka8r91BQ28JQHhGzga8tzkzM4dPqPzJb92kfo54GmISThXptl5z4N2NpxJPl/qsXhLoVB/04j/CAI5VN4h2FfHymRI1seulyPZ/qoXmEoUdh8kHljYMTF0mJW57ZKDOLHMKTYQNfdr3672xSIbOtIYisI1RQ4qLhGhkmvRTPoHBBVa+g9LaVdqjlModf4ClviOACZZvW2+GKJOSU42XdzIzb2QApxST6eRO+uhK5i7g4mXUwd03a8Yly3ke4+jAqSyncmtb9BzNmd2Yh2iMxFvWnljTJsG9vWW2MC55hRu2d+VxbbICzZ4kuJS6ayDK3u1D+dwWRAAfrUX9HhbnQyeHBIucWisQrTANGIBn0mCvzYvxmyFE15GS6tUPDRJzbpXKJ3CngqiqgPTZwN1U++w6JSWkqy2qgIu90XV0HGrOTK36ynMEZIqHtuCV5zbgIVGw= X-OriginatorOrg: oneplus.com X-MS-Exchange-CrossTenant-Network-Message-Id: d798f3bc-28f9-482d-d418-08d7edade411 X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 May 2020 08:59:01.1561 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 0423909d-296c-463e-ab5c-e5853a518df8 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: AtWQPx/9RrLqVeo/OuH0iVeoo4qKwXSJ9yakHVbQzB6xLRMCvOre7ol7T51jNsT3iQbqQkL5CGqHVE8qx/merwgNf9GF7IyXAgzjoa26McU= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SG2PR04MB3446 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Platform and setup conditions: Qualcomm's SM8150 platform under controlled conditions(i.e. only CPU0 and 6 turned on and set to max frequency, and DDR set to performance governor). --------------------------------------------------------------------------- --------------------------------------------------------------------------- Summary: We observed a ~61% improvement in executon time of clearing a hugepage in the case of arm64 if we increase the granularity i.e. the chunk size to 64KB from 4KB for each chunk clearing subroutine call. --------------------------------------------------------------------------- For the base build: clear_huge_page() ftrace profile -------------------------------- - CPU0: - Samples: 95 - Mean: 242.099 us - Std dev: 45.0096 us - CPU6: - Samples: 61 - Mean: 258.372 us - Std dev: 22.0754 us With patches [PATCH {1,2,3}/4] provided at the end where we just revert the forward-reverse traversal code we observed: clear_huge_page() ftrace profile -------------------------------- - CPU0: - Samples: 77 - Mean: 234.568 - Std dev: 6.52 - CPU6: - Samples: 81 - Mean: 259.437 - Std dev: 19.25 We were expecting a bit of an improvement for arm64's case because of our hypothesis that reverse traversal is considerably slower in arm64 but after Will Deacon's test code which showed similar timings for forward and reverse traversals we digged a bit deeper into this. I found that In the case of arm64 a page is cleared using a special clear_page.S assembly routine instead of an explicit call to memset. With the below patch we bypassed the assembly routine and oberserved improvement in execution time of clear_huge_page on CPU0. diff --git a/include/linux/highmem.h b/include/linux/highmem.h index ea5cdbd8c2c3..a0a97a95aee8 100644 --- a/include/linux/highmem.h +++ b/include/linux/highmem.h @@ -158,7 +158,7 @@ do { \ static inline void clear_user_highpage(struct page *page, unsigned long vaddr) { void *addr = kmap_atomic(page); - clear_user_page(addr, vaddr, page); + memset(addr, 0x0, PAGE_SIZE); kunmap_atomic(addr); } #endif For reference I will call the above patch v-exp. When v-exp is applied on base we observed: clear_huge_page() ftrace profile -------------------------------- - CPU0: - Samples: 71 - Mean: 124.657 us - Std dev: 0.494165 us - CPU6: - Samples: 60 - Mean: 252.153 us - Std dev: 17.7591 us When v-exp is applied on base along with [PATCH {1,2,3}/4] we observed: clear_huge_page() ftrace profile -------------------------------- - CPU0: - Samples: 103 - Mean: 127.034 - Std dev: 25.0312 - CPU6: - Samples: 80 - Mean: 250.88 - Std dev: 18.85 These timings for CPU0 made sense since according to Will Deacon's test code for 5GB the time was ~330us for our platform. So for 2 MB the time should be: 330*2/(5*1024) = 0.12890625 ms ~ 129 us. But the above tmings seemed somewhat unjustified for CPU6. As suggested earlier by Ying Huang to vary the chunk sizes for zeroing the hugepage, we observe ~61% improvement in the execution time for CPU6 and slight improvement in CPU0 when we vary the chunk_size from 4KB to 64KB with all the below patches i.e. [PATCH {1,2,3,4}/4]] ================ Results ================ Chunk Size = 4KB ---------------- CPU0: Samples: 103 Mean: 127.034 Std dev: 25.0312 CPU6: Samples: 80 Mean: 250.88 Std dev: 18.85 Chunk Size = 8KB ----------------- CPU0 Samples : 68 Mean : 120.627 Stddev : 2.12056 CPU6 Samples : 68 Mean : 241.897 Stddev : 24.8305 Chunk Size = 16KB ----------------- CPU0 Samples : 75 Mean : 121.632 Stddev : 44.83 CPU6 Samples : 75 Mean : 166.851 Stddev : 10.6432 Chunk Size = 32KB ----------------- CPU0 Samples : 68 Mean : 114.552 Stddev : 0.302009 CPU6 Samples : 68 Mean : 120.836 Stddev : 6.52462 Chunk Size = 64KB ----------------- CPU0 Samples : 70 Mean : 113.639 Stddev : 0.228294 CPU6 Samples : 70 Mean : 97.96 Stddev : 5.33412 ============================================================================= >From 2c0e618953316bd532cd3c0545e168e930515d42 Mon Sep 17 00:00:00 2001 From: Prathu Baronia Date: Tue, 21 Apr 2020 19:01:01 +0530 Subject: [PATCH 1/4] Revert "mm, clear_huge_page: move order algorithm into a separate function" This reverts commit c6ddfb6c58903262d2d77042c41dba58cf775d88. --- mm/memory.c | 90 ++++++++++++++++++++--------------------------------- 1 file changed, 34 insertions(+), 56 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index e8bfdf0d9d1d..e83fa2d01e1a 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4655,93 +4655,71 @@ EXPORT_SYMBOL(__might_fault); #endif #if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS) -/* - * Process all subpages of the specified huge page with the specified - * operation. The target subpage will be processed last to keep its - * cache lines hot. - */ -static inline void process_huge_page( - unsigned long addr_hint, unsigned int pages_per_huge_page, - void (*process_subpage)(unsigned long addr, int idx, void *arg), - void *arg) +static void clear_gigantic_page(struct page *page, + unsigned long addr, + unsigned int pages_per_huge_page) +{ + int i; + struct page *p = page; + + might_sleep(); + for (i = 0; i < pages_per_huge_page; + i++, p = mem_map_next(p, page, i)) { + cond_resched(); + clear_user_highpage(p, addr + i * PAGE_SIZE); + } +} +void clear_huge_page(struct page *page, + unsigned long addr_hint, unsigned int pages_per_huge_page) { int i, n, base, l; unsigned long addr = addr_hint & ~(((unsigned long)pages_per_huge_page << PAGE_SHIFT) - 1); - /* Process target subpage last to keep its cache lines hot */ + if (unlikely(pages_per_huge_page > MAX_ORDER_NR_PAGES)) { + clear_gigantic_page(page, addr, pages_per_huge_page); + return; + } + + /* Clear sub-page to access last to keep its cache lines hot */ might_sleep(); n = (addr_hint - addr) / PAGE_SIZE; if (2 * n <= pages_per_huge_page) { - /* If target subpage in first half of huge page */ + /* If sub-page to access in first half of huge page */ base = 0; l = n; - /* Process subpages at the end of huge page */ + /* Clear sub-pages at the end of huge page */ for (i = pages_per_huge_page - 1; i >= 2 * n; i--) { cond_resched(); - process_subpage(addr + i * PAGE_SIZE, i, arg); + clear_user_highpage(page + i, addr + i * PAGE_SIZE); } } else { - /* If target subpage in second half of huge page */ + /* If sub-page to access in second half of huge page */ base = pages_per_huge_page - 2 * (pages_per_huge_page - n); l = pages_per_huge_page - n; - /* Process subpages at the begin of huge page */ + /* Clear sub-pages at the begin of huge page */ for (i = 0; i < base; i++) { cond_resched(); - process_subpage(addr + i * PAGE_SIZE, i, arg); + clear_user_highpage(page + i, addr + i * PAGE_SIZE); } } /* - * Process remaining subpages in left-right-left-right pattern - * towards the target subpage + * Clear remaining sub-pages in left-right-left-right pattern + * towards the sub-page to access */ for (i = 0; i < l; i++) { int left_idx = base + i; int right_idx = base + 2 * l - 1 - i; cond_resched(); - process_subpage(addr + left_idx * PAGE_SIZE, left_idx, arg); + clear_user_highpage(page + left_idx, + addr + left_idx * PAGE_SIZE); cond_resched(); - process_subpage(addr + right_idx * PAGE_SIZE, right_idx, arg); + clear_user_highpage(page + right_idx, + addr + right_idx * PAGE_SIZE); } } -static void clear_gigantic_page(struct page *page, - unsigned long addr, - unsigned int pages_per_huge_page) -{ - int i; - struct page *p = page; - - might_sleep(); - for (i = 0; i < pages_per_huge_page; - i++, p = mem_map_next(p, page, i)) { - cond_resched(); - clear_user_highpage(p, addr + i * PAGE_SIZE); - } -} - -static void clear_subpage(unsigned long addr, int idx, void *arg) -{ - struct page *page = arg; - - clear_user_highpage(page + idx, addr); -} - -void clear_huge_page(struct page *page, - unsigned long addr_hint, unsigned int pages_per_huge_page) -{ - unsigned long addr = addr_hint & - ~(((unsigned long)pages_per_huge_page << PAGE_SHIFT) - 1); - - if (unlikely(pages_per_huge_page > MAX_ORDER_NR_PAGES)) { - clear_gigantic_page(page, addr, pages_per_huge_page); - return; - } - - process_huge_page(addr_hint, pages_per_huge_page, clear_subpage, page); -} - static void copy_user_gigantic_page(struct page *dst, struct page *src, unsigned long addr, struct vm_area_struct *vma, -- 2.17.1 >From ac357d010ba25b910c5830dcb8958497ea4b5bfa Mon Sep 17 00:00:00 2001 From: Prathu Baronia Date: Tue, 21 Apr 2020 19:03:24 +0530 Subject: [PATCH 2/4] Revert "mm, huge page: copy target sub-page last when copy huge page" This reverts commit c9f4cd71383576a916e7fca99c490fc92a289f5a. --- include/linux/mm.h | 3 +-- mm/huge_memory.c | 3 +-- mm/memory.c | 30 +++++++----------------------- 3 files changed, 9 insertions(+), 27 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index c54fb96cb1e6..243754937b0f 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2860,8 +2860,7 @@ extern void clear_huge_page(struct page *page, unsigned long addr_hint, unsigned int pages_per_huge_page); extern void copy_user_huge_page(struct page *dst, struct page *src, - unsigned long addr_hint, - struct vm_area_struct *vma, + unsigned long addr, struct vm_area_struct *vma, unsigned int pages_per_huge_page); extern long copy_huge_page_from_user(struct page *dst_page, const void __user *usr_src, diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 24ad53b4dfc0..844764f11e41 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1415,8 +1415,7 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd) if (!page) clear_huge_page(new_page, vmf->address, HPAGE_PMD_NR); else - copy_user_huge_page(new_page, page, vmf->address, - vma, HPAGE_PMD_NR); + copy_user_huge_page(new_page, page, haddr, vma, HPAGE_PMD_NR); __SetPageUptodate(new_page); mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma, vma->vm_mm, diff --git a/mm/memory.c b/mm/memory.c index e83fa2d01e1a..495880c3e945 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4739,31 +4739,11 @@ static void copy_user_gigantic_page(struct page *dst, struct page *src, } } -struct copy_subpage_arg { - struct page *dst; - struct page *src; - struct vm_area_struct *vma; -}; - -static void copy_subpage(unsigned long addr, int idx, void *arg) -{ - struct copy_subpage_arg *copy_arg = arg; - - copy_user_highpage(copy_arg->dst + idx, copy_arg->src + idx, - addr, copy_arg->vma); -} - void copy_user_huge_page(struct page *dst, struct page *src, - unsigned long addr_hint, struct vm_area_struct *vma, + unsigned long addr, struct vm_area_struct *vma, unsigned int pages_per_huge_page) { - unsigned long addr = addr_hint & - ~(((unsigned long)pages_per_huge_page << PAGE_SHIFT) - 1); - struct copy_subpage_arg arg = { - .dst = dst, - .src = src, - .vma = vma, - }; + int i; if (unlikely(pages_per_huge_page > MAX_ORDER_NR_PAGES)) { copy_user_gigantic_page(dst, src, addr, vma, @@ -4771,7 +4751,11 @@ void copy_user_huge_page(struct page *dst, struct page *src, return; } - process_huge_page(addr_hint, pages_per_huge_page, copy_subpage, &arg); + might_sleep(); + for (i = 0; i < pages_per_huge_page; i++) { + cond_resched(); + copy_user_highpage(dst + i, src + i, addr + i*PAGE_SIZE, vma); + } } long copy_huge_page_from_user(struct page *dst_page, -- 2.17.1 >From 14f0266254e4a2ea55f1cf33323205c77bec1a12 Mon Sep 17 00:00:00 2001 From: Prathu Baronia Date: Tue, 21 Apr 2020 19:06:30 +0530 Subject: [PATCH 3/4] Revert "mm: hugetlb: clear target sub-page last when clearing huge page" This reverts commit c79b57e462b5d2f47afa5f175cf1828f16e18612. --- include/linux/mm.h | 2 +- mm/huge_memory.c | 4 ++-- mm/memory.c | 42 ++++-------------------------------------- 3 files changed, 7 insertions(+), 41 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 243754937b0f..0b77926c3705 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2857,7 +2857,7 @@ enum mf_action_page_type { #if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS) extern void clear_huge_page(struct page *page, - unsigned long addr_hint, + unsigned long addr, unsigned int pages_per_huge_page); extern void copy_user_huge_page(struct page *dst, struct page *src, unsigned long addr, struct vm_area_struct *vma, diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 844764f11e41..a2131ee72a37 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -606,7 +606,7 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf, goto release; } - clear_huge_page(page, vmf->address, HPAGE_PMD_NR); + clear_huge_page(page, haddr, HPAGE_PMD_NR); /* * The memory barrier inside __SetPageUptodate makes sure that * clear_huge_page writes become visible before the set_pmd_at() @@ -1413,7 +1413,7 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd) count_memcg_events(memcg, THP_FAULT_ALLOC, 1); if (!page) - clear_huge_page(new_page, vmf->address, HPAGE_PMD_NR); + clear_huge_page(new_page, haddr, HPAGE_PMD_NR); else copy_user_huge_page(new_page, page, haddr, vma, HPAGE_PMD_NR); __SetPageUptodate(new_page); diff --git a/mm/memory.c b/mm/memory.c index 495880c3e945..a297569d7308 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4670,53 +4670,19 @@ static void clear_gigantic_page(struct page *page, } } void clear_huge_page(struct page *page, - unsigned long addr_hint, unsigned int pages_per_huge_page) + unsigned long addr, unsigned int pages_per_huge_page) { - int i, n, base, l; - unsigned long addr = addr_hint & - ~(((unsigned long)pages_per_huge_page << PAGE_SHIFT) - 1); + int i; if (unlikely(pages_per_huge_page > MAX_ORDER_NR_PAGES)) { clear_gigantic_page(page, addr, pages_per_huge_page); return; } - /* Clear sub-page to access last to keep its cache lines hot */ might_sleep(); - n = (addr_hint - addr) / PAGE_SIZE; - if (2 * n <= pages_per_huge_page) { - /* If sub-page to access in first half of huge page */ - base = 0; - l = n; - /* Clear sub-pages at the end of huge page */ - for (i = pages_per_huge_page - 1; i >= 2 * n; i--) { - cond_resched(); - clear_user_highpage(page + i, addr + i * PAGE_SIZE); - } - } else { - /* If sub-page to access in second half of huge page */ - base = pages_per_huge_page - 2 * (pages_per_huge_page - n); - l = pages_per_huge_page - n; - /* Clear sub-pages at the begin of huge page */ - for (i = 0; i < base; i++) { - cond_resched(); - clear_user_highpage(page + i, addr + i * PAGE_SIZE); - } - } - /* - * Clear remaining sub-pages in left-right-left-right pattern - * towards the sub-page to access - */ - for (i = 0; i < l; i++) { - int left_idx = base + i; - int right_idx = base + 2 * l - 1 - i; - - cond_resched(); - clear_user_highpage(page + left_idx, - addr + left_idx * PAGE_SIZE); + for (i = 0; i < pages_per_huge_page; i++) { cond_resched(); - clear_user_highpage(page + right_idx, - addr + right_idx * PAGE_SIZE); + clear_user_highpage(page + i, addr + i * PAGE_SIZE); } } -- 2.17.1 >From 1540b3a6b8732b03a480905b564611a14d09824a Mon Sep 17 00:00:00 2001 From: Prathu Baronia Date: Fri, 1 May 2020 11:50:58 +0530 Subject: [PATCH 4/4] Varied chunk size for clearing out a huge page. Signed-off-by: Prathu Baronia --- mm/memory.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index a297569d7308..b6f5a0cf4d40 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4673,6 +4673,9 @@ void clear_huge_page(struct page *page, unsigned long addr, unsigned int pages_per_huge_page) { int i; + void * addr_chunk; + int chunk_factor = 1; + int chunk_size = PAGE_SIZE*chunk_factor; if (unlikely(pages_per_huge_page > MAX_ORDER_NR_PAGES)) { clear_gigantic_page(page, addr, pages_per_huge_page); @@ -4680,9 +4683,11 @@ void clear_huge_page(struct page *page, } might_sleep(); - for (i = 0; i < pages_per_huge_page; i++) { + for (i = 0; i < pages_per_huge_page; i += chunk_factor) { cond_resched(); - clear_user_highpage(page + i, addr + i * PAGE_SIZE); + addr_chunk = kmap_atomic(page + i); + memset(addr_chunk, 0x0, chunk_size); + kunmap_atomic(addr_chunk); } } -- 2.17.1 -- Prathu Baronia OnePlus RnD