From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0734DC432C0 for ; Sun, 1 Dec 2019 01:57:09 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B0D19205ED for ; Sun, 1 Dec 2019 01:57:08 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="HjD6xP8A" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B0D19205ED Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 61DE46B0367; Sat, 30 Nov 2019 20:57:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5CE2A6B0369; Sat, 30 Nov 2019 20:57:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 50BDB6B036B; Sat, 30 Nov 2019 20:57:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0191.hostedemail.com [216.40.44.191]) by kanga.kvack.org (Postfix) with ESMTP id 385036B0367 for ; Sat, 30 Nov 2019 20:57:08 -0500 (EST) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with SMTP id E3723180AD817 for ; Sun, 1 Dec 2019 01:57:07 +0000 (UTC) X-FDA: 76214909694.05.scene21_3e3abac5c0610 X-HE-Tag: scene21_3e3abac5c0610 X-Filterd-Recvd-Size: 3532 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf42.hostedemail.com (Postfix) with ESMTP for ; Sun, 1 Dec 2019 01:57:07 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 82493215E5; Sun, 1 Dec 2019 01:57:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1575165426; bh=7+wvV3gZq5vFyvYUTM8y6xf9bmqVI1iWgdN4mIOvVH0=; h=Date:From:To:Subject:From; b=HjD6xP8AfTV8V/eyM/XH0vQ/MuZA/QH00B8yBnBMMowMsVtiAGmirJMQJ4zywcTmF CehhpEjTLVpdO3mfztEho7oXeD1HbzkQWEsZeA9ykxiX5G9atVHjEaFF7yiLpm0jSJ 7oshZL35rZtY+iMFeSgbLOBKQ73n/owYC1C1VGPQ= Date: Sat, 30 Nov 2019 17:57:06 -0800 From: akpm@linux-foundation.org To: akpm@linux-foundation.org, hzhongzhang@tencent.com, knightzhang@tencent.com, linux-mm@kvack.org, mike.kravetz@oracle.com, mm-commits@vger.kernel.org, tonnylu@tencent.com, torvalds@linux-foundation.org, willy@infradead.org Subject: [patch 132/158] mm/hugetlb: avoid looping to the same hugepage if !pages and !vmas Message-ID: <20191201015706.RdZ5u8gLH%akpm@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Zhigang Lu Subject: mm/hugetlb: avoid looping to the same hugepage if !pages and !vmas When mmapping an existing hugetlbfs file with MAP_POPULATE, we find it is very time consuming. For example, mmapping a 128GB file takes about 50 milliseconds. Sampling with perfevent shows it spends 99% time in the same_page loop in follow_hugetlb_page(). samples: 205 of event 'cycles', Event count (approx.): 136686374 - 99.04% test_mmap_huget [kernel.kallsyms] [k] follow_hugetlb_page follow_hugetlb_page __get_user_pages __mlock_vma_pages_range __mm_populate vm_mmap_pgoff sys_mmap_pgoff sys_mmap system_call_fastpath __mmap64 follow_hugetlb_page() is called with pages=NULL and vmas=NULL, so for each hugepage, we run into the same_page loop for pages_per_huge_page() times, but doing nothing. With this change, it takes less then 1 millisecond to mmap a 128GB file in hugetlbfs. Link: http://lkml.kernel.org/r/1567581712-5992-1-git-send-email-totty.lu@gmail.com Signed-off-by: Zhigang Lu Reviewed-by: Haozhong Zhang Reviewed-by: Zongming Zhang Reviewed-by: Mike Kravetz Acked-by: Matthew Wilcox Signed-off-by: Andrew Morton --- mm/hugetlb.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+) --- a/mm/hugetlb.c~mm-hugetlb-avoid-looping-to-the-same-hugepage-if-pages-and-vmas +++ a/mm/hugetlb.c @@ -4338,6 +4338,21 @@ long follow_hugetlb_page(struct mm_struc break; } } + + /* + * If subpage information not requested, update counters + * and skip the same_page loop below. + */ + if (!pages && !vmas && !pfn_offset && + (vaddr + huge_page_size(h) < vma->vm_end) && + (remainder >= pages_per_huge_page(h))) { + vaddr += huge_page_size(h); + remainder -= pages_per_huge_page(h); + i += pages_per_huge_page(h); + spin_unlock(ptl); + continue; + } + same_page: if (pages) { pages[i] = mem_map_offset(page, pfn_offset); _