From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 79CC6C6FD1D for ; Fri, 17 Mar 2023 10:58:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B97296B0082; Fri, 17 Mar 2023 06:58:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B476A6B0081; Fri, 17 Mar 2023 06:58:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A0F026B0082; Fri, 17 Mar 2023 06:58:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 92CF96B0080 for ; Fri, 17 Mar 2023 06:58:28 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 56CCD1216D0 for ; Fri, 17 Mar 2023 10:58:28 +0000 (UTC) X-FDA: 80578091496.25.35D3F31 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf28.hostedemail.com (Postfix) with ESMTP id B5C53C0004 for ; Fri, 17 Mar 2023 10:58:26 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf28.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679050706; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kELGhbvkyhE1A6aVwXMfE/mNgpiFCIycE4agf1r/kQg=; b=FZQ3er2IrsLJOxbHfowOacoaWGjdvXTLTi70ID3B72ePTo1qdMAAzXWt14fpXEIXEjWYAm kPFCLKMK9/l99qbzvmBrfTzzt9ZnQ3CyNuc6XAolmb4vnydB40PQw7hxJkcFnhYcA144yj MCuKVIVqSFDZBxoq5FO1oAx5db0Mtlk= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf28.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679050706; a=rsa-sha256; cv=none; b=i2KWNFxglNrvYyteXw7XwYZn+tUxJIUcL7CdVetnjt41OYc65YNj2FC3X0N4ZybXKr7mx9 LmmG0Z+XzvnqX8UfnWJfzJCVXQNeJWhcV4SO/jF5AnCcn7uOEAarrzENoIu8hUGhrzHgwG CYF29VpZKa7QtqI2lxLJkHis/JfG/No= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 948AD1A00; Fri, 17 Mar 2023 03:59:09 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id CE8093F885; Fri, 17 Mar 2023 03:58:24 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , "Matthew Wilcox (Oracle)" , "Yin, Fengwei" , Yu Zhao Cc: Ryan Roberts , linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org Subject: [RFC PATCH 4/6] mm: Implement folio_add_new_anon_rmap_range() Date: Fri, 17 Mar 2023 10:58:00 +0000 Message-Id: <20230317105802.2634004-5-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230317105802.2634004-1-ryan.roberts@arm.com> References: <20230317105802.2634004-1-ryan.roberts@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: B5C53C0004 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: tahwrbmok53chbreu9u5ns691twag5x5 X-HE-Tag: 1679050706-793486 X-HE-Meta: U2FsdGVkX1+HOmH/mg3GQ0DFaqm4ExGTKBsfgzoROMMTxM3coHlIQ+0viowTFe2ppii/XTTBdFIJtRwn/aRPRqFd99Hu/SZ2XJdGSv8fuZSsjeDtHu90XEknJ7eWG+OqGcKR7GAhQO/7rbeVRYVjFR3zpw7mn2jX8h6DqrgvbzJJhIAORtIRSiUDOQi0u4qr0RJl4kkb3bppCm49uEy5GYiQPC5m7V+46vEO0GLMbta5T5xchxc2XceC8wm8ujPAjl9GQsPyfWls7nUOKTppEa70RD1GqlktHb2mT61xKhqjYbWp1y5ck19nmk2PUjVaoO/QqhUVpC2IaW9yj0Jpg94VzAr7fDMuU5y6V0qRF4MqVAkHjlG0wKAPPILbTUrycZU3ZFRQfCwOhCRtx0IdIod5IeZK8tk3T8IjilheLk+WtNtdMO+MMGs3i3akqnzF0QobUaQ2xZc9LeDA5mN8GflKY4PBFFolKhmdfu3BHjXpYoYejOleWl6cs1H1POdKZ9hmasXzySYY8gD/1F48nH83w7NKRBvWZeyoAbzZifVZpwZuVeFPxe/LuXwnEMkKc52dyw/hkJzUV8RWm0YSFKx9Q6G+mKsuD7iCULA2/5VvwVj8PwNx6GerVTmSRo9WIlGM2ZvYczzMp3G3ng4EK1AEcx8RI0/anlA5k99nK6ozzAKa/4SgsuBnFT5xseU2JN9o/odgRIMsECDmuOg1NBRzIHLGbngUA2x2qE1NZq/foJZTLHYUBlC0Uya3Bfm8w75SxWRs+JP2Br8XeF1CGxvL4UMpAny/PN9wFt2mdGe59HHF7hK52DFLTx2Oet3kcEW0Q/ZxaRi5aYFWM/Ii1WgxltYlenujk1TPgHpcrM0Q7cNm03jrJL/s0kQhZ7hFhT+V8DRvFiNH4Q2pFr2Yk720QLoS4Dy0GjvILveEDUcDlW6VT18NocwFVKFPJHDgwwYIykbzGeIylB4uFBP NQ6NJQzi DmUvjVVIB92Glgb9mm1USWe+V5BmTocDkjRxJiLoSkD2h+YqIBgPuooY+8TrfOvOxOuXsAbG27W55GubH4rX+cflz9nwYSB/irtFfLpxf7Igv4y2KaSL/g7UKg64UcZ748/fesBTHSZZAjUmYrUBCQOY7Yjpd1KiV3lTYCUQTEiuy0FcMPyz7IEzFscnKk+fXK9KP2ZnZpHgE5r9kzgLKhxCECVMcQL7iW45C X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Like folio_add_new_anon_rmap() but batch-rmaps all the pages belonging to a folio, for effciency savings. Signed-off-by: Ryan Roberts --- include/linux/rmap.h | 2 ++ mm/rmap.c | 43 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 45 insertions(+) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index b87d01660412..d1d731650ce8 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -196,6 +196,8 @@ void page_add_new_anon_rmap(struct page *, struct vm_area_struct *, unsigned long address); void folio_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address); +void folio_add_new_anon_rmap_range(struct folio *folio, + struct vm_area_struct *vma, unsigned long address); void page_add_file_rmap(struct page *, struct vm_area_struct *, bool compound); void page_remove_rmap(struct page *, struct vm_area_struct *, diff --git a/mm/rmap.c b/mm/rmap.c index 8632e02661ac..05a0c0a700e7 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1302,6 +1302,49 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, __page_set_anon_rmap(folio, &folio->page, vma, address, 1); } +/** + * folio_add_new_anon_rmap_range - Add mapping to a new anonymous potentially + * large but definitely non-THP folio. + * @folio: The folio to add the mapping to. + * @vma: the vm area in which the mapping is added + * @address: the user virtual address of the first page in the folio + * + * Like folio_add_new_anon_rmap() but must only be called for new *non-THP* + * folios. Like folio_add_new_anon_rmap(), the inc-and-test is bypassed and the + * folio does not have to be locked. All pages in the folio are individually + * accounted. + * + * As the folio is new, it's assumed to be mapped exclusively by a single + * process. + */ +void folio_add_new_anon_rmap_range(struct folio *folio, + struct vm_area_struct *vma, unsigned long address) +{ + int i; + int nr = folio_nr_pages(folio); + struct page *page = &folio->page; + + VM_BUG_ON_VMA(address < vma->vm_start || + address + (nr << PAGE_SHIFT) > vma->vm_end, vma); + __folio_set_swapbacked(folio); + + if (folio_test_large(folio)) { + /* increment count (starts at 0) */ + atomic_set(&folio->_nr_pages_mapped, nr); + } + + for (i = 0; i < nr; i++) { + /* increment count (starts at -1) */ + atomic_set(&page->_mapcount, 0); + __page_set_anon_rmap(folio, page, vma, address, 1); + page++; + address += PAGE_SIZE; + } + + __lruvec_stat_mod_folio(folio, NR_ANON_MAPPED, nr); + +} + /** * page_add_file_rmap - add pte mapping to a file page * @page: the page to add the mapping to -- 2.25.1 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A71E3C6FD1D for ; Fri, 17 Mar 2023 10:59:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Ou7kTmivIRk/DkmgBh2IdSud+cZ7x8BOfhcdjKdAxGE=; b=h/th3iZGBXoQ8v MfdFYhvAO+F067WbMhRBUdLAv35YLnJEgWAS30JT5WoI2NP5JqgLpGeIFXSWb7U0W2Hc9Bc/nIu81 Uq7RiJ/31hcgNzrHWoCO97vRVpQlHsOns2uUWL/zQBrNSDpyJI57KKjHlVOhk8wHER4t4ur5m7nno vsJhBREVvuV+Y72xvJL0/J1LP1VmPJCpaJpl15rhL7mpiO+DjhKuSBD91j4WTer/6ocRxqAdfIu1W a7fc/fMJCPHjiOmzUdBJd3YzvJxxAW9+YpCX6z/EHkB2jISioU63LOOrnllV07omp5SVa06dK1lgM c5YBYvmWg2c7EPePke1w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1pd7nB-001yBg-2E; Fri, 17 Mar 2023 10:58:37 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1pd7n1-001y5D-33 for linux-arm-kernel@lists.infradead.org; Fri, 17 Mar 2023 10:58:31 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 948AD1A00; Fri, 17 Mar 2023 03:59:09 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id CE8093F885; Fri, 17 Mar 2023 03:58:24 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , "Matthew Wilcox (Oracle)" , "Yin, Fengwei" , Yu Zhao Cc: Ryan Roberts , linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org Subject: [RFC PATCH 4/6] mm: Implement folio_add_new_anon_rmap_range() Date: Fri, 17 Mar 2023 10:58:00 +0000 Message-Id: <20230317105802.2634004-5-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230317105802.2634004-1-ryan.roberts@arm.com> References: <20230317105802.2634004-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230317_035828_127482_4A7A2EAB X-CRM114-Status: GOOD ( 13.93 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Like folio_add_new_anon_rmap() but batch-rmaps all the pages belonging to a folio, for effciency savings. Signed-off-by: Ryan Roberts --- include/linux/rmap.h | 2 ++ mm/rmap.c | 43 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 45 insertions(+) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index b87d01660412..d1d731650ce8 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -196,6 +196,8 @@ void page_add_new_anon_rmap(struct page *, struct vm_area_struct *, unsigned long address); void folio_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address); +void folio_add_new_anon_rmap_range(struct folio *folio, + struct vm_area_struct *vma, unsigned long address); void page_add_file_rmap(struct page *, struct vm_area_struct *, bool compound); void page_remove_rmap(struct page *, struct vm_area_struct *, diff --git a/mm/rmap.c b/mm/rmap.c index 8632e02661ac..05a0c0a700e7 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1302,6 +1302,49 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, __page_set_anon_rmap(folio, &folio->page, vma, address, 1); } +/** + * folio_add_new_anon_rmap_range - Add mapping to a new anonymous potentially + * large but definitely non-THP folio. + * @folio: The folio to add the mapping to. + * @vma: the vm area in which the mapping is added + * @address: the user virtual address of the first page in the folio + * + * Like folio_add_new_anon_rmap() but must only be called for new *non-THP* + * folios. Like folio_add_new_anon_rmap(), the inc-and-test is bypassed and the + * folio does not have to be locked. All pages in the folio are individually + * accounted. + * + * As the folio is new, it's assumed to be mapped exclusively by a single + * process. + */ +void folio_add_new_anon_rmap_range(struct folio *folio, + struct vm_area_struct *vma, unsigned long address) +{ + int i; + int nr = folio_nr_pages(folio); + struct page *page = &folio->page; + + VM_BUG_ON_VMA(address < vma->vm_start || + address + (nr << PAGE_SHIFT) > vma->vm_end, vma); + __folio_set_swapbacked(folio); + + if (folio_test_large(folio)) { + /* increment count (starts at 0) */ + atomic_set(&folio->_nr_pages_mapped, nr); + } + + for (i = 0; i < nr; i++) { + /* increment count (starts at -1) */ + atomic_set(&page->_mapcount, 0); + __page_set_anon_rmap(folio, page, vma, address, 1); + page++; + address += PAGE_SIZE; + } + + __lruvec_stat_mod_folio(folio, NR_ANON_MAPPED, nr); + +} + /** * page_add_file_rmap - add pte mapping to a file page * @page: the page to add the mapping to -- 2.25.1 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel