From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AD2ACC43334 for ; Fri, 24 Jun 2022 17:37:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2CE028E0242; Fri, 24 Jun 2022 13:37:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 255F18E020E; Fri, 24 Jun 2022 13:37:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0A8F18E0242; Fri, 24 Jun 2022 13:37:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D1C2D8E020E for ; Fri, 24 Jun 2022 13:37:16 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 9F20B35630 for ; Fri, 24 Jun 2022 17:37:16 +0000 (UTC) X-FDA: 79613835672.09.16A3880 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf17.hostedemail.com (Postfix) with ESMTP id 2BBD740023 for ; Fri, 24 Jun 2022 17:37:16 +0000 (UTC) Received: by mail-yb1-f201.google.com with SMTP id a83-20020a251a56000000b0066c354a19ccso1666131yba.6 for ; Fri, 24 Jun 2022 10:37:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=Prs/g0eAXOv/ZKQzs92554GHZM/jn5SpuUFUCkOXcEo=; b=Px4lWScALFzFiAT+UIjCnXt+kA7DjkP32e6aEOW4CwgMESdfSnOQVe0TMA2O89nVFQ 2FSb4AFEEui36dp7cEO2cCeeW25R3wvs2P0Pw293UvGXo1bFkXyTw3c22GRwbp6ZeroB BKYDs88dfDTkiN+2kUji3gPFSrrfiflHbvNAwBPQkx1aAAnlNFHrCyf3UirmhSLZm9oL nyAorTU4OhiyKwV0sm0qey3GF9yE03pGEq+AkLOcUeZfuKYpmDNlnep1AHlKc5CHBoWs lchQ3wZR304ZcaHQH9yJaAqziuHYgjAFwrM+TJW+0jvW+Bqrs/0OMU3uJ876q5Jq/AM4 BUoQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=Prs/g0eAXOv/ZKQzs92554GHZM/jn5SpuUFUCkOXcEo=; b=qgymrKMrDHcKP3xFF41121ZeaI48nhlAd2YozyMvbdUqTKXi9LuVseGsu8Ug3iZM3J J+trOtCDYcaRjMOuibxFgWuVtL0/2wz7IWBrsEN9MeMLF4jBERsw2EVFTWw/Qzg6VDLn 1twsxx92IgG+GSekvMAbiLyuiT3nvHLUyYUUZ5S2Qo8frqQ+2pbFZNS5Uh0QCH4lyLkJ 1wcrKNs5S+7Ac6ehU42D4zuvXagpbJ7kwYs6nHT9XkyDI5O9Cbd5maFSpct7VA8t3TFg MNMZOnDgJGCSq0k08ucrKuOI1bx7zEQZjKVFTNjJLCMGgXBwNLSc6pNPTFRI1qi/GqZ0 yTaw== X-Gm-Message-State: AJIora/Y+zvgNZq1bStjVL+PgiOIkypuGbztuQAr6Pz8gKsChIqiWwJI lTuTvAyIjI+blN/ufzuUVwtHjCKU+H9LTjne X-Google-Smtp-Source: AGRyM1vNqp1siTq4+vM6xRRCFB8ViZUf9ItF5XtUfUff2x9P2l161GxyqwN1706DB9irdsmvY90a1AH0gr7PPFum X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a25:b19a:0:b0:668:a921:310d with SMTP id h26-20020a25b19a000000b00668a921310dmr271313ybj.341.1656092235523; Fri, 24 Jun 2022 10:37:15 -0700 (PDT) Date: Fri, 24 Jun 2022 17:36:34 +0000 In-Reply-To: <20220624173656.2033256-1-jthoughton@google.com> Message-Id: <20220624173656.2033256-5-jthoughton@google.com> Mime-Version: 1.0 References: <20220624173656.2033256-1-jthoughton@google.com> X-Mailer: git-send-email 2.37.0.rc0.161.g10f37bed90-goog Subject: [RFC PATCH 04/26] hugetlb: make huge_pte_lockptr take an explicit shift argument. From: James Houghton To: Mike Kravetz , Muchun Song , Peter Xu Cc: David Hildenbrand , David Rientjes , Axel Rasmussen , Mina Almasry , Jue Wang , Manish Mishra , "Dr . David Alan Gilbert" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, James Houghton Content-Type: text/plain; charset="UTF-8" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656092236; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Prs/g0eAXOv/ZKQzs92554GHZM/jn5SpuUFUCkOXcEo=; b=796c5QE199jdPZ1FDCsuhhpeApg1+nOXEzWLg4wsIRIUIcvcY7hFR0of85s027gf6rgQPP No6fUjLJYmHigjV3cP8OXPozQ/4UCviEtXhSAV2n5lyqdi0Bj9m102++oDBC+z6VBJGiGy BkfN6VHxVcOJSWQjLBfUTpMT2bYJWtU= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Px4lWScA; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of 3S_a1YgoKCCwRbPWcOPbWVOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--jthoughton.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3S_a1YgoKCCwRbPWcOPbWVOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--jthoughton.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656092236; a=rsa-sha256; cv=none; b=XLWeYnl6WgGsQwvfFvhs2cVehoQRMg0ajE1WxXVpDeHoOrTaYPX/HccaeU2S1/z5+++C5v 1Yk23+Kzl8ACauo0gWTsApCQDMRkl7i7JPHw09LkVcFouc0AE1/j3n8jQ3AiU3Lcl3osxt 5+TJhcWHobPNkmFxxB1H7UO/jmvPgug= X-Rspamd-Queue-Id: 2BBD740023 Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Px4lWScA; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of 3S_a1YgoKCCwRbPWcOPbWVOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--jthoughton.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3S_a1YgoKCCwRbPWcOPbWVOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--jthoughton.bounces.google.com X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 373nyokkfacnn8qhi8x99iwuuthzay4m X-HE-Tag: 1656092235-506515 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is needed to handle PTL locking with high-granularity mapping. We won't always be using the PMD-level PTL even if we're using the 2M hugepage hstate. It's possible that we're dealing with 4K PTEs, in which case, we need to lock the PTL for the 4K PTE. Signed-off-by: James Houghton --- arch/powerpc/mm/pgtable.c | 3 ++- include/linux/hugetlb.h | 19 ++++++++++++++----- mm/hugetlb.c | 9 +++++---- mm/migrate.c | 3 ++- mm/page_vma_mapped.c | 3 ++- 5 files changed, 25 insertions(+), 12 deletions(-) diff --git a/arch/powerpc/mm/pgtable.c b/arch/powerpc/mm/pgtable.c index e6166b71d36d..663d591a8f08 100644 --- a/arch/powerpc/mm/pgtable.c +++ b/arch/powerpc/mm/pgtable.c @@ -261,7 +261,8 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, psize = hstate_get_psize(h); #ifdef CONFIG_DEBUG_VM - assert_spin_locked(huge_pte_lockptr(h, vma->vm_mm, ptep)); + assert_spin_locked(huge_pte_lockptr(huge_page_shift(h), + vma->vm_mm, ptep)); #endif #else diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 498a4ae3d462..5fe1db46d8c9 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -868,12 +868,11 @@ static inline gfp_t htlb_modify_alloc_mask(struct hstate *h, gfp_t gfp_mask) return modified_mask; } -static inline spinlock_t *huge_pte_lockptr(struct hstate *h, +static inline spinlock_t *huge_pte_lockptr(unsigned int shift, struct mm_struct *mm, pte_t *pte) { - if (huge_page_size(h) == PMD_SIZE) + if (shift == PMD_SHIFT) return pmd_lockptr(mm, (pmd_t *) pte); - VM_BUG_ON(huge_page_size(h) == PAGE_SIZE); return &mm->page_table_lock; } @@ -1076,7 +1075,7 @@ static inline gfp_t htlb_modify_alloc_mask(struct hstate *h, gfp_t gfp_mask) return 0; } -static inline spinlock_t *huge_pte_lockptr(struct hstate *h, +static inline spinlock_t *huge_pte_lockptr(unsigned int shift, struct mm_struct *mm, pte_t *pte) { return &mm->page_table_lock; @@ -1116,7 +1115,17 @@ static inline spinlock_t *huge_pte_lock(struct hstate *h, { spinlock_t *ptl; - ptl = huge_pte_lockptr(h, mm, pte); + ptl = huge_pte_lockptr(huge_page_shift(h), mm, pte); + spin_lock(ptl); + return ptl; +} + +static inline spinlock_t *huge_pte_lock_shift(unsigned int shift, + struct mm_struct *mm, pte_t *pte) +{ + spinlock_t *ptl; + + ptl = huge_pte_lockptr(shift, mm, pte); spin_lock(ptl); return ptl; } diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 0eec34edf3b2..d6d0d4c03def 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4817,7 +4817,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, continue; dst_ptl = huge_pte_lock(h, dst, dst_pte); - src_ptl = huge_pte_lockptr(h, src, src_pte); + src_ptl = huge_pte_lockptr(huge_page_shift(h), src, src_pte); spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); entry = huge_ptep_get(src_pte); dst_entry = huge_ptep_get(dst_pte); @@ -4894,7 +4894,8 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, /* Install the new huge page if src pte stable */ dst_ptl = huge_pte_lock(h, dst, dst_pte); - src_ptl = huge_pte_lockptr(h, src, src_pte); + src_ptl = huge_pte_lockptr(huge_page_shift(h), + src, src_pte); spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); entry = huge_ptep_get(src_pte); if (!pte_same(src_pte_old, entry)) { @@ -4948,7 +4949,7 @@ static void move_huge_pte(struct vm_area_struct *vma, unsigned long old_addr, pte_t pte; dst_ptl = huge_pte_lock(h, mm, dst_pte); - src_ptl = huge_pte_lockptr(h, mm, src_pte); + src_ptl = huge_pte_lockptr(huge_page_shift(h), mm, src_pte); /* * We don't have to worry about the ordering of src and dst ptlocks @@ -6024,7 +6025,7 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, page_in_pagecache = true; } - ptl = huge_pte_lockptr(h, dst_mm, dst_pte); + ptl = huge_pte_lockptr(huge_page_shift(h), dst_mm, dst_pte); spin_lock(ptl); /* diff --git a/mm/migrate.c b/mm/migrate.c index e51588e95f57..a8a960992373 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -318,7 +318,8 @@ void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd, void migration_entry_wait_huge(struct vm_area_struct *vma, struct mm_struct *mm, pte_t *pte) { - spinlock_t *ptl = huge_pte_lockptr(hstate_vma(vma), mm, pte); + spinlock_t *ptl = huge_pte_lockptr(huge_page_shift(hstate_vma(vma)), + mm, pte); __migration_entry_wait(mm, pte, ptl); } diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index c10f839fc410..8921dd4e41b1 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -174,7 +174,8 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) if (!pvmw->pte) return false; - pvmw->ptl = huge_pte_lockptr(hstate, mm, pvmw->pte); + pvmw->ptl = huge_pte_lockptr(huge_page_shift(hstate), + mm, pvmw->pte); spin_lock(pvmw->ptl); if (!check_pte(pvmw)) return not_found(pvmw); -- 2.37.0.rc0.161.g10f37bed90-goog