From: Will Deacon <will@kernel.org> To: linux-arm-kernel@lists.infradead.org Cc: linux-arch@vger.kernel.org, Will Deacon <will@kernel.org>, Catalin Marinas <catalin.marinas@arm.com>, Mark Rutland <mark.rutland@arm.com>, Marc Zyngier <maz@kernel.org>, Peter Zijlstra <peterz@infradead.org>, stable@vger.kernel.org Subject: [PATCH 1/6] Revert "arm64: Remove unnecessary ISBs from set_{pte,pmd,pud}" Date: Tue, 27 Aug 2019 14:18:13 +0100 [thread overview] Message-ID: <20190827131818.14724-2-will@kernel.org> (raw) In-Reply-To: <20190827131818.14724-1-will@kernel.org> This reverts commit 24fe1b0efad4fcdd32ce46cffeab297f22581707. Commit 24fe1b0efad4fcdd ("arm64: Remove unnecessary ISBs from set_{pte,pmd,pud}") removed ISB instructions immediately following updates to the page table, on the grounds that they are not required by the architecture and a DSB alone is sufficient to ensure that subsequent data accesses use the new translation: DDI0487E_a, B2-128: | ... no instruction that appears in program order after the DSB | instruction can alter any state of the system or perform any part of | its functionality until the DSB completes other than: | | * Being fetched from memory and decoded | * Reading the general-purpose, SIMD and floating-point, | Special-purpose, or System registers that are directly or indirectly | read without causing side-effects. However, the same document also states the following: DDI0487E_a, B2-125: | DMB and DSB instructions affect reads and writes to the memory system | generated by Load/Store instructions and data or unified cache | maintenance instructions being executed by the PE. Instruction fetches | or accesses caused by a hardware translation table access are not | explicit accesses. which appears to claim that the DSB alone is insufficient. Unfortunately, some CPU designers have followed the second clause above, whereas in Linux we've been relying on the first. This means that our mapping sequence: MOV X0, <valid pte> STR X0, [Xptep] // Store new PTE to page table DSB ISHST LDR X1, [X2] // Translates using the new PTE can actually raise a translation fault on the load instruction because the translation can be performed speculatively before the page table update and then marked as "faulting" by the CPU. For user PTEs, this is ok because we can handle the spurious fault, but for kernel PTEs and intermediate table entries this results in a panic(). Revert the offending commit to reintroduce the missing barriers. Cc: <stable@vger.kernel.org> Fixes: 24fe1b0efad4fcdd ("arm64: Remove unnecessary ISBs from set_{pte,pmd,pud}") Signed-off-by: Will Deacon <will@kernel.org> --- arch/arm64/include/asm/pgtable.h | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 5fdcfe237338..feda7294320c 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -220,8 +220,10 @@ static inline void set_pte(pte_t *ptep, pte_t pte) * Only if the new pte is valid and kernel, otherwise TLB maintenance * or update_mmu_cache() have the necessary barriers. */ - if (pte_valid_not_user(pte)) + if (pte_valid_not_user(pte)) { dsb(ishst); + isb(); + } } extern void __sync_icache_dcache(pte_t pteval); @@ -481,8 +483,10 @@ static inline void set_pmd(pmd_t *pmdp, pmd_t pmd) WRITE_ONCE(*pmdp, pmd); - if (pmd_valid(pmd)) + if (pmd_valid(pmd)) { dsb(ishst); + isb(); + } } static inline void pmd_clear(pmd_t *pmdp) @@ -540,8 +544,10 @@ static inline void set_pud(pud_t *pudp, pud_t pud) WRITE_ONCE(*pudp, pud); - if (pud_valid(pud)) + if (pud_valid(pud)) { dsb(ishst); + isb(); + } } static inline void pud_clear(pud_t *pudp) -- 2.11.0
WARNING: multiple messages have this Message-ID (diff)
From: Will Deacon <will@kernel.org> To: linux-arm-kernel@lists.infradead.org Cc: linux-arch@vger.kernel.org, Mark Rutland <mark.rutland@arm.com>, Peter Zijlstra <peterz@infradead.org>, Catalin Marinas <catalin.marinas@arm.com>, stable@vger.kernel.org, Marc Zyngier <maz@kernel.org>, Will Deacon <will@kernel.org> Subject: [PATCH 1/6] Revert "arm64: Remove unnecessary ISBs from set_{pte, pmd, pud}" Date: Tue, 27 Aug 2019 14:18:13 +0100 [thread overview] Message-ID: <20190827131818.14724-2-will@kernel.org> (raw) In-Reply-To: <20190827131818.14724-1-will@kernel.org> This reverts commit 24fe1b0efad4fcdd32ce46cffeab297f22581707. Commit 24fe1b0efad4fcdd ("arm64: Remove unnecessary ISBs from set_{pte,pmd,pud}") removed ISB instructions immediately following updates to the page table, on the grounds that they are not required by the architecture and a DSB alone is sufficient to ensure that subsequent data accesses use the new translation: DDI0487E_a, B2-128: | ... no instruction that appears in program order after the DSB | instruction can alter any state of the system or perform any part of | its functionality until the DSB completes other than: | | * Being fetched from memory and decoded | * Reading the general-purpose, SIMD and floating-point, | Special-purpose, or System registers that are directly or indirectly | read without causing side-effects. However, the same document also states the following: DDI0487E_a, B2-125: | DMB and DSB instructions affect reads and writes to the memory system | generated by Load/Store instructions and data or unified cache | maintenance instructions being executed by the PE. Instruction fetches | or accesses caused by a hardware translation table access are not | explicit accesses. which appears to claim that the DSB alone is insufficient. Unfortunately, some CPU designers have followed the second clause above, whereas in Linux we've been relying on the first. This means that our mapping sequence: MOV X0, <valid pte> STR X0, [Xptep] // Store new PTE to page table DSB ISHST LDR X1, [X2] // Translates using the new PTE can actually raise a translation fault on the load instruction because the translation can be performed speculatively before the page table update and then marked as "faulting" by the CPU. For user PTEs, this is ok because we can handle the spurious fault, but for kernel PTEs and intermediate table entries this results in a panic(). Revert the offending commit to reintroduce the missing barriers. Cc: <stable@vger.kernel.org> Fixes: 24fe1b0efad4fcdd ("arm64: Remove unnecessary ISBs from set_{pte,pmd,pud}") Signed-off-by: Will Deacon <will@kernel.org> --- arch/arm64/include/asm/pgtable.h | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 5fdcfe237338..feda7294320c 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -220,8 +220,10 @@ static inline void set_pte(pte_t *ptep, pte_t pte) * Only if the new pte is valid and kernel, otherwise TLB maintenance * or update_mmu_cache() have the necessary barriers. */ - if (pte_valid_not_user(pte)) + if (pte_valid_not_user(pte)) { dsb(ishst); + isb(); + } } extern void __sync_icache_dcache(pte_t pteval); @@ -481,8 +483,10 @@ static inline void set_pmd(pmd_t *pmdp, pmd_t pmd) WRITE_ONCE(*pmdp, pmd); - if (pmd_valid(pmd)) + if (pmd_valid(pmd)) { dsb(ishst); + isb(); + } } static inline void pmd_clear(pmd_t *pmdp) @@ -540,8 +544,10 @@ static inline void set_pud(pud_t *pudp, pud_t pud) WRITE_ONCE(*pudp, pud); - if (pud_valid(pud)) + if (pud_valid(pud)) { dsb(ishst); + isb(); + } } static inline void pud_clear(pud_t *pudp) -- 2.11.0 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
next prev parent reply other threads:[~2019-08-27 13:19 UTC|newest] Thread overview: 36+ messages / expand[flat|nested] mbox.gz Atom feed top 2019-08-27 13:18 [PATCH 0/6] Fix TLB invalidation on arm64 Will Deacon 2019-08-27 13:18 ` Will Deacon 2019-08-27 13:18 ` Will Deacon 2019-08-27 13:18 ` Will Deacon [this message] 2019-08-27 13:18 ` [PATCH 1/6] Revert "arm64: Remove unnecessary ISBs from set_{pte, pmd, pud}" Will Deacon 2019-08-28 2:29 ` Sasha Levin 2019-08-28 2:29 ` Sasha Levin 2019-08-27 13:18 ` [PATCH 2/6] arm64: tlb: Ensure we execute an ISB following walk cache invalidation Will Deacon 2019-08-27 13:18 ` Will Deacon 2019-08-27 13:18 ` [PATCH 3/6] arm64: mm: Add ISB instruction to set_pgd() Will Deacon 2019-08-27 13:18 ` Will Deacon 2019-08-27 13:18 ` Will Deacon 2019-08-27 13:18 ` [PATCH 4/6] arm64: sysreg: Add some field definitions for PAR_EL1 Will Deacon 2019-08-27 13:18 ` Will Deacon 2019-08-27 13:18 ` Will Deacon 2019-08-27 13:18 ` [PATCH 5/6] arm64: mm: Ignore spurious translation faults taken from the kernel Will Deacon 2019-08-27 13:18 ` Will Deacon 2019-08-27 13:18 ` Will Deacon 2019-08-27 13:18 ` [PATCH 6/6] arm64: kvm: Replace hardcoded '1' with SYS_PAR_EL1_F Will Deacon 2019-08-27 13:18 ` Will Deacon 2019-08-27 13:18 ` Will Deacon 2019-08-27 16:19 ` [PATCH 0/6] Fix TLB invalidation on arm64 Mark Rutland 2019-08-27 16:19 ` Mark Rutland 2019-08-27 16:19 ` Mark Rutland 2019-08-28 0:35 ` Nicholas Piggin 2019-08-28 0:35 ` Nicholas Piggin 2019-08-28 0:35 ` Nicholas Piggin 2019-08-28 16:12 ` Will Deacon 2019-08-28 16:12 ` Will Deacon 2019-08-28 16:12 ` Will Deacon 2019-08-29 14:08 ` Nicholas Piggin 2019-08-29 14:08 ` Nicholas Piggin 2019-08-29 14:08 ` Nicholas Piggin 2019-08-30 12:40 ` Will Deacon 2019-08-30 12:40 ` Will Deacon 2019-08-30 12:40 ` Will Deacon
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20190827131818.14724-2-will@kernel.org \ --to=will@kernel.org \ --cc=catalin.marinas@arm.com \ --cc=linux-arch@vger.kernel.org \ --cc=linux-arm-kernel@lists.infradead.org \ --cc=mark.rutland@arm.com \ --cc=maz@kernel.org \ --cc=peterz@infradead.org \ --cc=stable@vger.kernel.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.