[PATCH v4 0/2] arm64: DC {ZVA,GVA,GZVA} shouldn't be used when DCZID_EL0.DZP == 1

[PATCH v4 0/2] arm64: DC {ZVA,GVA,GZVA} shouldn't be used when DCZID_EL0.DZP == 1

[Reiji Watanabe]

DCZID_EL0.DZP indicates whether or not use of DC {ZVA,GVA,GZVA} is
prohibited. However, clear_page(), mte_set_mem_tag_range(), and
mte_zero_clear_page_tags() use those instructions without checking it.

Fix those functions not to use DC {ZVA,GVA,GZVA} when DCZID_EL0.DZP == 1.

v4:
 - Fix mte_zero_clear_page_tags to use stz2g instead of stzg. [Catalin]
 - Fix mte_zero_clear_page_tags to use MTE_GRANULE_SIZE instead
   of 16. [Catalin]

v3: https://lore.kernel.org/all/20211119060927.3185381-1-reijiw@google.com/
 - Use stnp instead of stp in clear_page().

v2: https://lore.kernel.org/all/20211108071149.823570-1-reijiw@google.com/
 - Fix mte_set_mem_tag_range() and mte_zero_clear_page_tags() not to use
   DC {GVA,GZVA} when DCZID_EL0.DZP == 1. [Mark]
 - Fix a typo of the comment in clear_page(). [Mark]
 - Fix a loop of for stp in clear_page() for more consistency with
   the existing loop. [Mark]

v1: https://lore.kernel.org/all/20211026034844.1393437-1-reijiw@google.com/

Reiji Watanabe (2):
  arm64: clear_page() shouldn't use DC ZVA when DCZID_EL0.DZP == 1
  arm64: mte: DC {GVA,GZVA} shouldn't be used when DCZID_EL0.DZP == 1

 arch/arm64/include/asm/mte-kasan.h |  8 +++++---
 arch/arm64/lib/clear_page.S        | 10 ++++++++++
 arch/arm64/lib/mte.S               |  8 +++++++-
 3 files changed, 22 insertions(+), 4 deletions(-)


base-commit: 0fcfb00b28c0b7884635dacf38e46d60bf3d4eb1
-- 
2.34.1.400.ga245620fadb-goog


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v4 1/2] arm64: clear_page() shouldn't use DC ZVA when DCZID_EL0.DZP == 1

[Reiji Watanabe]

Currently, clear_page() uses DC ZVA instruction unconditionally.  But it
should make sure that DCZID_EL0.DZP, which indicates whether or not use
of DC ZVA instruction is prohibited, is zero when using the instruction.
Use STNP instead when DCZID_EL0.DZP == 1.

Fixes: f27bb139c387 ("arm64: Miscellaneous library functions")
Signed-off-by: Reiji Watanabe <reijiw@google.com>
Reviewed-by: Robin Murphy <robin.murphy@arm.com>
---
 arch/arm64/lib/clear_page.S | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/arch/arm64/lib/clear_page.S b/arch/arm64/lib/clear_page.S
index b84b179edba3..1fd5d790ab80 100644
--- a/arch/arm64/lib/clear_page.S
+++ b/arch/arm64/lib/clear_page.S
@@ -16,6 +16,7 @@
  */
 SYM_FUNC_START_PI(clear_page)
 	mrs	x1, dczid_el0
+	tbnz	x1, #4, 2f	/* Branch if DC ZVA is prohibited */
 	and	w1, w1, #0xf
 	mov	x2, #4
 	lsl	x1, x2, x1
@@ -25,5 +26,14 @@ SYM_FUNC_START_PI(clear_page)
 	tst	x0, #(PAGE_SIZE - 1)
 	b.ne	1b
 	ret
+
+2:	stnp	xzr, xzr, [x0]
+	stnp	xzr, xzr, [x0, #16]
+	stnp	xzr, xzr, [x0, #32]
+	stnp	xzr, xzr, [x0, #48]
+	add	x0, x0, #64
+	tst	x0, #(PAGE_SIZE - 1)
+	b.ne	2b
+	ret
 SYM_FUNC_END_PI(clear_page)
 EXPORT_SYMBOL(clear_page)
-- 
2.34.1.400.ga245620fadb-goog


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v4 2/2] arm64: mte: DC {GVA,GZVA} shouldn't be used when DCZID_EL0.DZP == 1

[Reiji Watanabe]

Currently, mte_set_mem_tag_range() and mte_zero_clear_page_tags() use
DC {GVA,GZVA} unconditionally.  But, they should make sure that
DCZID_EL0.DZP, which indicates whether or not use of those instructions
is prohibited, is zero when using those instructions.
Use ST{G,ZG,Z2G} instead when DCZID_EL0.DZP == 1.

Fixes: 013bb59dbb7c ("arm64: mte: handle tags zeroing at page allocation time")
Fixes: 3d0cca0b02ac ("kasan: speed up mte_set_mem_tag_range")
Signed-off-by: Reiji Watanabe <reijiw@google.com>
---
 arch/arm64/include/asm/mte-kasan.h | 8 +++++---
 arch/arm64/lib/mte.S               | 8 +++++++-
 2 files changed, 12 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/include/asm/mte-kasan.h b/arch/arm64/include/asm/mte-kasan.h
index 478b9bcf69ad..e4704a403237 100644
--- a/arch/arm64/include/asm/mte-kasan.h
+++ b/arch/arm64/include/asm/mte-kasan.h
@@ -84,10 +84,12 @@ static inline void __dc_gzva(u64 p)
 static inline void mte_set_mem_tag_range(void *addr, size_t size, u8 tag,
 					 bool init)
 {
-	u64 curr, mask, dczid_bs, end1, end2, end3;
+	u64 curr, mask, dczid, dczid_bs, dczid_dzp, end1, end2, end3;
 
 	/* Read DC G(Z)VA block size from the system register. */
-	dczid_bs = 4ul << (read_cpuid(DCZID_EL0) & 0xf);
+	dczid = read_cpuid(DCZID_EL0);
+	dczid_bs = 4ul << (dczid & 0xf);
+	dczid_dzp = (dczid >> 4) & 1;
 
 	curr = (u64)__tag_set(addr, tag);
 	mask = dczid_bs - 1;
@@ -106,7 +108,7 @@ static inline void mte_set_mem_tag_range(void *addr, size_t size, u8 tag,
 	 */
 #define SET_MEMTAG_RANGE(stg_post, dc_gva)		\
 	do {						\
-		if (size >= 2 * dczid_bs) {		\
+		if (!dczid_dzp && size >= 2 * dczid_bs) {\
 			do {				\
 				curr = stg_post(curr);	\
 			} while (curr < end1);		\
diff --git a/arch/arm64/lib/mte.S b/arch/arm64/lib/mte.S
index e83643b3995f..f531dcb95174 100644
--- a/arch/arm64/lib/mte.S
+++ b/arch/arm64/lib/mte.S
@@ -43,17 +43,23 @@ SYM_FUNC_END(mte_clear_page_tags)
  *	x0 - address to the beginning of the page
  */
 SYM_FUNC_START(mte_zero_clear_page_tags)
+	and	x0, x0, #(1 << MTE_TAG_SHIFT) - 1	// clear the tag
 	mrs	x1, dczid_el0
+	tbnz	x1, #4, 2f	// Branch if DC GZVA is prohibited
 	and	w1, w1, #0xf
 	mov	x2, #4
 	lsl	x1, x2, x1
-	and	x0, x0, #(1 << MTE_TAG_SHIFT) - 1	// clear the tag
 
 1:	dc	gzva, x0
 	add	x0, x0, x1
 	tst	x0, #(PAGE_SIZE - 1)
 	b.ne	1b
 	ret
+
+2:	stz2g	x0, [x0], #(MTE_GRANULE_SIZE * 2)
+	tst	x0, #(PAGE_SIZE - 1)
+	b.ne	2b
+	ret
 SYM_FUNC_END(mte_zero_clear_page_tags)
 
 /*
-- 
2.34.1.400.ga245620fadb-goog


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v4 0/2] arm64: DC {ZVA, GVA, GZVA} shouldn't be used when DCZID_EL0.DZP == 1

[Catalin Marinas]

On Sun, 5 Dec 2021 16:47:34 -0800, Reiji Watanabe wrote:
> DCZID_EL0.DZP indicates whether or not use of DC {ZVA,GVA,GZVA} is
> prohibited. However, clear_page(), mte_set_mem_tag_range(), and
> mte_zero_clear_page_tags() use those instructions without checking it.
> 
> Fix those functions not to use DC {ZVA,GVA,GZVA} when DCZID_EL0.DZP == 1.
> 
> v4:
>  - Fix mte_zero_clear_page_tags to use stz2g instead of stzg. [Catalin]
>  - Fix mte_zero_clear_page_tags to use MTE_GRANULE_SIZE instead
>    of 16. [Catalin]
> 
> [...]

Applied to arm64 (for-next/cache-ops-dzp), thanks!

[1/2] arm64: clear_page() shouldn't use DC ZVA when DCZID_EL0.DZP == 1
      https://git.kernel.org/arm64/c/f0616abd4e67
[2/2] arm64: mte: DC {GVA,GZVA} shouldn't be used when DCZID_EL0.DZP == 1
      https://git.kernel.org/arm64/c/685e2564daa1

-- 
Catalin


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel