[PATCH v2 0/4] arm64: accelerate crc32_be

[PATCH v2 0/4] arm64: accelerate crc32_be

[Kevin Bracey]

This series completes the crc32 helper acceleration by adding crc32_be.

There are plenty of users, for example OF.

To compensate for the extra supporting cruft in lib/crc32.c, a couple of minor
tidies.

Version 2 incorporating review from Ard Biesheuvel.

Kevin Bracey (4):
  lib/crc32.c: remove unneeded casts
  lib/crc32.c: Make crc32_be weak for arch override
  lib/crc32test.c: correct printed bytes count
  arm64: accelerate crc32_be

 arch/arm64/lib/crc32.S | 87 +++++++++++++++++++++++++++++++++++-------
 lib/crc32.c            | 14 +++----
 lib/crc32test.c        |  2 +-
 3 files changed, 80 insertions(+), 23 deletions(-)

-- 
2.25.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 1/4] lib/crc32.c: remove unneeded casts

[Kevin Bracey]

Casts were added in commit 8f243af42ade ("sections: fix const sections
for crc32 table") to cope with the tables not being const. They are no
longer required since commit f5e38b9284e1 ("lib: crc32: constify crc32
lookup table").

Signed-off-by: Kevin Bracey <kevin@bracey.fi>
---
 lib/crc32.c | 9 +++------
 1 file changed, 3 insertions(+), 6 deletions(-)

diff --git a/lib/crc32.c b/lib/crc32.c
index 2a68dfd3b96c..7f062a2639df 100644
--- a/lib/crc32.c
+++ b/lib/crc32.c
@@ -194,13 +194,11 @@ u32 __pure __weak __crc32c_le(u32 crc, unsigned char const *p, size_t len)
 #else
 u32 __pure __weak crc32_le(u32 crc, unsigned char const *p, size_t len)
 {
-	return crc32_le_generic(crc, p, len,
-			(const u32 (*)[256])crc32table_le, CRC32_POLY_LE);
+	return crc32_le_generic(crc, p, len, crc32table_le, CRC32_POLY_LE);
 }
 u32 __pure __weak __crc32c_le(u32 crc, unsigned char const *p, size_t len)
 {
-	return crc32_le_generic(crc, p, len,
-			(const u32 (*)[256])crc32ctable_le, CRC32C_POLY_LE);
+	return crc32_le_generic(crc, p, len, crc32ctable_le, CRC32C_POLY_LE);
 }
 #endif
 EXPORT_SYMBOL(crc32_le);
@@ -339,8 +337,7 @@ u32 __pure crc32_be(u32 crc, unsigned char const *p, size_t len)
 #else
 u32 __pure crc32_be(u32 crc, unsigned char const *p, size_t len)
 {
-	return crc32_be_generic(crc, p, len,
-			(const u32 (*)[256])crc32table_be, CRC32_POLY_BE);
+	return crc32_be_generic(crc, p, len, crc32table_be, CRC32_POLY_BE);
 }
 #endif
 EXPORT_SYMBOL(crc32_be);
-- 
2.25.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 2/4] lib/crc32.c: Make crc32_be weak for arch override

[Kevin Bracey]

crc32_le and __crc32c_le can be overridden - extend this to crc32_be.

Signed-off-by: Kevin Bracey <kevin@bracey.fi>
---
 lib/crc32.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/lib/crc32.c b/lib/crc32.c
index 7f062a2639df..5649847d0a8d 100644
--- a/lib/crc32.c
+++ b/lib/crc32.c
@@ -206,6 +206,7 @@ EXPORT_SYMBOL(__crc32c_le);
 
 u32 __pure crc32_le_base(u32, unsigned char const *, size_t) __alias(crc32_le);
 u32 __pure __crc32c_le_base(u32, unsigned char const *, size_t) __alias(__crc32c_le);
+u32 __pure crc32_be_base(u32, unsigned char const *, size_t) __alias(crc32_be);
 
 /*
  * This multiplies the polynomials x and y modulo the given modulus.
@@ -330,12 +331,12 @@ static inline u32 __pure crc32_be_generic(u32 crc, unsigned char const *p,
 }
 
 #if CRC_BE_BITS == 1
-u32 __pure crc32_be(u32 crc, unsigned char const *p, size_t len)
+u32 __pure __weak crc32_be(u32 crc, unsigned char const *p, size_t len)
 {
 	return crc32_be_generic(crc, p, len, NULL, CRC32_POLY_BE);
 }
 #else
-u32 __pure crc32_be(u32 crc, unsigned char const *p, size_t len)
+u32 __pure __weak crc32_be(u32 crc, unsigned char const *p, size_t len)
 {
 	return crc32_be_generic(crc, p, len, crc32table_be, CRC32_POLY_BE);
 }
-- 
2.25.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 3/4] lib/crc32test.c: correct printed bytes count

[Kevin Bracey]

crc32c_le self test had a stray multiply by two inherited from
the crc32_le+crc32_be test loop.

Signed-off-by: Kevin Bracey <kevin@bracey.fi>
---
 lib/crc32test.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/crc32test.c b/lib/crc32test.c
index 61ddce2cff77..9b4af79412c4 100644
--- a/lib/crc32test.c
+++ b/lib/crc32test.c
@@ -675,7 +675,7 @@ static int __init crc32c_test(void)
 
 	/* pre-warm the cache */
 	for (i = 0; i < 100; i++) {
-		bytes += 2*test[i].length;
+		bytes += test[i].length;
 
 		crc ^= __crc32c_le(test[i].crc, test_buf +
 		    test[i].start, test[i].length);
-- 
2.25.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 4/4] arm64: accelerate crc32_be

[Kevin Bracey]

It makes no sense to leave crc32_be using the generic code while we
only accelerate the little-endian ops.

Even though the big-endian form doesn't fit as smoothly into the arm64,
we can speed it up and avoid hitting the D cache.

Tested on Cortex-A53. Without acceleration:

    crc32: CRC_LE_BITS = 64, CRC_BE BITS = 64
    crc32: self tests passed, processed 225944 bytes in 192240 nsec
    crc32c: CRC_LE_BITS = 64
    crc32c: self tests passed, processed 112972 bytes in 21360 nsec

With acceleration:

    crc32: CRC_LE_BITS = 64, CRC_BE BITS = 64
    crc32: self tests passed, processed 225944 bytes in 53480 nsec
    crc32c: CRC_LE_BITS = 64
    crc32c: self tests passed, processed 112972 bytes in 21480 nsec

Signed-off-by: Kevin Bracey <kevin@bracey.fi>
Tested-by: Ard Biesheuvel <ardb@kernel.org>
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will@kernel.org>
---
 arch/arm64/lib/crc32.S | 87 +++++++++++++++++++++++++++++++++++-------
 1 file changed, 73 insertions(+), 14 deletions(-)

diff --git a/arch/arm64/lib/crc32.S b/arch/arm64/lib/crc32.S
index 0f9e10ecda23..8340dccff46f 100644
--- a/arch/arm64/lib/crc32.S
+++ b/arch/arm64/lib/crc32.S
@@ -11,7 +11,44 @@
 
 	.arch		armv8-a+crc
 
-	.macro		__crc32, c
+	.macro		byteorder, reg, be
+	.if		\be
+CPU_LE( rev		\reg, \reg	)
+	.else
+CPU_BE( rev		\reg, \reg	)
+	.endif
+	.endm
+
+	.macro		byteorder16, reg, be
+	.if		\be
+CPU_LE( rev16		\reg, \reg	)
+	.else
+CPU_BE( rev16		\reg, \reg	)
+	.endif
+	.endm
+
+	.macro		bitorder, reg, be
+	.if		\be
+	rbit		\reg, \reg
+	.endif
+	.endm
+
+	.macro		bitorder16, reg, be
+	.if		\be
+	rbit		\reg, \reg
+	lsr		\reg, \reg, #16
+	.endif
+	.endm
+
+	.macro		bitorder8, reg, be
+	.if		\be
+	rbit		\reg, \reg
+	lsr		\reg, \reg, #24
+	.endif
+	.endm
+
+	.macro		__crc32, c, be=0
+	bitorder	w0, \be
 	cmp		x2, #16
 	b.lt		8f			// less than 16 bytes
 
@@ -24,10 +61,14 @@
 	add		x8, x8, x1
 	add		x1, x1, x7
 	ldp		x5, x6, [x8]
-CPU_BE(	rev		x3, x3		)
-CPU_BE(	rev		x4, x4		)
-CPU_BE(	rev		x5, x5		)
-CPU_BE(	rev		x6, x6		)
+	byteorder	x3, \be
+	byteorder	x4, \be
+	byteorder	x5, \be
+	byteorder	x6, \be
+	bitorder	x3, \be
+	bitorder	x4, \be
+	bitorder	x5, \be
+	bitorder	x6, \be
 
 	tst		x7, #8
 	crc32\c\()x	w8, w0, x3
@@ -55,33 +96,43 @@ CPU_BE(	rev		x6, x6		)
 32:	ldp		x3, x4, [x1], #32
 	sub		x2, x2, #32
 	ldp		x5, x6, [x1, #-16]
-CPU_BE(	rev		x3, x3		)
-CPU_BE(	rev		x4, x4		)
-CPU_BE(	rev		x5, x5		)
-CPU_BE(	rev		x6, x6		)
+	byteorder	x3, \be
+	byteorder	x4, \be
+	byteorder	x5, \be
+	byteorder	x6, \be
+	bitorder	x3, \be
+	bitorder	x4, \be
+	bitorder	x5, \be
+	bitorder	x6, \be
 	crc32\c\()x	w0, w0, x3
 	crc32\c\()x	w0, w0, x4
 	crc32\c\()x	w0, w0, x5
 	crc32\c\()x	w0, w0, x6
 	cbnz		x2, 32b
-0:	ret
+0:	bitorder	w0, \be
+	ret
 
 8:	tbz		x2, #3, 4f
 	ldr		x3, [x1], #8
-CPU_BE(	rev		x3, x3		)
+	byteorder	x3, \be
+	bitorder	x3, \be
 	crc32\c\()x	w0, w0, x3
 4:	tbz		x2, #2, 2f
 	ldr		w3, [x1], #4
-CPU_BE(	rev		w3, w3		)
+	byteorder	w3, \be
+	bitorder	w3, \be
 	crc32\c\()w	w0, w0, w3
 2:	tbz		x2, #1, 1f
 	ldrh		w3, [x1], #2
-CPU_BE(	rev16		w3, w3		)
+	byteorder16	w3, \be
+	bitorder16	w3, \be
 	crc32\c\()h	w0, w0, w3
 1:	tbz		x2, #0, 0f
 	ldrb		w3, [x1]
+	bitorder8	w3, \be
 	crc32\c\()b	w0, w0, w3
-0:	ret
+0:	bitorder	w0, \be
+	ret
 	.endm
 
 	.align		5
@@ -99,3 +150,11 @@ alternative_if_not ARM64_HAS_CRC32
 alternative_else_nop_endif
 	__crc32		c
 SYM_FUNC_END(__crc32c_le)
+
+	.align		5
+SYM_FUNC_START(crc32_be)
+alternative_if_not ARM64_HAS_CRC32
+	b		crc32_be_base
+alternative_else_nop_endif
+	__crc32		be=1
+SYM_FUNC_END(crc32_be)
-- 
2.25.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 4/4] arm64: accelerate crc32_be

[Ard Biesheuvel]

On Thu, 13 Jan 2022 at 17:58, Kevin Bracey <kevin@bracey.fi> wrote:
>
> It makes no sense to leave crc32_be using the generic code while we
> only accelerate the little-endian ops.
>
> Even though the big-endian form doesn't fit as smoothly into the arm64,
> we can speed it up and avoid hitting the D cache.
>
> Tested on Cortex-A53. Without acceleration:
>
>     crc32: CRC_LE_BITS = 64, CRC_BE BITS = 64
>     crc32: self tests passed, processed 225944 bytes in 192240 nsec
>     crc32c: CRC_LE_BITS = 64
>     crc32c: self tests passed, processed 112972 bytes in 21360 nsec
>
> With acceleration:
>
>     crc32: CRC_LE_BITS = 64, CRC_BE BITS = 64
>     crc32: self tests passed, processed 225944 bytes in 53480 nsec
>     crc32c: CRC_LE_BITS = 64
>     crc32c: self tests passed, processed 112972 bytes in 21480 nsec
>
> Signed-off-by: Kevin Bracey <kevin@bracey.fi>
> Tested-by: Ard Biesheuvel <ardb@kernel.org>
> Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
> Cc: Catalin Marinas <catalin.marinas@arm.com>
> Cc: Will Deacon <will@kernel.org>

Catalin, Will,

We'll need an ack from you so we can take this whole set via Herbert's tree.

Thanks,
Ard.


> ---
>  arch/arm64/lib/crc32.S | 87 +++++++++++++++++++++++++++++++++++-------
>  1 file changed, 73 insertions(+), 14 deletions(-)
>
> diff --git a/arch/arm64/lib/crc32.S b/arch/arm64/lib/crc32.S
> index 0f9e10ecda23..8340dccff46f 100644
> --- a/arch/arm64/lib/crc32.S
> +++ b/arch/arm64/lib/crc32.S
> @@ -11,7 +11,44 @@
>
>         .arch           armv8-a+crc
>
> -       .macro          __crc32, c
> +       .macro          byteorder, reg, be
> +       .if             \be
> +CPU_LE( rev            \reg, \reg      )
> +       .else
> +CPU_BE( rev            \reg, \reg      )
> +       .endif
> +       .endm
> +
> +       .macro          byteorder16, reg, be
> +       .if             \be
> +CPU_LE( rev16          \reg, \reg      )
> +       .else
> +CPU_BE( rev16          \reg, \reg      )
> +       .endif
> +       .endm
> +
> +       .macro          bitorder, reg, be
> +       .if             \be
> +       rbit            \reg, \reg
> +       .endif
> +       .endm
> +
> +       .macro          bitorder16, reg, be
> +       .if             \be
> +       rbit            \reg, \reg
> +       lsr             \reg, \reg, #16
> +       .endif
> +       .endm
> +
> +       .macro          bitorder8, reg, be
> +       .if             \be
> +       rbit            \reg, \reg
> +       lsr             \reg, \reg, #24
> +       .endif
> +       .endm
> +
> +       .macro          __crc32, c, be=0
> +       bitorder        w0, \be
>         cmp             x2, #16
>         b.lt            8f                      // less than 16 bytes
>
> @@ -24,10 +61,14 @@
>         add             x8, x8, x1
>         add             x1, x1, x7
>         ldp             x5, x6, [x8]
> -CPU_BE(        rev             x3, x3          )
> -CPU_BE(        rev             x4, x4          )
> -CPU_BE(        rev             x5, x5          )
> -CPU_BE(        rev             x6, x6          )
> +       byteorder       x3, \be
> +       byteorder       x4, \be
> +       byteorder       x5, \be
> +       byteorder       x6, \be
> +       bitorder        x3, \be
> +       bitorder        x4, \be
> +       bitorder        x5, \be
> +       bitorder        x6, \be
>
>         tst             x7, #8
>         crc32\c\()x     w8, w0, x3
> @@ -55,33 +96,43 @@ CPU_BE(     rev             x6, x6          )
>  32:    ldp             x3, x4, [x1], #32
>         sub             x2, x2, #32
>         ldp             x5, x6, [x1, #-16]
> -CPU_BE(        rev             x3, x3          )
> -CPU_BE(        rev             x4, x4          )
> -CPU_BE(        rev             x5, x5          )
> -CPU_BE(        rev             x6, x6          )
> +       byteorder       x3, \be
> +       byteorder       x4, \be
> +       byteorder       x5, \be
> +       byteorder       x6, \be
> +       bitorder        x3, \be
> +       bitorder        x4, \be
> +       bitorder        x5, \be
> +       bitorder        x6, \be
>         crc32\c\()x     w0, w0, x3
>         crc32\c\()x     w0, w0, x4
>         crc32\c\()x     w0, w0, x5
>         crc32\c\()x     w0, w0, x6
>         cbnz            x2, 32b
> -0:     ret
> +0:     bitorder        w0, \be
> +       ret
>
>  8:     tbz             x2, #3, 4f
>         ldr             x3, [x1], #8
> -CPU_BE(        rev             x3, x3          )
> +       byteorder       x3, \be
> +       bitorder        x3, \be
>         crc32\c\()x     w0, w0, x3
>  4:     tbz             x2, #2, 2f
>         ldr             w3, [x1], #4
> -CPU_BE(        rev             w3, w3          )
> +       byteorder       w3, \be
> +       bitorder        w3, \be
>         crc32\c\()w     w0, w0, w3
>  2:     tbz             x2, #1, 1f
>         ldrh            w3, [x1], #2
> -CPU_BE(        rev16           w3, w3          )
> +       byteorder16     w3, \be
> +       bitorder16      w3, \be
>         crc32\c\()h     w0, w0, w3
>  1:     tbz             x2, #0, 0f
>         ldrb            w3, [x1]
> +       bitorder8       w3, \be
>         crc32\c\()b     w0, w0, w3
> -0:     ret
> +0:     bitorder        w0, \be
> +       ret
>         .endm
>
>         .align          5
> @@ -99,3 +150,11 @@ alternative_if_not ARM64_HAS_CRC32
>  alternative_else_nop_endif
>         __crc32         c
>  SYM_FUNC_END(__crc32c_le)
> +
> +       .align          5
> +SYM_FUNC_START(crc32_be)
> +alternative_if_not ARM64_HAS_CRC32
> +       b               crc32_be_base
> +alternative_else_nop_endif
> +       __crc32         be=1
> +SYM_FUNC_END(crc32_be)
> --
> 2.25.1
>

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 4/4] arm64: accelerate crc32_be

[Catalin Marinas]

On Thu, Jan 13, 2022 at 06:58:07PM +0200, Kevin Bracey wrote:
> It makes no sense to leave crc32_be using the generic code while we
> only accelerate the little-endian ops.
> 
> Even though the big-endian form doesn't fit as smoothly into the arm64,
> we can speed it up and avoid hitting the D cache.
> 
> Tested on Cortex-A53. Without acceleration:
> 
>     crc32: CRC_LE_BITS = 64, CRC_BE BITS = 64
>     crc32: self tests passed, processed 225944 bytes in 192240 nsec
>     crc32c: CRC_LE_BITS = 64
>     crc32c: self tests passed, processed 112972 bytes in 21360 nsec
> 
> With acceleration:
> 
>     crc32: CRC_LE_BITS = 64, CRC_BE BITS = 64
>     crc32: self tests passed, processed 225944 bytes in 53480 nsec
>     crc32c: CRC_LE_BITS = 64
>     crc32c: self tests passed, processed 112972 bytes in 21480 nsec
> 
> Signed-off-by: Kevin Bracey <kevin@bracey.fi>
> Tested-by: Ard Biesheuvel <ardb@kernel.org>
> Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
> Cc: Catalin Marinas <catalin.marinas@arm.com>
> Cc: Will Deacon <will@kernel.org>

Acked-by: Catalin Marinas <catalin.marinas@arm.com>

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel