[PATCH] crypto: arm64/aes-ce - deal with oversight in new CTR carry code

[PATCH] crypto: arm64/aes-ce - deal with oversight in new CTR carry code

[Ard Biesheuvel]

The new carry handling code in the CTR driver can deal with a carry
occurring in the 4x/5x parallel code path, by using a computed goto to
jump into the carry sequence at the right place as to only apply the
carry to a subset of the blocks being processed.

If the lower half of the counter wraps and ends up at exactly 0x0, a
carry needs to be applied to the counter, but not to the counter values
taken for the 4x/5x parallel sequence. In this case, the computed goto
skips all register assignments, and branches straight to the jump
instruction that gets us back to the fast path. This produces the
correct result, but due to the fact that this branch target does not
carry the correct BTI annotation, this fails when BTI is enabled.

Let's omit the computed goto entirely in this case, and jump straight
back to the fast path after applying the carry to the main counter.

Fixes: 5318d3db465d ("crypto: arm64/aes-ctr - improve tail handling")
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
 arch/arm64/crypto/aes-modes.S | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/arm64/crypto/aes-modes.S b/arch/arm64/crypto/aes-modes.S
index ab6c14ef9f4e..6d1a120c533d 100644
--- a/arch/arm64/crypto/aes-modes.S
+++ b/arch/arm64/crypto/aes-modes.S
@@ -359,6 +359,7 @@ ST5(	mov		v4.16b, vctr.16b		)
 	ins		vctr.d[0], x8
 
 	/* apply carry to N counter blocks for N := x12 */
+	cbz		x12, 2f
 	adr		x16, 1f
 	sub		x16, x16, x12, lsl #3
 	br		x16
-- 
2.31.0.208.g409f899ff0-goog

Re: [PATCH] crypto: arm64/aes-ce - deal with oversight in new CTR carry code

[Herbert Xu]

On Tue, Apr 06, 2021 at 04:25:23PM +0200, Ard Biesheuvel wrote:
> The new carry handling code in the CTR driver can deal with a carry
> occurring in the 4x/5x parallel code path, by using a computed goto to
> jump into the carry sequence at the right place as to only apply the
> carry to a subset of the blocks being processed.
> 
> If the lower half of the counter wraps and ends up at exactly 0x0, a
> carry needs to be applied to the counter, but not to the counter values
> taken for the 4x/5x parallel sequence. In this case, the computed goto
> skips all register assignments, and branches straight to the jump
> instruction that gets us back to the fast path. This produces the
> correct result, but due to the fact that this branch target does not
> carry the correct BTI annotation, this fails when BTI is enabled.
> 
> Let's omit the computed goto entirely in this case, and jump straight
> back to the fast path after applying the carry to the main counter.
> 
> Fixes: 5318d3db465d ("crypto: arm64/aes-ctr - improve tail handling")
> Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
> ---
>  arch/arm64/crypto/aes-modes.S | 1 +
>  1 file changed, 1 insertion(+)

Patch applied.  Thanks.
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt