Linux-ARM-Kernel Archive on
 help / color / Atom feed
From: Ard Biesheuvel <>
Cc: Ingo Molnar <>,
	Herbert Xu <>,
	Peter Zijlstra <>,
	Catalin Marinas <>,
	Sebastian Andrzej Siewior <>,, Ard Biesheuvel <>,
	Eric Biggers <>,
	Mark Brown <>,
	Thomas Gleixner <>,
	Will Deacon <>, Dave Martin <>,
Subject: [RFC PATCH 0/5] running kernel mode SIMD with softirqs disabled
Date: Fri, 18 Dec 2020 18:01:01 +0100
Message-ID: <> (raw)

[ TL;DR for the non-ARM folks on CC: disabling softirq processing when using
  SIMD in kernel mode could reduce complexity and improve performance, but we
  need to decide whether we can do this, and how much softirq processing
  latency we can tolerate. If we can find a satisfactory solution for this,
  we might do the same for x86 and 32-bit ARM as well. ]

The crypto API provides two ways to invoke symmetric encryption algorithms:
- synchronously, where the transformation is guaranteed to be done by the
  time the function returns;
- asynchronously, where the function may return with a -EINPROGRESS return code,
  and a completion will be signalled when the transformation is done.

The latter is mainly intended for h/w accelerators, where the throughput would
be severely limited by the latency otherwise. However, it is also being used
for software algorithms based on SIMD instructions, which cannot be issued from
any context (the rules are not the same on each architecture, but typically,
SIMD can be used in task context, or in softirq context if it was not taken
while the SIMD was already in use in kernel mode).

Many users of the crypto API exist in the kernel today that opt out of this
asynchronous interface (802.11, macsec, kerberos, sw kTLS), or use a library
interface which is fundamentally synchronous (wireguard). This means we end
up using a degraded mode for the contended case (a scalar fallback) as well
as the uncontended case (generic GCM/CCM/CTR chaining mode templates wrapped
around the SIMD cipher as opposed to accelerated implementations of the full
chaining modes in question). Note that scalar AES runs ~20x slower than the
SIMD instruction based version.

So let's address this for arm64, by reorganizing kernel mode SIMD support so
that the SIMD unit can always be assumed to be available. This means we need
to defer softirq processing when grabbing the NEON unit in task context, so
that any use of it in softirq context is guaranteed not to interrupt any code
that was already using the NEON.

This obviously impacts softirq processing latency, which is why the existing
conditional NEON yield support is modified to take pending softirqs into

As an example of how this impacts the code, the existing arm64 GCM driver is
updated to:
- Add yield support - currently, the pending softirq check is performed every
  64 bytes of input, which is way too often - one of the desired outcomes of
  this RFC is getting a reasonable ballpark for how long we want to run with
  softirqs disabled.
- Remove the existing scalar fallbacks, which are no longer needed.

- what did I miss or break horribly?
- does any of this matter for RT? AIUI, RT runs softirqs from a dedicated
  kthread, so I don't think it cares.
- what would be a reasonable upper bound to keep softirqs disabled? I suppose
  100s of cycles or less is overkill, but I'm not sure how to derive a better
- could we do the same on x86, now that kernel_fpu_begin/end is no longer

Cc: Dave Martin <>
Cc: Mark Brown <>
Cc: Herbert Xu <>
Cc: Eric Biggers <>
Cc: Will Deacon <>
Cc: Catalin Marinas <>
Cc: Thomas Gleixner <>
Cc: Peter Zijlstra <>
Cc: Sebastian Andrzej Siewior <>
Cc: Ingo Molnar <>

Ard Biesheuvel (5):
  crypto: aead - disallow en/decrypt for non-task or non-softirq context
  crypto: skcipher - disallow en/decrypt for non-task or non-softirq
  crypto: arm64/gcm-aes-ce - add NEON yield support
  arm64: fpsimd: run kernel mode NEON with softirqs disabled
  crypto: arm64/gcm-aes-ce - remove non-SIMD fallback path

 arch/arm64/crypto/ghash-ce-core.S  | 115 ++++++-----
 arch/arm64/crypto/ghash-ce-glue.c  | 209 +++++---------------
 arch/arm64/include/asm/assembler.h |  19 +-
 arch/arm64/kernel/asm-offsets.c    |   2 +
 arch/arm64/kernel/fpsimd.c         |   4 +-
 crypto/aead.c                      |  10 +
 crypto/skcipher.c                  |  10 +
 7 files changed, 155 insertions(+), 214 deletions(-)


linux-arm-kernel mailing list

             reply index

Thread overview: 14+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-12-18 17:01 Ard Biesheuvel [this message]
2020-12-18 17:01 ` [RFC PATCH 1/5] crypto: aead - disallow en/decrypt for non-task or non-softirq context Ard Biesheuvel
2020-12-18 17:01 ` [RFC PATCH 2/5] crypto: skcipher " Ard Biesheuvel
2020-12-18 17:01 ` [RFC PATCH 3/5] crypto: arm64/gcm-aes-ce - add NEON yield support Ard Biesheuvel
2020-12-18 17:01 ` [RFC PATCH 4/5] arm64: fpsimd: run kernel mode NEON with softirqs disabled Ard Biesheuvel
2021-01-19 16:00   ` Dave Martin
2021-01-19 16:29     ` Ard Biesheuvel
2021-01-20 15:44       ` Dave Martin
2021-02-15 18:30         ` Ard Biesheuvel
2020-12-18 17:01 ` [RFC PATCH 5/5] crypto: arm64/gcm-aes-ce - remove non-SIMD fallback path Ard Biesheuvel
2020-12-19  2:04 ` [RFC PATCH 0/5] running kernel mode SIMD with softirqs disabled Herbert Xu
2021-01-14  8:22   ` Ard Biesheuvel
2021-02-16 10:09 ` Peter Zijlstra
2021-02-16 10:35   ` Ard Biesheuvel

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-ARM-Kernel Archive on

Archives are clonable:
	git clone --mirror linux-arm-kernel/git/0.git
	git clone --mirror linux-arm-kernel/git/1.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-arm-kernel linux-arm-kernel/ \
	public-inbox-index linux-arm-kernel

Example config snippet for mirrors

Newsgroup available over NNTP:

AGPL code for this site: git clone