[PATCH 00/10] mm, arm64: Reduce ARCH_KMALLOC_MINALIGN below the cache line size

[PATCH 00/10] mm, arm64: Reduce ARCH_KMALLOC_MINALIGN below the cache line size

[Catalin Marinas]

Hi,

On arm64 ARCH_DMA_MINALIGN (and therefore ARCH_KMALLOC_MINALIGN) is 128.
While the majority of arm64 SoCs have a 64-byte cache line size (or
rather CWG - cache writeback granule), we chose a less than optimal
value in order to support all SoCs in a single kernel image.

The aim of this series is to allow smaller default ARCH_KMALLOC_MINALIGN
with kmalloc() caches configured at boot time to be safe when an SoC has
a larger DMA alignment requirement.

The first patch decouples ARCH_KMALLOC_MINALIGN from ARCH_DMA_MINALIGN
with the aim to only use the latter in DMA-specific compile-time
annotations. ARCH_KMALLOC_MINALIGN becomes the minimum (static)
guaranteed kmalloc() alignment but not necessarily safe for non-coherent
DMA. Patches 2-7 change some drivers/ code to use ARCH_DMA_MINALIGN
instead of ARCH_KMALLOC_MINALIGN.

Patch 8 introduces the dynamic arch_kmalloc_minalign() and the slab code
changes to set the corresponding minimum alignment on the newly created
kmalloc() caches. Patch 10 defines arch_kmalloc_minalign() for arm64
returning cache_line_size() together with reducing ARCH_KMALLOC_MINALIGN
to 64. ARCH_DMA_MINALIGN remains 128 on arm64.

I don't have access to it but there's the Fujitsu A64FX with a CWG of
256 (the arm64 cache_line_size() returns 256). This series will bump the
smallest kmalloc cache to kmalloc-256. The platform is known to be fully
cache coherent (or so I think) and we decided long ago not to bump
ARCH_DMA_MINALIGN to 256. If problematic, we could make the dynamic
kmalloc() alignment on arm64 min(ARCH_DMA_MINALIGN, cache_line_size()).

This series is beneficial to arm64 even if it's only reducing the
kmalloc() minimum alignment to 64. While it would be nice to reduce this
further to 8 (or 16) on SoCs known to be fully DMA coherent, detecting
this is via arch_setup_dma_ops() is problematic, especially with late
probed devices. I'd leave it for an additional RFC series on top of
this (there are ideas like bounce buffering for non-coherent devices if
the SoC was deemed coherent).

Thanks.

Catalin Marinas (10):
  mm/slab: Decouple ARCH_KMALLOC_MINALIGN from ARCH_DMA_MINALIGN
  drivers/base: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN
  drivers/gpu: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN
  drivers/md: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN
  drivers/spi: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN
  drivers/usb: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN
  crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN
  mm/slab: Allow dynamic kmalloc() minimum alignment
  mm/slab: Simplify create_kmalloc_cache() args and make it static
  arm64: Enable dynamic kmalloc() minimum alignment

 arch/arm64/include/asm/cache.h |  1 +
 arch/arm64/kernel/cacheinfo.c  |  7 ++++++
 drivers/base/devres.c          |  4 ++--
 drivers/gpu/drm/drm_managed.c  |  4 ++--
 drivers/md/dm-crypt.c          |  2 +-
 drivers/spi/spidev.c           |  2 +-
 drivers/usb/core/buffer.c      |  8 +++----
 drivers/usb/misc/usbtest.c     |  2 +-
 include/linux/crypto.h         |  2 +-
 include/linux/slab.h           | 25 ++++++++++++++++-----
 mm/slab.c                      |  6 +----
 mm/slab.h                      |  5 ++---
 mm/slab_common.c               | 40 ++++++++++++++++++++++------------
 13 files changed, 69 insertions(+), 39 deletions(-)

[PATCH 00/10] mm, arm64: Reduce ARCH_KMALLOC_MINALIGN below the cache line size

[Catalin Marinas]

Hi,

On arm64 ARCH_DMA_MINALIGN (and therefore ARCH_KMALLOC_MINALIGN) is 128.
While the majority of arm64 SoCs have a 64-byte cache line size (or
rather CWG - cache writeback granule), we chose a less than optimal
value in order to support all SoCs in a single kernel image.

The aim of this series is to allow smaller default ARCH_KMALLOC_MINALIGN
with kmalloc() caches configured at boot time to be safe when an SoC has
a larger DMA alignment requirement.

The first patch decouples ARCH_KMALLOC_MINALIGN from ARCH_DMA_MINALIGN
with the aim to only use the latter in DMA-specific compile-time
annotations. ARCH_KMALLOC_MINALIGN becomes the minimum (static)
guaranteed kmalloc() alignment but not necessarily safe for non-coherent
DMA. Patches 2-7 change some drivers/ code to use ARCH_DMA_MINALIGN
instead of ARCH_KMALLOC_MINALIGN.

Patch 8 introduces the dynamic arch_kmalloc_minalign() and the slab code
changes to set the corresponding minimum alignment on the newly created
kmalloc() caches. Patch 10 defines arch_kmalloc_minalign() for arm64
returning cache_line_size() together with reducing ARCH_KMALLOC_MINALIGN
to 64. ARCH_DMA_MINALIGN remains 128 on arm64.

I don't have access to it but there's the Fujitsu A64FX with a CWG of
256 (the arm64 cache_line_size() returns 256). This series will bump the
smallest kmalloc cache to kmalloc-256. The platform is known to be fully
cache coherent (or so I think) and we decided long ago not to bump
ARCH_DMA_MINALIGN to 256. If problematic, we could make the dynamic
kmalloc() alignment on arm64 min(ARCH_DMA_MINALIGN, cache_line_size()).

This series is beneficial to arm64 even if it's only reducing the
kmalloc() minimum alignment to 64. While it would be nice to reduce this
further to 8 (or 16) on SoCs known to be fully DMA coherent, detecting
this is via arch_setup_dma_ops() is problematic, especially with late
probed devices. I'd leave it for an additional RFC series on top of
this (there are ideas like bounce buffering for non-coherent devices if
the SoC was deemed coherent).

Thanks.

Catalin Marinas (10):
  mm/slab: Decouple ARCH_KMALLOC_MINALIGN from ARCH_DMA_MINALIGN
  drivers/base: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN
  drivers/gpu: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN
  drivers/md: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN
  drivers/spi: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN
  drivers/usb: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN
  crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN
  mm/slab: Allow dynamic kmalloc() minimum alignment
  mm/slab: Simplify create_kmalloc_cache() args and make it static
  arm64: Enable dynamic kmalloc() minimum alignment

 arch/arm64/include/asm/cache.h |  1 +
 arch/arm64/kernel/cacheinfo.c  |  7 ++++++
 drivers/base/devres.c          |  4 ++--
 drivers/gpu/drm/drm_managed.c  |  4 ++--
 drivers/md/dm-crypt.c          |  2 +-
 drivers/spi/spidev.c           |  2 +-
 drivers/usb/core/buffer.c      |  8 +++----
 drivers/usb/misc/usbtest.c     |  2 +-
 include/linux/crypto.h         |  2 +-
 include/linux/slab.h           | 25 ++++++++++++++++-----
 mm/slab.c                      |  6 +----
 mm/slab.h                      |  5 ++---
 mm/slab_common.c               | 40 ++++++++++++++++++++++------------
 13 files changed, 69 insertions(+), 39 deletions(-)


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH 01/10] mm/slab: Decouple ARCH_KMALLOC_MINALIGN from ARCH_DMA_MINALIGN

[Catalin Marinas]

In preparation for supporting a dynamic kmalloc() minimum alignment,
allow architectures to define ARCH_KMALLOC_MINALIGN independently of
ARCH_DMA_MINALIGN. In addition, always define ARCH_DMA_MINALIGN even if
an architecture does not override it.

After this patch, ARCH_DMA_MINALIGN is expected to be used in static
alignment annotations and defined by an architecture to be the maximum
alignment for all supported configurations/SoCs in a single Image.
ARCH_KMALLOC_MINALIGN, if different, is the minimum alignment guaranteed
by kmalloc().

Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
---
 include/linux/slab.h | 23 ++++++++++++++++++-----
 1 file changed, 18 insertions(+), 5 deletions(-)

diff --git a/include/linux/slab.h b/include/linux/slab.h
index 373b3ef99f4e..d58211bdeceb 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -187,17 +187,30 @@ bool kmem_valid_obj(void *object);
 void kmem_dump_obj(void *object);
 #endif
 
+/*
+ * slob does not support independent control of ARCH_KMALLOC_MINALIGN and
+ * ARCH_DMA_MINALIGN.
+ */
+#ifdef CONFIG_SLOB
+#undef ARCH_KMALLOC_MINALIGN
+#endif
+
 /*
  * Some archs want to perform DMA into kmalloc caches and need a guaranteed
  * alignment larger than the alignment of a 64-bit integer.
- * Setting ARCH_KMALLOC_MINALIGN in arch headers allows that.
+ * Setting ARCH_DMA_MINALIGN in arch headers allows that.
  */
-#if defined(ARCH_DMA_MINALIGN) && ARCH_DMA_MINALIGN > 8
+#ifndef ARCH_DMA_MINALIGN
+#define ARCH_DMA_MINALIGN __alignof__(unsigned long long)
+#elif ARCH_DMA_MINALIGN > 8 && !defined(ARCH_KMALLOC_MINALIGN)
 #define ARCH_KMALLOC_MINALIGN ARCH_DMA_MINALIGN
-#define KMALLOC_MIN_SIZE ARCH_DMA_MINALIGN
-#define KMALLOC_SHIFT_LOW ilog2(ARCH_DMA_MINALIGN)
-#else
+#endif
+
+#ifndef ARCH_KMALLOC_MINALIGN
 #define ARCH_KMALLOC_MINALIGN __alignof__(unsigned long long)
+#else
+#define KMALLOC_MIN_SIZE ARCH_KMALLOC_MINALIGN
+#define KMALLOC_SHIFT_LOW ilog2(KMALLOC_MIN_SIZE)
 #endif
 
 /*

[PATCH 01/10] mm/slab: Decouple ARCH_KMALLOC_MINALIGN from ARCH_DMA_MINALIGN

[Catalin Marinas]

In preparation for supporting a dynamic kmalloc() minimum alignment,
allow architectures to define ARCH_KMALLOC_MINALIGN independently of
ARCH_DMA_MINALIGN. In addition, always define ARCH_DMA_MINALIGN even if
an architecture does not override it.

After this patch, ARCH_DMA_MINALIGN is expected to be used in static
alignment annotations and defined by an architecture to be the maximum
alignment for all supported configurations/SoCs in a single Image.
ARCH_KMALLOC_MINALIGN, if different, is the minimum alignment guaranteed
by kmalloc().

Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
---
 include/linux/slab.h | 23 ++++++++++++++++++-----
 1 file changed, 18 insertions(+), 5 deletions(-)

diff --git a/include/linux/slab.h b/include/linux/slab.h
index 373b3ef99f4e..d58211bdeceb 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -187,17 +187,30 @@ bool kmem_valid_obj(void *object);
 void kmem_dump_obj(void *object);
 #endif
 
+/*
+ * slob does not support independent control of ARCH_KMALLOC_MINALIGN and
+ * ARCH_DMA_MINALIGN.
+ */
+#ifdef CONFIG_SLOB
+#undef ARCH_KMALLOC_MINALIGN
+#endif
+
 /*
  * Some archs want to perform DMA into kmalloc caches and need a guaranteed
  * alignment larger than the alignment of a 64-bit integer.
- * Setting ARCH_KMALLOC_MINALIGN in arch headers allows that.
+ * Setting ARCH_DMA_MINALIGN in arch headers allows that.
  */
-#if defined(ARCH_DMA_MINALIGN) && ARCH_DMA_MINALIGN > 8
+#ifndef ARCH_DMA_MINALIGN
+#define ARCH_DMA_MINALIGN __alignof__(unsigned long long)
+#elif ARCH_DMA_MINALIGN > 8 && !defined(ARCH_KMALLOC_MINALIGN)
 #define ARCH_KMALLOC_MINALIGN ARCH_DMA_MINALIGN
-#define KMALLOC_MIN_SIZE ARCH_DMA_MINALIGN
-#define KMALLOC_SHIFT_LOW ilog2(ARCH_DMA_MINALIGN)
-#else
+#endif
+
+#ifndef ARCH_KMALLOC_MINALIGN
 #define ARCH_KMALLOC_MINALIGN __alignof__(unsigned long long)
+#else
+#define KMALLOC_MIN_SIZE ARCH_KMALLOC_MINALIGN
+#define KMALLOC_SHIFT_LOW ilog2(KMALLOC_MIN_SIZE)
 #endif
 
 /*

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH 02/10] drivers/base: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA
operations while ARCH_KMALLOC_MINALIGN is the minimum kmalloc() objects
alignment.

Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: "Rafael J. Wysocki" <rafael@kernel.org>
---
 drivers/base/devres.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/base/devres.c b/drivers/base/devres.c
index 864d0b3f566e..0ed39464ad08 100644
--- a/drivers/base/devres.c
+++ b/drivers/base/devres.c
@@ -29,10 +29,10 @@ struct devres {
 	 * Some archs want to perform DMA into kmalloc caches
 	 * and need a guaranteed alignment larger than
 	 * the alignment of a 64-bit integer.
-	 * Thus we use ARCH_KMALLOC_MINALIGN here and get exactly the same
+	 * Thus we use ARCH_DMA_MINALIGN here and get at least the same
 	 * buffer alignment as if it was allocated by plain kmalloc().
 	 */
-	u8 __aligned(ARCH_KMALLOC_MINALIGN) data[];
+	u8 __aligned(ARCH_DMA_MINALIGN) data[];
 };
 
 struct devres_group {

[PATCH 02/10] drivers/base: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA
operations while ARCH_KMALLOC_MINALIGN is the minimum kmalloc() objects
alignment.

Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: "Rafael J. Wysocki" <rafael@kernel.org>
---
 drivers/base/devres.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/base/devres.c b/drivers/base/devres.c
index 864d0b3f566e..0ed39464ad08 100644
--- a/drivers/base/devres.c
+++ b/drivers/base/devres.c
@@ -29,10 +29,10 @@ struct devres {
 	 * Some archs want to perform DMA into kmalloc caches
 	 * and need a guaranteed alignment larger than
 	 * the alignment of a 64-bit integer.
-	 * Thus we use ARCH_KMALLOC_MINALIGN here and get exactly the same
+	 * Thus we use ARCH_DMA_MINALIGN here and get at least the same
 	 * buffer alignment as if it was allocated by plain kmalloc().
 	 */
-	u8 __aligned(ARCH_KMALLOC_MINALIGN) data[];
+	u8 __aligned(ARCH_DMA_MINALIGN) data[];
 };
 
 struct devres_group {

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH 03/10] drivers/gpu: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA
operations while ARCH_KMALLOC_MINALIGN is the minimum kmalloc() objects
alignment.

Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Cc: Daniel Vetter <daniel@ffwll.ch>
---
 drivers/gpu/drm/drm_managed.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/drm_managed.c b/drivers/gpu/drm/drm_managed.c
index 37d7db6223be..2d4c51f9d74e 100644
--- a/drivers/gpu/drm/drm_managed.c
+++ b/drivers/gpu/drm/drm_managed.c
@@ -48,10 +48,10 @@ struct drmres {
 	 * Some archs want to perform DMA into kmalloc caches
 	 * and need a guaranteed alignment larger than
 	 * the alignment of a 64-bit integer.
-	 * Thus we use ARCH_KMALLOC_MINALIGN here and get exactly the same
+	 * Thus we use ARCH_DMA_MINALIGN here and get at least the same
 	 * buffer alignment as if it was allocated by plain kmalloc().
 	 */
-	u8 __aligned(ARCH_KMALLOC_MINALIGN) data[];
+	u8 __aligned(ARCH_DMA_MINALIGN) data[];
 };
 
 static void free_dr(struct drmres *dr)

[PATCH 03/10] drivers/gpu: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA
operations while ARCH_KMALLOC_MINALIGN is the minimum kmalloc() objects
alignment.

Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Cc: Daniel Vetter <daniel@ffwll.ch>
---
 drivers/gpu/drm/drm_managed.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/drm_managed.c b/drivers/gpu/drm/drm_managed.c
index 37d7db6223be..2d4c51f9d74e 100644
--- a/drivers/gpu/drm/drm_managed.c
+++ b/drivers/gpu/drm/drm_managed.c
@@ -48,10 +48,10 @@ struct drmres {
 	 * Some archs want to perform DMA into kmalloc caches
 	 * and need a guaranteed alignment larger than
 	 * the alignment of a 64-bit integer.
-	 * Thus we use ARCH_KMALLOC_MINALIGN here and get exactly the same
+	 * Thus we use ARCH_DMA_MINALIGN here and get at least the same
 	 * buffer alignment as if it was allocated by plain kmalloc().
 	 */
-	u8 __aligned(ARCH_KMALLOC_MINALIGN) data[];
+	u8 __aligned(ARCH_DMA_MINALIGN) data[];
 };
 
 static void free_dr(struct drmres *dr)

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH 04/10] drivers/md: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA
operations while ARCH_KMALLOC_MINALIGN is the minimum kmalloc() objects
alignment.

Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Cc: Alasdair Kergon <agk@redhat.com>
Cc: Mike Snitzer <snitzer@kernel.org>
---
 drivers/md/dm-crypt.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/md/dm-crypt.c b/drivers/md/dm-crypt.c
index fb80539865d7..b6929dd1d283 100644
--- a/drivers/md/dm-crypt.c
+++ b/drivers/md/dm-crypt.c
@@ -3250,7 +3250,7 @@ static int crypt_ctr(struct dm_target *ti, unsigned int argc, char **argv)
 
 	cc->per_bio_data_size = ti->per_io_data_size =
 		ALIGN(sizeof(struct dm_crypt_io) + cc->dmreq_start + additional_req_size,
-		      ARCH_KMALLOC_MINALIGN);
+		      ARCH_DMA_MINALIGN);
 
 	ret = mempool_init(&cc->page_pool, BIO_MAX_VECS, crypt_page_alloc, crypt_page_free, cc);
 	if (ret) {

[PATCH 04/10] drivers/md: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA
operations while ARCH_KMALLOC_MINALIGN is the minimum kmalloc() objects
alignment.

Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Cc: Alasdair Kergon <agk@redhat.com>
Cc: Mike Snitzer <snitzer@kernel.org>
---
 drivers/md/dm-crypt.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/md/dm-crypt.c b/drivers/md/dm-crypt.c
index fb80539865d7..b6929dd1d283 100644
--- a/drivers/md/dm-crypt.c
+++ b/drivers/md/dm-crypt.c
@@ -3250,7 +3250,7 @@ static int crypt_ctr(struct dm_target *ti, unsigned int argc, char **argv)
 
 	cc->per_bio_data_size = ti->per_io_data_size =
 		ALIGN(sizeof(struct dm_crypt_io) + cc->dmreq_start + additional_req_size,
-		      ARCH_KMALLOC_MINALIGN);
+		      ARCH_DMA_MINALIGN);
 
 	ret = mempool_init(&cc->page_pool, BIO_MAX_VECS, crypt_page_alloc, crypt_page_free, cc);
 	if (ret) {

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH 05/10] drivers/spi: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA
operations while ARCH_KMALLOC_MINALIGN is the minimum kmalloc() objects
alignment.

Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Cc: Mark Brown <broonie@kernel.org>
---
 drivers/spi/spidev.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/spi/spidev.c b/drivers/spi/spidev.c
index 53a551714265..4baea157949b 100644
--- a/drivers/spi/spidev.c
+++ b/drivers/spi/spidev.c
@@ -227,7 +227,7 @@ static int spidev_message(struct spidev_data *spidev,
 		/* Ensure that also following allocations from rx_buf/tx_buf will meet
 		 * DMA alignment requirements.
 		 */
-		unsigned int len_aligned = ALIGN(u_tmp->len, ARCH_KMALLOC_MINALIGN);
+		unsigned int len_aligned = ALIGN(u_tmp->len, ARCH_DMA_MINALIGN);
 
 		k_tmp->len = u_tmp->len;
 

[PATCH 05/10] drivers/spi: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA
operations while ARCH_KMALLOC_MINALIGN is the minimum kmalloc() objects
alignment.

Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Cc: Mark Brown <broonie@kernel.org>
---
 drivers/spi/spidev.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/spi/spidev.c b/drivers/spi/spidev.c
index 53a551714265..4baea157949b 100644
--- a/drivers/spi/spidev.c
+++ b/drivers/spi/spidev.c
@@ -227,7 +227,7 @@ static int spidev_message(struct spidev_data *spidev,
 		/* Ensure that also following allocations from rx_buf/tx_buf will meet
 		 * DMA alignment requirements.
 		 */
-		unsigned int len_aligned = ALIGN(u_tmp->len, ARCH_KMALLOC_MINALIGN);
+		unsigned int len_aligned = ALIGN(u_tmp->len, ARCH_DMA_MINALIGN);
 
 		k_tmp->len = u_tmp->len;
 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH 06/10] drivers/usb: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA
operations while ARCH_KMALLOC_MINALIGN is the minimum kmalloc() objects
alignment.

Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
---
 drivers/usb/core/buffer.c  | 8 ++++----
 drivers/usb/misc/usbtest.c | 2 +-
 2 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/drivers/usb/core/buffer.c b/drivers/usb/core/buffer.c
index fbb087b728dc..e21d8d106977 100644
--- a/drivers/usb/core/buffer.c
+++ b/drivers/usb/core/buffer.c
@@ -34,13 +34,13 @@ void __init usb_init_pool_max(void)
 {
 	/*
 	 * The pool_max values must never be smaller than
-	 * ARCH_KMALLOC_MINALIGN.
+	 * ARCH_DMA_MINALIGN.
 	 */
-	if (ARCH_KMALLOC_MINALIGN <= 32)
+	if (ARCH_DMA_MINALIGN <= 32)
 		;			/* Original value is okay */
-	else if (ARCH_KMALLOC_MINALIGN <= 64)
+	else if (ARCH_DMA_MINALIGN <= 64)
 		pool_max[0] = 64;
-	else if (ARCH_KMALLOC_MINALIGN <= 128)
+	else if (ARCH_DMA_MINALIGN <= 128)
 		pool_max[0] = 0;	/* Don't use this pool */
 	else
 		BUILD_BUG();		/* We don't allow this */
diff --git a/drivers/usb/misc/usbtest.c b/drivers/usb/misc/usbtest.c
index 150090ee4ec1..e13f7f6904ff 100644
--- a/drivers/usb/misc/usbtest.c
+++ b/drivers/usb/misc/usbtest.c
@@ -378,7 +378,7 @@ static void simple_fill_buf(struct urb *urb)
 
 static inline unsigned long buffer_offset(void *buf)
 {
-	return (unsigned long)buf & (ARCH_KMALLOC_MINALIGN - 1);
+	return (unsigned long)buf & (ARCH_DMA_MINALIGN - 1);
 }
 
 static int check_guard_bytes(struct usbtest_dev *tdev, struct urb *urb)

[PATCH 06/10] drivers/usb: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA
operations while ARCH_KMALLOC_MINALIGN is the minimum kmalloc() objects
alignment.

Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
---
 drivers/usb/core/buffer.c  | 8 ++++----
 drivers/usb/misc/usbtest.c | 2 +-
 2 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/drivers/usb/core/buffer.c b/drivers/usb/core/buffer.c
index fbb087b728dc..e21d8d106977 100644
--- a/drivers/usb/core/buffer.c
+++ b/drivers/usb/core/buffer.c
@@ -34,13 +34,13 @@ void __init usb_init_pool_max(void)
 {
 	/*
 	 * The pool_max values must never be smaller than
-	 * ARCH_KMALLOC_MINALIGN.
+	 * ARCH_DMA_MINALIGN.
 	 */
-	if (ARCH_KMALLOC_MINALIGN <= 32)
+	if (ARCH_DMA_MINALIGN <= 32)
 		;			/* Original value is okay */
-	else if (ARCH_KMALLOC_MINALIGN <= 64)
+	else if (ARCH_DMA_MINALIGN <= 64)
 		pool_max[0] = 64;
-	else if (ARCH_KMALLOC_MINALIGN <= 128)
+	else if (ARCH_DMA_MINALIGN <= 128)
 		pool_max[0] = 0;	/* Don't use this pool */
 	else
 		BUILD_BUG();		/* We don't allow this */
diff --git a/drivers/usb/misc/usbtest.c b/drivers/usb/misc/usbtest.c
index 150090ee4ec1..e13f7f6904ff 100644
--- a/drivers/usb/misc/usbtest.c
+++ b/drivers/usb/misc/usbtest.c
@@ -378,7 +378,7 @@ static void simple_fill_buf(struct urb *urb)
 
 static inline unsigned long buffer_offset(void *buf)
 {
-	return (unsigned long)buf & (ARCH_KMALLOC_MINALIGN - 1);
+	return (unsigned long)buf & (ARCH_DMA_MINALIGN - 1);
 }
 
 static int check_guard_bytes(struct usbtest_dev *tdev, struct urb *urb)

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA
operations while ARCH_KMALLOC_MINALIGN is the minimum kmalloc() objects
alignment.

Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: "David S. Miller" <davem@davemloft.net>
---
 include/linux/crypto.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/include/linux/crypto.h b/include/linux/crypto.h
index 2324ab6f1846..654b9c355575 100644
--- a/include/linux/crypto.h
+++ b/include/linux/crypto.h
@@ -167,7 +167,7 @@
  * maintenance for non-coherent DMA (cache invalidation in particular) does not
  * affect data that may be accessed by the CPU concurrently.
  */
-#define CRYPTO_MINALIGN ARCH_KMALLOC_MINALIGN
+#define CRYPTO_MINALIGN ARCH_DMA_MINALIGN
 
 #define CRYPTO_MINALIGN_ATTR __attribute__ ((__aligned__(CRYPTO_MINALIGN)))
 

[PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA
operations while ARCH_KMALLOC_MINALIGN is the minimum kmalloc() objects
alignment.

Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: "David S. Miller" <davem@davemloft.net>
---
 include/linux/crypto.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/include/linux/crypto.h b/include/linux/crypto.h
index 2324ab6f1846..654b9c355575 100644
--- a/include/linux/crypto.h
+++ b/include/linux/crypto.h
@@ -167,7 +167,7 @@
  * maintenance for non-coherent DMA (cache invalidation in particular) does not
  * affect data that may be accessed by the CPU concurrently.
  */
-#define CRYPTO_MINALIGN ARCH_KMALLOC_MINALIGN
+#define CRYPTO_MINALIGN ARCH_DMA_MINALIGN
 
 #define CRYPTO_MINALIGN_ATTR __attribute__ ((__aligned__(CRYPTO_MINALIGN)))
 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH 08/10] mm/slab: Allow dynamic kmalloc() minimum alignment

[Catalin Marinas]

ARCH_KMALLOC_MINALIGN represents the minimum guaranteed kmalloc()
alignment but an architecture may require a larger run-time alignment.
Do not create kmalloc caches smaller than arch_kmalloc_minalign().

Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
---
 include/linux/slab.h |  2 ++
 mm/slab.c            |  6 +-----
 mm/slab.h            |  2 ++
 mm/slab_common.c     | 33 +++++++++++++++++++++++----------
 4 files changed, 28 insertions(+), 15 deletions(-)

diff --git a/include/linux/slab.h b/include/linux/slab.h
index d58211bdeceb..2137dba85691 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -332,6 +332,8 @@ enum kmalloc_cache_type {
 extern struct kmem_cache *
 kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_SHIFT_HIGH + 1];
 
+unsigned int arch_kmalloc_minalign(void);
+
 /*
  * Define gfp bits that should not be set for KMALLOC_NORMAL.
  */
diff --git a/mm/slab.c b/mm/slab.c
index b04e40078bdf..4aaeeb9c994d 100644
--- a/mm/slab.c
+++ b/mm/slab.c
@@ -1256,11 +1256,7 @@ void __init kmem_cache_init(void)
 	 * Initialize the caches that provide memory for the  kmem_cache_node
 	 * structures first.  Without this, further allocations will bug.
 	 */
-	kmalloc_caches[KMALLOC_NORMAL][INDEX_NODE] = create_kmalloc_cache(
-				kmalloc_info[INDEX_NODE].name[KMALLOC_NORMAL],
-				kmalloc_info[INDEX_NODE].size,
-				ARCH_KMALLOC_FLAGS, 0,
-				kmalloc_info[INDEX_NODE].size);
+	new_kmalloc_cache(INDEX_NODE, KMALLOC_NORMAL, ARCH_KMALLOC_FLAGS);
 	slab_state = PARTIAL_NODE;
 	setup_kmalloc_cache_index_table();
 
diff --git a/mm/slab.h b/mm/slab.h
index fd7ae2024897..e9238406602a 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -283,6 +283,8 @@ int __kmem_cache_create(struct kmem_cache *, slab_flags_t flags);
 struct kmem_cache *create_kmalloc_cache(const char *name, unsigned int size,
 			slab_flags_t flags, unsigned int useroffset,
 			unsigned int usersize);
+void __init new_kmalloc_cache(int idx, enum kmalloc_cache_type type,
+			      slab_flags_t flags);
 extern void create_boot_cache(struct kmem_cache *, const char *name,
 			unsigned int size, slab_flags_t flags,
 			unsigned int useroffset, unsigned int usersize);
diff --git a/mm/slab_common.c b/mm/slab_common.c
index 6ee64d6208b3..594d8a8a68d0 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -838,9 +838,18 @@ void __init setup_kmalloc_cache_index_table(void)
 	}
 }
 
-static void __init
+unsigned int __weak arch_kmalloc_minalign(void)
+{
+	return ARCH_KMALLOC_MINALIGN;
+}
+
+void __init
 new_kmalloc_cache(int idx, enum kmalloc_cache_type type, slab_flags_t flags)
 {
+	unsigned int minalign = arch_kmalloc_minalign();
+	unsigned int aligned_size = kmalloc_info[idx].size;
+	int aligned_idx = idx;
+
 	if (type == KMALLOC_RECLAIM) {
 		flags |= SLAB_RECLAIM_ACCOUNT;
 	} else if (IS_ENABLED(CONFIG_MEMCG_KMEM) && (type == KMALLOC_CGROUP)) {
@@ -851,10 +860,17 @@ new_kmalloc_cache(int idx, enum kmalloc_cache_type type, slab_flags_t flags)
 		flags |= SLAB_ACCOUNT;
 	}
 
-	kmalloc_caches[type][idx] = create_kmalloc_cache(
-					kmalloc_info[idx].name[type],
-					kmalloc_info[idx].size, flags, 0,
-					kmalloc_info[idx].size);
+	if (minalign > ARCH_KMALLOC_MINALIGN) {
+		aligned_size = ALIGN(aligned_size, minalign);
+		aligned_idx = __kmalloc_index(aligned_size, false);
+	}
+
+	if (!kmalloc_caches[type][aligned_idx])
+		kmalloc_caches[type][aligned_idx] = create_kmalloc_cache(
+					kmalloc_info[aligned_idx].name[type],
+					aligned_size, flags, 0, aligned_size);
+	if (idx != aligned_idx)
+		kmalloc_caches[type][idx] = kmalloc_caches[type][aligned_idx];
 
 	/*
 	 * If CONFIG_MEMCG_KMEM is enabled, disable cache merging for
@@ -904,11 +920,8 @@ void __init create_kmalloc_caches(slab_flags_t flags)
 		struct kmem_cache *s = kmalloc_caches[KMALLOC_NORMAL][i];
 
 		if (s) {
-			kmalloc_caches[KMALLOC_DMA][i] = create_kmalloc_cache(
-				kmalloc_info[i].name[KMALLOC_DMA],
-				kmalloc_info[i].size,
-				SLAB_CACHE_DMA | flags, 0,
-				kmalloc_info[i].size);
+			new_kmalloc_cache(i, KMALLOC_DMA,
+					  SLAB_CACHE_DMA | flags);
 		}
 	}
 #endif

[PATCH 08/10] mm/slab: Allow dynamic kmalloc() minimum alignment

[Catalin Marinas]

ARCH_KMALLOC_MINALIGN represents the minimum guaranteed kmalloc()
alignment but an architecture may require a larger run-time alignment.
Do not create kmalloc caches smaller than arch_kmalloc_minalign().

Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
---
 include/linux/slab.h |  2 ++
 mm/slab.c            |  6 +-----
 mm/slab.h            |  2 ++
 mm/slab_common.c     | 33 +++++++++++++++++++++++----------
 4 files changed, 28 insertions(+), 15 deletions(-)

diff --git a/include/linux/slab.h b/include/linux/slab.h
index d58211bdeceb..2137dba85691 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -332,6 +332,8 @@ enum kmalloc_cache_type {
 extern struct kmem_cache *
 kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_SHIFT_HIGH + 1];
 
+unsigned int arch_kmalloc_minalign(void);
+
 /*
  * Define gfp bits that should not be set for KMALLOC_NORMAL.
  */
diff --git a/mm/slab.c b/mm/slab.c
index b04e40078bdf..4aaeeb9c994d 100644
--- a/mm/slab.c
+++ b/mm/slab.c
@@ -1256,11 +1256,7 @@ void __init kmem_cache_init(void)
 	 * Initialize the caches that provide memory for the  kmem_cache_node
 	 * structures first.  Without this, further allocations will bug.
 	 */
-	kmalloc_caches[KMALLOC_NORMAL][INDEX_NODE] = create_kmalloc_cache(
-				kmalloc_info[INDEX_NODE].name[KMALLOC_NORMAL],
-				kmalloc_info[INDEX_NODE].size,
-				ARCH_KMALLOC_FLAGS, 0,
-				kmalloc_info[INDEX_NODE].size);
+	new_kmalloc_cache(INDEX_NODE, KMALLOC_NORMAL, ARCH_KMALLOC_FLAGS);
 	slab_state = PARTIAL_NODE;
 	setup_kmalloc_cache_index_table();
 
diff --git a/mm/slab.h b/mm/slab.h
index fd7ae2024897..e9238406602a 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -283,6 +283,8 @@ int __kmem_cache_create(struct kmem_cache *, slab_flags_t flags);
 struct kmem_cache *create_kmalloc_cache(const char *name, unsigned int size,
 			slab_flags_t flags, unsigned int useroffset,
 			unsigned int usersize);
+void __init new_kmalloc_cache(int idx, enum kmalloc_cache_type type,
+			      slab_flags_t flags);
 extern void create_boot_cache(struct kmem_cache *, const char *name,
 			unsigned int size, slab_flags_t flags,
 			unsigned int useroffset, unsigned int usersize);
diff --git a/mm/slab_common.c b/mm/slab_common.c
index 6ee64d6208b3..594d8a8a68d0 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -838,9 +838,18 @@ void __init setup_kmalloc_cache_index_table(void)
 	}
 }
 
-static void __init
+unsigned int __weak arch_kmalloc_minalign(void)
+{
+	return ARCH_KMALLOC_MINALIGN;
+}
+
+void __init
 new_kmalloc_cache(int idx, enum kmalloc_cache_type type, slab_flags_t flags)
 {
+	unsigned int minalign = arch_kmalloc_minalign();
+	unsigned int aligned_size = kmalloc_info[idx].size;
+	int aligned_idx = idx;
+
 	if (type == KMALLOC_RECLAIM) {
 		flags |= SLAB_RECLAIM_ACCOUNT;
 	} else if (IS_ENABLED(CONFIG_MEMCG_KMEM) && (type == KMALLOC_CGROUP)) {
@@ -851,10 +860,17 @@ new_kmalloc_cache(int idx, enum kmalloc_cache_type type, slab_flags_t flags)
 		flags |= SLAB_ACCOUNT;
 	}
 
-	kmalloc_caches[type][idx] = create_kmalloc_cache(
-					kmalloc_info[idx].name[type],
-					kmalloc_info[idx].size, flags, 0,
-					kmalloc_info[idx].size);
+	if (minalign > ARCH_KMALLOC_MINALIGN) {
+		aligned_size = ALIGN(aligned_size, minalign);
+		aligned_idx = __kmalloc_index(aligned_size, false);
+	}
+
+	if (!kmalloc_caches[type][aligned_idx])
+		kmalloc_caches[type][aligned_idx] = create_kmalloc_cache(
+					kmalloc_info[aligned_idx].name[type],
+					aligned_size, flags, 0, aligned_size);
+	if (idx != aligned_idx)
+		kmalloc_caches[type][idx] = kmalloc_caches[type][aligned_idx];
 
 	/*
 	 * If CONFIG_MEMCG_KMEM is enabled, disable cache merging for
@@ -904,11 +920,8 @@ void __init create_kmalloc_caches(slab_flags_t flags)
 		struct kmem_cache *s = kmalloc_caches[KMALLOC_NORMAL][i];
 
 		if (s) {
-			kmalloc_caches[KMALLOC_DMA][i] = create_kmalloc_cache(
-				kmalloc_info[i].name[KMALLOC_DMA],
-				kmalloc_info[i].size,
-				SLAB_CACHE_DMA | flags, 0,
-				kmalloc_info[i].size);
+			new_kmalloc_cache(i, KMALLOC_DMA,
+					  SLAB_CACHE_DMA | flags);
 		}
 	}
 #endif

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH 09/10] mm/slab: Simplify create_kmalloc_cache() args and make it static

[Catalin Marinas]

create_kmalloc_cache() is now only called from new_kmalloc_cache() in
the same file, so make it static. In addition, the useroffset argument
is always 0 while usersize is the same as size. Remove them.

Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
---
 mm/slab.h        | 3 ---
 mm/slab_common.c | 9 ++++-----
 2 files changed, 4 insertions(+), 8 deletions(-)

diff --git a/mm/slab.h b/mm/slab.h
index e9238406602a..d334b5e6d7aa 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -280,9 +280,6 @@ gfp_t kmalloc_fix_flags(gfp_t flags);
 /* Functions provided by the slab allocators */
 int __kmem_cache_create(struct kmem_cache *, slab_flags_t flags);
 
-struct kmem_cache *create_kmalloc_cache(const char *name, unsigned int size,
-			slab_flags_t flags, unsigned int useroffset,
-			unsigned int usersize);
 void __init new_kmalloc_cache(int idx, enum kmalloc_cache_type type,
 			      slab_flags_t flags);
 extern void create_boot_cache(struct kmem_cache *, const char *name,
diff --git a/mm/slab_common.c b/mm/slab_common.c
index 594d8a8a68d0..3a9f7a1e6490 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -655,16 +655,15 @@ void __init create_boot_cache(struct kmem_cache *s, const char *name,
 	s->refcount = -1;	/* Exempt from merging for now */
 }
 
-struct kmem_cache *__init create_kmalloc_cache(const char *name,
-		unsigned int size, slab_flags_t flags,
-		unsigned int useroffset, unsigned int usersize)
+static struct kmem_cache *__init create_kmalloc_cache(const char *name,
+		unsigned int size, slab_flags_t flags)
 {
 	struct kmem_cache *s = kmem_cache_zalloc(kmem_cache, GFP_NOWAIT);
 
 	if (!s)
 		panic("Out of memory when creating slab %s\n", name);
 
-	create_boot_cache(s, name, size, flags, useroffset, usersize);
+	create_boot_cache(s, name, size, flags, 0, size);
 	kasan_cache_create_kmalloc(s);
 	list_add(&s->list, &slab_caches);
 	s->refcount = 1;
@@ -868,7 +867,7 @@ new_kmalloc_cache(int idx, enum kmalloc_cache_type type, slab_flags_t flags)
 	if (!kmalloc_caches[type][aligned_idx])
 		kmalloc_caches[type][aligned_idx] = create_kmalloc_cache(
 					kmalloc_info[aligned_idx].name[type],
-					aligned_size, flags, 0, aligned_size);
+					aligned_size, flags);
 	if (idx != aligned_idx)
 		kmalloc_caches[type][idx] = kmalloc_caches[type][aligned_idx];
 

[PATCH 09/10] mm/slab: Simplify create_kmalloc_cache() args and make it static

[Catalin Marinas]

create_kmalloc_cache() is now only called from new_kmalloc_cache() in
the same file, so make it static. In addition, the useroffset argument
is always 0 while usersize is the same as size. Remove them.

Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
---
 mm/slab.h        | 3 ---
 mm/slab_common.c | 9 ++++-----
 2 files changed, 4 insertions(+), 8 deletions(-)

diff --git a/mm/slab.h b/mm/slab.h
index e9238406602a..d334b5e6d7aa 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -280,9 +280,6 @@ gfp_t kmalloc_fix_flags(gfp_t flags);
 /* Functions provided by the slab allocators */
 int __kmem_cache_create(struct kmem_cache *, slab_flags_t flags);
 
-struct kmem_cache *create_kmalloc_cache(const char *name, unsigned int size,
-			slab_flags_t flags, unsigned int useroffset,
-			unsigned int usersize);
 void __init new_kmalloc_cache(int idx, enum kmalloc_cache_type type,
 			      slab_flags_t flags);
 extern void create_boot_cache(struct kmem_cache *, const char *name,
diff --git a/mm/slab_common.c b/mm/slab_common.c
index 594d8a8a68d0..3a9f7a1e6490 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -655,16 +655,15 @@ void __init create_boot_cache(struct kmem_cache *s, const char *name,
 	s->refcount = -1;	/* Exempt from merging for now */
 }
 
-struct kmem_cache *__init create_kmalloc_cache(const char *name,
-		unsigned int size, slab_flags_t flags,
-		unsigned int useroffset, unsigned int usersize)
+static struct kmem_cache *__init create_kmalloc_cache(const char *name,
+		unsigned int size, slab_flags_t flags)
 {
 	struct kmem_cache *s = kmem_cache_zalloc(kmem_cache, GFP_NOWAIT);
 
 	if (!s)
 		panic("Out of memory when creating slab %s\n", name);
 
-	create_boot_cache(s, name, size, flags, useroffset, usersize);
+	create_boot_cache(s, name, size, flags, 0, size);
 	kasan_cache_create_kmalloc(s);
 	list_add(&s->list, &slab_caches);
 	s->refcount = 1;
@@ -868,7 +867,7 @@ new_kmalloc_cache(int idx, enum kmalloc_cache_type type, slab_flags_t flags)
 	if (!kmalloc_caches[type][aligned_idx])
 		kmalloc_caches[type][aligned_idx] = create_kmalloc_cache(
 					kmalloc_info[aligned_idx].name[type],
-					aligned_size, flags, 0, aligned_size);
+					aligned_size, flags);
 	if (idx != aligned_idx)
 		kmalloc_caches[type][idx] = kmalloc_caches[type][aligned_idx];
 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH 10/10] arm64: Enable dynamic kmalloc() minimum alignment

[Catalin Marinas]

Define ARCH_KMALLOC_MINALIGN as 64 since this would be the minimum
requirement across most arm64 SoCs. Define arch_kmalloc_minalign()
returning cache_line_size() to set the minimum run-time kmalloc()
alignment for those SoCs with bigger cache lines.

Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will@kernel.org>
---
 arch/arm64/include/asm/cache.h | 1 +
 arch/arm64/kernel/cacheinfo.c  | 7 +++++++
 2 files changed, 8 insertions(+)

diff --git a/arch/arm64/include/asm/cache.h b/arch/arm64/include/asm/cache.h
index a074459f8f2f..0bec986c9d51 100644
--- a/arch/arm64/include/asm/cache.h
+++ b/arch/arm64/include/asm/cache.h
@@ -48,6 +48,7 @@
  * the CPU.
  */
 #define ARCH_DMA_MINALIGN	(128)
+#define ARCH_KMALLOC_MINALIGN	(64)
 
 #ifdef CONFIG_KASAN_SW_TAGS
 #define ARCH_SLAB_MINALIGN	(1ULL << KASAN_SHADOW_SCALE_SHIFT)
diff --git a/arch/arm64/kernel/cacheinfo.c b/arch/arm64/kernel/cacheinfo.c
index 587543c6c51c..61211cd597f7 100644
--- a/arch/arm64/kernel/cacheinfo.c
+++ b/arch/arm64/kernel/cacheinfo.c
@@ -97,3 +97,10 @@ int populate_cache_leaves(unsigned int cpu)
 	}
 	return 0;
 }
+
+#ifndef CONFIG_SLOB
+unsigned int arch_kmalloc_minalign(void)
+{
+	return cache_line_size();
+}
+#endif

[PATCH 10/10] arm64: Enable dynamic kmalloc() minimum alignment

[Catalin Marinas]

Define ARCH_KMALLOC_MINALIGN as 64 since this would be the minimum
requirement across most arm64 SoCs. Define arch_kmalloc_minalign()
returning cache_line_size() to set the minimum run-time kmalloc()
alignment for those SoCs with bigger cache lines.

Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will@kernel.org>
---
 arch/arm64/include/asm/cache.h | 1 +
 arch/arm64/kernel/cacheinfo.c  | 7 +++++++
 2 files changed, 8 insertions(+)

diff --git a/arch/arm64/include/asm/cache.h b/arch/arm64/include/asm/cache.h
index a074459f8f2f..0bec986c9d51 100644
--- a/arch/arm64/include/asm/cache.h
+++ b/arch/arm64/include/asm/cache.h
@@ -48,6 +48,7 @@
  * the CPU.
  */
 #define ARCH_DMA_MINALIGN	(128)
+#define ARCH_KMALLOC_MINALIGN	(64)
 
 #ifdef CONFIG_KASAN_SW_TAGS
 #define ARCH_SLAB_MINALIGN	(1ULL << KASAN_SHADOW_SCALE_SHIFT)
diff --git a/arch/arm64/kernel/cacheinfo.c b/arch/arm64/kernel/cacheinfo.c
index 587543c6c51c..61211cd597f7 100644
--- a/arch/arm64/kernel/cacheinfo.c
+++ b/arch/arm64/kernel/cacheinfo.c
@@ -97,3 +97,10 @@ int populate_cache_leaves(unsigned int cpu)
 	}
 	return 0;
 }
+
+#ifndef CONFIG_SLOB
+unsigned int arch_kmalloc_minalign(void)
+{
+	return cache_line_size();
+}
+#endif

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 05/10] drivers/spi: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 270 bytes --]

On Tue, Apr 05, 2022 at 02:57:53PM +0100, Catalin Marinas wrote:
> ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA
> operations while ARCH_KMALLOC_MINALIGN is the minimum kmalloc() objects
> alignment.

Acked-by: Mark Brown <broonie@kernel.org>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH 05/10] drivers/spi: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Mark Brown]

[-- Attachment #1.1: Type: text/plain, Size: 270 bytes --]

On Tue, Apr 05, 2022 at 02:57:53PM +0100, Catalin Marinas wrote:
> ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA
> operations while ARCH_KMALLOC_MINALIGN is the minimum kmalloc() objects
> alignment.

Acked-by: Mark Brown <broonie@kernel.org>

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 176 bytes --]

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Herbert Xu]

On Tue, Apr 05, 2022 at 02:57:55PM +0100, Catalin Marinas wrote:
> ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA
> operations while ARCH_KMALLOC_MINALIGN is the minimum kmalloc() objects
> alignment.
> 
> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
> Cc: Herbert Xu <herbert@gondor.apana.org.au>
> Cc: "David S. Miller" <davem@davemloft.net>
> ---
>  include/linux/crypto.h | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/include/linux/crypto.h b/include/linux/crypto.h
> index 2324ab6f1846..654b9c355575 100644
> --- a/include/linux/crypto.h
> +++ b/include/linux/crypto.h
> @@ -167,7 +167,7 @@
>   * maintenance for non-coherent DMA (cache invalidation in particular) does not
>   * affect data that may be accessed by the CPU concurrently.
>   */
> -#define CRYPTO_MINALIGN ARCH_KMALLOC_MINALIGN
> +#define CRYPTO_MINALIGN ARCH_DMA_MINALIGN

I think this should remain as ARCH_KMALLOC_MINALIGN with the
comment above modified.  The reason is that we assume memory
returned by kmalloc is already aligned to this value.

Ard, you added the comment regarding the DMA requirement, so
does anything actually rely on this? If they do, they now need
to do their own alignment.

Thanks,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Herbert Xu]

On Tue, Apr 05, 2022 at 02:57:55PM +0100, Catalin Marinas wrote:
> ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA
> operations while ARCH_KMALLOC_MINALIGN is the minimum kmalloc() objects
> alignment.
> 
> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
> Cc: Herbert Xu <herbert@gondor.apana.org.au>
> Cc: "David S. Miller" <davem@davemloft.net>
> ---
>  include/linux/crypto.h | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/include/linux/crypto.h b/include/linux/crypto.h
> index 2324ab6f1846..654b9c355575 100644
> --- a/include/linux/crypto.h
> +++ b/include/linux/crypto.h
> @@ -167,7 +167,7 @@
>   * maintenance for non-coherent DMA (cache invalidation in particular) does not
>   * affect data that may be accessed by the CPU concurrently.
>   */
> -#define CRYPTO_MINALIGN ARCH_KMALLOC_MINALIGN
> +#define CRYPTO_MINALIGN ARCH_DMA_MINALIGN

I think this should remain as ARCH_KMALLOC_MINALIGN with the
comment above modified.  The reason is that we assume memory
returned by kmalloc is already aligned to this value.

Ard, you added the comment regarding the DMA requirement, so
does anything actually rely on this? If they do, they now need
to do their own alignment.

Thanks,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 01/10] mm/slab: Decouple ARCH_KMALLOC_MINALIGN from ARCH_DMA_MINALIGN

[Hyeonggon Yoo]

On Tue, Apr 05, 2022 at 02:57:49PM +0100, Catalin Marinas wrote:
> In preparation for supporting a dynamic kmalloc() minimum alignment,
> allow architectures to define ARCH_KMALLOC_MINALIGN independently of
> ARCH_DMA_MINALIGN. In addition, always define ARCH_DMA_MINALIGN even if
> an architecture does not override it.
>

[ +Cc slab maintainer/reviewers ]

I get why you want to set minimum alignment of kmalloc() dynamically.
That's because cache line size can be different and we cannot statically
know that, right?

But I don't get why you are trying to decouple ARCH_KMALLOC_MINALIGN
from ARCH_DMA_MINALIGN. kmalloc'ed buffer is always supposed to be DMA-safe.

I'm afraid this series may break some archs/drivers.

in Documentation/dma-api-howto.rst:
> 2) ARCH_DMA_MINALIGN
> 
>   Architectures must ensure that kmalloc'ed buffer is
>   DMA-safe. Drivers and subsystems depend on it. If an architecture
>   isn't fully DMA-coherent (i.e. hardware doesn't ensure that data in
>   the CPU cache is identical to data in main memory),
>   ARCH_DMA_MINALIGN must be set so that the memory allocator
>   makes sure that kmalloc'ed buffer doesn't share a cache line with
>   the others. See arch/arm/include/asm/cache.h as an example.
>
>   Note that ARCH_DMA_MINALIGN is about DMA memory alignment
>   constraints. You don't need to worry about the architecture data
>   alignment constraints (e.g. the alignment constraints about 64-bit
>   objects).

If I'm missing something, please let me know :)

Thanks,
Hyeonggon

> After this patch, ARCH_DMA_MINALIGN is expected to be used in static
> alignment annotations and defined by an architecture to be the maximum
> alignment for all supported configurations/SoCs in a single Image.
> ARCH_KMALLOC_MINALIGN, if different, is the minimum alignment guaranteed
> by kmalloc().
> 
> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
> Cc: Andrew Morton <akpm@linux-foundation.org>
> ---
>  include/linux/slab.h | 23 ++++++++++++++++++-----
>  1 file changed, 18 insertions(+), 5 deletions(-)
> 
> diff --git a/include/linux/slab.h b/include/linux/slab.h
> index 373b3ef99f4e..d58211bdeceb 100644
> --- a/include/linux/slab.h
> +++ b/include/linux/slab.h
> @@ -187,17 +187,30 @@ bool kmem_valid_obj(void *object);
>  void kmem_dump_obj(void *object);
>  #endif
>  
> +/*
> + * slob does not support independent control of ARCH_KMALLOC_MINALIGN and
> + * ARCH_DMA_MINALIGN.
> + */
> +#ifdef CONFIG_SLOB
> +#undef ARCH_KMALLOC_MINALIGN
> +#endif
> +
>  /*
>   * Some archs want to perform DMA into kmalloc caches and need a guaranteed
>   * alignment larger than the alignment of a 64-bit integer.
> - * Setting ARCH_KMALLOC_MINALIGN in arch headers allows that.
> + * Setting ARCH_DMA_MINALIGN in arch headers allows that.
>   */
> -#if defined(ARCH_DMA_MINALIGN) && ARCH_DMA_MINALIGN > 8
> +#ifndef ARCH_DMA_MINALIGN
> +#define ARCH_DMA_MINALIGN __alignof__(unsigned long long)
> +#elif ARCH_DMA_MINALIGN > 8 && !defined(ARCH_KMALLOC_MINALIGN)
>  #define ARCH_KMALLOC_MINALIGN ARCH_DMA_MINALIGN
> -#define KMALLOC_MIN_SIZE ARCH_DMA_MINALIGN
> -#define KMALLOC_SHIFT_LOW ilog2(ARCH_DMA_MINALIGN)
> -#else
> +#endif
> +
> +#ifndef ARCH_KMALLOC_MINALIGN
>  #define ARCH_KMALLOC_MINALIGN __alignof__(unsigned long long)
> +#else
> +#define KMALLOC_MIN_SIZE ARCH_KMALLOC_MINALIGN
> +#define KMALLOC_SHIFT_LOW ilog2(KMALLOC_MIN_SIZE)
>  #endif
>  
>  /*
> 

Re: [PATCH 01/10] mm/slab: Decouple ARCH_KMALLOC_MINALIGN from ARCH_DMA_MINALIGN

[Hyeonggon Yoo]

On Tue, Apr 05, 2022 at 02:57:49PM +0100, Catalin Marinas wrote:
> In preparation for supporting a dynamic kmalloc() minimum alignment,
> allow architectures to define ARCH_KMALLOC_MINALIGN independently of
> ARCH_DMA_MINALIGN. In addition, always define ARCH_DMA_MINALIGN even if
> an architecture does not override it.
>

[ +Cc slab maintainer/reviewers ]

I get why you want to set minimum alignment of kmalloc() dynamically.
That's because cache line size can be different and we cannot statically
know that, right?

But I don't get why you are trying to decouple ARCH_KMALLOC_MINALIGN
from ARCH_DMA_MINALIGN. kmalloc'ed buffer is always supposed to be DMA-safe.

I'm afraid this series may break some archs/drivers.

in Documentation/dma-api-howto.rst:
> 2) ARCH_DMA_MINALIGN
> 
>   Architectures must ensure that kmalloc'ed buffer is
>   DMA-safe. Drivers and subsystems depend on it. If an architecture
>   isn't fully DMA-coherent (i.e. hardware doesn't ensure that data in
>   the CPU cache is identical to data in main memory),
>   ARCH_DMA_MINALIGN must be set so that the memory allocator
>   makes sure that kmalloc'ed buffer doesn't share a cache line with
>   the others. See arch/arm/include/asm/cache.h as an example.
>
>   Note that ARCH_DMA_MINALIGN is about DMA memory alignment
>   constraints. You don't need to worry about the architecture data
>   alignment constraints (e.g. the alignment constraints about 64-bit
>   objects).

If I'm missing something, please let me know :)

Thanks,
Hyeonggon

> After this patch, ARCH_DMA_MINALIGN is expected to be used in static
> alignment annotations and defined by an architecture to be the maximum
> alignment for all supported configurations/SoCs in a single Image.
> ARCH_KMALLOC_MINALIGN, if different, is the minimum alignment guaranteed
> by kmalloc().
> 
> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
> Cc: Andrew Morton <akpm@linux-foundation.org>
> ---
>  include/linux/slab.h | 23 ++++++++++++++++++-----
>  1 file changed, 18 insertions(+), 5 deletions(-)
> 
> diff --git a/include/linux/slab.h b/include/linux/slab.h
> index 373b3ef99f4e..d58211bdeceb 100644
> --- a/include/linux/slab.h
> +++ b/include/linux/slab.h
> @@ -187,17 +187,30 @@ bool kmem_valid_obj(void *object);
>  void kmem_dump_obj(void *object);
>  #endif
>  
> +/*
> + * slob does not support independent control of ARCH_KMALLOC_MINALIGN and
> + * ARCH_DMA_MINALIGN.
> + */
> +#ifdef CONFIG_SLOB
> +#undef ARCH_KMALLOC_MINALIGN
> +#endif
> +
>  /*
>   * Some archs want to perform DMA into kmalloc caches and need a guaranteed
>   * alignment larger than the alignment of a 64-bit integer.
> - * Setting ARCH_KMALLOC_MINALIGN in arch headers allows that.
> + * Setting ARCH_DMA_MINALIGN in arch headers allows that.
>   */
> -#if defined(ARCH_DMA_MINALIGN) && ARCH_DMA_MINALIGN > 8
> +#ifndef ARCH_DMA_MINALIGN
> +#define ARCH_DMA_MINALIGN __alignof__(unsigned long long)
> +#elif ARCH_DMA_MINALIGN > 8 && !defined(ARCH_KMALLOC_MINALIGN)
>  #define ARCH_KMALLOC_MINALIGN ARCH_DMA_MINALIGN
> -#define KMALLOC_MIN_SIZE ARCH_DMA_MINALIGN
> -#define KMALLOC_SHIFT_LOW ilog2(ARCH_DMA_MINALIGN)
> -#else
> +#endif
> +
> +#ifndef ARCH_KMALLOC_MINALIGN
>  #define ARCH_KMALLOC_MINALIGN __alignof__(unsigned long long)
> +#else
> +#define KMALLOC_MIN_SIZE ARCH_KMALLOC_MINALIGN
> +#define KMALLOC_SHIFT_LOW ilog2(KMALLOC_MIN_SIZE)
>  #endif
>  
>  /*
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 01/10] mm/slab: Decouple ARCH_KMALLOC_MINALIGN from ARCH_DMA_MINALIGN

[kernel test robot]

Hi Catalin,

I love your patch! Perhaps something to improve:

[auto build test WARNING on hnaz-mm/master]

url:    https://github.com/intel-lab-lkp/linux/commits/Catalin-Marinas/mm-arm64-Reduce-ARCH_KMALLOC_MINALIGN-below-the-cache-line-size/20220405-220827
base:   https://github.com/hnaz/linux-mm master
config: riscv-buildonly-randconfig-r004-20220405 (https://download.01.org/0day-ci/archive/20220406/202204060957.sHGvCxYf-lkp@intel.com/config)
compiler: clang version 15.0.0 (https://github.com/llvm/llvm-project c4a1b07d0979e7ff20d7d541af666d822d66b566)
reproduce (this is a W=1 build):
        wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # install riscv cross compiling tool for clang build
        # apt-get install binutils-riscv64-linux-gnu
        # https://github.com/intel-lab-lkp/linux/commit/4b3dfcfcd58dc57d321559e467e85ce909539755
        git remote add linux-review https://github.com/intel-lab-lkp/linux
        git fetch --no-tags linux-review Catalin-Marinas/mm-arm64-Reduce-ARCH_KMALLOC_MINALIGN-below-the-cache-line-size/20220405-220827
        git checkout 4b3dfcfcd58dc57d321559e467e85ce909539755
        # save the config file to linux build tree
        mkdir build_dir
        COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 O=build_dir ARCH=riscv SHELL=/bin/bash

If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>

All warnings (new ones prefixed by >>, old ones prefixed by <<):

>> WARNING: modpost: vmlinux.o(.text+0xf8244): Section mismatch in reference from the function perf_swevent_set_period() to the function .init.text:set_reset_devices()
The function perf_swevent_set_period() references
the function __init set_reset_devices().
This is often because perf_swevent_set_period lacks a __init
annotation or the annotation of set_reset_devices is wrong.
--
>> WARNING: modpost: vmlinux.o(.text+0x18d1dc): Section mismatch in reference from the function generic_update_time() to the function .init.text:set_reset_devices()
The function generic_update_time() references
the function __init set_reset_devices().
This is often because generic_update_time lacks a __init
annotation or the annotation of set_reset_devices is wrong.
--
>> WARNING: modpost: vmlinux.o(.text+0x29fd1e): Section mismatch in reference from the function fat_remove_entries() to the function .init.text:set_reset_devices()
The function fat_remove_entries() references
the function __init set_reset_devices().
This is often because fat_remove_entries lacks a __init
annotation or the annotation of set_reset_devices is wrong.
--
>> WARNING: modpost: vmlinux.o(.text+0x2a46b0): Section mismatch in reference from the function fat_fill_inode() to the function .init.text:set_reset_devices()
The function fat_fill_inode() references
the function __init set_reset_devices().
This is often because fat_fill_inode lacks a __init
annotation or the annotation of set_reset_devices is wrong.
--
>> WARNING: modpost: vmlinux.o(.text+0x2a5e8a): Section mismatch in reference from the function fat_read_root() to the function .init.text:set_reset_devices()
The function fat_read_root() references
the function __init set_reset_devices().
This is often because fat_read_root lacks a __init
annotation or the annotation of set_reset_devices is wrong.
--
>> WARNING: modpost: vmlinux.o(.text+0x2a740e): Section mismatch in reference from the function fat_update_time() to the function .init.text:set_reset_devices()
The function fat_update_time() references
the function __init set_reset_devices().
This is often because fat_update_time lacks a __init
annotation or the annotation of set_reset_devices is wrong.
--
>> WARNING: modpost: vmlinux.o(.text+0x2a82aa): Section mismatch in reference from the function msdos_rename() to the function .init.text:set_reset_devices()
The function msdos_rename() references
the function __init set_reset_devices().
This is often because msdos_rename lacks a __init
annotation or the annotation of set_reset_devices is wrong.
--
>> WARNING: modpost: vmlinux.o(.text+0x2a8a1a): Section mismatch in reference from the function inode_inc_iversion() to the function .init.text:set_reset_devices()
The function inode_inc_iversion() references
the function __init set_reset_devices().
This is often because inode_inc_iversion lacks a __init
annotation or the annotation of set_reset_devices is wrong.
--
>> WARNING: modpost: vmlinux.o(.text+0x2aa83c): Section mismatch in reference from the function exfat_lookup() to the function .init.text:set_reset_devices()
The function exfat_lookup() references
the function __init set_reset_devices().
This is often because exfat_lookup lacks a __init
annotation or the annotation of set_reset_devices is wrong.
--
>> WARNING: modpost: vmlinux.o(.text+0x2aacd4): Section mismatch in reference from the function exfat_create() to the function .init.text:set_reset_devices()
The function exfat_create() references
the function __init set_reset_devices().
This is often because exfat_create lacks a __init
annotation or the annotation of set_reset_devices is wrong.
--
>> WARNING: modpost: vmlinux.o(.text+0x228148): Section mismatch in reference from the function ext4_readdir() to the function .init.text:set_reset_devices()
The function ext4_readdir() references
the function __init set_reset_devices().
This is often because ext4_readdir lacks a __init
annotation or the annotation of set_reset_devices is wrong.
..

Note: the below error/warnings can be found in parent commit:
<< WARNING: modpost: vmlinux.o(.text+0xb2412): Section mismatch in reference from the function bpf_check_uarg_tail_zero() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb24b8): Section mismatch in reference from the function bpf_map_area_alloc() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5b16): Section mismatch in reference from the function bpf_obj_pin() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5b54): Section mismatch in reference from the function bpf_obj_get() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5ba4): Section mismatch in reference from the function bpf_prog_attach() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5d2e): Section mismatch in reference from the function bpf_prog_detach() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5db8): Section mismatch in reference from the function bpf_prog_query() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5e38): Section mismatch in reference from the function bpf_prog_test_run() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5f1c): Section mismatch in reference from the function bpf_obj_get_next_id() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5fba): Section mismatch in reference from the function bpf_prog_get_fd_by_id() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb2412): Section mismatch in reference from the function bpf_check_uarg_tail_zero() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb24b8): Section mismatch in reference from the function bpf_map_area_alloc() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5b16): Section mismatch in reference from the function bpf_obj_pin() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5b54): Section mismatch in reference from the function bpf_obj_get() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5ba4): Section mismatch in reference from the function bpf_prog_attach() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5d2e): Section mismatch in reference from the function bpf_prog_detach() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5db8): Section mismatch in reference from the function bpf_prog_query() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5e38): Section mismatch in reference from the function bpf_prog_test_run() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5f1c): Section mismatch in reference from the function bpf_obj_get_next_id() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5fba): Section mismatch in reference from the function bpf_prog_get_fd_by_id() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb2412): Section mismatch in reference from the function bpf_check_uarg_tail_zero() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb24b8): Section mismatch in reference from the function bpf_map_area_alloc() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5b16): Section mismatch in reference from the function bpf_obj_pin() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5b54): Section mismatch in reference from the function bpf_obj_get() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5ba4): Section mismatch in reference from the function bpf_prog_attach() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5d2e): Section mismatch in reference from the function bpf_prog_detach() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5db8): Section mismatch in reference from the function bpf_prog_query() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5e38): Section mismatch in reference from the function bpf_prog_test_run() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5f1c): Section mismatch in reference from the function bpf_obj_get_next_id() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5fba): Section mismatch in reference from the function bpf_prog_get_fd_by_id() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb2412): Section mismatch in reference from the function bpf_check_uarg_tail_zero() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb24b8): Section mismatch in reference from the function bpf_map_area_alloc() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5b16): Section mismatch in reference from the function bpf_obj_pin() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5b54): Section mismatch in reference from the function bpf_obj_get() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5ba4): Section mismatch in reference from the function bpf_prog_attach() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5d2e): Section mismatch in reference from the function bpf_prog_detach() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5db8): Section mismatch in reference from the function bpf_prog_query() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5e38): Section mismatch in reference from the function bpf_prog_test_run() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5f1c): Section mismatch in reference from the function bpf_obj_get_next_id() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5fba): Section mismatch in reference from the function bpf_prog_get_fd_by_id() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb2412): Section mismatch in reference from the function bpf_check_uarg_tail_zero() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb24b8): Section mismatch in reference from the function bpf_map_area_alloc() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5b16): Section mismatch in reference from the function bpf_obj_pin() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5b54): Section mismatch in reference from the function bpf_obj_get() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5ba4): Section mismatch in reference from the function bpf_prog_attach() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5d2e): Section mismatch in reference from the function bpf_prog_detach() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5db8): Section mismatch in reference from the function bpf_prog_query() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5e38): Section mismatch in reference from the function bpf_prog_test_run() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5f1c): Section mismatch in reference from the function bpf_obj_get_next_id() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5fba): Section mismatch in reference from the function bpf_prog_get_fd_by_id() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb2412): Section mismatch in reference from the function bpf_check_uarg_tail_zero() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb24b8): Section mismatch in reference from the function bpf_map_area_alloc() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5b16): Section mismatch in reference from the function bpf_obj_pin() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5b54): Section mismatch in reference from the function bpf_obj_get() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5ba4): Section mismatch in reference from the function bpf_prog_attach() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5d2e): Section mismatch in reference from the function bpf_prog_detach() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5db8): Section mismatch in reference from the function bpf_prog_query() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5e38): Section mismatch in reference from the function bpf_prog_test_run() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5f1c): Section mismatch in reference from the function bpf_obj_get_next_id() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5fba): Section mismatch in reference from the function bpf_prog_get_fd_by_id() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb2412): Section mismatch in reference from the function bpf_check_uarg_tail_zero() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb24b8): Section mismatch in reference from the function bpf_map_area_alloc() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5b16): Section mismatch in reference from the function bpf_obj_pin() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5b54): Section mismatch in reference from the function bpf_obj_get() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5ba4): Section mismatch in reference from the function bpf_prog_attach() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5d2e): Section mismatch in reference from the function bpf_prog_detach() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5db8): Section mismatch in reference from the function bpf_prog_query() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5e38): Section mismatch in reference from the function bpf_prog_test_run() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5f1c): Section mismatch in reference from the function bpf_obj_get_next_id() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5fba): Section mismatch in reference from the function bpf_prog_get_fd_by_id() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb2412): Section mismatch in reference from the function bpf_check_uarg_tail_zero() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb24b8): Section mismatch in reference from the function bpf_map_area_alloc() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5b16): Section mismatch in reference from the function bpf_obj_pin() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5b54): Section mismatch in reference from the function bpf_obj_get() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5ba4): Section mismatch in reference from the function bpf_prog_attach() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5d2e): Section mismatch in reference from the function bpf_prog_detach() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5db8): Section mismatch in reference from the function bpf_prog_query() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5e38): Section mismatch in reference from the function bpf_prog_test_run() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5f1c): Section mismatch in reference from the function bpf_obj_get_next_id() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5fba): Section mismatch in reference from the function bpf_prog_get_fd_by_id() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb2412): Section mismatch in reference from the function bpf_check_uarg_tail_zero() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb24b8): Section mismatch in reference from the function bpf_map_area_alloc() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5b16): Section mismatch in reference from the function bpf_obj_pin() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5b54): Section mismatch in reference from the function bpf_obj_get() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5ba4): Section mismatch in reference from the function bpf_prog_attach() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5d2e): Section mismatch in reference from the function bpf_prog_detach() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5db8): Section mismatch in reference from the function bpf_prog_query() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5e38): Section mismatch in reference from the function bpf_prog_test_run() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5f1c): Section mismatch in reference from the function bpf_obj_get_next_id() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5fba): Section mismatch in reference from the function bpf_prog_get_fd_by_id() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb2412): Section mismatch in reference from the function bpf_check_uarg_tail_zero() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb24b8): Section mismatch in reference from the function bpf_map_area_alloc() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5b16): Section mismatch in reference from the function bpf_obj_pin() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5b54): Section mismatch in reference from the function bpf_obj_get() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5ba4): Section mismatch in reference from the function bpf_prog_attach() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5d2e): Section mismatch in reference from the function bpf_prog_detach() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5db8): Section mismatch in reference from the function bpf_prog_query() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5e38): Section mismatch in reference from the function bpf_prog_test_run() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5f1c): Section mismatch in reference from the function bpf_obj_get_next_id() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5fba): Section mismatch in reference from the function bpf_prog_get_fd_by_id() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb2412): Section mismatch in reference from the function bpf_check_uarg_tail_zero() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb24b8): Section mismatch in reference from the function bpf_map_area_alloc() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5b16): Section mismatch in reference from the function bpf_obj_pin() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5b54): Section mismatch in reference from the function bpf_obj_get() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5ba4): Section mismatch in reference from the function bpf_prog_attach() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5d2e): Section mismatch in reference from the function bpf_prog_detach() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5db8): Section mismatch in reference from the function bpf_prog_query() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5e38): Section mismatch in reference from the function bpf_prog_test_run() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5f1c): Section mismatch in reference from the function bpf_obj_get_next_id() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0xb5fba): Section mismatch in reference from the function bpf_prog_get_fd_by_id() to the function .init.text:set_reset_devices()

-- 
0-DAY CI Kernel Test Service
https://01.org/lkp

Re: [PATCH 09/10] mm/slab: Simplify create_kmalloc_cache() args and make it static

[kernel test robot]

Hi Catalin,

I love your patch! Perhaps something to improve:

[auto build test WARNING on hnaz-mm/master]

url:    https://github.com/intel-lab-lkp/linux/commits/Catalin-Marinas/mm-arm64-Reduce-ARCH_KMALLOC_MINALIGN-below-the-cache-line-size/20220405-220827
base:   https://github.com/hnaz/linux-mm master
config: riscv-buildonly-randconfig-r004-20220405 (https://download.01.org/0day-ci/archive/20220406/202204061348.mo7QK7AD-lkp@intel.com/config)
compiler: clang version 15.0.0 (https://github.com/llvm/llvm-project c4a1b07d0979e7ff20d7d541af666d822d66b566)
reproduce (this is a W=1 build):
        wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # install riscv cross compiling tool for clang build
        # apt-get install binutils-riscv64-linux-gnu
        # https://github.com/intel-lab-lkp/linux/commit/594fc15f21ce6de4d37971f717ff4223ab271823
        git remote add linux-review https://github.com/intel-lab-lkp/linux
        git fetch --no-tags linux-review Catalin-Marinas/mm-arm64-Reduce-ARCH_KMALLOC_MINALIGN-below-the-cache-line-size/20220405-220827
        git checkout 594fc15f21ce6de4d37971f717ff4223ab271823
        # save the config file to linux build tree
        mkdir build_dir
        COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 O=build_dir ARCH=riscv SHELL=/bin/bash

If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>

All warnings (new ones prefixed by >>, old ones prefixed by <<):

>> WARNING: modpost: vmlinux.o(.data+0x2836c8): Section mismatch in reference from the variable sctp_prot to the variable .init.text:.LBB19_63
The variable sctp_prot references
the variable __init .LBB19_63
If the reference is valid then annotate the
variable with or __refdata (see linux/init.h) or name the variable:


Note: the below error/warnings can be found in parent commit:
<< WARNING: modpost: vmlinux.o(.text+0xf8244): Section mismatch in reference from the function perf_swevent_set_period() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0x18d1e0): Section mismatch in reference from the function generic_update_time() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0x29fd22): Section mismatch in reference from the function fat_remove_entries() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0x2a46b4): Section mismatch in reference from the function fat_fill_inode() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0x2a5e8e): Section mismatch in reference from the function fat_read_root() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0x2a7412): Section mismatch in reference from the function fat_update_time() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0x2a82ae): Section mismatch in reference from the function msdos_rename() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0x2a8a1e): Section mismatch in reference from the function inode_inc_iversion() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0x2aa840): Section mismatch in reference from the function exfat_lookup() to the function .init.text:set_reset_devices()
<< WARNING: modpost: vmlinux.o(.text+0x2aacd8): Section mismatch in reference from the function exfat_create() to the function .init.text:set_reset_devices()

-- 
0-DAY CI Kernel Test Service
https://01.org/lkp

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Ard Biesheuvel]

On Wed, 6 Apr 2022 at 00:57, Herbert Xu <herbert@gondor.apana.org.au> wrote:
>
> On Tue, Apr 05, 2022 at 02:57:55PM +0100, Catalin Marinas wrote:
> > ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA
> > operations while ARCH_KMALLOC_MINALIGN is the minimum kmalloc() objects
> > alignment.
> >
> > Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
> > Cc: Herbert Xu <herbert@gondor.apana.org.au>
> > Cc: "David S. Miller" <davem@davemloft.net>
> > ---
> >  include/linux/crypto.h | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/include/linux/crypto.h b/include/linux/crypto.h
> > index 2324ab6f1846..654b9c355575 100644
> > --- a/include/linux/crypto.h
> > +++ b/include/linux/crypto.h
> > @@ -167,7 +167,7 @@
> >   * maintenance for non-coherent DMA (cache invalidation in particular) does not
> >   * affect data that may be accessed by the CPU concurrently.
> >   */
> > -#define CRYPTO_MINALIGN ARCH_KMALLOC_MINALIGN
> > +#define CRYPTO_MINALIGN ARCH_DMA_MINALIGN
>
> I think this should remain as ARCH_KMALLOC_MINALIGN with the
> comment above modified.  The reason is that we assume memory
> returned by kmalloc is already aligned to this value.
>
> Ard, you added the comment regarding the DMA requirement, so
> does anything actually rely on this? If they do, they now need
> to do their own alignment.
>

This patch looks incorrect to me, as ARCH_DMA_MINALIGN is not
#define'd on all architectures.

But I am fine with the intent: ARCH_DMA_MINALIGN will be >=
ARCH_KMALLOC_MINALIGN, and so the compile time layout of structs will
take the worst cast minimum DMA alignment into account, whereas their
placement in memory when they allocated dynamically may be aligned to
ARCH_KMALLOC_MINALIGN only. Since the latter will be based on the
actual cache geometry, this should be fine.

Apart from the 'shash desc on stack' issue solved by the patch that
also introduced the above comment(660d2062190d), I've never looked
into the actual memory footprint of the crypto related data structures
resulting from this alignment, but it seems to me that /if/ this is
significant, we should be able to punt this to the drivers that
actually need this, rather than impose it for the whole system. (This
would involve over-allocating the context struct, and aligning up the
pointer in the various xxx_ctx() getters iff needed by the driver in
question)

I'll put this on my list of things to look at.

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Ard Biesheuvel]

On Wed, 6 Apr 2022 at 00:57, Herbert Xu <herbert@gondor.apana.org.au> wrote:
>
> On Tue, Apr 05, 2022 at 02:57:55PM +0100, Catalin Marinas wrote:
> > ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA
> > operations while ARCH_KMALLOC_MINALIGN is the minimum kmalloc() objects
> > alignment.
> >
> > Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
> > Cc: Herbert Xu <herbert@gondor.apana.org.au>
> > Cc: "David S. Miller" <davem@davemloft.net>
> > ---
> >  include/linux/crypto.h | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/include/linux/crypto.h b/include/linux/crypto.h
> > index 2324ab6f1846..654b9c355575 100644
> > --- a/include/linux/crypto.h
> > +++ b/include/linux/crypto.h
> > @@ -167,7 +167,7 @@
> >   * maintenance for non-coherent DMA (cache invalidation in particular) does not
> >   * affect data that may be accessed by the CPU concurrently.
> >   */
> > -#define CRYPTO_MINALIGN ARCH_KMALLOC_MINALIGN
> > +#define CRYPTO_MINALIGN ARCH_DMA_MINALIGN
>
> I think this should remain as ARCH_KMALLOC_MINALIGN with the
> comment above modified.  The reason is that we assume memory
> returned by kmalloc is already aligned to this value.
>
> Ard, you added the comment regarding the DMA requirement, so
> does anything actually rely on this? If they do, they now need
> to do their own alignment.
>

This patch looks incorrect to me, as ARCH_DMA_MINALIGN is not
#define'd on all architectures.

But I am fine with the intent: ARCH_DMA_MINALIGN will be >=
ARCH_KMALLOC_MINALIGN, and so the compile time layout of structs will
take the worst cast minimum DMA alignment into account, whereas their
placement in memory when they allocated dynamically may be aligned to
ARCH_KMALLOC_MINALIGN only. Since the latter will be based on the
actual cache geometry, this should be fine.

Apart from the 'shash desc on stack' issue solved by the patch that
also introduced the above comment(660d2062190d), I've never looked
into the actual memory footprint of the crypto related data structures
resulting from this alignment, but it seems to me that /if/ this is
significant, we should be able to punt this to the drivers that
actually need this, rather than impose it for the whole system. (This
would involve over-allocating the context struct, and aligning up the
pointer in the various xxx_ctx() getters iff needed by the driver in
question)

I'll put this on my list of things to look at.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 01/10] mm/slab: Decouple ARCH_KMALLOC_MINALIGN from ARCH_DMA_MINALIGN

[Arnd Bergmann]

On Wed, Apr 6, 2022 at 1:59 AM Hyeonggon Yoo <42.hyeyoo@gmail.com> wrote:
>
> On Tue, Apr 05, 2022 at 02:57:49PM +0100, Catalin Marinas wrote:
> > In preparation for supporting a dynamic kmalloc() minimum alignment,
> > allow architectures to define ARCH_KMALLOC_MINALIGN independently of
> > ARCH_DMA_MINALIGN. In addition, always define ARCH_DMA_MINALIGN even if
> > an architecture does not override it.
> >
>
> [ +Cc slab maintainer/reviewers ]
>
> I get why you want to set minimum alignment of kmalloc() dynamically.
> That's because cache line size can be different and we cannot statically
> know that, right?
>
> But I don't get why you are trying to decouple ARCH_KMALLOC_MINALIGN
> from ARCH_DMA_MINALIGN. kmalloc'ed buffer is always supposed to be DMA-safe.
>
> I'm afraid this series may break some archs/drivers.
>
> in Documentation/dma-api-howto.rst:
> > 2) ARCH_DMA_MINALIGN
> >
> >   Architectures must ensure that kmalloc'ed buffer is
> >   DMA-safe. Drivers and subsystems depend on it. If an architecture
> >   isn't fully DMA-coherent (i.e. hardware doesn't ensure that data in
> >   the CPU cache is identical to data in main memory),
> >   ARCH_DMA_MINALIGN must be set so that the memory allocator
> >   makes sure that kmalloc'ed buffer doesn't share a cache line with
> >   the others. See arch/arm/include/asm/cache.h as an example.
> >
> >   Note that ARCH_DMA_MINALIGN is about DMA memory alignment
> >   constraints. You don't need to worry about the architecture data
> >   alignment constraints (e.g. the alignment constraints about 64-bit
> >   objects).
>
> If I'm missing something, please let me know :)

It helps in two ways:

- you can start with a relatively large hardcoded ARCH_DMA_MINALIGN
  of 128 or 256 bytes, depending on what the largest possible line size
  is for any machine you want to support, and then drop that down to
  32 or 64 bytes based on runtime detection. This should always be safe,
  and it means a very sizable chunk of wasted memory can be recovered.

- On systems that are fully cache coherent, there is no need to align
  kmallloc() allocations for DMA safety at all, on these, we can drop the
  size even below the cache line. This does not apply on most of the
  cheaper embedded or mobile SoCs, but it helps a lot on the machines
  you'd find in a data center.

        Arnd

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 01/10] mm/slab: Decouple ARCH_KMALLOC_MINALIGN from ARCH_DMA_MINALIGN

[Arnd Bergmann]

On Wed, Apr 6, 2022 at 1:59 AM Hyeonggon Yoo <42.hyeyoo@gmail.com> wrote:
>
> On Tue, Apr 05, 2022 at 02:57:49PM +0100, Catalin Marinas wrote:
> > In preparation for supporting a dynamic kmalloc() minimum alignment,
> > allow architectures to define ARCH_KMALLOC_MINALIGN independently of
> > ARCH_DMA_MINALIGN. In addition, always define ARCH_DMA_MINALIGN even if
> > an architecture does not override it.
> >
>
> [ +Cc slab maintainer/reviewers ]
>
> I get why you want to set minimum alignment of kmalloc() dynamically.
> That's because cache line size can be different and we cannot statically
> know that, right?
>
> But I don't get why you are trying to decouple ARCH_KMALLOC_MINALIGN
> from ARCH_DMA_MINALIGN. kmalloc'ed buffer is always supposed to be DMA-safe.
>
> I'm afraid this series may break some archs/drivers.
>
> in Documentation/dma-api-howto.rst:
> > 2) ARCH_DMA_MINALIGN
> >
> >   Architectures must ensure that kmalloc'ed buffer is
> >   DMA-safe. Drivers and subsystems depend on it. If an architecture
> >   isn't fully DMA-coherent (i.e. hardware doesn't ensure that data in
> >   the CPU cache is identical to data in main memory),
> >   ARCH_DMA_MINALIGN must be set so that the memory allocator
> >   makes sure that kmalloc'ed buffer doesn't share a cache line with
> >   the others. See arch/arm/include/asm/cache.h as an example.
> >
> >   Note that ARCH_DMA_MINALIGN is about DMA memory alignment
> >   constraints. You don't need to worry about the architecture data
> >   alignment constraints (e.g. the alignment constraints about 64-bit
> >   objects).
>
> If I'm missing something, please let me know :)

It helps in two ways:

- you can start with a relatively large hardcoded ARCH_DMA_MINALIGN
  of 128 or 256 bytes, depending on what the largest possible line size
  is for any machine you want to support, and then drop that down to
  32 or 64 bytes based on runtime detection. This should always be safe,
  and it means a very sizable chunk of wasted memory can be recovered.

- On systems that are fully cache coherent, there is no need to align
  kmallloc() allocations for DMA safety at all, on these, we can drop the
  size even below the cache line. This does not apply on most of the
  cheaper embedded or mobile SoCs, but it helps a lot on the machines
  you'd find in a data center.

        Arnd

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Wed, Apr 06, 2022 at 08:53:33AM +0200, Ard Biesheuvel wrote:
> On Wed, 6 Apr 2022 at 00:57, Herbert Xu <herbert@gondor.apana.org.au> wrote:
> > On Tue, Apr 05, 2022 at 02:57:55PM +0100, Catalin Marinas wrote:
> > > ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA
> > > operations while ARCH_KMALLOC_MINALIGN is the minimum kmalloc() objects
> > > alignment.
> > >
> > > Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
> > > Cc: Herbert Xu <herbert@gondor.apana.org.au>
> > > Cc: "David S. Miller" <davem@davemloft.net>
> > > ---
> > >  include/linux/crypto.h | 2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > >
> > > diff --git a/include/linux/crypto.h b/include/linux/crypto.h
> > > index 2324ab6f1846..654b9c355575 100644
> > > --- a/include/linux/crypto.h
> > > +++ b/include/linux/crypto.h
> > > @@ -167,7 +167,7 @@
> > >   * maintenance for non-coherent DMA (cache invalidation in particular) does not
> > >   * affect data that may be accessed by the CPU concurrently.
> > >   */
> > > -#define CRYPTO_MINALIGN ARCH_KMALLOC_MINALIGN
> > > +#define CRYPTO_MINALIGN ARCH_DMA_MINALIGN
> >
> > I think this should remain as ARCH_KMALLOC_MINALIGN with the
> > comment above modified.  The reason is that we assume memory
> > returned by kmalloc is already aligned to this value.
> >
> > Ard, you added the comment regarding the DMA requirement, so
> > does anything actually rely on this? If they do, they now need
> > to do their own alignment.
> 
> This patch looks incorrect to me, as ARCH_DMA_MINALIGN is not
> #define'd on all architectures.

It is after the first patch:

https://lore.kernel.org/all/20220405135758.774016-2-catalin.marinas@arm.com/

The series makes both ARCH_*_MINALIGN available irrespective of what an
arch defines. If one needs guaranteed static alignment for DMA, use the
DMA macro. If the minimum kmalloc() alignment is needed (e.g. to store
some flags in the lower pointer bits), use the KMALLOC macro. I grep'ed
through drivers/ and I've seen both cases (e.g.
drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c for the latter use-case).

> But I am fine with the intent: ARCH_DMA_MINALIGN will be >=
> ARCH_KMALLOC_MINALIGN, and so the compile time layout of structs will
> take the worst cast minimum DMA alignment into account, whereas their
> placement in memory when they allocated dynamically may be aligned to
> ARCH_KMALLOC_MINALIGN only. Since the latter will be based on the
> actual cache geometry, this should be fine.

That's the idea.

> Apart from the 'shash desc on stack' issue solved by the patch that
> also introduced the above comment(660d2062190d), I've never looked
> into the actual memory footprint of the crypto related data structures
> resulting from this alignment, but it seems to me that /if/ this is
> significant, we should be able to punt this to the drivers that
> actually need this, rather than impose it for the whole system. (This
> would involve over-allocating the context struct, and aligning up the
> pointer in the various xxx_ctx() getters iff needed by the driver in
> question)

Since ARCH_KMALLOC_MINALIGN on arm64 prior to this series is 128, there
is any change to the crypto code.

-- 
Catalin

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Wed, Apr 06, 2022 at 08:53:33AM +0200, Ard Biesheuvel wrote:
> On Wed, 6 Apr 2022 at 00:57, Herbert Xu <herbert@gondor.apana.org.au> wrote:
> > On Tue, Apr 05, 2022 at 02:57:55PM +0100, Catalin Marinas wrote:
> > > ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA
> > > operations while ARCH_KMALLOC_MINALIGN is the minimum kmalloc() objects
> > > alignment.
> > >
> > > Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
> > > Cc: Herbert Xu <herbert@gondor.apana.org.au>
> > > Cc: "David S. Miller" <davem@davemloft.net>
> > > ---
> > >  include/linux/crypto.h | 2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > >
> > > diff --git a/include/linux/crypto.h b/include/linux/crypto.h
> > > index 2324ab6f1846..654b9c355575 100644
> > > --- a/include/linux/crypto.h
> > > +++ b/include/linux/crypto.h
> > > @@ -167,7 +167,7 @@
> > >   * maintenance for non-coherent DMA (cache invalidation in particular) does not
> > >   * affect data that may be accessed by the CPU concurrently.
> > >   */
> > > -#define CRYPTO_MINALIGN ARCH_KMALLOC_MINALIGN
> > > +#define CRYPTO_MINALIGN ARCH_DMA_MINALIGN
> >
> > I think this should remain as ARCH_KMALLOC_MINALIGN with the
> > comment above modified.  The reason is that we assume memory
> > returned by kmalloc is already aligned to this value.
> >
> > Ard, you added the comment regarding the DMA requirement, so
> > does anything actually rely on this? If they do, they now need
> > to do their own alignment.
> 
> This patch looks incorrect to me, as ARCH_DMA_MINALIGN is not
> #define'd on all architectures.

It is after the first patch:

https://lore.kernel.org/all/20220405135758.774016-2-catalin.marinas@arm.com/

The series makes both ARCH_*_MINALIGN available irrespective of what an
arch defines. If one needs guaranteed static alignment for DMA, use the
DMA macro. If the minimum kmalloc() alignment is needed (e.g. to store
some flags in the lower pointer bits), use the KMALLOC macro. I grep'ed
through drivers/ and I've seen both cases (e.g.
drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c for the latter use-case).

> But I am fine with the intent: ARCH_DMA_MINALIGN will be >=
> ARCH_KMALLOC_MINALIGN, and so the compile time layout of structs will
> take the worst cast minimum DMA alignment into account, whereas their
> placement in memory when they allocated dynamically may be aligned to
> ARCH_KMALLOC_MINALIGN only. Since the latter will be based on the
> actual cache geometry, this should be fine.

That's the idea.

> Apart from the 'shash desc on stack' issue solved by the patch that
> also introduced the above comment(660d2062190d), I've never looked
> into the actual memory footprint of the crypto related data structures
> resulting from this alignment, but it seems to me that /if/ this is
> significant, we should be able to punt this to the drivers that
> actually need this, rather than impose it for the whole system. (This
> would involve over-allocating the context struct, and aligning up the
> pointer in the various xxx_ctx() getters iff needed by the driver in
> question)

Since ARCH_KMALLOC_MINALIGN on arm64 prior to this series is 128, there
is any change to the crypto code.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 01/10] mm/slab: Decouple ARCH_KMALLOC_MINALIGN from ARCH_DMA_MINALIGN

[Catalin Marinas]

On Wed, Apr 06, 2022 at 08:59:18AM +0900, Hyeonggon Yoo wrote:
> On Tue, Apr 05, 2022 at 02:57:49PM +0100, Catalin Marinas wrote:
> > In preparation for supporting a dynamic kmalloc() minimum alignment,
> > allow architectures to define ARCH_KMALLOC_MINALIGN independently of
> > ARCH_DMA_MINALIGN. In addition, always define ARCH_DMA_MINALIGN even if
> > an architecture does not override it.
> 
> [ +Cc slab maintainer/reviewers ]
> 
> I get why you want to set minimum alignment of kmalloc() dynamically.
> That's because cache line size can be different and we cannot statically
> know that, right?
> 
> But I don't get why you are trying to decouple ARCH_KMALLOC_MINALIGN
> from ARCH_DMA_MINALIGN. kmalloc'ed buffer is always supposed to be DMA-safe.

Arnd already replied. With this series, kmalloc'ed buffers are still
DMA-safe for the SoC the kernel is running on.

-- 
Catalin

Re: [PATCH 01/10] mm/slab: Decouple ARCH_KMALLOC_MINALIGN from ARCH_DMA_MINALIGN

[Catalin Marinas]

On Wed, Apr 06, 2022 at 08:59:18AM +0900, Hyeonggon Yoo wrote:
> On Tue, Apr 05, 2022 at 02:57:49PM +0100, Catalin Marinas wrote:
> > In preparation for supporting a dynamic kmalloc() minimum alignment,
> > allow architectures to define ARCH_KMALLOC_MINALIGN independently of
> > ARCH_DMA_MINALIGN. In addition, always define ARCH_DMA_MINALIGN even if
> > an architecture does not override it.
> 
> [ +Cc slab maintainer/reviewers ]
> 
> I get why you want to set minimum alignment of kmalloc() dynamically.
> That's because cache line size can be different and we cannot statically
> know that, right?
> 
> But I don't get why you are trying to decouple ARCH_KMALLOC_MINALIGN
> from ARCH_DMA_MINALIGN. kmalloc'ed buffer is always supposed to be DMA-safe.

Arnd already replied. With this series, kmalloc'ed buffers are still
DMA-safe for the SoC the kernel is running on.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 01/10] mm/slab: Decouple ARCH_KMALLOC_MINALIGN from ARCH_DMA_MINALIGN

[Catalin Marinas]

On Wed, Apr 06, 2022 at 10:01:45AM +0800, kernel test robot wrote:
> [auto build test WARNING on hnaz-mm/master]
> 
> url:    https://github.com/intel-lab-lkp/linux/commits/Catalin-Marinas/mm-arm64-Reduce-ARCH_KMALLOC_MINALIGN-below-the-cache-line-size/20220405-220827
> base:   https://github.com/hnaz/linux-mm master
> config: riscv-buildonly-randconfig-r004-20220405 (https://download.01.org/0day-ci/archive/20220406/202204060957.sHGvCxYf-lkp@intel.com/config)
> compiler: clang version 15.0.0 (https://github.com/llvm/llvm-project c4a1b07d0979e7ff20d7d541af666d822d66b566)
> reproduce (this is a W=1 build):
>         wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
>         chmod +x ~/bin/make.cross
>         # install riscv cross compiling tool for clang build
>         # apt-get install binutils-riscv64-linux-gnu
>         # https://github.com/intel-lab-lkp/linux/commit/4b3dfcfcd58dc57d321559e467e85ce909539755
>         git remote add linux-review https://github.com/intel-lab-lkp/linux
>         git fetch --no-tags linux-review Catalin-Marinas/mm-arm64-Reduce-ARCH_KMALLOC_MINALIGN-below-the-cache-line-size/20220405-220827
>         git checkout 4b3dfcfcd58dc57d321559e467e85ce909539755
>         # save the config file to linux build tree
>         mkdir build_dir
>         COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 O=build_dir ARCH=riscv SHELL=/bin/bash
> 
> If you fix the issue, kindly add following tag as appropriate
> Reported-by: kernel test robot <lkp@intel.com>
> 
> All warnings (new ones prefixed by >>, old ones prefixed by <<):
> 
> >> WARNING: modpost: vmlinux.o(.text+0xf8244): Section mismatch in reference from the function perf_swevent_set_period() to the function .init.text:set_reset_devices()
> The function perf_swevent_set_period() references
> the function __init set_reset_devices().
> This is often because perf_swevent_set_period lacks a __init
> annotation or the annotation of set_reset_devices is wrong.

I don't see how this relates to the patch which just defines
ARCH_DMA_MINALIGN. It may be something wrong in the tree you used as a
base. My patches are based on 5.18-rc1.

-- 
Catalin

Re: [PATCH 01/10] mm/slab: Decouple ARCH_KMALLOC_MINALIGN from ARCH_DMA_MINALIGN

[Catalin Marinas]

[-- Attachment #1: Type: text/plain, Size: 2246 bytes --]

On Wed, Apr 06, 2022 at 10:01:45AM +0800, kernel test robot wrote:
> [auto build test WARNING on hnaz-mm/master]
> 
> url:    https://github.com/intel-lab-lkp/linux/commits/Catalin-Marinas/mm-arm64-Reduce-ARCH_KMALLOC_MINALIGN-below-the-cache-line-size/20220405-220827
> base:   https://github.com/hnaz/linux-mm master
> config: riscv-buildonly-randconfig-r004-20220405 (https://download.01.org/0day-ci/archive/20220406/202204060957.sHGvCxYf-lkp(a)intel.com/config)
> compiler: clang version 15.0.0 (https://github.com/llvm/llvm-project c4a1b07d0979e7ff20d7d541af666d822d66b566)
> reproduce (this is a W=1 build):
>         wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
>         chmod +x ~/bin/make.cross
>         # install riscv cross compiling tool for clang build
>         # apt-get install binutils-riscv64-linux-gnu
>         # https://github.com/intel-lab-lkp/linux/commit/4b3dfcfcd58dc57d321559e467e85ce909539755
>         git remote add linux-review https://github.com/intel-lab-lkp/linux
>         git fetch --no-tags linux-review Catalin-Marinas/mm-arm64-Reduce-ARCH_KMALLOC_MINALIGN-below-the-cache-line-size/20220405-220827
>         git checkout 4b3dfcfcd58dc57d321559e467e85ce909539755
>         # save the config file to linux build tree
>         mkdir build_dir
>         COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 O=build_dir ARCH=riscv SHELL=/bin/bash
> 
> If you fix the issue, kindly add following tag as appropriate
> Reported-by: kernel test robot <lkp@intel.com>
> 
> All warnings (new ones prefixed by >>, old ones prefixed by <<):
> 
> >> WARNING: modpost: vmlinux.o(.text+0xf8244): Section mismatch in reference from the function perf_swevent_set_period() to the function .init.text:set_reset_devices()
> The function perf_swevent_set_period() references
> the function __init set_reset_devices().
> This is often because perf_swevent_set_period lacks a __init
> annotation or the annotation of set_reset_devices is wrong.

I don't see how this relates to the patch which just defines
ARCH_DMA_MINALIGN. It may be something wrong in the tree you used as a
base. My patches are based on 5.18-rc1.

-- 
Catalin

Re: [PATCH 09/10] mm/slab: Simplify create_kmalloc_cache() args and make it static

[Catalin Marinas]

On Wed, Apr 06, 2022 at 01:35:53PM +0800, kernel test robot wrote:
> Hi Catalin,
> 
> I love your patch! Perhaps something to improve:
> 
> [auto build test WARNING on hnaz-mm/master]
> 
> url:    https://github.com/intel-lab-lkp/linux/commits/Catalin-Marinas/mm-arm64-Reduce-ARCH_KMALLOC_MINALIGN-below-the-cache-line-size/20220405-220827
> base:   https://github.com/hnaz/linux-mm master
> config: riscv-buildonly-randconfig-r004-20220405 (https://download.01.org/0day-ci/archive/20220406/202204061348.mo7QK7AD-lkp@intel.com/config)
> compiler: clang version 15.0.0 (https://github.com/llvm/llvm-project c4a1b07d0979e7ff20d7d541af666d822d66b566)
> reproduce (this is a W=1 build):
>         wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
>         chmod +x ~/bin/make.cross
>         # install riscv cross compiling tool for clang build
>         # apt-get install binutils-riscv64-linux-gnu
>         # https://github.com/intel-lab-lkp/linux/commit/594fc15f21ce6de4d37971f717ff4223ab271823
>         git remote add linux-review https://github.com/intel-lab-lkp/linux
>         git fetch --no-tags linux-review Catalin-Marinas/mm-arm64-Reduce-ARCH_KMALLOC_MINALIGN-below-the-cache-line-size/20220405-220827
>         git checkout 594fc15f21ce6de4d37971f717ff4223ab271823
>         # save the config file to linux build tree
>         mkdir build_dir
>         COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 O=build_dir ARCH=riscv SHELL=/bin/bash
> 
> If you fix the issue, kindly add following tag as appropriate
> Reported-by: kernel test robot <lkp@intel.com>
> 
> All warnings (new ones prefixed by >>, old ones prefixed by <<):
> 
> >> WARNING: modpost: vmlinux.o(.data+0x2836c8): Section mismatch in reference from the variable sctp_prot to the variable .init.text:.LBB19_63
> The variable sctp_prot references
> the variable __init .LBB19_63
> If the reference is valid then annotate the
> variable with or __refdata (see linux/init.h) or name the variable:

Same with the other report, I don't think these are caused by my
patches.

Could you please try the patches again on top of 5.18-rc1?

Thanks.

-- 
Catalin

Re: [PATCH 09/10] mm/slab: Simplify create_kmalloc_cache() args and make it static

[Catalin Marinas]

[-- Attachment #1: Type: text/plain, Size: 2234 bytes --]

On Wed, Apr 06, 2022 at 01:35:53PM +0800, kernel test robot wrote:
> Hi Catalin,
> 
> I love your patch! Perhaps something to improve:
> 
> [auto build test WARNING on hnaz-mm/master]
> 
> url:    https://github.com/intel-lab-lkp/linux/commits/Catalin-Marinas/mm-arm64-Reduce-ARCH_KMALLOC_MINALIGN-below-the-cache-line-size/20220405-220827
> base:   https://github.com/hnaz/linux-mm master
> config: riscv-buildonly-randconfig-r004-20220405 (https://download.01.org/0day-ci/archive/20220406/202204061348.mo7QK7AD-lkp(a)intel.com/config)
> compiler: clang version 15.0.0 (https://github.com/llvm/llvm-project c4a1b07d0979e7ff20d7d541af666d822d66b566)
> reproduce (this is a W=1 build):
>         wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
>         chmod +x ~/bin/make.cross
>         # install riscv cross compiling tool for clang build
>         # apt-get install binutils-riscv64-linux-gnu
>         # https://github.com/intel-lab-lkp/linux/commit/594fc15f21ce6de4d37971f717ff4223ab271823
>         git remote add linux-review https://github.com/intel-lab-lkp/linux
>         git fetch --no-tags linux-review Catalin-Marinas/mm-arm64-Reduce-ARCH_KMALLOC_MINALIGN-below-the-cache-line-size/20220405-220827
>         git checkout 594fc15f21ce6de4d37971f717ff4223ab271823
>         # save the config file to linux build tree
>         mkdir build_dir
>         COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 O=build_dir ARCH=riscv SHELL=/bin/bash
> 
> If you fix the issue, kindly add following tag as appropriate
> Reported-by: kernel test robot <lkp@intel.com>
> 
> All warnings (new ones prefixed by >>, old ones prefixed by <<):
> 
> >> WARNING: modpost: vmlinux.o(.data+0x2836c8): Section mismatch in reference from the variable sctp_prot to the variable .init.text:.LBB19_63
> The variable sctp_prot references
> the variable __init .LBB19_63
> If the reference is valid then annotate the
> variable with or __refdata (see linux/init.h) or name the variable:

Same with the other report, I don't think these are caused by my
patches.

Could you please try the patches again on top of 5.18-rc1?

Thanks.

-- 
Catalin

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Ard Biesheuvel]

On Wed, 6 Apr 2022 at 10:49, Catalin Marinas <catalin.marinas@arm.com> wrote:
>
> On Wed, Apr 06, 2022 at 08:53:33AM +0200, Ard Biesheuvel wrote:
> > On Wed, 6 Apr 2022 at 00:57, Herbert Xu <herbert@gondor.apana.org.au> wrote:
> > > On Tue, Apr 05, 2022 at 02:57:55PM +0100, Catalin Marinas wrote:
> > > > ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA
> > > > operations while ARCH_KMALLOC_MINALIGN is the minimum kmalloc() objects
> > > > alignment.
> > > >
> > > > Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
> > > > Cc: Herbert Xu <herbert@gondor.apana.org.au>
> > > > Cc: "David S. Miller" <davem@davemloft.net>
> > > > ---
> > > >  include/linux/crypto.h | 2 +-
> > > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > >
> > > > diff --git a/include/linux/crypto.h b/include/linux/crypto.h
> > > > index 2324ab6f1846..654b9c355575 100644
> > > > --- a/include/linux/crypto.h
> > > > +++ b/include/linux/crypto.h
> > > > @@ -167,7 +167,7 @@
> > > >   * maintenance for non-coherent DMA (cache invalidation in particular) does not
> > > >   * affect data that may be accessed by the CPU concurrently.
> > > >   */
> > > > -#define CRYPTO_MINALIGN ARCH_KMALLOC_MINALIGN
> > > > +#define CRYPTO_MINALIGN ARCH_DMA_MINALIGN
> > >
> > > I think this should remain as ARCH_KMALLOC_MINALIGN with the
> > > comment above modified.  The reason is that we assume memory
> > > returned by kmalloc is already aligned to this value.
> > >
> > > Ard, you added the comment regarding the DMA requirement, so
> > > does anything actually rely on this? If they do, they now need
> > > to do their own alignment.
> >
> > This patch looks incorrect to me, as ARCH_DMA_MINALIGN is not
> > #define'd on all architectures.
>
> It is after the first patch:
>
> https://lore.kernel.org/all/20220405135758.774016-2-catalin.marinas@arm.com/
>

I wasn't cc'ed on that :-)

> The series makes both ARCH_*_MINALIGN available irrespective of what an
> arch defines. If one needs guaranteed static alignment for DMA, use the
> DMA macro. If the minimum kmalloc() alignment is needed (e.g. to store
> some flags in the lower pointer bits), use the KMALLOC macro. I grep'ed
> through drivers/ and I've seen both cases (e.g.
> drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c for the latter use-case).
>
> > But I am fine with the intent: ARCH_DMA_MINALIGN will be >=
> > ARCH_KMALLOC_MINALIGN, and so the compile time layout of structs will
> > take the worst cast minimum DMA alignment into account, whereas their
> > placement in memory when they allocated dynamically may be aligned to
> > ARCH_KMALLOC_MINALIGN only. Since the latter will be based on the
> > actual cache geometry, this should be fine.
>
> That's the idea.
>
> > Apart from the 'shash desc on stack' issue solved by the patch that
> > also introduced the above comment(660d2062190d), I've never looked
> > into the actual memory footprint of the crypto related data structures
> > resulting from this alignment, but it seems to me that /if/ this is
> > significant, we should be able to punt this to the drivers that
> > actually need this, rather than impose it for the whole system. (This
> > would involve over-allocating the context struct, and aligning up the
> > pointer in the various xxx_ctx() getters iff needed by the driver in
> > question)
>
> Since ARCH_KMALLOC_MINALIGN on arm64 prior to this series is 128, there
> is any change to the crypto code.
>

No, not currently. But what I started looking into today is avoiding
the need to impose DMA alignment on every single data structure
allocated by the crypto API, even if it is never used for DMA or
touched by a device. That seems rather wasteful as well.

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Ard Biesheuvel]

On Wed, 6 Apr 2022 at 10:49, Catalin Marinas <catalin.marinas@arm.com> wrote:
>
> On Wed, Apr 06, 2022 at 08:53:33AM +0200, Ard Biesheuvel wrote:
> > On Wed, 6 Apr 2022 at 00:57, Herbert Xu <herbert@gondor.apana.org.au> wrote:
> > > On Tue, Apr 05, 2022 at 02:57:55PM +0100, Catalin Marinas wrote:
> > > > ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA
> > > > operations while ARCH_KMALLOC_MINALIGN is the minimum kmalloc() objects
> > > > alignment.
> > > >
> > > > Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
> > > > Cc: Herbert Xu <herbert@gondor.apana.org.au>
> > > > Cc: "David S. Miller" <davem@davemloft.net>
> > > > ---
> > > >  include/linux/crypto.h | 2 +-
> > > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > >
> > > > diff --git a/include/linux/crypto.h b/include/linux/crypto.h
> > > > index 2324ab6f1846..654b9c355575 100644
> > > > --- a/include/linux/crypto.h
> > > > +++ b/include/linux/crypto.h
> > > > @@ -167,7 +167,7 @@
> > > >   * maintenance for non-coherent DMA (cache invalidation in particular) does not
> > > >   * affect data that may be accessed by the CPU concurrently.
> > > >   */
> > > > -#define CRYPTO_MINALIGN ARCH_KMALLOC_MINALIGN
> > > > +#define CRYPTO_MINALIGN ARCH_DMA_MINALIGN
> > >
> > > I think this should remain as ARCH_KMALLOC_MINALIGN with the
> > > comment above modified.  The reason is that we assume memory
> > > returned by kmalloc is already aligned to this value.
> > >
> > > Ard, you added the comment regarding the DMA requirement, so
> > > does anything actually rely on this? If they do, they now need
> > > to do their own alignment.
> >
> > This patch looks incorrect to me, as ARCH_DMA_MINALIGN is not
> > #define'd on all architectures.
>
> It is after the first patch:
>
> https://lore.kernel.org/all/20220405135758.774016-2-catalin.marinas@arm.com/
>

I wasn't cc'ed on that :-)

> The series makes both ARCH_*_MINALIGN available irrespective of what an
> arch defines. If one needs guaranteed static alignment for DMA, use the
> DMA macro. If the minimum kmalloc() alignment is needed (e.g. to store
> some flags in the lower pointer bits), use the KMALLOC macro. I grep'ed
> through drivers/ and I've seen both cases (e.g.
> drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c for the latter use-case).
>
> > But I am fine with the intent: ARCH_DMA_MINALIGN will be >=
> > ARCH_KMALLOC_MINALIGN, and so the compile time layout of structs will
> > take the worst cast minimum DMA alignment into account, whereas their
> > placement in memory when they allocated dynamically may be aligned to
> > ARCH_KMALLOC_MINALIGN only. Since the latter will be based on the
> > actual cache geometry, this should be fine.
>
> That's the idea.
>
> > Apart from the 'shash desc on stack' issue solved by the patch that
> > also introduced the above comment(660d2062190d), I've never looked
> > into the actual memory footprint of the crypto related data structures
> > resulting from this alignment, but it seems to me that /if/ this is
> > significant, we should be able to punt this to the drivers that
> > actually need this, rather than impose it for the whole system. (This
> > would involve over-allocating the context struct, and aligning up the
> > pointer in the various xxx_ctx() getters iff needed by the driver in
> > question)
>
> Since ARCH_KMALLOC_MINALIGN on arm64 prior to this series is 128, there
> is any change to the crypto code.
>

No, not currently. But what I started looking into today is avoiding
the need to impose DMA alignment on every single data structure
allocated by the crypto API, even if it is never used for DMA or
touched by a device. That seems rather wasteful as well.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 01/10] mm/slab: Decouple ARCH_KMALLOC_MINALIGN from ARCH_DMA_MINALIGN

[Hyeonggon Yoo]

On Wed, Apr 06, 2022 at 09:29:19AM +0200, Arnd Bergmann wrote:
> On Wed, Apr 6, 2022 at 1:59 AM Hyeonggon Yoo <42.hyeyoo@gmail.com> wrote:
> >
> > On Tue, Apr 05, 2022 at 02:57:49PM +0100, Catalin Marinas wrote:
> > > In preparation for supporting a dynamic kmalloc() minimum alignment,
> > > allow architectures to define ARCH_KMALLOC_MINALIGN independently of
> > > ARCH_DMA_MINALIGN. In addition, always define ARCH_DMA_MINALIGN even if
> > > an architecture does not override it.
> > >
> >
> > [ +Cc slab maintainer/reviewers ]
> >
> > I get why you want to set minimum alignment of kmalloc() dynamically.
> > That's because cache line size can be different and we cannot statically
> > know that, right?
> >
> > But I don't get why you are trying to decouple ARCH_KMALLOC_MINALIGN
> > from ARCH_DMA_MINALIGN. kmalloc'ed buffer is always supposed to be DMA-safe.
> >
> > I'm afraid this series may break some archs/drivers.
> >
> > in Documentation/dma-api-howto.rst:
> > > 2) ARCH_DMA_MINALIGN
> > >
> > >   Architectures must ensure that kmalloc'ed buffer is
> > >   DMA-safe. Drivers and subsystems depend on it. If an architecture
> > >   isn't fully DMA-coherent (i.e. hardware doesn't ensure that data in
> > >   the CPU cache is identical to data in main memory),
> > >   ARCH_DMA_MINALIGN must be set so that the memory allocator
> > >   makes sure that kmalloc'ed buffer doesn't share a cache line with
> > >   the others. See arch/arm/include/asm/cache.h as an example.
> > >
> > >   Note that ARCH_DMA_MINALIGN is about DMA memory alignment
> > >   constraints. You don't need to worry about the architecture data
> > >   alignment constraints (e.g. the alignment constraints about 64-bit
> > >   objects).
> >
> > If I'm missing something, please let me know :)
> 
> It helps in two ways:
> 
> - you can start with a relatively large hardcoded ARCH_DMA_MINALIGN
>   of 128 or 256 bytes, depending on what the largest possible line size
>   is for any machine you want to support, and then drop that down to
>   32 or 64 bytes based on runtime detection. This should always be safe,
>   and it means a very sizable chunk of wasted memory can be recovered.
>

I agree this part.

> - On systems that are fully cache coherent, there is no need to align
>   kmallloc() allocations for DMA safety at all, on these, we can drop the
>   size even below the cache line. This does not apply on most of the
>   cheaper embedded or mobile SoCs, but it helps a lot on the machines
>   you'd find in a data center.

Now I get the point. Thank you for explanation!
Going to review this series soon.

> 
>         Arnd

-- 
Thanks,
Hyeonggon

Re: [PATCH 01/10] mm/slab: Decouple ARCH_KMALLOC_MINALIGN from ARCH_DMA_MINALIGN

[Hyeonggon Yoo]

On Wed, Apr 06, 2022 at 09:29:19AM +0200, Arnd Bergmann wrote:
> On Wed, Apr 6, 2022 at 1:59 AM Hyeonggon Yoo <42.hyeyoo@gmail.com> wrote:
> >
> > On Tue, Apr 05, 2022 at 02:57:49PM +0100, Catalin Marinas wrote:
> > > In preparation for supporting a dynamic kmalloc() minimum alignment,
> > > allow architectures to define ARCH_KMALLOC_MINALIGN independently of
> > > ARCH_DMA_MINALIGN. In addition, always define ARCH_DMA_MINALIGN even if
> > > an architecture does not override it.
> > >
> >
> > [ +Cc slab maintainer/reviewers ]
> >
> > I get why you want to set minimum alignment of kmalloc() dynamically.
> > That's because cache line size can be different and we cannot statically
> > know that, right?
> >
> > But I don't get why you are trying to decouple ARCH_KMALLOC_MINALIGN
> > from ARCH_DMA_MINALIGN. kmalloc'ed buffer is always supposed to be DMA-safe.
> >
> > I'm afraid this series may break some archs/drivers.
> >
> > in Documentation/dma-api-howto.rst:
> > > 2) ARCH_DMA_MINALIGN
> > >
> > >   Architectures must ensure that kmalloc'ed buffer is
> > >   DMA-safe. Drivers and subsystems depend on it. If an architecture
> > >   isn't fully DMA-coherent (i.e. hardware doesn't ensure that data in
> > >   the CPU cache is identical to data in main memory),
> > >   ARCH_DMA_MINALIGN must be set so that the memory allocator
> > >   makes sure that kmalloc'ed buffer doesn't share a cache line with
> > >   the others. See arch/arm/include/asm/cache.h as an example.
> > >
> > >   Note that ARCH_DMA_MINALIGN is about DMA memory alignment
> > >   constraints. You don't need to worry about the architecture data
> > >   alignment constraints (e.g. the alignment constraints about 64-bit
> > >   objects).
> >
> > If I'm missing something, please let me know :)
> 
> It helps in two ways:
> 
> - you can start with a relatively large hardcoded ARCH_DMA_MINALIGN
>   of 128 or 256 bytes, depending on what the largest possible line size
>   is for any machine you want to support, and then drop that down to
>   32 or 64 bytes based on runtime detection. This should always be safe,
>   and it means a very sizable chunk of wasted memory can be recovered.
>

I agree this part.

> - On systems that are fully cache coherent, there is no need to align
>   kmallloc() allocations for DMA safety at all, on these, we can drop the
>   size even below the cache line. This does not apply on most of the
>   cheaper embedded or mobile SoCs, but it helps a lot on the machines
>   you'd find in a data center.

Now I get the point. Thank you for explanation!
Going to review this series soon.

> 
>         Arnd

-- 
Thanks,
Hyeonggon

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [kbuild-all] Re: [PATCH 01/10] mm/slab: Decouple ARCH_KMALLOC_MINALIGN from ARCH_DMA_MINALIGN

[Chen, Rong A]

Hi Catalin,

On 4/6/2022 4:56 PM, Catalin Marinas wrote:
> On Wed, Apr 06, 2022 at 10:01:45AM +0800, kernel test robot wrote:
>> [auto build test WARNING on hnaz-mm/master]
>>
>> url:    https://github.com/intel-lab-lkp/linux/commits/Catalin-Marinas/mm-arm64-Reduce-ARCH_KMALLOC_MINALIGN-below-the-cache-line-size/20220405-220827
>> base:   https://github.com/hnaz/linux-mm master
>> config: riscv-buildonly-randconfig-r004-20220405 (https://download.01.org/0day-ci/archive/20220406/202204060957.sHGvCxYf-lkp@intel.com/config)
>> compiler: clang version 15.0.0 (https://github.com/llvm/llvm-project c4a1b07d0979e7ff20d7d541af666d822d66b566)
>> reproduce (this is a W=1 build):
>>          wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
>>          chmod +x ~/bin/make.cross
>>          # install riscv cross compiling tool for clang build
>>          # apt-get install binutils-riscv64-linux-gnu
>>          # https://github.com/intel-lab-lkp/linux/commit/4b3dfcfcd58dc57d321559e467e85ce909539755
>>          git remote add linux-review https://github.com/intel-lab-lkp/linux
>>          git fetch --no-tags linux-review Catalin-Marinas/mm-arm64-Reduce-ARCH_KMALLOC_MINALIGN-below-the-cache-line-size/20220405-220827
>>          git checkout 4b3dfcfcd58dc57d321559e467e85ce909539755
>>          # save the config file to linux build tree
>>          mkdir build_dir
>>          COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 O=build_dir ARCH=riscv SHELL=/bin/bash
>>
>> If you fix the issue, kindly add following tag as appropriate
>> Reported-by: kernel test robot <lkp@intel.com>
>>
>> All warnings (new ones prefixed by >>, old ones prefixed by <<):
>>
>>>> WARNING: modpost: vmlinux.o(.text+0xf8244): Section mismatch in reference from the function perf_swevent_set_period() to the function .init.text:set_reset_devices()
>> The function perf_swevent_set_period() references
>> the function __init set_reset_devices().
>> This is often because perf_swevent_set_period lacks a __init
>> annotation or the annotation of set_reset_devices is wrong.
> 
> I don't see how this relates to the patch which just defines
> ARCH_DMA_MINALIGN. It may be something wrong in the tree you used as a
> base. My patches are based on 5.18-rc1.
> 

Sorry for the noise, we have received some feedback that the warning is
not related to the commit, we are going to add it to the ignore list.

Best Regards,
Rong Chen

Re: [PATCH 08/10] mm/slab: Allow dynamic kmalloc() minimum alignment

[Hyeonggon Yoo]

On Tue, Apr 05, 2022 at 02:57:56PM +0100, Catalin Marinas wrote:
> ARCH_KMALLOC_MINALIGN represents the minimum guaranteed kmalloc()
> alignment but an architecture may require a larger run-time alignment.
> Do not create kmalloc caches smaller than arch_kmalloc_minalign().
> 
> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
> Cc: Andrew Morton <akpm@linux-foundation.org>
> ---
>  include/linux/slab.h |  2 ++
>  mm/slab.c            |  6 +-----
>  mm/slab.h            |  2 ++
>  mm/slab_common.c     | 33 +++++++++++++++++++++++----------
>  4 files changed, 28 insertions(+), 15 deletions(-)
> 
> diff --git a/include/linux/slab.h b/include/linux/slab.h
> index d58211bdeceb..2137dba85691 100644
> --- a/include/linux/slab.h
> +++ b/include/linux/slab.h
> @@ -332,6 +332,8 @@ enum kmalloc_cache_type {
>  extern struct kmem_cache *
>  kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_SHIFT_HIGH + 1];
>  
> +unsigned int arch_kmalloc_minalign(void);
> +
>  /*
>   * Define gfp bits that should not be set for KMALLOC_NORMAL.
>   */
> diff --git a/mm/slab.c b/mm/slab.c
> index b04e40078bdf..4aaeeb9c994d 100644
> --- a/mm/slab.c
> +++ b/mm/slab.c
> @@ -1256,11 +1256,7 @@ void __init kmem_cache_init(void)
>  	 * Initialize the caches that provide memory for the  kmem_cache_node
>  	 * structures first.  Without this, further allocations will bug.
>  	 */
> -	kmalloc_caches[KMALLOC_NORMAL][INDEX_NODE] = create_kmalloc_cache(
> -				kmalloc_info[INDEX_NODE].name[KMALLOC_NORMAL],
> -				kmalloc_info[INDEX_NODE].size,
> -				ARCH_KMALLOC_FLAGS, 0,
> -				kmalloc_info[INDEX_NODE].size);
> +	new_kmalloc_cache(INDEX_NODE, KMALLOC_NORMAL, ARCH_KMALLOC_FLAGS);
>  	slab_state = PARTIAL_NODE;
>  	setup_kmalloc_cache_index_table();
>  
> diff --git a/mm/slab.h b/mm/slab.h
> index fd7ae2024897..e9238406602a 100644
> --- a/mm/slab.h
> +++ b/mm/slab.h
> @@ -283,6 +283,8 @@ int __kmem_cache_create(struct kmem_cache *, slab_flags_t flags);
>  struct kmem_cache *create_kmalloc_cache(const char *name, unsigned int size,
>  			slab_flags_t flags, unsigned int useroffset,
>  			unsigned int usersize);
> +void __init new_kmalloc_cache(int idx, enum kmalloc_cache_type type,
> +			      slab_flags_t flags);
>  extern void create_boot_cache(struct kmem_cache *, const char *name,
>  			unsigned int size, slab_flags_t flags,
>  			unsigned int useroffset, unsigned int usersize);
> diff --git a/mm/slab_common.c b/mm/slab_common.c
> index 6ee64d6208b3..594d8a8a68d0 100644
> --- a/mm/slab_common.c
> +++ b/mm/slab_common.c
> @@ -838,9 +838,18 @@ void __init setup_kmalloc_cache_index_table(void)
>  	}
>  }
>  
> -static void __init
> +unsigned int __weak arch_kmalloc_minalign(void)
> +{
> +	return ARCH_KMALLOC_MINALIGN;
> +}
> +

As ARCH_KMALLOC_ALIGN and arch_kmalloc_minalign() may not be same after
patch 10, I think s/ARCH_KMALLOC_ALIGN/arch_kmalloc_minalign/g
for every user of it would be more correct?

> +void __init
>  new_kmalloc_cache(int idx, enum kmalloc_cache_type type, slab_flags_t flags)
>  {
> +	unsigned int minalign = arch_kmalloc_minalign();
> +	unsigned int aligned_size = kmalloc_info[idx].size;
> +	int aligned_idx = idx;
> +
>  	if (type == KMALLOC_RECLAIM) {
>  		flags |= SLAB_RECLAIM_ACCOUNT;
>  	} else if (IS_ENABLED(CONFIG_MEMCG_KMEM) && (type == KMALLOC_CGROUP)) {
> @@ -851,10 +860,17 @@ new_kmalloc_cache(int idx, enum kmalloc_cache_type type, slab_flags_t flags)
>  		flags |= SLAB_ACCOUNT;
>  	}
>  
> -	kmalloc_caches[type][idx] = create_kmalloc_cache(
> -					kmalloc_info[idx].name[type],
> -					kmalloc_info[idx].size, flags, 0,
> -					kmalloc_info[idx].size);
> +	if (minalign > ARCH_KMALLOC_MINALIGN) {
> +		aligned_size = ALIGN(aligned_size, minalign);
> +		aligned_idx = __kmalloc_index(aligned_size, false);
> +	}
> +
> +	if (!kmalloc_caches[type][aligned_idx])
> +		kmalloc_caches[type][aligned_idx] = create_kmalloc_cache(
> +					kmalloc_info[aligned_idx].name[type],
> +					aligned_size, flags, 0, aligned_size);
> +	if (idx != aligned_idx)
> +		kmalloc_caches[type][idx] = kmalloc_caches[type][aligned_idx];

I would prefer detecting minimum kmalloc size in create_kmalloc_caches()
in runtime instead of changing behavior of new_kmalloc_cache().

>  	/*
>  	 * If CONFIG_MEMCG_KMEM is enabled, disable cache merging for
> @@ -904,11 +920,8 @@ void __init create_kmalloc_caches(slab_flags_t flags)
>  		struct kmem_cache *s = kmalloc_caches[KMALLOC_NORMAL][i];
>  
>  		if (s) {
> -			kmalloc_caches[KMALLOC_DMA][i] = create_kmalloc_cache(
> -				kmalloc_info[i].name[KMALLOC_DMA],
> -				kmalloc_info[i].size,
> -				SLAB_CACHE_DMA | flags, 0,
> -				kmalloc_info[i].size);
> +			new_kmalloc_cache(i, KMALLOC_DMA,
> +					  SLAB_CACHE_DMA | flags);
>  		}
>  	}
>  #endif

-- 
Thank you, You are awesome!
Hyeonggon :-)

Re: [PATCH 08/10] mm/slab: Allow dynamic kmalloc() minimum alignment

[Hyeonggon Yoo]

On Tue, Apr 05, 2022 at 02:57:56PM +0100, Catalin Marinas wrote:
> ARCH_KMALLOC_MINALIGN represents the minimum guaranteed kmalloc()
> alignment but an architecture may require a larger run-time alignment.
> Do not create kmalloc caches smaller than arch_kmalloc_minalign().
> 
> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
> Cc: Andrew Morton <akpm@linux-foundation.org>
> ---
>  include/linux/slab.h |  2 ++
>  mm/slab.c            |  6 +-----
>  mm/slab.h            |  2 ++
>  mm/slab_common.c     | 33 +++++++++++++++++++++++----------
>  4 files changed, 28 insertions(+), 15 deletions(-)
> 
> diff --git a/include/linux/slab.h b/include/linux/slab.h
> index d58211bdeceb..2137dba85691 100644
> --- a/include/linux/slab.h
> +++ b/include/linux/slab.h
> @@ -332,6 +332,8 @@ enum kmalloc_cache_type {
>  extern struct kmem_cache *
>  kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_SHIFT_HIGH + 1];
>  
> +unsigned int arch_kmalloc_minalign(void);
> +
>  /*
>   * Define gfp bits that should not be set for KMALLOC_NORMAL.
>   */
> diff --git a/mm/slab.c b/mm/slab.c
> index b04e40078bdf..4aaeeb9c994d 100644
> --- a/mm/slab.c
> +++ b/mm/slab.c
> @@ -1256,11 +1256,7 @@ void __init kmem_cache_init(void)
>  	 * Initialize the caches that provide memory for the  kmem_cache_node
>  	 * structures first.  Without this, further allocations will bug.
>  	 */
> -	kmalloc_caches[KMALLOC_NORMAL][INDEX_NODE] = create_kmalloc_cache(
> -				kmalloc_info[INDEX_NODE].name[KMALLOC_NORMAL],
> -				kmalloc_info[INDEX_NODE].size,
> -				ARCH_KMALLOC_FLAGS, 0,
> -				kmalloc_info[INDEX_NODE].size);
> +	new_kmalloc_cache(INDEX_NODE, KMALLOC_NORMAL, ARCH_KMALLOC_FLAGS);
>  	slab_state = PARTIAL_NODE;
>  	setup_kmalloc_cache_index_table();
>  
> diff --git a/mm/slab.h b/mm/slab.h
> index fd7ae2024897..e9238406602a 100644
> --- a/mm/slab.h
> +++ b/mm/slab.h
> @@ -283,6 +283,8 @@ int __kmem_cache_create(struct kmem_cache *, slab_flags_t flags);
>  struct kmem_cache *create_kmalloc_cache(const char *name, unsigned int size,
>  			slab_flags_t flags, unsigned int useroffset,
>  			unsigned int usersize);
> +void __init new_kmalloc_cache(int idx, enum kmalloc_cache_type type,
> +			      slab_flags_t flags);
>  extern void create_boot_cache(struct kmem_cache *, const char *name,
>  			unsigned int size, slab_flags_t flags,
>  			unsigned int useroffset, unsigned int usersize);
> diff --git a/mm/slab_common.c b/mm/slab_common.c
> index 6ee64d6208b3..594d8a8a68d0 100644
> --- a/mm/slab_common.c
> +++ b/mm/slab_common.c
> @@ -838,9 +838,18 @@ void __init setup_kmalloc_cache_index_table(void)
>  	}
>  }
>  
> -static void __init
> +unsigned int __weak arch_kmalloc_minalign(void)
> +{
> +	return ARCH_KMALLOC_MINALIGN;
> +}
> +

As ARCH_KMALLOC_ALIGN and arch_kmalloc_minalign() may not be same after
patch 10, I think s/ARCH_KMALLOC_ALIGN/arch_kmalloc_minalign/g
for every user of it would be more correct?

> +void __init
>  new_kmalloc_cache(int idx, enum kmalloc_cache_type type, slab_flags_t flags)
>  {
> +	unsigned int minalign = arch_kmalloc_minalign();
> +	unsigned int aligned_size = kmalloc_info[idx].size;
> +	int aligned_idx = idx;
> +
>  	if (type == KMALLOC_RECLAIM) {
>  		flags |= SLAB_RECLAIM_ACCOUNT;
>  	} else if (IS_ENABLED(CONFIG_MEMCG_KMEM) && (type == KMALLOC_CGROUP)) {
> @@ -851,10 +860,17 @@ new_kmalloc_cache(int idx, enum kmalloc_cache_type type, slab_flags_t flags)
>  		flags |= SLAB_ACCOUNT;
>  	}
>  
> -	kmalloc_caches[type][idx] = create_kmalloc_cache(
> -					kmalloc_info[idx].name[type],
> -					kmalloc_info[idx].size, flags, 0,
> -					kmalloc_info[idx].size);
> +	if (minalign > ARCH_KMALLOC_MINALIGN) {
> +		aligned_size = ALIGN(aligned_size, minalign);
> +		aligned_idx = __kmalloc_index(aligned_size, false);
> +	}
> +
> +	if (!kmalloc_caches[type][aligned_idx])
> +		kmalloc_caches[type][aligned_idx] = create_kmalloc_cache(
> +					kmalloc_info[aligned_idx].name[type],
> +					aligned_size, flags, 0, aligned_size);
> +	if (idx != aligned_idx)
> +		kmalloc_caches[type][idx] = kmalloc_caches[type][aligned_idx];

I would prefer detecting minimum kmalloc size in create_kmalloc_caches()
in runtime instead of changing behavior of new_kmalloc_cache().

>  	/*
>  	 * If CONFIG_MEMCG_KMEM is enabled, disable cache merging for
> @@ -904,11 +920,8 @@ void __init create_kmalloc_caches(slab_flags_t flags)
>  		struct kmem_cache *s = kmalloc_caches[KMALLOC_NORMAL][i];
>  
>  		if (s) {
> -			kmalloc_caches[KMALLOC_DMA][i] = create_kmalloc_cache(
> -				kmalloc_info[i].name[KMALLOC_DMA],
> -				kmalloc_info[i].size,
> -				SLAB_CACHE_DMA | flags, 0,
> -				kmalloc_info[i].size);
> +			new_kmalloc_cache(i, KMALLOC_DMA,
> +					  SLAB_CACHE_DMA | flags);
>  		}
>  	}
>  #endif

-- 
Thank you, You are awesome!
Hyeonggon :-)

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Herbert Xu]

On Wed, Apr 06, 2022 at 09:49:42AM +0100, Catalin Marinas wrote:
>
> Since ARCH_KMALLOC_MINALIGN on arm64 prior to this series is 128, there
> is any change to the crypto code.

But the crypto API assumes that memory returned by kmalloc is
automatically aligned to CRYPTO_MINALIGN, would this still be
the case if you change it to ARCH_DMA_MINALIGN?

Thanks,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Herbert Xu]

On Wed, Apr 06, 2022 at 09:49:42AM +0100, Catalin Marinas wrote:
>
> Since ARCH_KMALLOC_MINALIGN on arm64 prior to this series is 128, there
> is any change to the crypto code.

But the crypto API assumes that memory returned by kmalloc is
automatically aligned to CRYPTO_MINALIGN, would this still be
the case if you change it to ARCH_DMA_MINALIGN?

Thanks,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Muchun Song]

On Tue, Apr 05, 2022 at 02:57:55PM +0100, Catalin Marinas wrote:
> ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA
> operations while ARCH_KMALLOC_MINALIGN is the minimum kmalloc() objects
> alignment.
> 
> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
> Cc: Herbert Xu <herbert@gondor.apana.org.au>
> Cc: "David S. Miller" <davem@davemloft.net>
> ---
>  include/linux/crypto.h | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/include/linux/crypto.h b/include/linux/crypto.h
> index 2324ab6f1846..654b9c355575 100644
> --- a/include/linux/crypto.h
> +++ b/include/linux/crypto.h
> @@ -167,7 +167,7 @@
>   * maintenance for non-coherent DMA (cache invalidation in particular) does not
>   * affect data that may be accessed by the CPU concurrently.
>   */
> -#define CRYPTO_MINALIGN ARCH_KMALLOC_MINALIGN
> +#define CRYPTO_MINALIGN ARCH_DMA_MINALIGN
>

I don't think this should be changed since ARCH_KMALLOC_MINALIGN is
already aligned with the size what you need.

Thanks.

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Muchun Song]

On Tue, Apr 05, 2022 at 02:57:55PM +0100, Catalin Marinas wrote:
> ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA
> operations while ARCH_KMALLOC_MINALIGN is the minimum kmalloc() objects
> alignment.
> 
> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
> Cc: Herbert Xu <herbert@gondor.apana.org.au>
> Cc: "David S. Miller" <davem@davemloft.net>
> ---
>  include/linux/crypto.h | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/include/linux/crypto.h b/include/linux/crypto.h
> index 2324ab6f1846..654b9c355575 100644
> --- a/include/linux/crypto.h
> +++ b/include/linux/crypto.h
> @@ -167,7 +167,7 @@
>   * maintenance for non-coherent DMA (cache invalidation in particular) does not
>   * affect data that may be accessed by the CPU concurrently.
>   */
> -#define CRYPTO_MINALIGN ARCH_KMALLOC_MINALIGN
> +#define CRYPTO_MINALIGN ARCH_DMA_MINALIGN
>

I don't think this should be changed since ARCH_KMALLOC_MINALIGN is
already aligned with the size what you need.

Thanks.


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 08/10] mm/slab: Allow dynamic kmalloc() minimum alignment

[Catalin Marinas]

On Thu, Apr 07, 2022 at 03:46:37AM +0000, Hyeonggon Yoo wrote:
> On Tue, Apr 05, 2022 at 02:57:56PM +0100, Catalin Marinas wrote:
> > --- a/mm/slab_common.c
> > +++ b/mm/slab_common.c
> > @@ -838,9 +838,18 @@ void __init setup_kmalloc_cache_index_table(void)
> >  	}
> >  }
> >  
> > -static void __init
> > +unsigned int __weak arch_kmalloc_minalign(void)
> > +{
> > +	return ARCH_KMALLOC_MINALIGN;
> > +}
> > +
> 
> As ARCH_KMALLOC_ALIGN and arch_kmalloc_minalign() may not be same after
> patch 10, I think s/ARCH_KMALLOC_ALIGN/arch_kmalloc_minalign/g
> for every user of it would be more correct?

Not if the code currently using ARCH_KMALLOC_MINALIGN needs a constant.
Yes, there probably are a few places where the code can cope with a
dynamic arch_kmalloc_minalign() but there are two other cases where a
constant is needed:

1. As a BUILD_BUG check because the code is storing some flags in the
   bottom bits of a pointer. A smaller ARCH_KMALLOC_MINALIGN works just
   fine here.

2. As a static alignment for DMA requirements. That's where the newly
   exposed ARCH_DMA_MINALIGN should be used.

Note that this series doesn't make the situation any worse than before
since ARCH_DMA_MINALIGN stays at 128 bytes for arm64. Current users can
evolve to use a dynamic alignment in future patches. My main aim with
this series is to be able to create kmalloc-64 caches on arm64.

> > @@ -851,10 +860,17 @@ new_kmalloc_cache(int idx, enum kmalloc_cache_type type, slab_flags_t flags)
> >  		flags |= SLAB_ACCOUNT;
> >  	}
> >  
> > -	kmalloc_caches[type][idx] = create_kmalloc_cache(
> > -					kmalloc_info[idx].name[type],
> > -					kmalloc_info[idx].size, flags, 0,
> > -					kmalloc_info[idx].size);
> > +	if (minalign > ARCH_KMALLOC_MINALIGN) {
> > +		aligned_size = ALIGN(aligned_size, minalign);
> > +		aligned_idx = __kmalloc_index(aligned_size, false);
> > +	}
> > +
> > +	if (!kmalloc_caches[type][aligned_idx])
> > +		kmalloc_caches[type][aligned_idx] = create_kmalloc_cache(
> > +					kmalloc_info[aligned_idx].name[type],
> > +					aligned_size, flags, 0, aligned_size);
> > +	if (idx != aligned_idx)
> > +		kmalloc_caches[type][idx] = kmalloc_caches[type][aligned_idx];
> 
> I would prefer detecting minimum kmalloc size in create_kmalloc_caches()
> in runtime instead of changing behavior of new_kmalloc_cache().

That was my initial attempt but we have a couple of
create_kmalloc_cache() (not *_caches) calls directly, one of them in
mm/slab.c kmem_cache_init(). So I wanted all the minalign logic in a
single place, hence I replaced the explicit create_kmalloc_cache() call
with new_kmalloc_cache(). See this patch and patch 9 for some clean-up.

-- 
Catalin

Re: [PATCH 08/10] mm/slab: Allow dynamic kmalloc() minimum alignment

[Catalin Marinas]

On Thu, Apr 07, 2022 at 03:46:37AM +0000, Hyeonggon Yoo wrote:
> On Tue, Apr 05, 2022 at 02:57:56PM +0100, Catalin Marinas wrote:
> > --- a/mm/slab_common.c
> > +++ b/mm/slab_common.c
> > @@ -838,9 +838,18 @@ void __init setup_kmalloc_cache_index_table(void)
> >  	}
> >  }
> >  
> > -static void __init
> > +unsigned int __weak arch_kmalloc_minalign(void)
> > +{
> > +	return ARCH_KMALLOC_MINALIGN;
> > +}
> > +
> 
> As ARCH_KMALLOC_ALIGN and arch_kmalloc_minalign() may not be same after
> patch 10, I think s/ARCH_KMALLOC_ALIGN/arch_kmalloc_minalign/g
> for every user of it would be more correct?

Not if the code currently using ARCH_KMALLOC_MINALIGN needs a constant.
Yes, there probably are a few places where the code can cope with a
dynamic arch_kmalloc_minalign() but there are two other cases where a
constant is needed:

1. As a BUILD_BUG check because the code is storing some flags in the
   bottom bits of a pointer. A smaller ARCH_KMALLOC_MINALIGN works just
   fine here.

2. As a static alignment for DMA requirements. That's where the newly
   exposed ARCH_DMA_MINALIGN should be used.

Note that this series doesn't make the situation any worse than before
since ARCH_DMA_MINALIGN stays at 128 bytes for arm64. Current users can
evolve to use a dynamic alignment in future patches. My main aim with
this series is to be able to create kmalloc-64 caches on arm64.

> > @@ -851,10 +860,17 @@ new_kmalloc_cache(int idx, enum kmalloc_cache_type type, slab_flags_t flags)
> >  		flags |= SLAB_ACCOUNT;
> >  	}
> >  
> > -	kmalloc_caches[type][idx] = create_kmalloc_cache(
> > -					kmalloc_info[idx].name[type],
> > -					kmalloc_info[idx].size, flags, 0,
> > -					kmalloc_info[idx].size);
> > +	if (minalign > ARCH_KMALLOC_MINALIGN) {
> > +		aligned_size = ALIGN(aligned_size, minalign);
> > +		aligned_idx = __kmalloc_index(aligned_size, false);
> > +	}
> > +
> > +	if (!kmalloc_caches[type][aligned_idx])
> > +		kmalloc_caches[type][aligned_idx] = create_kmalloc_cache(
> > +					kmalloc_info[aligned_idx].name[type],
> > +					aligned_size, flags, 0, aligned_size);
> > +	if (idx != aligned_idx)
> > +		kmalloc_caches[type][idx] = kmalloc_caches[type][aligned_idx];
> 
> I would prefer detecting minimum kmalloc size in create_kmalloc_caches()
> in runtime instead of changing behavior of new_kmalloc_cache().

That was my initial attempt but we have a couple of
create_kmalloc_cache() (not *_caches) calls directly, one of them in
mm/slab.c kmem_cache_init(). So I wanted all the minalign logic in a
single place, hence I replaced the explicit create_kmalloc_cache() call
with new_kmalloc_cache(). See this patch and patch 9 for some clean-up.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 08/10] mm/slab: Allow dynamic kmalloc() minimum alignment

[Hyeonggon Yoo]

On Thu, Apr 07, 2022 at 09:50:23AM +0100, Catalin Marinas wrote:
> On Thu, Apr 07, 2022 at 03:46:37AM +0000, Hyeonggon Yoo wrote:
> > On Tue, Apr 05, 2022 at 02:57:56PM +0100, Catalin Marinas wrote:
> > > --- a/mm/slab_common.c
> > > +++ b/mm/slab_common.c
> > > @@ -838,9 +838,18 @@ void __init setup_kmalloc_cache_index_table(void)
> > >  	}
> > >  }
> > >  
> > > -static void __init
> > > +unsigned int __weak arch_kmalloc_minalign(void)
> > > +{
> > > +	return ARCH_KMALLOC_MINALIGN;
> > > +}
> > > +
> > 
> > As ARCH_KMALLOC_ALIGN and arch_kmalloc_minalign() may not be same after
> > patch 10, I think s/ARCH_KMALLOC_ALIGN/arch_kmalloc_minalign/g
> > for every user of it would be more correct?
> 
> Not if the code currently using ARCH_KMALLOC_MINALIGN needs a constant.
> Yes, there probably are a few places where the code can cope with a
> dynamic arch_kmalloc_minalign() but there are two other cases where a
> constant is needed:
> 
> 1. As a BUILD_BUG check because the code is storing some flags in the
>    bottom bits of a pointer. A smaller ARCH_KMALLOC_MINALIGN works just
>    fine here.
> 
> 2. As a static alignment for DMA requirements. That's where the newly
>    exposed ARCH_DMA_MINALIGN should be used.
> 
> Note that this series doesn't make the situation any worse than before
> since ARCH_DMA_MINALIGN stays at 128 bytes for arm64. Current users can
> evolve to use a dynamic alignment in future patches. My main aim with
> this series is to be able to create kmalloc-64 caches on arm64.

AFAIK there are bunch of drivers that directly calls kmalloc().
It becomes tricky when e.g.) a driver allocates just 32 bytes,
but architecture requires it to be 128-byte aligned.

That's why everything allocated from kmalloc() need to be aligned in
ARCH_DMA_MINALIGN. And It's too hard to update all of drivers
that depends on this fact.

So I'm yet skeptical on decoupling ARCH_DMA/KMALLOC_MINALIGN.
Instead of decoupling it, I'm more into dynamically decreasing it.

Please kindly let me know If I'm missing something ;-)

> > > @@ -851,10 +860,17 @@ new_kmalloc_cache(int idx, enum kmalloc_cache_type type, slab_flags_t flags)
> > >  		flags |= SLAB_ACCOUNT;
> > >  	}
> > >  
> > > -	kmalloc_caches[type][idx] = create_kmalloc_cache(
> > > -					kmalloc_info[idx].name[type],
> > > -					kmalloc_info[idx].size, flags, 0,
> > > -					kmalloc_info[idx].size);
> > > +	if (minalign > ARCH_KMALLOC_MINALIGN) {
> > > +		aligned_size = ALIGN(aligned_size, minalign);
> > > +		aligned_idx = __kmalloc_index(aligned_size, false);
> > > +	}
> > > +
> > > +	if (!kmalloc_caches[type][aligned_idx])
> > > +		kmalloc_caches[type][aligned_idx] = create_kmalloc_cache(
> > > +					kmalloc_info[aligned_idx].name[type],
> > > +					aligned_size, flags, 0, aligned_size);
> > > +	if (idx != aligned_idx)
> > > +		kmalloc_caches[type][idx] = kmalloc_caches[type][aligned_idx];
> > 
> > I would prefer detecting minimum kmalloc size in create_kmalloc_caches()
> > in runtime instead of changing behavior of new_kmalloc_cache().
> 
> That was my initial attempt but we have a couple of
> create_kmalloc_cache() (not *_caches) calls directly, one of them in
> mm/slab.c kmem_cache_init(). So I wanted all the minalign logic in a
> single place, hence I replaced the explicit create_kmalloc_cache() call
> with new_kmalloc_cache(). See this patch and patch 9 for some clean-up.
> 
> -- 
> Catalin

-- 
Thanks,
Hyeonggon

Re: [PATCH 08/10] mm/slab: Allow dynamic kmalloc() minimum alignment

[Hyeonggon Yoo]

On Thu, Apr 07, 2022 at 09:50:23AM +0100, Catalin Marinas wrote:
> On Thu, Apr 07, 2022 at 03:46:37AM +0000, Hyeonggon Yoo wrote:
> > On Tue, Apr 05, 2022 at 02:57:56PM +0100, Catalin Marinas wrote:
> > > --- a/mm/slab_common.c
> > > +++ b/mm/slab_common.c
> > > @@ -838,9 +838,18 @@ void __init setup_kmalloc_cache_index_table(void)
> > >  	}
> > >  }
> > >  
> > > -static void __init
> > > +unsigned int __weak arch_kmalloc_minalign(void)
> > > +{
> > > +	return ARCH_KMALLOC_MINALIGN;
> > > +}
> > > +
> > 
> > As ARCH_KMALLOC_ALIGN and arch_kmalloc_minalign() may not be same after
> > patch 10, I think s/ARCH_KMALLOC_ALIGN/arch_kmalloc_minalign/g
> > for every user of it would be more correct?
> 
> Not if the code currently using ARCH_KMALLOC_MINALIGN needs a constant.
> Yes, there probably are a few places where the code can cope with a
> dynamic arch_kmalloc_minalign() but there are two other cases where a
> constant is needed:
> 
> 1. As a BUILD_BUG check because the code is storing some flags in the
>    bottom bits of a pointer. A smaller ARCH_KMALLOC_MINALIGN works just
>    fine here.
> 
> 2. As a static alignment for DMA requirements. That's where the newly
>    exposed ARCH_DMA_MINALIGN should be used.
> 
> Note that this series doesn't make the situation any worse than before
> since ARCH_DMA_MINALIGN stays at 128 bytes for arm64. Current users can
> evolve to use a dynamic alignment in future patches. My main aim with
> this series is to be able to create kmalloc-64 caches on arm64.

AFAIK there are bunch of drivers that directly calls kmalloc().
It becomes tricky when e.g.) a driver allocates just 32 bytes,
but architecture requires it to be 128-byte aligned.

That's why everything allocated from kmalloc() need to be aligned in
ARCH_DMA_MINALIGN. And It's too hard to update all of drivers
that depends on this fact.

So I'm yet skeptical on decoupling ARCH_DMA/KMALLOC_MINALIGN.
Instead of decoupling it, I'm more into dynamically decreasing it.

Please kindly let me know If I'm missing something ;-)

> > > @@ -851,10 +860,17 @@ new_kmalloc_cache(int idx, enum kmalloc_cache_type type, slab_flags_t flags)
> > >  		flags |= SLAB_ACCOUNT;
> > >  	}
> > >  
> > > -	kmalloc_caches[type][idx] = create_kmalloc_cache(
> > > -					kmalloc_info[idx].name[type],
> > > -					kmalloc_info[idx].size, flags, 0,
> > > -					kmalloc_info[idx].size);
> > > +	if (minalign > ARCH_KMALLOC_MINALIGN) {
> > > +		aligned_size = ALIGN(aligned_size, minalign);
> > > +		aligned_idx = __kmalloc_index(aligned_size, false);
> > > +	}
> > > +
> > > +	if (!kmalloc_caches[type][aligned_idx])
> > > +		kmalloc_caches[type][aligned_idx] = create_kmalloc_cache(
> > > +					kmalloc_info[aligned_idx].name[type],
> > > +					aligned_size, flags, 0, aligned_size);
> > > +	if (idx != aligned_idx)
> > > +		kmalloc_caches[type][idx] = kmalloc_caches[type][aligned_idx];
> > 
> > I would prefer detecting minimum kmalloc size in create_kmalloc_caches()
> > in runtime instead of changing behavior of new_kmalloc_cache().
> 
> That was my initial attempt but we have a couple of
> create_kmalloc_cache() (not *_caches) calls directly, one of them in
> mm/slab.c kmem_cache_init(). So I wanted all the minalign logic in a
> single place, hence I replaced the explicit create_kmalloc_cache() call
> with new_kmalloc_cache(). See this patch and patch 9 for some clean-up.
> 
> -- 
> Catalin

-- 
Thanks,
Hyeonggon

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Thu, Apr 07, 2022 at 02:14:15PM +0800, Muchun Song wrote:
> On Tue, Apr 05, 2022 at 02:57:55PM +0100, Catalin Marinas wrote:
> > ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA
> > operations while ARCH_KMALLOC_MINALIGN is the minimum kmalloc() objects
> > alignment.
> > 
> > Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
> > Cc: Herbert Xu <herbert@gondor.apana.org.au>
> > Cc: "David S. Miller" <davem@davemloft.net>
> > ---
> >  include/linux/crypto.h | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/include/linux/crypto.h b/include/linux/crypto.h
> > index 2324ab6f1846..654b9c355575 100644
> > --- a/include/linux/crypto.h
> > +++ b/include/linux/crypto.h
> > @@ -167,7 +167,7 @@
> >   * maintenance for non-coherent DMA (cache invalidation in particular) does not
> >   * affect data that may be accessed by the CPU concurrently.
> >   */
> > -#define CRYPTO_MINALIGN ARCH_KMALLOC_MINALIGN
> > +#define CRYPTO_MINALIGN ARCH_DMA_MINALIGN
> 
> I don't think this should be changed since ARCH_KMALLOC_MINALIGN is
> already aligned with the size what you need.

With this series, ARCH_KMALLOC_MINALIGN is no longer safe for
non-coherent DMA on all arm64 SoCs, that's what ARCH_DMA_MINALIGN will
cover.

Now, looking at the comment for CRYPTO_MINALIGN, one aspect it covers is
the minimum alignment required by C for the crypto_tfm structure access.
So a smaller ARCH_KMALLOC_MINALIGN would do. But the other part of the
comment mentions in-structure alignment for non-coherent DMA. Here we'd
need the upper bound alignment, ARCH_DMA_MINALIGN.

I'll follow up on Herbert's email as I think he has a good point on
structure vs kmalloc() alignment.

-- 
Catalin

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Thu, Apr 07, 2022 at 02:14:15PM +0800, Muchun Song wrote:
> On Tue, Apr 05, 2022 at 02:57:55PM +0100, Catalin Marinas wrote:
> > ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA
> > operations while ARCH_KMALLOC_MINALIGN is the minimum kmalloc() objects
> > alignment.
> > 
> > Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
> > Cc: Herbert Xu <herbert@gondor.apana.org.au>
> > Cc: "David S. Miller" <davem@davemloft.net>
> > ---
> >  include/linux/crypto.h | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/include/linux/crypto.h b/include/linux/crypto.h
> > index 2324ab6f1846..654b9c355575 100644
> > --- a/include/linux/crypto.h
> > +++ b/include/linux/crypto.h
> > @@ -167,7 +167,7 @@
> >   * maintenance for non-coherent DMA (cache invalidation in particular) does not
> >   * affect data that may be accessed by the CPU concurrently.
> >   */
> > -#define CRYPTO_MINALIGN ARCH_KMALLOC_MINALIGN
> > +#define CRYPTO_MINALIGN ARCH_DMA_MINALIGN
> 
> I don't think this should be changed since ARCH_KMALLOC_MINALIGN is
> already aligned with the size what you need.

With this series, ARCH_KMALLOC_MINALIGN is no longer safe for
non-coherent DMA on all arm64 SoCs, that's what ARCH_DMA_MINALIGN will
cover.

Now, looking at the comment for CRYPTO_MINALIGN, one aspect it covers is
the minimum alignment required by C for the crypto_tfm structure access.
So a smaller ARCH_KMALLOC_MINALIGN would do. But the other part of the
comment mentions in-structure alignment for non-coherent DMA. Here we'd
need the upper bound alignment, ARCH_DMA_MINALIGN.

I'll follow up on Herbert's email as I think he has a good point on
structure vs kmalloc() alignment.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 08/10] mm/slab: Allow dynamic kmalloc() minimum alignment

[Catalin Marinas]

On Thu, Apr 07, 2022 at 06:18:16PM +0900, Hyeonggon Yoo wrote:
> On Thu, Apr 07, 2022 at 09:50:23AM +0100, Catalin Marinas wrote:
> > On Thu, Apr 07, 2022 at 03:46:37AM +0000, Hyeonggon Yoo wrote:
> > > On Tue, Apr 05, 2022 at 02:57:56PM +0100, Catalin Marinas wrote:
> > > > --- a/mm/slab_common.c
> > > > +++ b/mm/slab_common.c
> > > > @@ -838,9 +838,18 @@ void __init setup_kmalloc_cache_index_table(void)
> > > >  	}
> > > >  }
> > > >  
> > > > -static void __init
> > > > +unsigned int __weak arch_kmalloc_minalign(void)
> > > > +{
> > > > +	return ARCH_KMALLOC_MINALIGN;
> > > > +}
> > > > +
> > > 
> > > As ARCH_KMALLOC_ALIGN and arch_kmalloc_minalign() may not be same after
> > > patch 10, I think s/ARCH_KMALLOC_ALIGN/arch_kmalloc_minalign/g
> > > for every user of it would be more correct?
> > 
> > Not if the code currently using ARCH_KMALLOC_MINALIGN needs a constant.
> > Yes, there probably are a few places where the code can cope with a
> > dynamic arch_kmalloc_minalign() but there are two other cases where a
> > constant is needed:
> > 
> > 1. As a BUILD_BUG check because the code is storing some flags in the
> >    bottom bits of a pointer. A smaller ARCH_KMALLOC_MINALIGN works just
> >    fine here.
> > 
> > 2. As a static alignment for DMA requirements. That's where the newly
> >    exposed ARCH_DMA_MINALIGN should be used.
> > 
> > Note that this series doesn't make the situation any worse than before
> > since ARCH_DMA_MINALIGN stays at 128 bytes for arm64. Current users can
> > evolve to use a dynamic alignment in future patches. My main aim with
> > this series is to be able to create kmalloc-64 caches on arm64.
> 
> AFAIK there are bunch of drivers that directly calls kmalloc().

Well, lots of drivers call kmalloc() ;).

> It becomes tricky when e.g.) a driver allocates just 32 bytes,
> but architecture requires it to be 128-byte aligned.

That's the current behaviour, a 32 byte allocation would return an
object from kmalloc-128. I want to reduce this to at least kmalloc-64
(or smaller) if the CPU/SoC allows it.

> That's why everything allocated from kmalloc() need to be aligned in
> ARCH_DMA_MINALIGN.

I don't get your conclusion here. Would you mind explaining?

> So I'm yet skeptical on decoupling ARCH_DMA/KMALLOC_MINALIGN. Instead
> of decoupling it, I'm more into dynamically decreasing it.

The reason for decoupling is mostly that there are some static uses of
ARCH_KMALLOC_MINALIGN as per point 1 above. The other is the
__assume_kmalloc_alignment attribute. We shouldn't have such assumed
alignment larger than what a dynamic kmalloc() would return. To me it
makes a lot more sense for ARCH_KMALLOC_MINALIGN to be the minimum
guaranteed in a kernel build but kmalloc() returning a larger alignment
at run-time than the other way around.

Thanks.

-- 
Catalin

Re: [PATCH 08/10] mm/slab: Allow dynamic kmalloc() minimum alignment

[Catalin Marinas]

On Thu, Apr 07, 2022 at 06:18:16PM +0900, Hyeonggon Yoo wrote:
> On Thu, Apr 07, 2022 at 09:50:23AM +0100, Catalin Marinas wrote:
> > On Thu, Apr 07, 2022 at 03:46:37AM +0000, Hyeonggon Yoo wrote:
> > > On Tue, Apr 05, 2022 at 02:57:56PM +0100, Catalin Marinas wrote:
> > > > --- a/mm/slab_common.c
> > > > +++ b/mm/slab_common.c
> > > > @@ -838,9 +838,18 @@ void __init setup_kmalloc_cache_index_table(void)
> > > >  	}
> > > >  }
> > > >  
> > > > -static void __init
> > > > +unsigned int __weak arch_kmalloc_minalign(void)
> > > > +{
> > > > +	return ARCH_KMALLOC_MINALIGN;
> > > > +}
> > > > +
> > > 
> > > As ARCH_KMALLOC_ALIGN and arch_kmalloc_minalign() may not be same after
> > > patch 10, I think s/ARCH_KMALLOC_ALIGN/arch_kmalloc_minalign/g
> > > for every user of it would be more correct?
> > 
> > Not if the code currently using ARCH_KMALLOC_MINALIGN needs a constant.
> > Yes, there probably are a few places where the code can cope with a
> > dynamic arch_kmalloc_minalign() but there are two other cases where a
> > constant is needed:
> > 
> > 1. As a BUILD_BUG check because the code is storing some flags in the
> >    bottom bits of a pointer. A smaller ARCH_KMALLOC_MINALIGN works just
> >    fine here.
> > 
> > 2. As a static alignment for DMA requirements. That's where the newly
> >    exposed ARCH_DMA_MINALIGN should be used.
> > 
> > Note that this series doesn't make the situation any worse than before
> > since ARCH_DMA_MINALIGN stays at 128 bytes for arm64. Current users can
> > evolve to use a dynamic alignment in future patches. My main aim with
> > this series is to be able to create kmalloc-64 caches on arm64.
> 
> AFAIK there are bunch of drivers that directly calls kmalloc().

Well, lots of drivers call kmalloc() ;).

> It becomes tricky when e.g.) a driver allocates just 32 bytes,
> but architecture requires it to be 128-byte aligned.

That's the current behaviour, a 32 byte allocation would return an
object from kmalloc-128. I want to reduce this to at least kmalloc-64
(or smaller) if the CPU/SoC allows it.

> That's why everything allocated from kmalloc() need to be aligned in
> ARCH_DMA_MINALIGN.

I don't get your conclusion here. Would you mind explaining?

> So I'm yet skeptical on decoupling ARCH_DMA/KMALLOC_MINALIGN. Instead
> of decoupling it, I'm more into dynamically decreasing it.

The reason for decoupling is mostly that there are some static uses of
ARCH_KMALLOC_MINALIGN as per point 1 above. The other is the
__assume_kmalloc_alignment attribute. We shouldn't have such assumed
alignment larger than what a dynamic kmalloc() would return. To me it
makes a lot more sense for ARCH_KMALLOC_MINALIGN to be the minimum
guaranteed in a kernel build but kmalloc() returning a larger alignment
at run-time than the other way around.

Thanks.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Muchun Song]

On Thu, Apr 7, 2022 at 5:25 PM Catalin Marinas <catalin.marinas@arm.com> wrote:
>
> On Thu, Apr 07, 2022 at 02:14:15PM +0800, Muchun Song wrote:
> > On Tue, Apr 05, 2022 at 02:57:55PM +0100, Catalin Marinas wrote:
> > > ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA
> > > operations while ARCH_KMALLOC_MINALIGN is the minimum kmalloc() objects
> > > alignment.
> > >
> > > Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
> > > Cc: Herbert Xu <herbert@gondor.apana.org.au>
> > > Cc: "David S. Miller" <davem@davemloft.net>
> > > ---
> > >  include/linux/crypto.h | 2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > >
> > > diff --git a/include/linux/crypto.h b/include/linux/crypto.h
> > > index 2324ab6f1846..654b9c355575 100644
> > > --- a/include/linux/crypto.h
> > > +++ b/include/linux/crypto.h
> > > @@ -167,7 +167,7 @@
> > >   * maintenance for non-coherent DMA (cache invalidation in particular) does not
> > >   * affect data that may be accessed by the CPU concurrently.
> > >   */
> > > -#define CRYPTO_MINALIGN ARCH_KMALLOC_MINALIGN
> > > +#define CRYPTO_MINALIGN ARCH_DMA_MINALIGN
> >
> > I don't think this should be changed since ARCH_KMALLOC_MINALIGN is
> > already aligned with the size what you need.
>
> With this series, ARCH_KMALLOC_MINALIGN is no longer safe for
> non-coherent DMA on all arm64 SoCs, that's what ARCH_DMA_MINALIGN will
> cover.
>
> Now, looking at the comment for CRYPTO_MINALIGN, one aspect it covers is
> the minimum alignment required by C for the crypto_tfm structure access.
> So a smaller ARCH_KMALLOC_MINALIGN would do. But the other part of the
> comment mentions in-structure alignment for non-coherent DMA. Here we'd
> need the upper bound alignment, ARCH_DMA_MINALIGN.
>
> I'll follow up on Herbert's email as I think he has a good point on
> structure vs kmalloc() alignment.

Got it. Now I know what you want to do. You want to set
ARCH_KMALLOC_MINALIGN to 64, however, the smallest
size of kmem_cache depends on the cache line size at
runtime.  But we have to know the safe alignment at building
time.  So we have to make those align with ARCH_DMA_MINALIGN.
Right?  I think you are on the right road since most CPUs have
a 64-byte cache line.

Thanks.

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Muchun Song]

On Thu, Apr 7, 2022 at 5:25 PM Catalin Marinas <catalin.marinas@arm.com> wrote:
>
> On Thu, Apr 07, 2022 at 02:14:15PM +0800, Muchun Song wrote:
> > On Tue, Apr 05, 2022 at 02:57:55PM +0100, Catalin Marinas wrote:
> > > ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA
> > > operations while ARCH_KMALLOC_MINALIGN is the minimum kmalloc() objects
> > > alignment.
> > >
> > > Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
> > > Cc: Herbert Xu <herbert@gondor.apana.org.au>
> > > Cc: "David S. Miller" <davem@davemloft.net>
> > > ---
> > >  include/linux/crypto.h | 2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > >
> > > diff --git a/include/linux/crypto.h b/include/linux/crypto.h
> > > index 2324ab6f1846..654b9c355575 100644
> > > --- a/include/linux/crypto.h
> > > +++ b/include/linux/crypto.h
> > > @@ -167,7 +167,7 @@
> > >   * maintenance for non-coherent DMA (cache invalidation in particular) does not
> > >   * affect data that may be accessed by the CPU concurrently.
> > >   */
> > > -#define CRYPTO_MINALIGN ARCH_KMALLOC_MINALIGN
> > > +#define CRYPTO_MINALIGN ARCH_DMA_MINALIGN
> >
> > I don't think this should be changed since ARCH_KMALLOC_MINALIGN is
> > already aligned with the size what you need.
>
> With this series, ARCH_KMALLOC_MINALIGN is no longer safe for
> non-coherent DMA on all arm64 SoCs, that's what ARCH_DMA_MINALIGN will
> cover.
>
> Now, looking at the comment for CRYPTO_MINALIGN, one aspect it covers is
> the minimum alignment required by C for the crypto_tfm structure access.
> So a smaller ARCH_KMALLOC_MINALIGN would do. But the other part of the
> comment mentions in-structure alignment for non-coherent DMA. Here we'd
> need the upper bound alignment, ARCH_DMA_MINALIGN.
>
> I'll follow up on Herbert's email as I think he has a good point on
> structure vs kmalloc() alignment.

Got it. Now I know what you want to do. You want to set
ARCH_KMALLOC_MINALIGN to 64, however, the smallest
size of kmem_cache depends on the cache line size at
runtime.  But we have to know the safe alignment at building
time.  So we have to make those align with ARCH_DMA_MINALIGN.
Right?  I think you are on the right road since most CPUs have
a 64-byte cache line.

Thanks.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Thu, Apr 07, 2022 at 02:30:54PM +1000, Herbert Xu wrote:
> On Wed, Apr 06, 2022 at 09:49:42AM +0100, Catalin Marinas wrote:
> > is any change to the crypto code.
> 
> But the crypto API assumes that memory returned by kmalloc is
> automatically aligned to CRYPTO_MINALIGN, would this still be
> the case if you change it to ARCH_DMA_MINALIGN?

No but I think that's a valid point. Taking the crypto_tfm structure as
an example with ARCH_DMA_MINALIGN of 128:

#define CRYPTO_MINALIGN 128
#define CRYPTO_MINALIGN_ATTR __attribute__ ((__aligned__(CRYPTO_MINALIGN)))

struct crypto_tfm {
	u32 crt_flags;
	int node;
	void (*exit)(struct crypto_tfm *tfm);
	struct crypto_alg *__crt_alg;
	void *__crt_ctx[] CRYPTO_MINALIGN_ATTR;
};

The alignof(struct crypto_tfm) is 128. However, a kmalloc() would only
guarantee the smaller ARCH_KMALLOC_MINALIGN which, after this series,
would be 64 for arm64. From the DMA perspective there's no issue with
the smaller kmalloc() alignment since, if a crypto_tfm pointer is
DMA-aligned for the hardware it is running on, so would __ctr_ctx[] at
an offset multiple of the dynamic DMA alignment. If we used
ARCH_KMALLOC_MINALIGN instead and the hardware alignment requirement was
larger, than we would have a potential problem with non-coherent DMA.

The only issue is whether the compiler gets confused by a pointer to a
structure with a smaller alignment than alignof(struct ...). I don't see
a performance or correctness issue on arm64 here. It would be a problem
if instead of 16 we went down to 8 or 4 due to unaligned accesses but
from 128 to 64 (or even 16), I don't think it matters.

-- 
Catalin

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Thu, Apr 07, 2022 at 02:30:54PM +1000, Herbert Xu wrote:
> On Wed, Apr 06, 2022 at 09:49:42AM +0100, Catalin Marinas wrote:
> > is any change to the crypto code.
> 
> But the crypto API assumes that memory returned by kmalloc is
> automatically aligned to CRYPTO_MINALIGN, would this still be
> the case if you change it to ARCH_DMA_MINALIGN?

No but I think that's a valid point. Taking the crypto_tfm structure as
an example with ARCH_DMA_MINALIGN of 128:

#define CRYPTO_MINALIGN 128
#define CRYPTO_MINALIGN_ATTR __attribute__ ((__aligned__(CRYPTO_MINALIGN)))

struct crypto_tfm {
	u32 crt_flags;
	int node;
	void (*exit)(struct crypto_tfm *tfm);
	struct crypto_alg *__crt_alg;
	void *__crt_ctx[] CRYPTO_MINALIGN_ATTR;
};

The alignof(struct crypto_tfm) is 128. However, a kmalloc() would only
guarantee the smaller ARCH_KMALLOC_MINALIGN which, after this series,
would be 64 for arm64. From the DMA perspective there's no issue with
the smaller kmalloc() alignment since, if a crypto_tfm pointer is
DMA-aligned for the hardware it is running on, so would __ctr_ctx[] at
an offset multiple of the dynamic DMA alignment. If we used
ARCH_KMALLOC_MINALIGN instead and the hardware alignment requirement was
larger, than we would have a potential problem with non-coherent DMA.

The only issue is whether the compiler gets confused by a pointer to a
structure with a smaller alignment than alignof(struct ...). I don't see
a performance or correctness issue on arm64 here. It would be a problem
if instead of 16 we went down to 8 or 4 due to unaligned accesses but
from 128 to 64 (or even 16), I don't think it matters.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Thu, Apr 07, 2022 at 06:00:10PM +0800, Muchun Song wrote:
> On Thu, Apr 7, 2022 at 5:25 PM Catalin Marinas <catalin.marinas@arm.com> wrote:
> > On Thu, Apr 07, 2022 at 02:14:15PM +0800, Muchun Song wrote:
> > > On Tue, Apr 05, 2022 at 02:57:55PM +0100, Catalin Marinas wrote:
> > > > ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA
> > > > operations while ARCH_KMALLOC_MINALIGN is the minimum kmalloc() objects
> > > > alignment.
> > > >
> > > > Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
> > > > Cc: Herbert Xu <herbert@gondor.apana.org.au>
> > > > Cc: "David S. Miller" <davem@davemloft.net>
> > > > ---
> > > >  include/linux/crypto.h | 2 +-
> > > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > >
> > > > diff --git a/include/linux/crypto.h b/include/linux/crypto.h
> > > > index 2324ab6f1846..654b9c355575 100644
> > > > --- a/include/linux/crypto.h
> > > > +++ b/include/linux/crypto.h
> > > > @@ -167,7 +167,7 @@
> > > >   * maintenance for non-coherent DMA (cache invalidation in particular) does not
> > > >   * affect data that may be accessed by the CPU concurrently.
> > > >   */
> > > > -#define CRYPTO_MINALIGN ARCH_KMALLOC_MINALIGN
> > > > +#define CRYPTO_MINALIGN ARCH_DMA_MINALIGN
> > >
> > > I don't think this should be changed since ARCH_KMALLOC_MINALIGN is
> > > already aligned with the size what you need.
> >
> > With this series, ARCH_KMALLOC_MINALIGN is no longer safe for
> > non-coherent DMA on all arm64 SoCs, that's what ARCH_DMA_MINALIGN will
> > cover.
> >
> > Now, looking at the comment for CRYPTO_MINALIGN, one aspect it covers is
> > the minimum alignment required by C for the crypto_tfm structure access.
> > So a smaller ARCH_KMALLOC_MINALIGN would do. But the other part of the
> > comment mentions in-structure alignment for non-coherent DMA. Here we'd
> > need the upper bound alignment, ARCH_DMA_MINALIGN.
> >
> > I'll follow up on Herbert's email as I think he has a good point on
> > structure vs kmalloc() alignment.
> 
> Got it. Now I know what you want to do. You want to set
> ARCH_KMALLOC_MINALIGN to 64, however, the smallest
> size of kmem_cache depends on the cache line size at
> runtime.  But we have to know the safe alignment at building
> time.  So we have to make those align with ARCH_DMA_MINALIGN.
> Right?  

Right.

> I think you are on the right road since most CPUs have a 64-byte cache
> line.

Yeah, apart from about three SoCs with 128, the rest use 64-byte cache
lines.

Longer term we should try reduce such alignment below the cache line
size *if* the SoC fully DMA-coherent (like x86). In that case cache
maintenance for DMA doesn't exist, so there's no risk to smaller
allocations. Of course, one can use a kmem_cache_create() with
SLAB_HWCACHE_ALIGN and get the cacheline alignment for performance
reasons.

-- 
Catalin

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Thu, Apr 07, 2022 at 06:00:10PM +0800, Muchun Song wrote:
> On Thu, Apr 7, 2022 at 5:25 PM Catalin Marinas <catalin.marinas@arm.com> wrote:
> > On Thu, Apr 07, 2022 at 02:14:15PM +0800, Muchun Song wrote:
> > > On Tue, Apr 05, 2022 at 02:57:55PM +0100, Catalin Marinas wrote:
> > > > ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA
> > > > operations while ARCH_KMALLOC_MINALIGN is the minimum kmalloc() objects
> > > > alignment.
> > > >
> > > > Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
> > > > Cc: Herbert Xu <herbert@gondor.apana.org.au>
> > > > Cc: "David S. Miller" <davem@davemloft.net>
> > > > ---
> > > >  include/linux/crypto.h | 2 +-
> > > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > >
> > > > diff --git a/include/linux/crypto.h b/include/linux/crypto.h
> > > > index 2324ab6f1846..654b9c355575 100644
> > > > --- a/include/linux/crypto.h
> > > > +++ b/include/linux/crypto.h
> > > > @@ -167,7 +167,7 @@
> > > >   * maintenance for non-coherent DMA (cache invalidation in particular) does not
> > > >   * affect data that may be accessed by the CPU concurrently.
> > > >   */
> > > > -#define CRYPTO_MINALIGN ARCH_KMALLOC_MINALIGN
> > > > +#define CRYPTO_MINALIGN ARCH_DMA_MINALIGN
> > >
> > > I don't think this should be changed since ARCH_KMALLOC_MINALIGN is
> > > already aligned with the size what you need.
> >
> > With this series, ARCH_KMALLOC_MINALIGN is no longer safe for
> > non-coherent DMA on all arm64 SoCs, that's what ARCH_DMA_MINALIGN will
> > cover.
> >
> > Now, looking at the comment for CRYPTO_MINALIGN, one aspect it covers is
> > the minimum alignment required by C for the crypto_tfm structure access.
> > So a smaller ARCH_KMALLOC_MINALIGN would do. But the other part of the
> > comment mentions in-structure alignment for non-coherent DMA. Here we'd
> > need the upper bound alignment, ARCH_DMA_MINALIGN.
> >
> > I'll follow up on Herbert's email as I think he has a good point on
> > structure vs kmalloc() alignment.
> 
> Got it. Now I know what you want to do. You want to set
> ARCH_KMALLOC_MINALIGN to 64, however, the smallest
> size of kmem_cache depends on the cache line size at
> runtime.  But we have to know the safe alignment at building
> time.  So we have to make those align with ARCH_DMA_MINALIGN.
> Right?  

Right.

> I think you are on the right road since most CPUs have a 64-byte cache
> line.

Yeah, apart from about three SoCs with 128, the rest use 64-byte cache
lines.

Longer term we should try reduce such alignment below the cache line
size *if* the SoC fully DMA-coherent (like x86). In that case cache
maintenance for DMA doesn't exist, so there's no risk to smaller
allocations. Of course, one can use a kmem_cache_create() with
SLAB_HWCACHE_ALIGN and get the cacheline alignment for performance
reasons.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Herbert Xu]

On Thu, Apr 07, 2022 at 12:01:02PM +0100, Catalin Marinas wrote:
>
> The only issue is whether the compiler gets confused by a pointer to a
> structure with a smaller alignment than alignof(struct ...). I don't see
> a performance or correctness issue on arm64 here. It would be a problem
> if instead of 16 we went down to 8 or 4 due to unaligned accesses but
> from 128 to 64 (or even 16), I don't think it matters.

The issue is that there's code in the Crypto API which assumes
that all pointers returned by kmalloc are aligned to CRYPTO_MINALIGN,
if you break that then all that code would need to be modified.

However, I think it's better to change the code that assumes
CRYPTO_MINALIGN guarantees DMA alignment.

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Herbert Xu]

On Thu, Apr 07, 2022 at 12:01:02PM +0100, Catalin Marinas wrote:
>
> The only issue is whether the compiler gets confused by a pointer to a
> structure with a smaller alignment than alignof(struct ...). I don't see
> a performance or correctness issue on arm64 here. It would be a problem
> if instead of 16 we went down to 8 or 4 due to unaligned accesses but
> from 128 to 64 (or even 16), I don't think it matters.

The issue is that there's code in the Crypto API which assumes
that all pointers returned by kmalloc are aligned to CRYPTO_MINALIGN,
if you break that then all that code would need to be modified.

However, I think it's better to change the code that assumes
CRYPTO_MINALIGN guarantees DMA alignment.

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 08/10] mm/slab: Allow dynamic kmalloc() minimum alignment

[Hyeonggon Yoo]

On Thu, Apr 07, 2022 at 10:35:04AM +0100, Catalin Marinas wrote:
> On Thu, Apr 07, 2022 at 06:18:16PM +0900, Hyeonggon Yoo wrote:
> > On Thu, Apr 07, 2022 at 09:50:23AM +0100, Catalin Marinas wrote:
> > > On Thu, Apr 07, 2022 at 03:46:37AM +0000, Hyeonggon Yoo wrote:
> > > > On Tue, Apr 05, 2022 at 02:57:56PM +0100, Catalin Marinas wrote:
> > > > > --- a/mm/slab_common.c
> > > > > +++ b/mm/slab_common.c
> > > > > @@ -838,9 +838,18 @@ void __init setup_kmalloc_cache_index_table(void)
> > > > >  	}
> > > > >  }
> > > > >  
> > > > > -static void __init
> > > > > +unsigned int __weak arch_kmalloc_minalign(void)
> > > > > +{
> > > > > +	return ARCH_KMALLOC_MINALIGN;
> > > > > +}
> > > > > +
> > > > 
> > > > As ARCH_KMALLOC_ALIGN and arch_kmalloc_minalign() may not be same after
> > > > patch 10, I think s/ARCH_KMALLOC_ALIGN/arch_kmalloc_minalign/g
> > > > for every user of it would be more correct?
> > > 
> > > Not if the code currently using ARCH_KMALLOC_MINALIGN needs a constant.
> > > Yes, there probably are a few places where the code can cope with a
> > > dynamic arch_kmalloc_minalign() but there are two other cases where a
> > > constant is needed:
> > > 
> > > 1. As a BUILD_BUG check because the code is storing some flags in the
> > >    bottom bits of a pointer. A smaller ARCH_KMALLOC_MINALIGN works just
> > >    fine here.
> > > 
> > > 2. As a static alignment for DMA requirements. That's where the newly
> > >    exposed ARCH_DMA_MINALIGN should be used.
> > > 
> > > Note that this series doesn't make the situation any worse than before
> > > since ARCH_DMA_MINALIGN stays at 128 bytes for arm64. Current users can
> > > evolve to use a dynamic alignment in future patches. My main aim with
> > > this series is to be able to create kmalloc-64 caches on arm64.
> > 
> > AFAIK there are bunch of drivers that directly calls kmalloc().
> 
> Well, lots of drivers call kmalloc() ;).
> 
> > It becomes tricky when e.g.) a driver allocates just 32 bytes,
> > but architecture requires it to be 128-byte aligned.
> 
> That's the current behaviour, a 32 byte allocation would return an
> object from kmalloc-128. I want to reduce this to at least kmalloc-64
> (or smaller) if the CPU/SoC allows it.

Yeah I agree the change is worth :) Thanks for the work.

> > That's why everything allocated from kmalloc() need to be aligned in
> > ARCH_DMA_MINALIGN.
> 
> I don't get your conclusion here. Would you mind explaining?

What I wanted to say was that, why ARCH_DMA_MINALIGN should be
different from ARCH_KMALLOC_MINALIGN.

I thought the two were basically same thing. Instead of
decoupling them, I thought just decreasing them in runtime makes more sense.

> > So I'm yet skeptical on decoupling ARCH_DMA/KMALLOC_MINALIGN. Instead
> > of decoupling it, I'm more into dynamically decreasing it.
> 
> The reason for decoupling is mostly that there are some static uses of
> ARCH_KMALLOC_MINALIGN as per point 1 above. The other is the
> __assume_kmalloc_alignment attribute. We shouldn't have such assumed
> alignment larger than what a dynamic kmalloc() would return. To me it
> makes a lot more sense for ARCH_KMALLOC_MINALIGN to be the minimum
> guaranteed in a kernel build but kmalloc() returning a larger alignment
> at run-time than the other way around.

But yeah, considering the problems you mentioned, it seems unavoidable
to decouple them.

Thank you for explanation and I will review slab part soon.

> Thanks.
> 
> -- 
> Catalin

-- 
Thanks,
Hyeonggon

Re: [PATCH 08/10] mm/slab: Allow dynamic kmalloc() minimum alignment

[Hyeonggon Yoo]

On Thu, Apr 07, 2022 at 10:35:04AM +0100, Catalin Marinas wrote:
> On Thu, Apr 07, 2022 at 06:18:16PM +0900, Hyeonggon Yoo wrote:
> > On Thu, Apr 07, 2022 at 09:50:23AM +0100, Catalin Marinas wrote:
> > > On Thu, Apr 07, 2022 at 03:46:37AM +0000, Hyeonggon Yoo wrote:
> > > > On Tue, Apr 05, 2022 at 02:57:56PM +0100, Catalin Marinas wrote:
> > > > > --- a/mm/slab_common.c
> > > > > +++ b/mm/slab_common.c
> > > > > @@ -838,9 +838,18 @@ void __init setup_kmalloc_cache_index_table(void)
> > > > >  	}
> > > > >  }
> > > > >  
> > > > > -static void __init
> > > > > +unsigned int __weak arch_kmalloc_minalign(void)
> > > > > +{
> > > > > +	return ARCH_KMALLOC_MINALIGN;
> > > > > +}
> > > > > +
> > > > 
> > > > As ARCH_KMALLOC_ALIGN and arch_kmalloc_minalign() may not be same after
> > > > patch 10, I think s/ARCH_KMALLOC_ALIGN/arch_kmalloc_minalign/g
> > > > for every user of it would be more correct?
> > > 
> > > Not if the code currently using ARCH_KMALLOC_MINALIGN needs a constant.
> > > Yes, there probably are a few places where the code can cope with a
> > > dynamic arch_kmalloc_minalign() but there are two other cases where a
> > > constant is needed:
> > > 
> > > 1. As a BUILD_BUG check because the code is storing some flags in the
> > >    bottom bits of a pointer. A smaller ARCH_KMALLOC_MINALIGN works just
> > >    fine here.
> > > 
> > > 2. As a static alignment for DMA requirements. That's where the newly
> > >    exposed ARCH_DMA_MINALIGN should be used.
> > > 
> > > Note that this series doesn't make the situation any worse than before
> > > since ARCH_DMA_MINALIGN stays at 128 bytes for arm64. Current users can
> > > evolve to use a dynamic alignment in future patches. My main aim with
> > > this series is to be able to create kmalloc-64 caches on arm64.
> > 
> > AFAIK there are bunch of drivers that directly calls kmalloc().
> 
> Well, lots of drivers call kmalloc() ;).
> 
> > It becomes tricky when e.g.) a driver allocates just 32 bytes,
> > but architecture requires it to be 128-byte aligned.
> 
> That's the current behaviour, a 32 byte allocation would return an
> object from kmalloc-128. I want to reduce this to at least kmalloc-64
> (or smaller) if the CPU/SoC allows it.

Yeah I agree the change is worth :) Thanks for the work.

> > That's why everything allocated from kmalloc() need to be aligned in
> > ARCH_DMA_MINALIGN.
> 
> I don't get your conclusion here. Would you mind explaining?

What I wanted to say was that, why ARCH_DMA_MINALIGN should be
different from ARCH_KMALLOC_MINALIGN.

I thought the two were basically same thing. Instead of
decoupling them, I thought just decreasing them in runtime makes more sense.

> > So I'm yet skeptical on decoupling ARCH_DMA/KMALLOC_MINALIGN. Instead
> > of decoupling it, I'm more into dynamically decreasing it.
> 
> The reason for decoupling is mostly that there are some static uses of
> ARCH_KMALLOC_MINALIGN as per point 1 above. The other is the
> __assume_kmalloc_alignment attribute. We shouldn't have such assumed
> alignment larger than what a dynamic kmalloc() would return. To me it
> makes a lot more sense for ARCH_KMALLOC_MINALIGN to be the minimum
> guaranteed in a kernel build but kmalloc() returning a larger alignment
> at run-time than the other way around.

But yeah, considering the problems you mentioned, it seems unavoidable
to decouple them.

Thank you for explanation and I will review slab part soon.

> Thanks.
> 
> -- 
> Catalin

-- 
Thanks,
Hyeonggon

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 00/10] mm, arm64: Reduce ARCH_KMALLOC_MINALIGN below the cache line size

[Vlastimil Babka]

On 4/5/22 15:57, Catalin Marinas wrote:
> Hi,
> 
> On arm64 ARCH_DMA_MINALIGN (and therefore ARCH_KMALLOC_MINALIGN) is 128.
> While the majority of arm64 SoCs have a 64-byte cache line size (or
> rather CWG - cache writeback granule), we chose a less than optimal
> value in order to support all SoCs in a single kernel image.
> 
> The aim of this series is to allow smaller default ARCH_KMALLOC_MINALIGN
> with kmalloc() caches configured at boot time to be safe when an SoC has
> a larger DMA alignment requirement.
> 
> The first patch decouples ARCH_KMALLOC_MINALIGN from ARCH_DMA_MINALIGN
> with the aim to only use the latter in DMA-specific compile-time
> annotations. ARCH_KMALLOC_MINALIGN becomes the minimum (static)
> guaranteed kmalloc() alignment but not necessarily safe for non-coherent
> DMA. Patches 2-7 change some drivers/ code to use ARCH_DMA_MINALIGN
> instead of ARCH_KMALLOC_MINALIGN.
> 
> Patch 8 introduces the dynamic arch_kmalloc_minalign() and the slab code
> changes to set the corresponding minimum alignment on the newly created
> kmalloc() caches. Patch 10 defines arch_kmalloc_minalign() for arm64
> returning cache_line_size() together with reducing ARCH_KMALLOC_MINALIGN
> to 64. ARCH_DMA_MINALIGN remains 128 on arm64.
> 
> I don't have access to it but there's the Fujitsu A64FX with a CWG of
> 256 (the arm64 cache_line_size() returns 256). This series will bump the
> smallest kmalloc cache to kmalloc-256. The platform is known to be fully
> cache coherent (or so I think) and we decided long ago not to bump
> ARCH_DMA_MINALIGN to 256. If problematic, we could make the dynamic
> kmalloc() alignment on arm64 min(ARCH_DMA_MINALIGN, cache_line_size()).
> 
> This series is beneficial to arm64 even if it's only reducing the
> kmalloc() minimum alignment to 64. While it would be nice to reduce this
> further to 8 (or 16) on SoCs known to be fully DMA coherent, detecting
> this is via arch_setup_dma_ops() is problematic, especially with late
> probed devices. I'd leave it for an additional RFC series on top of
> this (there are ideas like bounce buffering for non-coherent devices if
> the SoC was deemed coherent).

Oh that sounds great, and perhaps should help with our SLOB problem as
detailed in this subthread [1]. To recap:

- we would like kfree() to work on allocations done by kmem_cache_alloc(),
in addition to kmalloc()
- for SLOB this would mean that kmem_cache_alloc() objects have to store
their alloc size (prepended to the allocated object) which is now done for
kmalloc() objects only - we don't have to store the size if
kmem_cache_free() gives us the kmem_cache pointer which contains the
per-cache size.
- due to ARCH_KMALLOC_MINALIGN and dma guarantees we should return
allocations aligned to ARCH_KMALLOC_MINALIGN and the prepended size header
should also not share their ARCH_KMALLOC_MINALIGN block with another
(shorter) allocation that has a different lifetime, for the dma coherency
reasons
- this is very wasteful especially with the 128 bytes alignment, and seems
we already violate it in some scenarios anyway [2]. Extending this to all
objects would be even more wasteful.

So this series would help here, especially if we can get to the 8/16 size.
But now I also wonder if keeping the name and meaning of "MINALIGN" is in
fact misleading and unnecessarily constraining us? What this is really about
is "granularity of exclusive access", no? Let's say the dma granularity is
64bytes, and there's a kmalloc(56). If SLOB find a 64-bytes aligned block,
uses the first 8 bytes for the size header and returns the remaining 56
bytes, then the returned pointer is not *aligned* to 64 bytes, but it's
still aligned enough for cpu accesses (which need only e.g. 8), and
non-coherent dma should be also safe because nobody will be accessing the 8
bytes header, until the user of the object calls kfree() which should happen
only when it's done with any dma operations. Is my reasoning correct and
would this be safe?

[1] https://lore.kernel.org/all/20211122013026.909933-1-rkovhaev@gmail.com/
[2] https://lore.kernel.org/all/d0927ca6-1710-5b2b-3682-6a80eb4e48d1@suse.cz/

Re: [PATCH 00/10] mm, arm64: Reduce ARCH_KMALLOC_MINALIGN below the cache line size

[Vlastimil Babka]

On 4/5/22 15:57, Catalin Marinas wrote:
> Hi,
> 
> On arm64 ARCH_DMA_MINALIGN (and therefore ARCH_KMALLOC_MINALIGN) is 128.
> While the majority of arm64 SoCs have a 64-byte cache line size (or
> rather CWG - cache writeback granule), we chose a less than optimal
> value in order to support all SoCs in a single kernel image.
> 
> The aim of this series is to allow smaller default ARCH_KMALLOC_MINALIGN
> with kmalloc() caches configured at boot time to be safe when an SoC has
> a larger DMA alignment requirement.
> 
> The first patch decouples ARCH_KMALLOC_MINALIGN from ARCH_DMA_MINALIGN
> with the aim to only use the latter in DMA-specific compile-time
> annotations. ARCH_KMALLOC_MINALIGN becomes the minimum (static)
> guaranteed kmalloc() alignment but not necessarily safe for non-coherent
> DMA. Patches 2-7 change some drivers/ code to use ARCH_DMA_MINALIGN
> instead of ARCH_KMALLOC_MINALIGN.
> 
> Patch 8 introduces the dynamic arch_kmalloc_minalign() and the slab code
> changes to set the corresponding minimum alignment on the newly created
> kmalloc() caches. Patch 10 defines arch_kmalloc_minalign() for arm64
> returning cache_line_size() together with reducing ARCH_KMALLOC_MINALIGN
> to 64. ARCH_DMA_MINALIGN remains 128 on arm64.
> 
> I don't have access to it but there's the Fujitsu A64FX with a CWG of
> 256 (the arm64 cache_line_size() returns 256). This series will bump the
> smallest kmalloc cache to kmalloc-256. The platform is known to be fully
> cache coherent (or so I think) and we decided long ago not to bump
> ARCH_DMA_MINALIGN to 256. If problematic, we could make the dynamic
> kmalloc() alignment on arm64 min(ARCH_DMA_MINALIGN, cache_line_size()).
> 
> This series is beneficial to arm64 even if it's only reducing the
> kmalloc() minimum alignment to 64. While it would be nice to reduce this
> further to 8 (or 16) on SoCs known to be fully DMA coherent, detecting
> this is via arch_setup_dma_ops() is problematic, especially with late
> probed devices. I'd leave it for an additional RFC series on top of
> this (there are ideas like bounce buffering for non-coherent devices if
> the SoC was deemed coherent).

Oh that sounds great, and perhaps should help with our SLOB problem as
detailed in this subthread [1]. To recap:

- we would like kfree() to work on allocations done by kmem_cache_alloc(),
in addition to kmalloc()
- for SLOB this would mean that kmem_cache_alloc() objects have to store
their alloc size (prepended to the allocated object) which is now done for
kmalloc() objects only - we don't have to store the size if
kmem_cache_free() gives us the kmem_cache pointer which contains the
per-cache size.
- due to ARCH_KMALLOC_MINALIGN and dma guarantees we should return
allocations aligned to ARCH_KMALLOC_MINALIGN and the prepended size header
should also not share their ARCH_KMALLOC_MINALIGN block with another
(shorter) allocation that has a different lifetime, for the dma coherency
reasons
- this is very wasteful especially with the 128 bytes alignment, and seems
we already violate it in some scenarios anyway [2]. Extending this to all
objects would be even more wasteful.

So this series would help here, especially if we can get to the 8/16 size.
But now I also wonder if keeping the name and meaning of "MINALIGN" is in
fact misleading and unnecessarily constraining us? What this is really about
is "granularity of exclusive access", no? Let's say the dma granularity is
64bytes, and there's a kmalloc(56). If SLOB find a 64-bytes aligned block,
uses the first 8 bytes for the size header and returns the remaining 56
bytes, then the returned pointer is not *aligned* to 64 bytes, but it's
still aligned enough for cpu accesses (which need only e.g. 8), and
non-coherent dma should be also safe because nobody will be accessing the 8
bytes header, until the user of the object calls kfree() which should happen
only when it's done with any dma operations. Is my reasoning correct and
would this be safe?

[1] https://lore.kernel.org/all/20211122013026.909933-1-rkovhaev@gmail.com/
[2] https://lore.kernel.org/all/d0927ca6-1710-5b2b-3682-6a80eb4e48d1@suse.cz/

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Thu, Apr 07, 2022 at 07:40:59PM +0800, Herbert Xu wrote:
> On Thu, Apr 07, 2022 at 12:01:02PM +0100, Catalin Marinas wrote:
> > The only issue is whether the compiler gets confused by a pointer to a
> > structure with a smaller alignment than alignof(struct ...). I don't see
> > a performance or correctness issue on arm64 here. It would be a problem
> > if instead of 16 we went down to 8 or 4 due to unaligned accesses but
> > from 128 to 64 (or even 16), I don't think it matters.
> 
> The issue is that there's code in the Crypto API which assumes
> that all pointers returned by kmalloc are aligned to CRYPTO_MINALIGN,
> if you break that then all that code would need to be modified.

I'm not familiar with the crypto API, trying to make sense of it now ;).

I can see in many cases that the kmalloc() caller aligns the requested
size to something like crypto_tfm_ctx_alignment(). So this would
guarantee a kmalloc() object aligned to CRYPTO_MINALIGN.

> However, I think it's better to change the code that assumes
> CRYPTO_MINALIGN guarantees DMA alignment.

I saw Ard already started to refactor some of these. But in the meantime
are there cases where the crypto code does a kmalloc() of less than
CRYPTO_MINALIGN and expects it to be CRYPTO_MINALIGN aligned?

-- 
Catalin

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Thu, Apr 07, 2022 at 07:40:59PM +0800, Herbert Xu wrote:
> On Thu, Apr 07, 2022 at 12:01:02PM +0100, Catalin Marinas wrote:
> > The only issue is whether the compiler gets confused by a pointer to a
> > structure with a smaller alignment than alignof(struct ...). I don't see
> > a performance or correctness issue on arm64 here. It would be a problem
> > if instead of 16 we went down to 8 or 4 due to unaligned accesses but
> > from 128 to 64 (or even 16), I don't think it matters.
> 
> The issue is that there's code in the Crypto API which assumes
> that all pointers returned by kmalloc are aligned to CRYPTO_MINALIGN,
> if you break that then all that code would need to be modified.

I'm not familiar with the crypto API, trying to make sense of it now ;).

I can see in many cases that the kmalloc() caller aligns the requested
size to something like crypto_tfm_ctx_alignment(). So this would
guarantee a kmalloc() object aligned to CRYPTO_MINALIGN.

> However, I think it's better to change the code that assumes
> CRYPTO_MINALIGN guarantees DMA alignment.

I saw Ard already started to refactor some of these. But in the meantime
are there cases where the crypto code does a kmalloc() of less than
CRYPTO_MINALIGN and expects it to be CRYPTO_MINALIGN aligned?

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 00/10] mm, arm64: Reduce ARCH_KMALLOC_MINALIGN below the cache line size

[Catalin Marinas]

On Thu, Apr 07, 2022 at 04:40:15PM +0200, Vlastimil Babka wrote:
> On 4/5/22 15:57, Catalin Marinas wrote:
> > This series is beneficial to arm64 even if it's only reducing the
> > kmalloc() minimum alignment to 64. While it would be nice to reduce this
> > further to 8 (or 16) on SoCs known to be fully DMA coherent, detecting
> > this is via arch_setup_dma_ops() is problematic, especially with late
> > probed devices. I'd leave it for an additional RFC series on top of
> > this (there are ideas like bounce buffering for non-coherent devices if
> > the SoC was deemed coherent).
[...]
> - due to ARCH_KMALLOC_MINALIGN and dma guarantees we should return
> allocations aligned to ARCH_KMALLOC_MINALIGN and the prepended size header
> should also not share their ARCH_KMALLOC_MINALIGN block with another
> (shorter) allocation that has a different lifetime, for the dma coherency
> reasons
> - this is very wasteful especially with the 128 bytes alignment, and seems
> we already violate it in some scenarios anyway [2]. Extending this to all
> objects would be even more wasteful.
> 
> So this series would help here, especially if we can get to the 8/16 size.

If we get to 8/16 size, it would only be for platforms that are fully
coherent. Otherwise, with non-coherent DMA, the minimum kmalloc()
alignment would still be the cache line size (typically 64) even if
ARCH_KMALLOC_MINALIGN is 8.

IIUC your point is that if ARCH_KMALLOC_MINALIGN is 8, kmalloc() could
return pointers 8-byte aligned only as long as DMA safety is preserved
(like not sharing the rest of the cache line with anything other
writers).

> But now I also wonder if keeping the name and meaning of "MINALIGN" is in
> fact misleading and unnecessarily constraining us? What this is really about
> is "granularity of exclusive access", no?

Not necessarily. Yes, in lots of cases it is about granularity of access
but there are others where the code does need the pointer returned
aligned to ARCH_DMA_MINALIGN (currently via ARCH_KMALLOC_MINALIGN).
Crypto seems to have such requirement (see the sub-thread with Herbert).
Some (all?) callers ask kmalloc() for the aligned size and there's an
expectation that if the size is a multiple of a power of two, kmalloc()
will return a pointer aligned to that power of two. I think we need to
preserve these semantics which may lead to some more wastage if you add
the header (e.g. a size of 3*64 returns a pointer either aligned to 192
or 256).

> Let's say the dma granularity is 64bytes, and there's a kmalloc(56).

In your example, the size is not a power of two (or multiple of), so I
guess there's no expectation for a 64-byte alignment (it can be 8)
unless DMA is involved. See below.

> If SLOB find a 64-bytes aligned block, uses the first 8 bytes for the
> size header and returns the remaining 56 bytes, then the returned
> pointer is not *aligned* to 64 bytes, but it's still aligned enough
> for cpu accesses (which need only e.g. 8), and non-coherent dma should
> be also safe because nobody will be accessing the 8 bytes header,
> until the user of the object calls kfree() which should happen only
> when it's done with any dma operations. Is my reasoning correct and
> would this be safe?

From the DMA perspective, it's not safe currently. Let's say we have an
inbound DMA transfer, the DMA API will invalidate the cache line prior
to DMA. In arm64 terms, it means that the cache line is discarded, not
flushed to memory. If the first 8 bytes had not been written back to
RAM, they'd be lost. If we can guarantee that no CPU write happens to
the cache line during the DMA transfer, we can change the DMA mapping
operation to do a clean+invalidate (flush the cacheline to RAM) first. I
guess this could be done with an IS_ENABLED(CONFIG_SLOB) check.

-- 
Catalin

Re: [PATCH 00/10] mm, arm64: Reduce ARCH_KMALLOC_MINALIGN below the cache line size

[Catalin Marinas]

On Thu, Apr 07, 2022 at 04:40:15PM +0200, Vlastimil Babka wrote:
> On 4/5/22 15:57, Catalin Marinas wrote:
> > This series is beneficial to arm64 even if it's only reducing the
> > kmalloc() minimum alignment to 64. While it would be nice to reduce this
> > further to 8 (or 16) on SoCs known to be fully DMA coherent, detecting
> > this is via arch_setup_dma_ops() is problematic, especially with late
> > probed devices. I'd leave it for an additional RFC series on top of
> > this (there are ideas like bounce buffering for non-coherent devices if
> > the SoC was deemed coherent).
[...]
> - due to ARCH_KMALLOC_MINALIGN and dma guarantees we should return
> allocations aligned to ARCH_KMALLOC_MINALIGN and the prepended size header
> should also not share their ARCH_KMALLOC_MINALIGN block with another
> (shorter) allocation that has a different lifetime, for the dma coherency
> reasons
> - this is very wasteful especially with the 128 bytes alignment, and seems
> we already violate it in some scenarios anyway [2]. Extending this to all
> objects would be even more wasteful.
> 
> So this series would help here, especially if we can get to the 8/16 size.

If we get to 8/16 size, it would only be for platforms that are fully
coherent. Otherwise, with non-coherent DMA, the minimum kmalloc()
alignment would still be the cache line size (typically 64) even if
ARCH_KMALLOC_MINALIGN is 8.

IIUC your point is that if ARCH_KMALLOC_MINALIGN is 8, kmalloc() could
return pointers 8-byte aligned only as long as DMA safety is preserved
(like not sharing the rest of the cache line with anything other
writers).

> But now I also wonder if keeping the name and meaning of "MINALIGN" is in
> fact misleading and unnecessarily constraining us? What this is really about
> is "granularity of exclusive access", no?

Not necessarily. Yes, in lots of cases it is about granularity of access
but there are others where the code does need the pointer returned
aligned to ARCH_DMA_MINALIGN (currently via ARCH_KMALLOC_MINALIGN).
Crypto seems to have such requirement (see the sub-thread with Herbert).
Some (all?) callers ask kmalloc() for the aligned size and there's an
expectation that if the size is a multiple of a power of two, kmalloc()
will return a pointer aligned to that power of two. I think we need to
preserve these semantics which may lead to some more wastage if you add
the header (e.g. a size of 3*64 returns a pointer either aligned to 192
or 256).

> Let's say the dma granularity is 64bytes, and there's a kmalloc(56).

In your example, the size is not a power of two (or multiple of), so I
guess there's no expectation for a 64-byte alignment (it can be 8)
unless DMA is involved. See below.

> If SLOB find a 64-bytes aligned block, uses the first 8 bytes for the
> size header and returns the remaining 56 bytes, then the returned
> pointer is not *aligned* to 64 bytes, but it's still aligned enough
> for cpu accesses (which need only e.g. 8), and non-coherent dma should
> be also safe because nobody will be accessing the 8 bytes header,
> until the user of the object calls kfree() which should happen only
> when it's done with any dma operations. Is my reasoning correct and
> would this be safe?

From the DMA perspective, it's not safe currently. Let's say we have an
inbound DMA transfer, the DMA API will invalidate the cache line prior
to DMA. In arm64 terms, it means that the cache line is discarded, not
flushed to memory. If the first 8 bytes had not been written back to
RAM, they'd be lost. If we can guarantee that no CPU write happens to
the cache line during the DMA transfer, we can change the DMA mapping
operation to do a clean+invalidate (flush the cacheline to RAM) first. I
guess this could be done with an IS_ENABLED(CONFIG_SLOB) check.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Herbert Xu]

On Thu, Apr 07, 2022 at 05:28:10PM +0100, Catalin Marinas wrote:
>
> I can see in many cases that the kmalloc() caller aligns the requested
> size to something like crypto_tfm_ctx_alignment(). So this would
> guarantee a kmalloc() object aligned to CRYPTO_MINALIGN.

crypto_tfm_ctx_alignment is basically the same as CRYPTO_MINALIGN.
We assume any kmalloced pointers to be aligned to that.

Specific algorithms may ask for an alignment greater than that
and we will use the knowledge that kmalloc is aligned to
CRYPTO_MINALIGN to derive the extra memory we need to get.

So if kmalloc no longer returns memory aligned to MINALIGN then
we'll get memory overruns.

> I saw Ard already started to refactor some of these. But in the meantime
> are there cases where the crypto code does a kmalloc() of less than
> CRYPTO_MINALIGN and expects it to be CRYPTO_MINALIGN aligned?

It's a fundamental assumption of the API.

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Herbert Xu]

On Thu, Apr 07, 2022 at 05:28:10PM +0100, Catalin Marinas wrote:
>
> I can see in many cases that the kmalloc() caller aligns the requested
> size to something like crypto_tfm_ctx_alignment(). So this would
> guarantee a kmalloc() object aligned to CRYPTO_MINALIGN.

crypto_tfm_ctx_alignment is basically the same as CRYPTO_MINALIGN.
We assume any kmalloced pointers to be aligned to that.

Specific algorithms may ask for an alignment greater than that
and we will use the knowledge that kmalloc is aligned to
CRYPTO_MINALIGN to derive the extra memory we need to get.

So if kmalloc no longer returns memory aligned to MINALIGN then
we'll get memory overruns.

> I saw Ard already started to refactor some of these. But in the meantime
> are there cases where the crypto code does a kmalloc() of less than
> CRYPTO_MINALIGN and expects it to be CRYPTO_MINALIGN aligned?

It's a fundamental assumption of the API.

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 01/10] mm/slab: Decouple ARCH_KMALLOC_MINALIGN from ARCH_DMA_MINALIGN

[Hyeonggon Yoo]

On Tue, Apr 05, 2022 at 02:57:49PM +0100, Catalin Marinas wrote:
> In preparation for supporting a dynamic kmalloc() minimum alignment,
> allow architectures to define ARCH_KMALLOC_MINALIGN independently of
> ARCH_DMA_MINALIGN. In addition, always define ARCH_DMA_MINALIGN even if
> an architecture does not override it.
> 
> After this patch, ARCH_DMA_MINALIGN is expected to be used in static
> alignment annotations and defined by an architecture to be the maximum
> alignment for all supported configurations/SoCs in a single Image.
> ARCH_KMALLOC_MINALIGN, if different, is the minimum alignment guaranteed
> by kmalloc().
> 
> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
> Cc: Andrew Morton <akpm@linux-foundation.org>
> ---
>  include/linux/slab.h | 23 ++++++++++++++++++-----
>  1 file changed, 18 insertions(+), 5 deletions(-)
> 
> diff --git a/include/linux/slab.h b/include/linux/slab.h
> index 373b3ef99f4e..d58211bdeceb 100644
> --- a/include/linux/slab.h
> +++ b/include/linux/slab.h
> @@ -187,17 +187,30 @@ bool kmem_valid_obj(void *object);
>  void kmem_dump_obj(void *object);
>  #endif
>  
> +/*
> + * slob does not support independent control of ARCH_KMALLOC_MINALIGN and
> + * ARCH_DMA_MINALIGN.
> + */
> +#ifdef CONFIG_SLOB
> +#undef ARCH_KMALLOC_MINALIGN
> +#endif

I think you should replace ARCH_KMALLOC_MINALIGN with ARCH_DMA_MINALIGN
in mm/slob.c too? Or detect minimum kmalloc alignment in runtime like SLAB/SLUB?

current code seem to break with SLOB on machines that has 128 byte cache lines
because ARCH_KMALLOC_MINALIGN is 64?

> +
>  /*
>   * Some archs want to perform DMA into kmalloc caches and need a guaranteed
>   * alignment larger than the alignment of a 64-bit integer.
> - * Setting ARCH_KMALLOC_MINALIGN in arch headers allows that.
> + * Setting ARCH_DMA_MINALIGN in arch headers allows that.
>   */
> -#if defined(ARCH_DMA_MINALIGN) && ARCH_DMA_MINALIGN > 8
> +#ifndef ARCH_DMA_MINALIGN
> +#define ARCH_DMA_MINALIGN __alignof__(unsigned long long)
> +#elif ARCH_DMA_MINALIGN > 8 && !defined(ARCH_KMALLOC_MINALIGN)
>  #define ARCH_KMALLOC_MINALIGN ARCH_DMA_MINALIGN
> -#define KMALLOC_MIN_SIZE ARCH_DMA_MINALIGN
> -#define KMALLOC_SHIFT_LOW ilog2(ARCH_DMA_MINALIGN)
> -#else
> +#endif
> +
> +#ifndef ARCH_KMALLOC_MINALIGN
>  #define ARCH_KMALLOC_MINALIGN __alignof__(unsigned long long)
> +#else
> +#define KMALLOC_MIN_SIZE ARCH_KMALLOC_MINALIGN
> +#define KMALLOC_SHIFT_LOW ilog2(KMALLOC_MIN_SIZE)
>  #endif
>  
>  /*
> 

-- 
Thanks,
Hyeonggon

Re: [PATCH 01/10] mm/slab: Decouple ARCH_KMALLOC_MINALIGN from ARCH_DMA_MINALIGN

[Hyeonggon Yoo]

On Tue, Apr 05, 2022 at 02:57:49PM +0100, Catalin Marinas wrote:
> In preparation for supporting a dynamic kmalloc() minimum alignment,
> allow architectures to define ARCH_KMALLOC_MINALIGN independently of
> ARCH_DMA_MINALIGN. In addition, always define ARCH_DMA_MINALIGN even if
> an architecture does not override it.
> 
> After this patch, ARCH_DMA_MINALIGN is expected to be used in static
> alignment annotations and defined by an architecture to be the maximum
> alignment for all supported configurations/SoCs in a single Image.
> ARCH_KMALLOC_MINALIGN, if different, is the minimum alignment guaranteed
> by kmalloc().
> 
> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
> Cc: Andrew Morton <akpm@linux-foundation.org>
> ---
>  include/linux/slab.h | 23 ++++++++++++++++++-----
>  1 file changed, 18 insertions(+), 5 deletions(-)
> 
> diff --git a/include/linux/slab.h b/include/linux/slab.h
> index 373b3ef99f4e..d58211bdeceb 100644
> --- a/include/linux/slab.h
> +++ b/include/linux/slab.h
> @@ -187,17 +187,30 @@ bool kmem_valid_obj(void *object);
>  void kmem_dump_obj(void *object);
>  #endif
>  
> +/*
> + * slob does not support independent control of ARCH_KMALLOC_MINALIGN and
> + * ARCH_DMA_MINALIGN.
> + */
> +#ifdef CONFIG_SLOB
> +#undef ARCH_KMALLOC_MINALIGN
> +#endif

I think you should replace ARCH_KMALLOC_MINALIGN with ARCH_DMA_MINALIGN
in mm/slob.c too? Or detect minimum kmalloc alignment in runtime like SLAB/SLUB?

current code seem to break with SLOB on machines that has 128 byte cache lines
because ARCH_KMALLOC_MINALIGN is 64?

> +
>  /*
>   * Some archs want to perform DMA into kmalloc caches and need a guaranteed
>   * alignment larger than the alignment of a 64-bit integer.
> - * Setting ARCH_KMALLOC_MINALIGN in arch headers allows that.
> + * Setting ARCH_DMA_MINALIGN in arch headers allows that.
>   */
> -#if defined(ARCH_DMA_MINALIGN) && ARCH_DMA_MINALIGN > 8
> +#ifndef ARCH_DMA_MINALIGN
> +#define ARCH_DMA_MINALIGN __alignof__(unsigned long long)
> +#elif ARCH_DMA_MINALIGN > 8 && !defined(ARCH_KMALLOC_MINALIGN)
>  #define ARCH_KMALLOC_MINALIGN ARCH_DMA_MINALIGN
> -#define KMALLOC_MIN_SIZE ARCH_DMA_MINALIGN
> -#define KMALLOC_SHIFT_LOW ilog2(ARCH_DMA_MINALIGN)
> -#else
> +#endif
> +
> +#ifndef ARCH_KMALLOC_MINALIGN
>  #define ARCH_KMALLOC_MINALIGN __alignof__(unsigned long long)
> +#else
> +#define KMALLOC_MIN_SIZE ARCH_KMALLOC_MINALIGN
> +#define KMALLOC_SHIFT_LOW ilog2(KMALLOC_MIN_SIZE)
>  #endif
>  
>  /*
> 

-- 
Thanks,
Hyeonggon

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Fri, Apr 08, 2022 at 11:25:29AM +0800, Herbert Xu wrote:
> On Thu, Apr 07, 2022 at 05:28:10PM +0100, Catalin Marinas wrote:
> > I can see in many cases that the kmalloc() caller aligns the requested
> > size to something like crypto_tfm_ctx_alignment(). So this would
> > guarantee a kmalloc() object aligned to CRYPTO_MINALIGN.
> 
> crypto_tfm_ctx_alignment is basically the same as CRYPTO_MINALIGN.
> We assume any kmalloced pointers to be aligned to that.
> 
> Specific algorithms may ask for an alignment greater than that
> and we will use the knowledge that kmalloc is aligned to
> CRYPTO_MINALIGN to derive the extra memory we need to get.
> 
> So if kmalloc no longer returns memory aligned to MINALIGN then
> we'll get memory overruns.

My point is that if the crypto code kmallocs a size aligned to
crypto_tfm_ctx_alignment() (and CRYPTO_MINALIGN), the slab allocator
will return memory aligned to CRYPTO_MINALIGN even if
ARCH_KMALLOC_MINALIGN is smaller.

Would the crypto code, say, do a kmalloc(64) and expect a 128 byte
alignment (when CRYPTO_MINALIGN == 128)? Or does it align the size to
CRYPTO_MINALIGN and do a kmalloc(128) directly? If it's the latter, I
don't think there's a problem.

-- 
Catalin

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Fri, Apr 08, 2022 at 11:25:29AM +0800, Herbert Xu wrote:
> On Thu, Apr 07, 2022 at 05:28:10PM +0100, Catalin Marinas wrote:
> > I can see in many cases that the kmalloc() caller aligns the requested
> > size to something like crypto_tfm_ctx_alignment(). So this would
> > guarantee a kmalloc() object aligned to CRYPTO_MINALIGN.
> 
> crypto_tfm_ctx_alignment is basically the same as CRYPTO_MINALIGN.
> We assume any kmalloced pointers to be aligned to that.
> 
> Specific algorithms may ask for an alignment greater than that
> and we will use the knowledge that kmalloc is aligned to
> CRYPTO_MINALIGN to derive the extra memory we need to get.
> 
> So if kmalloc no longer returns memory aligned to MINALIGN then
> we'll get memory overruns.

My point is that if the crypto code kmallocs a size aligned to
crypto_tfm_ctx_alignment() (and CRYPTO_MINALIGN), the slab allocator
will return memory aligned to CRYPTO_MINALIGN even if
ARCH_KMALLOC_MINALIGN is smaller.

Would the crypto code, say, do a kmalloc(64) and expect a 128 byte
alignment (when CRYPTO_MINALIGN == 128)? Or does it align the size to
CRYPTO_MINALIGN and do a kmalloc(128) directly? If it's the latter, I
don't think there's a problem.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 01/10] mm/slab: Decouple ARCH_KMALLOC_MINALIGN from ARCH_DMA_MINALIGN

[Hyeonggon Yoo]

On Fri, Apr 08, 2022 at 03:42:13PM +0900, Hyeonggon Yoo wrote:
> On Tue, Apr 05, 2022 at 02:57:49PM +0100, Catalin Marinas wrote:
> > In preparation for supporting a dynamic kmalloc() minimum alignment,
> > allow architectures to define ARCH_KMALLOC_MINALIGN independently of
> > ARCH_DMA_MINALIGN. In addition, always define ARCH_DMA_MINALIGN even if
> > an architecture does not override it.
> > 
> > After this patch, ARCH_DMA_MINALIGN is expected to be used in static
> > alignment annotations and defined by an architecture to be the maximum
> > alignment for all supported configurations/SoCs in a single Image.
> > ARCH_KMALLOC_MINALIGN, if different, is the minimum alignment guaranteed
> > by kmalloc().
> > 
> > Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
> > Cc: Andrew Morton <akpm@linux-foundation.org>
> > ---
> >  include/linux/slab.h | 23 ++++++++++++++++++-----
> >  1 file changed, 18 insertions(+), 5 deletions(-)
> > 
> > diff --git a/include/linux/slab.h b/include/linux/slab.h
> > index 373b3ef99f4e..d58211bdeceb 100644
> > --- a/include/linux/slab.h
> > +++ b/include/linux/slab.h
> > @@ -187,17 +187,30 @@ bool kmem_valid_obj(void *object);
> >  void kmem_dump_obj(void *object);
> >  #endif
> >  
> > +/*
> > + * slob does not support independent control of ARCH_KMALLOC_MINALIGN and
> > + * ARCH_DMA_MINALIGN.
> > + */
> > +#ifdef CONFIG_SLOB
> > +#undef ARCH_KMALLOC_MINALIGN
> > +#endif
> 

Sorry for the noise. Yeah, the code above is making
ARCH_KMALLOC_MINALIGN = ARCH_DMA_MINALIGN.

I was confused :(

> I think you should replace ARCH_KMALLOC_MINALIGN with ARCH_DMA_MINALIGN
> in mm/slob.c too? Or detect minimum kmalloc alignment in runtime like SLAB/SLUB?
> 
> current code seem to break with SLOB on machines that has 128 byte cache lines
> because ARCH_KMALLOC_MINALIGN is 64?
> 

> > +
> >  /*
> >   * Some archs want to perform DMA into kmalloc caches and need a guaranteed
> >   * alignment larger than the alignment of a 64-bit integer.
> > - * Setting ARCH_KMALLOC_MINALIGN in arch headers allows that.
> > + * Setting ARCH_DMA_MINALIGN in arch headers allows that.
> >   */
> > -#if defined(ARCH_DMA_MINALIGN) && ARCH_DMA_MINALIGN > 8
> > +#ifndef ARCH_DMA_MINALIGN
> > +#define ARCH_DMA_MINALIGN __alignof__(unsigned long long)
> > +#elif ARCH_DMA_MINALIGN > 8 && !defined(ARCH_KMALLOC_MINALIGN)
> >  #define ARCH_KMALLOC_MINALIGN ARCH_DMA_MINALIGN
> > -#define KMALLOC_MIN_SIZE ARCH_DMA_MINALIGN
> > -#define KMALLOC_SHIFT_LOW ilog2(ARCH_DMA_MINALIGN)
> > -#else
> > +#endif
> > +
> > +#ifndef ARCH_KMALLOC_MINALIGN
> >  #define ARCH_KMALLOC_MINALIGN __alignof__(unsigned long long)
> > +#else
> > +#define KMALLOC_MIN_SIZE ARCH_KMALLOC_MINALIGN
> > +#define KMALLOC_SHIFT_LOW ilog2(KMALLOC_MIN_SIZE)
> >  #endif
> >  
> >  /*
> > 
> 
> -- 
> Thanks,
> Hyeonggon

-- 
Thanks,
Hyeonggon

Re: [PATCH 01/10] mm/slab: Decouple ARCH_KMALLOC_MINALIGN from ARCH_DMA_MINALIGN

[Hyeonggon Yoo]

On Fri, Apr 08, 2022 at 03:42:13PM +0900, Hyeonggon Yoo wrote:
> On Tue, Apr 05, 2022 at 02:57:49PM +0100, Catalin Marinas wrote:
> > In preparation for supporting a dynamic kmalloc() minimum alignment,
> > allow architectures to define ARCH_KMALLOC_MINALIGN independently of
> > ARCH_DMA_MINALIGN. In addition, always define ARCH_DMA_MINALIGN even if
> > an architecture does not override it.
> > 
> > After this patch, ARCH_DMA_MINALIGN is expected to be used in static
> > alignment annotations and defined by an architecture to be the maximum
> > alignment for all supported configurations/SoCs in a single Image.
> > ARCH_KMALLOC_MINALIGN, if different, is the minimum alignment guaranteed
> > by kmalloc().
> > 
> > Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
> > Cc: Andrew Morton <akpm@linux-foundation.org>
> > ---
> >  include/linux/slab.h | 23 ++++++++++++++++++-----
> >  1 file changed, 18 insertions(+), 5 deletions(-)
> > 
> > diff --git a/include/linux/slab.h b/include/linux/slab.h
> > index 373b3ef99f4e..d58211bdeceb 100644
> > --- a/include/linux/slab.h
> > +++ b/include/linux/slab.h
> > @@ -187,17 +187,30 @@ bool kmem_valid_obj(void *object);
> >  void kmem_dump_obj(void *object);
> >  #endif
> >  
> > +/*
> > + * slob does not support independent control of ARCH_KMALLOC_MINALIGN and
> > + * ARCH_DMA_MINALIGN.
> > + */
> > +#ifdef CONFIG_SLOB
> > +#undef ARCH_KMALLOC_MINALIGN
> > +#endif
> 

Sorry for the noise. Yeah, the code above is making
ARCH_KMALLOC_MINALIGN = ARCH_DMA_MINALIGN.

I was confused :(

> I think you should replace ARCH_KMALLOC_MINALIGN with ARCH_DMA_MINALIGN
> in mm/slob.c too? Or detect minimum kmalloc alignment in runtime like SLAB/SLUB?
> 
> current code seem to break with SLOB on machines that has 128 byte cache lines
> because ARCH_KMALLOC_MINALIGN is 64?
> 

> > +
> >  /*
> >   * Some archs want to perform DMA into kmalloc caches and need a guaranteed
> >   * alignment larger than the alignment of a 64-bit integer.
> > - * Setting ARCH_KMALLOC_MINALIGN in arch headers allows that.
> > + * Setting ARCH_DMA_MINALIGN in arch headers allows that.
> >   */
> > -#if defined(ARCH_DMA_MINALIGN) && ARCH_DMA_MINALIGN > 8
> > +#ifndef ARCH_DMA_MINALIGN
> > +#define ARCH_DMA_MINALIGN __alignof__(unsigned long long)
> > +#elif ARCH_DMA_MINALIGN > 8 && !defined(ARCH_KMALLOC_MINALIGN)
> >  #define ARCH_KMALLOC_MINALIGN ARCH_DMA_MINALIGN
> > -#define KMALLOC_MIN_SIZE ARCH_DMA_MINALIGN
> > -#define KMALLOC_SHIFT_LOW ilog2(ARCH_DMA_MINALIGN)
> > -#else
> > +#endif
> > +
> > +#ifndef ARCH_KMALLOC_MINALIGN
> >  #define ARCH_KMALLOC_MINALIGN __alignof__(unsigned long long)
> > +#else
> > +#define KMALLOC_MIN_SIZE ARCH_KMALLOC_MINALIGN
> > +#define KMALLOC_SHIFT_LOW ilog2(KMALLOC_MIN_SIZE)
> >  #endif
> >  
> >  /*
> > 
> 
> -- 
> Thanks,
> Hyeonggon

-- 
Thanks,
Hyeonggon

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Herbert Xu]

On Fri, Apr 08, 2022 at 10:04:54AM +0100, Catalin Marinas wrote:
>
> My point is that if the crypto code kmallocs a size aligned to
> crypto_tfm_ctx_alignment() (and CRYPTO_MINALIGN), the slab allocator
> will return memory aligned to CRYPTO_MINALIGN even if
> ARCH_KMALLOC_MINALIGN is smaller.

No we don't align the size to CRYPTO_MINALIGN at all.  We simply
assume that this is the alignment returned by kmalloc.

> Would the crypto code, say, do a kmalloc(64) and expect a 128 byte
> alignment (when CRYPTO_MINALIGN == 128)? Or does it align the size to
> CRYPTO_MINALIGN and do a kmalloc(128) directly? If it's the latter, I
> don't think there's a problem.

It's the former.

I think you can still make the change you want, but first you need
to modify the affected drivers to specify their actual alignment
requirement explicitly through cra_alignmask and then use the
correct methods to access the context pointer.

Basically these drivers have been broken from day one, but their
brokenness has been hidden by the extra-large KMALLOC_MINALIGN
value on arm.  So to reduce the KMALLOC_MINALIGN value, you have
to modify the drivers and set the cra_alignmask value.

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Herbert Xu]

On Fri, Apr 08, 2022 at 10:04:54AM +0100, Catalin Marinas wrote:
>
> My point is that if the crypto code kmallocs a size aligned to
> crypto_tfm_ctx_alignment() (and CRYPTO_MINALIGN), the slab allocator
> will return memory aligned to CRYPTO_MINALIGN even if
> ARCH_KMALLOC_MINALIGN is smaller.

No we don't align the size to CRYPTO_MINALIGN at all.  We simply
assume that this is the alignment returned by kmalloc.

> Would the crypto code, say, do a kmalloc(64) and expect a 128 byte
> alignment (when CRYPTO_MINALIGN == 128)? Or does it align the size to
> CRYPTO_MINALIGN and do a kmalloc(128) directly? If it's the latter, I
> don't think there's a problem.

It's the former.

I think you can still make the change you want, but first you need
to modify the affected drivers to specify their actual alignment
requirement explicitly through cra_alignmask and then use the
correct methods to access the context pointer.

Basically these drivers have been broken from day one, but their
brokenness has been hidden by the extra-large KMALLOC_MINALIGN
value on arm.  So to reduce the KMALLOC_MINALIGN value, you have
to modify the drivers and set the cra_alignmask value.

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 01/10] mm/slab: Decouple ARCH_KMALLOC_MINALIGN from ARCH_DMA_MINALIGN

[Catalin Marinas]

On Fri, Apr 08, 2022 at 03:42:13PM +0900, Hyeonggon Yoo wrote:
> On Tue, Apr 05, 2022 at 02:57:49PM +0100, Catalin Marinas wrote:
> > diff --git a/include/linux/slab.h b/include/linux/slab.h
> > index 373b3ef99f4e..d58211bdeceb 100644
> > --- a/include/linux/slab.h
> > +++ b/include/linux/slab.h
> > @@ -187,17 +187,30 @@ bool kmem_valid_obj(void *object);
> >  void kmem_dump_obj(void *object);
> >  #endif
> >  
> > +/*
> > + * slob does not support independent control of ARCH_KMALLOC_MINALIGN and
> > + * ARCH_DMA_MINALIGN.
> > + */
> > +#ifdef CONFIG_SLOB
> > +#undef ARCH_KMALLOC_MINALIGN
> > +#endif
> 
> I think you should replace ARCH_KMALLOC_MINALIGN with ARCH_DMA_MINALIGN
> in mm/slob.c too? Or detect minimum kmalloc alignment in runtime like SLAB/SLUB?

One step at a time. The slob approach is a bit different, doesn't
generate kmalloc-* caches, so I did not look at it yet. Also based on
Vlastimil's email, there is some reworking going on in there already.

> current code seem to break with SLOB on machines that has 128 byte cache lines
> because ARCH_KMALLOC_MINALIGN is 64?

Does it? The point of the #undef above was precisely to make sure
ARCH_KMALLOC_MINALIGN stays the same as ARCH_DMA_MINALIGN when
CONFIG_SLOB is enabled.

-- 
Catalin

Re: [PATCH 01/10] mm/slab: Decouple ARCH_KMALLOC_MINALIGN from ARCH_DMA_MINALIGN

[Catalin Marinas]

On Fri, Apr 08, 2022 at 03:42:13PM +0900, Hyeonggon Yoo wrote:
> On Tue, Apr 05, 2022 at 02:57:49PM +0100, Catalin Marinas wrote:
> > diff --git a/include/linux/slab.h b/include/linux/slab.h
> > index 373b3ef99f4e..d58211bdeceb 100644
> > --- a/include/linux/slab.h
> > +++ b/include/linux/slab.h
> > @@ -187,17 +187,30 @@ bool kmem_valid_obj(void *object);
> >  void kmem_dump_obj(void *object);
> >  #endif
> >  
> > +/*
> > + * slob does not support independent control of ARCH_KMALLOC_MINALIGN and
> > + * ARCH_DMA_MINALIGN.
> > + */
> > +#ifdef CONFIG_SLOB
> > +#undef ARCH_KMALLOC_MINALIGN
> > +#endif
> 
> I think you should replace ARCH_KMALLOC_MINALIGN with ARCH_DMA_MINALIGN
> in mm/slob.c too? Or detect minimum kmalloc alignment in runtime like SLAB/SLUB?

One step at a time. The slob approach is a bit different, doesn't
generate kmalloc-* caches, so I did not look at it yet. Also based on
Vlastimil's email, there is some reworking going on in there already.

> current code seem to break with SLOB on machines that has 128 byte cache lines
> because ARCH_KMALLOC_MINALIGN is 64?

Does it? The point of the #undef above was precisely to make sure
ARCH_KMALLOC_MINALIGN stays the same as ARCH_DMA_MINALIGN when
CONFIG_SLOB is enabled.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 00/10] mm, arm64: Reduce ARCH_KMALLOC_MINALIGN below the cache line size

[Vlastimil Babka]

On 4/7/22 19:48, Catalin Marinas wrote:
> On Thu, Apr 07, 2022 at 04:40:15PM +0200, Vlastimil Babka wrote:
>> On 4/5/22 15:57, Catalin Marinas wrote:
>> > This series is beneficial to arm64 even if it's only reducing the
>> > kmalloc() minimum alignment to 64. While it would be nice to reduce this
>> > further to 8 (or 16) on SoCs known to be fully DMA coherent, detecting
>> > this is via arch_setup_dma_ops() is problematic, especially with late
>> > probed devices. I'd leave it for an additional RFC series on top of
>> > this (there are ideas like bounce buffering for non-coherent devices if
>> > the SoC was deemed coherent).
> [...]
>> - due to ARCH_KMALLOC_MINALIGN and dma guarantees we should return
>> allocations aligned to ARCH_KMALLOC_MINALIGN and the prepended size header
>> should also not share their ARCH_KMALLOC_MINALIGN block with another
>> (shorter) allocation that has a different lifetime, for the dma coherency
>> reasons
>> - this is very wasteful especially with the 128 bytes alignment, and seems
>> we already violate it in some scenarios anyway [2]. Extending this to all
>> objects would be even more wasteful.
>> 
>> So this series would help here, especially if we can get to the 8/16 size.
> 
> If we get to 8/16 size, it would only be for platforms that are fully
> coherent. Otherwise, with non-coherent DMA, the minimum kmalloc()
> alignment would still be the cache line size (typically 64) even if
> ARCH_KMALLOC_MINALIGN is 8.

OK.

> IIUC your point is that if ARCH_KMALLOC_MINALIGN is 8, kmalloc() could
> return pointers 8-byte aligned only as long as DMA safety is preserved
> (like not sharing the rest of the cache line with anything other
> writers).

Yeah IIUC we now have no way to distinguish callers of kmalloc() that
actually rely on the allocations being DMA safe, so we can only assume it's
potentially all of them. Because __GFP_DMA is something different than coherency.

>> But now I also wonder if keeping the name and meaning of "MINALIGN" is in
>> fact misleading and unnecessarily constraining us? What this is really about
>> is "granularity of exclusive access", no?
> 
> Not necessarily. Yes, in lots of cases it is about granularity of access
> but there are others where the code does need the pointer returned
> aligned to ARCH_DMA_MINALIGN (currently via ARCH_KMALLOC_MINALIGN).
> Crypto seems to have such requirement (see the sub-thread with Herbert).
> Some (all?) callers ask kmalloc() for the aligned size and there's an
> expectation that if the size is a multiple of a power of two, kmalloc()
> will return a pointer aligned to that power of two. I think we need to
> preserve these semantics which may lead to some more wastage if you add
> the header (e.g. a size of 3*64 returns a pointer either aligned to 192
> or 256).

Agree, I wasn't suggesting we start ignoring the (explicit or implicit
power-of-2) alignment requests. In that case the size header would still be
in the preceding (relative to the alignment address, e.g. 64bytes) block.
But if dma safety didn't require 64byte granularity, the rest of that block
could be used by another allocation, so that would reduce the wastage.

>> Let's say the dma granularity is 64bytes, and there's a kmalloc(56).
> 
> In your example, the size is not a power of two (or multiple of), so I
> guess there's no expectation for a 64-byte alignment (it can be 8)
> unless DMA is involved. See below.

Yes, and AFAIU we cannot rule out the DMA involvement, unfortunately.

>> If SLOB find a 64-bytes aligned block, uses the first 8 bytes for the
>> size header and returns the remaining 56 bytes, then the returned
>> pointer is not *aligned* to 64 bytes, but it's still aligned enough
>> for cpu accesses (which need only e.g. 8), and non-coherent dma should
>> be also safe because nobody will be accessing the 8 bytes header,
>> until the user of the object calls kfree() which should happen only
>> when it's done with any dma operations. Is my reasoning correct and
>> would this be safe?
> 
> From the DMA perspective, it's not safe currently. Let's say we have an
> inbound DMA transfer, the DMA API will invalidate the cache line prior
> to DMA. In arm64 terms, it means that the cache line is discarded, not
> flushed to memory. If the first 8 bytes had not been written back to
> RAM, they'd be lost.

Thanks for the insight, that's what I was looking for.

> If we can guarantee that no CPU write happens to
> the cache line during the DMA transfer, we can change the DMA mapping
> operation to do a clean+invalidate (flush the cacheline to RAM) first. I
> guess this could be done with an IS_ENABLED(CONFIG_SLOB) check.
 
I see. Maybe we'll think of another solution for SLOB.

Re: [PATCH 00/10] mm, arm64: Reduce ARCH_KMALLOC_MINALIGN below the cache line size

[Vlastimil Babka]

On 4/7/22 19:48, Catalin Marinas wrote:
> On Thu, Apr 07, 2022 at 04:40:15PM +0200, Vlastimil Babka wrote:
>> On 4/5/22 15:57, Catalin Marinas wrote:
>> > This series is beneficial to arm64 even if it's only reducing the
>> > kmalloc() minimum alignment to 64. While it would be nice to reduce this
>> > further to 8 (or 16) on SoCs known to be fully DMA coherent, detecting
>> > this is via arch_setup_dma_ops() is problematic, especially with late
>> > probed devices. I'd leave it for an additional RFC series on top of
>> > this (there are ideas like bounce buffering for non-coherent devices if
>> > the SoC was deemed coherent).
> [...]
>> - due to ARCH_KMALLOC_MINALIGN and dma guarantees we should return
>> allocations aligned to ARCH_KMALLOC_MINALIGN and the prepended size header
>> should also not share their ARCH_KMALLOC_MINALIGN block with another
>> (shorter) allocation that has a different lifetime, for the dma coherency
>> reasons
>> - this is very wasteful especially with the 128 bytes alignment, and seems
>> we already violate it in some scenarios anyway [2]. Extending this to all
>> objects would be even more wasteful.
>> 
>> So this series would help here, especially if we can get to the 8/16 size.
> 
> If we get to 8/16 size, it would only be for platforms that are fully
> coherent. Otherwise, with non-coherent DMA, the minimum kmalloc()
> alignment would still be the cache line size (typically 64) even if
> ARCH_KMALLOC_MINALIGN is 8.

OK.

> IIUC your point is that if ARCH_KMALLOC_MINALIGN is 8, kmalloc() could
> return pointers 8-byte aligned only as long as DMA safety is preserved
> (like not sharing the rest of the cache line with anything other
> writers).

Yeah IIUC we now have no way to distinguish callers of kmalloc() that
actually rely on the allocations being DMA safe, so we can only assume it's
potentially all of them. Because __GFP_DMA is something different than coherency.

>> But now I also wonder if keeping the name and meaning of "MINALIGN" is in
>> fact misleading and unnecessarily constraining us? What this is really about
>> is "granularity of exclusive access", no?
> 
> Not necessarily. Yes, in lots of cases it is about granularity of access
> but there are others where the code does need the pointer returned
> aligned to ARCH_DMA_MINALIGN (currently via ARCH_KMALLOC_MINALIGN).
> Crypto seems to have such requirement (see the sub-thread with Herbert).
> Some (all?) callers ask kmalloc() for the aligned size and there's an
> expectation that if the size is a multiple of a power of two, kmalloc()
> will return a pointer aligned to that power of two. I think we need to
> preserve these semantics which may lead to some more wastage if you add
> the header (e.g. a size of 3*64 returns a pointer either aligned to 192
> or 256).

Agree, I wasn't suggesting we start ignoring the (explicit or implicit
power-of-2) alignment requests. In that case the size header would still be
in the preceding (relative to the alignment address, e.g. 64bytes) block.
But if dma safety didn't require 64byte granularity, the rest of that block
could be used by another allocation, so that would reduce the wastage.

>> Let's say the dma granularity is 64bytes, and there's a kmalloc(56).
> 
> In your example, the size is not a power of two (or multiple of), so I
> guess there's no expectation for a 64-byte alignment (it can be 8)
> unless DMA is involved. See below.

Yes, and AFAIU we cannot rule out the DMA involvement, unfortunately.

>> If SLOB find a 64-bytes aligned block, uses the first 8 bytes for the
>> size header and returns the remaining 56 bytes, then the returned
>> pointer is not *aligned* to 64 bytes, but it's still aligned enough
>> for cpu accesses (which need only e.g. 8), and non-coherent dma should
>> be also safe because nobody will be accessing the 8 bytes header,
>> until the user of the object calls kfree() which should happen only
>> when it's done with any dma operations. Is my reasoning correct and
>> would this be safe?
> 
> From the DMA perspective, it's not safe currently. Let's say we have an
> inbound DMA transfer, the DMA API will invalidate the cache line prior
> to DMA. In arm64 terms, it means that the cache line is discarded, not
> flushed to memory. If the first 8 bytes had not been written back to
> RAM, they'd be lost.

Thanks for the insight, that's what I was looking for.

> If we can guarantee that no CPU write happens to
> the cache line during the DMA transfer, we can change the DMA mapping
> operation to do a clean+invalidate (flush the cacheline to RAM) first. I
> guess this could be done with an IS_ENABLED(CONFIG_SLOB) check.
 
I see. Maybe we'll think of another solution for SLOB.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 01/10] mm/slab: Decouple ARCH_KMALLOC_MINALIGN from ARCH_DMA_MINALIGN

[Hyeonggon Yoo]

On Tue, Apr 05, 2022 at 02:57:49PM +0100, Catalin Marinas wrote:
> In preparation for supporting a dynamic kmalloc() minimum alignment,
> allow architectures to define ARCH_KMALLOC_MINALIGN independently of
> ARCH_DMA_MINALIGN. In addition, always define ARCH_DMA_MINALIGN even if
> an architecture does not override it.
> 
> After this patch, ARCH_DMA_MINALIGN is expected to be used in static
> alignment annotations and defined by an architecture to be the maximum
> alignment for all supported configurations/SoCs in a single Image.
> ARCH_KMALLOC_MINALIGN, if different, is the minimum alignment guaranteed
> by kmalloc().
> 
> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
> Cc: Andrew Morton <akpm@linux-foundation.org>
> ---
>  include/linux/slab.h | 23 ++++++++++++++++++-----
>  1 file changed, 18 insertions(+), 5 deletions(-)
> 
> diff --git a/include/linux/slab.h b/include/linux/slab.h
> index 373b3ef99f4e..d58211bdeceb 100644
> --- a/include/linux/slab.h
> +++ b/include/linux/slab.h
> @@ -187,17 +187,30 @@ bool kmem_valid_obj(void *object);
>  void kmem_dump_obj(void *object);
>  #endif
>  
> +/*
> + * slob does not support independent control of ARCH_KMALLOC_MINALIGN and
> + * ARCH_DMA_MINALIGN.
> + */
> +#ifdef CONFIG_SLOB
> +#undef ARCH_KMALLOC_MINALIGN
> +#endif
> +
>  /*
>   * Some archs want to perform DMA into kmalloc caches and need a guaranteed
>   * alignment larger than the alignment of a 64-bit integer.
> - * Setting ARCH_KMALLOC_MINALIGN in arch headers allows that.
> + * Setting ARCH_DMA_MINALIGN in arch headers allows that.
>   */
> -#if defined(ARCH_DMA_MINALIGN) && ARCH_DMA_MINALIGN > 8
> +#ifndef ARCH_DMA_MINALIGN
> +#define ARCH_DMA_MINALIGN __alignof__(unsigned long long)
> +#elif ARCH_DMA_MINALIGN > 8 && !defined(ARCH_KMALLOC_MINALIGN)
>  #define ARCH_KMALLOC_MINALIGN ARCH_DMA_MINALIGN
> -#define KMALLOC_MIN_SIZE ARCH_DMA_MINALIGN
> -#define KMALLOC_SHIFT_LOW ilog2(ARCH_DMA_MINALIGN)
> -#else
> +#endif
> +
> +#ifndef ARCH_KMALLOC_MINALIGN
>  #define ARCH_KMALLOC_MINALIGN __alignof__(unsigned long long)
> +#else
> +#define KMALLOC_MIN_SIZE ARCH_KMALLOC_MINALIGN
> +#define KMALLOC_SHIFT_LOW ilog2(KMALLOC_MIN_SIZE)
>  #endif
>  
>  /*
>

Sorry for the noise I made due to misunderstanding :)
Now this patch looks good to me and I think it's worth adding.

Reviewed-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>

and works fine with SLAB/SLOB/SLUB on my arm64 machine.

Tested-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>

Thanks!

-- 
Thanks,
Hyeonggon

Re: [PATCH 01/10] mm/slab: Decouple ARCH_KMALLOC_MINALIGN from ARCH_DMA_MINALIGN

[Hyeonggon Yoo]

On Tue, Apr 05, 2022 at 02:57:49PM +0100, Catalin Marinas wrote:
> In preparation for supporting a dynamic kmalloc() minimum alignment,
> allow architectures to define ARCH_KMALLOC_MINALIGN independently of
> ARCH_DMA_MINALIGN. In addition, always define ARCH_DMA_MINALIGN even if
> an architecture does not override it.
> 
> After this patch, ARCH_DMA_MINALIGN is expected to be used in static
> alignment annotations and defined by an architecture to be the maximum
> alignment for all supported configurations/SoCs in a single Image.
> ARCH_KMALLOC_MINALIGN, if different, is the minimum alignment guaranteed
> by kmalloc().
> 
> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
> Cc: Andrew Morton <akpm@linux-foundation.org>
> ---
>  include/linux/slab.h | 23 ++++++++++++++++++-----
>  1 file changed, 18 insertions(+), 5 deletions(-)
> 
> diff --git a/include/linux/slab.h b/include/linux/slab.h
> index 373b3ef99f4e..d58211bdeceb 100644
> --- a/include/linux/slab.h
> +++ b/include/linux/slab.h
> @@ -187,17 +187,30 @@ bool kmem_valid_obj(void *object);
>  void kmem_dump_obj(void *object);
>  #endif
>  
> +/*
> + * slob does not support independent control of ARCH_KMALLOC_MINALIGN and
> + * ARCH_DMA_MINALIGN.
> + */
> +#ifdef CONFIG_SLOB
> +#undef ARCH_KMALLOC_MINALIGN
> +#endif
> +
>  /*
>   * Some archs want to perform DMA into kmalloc caches and need a guaranteed
>   * alignment larger than the alignment of a 64-bit integer.
> - * Setting ARCH_KMALLOC_MINALIGN in arch headers allows that.
> + * Setting ARCH_DMA_MINALIGN in arch headers allows that.
>   */
> -#if defined(ARCH_DMA_MINALIGN) && ARCH_DMA_MINALIGN > 8
> +#ifndef ARCH_DMA_MINALIGN
> +#define ARCH_DMA_MINALIGN __alignof__(unsigned long long)
> +#elif ARCH_DMA_MINALIGN > 8 && !defined(ARCH_KMALLOC_MINALIGN)
>  #define ARCH_KMALLOC_MINALIGN ARCH_DMA_MINALIGN
> -#define KMALLOC_MIN_SIZE ARCH_DMA_MINALIGN
> -#define KMALLOC_SHIFT_LOW ilog2(ARCH_DMA_MINALIGN)
> -#else
> +#endif
> +
> +#ifndef ARCH_KMALLOC_MINALIGN
>  #define ARCH_KMALLOC_MINALIGN __alignof__(unsigned long long)
> +#else
> +#define KMALLOC_MIN_SIZE ARCH_KMALLOC_MINALIGN
> +#define KMALLOC_SHIFT_LOW ilog2(KMALLOC_MIN_SIZE)
>  #endif
>  
>  /*
>

Sorry for the noise I made due to misunderstanding :)
Now this patch looks good to me and I think it's worth adding.

Reviewed-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>

and works fine with SLAB/SLOB/SLUB on my arm64 machine.

Tested-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>

Thanks!

-- 
Thanks,
Hyeonggon

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 08/10] mm/slab: Allow dynamic kmalloc() minimum alignment

[Hyeonggon Yoo]

On Tue, Apr 05, 2022 at 02:57:56PM +0100, Catalin Marinas wrote:
> ARCH_KMALLOC_MINALIGN represents the minimum guaranteed kmalloc()
> alignment but an architecture may require a larger run-time alignment.
> Do not create kmalloc caches smaller than arch_kmalloc_minalign().
> 
> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
> Cc: Andrew Morton <akpm@linux-foundation.org>
> ---
>  include/linux/slab.h |  2 ++
>  mm/slab.c            |  6 +-----
>  mm/slab.h            |  2 ++
>  mm/slab_common.c     | 33 +++++++++++++++++++++++----------
>  4 files changed, 28 insertions(+), 15 deletions(-)
> 
> diff --git a/include/linux/slab.h b/include/linux/slab.h
> index d58211bdeceb..2137dba85691 100644
> --- a/include/linux/slab.h
> +++ b/include/linux/slab.h
> @@ -332,6 +332,8 @@ enum kmalloc_cache_type {
>  extern struct kmem_cache *
>  kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_SHIFT_HIGH + 1];
>  
> +unsigned int arch_kmalloc_minalign(void);
> +
>  /*
>   * Define gfp bits that should not be set for KMALLOC_NORMAL.
>   */
> diff --git a/mm/slab.c b/mm/slab.c
> index b04e40078bdf..4aaeeb9c994d 100644
> --- a/mm/slab.c
> +++ b/mm/slab.c
> @@ -1256,11 +1256,7 @@ void __init kmem_cache_init(void)
>  	 * Initialize the caches that provide memory for the  kmem_cache_node
>  	 * structures first.  Without this, further allocations will bug.
>  	 */
> -	kmalloc_caches[KMALLOC_NORMAL][INDEX_NODE] = create_kmalloc_cache(
> -				kmalloc_info[INDEX_NODE].name[KMALLOC_NORMAL],
> -				kmalloc_info[INDEX_NODE].size,
> -				ARCH_KMALLOC_FLAGS, 0,
> -				kmalloc_info[INDEX_NODE].size);
> +	new_kmalloc_cache(INDEX_NODE, KMALLOC_NORMAL, ARCH_KMALLOC_FLAGS);
>  	slab_state = PARTIAL_NODE;
>  	setup_kmalloc_cache_index_table();
>  
> diff --git a/mm/slab.h b/mm/slab.h
> index fd7ae2024897..e9238406602a 100644
> --- a/mm/slab.h
> +++ b/mm/slab.h
> @@ -283,6 +283,8 @@ int __kmem_cache_create(struct kmem_cache *, slab_flags_t flags);
>  struct kmem_cache *create_kmalloc_cache(const char *name, unsigned int size,
>  			slab_flags_t flags, unsigned int useroffset,
>  			unsigned int usersize);
> +void __init new_kmalloc_cache(int idx, enum kmalloc_cache_type type,
> +			      slab_flags_t flags);
>  extern void create_boot_cache(struct kmem_cache *, const char *name,
>  			unsigned int size, slab_flags_t flags,
>  			unsigned int useroffset, unsigned int usersize);
> diff --git a/mm/slab_common.c b/mm/slab_common.c
> index 6ee64d6208b3..594d8a8a68d0 100644
> --- a/mm/slab_common.c
> +++ b/mm/slab_common.c
> @@ -838,9 +838,18 @@ void __init setup_kmalloc_cache_index_table(void)
>  	}
>  }
>  
> -static void __init
> +unsigned int __weak arch_kmalloc_minalign(void)
> +{
> +	return ARCH_KMALLOC_MINALIGN;
> +}
> +
> +void __init
>  new_kmalloc_cache(int idx, enum kmalloc_cache_type type, slab_flags_t flags)
>  {
> +	unsigned int minalign = arch_kmalloc_minalign();
> +	unsigned int aligned_size = kmalloc_info[idx].size;
> +	int aligned_idx = idx;
> +
>  	if (type == KMALLOC_RECLAIM) {
>  		flags |= SLAB_RECLAIM_ACCOUNT;
>  	} else if (IS_ENABLED(CONFIG_MEMCG_KMEM) && (type == KMALLOC_CGROUP)) {
> @@ -851,10 +860,17 @@ new_kmalloc_cache(int idx, enum kmalloc_cache_type type, slab_flags_t flags)
>  		flags |= SLAB_ACCOUNT;
>  	}
>  
> -	kmalloc_caches[type][idx] = create_kmalloc_cache(
> -					kmalloc_info[idx].name[type],
> -					kmalloc_info[idx].size, flags, 0,
> -					kmalloc_info[idx].size);
> +	if (minalign > ARCH_KMALLOC_MINALIGN) {
> +		aligned_size = ALIGN(aligned_size, minalign);
> +		aligned_idx = __kmalloc_index(aligned_size, false);
> +	}
> +
> +	if (!kmalloc_caches[type][aligned_idx])
> +		kmalloc_caches[type][aligned_idx] = create_kmalloc_cache(
> +					kmalloc_info[aligned_idx].name[type],
> +					aligned_size, flags, 0, aligned_size);
> +	if (idx != aligned_idx)
> +		kmalloc_caches[type][idx] = kmalloc_caches[type][aligned_idx];
>  
>  	/*
>  	 * If CONFIG_MEMCG_KMEM is enabled, disable cache merging for
> @@ -904,11 +920,8 @@ void __init create_kmalloc_caches(slab_flags_t flags)
>  		struct kmem_cache *s = kmalloc_caches[KMALLOC_NORMAL][i];
>  
>  		if (s) {
> -			kmalloc_caches[KMALLOC_DMA][i] = create_kmalloc_cache(
> -				kmalloc_info[i].name[KMALLOC_DMA],
> -				kmalloc_info[i].size,
> -				SLAB_CACHE_DMA | flags, 0,
> -				kmalloc_info[i].size);
> +			new_kmalloc_cache(i, KMALLOC_DMA,
> +					  SLAB_CACHE_DMA | flags);
>  		}
>  	}
>  #endif

Looks good to me.

Reviewed-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>

And works fine with SLAB/SLUB/SLOB on my arm64 machine.

Tested-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>

Thanks!

-- 
Thanks,
Hyeonggon

Re: [PATCH 08/10] mm/slab: Allow dynamic kmalloc() minimum alignment

[Hyeonggon Yoo]

On Tue, Apr 05, 2022 at 02:57:56PM +0100, Catalin Marinas wrote:
> ARCH_KMALLOC_MINALIGN represents the minimum guaranteed kmalloc()
> alignment but an architecture may require a larger run-time alignment.
> Do not create kmalloc caches smaller than arch_kmalloc_minalign().
> 
> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
> Cc: Andrew Morton <akpm@linux-foundation.org>
> ---
>  include/linux/slab.h |  2 ++
>  mm/slab.c            |  6 +-----
>  mm/slab.h            |  2 ++
>  mm/slab_common.c     | 33 +++++++++++++++++++++++----------
>  4 files changed, 28 insertions(+), 15 deletions(-)
> 
> diff --git a/include/linux/slab.h b/include/linux/slab.h
> index d58211bdeceb..2137dba85691 100644
> --- a/include/linux/slab.h
> +++ b/include/linux/slab.h
> @@ -332,6 +332,8 @@ enum kmalloc_cache_type {
>  extern struct kmem_cache *
>  kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_SHIFT_HIGH + 1];
>  
> +unsigned int arch_kmalloc_minalign(void);
> +
>  /*
>   * Define gfp bits that should not be set for KMALLOC_NORMAL.
>   */
> diff --git a/mm/slab.c b/mm/slab.c
> index b04e40078bdf..4aaeeb9c994d 100644
> --- a/mm/slab.c
> +++ b/mm/slab.c
> @@ -1256,11 +1256,7 @@ void __init kmem_cache_init(void)
>  	 * Initialize the caches that provide memory for the  kmem_cache_node
>  	 * structures first.  Without this, further allocations will bug.
>  	 */
> -	kmalloc_caches[KMALLOC_NORMAL][INDEX_NODE] = create_kmalloc_cache(
> -				kmalloc_info[INDEX_NODE].name[KMALLOC_NORMAL],
> -				kmalloc_info[INDEX_NODE].size,
> -				ARCH_KMALLOC_FLAGS, 0,
> -				kmalloc_info[INDEX_NODE].size);
> +	new_kmalloc_cache(INDEX_NODE, KMALLOC_NORMAL, ARCH_KMALLOC_FLAGS);
>  	slab_state = PARTIAL_NODE;
>  	setup_kmalloc_cache_index_table();
>  
> diff --git a/mm/slab.h b/mm/slab.h
> index fd7ae2024897..e9238406602a 100644
> --- a/mm/slab.h
> +++ b/mm/slab.h
> @@ -283,6 +283,8 @@ int __kmem_cache_create(struct kmem_cache *, slab_flags_t flags);
>  struct kmem_cache *create_kmalloc_cache(const char *name, unsigned int size,
>  			slab_flags_t flags, unsigned int useroffset,
>  			unsigned int usersize);
> +void __init new_kmalloc_cache(int idx, enum kmalloc_cache_type type,
> +			      slab_flags_t flags);
>  extern void create_boot_cache(struct kmem_cache *, const char *name,
>  			unsigned int size, slab_flags_t flags,
>  			unsigned int useroffset, unsigned int usersize);
> diff --git a/mm/slab_common.c b/mm/slab_common.c
> index 6ee64d6208b3..594d8a8a68d0 100644
> --- a/mm/slab_common.c
> +++ b/mm/slab_common.c
> @@ -838,9 +838,18 @@ void __init setup_kmalloc_cache_index_table(void)
>  	}
>  }
>  
> -static void __init
> +unsigned int __weak arch_kmalloc_minalign(void)
> +{
> +	return ARCH_KMALLOC_MINALIGN;
> +}
> +
> +void __init
>  new_kmalloc_cache(int idx, enum kmalloc_cache_type type, slab_flags_t flags)
>  {
> +	unsigned int minalign = arch_kmalloc_minalign();
> +	unsigned int aligned_size = kmalloc_info[idx].size;
> +	int aligned_idx = idx;
> +
>  	if (type == KMALLOC_RECLAIM) {
>  		flags |= SLAB_RECLAIM_ACCOUNT;
>  	} else if (IS_ENABLED(CONFIG_MEMCG_KMEM) && (type == KMALLOC_CGROUP)) {
> @@ -851,10 +860,17 @@ new_kmalloc_cache(int idx, enum kmalloc_cache_type type, slab_flags_t flags)
>  		flags |= SLAB_ACCOUNT;
>  	}
>  
> -	kmalloc_caches[type][idx] = create_kmalloc_cache(
> -					kmalloc_info[idx].name[type],
> -					kmalloc_info[idx].size, flags, 0,
> -					kmalloc_info[idx].size);
> +	if (minalign > ARCH_KMALLOC_MINALIGN) {
> +		aligned_size = ALIGN(aligned_size, minalign);
> +		aligned_idx = __kmalloc_index(aligned_size, false);
> +	}
> +
> +	if (!kmalloc_caches[type][aligned_idx])
> +		kmalloc_caches[type][aligned_idx] = create_kmalloc_cache(
> +					kmalloc_info[aligned_idx].name[type],
> +					aligned_size, flags, 0, aligned_size);
> +	if (idx != aligned_idx)
> +		kmalloc_caches[type][idx] = kmalloc_caches[type][aligned_idx];
>  
>  	/*
>  	 * If CONFIG_MEMCG_KMEM is enabled, disable cache merging for
> @@ -904,11 +920,8 @@ void __init create_kmalloc_caches(slab_flags_t flags)
>  		struct kmem_cache *s = kmalloc_caches[KMALLOC_NORMAL][i];
>  
>  		if (s) {
> -			kmalloc_caches[KMALLOC_DMA][i] = create_kmalloc_cache(
> -				kmalloc_info[i].name[KMALLOC_DMA],
> -				kmalloc_info[i].size,
> -				SLAB_CACHE_DMA | flags, 0,
> -				kmalloc_info[i].size);
> +			new_kmalloc_cache(i, KMALLOC_DMA,
> +					  SLAB_CACHE_DMA | flags);
>  		}
>  	}
>  #endif

Looks good to me.

Reviewed-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>

And works fine with SLAB/SLUB/SLOB on my arm64 machine.

Tested-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>

Thanks!

-- 
Thanks,
Hyeonggon

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 01/10] mm/slab: Decouple ARCH_KMALLOC_MINALIGN from ARCH_DMA_MINALIGN

[Catalin Marinas]

On Mon, Apr 11, 2022 at 07:37:01PM +0900, Hyeonggon Yoo wrote:
> On Tue, Apr 05, 2022 at 02:57:49PM +0100, Catalin Marinas wrote:
> > In preparation for supporting a dynamic kmalloc() minimum alignment,
> > allow architectures to define ARCH_KMALLOC_MINALIGN independently of
> > ARCH_DMA_MINALIGN. In addition, always define ARCH_DMA_MINALIGN even if
> > an architecture does not override it.
> > 
> > After this patch, ARCH_DMA_MINALIGN is expected to be used in static
> > alignment annotations and defined by an architecture to be the maximum
> > alignment for all supported configurations/SoCs in a single Image.
> > ARCH_KMALLOC_MINALIGN, if different, is the minimum alignment guaranteed
> > by kmalloc().
> > 
> > Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
> > Cc: Andrew Morton <akpm@linux-foundation.org>
[...]
> Sorry for the noise I made due to misunderstanding :)
> Now this patch looks good to me and I think it's worth adding.
> 
> Reviewed-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
> 
> and works fine with SLAB/SLOB/SLUB on my arm64 machine.
> 
> Tested-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>

Thanks for the review and test. We still need to solve the potential
crypto issues raised by Herbert before making the change.

-- 
Catalin

Re: [PATCH 01/10] mm/slab: Decouple ARCH_KMALLOC_MINALIGN from ARCH_DMA_MINALIGN

[Catalin Marinas]

On Mon, Apr 11, 2022 at 07:37:01PM +0900, Hyeonggon Yoo wrote:
> On Tue, Apr 05, 2022 at 02:57:49PM +0100, Catalin Marinas wrote:
> > In preparation for supporting a dynamic kmalloc() minimum alignment,
> > allow architectures to define ARCH_KMALLOC_MINALIGN independently of
> > ARCH_DMA_MINALIGN. In addition, always define ARCH_DMA_MINALIGN even if
> > an architecture does not override it.
> > 
> > After this patch, ARCH_DMA_MINALIGN is expected to be used in static
> > alignment annotations and defined by an architecture to be the maximum
> > alignment for all supported configurations/SoCs in a single Image.
> > ARCH_KMALLOC_MINALIGN, if different, is the minimum alignment guaranteed
> > by kmalloc().
> > 
> > Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
> > Cc: Andrew Morton <akpm@linux-foundation.org>
[...]
> Sorry for the noise I made due to misunderstanding :)
> Now this patch looks good to me and I think it's worth adding.
> 
> Reviewed-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
> 
> and works fine with SLAB/SLOB/SLUB on my arm64 machine.
> 
> Tested-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>

Thanks for the review and test. We still need to solve the potential
crypto issues raised by Herbert before making the change.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 02/10] drivers/base: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Andy Shevchenko]

On Wed, Apr 6, 2022 at 2:30 PM Catalin Marinas <catalin.marinas@arm.com> wrote:
>
> ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA
> operations while ARCH_KMALLOC_MINALIGN is the minimum kmalloc() objects
> alignment.

...

> -        * Thus we use ARCH_KMALLOC_MINALIGN here and get exactly the same
> +        * Thus we use ARCH_DMA_MINALIGN here and get at least the same
>          * buffer alignment as if it was allocated by plain kmalloc().

But then it becomes not true either, because the kmalloc() has other
alignment constraints.

-- 
With Best Regards,
Andy Shevchenko

Re: [PATCH 02/10] drivers/base: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Andy Shevchenko]

On Wed, Apr 6, 2022 at 2:30 PM Catalin Marinas <catalin.marinas@arm.com> wrote:
>
> ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA
> operations while ARCH_KMALLOC_MINALIGN is the minimum kmalloc() objects
> alignment.

...

> -        * Thus we use ARCH_KMALLOC_MINALIGN here and get exactly the same
> +        * Thus we use ARCH_DMA_MINALIGN here and get at least the same
>          * buffer alignment as if it was allocated by plain kmalloc().

But then it becomes not true either, because the kmalloc() has other
alignment constraints.

-- 
With Best Regards,
Andy Shevchenko

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 02/10] drivers/base: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Mon, Apr 11, 2022 at 05:57:08PM +0300, Andy Shevchenko wrote:
> On Wed, Apr 6, 2022 at 2:30 PM Catalin Marinas <catalin.marinas@arm.com> wrote:
> > ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA
> > operations while ARCH_KMALLOC_MINALIGN is the minimum kmalloc() objects
> > alignment.
> 
> ...
> 
> > -        * Thus we use ARCH_KMALLOC_MINALIGN here and get exactly the same
> > +        * Thus we use ARCH_DMA_MINALIGN here and get at least the same
> >          * buffer alignment as if it was allocated by plain kmalloc().
> 
> But then it becomes not true either, because the kmalloc() has other
> alignment constraints.

Maybe the comment could be improved a bit but I think it's still valid.
After this patch, struct devres becomes:

struct devres {
	struct devres_node		node;
	u8 __aligned(ARCH_DMA_MINALIGN) data[];
};

While we no longer guarantee the ARCH_DMA_MINALIGN alignment (which is
too big on most arm64 SoCs), what we need is for devres.data[] to be
aligned to the newly introduced arch_kmalloc_minalign(). This would give
us the DMA safety guarantees.

Since devres.data[] is at an offset multiple of ARCH_DMA_MINALIGN, in
order for the array to be aligned to arch_kmalloc_minalign(), all we
need is for ARCH_DMA_MINALIGN to be a multiple of
arch_kmalloc_minalign(). I actually had to write down some simple
equations to convince myself.

devres.data[] is at an offset multiple of ARCH_DMA_MINALIGN (after this
patch), even when struct devres is included in another structure, so we
have:

  offsetof(struct devres, data) = m * ARCH_DMA_MINALIGN

ARCH_DMA_MINALIGN is a power of two while arch_kmalloc_minalign() is
also a power of two, equal to or less than ARCH_DMA_MINALIGN:

  ARCH_DMA_MINALIGN = n * arch_kmalloc_minalign()

A kmalloc()'ed object of struct devres (or a container of) is aligned to
arch_kmalloc_minalign() by definition so:

  kmalloc() = p * arch_kmalloc_minalign()

From the above, we can conclude that the data[] pointer is at a multiple
of arch_kmalloc_minalign():

  devres.data = (p + m * n) * arch_kmalloc_minalign()

Where m, n, p are all positive integers (n is also power of two).

If we did not change the devres structure, the alignment of
ARCH_KMALLOC_MINALIGN would no no be longer sufficient since the dynamic
arch_kmalloc_minalign() can be greater than ARCH_KMALLOC_MINALIGN on
specific SoCs (the first offsetof equation is no longer true).

-- 
Catalin

Re: [PATCH 02/10] drivers/base: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Mon, Apr 11, 2022 at 05:57:08PM +0300, Andy Shevchenko wrote:
> On Wed, Apr 6, 2022 at 2:30 PM Catalin Marinas <catalin.marinas@arm.com> wrote:
> > ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA
> > operations while ARCH_KMALLOC_MINALIGN is the minimum kmalloc() objects
> > alignment.
> 
> ...
> 
> > -        * Thus we use ARCH_KMALLOC_MINALIGN here and get exactly the same
> > +        * Thus we use ARCH_DMA_MINALIGN here and get at least the same
> >          * buffer alignment as if it was allocated by plain kmalloc().
> 
> But then it becomes not true either, because the kmalloc() has other
> alignment constraints.

Maybe the comment could be improved a bit but I think it's still valid.
After this patch, struct devres becomes:

struct devres {
	struct devres_node		node;
	u8 __aligned(ARCH_DMA_MINALIGN) data[];
};

While we no longer guarantee the ARCH_DMA_MINALIGN alignment (which is
too big on most arm64 SoCs), what we need is for devres.data[] to be
aligned to the newly introduced arch_kmalloc_minalign(). This would give
us the DMA safety guarantees.

Since devres.data[] is at an offset multiple of ARCH_DMA_MINALIGN, in
order for the array to be aligned to arch_kmalloc_minalign(), all we
need is for ARCH_DMA_MINALIGN to be a multiple of
arch_kmalloc_minalign(). I actually had to write down some simple
equations to convince myself.

devres.data[] is at an offset multiple of ARCH_DMA_MINALIGN (after this
patch), even when struct devres is included in another structure, so we
have:

  offsetof(struct devres, data) = m * ARCH_DMA_MINALIGN

ARCH_DMA_MINALIGN is a power of two while arch_kmalloc_minalign() is
also a power of two, equal to or less than ARCH_DMA_MINALIGN:

  ARCH_DMA_MINALIGN = n * arch_kmalloc_minalign()

A kmalloc()'ed object of struct devres (or a container of) is aligned to
arch_kmalloc_minalign() by definition so:

  kmalloc() = p * arch_kmalloc_minalign()

From the above, we can conclude that the data[] pointer is at a multiple
of arch_kmalloc_minalign():

  devres.data = (p + m * n) * arch_kmalloc_minalign()

Where m, n, p are all positive integers (n is also power of two).

If we did not change the devres structure, the alignment of
ARCH_KMALLOC_MINALIGN would no no be longer sufficient since the dynamic
arch_kmalloc_minalign() can be greater than ARCH_KMALLOC_MINALIGN on
specific SoCs (the first offsetof equation is no longer true).

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Fri, Apr 08, 2022 at 05:11:28PM +0800, Herbert Xu wrote:
> On Fri, Apr 08, 2022 at 10:04:54AM +0100, Catalin Marinas wrote:
> > My point is that if the crypto code kmallocs a size aligned to
> > crypto_tfm_ctx_alignment() (and CRYPTO_MINALIGN), the slab allocator
> > will return memory aligned to CRYPTO_MINALIGN even if
> > ARCH_KMALLOC_MINALIGN is smaller.
> 
> No we don't align the size to CRYPTO_MINALIGN at all.  We simply
> assume that this is the alignment returned by kmalloc.
> 
> > Would the crypto code, say, do a kmalloc(64) and expect a 128 byte
> > alignment (when CRYPTO_MINALIGN == 128)? Or does it align the size to
> > CRYPTO_MINALIGN and do a kmalloc(128) directly? If it's the latter, I
> > don't think there's a problem.
> 
> It's the former.

Does this only matter for DMA? If there other unrelated alignment
requirements, I think those drivers should be fixed for their own
cra_alignmask independent of the CPU cache line and DMA needs (e.g.
some 1024-bit vectors that would benefit from an aligned load).

With this series, the dynamic arch_kmalloc_minalign() still provides the
DMA-safe alignment even if it would be smaller than the default
CRYPTO_MINALIGN of 128. Let's say we have a structure:

struct crypto_something {
	u8	some_field;
	u8	data[] CRYPTO_MINALIGN_ATTR;
};

The sizeof(struct crypto_something) is automatically a multiple of
CRYPTO_MINALIGN (128 bytes for arm64). While kmalloc() could return a
smaller alignment, arch_kmalloc_minalign(), the data pointer above is
still aligned to arch_kmalloc_minalign() and DMA-safe since
CRYPTO_MINALIGN is a multiple of this (similar to the struct devres
case, https://lore.kernel.org/all/YlRn2Wal4ezjvomZ@arm.com/).

Even if such struct is included in another struct, the alignment and
sizeof is inherited by the containing object.

So let's assume the driver needs 64 bytes of data in addition to the
struct, it would allocate:

	kmalloc(sizeof(struct crypto_something) + 64);

Prior to this series, kmalloc() would return a 256-byte aligned pointer.
With this series, if the cache line size on a SoC is 64-byte, the
allocation falls under the kmalloc-192 cache, so 'data' would be 64-byte
aligned which is safe for DMA.

> I think you can still make the change you want, but first you need
> to modify the affected drivers to specify their actual alignment
> requirement explicitly through cra_alignmask and then use the
> correct methods to access the context pointer.

At a quick grep, most cra_alignmask values are currently 15 or smaller.
I'm not convinced the driver needs to know about the CPU cache
alignment. We could set cra_alignmask to CRYPTO_MINALIGN but that would
incur unnecessary overhead via function like setkey_unaligned() when the
arch_kmalloc_minalign() was already sufficient for DMA safety.

Maybe I miss some use-cases or I focus too much on DMA safety.

Thanks.

-- 
Catalin

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Fri, Apr 08, 2022 at 05:11:28PM +0800, Herbert Xu wrote:
> On Fri, Apr 08, 2022 at 10:04:54AM +0100, Catalin Marinas wrote:
> > My point is that if the crypto code kmallocs a size aligned to
> > crypto_tfm_ctx_alignment() (and CRYPTO_MINALIGN), the slab allocator
> > will return memory aligned to CRYPTO_MINALIGN even if
> > ARCH_KMALLOC_MINALIGN is smaller.
> 
> No we don't align the size to CRYPTO_MINALIGN at all.  We simply
> assume that this is the alignment returned by kmalloc.
> 
> > Would the crypto code, say, do a kmalloc(64) and expect a 128 byte
> > alignment (when CRYPTO_MINALIGN == 128)? Or does it align the size to
> > CRYPTO_MINALIGN and do a kmalloc(128) directly? If it's the latter, I
> > don't think there's a problem.
> 
> It's the former.

Does this only matter for DMA? If there other unrelated alignment
requirements, I think those drivers should be fixed for their own
cra_alignmask independent of the CPU cache line and DMA needs (e.g.
some 1024-bit vectors that would benefit from an aligned load).

With this series, the dynamic arch_kmalloc_minalign() still provides the
DMA-safe alignment even if it would be smaller than the default
CRYPTO_MINALIGN of 128. Let's say we have a structure:

struct crypto_something {
	u8	some_field;
	u8	data[] CRYPTO_MINALIGN_ATTR;
};

The sizeof(struct crypto_something) is automatically a multiple of
CRYPTO_MINALIGN (128 bytes for arm64). While kmalloc() could return a
smaller alignment, arch_kmalloc_minalign(), the data pointer above is
still aligned to arch_kmalloc_minalign() and DMA-safe since
CRYPTO_MINALIGN is a multiple of this (similar to the struct devres
case, https://lore.kernel.org/all/YlRn2Wal4ezjvomZ@arm.com/).

Even if such struct is included in another struct, the alignment and
sizeof is inherited by the containing object.

So let's assume the driver needs 64 bytes of data in addition to the
struct, it would allocate:

	kmalloc(sizeof(struct crypto_something) + 64);

Prior to this series, kmalloc() would return a 256-byte aligned pointer.
With this series, if the cache line size on a SoC is 64-byte, the
allocation falls under the kmalloc-192 cache, so 'data' would be 64-byte
aligned which is safe for DMA.

> I think you can still make the change you want, but first you need
> to modify the affected drivers to specify their actual alignment
> requirement explicitly through cra_alignmask and then use the
> correct methods to access the context pointer.

At a quick grep, most cra_alignmask values are currently 15 or smaller.
I'm not convinced the driver needs to know about the CPU cache
alignment. We could set cra_alignmask to CRYPTO_MINALIGN but that would
incur unnecessary overhead via function like setkey_unaligned() when the
arch_kmalloc_minalign() was already sufficient for DMA safety.

Maybe I miss some use-cases or I focus too much on DMA safety.

Thanks.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Herbert Xu]

On Tue, Apr 12, 2022 at 10:32:58AM +0100, Catalin Marinas wrote:
>
> At a quick grep, most cra_alignmask values are currently 15 or smaller.
> I'm not convinced the driver needs to know about the CPU cache
> alignment. We could set cra_alignmask to CRYPTO_MINALIGN but that would
> incur unnecessary overhead via function like setkey_unaligned() when the
> arch_kmalloc_minalign() was already sufficient for DMA safety.
> 
> Maybe I miss some use-cases or I focus too much on DMA safety.

The alignment was never designed for DMA.  It's mainly for CPUs
that provide accelerated instructions that require a certain
amount of alignment, most commonly 16 bytes.

Therefore CRYPTO_MINALIGN was never meant to be a guarantee for
DMA alignment.  Any drivers relying on this is simply broken.

I understand that on ARM for historical reasons you have had a
situation that CRYPTO_MINALIGN happened to be large enough to
guarantee DMA alignment.  I totally agree with your patch to
try to fix this but it should not penalise other architectures
and raise the CRYPTO_MINALIGN unnecessarily.

I think if CRYPTO_MINALIGN makes those drivers work then so
should cra_alignmask.  And that would be a relatively easy
patch to do.

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Herbert Xu]

On Tue, Apr 12, 2022 at 10:32:58AM +0100, Catalin Marinas wrote:
>
> At a quick grep, most cra_alignmask values are currently 15 or smaller.
> I'm not convinced the driver needs to know about the CPU cache
> alignment. We could set cra_alignmask to CRYPTO_MINALIGN but that would
> incur unnecessary overhead via function like setkey_unaligned() when the
> arch_kmalloc_minalign() was already sufficient for DMA safety.
> 
> Maybe I miss some use-cases or I focus too much on DMA safety.

The alignment was never designed for DMA.  It's mainly for CPUs
that provide accelerated instructions that require a certain
amount of alignment, most commonly 16 bytes.

Therefore CRYPTO_MINALIGN was never meant to be a guarantee for
DMA alignment.  Any drivers relying on this is simply broken.

I understand that on ARM for historical reasons you have had a
situation that CRYPTO_MINALIGN happened to be large enough to
guarantee DMA alignment.  I totally agree with your patch to
try to fix this but it should not penalise other architectures
and raise the CRYPTO_MINALIGN unnecessarily.

I think if CRYPTO_MINALIGN makes those drivers work then so
should cra_alignmask.  And that would be a relatively easy
patch to do.

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Tue, Apr 12, 2022 at 05:40:58PM +0800, Herbert Xu wrote:
> On Tue, Apr 12, 2022 at 10:32:58AM +0100, Catalin Marinas wrote:
> > At a quick grep, most cra_alignmask values are currently 15 or smaller.
> > I'm not convinced the driver needs to know about the CPU cache
> > alignment. We could set cra_alignmask to CRYPTO_MINALIGN but that would
> > incur unnecessary overhead via function like setkey_unaligned() when the
> > arch_kmalloc_minalign() was already sufficient for DMA safety.
> > 
> > Maybe I miss some use-cases or I focus too much on DMA safety.
> 
> The alignment was never designed for DMA.  It's mainly for CPUs
> that provide accelerated instructions that require a certain
> amount of alignment, most commonly 16 bytes.
> 
> Therefore CRYPTO_MINALIGN was never meant to be a guarantee for
> DMA alignment.  Any drivers relying on this is simply broken.

Thanks for the clarification (the comment in crypto.h implies that it
covers this aspect as well).

> I understand that on ARM for historical reasons you have had a
> situation that CRYPTO_MINALIGN happened to be large enough to
> guarantee DMA alignment.  I totally agree with your patch to
> try to fix this but it should not penalise other architectures
> and raise the CRYPTO_MINALIGN unnecessarily.

This series does not penalise any architecture. It doesn't even make
arm64 any worse than it currently is.

For arm64, before the series:

  ARCH_DMA_MINALIGN == 128
  ARCH_KMALLOC_MINALIGN == 128
  CRYPTO_MINALIGN == 128

After the series, on arm64:

  ARCH_DMA_MINALIGN == 128
  ARCH_KMALLOC_MINALIGN == 64
  CRYPTO_MINALIGN == 128

For x86, before the series:

  ARCH_DMA_MINALIGN undefined
  ARCH_KMALLOC_MINALIGN == 8
  CRYPTO_MINALIGN == 8

After the series, on x86 (or any architecture that doesn't define
ARCH_DMA_MINALIGN):

  ARCH_DMA_MINALIGN == 8
  ARCH_KMALLOC_MINALIGN == 8
  CRYPTO_MINALIGN == 8

For other architectures that define ARCH_DMA_MINALIGN to 'N' but do not
define ARCH_KMALLOC_MINALIGN, before and after this series:

  ARCH_DMA_MINALIGN == N
  ARCH_KMALLOC_MINALIGN == N
  CRYPTO_MINALIGN == N

> I think if CRYPTO_MINALIGN makes those drivers work then so
> should cra_alignmask.  And that would be a relatively easy
> patch to do.

Yes, the patch would be simple, subject to figuring out which drivers
and what alignment they actually need (any help appreciated).

There are already arm64 vendor kernels that change ARCH_DMA_MINALIGN to
64, hence ARCH_KMALLOC_MINALIGN and CRYPTO_MINALIGN become 64. We also
discussed a Kconfig option for this in the past. Would that have broken
any crypto drivers that rely on a strict 128 byte alignment?

Thanks.

-- 
Catalin

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Tue, Apr 12, 2022 at 05:40:58PM +0800, Herbert Xu wrote:
> On Tue, Apr 12, 2022 at 10:32:58AM +0100, Catalin Marinas wrote:
> > At a quick grep, most cra_alignmask values are currently 15 or smaller.
> > I'm not convinced the driver needs to know about the CPU cache
> > alignment. We could set cra_alignmask to CRYPTO_MINALIGN but that would
> > incur unnecessary overhead via function like setkey_unaligned() when the
> > arch_kmalloc_minalign() was already sufficient for DMA safety.
> > 
> > Maybe I miss some use-cases or I focus too much on DMA safety.
> 
> The alignment was never designed for DMA.  It's mainly for CPUs
> that provide accelerated instructions that require a certain
> amount of alignment, most commonly 16 bytes.
> 
> Therefore CRYPTO_MINALIGN was never meant to be a guarantee for
> DMA alignment.  Any drivers relying on this is simply broken.

Thanks for the clarification (the comment in crypto.h implies that it
covers this aspect as well).

> I understand that on ARM for historical reasons you have had a
> situation that CRYPTO_MINALIGN happened to be large enough to
> guarantee DMA alignment.  I totally agree with your patch to
> try to fix this but it should not penalise other architectures
> and raise the CRYPTO_MINALIGN unnecessarily.

This series does not penalise any architecture. It doesn't even make
arm64 any worse than it currently is.

For arm64, before the series:

  ARCH_DMA_MINALIGN == 128
  ARCH_KMALLOC_MINALIGN == 128
  CRYPTO_MINALIGN == 128

After the series, on arm64:

  ARCH_DMA_MINALIGN == 128
  ARCH_KMALLOC_MINALIGN == 64
  CRYPTO_MINALIGN == 128

For x86, before the series:

  ARCH_DMA_MINALIGN undefined
  ARCH_KMALLOC_MINALIGN == 8
  CRYPTO_MINALIGN == 8

After the series, on x86 (or any architecture that doesn't define
ARCH_DMA_MINALIGN):

  ARCH_DMA_MINALIGN == 8
  ARCH_KMALLOC_MINALIGN == 8
  CRYPTO_MINALIGN == 8

For other architectures that define ARCH_DMA_MINALIGN to 'N' but do not
define ARCH_KMALLOC_MINALIGN, before and after this series:

  ARCH_DMA_MINALIGN == N
  ARCH_KMALLOC_MINALIGN == N
  CRYPTO_MINALIGN == N

> I think if CRYPTO_MINALIGN makes those drivers work then so
> should cra_alignmask.  And that would be a relatively easy
> patch to do.

Yes, the patch would be simple, subject to figuring out which drivers
and what alignment they actually need (any help appreciated).

There are already arm64 vendor kernels that change ARCH_DMA_MINALIGN to
64, hence ARCH_KMALLOC_MINALIGN and CRYPTO_MINALIGN become 64. We also
discussed a Kconfig option for this in the past. Would that have broken
any crypto drivers that rely on a strict 128 byte alignment?

Thanks.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Herbert Xu]

On Tue, Apr 12, 2022 at 11:02:54AM +0100, Catalin Marinas wrote:
>
> This series does not penalise any architecture. It doesn't even make
> arm64 any worse than it currently is.

Right, the patch as it stands doesn't change anything.  However,
it is also broken as it stands.  As I said before, CRYPTO_MINALIGN
is not something that is guaranteed by the Crypto API, it is simply
a statement of whatever kmalloc returns.

So if kmalloc is no longer returning CRYPTO_MINALIGN-aligned
memory, then those drivers that need this alignment for DMA
will break anyway.

If you want the Crypto API to guarantee alignment over and above
that returned by kmalloc, the correct way is to use cra_alignmask.

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Herbert Xu]

On Tue, Apr 12, 2022 at 11:02:54AM +0100, Catalin Marinas wrote:
>
> This series does not penalise any architecture. It doesn't even make
> arm64 any worse than it currently is.

Right, the patch as it stands doesn't change anything.  However,
it is also broken as it stands.  As I said before, CRYPTO_MINALIGN
is not something that is guaranteed by the Crypto API, it is simply
a statement of whatever kmalloc returns.

So if kmalloc is no longer returning CRYPTO_MINALIGN-aligned
memory, then those drivers that need this alignment for DMA
will break anyway.

If you want the Crypto API to guarantee alignment over and above
that returned by kmalloc, the correct way is to use cra_alignmask.

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Tue, Apr 12, 2022 at 11:02:54AM +0100, Catalin Marinas wrote:
> On Tue, Apr 12, 2022 at 05:40:58PM +0800, Herbert Xu wrote:
> > I think if CRYPTO_MINALIGN makes those drivers work then so
> > should cra_alignmask.  And that would be a relatively easy
> > patch to do.
> 
> Yes, the patch would be simple, subject to figuring out which drivers
> and what alignment they actually need (any help appreciated).
> 
> There are already arm64 vendor kernels that change ARCH_DMA_MINALIGN to
> 64, hence ARCH_KMALLOC_MINALIGN and CRYPTO_MINALIGN become 64. We also
> discussed a Kconfig option for this in the past. Would that have broken
> any crypto drivers that rely on a strict 128 byte alignment?

Actually, I think with a cra_alignmask of 127 (the arm64
CRYPTO_MINALIGN-1), crypto_check_alg() will fail since
MAX_ALGAPI_ALIGNMASK is 63.

-- 
Catalin

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Tue, Apr 12, 2022 at 11:02:54AM +0100, Catalin Marinas wrote:
> On Tue, Apr 12, 2022 at 05:40:58PM +0800, Herbert Xu wrote:
> > I think if CRYPTO_MINALIGN makes those drivers work then so
> > should cra_alignmask.  And that would be a relatively easy
> > patch to do.
> 
> Yes, the patch would be simple, subject to figuring out which drivers
> and what alignment they actually need (any help appreciated).
> 
> There are already arm64 vendor kernels that change ARCH_DMA_MINALIGN to
> 64, hence ARCH_KMALLOC_MINALIGN and CRYPTO_MINALIGN become 64. We also
> discussed a Kconfig option for this in the past. Would that have broken
> any crypto drivers that rely on a strict 128 byte alignment?

Actually, I think with a cra_alignmask of 127 (the arm64
CRYPTO_MINALIGN-1), crypto_check_alg() will fail since
MAX_ALGAPI_ALIGNMASK is 63.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Tue, Apr 12, 2022 at 06:18:46PM +0800, Herbert Xu wrote:
> On Tue, Apr 12, 2022 at 11:02:54AM +0100, Catalin Marinas wrote:
> > This series does not penalise any architecture. It doesn't even make
> > arm64 any worse than it currently is.
> 
> Right, the patch as it stands doesn't change anything.  However,
> it is also broken as it stands.  As I said before, CRYPTO_MINALIGN
> is not something that is guaranteed by the Crypto API, it is simply
> a statement of whatever kmalloc returns.

I agree that CRYPTO_MINALIGN is not guaranteed by the Crypto API. What
I'm debating is the intended use for CRYPTO_MINALIGN in some (most?) of
the drivers. It's not just about kmalloc() but also a build-time offset
of buffers within structures to guarantee DMA safety. This can't be
fixed by cra_alignmask.

We could leave CRYPTO_MINALIGN as ARCH_KMALLOC_MINALIGN and that matches
it being just a statement of the kmalloc() minimum alignment. But since
it is also overloaded with the DMA in-structure offset alignment, we'd
need a new CRYPTO_DMA_MINALIGN (and _ATTR) to annotate those structures.
I have a suspicion there'll be fewer of the original CRYPTO_MINALIGN
uses left, hence my approach to making this bigger from the start.

There's also Ard's series introducing CRYPTO_REQ_MINALIGN while leaving
CRYPT_MINALIGN for DMA-safe offsets (IIUC):

https://lore.kernel.org/r/20220406142715.2270256-1-ardb@kernel.org

> So if kmalloc is no longer returning CRYPTO_MINALIGN-aligned
> memory, then those drivers that need this alignment for DMA
> will break anyway.

No. As per one of my previous emails, kmalloc() will preserve the DMA
alignment for an SoC even if smaller than CRYPTO_MINALIGN (or a new
CRYPTO_DMA_MINALIGN). Since kmalloc() returns DMA-safe pointers and
CRYPTO_MINALIGN (or a new CRYPTO_DMA_MINALIGN) is DMA-safe, so would an
offset from a pointer returned by kmalloc().

> If you want the Crypto API to guarantee alignment over and above
> that returned by kmalloc, the correct way is to use cra_alignmask.

For kmalloc(), this would work, but for the current CRYPTO_MINALIGN_ATTR
uses it won't.

Thanks.

-- 
Catalin

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Tue, Apr 12, 2022 at 06:18:46PM +0800, Herbert Xu wrote:
> On Tue, Apr 12, 2022 at 11:02:54AM +0100, Catalin Marinas wrote:
> > This series does not penalise any architecture. It doesn't even make
> > arm64 any worse than it currently is.
> 
> Right, the patch as it stands doesn't change anything.  However,
> it is also broken as it stands.  As I said before, CRYPTO_MINALIGN
> is not something that is guaranteed by the Crypto API, it is simply
> a statement of whatever kmalloc returns.

I agree that CRYPTO_MINALIGN is not guaranteed by the Crypto API. What
I'm debating is the intended use for CRYPTO_MINALIGN in some (most?) of
the drivers. It's not just about kmalloc() but also a build-time offset
of buffers within structures to guarantee DMA safety. This can't be
fixed by cra_alignmask.

We could leave CRYPTO_MINALIGN as ARCH_KMALLOC_MINALIGN and that matches
it being just a statement of the kmalloc() minimum alignment. But since
it is also overloaded with the DMA in-structure offset alignment, we'd
need a new CRYPTO_DMA_MINALIGN (and _ATTR) to annotate those structures.
I have a suspicion there'll be fewer of the original CRYPTO_MINALIGN
uses left, hence my approach to making this bigger from the start.

There's also Ard's series introducing CRYPTO_REQ_MINALIGN while leaving
CRYPT_MINALIGN for DMA-safe offsets (IIUC):

https://lore.kernel.org/r/20220406142715.2270256-1-ardb@kernel.org

> So if kmalloc is no longer returning CRYPTO_MINALIGN-aligned
> memory, then those drivers that need this alignment for DMA
> will break anyway.

No. As per one of my previous emails, kmalloc() will preserve the DMA
alignment for an SoC even if smaller than CRYPTO_MINALIGN (or a new
CRYPTO_DMA_MINALIGN). Since kmalloc() returns DMA-safe pointers and
CRYPTO_MINALIGN (or a new CRYPTO_DMA_MINALIGN) is DMA-safe, so would an
offset from a pointer returned by kmalloc().

> If you want the Crypto API to guarantee alignment over and above
> that returned by kmalloc, the correct way is to use cra_alignmask.

For kmalloc(), this would work, but for the current CRYPTO_MINALIGN_ATTR
uses it won't.

Thanks.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Ard Biesheuvel]

On Tue, 12 Apr 2022 at 14:31, Catalin Marinas <catalin.marinas@arm.com> wrote:
>
> On Tue, Apr 12, 2022 at 06:18:46PM +0800, Herbert Xu wrote:
> > On Tue, Apr 12, 2022 at 11:02:54AM +0100, Catalin Marinas wrote:
> > > This series does not penalise any architecture. It doesn't even make
> > > arm64 any worse than it currently is.
> >
> > Right, the patch as it stands doesn't change anything.  However,
> > it is also broken as it stands.  As I said before, CRYPTO_MINALIGN
> > is not something that is guaranteed by the Crypto API, it is simply
> > a statement of whatever kmalloc returns.
>
> I agree that CRYPTO_MINALIGN is not guaranteed by the Crypto API. What
> I'm debating is the intended use for CRYPTO_MINALIGN in some (most?) of
> the drivers. It's not just about kmalloc() but also a build-time offset
> of buffers within structures to guarantee DMA safety. This can't be
> fixed by cra_alignmask.
>
> We could leave CRYPTO_MINALIGN as ARCH_KMALLOC_MINALIGN and that matches
> it being just a statement of the kmalloc() minimum alignment. But since
> it is also overloaded with the DMA in-structure offset alignment, we'd
> need a new CRYPTO_DMA_MINALIGN (and _ATTR) to annotate those structures.
> I have a suspicion there'll be fewer of the original CRYPTO_MINALIGN
> uses left, hence my approach to making this bigger from the start.
>
> There's also Ard's series introducing CRYPTO_REQ_MINALIGN while leaving
> CRYPT_MINALIGN for DMA-safe offsets (IIUC):
>
> https://lore.kernel.org/r/20220406142715.2270256-1-ardb@kernel.org
>
> > So if kmalloc is no longer returning CRYPTO_MINALIGN-aligned
> > memory, then those drivers that need this alignment for DMA
> > will break anyway.
>

One thing to note here is that minimum DMA *alignment* is not the same
as the padding to cache writeback granule (CWG) that is needed when
dealing with non-cache coherent inbound DMA.

The former is typically a property of the peripheral IP, and is
something that the driver needs to deal with, potentially by setting
cra_alignmask to ensure that the input and output buffers are placed
such that they can accessed via DMA by the peripheral.

The latter is a property of the CPU's cache hierarchy, not only the
size of the CWG, but also whether or not DMA is cache coherent to
begin with. This is not something the driver should usually have to
care about if it uses the DMA API correctly.

The reason why CRYPTO_MINALIGN matters for DMA in spite of this is
that some drivers not only use DMA for processing the bulk of the data
(typically presented using scatterlists) but sometimes also use DMA to
map parts of the request and TFM structures, which carry control data
used by the CPU to manage the crypto requests. Doing a non-coherent
DMA write into such a structure may blow away 64 or 128 bytes of data,
even if the write itself is much smaller, due to the fact that we need
to perform cache invalidation in order for the CPU to be able to
observe what the device wrote to that memory, and the invalidated
cache lines may be shared with other data items, and may become dirty
while the DMA mapping is active.

This is what I am addressing with my patch series, i.e., padding out
the driver owned parts of the struct to the CWG size so that cache
invalidation does not affect data owned by other layers in the crypto
cake, but only at runtime. By doing this consistently for TFM and
request structures, we should be able to disregard ARCH_DMA_MINALIGN
entirely when it comes to defining CRYPTO_MINALIGN, as it is highly
unlikely that any of these peripherals would require more than 8 or 16
bytes of alignment for the DMA operations themselves.




> No. As per one of my previous emails, kmalloc() will preserve the DMA
> alignment for an SoC even if smaller than CRYPTO_MINALIGN (or a new
> CRYPTO_DMA_MINALIGN). Since kmalloc() returns DMA-safe pointers and
> CRYPTO_MINALIGN (or a new CRYPTO_DMA_MINALIGN) is DMA-safe, so would an
> offset from a pointer returned by kmalloc().
>
> > If you want the Crypto API to guarantee alignment over and above
> > that returned by kmalloc, the correct way is to use cra_alignmask.
>
> For kmalloc(), this would work, but for the current CRYPTO_MINALIGN_ATTR
> uses it won't.
>
> Thanks.
>
> --
> Catalin

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Ard Biesheuvel]

On Tue, 12 Apr 2022 at 14:31, Catalin Marinas <catalin.marinas@arm.com> wrote:
>
> On Tue, Apr 12, 2022 at 06:18:46PM +0800, Herbert Xu wrote:
> > On Tue, Apr 12, 2022 at 11:02:54AM +0100, Catalin Marinas wrote:
> > > This series does not penalise any architecture. It doesn't even make
> > > arm64 any worse than it currently is.
> >
> > Right, the patch as it stands doesn't change anything.  However,
> > it is also broken as it stands.  As I said before, CRYPTO_MINALIGN
> > is not something that is guaranteed by the Crypto API, it is simply
> > a statement of whatever kmalloc returns.
>
> I agree that CRYPTO_MINALIGN is not guaranteed by the Crypto API. What
> I'm debating is the intended use for CRYPTO_MINALIGN in some (most?) of
> the drivers. It's not just about kmalloc() but also a build-time offset
> of buffers within structures to guarantee DMA safety. This can't be
> fixed by cra_alignmask.
>
> We could leave CRYPTO_MINALIGN as ARCH_KMALLOC_MINALIGN and that matches
> it being just a statement of the kmalloc() minimum alignment. But since
> it is also overloaded with the DMA in-structure offset alignment, we'd
> need a new CRYPTO_DMA_MINALIGN (and _ATTR) to annotate those structures.
> I have a suspicion there'll be fewer of the original CRYPTO_MINALIGN
> uses left, hence my approach to making this bigger from the start.
>
> There's also Ard's series introducing CRYPTO_REQ_MINALIGN while leaving
> CRYPT_MINALIGN for DMA-safe offsets (IIUC):
>
> https://lore.kernel.org/r/20220406142715.2270256-1-ardb@kernel.org
>
> > So if kmalloc is no longer returning CRYPTO_MINALIGN-aligned
> > memory, then those drivers that need this alignment for DMA
> > will break anyway.
>

One thing to note here is that minimum DMA *alignment* is not the same
as the padding to cache writeback granule (CWG) that is needed when
dealing with non-cache coherent inbound DMA.

The former is typically a property of the peripheral IP, and is
something that the driver needs to deal with, potentially by setting
cra_alignmask to ensure that the input and output buffers are placed
such that they can accessed via DMA by the peripheral.

The latter is a property of the CPU's cache hierarchy, not only the
size of the CWG, but also whether or not DMA is cache coherent to
begin with. This is not something the driver should usually have to
care about if it uses the DMA API correctly.

The reason why CRYPTO_MINALIGN matters for DMA in spite of this is
that some drivers not only use DMA for processing the bulk of the data
(typically presented using scatterlists) but sometimes also use DMA to
map parts of the request and TFM structures, which carry control data
used by the CPU to manage the crypto requests. Doing a non-coherent
DMA write into such a structure may blow away 64 or 128 bytes of data,
even if the write itself is much smaller, due to the fact that we need
to perform cache invalidation in order for the CPU to be able to
observe what the device wrote to that memory, and the invalidated
cache lines may be shared with other data items, and may become dirty
while the DMA mapping is active.

This is what I am addressing with my patch series, i.e., padding out
the driver owned parts of the struct to the CWG size so that cache
invalidation does not affect data owned by other layers in the crypto
cake, but only at runtime. By doing this consistently for TFM and
request structures, we should be able to disregard ARCH_DMA_MINALIGN
entirely when it comes to defining CRYPTO_MINALIGN, as it is highly
unlikely that any of these peripherals would require more than 8 or 16
bytes of alignment for the DMA operations themselves.




> No. As per one of my previous emails, kmalloc() will preserve the DMA
> alignment for an SoC even if smaller than CRYPTO_MINALIGN (or a new
> CRYPTO_DMA_MINALIGN). Since kmalloc() returns DMA-safe pointers and
> CRYPTO_MINALIGN (or a new CRYPTO_DMA_MINALIGN) is DMA-safe, so would an
> offset from a pointer returned by kmalloc().
>
> > If you want the Crypto API to guarantee alignment over and above
> > that returned by kmalloc, the correct way is to use cra_alignmask.
>
> For kmalloc(), this would work, but for the current CRYPTO_MINALIGN_ATTR
> uses it won't.
>
> Thanks.
>
> --
> Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Wed, Apr 13, 2022 at 12:01:25AM +0200, Ard Biesheuvel wrote:
> On Tue, 12 Apr 2022 at 14:31, Catalin Marinas <catalin.marinas@arm.com> wrote:
> > On Tue, Apr 12, 2022 at 06:18:46PM +0800, Herbert Xu wrote:
> > > On Tue, Apr 12, 2022 at 11:02:54AM +0100, Catalin Marinas wrote:
> > > > This series does not penalise any architecture. It doesn't even make
> > > > arm64 any worse than it currently is.
> > >
> > > Right, the patch as it stands doesn't change anything.  However,
> > > it is also broken as it stands.  As I said before, CRYPTO_MINALIGN
> > > is not something that is guaranteed by the Crypto API, it is simply
> > > a statement of whatever kmalloc returns.
> >
> > I agree that CRYPTO_MINALIGN is not guaranteed by the Crypto API. What
> > I'm debating is the intended use for CRYPTO_MINALIGN in some (most?) of
> > the drivers. It's not just about kmalloc() but also a build-time offset
> > of buffers within structures to guarantee DMA safety. This can't be
> > fixed by cra_alignmask.
> >
> > We could leave CRYPTO_MINALIGN as ARCH_KMALLOC_MINALIGN and that matches
> > it being just a statement of the kmalloc() minimum alignment. But since
> > it is also overloaded with the DMA in-structure offset alignment, we'd
> > need a new CRYPTO_DMA_MINALIGN (and _ATTR) to annotate those structures.
> > I have a suspicion there'll be fewer of the original CRYPTO_MINALIGN
> > uses left, hence my approach to making this bigger from the start.
> >
> > There's also Ard's series introducing CRYPTO_REQ_MINALIGN while leaving
> > CRYPT_MINALIGN for DMA-safe offsets (IIUC):
> >
> > https://lore.kernel.org/r/20220406142715.2270256-1-ardb@kernel.org
> >
> > > So if kmalloc is no longer returning CRYPTO_MINALIGN-aligned
> > > memory, then those drivers that need this alignment for DMA
> > > will break anyway.
> 
> One thing to note here is that minimum DMA *alignment* is not the same
> as the padding to cache writeback granule (CWG) that is needed when
> dealing with non-cache coherent inbound DMA.
> 
> The former is typically a property of the peripheral IP, and is
> something that the driver needs to deal with, potentially by setting
> cra_alignmask to ensure that the input and output buffers are placed
> such that they can accessed via DMA by the peripheral.

Good point, thanks for making this clear. AFAICT, the requirement for
bus master access in the crypto code doesn't go above 16 (a
cra_alignmask of 15).

> The latter is a property of the CPU's cache hierarchy, not only the
> size of the CWG, but also whether or not DMA is cache coherent to
> begin with. This is not something the driver should usually have to
> care about if it uses the DMA API correctly.

I agree. There is also an implicit expectation that the DMA API works on
kmalloc'ed buffers and that's what ARCH_DMA_MINALIGN is for (and the
dynamic arch_kmalloc_minalign() in this series). But the key point is
that the driver doesn't need to know the CPU cache topology, coherency,
the DMA API and kmalloc() take care of these.

> The reason why CRYPTO_MINALIGN matters for DMA in spite of this is
> that some drivers not only use DMA for processing the bulk of the data
> (typically presented using scatterlists) but sometimes also use DMA to
> map parts of the request and TFM structures, which carry control data
> used by the CPU to manage the crypto requests. Doing a non-coherent
> DMA write into such a structure may blow away 64 or 128 bytes of data,
> even if the write itself is much smaller, due to the fact that we need
> to perform cache invalidation in order for the CPU to be able to
> observe what the device wrote to that memory, and the invalidated
> cache lines may be shared with other data items, and may become dirty
> while the DMA mapping is active.

Indeed.

> This is what I am addressing with my patch series, i.e., padding out
> the driver owned parts of the struct to the CWG size so that cache
> invalidation does not affect data owned by other layers in the crypto
> cake, but only at runtime. By doing this consistently for TFM and
> request structures, we should be able to disregard ARCH_DMA_MINALIGN
> entirely when it comes to defining CRYPTO_MINALIGN, as it is highly
> unlikely that any of these peripherals would require more than 8 or 16
> bytes of alignment for the DMA operations themselves.

Your series would be a nice improvement and eventually get
CRYPTO_MINALIGN to ARCH_SLAB_MINALIGN. In the meantime, with my series,
we need CRYPTO_MINALIGN to be ARCH_DMA_MINALIGN on arm64, unless we add
a new CRYPT_DMA_MINALIGN but I don't see much point, especially if you
plan to reduce this alignment anyway.


Let's go back to restating the crypto code alignment requirements, as I
understand them (please correct):

1. Bus master accesses (DMA, CPUs that can't do efficient unaligned
   accesses). That's what cra_alignmask is for. If there's a driver that
   relies on an implicit kmalloc() alignment higher than cra_alignmask,
   it is already broken on x86 and a few other architectures that don't
   define ARCH_DMA_MINALIGN. But it won't be any worse with this series
   since it only brings the arm64 kmalloc() alignment down from 128 to
   64.

2. Non-coherent DMA and cache invalidation. With my series, kmalloc'ed
   buffers are DMA-safe for the CPU/SoC the kernel is running on.

3. DMA into buffers embedded in TFM structures. Since the offset of
   those buffers is decided at compile-time, we need a
   CRYPTO_MINALIGN_ATTR that covers both bus master alignment
   requirements and the highest cache line size (CWG) for a non-coherent
   DMA SoC. Ard's series would simplify the latter but, before then,
   CRYPTO_MINALIGN needs to stay as the larger ARCH_DMA_MINALIGN.

With my series, there is no change to the value of CRYPTO_MINALIGN for
arm64 or any other architecture, so point 3 is unaffected. The series
does change the kmalloc() alignment and that may be smaller than
CRYPTO_MINALIGN but neither of points 1 or 2 above are affected since
(a) we still have a sufficiently large ARCH_KMALLOC_MINALIGN of 64 and
(b) the kmalloc'ed buffers are safe for non-coherent DMA.

Herbert, Ard, if I missed anything please let me know but based on my
understanding, this series is safe for the crypto code.

Thanks.

-- 
Catalin

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Wed, Apr 13, 2022 at 12:01:25AM +0200, Ard Biesheuvel wrote:
> On Tue, 12 Apr 2022 at 14:31, Catalin Marinas <catalin.marinas@arm.com> wrote:
> > On Tue, Apr 12, 2022 at 06:18:46PM +0800, Herbert Xu wrote:
> > > On Tue, Apr 12, 2022 at 11:02:54AM +0100, Catalin Marinas wrote:
> > > > This series does not penalise any architecture. It doesn't even make
> > > > arm64 any worse than it currently is.
> > >
> > > Right, the patch as it stands doesn't change anything.  However,
> > > it is also broken as it stands.  As I said before, CRYPTO_MINALIGN
> > > is not something that is guaranteed by the Crypto API, it is simply
> > > a statement of whatever kmalloc returns.
> >
> > I agree that CRYPTO_MINALIGN is not guaranteed by the Crypto API. What
> > I'm debating is the intended use for CRYPTO_MINALIGN in some (most?) of
> > the drivers. It's not just about kmalloc() but also a build-time offset
> > of buffers within structures to guarantee DMA safety. This can't be
> > fixed by cra_alignmask.
> >
> > We could leave CRYPTO_MINALIGN as ARCH_KMALLOC_MINALIGN and that matches
> > it being just a statement of the kmalloc() minimum alignment. But since
> > it is also overloaded with the DMA in-structure offset alignment, we'd
> > need a new CRYPTO_DMA_MINALIGN (and _ATTR) to annotate those structures.
> > I have a suspicion there'll be fewer of the original CRYPTO_MINALIGN
> > uses left, hence my approach to making this bigger from the start.
> >
> > There's also Ard's series introducing CRYPTO_REQ_MINALIGN while leaving
> > CRYPT_MINALIGN for DMA-safe offsets (IIUC):
> >
> > https://lore.kernel.org/r/20220406142715.2270256-1-ardb@kernel.org
> >
> > > So if kmalloc is no longer returning CRYPTO_MINALIGN-aligned
> > > memory, then those drivers that need this alignment for DMA
> > > will break anyway.
> 
> One thing to note here is that minimum DMA *alignment* is not the same
> as the padding to cache writeback granule (CWG) that is needed when
> dealing with non-cache coherent inbound DMA.
> 
> The former is typically a property of the peripheral IP, and is
> something that the driver needs to deal with, potentially by setting
> cra_alignmask to ensure that the input and output buffers are placed
> such that they can accessed via DMA by the peripheral.

Good point, thanks for making this clear. AFAICT, the requirement for
bus master access in the crypto code doesn't go above 16 (a
cra_alignmask of 15).

> The latter is a property of the CPU's cache hierarchy, not only the
> size of the CWG, but also whether or not DMA is cache coherent to
> begin with. This is not something the driver should usually have to
> care about if it uses the DMA API correctly.

I agree. There is also an implicit expectation that the DMA API works on
kmalloc'ed buffers and that's what ARCH_DMA_MINALIGN is for (and the
dynamic arch_kmalloc_minalign() in this series). But the key point is
that the driver doesn't need to know the CPU cache topology, coherency,
the DMA API and kmalloc() take care of these.

> The reason why CRYPTO_MINALIGN matters for DMA in spite of this is
> that some drivers not only use DMA for processing the bulk of the data
> (typically presented using scatterlists) but sometimes also use DMA to
> map parts of the request and TFM structures, which carry control data
> used by the CPU to manage the crypto requests. Doing a non-coherent
> DMA write into such a structure may blow away 64 or 128 bytes of data,
> even if the write itself is much smaller, due to the fact that we need
> to perform cache invalidation in order for the CPU to be able to
> observe what the device wrote to that memory, and the invalidated
> cache lines may be shared with other data items, and may become dirty
> while the DMA mapping is active.

Indeed.

> This is what I am addressing with my patch series, i.e., padding out
> the driver owned parts of the struct to the CWG size so that cache
> invalidation does not affect data owned by other layers in the crypto
> cake, but only at runtime. By doing this consistently for TFM and
> request structures, we should be able to disregard ARCH_DMA_MINALIGN
> entirely when it comes to defining CRYPTO_MINALIGN, as it is highly
> unlikely that any of these peripherals would require more than 8 or 16
> bytes of alignment for the DMA operations themselves.

Your series would be a nice improvement and eventually get
CRYPTO_MINALIGN to ARCH_SLAB_MINALIGN. In the meantime, with my series,
we need CRYPTO_MINALIGN to be ARCH_DMA_MINALIGN on arm64, unless we add
a new CRYPT_DMA_MINALIGN but I don't see much point, especially if you
plan to reduce this alignment anyway.


Let's go back to restating the crypto code alignment requirements, as I
understand them (please correct):

1. Bus master accesses (DMA, CPUs that can't do efficient unaligned
   accesses). That's what cra_alignmask is for. If there's a driver that
   relies on an implicit kmalloc() alignment higher than cra_alignmask,
   it is already broken on x86 and a few other architectures that don't
   define ARCH_DMA_MINALIGN. But it won't be any worse with this series
   since it only brings the arm64 kmalloc() alignment down from 128 to
   64.

2. Non-coherent DMA and cache invalidation. With my series, kmalloc'ed
   buffers are DMA-safe for the CPU/SoC the kernel is running on.

3. DMA into buffers embedded in TFM structures. Since the offset of
   those buffers is decided at compile-time, we need a
   CRYPTO_MINALIGN_ATTR that covers both bus master alignment
   requirements and the highest cache line size (CWG) for a non-coherent
   DMA SoC. Ard's series would simplify the latter but, before then,
   CRYPTO_MINALIGN needs to stay as the larger ARCH_DMA_MINALIGN.

With my series, there is no change to the value of CRYPTO_MINALIGN for
arm64 or any other architecture, so point 3 is unaffected. The series
does change the kmalloc() alignment and that may be smaller than
CRYPTO_MINALIGN but neither of points 1 or 2 above are affected since
(a) we still have a sufficiently large ARCH_KMALLOC_MINALIGN of 64 and
(b) the kmalloc'ed buffers are safe for non-coherent DMA.

Herbert, Ard, if I missed anything please let me know but based on my
understanding, this series is safe for the crypto code.

Thanks.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Linus Torvalds]

On Tue, Apr 12, 2022 at 10:47 PM Catalin Marinas
<catalin.marinas@arm.com> wrote:
>
> I agree. There is also an implicit expectation that the DMA API works on
> kmalloc'ed buffers and that's what ARCH_DMA_MINALIGN is for (and the
> dynamic arch_kmalloc_minalign() in this series). But the key point is
> that the driver doesn't need to know the CPU cache topology, coherency,
> the DMA API and kmalloc() take care of these.

Honestly, I think it would probably be worth discussing the "kmalloc
DMA alignment" issues.

99.9% of kmalloc users don't want to do DMA.

And there's actually a fair amount of small kmalloc for random stuff.
Right now on my laptop, I have

    kmalloc-8          16907  18432      8  512    1 : ...

according to slabinfo, so almost 17 _thousand_ allocations of 8 bytes.

It's all kinds of sad if those allocations need to be 64 bytes in size
just because of some silly DMA alignment issue, when none of them want
it.

Yeah, yeah, wasting a megabyte of memory is "just a megabyte" these
days. Which is crazy. It's literally memory that could have been used
for something much more useful than just pure and utter waste.

I think we could and should just say "people who actually require DMA
accesses should say so at kmalloc time". We literally have that
GFP_DMA and ZOME_DMA for various historical reasons, so we've been
able to do that before.

No, that historical GFP_DMA isn't what arm64 wants - it's the old
crazy "legacy 16MB DMA" thing that ISA DMA used to have.

But the basic issue was true then, and is true now - DMA allocations
are fairly special, and should not be that hard to just mark as such.

We could add a trivial wrapper function like

     static void *dma_kmalloc(size_t size)
     { return kmalloc(size | (ARCH_DMA_MINALIGN-1); }

which now means that the size argument is guaranteed to be big enough
(not not overflow to zero) that you get that aligned memory
allocation.

We could perhaps even have other special rules. Including really
specific ones, like saying

 - allocations smaller than 32 bytes are not DMA coherent, because we pack them

which would allow those small allocations to not pointlessly waste memory.

I dunno. But it's ridiculous that arm64 wastes so much memory when
it's approximately never needed.

                 Linus

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Linus Torvalds]

On Tue, Apr 12, 2022 at 10:47 PM Catalin Marinas
<catalin.marinas@arm.com> wrote:
>
> I agree. There is also an implicit expectation that the DMA API works on
> kmalloc'ed buffers and that's what ARCH_DMA_MINALIGN is for (and the
> dynamic arch_kmalloc_minalign() in this series). But the key point is
> that the driver doesn't need to know the CPU cache topology, coherency,
> the DMA API and kmalloc() take care of these.

Honestly, I think it would probably be worth discussing the "kmalloc
DMA alignment" issues.

99.9% of kmalloc users don't want to do DMA.

And there's actually a fair amount of small kmalloc for random stuff.
Right now on my laptop, I have

    kmalloc-8          16907  18432      8  512    1 : ...

according to slabinfo, so almost 17 _thousand_ allocations of 8 bytes.

It's all kinds of sad if those allocations need to be 64 bytes in size
just because of some silly DMA alignment issue, when none of them want
it.

Yeah, yeah, wasting a megabyte of memory is "just a megabyte" these
days. Which is crazy. It's literally memory that could have been used
for something much more useful than just pure and utter waste.

I think we could and should just say "people who actually require DMA
accesses should say so at kmalloc time". We literally have that
GFP_DMA and ZOME_DMA for various historical reasons, so we've been
able to do that before.

No, that historical GFP_DMA isn't what arm64 wants - it's the old
crazy "legacy 16MB DMA" thing that ISA DMA used to have.

But the basic issue was true then, and is true now - DMA allocations
are fairly special, and should not be that hard to just mark as such.

We could add a trivial wrapper function like

     static void *dma_kmalloc(size_t size)
     { return kmalloc(size | (ARCH_DMA_MINALIGN-1); }

which now means that the size argument is guaranteed to be big enough
(not not overflow to zero) that you get that aligned memory
allocation.

We could perhaps even have other special rules. Including really
specific ones, like saying

 - allocations smaller than 32 bytes are not DMA coherent, because we pack them

which would allow those small allocations to not pointlessly waste memory.

I dunno. But it's ridiculous that arm64 wastes so much memory when
it's approximately never needed.

                 Linus

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Greg Kroah-Hartman]

On Wed, Apr 13, 2022 at 09:53:24AM -1000, Linus Torvalds wrote:
> On Tue, Apr 12, 2022 at 10:47 PM Catalin Marinas
> <catalin.marinas@arm.com> wrote:
> >
> > I agree. There is also an implicit expectation that the DMA API works on
> > kmalloc'ed buffers and that's what ARCH_DMA_MINALIGN is for (and the
> > dynamic arch_kmalloc_minalign() in this series). But the key point is
> > that the driver doesn't need to know the CPU cache topology, coherency,
> > the DMA API and kmalloc() take care of these.
> 
> Honestly, I think it would probably be worth discussing the "kmalloc
> DMA alignment" issues.
> 
> 99.9% of kmalloc users don't want to do DMA.
> 
> And there's actually a fair amount of small kmalloc for random stuff.
> Right now on my laptop, I have
> 
>     kmalloc-8          16907  18432      8  512    1 : ...
> 
> according to slabinfo, so almost 17 _thousand_ allocations of 8 bytes.
> 
> It's all kinds of sad if those allocations need to be 64 bytes in size
> just because of some silly DMA alignment issue, when none of them want
> it.
> 
> Yeah, yeah, wasting a megabyte of memory is "just a megabyte" these
> days. Which is crazy. It's literally memory that could have been used
> for something much more useful than just pure and utter waste.
> 
> I think we could and should just say "people who actually require DMA
> accesses should say so at kmalloc time". We literally have that
> GFP_DMA and ZOME_DMA for various historical reasons, so we've been
> able to do that before.
> 
> No, that historical GFP_DMA isn't what arm64 wants - it's the old
> crazy "legacy 16MB DMA" thing that ISA DMA used to have.
> 
> But the basic issue was true then, and is true now - DMA allocations
> are fairly special, and should not be that hard to just mark as such.

"fairly special" == "all USB transactions", so it will take a lot of
auditing here.  I think also many SPI controllers require this and maybe
I2C?  Perhaps other bus types do as well.

So please don't make this change without some way of figuring out just
what drivers need to be fixed up, as it's going to be a lot...

thanks,

greg k-h

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Greg Kroah-Hartman]

On Wed, Apr 13, 2022 at 09:53:24AM -1000, Linus Torvalds wrote:
> On Tue, Apr 12, 2022 at 10:47 PM Catalin Marinas
> <catalin.marinas@arm.com> wrote:
> >
> > I agree. There is also an implicit expectation that the DMA API works on
> > kmalloc'ed buffers and that's what ARCH_DMA_MINALIGN is for (and the
> > dynamic arch_kmalloc_minalign() in this series). But the key point is
> > that the driver doesn't need to know the CPU cache topology, coherency,
> > the DMA API and kmalloc() take care of these.
> 
> Honestly, I think it would probably be worth discussing the "kmalloc
> DMA alignment" issues.
> 
> 99.9% of kmalloc users don't want to do DMA.
> 
> And there's actually a fair amount of small kmalloc for random stuff.
> Right now on my laptop, I have
> 
>     kmalloc-8          16907  18432      8  512    1 : ...
> 
> according to slabinfo, so almost 17 _thousand_ allocations of 8 bytes.
> 
> It's all kinds of sad if those allocations need to be 64 bytes in size
> just because of some silly DMA alignment issue, when none of them want
> it.
> 
> Yeah, yeah, wasting a megabyte of memory is "just a megabyte" these
> days. Which is crazy. It's literally memory that could have been used
> for something much more useful than just pure and utter waste.
> 
> I think we could and should just say "people who actually require DMA
> accesses should say so at kmalloc time". We literally have that
> GFP_DMA and ZOME_DMA for various historical reasons, so we've been
> able to do that before.
> 
> No, that historical GFP_DMA isn't what arm64 wants - it's the old
> crazy "legacy 16MB DMA" thing that ISA DMA used to have.
> 
> But the basic issue was true then, and is true now - DMA allocations
> are fairly special, and should not be that hard to just mark as such.

"fairly special" == "all USB transactions", so it will take a lot of
auditing here.  I think also many SPI controllers require this and maybe
I2C?  Perhaps other bus types do as well.

So please don't make this change without some way of figuring out just
what drivers need to be fixed up, as it's going to be a lot...

thanks,

greg k-h

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Ard Biesheuvel]

On Thu, 14 Apr 2022 at 07:38, Greg Kroah-Hartman
<gregkh@linuxfoundation.org> wrote:
>
> On Wed, Apr 13, 2022 at 09:53:24AM -1000, Linus Torvalds wrote:
> > On Tue, Apr 12, 2022 at 10:47 PM Catalin Marinas
> > <catalin.marinas@arm.com> wrote:
> > >
> > > I agree. There is also an implicit expectation that the DMA API works on
> > > kmalloc'ed buffers and that's what ARCH_DMA_MINALIGN is for (and the
> > > dynamic arch_kmalloc_minalign() in this series). But the key point is
> > > that the driver doesn't need to know the CPU cache topology, coherency,
> > > the DMA API and kmalloc() take care of these.
> >
> > Honestly, I think it would probably be worth discussing the "kmalloc
> > DMA alignment" issues.
> >
> > 99.9% of kmalloc users don't want to do DMA.
> >
> > And there's actually a fair amount of small kmalloc for random stuff.
> > Right now on my laptop, I have
> >
> >     kmalloc-8          16907  18432      8  512    1 : ...
> >
> > according to slabinfo, so almost 17 _thousand_ allocations of 8 bytes.
> >
> > It's all kinds of sad if those allocations need to be 64 bytes in size
> > just because of some silly DMA alignment issue, when none of them want
> > it.
> >

Actually, the alignment for non-cache coherent DMA is 128 bytes on
arm64, not 64 bytes.

> > Yeah, yeah, wasting a megabyte of memory is "just a megabyte" these
> > days. Which is crazy. It's literally memory that could have been used
> > for something much more useful than just pure and utter waste.
> >
> > I think we could and should just say "people who actually require DMA
> > accesses should say so at kmalloc time". We literally have that
> > GFP_DMA and ZOME_DMA for various historical reasons, so we've been
> > able to do that before.
> >
> > No, that historical GFP_DMA isn't what arm64 wants - it's the old
> > crazy "legacy 16MB DMA" thing that ISA DMA used to have.
> >
> > But the basic issue was true then, and is true now - DMA allocations
> > are fairly special, and should not be that hard to just mark as such.
>
> "fairly special" == "all USB transactions", so it will take a lot of
> auditing here.  I think also many SPI controllers require this and maybe
> I2C?  Perhaps other bus types do as well.
>
> So please don't make this change without some way of figuring out just
> what drivers need to be fixed up, as it's going to be a lot...
>

Yeah, the current de facto contract of being able to DMA map anything
that was allocated via the linear map makes it quite hard to enforce
the use of dma_kmalloc() for this.

What we might do, given the fact that only inbound non-cache coherent
DMA is problematic, is dropping the kmalloc alignment to 8 like on
x86, and falling back to bounce buffering when a misaligned, non-cache
coherent inbound DMA mapping is created, using the SWIOTLB bounce
buffering code that we already have, and is already in use on most
affected systems for other reasons (i.e., DMA addressing limits)

This will cause some performance regressions, but in a way that seems
fixable to me: taking network drivers as an example, the RX buffers
that are filled using inbound DMA are typically owned by the driver
itself, which could be updated to round up its allocations and DMA
mappings. Block devices typically operate on quantities that are
aligned sufficiently already. In other cases, we will likely notice
if/when this fallback is taken on a hot path, but if we don't, at
least we know a bounce buffer is being used whenever we cannot perform
the DMA safely in-place.

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Ard Biesheuvel]

On Thu, 14 Apr 2022 at 07:38, Greg Kroah-Hartman
<gregkh@linuxfoundation.org> wrote:
>
> On Wed, Apr 13, 2022 at 09:53:24AM -1000, Linus Torvalds wrote:
> > On Tue, Apr 12, 2022 at 10:47 PM Catalin Marinas
> > <catalin.marinas@arm.com> wrote:
> > >
> > > I agree. There is also an implicit expectation that the DMA API works on
> > > kmalloc'ed buffers and that's what ARCH_DMA_MINALIGN is for (and the
> > > dynamic arch_kmalloc_minalign() in this series). But the key point is
> > > that the driver doesn't need to know the CPU cache topology, coherency,
> > > the DMA API and kmalloc() take care of these.
> >
> > Honestly, I think it would probably be worth discussing the "kmalloc
> > DMA alignment" issues.
> >
> > 99.9% of kmalloc users don't want to do DMA.
> >
> > And there's actually a fair amount of small kmalloc for random stuff.
> > Right now on my laptop, I have
> >
> >     kmalloc-8          16907  18432      8  512    1 : ...
> >
> > according to slabinfo, so almost 17 _thousand_ allocations of 8 bytes.
> >
> > It's all kinds of sad if those allocations need to be 64 bytes in size
> > just because of some silly DMA alignment issue, when none of them want
> > it.
> >

Actually, the alignment for non-cache coherent DMA is 128 bytes on
arm64, not 64 bytes.

> > Yeah, yeah, wasting a megabyte of memory is "just a megabyte" these
> > days. Which is crazy. It's literally memory that could have been used
> > for something much more useful than just pure and utter waste.
> >
> > I think we could and should just say "people who actually require DMA
> > accesses should say so at kmalloc time". We literally have that
> > GFP_DMA and ZOME_DMA for various historical reasons, so we've been
> > able to do that before.
> >
> > No, that historical GFP_DMA isn't what arm64 wants - it's the old
> > crazy "legacy 16MB DMA" thing that ISA DMA used to have.
> >
> > But the basic issue was true then, and is true now - DMA allocations
> > are fairly special, and should not be that hard to just mark as such.
>
> "fairly special" == "all USB transactions", so it will take a lot of
> auditing here.  I think also many SPI controllers require this and maybe
> I2C?  Perhaps other bus types do as well.
>
> So please don't make this change without some way of figuring out just
> what drivers need to be fixed up, as it's going to be a lot...
>

Yeah, the current de facto contract of being able to DMA map anything
that was allocated via the linear map makes it quite hard to enforce
the use of dma_kmalloc() for this.

What we might do, given the fact that only inbound non-cache coherent
DMA is problematic, is dropping the kmalloc alignment to 8 like on
x86, and falling back to bounce buffering when a misaligned, non-cache
coherent inbound DMA mapping is created, using the SWIOTLB bounce
buffering code that we already have, and is already in use on most
affected systems for other reasons (i.e., DMA addressing limits)

This will cause some performance regressions, but in a way that seems
fixable to me: taking network drivers as an example, the RX buffers
that are filled using inbound DMA are typically owned by the driver
itself, which could be updated to round up its allocations and DMA
mappings. Block devices typically operate on quantities that are
aligned sufficiently already. In other cases, we will likely notice
if/when this fallback is taken on a hot path, but if we don't, at
least we know a bounce buffer is being used whenever we cannot perform
the DMA safely in-place.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Greg Kroah-Hartman]

On Thu, Apr 14, 2022 at 03:52:53PM +0200, Ard Biesheuvel wrote:
> On Thu, 14 Apr 2022 at 07:38, Greg Kroah-Hartman
> <gregkh@linuxfoundation.org> wrote:
> >
> > On Wed, Apr 13, 2022 at 09:53:24AM -1000, Linus Torvalds wrote:
> > > On Tue, Apr 12, 2022 at 10:47 PM Catalin Marinas
> > > <catalin.marinas@arm.com> wrote:
> > > >
> > > > I agree. There is also an implicit expectation that the DMA API works on
> > > > kmalloc'ed buffers and that's what ARCH_DMA_MINALIGN is for (and the
> > > > dynamic arch_kmalloc_minalign() in this series). But the key point is
> > > > that the driver doesn't need to know the CPU cache topology, coherency,
> > > > the DMA API and kmalloc() take care of these.
> > >
> > > Honestly, I think it would probably be worth discussing the "kmalloc
> > > DMA alignment" issues.
> > >
> > > 99.9% of kmalloc users don't want to do DMA.
> > >
> > > And there's actually a fair amount of small kmalloc for random stuff.
> > > Right now on my laptop, I have
> > >
> > >     kmalloc-8          16907  18432      8  512    1 : ...
> > >
> > > according to slabinfo, so almost 17 _thousand_ allocations of 8 bytes.
> > >
> > > It's all kinds of sad if those allocations need to be 64 bytes in size
> > > just because of some silly DMA alignment issue, when none of them want
> > > it.
> > >
> 
> Actually, the alignment for non-cache coherent DMA is 128 bytes on
> arm64, not 64 bytes.
> 
> > > Yeah, yeah, wasting a megabyte of memory is "just a megabyte" these
> > > days. Which is crazy. It's literally memory that could have been used
> > > for something much more useful than just pure and utter waste.
> > >
> > > I think we could and should just say "people who actually require DMA
> > > accesses should say so at kmalloc time". We literally have that
> > > GFP_DMA and ZOME_DMA for various historical reasons, so we've been
> > > able to do that before.
> > >
> > > No, that historical GFP_DMA isn't what arm64 wants - it's the old
> > > crazy "legacy 16MB DMA" thing that ISA DMA used to have.
> > >
> > > But the basic issue was true then, and is true now - DMA allocations
> > > are fairly special, and should not be that hard to just mark as such.
> >
> > "fairly special" == "all USB transactions", so it will take a lot of
> > auditing here.  I think also many SPI controllers require this and maybe
> > I2C?  Perhaps other bus types do as well.
> >
> > So please don't make this change without some way of figuring out just
> > what drivers need to be fixed up, as it's going to be a lot...
> >
> 
> Yeah, the current de facto contract of being able to DMA map anything
> that was allocated via the linear map makes it quite hard to enforce
> the use of dma_kmalloc() for this.
> 
> What we might do, given the fact that only inbound non-cache coherent
> DMA is problematic, is dropping the kmalloc alignment to 8 like on
> x86, and falling back to bounce buffering when a misaligned, non-cache
> coherent inbound DMA mapping is created, using the SWIOTLB bounce
> buffering code that we already have, and is already in use on most
> affected systems for other reasons (i.e., DMA addressing limits)

Ick, that's a mess.

> This will cause some performance regressions, but in a way that seems
> fixable to me: taking network drivers as an example, the RX buffers
> that are filled using inbound DMA are typically owned by the driver
> itself, which could be updated to round up its allocations and DMA
> mappings. Block devices typically operate on quantities that are
> aligned sufficiently already. In other cases, we will likely notice
> if/when this fallback is taken on a hot path, but if we don't, at
> least we know a bounce buffer is being used whenever we cannot perform
> the DMA safely in-place.

We can move to having an "allocator-per-bus" for memory like this to
allow the bus to know if this is a DMA requirement or not.

So for all USB drivers, we would have:
	usb_kmalloc(size, flags);
and then it might even be easier to verify with static tools that the
USB drivers are sending only properly allocated data.  Same for SPI and
other busses.

https://lore.kernel.org/r/230a9486fc68ea0182df46255e42a51099403642.1648032613.git.christophe.leroy@csgroup.eu
is an example of a SPI driver that has been there "for forever" yet
always got it wrong.  If we could have had some way to know "only memory
allocated with this function is allowed on the bus" that would have
fixed the issue a long time ago.

Anyway, just an idea, it's up to you all if this is worth it or not.
Adding performance regressions at the expense of memory size feels like
a rough trade-off to go through until things are fixed up.

greg k-h

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Greg Kroah-Hartman]

On Thu, Apr 14, 2022 at 03:52:53PM +0200, Ard Biesheuvel wrote:
> On Thu, 14 Apr 2022 at 07:38, Greg Kroah-Hartman
> <gregkh@linuxfoundation.org> wrote:
> >
> > On Wed, Apr 13, 2022 at 09:53:24AM -1000, Linus Torvalds wrote:
> > > On Tue, Apr 12, 2022 at 10:47 PM Catalin Marinas
> > > <catalin.marinas@arm.com> wrote:
> > > >
> > > > I agree. There is also an implicit expectation that the DMA API works on
> > > > kmalloc'ed buffers and that's what ARCH_DMA_MINALIGN is for (and the
> > > > dynamic arch_kmalloc_minalign() in this series). But the key point is
> > > > that the driver doesn't need to know the CPU cache topology, coherency,
> > > > the DMA API and kmalloc() take care of these.
> > >
> > > Honestly, I think it would probably be worth discussing the "kmalloc
> > > DMA alignment" issues.
> > >
> > > 99.9% of kmalloc users don't want to do DMA.
> > >
> > > And there's actually a fair amount of small kmalloc for random stuff.
> > > Right now on my laptop, I have
> > >
> > >     kmalloc-8          16907  18432      8  512    1 : ...
> > >
> > > according to slabinfo, so almost 17 _thousand_ allocations of 8 bytes.
> > >
> > > It's all kinds of sad if those allocations need to be 64 bytes in size
> > > just because of some silly DMA alignment issue, when none of them want
> > > it.
> > >
> 
> Actually, the alignment for non-cache coherent DMA is 128 bytes on
> arm64, not 64 bytes.
> 
> > > Yeah, yeah, wasting a megabyte of memory is "just a megabyte" these
> > > days. Which is crazy. It's literally memory that could have been used
> > > for something much more useful than just pure and utter waste.
> > >
> > > I think we could and should just say "people who actually require DMA
> > > accesses should say so at kmalloc time". We literally have that
> > > GFP_DMA and ZOME_DMA for various historical reasons, so we've been
> > > able to do that before.
> > >
> > > No, that historical GFP_DMA isn't what arm64 wants - it's the old
> > > crazy "legacy 16MB DMA" thing that ISA DMA used to have.
> > >
> > > But the basic issue was true then, and is true now - DMA allocations
> > > are fairly special, and should not be that hard to just mark as such.
> >
> > "fairly special" == "all USB transactions", so it will take a lot of
> > auditing here.  I think also many SPI controllers require this and maybe
> > I2C?  Perhaps other bus types do as well.
> >
> > So please don't make this change without some way of figuring out just
> > what drivers need to be fixed up, as it's going to be a lot...
> >
> 
> Yeah, the current de facto contract of being able to DMA map anything
> that was allocated via the linear map makes it quite hard to enforce
> the use of dma_kmalloc() for this.
> 
> What we might do, given the fact that only inbound non-cache coherent
> DMA is problematic, is dropping the kmalloc alignment to 8 like on
> x86, and falling back to bounce buffering when a misaligned, non-cache
> coherent inbound DMA mapping is created, using the SWIOTLB bounce
> buffering code that we already have, and is already in use on most
> affected systems for other reasons (i.e., DMA addressing limits)

Ick, that's a mess.

> This will cause some performance regressions, but in a way that seems
> fixable to me: taking network drivers as an example, the RX buffers
> that are filled using inbound DMA are typically owned by the driver
> itself, which could be updated to round up its allocations and DMA
> mappings. Block devices typically operate on quantities that are
> aligned sufficiently already. In other cases, we will likely notice
> if/when this fallback is taken on a hot path, but if we don't, at
> least we know a bounce buffer is being used whenever we cannot perform
> the DMA safely in-place.

We can move to having an "allocator-per-bus" for memory like this to
allow the bus to know if this is a DMA requirement or not.

So for all USB drivers, we would have:
	usb_kmalloc(size, flags);
and then it might even be easier to verify with static tools that the
USB drivers are sending only properly allocated data.  Same for SPI and
other busses.

https://lore.kernel.org/r/230a9486fc68ea0182df46255e42a51099403642.1648032613.git.christophe.leroy@csgroup.eu
is an example of a SPI driver that has been there "for forever" yet
always got it wrong.  If we could have had some way to know "only memory
allocated with this function is allowed on the bus" that would have
fixed the issue a long time ago.

Anyway, just an idea, it's up to you all if this is worth it or not.
Adding performance regressions at the expense of memory size feels like
a rough trade-off to go through until things are fixed up.

greg k-h

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Ard Biesheuvel]

On Wed, 13 Apr 2022 at 10:47, Catalin Marinas <catalin.marinas@arm.com> wrote:
..
>
> Let's go back to restating the crypto code alignment requirements, as I
> understand them (please correct):
>
> 1. Bus master accesses (DMA, CPUs that can't do efficient unaligned
>    accesses). That's what cra_alignmask is for. If there's a driver that
>    relies on an implicit kmalloc() alignment higher than cra_alignmask,
>    it is already broken on x86 and a few other architectures that don't
>    define ARCH_DMA_MINALIGN. But it won't be any worse with this series
>    since it only brings the arm64 kmalloc() alignment down from 128 to
>    64.
>
> 2. Non-coherent DMA and cache invalidation. With my series, kmalloc'ed
>    buffers are DMA-safe for the CPU/SoC the kernel is running on.
>
> 3. DMA into buffers embedded in TFM structures. Since the offset of
>    those buffers is decided at compile-time, we need a
>    CRYPTO_MINALIGN_ATTR that covers both bus master alignment
>    requirements and the highest cache line size (CWG) for a non-coherent
>    DMA SoC. Ard's series would simplify the latter but, before then,
>    CRYPTO_MINALIGN needs to stay as the larger ARCH_DMA_MINALIGN.
>
> With my series, there is no change to the value of CRYPTO_MINALIGN for
> arm64 or any other architecture, so point 3 is unaffected. The series
> does change the kmalloc() alignment and that may be smaller than
> CRYPTO_MINALIGN but neither of points 1 or 2 above are affected since
> (a) we still have a sufficiently large ARCH_KMALLOC_MINALIGN of 64 and
> (b) the kmalloc'ed buffers are safe for non-coherent DMA.
>
> Herbert, Ard, if I missed anything please let me know but based on my
> understanding, this series is safe for the crypto code.
>

Agreed.

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Ard Biesheuvel]

On Wed, 13 Apr 2022 at 10:47, Catalin Marinas <catalin.marinas@arm.com> wrote:
..
>
> Let's go back to restating the crypto code alignment requirements, as I
> understand them (please correct):
>
> 1. Bus master accesses (DMA, CPUs that can't do efficient unaligned
>    accesses). That's what cra_alignmask is for. If there's a driver that
>    relies on an implicit kmalloc() alignment higher than cra_alignmask,
>    it is already broken on x86 and a few other architectures that don't
>    define ARCH_DMA_MINALIGN. But it won't be any worse with this series
>    since it only brings the arm64 kmalloc() alignment down from 128 to
>    64.
>
> 2. Non-coherent DMA and cache invalidation. With my series, kmalloc'ed
>    buffers are DMA-safe for the CPU/SoC the kernel is running on.
>
> 3. DMA into buffers embedded in TFM structures. Since the offset of
>    those buffers is decided at compile-time, we need a
>    CRYPTO_MINALIGN_ATTR that covers both bus master alignment
>    requirements and the highest cache line size (CWG) for a non-coherent
>    DMA SoC. Ard's series would simplify the latter but, before then,
>    CRYPTO_MINALIGN needs to stay as the larger ARCH_DMA_MINALIGN.
>
> With my series, there is no change to the value of CRYPTO_MINALIGN for
> arm64 or any other architecture, so point 3 is unaffected. The series
> does change the kmalloc() alignment and that may be smaller than
> CRYPTO_MINALIGN but neither of points 1 or 2 above are affected since
> (a) we still have a sufficiently large ARCH_KMALLOC_MINALIGN of 64 and
> (b) the kmalloc'ed buffers are safe for non-coherent DMA.
>
> Herbert, Ard, if I missed anything please let me know but based on my
> understanding, this series is safe for the crypto code.
>

Agreed.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Ard Biesheuvel]

On Thu, 14 Apr 2022 at 16:27, Greg Kroah-Hartman
<gregkh@linuxfoundation.org> wrote:
>
> On Thu, Apr 14, 2022 at 03:52:53PM +0200, Ard Biesheuvel wrote:
> > On Thu, 14 Apr 2022 at 07:38, Greg Kroah-Hartman
> > <gregkh@linuxfoundation.org> wrote:
> > >
> > > On Wed, Apr 13, 2022 at 09:53:24AM -1000, Linus Torvalds wrote:
...
> > > > Honestly, I think it would probably be worth discussing the "kmalloc
> > > > DMA alignment" issues.
> > > >
> > > > 99.9% of kmalloc users don't want to do DMA.
> > > >
> > > > And there's actually a fair amount of small kmalloc for random stuff.
> > > > Right now on my laptop, I have
> > > >
> > > >     kmalloc-8          16907  18432      8  512    1 : ...
> > > >
> > > > according to slabinfo, so almost 17 _thousand_ allocations of 8 bytes.
> > > >
> > > > It's all kinds of sad if those allocations need to be 64 bytes in size
> > > > just because of some silly DMA alignment issue, when none of them want
> > > > it.
> > > >
> >
> > Actually, the alignment for non-cache coherent DMA is 128 bytes on
> > arm64, not 64 bytes.
> >
> > > > Yeah, yeah, wasting a megabyte of memory is "just a megabyte" these
> > > > days. Which is crazy. It's literally memory that could have been used
> > > > for something much more useful than just pure and utter waste.
> > > >
> > > > I think we could and should just say "people who actually require DMA
> > > > accesses should say so at kmalloc time". We literally have that
> > > > GFP_DMA and ZOME_DMA for various historical reasons, so we've been
> > > > able to do that before.
> > > >
> > > > No, that historical GFP_DMA isn't what arm64 wants - it's the old
> > > > crazy "legacy 16MB DMA" thing that ISA DMA used to have.
> > > >
> > > > But the basic issue was true then, and is true now - DMA allocations
> > > > are fairly special, and should not be that hard to just mark as such.
> > >
> > > "fairly special" == "all USB transactions", so it will take a lot of
> > > auditing here.  I think also many SPI controllers require this and maybe
> > > I2C?  Perhaps other bus types do as well.
> > >
> > > So please don't make this change without some way of figuring out just
> > > what drivers need to be fixed up, as it's going to be a lot...
> > >
> >
> > Yeah, the current de facto contract of being able to DMA map anything
> > that was allocated via the linear map makes it quite hard to enforce
> > the use of dma_kmalloc() for this.
> >
> > What we might do, given the fact that only inbound non-cache coherent
> > DMA is problematic, is dropping the kmalloc alignment to 8 like on
> > x86, and falling back to bounce buffering when a misaligned, non-cache
> > coherent inbound DMA mapping is created, using the SWIOTLB bounce
> > buffering code that we already have, and is already in use on most
> > affected systems for other reasons (i.e., DMA addressing limits)
>
> Ick, that's a mess.
>
> > This will cause some performance regressions, but in a way that seems
> > fixable to me: taking network drivers as an example, the RX buffers
> > that are filled using inbound DMA are typically owned by the driver
> > itself, which could be updated to round up its allocations and DMA
> > mappings. Block devices typically operate on quantities that are
> > aligned sufficiently already. In other cases, we will likely notice
> > if/when this fallback is taken on a hot path, but if we don't, at
> > least we know a bounce buffer is being used whenever we cannot perform
> > the DMA safely in-place.
>
> We can move to having an "allocator-per-bus" for memory like this to
> allow the bus to know if this is a DMA requirement or not.
>
> So for all USB drivers, we would have:
>         usb_kmalloc(size, flags);
> and then it might even be easier to verify with static tools that the
> USB drivers are sending only properly allocated data.  Same for SPI and
> other busses.
>

As I pointed out earlier in the thread, alignment/padding requirements
for non-coherent DMA are a property of the CPU's cache hierarchy, not
of the device. So I'm not sure I follow how a per-subsystem
distinction would help here. In the case of USB especially, would that
mean that block, media and networking subsystems would need to be
aware of the USB-ness of the underlying transport?

> https://lore.kernel.org/r/230a9486fc68ea0182df46255e42a51099403642.1648032613.git.christophe.leroy@csgroup.eu
> is an example of a SPI driver that has been there "for forever" yet
> always got it wrong.  If we could have had some way to know "only memory
> allocated with this function is allowed on the bus" that would have
> fixed the issue a long time ago.
>
> Anyway, just an idea, it's up to you all if this is worth it or not.
> Adding performance regressions at the expense of memory size feels like
> a rough trade-off to go through until things are fixed up.
>

Yeah, I hear you. But we already have distinct classes of memory,
i.e., vmalloc vs kmalloc, where only the latter is permitted for DMA
(which is why VMAP stack broke that SPI driver), and I'm not sure
adding more types is going to make this tractable going forward.

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Ard Biesheuvel]

On Thu, 14 Apr 2022 at 16:27, Greg Kroah-Hartman
<gregkh@linuxfoundation.org> wrote:
>
> On Thu, Apr 14, 2022 at 03:52:53PM +0200, Ard Biesheuvel wrote:
> > On Thu, 14 Apr 2022 at 07:38, Greg Kroah-Hartman
> > <gregkh@linuxfoundation.org> wrote:
> > >
> > > On Wed, Apr 13, 2022 at 09:53:24AM -1000, Linus Torvalds wrote:
...
> > > > Honestly, I think it would probably be worth discussing the "kmalloc
> > > > DMA alignment" issues.
> > > >
> > > > 99.9% of kmalloc users don't want to do DMA.
> > > >
> > > > And there's actually a fair amount of small kmalloc for random stuff.
> > > > Right now on my laptop, I have
> > > >
> > > >     kmalloc-8          16907  18432      8  512    1 : ...
> > > >
> > > > according to slabinfo, so almost 17 _thousand_ allocations of 8 bytes.
> > > >
> > > > It's all kinds of sad if those allocations need to be 64 bytes in size
> > > > just because of some silly DMA alignment issue, when none of them want
> > > > it.
> > > >
> >
> > Actually, the alignment for non-cache coherent DMA is 128 bytes on
> > arm64, not 64 bytes.
> >
> > > > Yeah, yeah, wasting a megabyte of memory is "just a megabyte" these
> > > > days. Which is crazy. It's literally memory that could have been used
> > > > for something much more useful than just pure and utter waste.
> > > >
> > > > I think we could and should just say "people who actually require DMA
> > > > accesses should say so at kmalloc time". We literally have that
> > > > GFP_DMA and ZOME_DMA for various historical reasons, so we've been
> > > > able to do that before.
> > > >
> > > > No, that historical GFP_DMA isn't what arm64 wants - it's the old
> > > > crazy "legacy 16MB DMA" thing that ISA DMA used to have.
> > > >
> > > > But the basic issue was true then, and is true now - DMA allocations
> > > > are fairly special, and should not be that hard to just mark as such.
> > >
> > > "fairly special" == "all USB transactions", so it will take a lot of
> > > auditing here.  I think also many SPI controllers require this and maybe
> > > I2C?  Perhaps other bus types do as well.
> > >
> > > So please don't make this change without some way of figuring out just
> > > what drivers need to be fixed up, as it's going to be a lot...
> > >
> >
> > Yeah, the current de facto contract of being able to DMA map anything
> > that was allocated via the linear map makes it quite hard to enforce
> > the use of dma_kmalloc() for this.
> >
> > What we might do, given the fact that only inbound non-cache coherent
> > DMA is problematic, is dropping the kmalloc alignment to 8 like on
> > x86, and falling back to bounce buffering when a misaligned, non-cache
> > coherent inbound DMA mapping is created, using the SWIOTLB bounce
> > buffering code that we already have, and is already in use on most
> > affected systems for other reasons (i.e., DMA addressing limits)
>
> Ick, that's a mess.
>
> > This will cause some performance regressions, but in a way that seems
> > fixable to me: taking network drivers as an example, the RX buffers
> > that are filled using inbound DMA are typically owned by the driver
> > itself, which could be updated to round up its allocations and DMA
> > mappings. Block devices typically operate on quantities that are
> > aligned sufficiently already. In other cases, we will likely notice
> > if/when this fallback is taken on a hot path, but if we don't, at
> > least we know a bounce buffer is being used whenever we cannot perform
> > the DMA safely in-place.
>
> We can move to having an "allocator-per-bus" for memory like this to
> allow the bus to know if this is a DMA requirement or not.
>
> So for all USB drivers, we would have:
>         usb_kmalloc(size, flags);
> and then it might even be easier to verify with static tools that the
> USB drivers are sending only properly allocated data.  Same for SPI and
> other busses.
>

As I pointed out earlier in the thread, alignment/padding requirements
for non-coherent DMA are a property of the CPU's cache hierarchy, not
of the device. So I'm not sure I follow how a per-subsystem
distinction would help here. In the case of USB especially, would that
mean that block, media and networking subsystems would need to be
aware of the USB-ness of the underlying transport?

> https://lore.kernel.org/r/230a9486fc68ea0182df46255e42a51099403642.1648032613.git.christophe.leroy@csgroup.eu
> is an example of a SPI driver that has been there "for forever" yet
> always got it wrong.  If we could have had some way to know "only memory
> allocated with this function is allowed on the bus" that would have
> fixed the issue a long time ago.
>
> Anyway, just an idea, it's up to you all if this is worth it or not.
> Adding performance regressions at the expense of memory size feels like
> a rough trade-off to go through until things are fixed up.
>

Yeah, I hear you. But we already have distinct classes of memory,
i.e., vmalloc vs kmalloc, where only the latter is permitted for DMA
(which is why VMAP stack broke that SPI driver), and I'm not sure
adding more types is going to make this tractable going forward.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Greg Kroah-Hartman]

On Thu, Apr 14, 2022 at 04:36:46PM +0200, Ard Biesheuvel wrote:
> On Thu, 14 Apr 2022 at 16:27, Greg Kroah-Hartman
> <gregkh@linuxfoundation.org> wrote:
> >
> > On Thu, Apr 14, 2022 at 03:52:53PM +0200, Ard Biesheuvel wrote:
> > > On Thu, 14 Apr 2022 at 07:38, Greg Kroah-Hartman
> > > <gregkh@linuxfoundation.org> wrote:
> > > >
> > > > On Wed, Apr 13, 2022 at 09:53:24AM -1000, Linus Torvalds wrote:
> ...
> > > > > Honestly, I think it would probably be worth discussing the "kmalloc
> > > > > DMA alignment" issues.
> > > > >
> > > > > 99.9% of kmalloc users don't want to do DMA.
> > > > >
> > > > > And there's actually a fair amount of small kmalloc for random stuff.
> > > > > Right now on my laptop, I have
> > > > >
> > > > >     kmalloc-8          16907  18432      8  512    1 : ...
> > > > >
> > > > > according to slabinfo, so almost 17 _thousand_ allocations of 8 bytes.
> > > > >
> > > > > It's all kinds of sad if those allocations need to be 64 bytes in size
> > > > > just because of some silly DMA alignment issue, when none of them want
> > > > > it.
> > > > >
> > >
> > > Actually, the alignment for non-cache coherent DMA is 128 bytes on
> > > arm64, not 64 bytes.
> > >
> > > > > Yeah, yeah, wasting a megabyte of memory is "just a megabyte" these
> > > > > days. Which is crazy. It's literally memory that could have been used
> > > > > for something much more useful than just pure and utter waste.
> > > > >
> > > > > I think we could and should just say "people who actually require DMA
> > > > > accesses should say so at kmalloc time". We literally have that
> > > > > GFP_DMA and ZOME_DMA for various historical reasons, so we've been
> > > > > able to do that before.
> > > > >
> > > > > No, that historical GFP_DMA isn't what arm64 wants - it's the old
> > > > > crazy "legacy 16MB DMA" thing that ISA DMA used to have.
> > > > >
> > > > > But the basic issue was true then, and is true now - DMA allocations
> > > > > are fairly special, and should not be that hard to just mark as such.
> > > >
> > > > "fairly special" == "all USB transactions", so it will take a lot of
> > > > auditing here.  I think also many SPI controllers require this and maybe
> > > > I2C?  Perhaps other bus types do as well.
> > > >
> > > > So please don't make this change without some way of figuring out just
> > > > what drivers need to be fixed up, as it's going to be a lot...
> > > >
> > >
> > > Yeah, the current de facto contract of being able to DMA map anything
> > > that was allocated via the linear map makes it quite hard to enforce
> > > the use of dma_kmalloc() for this.
> > >
> > > What we might do, given the fact that only inbound non-cache coherent
> > > DMA is problematic, is dropping the kmalloc alignment to 8 like on
> > > x86, and falling back to bounce buffering when a misaligned, non-cache
> > > coherent inbound DMA mapping is created, using the SWIOTLB bounce
> > > buffering code that we already have, and is already in use on most
> > > affected systems for other reasons (i.e., DMA addressing limits)
> >
> > Ick, that's a mess.
> >
> > > This will cause some performance regressions, but in a way that seems
> > > fixable to me: taking network drivers as an example, the RX buffers
> > > that are filled using inbound DMA are typically owned by the driver
> > > itself, which could be updated to round up its allocations and DMA
> > > mappings. Block devices typically operate on quantities that are
> > > aligned sufficiently already. In other cases, we will likely notice
> > > if/when this fallback is taken on a hot path, but if we don't, at
> > > least we know a bounce buffer is being used whenever we cannot perform
> > > the DMA safely in-place.
> >
> > We can move to having an "allocator-per-bus" for memory like this to
> > allow the bus to know if this is a DMA requirement or not.
> >
> > So for all USB drivers, we would have:
> >         usb_kmalloc(size, flags);
> > and then it might even be easier to verify with static tools that the
> > USB drivers are sending only properly allocated data.  Same for SPI and
> > other busses.
> >
> 
> As I pointed out earlier in the thread, alignment/padding requirements
> for non-coherent DMA are a property of the CPU's cache hierarchy, not
> of the device. So I'm not sure I follow how a per-subsystem
> distinction would help here. In the case of USB especially, would that
> mean that block, media and networking subsystems would need to be
> aware of the USB-ness of the underlying transport?

That's what we have required today, yes.  That's only because we knew
that for some USB controllers, that was a requirement and we had no way
of passing that information back up the stack so we just made it a
requirement.

But I do agree this is messy.  It's even messier for things like USB
where it's not the USB device itself that matters, it's the USB
controller that the USB device is attached to.  And that can be _way_ up
the device hierarchy.  Attach something like a NFS mount over a PPP
network connection on a USB to serial device and ugh, where do you
begin?  :)

And is this always just an issue of the CPU cache hierarchy?  And not the
specific bridge that a device is connected to that CPU on?  Or am I
saying the same thing here?

I mean take a USB controller for example.  We could have a system where
one USB controller is on a PCI bus, while another is on a "platform"
bus.  Both of those are connected to the CPU in different ways and so
could have different DMA rules.  Do we downgrade everything in the
system for the worst connection possible?

Again, consider a USB driver allocating memory to transfer stuff, should
it somehow know the cache hierarchy that it is connected to?  Right now
we punt and do not do that at the expense of a bit of potentially
wasted memory for small allocations.


> > https://lore.kernel.org/r/230a9486fc68ea0182df46255e42a51099403642.1648032613.git.christophe.leroy@csgroup.eu
> > is an example of a SPI driver that has been there "for forever" yet
> > always got it wrong.  If we could have had some way to know "only memory
> > allocated with this function is allowed on the bus" that would have
> > fixed the issue a long time ago.
> >
> > Anyway, just an idea, it's up to you all if this is worth it or not.
> > Adding performance regressions at the expense of memory size feels like
> > a rough trade-off to go through until things are fixed up.
> >
> 
> Yeah, I hear you. But we already have distinct classes of memory,
> i.e., vmalloc vs kmalloc, where only the latter is permitted for DMA
> (which is why VMAP stack broke that SPI driver), and I'm not sure
> adding more types is going to make this tractable going forward.

Sorry, it would not be a "real" type of memory at all, just something
with a static tag on it at the compiler level (like we do for endian
stuff with sparse today).  The real memory would be allocated with the
normal kmalloc() and friends it's just that static tools can be run to
verify "yes, all of the calls to usb_submit_urb() were done with memory
allocated with usb_kmalloc(), and not from the stack"

No runtime changes needed for different types, that would be a mess.
Let's lean on static tools when we can.

thanks,

greg k-h

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Greg Kroah-Hartman]

On Thu, Apr 14, 2022 at 04:36:46PM +0200, Ard Biesheuvel wrote:
> On Thu, 14 Apr 2022 at 16:27, Greg Kroah-Hartman
> <gregkh@linuxfoundation.org> wrote:
> >
> > On Thu, Apr 14, 2022 at 03:52:53PM +0200, Ard Biesheuvel wrote:
> > > On Thu, 14 Apr 2022 at 07:38, Greg Kroah-Hartman
> > > <gregkh@linuxfoundation.org> wrote:
> > > >
> > > > On Wed, Apr 13, 2022 at 09:53:24AM -1000, Linus Torvalds wrote:
> ...
> > > > > Honestly, I think it would probably be worth discussing the "kmalloc
> > > > > DMA alignment" issues.
> > > > >
> > > > > 99.9% of kmalloc users don't want to do DMA.
> > > > >
> > > > > And there's actually a fair amount of small kmalloc for random stuff.
> > > > > Right now on my laptop, I have
> > > > >
> > > > >     kmalloc-8          16907  18432      8  512    1 : ...
> > > > >
> > > > > according to slabinfo, so almost 17 _thousand_ allocations of 8 bytes.
> > > > >
> > > > > It's all kinds of sad if those allocations need to be 64 bytes in size
> > > > > just because of some silly DMA alignment issue, when none of them want
> > > > > it.
> > > > >
> > >
> > > Actually, the alignment for non-cache coherent DMA is 128 bytes on
> > > arm64, not 64 bytes.
> > >
> > > > > Yeah, yeah, wasting a megabyte of memory is "just a megabyte" these
> > > > > days. Which is crazy. It's literally memory that could have been used
> > > > > for something much more useful than just pure and utter waste.
> > > > >
> > > > > I think we could and should just say "people who actually require DMA
> > > > > accesses should say so at kmalloc time". We literally have that
> > > > > GFP_DMA and ZOME_DMA for various historical reasons, so we've been
> > > > > able to do that before.
> > > > >
> > > > > No, that historical GFP_DMA isn't what arm64 wants - it's the old
> > > > > crazy "legacy 16MB DMA" thing that ISA DMA used to have.
> > > > >
> > > > > But the basic issue was true then, and is true now - DMA allocations
> > > > > are fairly special, and should not be that hard to just mark as such.
> > > >
> > > > "fairly special" == "all USB transactions", so it will take a lot of
> > > > auditing here.  I think also many SPI controllers require this and maybe
> > > > I2C?  Perhaps other bus types do as well.
> > > >
> > > > So please don't make this change without some way of figuring out just
> > > > what drivers need to be fixed up, as it's going to be a lot...
> > > >
> > >
> > > Yeah, the current de facto contract of being able to DMA map anything
> > > that was allocated via the linear map makes it quite hard to enforce
> > > the use of dma_kmalloc() for this.
> > >
> > > What we might do, given the fact that only inbound non-cache coherent
> > > DMA is problematic, is dropping the kmalloc alignment to 8 like on
> > > x86, and falling back to bounce buffering when a misaligned, non-cache
> > > coherent inbound DMA mapping is created, using the SWIOTLB bounce
> > > buffering code that we already have, and is already in use on most
> > > affected systems for other reasons (i.e., DMA addressing limits)
> >
> > Ick, that's a mess.
> >
> > > This will cause some performance regressions, but in a way that seems
> > > fixable to me: taking network drivers as an example, the RX buffers
> > > that are filled using inbound DMA are typically owned by the driver
> > > itself, which could be updated to round up its allocations and DMA
> > > mappings. Block devices typically operate on quantities that are
> > > aligned sufficiently already. In other cases, we will likely notice
> > > if/when this fallback is taken on a hot path, but if we don't, at
> > > least we know a bounce buffer is being used whenever we cannot perform
> > > the DMA safely in-place.
> >
> > We can move to having an "allocator-per-bus" for memory like this to
> > allow the bus to know if this is a DMA requirement or not.
> >
> > So for all USB drivers, we would have:
> >         usb_kmalloc(size, flags);
> > and then it might even be easier to verify with static tools that the
> > USB drivers are sending only properly allocated data.  Same for SPI and
> > other busses.
> >
> 
> As I pointed out earlier in the thread, alignment/padding requirements
> for non-coherent DMA are a property of the CPU's cache hierarchy, not
> of the device. So I'm not sure I follow how a per-subsystem
> distinction would help here. In the case of USB especially, would that
> mean that block, media and networking subsystems would need to be
> aware of the USB-ness of the underlying transport?

That's what we have required today, yes.  That's only because we knew
that for some USB controllers, that was a requirement and we had no way
of passing that information back up the stack so we just made it a
requirement.

But I do agree this is messy.  It's even messier for things like USB
where it's not the USB device itself that matters, it's the USB
controller that the USB device is attached to.  And that can be _way_ up
the device hierarchy.  Attach something like a NFS mount over a PPP
network connection on a USB to serial device and ugh, where do you
begin?  :)

And is this always just an issue of the CPU cache hierarchy?  And not the
specific bridge that a device is connected to that CPU on?  Or am I
saying the same thing here?

I mean take a USB controller for example.  We could have a system where
one USB controller is on a PCI bus, while another is on a "platform"
bus.  Both of those are connected to the CPU in different ways and so
could have different DMA rules.  Do we downgrade everything in the
system for the worst connection possible?

Again, consider a USB driver allocating memory to transfer stuff, should
it somehow know the cache hierarchy that it is connected to?  Right now
we punt and do not do that at the expense of a bit of potentially
wasted memory for small allocations.


> > https://lore.kernel.org/r/230a9486fc68ea0182df46255e42a51099403642.1648032613.git.christophe.leroy@csgroup.eu
> > is an example of a SPI driver that has been there "for forever" yet
> > always got it wrong.  If we could have had some way to know "only memory
> > allocated with this function is allowed on the bus" that would have
> > fixed the issue a long time ago.
> >
> > Anyway, just an idea, it's up to you all if this is worth it or not.
> > Adding performance regressions at the expense of memory size feels like
> > a rough trade-off to go through until things are fixed up.
> >
> 
> Yeah, I hear you. But we already have distinct classes of memory,
> i.e., vmalloc vs kmalloc, where only the latter is permitted for DMA
> (which is why VMAP stack broke that SPI driver), and I'm not sure
> adding more types is going to make this tractable going forward.

Sorry, it would not be a "real" type of memory at all, just something
with a static tag on it at the compiler level (like we do for endian
stuff with sparse today).  The real memory would be allocated with the
normal kmalloc() and friends it's just that static tools can be run to
verify "yes, all of the calls to usb_submit_urb() were done with memory
allocated with usb_kmalloc(), and not from the stack"

No runtime changes needed for different types, that would be a mess.
Let's lean on static tools when we can.

thanks,

greg k-h

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Ard Biesheuvel]

On Thu, 14 Apr 2022 at 16:53, Greg Kroah-Hartman
<gregkh@linuxfoundation.org> wrote:
>
> On Thu, Apr 14, 2022 at 04:36:46PM +0200, Ard Biesheuvel wrote:
> > On Thu, 14 Apr 2022 at 16:27, Greg Kroah-Hartman
> > <gregkh@linuxfoundation.org> wrote:
> > >
> > > On Thu, Apr 14, 2022 at 03:52:53PM +0200, Ard Biesheuvel wrote:
...
> > > > What we might do, given the fact that only inbound non-cache coherent
> > > > DMA is problematic, is dropping the kmalloc alignment to 8 like on
> > > > x86, and falling back to bounce buffering when a misaligned, non-cache
> > > > coherent inbound DMA mapping is created, using the SWIOTLB bounce
> > > > buffering code that we already have, and is already in use on most
> > > > affected systems for other reasons (i.e., DMA addressing limits)
> > >
> > > Ick, that's a mess.
> > >
> > > > This will cause some performance regressions, but in a way that seems
> > > > fixable to me: taking network drivers as an example, the RX buffers
> > > > that are filled using inbound DMA are typically owned by the driver
> > > > itself, which could be updated to round up its allocations and DMA
> > > > mappings. Block devices typically operate on quantities that are
> > > > aligned sufficiently already. In other cases, we will likely notice
> > > > if/when this fallback is taken on a hot path, but if we don't, at
> > > > least we know a bounce buffer is being used whenever we cannot perform
> > > > the DMA safely in-place.
> > >
> > > We can move to having an "allocator-per-bus" for memory like this to
> > > allow the bus to know if this is a DMA requirement or not.
> > >
> > > So for all USB drivers, we would have:
> > >         usb_kmalloc(size, flags);
> > > and then it might even be easier to verify with static tools that the
> > > USB drivers are sending only properly allocated data.  Same for SPI and
> > > other busses.
> > >
> >
> > As I pointed out earlier in the thread, alignment/padding requirements
> > for non-coherent DMA are a property of the CPU's cache hierarchy, not
> > of the device. So I'm not sure I follow how a per-subsystem
> > distinction would help here. In the case of USB especially, would that
> > mean that block, media and networking subsystems would need to be
> > aware of the USB-ness of the underlying transport?
>
> That's what we have required today, yes.  That's only because we knew
> that for some USB controllers, that was a requirement and we had no way
> of passing that information back up the stack so we just made it a
> requirement.
>
> But I do agree this is messy.  It's even messier for things like USB
> where it's not the USB device itself that matters, it's the USB
> controller that the USB device is attached to.  And that can be _way_ up
> the device hierarchy.  Attach something like a NFS mount over a PPP
> network connection on a USB to serial device and ugh, where do you
> begin?  :)
>

Exactly.

> And is this always just an issue of the CPU cache hierarchy?  And not the
> specific bridge that a device is connected to that CPU on?  Or am I
> saying the same thing here?
>

Yes, this is a system property not a device property, and the driver
typically doesn't have any knowledge of this. For example, if a PCI
host bridge happens to be integrated in a non-cache coherent way, any
PCI device plugged into it becomes non-coherent, and the associated
driver needs to do the right thing. This is why we rely on the DMA
layer to take care of this.

> I mean take a USB controller for example.  We could have a system where
> one USB controller is on a PCI bus, while another is on a "platform"
> bus.  Both of those are connected to the CPU in different ways and so
> could have different DMA rules.  Do we downgrade everything in the
> system for the worst connection possible?
>

No, we currently support a mix of coherent and non-coherent just fine,
and this shouldn't change. It's just that the mere fact that
non-coherent devices might exist is increasing the memory footprint of
all kmalloc allocations.

> Again, consider a USB driver allocating memory to transfer stuff, should
> it somehow know the cache hierarchy that it is connected to?  Right now
> we punt and do not do that at the expense of a bit of potentially
> wasted memory for small allocations.
>

This whole discussion is based on the premise that this is an expense
we would prefer to avoid. Currently, every kmalloc allocation is
rounded up to 128 bytes on arm64, while x86 uses only 8.

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Ard Biesheuvel]

On Thu, 14 Apr 2022 at 16:53, Greg Kroah-Hartman
<gregkh@linuxfoundation.org> wrote:
>
> On Thu, Apr 14, 2022 at 04:36:46PM +0200, Ard Biesheuvel wrote:
> > On Thu, 14 Apr 2022 at 16:27, Greg Kroah-Hartman
> > <gregkh@linuxfoundation.org> wrote:
> > >
> > > On Thu, Apr 14, 2022 at 03:52:53PM +0200, Ard Biesheuvel wrote:
...
> > > > What we might do, given the fact that only inbound non-cache coherent
> > > > DMA is problematic, is dropping the kmalloc alignment to 8 like on
> > > > x86, and falling back to bounce buffering when a misaligned, non-cache
> > > > coherent inbound DMA mapping is created, using the SWIOTLB bounce
> > > > buffering code that we already have, and is already in use on most
> > > > affected systems for other reasons (i.e., DMA addressing limits)
> > >
> > > Ick, that's a mess.
> > >
> > > > This will cause some performance regressions, but in a way that seems
> > > > fixable to me: taking network drivers as an example, the RX buffers
> > > > that are filled using inbound DMA are typically owned by the driver
> > > > itself, which could be updated to round up its allocations and DMA
> > > > mappings. Block devices typically operate on quantities that are
> > > > aligned sufficiently already. In other cases, we will likely notice
> > > > if/when this fallback is taken on a hot path, but if we don't, at
> > > > least we know a bounce buffer is being used whenever we cannot perform
> > > > the DMA safely in-place.
> > >
> > > We can move to having an "allocator-per-bus" for memory like this to
> > > allow the bus to know if this is a DMA requirement or not.
> > >
> > > So for all USB drivers, we would have:
> > >         usb_kmalloc(size, flags);
> > > and then it might even be easier to verify with static tools that the
> > > USB drivers are sending only properly allocated data.  Same for SPI and
> > > other busses.
> > >
> >
> > As I pointed out earlier in the thread, alignment/padding requirements
> > for non-coherent DMA are a property of the CPU's cache hierarchy, not
> > of the device. So I'm not sure I follow how a per-subsystem
> > distinction would help here. In the case of USB especially, would that
> > mean that block, media and networking subsystems would need to be
> > aware of the USB-ness of the underlying transport?
>
> That's what we have required today, yes.  That's only because we knew
> that for some USB controllers, that was a requirement and we had no way
> of passing that information back up the stack so we just made it a
> requirement.
>
> But I do agree this is messy.  It's even messier for things like USB
> where it's not the USB device itself that matters, it's the USB
> controller that the USB device is attached to.  And that can be _way_ up
> the device hierarchy.  Attach something like a NFS mount over a PPP
> network connection on a USB to serial device and ugh, where do you
> begin?  :)
>

Exactly.

> And is this always just an issue of the CPU cache hierarchy?  And not the
> specific bridge that a device is connected to that CPU on?  Or am I
> saying the same thing here?
>

Yes, this is a system property not a device property, and the driver
typically doesn't have any knowledge of this. For example, if a PCI
host bridge happens to be integrated in a non-cache coherent way, any
PCI device plugged into it becomes non-coherent, and the associated
driver needs to do the right thing. This is why we rely on the DMA
layer to take care of this.

> I mean take a USB controller for example.  We could have a system where
> one USB controller is on a PCI bus, while another is on a "platform"
> bus.  Both of those are connected to the CPU in different ways and so
> could have different DMA rules.  Do we downgrade everything in the
> system for the worst connection possible?
>

No, we currently support a mix of coherent and non-coherent just fine,
and this shouldn't change. It's just that the mere fact that
non-coherent devices might exist is increasing the memory footprint of
all kmalloc allocations.

> Again, consider a USB driver allocating memory to transfer stuff, should
> it somehow know the cache hierarchy that it is connected to?  Right now
> we punt and do not do that at the expense of a bit of potentially
> wasted memory for small allocations.
>

This whole discussion is based on the premise that this is an expense
we would prefer to avoid. Currently, every kmalloc allocation is
rounded up to 128 bytes on arm64, while x86 uses only 8.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Ard Biesheuvel]

On Thu, 14 Apr 2022 at 17:01, Ard Biesheuvel <ardb@kernel.org> wrote:
>
> On Thu, 14 Apr 2022 at 16:53, Greg Kroah-Hartman
> <gregkh@linuxfoundation.org> wrote:
> >
> > On Thu, Apr 14, 2022 at 04:36:46PM +0200, Ard Biesheuvel wrote:
> > > On Thu, 14 Apr 2022 at 16:27, Greg Kroah-Hartman
> > > <gregkh@linuxfoundation.org> wrote:
> > > >
> > > > On Thu, Apr 14, 2022 at 03:52:53PM +0200, Ard Biesheuvel wrote:
> ...
> > > > > What we might do, given the fact that only inbound non-cache coherent
> > > > > DMA is problematic, is dropping the kmalloc alignment to 8 like on
> > > > > x86, and falling back to bounce buffering when a misaligned, non-cache
> > > > > coherent inbound DMA mapping is created, using the SWIOTLB bounce
> > > > > buffering code that we already have, and is already in use on most
> > > > > affected systems for other reasons (i.e., DMA addressing limits)
> > > >
> > > > Ick, that's a mess.
> > > >
> > > > > This will cause some performance regressions, but in a way that seems
> > > > > fixable to me: taking network drivers as an example, the RX buffers
> > > > > that are filled using inbound DMA are typically owned by the driver
> > > > > itself, which could be updated to round up its allocations and DMA
> > > > > mappings. Block devices typically operate on quantities that are
> > > > > aligned sufficiently already. In other cases, we will likely notice
> > > > > if/when this fallback is taken on a hot path, but if we don't, at
> > > > > least we know a bounce buffer is being used whenever we cannot perform
> > > > > the DMA safely in-place.
> > > >
> > > > We can move to having an "allocator-per-bus" for memory like this to
> > > > allow the bus to know if this is a DMA requirement or not.
> > > >
> > > > So for all USB drivers, we would have:
> > > >         usb_kmalloc(size, flags);
> > > > and then it might even be easier to verify with static tools that the
> > > > USB drivers are sending only properly allocated data.  Same for SPI and
> > > > other busses.
> > > >
> > >
> > > As I pointed out earlier in the thread, alignment/padding requirements
> > > for non-coherent DMA are a property of the CPU's cache hierarchy, not
> > > of the device. So I'm not sure I follow how a per-subsystem
> > > distinction would help here. In the case of USB especially, would that
> > > mean that block, media and networking subsystems would need to be
> > > aware of the USB-ness of the underlying transport?
> >
> > That's what we have required today, yes.  That's only because we knew
> > that for some USB controllers, that was a requirement and we had no way
> > of passing that information back up the stack so we just made it a
> > requirement.
> >
> > But I do agree this is messy.  It's even messier for things like USB
> > where it's not the USB device itself that matters, it's the USB
> > controller that the USB device is attached to.  And that can be _way_ up
> > the device hierarchy.  Attach something like a NFS mount over a PPP
> > network connection on a USB to serial device and ugh, where do you
> > begin?  :)
> >
>
> Exactly.
>
> > And is this always just an issue of the CPU cache hierarchy?  And not the
> > specific bridge that a device is connected to that CPU on?  Or am I
> > saying the same thing here?
> >
>
> Yes, this is a system property not a device property, and the driver
> typically doesn't have any knowledge of this. For example, if a PCI
> host bridge happens to be integrated in a non-cache coherent way, any
> PCI device plugged into it becomes non-coherent, and the associated
> driver needs to do the right thing. This is why we rely on the DMA
> layer to take care of this.
>
> > I mean take a USB controller for example.  We could have a system where
> > one USB controller is on a PCI bus, while another is on a "platform"
> > bus.  Both of those are connected to the CPU in different ways and so
> > could have different DMA rules.  Do we downgrade everything in the
> > system for the worst connection possible?
> >
>
> No, we currently support a mix of coherent and non-coherent just fine,
> and this shouldn't change. It's just that the mere fact that
> non-coherent devices might exist is increasing the memory footprint of
> all kmalloc allocations.
>
> > Again, consider a USB driver allocating memory to transfer stuff, should
> > it somehow know the cache hierarchy that it is connected to?  Right now
> > we punt and do not do that at the expense of a bit of potentially
> > wasted memory for small allocations.
> >
>
> This whole discussion is based on the premise that this is an expense
> we would prefer to avoid. Currently, every kmalloc allocation is
> rounded up to 128 bytes on arm64, while x86 uses only 8.

I guess I didn't answer that last question. Yes, I guess dma_kmalloc()
should be used in such cases. Combined with my bounce buffering hack,
the penalty for using plain kmalloc() instead would be a potential
performance hit when used for inbound DMA, instead of data corruption
(if we'd reduce the kmalloc() alignment when introducing
dma_kmalloc())

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Ard Biesheuvel]

On Thu, 14 Apr 2022 at 17:01, Ard Biesheuvel <ardb@kernel.org> wrote:
>
> On Thu, 14 Apr 2022 at 16:53, Greg Kroah-Hartman
> <gregkh@linuxfoundation.org> wrote:
> >
> > On Thu, Apr 14, 2022 at 04:36:46PM +0200, Ard Biesheuvel wrote:
> > > On Thu, 14 Apr 2022 at 16:27, Greg Kroah-Hartman
> > > <gregkh@linuxfoundation.org> wrote:
> > > >
> > > > On Thu, Apr 14, 2022 at 03:52:53PM +0200, Ard Biesheuvel wrote:
> ...
> > > > > What we might do, given the fact that only inbound non-cache coherent
> > > > > DMA is problematic, is dropping the kmalloc alignment to 8 like on
> > > > > x86, and falling back to bounce buffering when a misaligned, non-cache
> > > > > coherent inbound DMA mapping is created, using the SWIOTLB bounce
> > > > > buffering code that we already have, and is already in use on most
> > > > > affected systems for other reasons (i.e., DMA addressing limits)
> > > >
> > > > Ick, that's a mess.
> > > >
> > > > > This will cause some performance regressions, but in a way that seems
> > > > > fixable to me: taking network drivers as an example, the RX buffers
> > > > > that are filled using inbound DMA are typically owned by the driver
> > > > > itself, which could be updated to round up its allocations and DMA
> > > > > mappings. Block devices typically operate on quantities that are
> > > > > aligned sufficiently already. In other cases, we will likely notice
> > > > > if/when this fallback is taken on a hot path, but if we don't, at
> > > > > least we know a bounce buffer is being used whenever we cannot perform
> > > > > the DMA safely in-place.
> > > >
> > > > We can move to having an "allocator-per-bus" for memory like this to
> > > > allow the bus to know if this is a DMA requirement or not.
> > > >
> > > > So for all USB drivers, we would have:
> > > >         usb_kmalloc(size, flags);
> > > > and then it might even be easier to verify with static tools that the
> > > > USB drivers are sending only properly allocated data.  Same for SPI and
> > > > other busses.
> > > >
> > >
> > > As I pointed out earlier in the thread, alignment/padding requirements
> > > for non-coherent DMA are a property of the CPU's cache hierarchy, not
> > > of the device. So I'm not sure I follow how a per-subsystem
> > > distinction would help here. In the case of USB especially, would that
> > > mean that block, media and networking subsystems would need to be
> > > aware of the USB-ness of the underlying transport?
> >
> > That's what we have required today, yes.  That's only because we knew
> > that for some USB controllers, that was a requirement and we had no way
> > of passing that information back up the stack so we just made it a
> > requirement.
> >
> > But I do agree this is messy.  It's even messier for things like USB
> > where it's not the USB device itself that matters, it's the USB
> > controller that the USB device is attached to.  And that can be _way_ up
> > the device hierarchy.  Attach something like a NFS mount over a PPP
> > network connection on a USB to serial device and ugh, where do you
> > begin?  :)
> >
>
> Exactly.
>
> > And is this always just an issue of the CPU cache hierarchy?  And not the
> > specific bridge that a device is connected to that CPU on?  Or am I
> > saying the same thing here?
> >
>
> Yes, this is a system property not a device property, and the driver
> typically doesn't have any knowledge of this. For example, if a PCI
> host bridge happens to be integrated in a non-cache coherent way, any
> PCI device plugged into it becomes non-coherent, and the associated
> driver needs to do the right thing. This is why we rely on the DMA
> layer to take care of this.
>
> > I mean take a USB controller for example.  We could have a system where
> > one USB controller is on a PCI bus, while another is on a "platform"
> > bus.  Both of those are connected to the CPU in different ways and so
> > could have different DMA rules.  Do we downgrade everything in the
> > system for the worst connection possible?
> >
>
> No, we currently support a mix of coherent and non-coherent just fine,
> and this shouldn't change. It's just that the mere fact that
> non-coherent devices might exist is increasing the memory footprint of
> all kmalloc allocations.
>
> > Again, consider a USB driver allocating memory to transfer stuff, should
> > it somehow know the cache hierarchy that it is connected to?  Right now
> > we punt and do not do that at the expense of a bit of potentially
> > wasted memory for small allocations.
> >
>
> This whole discussion is based on the premise that this is an expense
> we would prefer to avoid. Currently, every kmalloc allocation is
> rounded up to 128 bytes on arm64, while x86 uses only 8.

I guess I didn't answer that last question. Yes, I guess dma_kmalloc()
should be used in such cases. Combined with my bounce buffering hack,
the penalty for using plain kmalloc() instead would be a potential
performance hit when used for inbound DMA, instead of data corruption
(if we'd reduce the kmalloc() alignment when introducing
dma_kmalloc())

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Wed, Apr 13, 2022 at 09:53:24AM -1000, Linus Torvalds wrote:
> On Tue, Apr 12, 2022 at 10:47 PM Catalin Marinas
> <catalin.marinas@arm.com> wrote:
> > I agree. There is also an implicit expectation that the DMA API works on
> > kmalloc'ed buffers and that's what ARCH_DMA_MINALIGN is for (and the
> > dynamic arch_kmalloc_minalign() in this series). But the key point is
> > that the driver doesn't need to know the CPU cache topology, coherency,
> > the DMA API and kmalloc() take care of these.
> 
> Honestly, I think it would probably be worth discussing the "kmalloc
> DMA alignment" issues.
> 
> 99.9% of kmalloc users don't want to do DMA.
> 
> And there's actually a fair amount of small kmalloc for random stuff.
> Right now on my laptop, I have
> 
>     kmalloc-8          16907  18432      8  512    1 : ...
> 
> according to slabinfo, so almost 17 _thousand_ allocations of 8 bytes.
> 
> It's all kinds of sad if those allocations need to be 64 bytes in size
> just because of some silly DMA alignment issue, when none of them want
> it.

It's a lot worse, ARCH_KMALLOC_MINALIGN is currently 128 bytes on arm64.
I want to at least get it down to 64 with this series while preserving
the current kmalloc() semantics.

If we know the SoC is fully coherent (a bit tricky with late probed
devices), we could get the alignment down to 8. In the mobile space,
unfortunately, most DMA is non-coherent.

I think it's worth investigating the __dma annotations that Greg
suggested, though I have a suspicion it either is too difficult to track
or we just end up with this annotation everywhere. There are cases where
the memory is allocated outside the driver that knows the DMA needs,
though I guess these are either full page allocations or
kmem_cache_alloc() (e.g. page cache pages, skb).

There's also Ard's suggestion to bounce the (inbound DMA) buffer if not
aligned. That's doable but dma_map_single(), for example, only gets the
size of some random structure/buffer. If the size is below
ARCH_DMA_MINALIGN (or cache_line_size()), the DMA API implementation
would have to retrieve the slab cache, check the real allocation size
and then bounce if necessary.

Irrespective of which option we go for, I think at least part of this
series decoupling ARCH_KMALLOC_MINALIGN from ARCH_DMA_MINALIGN is still
needed since currently the minalign is used in some compile time
attributes. Even getting the kmalloc() size down to 64 is a significant
improvement over 128.

Subsequently I'd attempt Ard's bouncing idea as a quick workaround and
assess the bouncing overhead on some real platforms. This may be needed
before we track down all places to use dma_kmalloc().

I need to think some more on Greg's __dma annotation, as I said the
allocation may be decoupled from the driver in some cases.

-- 
Catalin

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Wed, Apr 13, 2022 at 09:53:24AM -1000, Linus Torvalds wrote:
> On Tue, Apr 12, 2022 at 10:47 PM Catalin Marinas
> <catalin.marinas@arm.com> wrote:
> > I agree. There is also an implicit expectation that the DMA API works on
> > kmalloc'ed buffers and that's what ARCH_DMA_MINALIGN is for (and the
> > dynamic arch_kmalloc_minalign() in this series). But the key point is
> > that the driver doesn't need to know the CPU cache topology, coherency,
> > the DMA API and kmalloc() take care of these.
> 
> Honestly, I think it would probably be worth discussing the "kmalloc
> DMA alignment" issues.
> 
> 99.9% of kmalloc users don't want to do DMA.
> 
> And there's actually a fair amount of small kmalloc for random stuff.
> Right now on my laptop, I have
> 
>     kmalloc-8          16907  18432      8  512    1 : ...
> 
> according to slabinfo, so almost 17 _thousand_ allocations of 8 bytes.
> 
> It's all kinds of sad if those allocations need to be 64 bytes in size
> just because of some silly DMA alignment issue, when none of them want
> it.

It's a lot worse, ARCH_KMALLOC_MINALIGN is currently 128 bytes on arm64.
I want to at least get it down to 64 with this series while preserving
the current kmalloc() semantics.

If we know the SoC is fully coherent (a bit tricky with late probed
devices), we could get the alignment down to 8. In the mobile space,
unfortunately, most DMA is non-coherent.

I think it's worth investigating the __dma annotations that Greg
suggested, though I have a suspicion it either is too difficult to track
or we just end up with this annotation everywhere. There are cases where
the memory is allocated outside the driver that knows the DMA needs,
though I guess these are either full page allocations or
kmem_cache_alloc() (e.g. page cache pages, skb).

There's also Ard's suggestion to bounce the (inbound DMA) buffer if not
aligned. That's doable but dma_map_single(), for example, only gets the
size of some random structure/buffer. If the size is below
ARCH_DMA_MINALIGN (or cache_line_size()), the DMA API implementation
would have to retrieve the slab cache, check the real allocation size
and then bounce if necessary.

Irrespective of which option we go for, I think at least part of this
series decoupling ARCH_KMALLOC_MINALIGN from ARCH_DMA_MINALIGN is still
needed since currently the minalign is used in some compile time
attributes. Even getting the kmalloc() size down to 64 is a significant
improvement over 128.

Subsequently I'd attempt Ard's bouncing idea as a quick workaround and
assess the bouncing overhead on some real platforms. This may be needed
before we track down all places to use dma_kmalloc().

I need to think some more on Greg's __dma annotation, as I said the
allocation may be decoupled from the driver in some cases.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Linus Torvalds]

On Thu, Apr 14, 2022 at 12:49 PM Catalin Marinas
<catalin.marinas@arm.com> wrote:
>
> It's a lot worse, ARCH_KMALLOC_MINALIGN is currently 128 bytes on arm64.
> I want to at least get it down to 64 with this series while preserving
> the current kmalloc() semantics.

So here's a thought - maybe we could do the reverse of GFP_DMA, and
add a flag to the places that want small allocations and know they
don't need DMA?

Because even 64 bytes is _huge_ for those small allocations. And 128
bytes is just insane.

Maybe we don't have a ton of them, but I just checked my desktop, and
if my laptop had 17k 8-byte allocation, on my desktop it's currently
sitting at 43k of them. And I've got a lot of 16- and 32-byte ones
too:

  kmalloc-32         51529  51712     32  128    1 :
  kmalloc-16         45056  45056     16  256    1 :
  kmalloc-8          43008  43008      8  512    1 :

Don't ask me what they are. It's probably fairly easy to find out, and
it's probably something silly like sysfs private pointer data or
whatever.

If I did my math right, with a 128-byte minimum allocation, that is
16MB of wasted memory.

Yeah, yeah, I've got 64GB or RAM in this thing, and there are things
that take a lot more memory than the above (mem_map etc), and 64MB is
peanuts at just 0.1% of RAM.

Maybe nobody cares. But I really feel bad about that kind of egregious
waste. The mem_map[] array may be big, it may use 1.5% of the memory I
have, but at least it *does* something.

And it could be that if I have 150k of those smallish allocations, a
server with lots of active users might have millions. Not having
looked at where they come from, maybe that isn't the case, but it
*might* be.

Maybe adding something like a

        static int warn_every_1k = 0;
        WARN_ON(size < 32 && (1023 & ++warn_every_1k));

to kmalloc() would give us a statistical view of "lots of these small
allocations" thing, and we could add GFP_NODMA to them. There probably
aren't that many places that have those small allocations, and it's
certainly safer to annotate "this is not for DMA" than have the
requirement that all DMA allocations must be marked.

Then just teach kmalloc() something like

        align = (gfp_flags & __GFP_NODMA) ? 8 : 128;

on arm.

Hmm?

              Linus

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Linus Torvalds]

On Thu, Apr 14, 2022 at 12:49 PM Catalin Marinas
<catalin.marinas@arm.com> wrote:
>
> It's a lot worse, ARCH_KMALLOC_MINALIGN is currently 128 bytes on arm64.
> I want to at least get it down to 64 with this series while preserving
> the current kmalloc() semantics.

So here's a thought - maybe we could do the reverse of GFP_DMA, and
add a flag to the places that want small allocations and know they
don't need DMA?

Because even 64 bytes is _huge_ for those small allocations. And 128
bytes is just insane.

Maybe we don't have a ton of them, but I just checked my desktop, and
if my laptop had 17k 8-byte allocation, on my desktop it's currently
sitting at 43k of them. And I've got a lot of 16- and 32-byte ones
too:

  kmalloc-32         51529  51712     32  128    1 :
  kmalloc-16         45056  45056     16  256    1 :
  kmalloc-8          43008  43008      8  512    1 :

Don't ask me what they are. It's probably fairly easy to find out, and
it's probably something silly like sysfs private pointer data or
whatever.

If I did my math right, with a 128-byte minimum allocation, that is
16MB of wasted memory.

Yeah, yeah, I've got 64GB or RAM in this thing, and there are things
that take a lot more memory than the above (mem_map etc), and 64MB is
peanuts at just 0.1% of RAM.

Maybe nobody cares. But I really feel bad about that kind of egregious
waste. The mem_map[] array may be big, it may use 1.5% of the memory I
have, but at least it *does* something.

And it could be that if I have 150k of those smallish allocations, a
server with lots of active users might have millions. Not having
looked at where they come from, maybe that isn't the case, but it
*might* be.

Maybe adding something like a

        static int warn_every_1k = 0;
        WARN_ON(size < 32 && (1023 & ++warn_every_1k));

to kmalloc() would give us a statistical view of "lots of these small
allocations" thing, and we could add GFP_NODMA to them. There probably
aren't that many places that have those small allocations, and it's
certainly safer to annotate "this is not for DMA" than have the
requirement that all DMA allocations must be marked.

Then just teach kmalloc() something like

        align = (gfp_flags & __GFP_NODMA) ? 8 : 128;

on arm.

Hmm?

              Linus

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Ard Biesheuvel]

On Fri, 15 Apr 2022 at 00:26, Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> On Thu, Apr 14, 2022 at 12:49 PM Catalin Marinas
> <catalin.marinas@arm.com> wrote:
> >
> > It's a lot worse, ARCH_KMALLOC_MINALIGN is currently 128 bytes on arm64.
> > I want to at least get it down to 64 with this series while preserving
> > the current kmalloc() semantics.
>
> So here's a thought - maybe we could do the reverse of GFP_DMA, and
> add a flag to the places that want small allocations and know they
> don't need DMA?
>
> Because even 64 bytes is _huge_ for those small allocations. And 128
> bytes is just insane.
>
> Maybe we don't have a ton of them, but I just checked my desktop, and
> if my laptop had 17k 8-byte allocation, on my desktop it's currently
> sitting at 43k of them. And I've got a lot of 16- and 32-byte ones
> too:
>
>   kmalloc-32         51529  51712     32  128    1 :
>   kmalloc-16         45056  45056     16  256    1 :
>   kmalloc-8          43008  43008      8  512    1 :
>
> Don't ask me what they are. It's probably fairly easy to find out, and
> it's probably something silly like sysfs private pointer data or
> whatever.
>
> If I did my math right, with a 128-byte minimum allocation, that is
> 16MB of wasted memory.
>
> Yeah, yeah, I've got 64GB or RAM in this thing, and there are things
> that take a lot more memory than the above (mem_map etc), and 64MB is
> peanuts at just 0.1% of RAM.
>

Actually, I think the impact on D-cache efficiency is a much bigger
concern, although I don't have any data to back that up. On arm64,
every 8 byte allocation takes up an entire cacheline, pushing out
other data that we could meaningfully keep there. And the doubling to
128 that is unnecessary on most arm64 systems probably has an
additional negative effect, as it means those allocations are heavily
skewed to use even indexes into the cache sets, causing extra
contention.

> Maybe nobody cares. But I really feel bad about that kind of egregious
> waste. The mem_map[] array may be big, it may use 1.5% of the memory I
> have, but at least it *does* something.
>
> And it could be that if I have 150k of those smallish allocations, a
> server with lots of active users might have millions. Not having
> looked at where they come from, maybe that isn't the case, but it
> *might* be.
>
> Maybe adding something like a
>
>         static int warn_every_1k = 0;
>         WARN_ON(size < 32 && (1023 & ++warn_every_1k));
>
> to kmalloc() would give us a statistical view of "lots of these small
> allocations" thing, and we could add GFP_NODMA to them. There probably
> aren't that many places that have those small allocations, and it's
> certainly safer to annotate "this is not for DMA" than have the
> requirement that all DMA allocations must be marked.
>
> Then just teach kmalloc() something like
>
>         align = (gfp_flags & __GFP_NODMA) ? 8 : 128;
>
> on arm.
>

Sounds like a lot of churn as well, tbh. But perhaps there are a few
hot spots that we can fix that would alleviate the bulk of the issue,
who knows?

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Ard Biesheuvel]

On Fri, 15 Apr 2022 at 00:26, Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> On Thu, Apr 14, 2022 at 12:49 PM Catalin Marinas
> <catalin.marinas@arm.com> wrote:
> >
> > It's a lot worse, ARCH_KMALLOC_MINALIGN is currently 128 bytes on arm64.
> > I want to at least get it down to 64 with this series while preserving
> > the current kmalloc() semantics.
>
> So here's a thought - maybe we could do the reverse of GFP_DMA, and
> add a flag to the places that want small allocations and know they
> don't need DMA?
>
> Because even 64 bytes is _huge_ for those small allocations. And 128
> bytes is just insane.
>
> Maybe we don't have a ton of them, but I just checked my desktop, and
> if my laptop had 17k 8-byte allocation, on my desktop it's currently
> sitting at 43k of them. And I've got a lot of 16- and 32-byte ones
> too:
>
>   kmalloc-32         51529  51712     32  128    1 :
>   kmalloc-16         45056  45056     16  256    1 :
>   kmalloc-8          43008  43008      8  512    1 :
>
> Don't ask me what they are. It's probably fairly easy to find out, and
> it's probably something silly like sysfs private pointer data or
> whatever.
>
> If I did my math right, with a 128-byte minimum allocation, that is
> 16MB of wasted memory.
>
> Yeah, yeah, I've got 64GB or RAM in this thing, and there are things
> that take a lot more memory than the above (mem_map etc), and 64MB is
> peanuts at just 0.1% of RAM.
>

Actually, I think the impact on D-cache efficiency is a much bigger
concern, although I don't have any data to back that up. On arm64,
every 8 byte allocation takes up an entire cacheline, pushing out
other data that we could meaningfully keep there. And the doubling to
128 that is unnecessary on most arm64 systems probably has an
additional negative effect, as it means those allocations are heavily
skewed to use even indexes into the cache sets, causing extra
contention.

> Maybe nobody cares. But I really feel bad about that kind of egregious
> waste. The mem_map[] array may be big, it may use 1.5% of the memory I
> have, but at least it *does* something.
>
> And it could be that if I have 150k of those smallish allocations, a
> server with lots of active users might have millions. Not having
> looked at where they come from, maybe that isn't the case, but it
> *might* be.
>
> Maybe adding something like a
>
>         static int warn_every_1k = 0;
>         WARN_ON(size < 32 && (1023 & ++warn_every_1k));
>
> to kmalloc() would give us a statistical view of "lots of these small
> allocations" thing, and we could add GFP_NODMA to them. There probably
> aren't that many places that have those small allocations, and it's
> certainly safer to annotate "this is not for DMA" than have the
> requirement that all DMA allocations must be marked.
>
> Then just teach kmalloc() something like
>
>         align = (gfp_flags & __GFP_NODMA) ? 8 : 128;
>
> on arm.
>

Sounds like a lot of churn as well, tbh. But perhaps there are a few
hot spots that we can fix that would alleviate the bulk of the issue,
who knows?

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Herbert Xu]

On Wed, Apr 13, 2022 at 09:47:29AM +0100, Catalin Marinas wrote:
>
> With my series, there is no change to the value of CRYPTO_MINALIGN for
> arm64 or any other architecture, so point 3 is unaffected. The series
> does change the kmalloc() alignment and that may be smaller than
> CRYPTO_MINALIGN but neither of points 1 or 2 above are affected since
> (a) we still have a sufficiently large ARCH_KMALLOC_MINALIGN of 64 and
> (b) the kmalloc'ed buffers are safe for non-coherent DMA.
> 
> Herbert, Ard, if I missed anything please let me know but based on my
> understanding, this series is safe for the crypto code.

Sorry, but you can't change CRYPTO_MINALIGN to a value greater
than the minimum alignment returned by kmalloc.  That simply
doesn't work.  There is no magic in the Crypto API that makes
this work.

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Herbert Xu]

On Wed, Apr 13, 2022 at 09:47:29AM +0100, Catalin Marinas wrote:
>
> With my series, there is no change to the value of CRYPTO_MINALIGN for
> arm64 or any other architecture, so point 3 is unaffected. The series
> does change the kmalloc() alignment and that may be smaller than
> CRYPTO_MINALIGN but neither of points 1 or 2 above are affected since
> (a) we still have a sufficiently large ARCH_KMALLOC_MINALIGN of 64 and
> (b) the kmalloc'ed buffers are safe for non-coherent DMA.
> 
> Herbert, Ard, if I missed anything please let me know but based on my
> understanding, this series is safe for the crypto code.

Sorry, but you can't change CRYPTO_MINALIGN to a value greater
than the minimum alignment returned by kmalloc.  That simply
doesn't work.  There is no magic in the Crypto API that makes
this work.

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Ard Biesheuvel]

On Fri, 15 Apr 2022 at 08:51, Herbert Xu <herbert@gondor.apana.org.au> wrote:
>
> On Wed, Apr 13, 2022 at 09:47:29AM +0100, Catalin Marinas wrote:
> >
> > With my series, there is no change to the value of CRYPTO_MINALIGN for
> > arm64 or any other architecture, so point 3 is unaffected. The series
> > does change the kmalloc() alignment and that may be smaller than
> > CRYPTO_MINALIGN but neither of points 1 or 2 above are affected since
> > (a) we still have a sufficiently large ARCH_KMALLOC_MINALIGN of 64 and
> > (b) the kmalloc'ed buffers are safe for non-coherent DMA.
> >
> > Herbert, Ard, if I missed anything please let me know but based on my
> > understanding, this series is safe for the crypto code.
>
> Sorry, but you can't change CRYPTO_MINALIGN to a value greater
> than the minimum alignment returned by kmalloc.  That simply
> doesn't work.  There is no magic in the Crypto API that makes
> this work.
>

I'm not sure I understand what would go wrong if that assumption no
longer holds, but if CRYPTO_MINALIGN needs to remain equal to
ARCH_KMALLOC_MINALIGN, let's at least decouple it from
ARCH_DMA_MINALIGN, as I do in my series. As I pointed out before,
ARCH_DMA_MINALIGN has nothing to do with DMA addressing capabilities
of individual masters, it is simply a worst case cacheline size that
needs to be taken into account to avoid corruption when doing cache
invalidation for non-cache coherent inbound DMA.

I'll rename the flag I proposed from CRYPTO_ALG_NEED_DMA_ALIGNMENT to
CRYPTO_ALG_NEED_DMA_PADDING to make this clearer, and given that only
a few drivers should be relying on DMA to write into request/TFM
context structures, hopefully we can fix those to stop doing that, and
get rid of this flag again entirely.

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Ard Biesheuvel]

On Fri, 15 Apr 2022 at 08:51, Herbert Xu <herbert@gondor.apana.org.au> wrote:
>
> On Wed, Apr 13, 2022 at 09:47:29AM +0100, Catalin Marinas wrote:
> >
> > With my series, there is no change to the value of CRYPTO_MINALIGN for
> > arm64 or any other architecture, so point 3 is unaffected. The series
> > does change the kmalloc() alignment and that may be smaller than
> > CRYPTO_MINALIGN but neither of points 1 or 2 above are affected since
> > (a) we still have a sufficiently large ARCH_KMALLOC_MINALIGN of 64 and
> > (b) the kmalloc'ed buffers are safe for non-coherent DMA.
> >
> > Herbert, Ard, if I missed anything please let me know but based on my
> > understanding, this series is safe for the crypto code.
>
> Sorry, but you can't change CRYPTO_MINALIGN to a value greater
> than the minimum alignment returned by kmalloc.  That simply
> doesn't work.  There is no magic in the Crypto API that makes
> this work.
>

I'm not sure I understand what would go wrong if that assumption no
longer holds, but if CRYPTO_MINALIGN needs to remain equal to
ARCH_KMALLOC_MINALIGN, let's at least decouple it from
ARCH_DMA_MINALIGN, as I do in my series. As I pointed out before,
ARCH_DMA_MINALIGN has nothing to do with DMA addressing capabilities
of individual masters, it is simply a worst case cacheline size that
needs to be taken into account to avoid corruption when doing cache
invalidation for non-cache coherent inbound DMA.

I'll rename the flag I proposed from CRYPTO_ALG_NEED_DMA_ALIGNMENT to
CRYPTO_ALG_NEED_DMA_PADDING to make this clearer, and given that only
a few drivers should be relying on DMA to write into request/TFM
context structures, hopefully we can fix those to stop doing that, and
get rid of this flag again entirely.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Herbert Xu]

On Fri, Apr 15, 2022 at 09:49:12AM +0200, Ard Biesheuvel wrote:
>
> I'm not sure I understand what would go wrong if that assumption no
> longer holds.

It's very simple, we don't do anything to the pointer returned
by kmalloc before returning it as a tfm or other object with
an alignment of CRYPTO_MINALIGN.  IOW if kmalloc starts returning
pointers that are not aligned to CRYPTO_MINALIGN then we'd be
lying to the compiler.

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Herbert Xu]

On Fri, Apr 15, 2022 at 09:49:12AM +0200, Ard Biesheuvel wrote:
>
> I'm not sure I understand what would go wrong if that assumption no
> longer holds.

It's very simple, we don't do anything to the pointer returned
by kmalloc before returning it as a tfm or other object with
an alignment of CRYPTO_MINALIGN.  IOW if kmalloc starts returning
pointers that are not aligned to CRYPTO_MINALIGN then we'd be
lying to the compiler.

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Ard Biesheuvel]

On Fri, 15 Apr 2022 at 09:52, Herbert Xu <herbert@gondor.apana.org.au> wrote:
>
> On Fri, Apr 15, 2022 at 09:49:12AM +0200, Ard Biesheuvel wrote:
> >
> > I'm not sure I understand what would go wrong if that assumption no
> > longer holds.
>
> It's very simple, we don't do anything to the pointer returned
> by kmalloc before returning it as a tfm or other object with
> an alignment of CRYPTO_MINALIGN.  IOW if kmalloc starts returning
> pointers that are not aligned to CRYPTO_MINALIGN then we'd be
> lying to the compiler.
>

I guess that should be fixable. GIven that this is about padding
rather than alignment, we could do something like

struct crypto_request {
  union {
      struct {
        ... fields ...
      };
      u8 __padding[ARCH_DMA_MINALIGN];
   };
    void __ctx[]  __align(CRYPTO_MINALIGN);
};

And then hopefully, we can get rid of the padding once we fix drivers
doing non-cache coherent inbound DMA into those structures.

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Ard Biesheuvel]

On Fri, 15 Apr 2022 at 09:52, Herbert Xu <herbert@gondor.apana.org.au> wrote:
>
> On Fri, Apr 15, 2022 at 09:49:12AM +0200, Ard Biesheuvel wrote:
> >
> > I'm not sure I understand what would go wrong if that assumption no
> > longer holds.
>
> It's very simple, we don't do anything to the pointer returned
> by kmalloc before returning it as a tfm or other object with
> an alignment of CRYPTO_MINALIGN.  IOW if kmalloc starts returning
> pointers that are not aligned to CRYPTO_MINALIGN then we'd be
> lying to the compiler.
>

I guess that should be fixable. GIven that this is about padding
rather than alignment, we could do something like

struct crypto_request {
  union {
      struct {
        ... fields ...
      };
      u8 __padding[ARCH_DMA_MINALIGN];
   };
    void __ctx[]  __align(CRYPTO_MINALIGN);
};

And then hopefully, we can get rid of the padding once we fix drivers
doing non-cache coherent inbound DMA into those structures.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Herbert Xu]

On Fri, Apr 15, 2022 at 10:05:21AM +0200, Ard Biesheuvel wrote:
>
> I guess that should be fixable. GIven that this is about padding
> rather than alignment, we could do something like
> 
> struct crypto_request {
>   union {
>       struct {
>         ... fields ...
>       };
>       u8 __padding[ARCH_DMA_MINALIGN];
>    };
>     void __ctx[]  __align(CRYPTO_MINALIGN);
> };
> 
> And then hopefully, we can get rid of the padding once we fix drivers
> doing non-cache coherent inbound DMA into those structures.

Sorry, I don't think this works.  kmalloc can still return something
that's not ARCH_DMA_MINALIGN-aligned, and therefore __ctx won't be
aligned correctly.

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Herbert Xu]

On Fri, Apr 15, 2022 at 10:05:21AM +0200, Ard Biesheuvel wrote:
>
> I guess that should be fixable. GIven that this is about padding
> rather than alignment, we could do something like
> 
> struct crypto_request {
>   union {
>       struct {
>         ... fields ...
>       };
>       u8 __padding[ARCH_DMA_MINALIGN];
>    };
>     void __ctx[]  __align(CRYPTO_MINALIGN);
> };
> 
> And then hopefully, we can get rid of the padding once we fix drivers
> doing non-cache coherent inbound DMA into those structures.

Sorry, I don't think this works.  kmalloc can still return something
that's not ARCH_DMA_MINALIGN-aligned, and therefore __ctx won't be
aligned correctly.

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Fri, Apr 15, 2022 at 03:51:54PM +0800, Herbert Xu wrote:
> On Fri, Apr 15, 2022 at 09:49:12AM +0200, Ard Biesheuvel wrote:
> > I'm not sure I understand what would go wrong if that assumption no
> > longer holds.
> 
> It's very simple, we don't do anything to the pointer returned
> by kmalloc before returning it as a tfm or other object with
> an alignment of CRYPTO_MINALIGN.  IOW if kmalloc starts returning
> pointers that are not aligned to CRYPTO_MINALIGN then we'd be
> lying to the compiler.

I agree that it would be lying to the compiler, but I don't think this
matters for arm64 where the CPU can do unaligned accesses just fine. We
don't even end up with unaligned accesses here. Let's say we have:

struct x {
	...
} __attribute__ ((__aligned__ (128)));

and the kmalloc(sizeof(struct x)) returns a 64-byte aligned pointer. The
compiler-generated code won't have any problem on arm64 accessing the
struct x members. As I said a few times, it's not affecting any other
architecture and not breaking arm64 either.

Anyway, let's agree to disagree. I'll look into keeping CRYPTO_MINALIGN
as ARCH_KMALLOC_MINALIGN and introduce a CRYPTO_DMA_MINALIGN (or just
use ARCH_DMA_MINALIGN directly) together with something like Linus'
dma_kmalloc() in places where an object aligned to ARCH_DMA_MINALIGN is
needed in the crypto code.

-- 
Catalin

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Fri, Apr 15, 2022 at 03:51:54PM +0800, Herbert Xu wrote:
> On Fri, Apr 15, 2022 at 09:49:12AM +0200, Ard Biesheuvel wrote:
> > I'm not sure I understand what would go wrong if that assumption no
> > longer holds.
> 
> It's very simple, we don't do anything to the pointer returned
> by kmalloc before returning it as a tfm or other object with
> an alignment of CRYPTO_MINALIGN.  IOW if kmalloc starts returning
> pointers that are not aligned to CRYPTO_MINALIGN then we'd be
> lying to the compiler.

I agree that it would be lying to the compiler, but I don't think this
matters for arm64 where the CPU can do unaligned accesses just fine. We
don't even end up with unaligned accesses here. Let's say we have:

struct x {
	...
} __attribute__ ((__aligned__ (128)));

and the kmalloc(sizeof(struct x)) returns a 64-byte aligned pointer. The
compiler-generated code won't have any problem on arm64 accessing the
struct x members. As I said a few times, it's not affecting any other
architecture and not breaking arm64 either.

Anyway, let's agree to disagree. I'll look into keeping CRYPTO_MINALIGN
as ARCH_KMALLOC_MINALIGN and introduce a CRYPTO_DMA_MINALIGN (or just
use ARCH_DMA_MINALIGN directly) together with something like Linus'
dma_kmalloc() in places where an object aligned to ARCH_DMA_MINALIGN is
needed in the crypto code.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Ard Biesheuvel]

On Fri, 15 Apr 2022 at 10:12, Herbert Xu <herbert@gondor.apana.org.au> wrote:
>
> On Fri, Apr 15, 2022 at 10:05:21AM +0200, Ard Biesheuvel wrote:
> >
> > I guess that should be fixable. GIven that this is about padding
> > rather than alignment, we could do something like
> >
> > struct crypto_request {
> >   union {
> >       struct {
> >         ... fields ...
> >       };
> >       u8 __padding[ARCH_DMA_MINALIGN];
> >    };
> >     void __ctx[]  __align(CRYPTO_MINALIGN);
> > };
> >
> > And then hopefully, we can get rid of the padding once we fix drivers
> > doing non-cache coherent inbound DMA into those structures.
>
> Sorry, I don't think this works.  kmalloc can still return something
> that's not ARCH_DMA_MINALIGN-aligned, and therefore __ctx won't be
> aligned correctly.
>

That is the whole point, really: ARCH_DMA_MINALIGN==128 does not mean
__ctx needs to be aligned to 128 bytes, it only means that it should
not share a 128 byte cacheline with the preceding fields. So if
kmalloc() returns buffers that are aligned to whatever alignment the
platform requires (which will be 64 in most cases), the above
arrangement ensures that, without requiring that CRYPTO_MINALIGN ==
ARCH_DMA_MINALIGN.

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Ard Biesheuvel]

On Fri, 15 Apr 2022 at 10:12, Herbert Xu <herbert@gondor.apana.org.au> wrote:
>
> On Fri, Apr 15, 2022 at 10:05:21AM +0200, Ard Biesheuvel wrote:
> >
> > I guess that should be fixable. GIven that this is about padding
> > rather than alignment, we could do something like
> >
> > struct crypto_request {
> >   union {
> >       struct {
> >         ... fields ...
> >       };
> >       u8 __padding[ARCH_DMA_MINALIGN];
> >    };
> >     void __ctx[]  __align(CRYPTO_MINALIGN);
> > };
> >
> > And then hopefully, we can get rid of the padding once we fix drivers
> > doing non-cache coherent inbound DMA into those structures.
>
> Sorry, I don't think this works.  kmalloc can still return something
> that's not ARCH_DMA_MINALIGN-aligned, and therefore __ctx won't be
> aligned correctly.
>

That is the whole point, really: ARCH_DMA_MINALIGN==128 does not mean
__ctx needs to be aligned to 128 bytes, it only means that it should
not share a 128 byte cacheline with the preceding fields. So if
kmalloc() returns buffers that are aligned to whatever alignment the
platform requires (which will be 64 in most cases), the above
arrangement ensures that, without requiring that CRYPTO_MINALIGN ==
ARCH_DMA_MINALIGN.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Ard Biesheuvel]

On Fri, 15 Apr 2022 at 11:51, Ard Biesheuvel <ardb@kernel.org> wrote:
>
> On Fri, 15 Apr 2022 at 10:12, Herbert Xu <herbert@gondor.apana.org.au> wrote:
> >
> > On Fri, Apr 15, 2022 at 10:05:21AM +0200, Ard Biesheuvel wrote:
> > >
> > > I guess that should be fixable. GIven that this is about padding
> > > rather than alignment, we could do something like
> > >
> > > struct crypto_request {
> > >   union {
> > >       struct {
> > >         ... fields ...
> > >       };
> > >       u8 __padding[ARCH_DMA_MINALIGN];
> > >    };
> > >     void __ctx[]  __align(CRYPTO_MINALIGN);
> > > };
> > >
> > > And then hopefully, we can get rid of the padding once we fix drivers
> > > doing non-cache coherent inbound DMA into those structures.
> >
> > Sorry, I don't think this works.  kmalloc can still return something
> > that's not ARCH_DMA_MINALIGN-aligned, and therefore __ctx won't be
> > aligned correctly.
> >
>
> That is the whole point, really: ARCH_DMA_MINALIGN==128 does not mean
> __ctx needs to be aligned to 128 bytes, it only means that it should
> not share a 128 byte cacheline with the preceding fields.

Let's rephrase that as 'must not share a cacheline with the preceding
fields, and the worst case we expect to have to deal with is a
cacheline size of 128 bytes'

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Ard Biesheuvel]

On Fri, 15 Apr 2022 at 11:51, Ard Biesheuvel <ardb@kernel.org> wrote:
>
> On Fri, 15 Apr 2022 at 10:12, Herbert Xu <herbert@gondor.apana.org.au> wrote:
> >
> > On Fri, Apr 15, 2022 at 10:05:21AM +0200, Ard Biesheuvel wrote:
> > >
> > > I guess that should be fixable. GIven that this is about padding
> > > rather than alignment, we could do something like
> > >
> > > struct crypto_request {
> > >   union {
> > >       struct {
> > >         ... fields ...
> > >       };
> > >       u8 __padding[ARCH_DMA_MINALIGN];
> > >    };
> > >     void __ctx[]  __align(CRYPTO_MINALIGN);
> > > };
> > >
> > > And then hopefully, we can get rid of the padding once we fix drivers
> > > doing non-cache coherent inbound DMA into those structures.
> >
> > Sorry, I don't think this works.  kmalloc can still return something
> > that's not ARCH_DMA_MINALIGN-aligned, and therefore __ctx won't be
> > aligned correctly.
> >
>
> That is the whole point, really: ARCH_DMA_MINALIGN==128 does not mean
> __ctx needs to be aligned to 128 bytes, it only means that it should
> not share a 128 byte cacheline with the preceding fields.

Let's rephrase that as 'must not share a cacheline with the preceding
fields, and the worst case we expect to have to deal with is a
cacheline size of 128 bytes'

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Herbert Xu]

On Fri, Apr 15, 2022 at 11:51:49AM +0200, Ard Biesheuvel wrote:
>
> That is the whole point, really: ARCH_DMA_MINALIGN==128 does not mean
> __ctx needs to be aligned to 128 bytes, it only means that it should
> not share a 128 byte cacheline with the preceding fields. So if
> kmalloc() returns buffers that are aligned to whatever alignment the
> platform requires (which will be 64 in most cases), the above
> arrangement ensures that, without requiring that CRYPTO_MINALIGN ==
> ARCH_DMA_MINALIGN.

What if they started sharing a cacheline with the subsequent object?
I guess you could add some more padding at the end though.

I could accept this as a temporary solution, if you volunteer to
modify all the affected drivers so we can get rid of this bandaid.

Thanks,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Herbert Xu]

On Fri, Apr 15, 2022 at 11:51:49AM +0200, Ard Biesheuvel wrote:
>
> That is the whole point, really: ARCH_DMA_MINALIGN==128 does not mean
> __ctx needs to be aligned to 128 bytes, it only means that it should
> not share a 128 byte cacheline with the preceding fields. So if
> kmalloc() returns buffers that are aligned to whatever alignment the
> platform requires (which will be 64 in most cases), the above
> arrangement ensures that, without requiring that CRYPTO_MINALIGN ==
> ARCH_DMA_MINALIGN.

What if they started sharing a cacheline with the subsequent object?
I guess you could add some more padding at the end though.

I could accept this as a temporary solution, if you volunteer to
modify all the affected drivers so we can get rid of this bandaid.

Thanks,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Ard Biesheuvel]

On Fri, 15 Apr 2022 at 12:12, Herbert Xu <herbert@gondor.apana.org.au> wrote:
>
> On Fri, Apr 15, 2022 at 11:51:49AM +0200, Ard Biesheuvel wrote:
> >
> > That is the whole point, really: ARCH_DMA_MINALIGN==128 does not mean
> > __ctx needs to be aligned to 128 bytes, it only means that it should
> > not share a 128 byte cacheline with the preceding fields. So if
> > kmalloc() returns buffers that are aligned to whatever alignment the
> > platform requires (which will be 64 in most cases), the above
> > arrangement ensures that, without requiring that CRYPTO_MINALIGN ==
> > ARCH_DMA_MINALIGN.
>
> What if they started sharing a cacheline with the subsequent object?
> I guess you could add some more padding at the end though.
>

Subsequent objects are owned by the driver, and it is the
responsibility of the driver not to modify the fields while it is also
mapped for DMA (and we have had issues in the past where drivers
violated this rule). So as long as ARCH_KMALLOC_ALIGN guarantees
actual DMA minimum alignment for both the start and the end, we
shouldn't need any explicit padding at the end.

> I could accept this as a temporary solution, if you volunteer to
> modify all the affected drivers so we can get rid of this bandaid.
>

I'l do a scan of drivers/crypto to figure out how much we are relying
on this to begin with.

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Ard Biesheuvel]

On Fri, 15 Apr 2022 at 12:12, Herbert Xu <herbert@gondor.apana.org.au> wrote:
>
> On Fri, Apr 15, 2022 at 11:51:49AM +0200, Ard Biesheuvel wrote:
> >
> > That is the whole point, really: ARCH_DMA_MINALIGN==128 does not mean
> > __ctx needs to be aligned to 128 bytes, it only means that it should
> > not share a 128 byte cacheline with the preceding fields. So if
> > kmalloc() returns buffers that are aligned to whatever alignment the
> > platform requires (which will be 64 in most cases), the above
> > arrangement ensures that, without requiring that CRYPTO_MINALIGN ==
> > ARCH_DMA_MINALIGN.
>
> What if they started sharing a cacheline with the subsequent object?
> I guess you could add some more padding at the end though.
>

Subsequent objects are owned by the driver, and it is the
responsibility of the driver not to modify the fields while it is also
mapped for DMA (and we have had issues in the past where drivers
violated this rule). So as long as ARCH_KMALLOC_ALIGN guarantees
actual DMA minimum alignment for both the start and the end, we
shouldn't need any explicit padding at the end.

> I could accept this as a temporary solution, if you volunteer to
> modify all the affected drivers so we can get rid of this bandaid.
>

I'l do a scan of drivers/crypto to figure out how much we are relying
on this to begin with.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Herbert Xu]

On Fri, Apr 15, 2022 at 12:22:27PM +0200, Ard Biesheuvel wrote:
>
> Subsequent objects are owned by the driver, and it is the
> responsibility of the driver not to modify the fields while it is also
> mapped for DMA (and we have had issues in the past where drivers
> violated this rule). So as long as ARCH_KMALLOC_ALIGN guarantees
> actual DMA minimum alignment for both the start and the end, we
> shouldn't need any explicit padding at the end.

I don't understand why this is guaranteed.  The driver context
size is arbitrary so it could end in the middle of a cacheline.
The slab allocator could well lay it out so that the next kmalloc
object starts right after the end of the context, in which case
they would share a cache-line.

The next kmalloc object could be (and in fact is likely to be)
of the same type.

Previously this wasn't possible because kmalloc guaranteed
alignment.

> I'l do a scan of drivers/crypto to figure out how much we are relying
> on this to begin with.

Thanks,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Herbert Xu]

On Fri, Apr 15, 2022 at 12:22:27PM +0200, Ard Biesheuvel wrote:
>
> Subsequent objects are owned by the driver, and it is the
> responsibility of the driver not to modify the fields while it is also
> mapped for DMA (and we have had issues in the past where drivers
> violated this rule). So as long as ARCH_KMALLOC_ALIGN guarantees
> actual DMA minimum alignment for both the start and the end, we
> shouldn't need any explicit padding at the end.

I don't understand why this is guaranteed.  The driver context
size is arbitrary so it could end in the middle of a cacheline.
The slab allocator could well lay it out so that the next kmalloc
object starts right after the end of the context, in which case
they would share a cache-line.

The next kmalloc object could be (and in fact is likely to be)
of the same type.

Previously this wasn't possible because kmalloc guaranteed
alignment.

> I'l do a scan of drivers/crypto to figure out how much we are relying
> on this to begin with.

Thanks,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Arnd Bergmann]

On Fri, Apr 15, 2022 at 12:25 AM Linus Torvalds
<torvalds@linux-foundation.org> wrote:
> >
> And it could be that if I have 150k of those smallish allocations, a
> server with lots of active users might have millions. Not having
> looked at where they come from, maybe that isn't the case, but it
> *might* be.
>
> Maybe adding something like a
>
>         static int warn_every_1k = 0;
>         WARN_ON(size < 32 && (1023 & ++warn_every_1k));
>
> to kmalloc() would give us a statistical view of "lots of these small
> allocations" thing, and we could add GFP_NODMA to them. There probably
> aren't that many places that have those small allocations, and it's
> certainly safer to annotate "this is not for DMA" than have the
> requirement that all DMA allocations must be marked.

I think finding out the allocations is one of the most common examples
for ftrace. I followed the instructions from
https://www.kernel.org/doc/Documentation/trace/events.txt to
show me a histogram of all allocations under 256 bytes, which
(one kernel compile later) gives me something like

$echo 'hist:key=call_site.sym-offset:val=bytes_req:sort=bytes_req.descending
if bytes_req<256' >   \
/sys/kernel/debug/tracing/events/kmem/kmalloc/trigger
$ make -skj30
...
$ head  /sys/kernel/debug/tracing/events/kmem/kmalloc/hist
{ call_site: [ffffffffc04e457f]
btrfs_delete_delayed_dir_index+0xbf/0x1e0 [btrfs]       } hitcount:
 146914  bytes_req:   16454368
{ call_site: [ffffffffbbe601a3] generic_file_buffered_read+0x463/0x4a0
                 } hitcount:      98187  bytes_req:   14906232
{ call_site: [ffffffffc0497b81] btrfs_buffered_write+0x131/0x7e0
[btrfs]                } hitcount:     156513  bytes_req:   10038544
{ call_site: [ffffffffc05125c9] btrfs_alloc_block_rsv+0x29/0x60
[btrfs]                 } hitcount:     155044  bytes_req:    8682464
{ call_site: [ffffffffbbfe7272] kernfs_fop_open+0xc2/0x290
                 } hitcount:      38764  bytes_req:    5892128
{ call_site: [ffffffffbbfb6ea2] load_elf_binary+0x242/0xed0
                 } hitcount:      58276  bytes_req:    3729664
{ call_site: [ffffffffc04b52d0] __btrfs_map_block+0x1f0/0xb60 [btrfs]
                 } hitcount:      29289  bytes_req:    3521656
{ call_site: [ffffffffbbf7ac7e] inotify_handle_inode_event+0x7e/0x210
                 } hitcount:      61688  bytes_req:    2986992
{ call_site: [ffffffffbbf2fa35] alloc_pipe_info+0x65/0x230
                 } hitcount:      13139  bytes_req:    2312464
{ call_site: [ffffffffbc0cd3ec] security_task_alloc+0x9c/0x100
                 } hitcount:      60475  bytes_req:    2177100
{ call_site: [ffffffffbc0cd5f6] security_prepare_creds+0x76/0xa0
                 } hitcount:     266124  bytes_req:    2128992
{ call_site: [ffffffffbbfe710e] kernfs_get_open_node+0x7e/0x120
                 } hitcount:      38764  bytes_req:    1860672
{ call_site: [ffffffffc04e1fbd] btrfs_alloc_delayed_item+0x1d/0x50
[btrfs]              } hitcount:      11859  bytes_req:    1833383
{ call_site: [ffffffffc046595d] split_item+0x8d/0x2e0 [btrfs]
                 } hitcount:      14049  bytes_req:    1716288
{ call_site: [ffffffffbbfb6dbc] load_elf_binary+0x15c/0xed0
                 } hitcount:      58276  bytes_req:    1631728
{ call_site: [ffffffffbbf40e79] __d_alloc+0x179/0x1f0
                 } hitcount:      24814  bytes_req:    1280649
{ call_site: [ffffffffbbf5203f] single_open+0x2f/0xa0
                 } hitcount:      34541  bytes_req:    1105312
{ call_site: [ffffffffc047ad0a] btrfs_wq_submit_bio+0x4a/0xe0 [btrfs]
                 } hitcount:       7746  bytes_req:    1053456
{ call_site: [ffffffffbc519e95] xhci_urb_enqueue+0xf5/0x3c0
                 } hitcount:       5511  bytes_req:     484968
{ call_site: [ffffffffc0482935] btrfs_opendir+0x25/0x70 [btrfs]
                 } hitcount:      60245  bytes_req:     481960
{ call_site: [ffffffffc04c44ff] overwrite_item+0x1cf/0x5c0 [btrfs]
                 } hitcount:       7378  bytes_req:     364305
{ call_site: [ffffffffc04c4514] overwrite_item+0x1e4/0x5c0 [btrfs]
                 } hitcount:       7378  bytes_req:     364305
{ call_site: [ffffffffc04e207f] btrfs_wq_run_delayed_node+0x2f/0x80
[btrfs]             } hitcount:       3427  bytes_req:     356408
{ call_site: [ffffffffbbe7e96d] shmem_symlink+0xbd/0x250
                 } hitcount:       5169  bytes_req:     242943
{ call_site: [ffffffffc03e0526] hid_input_field+0x56/0x290 [hid]
                 } hitcount:      11004  bytes_req:     175760

I think these are all safe for the GFP_NODMA approach you suggest, maybe
not the xhci_urb_enqueue one.

          Arnd

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Arnd Bergmann]

On Fri, Apr 15, 2022 at 12:25 AM Linus Torvalds
<torvalds@linux-foundation.org> wrote:
> >
> And it could be that if I have 150k of those smallish allocations, a
> server with lots of active users might have millions. Not having
> looked at where they come from, maybe that isn't the case, but it
> *might* be.
>
> Maybe adding something like a
>
>         static int warn_every_1k = 0;
>         WARN_ON(size < 32 && (1023 & ++warn_every_1k));
>
> to kmalloc() would give us a statistical view of "lots of these small
> allocations" thing, and we could add GFP_NODMA to them. There probably
> aren't that many places that have those small allocations, and it's
> certainly safer to annotate "this is not for DMA" than have the
> requirement that all DMA allocations must be marked.

I think finding out the allocations is one of the most common examples
for ftrace. I followed the instructions from
https://www.kernel.org/doc/Documentation/trace/events.txt to
show me a histogram of all allocations under 256 bytes, which
(one kernel compile later) gives me something like

$echo 'hist:key=call_site.sym-offset:val=bytes_req:sort=bytes_req.descending
if bytes_req<256' >   \
/sys/kernel/debug/tracing/events/kmem/kmalloc/trigger
$ make -skj30
...
$ head  /sys/kernel/debug/tracing/events/kmem/kmalloc/hist
{ call_site: [ffffffffc04e457f]
btrfs_delete_delayed_dir_index+0xbf/0x1e0 [btrfs]       } hitcount:
 146914  bytes_req:   16454368
{ call_site: [ffffffffbbe601a3] generic_file_buffered_read+0x463/0x4a0
                 } hitcount:      98187  bytes_req:   14906232
{ call_site: [ffffffffc0497b81] btrfs_buffered_write+0x131/0x7e0
[btrfs]                } hitcount:     156513  bytes_req:   10038544
{ call_site: [ffffffffc05125c9] btrfs_alloc_block_rsv+0x29/0x60
[btrfs]                 } hitcount:     155044  bytes_req:    8682464
{ call_site: [ffffffffbbfe7272] kernfs_fop_open+0xc2/0x290
                 } hitcount:      38764  bytes_req:    5892128
{ call_site: [ffffffffbbfb6ea2] load_elf_binary+0x242/0xed0
                 } hitcount:      58276  bytes_req:    3729664
{ call_site: [ffffffffc04b52d0] __btrfs_map_block+0x1f0/0xb60 [btrfs]
                 } hitcount:      29289  bytes_req:    3521656
{ call_site: [ffffffffbbf7ac7e] inotify_handle_inode_event+0x7e/0x210
                 } hitcount:      61688  bytes_req:    2986992
{ call_site: [ffffffffbbf2fa35] alloc_pipe_info+0x65/0x230
                 } hitcount:      13139  bytes_req:    2312464
{ call_site: [ffffffffbc0cd3ec] security_task_alloc+0x9c/0x100
                 } hitcount:      60475  bytes_req:    2177100
{ call_site: [ffffffffbc0cd5f6] security_prepare_creds+0x76/0xa0
                 } hitcount:     266124  bytes_req:    2128992
{ call_site: [ffffffffbbfe710e] kernfs_get_open_node+0x7e/0x120
                 } hitcount:      38764  bytes_req:    1860672
{ call_site: [ffffffffc04e1fbd] btrfs_alloc_delayed_item+0x1d/0x50
[btrfs]              } hitcount:      11859  bytes_req:    1833383
{ call_site: [ffffffffc046595d] split_item+0x8d/0x2e0 [btrfs]
                 } hitcount:      14049  bytes_req:    1716288
{ call_site: [ffffffffbbfb6dbc] load_elf_binary+0x15c/0xed0
                 } hitcount:      58276  bytes_req:    1631728
{ call_site: [ffffffffbbf40e79] __d_alloc+0x179/0x1f0
                 } hitcount:      24814  bytes_req:    1280649
{ call_site: [ffffffffbbf5203f] single_open+0x2f/0xa0
                 } hitcount:      34541  bytes_req:    1105312
{ call_site: [ffffffffc047ad0a] btrfs_wq_submit_bio+0x4a/0xe0 [btrfs]
                 } hitcount:       7746  bytes_req:    1053456
{ call_site: [ffffffffbc519e95] xhci_urb_enqueue+0xf5/0x3c0
                 } hitcount:       5511  bytes_req:     484968
{ call_site: [ffffffffc0482935] btrfs_opendir+0x25/0x70 [btrfs]
                 } hitcount:      60245  bytes_req:     481960
{ call_site: [ffffffffc04c44ff] overwrite_item+0x1cf/0x5c0 [btrfs]
                 } hitcount:       7378  bytes_req:     364305
{ call_site: [ffffffffc04c4514] overwrite_item+0x1e4/0x5c0 [btrfs]
                 } hitcount:       7378  bytes_req:     364305
{ call_site: [ffffffffc04e207f] btrfs_wq_run_delayed_node+0x2f/0x80
[btrfs]             } hitcount:       3427  bytes_req:     356408
{ call_site: [ffffffffbbe7e96d] shmem_symlink+0xbd/0x250
                 } hitcount:       5169  bytes_req:     242943
{ call_site: [ffffffffc03e0526] hid_input_field+0x56/0x290 [hid]
                 } hitcount:      11004  bytes_req:     175760

I think these are all safe for the GFP_NODMA approach you suggest, maybe
not the xhci_urb_enqueue one.

          Arnd

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Ard Biesheuvel]

On Fri, 15 Apr 2022 at 12:45, Herbert Xu <herbert@gondor.apana.org.au> wrote:
>
> On Fri, Apr 15, 2022 at 12:22:27PM +0200, Ard Biesheuvel wrote:
> >
> > Subsequent objects are owned by the driver, and it is the
> > responsibility of the driver not to modify the fields while it is also
> > mapped for DMA (and we have had issues in the past where drivers
> > violated this rule). So as long as ARCH_KMALLOC_ALIGN guarantees
> > actual DMA minimum alignment for both the start and the end, we
> > shouldn't need any explicit padding at the end.
>
> I don't understand why this is guaranteed.  The driver context
> size is arbitrary so it could end in the middle of a cacheline.
> The slab allocator could well lay it out so that the next kmalloc
> object starts right after the end of the context, in which case
> they would share a cache-line.
>

If this is the case, things are already broken today. We never take
ARCH_DMA_MINALIGN into account when adding the driver ctx size to the
overall allocation size.

> The next kmalloc object could be (and in fact is likely to be)
> of the same type.
>
> Previously this wasn't possible because kmalloc guaranteed
> alignment.
>

Either it does or it doesn't. If kmalloc() guarantees the actual DMA
alignment at both ends, the situation you describe cannot occur, given
that the driver's slice of the request/TFM structure would be padded
up to actual DMA alignment, in spite of whether or not
ARCH_DMA_MINALIGN exceeds that. So it would never share a cacheline in
practice, even though they might live in the same 128 byte aligned
region on a system that has a minimum DMA alignment that is lower than
that.

If kmalloc() does not guarantee that the end of the buffer is aligned
to actual DMA alignment, things are already broken today.

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Ard Biesheuvel]

On Fri, 15 Apr 2022 at 12:45, Herbert Xu <herbert@gondor.apana.org.au> wrote:
>
> On Fri, Apr 15, 2022 at 12:22:27PM +0200, Ard Biesheuvel wrote:
> >
> > Subsequent objects are owned by the driver, and it is the
> > responsibility of the driver not to modify the fields while it is also
> > mapped for DMA (and we have had issues in the past where drivers
> > violated this rule). So as long as ARCH_KMALLOC_ALIGN guarantees
> > actual DMA minimum alignment for both the start and the end, we
> > shouldn't need any explicit padding at the end.
>
> I don't understand why this is guaranteed.  The driver context
> size is arbitrary so it could end in the middle of a cacheline.
> The slab allocator could well lay it out so that the next kmalloc
> object starts right after the end of the context, in which case
> they would share a cache-line.
>

If this is the case, things are already broken today. We never take
ARCH_DMA_MINALIGN into account when adding the driver ctx size to the
overall allocation size.

> The next kmalloc object could be (and in fact is likely to be)
> of the same type.
>
> Previously this wasn't possible because kmalloc guaranteed
> alignment.
>

Either it does or it doesn't. If kmalloc() guarantees the actual DMA
alignment at both ends, the situation you describe cannot occur, given
that the driver's slice of the request/TFM structure would be padded
up to actual DMA alignment, in spite of whether or not
ARCH_DMA_MINALIGN exceeds that. So it would never share a cacheline in
practice, even though they might live in the same 128 byte aligned
region on a system that has a minimum DMA alignment that is lower than
that.

If kmalloc() does not guarantee that the end of the buffer is aligned
to actual DMA alignment, things are already broken today.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Fri, Apr 15, 2022 at 10:05:21AM +0200, Ard Biesheuvel wrote:
> On Fri, 15 Apr 2022 at 09:52, Herbert Xu <herbert@gondor.apana.org.au> wrote:
> > On Fri, Apr 15, 2022 at 09:49:12AM +0200, Ard Biesheuvel wrote:
> > > I'm not sure I understand what would go wrong if that assumption no
> > > longer holds.
> >
> > It's very simple, we don't do anything to the pointer returned
> > by kmalloc before returning it as a tfm or other object with
> > an alignment of CRYPTO_MINALIGN.  IOW if kmalloc starts returning
> > pointers that are not aligned to CRYPTO_MINALIGN then we'd be
> > lying to the compiler.
> 
> I guess that should be fixable. GIven that this is about padding
> rather than alignment, we could do something like
> 
> struct crypto_request {
>   union {
>       struct {
>         ... fields ...
>       };
>       u8 __padding[ARCH_DMA_MINALIGN];
>    };
>     void __ctx[]  __align(CRYPTO_MINALIGN);
> };
> 
> And then hopefully, we can get rid of the padding once we fix drivers
> doing non-cache coherent inbound DMA into those structures.

But if we keep CRYPTO_MINALIGN as 128, don't we get the padding
automatically?

struct crypto_request {
	...
	void *__ctx[] CRYPTO_MINALIGN_ATTR;
};

__alignof__(struct crypto_request) == 128;
sizeof(struct crypto_request) == N * 128

The same alignment and size is true for a structure like:

struct crypto_alg {
	...
} CRYPTO_MINALIGN_ATTR;

Any kmalloc() of sizeof(the above structures) will return a pointer
aligned to 128, irrespective of what ARCH_KMALLOC_MINALIGN is.

The problem is if you have a structure without any alignment attribute
(just ABI default), making its sizeof() smaller than ARCH_DMA_MINALIGN.
In this case kmalloc() could return a pointer aligned to something
smaller. Is this the case in the crypto code today? I can see it uses
the right alignment annotations already, no need for kmalloc() hacks.

-- 
Catalin

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Fri, Apr 15, 2022 at 10:05:21AM +0200, Ard Biesheuvel wrote:
> On Fri, 15 Apr 2022 at 09:52, Herbert Xu <herbert@gondor.apana.org.au> wrote:
> > On Fri, Apr 15, 2022 at 09:49:12AM +0200, Ard Biesheuvel wrote:
> > > I'm not sure I understand what would go wrong if that assumption no
> > > longer holds.
> >
> > It's very simple, we don't do anything to the pointer returned
> > by kmalloc before returning it as a tfm or other object with
> > an alignment of CRYPTO_MINALIGN.  IOW if kmalloc starts returning
> > pointers that are not aligned to CRYPTO_MINALIGN then we'd be
> > lying to the compiler.
> 
> I guess that should be fixable. GIven that this is about padding
> rather than alignment, we could do something like
> 
> struct crypto_request {
>   union {
>       struct {
>         ... fields ...
>       };
>       u8 __padding[ARCH_DMA_MINALIGN];
>    };
>     void __ctx[]  __align(CRYPTO_MINALIGN);
> };
> 
> And then hopefully, we can get rid of the padding once we fix drivers
> doing non-cache coherent inbound DMA into those structures.

But if we keep CRYPTO_MINALIGN as 128, don't we get the padding
automatically?

struct crypto_request {
	...
	void *__ctx[] CRYPTO_MINALIGN_ATTR;
};

__alignof__(struct crypto_request) == 128;
sizeof(struct crypto_request) == N * 128

The same alignment and size is true for a structure like:

struct crypto_alg {
	...
} CRYPTO_MINALIGN_ATTR;

Any kmalloc() of sizeof(the above structures) will return a pointer
aligned to 128, irrespective of what ARCH_KMALLOC_MINALIGN is.

The problem is if you have a structure without any alignment attribute
(just ABI default), making its sizeof() smaller than ARCH_DMA_MINALIGN.
In this case kmalloc() could return a pointer aligned to something
smaller. Is this the case in the crypto code today? I can see it uses
the right alignment annotations already, no need for kmalloc() hacks.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Ard Biesheuvel]

On Fri, 15 Apr 2022 at 14:19, Catalin Marinas <catalin.marinas@arm.com> wrote:
>
> On Fri, Apr 15, 2022 at 10:05:21AM +0200, Ard Biesheuvel wrote:
> > On Fri, 15 Apr 2022 at 09:52, Herbert Xu <herbert@gondor.apana.org.au> wrote:
> > > On Fri, Apr 15, 2022 at 09:49:12AM +0200, Ard Biesheuvel wrote:
> > > > I'm not sure I understand what would go wrong if that assumption no
> > > > longer holds.
> > >
> > > It's very simple, we don't do anything to the pointer returned
> > > by kmalloc before returning it as a tfm or other object with
> > > an alignment of CRYPTO_MINALIGN.  IOW if kmalloc starts returning
> > > pointers that are not aligned to CRYPTO_MINALIGN then we'd be
> > > lying to the compiler.
> >
> > I guess that should be fixable. GIven that this is about padding
> > rather than alignment, we could do something like
> >
> > struct crypto_request {
> >   union {
> >       struct {
> >         ... fields ...
> >       };
> >       u8 __padding[ARCH_DMA_MINALIGN];
> >    };
> >     void __ctx[]  __align(CRYPTO_MINALIGN);
> > };
> >
> > And then hopefully, we can get rid of the padding once we fix drivers
> > doing non-cache coherent inbound DMA into those structures.
>
> But if we keep CRYPTO_MINALIGN as 128, don't we get the padding
> automatically?
>

I suppose, yes.

> struct crypto_request {
>         ...
>         void *__ctx[] CRYPTO_MINALIGN_ATTR;
> };
>
> __alignof__(struct crypto_request) == 128;
> sizeof(struct crypto_request) == N * 128
>
> The same alignment and size is true for a structure like:
>
> struct crypto_alg {
>         ...
> } CRYPTO_MINALIGN_ATTR;
>
> Any kmalloc() of sizeof(the above structures) will return a pointer
> aligned to 128, irrespective of what ARCH_KMALLOC_MINALIGN is.
>
> The problem is if you have a structure without any alignment attribute
> (just ABI default), making its sizeof() smaller than ARCH_DMA_MINALIGN.
> In this case kmalloc() could return a pointer aligned to something
> smaller. Is this the case in the crypto code today? I can see it uses
> the right alignment annotations already, no need for kmalloc() hacks.
>

As long as CRYPTO_MINALIGN >= ARCH_KMALLOC_MINALIGN, we won't be lying
to the compiler when casting kmalloc buffers to these struct types.

I'd still like to fix the bad DMA behavior but I suppose it is a separate issue.

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Ard Biesheuvel]

On Fri, 15 Apr 2022 at 14:19, Catalin Marinas <catalin.marinas@arm.com> wrote:
>
> On Fri, Apr 15, 2022 at 10:05:21AM +0200, Ard Biesheuvel wrote:
> > On Fri, 15 Apr 2022 at 09:52, Herbert Xu <herbert@gondor.apana.org.au> wrote:
> > > On Fri, Apr 15, 2022 at 09:49:12AM +0200, Ard Biesheuvel wrote:
> > > > I'm not sure I understand what would go wrong if that assumption no
> > > > longer holds.
> > >
> > > It's very simple, we don't do anything to the pointer returned
> > > by kmalloc before returning it as a tfm or other object with
> > > an alignment of CRYPTO_MINALIGN.  IOW if kmalloc starts returning
> > > pointers that are not aligned to CRYPTO_MINALIGN then we'd be
> > > lying to the compiler.
> >
> > I guess that should be fixable. GIven that this is about padding
> > rather than alignment, we could do something like
> >
> > struct crypto_request {
> >   union {
> >       struct {
> >         ... fields ...
> >       };
> >       u8 __padding[ARCH_DMA_MINALIGN];
> >    };
> >     void __ctx[]  __align(CRYPTO_MINALIGN);
> > };
> >
> > And then hopefully, we can get rid of the padding once we fix drivers
> > doing non-cache coherent inbound DMA into those structures.
>
> But if we keep CRYPTO_MINALIGN as 128, don't we get the padding
> automatically?
>

I suppose, yes.

> struct crypto_request {
>         ...
>         void *__ctx[] CRYPTO_MINALIGN_ATTR;
> };
>
> __alignof__(struct crypto_request) == 128;
> sizeof(struct crypto_request) == N * 128
>
> The same alignment and size is true for a structure like:
>
> struct crypto_alg {
>         ...
> } CRYPTO_MINALIGN_ATTR;
>
> Any kmalloc() of sizeof(the above structures) will return a pointer
> aligned to 128, irrespective of what ARCH_KMALLOC_MINALIGN is.
>
> The problem is if you have a structure without any alignment attribute
> (just ABI default), making its sizeof() smaller than ARCH_DMA_MINALIGN.
> In this case kmalloc() could return a pointer aligned to something
> smaller. Is this the case in the crypto code today? I can see it uses
> the right alignment annotations already, no need for kmalloc() hacks.
>

As long as CRYPTO_MINALIGN >= ARCH_KMALLOC_MINALIGN, we won't be lying
to the compiler when casting kmalloc buffers to these struct types.

I'd still like to fix the bad DMA behavior but I suppose it is a separate issue.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Fri, Apr 15, 2022 at 10:51:40AM +0100, Catalin Marinas wrote:
> On Fri, Apr 15, 2022 at 03:51:54PM +0800, Herbert Xu wrote:
> > On Fri, Apr 15, 2022 at 09:49:12AM +0200, Ard Biesheuvel wrote:
> > > I'm not sure I understand what would go wrong if that assumption no
> > > longer holds.
> > 
> > It's very simple, we don't do anything to the pointer returned
> > by kmalloc before returning it as a tfm or other object with
> > an alignment of CRYPTO_MINALIGN.  IOW if kmalloc starts returning
> > pointers that are not aligned to CRYPTO_MINALIGN then we'd be
> > lying to the compiler.
> 
> I agree that it would be lying to the compiler, but I don't think this
> matters for arm64 where the CPU can do unaligned accesses just fine. We
> don't even end up with unaligned accesses here. Let's say we have:
> 
> struct x {
> 	...
> } __attribute__ ((__aligned__ (128)));
> 
> and the kmalloc(sizeof(struct x)) returns a 64-byte aligned pointer.

This needs a clarification. For the above structure, kmalloc() will
return a 128-byte aligned pointer since sizeof(x) is a multiple of 128.
The potential problem is if you have something like:

	kmalloc(sizeof(struct x) + 64);

The above could end up as a kmalloc(192) which is available with an
ARCH_KMALLOC_MINALIGN of 64. If that's a real use-case, I can change the
slab patch to not create the 192 (or 48 if we go for an even smaller
ARCH_KMALLOC_MINALIGN) caches and we'd always have ARCH_DMA_MINALIGN
guarantee if the structure itself is correctly aligned. No lying to the
compiler.

-- 
Catalin

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Fri, Apr 15, 2022 at 10:51:40AM +0100, Catalin Marinas wrote:
> On Fri, Apr 15, 2022 at 03:51:54PM +0800, Herbert Xu wrote:
> > On Fri, Apr 15, 2022 at 09:49:12AM +0200, Ard Biesheuvel wrote:
> > > I'm not sure I understand what would go wrong if that assumption no
> > > longer holds.
> > 
> > It's very simple, we don't do anything to the pointer returned
> > by kmalloc before returning it as a tfm or other object with
> > an alignment of CRYPTO_MINALIGN.  IOW if kmalloc starts returning
> > pointers that are not aligned to CRYPTO_MINALIGN then we'd be
> > lying to the compiler.
> 
> I agree that it would be lying to the compiler, but I don't think this
> matters for arm64 where the CPU can do unaligned accesses just fine. We
> don't even end up with unaligned accesses here. Let's say we have:
> 
> struct x {
> 	...
> } __attribute__ ((__aligned__ (128)));
> 
> and the kmalloc(sizeof(struct x)) returns a 64-byte aligned pointer.

This needs a clarification. For the above structure, kmalloc() will
return a 128-byte aligned pointer since sizeof(x) is a multiple of 128.
The potential problem is if you have something like:

	kmalloc(sizeof(struct x) + 64);

The above could end up as a kmalloc(192) which is available with an
ARCH_KMALLOC_MINALIGN of 64. If that's a real use-case, I can change the
slab patch to not create the 192 (or 48 if we go for an even smaller
ARCH_KMALLOC_MINALIGN) caches and we'd always have ARCH_DMA_MINALIGN
guarantee if the structure itself is correctly aligned. No lying to the
compiler.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Thu, Apr 14, 2022 at 03:25:59PM -0700, Linus Torvalds wrote:
> On Thu, Apr 14, 2022 at 12:49 PM Catalin Marinas
> <catalin.marinas@arm.com> wrote:
> > It's a lot worse, ARCH_KMALLOC_MINALIGN is currently 128 bytes on arm64.
> > I want to at least get it down to 64 with this series while preserving
> > the current kmalloc() semantics.
> 
> So here's a thought - maybe we could do the reverse of GFP_DMA, and
> add a flag to the places that want small allocations and know they
> don't need DMA?

I wonder whether that's a lot more churn than trying to identify places
where a small kmalloc()'ed buffer is passed to the DMA API. DMA into
kmalloc() buffers should be a small fraction of the total kmalloc()
uses.

For kmem_cache we have the SLAB_HWCACHE_ALIGN flag. We can add a similar
GFP_ flag as that's what we care about for DMA safety. It doesn't even
need to force the alignment to ARCH_DMA_MINALIGN but just
cache_line_size() (typically 64 on arm64 while ARCH_DMA_MINALIGN is 128
for about three platforms that have this requirement).

Functions like dma_map_single() can be made to track down the origin of
the buffer when size < cache_line_size() and warn if the slab is not
correctly aligned.

-- 
Catalin

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Thu, Apr 14, 2022 at 03:25:59PM -0700, Linus Torvalds wrote:
> On Thu, Apr 14, 2022 at 12:49 PM Catalin Marinas
> <catalin.marinas@arm.com> wrote:
> > It's a lot worse, ARCH_KMALLOC_MINALIGN is currently 128 bytes on arm64.
> > I want to at least get it down to 64 with this series while preserving
> > the current kmalloc() semantics.
> 
> So here's a thought - maybe we could do the reverse of GFP_DMA, and
> add a flag to the places that want small allocations and know they
> don't need DMA?

I wonder whether that's a lot more churn than trying to identify places
where a small kmalloc()'ed buffer is passed to the DMA API. DMA into
kmalloc() buffers should be a small fraction of the total kmalloc()
uses.

For kmem_cache we have the SLAB_HWCACHE_ALIGN flag. We can add a similar
GFP_ flag as that's what we care about for DMA safety. It doesn't even
need to force the alignment to ARCH_DMA_MINALIGN but just
cache_line_size() (typically 64 on arm64 while ARCH_DMA_MINALIGN is 128
for about three platforms that have this requirement).

Functions like dma_map_single() can be made to track down the origin of
the buffer when size < cache_line_size() and warn if the slab is not
correctly aligned.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Herbert Xu]

On Fri, Apr 15, 2022 at 01:38:15PM +0200, Ard Biesheuvel wrote:
>
> If this is the case, things are already broken today. We never take
> ARCH_DMA_MINALIGN into account when adding the driver ctx size to the
> overall allocation size.

No it's not broken because kmalloc guarantees alignment.  For
example, if ARCH_DMA_MINALIGN is 128 bytes, then kmalloc will
always return a pointer that's 128-byte aligned.  That guarantees
this object and the next object are on different cache-lines.

If you reduce the kmalloc minimum alignment to 64 bytes, then
the two neighbouring objects can share cache-lines, even if
each object is bigger than 128 bytes (e.g., if they were 192
bytes each).

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Herbert Xu]

On Fri, Apr 15, 2022 at 01:38:15PM +0200, Ard Biesheuvel wrote:
>
> If this is the case, things are already broken today. We never take
> ARCH_DMA_MINALIGN into account when adding the driver ctx size to the
> overall allocation size.

No it's not broken because kmalloc guarantees alignment.  For
example, if ARCH_DMA_MINALIGN is 128 bytes, then kmalloc will
always return a pointer that's 128-byte aligned.  That guarantees
this object and the next object are on different cache-lines.

If you reduce the kmalloc minimum alignment to 64 bytes, then
the two neighbouring objects can share cache-lines, even if
each object is bigger than 128 bytes (e.g., if they were 192
bytes each).

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Herbert Xu]

On Fri, Apr 15, 2022 at 01:31:32PM +0100, Catalin Marinas wrote:
>
> This needs a clarification. For the above structure, kmalloc() will
> return a 128-byte aligned pointer since sizeof(x) is a multiple of 128.
> The potential problem is if you have something like:
> 
> 	kmalloc(sizeof(struct x) + 64);
> 
> The above could end up as a kmalloc(192) which is available with an
> ARCH_KMALLOC_MINALIGN of 64. If that's a real use-case, I can change the
> slab patch to not create the 192 (or 48 if we go for an even smaller
> ARCH_KMALLOC_MINALIGN) caches and we'd always have ARCH_DMA_MINALIGN
> guarantee if the structure itself is correctly aligned. No lying to the
> compiler.

Yes I suppose that should work:

1) Enlarge each crypto API object so that they're >= 128 bytes;
2) Modify kmalloc so that for sizes >= 128 bytes they're padded
to multiples of 128.

But it really feels like a hack.

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Herbert Xu]

On Fri, Apr 15, 2022 at 01:31:32PM +0100, Catalin Marinas wrote:
>
> This needs a clarification. For the above structure, kmalloc() will
> return a 128-byte aligned pointer since sizeof(x) is a multiple of 128.
> The potential problem is if you have something like:
> 
> 	kmalloc(sizeof(struct x) + 64);
> 
> The above could end up as a kmalloc(192) which is available with an
> ARCH_KMALLOC_MINALIGN of 64. If that's a real use-case, I can change the
> slab patch to not create the 192 (or 48 if we go for an even smaller
> ARCH_KMALLOC_MINALIGN) caches and we'd always have ARCH_DMA_MINALIGN
> guarantee if the structure itself is correctly aligned. No lying to the
> compiler.

Yes I suppose that should work:

1) Enlarge each crypto API object so that they're >= 128 bytes;
2) Modify kmalloc so that for sizes >= 128 bytes they're padded
to multiples of 128.

But it really feels like a hack.

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Sun, Apr 17, 2022 at 04:08:22PM +0800, Herbert Xu wrote:
> On Fri, Apr 15, 2022 at 01:38:15PM +0200, Ard Biesheuvel wrote:
> > If this is the case, things are already broken today. We never take
> > ARCH_DMA_MINALIGN into account when adding the driver ctx size to the
> > overall allocation size.
> 
> No it's not broken because kmalloc guarantees alignment.  For
> example, if ARCH_DMA_MINALIGN is 128 bytes, then kmalloc will
> always return a pointer that's 128-byte aligned.  That guarantees
> this object and the next object are on different cache-lines.
> 
> If you reduce the kmalloc minimum alignment to 64 bytes, then
> the two neighbouring objects can share cache-lines, even if
> each object is bigger than 128 bytes (e.g., if they were 192
> bytes each).

Not with my series, the non-sharing of cache lines is preserved.
kmalloc() still returns objects aligned to a cache-line.
ARCH_DMA_MINALIGN was chosen as the cover-all value for all SoCs
supported but I want to reduce the kmalloc() alignment to a cache line
size if a platform has a cache line smaller than ARCH_DMA_MINALIGN (most
arm64 SoCs have a cache line of 64 bytes rather than 128).

-- 
Catalin

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Sun, Apr 17, 2022 at 04:08:22PM +0800, Herbert Xu wrote:
> On Fri, Apr 15, 2022 at 01:38:15PM +0200, Ard Biesheuvel wrote:
> > If this is the case, things are already broken today. We never take
> > ARCH_DMA_MINALIGN into account when adding the driver ctx size to the
> > overall allocation size.
> 
> No it's not broken because kmalloc guarantees alignment.  For
> example, if ARCH_DMA_MINALIGN is 128 bytes, then kmalloc will
> always return a pointer that's 128-byte aligned.  That guarantees
> this object and the next object are on different cache-lines.
> 
> If you reduce the kmalloc minimum alignment to 64 bytes, then
> the two neighbouring objects can share cache-lines, even if
> each object is bigger than 128 bytes (e.g., if they were 192
> bytes each).

Not with my series, the non-sharing of cache lines is preserved.
kmalloc() still returns objects aligned to a cache-line.
ARCH_DMA_MINALIGN was chosen as the cover-all value for all SoCs
supported but I want to reduce the kmalloc() alignment to a cache line
size if a platform has a cache line smaller than ARCH_DMA_MINALIGN (most
arm64 SoCs have a cache line of 64 bytes rather than 128).

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Herbert Xu]

On Sun, Apr 17, 2022 at 09:31:09AM +0100, Catalin Marinas wrote:
>
> Not with my series, the non-sharing of cache lines is preserved.
> kmalloc() still returns objects aligned to a cache-line.
> ARCH_DMA_MINALIGN was chosen as the cover-all value for all SoCs
> supported but I want to reduce the kmalloc() alignment to a cache line
> size if a platform has a cache line smaller than ARCH_DMA_MINALIGN (most
> arm64 SoCs have a cache line of 64 bytes rather than 128).

OK, but then you don't need to play with CRYPTO_MINALIGN at all,
right? All you need to do is add the padding between the Crypto
API fields and the context structure, right?

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Herbert Xu]

On Sun, Apr 17, 2022 at 09:31:09AM +0100, Catalin Marinas wrote:
>
> Not with my series, the non-sharing of cache lines is preserved.
> kmalloc() still returns objects aligned to a cache-line.
> ARCH_DMA_MINALIGN was chosen as the cover-all value for all SoCs
> supported but I want to reduce the kmalloc() alignment to a cache line
> size if a platform has a cache line smaller than ARCH_DMA_MINALIGN (most
> arm64 SoCs have a cache line of 64 bytes rather than 128).

OK, but then you don't need to play with CRYPTO_MINALIGN at all,
right? All you need to do is add the padding between the Crypto
API fields and the context structure, right?

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Sun, Apr 17, 2022 at 04:11:22PM +0800, Herbert Xu wrote:
> On Fri, Apr 15, 2022 at 01:31:32PM +0100, Catalin Marinas wrote:
> >
> > This needs a clarification. For the above structure, kmalloc() will
> > return a 128-byte aligned pointer since sizeof(x) is a multiple of 128.
> > The potential problem is if you have something like:
> > 
> > 	kmalloc(sizeof(struct x) + 64);
> > 
> > The above could end up as a kmalloc(192) which is available with an
> > ARCH_KMALLOC_MINALIGN of 64. If that's a real use-case, I can change the
> > slab patch to not create the 192 (or 48 if we go for an even smaller
> > ARCH_KMALLOC_MINALIGN) caches and we'd always have ARCH_DMA_MINALIGN
> > guarantee if the structure itself is correctly aligned. No lying to the
> > compiler.
> 
> Yes I suppose that should work:
> 
> 1) Enlarge each crypto API object so that they're >= 128 bytes;

I don't think we need to do anything here. A structure like:

struct x {
	char y;
	char z[] CRYPTO_MINALIGN_ATTR;
};

is already of size 128. Without CRYPTO_MINALIGN_ATTR, its size would be
1 but otherwise the whole structure inherits the alignment of its
member and this translates into an aligned size.

> 2) Modify kmalloc so that for sizes >= 128 bytes they're padded
> to multiples of 128.

This doesn't look like a hack, we want to honour the power of 2
alignments.

-- 
Catalin

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Sun, Apr 17, 2022 at 04:11:22PM +0800, Herbert Xu wrote:
> On Fri, Apr 15, 2022 at 01:31:32PM +0100, Catalin Marinas wrote:
> >
> > This needs a clarification. For the above structure, kmalloc() will
> > return a 128-byte aligned pointer since sizeof(x) is a multiple of 128.
> > The potential problem is if you have something like:
> > 
> > 	kmalloc(sizeof(struct x) + 64);
> > 
> > The above could end up as a kmalloc(192) which is available with an
> > ARCH_KMALLOC_MINALIGN of 64. If that's a real use-case, I can change the
> > slab patch to not create the 192 (or 48 if we go for an even smaller
> > ARCH_KMALLOC_MINALIGN) caches and we'd always have ARCH_DMA_MINALIGN
> > guarantee if the structure itself is correctly aligned. No lying to the
> > compiler.
> 
> Yes I suppose that should work:
> 
> 1) Enlarge each crypto API object so that they're >= 128 bytes;

I don't think we need to do anything here. A structure like:

struct x {
	char y;
	char z[] CRYPTO_MINALIGN_ATTR;
};

is already of size 128. Without CRYPTO_MINALIGN_ATTR, its size would be
1 but otherwise the whole structure inherits the alignment of its
member and this translates into an aligned size.

> 2) Modify kmalloc so that for sizes >= 128 bytes they're padded
> to multiples of 128.

This doesn't look like a hack, we want to honour the power of 2
alignments.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Herbert Xu]

On Sun, Apr 17, 2022 at 09:38:40AM +0100, Catalin Marinas wrote:
> 
> I don't think we need to do anything here. A structure like:
> 
> struct x {
> 	char y;
> 	char z[] CRYPTO_MINALIGN_ATTR;
> };
> 
> is already of size 128. Without CRYPTO_MINALIGN_ATTR, its size would be
> 1 but otherwise the whole structure inherits the alignment of its
> member and this translates into an aligned size.

No we should not lie to the compiler, we have code elsewhere
that uses the alignment to compute the amount of extra padding
needed to create greater padding.  If CRYPTO_MINALIGN is misleading
then that calculation will fall apart.

So keep CRYPTO_MINALIGN at whatever alignment you lower kmalloc
to, and then add the padding you need to separate the Crypto API
bits from the context.  In fact, that is exactly what cra_alignmask
is supposed to do.

Sure we currently limit the maximum alignment to 64 bytes because
of stack usage but we can certainly look into increasing it as
that's what you're doing here anyway.

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Herbert Xu]

On Sun, Apr 17, 2022 at 09:38:40AM +0100, Catalin Marinas wrote:
> 
> I don't think we need to do anything here. A structure like:
> 
> struct x {
> 	char y;
> 	char z[] CRYPTO_MINALIGN_ATTR;
> };
> 
> is already of size 128. Without CRYPTO_MINALIGN_ATTR, its size would be
> 1 but otherwise the whole structure inherits the alignment of its
> member and this translates into an aligned size.

No we should not lie to the compiler, we have code elsewhere
that uses the alignment to compute the amount of extra padding
needed to create greater padding.  If CRYPTO_MINALIGN is misleading
then that calculation will fall apart.

So keep CRYPTO_MINALIGN at whatever alignment you lower kmalloc
to, and then add the padding you need to separate the Crypto API
bits from the context.  In fact, that is exactly what cra_alignmask
is supposed to do.

Sure we currently limit the maximum alignment to 64 bytes because
of stack usage but we can certainly look into increasing it as
that's what you're doing here anyway.

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Sun, Apr 17, 2022 at 04:35:58PM +0800, Herbert Xu wrote:
> On Sun, Apr 17, 2022 at 09:31:09AM +0100, Catalin Marinas wrote:
> >
> > Not with my series, the non-sharing of cache lines is preserved.
> > kmalloc() still returns objects aligned to a cache-line.
> > ARCH_DMA_MINALIGN was chosen as the cover-all value for all SoCs
> > supported but I want to reduce the kmalloc() alignment to a cache line
> > size if a platform has a cache line smaller than ARCH_DMA_MINALIGN (most
> > arm64 SoCs have a cache line of 64 bytes rather than 128).
> 
> OK, but then you don't need to play with CRYPTO_MINALIGN at all,
> right? All you need to do is add the padding between the Crypto
> API fields and the context structure, right?

Right, if that's what you prefer. Something like:

diff --git a/include/linux/crypto.h b/include/linux/crypto.h
index 2324ab6f1846..bb645b2f2718 100644
--- a/include/linux/crypto.h
+++ b/include/linux/crypto.h
@@ -645,7 +645,7 @@ struct crypto_tfm {
 	
 	struct crypto_alg *__crt_alg;
 
-	void *__crt_ctx[] CRYPTO_MINALIGN_ATTR;
+	void *__crt_ctx[] __aligned(ARCH_DMA_MINALIGN);
 };

But once we do that, are there any other CRYPTO_MINALIGN left around?

-- 
Catalin

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Sun, Apr 17, 2022 at 04:35:58PM +0800, Herbert Xu wrote:
> On Sun, Apr 17, 2022 at 09:31:09AM +0100, Catalin Marinas wrote:
> >
> > Not with my series, the non-sharing of cache lines is preserved.
> > kmalloc() still returns objects aligned to a cache-line.
> > ARCH_DMA_MINALIGN was chosen as the cover-all value for all SoCs
> > supported but I want to reduce the kmalloc() alignment to a cache line
> > size if a platform has a cache line smaller than ARCH_DMA_MINALIGN (most
> > arm64 SoCs have a cache line of 64 bytes rather than 128).
> 
> OK, but then you don't need to play with CRYPTO_MINALIGN at all,
> right? All you need to do is add the padding between the Crypto
> API fields and the context structure, right?

Right, if that's what you prefer. Something like:

diff --git a/include/linux/crypto.h b/include/linux/crypto.h
index 2324ab6f1846..bb645b2f2718 100644
--- a/include/linux/crypto.h
+++ b/include/linux/crypto.h
@@ -645,7 +645,7 @@ struct crypto_tfm {
 	
 	struct crypto_alg *__crt_alg;
 
-	void *__crt_ctx[] CRYPTO_MINALIGN_ATTR;
+	void *__crt_ctx[] __aligned(ARCH_DMA_MINALIGN);
 };

But once we do that, are there any other CRYPTO_MINALIGN left around?

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Herbert Xu]

On Sun, Apr 17, 2022 at 09:50:50AM +0100, Catalin Marinas wrote:
>
> Right, if that's what you prefer. Something like:
> 
> diff --git a/include/linux/crypto.h b/include/linux/crypto.h
> index 2324ab6f1846..bb645b2f2718 100644
> --- a/include/linux/crypto.h
> +++ b/include/linux/crypto.h
> @@ -645,7 +645,7 @@ struct crypto_tfm {
>  	
>  	struct crypto_alg *__crt_alg;
>  
> -	void *__crt_ctx[] CRYPTO_MINALIGN_ATTR;
> +	void *__crt_ctx[] __aligned(ARCH_DMA_MINALIGN);
>  };
> 
> But once we do that, are there any other CRYPTO_MINALIGN left around?

This is still implying the whole structure is aligned to the given
value, which it is not.

Please just add the padding as needed.

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Herbert Xu]

On Sun, Apr 17, 2022 at 09:50:50AM +0100, Catalin Marinas wrote:
>
> Right, if that's what you prefer. Something like:
> 
> diff --git a/include/linux/crypto.h b/include/linux/crypto.h
> index 2324ab6f1846..bb645b2f2718 100644
> --- a/include/linux/crypto.h
> +++ b/include/linux/crypto.h
> @@ -645,7 +645,7 @@ struct crypto_tfm {
>  	
>  	struct crypto_alg *__crt_alg;
>  
> -	void *__crt_ctx[] CRYPTO_MINALIGN_ATTR;
> +	void *__crt_ctx[] __aligned(ARCH_DMA_MINALIGN);
>  };
> 
> But once we do that, are there any other CRYPTO_MINALIGN left around?

This is still implying the whole structure is aligned to the given
value, which it is not.

Please just add the padding as needed.

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Sun, Apr 17, 2022 at 04:43:33PM +0800, Herbert Xu wrote:
> On Sun, Apr 17, 2022 at 09:38:40AM +0100, Catalin Marinas wrote:
> > I don't think we need to do anything here. A structure like:
> > 
> > struct x {
> > 	char y;
> > 	char z[] CRYPTO_MINALIGN_ATTR;
> > };
> > 
> > is already of size 128. Without CRYPTO_MINALIGN_ATTR, its size would be
> > 1 but otherwise the whole structure inherits the alignment of its
> > member and this translates into an aligned size.
> 
> No we should not lie to the compiler,

We won't if we ensure that a structure with sizeof() >= 128 is aligned
to 128.

> we have code elsewhere
> that uses the alignment to compute the amount of extra padding
> needed to create greater padding.  If CRYPTO_MINALIGN is misleading
> then that calculation will fall apart.

There is no direct CRYPTO_MINALIGN use for any extra padding AFAICT.
There is an indirect use via __alignof__(ctx) like in
crypto_tfm_ctx_alignment() but IIUC in that case CRYPTO_MINALIGN is a
statement of what you want rather than what you get from kmalloc(). So
if you want 128 alignment of tfm->__crt_ctx, you should say so by either
changing the attribute to __aligned(ARCH_DMA_MINALIGN) or keeping
CRYPTO_MINALIGN as 128.

There is the union padding that Ard suggested but I don't think it buys
us much, the __aligned(ARCH_DMA_MINALIGN) gives you the padding and the
kmalloc() rules the alignment (subject to removing kmalloc-192). The
code that allocates these would need to place the structure aligned
anyway, irrespective of whether we use the padding or the
__aligned(ARCH_DMA_MINALIGN).

> So keep CRYPTO_MINALIGN at whatever alignment you lower kmalloc
> to, and then add the padding you need to separate the Crypto API
> bits from the context.  In fact, that is exactly what cra_alignmask
> is supposed to do.

I disagree on the cra_alignmask intention here, though I may be wrong as
I did not write the code. Yes, you could make it ARCH_DMA_MINALIGN
everywhere but IMHO that's not what it is supposed to do. The driver
only knows about the bus master alignment requirements (typically
smaller than a cache line) while the architecture-defined
ARCH_DMA_MINALIGN cares about the non-coherent DMA requirements.

> Sure we currently limit the maximum alignment to 64 bytes because
> of stack usage but we can certainly look into increasing it as
> that's what you're doing here anyway.

I'm not actually increasing CRYPTO_MINALIGN, just trying to keep it as
the current value of 128 for arm64.

-- 
Catalin

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Sun, Apr 17, 2022 at 04:43:33PM +0800, Herbert Xu wrote:
> On Sun, Apr 17, 2022 at 09:38:40AM +0100, Catalin Marinas wrote:
> > I don't think we need to do anything here. A structure like:
> > 
> > struct x {
> > 	char y;
> > 	char z[] CRYPTO_MINALIGN_ATTR;
> > };
> > 
> > is already of size 128. Without CRYPTO_MINALIGN_ATTR, its size would be
> > 1 but otherwise the whole structure inherits the alignment of its
> > member and this translates into an aligned size.
> 
> No we should not lie to the compiler,

We won't if we ensure that a structure with sizeof() >= 128 is aligned
to 128.

> we have code elsewhere
> that uses the alignment to compute the amount of extra padding
> needed to create greater padding.  If CRYPTO_MINALIGN is misleading
> then that calculation will fall apart.

There is no direct CRYPTO_MINALIGN use for any extra padding AFAICT.
There is an indirect use via __alignof__(ctx) like in
crypto_tfm_ctx_alignment() but IIUC in that case CRYPTO_MINALIGN is a
statement of what you want rather than what you get from kmalloc(). So
if you want 128 alignment of tfm->__crt_ctx, you should say so by either
changing the attribute to __aligned(ARCH_DMA_MINALIGN) or keeping
CRYPTO_MINALIGN as 128.

There is the union padding that Ard suggested but I don't think it buys
us much, the __aligned(ARCH_DMA_MINALIGN) gives you the padding and the
kmalloc() rules the alignment (subject to removing kmalloc-192). The
code that allocates these would need to place the structure aligned
anyway, irrespective of whether we use the padding or the
__aligned(ARCH_DMA_MINALIGN).

> So keep CRYPTO_MINALIGN at whatever alignment you lower kmalloc
> to, and then add the padding you need to separate the Crypto API
> bits from the context.  In fact, that is exactly what cra_alignmask
> is supposed to do.

I disagree on the cra_alignmask intention here, though I may be wrong as
I did not write the code. Yes, you could make it ARCH_DMA_MINALIGN
everywhere but IMHO that's not what it is supposed to do. The driver
only knows about the bus master alignment requirements (typically
smaller than a cache line) while the architecture-defined
ARCH_DMA_MINALIGN cares about the non-coherent DMA requirements.

> Sure we currently limit the maximum alignment to 64 bytes because
> of stack usage but we can certainly look into increasing it as
> that's what you're doing here anyway.

I'm not actually increasing CRYPTO_MINALIGN, just trying to keep it as
the current value of 128 for arm64.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Sun, Apr 17, 2022 at 04:58:29PM +0800, Herbert Xu wrote:
> On Sun, Apr 17, 2022 at 09:50:50AM +0100, Catalin Marinas wrote:
> >
> > Right, if that's what you prefer. Something like:
> > 
> > diff --git a/include/linux/crypto.h b/include/linux/crypto.h
> > index 2324ab6f1846..bb645b2f2718 100644
> > --- a/include/linux/crypto.h
> > +++ b/include/linux/crypto.h
> > @@ -645,7 +645,7 @@ struct crypto_tfm {
> >  	
> >  	struct crypto_alg *__crt_alg;
> >  
> > -	void *__crt_ctx[] CRYPTO_MINALIGN_ATTR;
> > +	void *__crt_ctx[] __aligned(ARCH_DMA_MINALIGN);
> >  };
> > 
> > But once we do that, are there any other CRYPTO_MINALIGN left around?
> 
> This is still implying the whole structure is aligned to the given
> value, which it is not.
> 
> Please just add the padding as needed.

Do you mean as per Ard's proposal here:

https://lore.kernel.org/r/CAMj1kXH0x5Va7Wgs+mU1ONDwwsazOBuN4z4ihVzO2uG-n41Kbg@mail.gmail.com

struct crypto_request {
	union {
		struct {
			... fields ...
		};
		u8 __padding[ARCH_DMA_MINALIGN];
	};
	void __ctx[]  __aligned(CRYPTO_MINALIGN);
};

If CRYPTO_MINALIGN is lowered to, say, 8 (to be the same as lowest
ARCH_KMALLOC_MINALIGN), the __alignof__(req->__ctx) would be 8.
Functions like crypto_tfm_ctx_alignment() will return 8 when what you
need is 128. We can change those functions to return ARCH_DMA_MINALIGN
instead or always bump cra_alignmask to ARCH_DMA_MINALIGN-1.

-- 
Catalin

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Sun, Apr 17, 2022 at 04:58:29PM +0800, Herbert Xu wrote:
> On Sun, Apr 17, 2022 at 09:50:50AM +0100, Catalin Marinas wrote:
> >
> > Right, if that's what you prefer. Something like:
> > 
> > diff --git a/include/linux/crypto.h b/include/linux/crypto.h
> > index 2324ab6f1846..bb645b2f2718 100644
> > --- a/include/linux/crypto.h
> > +++ b/include/linux/crypto.h
> > @@ -645,7 +645,7 @@ struct crypto_tfm {
> >  	
> >  	struct crypto_alg *__crt_alg;
> >  
> > -	void *__crt_ctx[] CRYPTO_MINALIGN_ATTR;
> > +	void *__crt_ctx[] __aligned(ARCH_DMA_MINALIGN);
> >  };
> > 
> > But once we do that, are there any other CRYPTO_MINALIGN left around?
> 
> This is still implying the whole structure is aligned to the given
> value, which it is not.
> 
> Please just add the padding as needed.

Do you mean as per Ard's proposal here:

https://lore.kernel.org/r/CAMj1kXH0x5Va7Wgs+mU1ONDwwsazOBuN4z4ihVzO2uG-n41Kbg@mail.gmail.com

struct crypto_request {
	union {
		struct {
			... fields ...
		};
		u8 __padding[ARCH_DMA_MINALIGN];
	};
	void __ctx[]  __aligned(CRYPTO_MINALIGN);
};

If CRYPTO_MINALIGN is lowered to, say, 8 (to be the same as lowest
ARCH_KMALLOC_MINALIGN), the __alignof__(req->__ctx) would be 8.
Functions like crypto_tfm_ctx_alignment() will return 8 when what you
need is 128. We can change those functions to return ARCH_DMA_MINALIGN
instead or always bump cra_alignmask to ARCH_DMA_MINALIGN-1.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Herbert Xu]

On Sun, Apr 17, 2022 at 05:30:27PM +0100, Catalin Marinas wrote:
>
> Do you mean as per Ard's proposal here:
> 
> https://lore.kernel.org/r/CAMj1kXH0x5Va7Wgs+mU1ONDwwsazOBuN4z4ihVzO2uG-n41Kbg@mail.gmail.com
> 
> struct crypto_request {
> 	union {
> 		struct {
> 			... fields ...
> 		};
> 		u8 __padding[ARCH_DMA_MINALIGN];
> 	};
> 	void __ctx[]  __aligned(CRYPTO_MINALIGN);
> };
> 
> If CRYPTO_MINALIGN is lowered to, say, 8 (to be the same as lowest
> ARCH_KMALLOC_MINALIGN), the __alignof__(req->__ctx) would be 8.
> Functions like crypto_tfm_ctx_alignment() will return 8 when what you
> need is 128. We can change those functions to return ARCH_DMA_MINALIGN
> instead or always bump cra_alignmask to ARCH_DMA_MINALIGN-1.

OK, at this point I think we need to let the code do the talking :)

I've seen Ard's patches already and I think I understand what your
needs are.  So let me whip up some code to show you guys what I
think needs to be done.

Please bear with me for a few days.

Thanks,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Herbert Xu]

On Sun, Apr 17, 2022 at 05:30:27PM +0100, Catalin Marinas wrote:
>
> Do you mean as per Ard's proposal here:
> 
> https://lore.kernel.org/r/CAMj1kXH0x5Va7Wgs+mU1ONDwwsazOBuN4z4ihVzO2uG-n41Kbg@mail.gmail.com
> 
> struct crypto_request {
> 	union {
> 		struct {
> 			... fields ...
> 		};
> 		u8 __padding[ARCH_DMA_MINALIGN];
> 	};
> 	void __ctx[]  __aligned(CRYPTO_MINALIGN);
> };
> 
> If CRYPTO_MINALIGN is lowered to, say, 8 (to be the same as lowest
> ARCH_KMALLOC_MINALIGN), the __alignof__(req->__ctx) would be 8.
> Functions like crypto_tfm_ctx_alignment() will return 8 when what you
> need is 128. We can change those functions to return ARCH_DMA_MINALIGN
> instead or always bump cra_alignmask to ARCH_DMA_MINALIGN-1.

OK, at this point I think we need to let the code do the talking :)

I've seen Ard's patches already and I think I understand what your
needs are.  So let me whip up some code to show you guys what I
think needs to be done.

Please bear with me for a few days.

Thanks,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Mon, Apr 18, 2022 at 04:37:17PM +0800, Herbert Xu wrote:
> On Sun, Apr 17, 2022 at 05:30:27PM +0100, Catalin Marinas wrote:
> >
> > Do you mean as per Ard's proposal here:
> > 
> > https://lore.kernel.org/r/CAMj1kXH0x5Va7Wgs+mU1ONDwwsazOBuN4z4ihVzO2uG-n41Kbg@mail.gmail.com
> > 
> > struct crypto_request {
> > 	union {
> > 		struct {
> > 			... fields ...
> > 		};
> > 		u8 __padding[ARCH_DMA_MINALIGN];
> > 	};
> > 	void __ctx[]  __aligned(CRYPTO_MINALIGN);
> > };
> > 
> > If CRYPTO_MINALIGN is lowered to, say, 8 (to be the same as lowest
> > ARCH_KMALLOC_MINALIGN), the __alignof__(req->__ctx) would be 8.
> > Functions like crypto_tfm_ctx_alignment() will return 8 when what you
> > need is 128. We can change those functions to return ARCH_DMA_MINALIGN
> > instead or always bump cra_alignmask to ARCH_DMA_MINALIGN-1.
> 
> OK, at this point I think we need to let the code do the talking :)
> 
> I've seen Ard's patches already and I think I understand what your
> needs are.  So let me whip up some code to show you guys what I
> think needs to be done.
> 
> Please bear with me for a few days.

Thanks Herbert, that's great. Whenever you have time, I'll be busy this
week with collecting arm64 patches for the upcoming merging window
anyway.

-- 
Catalin

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Mon, Apr 18, 2022 at 04:37:17PM +0800, Herbert Xu wrote:
> On Sun, Apr 17, 2022 at 05:30:27PM +0100, Catalin Marinas wrote:
> >
> > Do you mean as per Ard's proposal here:
> > 
> > https://lore.kernel.org/r/CAMj1kXH0x5Va7Wgs+mU1ONDwwsazOBuN4z4ihVzO2uG-n41Kbg@mail.gmail.com
> > 
> > struct crypto_request {
> > 	union {
> > 		struct {
> > 			... fields ...
> > 		};
> > 		u8 __padding[ARCH_DMA_MINALIGN];
> > 	};
> > 	void __ctx[]  __aligned(CRYPTO_MINALIGN);
> > };
> > 
> > If CRYPTO_MINALIGN is lowered to, say, 8 (to be the same as lowest
> > ARCH_KMALLOC_MINALIGN), the __alignof__(req->__ctx) would be 8.
> > Functions like crypto_tfm_ctx_alignment() will return 8 when what you
> > need is 128. We can change those functions to return ARCH_DMA_MINALIGN
> > instead or always bump cra_alignmask to ARCH_DMA_MINALIGN-1.
> 
> OK, at this point I think we need to let the code do the talking :)
> 
> I've seen Ard's patches already and I think I understand what your
> needs are.  So let me whip up some code to show you guys what I
> think needs to be done.
> 
> Please bear with me for a few days.

Thanks Herbert, that's great. Whenever you have time, I'll be busy this
week with collecting arm64 patches for the upcoming merging window
anyway.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Mon, Apr 18, 2022 at 04:37:17PM +0800, Herbert Xu wrote:
> On Sun, Apr 17, 2022 at 05:30:27PM +0100, Catalin Marinas wrote:
> > Do you mean as per Ard's proposal here:
> > 
> > https://lore.kernel.org/r/CAMj1kXH0x5Va7Wgs+mU1ONDwwsazOBuN4z4ihVzO2uG-n41Kbg@mail.gmail.com
> > 
> > struct crypto_request {
> > 	union {
> > 		struct {
> > 			... fields ...
> > 		};
> > 		u8 __padding[ARCH_DMA_MINALIGN];
> > 	};
> > 	void __ctx[]  __aligned(CRYPTO_MINALIGN);
> > };
> > 
> > If CRYPTO_MINALIGN is lowered to, say, 8 (to be the same as lowest
> > ARCH_KMALLOC_MINALIGN), the __alignof__(req->__ctx) would be 8.
> > Functions like crypto_tfm_ctx_alignment() will return 8 when what you
> > need is 128. We can change those functions to return ARCH_DMA_MINALIGN
> > instead or always bump cra_alignmask to ARCH_DMA_MINALIGN-1.
> 
> OK, at this point I think we need to let the code do the talking :)
> 
> I've seen Ard's patches already and I think I understand what your
> needs are.  So let me whip up some code to show you guys what I
> think needs to be done.

BTW before you have a go at this, there's also Linus' idea that does not
change the crypto code (at least not functionally). Of course, you and
Ard can still try to figure out how to reduce the padding but if we go
with Linus' idea of a new GFP_NODMA flag, there won't be any changes to
the crypto code as long as it doesn't pass such flag. So, the options:

1. Change ARCH_KMALLOC_MINALIGN to 8 (or ARCH_SLAB_MINALIGN if higher)
   while keeping ARCH_DMA_MINALIGN to 128. By default kmalloc() will
   honour the 128-byte alignment, unless GDP_NODMA is passed. This still
   requires changing CRYPTO_MINALIGN to ARCH_DMA_MINALIGN but there is
   no functional change, kmalloc() without the new flag will return
   CRYPTO_MINALIGN-aligned pointers.

2. Leave ARCH_KMALLOC_MINALIGN as ARCH_DMA_MINALIGN (128) and introduce
   a new GFP_PACKED (I think it fits better than 'NODMA') flag that
   reduces the minimum kmalloc() below ARCH_KMALLOC_MINALIGN (and
   probably at least ARCH_SLAB_MINALIGN). It's equivalent to (1) but
   does not touch the crypto code at all.

(1) and (2) are the same, just minor naming difference. Happy to go with
any of them. They still have the downside that we need to add the new
GFP_ flag to those hotspots that allocate small objects (Arnd provided
an idea on how to find them with ftrace) but at least we know it won't
inadvertently break anything.

-- 
Catalin

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Mon, Apr 18, 2022 at 04:37:17PM +0800, Herbert Xu wrote:
> On Sun, Apr 17, 2022 at 05:30:27PM +0100, Catalin Marinas wrote:
> > Do you mean as per Ard's proposal here:
> > 
> > https://lore.kernel.org/r/CAMj1kXH0x5Va7Wgs+mU1ONDwwsazOBuN4z4ihVzO2uG-n41Kbg@mail.gmail.com
> > 
> > struct crypto_request {
> > 	union {
> > 		struct {
> > 			... fields ...
> > 		};
> > 		u8 __padding[ARCH_DMA_MINALIGN];
> > 	};
> > 	void __ctx[]  __aligned(CRYPTO_MINALIGN);
> > };
> > 
> > If CRYPTO_MINALIGN is lowered to, say, 8 (to be the same as lowest
> > ARCH_KMALLOC_MINALIGN), the __alignof__(req->__ctx) would be 8.
> > Functions like crypto_tfm_ctx_alignment() will return 8 when what you
> > need is 128. We can change those functions to return ARCH_DMA_MINALIGN
> > instead or always bump cra_alignmask to ARCH_DMA_MINALIGN-1.
> 
> OK, at this point I think we need to let the code do the talking :)
> 
> I've seen Ard's patches already and I think I understand what your
> needs are.  So let me whip up some code to show you guys what I
> think needs to be done.

BTW before you have a go at this, there's also Linus' idea that does not
change the crypto code (at least not functionally). Of course, you and
Ard can still try to figure out how to reduce the padding but if we go
with Linus' idea of a new GFP_NODMA flag, there won't be any changes to
the crypto code as long as it doesn't pass such flag. So, the options:

1. Change ARCH_KMALLOC_MINALIGN to 8 (or ARCH_SLAB_MINALIGN if higher)
   while keeping ARCH_DMA_MINALIGN to 128. By default kmalloc() will
   honour the 128-byte alignment, unless GDP_NODMA is passed. This still
   requires changing CRYPTO_MINALIGN to ARCH_DMA_MINALIGN but there is
   no functional change, kmalloc() without the new flag will return
   CRYPTO_MINALIGN-aligned pointers.

2. Leave ARCH_KMALLOC_MINALIGN as ARCH_DMA_MINALIGN (128) and introduce
   a new GFP_PACKED (I think it fits better than 'NODMA') flag that
   reduces the minimum kmalloc() below ARCH_KMALLOC_MINALIGN (and
   probably at least ARCH_SLAB_MINALIGN). It's equivalent to (1) but
   does not touch the crypto code at all.

(1) and (2) are the same, just minor naming difference. Happy to go with
any of them. They still have the downside that we need to add the new
GFP_ flag to those hotspots that allocate small objects (Arnd provided
an idea on how to find them with ftrace) but at least we know it won't
inadvertently break anything.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Ard Biesheuvel]

On Mon, 18 Apr 2022 at 18:44, Catalin Marinas <catalin.marinas@arm.com> wrote:
>
> On Mon, Apr 18, 2022 at 04:37:17PM +0800, Herbert Xu wrote:
> > On Sun, Apr 17, 2022 at 05:30:27PM +0100, Catalin Marinas wrote:
> > > Do you mean as per Ard's proposal here:
> > >
> > > https://lore.kernel.org/r/CAMj1kXH0x5Va7Wgs+mU1ONDwwsazOBuN4z4ihVzO2uG-n41Kbg@mail.gmail.com
> > >
> > > struct crypto_request {
> > >     union {
> > >             struct {
> > >                     ... fields ...
> > >             };
> > >             u8 __padding[ARCH_DMA_MINALIGN];
> > >     };
> > >     void __ctx[]  __aligned(CRYPTO_MINALIGN);
> > > };
> > >
> > > If CRYPTO_MINALIGN is lowered to, say, 8 (to be the same as lowest
> > > ARCH_KMALLOC_MINALIGN), the __alignof__(req->__ctx) would be 8.
> > > Functions like crypto_tfm_ctx_alignment() will return 8 when what you
> > > need is 128. We can change those functions to return ARCH_DMA_MINALIGN
> > > instead or always bump cra_alignmask to ARCH_DMA_MINALIGN-1.
> >
> > OK, at this point I think we need to let the code do the talking :)
> >
> > I've seen Ard's patches already and I think I understand what your
> > needs are.  So let me whip up some code to show you guys what I
> > think needs to be done.
>
> BTW before you have a go at this, there's also Linus' idea that does not
> change the crypto code (at least not functionally). Of course, you and
> Ard can still try to figure out how to reduce the padding but if we go
> with Linus' idea of a new GFP_NODMA flag, there won't be any changes to
> the crypto code as long as it doesn't pass such flag. So, the options:
>
> 1. Change ARCH_KMALLOC_MINALIGN to 8 (or ARCH_SLAB_MINALIGN if higher)
>    while keeping ARCH_DMA_MINALIGN to 128. By default kmalloc() will
>    honour the 128-byte alignment, unless GDP_NODMA is passed. This still
>    requires changing CRYPTO_MINALIGN to ARCH_DMA_MINALIGN but there is
>    no functional change, kmalloc() without the new flag will return
>    CRYPTO_MINALIGN-aligned pointers.
>
> 2. Leave ARCH_KMALLOC_MINALIGN as ARCH_DMA_MINALIGN (128) and introduce
>    a new GFP_PACKED (I think it fits better than 'NODMA') flag that
>    reduces the minimum kmalloc() below ARCH_KMALLOC_MINALIGN (and
>    probably at least ARCH_SLAB_MINALIGN). It's equivalent to (1) but
>    does not touch the crypto code at all.
>
> (1) and (2) are the same, just minor naming difference. Happy to go with
> any of them. They still have the downside that we need to add the new
> GFP_ flag to those hotspots that allocate small objects (Arnd provided
> an idea on how to find them with ftrace) but at least we know it won't
> inadvertently break anything.
>

I'm not sure GFP_NODMA adds much here.

The way I see it, the issue in the crypto code is that we are relying
on a ARCH_KMALLOC_ALIGN aligned zero length __ctx[] array for three
different things:
- ensuring/informing the compiler that top-level request/TFM
structures are aligned to ARCH_KMALLOC_ALIGN,
- adding padding to ensure that driver context structures that are
embedded in those top-level request/TFM structures are sufficiently
aligned so that any member C types appear at the expected alignment
(but those structures are not usually defined as being aligned to
ARCH_KMALLOC_ALIGN)
- adding padding to ensure that these driver context structures do not
share cache lines with the preceding top-level struct.

One thing to note here is that the padding is only necessary when the
driver context size > 0, and has nothing to do with the alignment of
the top-level struct.

Using a single aligned ctx member was a nice way to accomplish all of
these when it was introduced, but I think it might be better to get
rid of it, and move the padding logic to the static inline helpers
instead.

So something like

struct skcipher_request {
  ...
} CRYPTO_MINALIGN_ATTR;

which states/ensures the alignment of the struct, and

void *skcipher_request_ctx(struct skcipher_request *req) {
  return (void *)PTR_ALIGN(req + 1, ARCH_DMA_MINALIGN);
}

to get at the context struct, instead of using a struct field.

Then, we could update skcipher_request_alloc() to only round up
sizeof(struct skipher_request) to ARCH_DMA_MINALIGN if the reqsize is
>0 to begin with, and if it is, to also round reqsize up to
ARCH_DMA_MINALIGN when accessed. That way, we spell out the DMA
padding requirements with relying on aligned struct members.

If we do it this way, we have a clear distinction between expectations
about what kmalloc returns in terms of alignment, and adding padding
to influence the placement of the context struct. It also makes it
easier to either apply the changes I proposed in the series I sent out
a couple of weeks ago, or get rid of DMA alignment for request/TFM
structures altogether, if we manage to identify and fix the drivers
that are relying on this. In any case, it decouples these two things
in a way that allows Catalin to make his kmalloc changes without
having to redefine CRYPTO_MINALIGN to ARCH_DMA_MINALIGN.

-- 
Ard.

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Ard Biesheuvel]

On Mon, 18 Apr 2022 at 18:44, Catalin Marinas <catalin.marinas@arm.com> wrote:
>
> On Mon, Apr 18, 2022 at 04:37:17PM +0800, Herbert Xu wrote:
> > On Sun, Apr 17, 2022 at 05:30:27PM +0100, Catalin Marinas wrote:
> > > Do you mean as per Ard's proposal here:
> > >
> > > https://lore.kernel.org/r/CAMj1kXH0x5Va7Wgs+mU1ONDwwsazOBuN4z4ihVzO2uG-n41Kbg@mail.gmail.com
> > >
> > > struct crypto_request {
> > >     union {
> > >             struct {
> > >                     ... fields ...
> > >             };
> > >             u8 __padding[ARCH_DMA_MINALIGN];
> > >     };
> > >     void __ctx[]  __aligned(CRYPTO_MINALIGN);
> > > };
> > >
> > > If CRYPTO_MINALIGN is lowered to, say, 8 (to be the same as lowest
> > > ARCH_KMALLOC_MINALIGN), the __alignof__(req->__ctx) would be 8.
> > > Functions like crypto_tfm_ctx_alignment() will return 8 when what you
> > > need is 128. We can change those functions to return ARCH_DMA_MINALIGN
> > > instead or always bump cra_alignmask to ARCH_DMA_MINALIGN-1.
> >
> > OK, at this point I think we need to let the code do the talking :)
> >
> > I've seen Ard's patches already and I think I understand what your
> > needs are.  So let me whip up some code to show you guys what I
> > think needs to be done.
>
> BTW before you have a go at this, there's also Linus' idea that does not
> change the crypto code (at least not functionally). Of course, you and
> Ard can still try to figure out how to reduce the padding but if we go
> with Linus' idea of a new GFP_NODMA flag, there won't be any changes to
> the crypto code as long as it doesn't pass such flag. So, the options:
>
> 1. Change ARCH_KMALLOC_MINALIGN to 8 (or ARCH_SLAB_MINALIGN if higher)
>    while keeping ARCH_DMA_MINALIGN to 128. By default kmalloc() will
>    honour the 128-byte alignment, unless GDP_NODMA is passed. This still
>    requires changing CRYPTO_MINALIGN to ARCH_DMA_MINALIGN but there is
>    no functional change, kmalloc() without the new flag will return
>    CRYPTO_MINALIGN-aligned pointers.
>
> 2. Leave ARCH_KMALLOC_MINALIGN as ARCH_DMA_MINALIGN (128) and introduce
>    a new GFP_PACKED (I think it fits better than 'NODMA') flag that
>    reduces the minimum kmalloc() below ARCH_KMALLOC_MINALIGN (and
>    probably at least ARCH_SLAB_MINALIGN). It's equivalent to (1) but
>    does not touch the crypto code at all.
>
> (1) and (2) are the same, just minor naming difference. Happy to go with
> any of them. They still have the downside that we need to add the new
> GFP_ flag to those hotspots that allocate small objects (Arnd provided
> an idea on how to find them with ftrace) but at least we know it won't
> inadvertently break anything.
>

I'm not sure GFP_NODMA adds much here.

The way I see it, the issue in the crypto code is that we are relying
on a ARCH_KMALLOC_ALIGN aligned zero length __ctx[] array for three
different things:
- ensuring/informing the compiler that top-level request/TFM
structures are aligned to ARCH_KMALLOC_ALIGN,
- adding padding to ensure that driver context structures that are
embedded in those top-level request/TFM structures are sufficiently
aligned so that any member C types appear at the expected alignment
(but those structures are not usually defined as being aligned to
ARCH_KMALLOC_ALIGN)
- adding padding to ensure that these driver context structures do not
share cache lines with the preceding top-level struct.

One thing to note here is that the padding is only necessary when the
driver context size > 0, and has nothing to do with the alignment of
the top-level struct.

Using a single aligned ctx member was a nice way to accomplish all of
these when it was introduced, but I think it might be better to get
rid of it, and move the padding logic to the static inline helpers
instead.

So something like

struct skcipher_request {
  ...
} CRYPTO_MINALIGN_ATTR;

which states/ensures the alignment of the struct, and

void *skcipher_request_ctx(struct skcipher_request *req) {
  return (void *)PTR_ALIGN(req + 1, ARCH_DMA_MINALIGN);
}

to get at the context struct, instead of using a struct field.

Then, we could update skcipher_request_alloc() to only round up
sizeof(struct skipher_request) to ARCH_DMA_MINALIGN if the reqsize is
>0 to begin with, and if it is, to also round reqsize up to
ARCH_DMA_MINALIGN when accessed. That way, we spell out the DMA
padding requirements with relying on aligned struct members.

If we do it this way, we have a clear distinction between expectations
about what kmalloc returns in terms of alignment, and adding padding
to influence the placement of the context struct. It also makes it
easier to either apply the changes I proposed in the series I sent out
a couple of weeks ago, or get rid of DMA alignment for request/TFM
structures altogether, if we manage to identify and fix the drivers
that are relying on this. In any case, it decouples these two things
in a way that allows Catalin to make his kmalloc changes without
having to redefine CRYPTO_MINALIGN to ARCH_DMA_MINALIGN.

-- 
Ard.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Tue, Apr 19, 2022 at 11:50:11PM +0200, Ard Biesheuvel wrote:
> On Mon, 18 Apr 2022 at 18:44, Catalin Marinas <catalin.marinas@arm.com> wrote:
> > On Mon, Apr 18, 2022 at 04:37:17PM +0800, Herbert Xu wrote:
> > > I've seen Ard's patches already and I think I understand what your
> > > needs are.  So let me whip up some code to show you guys what I
> > > think needs to be done.
> >
> > BTW before you have a go at this, there's also Linus' idea that does not
> > change the crypto code (at least not functionally). Of course, you and
> > Ard can still try to figure out how to reduce the padding but if we go
> > with Linus' idea of a new GFP_NODMA flag, there won't be any changes to
> > the crypto code as long as it doesn't pass such flag. So, the options:
> >
> > 1. Change ARCH_KMALLOC_MINALIGN to 8 (or ARCH_SLAB_MINALIGN if higher)
> >    while keeping ARCH_DMA_MINALIGN to 128. By default kmalloc() will
> >    honour the 128-byte alignment, unless GDP_NODMA is passed. This still
> >    requires changing CRYPTO_MINALIGN to ARCH_DMA_MINALIGN but there is
> >    no functional change, kmalloc() without the new flag will return
> >    CRYPTO_MINALIGN-aligned pointers.
> >
> > 2. Leave ARCH_KMALLOC_MINALIGN as ARCH_DMA_MINALIGN (128) and introduce
> >    a new GFP_PACKED (I think it fits better than 'NODMA') flag that
> >    reduces the minimum kmalloc() below ARCH_KMALLOC_MINALIGN (and
> >    probably at least ARCH_SLAB_MINALIGN). It's equivalent to (1) but
> >    does not touch the crypto code at all.
> >
> > (1) and (2) are the same, just minor naming difference. Happy to go with
> > any of them. They still have the downside that we need to add the new
> > GFP_ flag to those hotspots that allocate small objects (Arnd provided
> > an idea on how to find them with ftrace) but at least we know it won't
> > inadvertently break anything.
> 
> I'm not sure GFP_NODMA adds much here.

What it buys is that the crypto code won't need to be changed
immediately, so we can go ahead with the kmalloc() optimisation while
you and Herbert figure out how to refactor the crypto code.

Another option is to make the change but pass a new GFP_DMA_MINALIGN
flag to all kmalloc() calls in the crypto code (or a new dma_kmalloc()
that Linus suggested). This way the allocations in the crypto code are
guaranteed to be ARCH_DMA_MINALIGN (we'd still change CRYPTO_MINALIGN to
this).

> The way I see it, the issue in the crypto code is that we are relying
> on a ARCH_KMALLOC_ALIGN aligned zero length __ctx[] array for three
> different things:
> - ensuring/informing the compiler that top-level request/TFM
> structures are aligned to ARCH_KMALLOC_ALIGN,
> - adding padding to ensure that driver context structures that are
> embedded in those top-level request/TFM structures are sufficiently
> aligned so that any member C types appear at the expected alignment
> (but those structures are not usually defined as being aligned to
> ARCH_KMALLOC_ALIGN)

In this case, a void * array type would cover most structures that don't
have special alignment needs.

In both the above cases, we can get ARCH_KMALLOC_MINALIGN down to 8.

> - adding padding to ensure that these driver context structures do not
> share cache lines with the preceding top-level struct.
> 
> One thing to note here is that the padding is only necessary when the
> driver context size > 0, and has nothing to do with the alignment of
> the top-level struct.
> 
> Using a single aligned ctx member was a nice way to accomplish all of
> these when it was introduced, but I think it might be better to get
> rid of it, and move the padding logic to the static inline helpers
> instead.
> 
> So something like
> 
> struct skcipher_request {
>   ...
> } CRYPTO_MINALIGN_ATTR;
> 
> which states/ensures the alignment of the struct, and
> 
> void *skcipher_request_ctx(struct skcipher_request *req) {
>   return (void *)PTR_ALIGN(req + 1, ARCH_DMA_MINALIGN);
> }
> 
> to get at the context struct, instead of using a struct field.
> 
> Then, we could update skcipher_request_alloc() to only round up
> sizeof(struct skipher_request) to ARCH_DMA_MINALIGN if the reqsize is
> >0 to begin with, and if it is, to also round reqsize up to
> ARCH_DMA_MINALIGN when accessed. That way, we spell out the DMA
> padding requirements with relying on aligned struct members.

I think this should work and CRYPTO_MINALIGN can go down to whatever
ARCH_KMALLOC_MINALIGN without breaking the non-coherent DMA.

-- 
Catalin

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Tue, Apr 19, 2022 at 11:50:11PM +0200, Ard Biesheuvel wrote:
> On Mon, 18 Apr 2022 at 18:44, Catalin Marinas <catalin.marinas@arm.com> wrote:
> > On Mon, Apr 18, 2022 at 04:37:17PM +0800, Herbert Xu wrote:
> > > I've seen Ard's patches already and I think I understand what your
> > > needs are.  So let me whip up some code to show you guys what I
> > > think needs to be done.
> >
> > BTW before you have a go at this, there's also Linus' idea that does not
> > change the crypto code (at least not functionally). Of course, you and
> > Ard can still try to figure out how to reduce the padding but if we go
> > with Linus' idea of a new GFP_NODMA flag, there won't be any changes to
> > the crypto code as long as it doesn't pass such flag. So, the options:
> >
> > 1. Change ARCH_KMALLOC_MINALIGN to 8 (or ARCH_SLAB_MINALIGN if higher)
> >    while keeping ARCH_DMA_MINALIGN to 128. By default kmalloc() will
> >    honour the 128-byte alignment, unless GDP_NODMA is passed. This still
> >    requires changing CRYPTO_MINALIGN to ARCH_DMA_MINALIGN but there is
> >    no functional change, kmalloc() without the new flag will return
> >    CRYPTO_MINALIGN-aligned pointers.
> >
> > 2. Leave ARCH_KMALLOC_MINALIGN as ARCH_DMA_MINALIGN (128) and introduce
> >    a new GFP_PACKED (I think it fits better than 'NODMA') flag that
> >    reduces the minimum kmalloc() below ARCH_KMALLOC_MINALIGN (and
> >    probably at least ARCH_SLAB_MINALIGN). It's equivalent to (1) but
> >    does not touch the crypto code at all.
> >
> > (1) and (2) are the same, just minor naming difference. Happy to go with
> > any of them. They still have the downside that we need to add the new
> > GFP_ flag to those hotspots that allocate small objects (Arnd provided
> > an idea on how to find them with ftrace) but at least we know it won't
> > inadvertently break anything.
> 
> I'm not sure GFP_NODMA adds much here.

What it buys is that the crypto code won't need to be changed
immediately, so we can go ahead with the kmalloc() optimisation while
you and Herbert figure out how to refactor the crypto code.

Another option is to make the change but pass a new GFP_DMA_MINALIGN
flag to all kmalloc() calls in the crypto code (or a new dma_kmalloc()
that Linus suggested). This way the allocations in the crypto code are
guaranteed to be ARCH_DMA_MINALIGN (we'd still change CRYPTO_MINALIGN to
this).

> The way I see it, the issue in the crypto code is that we are relying
> on a ARCH_KMALLOC_ALIGN aligned zero length __ctx[] array for three
> different things:
> - ensuring/informing the compiler that top-level request/TFM
> structures are aligned to ARCH_KMALLOC_ALIGN,
> - adding padding to ensure that driver context structures that are
> embedded in those top-level request/TFM structures are sufficiently
> aligned so that any member C types appear at the expected alignment
> (but those structures are not usually defined as being aligned to
> ARCH_KMALLOC_ALIGN)

In this case, a void * array type would cover most structures that don't
have special alignment needs.

In both the above cases, we can get ARCH_KMALLOC_MINALIGN down to 8.

> - adding padding to ensure that these driver context structures do not
> share cache lines with the preceding top-level struct.
> 
> One thing to note here is that the padding is only necessary when the
> driver context size > 0, and has nothing to do with the alignment of
> the top-level struct.
> 
> Using a single aligned ctx member was a nice way to accomplish all of
> these when it was introduced, but I think it might be better to get
> rid of it, and move the padding logic to the static inline helpers
> instead.
> 
> So something like
> 
> struct skcipher_request {
>   ...
> } CRYPTO_MINALIGN_ATTR;
> 
> which states/ensures the alignment of the struct, and
> 
> void *skcipher_request_ctx(struct skcipher_request *req) {
>   return (void *)PTR_ALIGN(req + 1, ARCH_DMA_MINALIGN);
> }
> 
> to get at the context struct, instead of using a struct field.
> 
> Then, we could update skcipher_request_alloc() to only round up
> sizeof(struct skipher_request) to ARCH_DMA_MINALIGN if the reqsize is
> >0 to begin with, and if it is, to also round reqsize up to
> ARCH_DMA_MINALIGN when accessed. That way, we spell out the DMA
> padding requirements with relying on aligned struct members.

I think this should work and CRYPTO_MINALIGN can go down to whatever
ARCH_KMALLOC_MINALIGN without breaking the non-coherent DMA.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Arnd Bergmann]

On Tue, Apr 19, 2022 at 11:50 PM Ard Biesheuvel <ardb@kernel.org> wrote:
> On Mon, 18 Apr 2022 at 18:44, Catalin Marinas <catalin.marinas@arm.com> wrote:
> > On Mon, Apr 18, 2022 at 04:37:17PM +0800, Herbert Xu wrote:

> > BTW before you have a go at this, there's also Linus' idea that does not
> > change the crypto code (at least not functionally). Of course, you and
> > Ard can still try to figure out how to reduce the padding but if we go
> > with Linus' idea of a new GFP_NODMA flag, there won't be any changes to
> > the crypto code as long as it doesn't pass such flag. So, the options:
> >
> > 1. Change ARCH_KMALLOC_MINALIGN to 8 (or ARCH_SLAB_MINALIGN if higher)
> >    while keeping ARCH_DMA_MINALIGN to 128. By default kmalloc() will
> >    honour the 128-byte alignment, unless GDP_NODMA is passed. This still
> >    requires changing CRYPTO_MINALIGN to ARCH_DMA_MINALIGN but there is
> >    no functional change, kmalloc() without the new flag will return
> >    CRYPTO_MINALIGN-aligned pointers.
> >
> > 2. Leave ARCH_KMALLOC_MINALIGN as ARCH_DMA_MINALIGN (128) and introduce
> >    a new GFP_PACKED (I think it fits better than 'NODMA') flag that
> >    reduces the minimum kmalloc() below ARCH_KMALLOC_MINALIGN (and
> >    probably at least ARCH_SLAB_MINALIGN). It's equivalent to (1) but
> >    does not touch the crypto code at all.
> >
> > (1) and (2) are the same, just minor naming difference. Happy to go with
> > any of them. They still have the downside that we need to add the new
> > GFP_ flag to those hotspots that allocate small objects (Arnd provided
> > an idea on how to find them with ftrace) but at least we know it won't
> > inadvertently break anything.

Right, both of these seem reasonable to me.

> I'm not sure GFP_NODMA adds much here.
>
> The way I see it, the issue in the crypto code is that we are relying
> on a ARCH_KMALLOC_ALIGN aligned zero length __ctx[] array for three
> different things:
...

Right. So as long as the crypto subsystem has additional alignment
requirement, it won't benefit from GFP_NODMA. For everything else,
GFP_NODMA would however have a very direct and measuable
impact on memory consumption.

Your proposed changes to the crypto subsystem all seem helpful
as well, just mostly orthogonal to the savings elsewhere. I don't know
how much memory is permanently tied up in overaligned crypto
data structures, but my guess is that it's not a lot on most systems.

Improving the alignment for crypto would however likely help
with stack usage on on-stack structures, and with performance
when the amount of ctx memory to clear for each operation
becomes smaller.

       Arnd

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Arnd Bergmann]

On Tue, Apr 19, 2022 at 11:50 PM Ard Biesheuvel <ardb@kernel.org> wrote:
> On Mon, 18 Apr 2022 at 18:44, Catalin Marinas <catalin.marinas@arm.com> wrote:
> > On Mon, Apr 18, 2022 at 04:37:17PM +0800, Herbert Xu wrote:

> > BTW before you have a go at this, there's also Linus' idea that does not
> > change the crypto code (at least not functionally). Of course, you and
> > Ard can still try to figure out how to reduce the padding but if we go
> > with Linus' idea of a new GFP_NODMA flag, there won't be any changes to
> > the crypto code as long as it doesn't pass such flag. So, the options:
> >
> > 1. Change ARCH_KMALLOC_MINALIGN to 8 (or ARCH_SLAB_MINALIGN if higher)
> >    while keeping ARCH_DMA_MINALIGN to 128. By default kmalloc() will
> >    honour the 128-byte alignment, unless GDP_NODMA is passed. This still
> >    requires changing CRYPTO_MINALIGN to ARCH_DMA_MINALIGN but there is
> >    no functional change, kmalloc() without the new flag will return
> >    CRYPTO_MINALIGN-aligned pointers.
> >
> > 2. Leave ARCH_KMALLOC_MINALIGN as ARCH_DMA_MINALIGN (128) and introduce
> >    a new GFP_PACKED (I think it fits better than 'NODMA') flag that
> >    reduces the minimum kmalloc() below ARCH_KMALLOC_MINALIGN (and
> >    probably at least ARCH_SLAB_MINALIGN). It's equivalent to (1) but
> >    does not touch the crypto code at all.
> >
> > (1) and (2) are the same, just minor naming difference. Happy to go with
> > any of them. They still have the downside that we need to add the new
> > GFP_ flag to those hotspots that allocate small objects (Arnd provided
> > an idea on how to find them with ftrace) but at least we know it won't
> > inadvertently break anything.

Right, both of these seem reasonable to me.

> I'm not sure GFP_NODMA adds much here.
>
> The way I see it, the issue in the crypto code is that we are relying
> on a ARCH_KMALLOC_ALIGN aligned zero length __ctx[] array for three
> different things:
...

Right. So as long as the crypto subsystem has additional alignment
requirement, it won't benefit from GFP_NODMA. For everything else,
GFP_NODMA would however have a very direct and measuable
impact on memory consumption.

Your proposed changes to the crypto subsystem all seem helpful
as well, just mostly orthogonal to the savings elsewhere. I don't know
how much memory is permanently tied up in overaligned crypto
data structures, but my guess is that it's not a lot on most systems.

Improving the alignment for crypto would however likely help
with stack usage on on-stack structures, and with performance
when the amount of ctx memory to clear for each operation
becomes smaller.

       Arnd

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Thu, Apr 14, 2022 at 03:25:59PM -0700, Linus Torvalds wrote:
> On Thu, Apr 14, 2022 at 12:49 PM Catalin Marinas
> <catalin.marinas@arm.com> wrote:
> > It's a lot worse, ARCH_KMALLOC_MINALIGN is currently 128 bytes on arm64.
> > I want to at least get it down to 64 with this series while preserving
> > the current kmalloc() semantics.
> 
> So here's a thought - maybe we could do the reverse of GFP_DMA, and
> add a flag to the places that want small allocations and know they
> don't need DMA?

Quick diff below. I'll test it some more with all sl*b and post a proper
patch with description tomorrow. But the basic idea is that
ARCH_KMALLOC_MINALIGN remains the same as ARCH_DMA_MINALIGN so that I
don't have to change existing users. KMALLOC_MIN_SIZE is decoupled from
ARCH_KMALLOC_MINALIGN and now we have caches all the way to kmalloc-8
(with slub). Callers would have to pass __GFP_PACKED to get an object
with alignment below ARCH_KMALLOC_MINALIGN.

Without any kmalloc() callers modified, the kmalloc caches look like
(only booted in a VM, not much activity):

kmalloc-128        12055  12096    128   32
kmalloc-96             0      0     96   42
kmalloc-64             0      0     64   64
kmalloc-32             0      0     32  128
kmalloc-16             0      0     16  256
kmalloc-8              0      0      8  512

With kstrdup() modified to pass __GFP_PACKED (as per the last hunk in
the diff below), I get just after boot:

kmalloc-128         8966   9056    128   32
kmalloc-96             0      0     96   42
kmalloc-64           192    192     64   64
kmalloc-32           768    768     32  128
kmalloc-16          2048   2048     16  256
kmalloc-8           2560   2560      8  512

So that's probably the simplest approach and using the ftrace histogram
we can add the flag to more places.

--------------------8<-----------------------------------------------
diff --git a/include/linux/gfp.h b/include/linux/gfp.h
index 761f8f1885c7..7c9f47ef3a53 100644
--- a/include/linux/gfp.h
+++ b/include/linux/gfp.h
@@ -63,8 +63,9 @@ struct vm_area_struct;
 #define ___GFP_SKIP_KASAN_UNPOISON	0
 #define ___GFP_SKIP_KASAN_POISON	0
 #endif
+#define ___GFP_PACKED		0x8000000u
 #ifdef CONFIG_LOCKDEP
-#define ___GFP_NOLOCKDEP	0x8000000u
+#define ___GFP_NOLOCKDEP	0x10000000u
 #else
 #define ___GFP_NOLOCKDEP	0
 #endif
@@ -251,6 +252,10 @@ struct vm_area_struct;
  *
  * %__GFP_SKIP_KASAN_POISON makes KASAN skip poisoning on page deallocation.
  * Typically, used for userspace pages. Only effective in HW_TAGS mode.
+ *
+ * %__GFP_PACKED returns a pointer aligned to the smaller KMALLOC_MIN_SIZE
+ * rather than ARCH_KMALLOC_MINALIGN. Beneficial for small object allocation
+ * on architectures that define ARCH_DMA_MINALIGN.
  */
 #define __GFP_NOWARN	((__force gfp_t)___GFP_NOWARN)
 #define __GFP_COMP	((__force gfp_t)___GFP_COMP)
@@ -259,12 +264,13 @@ struct vm_area_struct;
 #define __GFP_SKIP_ZERO ((__force gfp_t)___GFP_SKIP_ZERO)
 #define __GFP_SKIP_KASAN_UNPOISON ((__force gfp_t)___GFP_SKIP_KASAN_UNPOISON)
 #define __GFP_SKIP_KASAN_POISON   ((__force gfp_t)___GFP_SKIP_KASAN_POISON)
+#define __GFP_PACKED	((__force gfp_t)___GFP_PACKED)
 
 /* Disable lockdep for GFP context tracking */
 #define __GFP_NOLOCKDEP ((__force gfp_t)___GFP_NOLOCKDEP)
 
 /* Room for N __GFP_FOO bits */
-#define __GFP_BITS_SHIFT (27 + IS_ENABLED(CONFIG_LOCKDEP))
+#define __GFP_BITS_SHIFT (28 + IS_ENABLED(CONFIG_LOCKDEP))
 #define __GFP_BITS_MASK ((__force gfp_t)((1 << __GFP_BITS_SHIFT) - 1))
 
 /**
diff --git a/include/linux/slab.h b/include/linux/slab.h
index 373b3ef99f4e..7bd3a33cdb9d 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -194,8 +194,6 @@ void kmem_dump_obj(void *object);
  */
 #if defined(ARCH_DMA_MINALIGN) && ARCH_DMA_MINALIGN > 8
 #define ARCH_KMALLOC_MINALIGN ARCH_DMA_MINALIGN
-#define KMALLOC_MIN_SIZE ARCH_DMA_MINALIGN
-#define KMALLOC_SHIFT_LOW ilog2(ARCH_DMA_MINALIGN)
 #else
 #define ARCH_KMALLOC_MINALIGN __alignof__(unsigned long long)
 #endif
@@ -364,12 +362,14 @@ static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags)
  * Callers where !size_is_constant should only be test modules, where runtime
  * overheads of __kmalloc_index() can be tolerated.  Also see kmalloc_slab().
  */
-static __always_inline unsigned int __kmalloc_index(size_t size,
+static __always_inline unsigned int __kmalloc_index(size_t size, gfp_t flags,
 						    bool size_is_constant)
 {
 	if (!size)
 		return 0;
 
+	if (ARCH_KMALLOC_MINALIGN > KMALLOC_MIN_SIZE && !(flags & __GFP_PACKED))
+		size = ALIGN(size, ARCH_KMALLOC_MINALIGN);
 	if (size <= KMALLOC_MIN_SIZE)
 		return KMALLOC_SHIFT_LOW;
 
@@ -409,7 +409,7 @@ static __always_inline unsigned int __kmalloc_index(size_t size,
 	/* Will never be reached. Needed because the compiler may complain */
 	return -1;
 }
-#define kmalloc_index(s) __kmalloc_index(s, true)
+#define kmalloc_index(s, f) __kmalloc_index(s, f, true)
 #endif /* !CONFIG_SLOB */
 
 void *__kmalloc(size_t size, gfp_t flags) __assume_kmalloc_alignment __alloc_size(1);
@@ -573,7 +573,7 @@ static __always_inline __alloc_size(1) void *kmalloc(size_t size, gfp_t flags)
 		if (size > KMALLOC_MAX_CACHE_SIZE)
 			return kmalloc_large(size, flags);
 #ifndef CONFIG_SLOB
-		index = kmalloc_index(size);
+		index = kmalloc_index(size, flags);
 
 		if (!index)
 			return ZERO_SIZE_PTR;
@@ -591,7 +591,7 @@ static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t fla
 #ifndef CONFIG_SLOB
 	if (__builtin_constant_p(size) &&
 		size <= KMALLOC_MAX_CACHE_SIZE) {
-		unsigned int i = kmalloc_index(size);
+		unsigned int i = kmalloc_index(size, flags);
 
 		if (!i)
 			return ZERO_SIZE_PTR;
diff --git a/mm/slab_common.c b/mm/slab_common.c
index 6ee64d6208b3..d5da402c8aae 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -630,7 +630,7 @@ void __init create_boot_cache(struct kmem_cache *s, const char *name,
 		unsigned int useroffset, unsigned int usersize)
 {
 	int err;
-	unsigned int align = ARCH_KMALLOC_MINALIGN;
+	unsigned int align = KMALLOC_MIN_SIZE;
 
 	s->name = name;
 	s->size = s->object_size = size;
@@ -722,6 +722,8 @@ struct kmem_cache *kmalloc_slab(size_t size, gfp_t flags)
 {
 	unsigned int index;
 
+	if (ARCH_KMALLOC_MINALIGN > KMALLOC_MIN_SIZE && !(flags & __GFP_PACKED))
+		size = ALIGN(size, ARCH_KMALLOC_MINALIGN);
 	if (size <= 192) {
 		if (!size)
 			return ZERO_SIZE_PTR;
diff --git a/mm/util.c b/mm/util.c
index 54e5e761a9a9..27eb1731de27 100644
--- a/mm/util.c
+++ b/mm/util.c
@@ -57,7 +57,7 @@ char *kstrdup(const char *s, gfp_t gfp)
 		return NULL;
 
 	len = strlen(s) + 1;
-	buf = kmalloc_track_caller(len, gfp);
+	buf = kmalloc_track_caller(len, gfp | __GFP_PACKED);
 	if (buf)
 		memcpy(buf, s, len);
 	return buf;

-- 
Catalin

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Thu, Apr 14, 2022 at 03:25:59PM -0700, Linus Torvalds wrote:
> On Thu, Apr 14, 2022 at 12:49 PM Catalin Marinas
> <catalin.marinas@arm.com> wrote:
> > It's a lot worse, ARCH_KMALLOC_MINALIGN is currently 128 bytes on arm64.
> > I want to at least get it down to 64 with this series while preserving
> > the current kmalloc() semantics.
> 
> So here's a thought - maybe we could do the reverse of GFP_DMA, and
> add a flag to the places that want small allocations and know they
> don't need DMA?

Quick diff below. I'll test it some more with all sl*b and post a proper
patch with description tomorrow. But the basic idea is that
ARCH_KMALLOC_MINALIGN remains the same as ARCH_DMA_MINALIGN so that I
don't have to change existing users. KMALLOC_MIN_SIZE is decoupled from
ARCH_KMALLOC_MINALIGN and now we have caches all the way to kmalloc-8
(with slub). Callers would have to pass __GFP_PACKED to get an object
with alignment below ARCH_KMALLOC_MINALIGN.

Without any kmalloc() callers modified, the kmalloc caches look like
(only booted in a VM, not much activity):

kmalloc-128        12055  12096    128   32
kmalloc-96             0      0     96   42
kmalloc-64             0      0     64   64
kmalloc-32             0      0     32  128
kmalloc-16             0      0     16  256
kmalloc-8              0      0      8  512

With kstrdup() modified to pass __GFP_PACKED (as per the last hunk in
the diff below), I get just after boot:

kmalloc-128         8966   9056    128   32
kmalloc-96             0      0     96   42
kmalloc-64           192    192     64   64
kmalloc-32           768    768     32  128
kmalloc-16          2048   2048     16  256
kmalloc-8           2560   2560      8  512

So that's probably the simplest approach and using the ftrace histogram
we can add the flag to more places.

--------------------8<-----------------------------------------------
diff --git a/include/linux/gfp.h b/include/linux/gfp.h
index 761f8f1885c7..7c9f47ef3a53 100644
--- a/include/linux/gfp.h
+++ b/include/linux/gfp.h
@@ -63,8 +63,9 @@ struct vm_area_struct;
 #define ___GFP_SKIP_KASAN_UNPOISON	0
 #define ___GFP_SKIP_KASAN_POISON	0
 #endif
+#define ___GFP_PACKED		0x8000000u
 #ifdef CONFIG_LOCKDEP
-#define ___GFP_NOLOCKDEP	0x8000000u
+#define ___GFP_NOLOCKDEP	0x10000000u
 #else
 #define ___GFP_NOLOCKDEP	0
 #endif
@@ -251,6 +252,10 @@ struct vm_area_struct;
  *
  * %__GFP_SKIP_KASAN_POISON makes KASAN skip poisoning on page deallocation.
  * Typically, used for userspace pages. Only effective in HW_TAGS mode.
+ *
+ * %__GFP_PACKED returns a pointer aligned to the smaller KMALLOC_MIN_SIZE
+ * rather than ARCH_KMALLOC_MINALIGN. Beneficial for small object allocation
+ * on architectures that define ARCH_DMA_MINALIGN.
  */
 #define __GFP_NOWARN	((__force gfp_t)___GFP_NOWARN)
 #define __GFP_COMP	((__force gfp_t)___GFP_COMP)
@@ -259,12 +264,13 @@ struct vm_area_struct;
 #define __GFP_SKIP_ZERO ((__force gfp_t)___GFP_SKIP_ZERO)
 #define __GFP_SKIP_KASAN_UNPOISON ((__force gfp_t)___GFP_SKIP_KASAN_UNPOISON)
 #define __GFP_SKIP_KASAN_POISON   ((__force gfp_t)___GFP_SKIP_KASAN_POISON)
+#define __GFP_PACKED	((__force gfp_t)___GFP_PACKED)
 
 /* Disable lockdep for GFP context tracking */
 #define __GFP_NOLOCKDEP ((__force gfp_t)___GFP_NOLOCKDEP)
 
 /* Room for N __GFP_FOO bits */
-#define __GFP_BITS_SHIFT (27 + IS_ENABLED(CONFIG_LOCKDEP))
+#define __GFP_BITS_SHIFT (28 + IS_ENABLED(CONFIG_LOCKDEP))
 #define __GFP_BITS_MASK ((__force gfp_t)((1 << __GFP_BITS_SHIFT) - 1))
 
 /**
diff --git a/include/linux/slab.h b/include/linux/slab.h
index 373b3ef99f4e..7bd3a33cdb9d 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -194,8 +194,6 @@ void kmem_dump_obj(void *object);
  */
 #if defined(ARCH_DMA_MINALIGN) && ARCH_DMA_MINALIGN > 8
 #define ARCH_KMALLOC_MINALIGN ARCH_DMA_MINALIGN
-#define KMALLOC_MIN_SIZE ARCH_DMA_MINALIGN
-#define KMALLOC_SHIFT_LOW ilog2(ARCH_DMA_MINALIGN)
 #else
 #define ARCH_KMALLOC_MINALIGN __alignof__(unsigned long long)
 #endif
@@ -364,12 +362,14 @@ static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags)
  * Callers where !size_is_constant should only be test modules, where runtime
  * overheads of __kmalloc_index() can be tolerated.  Also see kmalloc_slab().
  */
-static __always_inline unsigned int __kmalloc_index(size_t size,
+static __always_inline unsigned int __kmalloc_index(size_t size, gfp_t flags,
 						    bool size_is_constant)
 {
 	if (!size)
 		return 0;
 
+	if (ARCH_KMALLOC_MINALIGN > KMALLOC_MIN_SIZE && !(flags & __GFP_PACKED))
+		size = ALIGN(size, ARCH_KMALLOC_MINALIGN);
 	if (size <= KMALLOC_MIN_SIZE)
 		return KMALLOC_SHIFT_LOW;
 
@@ -409,7 +409,7 @@ static __always_inline unsigned int __kmalloc_index(size_t size,
 	/* Will never be reached. Needed because the compiler may complain */
 	return -1;
 }
-#define kmalloc_index(s) __kmalloc_index(s, true)
+#define kmalloc_index(s, f) __kmalloc_index(s, f, true)
 #endif /* !CONFIG_SLOB */
 
 void *__kmalloc(size_t size, gfp_t flags) __assume_kmalloc_alignment __alloc_size(1);
@@ -573,7 +573,7 @@ static __always_inline __alloc_size(1) void *kmalloc(size_t size, gfp_t flags)
 		if (size > KMALLOC_MAX_CACHE_SIZE)
 			return kmalloc_large(size, flags);
 #ifndef CONFIG_SLOB
-		index = kmalloc_index(size);
+		index = kmalloc_index(size, flags);
 
 		if (!index)
 			return ZERO_SIZE_PTR;
@@ -591,7 +591,7 @@ static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t fla
 #ifndef CONFIG_SLOB
 	if (__builtin_constant_p(size) &&
 		size <= KMALLOC_MAX_CACHE_SIZE) {
-		unsigned int i = kmalloc_index(size);
+		unsigned int i = kmalloc_index(size, flags);
 
 		if (!i)
 			return ZERO_SIZE_PTR;
diff --git a/mm/slab_common.c b/mm/slab_common.c
index 6ee64d6208b3..d5da402c8aae 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -630,7 +630,7 @@ void __init create_boot_cache(struct kmem_cache *s, const char *name,
 		unsigned int useroffset, unsigned int usersize)
 {
 	int err;
-	unsigned int align = ARCH_KMALLOC_MINALIGN;
+	unsigned int align = KMALLOC_MIN_SIZE;
 
 	s->name = name;
 	s->size = s->object_size = size;
@@ -722,6 +722,8 @@ struct kmem_cache *kmalloc_slab(size_t size, gfp_t flags)
 {
 	unsigned int index;
 
+	if (ARCH_KMALLOC_MINALIGN > KMALLOC_MIN_SIZE && !(flags & __GFP_PACKED))
+		size = ALIGN(size, ARCH_KMALLOC_MINALIGN);
 	if (size <= 192) {
 		if (!size)
 			return ZERO_SIZE_PTR;
diff --git a/mm/util.c b/mm/util.c
index 54e5e761a9a9..27eb1731de27 100644
--- a/mm/util.c
+++ b/mm/util.c
@@ -57,7 +57,7 @@ char *kstrdup(const char *s, gfp_t gfp)
 		return NULL;
 
 	len = strlen(s) + 1;
-	buf = kmalloc_track_caller(len, gfp);
+	buf = kmalloc_track_caller(len, gfp | __GFP_PACKED);
 	if (buf)
 		memcpy(buf, s, len);
 	return buf;

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Linus Torvalds]

On Wed, Apr 20, 2022 at 12:08 PM Catalin Marinas
<catalin.marinas@arm.com> wrote:
>
> With kstrdup() modified to pass __GFP_PACKED (as per the last hunk in
> the diff below), I get just after boot:
>
> kmalloc-128         8966   9056    128   32
> kmalloc-96             0      0     96   42
> kmalloc-64           192    192     64   64
> kmalloc-32           768    768     32  128
> kmalloc-16          2048   2048     16  256
> kmalloc-8           2560   2560      8  512
>
> So that's probably the simplest approach and using the ftrace histogram
> we can add the flag to more places.

I agree that this seems to be the safest thing to do, and maybe
__GFP_PACKED is a better flag name than __GFP_NODMA.

That said, It worries me a bit in that to me "PACKED" implies "no
alignment at all". And I could _easily_ see people still wanting to do
8-byte allocations that have 8-byte alignment because it's some kind
of private pointer thing or whatever.

For "kstrdup()", a flag like __GFP_PACKED makes 100% sense, since it
literally wants byte alignment.

But what about those "random small structures" cases?

Anyway, I'm perfectly happy calling it __GFP_PACKED, but at a minimum
document that "packed" in this case still means "__alignof__(unsigned
long long)" or something like that?

                      Linus

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Linus Torvalds]

On Wed, Apr 20, 2022 at 12:08 PM Catalin Marinas
<catalin.marinas@arm.com> wrote:
>
> With kstrdup() modified to pass __GFP_PACKED (as per the last hunk in
> the diff below), I get just after boot:
>
> kmalloc-128         8966   9056    128   32
> kmalloc-96             0      0     96   42
> kmalloc-64           192    192     64   64
> kmalloc-32           768    768     32  128
> kmalloc-16          2048   2048     16  256
> kmalloc-8           2560   2560      8  512
>
> So that's probably the simplest approach and using the ftrace histogram
> we can add the flag to more places.

I agree that this seems to be the safest thing to do, and maybe
__GFP_PACKED is a better flag name than __GFP_NODMA.

That said, It worries me a bit in that to me "PACKED" implies "no
alignment at all". And I could _easily_ see people still wanting to do
8-byte allocations that have 8-byte alignment because it's some kind
of private pointer thing or whatever.

For "kstrdup()", a flag like __GFP_PACKED makes 100% sense, since it
literally wants byte alignment.

But what about those "random small structures" cases?

Anyway, I'm perfectly happy calling it __GFP_PACKED, but at a minimum
document that "packed" in this case still means "__alignof__(unsigned
long long)" or something like that?

                      Linus

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Christoph Hellwig]

Btw, there is another option:  Most real systems already require having
swiotlb to bounce buffer in some cases.  We could simply force bounce
buffering in the dma mapping code for too small or not properly aligned
transfers and just decrease the dma alignment.

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Christoph Hellwig]

Btw, there is another option:  Most real systems already require having
swiotlb to bounce buffer in some cases.  We could simply force bounce
buffering in the dma mapping code for too small or not properly aligned
transfers and just decrease the dma alignment.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Arnd Bergmann]

On Thu, Apr 21, 2022 at 9:20 AM Christoph Hellwig <hch@infradead.org> wrote:
>
> Btw, there is another option:  Most real systems already require having
> swiotlb to bounce buffer in some cases.  We could simply force bounce
> buffering in the dma mapping code for too small or not properly aligned
> transfers and just decrease the dma alignment.

I like the idea because these days we already rely on bounce buffering
for sub-page buffers in many iommu based cases for strict isolation
purposes, as well as most 64-bit machines that lack an iommu.

Does this work on all 32-bit architectures as well? I see that you added
swiotlb for ARM LPASE systems in 2019, but I don't know if that has any
additional requirements for the other 32-bit architectures that don't
select SWIOTLB today.

      Arnd

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Arnd Bergmann]

On Thu, Apr 21, 2022 at 9:20 AM Christoph Hellwig <hch@infradead.org> wrote:
>
> Btw, there is another option:  Most real systems already require having
> swiotlb to bounce buffer in some cases.  We could simply force bounce
> buffering in the dma mapping code for too small or not properly aligned
> transfers and just decrease the dma alignment.

I like the idea because these days we already rely on bounce buffering
for sub-page buffers in many iommu based cases for strict isolation
purposes, as well as most 64-bit machines that lack an iommu.

Does this work on all 32-bit architectures as well? I see that you added
swiotlb for ARM LPASE systems in 2019, but I don't know if that has any
additional requirements for the other 32-bit architectures that don't
select SWIOTLB today.

      Arnd

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Christoph Hellwig]

On Thu, Apr 21, 2022 at 09:36:46AM +0200, Arnd Bergmann wrote:
> Does this work on all 32-bit architectures as well? I see that you added
> swiotlb for ARM LPASE systems in 2019, but I don't know if that has any
> additional requirements for the other 32-bit architectures that don't
> select SWIOTLB today.

We'll need to call swiotlb_init for all these cases, but there is no other
fundamental requirement.  We can probably do with a way smaller buffer
if bouncing is only needed for misaligned allocations.

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Christoph Hellwig]

On Thu, Apr 21, 2022 at 09:36:46AM +0200, Arnd Bergmann wrote:
> Does this work on all 32-bit architectures as well? I see that you added
> swiotlb for ARM LPASE systems in 2019, but I don't know if that has any
> additional requirements for the other 32-bit architectures that don't
> select SWIOTLB today.

We'll need to call swiotlb_init for all these cases, but there is no other
fundamental requirement.  We can probably do with a way smaller buffer
if bouncing is only needed for misaligned allocations.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Ard Biesheuvel]

On Thu, 21 Apr 2022 at 09:20, Christoph Hellwig <hch@infradead.org> wrote:
>
> Btw, there is another option:  Most real systems already require having
> swiotlb to bounce buffer in some cases.  We could simply force bounce
> buffering in the dma mapping code for too small or not properly aligned
> transfers and just decrease the dma alignment.

Strongly agree. As I pointed out before, we'd only need to do this for
misaligned, non-cache coherent inbound DMA, and we'd only have to
worry about performance regressions, not data corruption issues. And
given the natural alignment of block I/O, and the fact that network
drivers typically allocate and map their own RX buffers (which means
they could reasonably be fixed if a performance bottleneck pops up), I
think the risk for showstopper performance regressions is likely to be
acceptable.

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Ard Biesheuvel]

On Thu, 21 Apr 2022 at 09:20, Christoph Hellwig <hch@infradead.org> wrote:
>
> Btw, there is another option:  Most real systems already require having
> swiotlb to bounce buffer in some cases.  We could simply force bounce
> buffering in the dma mapping code for too small or not properly aligned
> transfers and just decrease the dma alignment.

Strongly agree. As I pointed out before, we'd only need to do this for
misaligned, non-cache coherent inbound DMA, and we'd only have to
worry about performance regressions, not data corruption issues. And
given the natural alignment of block I/O, and the fact that network
drivers typically allocate and map their own RX buffers (which means
they could reasonably be fixed if a performance bottleneck pops up), I
think the risk for showstopper performance regressions is likely to be
acceptable.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Thu, Apr 21, 2022 at 12:20:22AM -0700, Christoph Hellwig wrote:
> Btw, there is another option:  Most real systems already require having
> swiotlb to bounce buffer in some cases.  We could simply force bounce
> buffering in the dma mapping code for too small or not properly aligned
> transfers and just decrease the dma alignment.

We can force bounce if size is small but checking the alignment is
trickier. Normally the beginning of the buffer is aligned but the end is
at some sizeof() distance. We need to know whether the end is in a
kmalloc-128 cache and that requires reaching out to the slab internals.
That's doable and not expensive but it needs to be done for every small
size getting to the DMA API, something like (for mm/slub.c):

	folio = virt_to_folio(x);
	slab = folio_slab(folio);
	if (slab->slab_cache->align < ARCH_DMA_MINALIGN)
		... bounce ...

(and a bit different for mm/slab.c)

If we scrap ARCH_DMA_MINALIGN altogether from arm64, we can check the
alignment against cache_line_size(), though I'd rather keep it for code
that wants to avoid bouncing and goes for this compile-time alignment.

I think we are down to four options (1 and 2 can be combined):

1. ARCH_DMA_MINALIGN == 128, dynamic arch_kmalloc_minalign() to reduce
   kmalloc() alignment to 64 on most arm64 SoC - this series.

2. ARCH_DMA_MINALIGN == 128, ARCH_KMALLOC_MINALIGN == 128, add explicit
   __GFP_PACKED for small allocations. It can be combined with (1) so
   that allocations without __GFP_PACKED can still get 64-byte
   alignment.

3. ARCH_DMA_MINALIGN == 128, ARCH_KMALLOC_MINALIGN == 8, swiotlb bounce.

4. undef ARCH_DMA_MINALIGN, ARCH_KMALLOC_MINALIGN == 8, swiotlb bounce.

(3) and (4) don't require histogram analysis. Between them, I have a
preference for (3) as it gives drivers a chance to avoid the bounce.

If (2) is feasible, we don't need to bother with any bouncing or
structure alignments, it's an opt-in by the driver/subsystem. However,
it may be tedious to analyse the hot spots. While there are a few
obvious places (kstrdup), I don't have access to a multitude of devices
that may exercise the drivers and subsystems.

With (3) the risk is someone complaining about performance or even
running out of swiotlb space on some SoCs (I guess the fall-back can be
another kmalloc() with an appropriate size).

I guess we can limit the choice to either (2) or (3). I have (2) already
(needs some more testing). I can attempt (3) and try to run it on some
real hardware to see the perf impact.

-- 
Catalin

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Thu, Apr 21, 2022 at 12:20:22AM -0700, Christoph Hellwig wrote:
> Btw, there is another option:  Most real systems already require having
> swiotlb to bounce buffer in some cases.  We could simply force bounce
> buffering in the dma mapping code for too small or not properly aligned
> transfers and just decrease the dma alignment.

We can force bounce if size is small but checking the alignment is
trickier. Normally the beginning of the buffer is aligned but the end is
at some sizeof() distance. We need to know whether the end is in a
kmalloc-128 cache and that requires reaching out to the slab internals.
That's doable and not expensive but it needs to be done for every small
size getting to the DMA API, something like (for mm/slub.c):

	folio = virt_to_folio(x);
	slab = folio_slab(folio);
	if (slab->slab_cache->align < ARCH_DMA_MINALIGN)
		... bounce ...

(and a bit different for mm/slab.c)

If we scrap ARCH_DMA_MINALIGN altogether from arm64, we can check the
alignment against cache_line_size(), though I'd rather keep it for code
that wants to avoid bouncing and goes for this compile-time alignment.

I think we are down to four options (1 and 2 can be combined):

1. ARCH_DMA_MINALIGN == 128, dynamic arch_kmalloc_minalign() to reduce
   kmalloc() alignment to 64 on most arm64 SoC - this series.

2. ARCH_DMA_MINALIGN == 128, ARCH_KMALLOC_MINALIGN == 128, add explicit
   __GFP_PACKED for small allocations. It can be combined with (1) so
   that allocations without __GFP_PACKED can still get 64-byte
   alignment.

3. ARCH_DMA_MINALIGN == 128, ARCH_KMALLOC_MINALIGN == 8, swiotlb bounce.

4. undef ARCH_DMA_MINALIGN, ARCH_KMALLOC_MINALIGN == 8, swiotlb bounce.

(3) and (4) don't require histogram analysis. Between them, I have a
preference for (3) as it gives drivers a chance to avoid the bounce.

If (2) is feasible, we don't need to bother with any bouncing or
structure alignments, it's an opt-in by the driver/subsystem. However,
it may be tedious to analyse the hot spots. While there are a few
obvious places (kstrdup), I don't have access to a multitude of devices
that may exercise the drivers and subsystems.

With (3) the risk is someone complaining about performance or even
running out of swiotlb space on some SoCs (I guess the fall-back can be
another kmalloc() with an appropriate size).

I guess we can limit the choice to either (2) or (3). I have (2) already
(needs some more testing). I can attempt (3) and try to run it on some
real hardware to see the perf impact.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Arnd Bergmann]

On Thu, Apr 21, 2022 at 1:06 PM Catalin Marinas <catalin.marinas@arm.com> wrote:
>
> On Thu, Apr 21, 2022 at 12:20:22AM -0700, Christoph Hellwig wrote:
> > Btw, there is another option:  Most real systems already require having
> > swiotlb to bounce buffer in some cases.  We could simply force bounce
> > buffering in the dma mapping code for too small or not properly aligned
> > transfers and just decrease the dma alignment.
>
> We can force bounce if size is small but checking the alignment is
> trickier. Normally the beginning of the buffer is aligned but the end is
> at some sizeof() distance. We need to know whether the end is in a
> kmalloc-128 cache and that requires reaching out to the slab internals.
> That's doable and not expensive but it needs to be done for every small
> size getting to the DMA API, something like (for mm/slub.c):
>
>         folio = virt_to_folio(x);
>         slab = folio_slab(folio);
>         if (slab->slab_cache->align < ARCH_DMA_MINALIGN)
>                 ... bounce ...
>
> (and a bit different for mm/slab.c)

I think the decision to bounce or not can be based on the actual
cache line size at runtime, so most commonly 64 bytes on arm64,
even though the compile-time limit is 128 bytes.

We also know that larger slabs are all cacheline aligned, so simply
comparing the transfer size is enough to rule out most, in this case
any transfer larger than 96 bytes must come from the kmalloc-128
or larger cache, so that works like before.

For transfers <=96 bytes, the possibilities are:

1.kmalloc-32 or smaller, always needs to bounce
2. kmalloc-96, but at least one byte in partial cache line,
    need to bounce
3. kmalloc-64, may skip the bounce.
4. kmalloc-128 or larger, or not a slab cache but a partial
    transfer, may skip the bounce.

I would guess that the first case is the most common here,
so unless bouncing one or two cache lines is extremely
expensive, I don't expect it to be worth optimizing for the latter
two cases.

       Arnd

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Arnd Bergmann]

On Thu, Apr 21, 2022 at 1:06 PM Catalin Marinas <catalin.marinas@arm.com> wrote:
>
> On Thu, Apr 21, 2022 at 12:20:22AM -0700, Christoph Hellwig wrote:
> > Btw, there is another option:  Most real systems already require having
> > swiotlb to bounce buffer in some cases.  We could simply force bounce
> > buffering in the dma mapping code for too small or not properly aligned
> > transfers and just decrease the dma alignment.
>
> We can force bounce if size is small but checking the alignment is
> trickier. Normally the beginning of the buffer is aligned but the end is
> at some sizeof() distance. We need to know whether the end is in a
> kmalloc-128 cache and that requires reaching out to the slab internals.
> That's doable and not expensive but it needs to be done for every small
> size getting to the DMA API, something like (for mm/slub.c):
>
>         folio = virt_to_folio(x);
>         slab = folio_slab(folio);
>         if (slab->slab_cache->align < ARCH_DMA_MINALIGN)
>                 ... bounce ...
>
> (and a bit different for mm/slab.c)

I think the decision to bounce or not can be based on the actual
cache line size at runtime, so most commonly 64 bytes on arm64,
even though the compile-time limit is 128 bytes.

We also know that larger slabs are all cacheline aligned, so simply
comparing the transfer size is enough to rule out most, in this case
any transfer larger than 96 bytes must come from the kmalloc-128
or larger cache, so that works like before.

For transfers <=96 bytes, the possibilities are:

1.kmalloc-32 or smaller, always needs to bounce
2. kmalloc-96, but at least one byte in partial cache line,
    need to bounce
3. kmalloc-64, may skip the bounce.
4. kmalloc-128 or larger, or not a slab cache but a partial
    transfer, may skip the bounce.

I would guess that the first case is the most common here,
so unless bouncing one or two cache lines is extremely
expensive, I don't expect it to be worth optimizing for the latter
two cases.

       Arnd

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Thu, Apr 21, 2022 at 02:28:45PM +0200, Arnd Bergmann wrote:
> On Thu, Apr 21, 2022 at 1:06 PM Catalin Marinas <catalin.marinas@arm.com> wrote:
> > On Thu, Apr 21, 2022 at 12:20:22AM -0700, Christoph Hellwig wrote:
> > > Btw, there is another option:  Most real systems already require having
> > > swiotlb to bounce buffer in some cases.  We could simply force bounce
> > > buffering in the dma mapping code for too small or not properly aligned
> > > transfers and just decrease the dma alignment.
> >
> > We can force bounce if size is small but checking the alignment is
> > trickier. Normally the beginning of the buffer is aligned but the end is
> > at some sizeof() distance. We need to know whether the end is in a
> > kmalloc-128 cache and that requires reaching out to the slab internals.
> > That's doable and not expensive but it needs to be done for every small
> > size getting to the DMA API, something like (for mm/slub.c):
> >
> >         folio = virt_to_folio(x);
> >         slab = folio_slab(folio);
> >         if (slab->slab_cache->align < ARCH_DMA_MINALIGN)
> >                 ... bounce ...
> >
> > (and a bit different for mm/slab.c)
> 
> I think the decision to bounce or not can be based on the actual
> cache line size at runtime, so most commonly 64 bytes on arm64,
> even though the compile-time limit is 128 bytes.
> 
> We also know that larger slabs are all cacheline aligned, so simply
> comparing the transfer size is enough to rule out most, in this case
> any transfer larger than 96 bytes must come from the kmalloc-128
> or larger cache, so that works like before.

There's also the case with 128-byte cache lines and kmalloc-192.

> For transfers <=96 bytes, the possibilities are:
> 
> 1.kmalloc-32 or smaller, always needs to bounce
> 2. kmalloc-96, but at least one byte in partial cache line,
>     need to bounce
> 3. kmalloc-64, may skip the bounce.
> 4. kmalloc-128 or larger, or not a slab cache but a partial
>     transfer, may skip the bounce.
> 
> I would guess that the first case is the most common here,
> so unless bouncing one or two cache lines is extremely
> expensive, I don't expect it to be worth optimizing for the latter
> two cases.

I think so. If someone complains of a performance regression, we can
look at optimising the bounce. I have a suspicion the cost of copying
two cache lines is small compared to swiotlb_find_slots() etc.

-- 
Catalin

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Thu, Apr 21, 2022 at 02:28:45PM +0200, Arnd Bergmann wrote:
> On Thu, Apr 21, 2022 at 1:06 PM Catalin Marinas <catalin.marinas@arm.com> wrote:
> > On Thu, Apr 21, 2022 at 12:20:22AM -0700, Christoph Hellwig wrote:
> > > Btw, there is another option:  Most real systems already require having
> > > swiotlb to bounce buffer in some cases.  We could simply force bounce
> > > buffering in the dma mapping code for too small or not properly aligned
> > > transfers and just decrease the dma alignment.
> >
> > We can force bounce if size is small but checking the alignment is
> > trickier. Normally the beginning of the buffer is aligned but the end is
> > at some sizeof() distance. We need to know whether the end is in a
> > kmalloc-128 cache and that requires reaching out to the slab internals.
> > That's doable and not expensive but it needs to be done for every small
> > size getting to the DMA API, something like (for mm/slub.c):
> >
> >         folio = virt_to_folio(x);
> >         slab = folio_slab(folio);
> >         if (slab->slab_cache->align < ARCH_DMA_MINALIGN)
> >                 ... bounce ...
> >
> > (and a bit different for mm/slab.c)
> 
> I think the decision to bounce or not can be based on the actual
> cache line size at runtime, so most commonly 64 bytes on arm64,
> even though the compile-time limit is 128 bytes.
> 
> We also know that larger slabs are all cacheline aligned, so simply
> comparing the transfer size is enough to rule out most, in this case
> any transfer larger than 96 bytes must come from the kmalloc-128
> or larger cache, so that works like before.

There's also the case with 128-byte cache lines and kmalloc-192.

> For transfers <=96 bytes, the possibilities are:
> 
> 1.kmalloc-32 or smaller, always needs to bounce
> 2. kmalloc-96, but at least one byte in partial cache line,
>     need to bounce
> 3. kmalloc-64, may skip the bounce.
> 4. kmalloc-128 or larger, or not a slab cache but a partial
>     transfer, may skip the bounce.
> 
> I would guess that the first case is the most common here,
> so unless bouncing one or two cache lines is extremely
> expensive, I don't expect it to be worth optimizing for the latter
> two cases.

I think so. If someone complains of a performance regression, we can
look at optimising the bounce. I have a suspicion the cost of copying
two cache lines is small compared to swiotlb_find_slots() etc.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Arnd Bergmann]

On Thu, Apr 21, 2022 at 3:25 PM Catalin Marinas <catalin.marinas@arm.com> wrote:
> On Thu, Apr 21, 2022 at 02:28:45PM +0200, Arnd Bergmann wrote:
> > We also know that larger slabs are all cacheline aligned, so simply
> > comparing the transfer size is enough to rule out most, in this case
> > any transfer larger than 96 bytes must come from the kmalloc-128
> > or larger cache, so that works like before.
>
> There's also the case with 128-byte cache lines and kmalloc-192.

Sure, but that's much less common, as the few machines with 128 byte
cache lines tend to also have cache coherent devices IIRC, so we'd
skip the bounce buffer entirely.

> > For transfers <=96 bytes, the possibilities are:
> >
> > 1.kmalloc-32 or smaller, always needs to bounce
> > 2. kmalloc-96, but at least one byte in partial cache line,
> >     need to bounce
> > 3. kmalloc-64, may skip the bounce.
> > 4. kmalloc-128 or larger, or not a slab cache but a partial
> >     transfer, may skip the bounce.
> >
> > I would guess that the first case is the most common here,
> > so unless bouncing one or two cache lines is extremely
> > expensive, I don't expect it to be worth optimizing for the latter
> > two cases.
>
> I think so. If someone complains of a performance regression, we can
> look at optimising the bounce. I have a suspicion the cost of copying
> two cache lines is small compared to swiotlb_find_slots() etc.

That is possible, and we'd definitely have to watch out for
performance regressions, I'm just skeptical that the cases that
suffer from the extra bouncer buffering on 33..64 byte allocations
benefit much from having a special case if the 1...32 and 65..96
byte allocations are still slow.

Another simpler way to do this might be to just not create the
kmalloc-96 (or kmalloc-192) caches, and assuming that any
transfer >=33 (or 65) bytes is safe.

       Arnd

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Arnd Bergmann]

On Thu, Apr 21, 2022 at 3:25 PM Catalin Marinas <catalin.marinas@arm.com> wrote:
> On Thu, Apr 21, 2022 at 02:28:45PM +0200, Arnd Bergmann wrote:
> > We also know that larger slabs are all cacheline aligned, so simply
> > comparing the transfer size is enough to rule out most, in this case
> > any transfer larger than 96 bytes must come from the kmalloc-128
> > or larger cache, so that works like before.
>
> There's also the case with 128-byte cache lines and kmalloc-192.

Sure, but that's much less common, as the few machines with 128 byte
cache lines tend to also have cache coherent devices IIRC, so we'd
skip the bounce buffer entirely.

> > For transfers <=96 bytes, the possibilities are:
> >
> > 1.kmalloc-32 or smaller, always needs to bounce
> > 2. kmalloc-96, but at least one byte in partial cache line,
> >     need to bounce
> > 3. kmalloc-64, may skip the bounce.
> > 4. kmalloc-128 or larger, or not a slab cache but a partial
> >     transfer, may skip the bounce.
> >
> > I would guess that the first case is the most common here,
> > so unless bouncing one or two cache lines is extremely
> > expensive, I don't expect it to be worth optimizing for the latter
> > two cases.
>
> I think so. If someone complains of a performance regression, we can
> look at optimising the bounce. I have a suspicion the cost of copying
> two cache lines is small compared to swiotlb_find_slots() etc.

That is possible, and we'd definitely have to watch out for
performance regressions, I'm just skeptical that the cases that
suffer from the extra bouncer buffering on 33..64 byte allocations
benefit much from having a special case if the 1...32 and 65..96
byte allocations are still slow.

Another simpler way to do this might be to just not create the
kmalloc-96 (or kmalloc-192) caches, and assuming that any
transfer >=33 (or 65) bytes is safe.

       Arnd

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Thu, Apr 21, 2022 at 03:47:30PM +0200, Arnd Bergmann wrote:
> On Thu, Apr 21, 2022 at 3:25 PM Catalin Marinas <catalin.marinas@arm.com> wrote:
> > On Thu, Apr 21, 2022 at 02:28:45PM +0200, Arnd Bergmann wrote:
> > > We also know that larger slabs are all cacheline aligned, so simply
> > > comparing the transfer size is enough to rule out most, in this case
> > > any transfer larger than 96 bytes must come from the kmalloc-128
> > > or larger cache, so that works like before.
> >
> > There's also the case with 128-byte cache lines and kmalloc-192.
> 
> Sure, but that's much less common, as the few machines with 128 byte
> cache lines tend to also have cache coherent devices IIRC, so we'd
> skip the bounce buffer entirely.

Do you know which machines still have 128-byte cache lines _and_
non-coherent DMA? If there isn't any that matters, I'd reduce
ARCH_DMA_MINALIGN to 64 now (while trying to get to even smaller kmalloc
caches).

> > > For transfers <=96 bytes, the possibilities are:
> > >
> > > 1.kmalloc-32 or smaller, always needs to bounce
> > > 2. kmalloc-96, but at least one byte in partial cache line,
> > >     need to bounce
> > > 3. kmalloc-64, may skip the bounce.
> > > 4. kmalloc-128 or larger, or not a slab cache but a partial
> > >     transfer, may skip the bounce.
> > >
> > > I would guess that the first case is the most common here,
> > > so unless bouncing one or two cache lines is extremely
> > > expensive, I don't expect it to be worth optimizing for the latter
> > > two cases.
> >
> > I think so. If someone complains of a performance regression, we can
> > look at optimising the bounce. I have a suspicion the cost of copying
> > two cache lines is small compared to swiotlb_find_slots() etc.
> 
> That is possible, and we'd definitely have to watch out for
> performance regressions, I'm just skeptical that the cases that
> suffer from the extra bouncer buffering on 33..64 byte allocations
> benefit much from having a special case if the 1...32 and 65..96
> byte allocations are still slow.
> 
> Another simpler way to do this might be to just not create the
> kmalloc-96 (or kmalloc-192) caches, and assuming that any
> transfer >=33 (or 65) bytes is safe.

I'll give the dma bounce idea a go next week, see how it looks.

-- 
Catalin

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Thu, Apr 21, 2022 at 03:47:30PM +0200, Arnd Bergmann wrote:
> On Thu, Apr 21, 2022 at 3:25 PM Catalin Marinas <catalin.marinas@arm.com> wrote:
> > On Thu, Apr 21, 2022 at 02:28:45PM +0200, Arnd Bergmann wrote:
> > > We also know that larger slabs are all cacheline aligned, so simply
> > > comparing the transfer size is enough to rule out most, in this case
> > > any transfer larger than 96 bytes must come from the kmalloc-128
> > > or larger cache, so that works like before.
> >
> > There's also the case with 128-byte cache lines and kmalloc-192.
> 
> Sure, but that's much less common, as the few machines with 128 byte
> cache lines tend to also have cache coherent devices IIRC, so we'd
> skip the bounce buffer entirely.

Do you know which machines still have 128-byte cache lines _and_
non-coherent DMA? If there isn't any that matters, I'd reduce
ARCH_DMA_MINALIGN to 64 now (while trying to get to even smaller kmalloc
caches).

> > > For transfers <=96 bytes, the possibilities are:
> > >
> > > 1.kmalloc-32 or smaller, always needs to bounce
> > > 2. kmalloc-96, but at least one byte in partial cache line,
> > >     need to bounce
> > > 3. kmalloc-64, may skip the bounce.
> > > 4. kmalloc-128 or larger, or not a slab cache but a partial
> > >     transfer, may skip the bounce.
> > >
> > > I would guess that the first case is the most common here,
> > > so unless bouncing one or two cache lines is extremely
> > > expensive, I don't expect it to be worth optimizing for the latter
> > > two cases.
> >
> > I think so. If someone complains of a performance regression, we can
> > look at optimising the bounce. I have a suspicion the cost of copying
> > two cache lines is small compared to swiotlb_find_slots() etc.
> 
> That is possible, and we'd definitely have to watch out for
> performance regressions, I'm just skeptical that the cases that
> suffer from the extra bouncer buffering on 33..64 byte allocations
> benefit much from having a special case if the 1...32 and 65..96
> byte allocations are still slow.
> 
> Another simpler way to do this might be to just not create the
> kmalloc-96 (or kmalloc-192) caches, and assuming that any
> transfer >=33 (or 65) bytes is safe.

I'll give the dma bounce idea a go next week, see how it looks.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Arnd Bergmann]

On Thu, Apr 21, 2022 at 4:44 PM Catalin Marinas <catalin.marinas@arm.com> wrote:
> On Thu, Apr 21, 2022 at 03:47:30PM +0200, Arnd Bergmann wrote:
> > On Thu, Apr 21, 2022 at 3:25 PM Catalin Marinas <catalin.marinas@arm.com> wrote:
> > > On Thu, Apr 21, 2022 at 02:28:45PM +0200, Arnd Bergmann wrote:
> > > > We also know that larger slabs are all cacheline aligned, so simply
> > > > comparing the transfer size is enough to rule out most, in this case
> > > > any transfer larger than 96 bytes must come from the kmalloc-128
> > > > or larger cache, so that works like before.
> > >
> > > There's also the case with 128-byte cache lines and kmalloc-192.
> >
> > Sure, but that's much less common, as the few machines with 128 byte
> > cache lines tend to also have cache coherent devices IIRC, so we'd
> > skip the bounce buffer entirely.
>
> Do you know which machines still have 128-byte cache lines _and_
> non-coherent DMA? If there isn't any that matters, I'd reduce
> ARCH_DMA_MINALIGN to 64 now (while trying to get to even smaller kmalloc
> caches).

I think the last time this came up, someone pointed out one of the
Qualcomm Snapdragon phone chips with their custom cores.

        Arnd

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Arnd Bergmann]

On Thu, Apr 21, 2022 at 4:44 PM Catalin Marinas <catalin.marinas@arm.com> wrote:
> On Thu, Apr 21, 2022 at 03:47:30PM +0200, Arnd Bergmann wrote:
> > On Thu, Apr 21, 2022 at 3:25 PM Catalin Marinas <catalin.marinas@arm.com> wrote:
> > > On Thu, Apr 21, 2022 at 02:28:45PM +0200, Arnd Bergmann wrote:
> > > > We also know that larger slabs are all cacheline aligned, so simply
> > > > comparing the transfer size is enough to rule out most, in this case
> > > > any transfer larger than 96 bytes must come from the kmalloc-128
> > > > or larger cache, so that works like before.
> > >
> > > There's also the case with 128-byte cache lines and kmalloc-192.
> >
> > Sure, but that's much less common, as the few machines with 128 byte
> > cache lines tend to also have cache coherent devices IIRC, so we'd
> > skip the bounce buffer entirely.
>
> Do you know which machines still have 128-byte cache lines _and_
> non-coherent DMA? If there isn't any that matters, I'd reduce
> ARCH_DMA_MINALIGN to 64 now (while trying to get to even smaller kmalloc
> caches).

I think the last time this came up, someone pointed out one of the
Qualcomm Snapdragon phone chips with their custom cores.

        Arnd

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[RFC PATCH 0/7] crypto: Add helpers for allocating with DMA alignment

[Herbert Xu]

This patch series adds helpers to allow drivers to explicitly
request ARCH_DMA_MINALIGN when allocating memory through the
Crypto API.

Note that I've only converted one file in one driver as this
is only meant to show how it's done and find out what else we
may need.

Thanks,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

[RFC PATCH 0/7] crypto: Add helpers for allocating with DMA alignment

[Herbert Xu]

This patch series adds helpers to allow drivers to explicitly
request ARCH_DMA_MINALIGN when allocating memory through the
Crypto API.

Note that I've only converted one file in one driver as this
is only meant to show how it's done and find out what else we
may need.

Thanks,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[RFC PATCH 1/7] crypto: Prepare to move crypto_tfm_ctx

[Herbert Xu]

The helper crypto_tfm_ctx is only used by the Crypto API algorithm
code and should really be in algapi.h.  However, for historical
reasons many files relied on it to be in crypto.h.  This patch
changes those files to use algapi.h instead in prepartion for a
move.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
---

 arch/arm/crypto/aes-cipher-glue.c      |    2 +-
 arch/arm64/crypto/aes-ce-glue.c        |    2 +-
 arch/arm64/crypto/aes-cipher-glue.c    |    2 +-
 arch/arm64/crypto/sm4-ce-cipher-glue.c |    2 +-
 arch/x86/crypto/twofish_glue.c         |    2 +-
 crypto/aes_generic.c                   |    2 +-
 crypto/aes_ti.c                        |    2 +-
 crypto/anubis.c                        |    2 +-
 crypto/blowfish_common.c               |    3 ++-
 crypto/blowfish_generic.c              |    3 ++-
 crypto/camellia_generic.c              |    2 +-
 crypto/cast5_generic.c                 |    2 +-
 crypto/cast6_generic.c                 |    2 +-
 crypto/des_generic.c                   |    2 +-
 crypto/fcrypt.c                        |    2 +-
 crypto/khazad.c                        |    2 +-
 crypto/seed.c                          |    2 +-
 crypto/serpent_generic.c               |    2 +-
 crypto/sm4_generic.c                   |    2 +-
 crypto/tea.c                           |    2 +-
 crypto/twofish_common.c                |    2 +-
 crypto/twofish_generic.c               |    2 +-
 drivers/crypto/nx/nx-842.h             |    2 +-
 include/crypto/internal/acompress.h    |    2 ++
 include/crypto/internal/scompress.h    |    3 ++-
 25 files changed, 29 insertions(+), 24 deletions(-)

diff --git a/arch/arm/crypto/aes-cipher-glue.c b/arch/arm/crypto/aes-cipher-glue.c
index 8cd00f56800e7..6dfaef2d8f913 100644
--- a/arch/arm/crypto/aes-cipher-glue.c
+++ b/arch/arm/crypto/aes-cipher-glue.c
@@ -7,7 +7,7 @@
  */
 
 #include <crypto/aes.h>
-#include <linux/crypto.h>
+#include <crypto/algapi.h>
 #include <linux/module.h>
 
 asmlinkage void __aes_arm_encrypt(u32 *rk, int rounds, const u8 *in, u8 *out);
diff --git a/arch/arm64/crypto/aes-ce-glue.c b/arch/arm64/crypto/aes-ce-glue.c
index 56a5f6f0b0c12..e921823ca103a 100644
--- a/arch/arm64/crypto/aes-ce-glue.c
+++ b/arch/arm64/crypto/aes-ce-glue.c
@@ -9,9 +9,9 @@
 #include <asm/simd.h>
 #include <asm/unaligned.h>
 #include <crypto/aes.h>
+#include <crypto/algapi.h>
 #include <crypto/internal/simd.h>
 #include <linux/cpufeature.h>
-#include <linux/crypto.h>
 #include <linux/module.h>
 
 #include "aes-ce-setkey.h"
diff --git a/arch/arm64/crypto/aes-cipher-glue.c b/arch/arm64/crypto/aes-cipher-glue.c
index 8caf6dfefce88..4ec55e568941c 100644
--- a/arch/arm64/crypto/aes-cipher-glue.c
+++ b/arch/arm64/crypto/aes-cipher-glue.c
@@ -6,7 +6,7 @@
  */
 
 #include <crypto/aes.h>
-#include <linux/crypto.h>
+#include <crypto/algapi.h>
 #include <linux/module.h>
 
 asmlinkage void __aes_arm64_encrypt(u32 *rk, u8 *out, const u8 *in, int rounds);
diff --git a/arch/arm64/crypto/sm4-ce-cipher-glue.c b/arch/arm64/crypto/sm4-ce-cipher-glue.c
index 76a34ef4abbbf..c31d76fb5a177 100644
--- a/arch/arm64/crypto/sm4-ce-cipher-glue.c
+++ b/arch/arm64/crypto/sm4-ce-cipher-glue.c
@@ -2,11 +2,11 @@
 
 #include <asm/neon.h>
 #include <asm/simd.h>
+#include <crypto/algapi.h>
 #include <crypto/sm4.h>
 #include <crypto/internal/simd.h>
 #include <linux/module.h>
 #include <linux/cpufeature.h>
-#include <linux/crypto.h>
 #include <linux/types.h>
 
 MODULE_ALIAS_CRYPTO("sm4");
diff --git a/arch/x86/crypto/twofish_glue.c b/arch/x86/crypto/twofish_glue.c
index f9c4adc274040..0614beece2793 100644
--- a/arch/x86/crypto/twofish_glue.c
+++ b/arch/x86/crypto/twofish_glue.c
@@ -38,8 +38,8 @@
  * Third Edition.
  */
 
+#include <crypto/algapi.h>
 #include <crypto/twofish.h>
-#include <linux/crypto.h>
 #include <linux/init.h>
 #include <linux/module.h>
 #include <linux/types.h>
diff --git a/crypto/aes_generic.c b/crypto/aes_generic.c
index 27ab279318137..666474b81c6aa 100644
--- a/crypto/aes_generic.c
+++ b/crypto/aes_generic.c
@@ -48,11 +48,11 @@
  */
 
 #include <crypto/aes.h>
+#include <crypto/algapi.h>
 #include <linux/module.h>
 #include <linux/init.h>
 #include <linux/types.h>
 #include <linux/errno.h>
-#include <linux/crypto.h>
 #include <asm/byteorder.h>
 #include <asm/unaligned.h>
 
diff --git a/crypto/aes_ti.c b/crypto/aes_ti.c
index 205c2c257d492..a3b342f92fab6 100644
--- a/crypto/aes_ti.c
+++ b/crypto/aes_ti.c
@@ -6,7 +6,7 @@
  */
 
 #include <crypto/aes.h>
-#include <linux/crypto.h>
+#include <crypto/algapi.h>
 #include <linux/module.h>
 
 static int aesti_set_key(struct crypto_tfm *tfm, const u8 *in_key,
diff --git a/crypto/anubis.c b/crypto/anubis.c
index 5da0241ef453c..9f0cf61bbc6e2 100644
--- a/crypto/anubis.c
+++ b/crypto/anubis.c
@@ -29,11 +29,11 @@
  *
  */
 
+#include <crypto/algapi.h>
 #include <linux/init.h>
 #include <linux/module.h>
 #include <linux/mm.h>
 #include <asm/byteorder.h>
-#include <linux/crypto.h>
 #include <linux/types.h>
 
 #define ANUBIS_MIN_KEY_SIZE	16
diff --git a/crypto/blowfish_common.c b/crypto/blowfish_common.c
index 1c072012baff4..c0208ce269a33 100644
--- a/crypto/blowfish_common.c
+++ b/crypto/blowfish_common.c
@@ -14,11 +14,12 @@
  * Copyright (c) Kyle McMartin <kyle@debian.org>
  * Copyright (c) 2002 James Morris <jmorris@intercode.com.au>
  */
+
+#include <crypto/algapi.h>
 #include <linux/init.h>
 #include <linux/module.h>
 #include <linux/mm.h>
 #include <asm/byteorder.h>
-#include <linux/crypto.h>
 #include <linux/types.h>
 #include <crypto/blowfish.h>
 
diff --git a/crypto/blowfish_generic.c b/crypto/blowfish_generic.c
index 003b52c6880ea..0e74c7242e77e 100644
--- a/crypto/blowfish_generic.c
+++ b/crypto/blowfish_generic.c
@@ -11,11 +11,12 @@
  * Copyright (c) Kyle McMartin <kyle@debian.org>
  * Copyright (c) 2002 James Morris <jmorris@intercode.com.au>
  */
+
+#include <crypto/algapi.h>
 #include <linux/init.h>
 #include <linux/module.h>
 #include <linux/mm.h>
 #include <asm/unaligned.h>
-#include <linux/crypto.h>
 #include <linux/types.h>
 #include <crypto/blowfish.h>
 
diff --git a/crypto/camellia_generic.c b/crypto/camellia_generic.c
index fd1a88af9e77f..c04670cf51acf 100644
--- a/crypto/camellia_generic.c
+++ b/crypto/camellia_generic.c
@@ -9,7 +9,7 @@
  *  https://info.isl.ntt.co.jp/crypt/eng/camellia/specifications.html
  */
 
-#include <linux/crypto.h>
+#include <crypto/algapi.h>
 #include <linux/errno.h>
 #include <linux/init.h>
 #include <linux/kernel.h>
diff --git a/crypto/cast5_generic.c b/crypto/cast5_generic.c
index 0257c14cefc25..085a1eedae03b 100644
--- a/crypto/cast5_generic.c
+++ b/crypto/cast5_generic.c
@@ -14,8 +14,8 @@
 
 
 #include <asm/unaligned.h>
+#include <crypto/algapi.h>
 #include <linux/init.h>
-#include <linux/crypto.h>
 #include <linux/module.h>
 #include <linux/errno.h>
 #include <linux/string.h>
diff --git a/crypto/cast6_generic.c b/crypto/cast6_generic.c
index 75346380aa0bc..34f1ab53e3a71 100644
--- a/crypto/cast6_generic.c
+++ b/crypto/cast6_generic.c
@@ -11,8 +11,8 @@
 
 
 #include <asm/unaligned.h>
+#include <crypto/algapi.h>
 #include <linux/init.h>
-#include <linux/crypto.h>
 #include <linux/module.h>
 #include <linux/errno.h>
 #include <linux/string.h>
diff --git a/crypto/des_generic.c b/crypto/des_generic.c
index c85354a5e94c7..1274e18d3eb90 100644
--- a/crypto/des_generic.c
+++ b/crypto/des_generic.c
@@ -8,11 +8,11 @@
  */
 
 #include <asm/byteorder.h>
+#include <crypto/algapi.h>
 #include <linux/bitops.h>
 #include <linux/init.h>
 #include <linux/module.h>
 #include <linux/errno.h>
-#include <linux/crypto.h>
 
 #include <crypto/internal/des.h>
 
diff --git a/crypto/fcrypt.c b/crypto/fcrypt.c
index 76a04d000c0d3..95a16e88899ba 100644
--- a/crypto/fcrypt.c
+++ b/crypto/fcrypt.c
@@ -43,10 +43,10 @@
  */
 
 #include <asm/byteorder.h>
+#include <crypto/algapi.h>
 #include <linux/bitops.h>
 #include <linux/init.h>
 #include <linux/module.h>
-#include <linux/crypto.h>
 
 #define ROUNDS 16
 
diff --git a/crypto/khazad.c b/crypto/khazad.c
index f19339954c89e..70cafe73f9740 100644
--- a/crypto/khazad.c
+++ b/crypto/khazad.c
@@ -19,11 +19,11 @@
  *
  */
 
+#include <crypto/algapi.h>
 #include <linux/init.h>
 #include <linux/module.h>
 #include <linux/mm.h>
 #include <asm/byteorder.h>
-#include <linux/crypto.h>
 #include <linux/types.h>
 
 #define KHAZAD_KEY_SIZE		16
diff --git a/crypto/seed.c b/crypto/seed.c
index 27720140820ef..d0506ade2a5f8 100644
--- a/crypto/seed.c
+++ b/crypto/seed.c
@@ -8,11 +8,11 @@
  * Copyright (C) 2007 Korea Information Security Agency (KISA).
  */
 
+#include <crypto/algapi.h>
 #include <linux/module.h>
 #include <linux/init.h>
 #include <linux/types.h>
 #include <linux/errno.h>
-#include <linux/crypto.h>
 #include <asm/byteorder.h>
 
 #define SEED_NUM_KCONSTANTS	16
diff --git a/crypto/serpent_generic.c b/crypto/serpent_generic.c
index 45f98b750053e..c6bca47931e21 100644
--- a/crypto/serpent_generic.c
+++ b/crypto/serpent_generic.c
@@ -7,11 +7,11 @@
  * Copyright (C) 2002 Dag Arne Osvik <osvik@ii.uib.no>
  */
 
+#include <crypto/algapi.h>
 #include <linux/init.h>
 #include <linux/module.h>
 #include <linux/errno.h>
 #include <asm/unaligned.h>
-#include <linux/crypto.h>
 #include <linux/types.h>
 #include <crypto/serpent.h>
 
diff --git a/crypto/sm4_generic.c b/crypto/sm4_generic.c
index 4a6480a27fee5..560eba37dc55e 100644
--- a/crypto/sm4_generic.c
+++ b/crypto/sm4_generic.c
@@ -7,12 +7,12 @@
  * All rights reserved.
  */
 
+#include <crypto/algapi.h>
 #include <crypto/sm4.h>
 #include <linux/module.h>
 #include <linux/init.h>
 #include <linux/types.h>
 #include <linux/errno.h>
-#include <linux/crypto.h>
 #include <asm/byteorder.h>
 #include <asm/unaligned.h>
 
diff --git a/crypto/tea.c b/crypto/tea.c
index 02efc5d816903..896f863f3067c 100644
--- a/crypto/tea.c
+++ b/crypto/tea.c
@@ -14,11 +14,11 @@
  * Copyright (c) 2004 Aaron Grothe ajgrothe@yahoo.com
  */
 
+#include <crypto/algapi.h>
 #include <linux/init.h>
 #include <linux/module.h>
 #include <linux/mm.h>
 #include <asm/byteorder.h>
-#include <linux/crypto.h>
 #include <linux/types.h>
 
 #define TEA_KEY_SIZE		16
diff --git a/crypto/twofish_common.c b/crypto/twofish_common.c
index d23fa531b91f1..3a95e232aa236 100644
--- a/crypto/twofish_common.c
+++ b/crypto/twofish_common.c
@@ -25,9 +25,9 @@
  * Third Edition.
  */
 
+#include <crypto/algapi.h>
 #include <crypto/twofish.h>
 #include <linux/bitops.h>
-#include <linux/crypto.h>
 #include <linux/errno.h>
 #include <linux/init.h>
 #include <linux/kernel.h>
diff --git a/crypto/twofish_generic.c b/crypto/twofish_generic.c
index 86b2f067a4162..557915e4062d7 100644
--- a/crypto/twofish_generic.c
+++ b/crypto/twofish_generic.c
@@ -25,12 +25,12 @@
  */
 
 #include <asm/unaligned.h>
+#include <crypto/algapi.h>
 #include <crypto/twofish.h>
 #include <linux/module.h>
 #include <linux/init.h>
 #include <linux/types.h>
 #include <linux/errno.h>
-#include <linux/crypto.h>
 #include <linux/bitops.h>
 
 /* Macros to compute the g() function in the encryption and decryption
diff --git a/drivers/crypto/nx/nx-842.h b/drivers/crypto/nx/nx-842.h
index b66f19ac600f2..7590bfb24d79b 100644
--- a/drivers/crypto/nx/nx-842.h
+++ b/drivers/crypto/nx/nx-842.h
@@ -3,10 +3,10 @@
 #ifndef __NX_842_H__
 #define __NX_842_H__
 
+#include <crypto/algapi.h>
 #include <linux/kernel.h>
 #include <linux/init.h>
 #include <linux/module.h>
-#include <linux/crypto.h>
 #include <linux/of.h>
 #include <linux/slab.h>
 #include <linux/io.h>
diff --git a/include/crypto/internal/acompress.h b/include/crypto/internal/acompress.h
index cfc47e18820fb..49339003bd2ce 100644
--- a/include/crypto/internal/acompress.h
+++ b/include/crypto/internal/acompress.h
@@ -8,7 +8,9 @@
  */
 #ifndef _CRYPTO_ACOMP_INT_H
 #define _CRYPTO_ACOMP_INT_H
+
 #include <crypto/acompress.h>
+#include <crypto/algapi.h>
 
 /*
  * Transform internal helpers.
diff --git a/include/crypto/internal/scompress.h b/include/crypto/internal/scompress.h
index f834274c2493f..252cc949d4ee5 100644
--- a/include/crypto/internal/scompress.h
+++ b/include/crypto/internal/scompress.h
@@ -8,7 +8,8 @@
  */
 #ifndef _CRYPTO_SCOMP_INT_H
 #define _CRYPTO_SCOMP_INT_H
-#include <linux/crypto.h>
+
+#include <crypto/algapi.h>
 
 #define SCOMP_SCRATCH_SIZE	131072
 

[RFC PATCH 1/7] crypto: Prepare to move crypto_tfm_ctx

[Herbert Xu]

The helper crypto_tfm_ctx is only used by the Crypto API algorithm
code and should really be in algapi.h.  However, for historical
reasons many files relied on it to be in crypto.h.  This patch
changes those files to use algapi.h instead in prepartion for a
move.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
---

 arch/arm/crypto/aes-cipher-glue.c      |    2 +-
 arch/arm64/crypto/aes-ce-glue.c        |    2 +-
 arch/arm64/crypto/aes-cipher-glue.c    |    2 +-
 arch/arm64/crypto/sm4-ce-cipher-glue.c |    2 +-
 arch/x86/crypto/twofish_glue.c         |    2 +-
 crypto/aes_generic.c                   |    2 +-
 crypto/aes_ti.c                        |    2 +-
 crypto/anubis.c                        |    2 +-
 crypto/blowfish_common.c               |    3 ++-
 crypto/blowfish_generic.c              |    3 ++-
 crypto/camellia_generic.c              |    2 +-
 crypto/cast5_generic.c                 |    2 +-
 crypto/cast6_generic.c                 |    2 +-
 crypto/des_generic.c                   |    2 +-
 crypto/fcrypt.c                        |    2 +-
 crypto/khazad.c                        |    2 +-
 crypto/seed.c                          |    2 +-
 crypto/serpent_generic.c               |    2 +-
 crypto/sm4_generic.c                   |    2 +-
 crypto/tea.c                           |    2 +-
 crypto/twofish_common.c                |    2 +-
 crypto/twofish_generic.c               |    2 +-
 drivers/crypto/nx/nx-842.h             |    2 +-
 include/crypto/internal/acompress.h    |    2 ++
 include/crypto/internal/scompress.h    |    3 ++-
 25 files changed, 29 insertions(+), 24 deletions(-)

diff --git a/arch/arm/crypto/aes-cipher-glue.c b/arch/arm/crypto/aes-cipher-glue.c
index 8cd00f56800e7..6dfaef2d8f913 100644
--- a/arch/arm/crypto/aes-cipher-glue.c
+++ b/arch/arm/crypto/aes-cipher-glue.c
@@ -7,7 +7,7 @@
  */
 
 #include <crypto/aes.h>
-#include <linux/crypto.h>
+#include <crypto/algapi.h>
 #include <linux/module.h>
 
 asmlinkage void __aes_arm_encrypt(u32 *rk, int rounds, const u8 *in, u8 *out);
diff --git a/arch/arm64/crypto/aes-ce-glue.c b/arch/arm64/crypto/aes-ce-glue.c
index 56a5f6f0b0c12..e921823ca103a 100644
--- a/arch/arm64/crypto/aes-ce-glue.c
+++ b/arch/arm64/crypto/aes-ce-glue.c
@@ -9,9 +9,9 @@
 #include <asm/simd.h>
 #include <asm/unaligned.h>
 #include <crypto/aes.h>
+#include <crypto/algapi.h>
 #include <crypto/internal/simd.h>
 #include <linux/cpufeature.h>
-#include <linux/crypto.h>
 #include <linux/module.h>
 
 #include "aes-ce-setkey.h"
diff --git a/arch/arm64/crypto/aes-cipher-glue.c b/arch/arm64/crypto/aes-cipher-glue.c
index 8caf6dfefce88..4ec55e568941c 100644
--- a/arch/arm64/crypto/aes-cipher-glue.c
+++ b/arch/arm64/crypto/aes-cipher-glue.c
@@ -6,7 +6,7 @@
  */
 
 #include <crypto/aes.h>
-#include <linux/crypto.h>
+#include <crypto/algapi.h>
 #include <linux/module.h>
 
 asmlinkage void __aes_arm64_encrypt(u32 *rk, u8 *out, const u8 *in, int rounds);
diff --git a/arch/arm64/crypto/sm4-ce-cipher-glue.c b/arch/arm64/crypto/sm4-ce-cipher-glue.c
index 76a34ef4abbbf..c31d76fb5a177 100644
--- a/arch/arm64/crypto/sm4-ce-cipher-glue.c
+++ b/arch/arm64/crypto/sm4-ce-cipher-glue.c
@@ -2,11 +2,11 @@
 
 #include <asm/neon.h>
 #include <asm/simd.h>
+#include <crypto/algapi.h>
 #include <crypto/sm4.h>
 #include <crypto/internal/simd.h>
 #include <linux/module.h>
 #include <linux/cpufeature.h>
-#include <linux/crypto.h>
 #include <linux/types.h>
 
 MODULE_ALIAS_CRYPTO("sm4");
diff --git a/arch/x86/crypto/twofish_glue.c b/arch/x86/crypto/twofish_glue.c
index f9c4adc274040..0614beece2793 100644
--- a/arch/x86/crypto/twofish_glue.c
+++ b/arch/x86/crypto/twofish_glue.c
@@ -38,8 +38,8 @@
  * Third Edition.
  */
 
+#include <crypto/algapi.h>
 #include <crypto/twofish.h>
-#include <linux/crypto.h>
 #include <linux/init.h>
 #include <linux/module.h>
 #include <linux/types.h>
diff --git a/crypto/aes_generic.c b/crypto/aes_generic.c
index 27ab279318137..666474b81c6aa 100644
--- a/crypto/aes_generic.c
+++ b/crypto/aes_generic.c
@@ -48,11 +48,11 @@
  */
 
 #include <crypto/aes.h>
+#include <crypto/algapi.h>
 #include <linux/module.h>
 #include <linux/init.h>
 #include <linux/types.h>
 #include <linux/errno.h>
-#include <linux/crypto.h>
 #include <asm/byteorder.h>
 #include <asm/unaligned.h>
 
diff --git a/crypto/aes_ti.c b/crypto/aes_ti.c
index 205c2c257d492..a3b342f92fab6 100644
--- a/crypto/aes_ti.c
+++ b/crypto/aes_ti.c
@@ -6,7 +6,7 @@
  */
 
 #include <crypto/aes.h>
-#include <linux/crypto.h>
+#include <crypto/algapi.h>
 #include <linux/module.h>
 
 static int aesti_set_key(struct crypto_tfm *tfm, const u8 *in_key,
diff --git a/crypto/anubis.c b/crypto/anubis.c
index 5da0241ef453c..9f0cf61bbc6e2 100644
--- a/crypto/anubis.c
+++ b/crypto/anubis.c
@@ -29,11 +29,11 @@
  *
  */
 
+#include <crypto/algapi.h>
 #include <linux/init.h>
 #include <linux/module.h>
 #include <linux/mm.h>
 #include <asm/byteorder.h>
-#include <linux/crypto.h>
 #include <linux/types.h>
 
 #define ANUBIS_MIN_KEY_SIZE	16
diff --git a/crypto/blowfish_common.c b/crypto/blowfish_common.c
index 1c072012baff4..c0208ce269a33 100644
--- a/crypto/blowfish_common.c
+++ b/crypto/blowfish_common.c
@@ -14,11 +14,12 @@
  * Copyright (c) Kyle McMartin <kyle@debian.org>
  * Copyright (c) 2002 James Morris <jmorris@intercode.com.au>
  */
+
+#include <crypto/algapi.h>
 #include <linux/init.h>
 #include <linux/module.h>
 #include <linux/mm.h>
 #include <asm/byteorder.h>
-#include <linux/crypto.h>
 #include <linux/types.h>
 #include <crypto/blowfish.h>
 
diff --git a/crypto/blowfish_generic.c b/crypto/blowfish_generic.c
index 003b52c6880ea..0e74c7242e77e 100644
--- a/crypto/blowfish_generic.c
+++ b/crypto/blowfish_generic.c
@@ -11,11 +11,12 @@
  * Copyright (c) Kyle McMartin <kyle@debian.org>
  * Copyright (c) 2002 James Morris <jmorris@intercode.com.au>
  */
+
+#include <crypto/algapi.h>
 #include <linux/init.h>
 #include <linux/module.h>
 #include <linux/mm.h>
 #include <asm/unaligned.h>
-#include <linux/crypto.h>
 #include <linux/types.h>
 #include <crypto/blowfish.h>
 
diff --git a/crypto/camellia_generic.c b/crypto/camellia_generic.c
index fd1a88af9e77f..c04670cf51acf 100644
--- a/crypto/camellia_generic.c
+++ b/crypto/camellia_generic.c
@@ -9,7 +9,7 @@
  *  https://info.isl.ntt.co.jp/crypt/eng/camellia/specifications.html
  */
 
-#include <linux/crypto.h>
+#include <crypto/algapi.h>
 #include <linux/errno.h>
 #include <linux/init.h>
 #include <linux/kernel.h>
diff --git a/crypto/cast5_generic.c b/crypto/cast5_generic.c
index 0257c14cefc25..085a1eedae03b 100644
--- a/crypto/cast5_generic.c
+++ b/crypto/cast5_generic.c
@@ -14,8 +14,8 @@
 
 
 #include <asm/unaligned.h>
+#include <crypto/algapi.h>
 #include <linux/init.h>
-#include <linux/crypto.h>
 #include <linux/module.h>
 #include <linux/errno.h>
 #include <linux/string.h>
diff --git a/crypto/cast6_generic.c b/crypto/cast6_generic.c
index 75346380aa0bc..34f1ab53e3a71 100644
--- a/crypto/cast6_generic.c
+++ b/crypto/cast6_generic.c
@@ -11,8 +11,8 @@
 
 
 #include <asm/unaligned.h>
+#include <crypto/algapi.h>
 #include <linux/init.h>
-#include <linux/crypto.h>
 #include <linux/module.h>
 #include <linux/errno.h>
 #include <linux/string.h>
diff --git a/crypto/des_generic.c b/crypto/des_generic.c
index c85354a5e94c7..1274e18d3eb90 100644
--- a/crypto/des_generic.c
+++ b/crypto/des_generic.c
@@ -8,11 +8,11 @@
  */
 
 #include <asm/byteorder.h>
+#include <crypto/algapi.h>
 #include <linux/bitops.h>
 #include <linux/init.h>
 #include <linux/module.h>
 #include <linux/errno.h>
-#include <linux/crypto.h>
 
 #include <crypto/internal/des.h>
 
diff --git a/crypto/fcrypt.c b/crypto/fcrypt.c
index 76a04d000c0d3..95a16e88899ba 100644
--- a/crypto/fcrypt.c
+++ b/crypto/fcrypt.c
@@ -43,10 +43,10 @@
  */
 
 #include <asm/byteorder.h>
+#include <crypto/algapi.h>
 #include <linux/bitops.h>
 #include <linux/init.h>
 #include <linux/module.h>
-#include <linux/crypto.h>
 
 #define ROUNDS 16
 
diff --git a/crypto/khazad.c b/crypto/khazad.c
index f19339954c89e..70cafe73f9740 100644
--- a/crypto/khazad.c
+++ b/crypto/khazad.c
@@ -19,11 +19,11 @@
  *
  */
 
+#include <crypto/algapi.h>
 #include <linux/init.h>
 #include <linux/module.h>
 #include <linux/mm.h>
 #include <asm/byteorder.h>
-#include <linux/crypto.h>
 #include <linux/types.h>
 
 #define KHAZAD_KEY_SIZE		16
diff --git a/crypto/seed.c b/crypto/seed.c
index 27720140820ef..d0506ade2a5f8 100644
--- a/crypto/seed.c
+++ b/crypto/seed.c
@@ -8,11 +8,11 @@
  * Copyright (C) 2007 Korea Information Security Agency (KISA).
  */
 
+#include <crypto/algapi.h>
 #include <linux/module.h>
 #include <linux/init.h>
 #include <linux/types.h>
 #include <linux/errno.h>
-#include <linux/crypto.h>
 #include <asm/byteorder.h>
 
 #define SEED_NUM_KCONSTANTS	16
diff --git a/crypto/serpent_generic.c b/crypto/serpent_generic.c
index 45f98b750053e..c6bca47931e21 100644
--- a/crypto/serpent_generic.c
+++ b/crypto/serpent_generic.c
@@ -7,11 +7,11 @@
  * Copyright (C) 2002 Dag Arne Osvik <osvik@ii.uib.no>
  */
 
+#include <crypto/algapi.h>
 #include <linux/init.h>
 #include <linux/module.h>
 #include <linux/errno.h>
 #include <asm/unaligned.h>
-#include <linux/crypto.h>
 #include <linux/types.h>
 #include <crypto/serpent.h>
 
diff --git a/crypto/sm4_generic.c b/crypto/sm4_generic.c
index 4a6480a27fee5..560eba37dc55e 100644
--- a/crypto/sm4_generic.c
+++ b/crypto/sm4_generic.c
@@ -7,12 +7,12 @@
  * All rights reserved.
  */
 
+#include <crypto/algapi.h>
 #include <crypto/sm4.h>
 #include <linux/module.h>
 #include <linux/init.h>
 #include <linux/types.h>
 #include <linux/errno.h>
-#include <linux/crypto.h>
 #include <asm/byteorder.h>
 #include <asm/unaligned.h>
 
diff --git a/crypto/tea.c b/crypto/tea.c
index 02efc5d816903..896f863f3067c 100644
--- a/crypto/tea.c
+++ b/crypto/tea.c
@@ -14,11 +14,11 @@
  * Copyright (c) 2004 Aaron Grothe ajgrothe@yahoo.com
  */
 
+#include <crypto/algapi.h>
 #include <linux/init.h>
 #include <linux/module.h>
 #include <linux/mm.h>
 #include <asm/byteorder.h>
-#include <linux/crypto.h>
 #include <linux/types.h>
 
 #define TEA_KEY_SIZE		16
diff --git a/crypto/twofish_common.c b/crypto/twofish_common.c
index d23fa531b91f1..3a95e232aa236 100644
--- a/crypto/twofish_common.c
+++ b/crypto/twofish_common.c
@@ -25,9 +25,9 @@
  * Third Edition.
  */
 
+#include <crypto/algapi.h>
 #include <crypto/twofish.h>
 #include <linux/bitops.h>
-#include <linux/crypto.h>
 #include <linux/errno.h>
 #include <linux/init.h>
 #include <linux/kernel.h>
diff --git a/crypto/twofish_generic.c b/crypto/twofish_generic.c
index 86b2f067a4162..557915e4062d7 100644
--- a/crypto/twofish_generic.c
+++ b/crypto/twofish_generic.c
@@ -25,12 +25,12 @@
  */
 
 #include <asm/unaligned.h>
+#include <crypto/algapi.h>
 #include <crypto/twofish.h>
 #include <linux/module.h>
 #include <linux/init.h>
 #include <linux/types.h>
 #include <linux/errno.h>
-#include <linux/crypto.h>
 #include <linux/bitops.h>
 
 /* Macros to compute the g() function in the encryption and decryption
diff --git a/drivers/crypto/nx/nx-842.h b/drivers/crypto/nx/nx-842.h
index b66f19ac600f2..7590bfb24d79b 100644
--- a/drivers/crypto/nx/nx-842.h
+++ b/drivers/crypto/nx/nx-842.h
@@ -3,10 +3,10 @@
 #ifndef __NX_842_H__
 #define __NX_842_H__
 
+#include <crypto/algapi.h>
 #include <linux/kernel.h>
 #include <linux/init.h>
 #include <linux/module.h>
-#include <linux/crypto.h>
 #include <linux/of.h>
 #include <linux/slab.h>
 #include <linux/io.h>
diff --git a/include/crypto/internal/acompress.h b/include/crypto/internal/acompress.h
index cfc47e18820fb..49339003bd2ce 100644
--- a/include/crypto/internal/acompress.h
+++ b/include/crypto/internal/acompress.h
@@ -8,7 +8,9 @@
  */
 #ifndef _CRYPTO_ACOMP_INT_H
 #define _CRYPTO_ACOMP_INT_H
+
 #include <crypto/acompress.h>
+#include <crypto/algapi.h>
 
 /*
  * Transform internal helpers.
diff --git a/include/crypto/internal/scompress.h b/include/crypto/internal/scompress.h
index f834274c2493f..252cc949d4ee5 100644
--- a/include/crypto/internal/scompress.h
+++ b/include/crypto/internal/scompress.h
@@ -8,7 +8,8 @@
  */
 #ifndef _CRYPTO_SCOMP_INT_H
 #define _CRYPTO_SCOMP_INT_H
-#include <linux/crypto.h>
+
+#include <crypto/algapi.h>
 
 #define SCOMP_SCRATCH_SIZE	131072
 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[RFC PATCH 2/7] crypto: api - Add crypto_tfm_ctx_dma

[Herbert Xu]

This patch adds the helper crypto_tfm_ctx_dma.  It's similar to
crypto_tfm_ctx_aligned but the alignment is hard-coded to the
macro ARCH_DMA_MINALIGN.

This patch also moves crypto_tfm_ctx into algapi.h.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
---

 include/crypto/algapi.h |   28 ++++++++++++++++++++++++++--
 include/linux/crypto.h  |    5 -----
 2 files changed, 26 insertions(+), 7 deletions(-)

diff --git a/include/crypto/algapi.h b/include/crypto/algapi.h
index f50c5d1725da5..cdf12e51c53a0 100644
--- a/include/crypto/algapi.h
+++ b/include/crypto/algapi.h
@@ -189,10 +189,34 @@ static inline void crypto_xor_cpy(u8 *dst, const u8 *src1, const u8 *src2,
 	}
 }
 
+static inline void *crypto_tfm_ctx(struct crypto_tfm *tfm)
+{
+	return tfm->__crt_ctx;
+}
+
+static inline void *crypto_tfm_ctx_align(struct crypto_tfm *tfm,
+					 unsigned int align)
+{
+	if (align <= crypto_tfm_ctx_alignment())
+		align = 1;
+
+	return PTR_ALIGN(crypto_tfm_ctx(tfm), align);
+}
+
 static inline void *crypto_tfm_ctx_aligned(struct crypto_tfm *tfm)
 {
-	return PTR_ALIGN(crypto_tfm_ctx(tfm),
-			 crypto_tfm_alg_alignmask(tfm) + 1);
+	return crypto_tfm_ctx_align(tfm, crypto_tfm_alg_alignmask(tfm) + 1);
+}
+
+static inline void *crypto_tfm_ctx_dma(struct crypto_tfm *tfm)
+{
+	unsigned int align = 1;
+
+#ifdef ARCH_DMA_MINALIGN
+	align = ARCH_DMA_MINALIGN;
+#endif
+
+	return crypto_tfm_ctx_align(tfm, align);
 }
 
 static inline struct crypto_instance *crypto_tfm_alg_instance(
diff --git a/include/linux/crypto.h b/include/linux/crypto.h
index 2324ab6f1846b..5d1e961f810ec 100644
--- a/include/linux/crypto.h
+++ b/include/linux/crypto.h
@@ -714,11 +714,6 @@ static inline void crypto_tfm_clear_flags(struct crypto_tfm *tfm, u32 flags)
 	tfm->crt_flags &= ~flags;
 }
 
-static inline void *crypto_tfm_ctx(struct crypto_tfm *tfm)
-{
-	return tfm->__crt_ctx;
-}
-
 static inline unsigned int crypto_tfm_ctx_alignment(void)
 {
 	struct crypto_tfm *tfm;

[RFC PATCH 2/7] crypto: api - Add crypto_tfm_ctx_dma

[Herbert Xu]

This patch adds the helper crypto_tfm_ctx_dma.  It's similar to
crypto_tfm_ctx_aligned but the alignment is hard-coded to the
macro ARCH_DMA_MINALIGN.

This patch also moves crypto_tfm_ctx into algapi.h.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
---

 include/crypto/algapi.h |   28 ++++++++++++++++++++++++++--
 include/linux/crypto.h  |    5 -----
 2 files changed, 26 insertions(+), 7 deletions(-)

diff --git a/include/crypto/algapi.h b/include/crypto/algapi.h
index f50c5d1725da5..cdf12e51c53a0 100644
--- a/include/crypto/algapi.h
+++ b/include/crypto/algapi.h
@@ -189,10 +189,34 @@ static inline void crypto_xor_cpy(u8 *dst, const u8 *src1, const u8 *src2,
 	}
 }
 
+static inline void *crypto_tfm_ctx(struct crypto_tfm *tfm)
+{
+	return tfm->__crt_ctx;
+}
+
+static inline void *crypto_tfm_ctx_align(struct crypto_tfm *tfm,
+					 unsigned int align)
+{
+	if (align <= crypto_tfm_ctx_alignment())
+		align = 1;
+
+	return PTR_ALIGN(crypto_tfm_ctx(tfm), align);
+}
+
 static inline void *crypto_tfm_ctx_aligned(struct crypto_tfm *tfm)
 {
-	return PTR_ALIGN(crypto_tfm_ctx(tfm),
-			 crypto_tfm_alg_alignmask(tfm) + 1);
+	return crypto_tfm_ctx_align(tfm, crypto_tfm_alg_alignmask(tfm) + 1);
+}
+
+static inline void *crypto_tfm_ctx_dma(struct crypto_tfm *tfm)
+{
+	unsigned int align = 1;
+
+#ifdef ARCH_DMA_MINALIGN
+	align = ARCH_DMA_MINALIGN;
+#endif
+
+	return crypto_tfm_ctx_align(tfm, align);
 }
 
 static inline struct crypto_instance *crypto_tfm_alg_instance(
diff --git a/include/linux/crypto.h b/include/linux/crypto.h
index 2324ab6f1846b..5d1e961f810ec 100644
--- a/include/linux/crypto.h
+++ b/include/linux/crypto.h
@@ -714,11 +714,6 @@ static inline void crypto_tfm_clear_flags(struct crypto_tfm *tfm, u32 flags)
 	tfm->crt_flags &= ~flags;
 }
 
-static inline void *crypto_tfm_ctx(struct crypto_tfm *tfm)
-{
-	return tfm->__crt_ctx;
-}
-
 static inline unsigned int crypto_tfm_ctx_alignment(void)
 {
 	struct crypto_tfm *tfm;

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[RFC PATCH 3/7] crypto: aead - Add ctx helpers with DMA alignment

[Herbert Xu]

This patch adds helpers to access the aead context structure and
request context structure with an added alignment for DMA access.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
---

 include/crypto/internal/aead.h |   28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/include/crypto/internal/aead.h b/include/crypto/internal/aead.h
index 27b7b0224ea6f..be225f3f7fdbf 100644
--- a/include/crypto/internal/aead.h
+++ b/include/crypto/internal/aead.h
@@ -39,6 +39,11 @@ static inline void *crypto_aead_ctx(struct crypto_aead *tfm)
 	return crypto_tfm_ctx(&tfm->base);
 }
 
+static inline void *crypto_aead_ctx_dma(struct crypto_aead *tfm)
+{
+	return crypto_tfm_ctx_dma(&tfm->base);
+}
+
 static inline struct crypto_instance *aead_crypto_instance(
 	struct aead_instance *inst)
 {
@@ -65,6 +70,20 @@ static inline void *aead_request_ctx(struct aead_request *req)
 	return req->__ctx;
 }
 
+static inline void *aead_request_ctx_dma(struct aead_request *req)
+{
+	unsigned int align = 1;
+
+#ifdef ARCH_DMA_MINALIGN
+	align = ARCH_DMA_MINALIGN;
+#endif
+
+	if (align <= crypto_tfm_ctx_alignment())
+		align = 1;
+
+	return PTR_ALIGN(aead_request_ctx(req), align);
+}
+
 static inline void aead_request_complete(struct aead_request *req, int err)
 {
 	req->base.complete(&req->base, err);
@@ -108,6 +127,15 @@ static inline void crypto_aead_set_reqsize(struct crypto_aead *aead,
 	aead->reqsize = reqsize;
 }
 
+static inline void crypto_aead_set_reqsize_dma(struct crypto_aead *aead,
+					       unsigned int reqsize)
+{
+#ifdef ARCH_DMA_MINALIGN
+	reqsize += ARCH_DMA_MINALIGN & ~(crypto_tfm_ctx_alignment() - 1);
+#endif
+	aead->reqsize = reqsize;
+}
+
 static inline void aead_init_queue(struct aead_queue *queue,
 				   unsigned int max_qlen)
 {

[RFC PATCH 3/7] crypto: aead - Add ctx helpers with DMA alignment

[Herbert Xu]

This patch adds helpers to access the aead context structure and
request context structure with an added alignment for DMA access.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
---

 include/crypto/internal/aead.h |   28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/include/crypto/internal/aead.h b/include/crypto/internal/aead.h
index 27b7b0224ea6f..be225f3f7fdbf 100644
--- a/include/crypto/internal/aead.h
+++ b/include/crypto/internal/aead.h
@@ -39,6 +39,11 @@ static inline void *crypto_aead_ctx(struct crypto_aead *tfm)
 	return crypto_tfm_ctx(&tfm->base);
 }
 
+static inline void *crypto_aead_ctx_dma(struct crypto_aead *tfm)
+{
+	return crypto_tfm_ctx_dma(&tfm->base);
+}
+
 static inline struct crypto_instance *aead_crypto_instance(
 	struct aead_instance *inst)
 {
@@ -65,6 +70,20 @@ static inline void *aead_request_ctx(struct aead_request *req)
 	return req->__ctx;
 }
 
+static inline void *aead_request_ctx_dma(struct aead_request *req)
+{
+	unsigned int align = 1;
+
+#ifdef ARCH_DMA_MINALIGN
+	align = ARCH_DMA_MINALIGN;
+#endif
+
+	if (align <= crypto_tfm_ctx_alignment())
+		align = 1;
+
+	return PTR_ALIGN(aead_request_ctx(req), align);
+}
+
 static inline void aead_request_complete(struct aead_request *req, int err)
 {
 	req->base.complete(&req->base, err);
@@ -108,6 +127,15 @@ static inline void crypto_aead_set_reqsize(struct crypto_aead *aead,
 	aead->reqsize = reqsize;
 }
 
+static inline void crypto_aead_set_reqsize_dma(struct crypto_aead *aead,
+					       unsigned int reqsize)
+{
+#ifdef ARCH_DMA_MINALIGN
+	reqsize += ARCH_DMA_MINALIGN & ~(crypto_tfm_ctx_alignment() - 1);
+#endif
+	aead->reqsize = reqsize;
+}
+
 static inline void aead_init_queue(struct aead_queue *queue,
 				   unsigned int max_qlen)
 {

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[RFC PATCH 4/7] crypto: hash - Add ctx helpers with DMA alignment

[Herbert Xu]

This patch adds helpers to access the ahash context structure and
request context structure with an added alignment for DMA access.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
---

 include/crypto/internal/hash.h |   28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/include/crypto/internal/hash.h b/include/crypto/internal/hash.h
index 25806141db591..f635a61fd1a0e 100644
--- a/include/crypto/internal/hash.h
+++ b/include/crypto/internal/hash.h
@@ -134,6 +134,11 @@ static inline void *crypto_ahash_ctx(struct crypto_ahash *tfm)
 	return crypto_tfm_ctx(crypto_ahash_tfm(tfm));
 }
 
+static inline void *crypto_ahash_ctx_dma(struct crypto_ahash *tfm)
+{
+	return crypto_tfm_ctx_dma(crypto_ahash_tfm(tfm));
+}
+
 static inline struct ahash_alg *__crypto_ahash_alg(struct crypto_alg *alg)
 {
 	return container_of(__crypto_hash_alg_common(alg), struct ahash_alg,
@@ -146,6 +151,15 @@ static inline void crypto_ahash_set_reqsize(struct crypto_ahash *tfm,
 	tfm->reqsize = reqsize;
 }
 
+static inline void crypto_ahash_set_reqsize_dma(struct crypto_ahash *ahash,
+						unsigned int reqsize)
+{
+#ifdef ARCH_DMA_MINALIGN
+	reqsize += ARCH_DMA_MINALIGN & ~(crypto_tfm_ctx_alignment() - 1);
+#endif
+	ahash->reqsize = reqsize;
+}
+
 static inline struct crypto_instance *ahash_crypto_instance(
 	struct ahash_instance *inst)
 {
@@ -169,6 +183,20 @@ static inline void *ahash_instance_ctx(struct ahash_instance *inst)
 	return crypto_instance_ctx(ahash_crypto_instance(inst));
 }
 
+static inline void *ahash_request_ctx_dma(struct ahash_request *req)
+{
+	unsigned int align = 1;
+
+#ifdef ARCH_DMA_MINALIGN
+	align = ARCH_DMA_MINALIGN;
+#endif
+
+	if (align <= crypto_tfm_ctx_alignment())
+		align = 1;
+
+	return PTR_ALIGN(ahash_request_ctx(req), align);
+}
+
 static inline void ahash_request_complete(struct ahash_request *req, int err)
 {
 	req->base.complete(&req->base, err);

[RFC PATCH 4/7] crypto: hash - Add ctx helpers with DMA alignment

[Herbert Xu]

This patch adds helpers to access the ahash context structure and
request context structure with an added alignment for DMA access.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
---

 include/crypto/internal/hash.h |   28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/include/crypto/internal/hash.h b/include/crypto/internal/hash.h
index 25806141db591..f635a61fd1a0e 100644
--- a/include/crypto/internal/hash.h
+++ b/include/crypto/internal/hash.h
@@ -134,6 +134,11 @@ static inline void *crypto_ahash_ctx(struct crypto_ahash *tfm)
 	return crypto_tfm_ctx(crypto_ahash_tfm(tfm));
 }
 
+static inline void *crypto_ahash_ctx_dma(struct crypto_ahash *tfm)
+{
+	return crypto_tfm_ctx_dma(crypto_ahash_tfm(tfm));
+}
+
 static inline struct ahash_alg *__crypto_ahash_alg(struct crypto_alg *alg)
 {
 	return container_of(__crypto_hash_alg_common(alg), struct ahash_alg,
@@ -146,6 +151,15 @@ static inline void crypto_ahash_set_reqsize(struct crypto_ahash *tfm,
 	tfm->reqsize = reqsize;
 }
 
+static inline void crypto_ahash_set_reqsize_dma(struct crypto_ahash *ahash,
+						unsigned int reqsize)
+{
+#ifdef ARCH_DMA_MINALIGN
+	reqsize += ARCH_DMA_MINALIGN & ~(crypto_tfm_ctx_alignment() - 1);
+#endif
+	ahash->reqsize = reqsize;
+}
+
 static inline struct crypto_instance *ahash_crypto_instance(
 	struct ahash_instance *inst)
 {
@@ -169,6 +183,20 @@ static inline void *ahash_instance_ctx(struct ahash_instance *inst)
 	return crypto_instance_ctx(ahash_crypto_instance(inst));
 }
 
+static inline void *ahash_request_ctx_dma(struct ahash_request *req)
+{
+	unsigned int align = 1;
+
+#ifdef ARCH_DMA_MINALIGN
+	align = ARCH_DMA_MINALIGN;
+#endif
+
+	if (align <= crypto_tfm_ctx_alignment())
+		align = 1;
+
+	return PTR_ALIGN(ahash_request_ctx(req), align);
+}
+
 static inline void ahash_request_complete(struct ahash_request *req, int err)
 {
 	req->base.complete(&req->base, err);

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[RFC PATCH 5/7] crypto: skcipher - Add ctx helpers with DMA alignment

[Herbert Xu]

This patch adds helpers to access the skcipher context structure and
request context structure with an added alignment for DMA access.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
---

 include/crypto/internal/skcipher.h |   28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/include/crypto/internal/skcipher.h b/include/crypto/internal/skcipher.h
index a2339f80a6159..719822e42dc46 100644
--- a/include/crypto/internal/skcipher.h
+++ b/include/crypto/internal/skcipher.h
@@ -122,6 +122,15 @@ static inline void crypto_skcipher_set_reqsize(
 	skcipher->reqsize = reqsize;
 }
 
+static inline void crypto_skcipher_set_reqsize_dma(
+	struct crypto_skcipher *skcipher, unsigned int reqsize)
+{
+#ifdef ARCH_DMA_MINALIGN
+	reqsize += ARCH_DMA_MINALIGN & ~(crypto_tfm_ctx_alignment() - 1);
+#endif
+	skcipher->reqsize = reqsize;
+}
+
 int crypto_register_skcipher(struct skcipher_alg *alg);
 void crypto_unregister_skcipher(struct skcipher_alg *alg);
 int crypto_register_skciphers(struct skcipher_alg *algs, int count);
@@ -151,11 +160,30 @@ static inline void *crypto_skcipher_ctx(struct crypto_skcipher *tfm)
 	return crypto_tfm_ctx(&tfm->base);
 }
 
+static inline void *crypto_skcipher_ctx_dma(struct crypto_skcipher *tfm)
+{
+	return crypto_tfm_ctx_dma(&tfm->base);
+}
+
 static inline void *skcipher_request_ctx(struct skcipher_request *req)
 {
 	return req->__ctx;
 }
 
+static inline void *skcipher_request_ctx_dma(struct skcipher_request *req)
+{
+	unsigned int align = 1;
+
+#ifdef ARCH_DMA_MINALIGN
+	align = ARCH_DMA_MINALIGN;
+#endif
+
+	if (align <= crypto_tfm_ctx_alignment())
+		align = 1;
+
+	return PTR_ALIGN(skcipher_request_ctx(req), align);
+}
+
 static inline u32 skcipher_request_flags(struct skcipher_request *req)
 {
 	return req->base.flags;

[RFC PATCH 5/7] crypto: skcipher - Add ctx helpers with DMA alignment

[Herbert Xu]

This patch adds helpers to access the skcipher context structure and
request context structure with an added alignment for DMA access.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
---

 include/crypto/internal/skcipher.h |   28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/include/crypto/internal/skcipher.h b/include/crypto/internal/skcipher.h
index a2339f80a6159..719822e42dc46 100644
--- a/include/crypto/internal/skcipher.h
+++ b/include/crypto/internal/skcipher.h
@@ -122,6 +122,15 @@ static inline void crypto_skcipher_set_reqsize(
 	skcipher->reqsize = reqsize;
 }
 
+static inline void crypto_skcipher_set_reqsize_dma(
+	struct crypto_skcipher *skcipher, unsigned int reqsize)
+{
+#ifdef ARCH_DMA_MINALIGN
+	reqsize += ARCH_DMA_MINALIGN & ~(crypto_tfm_ctx_alignment() - 1);
+#endif
+	skcipher->reqsize = reqsize;
+}
+
 int crypto_register_skcipher(struct skcipher_alg *alg);
 void crypto_unregister_skcipher(struct skcipher_alg *alg);
 int crypto_register_skciphers(struct skcipher_alg *algs, int count);
@@ -151,11 +160,30 @@ static inline void *crypto_skcipher_ctx(struct crypto_skcipher *tfm)
 	return crypto_tfm_ctx(&tfm->base);
 }
 
+static inline void *crypto_skcipher_ctx_dma(struct crypto_skcipher *tfm)
+{
+	return crypto_tfm_ctx_dma(&tfm->base);
+}
+
 static inline void *skcipher_request_ctx(struct skcipher_request *req)
 {
 	return req->__ctx;
 }
 
+static inline void *skcipher_request_ctx_dma(struct skcipher_request *req)
+{
+	unsigned int align = 1;
+
+#ifdef ARCH_DMA_MINALIGN
+	align = ARCH_DMA_MINALIGN;
+#endif
+
+	if (align <= crypto_tfm_ctx_alignment())
+		align = 1;
+
+	return PTR_ALIGN(skcipher_request_ctx(req), align);
+}
+
 static inline u32 skcipher_request_flags(struct skcipher_request *req)
 {
 	return req->base.flags;

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[RFC PATCH 6/7] crypto: api - Increase MAX_ALGAPI_ALIGNMASK to 127

[Herbert Xu]

Previously we limited the maximum alignment mask to 63.  This
is mostly due to stack usage for shash.  This patch introduces
a separate limit for shash algorithms and increases the general
limit to 127 which is the value that we need for DMA allocations
on arm64.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
---

 crypto/shash.c          |    9 +++++++--
 include/crypto/algapi.h |    2 +-
 2 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/crypto/shash.c b/crypto/shash.c
index 0a0a50cb694f0..0a10477eac060 100644
--- a/crypto/shash.c
+++ b/crypto/shash.c
@@ -18,6 +18,8 @@
 
 #include "internal.h"
 
+#define MAX_SHASH_ALIGNMASK 63
+
 static const struct crypto_type crypto_shash_type;
 
 static int shash_no_setkey(struct crypto_shash *tfm, const u8 *key,
@@ -100,7 +102,7 @@ static int shash_update_unaligned(struct shash_desc *desc, const u8 *data,
 	 * We cannot count on __aligned() working for large values:
 	 * https://patchwork.kernel.org/patch/9507697/
 	 */
-	u8 ubuf[MAX_ALGAPI_ALIGNMASK * 2];
+	u8 ubuf[MAX_SHASH_ALIGNMASK * 2];
 	u8 *buf = PTR_ALIGN(&ubuf[0], alignmask + 1);
 	int err;
 
@@ -142,7 +144,7 @@ static int shash_final_unaligned(struct shash_desc *desc, u8 *out)
 	 * We cannot count on __aligned() working for large values:
 	 * https://patchwork.kernel.org/patch/9507697/
 	 */
-	u8 ubuf[MAX_ALGAPI_ALIGNMASK + HASH_MAX_DIGESTSIZE];
+	u8 ubuf[MAX_SHASH_ALIGNMASK + HASH_MAX_DIGESTSIZE];
 	u8 *buf = PTR_ALIGN(&ubuf[0], alignmask + 1);
 	int err;
 
@@ -530,6 +532,9 @@ static int shash_prepare_alg(struct shash_alg *alg)
 	    alg->statesize > HASH_MAX_STATESIZE)
 		return -EINVAL;
 
+	if (base->cra_alignmask > MAX_SHASH_ALIGNMASK)
+		return -EINVAL;
+
 	if ((alg->export && !alg->import) || (alg->import && !alg->export))
 		return -EINVAL;
 
diff --git a/include/crypto/algapi.h b/include/crypto/algapi.h
index cdf12e51c53a0..16cfd823ee911 100644
--- a/include/crypto/algapi.h
+++ b/include/crypto/algapi.h
@@ -21,7 +21,7 @@
  * algs and architectures. Ciphers have a lower maximum size.
  */
 #define MAX_ALGAPI_BLOCKSIZE		160
-#define MAX_ALGAPI_ALIGNMASK		63
+#define MAX_ALGAPI_ALIGNMASK		127
 #define MAX_CIPHER_BLOCKSIZE		16
 #define MAX_CIPHER_ALIGNMASK		15
 

[RFC PATCH 6/7] crypto: api - Increase MAX_ALGAPI_ALIGNMASK to 127

[Herbert Xu]

Previously we limited the maximum alignment mask to 63.  This
is mostly due to stack usage for shash.  This patch introduces
a separate limit for shash algorithms and increases the general
limit to 127 which is the value that we need for DMA allocations
on arm64.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
---

 crypto/shash.c          |    9 +++++++--
 include/crypto/algapi.h |    2 +-
 2 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/crypto/shash.c b/crypto/shash.c
index 0a0a50cb694f0..0a10477eac060 100644
--- a/crypto/shash.c
+++ b/crypto/shash.c
@@ -18,6 +18,8 @@
 
 #include "internal.h"
 
+#define MAX_SHASH_ALIGNMASK 63
+
 static const struct crypto_type crypto_shash_type;
 
 static int shash_no_setkey(struct crypto_shash *tfm, const u8 *key,
@@ -100,7 +102,7 @@ static int shash_update_unaligned(struct shash_desc *desc, const u8 *data,
 	 * We cannot count on __aligned() working for large values:
 	 * https://patchwork.kernel.org/patch/9507697/
 	 */
-	u8 ubuf[MAX_ALGAPI_ALIGNMASK * 2];
+	u8 ubuf[MAX_SHASH_ALIGNMASK * 2];
 	u8 *buf = PTR_ALIGN(&ubuf[0], alignmask + 1);
 	int err;
 
@@ -142,7 +144,7 @@ static int shash_final_unaligned(struct shash_desc *desc, u8 *out)
 	 * We cannot count on __aligned() working for large values:
 	 * https://patchwork.kernel.org/patch/9507697/
 	 */
-	u8 ubuf[MAX_ALGAPI_ALIGNMASK + HASH_MAX_DIGESTSIZE];
+	u8 ubuf[MAX_SHASH_ALIGNMASK + HASH_MAX_DIGESTSIZE];
 	u8 *buf = PTR_ALIGN(&ubuf[0], alignmask + 1);
 	int err;
 
@@ -530,6 +532,9 @@ static int shash_prepare_alg(struct shash_alg *alg)
 	    alg->statesize > HASH_MAX_STATESIZE)
 		return -EINVAL;
 
+	if (base->cra_alignmask > MAX_SHASH_ALIGNMASK)
+		return -EINVAL;
+
 	if ((alg->export && !alg->import) || (alg->import && !alg->export))
 		return -EINVAL;
 
diff --git a/include/crypto/algapi.h b/include/crypto/algapi.h
index cdf12e51c53a0..16cfd823ee911 100644
--- a/include/crypto/algapi.h
+++ b/include/crypto/algapi.h
@@ -21,7 +21,7 @@
  * algs and architectures. Ciphers have a lower maximum size.
  */
 #define MAX_ALGAPI_BLOCKSIZE		160
-#define MAX_ALGAPI_ALIGNMASK		63
+#define MAX_ALGAPI_ALIGNMASK		127
 #define MAX_CIPHER_BLOCKSIZE		16
 #define MAX_CIPHER_ALIGNMASK		15
 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[RFC PATCH 7/7] crypto: caam - Explicitly request DMA alignment

[Herbert Xu]

This patch uses API helpers to explicitly request for DMA alignment
when allocating memory through the Crypto API.  Previously it was
implicitly assumed that all kmalloc memory is aligned for DMA.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
---

 drivers/crypto/caam/caamalg.c |  101 ++++++++++++++++++++++--------------------
 1 file changed, 55 insertions(+), 46 deletions(-)

diff --git a/drivers/crypto/caam/caamalg.c b/drivers/crypto/caam/caamalg.c
index d3d8bb0a69900..c12c678dcb0fd 100644
--- a/drivers/crypto/caam/caamalg.c
+++ b/drivers/crypto/caam/caamalg.c
@@ -131,7 +131,7 @@ struct caam_aead_req_ctx {
 
 static int aead_null_set_sh_desc(struct crypto_aead *aead)
 {
-	struct caam_ctx *ctx = crypto_aead_ctx(aead);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(aead);
 	struct device *jrdev = ctx->jrdev;
 	struct caam_drv_private *ctrlpriv = dev_get_drvdata(jrdev->parent);
 	u32 *desc;
@@ -184,7 +184,7 @@ static int aead_set_sh_desc(struct crypto_aead *aead)
 	struct caam_aead_alg *alg = container_of(crypto_aead_alg(aead),
 						 struct caam_aead_alg, aead);
 	unsigned int ivsize = crypto_aead_ivsize(aead);
-	struct caam_ctx *ctx = crypto_aead_ctx(aead);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(aead);
 	struct device *jrdev = ctx->jrdev;
 	struct caam_drv_private *ctrlpriv = dev_get_drvdata(jrdev->parent);
 	u32 ctx1_iv_off = 0;
@@ -312,7 +312,7 @@ static int aead_set_sh_desc(struct crypto_aead *aead)
 static int aead_setauthsize(struct crypto_aead *authenc,
 				    unsigned int authsize)
 {
-	struct caam_ctx *ctx = crypto_aead_ctx(authenc);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(authenc);
 
 	ctx->authsize = authsize;
 	aead_set_sh_desc(authenc);
@@ -322,7 +322,7 @@ static int aead_setauthsize(struct crypto_aead *authenc,
 
 static int gcm_set_sh_desc(struct crypto_aead *aead)
 {
-	struct caam_ctx *ctx = crypto_aead_ctx(aead);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(aead);
 	struct device *jrdev = ctx->jrdev;
 	unsigned int ivsize = crypto_aead_ivsize(aead);
 	u32 *desc;
@@ -372,7 +372,7 @@ static int gcm_set_sh_desc(struct crypto_aead *aead)
 
 static int gcm_setauthsize(struct crypto_aead *authenc, unsigned int authsize)
 {
-	struct caam_ctx *ctx = crypto_aead_ctx(authenc);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(authenc);
 	int err;
 
 	err = crypto_gcm_check_authsize(authsize);
@@ -387,7 +387,7 @@ static int gcm_setauthsize(struct crypto_aead *authenc, unsigned int authsize)
 
 static int rfc4106_set_sh_desc(struct crypto_aead *aead)
 {
-	struct caam_ctx *ctx = crypto_aead_ctx(aead);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(aead);
 	struct device *jrdev = ctx->jrdev;
 	unsigned int ivsize = crypto_aead_ivsize(aead);
 	u32 *desc;
@@ -440,7 +440,7 @@ static int rfc4106_set_sh_desc(struct crypto_aead *aead)
 static int rfc4106_setauthsize(struct crypto_aead *authenc,
 			       unsigned int authsize)
 {
-	struct caam_ctx *ctx = crypto_aead_ctx(authenc);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(authenc);
 	int err;
 
 	err = crypto_rfc4106_check_authsize(authsize);
@@ -455,7 +455,7 @@ static int rfc4106_setauthsize(struct crypto_aead *authenc,
 
 static int rfc4543_set_sh_desc(struct crypto_aead *aead)
 {
-	struct caam_ctx *ctx = crypto_aead_ctx(aead);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(aead);
 	struct device *jrdev = ctx->jrdev;
 	unsigned int ivsize = crypto_aead_ivsize(aead);
 	u32 *desc;
@@ -508,7 +508,7 @@ static int rfc4543_set_sh_desc(struct crypto_aead *aead)
 static int rfc4543_setauthsize(struct crypto_aead *authenc,
 			       unsigned int authsize)
 {
-	struct caam_ctx *ctx = crypto_aead_ctx(authenc);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(authenc);
 
 	if (authsize != 16)
 		return -EINVAL;
@@ -521,7 +521,7 @@ static int rfc4543_setauthsize(struct crypto_aead *authenc,
 
 static int chachapoly_set_sh_desc(struct crypto_aead *aead)
 {
-	struct caam_ctx *ctx = crypto_aead_ctx(aead);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(aead);
 	struct device *jrdev = ctx->jrdev;
 	unsigned int ivsize = crypto_aead_ivsize(aead);
 	u32 *desc;
@@ -547,7 +547,7 @@ static int chachapoly_set_sh_desc(struct crypto_aead *aead)
 static int chachapoly_setauthsize(struct crypto_aead *aead,
 				  unsigned int authsize)
 {
-	struct caam_ctx *ctx = crypto_aead_ctx(aead);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(aead);
 
 	if (authsize != POLY1305_DIGEST_SIZE)
 		return -EINVAL;
@@ -559,7 +559,7 @@ static int chachapoly_setauthsize(struct crypto_aead *aead,
 static int chachapoly_setkey(struct crypto_aead *aead, const u8 *key,
 			     unsigned int keylen)
 {
-	struct caam_ctx *ctx = crypto_aead_ctx(aead);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(aead);
 	unsigned int ivsize = crypto_aead_ivsize(aead);
 	unsigned int saltlen = CHACHAPOLY_IV_SIZE - ivsize;
 
@@ -575,7 +575,7 @@ static int chachapoly_setkey(struct crypto_aead *aead, const u8 *key,
 static int aead_setkey(struct crypto_aead *aead,
 			       const u8 *key, unsigned int keylen)
 {
-	struct caam_ctx *ctx = crypto_aead_ctx(aead);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(aead);
 	struct device *jrdev = ctx->jrdev;
 	struct caam_drv_private *ctrlpriv = dev_get_drvdata(jrdev->parent);
 	struct crypto_authenc_keys keys;
@@ -656,7 +656,7 @@ static int des3_aead_setkey(struct crypto_aead *aead, const u8 *key,
 static int gcm_setkey(struct crypto_aead *aead,
 		      const u8 *key, unsigned int keylen)
 {
-	struct caam_ctx *ctx = crypto_aead_ctx(aead);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(aead);
 	struct device *jrdev = ctx->jrdev;
 	int err;
 
@@ -677,7 +677,7 @@ static int gcm_setkey(struct crypto_aead *aead,
 static int rfc4106_setkey(struct crypto_aead *aead,
 			  const u8 *key, unsigned int keylen)
 {
-	struct caam_ctx *ctx = crypto_aead_ctx(aead);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(aead);
 	struct device *jrdev = ctx->jrdev;
 	int err;
 
@@ -703,7 +703,7 @@ static int rfc4106_setkey(struct crypto_aead *aead,
 static int rfc4543_setkey(struct crypto_aead *aead,
 			  const u8 *key, unsigned int keylen)
 {
-	struct caam_ctx *ctx = crypto_aead_ctx(aead);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(aead);
 	struct device *jrdev = ctx->jrdev;
 	int err;
 
@@ -729,7 +729,7 @@ static int rfc4543_setkey(struct crypto_aead *aead,
 static int skcipher_setkey(struct crypto_skcipher *skcipher, const u8 *key,
 			   unsigned int keylen, const u32 ctx1_iv_off)
 {
-	struct caam_ctx *ctx = crypto_skcipher_ctx(skcipher);
+	struct caam_ctx *ctx = crypto_skcipher_ctx_dma(skcipher);
 	struct caam_skcipher_alg *alg =
 		container_of(crypto_skcipher_alg(skcipher), typeof(*alg),
 			     skcipher);
@@ -832,7 +832,7 @@ static int des3_skcipher_setkey(struct crypto_skcipher *skcipher,
 static int xts_skcipher_setkey(struct crypto_skcipher *skcipher, const u8 *key,
 			       unsigned int keylen)
 {
-	struct caam_ctx *ctx = crypto_skcipher_ctx(skcipher);
+	struct caam_ctx *ctx = crypto_skcipher_ctx_dma(skcipher);
 	struct device *jrdev = ctx->jrdev;
 	struct caam_drv_private *ctrlpriv = dev_get_drvdata(jrdev->parent);
 	u32 *desc;
@@ -970,7 +970,7 @@ static void aead_crypt_done(struct device *jrdev, u32 *desc, u32 err,
 			    void *context)
 {
 	struct aead_request *req = context;
-	struct caam_aead_req_ctx *rctx = aead_request_ctx(req);
+	struct caam_aead_req_ctx *rctx = aead_request_ctx_dma(req);
 	struct caam_drv_private_jr *jrp = dev_get_drvdata(jrdev);
 	struct aead_edesc *edesc;
 	int ecode = 0;
@@ -1003,7 +1003,7 @@ static void skcipher_crypt_done(struct device *jrdev, u32 *desc, u32 err,
 {
 	struct skcipher_request *req = context;
 	struct skcipher_edesc *edesc;
-	struct caam_skcipher_req_ctx *rctx = skcipher_request_ctx(req);
+	struct caam_skcipher_req_ctx *rctx = skcipher_request_ctx_dma(req);
 	struct crypto_skcipher *skcipher = crypto_skcipher_reqtfm(req);
 	struct caam_drv_private_jr *jrp = dev_get_drvdata(jrdev);
 	int ivsize = crypto_skcipher_ivsize(skcipher);
@@ -1057,7 +1057,7 @@ static void init_aead_job(struct aead_request *req,
 			  bool all_contig, bool encrypt)
 {
 	struct crypto_aead *aead = crypto_aead_reqtfm(req);
-	struct caam_ctx *ctx = crypto_aead_ctx(aead);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(aead);
 	int authsize = ctx->authsize;
 	u32 *desc = edesc->hw_desc;
 	u32 out_options, in_options;
@@ -1118,7 +1118,7 @@ static void init_gcm_job(struct aead_request *req,
 			 bool all_contig, bool encrypt)
 {
 	struct crypto_aead *aead = crypto_aead_reqtfm(req);
-	struct caam_ctx *ctx = crypto_aead_ctx(aead);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(aead);
 	unsigned int ivsize = crypto_aead_ivsize(aead);
 	u32 *desc = edesc->hw_desc;
 	bool generic_gcm = (ivsize == GCM_AES_IV_SIZE);
@@ -1185,7 +1185,7 @@ static void init_authenc_job(struct aead_request *req,
 	struct caam_aead_alg *alg = container_of(crypto_aead_alg(aead),
 						 struct caam_aead_alg, aead);
 	unsigned int ivsize = crypto_aead_ivsize(aead);
-	struct caam_ctx *ctx = crypto_aead_ctx(aead);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(aead);
 	struct caam_drv_private *ctrlpriv = dev_get_drvdata(ctx->jrdev->parent);
 	const bool ctr_mode = ((ctx->cdata.algtype & OP_ALG_AAI_MASK) ==
 			       OP_ALG_AAI_CTR_MOD128);
@@ -1234,7 +1234,7 @@ static void init_skcipher_job(struct skcipher_request *req,
 			      const bool encrypt)
 {
 	struct crypto_skcipher *skcipher = crypto_skcipher_reqtfm(req);
-	struct caam_ctx *ctx = crypto_skcipher_ctx(skcipher);
+	struct caam_ctx *ctx = crypto_skcipher_ctx_dma(skcipher);
 	struct device *jrdev = ctx->jrdev;
 	int ivsize = crypto_skcipher_ivsize(skcipher);
 	u32 *desc = edesc->hw_desc;
@@ -1290,9 +1290,9 @@ static struct aead_edesc *aead_edesc_alloc(struct aead_request *req,
 					   bool encrypt)
 {
 	struct crypto_aead *aead = crypto_aead_reqtfm(req);
-	struct caam_ctx *ctx = crypto_aead_ctx(aead);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(aead);
 	struct device *jrdev = ctx->jrdev;
-	struct caam_aead_req_ctx *rctx = aead_request_ctx(req);
+	struct caam_aead_req_ctx *rctx = aead_request_ctx_dma(req);
 	gfp_t flags = (req->base.flags & CRYPTO_TFM_REQ_MAY_SLEEP) ?
 		       GFP_KERNEL : GFP_ATOMIC;
 	int src_nents, mapped_src_nents, dst_nents = 0, mapped_dst_nents = 0;
@@ -1429,7 +1429,7 @@ static struct aead_edesc *aead_edesc_alloc(struct aead_request *req,
 static int aead_enqueue_req(struct device *jrdev, struct aead_request *req)
 {
 	struct caam_drv_private_jr *jrpriv = dev_get_drvdata(jrdev);
-	struct caam_aead_req_ctx *rctx = aead_request_ctx(req);
+	struct caam_aead_req_ctx *rctx = aead_request_ctx_dma(req);
 	struct aead_edesc *edesc = rctx->edesc;
 	u32 *desc = edesc->hw_desc;
 	int ret;
@@ -1457,7 +1457,7 @@ static inline int chachapoly_crypt(struct aead_request *req, bool encrypt)
 {
 	struct aead_edesc *edesc;
 	struct crypto_aead *aead = crypto_aead_reqtfm(req);
-	struct caam_ctx *ctx = crypto_aead_ctx(aead);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(aead);
 	struct device *jrdev = ctx->jrdev;
 	bool all_contig;
 	u32 *desc;
@@ -1491,7 +1491,7 @@ static inline int aead_crypt(struct aead_request *req, bool encrypt)
 {
 	struct aead_edesc *edesc;
 	struct crypto_aead *aead = crypto_aead_reqtfm(req);
-	struct caam_ctx *ctx = crypto_aead_ctx(aead);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(aead);
 	struct device *jrdev = ctx->jrdev;
 	bool all_contig;
 
@@ -1524,8 +1524,8 @@ static int aead_decrypt(struct aead_request *req)
 static int aead_do_one_req(struct crypto_engine *engine, void *areq)
 {
 	struct aead_request *req = aead_request_cast(areq);
-	struct caam_ctx *ctx = crypto_aead_ctx(crypto_aead_reqtfm(req));
-	struct caam_aead_req_ctx *rctx = aead_request_ctx(req);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(crypto_aead_reqtfm(req));
+	struct caam_aead_req_ctx *rctx = aead_request_ctx_dma(req);
 	u32 *desc = rctx->edesc->hw_desc;
 	int ret;
 
@@ -1550,7 +1550,7 @@ static inline int gcm_crypt(struct aead_request *req, bool encrypt)
 {
 	struct aead_edesc *edesc;
 	struct crypto_aead *aead = crypto_aead_reqtfm(req);
-	struct caam_ctx *ctx = crypto_aead_ctx(aead);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(aead);
 	struct device *jrdev = ctx->jrdev;
 	bool all_contig;
 
@@ -1597,8 +1597,8 @@ static struct skcipher_edesc *skcipher_edesc_alloc(struct skcipher_request *req,
 						   int desc_bytes)
 {
 	struct crypto_skcipher *skcipher = crypto_skcipher_reqtfm(req);
-	struct caam_ctx *ctx = crypto_skcipher_ctx(skcipher);
-	struct caam_skcipher_req_ctx *rctx = skcipher_request_ctx(req);
+	struct caam_ctx *ctx = crypto_skcipher_ctx_dma(skcipher);
+	struct caam_skcipher_req_ctx *rctx = skcipher_request_ctx_dma(req);
 	struct device *jrdev = ctx->jrdev;
 	gfp_t flags = (req->base.flags & CRYPTO_TFM_REQ_MAY_SLEEP) ?
 		       GFP_KERNEL : GFP_ATOMIC;
@@ -1756,8 +1756,8 @@ static struct skcipher_edesc *skcipher_edesc_alloc(struct skcipher_request *req,
 static int skcipher_do_one_req(struct crypto_engine *engine, void *areq)
 {
 	struct skcipher_request *req = skcipher_request_cast(areq);
-	struct caam_ctx *ctx = crypto_skcipher_ctx(crypto_skcipher_reqtfm(req));
-	struct caam_skcipher_req_ctx *rctx = skcipher_request_ctx(req);
+	struct caam_ctx *ctx = crypto_skcipher_ctx_dma(crypto_skcipher_reqtfm(req));
+	struct caam_skcipher_req_ctx *rctx = skcipher_request_ctx_dma(req);
 	u32 *desc = rctx->edesc->hw_desc;
 	int ret;
 
@@ -1790,7 +1790,7 @@ static inline int skcipher_crypt(struct skcipher_request *req, bool encrypt)
 {
 	struct skcipher_edesc *edesc;
 	struct crypto_skcipher *skcipher = crypto_skcipher_reqtfm(req);
-	struct caam_ctx *ctx = crypto_skcipher_ctx(skcipher);
+	struct caam_ctx *ctx = crypto_skcipher_ctx_dma(skcipher);
 	struct device *jrdev = ctx->jrdev;
 	struct caam_drv_private_jr *jrpriv = dev_get_drvdata(jrdev);
 	struct caam_drv_private *ctrlpriv = dev_get_drvdata(jrdev->parent);
@@ -1807,7 +1807,8 @@ static inline int skcipher_crypt(struct skcipher_request *req, bool encrypt)
 
 	if (ctx->fallback && ((ctrlpriv->era <= 8 && xts_skcipher_ivsize(req)) ||
 			      ctx->xts_key_fallback)) {
-		struct caam_skcipher_req_ctx *rctx = skcipher_request_ctx(req);
+		struct caam_skcipher_req_ctx *rctx =
+			skcipher_request_ctx_dma(req);
 
 		skcipher_request_set_tfm(&rctx->fallback_req, ctx->fallback);
 		skcipher_request_set_callback(&rctx->fallback_req,
@@ -3397,7 +3398,7 @@ static int caam_cra_init(struct crypto_skcipher *tfm)
 	struct skcipher_alg *alg = crypto_skcipher_alg(tfm);
 	struct caam_skcipher_alg *caam_alg =
 		container_of(alg, typeof(*caam_alg), skcipher);
-	struct caam_ctx *ctx = crypto_skcipher_ctx(tfm);
+	struct caam_ctx *ctx = crypto_skcipher_ctx_dma(tfm);
 	u32 alg_aai = caam_alg->caam.class1_alg_type & OP_ALG_AAI_MASK;
 	int ret = 0;
 
@@ -3416,10 +3417,12 @@ static int caam_cra_init(struct crypto_skcipher *tfm)
 		}
 
 		ctx->fallback = fallback;
-		crypto_skcipher_set_reqsize(tfm, sizeof(struct caam_skcipher_req_ctx) +
-					    crypto_skcipher_reqsize(fallback));
+		crypto_skcipher_set_reqsize_dma(
+			tfm, sizeof(struct caam_skcipher_req_ctx) +
+			     crypto_skcipher_reqsize(fallback));
 	} else {
-		crypto_skcipher_set_reqsize(tfm, sizeof(struct caam_skcipher_req_ctx));
+		crypto_skcipher_set_reqsize_dma(
+			tfm, sizeof(struct caam_skcipher_req_ctx));
 	}
 
 	ret = caam_init_common(ctx, &caam_alg->caam, false);
@@ -3434,9 +3437,9 @@ static int caam_aead_init(struct crypto_aead *tfm)
 	struct aead_alg *alg = crypto_aead_alg(tfm);
 	struct caam_aead_alg *caam_alg =
 		 container_of(alg, struct caam_aead_alg, aead);
-	struct caam_ctx *ctx = crypto_aead_ctx(tfm);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(tfm);
 
-	crypto_aead_set_reqsize(tfm, sizeof(struct caam_aead_req_ctx));
+	crypto_aead_set_reqsize_dma(tfm, sizeof(struct caam_aead_req_ctx));
 
 	ctx->enginectx.op.do_one_request = aead_do_one_req;
 
@@ -3454,7 +3457,7 @@ static void caam_exit_common(struct caam_ctx *ctx)
 
 static void caam_cra_exit(struct crypto_skcipher *tfm)
 {
-	struct caam_ctx *ctx = crypto_skcipher_ctx(tfm);
+	struct caam_ctx *ctx = crypto_skcipher_ctx_dma(tfm);
 
 	if (ctx->fallback)
 		crypto_free_skcipher(ctx->fallback);
@@ -3463,7 +3466,7 @@ static void caam_cra_exit(struct crypto_skcipher *tfm)
 
 static void caam_aead_exit(struct crypto_aead *tfm)
 {
-	caam_exit_common(crypto_aead_ctx(tfm));
+	caam_exit_common(crypto_aead_ctx_dma(tfm));
 }
 
 void caam_algapi_exit(void)
@@ -3492,6 +3495,9 @@ static void caam_skcipher_alg_init(struct caam_skcipher_alg *t_alg)
 	alg->base.cra_module = THIS_MODULE;
 	alg->base.cra_priority = CAAM_CRA_PRIORITY;
 	alg->base.cra_ctxsize = sizeof(struct caam_ctx);
+#ifdef ARCH_DMA_MINALIGN
+	alg->base.cra_alignmask = ARCH_DMA_MINALIGN - 1;
+#endif
 	alg->base.cra_flags |= (CRYPTO_ALG_ASYNC | CRYPTO_ALG_ALLOCATES_MEMORY |
 			      CRYPTO_ALG_KERN_DRIVER_ONLY);
 
@@ -3506,6 +3512,9 @@ static void caam_aead_alg_init(struct caam_aead_alg *t_alg)
 	alg->base.cra_module = THIS_MODULE;
 	alg->base.cra_priority = CAAM_CRA_PRIORITY;
 	alg->base.cra_ctxsize = sizeof(struct caam_ctx);
+#ifdef ARCH_DMA_MINALIGN
+	alg->base.cra_alignmask = ARCH_DMA_MINALIGN - 1;
+#endif
 	alg->base.cra_flags = CRYPTO_ALG_ASYNC | CRYPTO_ALG_ALLOCATES_MEMORY |
 			      CRYPTO_ALG_KERN_DRIVER_ONLY;
 

[RFC PATCH 7/7] crypto: caam - Explicitly request DMA alignment

[Herbert Xu]

This patch uses API helpers to explicitly request for DMA alignment
when allocating memory through the Crypto API.  Previously it was
implicitly assumed that all kmalloc memory is aligned for DMA.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
---

 drivers/crypto/caam/caamalg.c |  101 ++++++++++++++++++++++--------------------
 1 file changed, 55 insertions(+), 46 deletions(-)

diff --git a/drivers/crypto/caam/caamalg.c b/drivers/crypto/caam/caamalg.c
index d3d8bb0a69900..c12c678dcb0fd 100644
--- a/drivers/crypto/caam/caamalg.c
+++ b/drivers/crypto/caam/caamalg.c
@@ -131,7 +131,7 @@ struct caam_aead_req_ctx {
 
 static int aead_null_set_sh_desc(struct crypto_aead *aead)
 {
-	struct caam_ctx *ctx = crypto_aead_ctx(aead);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(aead);
 	struct device *jrdev = ctx->jrdev;
 	struct caam_drv_private *ctrlpriv = dev_get_drvdata(jrdev->parent);
 	u32 *desc;
@@ -184,7 +184,7 @@ static int aead_set_sh_desc(struct crypto_aead *aead)
 	struct caam_aead_alg *alg = container_of(crypto_aead_alg(aead),
 						 struct caam_aead_alg, aead);
 	unsigned int ivsize = crypto_aead_ivsize(aead);
-	struct caam_ctx *ctx = crypto_aead_ctx(aead);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(aead);
 	struct device *jrdev = ctx->jrdev;
 	struct caam_drv_private *ctrlpriv = dev_get_drvdata(jrdev->parent);
 	u32 ctx1_iv_off = 0;
@@ -312,7 +312,7 @@ static int aead_set_sh_desc(struct crypto_aead *aead)
 static int aead_setauthsize(struct crypto_aead *authenc,
 				    unsigned int authsize)
 {
-	struct caam_ctx *ctx = crypto_aead_ctx(authenc);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(authenc);
 
 	ctx->authsize = authsize;
 	aead_set_sh_desc(authenc);
@@ -322,7 +322,7 @@ static int aead_setauthsize(struct crypto_aead *authenc,
 
 static int gcm_set_sh_desc(struct crypto_aead *aead)
 {
-	struct caam_ctx *ctx = crypto_aead_ctx(aead);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(aead);
 	struct device *jrdev = ctx->jrdev;
 	unsigned int ivsize = crypto_aead_ivsize(aead);
 	u32 *desc;
@@ -372,7 +372,7 @@ static int gcm_set_sh_desc(struct crypto_aead *aead)
 
 static int gcm_setauthsize(struct crypto_aead *authenc, unsigned int authsize)
 {
-	struct caam_ctx *ctx = crypto_aead_ctx(authenc);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(authenc);
 	int err;
 
 	err = crypto_gcm_check_authsize(authsize);
@@ -387,7 +387,7 @@ static int gcm_setauthsize(struct crypto_aead *authenc, unsigned int authsize)
 
 static int rfc4106_set_sh_desc(struct crypto_aead *aead)
 {
-	struct caam_ctx *ctx = crypto_aead_ctx(aead);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(aead);
 	struct device *jrdev = ctx->jrdev;
 	unsigned int ivsize = crypto_aead_ivsize(aead);
 	u32 *desc;
@@ -440,7 +440,7 @@ static int rfc4106_set_sh_desc(struct crypto_aead *aead)
 static int rfc4106_setauthsize(struct crypto_aead *authenc,
 			       unsigned int authsize)
 {
-	struct caam_ctx *ctx = crypto_aead_ctx(authenc);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(authenc);
 	int err;
 
 	err = crypto_rfc4106_check_authsize(authsize);
@@ -455,7 +455,7 @@ static int rfc4106_setauthsize(struct crypto_aead *authenc,
 
 static int rfc4543_set_sh_desc(struct crypto_aead *aead)
 {
-	struct caam_ctx *ctx = crypto_aead_ctx(aead);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(aead);
 	struct device *jrdev = ctx->jrdev;
 	unsigned int ivsize = crypto_aead_ivsize(aead);
 	u32 *desc;
@@ -508,7 +508,7 @@ static int rfc4543_set_sh_desc(struct crypto_aead *aead)
 static int rfc4543_setauthsize(struct crypto_aead *authenc,
 			       unsigned int authsize)
 {
-	struct caam_ctx *ctx = crypto_aead_ctx(authenc);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(authenc);
 
 	if (authsize != 16)
 		return -EINVAL;
@@ -521,7 +521,7 @@ static int rfc4543_setauthsize(struct crypto_aead *authenc,
 
 static int chachapoly_set_sh_desc(struct crypto_aead *aead)
 {
-	struct caam_ctx *ctx = crypto_aead_ctx(aead);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(aead);
 	struct device *jrdev = ctx->jrdev;
 	unsigned int ivsize = crypto_aead_ivsize(aead);
 	u32 *desc;
@@ -547,7 +547,7 @@ static int chachapoly_set_sh_desc(struct crypto_aead *aead)
 static int chachapoly_setauthsize(struct crypto_aead *aead,
 				  unsigned int authsize)
 {
-	struct caam_ctx *ctx = crypto_aead_ctx(aead);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(aead);
 
 	if (authsize != POLY1305_DIGEST_SIZE)
 		return -EINVAL;
@@ -559,7 +559,7 @@ static int chachapoly_setauthsize(struct crypto_aead *aead,
 static int chachapoly_setkey(struct crypto_aead *aead, const u8 *key,
 			     unsigned int keylen)
 {
-	struct caam_ctx *ctx = crypto_aead_ctx(aead);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(aead);
 	unsigned int ivsize = crypto_aead_ivsize(aead);
 	unsigned int saltlen = CHACHAPOLY_IV_SIZE - ivsize;
 
@@ -575,7 +575,7 @@ static int chachapoly_setkey(struct crypto_aead *aead, const u8 *key,
 static int aead_setkey(struct crypto_aead *aead,
 			       const u8 *key, unsigned int keylen)
 {
-	struct caam_ctx *ctx = crypto_aead_ctx(aead);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(aead);
 	struct device *jrdev = ctx->jrdev;
 	struct caam_drv_private *ctrlpriv = dev_get_drvdata(jrdev->parent);
 	struct crypto_authenc_keys keys;
@@ -656,7 +656,7 @@ static int des3_aead_setkey(struct crypto_aead *aead, const u8 *key,
 static int gcm_setkey(struct crypto_aead *aead,
 		      const u8 *key, unsigned int keylen)
 {
-	struct caam_ctx *ctx = crypto_aead_ctx(aead);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(aead);
 	struct device *jrdev = ctx->jrdev;
 	int err;
 
@@ -677,7 +677,7 @@ static int gcm_setkey(struct crypto_aead *aead,
 static int rfc4106_setkey(struct crypto_aead *aead,
 			  const u8 *key, unsigned int keylen)
 {
-	struct caam_ctx *ctx = crypto_aead_ctx(aead);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(aead);
 	struct device *jrdev = ctx->jrdev;
 	int err;
 
@@ -703,7 +703,7 @@ static int rfc4106_setkey(struct crypto_aead *aead,
 static int rfc4543_setkey(struct crypto_aead *aead,
 			  const u8 *key, unsigned int keylen)
 {
-	struct caam_ctx *ctx = crypto_aead_ctx(aead);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(aead);
 	struct device *jrdev = ctx->jrdev;
 	int err;
 
@@ -729,7 +729,7 @@ static int rfc4543_setkey(struct crypto_aead *aead,
 static int skcipher_setkey(struct crypto_skcipher *skcipher, const u8 *key,
 			   unsigned int keylen, const u32 ctx1_iv_off)
 {
-	struct caam_ctx *ctx = crypto_skcipher_ctx(skcipher);
+	struct caam_ctx *ctx = crypto_skcipher_ctx_dma(skcipher);
 	struct caam_skcipher_alg *alg =
 		container_of(crypto_skcipher_alg(skcipher), typeof(*alg),
 			     skcipher);
@@ -832,7 +832,7 @@ static int des3_skcipher_setkey(struct crypto_skcipher *skcipher,
 static int xts_skcipher_setkey(struct crypto_skcipher *skcipher, const u8 *key,
 			       unsigned int keylen)
 {
-	struct caam_ctx *ctx = crypto_skcipher_ctx(skcipher);
+	struct caam_ctx *ctx = crypto_skcipher_ctx_dma(skcipher);
 	struct device *jrdev = ctx->jrdev;
 	struct caam_drv_private *ctrlpriv = dev_get_drvdata(jrdev->parent);
 	u32 *desc;
@@ -970,7 +970,7 @@ static void aead_crypt_done(struct device *jrdev, u32 *desc, u32 err,
 			    void *context)
 {
 	struct aead_request *req = context;
-	struct caam_aead_req_ctx *rctx = aead_request_ctx(req);
+	struct caam_aead_req_ctx *rctx = aead_request_ctx_dma(req);
 	struct caam_drv_private_jr *jrp = dev_get_drvdata(jrdev);
 	struct aead_edesc *edesc;
 	int ecode = 0;
@@ -1003,7 +1003,7 @@ static void skcipher_crypt_done(struct device *jrdev, u32 *desc, u32 err,
 {
 	struct skcipher_request *req = context;
 	struct skcipher_edesc *edesc;
-	struct caam_skcipher_req_ctx *rctx = skcipher_request_ctx(req);
+	struct caam_skcipher_req_ctx *rctx = skcipher_request_ctx_dma(req);
 	struct crypto_skcipher *skcipher = crypto_skcipher_reqtfm(req);
 	struct caam_drv_private_jr *jrp = dev_get_drvdata(jrdev);
 	int ivsize = crypto_skcipher_ivsize(skcipher);
@@ -1057,7 +1057,7 @@ static void init_aead_job(struct aead_request *req,
 			  bool all_contig, bool encrypt)
 {
 	struct crypto_aead *aead = crypto_aead_reqtfm(req);
-	struct caam_ctx *ctx = crypto_aead_ctx(aead);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(aead);
 	int authsize = ctx->authsize;
 	u32 *desc = edesc->hw_desc;
 	u32 out_options, in_options;
@@ -1118,7 +1118,7 @@ static void init_gcm_job(struct aead_request *req,
 			 bool all_contig, bool encrypt)
 {
 	struct crypto_aead *aead = crypto_aead_reqtfm(req);
-	struct caam_ctx *ctx = crypto_aead_ctx(aead);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(aead);
 	unsigned int ivsize = crypto_aead_ivsize(aead);
 	u32 *desc = edesc->hw_desc;
 	bool generic_gcm = (ivsize == GCM_AES_IV_SIZE);
@@ -1185,7 +1185,7 @@ static void init_authenc_job(struct aead_request *req,
 	struct caam_aead_alg *alg = container_of(crypto_aead_alg(aead),
 						 struct caam_aead_alg, aead);
 	unsigned int ivsize = crypto_aead_ivsize(aead);
-	struct caam_ctx *ctx = crypto_aead_ctx(aead);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(aead);
 	struct caam_drv_private *ctrlpriv = dev_get_drvdata(ctx->jrdev->parent);
 	const bool ctr_mode = ((ctx->cdata.algtype & OP_ALG_AAI_MASK) ==
 			       OP_ALG_AAI_CTR_MOD128);
@@ -1234,7 +1234,7 @@ static void init_skcipher_job(struct skcipher_request *req,
 			      const bool encrypt)
 {
 	struct crypto_skcipher *skcipher = crypto_skcipher_reqtfm(req);
-	struct caam_ctx *ctx = crypto_skcipher_ctx(skcipher);
+	struct caam_ctx *ctx = crypto_skcipher_ctx_dma(skcipher);
 	struct device *jrdev = ctx->jrdev;
 	int ivsize = crypto_skcipher_ivsize(skcipher);
 	u32 *desc = edesc->hw_desc;
@@ -1290,9 +1290,9 @@ static struct aead_edesc *aead_edesc_alloc(struct aead_request *req,
 					   bool encrypt)
 {
 	struct crypto_aead *aead = crypto_aead_reqtfm(req);
-	struct caam_ctx *ctx = crypto_aead_ctx(aead);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(aead);
 	struct device *jrdev = ctx->jrdev;
-	struct caam_aead_req_ctx *rctx = aead_request_ctx(req);
+	struct caam_aead_req_ctx *rctx = aead_request_ctx_dma(req);
 	gfp_t flags = (req->base.flags & CRYPTO_TFM_REQ_MAY_SLEEP) ?
 		       GFP_KERNEL : GFP_ATOMIC;
 	int src_nents, mapped_src_nents, dst_nents = 0, mapped_dst_nents = 0;
@@ -1429,7 +1429,7 @@ static struct aead_edesc *aead_edesc_alloc(struct aead_request *req,
 static int aead_enqueue_req(struct device *jrdev, struct aead_request *req)
 {
 	struct caam_drv_private_jr *jrpriv = dev_get_drvdata(jrdev);
-	struct caam_aead_req_ctx *rctx = aead_request_ctx(req);
+	struct caam_aead_req_ctx *rctx = aead_request_ctx_dma(req);
 	struct aead_edesc *edesc = rctx->edesc;
 	u32 *desc = edesc->hw_desc;
 	int ret;
@@ -1457,7 +1457,7 @@ static inline int chachapoly_crypt(struct aead_request *req, bool encrypt)
 {
 	struct aead_edesc *edesc;
 	struct crypto_aead *aead = crypto_aead_reqtfm(req);
-	struct caam_ctx *ctx = crypto_aead_ctx(aead);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(aead);
 	struct device *jrdev = ctx->jrdev;
 	bool all_contig;
 	u32 *desc;
@@ -1491,7 +1491,7 @@ static inline int aead_crypt(struct aead_request *req, bool encrypt)
 {
 	struct aead_edesc *edesc;
 	struct crypto_aead *aead = crypto_aead_reqtfm(req);
-	struct caam_ctx *ctx = crypto_aead_ctx(aead);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(aead);
 	struct device *jrdev = ctx->jrdev;
 	bool all_contig;
 
@@ -1524,8 +1524,8 @@ static int aead_decrypt(struct aead_request *req)
 static int aead_do_one_req(struct crypto_engine *engine, void *areq)
 {
 	struct aead_request *req = aead_request_cast(areq);
-	struct caam_ctx *ctx = crypto_aead_ctx(crypto_aead_reqtfm(req));
-	struct caam_aead_req_ctx *rctx = aead_request_ctx(req);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(crypto_aead_reqtfm(req));
+	struct caam_aead_req_ctx *rctx = aead_request_ctx_dma(req);
 	u32 *desc = rctx->edesc->hw_desc;
 	int ret;
 
@@ -1550,7 +1550,7 @@ static inline int gcm_crypt(struct aead_request *req, bool encrypt)
 {
 	struct aead_edesc *edesc;
 	struct crypto_aead *aead = crypto_aead_reqtfm(req);
-	struct caam_ctx *ctx = crypto_aead_ctx(aead);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(aead);
 	struct device *jrdev = ctx->jrdev;
 	bool all_contig;
 
@@ -1597,8 +1597,8 @@ static struct skcipher_edesc *skcipher_edesc_alloc(struct skcipher_request *req,
 						   int desc_bytes)
 {
 	struct crypto_skcipher *skcipher = crypto_skcipher_reqtfm(req);
-	struct caam_ctx *ctx = crypto_skcipher_ctx(skcipher);
-	struct caam_skcipher_req_ctx *rctx = skcipher_request_ctx(req);
+	struct caam_ctx *ctx = crypto_skcipher_ctx_dma(skcipher);
+	struct caam_skcipher_req_ctx *rctx = skcipher_request_ctx_dma(req);
 	struct device *jrdev = ctx->jrdev;
 	gfp_t flags = (req->base.flags & CRYPTO_TFM_REQ_MAY_SLEEP) ?
 		       GFP_KERNEL : GFP_ATOMIC;
@@ -1756,8 +1756,8 @@ static struct skcipher_edesc *skcipher_edesc_alloc(struct skcipher_request *req,
 static int skcipher_do_one_req(struct crypto_engine *engine, void *areq)
 {
 	struct skcipher_request *req = skcipher_request_cast(areq);
-	struct caam_ctx *ctx = crypto_skcipher_ctx(crypto_skcipher_reqtfm(req));
-	struct caam_skcipher_req_ctx *rctx = skcipher_request_ctx(req);
+	struct caam_ctx *ctx = crypto_skcipher_ctx_dma(crypto_skcipher_reqtfm(req));
+	struct caam_skcipher_req_ctx *rctx = skcipher_request_ctx_dma(req);
 	u32 *desc = rctx->edesc->hw_desc;
 	int ret;
 
@@ -1790,7 +1790,7 @@ static inline int skcipher_crypt(struct skcipher_request *req, bool encrypt)
 {
 	struct skcipher_edesc *edesc;
 	struct crypto_skcipher *skcipher = crypto_skcipher_reqtfm(req);
-	struct caam_ctx *ctx = crypto_skcipher_ctx(skcipher);
+	struct caam_ctx *ctx = crypto_skcipher_ctx_dma(skcipher);
 	struct device *jrdev = ctx->jrdev;
 	struct caam_drv_private_jr *jrpriv = dev_get_drvdata(jrdev);
 	struct caam_drv_private *ctrlpriv = dev_get_drvdata(jrdev->parent);
@@ -1807,7 +1807,8 @@ static inline int skcipher_crypt(struct skcipher_request *req, bool encrypt)
 
 	if (ctx->fallback && ((ctrlpriv->era <= 8 && xts_skcipher_ivsize(req)) ||
 			      ctx->xts_key_fallback)) {
-		struct caam_skcipher_req_ctx *rctx = skcipher_request_ctx(req);
+		struct caam_skcipher_req_ctx *rctx =
+			skcipher_request_ctx_dma(req);
 
 		skcipher_request_set_tfm(&rctx->fallback_req, ctx->fallback);
 		skcipher_request_set_callback(&rctx->fallback_req,
@@ -3397,7 +3398,7 @@ static int caam_cra_init(struct crypto_skcipher *tfm)
 	struct skcipher_alg *alg = crypto_skcipher_alg(tfm);
 	struct caam_skcipher_alg *caam_alg =
 		container_of(alg, typeof(*caam_alg), skcipher);
-	struct caam_ctx *ctx = crypto_skcipher_ctx(tfm);
+	struct caam_ctx *ctx = crypto_skcipher_ctx_dma(tfm);
 	u32 alg_aai = caam_alg->caam.class1_alg_type & OP_ALG_AAI_MASK;
 	int ret = 0;
 
@@ -3416,10 +3417,12 @@ static int caam_cra_init(struct crypto_skcipher *tfm)
 		}
 
 		ctx->fallback = fallback;
-		crypto_skcipher_set_reqsize(tfm, sizeof(struct caam_skcipher_req_ctx) +
-					    crypto_skcipher_reqsize(fallback));
+		crypto_skcipher_set_reqsize_dma(
+			tfm, sizeof(struct caam_skcipher_req_ctx) +
+			     crypto_skcipher_reqsize(fallback));
 	} else {
-		crypto_skcipher_set_reqsize(tfm, sizeof(struct caam_skcipher_req_ctx));
+		crypto_skcipher_set_reqsize_dma(
+			tfm, sizeof(struct caam_skcipher_req_ctx));
 	}
 
 	ret = caam_init_common(ctx, &caam_alg->caam, false);
@@ -3434,9 +3437,9 @@ static int caam_aead_init(struct crypto_aead *tfm)
 	struct aead_alg *alg = crypto_aead_alg(tfm);
 	struct caam_aead_alg *caam_alg =
 		 container_of(alg, struct caam_aead_alg, aead);
-	struct caam_ctx *ctx = crypto_aead_ctx(tfm);
+	struct caam_ctx *ctx = crypto_aead_ctx_dma(tfm);
 
-	crypto_aead_set_reqsize(tfm, sizeof(struct caam_aead_req_ctx));
+	crypto_aead_set_reqsize_dma(tfm, sizeof(struct caam_aead_req_ctx));
 
 	ctx->enginectx.op.do_one_request = aead_do_one_req;
 
@@ -3454,7 +3457,7 @@ static void caam_exit_common(struct caam_ctx *ctx)
 
 static void caam_cra_exit(struct crypto_skcipher *tfm)
 {
-	struct caam_ctx *ctx = crypto_skcipher_ctx(tfm);
+	struct caam_ctx *ctx = crypto_skcipher_ctx_dma(tfm);
 
 	if (ctx->fallback)
 		crypto_free_skcipher(ctx->fallback);
@@ -3463,7 +3466,7 @@ static void caam_cra_exit(struct crypto_skcipher *tfm)
 
 static void caam_aead_exit(struct crypto_aead *tfm)
 {
-	caam_exit_common(crypto_aead_ctx(tfm));
+	caam_exit_common(crypto_aead_ctx_dma(tfm));
 }
 
 void caam_algapi_exit(void)
@@ -3492,6 +3495,9 @@ static void caam_skcipher_alg_init(struct caam_skcipher_alg *t_alg)
 	alg->base.cra_module = THIS_MODULE;
 	alg->base.cra_priority = CAAM_CRA_PRIORITY;
 	alg->base.cra_ctxsize = sizeof(struct caam_ctx);
+#ifdef ARCH_DMA_MINALIGN
+	alg->base.cra_alignmask = ARCH_DMA_MINALIGN - 1;
+#endif
 	alg->base.cra_flags |= (CRYPTO_ALG_ASYNC | CRYPTO_ALG_ALLOCATES_MEMORY |
 			      CRYPTO_ALG_KERN_DRIVER_ONLY);
 
@@ -3506,6 +3512,9 @@ static void caam_aead_alg_init(struct caam_aead_alg *t_alg)
 	alg->base.cra_module = THIS_MODULE;
 	alg->base.cra_priority = CAAM_CRA_PRIORITY;
 	alg->base.cra_ctxsize = sizeof(struct caam_ctx);
+#ifdef ARCH_DMA_MINALIGN
+	alg->base.cra_alignmask = ARCH_DMA_MINALIGN - 1;
+#endif
 	alg->base.cra_flags = CRYPTO_ALG_ASYNC | CRYPTO_ALG_ALLOCATES_MEMORY |
 			      CRYPTO_ALG_KERN_DRIVER_ONLY;
 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [RFC PATCH 2/7] crypto: api - Add crypto_tfm_ctx_dma

[Catalin Marinas]

Hi Herbert,

Thanks for putting this together.

On Tue, May 10, 2022 at 07:07:10PM +0800, Herbert Xu wrote:
> diff --git a/include/crypto/algapi.h b/include/crypto/algapi.h
> index f50c5d1725da5..cdf12e51c53a0 100644
> --- a/include/crypto/algapi.h
> +++ b/include/crypto/algapi.h
> @@ -189,10 +189,34 @@ static inline void crypto_xor_cpy(u8 *dst, const u8 *src1, const u8 *src2,
>  	}
>  }
>  
> +static inline void *crypto_tfm_ctx(struct crypto_tfm *tfm)
> +{
> +	return tfm->__crt_ctx;
> +}
> +
> +static inline void *crypto_tfm_ctx_align(struct crypto_tfm *tfm,
> +					 unsigned int align)
> +{
> +	if (align <= crypto_tfm_ctx_alignment())
> +		align = 1;
> +
> +	return PTR_ALIGN(crypto_tfm_ctx(tfm), align);
> +}
> +
>  static inline void *crypto_tfm_ctx_aligned(struct crypto_tfm *tfm)
>  {
> -	return PTR_ALIGN(crypto_tfm_ctx(tfm),
> -			 crypto_tfm_alg_alignmask(tfm) + 1);
> +	return crypto_tfm_ctx_align(tfm, crypto_tfm_alg_alignmask(tfm) + 1);
> +}
> +
> +static inline void *crypto_tfm_ctx_dma(struct crypto_tfm *tfm)
> +{
> +	unsigned int align = 1;
> +
> +#ifdef ARCH_DMA_MINALIGN
> +	align = ARCH_DMA_MINALIGN;
> +#endif
> +
> +	return crypto_tfm_ctx_align(tfm, align);
>  }

Is there a case where a driver needs the minimum alignment between
ARCH_DMA_MINALIGN and crypto_tfm_alg_alignmask()+1? Maybe for platforms
where ARCH_DMA_MINALIGN is 8 (fully coherent) but the device's bus
master alignment requirements are higher.

My plan is to have ARCH_DMA_MINALIGN always defined but potentially
higher than ARCH_KMALLOC_MINALIGN on specific architectures. I think
crypto_tfm_ctx_dma() should use ARCH_KMALLOC_MINALIGN (and no #ifdefs)
until I get my patches sorted and I'll replace it with ARCH_DMA_MINALIGN
once it's defined globally (still no #ifdefs). Currently in mainline
it's ARCH_KMALLOC_MINALIGN that gives the static DMA alignment.

With the explicit crypto_tfm_ctx_dma(), can CRYPTO_MINALIGN_ATTR be
dropped entirely? This may be beneficial in reducing the structure size
when no DMA is required.

-- 
Catalin

Re: [RFC PATCH 2/7] crypto: api - Add crypto_tfm_ctx_dma

[Catalin Marinas]

Hi Herbert,

Thanks for putting this together.

On Tue, May 10, 2022 at 07:07:10PM +0800, Herbert Xu wrote:
> diff --git a/include/crypto/algapi.h b/include/crypto/algapi.h
> index f50c5d1725da5..cdf12e51c53a0 100644
> --- a/include/crypto/algapi.h
> +++ b/include/crypto/algapi.h
> @@ -189,10 +189,34 @@ static inline void crypto_xor_cpy(u8 *dst, const u8 *src1, const u8 *src2,
>  	}
>  }
>  
> +static inline void *crypto_tfm_ctx(struct crypto_tfm *tfm)
> +{
> +	return tfm->__crt_ctx;
> +}
> +
> +static inline void *crypto_tfm_ctx_align(struct crypto_tfm *tfm,
> +					 unsigned int align)
> +{
> +	if (align <= crypto_tfm_ctx_alignment())
> +		align = 1;
> +
> +	return PTR_ALIGN(crypto_tfm_ctx(tfm), align);
> +}
> +
>  static inline void *crypto_tfm_ctx_aligned(struct crypto_tfm *tfm)
>  {
> -	return PTR_ALIGN(crypto_tfm_ctx(tfm),
> -			 crypto_tfm_alg_alignmask(tfm) + 1);
> +	return crypto_tfm_ctx_align(tfm, crypto_tfm_alg_alignmask(tfm) + 1);
> +}
> +
> +static inline void *crypto_tfm_ctx_dma(struct crypto_tfm *tfm)
> +{
> +	unsigned int align = 1;
> +
> +#ifdef ARCH_DMA_MINALIGN
> +	align = ARCH_DMA_MINALIGN;
> +#endif
> +
> +	return crypto_tfm_ctx_align(tfm, align);
>  }

Is there a case where a driver needs the minimum alignment between
ARCH_DMA_MINALIGN and crypto_tfm_alg_alignmask()+1? Maybe for platforms
where ARCH_DMA_MINALIGN is 8 (fully coherent) but the device's bus
master alignment requirements are higher.

My plan is to have ARCH_DMA_MINALIGN always defined but potentially
higher than ARCH_KMALLOC_MINALIGN on specific architectures. I think
crypto_tfm_ctx_dma() should use ARCH_KMALLOC_MINALIGN (and no #ifdefs)
until I get my patches sorted and I'll replace it with ARCH_DMA_MINALIGN
once it's defined globally (still no #ifdefs). Currently in mainline
it's ARCH_KMALLOC_MINALIGN that gives the static DMA alignment.

With the explicit crypto_tfm_ctx_dma(), can CRYPTO_MINALIGN_ATTR be
dropped entirely? This may be beneficial in reducing the structure size
when no DMA is required.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [RFC PATCH 2/7] crypto: api - Add crypto_tfm_ctx_dma

[Herbert Xu]

On Tue, May 10, 2022 at 06:10:34PM +0100, Catalin Marinas wrote:
>
> Is there a case where a driver needs the minimum alignment between
> ARCH_DMA_MINALIGN and crypto_tfm_alg_alignmask()+1? Maybe for platforms
> where ARCH_DMA_MINALIGN is 8 (fully coherent) but the device's bus
> master alignment requirements are higher.

Yes, for example on x86 aesni requires 16-byte alignment.

> My plan is to have ARCH_DMA_MINALIGN always defined but potentially
> higher than ARCH_KMALLOC_MINALIGN on specific architectures. I think
> crypto_tfm_ctx_dma() should use ARCH_KMALLOC_MINALIGN (and no #ifdefs)
> until I get my patches sorted and I'll replace it with ARCH_DMA_MINALIGN
> once it's defined globally (still no #ifdefs). Currently in mainline
> it's ARCH_KMALLOC_MINALIGN that gives the static DMA alignment.
> 
> With the explicit crypto_tfm_ctx_dma(), can CRYPTO_MINALIGN_ATTR be
> dropped entirely? This may be beneficial in reducing the structure size
> when no DMA is required.

We always need CRYPTO_MINALIGN to reflect what alignment kmalloc
guarantees.  It is used to minimise the amount of extra padding
for users such aesni.

This shouldn't have any impact on your plans though as once the
drivers in question switch over to the DMA helpers you can safely
lower ARCH_KMALLOC_MINALIGN.

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [RFC PATCH 2/7] crypto: api - Add crypto_tfm_ctx_dma

[Herbert Xu]

On Tue, May 10, 2022 at 06:10:34PM +0100, Catalin Marinas wrote:
>
> Is there a case where a driver needs the minimum alignment between
> ARCH_DMA_MINALIGN and crypto_tfm_alg_alignmask()+1? Maybe for platforms
> where ARCH_DMA_MINALIGN is 8 (fully coherent) but the device's bus
> master alignment requirements are higher.

Yes, for example on x86 aesni requires 16-byte alignment.

> My plan is to have ARCH_DMA_MINALIGN always defined but potentially
> higher than ARCH_KMALLOC_MINALIGN on specific architectures. I think
> crypto_tfm_ctx_dma() should use ARCH_KMALLOC_MINALIGN (and no #ifdefs)
> until I get my patches sorted and I'll replace it with ARCH_DMA_MINALIGN
> once it's defined globally (still no #ifdefs). Currently in mainline
> it's ARCH_KMALLOC_MINALIGN that gives the static DMA alignment.
> 
> With the explicit crypto_tfm_ctx_dma(), can CRYPTO_MINALIGN_ATTR be
> dropped entirely? This may be beneficial in reducing the structure size
> when no DMA is required.

We always need CRYPTO_MINALIGN to reflect what alignment kmalloc
guarantees.  It is used to minimise the amount of extra padding
for users such aesni.

This shouldn't have any impact on your plans though as once the
drivers in question switch over to the DMA helpers you can safely
lower ARCH_KMALLOC_MINALIGN.

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Isaac Manjarres]

On Sun, Apr 17, 2022 at 05:29:01PM +0100, Catalin Marinas wrote:
> On Sun, Apr 17, 2022 at 04:43:33PM +0800, Herbert Xu wrote:
> > On Sun, Apr 17, 2022 at 09:38:40AM +0100, Catalin Marinas wrote:
> > > I don't think we need to do anything here. A structure like:
> > > 
> > > struct x {
> > > 	char y;
> > > 	char z[] CRYPTO_MINALIGN_ATTR;
> > > };
> > > 
> > > is already of size 128. Without CRYPTO_MINALIGN_ATTR, its size would be
> > > 1 but otherwise the whole structure inherits the alignment of its
> > > member and this translates into an aligned size.
> > 
> > No we should not lie to the compiler,
> 
> We won't if we ensure that a structure with sizeof() >= 128 is aligned
> to 128.
> 
Right. kmalloc() should return a 128 byte aligned pointer as long as
the size of the allocation is >= 128 bytes, and the kmalloc-192 cache
isn't present. So, the current behavior that crypto is relying on
wouldn't change, so I agree with Catalin that we wouldn't be lying to
the compiler if we move forward with getting rid of kmalloc-192.

FWIW, I did a comparison on my machine with and without kmalloc-192, and
the amount of memory usage that increased from allocations being redirected to
kmalloc-256 was about 0.4-0.5 MB, which doesn't seem too bad.

> > we have code elsewhere
> > that uses the alignment to compute the amount of extra padding
> > needed to create greater padding.  If CRYPTO_MINALIGN is misleading
> > then that calculation will fall apart.
I don't think it would be misleading. If all of your allocations
are >= CRYPTO_MINALIGN == ARCH_DMA_MINALIGN in size, and
kmalloc()--with kmalloc-192 removed--returns buffers that are aligned to a
power of 2, and are big enough to accomodate your allocation, then wouldn't
they always be CYRPTO_MINALIGN'ed, so your calculation would still be fine?

--Isaac

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Isaac Manjarres]

On Sun, Apr 17, 2022 at 05:29:01PM +0100, Catalin Marinas wrote:
> On Sun, Apr 17, 2022 at 04:43:33PM +0800, Herbert Xu wrote:
> > On Sun, Apr 17, 2022 at 09:38:40AM +0100, Catalin Marinas wrote:
> > > I don't think we need to do anything here. A structure like:
> > > 
> > > struct x {
> > > 	char y;
> > > 	char z[] CRYPTO_MINALIGN_ATTR;
> > > };
> > > 
> > > is already of size 128. Without CRYPTO_MINALIGN_ATTR, its size would be
> > > 1 but otherwise the whole structure inherits the alignment of its
> > > member and this translates into an aligned size.
> > 
> > No we should not lie to the compiler,
> 
> We won't if we ensure that a structure with sizeof() >= 128 is aligned
> to 128.
> 
Right. kmalloc() should return a 128 byte aligned pointer as long as
the size of the allocation is >= 128 bytes, and the kmalloc-192 cache
isn't present. So, the current behavior that crypto is relying on
wouldn't change, so I agree with Catalin that we wouldn't be lying to
the compiler if we move forward with getting rid of kmalloc-192.

FWIW, I did a comparison on my machine with and without kmalloc-192, and
the amount of memory usage that increased from allocations being redirected to
kmalloc-256 was about 0.4-0.5 MB, which doesn't seem too bad.

> > we have code elsewhere
> > that uses the alignment to compute the amount of extra padding
> > needed to create greater padding.  If CRYPTO_MINALIGN is misleading
> > then that calculation will fall apart.
I don't think it would be misleading. If all of your allocations
are >= CRYPTO_MINALIGN == ARCH_DMA_MINALIGN in size, and
kmalloc()--with kmalloc-192 removed--returns buffers that are aligned to a
power of 2, and are big enough to accomodate your allocation, then wouldn't
they always be CYRPTO_MINALIGN'ed, so your calculation would still be fine?

--Isaac

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Herbert Xu]

On Fri, Jul 15, 2022 at 03:23:25PM -0700, Isaac Manjarres wrote:
>
> isn't present. So, the current behavior that crypto is relying on
> wouldn't change, so I agree with Catalin that we wouldn't be lying to
> the compiler if we move forward with getting rid of kmalloc-192.

There is no guarantee that crypto will always be allocating
structures > 128 bytes.

But thanks for the reminder, I do need to push the patches along.

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Herbert Xu]

On Fri, Jul 15, 2022 at 03:23:25PM -0700, Isaac Manjarres wrote:
>
> isn't present. So, the current behavior that crypto is relying on
> wouldn't change, so I agree with Catalin that we wouldn't be lying to
> the compiler if we move forward with getting rid of kmalloc-192.

There is no guarantee that crypto will always be allocating
structures > 128 bytes.

But thanks for the reminder, I do need to push the patches along.

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Sat, Jul 16, 2022 at 11:25:35AM +0800, Herbert Xu wrote:
> On Fri, Jul 15, 2022 at 03:23:25PM -0700, Isaac Manjarres wrote:
> > isn't present. So, the current behavior that crypto is relying on
> > wouldn't change, so I agree with Catalin that we wouldn't be lying to
> > the compiler if we move forward with getting rid of kmalloc-192.
> 
> There is no guarantee that crypto will always be allocating
> structures > 128 bytes.
> 
> But thanks for the reminder, I do need to push the patches along.

So do I but holidays get in the way ;). I plan to refresh my kmalloc
minalign series at the end of August.

One significant change I have though is that now ARCH_KMALLOC_MINALIGN
now goes down all the way to 8 and using swiotlb bounce buffering if the
DMA mapping size is small.

-- 
Catalin

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Sat, Jul 16, 2022 at 11:25:35AM +0800, Herbert Xu wrote:
> On Fri, Jul 15, 2022 at 03:23:25PM -0700, Isaac Manjarres wrote:
> > isn't present. So, the current behavior that crypto is relying on
> > wouldn't change, so I agree with Catalin that we wouldn't be lying to
> > the compiler if we move forward with getting rid of kmalloc-192.
> 
> There is no guarantee that crypto will always be allocating
> structures > 128 bytes.
> 
> But thanks for the reminder, I do need to push the patches along.

So do I but holidays get in the way ;). I plan to refresh my kmalloc
minalign series at the end of August.

One significant change I have though is that now ARCH_KMALLOC_MINALIGN
now goes down all the way to 8 and using swiotlb bounce buffering if the
DMA mapping size is small.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Isaac Manjarres]

On Mon, Jul 18, 2022 at 06:53:43PM +0100, Catalin Marinas wrote:
> So do I but holidays get in the way ;). I plan to refresh my kmalloc
> minalign series at the end of August.
> 
> One significant change I have though is that now ARCH_KMALLOC_MINALIGN
> now goes down all the way to 8 and using swiotlb bounce buffering if the
> DMA mapping size is small.

Hi Catalin,

This sounds like a good idea for optimizing kmalloc's memory usage
beyond what this series originally achieved. I'm sure a few other things
have come up in the meantime, but I was curious to know if you had a
chance to get back to this?

Thanks,
Isaac

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Isaac Manjarres]

On Mon, Jul 18, 2022 at 06:53:43PM +0100, Catalin Marinas wrote:
> So do I but holidays get in the way ;). I plan to refresh my kmalloc
> minalign series at the end of August.
> 
> One significant change I have though is that now ARCH_KMALLOC_MINALIGN
> now goes down all the way to 8 and using swiotlb bounce buffering if the
> DMA mapping size is small.

Hi Catalin,

This sounds like a good idea for optimizing kmalloc's memory usage
beyond what this series originally achieved. I'm sure a few other things
have come up in the meantime, but I was curious to know if you had a
chance to get back to this?

Thanks,
Isaac

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Tue, Sep 20, 2022 at 05:47:33PM -0700, Isaac Manjarres wrote:
> On Mon, Jul 18, 2022 at 06:53:43PM +0100, Catalin Marinas wrote:
> > So do I but holidays get in the way ;). I plan to refresh my kmalloc
> > minalign series at the end of August.
> > 
> > One significant change I have though is that now ARCH_KMALLOC_MINALIGN
> > now goes down all the way to 8 and using swiotlb bounce buffering if the
> > DMA mapping size is small.
> 
> This sounds like a good idea for optimizing kmalloc's memory usage
> beyond what this series originally achieved. I'm sure a few other things
> have come up in the meantime, but I was curious to know if you had a
> chance to get back to this?

I started refreshing the series but I got stuck on having to do bouncing
for small buffers even if when they go through the iommu (and I don't
have the set up to test it yet).

I hope to post something when the merging window closes. In the
meantime, I pushed my work-in-progress patches to:

git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux devel/kmalloc-minalign-v2-wip

Apart from the iommu think, I still need to figure out whether with an
ARCH_DMA_MINALIGN of 128 we need to disable the kmalloc-192 cache (so
far I don't think it's needed). There are no additional changes to the
crypto code from the last series, I still set CRYPTO_MINALIGN to
ARCH_DMA_MINALIGN (without any other patches, crypto DMA will break; so
I need to see how it interacts with Herbert's series).

Anyway, I hope for more discussions once 6.1-rc1 is out.

-- 
Catalin

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Tue, Sep 20, 2022 at 05:47:33PM -0700, Isaac Manjarres wrote:
> On Mon, Jul 18, 2022 at 06:53:43PM +0100, Catalin Marinas wrote:
> > So do I but holidays get in the way ;). I plan to refresh my kmalloc
> > minalign series at the end of August.
> > 
> > One significant change I have though is that now ARCH_KMALLOC_MINALIGN
> > now goes down all the way to 8 and using swiotlb bounce buffering if the
> > DMA mapping size is small.
> 
> This sounds like a good idea for optimizing kmalloc's memory usage
> beyond what this series originally achieved. I'm sure a few other things
> have come up in the meantime, but I was curious to know if you had a
> chance to get back to this?

I started refreshing the series but I got stuck on having to do bouncing
for small buffers even if when they go through the iommu (and I don't
have the set up to test it yet).

I hope to post something when the merging window closes. In the
meantime, I pushed my work-in-progress patches to:

git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux devel/kmalloc-minalign-v2-wip

Apart from the iommu think, I still need to figure out whether with an
ARCH_DMA_MINALIGN of 128 we need to disable the kmalloc-192 cache (so
far I don't think it's needed). There are no additional changes to the
crypto code from the last series, I still set CRYPTO_MINALIGN to
ARCH_DMA_MINALIGN (without any other patches, crypto DMA will break; so
I need to see how it interacts with Herbert's series).

Anyway, I hope for more discussions once 6.1-rc1 is out.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Linus Torvalds]

On Fri, Sep 30, 2022 at 11:33 AM Catalin Marinas
<catalin.marinas@arm.com> wrote:
>
> I started refreshing the series but I got stuck on having to do bouncing
> for small buffers even if when they go through the iommu (and I don't
> have the set up to test it yet).

May I suggest doing that "force bouncing" and "change kmalloc to have
a 8-byte minalign" to be the two first commits?

IOW, if we force bouncing for unaligned DMA, then that *should* mean
that allocation alignment is no longer a correctness issue, it's
purely a performance one due to the bouncing.

So then the rest of the series should be about "ok, this is actually a
hot enough allocation that I want to force alignment", and be purely
about performance, not correctness.

No?

               Linus

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Linus Torvalds]

On Fri, Sep 30, 2022 at 11:33 AM Catalin Marinas
<catalin.marinas@arm.com> wrote:
>
> I started refreshing the series but I got stuck on having to do bouncing
> for small buffers even if when they go through the iommu (and I don't
> have the set up to test it yet).

May I suggest doing that "force bouncing" and "change kmalloc to have
a 8-byte minalign" to be the two first commits?

IOW, if we force bouncing for unaligned DMA, then that *should* mean
that allocation alignment is no longer a correctness issue, it's
purely a performance one due to the bouncing.

So then the rest of the series should be about "ok, this is actually a
hot enough allocation that I want to force alignment", and be purely
about performance, not correctness.

No?

               Linus

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Fri, Sep 30, 2022 at 12:35:45PM -0700, Linus Torvalds wrote:
> On Fri, Sep 30, 2022 at 11:33 AM Catalin Marinas
> <catalin.marinas@arm.com> wrote:
> > I started refreshing the series but I got stuck on having to do bouncing
> > for small buffers even if when they go through the iommu (and I don't
> > have the set up to test it yet).
> 
> May I suggest doing that "force bouncing" and "change kmalloc to have
> a 8-byte minalign" to be the two first commits?
> 
> IOW, if we force bouncing for unaligned DMA, then that *should* mean
> that allocation alignment is no longer a correctness issue, it's
> purely a performance one due to the bouncing.

I've been thinking about this and even ended up with a CBMC model
(included below; it found a bug in dma_kmalloc_needs_bounce()).

The "force bouncing" in my series currently only checks for small
(potentially kmalloc'ed) sizes under the assumption that intra-object
DMA buffers were properly aligned to 128. So for something like below:

struct devres {
	struct devres_node		node;
	u8 __aligned(ARCH_DMA_MINALIGN) data[];
};

we'd need ARCH_DMA_MINALIGN of 128 even if ARCH_KMALLOC_MINALIGN is 8.
Original the code has __aligned(ARCH_KMALLOC_MINALIGN), so lowering the
latter to 8 without any changes would be problematic (the sizeof(devres)
may be sufficiently large to look cacheline-aligned).

If data[] contains a single DMA buffer, dma_kmalloc_needs_bounce() can
get the start of the buffer as another parameter and check that it's a
multiple of cache_line_size().

However, things get more complicated if data[] is used for several
sub-allocations of multiples of ARCH_KMALLOC_MINALIGN. Not much to do
with kmalloc() caches at this point. I haven't got my head around the
crypto code but it looked to me like it needs ARCH_DMA_MINALIGN in some
places if we are to lower ARCH_KMALLOC_MINALIGN. We could attempt to
force bouncing in dma_kmalloc_needs_bounce() by:

	if (ptr % dma_align != || size % dma_align != 0)
		return true;

but that's orthogonal to the kmalloc caches. I tried this some years ago
and IIRC many buffers get bounced even with ARCH_KMALLOC_MINALIGN of 128
because drivers don't necessarily have cacheline-aligned sized
structures shared with devices (but they are allocated from a
cacheline-aligned slab). So this check results in unnecessary bouncing.

So my series attempts to (1) fix the (static) alignment for intra-object
buffers by changing a few ARCH_KMALLOC_MINALIGN uses to
ARCH_DMA_MINALIGN and (2) address the kmalloc() DMA safety by bouncing
non-cacheline-aligned sizes. I don't think we can do (2) first as the
logic for handling (1) in the absence of a large ARCH_DMA_MINALIGN is
different.

And that's the CMBC model:

------------------------------------8<----------------------------
// SPDX-License-Identifier: GPL-2.0-only
/*
 * Check with:
 *   cbmc --trace dma-bounce.c
 */

#define PAGE_SIZE		4096

#define ARCH_KMALLOC_MINALIGN	8
#define ARCH_DMA_MINALIGN	128

#define KMALLOC_MIN_SIZE	ARCH_KMALLOC_MINALIGN
#define KMALLOC_SHIFT_LOW	3
#define KMALLOC_SHIFT_HIGH	25
#define KMALLOC_MAX_SIZE	(1UL << KMALLOC_SHIFT_HIGH)

#define INIT_KMALLOC_INFO(__size, __short_size)			\
{								\
	.size = __size,						\
}

static unsigned int nondet_uint(void);

struct kmalloc_info_struct {
	unsigned int size;
};

struct kmalloc_slab {
	unsigned int ptr;
	unsigned int size;
};

static const struct kmalloc_info_struct kmalloc_info[] = {
	INIT_KMALLOC_INFO(0, 0),
	INIT_KMALLOC_INFO(96, 96),
	INIT_KMALLOC_INFO(192, 192),
	INIT_KMALLOC_INFO(8, 8),
	INIT_KMALLOC_INFO(16, 16),
	INIT_KMALLOC_INFO(32, 32),
	INIT_KMALLOC_INFO(64, 64),
	INIT_KMALLOC_INFO(128, 128),
	INIT_KMALLOC_INFO(256, 256),
	INIT_KMALLOC_INFO(512, 512),
	INIT_KMALLOC_INFO(1024, 1k),
	INIT_KMALLOC_INFO(2048, 2k),
	INIT_KMALLOC_INFO(4096, 4k),
	INIT_KMALLOC_INFO(8192, 8k),
	INIT_KMALLOC_INFO(16384, 16k),
	INIT_KMALLOC_INFO(32768, 32k),
	INIT_KMALLOC_INFO(65536, 64k),
	INIT_KMALLOC_INFO(131072, 128k),
	INIT_KMALLOC_INFO(262144, 256k),
	INIT_KMALLOC_INFO(524288, 512k),
	INIT_KMALLOC_INFO(1048576, 1M),
	INIT_KMALLOC_INFO(2097152, 2M),
	INIT_KMALLOC_INFO(4194304, 4M),
	INIT_KMALLOC_INFO(8388608, 8M),
	INIT_KMALLOC_INFO(16777216, 16M),
	INIT_KMALLOC_INFO(33554432, 32M)
};

static unsigned int cache_line_size(void)
{
	static const unsigned int cls = nondet_uint();

	__CPROVER_assume(cls == 32 || cls == 64 || cls == 128);

	return cls;
}

static unsigned int kmalloc_index(unsigned int size)
{
	if (!size)
		return 0;

	if (size <= KMALLOC_MIN_SIZE)
		return KMALLOC_SHIFT_LOW;

	if (KMALLOC_MIN_SIZE <= 32 && size > 64 && size <= 96)
		return 1;
	if (KMALLOC_MIN_SIZE <= 64 && size > 128 && size <= 192)
		return 2;
	if (size <=          8) return 3;
	if (size <=         16) return 4;
	if (size <=         32) return 5;
	if (size <=         64) return 6;
	if (size <=        128) return 7;
	if (size <=        256) return 8;
	if (size <=        512) return 9;
	if (size <=       1024) return 10;
	if (size <=   2 * 1024) return 11;
	if (size <=   4 * 1024) return 12;
	if (size <=   8 * 1024) return 13;
	if (size <=  16 * 1024) return 14;
	if (size <=  32 * 1024) return 15;
	if (size <=  64 * 1024) return 16;
	if (size <= 128 * 1024) return 17;
	if (size <= 256 * 1024) return 18;
	if (size <= 512 * 1024) return 19;
	if (size <= 1024 * 1024) return 20;
	if (size <=  2 * 1024 * 1024) return 21;
	if (size <=  4 * 1024 * 1024) return 22;
	if (size <=  8 * 1024 * 1024) return 23;
	if (size <=  16 * 1024 * 1024) return 24;
	if (size <=  32 * 1024 * 1024) return 25;

	__CPROVER_assert(0, "Invalid kmalloc() size");

	return -1;
}

unsigned int kmalloc(unsigned int size, struct kmalloc_slab *slab)
{
	unsigned int nr = nondet_uint();

	slab->size = kmalloc_info[kmalloc_index(size)].size;
	slab->ptr = nr * slab->size;

	__CPROVER_assume(slab->ptr < PAGE_SIZE);
	__CPROVER_assume(slab->ptr % slab->size == 0);

	return slab->ptr;
}

/*
 * Implemented only for 32, 64 and 128 cache line sizes.
 */
int dma_kmalloc_needs_bounce(unsigned int size)
{
	unsigned int dma_align = cache_line_size();

	/*
	 * Less than half dma_align, there's definitely a smaller kmalloc()
	 * cache.
	 */
	if (size <= dma_align / 2)
		return 1;

	/*
	 * From this point, any kmalloc cache size is 32-byte aligned.
	 */
	if (dma_align == 32)
		return 0;

	/*
	 * dma_align == 64 => 96 needs bouncing.
	 * dma_align == 128 => 96 and 192 need bouncing.
	 */
	if (size > 64 && size <= 96)
		return 1;
	if (dma_align == 128 && size > 128 && size <= 192)
		return 1;

	return 0;
}

/*
 * Simulate DMA cache maintenance. The 'slab' object is only used for
 * verification.
 */
void dma_map_single(unsigned int ptr, unsigned int size,
		    struct kmalloc_slab *slab)
{
	unsigned int mask = cache_line_size() - 1;

	if (dma_kmalloc_needs_bounce(size)) {
		/* was the bounce really necessary? */
		__CPROVER_assert((ptr & mask) != 0 || (size & mask) != 0,
				 "Bouncing aligned DMA buffer");
		return;
	}

	/*
	 * Check for cache maintenance outside the kmalloc'ed object. We don't
	 * care about intra-object overlap, it's the caller's responsibility
	 * to ensure alignment.
	 */
	__CPROVER_assert((ptr & ~mask) >= slab->ptr, "DMA cache maintenance underflow");
	__CPROVER_assert(((ptr + size + mask) & ~mask) <= slab->ptr + slab->size,
			 "DMA cache maintenance overflow");
}

int main(void)
{
	struct kmalloc_slab slab;
	unsigned int size = nondet_uint();
	unsigned int offset = nondet_uint();
	unsigned int ptr;

	__CPROVER_assume(size <= KMALLOC_MAX_SIZE);
	__CPROVER_assume(offset < size);
	__CPROVER_assume(offset % ARCH_DMA_MINALIGN == 0);

	ptr = kmalloc(size, &slab);
	dma_map_single(ptr + offset, size - offset, &slab);

	return 0;
}

-- 
Catalin

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Fri, Sep 30, 2022 at 12:35:45PM -0700, Linus Torvalds wrote:
> On Fri, Sep 30, 2022 at 11:33 AM Catalin Marinas
> <catalin.marinas@arm.com> wrote:
> > I started refreshing the series but I got stuck on having to do bouncing
> > for small buffers even if when they go through the iommu (and I don't
> > have the set up to test it yet).
> 
> May I suggest doing that "force bouncing" and "change kmalloc to have
> a 8-byte minalign" to be the two first commits?
> 
> IOW, if we force bouncing for unaligned DMA, then that *should* mean
> that allocation alignment is no longer a correctness issue, it's
> purely a performance one due to the bouncing.

I've been thinking about this and even ended up with a CBMC model
(included below; it found a bug in dma_kmalloc_needs_bounce()).

The "force bouncing" in my series currently only checks for small
(potentially kmalloc'ed) sizes under the assumption that intra-object
DMA buffers were properly aligned to 128. So for something like below:

struct devres {
	struct devres_node		node;
	u8 __aligned(ARCH_DMA_MINALIGN) data[];
};

we'd need ARCH_DMA_MINALIGN of 128 even if ARCH_KMALLOC_MINALIGN is 8.
Original the code has __aligned(ARCH_KMALLOC_MINALIGN), so lowering the
latter to 8 without any changes would be problematic (the sizeof(devres)
may be sufficiently large to look cacheline-aligned).

If data[] contains a single DMA buffer, dma_kmalloc_needs_bounce() can
get the start of the buffer as another parameter and check that it's a
multiple of cache_line_size().

However, things get more complicated if data[] is used for several
sub-allocations of multiples of ARCH_KMALLOC_MINALIGN. Not much to do
with kmalloc() caches at this point. I haven't got my head around the
crypto code but it looked to me like it needs ARCH_DMA_MINALIGN in some
places if we are to lower ARCH_KMALLOC_MINALIGN. We could attempt to
force bouncing in dma_kmalloc_needs_bounce() by:

	if (ptr % dma_align != || size % dma_align != 0)
		return true;

but that's orthogonal to the kmalloc caches. I tried this some years ago
and IIRC many buffers get bounced even with ARCH_KMALLOC_MINALIGN of 128
because drivers don't necessarily have cacheline-aligned sized
structures shared with devices (but they are allocated from a
cacheline-aligned slab). So this check results in unnecessary bouncing.

So my series attempts to (1) fix the (static) alignment for intra-object
buffers by changing a few ARCH_KMALLOC_MINALIGN uses to
ARCH_DMA_MINALIGN and (2) address the kmalloc() DMA safety by bouncing
non-cacheline-aligned sizes. I don't think we can do (2) first as the
logic for handling (1) in the absence of a large ARCH_DMA_MINALIGN is
different.

And that's the CMBC model:

------------------------------------8<----------------------------
// SPDX-License-Identifier: GPL-2.0-only
/*
 * Check with:
 *   cbmc --trace dma-bounce.c
 */

#define PAGE_SIZE		4096

#define ARCH_KMALLOC_MINALIGN	8
#define ARCH_DMA_MINALIGN	128

#define KMALLOC_MIN_SIZE	ARCH_KMALLOC_MINALIGN
#define KMALLOC_SHIFT_LOW	3
#define KMALLOC_SHIFT_HIGH	25
#define KMALLOC_MAX_SIZE	(1UL << KMALLOC_SHIFT_HIGH)

#define INIT_KMALLOC_INFO(__size, __short_size)			\
{								\
	.size = __size,						\
}

static unsigned int nondet_uint(void);

struct kmalloc_info_struct {
	unsigned int size;
};

struct kmalloc_slab {
	unsigned int ptr;
	unsigned int size;
};

static const struct kmalloc_info_struct kmalloc_info[] = {
	INIT_KMALLOC_INFO(0, 0),
	INIT_KMALLOC_INFO(96, 96),
	INIT_KMALLOC_INFO(192, 192),
	INIT_KMALLOC_INFO(8, 8),
	INIT_KMALLOC_INFO(16, 16),
	INIT_KMALLOC_INFO(32, 32),
	INIT_KMALLOC_INFO(64, 64),
	INIT_KMALLOC_INFO(128, 128),
	INIT_KMALLOC_INFO(256, 256),
	INIT_KMALLOC_INFO(512, 512),
	INIT_KMALLOC_INFO(1024, 1k),
	INIT_KMALLOC_INFO(2048, 2k),
	INIT_KMALLOC_INFO(4096, 4k),
	INIT_KMALLOC_INFO(8192, 8k),
	INIT_KMALLOC_INFO(16384, 16k),
	INIT_KMALLOC_INFO(32768, 32k),
	INIT_KMALLOC_INFO(65536, 64k),
	INIT_KMALLOC_INFO(131072, 128k),
	INIT_KMALLOC_INFO(262144, 256k),
	INIT_KMALLOC_INFO(524288, 512k),
	INIT_KMALLOC_INFO(1048576, 1M),
	INIT_KMALLOC_INFO(2097152, 2M),
	INIT_KMALLOC_INFO(4194304, 4M),
	INIT_KMALLOC_INFO(8388608, 8M),
	INIT_KMALLOC_INFO(16777216, 16M),
	INIT_KMALLOC_INFO(33554432, 32M)
};

static unsigned int cache_line_size(void)
{
	static const unsigned int cls = nondet_uint();

	__CPROVER_assume(cls == 32 || cls == 64 || cls == 128);

	return cls;
}

static unsigned int kmalloc_index(unsigned int size)
{
	if (!size)
		return 0;

	if (size <= KMALLOC_MIN_SIZE)
		return KMALLOC_SHIFT_LOW;

	if (KMALLOC_MIN_SIZE <= 32 && size > 64 && size <= 96)
		return 1;
	if (KMALLOC_MIN_SIZE <= 64 && size > 128 && size <= 192)
		return 2;
	if (size <=          8) return 3;
	if (size <=         16) return 4;
	if (size <=         32) return 5;
	if (size <=         64) return 6;
	if (size <=        128) return 7;
	if (size <=        256) return 8;
	if (size <=        512) return 9;
	if (size <=       1024) return 10;
	if (size <=   2 * 1024) return 11;
	if (size <=   4 * 1024) return 12;
	if (size <=   8 * 1024) return 13;
	if (size <=  16 * 1024) return 14;
	if (size <=  32 * 1024) return 15;
	if (size <=  64 * 1024) return 16;
	if (size <= 128 * 1024) return 17;
	if (size <= 256 * 1024) return 18;
	if (size <= 512 * 1024) return 19;
	if (size <= 1024 * 1024) return 20;
	if (size <=  2 * 1024 * 1024) return 21;
	if (size <=  4 * 1024 * 1024) return 22;
	if (size <=  8 * 1024 * 1024) return 23;
	if (size <=  16 * 1024 * 1024) return 24;
	if (size <=  32 * 1024 * 1024) return 25;

	__CPROVER_assert(0, "Invalid kmalloc() size");

	return -1;
}

unsigned int kmalloc(unsigned int size, struct kmalloc_slab *slab)
{
	unsigned int nr = nondet_uint();

	slab->size = kmalloc_info[kmalloc_index(size)].size;
	slab->ptr = nr * slab->size;

	__CPROVER_assume(slab->ptr < PAGE_SIZE);
	__CPROVER_assume(slab->ptr % slab->size == 0);

	return slab->ptr;
}

/*
 * Implemented only for 32, 64 and 128 cache line sizes.
 */
int dma_kmalloc_needs_bounce(unsigned int size)
{
	unsigned int dma_align = cache_line_size();

	/*
	 * Less than half dma_align, there's definitely a smaller kmalloc()
	 * cache.
	 */
	if (size <= dma_align / 2)
		return 1;

	/*
	 * From this point, any kmalloc cache size is 32-byte aligned.
	 */
	if (dma_align == 32)
		return 0;

	/*
	 * dma_align == 64 => 96 needs bouncing.
	 * dma_align == 128 => 96 and 192 need bouncing.
	 */
	if (size > 64 && size <= 96)
		return 1;
	if (dma_align == 128 && size > 128 && size <= 192)
		return 1;

	return 0;
}

/*
 * Simulate DMA cache maintenance. The 'slab' object is only used for
 * verification.
 */
void dma_map_single(unsigned int ptr, unsigned int size,
		    struct kmalloc_slab *slab)
{
	unsigned int mask = cache_line_size() - 1;

	if (dma_kmalloc_needs_bounce(size)) {
		/* was the bounce really necessary? */
		__CPROVER_assert((ptr & mask) != 0 || (size & mask) != 0,
				 "Bouncing aligned DMA buffer");
		return;
	}

	/*
	 * Check for cache maintenance outside the kmalloc'ed object. We don't
	 * care about intra-object overlap, it's the caller's responsibility
	 * to ensure alignment.
	 */
	__CPROVER_assert((ptr & ~mask) >= slab->ptr, "DMA cache maintenance underflow");
	__CPROVER_assert(((ptr + size + mask) & ~mask) <= slab->ptr + slab->size,
			 "DMA cache maintenance overflow");
}

int main(void)
{
	struct kmalloc_slab slab;
	unsigned int size = nondet_uint();
	unsigned int offset = nondet_uint();
	unsigned int ptr;

	__CPROVER_assume(size <= KMALLOC_MAX_SIZE);
	__CPROVER_assume(offset < size);
	__CPROVER_assume(offset % ARCH_DMA_MINALIGN == 0);

	ptr = kmalloc(size, &slab);
	dma_map_single(ptr + offset, size - offset, &slab);

	return 0;
}

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Linus Torvalds]

On Sat, Oct 1, 2022 at 3:30 PM Catalin Marinas <catalin.marinas@arm.com> wrote:
>
> The "force bouncing" in my series currently only checks for small
> (potentially kmalloc'ed) sizes under the assumption that intra-object
> DMA buffers were properly aligned to 128. So for something like below:

Ahh, so your forced bouncing isn't actually safe.

I would have hoped (but obviously never checked) that the force
bouncing be made really safe and look at the actual alignment of the
DMA (comparing it to the hardware coherency requirements), so that
alignment at allocation time simply wouldn't matter.

At that point, places like the ones you found would still work, they'd
just cause bouncing.

At which point you'd then have a choice of

 (a) just let it bounce

 (b) marking the allocations that led to them

and (a) might actually be perfectly fine in a lot of situations.
That's particularly true for the "random drivers" situation that may
not be all that relevant in real life, which is a *big* deal. Not
because of any performance issues, but simply because of kernel
developers not having to worry their pretty little heads about stuff
that doesn't really matter.

In fact, (a) might be perfectly ok even for drivers that *do* matter,
if they just aren't all that performance-critical and the situation
doesn't come up a lot (maybe it's a special management ioctl or
similar that just causes the possibility to come up, and it's
important that it *works*, but having a few bounces occasionally
doesn't actually matter, and all the regular IO goes the normal path).

And (b) would be triggered by actual data. Which could be fairly easy
to gather with a statistical model. For example, just making
dma_map_xyz() have a debug mode where it prints out the stack trace of
these bounces once every minute or so - statistically the call trace
will be one of the hot ones. Or, better yet, just use tracing to do
it.

That would allow us to say "DMA is immaterial for _correct_ alignment,
because we always fix it up if required", but then also find
situations where we might want to give it a gentle helper nudge.

But hey, if you're comfortable with your approach, that's fine too.
Anything that gets rid of the absolutely insane "you can't do small
allocations" is an improvement.

                   Linus

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Linus Torvalds]

On Sat, Oct 1, 2022 at 3:30 PM Catalin Marinas <catalin.marinas@arm.com> wrote:
>
> The "force bouncing" in my series currently only checks for small
> (potentially kmalloc'ed) sizes under the assumption that intra-object
> DMA buffers were properly aligned to 128. So for something like below:

Ahh, so your forced bouncing isn't actually safe.

I would have hoped (but obviously never checked) that the force
bouncing be made really safe and look at the actual alignment of the
DMA (comparing it to the hardware coherency requirements), so that
alignment at allocation time simply wouldn't matter.

At that point, places like the ones you found would still work, they'd
just cause bouncing.

At which point you'd then have a choice of

 (a) just let it bounce

 (b) marking the allocations that led to them

and (a) might actually be perfectly fine in a lot of situations.
That's particularly true for the "random drivers" situation that may
not be all that relevant in real life, which is a *big* deal. Not
because of any performance issues, but simply because of kernel
developers not having to worry their pretty little heads about stuff
that doesn't really matter.

In fact, (a) might be perfectly ok even for drivers that *do* matter,
if they just aren't all that performance-critical and the situation
doesn't come up a lot (maybe it's a special management ioctl or
similar that just causes the possibility to come up, and it's
important that it *works*, but having a few bounces occasionally
doesn't actually matter, and all the regular IO goes the normal path).

And (b) would be triggered by actual data. Which could be fairly easy
to gather with a statistical model. For example, just making
dma_map_xyz() have a debug mode where it prints out the stack trace of
these bounces once every minute or so - statistically the call trace
will be one of the hot ones. Or, better yet, just use tracing to do
it.

That would allow us to say "DMA is immaterial for _correct_ alignment,
because we always fix it up if required", but then also find
situations where we might want to give it a gentle helper nudge.

But hey, if you're comfortable with your approach, that's fine too.
Anything that gets rid of the absolutely insane "you can't do small
allocations" is an improvement.

                   Linus

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Ard Biesheuvel]

On Sun, 2 Oct 2022 at 19:00, Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> On Sat, Oct 1, 2022 at 3:30 PM Catalin Marinas <catalin.marinas@arm.com> wrote:
> >
> > The "force bouncing" in my series currently only checks for small
> > (potentially kmalloc'ed) sizes under the assumption that intra-object
> > DMA buffers were properly aligned to 128. So for something like below:
>
> Ahh, so your forced bouncing isn't actually safe.
>
> I would have hoped (but obviously never checked) that the force
> bouncing be made really safe and look at the actual alignment of the
> DMA (comparing it to the hardware coherency requirements), so that
> alignment at allocation time simply wouldn't matter.
>
> At that point, places like the ones you found would still work, they'd
> just cause bouncing.
>
> At which point you'd then have a choice of
>
>  (a) just let it bounce
>
>  (b) marking the allocations that led to them
>
> and (a) might actually be perfectly fine in a lot of situations.

Non-coherent DMA for networking is going to be fun, though.
Fortunately, we'll only need to bounce for inbound DMA (where the
cache invalidation might otherwise corrupt adjacent unrelated data),
and for inbound networking, the driver typically owns and maps the
buffers, and so in most cases, it can hopefully be easily modified to
round up the allocations and the length values passed to
dma_map_xxx(). And for outbound, we can just clean the SKB from the
caches without the risk of corruption.

It does mean that there is likely going to be a long tail of network
drivers that will non-negligibly regress in performance and in memory
footprint (as the entire RX ring will be double buffered all the time)
until they get updated.

-- 
Ard.

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Ard Biesheuvel]

On Sun, 2 Oct 2022 at 19:00, Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> On Sat, Oct 1, 2022 at 3:30 PM Catalin Marinas <catalin.marinas@arm.com> wrote:
> >
> > The "force bouncing" in my series currently only checks for small
> > (potentially kmalloc'ed) sizes under the assumption that intra-object
> > DMA buffers were properly aligned to 128. So for something like below:
>
> Ahh, so your forced bouncing isn't actually safe.
>
> I would have hoped (but obviously never checked) that the force
> bouncing be made really safe and look at the actual alignment of the
> DMA (comparing it to the hardware coherency requirements), so that
> alignment at allocation time simply wouldn't matter.
>
> At that point, places like the ones you found would still work, they'd
> just cause bouncing.
>
> At which point you'd then have a choice of
>
>  (a) just let it bounce
>
>  (b) marking the allocations that led to them
>
> and (a) might actually be perfectly fine in a lot of situations.

Non-coherent DMA for networking is going to be fun, though.
Fortunately, we'll only need to bounce for inbound DMA (where the
cache invalidation might otherwise corrupt adjacent unrelated data),
and for inbound networking, the driver typically owns and maps the
buffers, and so in most cases, it can hopefully be easily modified to
round up the allocations and the length values passed to
dma_map_xxx(). And for outbound, we can just clean the SKB from the
caches without the risk of corruption.

It does mean that there is likely going to be a long tail of network
drivers that will non-negligibly regress in performance and in memory
footprint (as the entire RX ring will be double buffered all the time)
until they get updated.

-- 
Ard.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Linus Torvalds]

On Sun, Oct 2, 2022 at 3:09 PM Ard Biesheuvel <ardb@kernel.org> wrote:
>
> Non-coherent DMA for networking is going to be fun, though.

I agree that networking is likely the main performance issue, but I
suspect 99% of the cases would come from __alloc_skb().

You might want to have help from the network drivers for the "allocate
for RX vs TX", since it ends up having very different DMA coherence
issues, as you point out.

The code actually already has a SKB_ALLOC_RX flag, but despite the
name it doesn't really mean what you'd think it means.

Similarly, that code already has magic stuff to try to be
cacheline-aligned for accesses, but it's not really for DMA coherency
reasons, just purely for performance reasons (trying to make sure that
the header accesses stay in one cacheline etc).

And to be honest, it's been years and years since I did any networking, so...

                  Linus

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Linus Torvalds]

On Sun, Oct 2, 2022 at 3:09 PM Ard Biesheuvel <ardb@kernel.org> wrote:
>
> Non-coherent DMA for networking is going to be fun, though.

I agree that networking is likely the main performance issue, but I
suspect 99% of the cases would come from __alloc_skb().

You might want to have help from the network drivers for the "allocate
for RX vs TX", since it ends up having very different DMA coherence
issues, as you point out.

The code actually already has a SKB_ALLOC_RX flag, but despite the
name it doesn't really mean what you'd think it means.

Similarly, that code already has magic stuff to try to be
cacheline-aligned for accesses, but it's not really for DMA coherency
reasons, just purely for performance reasons (trying to make sure that
the header accesses stay in one cacheline etc).

And to be honest, it's been years and years since I did any networking, so...

                  Linus

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Sun, Oct 02, 2022 at 03:24:57PM -0700, Linus Torvalds wrote:
> On Sun, Oct 2, 2022 at 3:09 PM Ard Biesheuvel <ardb@kernel.org> wrote:
> > Non-coherent DMA for networking is going to be fun, though.
> 
> I agree that networking is likely the main performance issue, but I
> suspect 99% of the cases would come from __alloc_skb().

The problem is not the allocation but rather having a generic enough
dma_needs_bounce() check. It won't be able to tell whether some 1500
byte range is for network or for crypto code that uses a small
ARCH_KMALLOC_MINALIGN. Getting the actual object size (e.g. with
ksize()) doesn't tell the full story on how safe the DMA is.

> Similarly, that code already has magic stuff to try to be
> cacheline-aligned for accesses, but it's not really for DMA coherency
> reasons, just purely for performance reasons (trying to make sure that
> the header accesses stay in one cacheline etc).

Yeah, __skb_alloc() ends up using SMP_CACHE_BYTES for data alignment
(via SKB_DATA_ALIGN). I have a suspicion this may break on SoCs with a
128-byte cache line but I haven't seen any report yet (there aren't many
such systems).

-- 
Catalin

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Sun, Oct 02, 2022 at 03:24:57PM -0700, Linus Torvalds wrote:
> On Sun, Oct 2, 2022 at 3:09 PM Ard Biesheuvel <ardb@kernel.org> wrote:
> > Non-coherent DMA for networking is going to be fun, though.
> 
> I agree that networking is likely the main performance issue, but I
> suspect 99% of the cases would come from __alloc_skb().

The problem is not the allocation but rather having a generic enough
dma_needs_bounce() check. It won't be able to tell whether some 1500
byte range is for network or for crypto code that uses a small
ARCH_KMALLOC_MINALIGN. Getting the actual object size (e.g. with
ksize()) doesn't tell the full story on how safe the DMA is.

> Similarly, that code already has magic stuff to try to be
> cacheline-aligned for accesses, but it's not really for DMA coherency
> reasons, just purely for performance reasons (trying to make sure that
> the header accesses stay in one cacheline etc).

Yeah, __skb_alloc() ends up using SMP_CACHE_BYTES for data alignment
(via SKB_DATA_ALIGN). I have a suspicion this may break on SoCs with a
128-byte cache line but I haven't seen any report yet (there aren't many
such systems).

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Isaac Manjarres]

On Fri, Sep 30, 2022 at 07:32:50PM +0100, Catalin Marinas wrote:
> I started refreshing the series but I got stuck on having to do bouncing
> for small buffers even if when they go through the iommu (and I don't
> have the set up to test it yet).

For devices that go through the IOMMU, are you planning on adding
similar logic as you did in the direct-DMA path to bounce the buffer
prior to calling into whatever DMA ops are registered for the device?

Also, there are devices with ARM64 CPUs that disable SWIOTLB usage because
none of the peripherals that they engage in DMA with need bounce buffering,
and also to reclaim the default 64 MB of memory that SWIOTLB uses. With
this approach, SWIOTLB usage will become mandatory if those devices need
to perform non-coherent DMA transactions that may not necessarily be DMA
aligned (e.g. small buffers), correct?

If so, would there be concerns that the memory savings we get back from
reducing the memory footprint of kmalloc might be defeated by how much
memory is needed for bounce buffering? I understand that we can use the
"swiotlb=num_slabs" command line parameter to minimize the amount of
memory allocated for bounce buffering. If this is the only way to
minimize this impact, how much memory would you recommend to allocate
for bounce buffering on a system that will only use bounce buffers for
non-DMA-aligned buffers?

Thanks,
Isaac

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Isaac Manjarres]

On Fri, Sep 30, 2022 at 07:32:50PM +0100, Catalin Marinas wrote:
> I started refreshing the series but I got stuck on having to do bouncing
> for small buffers even if when they go through the iommu (and I don't
> have the set up to test it yet).

For devices that go through the IOMMU, are you planning on adding
similar logic as you did in the direct-DMA path to bounce the buffer
prior to calling into whatever DMA ops are registered for the device?

Also, there are devices with ARM64 CPUs that disable SWIOTLB usage because
none of the peripherals that they engage in DMA with need bounce buffering,
and also to reclaim the default 64 MB of memory that SWIOTLB uses. With
this approach, SWIOTLB usage will become mandatory if those devices need
to perform non-coherent DMA transactions that may not necessarily be DMA
aligned (e.g. small buffers), correct?

If so, would there be concerns that the memory savings we get back from
reducing the memory footprint of kmalloc might be defeated by how much
memory is needed for bounce buffering? I understand that we can use the
"swiotlb=num_slabs" command line parameter to minimize the amount of
memory allocated for bounce buffering. If this is the only way to
minimize this impact, how much memory would you recommend to allocate
for bounce buffering on a system that will only use bounce buffers for
non-DMA-aligned buffers?

Thanks,
Isaac

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Wed, Oct 12, 2022 at 10:45:45AM -0700, Isaac Manjarres wrote:
> On Fri, Sep 30, 2022 at 07:32:50PM +0100, Catalin Marinas wrote:
> > I started refreshing the series but I got stuck on having to do bouncing
> > for small buffers even if when they go through the iommu (and I don't
> > have the set up to test it yet).
> 
> For devices that go through the IOMMU, are you planning on adding
> similar logic as you did in the direct-DMA path to bounce the buffer
> prior to calling into whatever DMA ops are registered for the device?

Yes.

> Also, there are devices with ARM64 CPUs that disable SWIOTLB usage because
> none of the peripherals that they engage in DMA with need bounce buffering,
> and also to reclaim the default 64 MB of memory that SWIOTLB uses. With
> this approach, SWIOTLB usage will become mandatory if those devices need
> to perform non-coherent DMA transactions that may not necessarily be DMA
> aligned (e.g. small buffers), correct?

Correct. I've been thinking about this and a way around is to combine
the original series (dynamic kmalloc_minalign) with the new one so that
the arch code can lower the minimum alignment either to 8 if swiotlb is
available (usually in server space with more RAM) or the cache line size
if there is no bounce buffer.

> If so, would there be concerns that the memory savings we get back from
> reducing the memory footprint of kmalloc might be defeated by how much
> memory is needed for bounce buffering?

It's not necessarily about the saved memory but also locality of the
small buffer allocations, less cache and TLB pressure.

> I understand that we can use the
> "swiotlb=num_slabs" command line parameter to minimize the amount of
> memory allocated for bounce buffering. If this is the only way to
> minimize this impact, how much memory would you recommend to allocate
> for bounce buffering on a system that will only use bounce buffers for
> non-DMA-aligned buffers?

It's hard to tell, it would need to be guessed by trial and error on
specific hardware if you want to lower it. Another issue is that IIRC
the swiotlb is allocated in 2K slots, so you may need a lot more bounce
buffers than the actual memory allocated.

I wonder whether swiotlb is actually the best option for bouncing
unaligned buffers. We could use something like mempool_alloc() instead
if we stick to small buffers rather than any (even large) buffer that's
not aligned to a cache line. Or just go for kmem_cache_alloc() directly.
A downside is that we may need GFP_ATOMIC for such allocations, so
higher risk of failure.

-- 
Catalin

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Wed, Oct 12, 2022 at 10:45:45AM -0700, Isaac Manjarres wrote:
> On Fri, Sep 30, 2022 at 07:32:50PM +0100, Catalin Marinas wrote:
> > I started refreshing the series but I got stuck on having to do bouncing
> > for small buffers even if when they go through the iommu (and I don't
> > have the set up to test it yet).
> 
> For devices that go through the IOMMU, are you planning on adding
> similar logic as you did in the direct-DMA path to bounce the buffer
> prior to calling into whatever DMA ops are registered for the device?

Yes.

> Also, there are devices with ARM64 CPUs that disable SWIOTLB usage because
> none of the peripherals that they engage in DMA with need bounce buffering,
> and also to reclaim the default 64 MB of memory that SWIOTLB uses. With
> this approach, SWIOTLB usage will become mandatory if those devices need
> to perform non-coherent DMA transactions that may not necessarily be DMA
> aligned (e.g. small buffers), correct?

Correct. I've been thinking about this and a way around is to combine
the original series (dynamic kmalloc_minalign) with the new one so that
the arch code can lower the minimum alignment either to 8 if swiotlb is
available (usually in server space with more RAM) or the cache line size
if there is no bounce buffer.

> If so, would there be concerns that the memory savings we get back from
> reducing the memory footprint of kmalloc might be defeated by how much
> memory is needed for bounce buffering?

It's not necessarily about the saved memory but also locality of the
small buffer allocations, less cache and TLB pressure.

> I understand that we can use the
> "swiotlb=num_slabs" command line parameter to minimize the amount of
> memory allocated for bounce buffering. If this is the only way to
> minimize this impact, how much memory would you recommend to allocate
> for bounce buffering on a system that will only use bounce buffers for
> non-DMA-aligned buffers?

It's hard to tell, it would need to be guessed by trial and error on
specific hardware if you want to lower it. Another issue is that IIRC
the swiotlb is allocated in 2K slots, so you may need a lot more bounce
buffers than the actual memory allocated.

I wonder whether swiotlb is actually the best option for bouncing
unaligned buffers. We could use something like mempool_alloc() instead
if we stick to small buffers rather than any (even large) buffer that's
not aligned to a cache line. Or just go for kmem_cache_alloc() directly.
A downside is that we may need GFP_ATOMIC for such allocations, so
higher risk of failure.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Saravana Kannan]

On Thu, Oct 13, 2022 at 9:57 AM Catalin Marinas <catalin.marinas@arm.com> wrote:
>
> On Wed, Oct 12, 2022 at 10:45:45AM -0700, Isaac Manjarres wrote:
> > On Fri, Sep 30, 2022 at 07:32:50PM +0100, Catalin Marinas wrote:
> > > I started refreshing the series but I got stuck on having to do bouncing
> > > for small buffers even if when they go through the iommu (and I don't
> > > have the set up to test it yet).
> >
> > For devices that go through the IOMMU, are you planning on adding
> > similar logic as you did in the direct-DMA path to bounce the buffer
> > prior to calling into whatever DMA ops are registered for the device?
>
> Yes.
>
> > Also, there are devices with ARM64 CPUs that disable SWIOTLB usage because
> > none of the peripherals that they engage in DMA with need bounce buffering,
> > and also to reclaim the default 64 MB of memory that SWIOTLB uses. With
> > this approach, SWIOTLB usage will become mandatory if those devices need
> > to perform non-coherent DMA transactions that may not necessarily be DMA
> > aligned (e.g. small buffers), correct?
>
> Correct. I've been thinking about this and a way around is to combine
> the original series (dynamic kmalloc_minalign) with the new one so that
> the arch code can lower the minimum alignment either to 8 if swiotlb is
> available (usually in server space with more RAM) or the cache line size
> if there is no bounce buffer.
>
> > If so, would there be concerns that the memory savings we get back from
> > reducing the memory footprint of kmalloc might be defeated by how much
> > memory is needed for bounce buffering?
>
> It's not necessarily about the saved memory but also locality of the
> small buffer allocations, less cache and TLB pressure.

Part of the pushback we get when we try to move some of the Android
ecosystem from 32-bit to 64-bit is the memory usage increase. So,
while the main goal might not be memory savings, it'll be good to keep
that in mind too. I'd definitely not want this patch series to make
things worse. Ideally, it'd make things better. 10MB is considered a
lot on some of the super low speced devices.

> > I understand that we can use the
> > "swiotlb=num_slabs" command line parameter to minimize the amount of
> > memory allocated for bounce buffering. If this is the only way to
> > minimize this impact, how much memory would you recommend to allocate
> > for bounce buffering on a system that will only use bounce buffers for
> > non-DMA-aligned buffers?
>
> It's hard to tell, it would need to be guessed by trial and error on
> specific hardware if you want to lower it. Another issue is that IIRC
> the swiotlb is allocated in 2K slots, so you may need a lot more bounce
> buffers than the actual memory allocated.
>
> I wonder whether swiotlb is actually the best option for bouncing
> unaligned buffers. We could use something like mempool_alloc() instead
> if we stick to small buffers rather than any (even large) buffer that's
> not aligned to a cache line. Or just go for kmem_cache_alloc() directly.
> A downside is that we may need GFP_ATOMIC for such allocations, so
> higher risk of failure.

Yeah, a temporary kmem_cache_alloc() to bounce buffers off of feels
like a better idea than swiotlb. Especially for small allocations (say
8 byte allocations) that might have gone into the kmem-cache-64 if we
hadn't dropped KMALLOC_MIN_ALIGN to 8.

-Saravana

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Saravana Kannan]

On Thu, Oct 13, 2022 at 9:57 AM Catalin Marinas <catalin.marinas@arm.com> wrote:
>
> On Wed, Oct 12, 2022 at 10:45:45AM -0700, Isaac Manjarres wrote:
> > On Fri, Sep 30, 2022 at 07:32:50PM +0100, Catalin Marinas wrote:
> > > I started refreshing the series but I got stuck on having to do bouncing
> > > for small buffers even if when they go through the iommu (and I don't
> > > have the set up to test it yet).
> >
> > For devices that go through the IOMMU, are you planning on adding
> > similar logic as you did in the direct-DMA path to bounce the buffer
> > prior to calling into whatever DMA ops are registered for the device?
>
> Yes.
>
> > Also, there are devices with ARM64 CPUs that disable SWIOTLB usage because
> > none of the peripherals that they engage in DMA with need bounce buffering,
> > and also to reclaim the default 64 MB of memory that SWIOTLB uses. With
> > this approach, SWIOTLB usage will become mandatory if those devices need
> > to perform non-coherent DMA transactions that may not necessarily be DMA
> > aligned (e.g. small buffers), correct?
>
> Correct. I've been thinking about this and a way around is to combine
> the original series (dynamic kmalloc_minalign) with the new one so that
> the arch code can lower the minimum alignment either to 8 if swiotlb is
> available (usually in server space with more RAM) or the cache line size
> if there is no bounce buffer.
>
> > If so, would there be concerns that the memory savings we get back from
> > reducing the memory footprint of kmalloc might be defeated by how much
> > memory is needed for bounce buffering?
>
> It's not necessarily about the saved memory but also locality of the
> small buffer allocations, less cache and TLB pressure.

Part of the pushback we get when we try to move some of the Android
ecosystem from 32-bit to 64-bit is the memory usage increase. So,
while the main goal might not be memory savings, it'll be good to keep
that in mind too. I'd definitely not want this patch series to make
things worse. Ideally, it'd make things better. 10MB is considered a
lot on some of the super low speced devices.

> > I understand that we can use the
> > "swiotlb=num_slabs" command line parameter to minimize the amount of
> > memory allocated for bounce buffering. If this is the only way to
> > minimize this impact, how much memory would you recommend to allocate
> > for bounce buffering on a system that will only use bounce buffers for
> > non-DMA-aligned buffers?
>
> It's hard to tell, it would need to be guessed by trial and error on
> specific hardware if you want to lower it. Another issue is that IIRC
> the swiotlb is allocated in 2K slots, so you may need a lot more bounce
> buffers than the actual memory allocated.
>
> I wonder whether swiotlb is actually the best option for bouncing
> unaligned buffers. We could use something like mempool_alloc() instead
> if we stick to small buffers rather than any (even large) buffer that's
> not aligned to a cache line. Or just go for kmem_cache_alloc() directly.
> A downside is that we may need GFP_ATOMIC for such allocations, so
> higher risk of failure.

Yeah, a temporary kmem_cache_alloc() to bounce buffers off of feels
like a better idea than swiotlb. Especially for small allocations (say
8 byte allocations) that might have gone into the kmem-cache-64 if we
hadn't dropped KMALLOC_MIN_ALIGN to 8.

-Saravana

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Thu, Oct 13, 2022 at 11:58:22AM -0700, Saravana Kannan wrote:
> On Thu, Oct 13, 2022 at 9:57 AM Catalin Marinas <catalin.marinas@arm.com> wrote:
> > > If so, would there be concerns that the memory savings we get back from
> > > reducing the memory footprint of kmalloc might be defeated by how much
> > > memory is needed for bounce buffering?
> >
> > It's not necessarily about the saved memory but also locality of the
> > small buffer allocations, less cache and TLB pressure.
> 
> Part of the pushback we get when we try to move some of the Android
> ecosystem from 32-bit to 64-bit is the memory usage increase. So,
> while the main goal might not be memory savings, it'll be good to keep
> that in mind too. I'd definitely not want this patch series to make
> things worse. Ideally, it'd make things better. 10MB is considered a
> lot on some of the super low speced devices.

Well, we can still add the option to skip allocating from the small
kmalloc caches if there's no swiotlb available.

> > I wonder whether swiotlb is actually the best option for bouncing
> > unaligned buffers. We could use something like mempool_alloc() instead
> > if we stick to small buffers rather than any (even large) buffer that's
> > not aligned to a cache line. Or just go for kmem_cache_alloc() directly.
> > A downside is that we may need GFP_ATOMIC for such allocations, so
> > higher risk of failure.
> 
> Yeah, a temporary kmem_cache_alloc() to bounce buffers off of feels
> like a better idea than swiotlb. Especially for small allocations (say
> 8 byte allocations) that might have gone into the kmem-cache-64 if we
> hadn't dropped KMALLOC_MIN_ALIGN to 8.

I now remembered why this isn't trivial. On the dma_unmap_*() side, we
can't easily tell whether the buffer was bounced and it needs freeing.
The swiotlb solves this by checking whether the address is within the
pre-allocated swiotlb range. With kmem_caches, we could dig into the
slab internals and check whether the pointer is part of a slab page,
then check with kmem_cache that was. It looks too complicated and I'm
rather tempted to just go for swiotlb if available or prevent the
creation of small kmalloc caches if no swiotlb (TBH, even the initial
series was an improvement dropping KMALLOC_MIN_ALIGN from 128 to 64).

-- 
Catalin

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Thu, Oct 13, 2022 at 11:58:22AM -0700, Saravana Kannan wrote:
> On Thu, Oct 13, 2022 at 9:57 AM Catalin Marinas <catalin.marinas@arm.com> wrote:
> > > If so, would there be concerns that the memory savings we get back from
> > > reducing the memory footprint of kmalloc might be defeated by how much
> > > memory is needed for bounce buffering?
> >
> > It's not necessarily about the saved memory but also locality of the
> > small buffer allocations, less cache and TLB pressure.
> 
> Part of the pushback we get when we try to move some of the Android
> ecosystem from 32-bit to 64-bit is the memory usage increase. So,
> while the main goal might not be memory savings, it'll be good to keep
> that in mind too. I'd definitely not want this patch series to make
> things worse. Ideally, it'd make things better. 10MB is considered a
> lot on some of the super low speced devices.

Well, we can still add the option to skip allocating from the small
kmalloc caches if there's no swiotlb available.

> > I wonder whether swiotlb is actually the best option for bouncing
> > unaligned buffers. We could use something like mempool_alloc() instead
> > if we stick to small buffers rather than any (even large) buffer that's
> > not aligned to a cache line. Or just go for kmem_cache_alloc() directly.
> > A downside is that we may need GFP_ATOMIC for such allocations, so
> > higher risk of failure.
> 
> Yeah, a temporary kmem_cache_alloc() to bounce buffers off of feels
> like a better idea than swiotlb. Especially for small allocations (say
> 8 byte allocations) that might have gone into the kmem-cache-64 if we
> hadn't dropped KMALLOC_MIN_ALIGN to 8.

I now remembered why this isn't trivial. On the dma_unmap_*() side, we
can't easily tell whether the buffer was bounced and it needs freeing.
The swiotlb solves this by checking whether the address is within the
pre-allocated swiotlb range. With kmem_caches, we could dig into the
slab internals and check whether the pointer is part of a slab page,
then check with kmem_cache that was. It looks too complicated and I'm
rather tempted to just go for swiotlb if available or prevent the
creation of small kmalloc caches if no swiotlb (TBH, even the initial
series was an improvement dropping KMALLOC_MIN_ALIGN from 128 to 64).

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Saravana Kannan]

On Fri, Oct 14, 2022 at 9:25 AM Catalin Marinas <catalin.marinas@arm.com> wrote:
>
> On Thu, Oct 13, 2022 at 11:58:22AM -0700, Saravana Kannan wrote:
> > On Thu, Oct 13, 2022 at 9:57 AM Catalin Marinas <catalin.marinas@arm.com> wrote:
> > > > If so, would there be concerns that the memory savings we get back from
> > > > reducing the memory footprint of kmalloc might be defeated by how much
> > > > memory is needed for bounce buffering?
> > >
> > > It's not necessarily about the saved memory but also locality of the
> > > small buffer allocations, less cache and TLB pressure.
> >
> > Part of the pushback we get when we try to move some of the Android
> > ecosystem from 32-bit to 64-bit is the memory usage increase. So,
> > while the main goal might not be memory savings, it'll be good to keep
> > that in mind too. I'd definitely not want this patch series to make
> > things worse. Ideally, it'd make things better. 10MB is considered a
> > lot on some of the super low speced devices.
>
> Well, we can still add the option to skip allocating from the small
> kmalloc caches if there's no swiotlb available.

This seems like a good compromise.

> > > I wonder whether swiotlb is actually the best option for bouncing
> > > unaligned buffers. We could use something like mempool_alloc() instead
> > > if we stick to small buffers rather than any (even large) buffer that's
> > > not aligned to a cache line. Or just go for kmem_cache_alloc() directly.
> > > A downside is that we may need GFP_ATOMIC for such allocations, so
> > > higher risk of failure.
> >
> > Yeah, a temporary kmem_cache_alloc() to bounce buffers off of feels
> > like a better idea than swiotlb. Especially for small allocations (say
> > 8 byte allocations) that might have gone into the kmem-cache-64 if we
> > hadn't dropped KMALLOC_MIN_ALIGN to 8.
>
> I now remembered why this isn't trivial. On the dma_unmap_*() side, we
> can't easily tell whether the buffer was bounced and it needs freeing.
> The swiotlb solves this by checking whether the address is within the
> pre-allocated swiotlb range. With kmem_caches, we could dig into the
> slab internals and check whether the pointer is part of a slab page,
> then check with kmem_cache that was. It looks too complicated and I'm
> rather tempted to just go for swiotlb if available or prevent the
> creation of small kmalloc caches if no swiotlb (TBH, even the initial
> series was an improvement dropping KMALLOC_MIN_ALIGN from 128 to 64).

Agreed. Even allowing a 64-byte kmalloc cache on a system with a
64-byte cacheline size saves quite a bit of memory.

-Saravana

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Saravana Kannan]

On Fri, Oct 14, 2022 at 9:25 AM Catalin Marinas <catalin.marinas@arm.com> wrote:
>
> On Thu, Oct 13, 2022 at 11:58:22AM -0700, Saravana Kannan wrote:
> > On Thu, Oct 13, 2022 at 9:57 AM Catalin Marinas <catalin.marinas@arm.com> wrote:
> > > > If so, would there be concerns that the memory savings we get back from
> > > > reducing the memory footprint of kmalloc might be defeated by how much
> > > > memory is needed for bounce buffering?
> > >
> > > It's not necessarily about the saved memory but also locality of the
> > > small buffer allocations, less cache and TLB pressure.
> >
> > Part of the pushback we get when we try to move some of the Android
> > ecosystem from 32-bit to 64-bit is the memory usage increase. So,
> > while the main goal might not be memory savings, it'll be good to keep
> > that in mind too. I'd definitely not want this patch series to make
> > things worse. Ideally, it'd make things better. 10MB is considered a
> > lot on some of the super low speced devices.
>
> Well, we can still add the option to skip allocating from the small
> kmalloc caches if there's no swiotlb available.

This seems like a good compromise.

> > > I wonder whether swiotlb is actually the best option for bouncing
> > > unaligned buffers. We could use something like mempool_alloc() instead
> > > if we stick to small buffers rather than any (even large) buffer that's
> > > not aligned to a cache line. Or just go for kmem_cache_alloc() directly.
> > > A downside is that we may need GFP_ATOMIC for such allocations, so
> > > higher risk of failure.
> >
> > Yeah, a temporary kmem_cache_alloc() to bounce buffers off of feels
> > like a better idea than swiotlb. Especially for small allocations (say
> > 8 byte allocations) that might have gone into the kmem-cache-64 if we
> > hadn't dropped KMALLOC_MIN_ALIGN to 8.
>
> I now remembered why this isn't trivial. On the dma_unmap_*() side, we
> can't easily tell whether the buffer was bounced and it needs freeing.
> The swiotlb solves this by checking whether the address is within the
> pre-allocated swiotlb range. With kmem_caches, we could dig into the
> slab internals and check whether the pointer is part of a slab page,
> then check with kmem_cache that was. It looks too complicated and I'm
> rather tempted to just go for swiotlb if available or prevent the
> creation of small kmalloc caches if no swiotlb (TBH, even the initial
> series was an improvement dropping KMALLOC_MIN_ALIGN from 128 to 64).

Agreed. Even allowing a 64-byte kmalloc cache on a system with a
64-byte cacheline size saves quite a bit of memory.

-Saravana

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Linus Torvalds]

On Fri, Oct 14, 2022 at 1:24 PM Saravana Kannan <saravanak@google.com> wrote:
>
> Agreed. Even allowing a 64-byte kmalloc cache on a system with a
> 64-byte cacheline size saves quite a bit of memory.

Well, the *really* trivial thing to do is to just say "if the platform
is DMA coherent, just allow any size kmalloc cache". And just
consciously leave the broken garbage behind.

Because it's not just 64-byte kmalloc. It's things like 'avtab_node'
that is 24 bytes, and that on my system currently uses about 3MB of
memory simply because there's a _lot_ of them.

I've got 1.8MB in "kmalloc-32" too, and about 1MB in "kamlloc-16", fwiw. That's

Yeah, yeah, this is on a 64GB machine and so none of that matters (and
some of these things are very much "scales with memory", but these
small allocations aren't actually all that unusual.

And yes, the above is obviously on my x86-64 machine.

My arm64 laptop doesn't have the small kmallocs, and as a result the
"kmalloc-128" has 633 _thousand_ entries, and takes up 77MB of RAM on
my 16GB laptop. I'm assuming (but have no proof) more than 50% of that
is just wasted memory.

I suspect that DMA is cache coherent on this thing, and that wasted
space is just *stupid* wasted space, but hey, I don't actually know. I
just use the Asahi people's patches.

               Linus

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Linus Torvalds]

On Fri, Oct 14, 2022 at 1:24 PM Saravana Kannan <saravanak@google.com> wrote:
>
> Agreed. Even allowing a 64-byte kmalloc cache on a system with a
> 64-byte cacheline size saves quite a bit of memory.

Well, the *really* trivial thing to do is to just say "if the platform
is DMA coherent, just allow any size kmalloc cache". And just
consciously leave the broken garbage behind.

Because it's not just 64-byte kmalloc. It's things like 'avtab_node'
that is 24 bytes, and that on my system currently uses about 3MB of
memory simply because there's a _lot_ of them.

I've got 1.8MB in "kmalloc-32" too, and about 1MB in "kamlloc-16", fwiw. That's

Yeah, yeah, this is on a 64GB machine and so none of that matters (and
some of these things are very much "scales with memory", but these
small allocations aren't actually all that unusual.

And yes, the above is obviously on my x86-64 machine.

My arm64 laptop doesn't have the small kmallocs, and as a result the
"kmalloc-128" has 633 _thousand_ entries, and takes up 77MB of RAM on
my 16GB laptop. I'm assuming (but have no proof) more than 50% of that
is just wasted memory.

I suspect that DMA is cache coherent on this thing, and that wasted
space is just *stupid* wasted space, but hey, I don't actually know. I
just use the Asahi people's patches.

               Linus

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Fri, Oct 14, 2022 at 01:44:25PM -0700, Linus Torvalds wrote:
> On Fri, Oct 14, 2022 at 1:24 PM Saravana Kannan <saravanak@google.com> wrote:
> > Agreed. Even allowing a 64-byte kmalloc cache on a system with a
> > 64-byte cacheline size saves quite a bit of memory.
> 
> Well, the *really* trivial thing to do is to just say "if the platform
> is DMA coherent, just allow any size kmalloc cache". And just
> consciously leave the broken garbage behind.

The problem is we don't have a reliable way to tell whether the platform
is DMA-coherent. The CPU IDs don't really say much and in most cases
it's a property of the interconnect/bus and device. We describe the DMA
coherency in DT or ACPI and the latter is somewhat better as it assumes
coherent by default. But for DT, not having a 'dma-coherent' property
means non-coherent DMA (or no DMA at all). We can't even tell whether
the device is going to do any DMA, arch_setup_dma_ops() is called even
for devices like the PMU. We could look into defining new properties
(e.g. "no-dma") and adjust the DTs but we may also have late probed
devices, long after the slab allocator was initialised. A big
'dma-coherent' property on the top node may work but most Android
systems won't benefit from this (your laptop may, I haven't checked).

I think the best bet is still either (a) bounce for small sizes or (b) a
new GFP_NODMA/PACKED/etc. flag for the hot small allocations. (a) is
somehow more universal but lots (most) Android devices are deployed with
no swiotlb buffer as the vendor knows the device needs and don't need
extra buffer waste. Not sure how reliable it would be to trigger another
slab allocation on the dma_map_*() calls for the bounce (it may need to
be GFP_ATOMIC). Option (b) looks more appealing on such systems, though
a lot more churn throughout the kernel.

-- 
Catalin

Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

[Catalin Marinas]

On Fri, Oct 14, 2022 at 01:44:25PM -0700, Linus Torvalds wrote:
> On Fri, Oct 14, 2022 at 1:24 PM Saravana Kannan <saravanak@google.com> wrote:
> > Agreed. Even allowing a 64-byte kmalloc cache on a system with a
> > 64-byte cacheline size saves quite a bit of memory.
> 
> Well, the *really* trivial thing to do is to just say "if the platform
> is DMA coherent, just allow any size kmalloc cache". And just
> consciously leave the broken garbage behind.

The problem is we don't have a reliable way to tell whether the platform
is DMA-coherent. The CPU IDs don't really say much and in most cases
it's a property of the interconnect/bus and device. We describe the DMA
coherency in DT or ACPI and the latter is somewhat better as it assumes
coherent by default. But for DT, not having a 'dma-coherent' property
means non-coherent DMA (or no DMA at all). We can't even tell whether
the device is going to do any DMA, arch_setup_dma_ops() is called even
for devices like the PMU. We could look into defining new properties
(e.g. "no-dma") and adjust the DTs but we may also have late probed
devices, long after the slab allocator was initialised. A big
'dma-coherent' property on the top node may work but most Android
systems won't benefit from this (your laptop may, I haven't checked).

I think the best bet is still either (a) bounce for small sizes or (b) a
new GFP_NODMA/PACKED/etc. flag for the hot small allocations. (a) is
somehow more universal but lots (most) Android devices are deployed with
no swiotlb buffer as the vendor knows the device needs and don't need
extra buffer waste. Not sure how reliable it would be to trigger another
slab allocation on the dma_map_*() calls for the bounce (it may need to
be GFP_ATOMIC). Option (b) looks more appealing on such systems, though
a lot more churn throughout the kernel.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel