* [PATCH] riscv: add ARCH_DMA_MINALIGN support @ 2021-08-07 14:55 ` Xianting Tian 0 siblings, 0 replies; 16+ messages in thread From: Xianting Tian @ 2021-08-07 14:55 UTC (permalink / raw) To: paul.walmsley, palmer, aou; +Cc: linux-riscv, linux-kernel, Xianting Tian Introduce ARCH_DMA_MINALIGN to riscv arch. Signed-off-by: Xianting Tian <xianting.tian@linux.alibaba.com> --- arch/riscv/include/asm/cache.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/riscv/include/asm/cache.h b/arch/riscv/include/asm/cache.h index 9b58b1045..2945bbe2b 100644 --- a/arch/riscv/include/asm/cache.h +++ b/arch/riscv/include/asm/cache.h @@ -11,6 +11,8 @@ #define L1_CACHE_BYTES (1 << L1_CACHE_SHIFT) +#define ARCH_DMA_MINALIGN L1_CACHE_BYTES + /* * RISC-V requires the stack pointer to be 16-byte aligned, so ensure that * the flat loader aligns it accordingly. -- 2.17.1 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH] riscv: add ARCH_DMA_MINALIGN support @ 2021-08-07 14:55 ` Xianting Tian 0 siblings, 0 replies; 16+ messages in thread From: Xianting Tian @ 2021-08-07 14:55 UTC (permalink / raw) To: paul.walmsley, palmer, aou; +Cc: linux-riscv, linux-kernel, Xianting Tian Introduce ARCH_DMA_MINALIGN to riscv arch. Signed-off-by: Xianting Tian <xianting.tian@linux.alibaba.com> --- arch/riscv/include/asm/cache.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/riscv/include/asm/cache.h b/arch/riscv/include/asm/cache.h index 9b58b1045..2945bbe2b 100644 --- a/arch/riscv/include/asm/cache.h +++ b/arch/riscv/include/asm/cache.h @@ -11,6 +11,8 @@ #define L1_CACHE_BYTES (1 << L1_CACHE_SHIFT) +#define ARCH_DMA_MINALIGN L1_CACHE_BYTES + /* * RISC-V requires the stack pointer to be 16-byte aligned, so ensure that * the flat loader aligns it accordingly. -- 2.17.1 _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [PATCH] riscv: add ARCH_DMA_MINALIGN support 2021-08-07 14:55 ` Xianting Tian @ 2021-08-08 16:30 ` Jisheng Zhang -1 siblings, 0 replies; 16+ messages in thread From: Jisheng Zhang @ 2021-08-08 16:30 UTC (permalink / raw) To: Xianting Tian; +Cc: paul.walmsley, palmer, aou, linux-riscv, linux-kernel On Sat, 7 Aug 2021 22:55:37 +0800 Xianting Tian <xianting.tian@linux.alibaba.com> wrote: > Introduce ARCH_DMA_MINALIGN to riscv arch. > > Signed-off-by: Xianting Tian <xianting.tian@linux.alibaba.com> > --- > arch/riscv/include/asm/cache.h | 2 ++ > 1 file changed, 2 insertions(+) > > diff --git a/arch/riscv/include/asm/cache.h b/arch/riscv/include/asm/cache.h > index 9b58b1045..2945bbe2b 100644 > --- a/arch/riscv/include/asm/cache.h > +++ b/arch/riscv/include/asm/cache.h > @@ -11,6 +11,8 @@ > > #define L1_CACHE_BYTES (1 << L1_CACHE_SHIFT) > > +#define ARCH_DMA_MINALIGN L1_CACHE_BYTES It's not a good idea to blindly set this for all riscv. For "coherent" platforms, this is not necessary and will waste memory. > + > /* > * RISC-V requires the stack pointer to be 16-byte aligned, so ensure that > * the flat loader aligns it accordingly. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH] riscv: add ARCH_DMA_MINALIGN support @ 2021-08-08 16:30 ` Jisheng Zhang 0 siblings, 0 replies; 16+ messages in thread From: Jisheng Zhang @ 2021-08-08 16:30 UTC (permalink / raw) To: Xianting Tian; +Cc: paul.walmsley, palmer, aou, linux-riscv, linux-kernel On Sat, 7 Aug 2021 22:55:37 +0800 Xianting Tian <xianting.tian@linux.alibaba.com> wrote: > Introduce ARCH_DMA_MINALIGN to riscv arch. > > Signed-off-by: Xianting Tian <xianting.tian@linux.alibaba.com> > --- > arch/riscv/include/asm/cache.h | 2 ++ > 1 file changed, 2 insertions(+) > > diff --git a/arch/riscv/include/asm/cache.h b/arch/riscv/include/asm/cache.h > index 9b58b1045..2945bbe2b 100644 > --- a/arch/riscv/include/asm/cache.h > +++ b/arch/riscv/include/asm/cache.h > @@ -11,6 +11,8 @@ > > #define L1_CACHE_BYTES (1 << L1_CACHE_SHIFT) > > +#define ARCH_DMA_MINALIGN L1_CACHE_BYTES It's not a good idea to blindly set this for all riscv. For "coherent" platforms, this is not necessary and will waste memory. > + > /* > * RISC-V requires the stack pointer to be 16-byte aligned, so ensure that > * the flat loader aligns it accordingly. _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH] riscv: add ARCH_DMA_MINALIGN support 2021-08-08 16:30 ` Jisheng Zhang @ 2021-08-09 1:55 ` Xianting TIan -1 siblings, 0 replies; 16+ messages in thread From: Xianting TIan @ 2021-08-09 1:55 UTC (permalink / raw) To: Jisheng Zhang Cc: paul.walmsley, palmer, aou, linux-riscv, linux-kernel, guoren, arnd 在 2021/8/9 上午12:30, Jisheng Zhang 写道: > On Sat, 7 Aug 2021 22:55:37 +0800 > Xianting Tian <xianting.tian@linux.alibaba.com> wrote: > >> Introduce ARCH_DMA_MINALIGN to riscv arch. >> >> Signed-off-by: Xianting Tian <xianting.tian@linux.alibaba.com> >> --- >> arch/riscv/include/asm/cache.h | 2 ++ >> 1 file changed, 2 insertions(+) >> >> diff --git a/arch/riscv/include/asm/cache.h b/arch/riscv/include/asm/cache.h >> index 9b58b1045..2945bbe2b 100644 >> --- a/arch/riscv/include/asm/cache.h >> +++ b/arch/riscv/include/asm/cache.h >> @@ -11,6 +11,8 @@ >> >> #define L1_CACHE_BYTES (1 << L1_CACHE_SHIFT) >> >> +#define ARCH_DMA_MINALIGN L1_CACHE_BYTES > It's not a good idea to blindly set this for all riscv. For "coherent" > platforms, this is not necessary and will waste memory. > thanks for the reply, So riscv is the "coherent" platform? I submit this patch as I got a fix suggestion of another patch to use ARCH_DMA_MINALIGN, but riscv doesn't define it. https://lkml.org/lkml/2021/8/6/723 <https://lkml.org/lkml/2021/8/6/723> Considering the portability of the code, in my opinion, it is better to define it for riscv if it is not "coherent" platform. >> + >> /* >> * RISC-V requires the stack pointer to be 16-byte aligned, so ensure that >> * the flat loader aligns it accordingly. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH] riscv: add ARCH_DMA_MINALIGN support @ 2021-08-09 1:55 ` Xianting TIan 0 siblings, 0 replies; 16+ messages in thread From: Xianting TIan @ 2021-08-09 1:55 UTC (permalink / raw) To: Jisheng Zhang Cc: paul.walmsley, palmer, aou, linux-riscv, linux-kernel, guoren, arnd 在 2021/8/9 上午12:30, Jisheng Zhang 写道: > On Sat, 7 Aug 2021 22:55:37 +0800 > Xianting Tian <xianting.tian@linux.alibaba.com> wrote: > >> Introduce ARCH_DMA_MINALIGN to riscv arch. >> >> Signed-off-by: Xianting Tian <xianting.tian@linux.alibaba.com> >> --- >> arch/riscv/include/asm/cache.h | 2 ++ >> 1 file changed, 2 insertions(+) >> >> diff --git a/arch/riscv/include/asm/cache.h b/arch/riscv/include/asm/cache.h >> index 9b58b1045..2945bbe2b 100644 >> --- a/arch/riscv/include/asm/cache.h >> +++ b/arch/riscv/include/asm/cache.h >> @@ -11,6 +11,8 @@ >> >> #define L1_CACHE_BYTES (1 << L1_CACHE_SHIFT) >> >> +#define ARCH_DMA_MINALIGN L1_CACHE_BYTES > It's not a good idea to blindly set this for all riscv. For "coherent" > platforms, this is not necessary and will waste memory. > thanks for the reply, So riscv is the "coherent" platform? I submit this patch as I got a fix suggestion of another patch to use ARCH_DMA_MINALIGN, but riscv doesn't define it. https://lkml.org/lkml/2021/8/6/723 <https://lkml.org/lkml/2021/8/6/723> Considering the portability of the code, in my opinion, it is better to define it for riscv if it is not "coherent" platform. >> + >> /* >> * RISC-V requires the stack pointer to be 16-byte aligned, so ensure that >> * the flat loader aligns it accordingly. _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH] riscv: add ARCH_DMA_MINALIGN support 2021-08-09 1:55 ` Xianting TIan @ 2021-08-10 1:30 ` Guo Ren -1 siblings, 0 replies; 16+ messages in thread From: Guo Ren @ 2021-08-10 1:30 UTC (permalink / raw) To: Xianting TIan Cc: Jisheng Zhang, Paul Walmsley, Palmer Dabbelt, Albert Ou, linux-riscv, Linux Kernel Mailing List, Arnd Bergmann On Mon, Aug 9, 2021 at 9:55 AM Xianting TIan <xianting.tian@linux.alibaba.com> wrote: > > > 在 2021/8/9 上午12:30, Jisheng Zhang 写道: > > On Sat, 7 Aug 2021 22:55:37 +0800 > > Xianting Tian <xianting.tian@linux.alibaba.com> wrote: > > > >> Introduce ARCH_DMA_MINALIGN to riscv arch. > >> > >> Signed-off-by: Xianting Tian <xianting.tian@linux.alibaba.com> > >> --- > >> arch/riscv/include/asm/cache.h | 2 ++ > >> 1 file changed, 2 insertions(+) > >> > >> diff --git a/arch/riscv/include/asm/cache.h b/arch/riscv/include/asm/cache.h > >> index 9b58b1045..2945bbe2b 100644 > >> --- a/arch/riscv/include/asm/cache.h > >> +++ b/arch/riscv/include/asm/cache.h > >> @@ -11,6 +11,8 @@ > >> > >> #define L1_CACHE_BYTES (1 << L1_CACHE_SHIFT) > >> > >> +#define ARCH_DMA_MINALIGN L1_CACHE_BYTES > > It's not a good idea to blindly set this for all riscv. For "coherent" > > platforms, this is not necessary and will waste memory. quote from slab.h: * Some archs want to perform DMA into kmalloc caches and need a guaranteed * alignment larger than the alignment of a 64-bit integer. * Setting ARCH_KMALLOC_MINALIGN in arch headers allows that. ARCH_DMA_MINALIGN is for the whole system, maybe we could give a DMA_MINALIGN Kconfig entry in arch/riscv? > > > thanks for the reply, > > So riscv is the "coherent" platform? > > I submit this patch as I got a fix suggestion of another patch to use > ARCH_DMA_MINALIGN, but riscv doesn't define it. > > https://lkml.org/lkml/2021/8/6/723 <https://lkml.org/lkml/2021/8/6/723> > > Considering the portability of the code, in my opinion, it is better to > define it for riscv if it is not "coherent" platform. > > >> + > >> /* > >> * RISC-V requires the stack pointer to be 16-byte aligned, so ensure that > >> * the flat loader aligns it accordingly. -- Best Regards Guo Ren ML: https://lore.kernel.org/linux-csky/ ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH] riscv: add ARCH_DMA_MINALIGN support @ 2021-08-10 1:30 ` Guo Ren 0 siblings, 0 replies; 16+ messages in thread From: Guo Ren @ 2021-08-10 1:30 UTC (permalink / raw) To: Xianting TIan Cc: Jisheng Zhang, Paul Walmsley, Palmer Dabbelt, Albert Ou, linux-riscv, Linux Kernel Mailing List, Arnd Bergmann On Mon, Aug 9, 2021 at 9:55 AM Xianting TIan <xianting.tian@linux.alibaba.com> wrote: > > > 在 2021/8/9 上午12:30, Jisheng Zhang 写道: > > On Sat, 7 Aug 2021 22:55:37 +0800 > > Xianting Tian <xianting.tian@linux.alibaba.com> wrote: > > > >> Introduce ARCH_DMA_MINALIGN to riscv arch. > >> > >> Signed-off-by: Xianting Tian <xianting.tian@linux.alibaba.com> > >> --- > >> arch/riscv/include/asm/cache.h | 2 ++ > >> 1 file changed, 2 insertions(+) > >> > >> diff --git a/arch/riscv/include/asm/cache.h b/arch/riscv/include/asm/cache.h > >> index 9b58b1045..2945bbe2b 100644 > >> --- a/arch/riscv/include/asm/cache.h > >> +++ b/arch/riscv/include/asm/cache.h > >> @@ -11,6 +11,8 @@ > >> > >> #define L1_CACHE_BYTES (1 << L1_CACHE_SHIFT) > >> > >> +#define ARCH_DMA_MINALIGN L1_CACHE_BYTES > > It's not a good idea to blindly set this for all riscv. For "coherent" > > platforms, this is not necessary and will waste memory. quote from slab.h: * Some archs want to perform DMA into kmalloc caches and need a guaranteed * alignment larger than the alignment of a 64-bit integer. * Setting ARCH_KMALLOC_MINALIGN in arch headers allows that. ARCH_DMA_MINALIGN is for the whole system, maybe we could give a DMA_MINALIGN Kconfig entry in arch/riscv? > > > thanks for the reply, > > So riscv is the "coherent" platform? > > I submit this patch as I got a fix suggestion of another patch to use > ARCH_DMA_MINALIGN, but riscv doesn't define it. > > https://lkml.org/lkml/2021/8/6/723 <https://lkml.org/lkml/2021/8/6/723> > > Considering the portability of the code, in my opinion, it is better to > define it for riscv if it is not "coherent" platform. > > >> + > >> /* > >> * RISC-V requires the stack pointer to be 16-byte aligned, so ensure that > >> * the flat loader aligns it accordingly. -- Best Regards Guo Ren ML: https://lore.kernel.org/linux-csky/ _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH] riscv: add ARCH_DMA_MINALIGN support 2021-08-08 16:30 ` Jisheng Zhang @ 2021-08-09 6:20 ` Xianting TIan -1 siblings, 0 replies; 16+ messages in thread From: Xianting TIan @ 2021-08-09 6:20 UTC (permalink / raw) To: Jisheng Zhang; +Cc: paul.walmsley, palmer, aou, linux-riscv, linux-kernel 在 2021/8/9 上午12:30, Jisheng Zhang 写道: > On Sat, 7 Aug 2021 22:55:37 +0800 > Xianting Tian <xianting.tian@linux.alibaba.com> wrote: > >> Introduce ARCH_DMA_MINALIGN to riscv arch. >> >> Signed-off-by: Xianting Tian <xianting.tian@linux.alibaba.com> >> --- >> arch/riscv/include/asm/cache.h | 2 ++ >> 1 file changed, 2 insertions(+) >> >> diff --git a/arch/riscv/include/asm/cache.h b/arch/riscv/include/asm/cache.h >> index 9b58b1045..2945bbe2b 100644 >> --- a/arch/riscv/include/asm/cache.h >> +++ b/arch/riscv/include/asm/cache.h >> @@ -11,6 +11,8 @@ >> >> #define L1_CACHE_BYTES (1 << L1_CACHE_SHIFT) >> >> +#define ARCH_DMA_MINALIGN L1_CACHE_BYTES > It's not a good idea to blindly set this for all riscv. For "coherent" > platforms, this is not necessary and will waste memory. I checked ARCH_DMA_MINALIGN definition, "If an architecture isn't fully DMA-coherent, ARCH_DMA_MINALIGN must be set". so that the memory allocator makes sure that kmalloc'ed buffer doesn't share a cache line with the others. Documentation/core-api/dma-api-howto.rst 2) ARCH_DMA_MINALIGN Architectures must ensure that kmalloc'ed buffer is DMA-safe. Drivers and subsystems depend on it. If an architecture isn't fully DMA-coherent (i.e. hardware doesn't ensure that data in the CPU cache is identical to data in main memory), ARCH_DMA_MINALIGN must be set so that the memory allocator makes sure that kmalloc'ed buffer doesn't share a cache line with the others. See arch/arm/include/asm/cache.h as an example. Note that ARCH_DMA_MINALIGN is about DMA memory alignment constraints. You don't need to worry about the architecture data alignment constraints (e.g. the alignment constraints about 64-bit objects). > >> + >> /* >> * RISC-V requires the stack pointer to be 16-byte aligned, so ensure that >> * the flat loader aligns it accordingly. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH] riscv: add ARCH_DMA_MINALIGN support @ 2021-08-09 6:20 ` Xianting TIan 0 siblings, 0 replies; 16+ messages in thread From: Xianting TIan @ 2021-08-09 6:20 UTC (permalink / raw) To: Jisheng Zhang; +Cc: paul.walmsley, palmer, aou, linux-riscv, linux-kernel 在 2021/8/9 上午12:30, Jisheng Zhang 写道: > On Sat, 7 Aug 2021 22:55:37 +0800 > Xianting Tian <xianting.tian@linux.alibaba.com> wrote: > >> Introduce ARCH_DMA_MINALIGN to riscv arch. >> >> Signed-off-by: Xianting Tian <xianting.tian@linux.alibaba.com> >> --- >> arch/riscv/include/asm/cache.h | 2 ++ >> 1 file changed, 2 insertions(+) >> >> diff --git a/arch/riscv/include/asm/cache.h b/arch/riscv/include/asm/cache.h >> index 9b58b1045..2945bbe2b 100644 >> --- a/arch/riscv/include/asm/cache.h >> +++ b/arch/riscv/include/asm/cache.h >> @@ -11,6 +11,8 @@ >> >> #define L1_CACHE_BYTES (1 << L1_CACHE_SHIFT) >> >> +#define ARCH_DMA_MINALIGN L1_CACHE_BYTES > It's not a good idea to blindly set this for all riscv. For "coherent" > platforms, this is not necessary and will waste memory. I checked ARCH_DMA_MINALIGN definition, "If an architecture isn't fully DMA-coherent, ARCH_DMA_MINALIGN must be set". so that the memory allocator makes sure that kmalloc'ed buffer doesn't share a cache line with the others. Documentation/core-api/dma-api-howto.rst 2) ARCH_DMA_MINALIGN Architectures must ensure that kmalloc'ed buffer is DMA-safe. Drivers and subsystems depend on it. If an architecture isn't fully DMA-coherent (i.e. hardware doesn't ensure that data in the CPU cache is identical to data in main memory), ARCH_DMA_MINALIGN must be set so that the memory allocator makes sure that kmalloc'ed buffer doesn't share a cache line with the others. See arch/arm/include/asm/cache.h as an example. Note that ARCH_DMA_MINALIGN is about DMA memory alignment constraints. You don't need to worry about the architecture data alignment constraints (e.g. the alignment constraints about 64-bit objects). > >> + >> /* >> * RISC-V requires the stack pointer to be 16-byte aligned, so ensure that >> * the flat loader aligns it accordingly. _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH] riscv: add ARCH_DMA_MINALIGN support 2021-08-09 6:20 ` Xianting TIan @ 2021-08-09 7:49 ` Arnd Bergmann -1 siblings, 0 replies; 16+ messages in thread From: Arnd Bergmann @ 2021-08-09 7:49 UTC (permalink / raw) To: Xianting TIan Cc: Jisheng Zhang, Paul Walmsley, Palmer Dabbelt, Albert Ou, linux-riscv, Linux Kernel Mailing List On Mon, Aug 9, 2021 at 8:20 AM Xianting TIan <xianting.tian@linux.alibaba.com> wrote: > > >> +#define ARCH_DMA_MINALIGN L1_CACHE_BYTES > > It's not a good idea to blindly set this for all riscv. For "coherent" > > platforms, this is not necessary and will waste memory. > > I checked ARCH_DMA_MINALIGN definition, "If an architecture isn't fully > DMA-coherent, ARCH_DMA_MINALIGN must be set". > > so that the memory allocator makes sure that kmalloc'ed buffer doesn't > share a cache line with the others. > > Documentation/core-api/dma-api-howto.rst > > 2) ARCH_DMA_MINALIGN > > Architectures must ensure that kmalloc'ed buffer is > DMA-safe. Drivers and subsystems depend on it. If an architecture > isn't fully DMA-coherent (i.e. hardware doesn't ensure that data in > the CPU cache is identical to data in main memory), > ARCH_DMA_MINALIGN must be set so that the memory allocator > makes sure that kmalloc'ed buffer doesn't share a cache line with > the others. See arch/arm/include/asm/cache.h as an example. > > Note that ARCH_DMA_MINALIGN is about DMA memory alignment > constraints. You don't need to worry about the architecture data > alignment constraints (e.g. the alignment constraints about 64-bit > objects). The platform spec [1] says about this: | Memory accesses by I/O masters can be coherent or non-coherent | with respect to all hart-related caches. So the kernel in its default configuration can not assume that DMA is cache coherent on RISC-V. Making this configurable implies that a kernel that is configured for cache-coherent machines can no longer run on all hardware that follows the platform spec. We have the same problem on arm64, where most of the server parts are cache coherent, but the majority of the low-end embedded devices are not, and we require that a single kernel ran run on all of the above. One idea that we have discussed several times is to start the kernel without the small kmalloc caches and defer their creation until a later point in the boot process after determining whether any non-coherent devices have been discovered. Any in-kernel structures that have an explicit ARCH_DMA_MINALIGN alignment won't benefit from this, but any subsequent kmalloc() calls can use the smaller caches. The tricky bit is finding out whether /everything/ on the system is cache-coherent or not, since we do not have a global flag for that in the DT. See [2] for a recent discussion. Arnd [1] https://github.com/riscv/riscv-platform-specs/blob/main/riscv-platform-spec.adoc#architecture [2] https://lore.kernel.org/linux-arm-kernel/20210527124356.22367-1-will@kernel.org/ ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH] riscv: add ARCH_DMA_MINALIGN support @ 2021-08-09 7:49 ` Arnd Bergmann 0 siblings, 0 replies; 16+ messages in thread From: Arnd Bergmann @ 2021-08-09 7:49 UTC (permalink / raw) To: Xianting TIan Cc: Jisheng Zhang, Paul Walmsley, Palmer Dabbelt, Albert Ou, linux-riscv, Linux Kernel Mailing List On Mon, Aug 9, 2021 at 8:20 AM Xianting TIan <xianting.tian@linux.alibaba.com> wrote: > > >> +#define ARCH_DMA_MINALIGN L1_CACHE_BYTES > > It's not a good idea to blindly set this for all riscv. For "coherent" > > platforms, this is not necessary and will waste memory. > > I checked ARCH_DMA_MINALIGN definition, "If an architecture isn't fully > DMA-coherent, ARCH_DMA_MINALIGN must be set". > > so that the memory allocator makes sure that kmalloc'ed buffer doesn't > share a cache line with the others. > > Documentation/core-api/dma-api-howto.rst > > 2) ARCH_DMA_MINALIGN > > Architectures must ensure that kmalloc'ed buffer is > DMA-safe. Drivers and subsystems depend on it. If an architecture > isn't fully DMA-coherent (i.e. hardware doesn't ensure that data in > the CPU cache is identical to data in main memory), > ARCH_DMA_MINALIGN must be set so that the memory allocator > makes sure that kmalloc'ed buffer doesn't share a cache line with > the others. See arch/arm/include/asm/cache.h as an example. > > Note that ARCH_DMA_MINALIGN is about DMA memory alignment > constraints. You don't need to worry about the architecture data > alignment constraints (e.g. the alignment constraints about 64-bit > objects). The platform spec [1] says about this: | Memory accesses by I/O masters can be coherent or non-coherent | with respect to all hart-related caches. So the kernel in its default configuration can not assume that DMA is cache coherent on RISC-V. Making this configurable implies that a kernel that is configured for cache-coherent machines can no longer run on all hardware that follows the platform spec. We have the same problem on arm64, where most of the server parts are cache coherent, but the majority of the low-end embedded devices are not, and we require that a single kernel ran run on all of the above. One idea that we have discussed several times is to start the kernel without the small kmalloc caches and defer their creation until a later point in the boot process after determining whether any non-coherent devices have been discovered. Any in-kernel structures that have an explicit ARCH_DMA_MINALIGN alignment won't benefit from this, but any subsequent kmalloc() calls can use the smaller caches. The tricky bit is finding out whether /everything/ on the system is cache-coherent or not, since we do not have a global flag for that in the DT. See [2] for a recent discussion. Arnd [1] https://github.com/riscv/riscv-platform-specs/blob/main/riscv-platform-spec.adoc#architecture [2] https://lore.kernel.org/linux-arm-kernel/20210527124356.22367-1-will@kernel.org/ _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH] riscv: add ARCH_DMA_MINALIGN support 2021-08-09 7:49 ` Arnd Bergmann @ 2021-08-09 9:00 ` Xianting TIan -1 siblings, 0 replies; 16+ messages in thread From: Xianting TIan @ 2021-08-09 9:00 UTC (permalink / raw) To: Arnd Bergmann Cc: Jisheng Zhang, Paul Walmsley, Palmer Dabbelt, Albert Ou, linux-riscv, Linux Kernel Mailing List 在 2021/8/9 下午3:49, Arnd Bergmann 写道: > On Mon, Aug 9, 2021 at 8:20 AM Xianting TIan > <xianting.tian@linux.alibaba.com> wrote: >>>> +#define ARCH_DMA_MINALIGN L1_CACHE_BYTES >>> It's not a good idea to blindly set this for all riscv. For "coherent" >>> platforms, this is not necessary and will waste memory. >> I checked ARCH_DMA_MINALIGN definition, "If an architecture isn't fully >> DMA-coherent, ARCH_DMA_MINALIGN must be set". >> >> so that the memory allocator makes sure that kmalloc'ed buffer doesn't >> share a cache line with the others. >> >> Documentation/core-api/dma-api-howto.rst >> >> 2) ARCH_DMA_MINALIGN >> >> Architectures must ensure that kmalloc'ed buffer is >> DMA-safe. Drivers and subsystems depend on it. If an architecture >> isn't fully DMA-coherent (i.e. hardware doesn't ensure that data in >> the CPU cache is identical to data in main memory), >> ARCH_DMA_MINALIGN must be set so that the memory allocator >> makes sure that kmalloc'ed buffer doesn't share a cache line with >> the others. See arch/arm/include/asm/cache.h as an example. >> >> Note that ARCH_DMA_MINALIGN is about DMA memory alignment >> constraints. You don't need to worry about the architecture data >> alignment constraints (e.g. the alignment constraints about 64-bit >> objects). > The platform spec [1] says about this: > > | Memory accesses by I/O masters can be coherent or non-coherent > | with respect to all hart-related caches. > > So the kernel in its default configuration can not assume that DMA is > cache coherent on RISC-V. Making this configurable implies that > a kernel that is configured for cache-coherent machines can no longer > run on all hardware that follows the platform spec. > > We have the same problem on arm64, where most of the server parts > are cache coherent, but the majority of the low-end embedded devices > are not, and we require that a single kernel ran run on all of the above. > > One idea that we have discussed several times is to start the kernel > without the small kmalloc caches and defer their creation until a > later point in the boot process after determining whether any > non-coherent devices have been discovered. Any in-kernel structures > that have an explicit ARCH_DMA_MINALIGN alignment won't > benefit from this, but any subsequent kmalloc() calls can use the > smaller caches. The tricky bit is finding out whether /everything/ on > the system is cache-coherent or not, since we do not have a global > flag for that in the DT. See [2] for a recent discussion. > > Arnd > > [1] https://github.com/riscv/riscv-platform-specs/blob/main/riscv-platform-spec.adoc#architecture > [2] https://lore.kernel.org/linux-arm-kernel/20210527124356.22367-1-will@kernel.org/ Arnd, thanks for info, according to the description, seems we need to apply this patch to riscv. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH] riscv: add ARCH_DMA_MINALIGN support @ 2021-08-09 9:00 ` Xianting TIan 0 siblings, 0 replies; 16+ messages in thread From: Xianting TIan @ 2021-08-09 9:00 UTC (permalink / raw) To: Arnd Bergmann Cc: Jisheng Zhang, Paul Walmsley, Palmer Dabbelt, Albert Ou, linux-riscv, Linux Kernel Mailing List 在 2021/8/9 下午3:49, Arnd Bergmann 写道: > On Mon, Aug 9, 2021 at 8:20 AM Xianting TIan > <xianting.tian@linux.alibaba.com> wrote: >>>> +#define ARCH_DMA_MINALIGN L1_CACHE_BYTES >>> It's not a good idea to blindly set this for all riscv. For "coherent" >>> platforms, this is not necessary and will waste memory. >> I checked ARCH_DMA_MINALIGN definition, "If an architecture isn't fully >> DMA-coherent, ARCH_DMA_MINALIGN must be set". >> >> so that the memory allocator makes sure that kmalloc'ed buffer doesn't >> share a cache line with the others. >> >> Documentation/core-api/dma-api-howto.rst >> >> 2) ARCH_DMA_MINALIGN >> >> Architectures must ensure that kmalloc'ed buffer is >> DMA-safe. Drivers and subsystems depend on it. If an architecture >> isn't fully DMA-coherent (i.e. hardware doesn't ensure that data in >> the CPU cache is identical to data in main memory), >> ARCH_DMA_MINALIGN must be set so that the memory allocator >> makes sure that kmalloc'ed buffer doesn't share a cache line with >> the others. See arch/arm/include/asm/cache.h as an example. >> >> Note that ARCH_DMA_MINALIGN is about DMA memory alignment >> constraints. You don't need to worry about the architecture data >> alignment constraints (e.g. the alignment constraints about 64-bit >> objects). > The platform spec [1] says about this: > > | Memory accesses by I/O masters can be coherent or non-coherent > | with respect to all hart-related caches. > > So the kernel in its default configuration can not assume that DMA is > cache coherent on RISC-V. Making this configurable implies that > a kernel that is configured for cache-coherent machines can no longer > run on all hardware that follows the platform spec. > > We have the same problem on arm64, where most of the server parts > are cache coherent, but the majority of the low-end embedded devices > are not, and we require that a single kernel ran run on all of the above. > > One idea that we have discussed several times is to start the kernel > without the small kmalloc caches and defer their creation until a > later point in the boot process after determining whether any > non-coherent devices have been discovered. Any in-kernel structures > that have an explicit ARCH_DMA_MINALIGN alignment won't > benefit from this, but any subsequent kmalloc() calls can use the > smaller caches. The tricky bit is finding out whether /everything/ on > the system is cache-coherent or not, since we do not have a global > flag for that in the DT. See [2] for a recent discussion. > > Arnd > > [1] https://github.com/riscv/riscv-platform-specs/blob/main/riscv-platform-spec.adoc#architecture > [2] https://lore.kernel.org/linux-arm-kernel/20210527124356.22367-1-will@kernel.org/ Arnd, thanks for info, according to the description, seems we need to apply this patch to riscv. _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH] riscv: add ARCH_DMA_MINALIGN support 2021-08-09 7:49 ` Arnd Bergmann @ 2021-08-09 19:19 ` Atish Patra -1 siblings, 0 replies; 16+ messages in thread From: Atish Patra @ 2021-08-09 19:19 UTC (permalink / raw) To: Arnd Bergmann Cc: Xianting TIan, Jisheng Zhang, Paul Walmsley, Palmer Dabbelt, Albert Ou, linux-riscv, Linux Kernel Mailing List On Mon, Aug 9, 2021 at 12:50 AM Arnd Bergmann <arnd@kernel.org> wrote: > > On Mon, Aug 9, 2021 at 8:20 AM Xianting TIan > <xianting.tian@linux.alibaba.com> wrote: > > > > >> +#define ARCH_DMA_MINALIGN L1_CACHE_BYTES > > > It's not a good idea to blindly set this for all riscv. For "coherent" > > > platforms, this is not necessary and will waste memory. > > > > I checked ARCH_DMA_MINALIGN definition, "If an architecture isn't fully > > DMA-coherent, ARCH_DMA_MINALIGN must be set". > > > > so that the memory allocator makes sure that kmalloc'ed buffer doesn't > > share a cache line with the others. > > > > Documentation/core-api/dma-api-howto.rst > > > > 2) ARCH_DMA_MINALIGN > > > > Architectures must ensure that kmalloc'ed buffer is > > DMA-safe. Drivers and subsystems depend on it. If an architecture > > isn't fully DMA-coherent (i.e. hardware doesn't ensure that data in > > the CPU cache is identical to data in main memory), > > ARCH_DMA_MINALIGN must be set so that the memory allocator > > makes sure that kmalloc'ed buffer doesn't share a cache line with > > the others. See arch/arm/include/asm/cache.h as an example. > > > > Note that ARCH_DMA_MINALIGN is about DMA memory alignment > > constraints. You don't need to worry about the architecture data > > alignment constraints (e.g. the alignment constraints about 64-bit > > objects). > > The platform spec [1] says about this: > > | Memory accesses by I/O masters can be coherent or non-coherent > | with respect to all hart-related caches. > > So the kernel in its default configuration can not assume that DMA is > cache coherent on RISC-V. Making this configurable implies that > a kernel that is configured for cache-coherent machines can no longer > run on all hardware that follows the platform spec. > > We have the same problem on arm64, where most of the server parts > are cache coherent, but the majority of the low-end embedded devices > are not, and we require that a single kernel ran run on all of the above. > > One idea that we have discussed several times is to start the kernel > without the small kmalloc caches and defer their creation until a > later point in the boot process after determining whether any > non-coherent devices have been discovered. Any in-kernel structures > that have an explicit ARCH_DMA_MINALIGN alignment won't > benefit from this, but any subsequent kmalloc() calls can use the > smaller caches. The tricky bit is finding out whether /everything/ on > the system is cache-coherent or not, since we do not have a global > flag for that in the DT. See [2] for a recent discussion. Can we add a new DT property to indicate the system is fully cache-coherent ? That will be helpful for RISC-V as well. We already have platforms like hifive unleashed/unmatched that are coherent while beagleV is not. The workaround to support both platforms in a single image was not very pretty [1]. [1] https://patchwork.kernel.org/project/linux-riscv/list/?series=520541 > > Arnd > > [1] https://github.com/riscv/riscv-platform-specs/blob/main/riscv-platform-spec.adoc#architecture > [2] https://lore.kernel.org/linux-arm-kernel/20210527124356.22367-1-will@kernel.org/ > > _______________________________________________ > linux-riscv mailing list > linux-riscv@lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-riscv -- Regards, Atish ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH] riscv: add ARCH_DMA_MINALIGN support @ 2021-08-09 19:19 ` Atish Patra 0 siblings, 0 replies; 16+ messages in thread From: Atish Patra @ 2021-08-09 19:19 UTC (permalink / raw) To: Arnd Bergmann Cc: Xianting TIan, Jisheng Zhang, Paul Walmsley, Palmer Dabbelt, Albert Ou, linux-riscv, Linux Kernel Mailing List On Mon, Aug 9, 2021 at 12:50 AM Arnd Bergmann <arnd@kernel.org> wrote: > > On Mon, Aug 9, 2021 at 8:20 AM Xianting TIan > <xianting.tian@linux.alibaba.com> wrote: > > > > >> +#define ARCH_DMA_MINALIGN L1_CACHE_BYTES > > > It's not a good idea to blindly set this for all riscv. For "coherent" > > > platforms, this is not necessary and will waste memory. > > > > I checked ARCH_DMA_MINALIGN definition, "If an architecture isn't fully > > DMA-coherent, ARCH_DMA_MINALIGN must be set". > > > > so that the memory allocator makes sure that kmalloc'ed buffer doesn't > > share a cache line with the others. > > > > Documentation/core-api/dma-api-howto.rst > > > > 2) ARCH_DMA_MINALIGN > > > > Architectures must ensure that kmalloc'ed buffer is > > DMA-safe. Drivers and subsystems depend on it. If an architecture > > isn't fully DMA-coherent (i.e. hardware doesn't ensure that data in > > the CPU cache is identical to data in main memory), > > ARCH_DMA_MINALIGN must be set so that the memory allocator > > makes sure that kmalloc'ed buffer doesn't share a cache line with > > the others. See arch/arm/include/asm/cache.h as an example. > > > > Note that ARCH_DMA_MINALIGN is about DMA memory alignment > > constraints. You don't need to worry about the architecture data > > alignment constraints (e.g. the alignment constraints about 64-bit > > objects). > > The platform spec [1] says about this: > > | Memory accesses by I/O masters can be coherent or non-coherent > | with respect to all hart-related caches. > > So the kernel in its default configuration can not assume that DMA is > cache coherent on RISC-V. Making this configurable implies that > a kernel that is configured for cache-coherent machines can no longer > run on all hardware that follows the platform spec. > > We have the same problem on arm64, where most of the server parts > are cache coherent, but the majority of the low-end embedded devices > are not, and we require that a single kernel ran run on all of the above. > > One idea that we have discussed several times is to start the kernel > without the small kmalloc caches and defer their creation until a > later point in the boot process after determining whether any > non-coherent devices have been discovered. Any in-kernel structures > that have an explicit ARCH_DMA_MINALIGN alignment won't > benefit from this, but any subsequent kmalloc() calls can use the > smaller caches. The tricky bit is finding out whether /everything/ on > the system is cache-coherent or not, since we do not have a global > flag for that in the DT. See [2] for a recent discussion. Can we add a new DT property to indicate the system is fully cache-coherent ? That will be helpful for RISC-V as well. We already have platforms like hifive unleashed/unmatched that are coherent while beagleV is not. The workaround to support both platforms in a single image was not very pretty [1]. [1] https://patchwork.kernel.org/project/linux-riscv/list/?series=520541 > > Arnd > > [1] https://github.com/riscv/riscv-platform-specs/blob/main/riscv-platform-spec.adoc#architecture > [2] https://lore.kernel.org/linux-arm-kernel/20210527124356.22367-1-will@kernel.org/ > > _______________________________________________ > linux-riscv mailing list > linux-riscv@lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-riscv -- Regards, Atish _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2021-08-10 1:31 UTC | newest] Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-08-07 14:55 [PATCH] riscv: add ARCH_DMA_MINALIGN support Xianting Tian 2021-08-07 14:55 ` Xianting Tian 2021-08-08 16:30 ` Jisheng Zhang 2021-08-08 16:30 ` Jisheng Zhang 2021-08-09 1:55 ` Xianting TIan 2021-08-09 1:55 ` Xianting TIan 2021-08-10 1:30 ` Guo Ren 2021-08-10 1:30 ` Guo Ren 2021-08-09 6:20 ` Xianting TIan 2021-08-09 6:20 ` Xianting TIan 2021-08-09 7:49 ` Arnd Bergmann 2021-08-09 7:49 ` Arnd Bergmann 2021-08-09 9:00 ` Xianting TIan 2021-08-09 9:00 ` Xianting TIan 2021-08-09 19:19 ` Atish Patra 2021-08-09 19:19 ` Atish Patra
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.