Re: [PATCH] Revert "arm64: Increase the max granular size"

From: "Chalamarla, Tirumalesh" <Tirumalesh.Chalamarla@caviumnetworks.com>
To: Catalin Marinas <catalin.marinas@arm.com>,
	Will Deacon <will.deacon@arm.com>
Cc: Ganesh Mahendran <opensource.ganesh@gmail.com>,
	"stable@vger.kernel.org" <stable@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"linux-arm-kernel@lists.infradead.org" 
	<linux-arm-kernel@lists.infradead.org>
Subject: Re: [PATCH] Revert "arm64: Increase the max granular size"
Date: Mon, 21 Mar 2016 17:39:20 +0000	[thread overview]
Message-ID: <C1489C4D-9980-4889-B2F0-B071E58FD733@caviumnetworks.com> (raw)
In-Reply-To: <20160321173317.GF25466@e104818-lin.cambridge.arm.com>

On 3/21/16, 10:33 AM, "Catalin Marinas" <catalin.marinas@arm.com> wrote:

>On Mon, Mar 21, 2016 at 05:23:01PM +0000, Will Deacon wrote:
>> On Mon, Mar 21, 2016 at 05:14:03PM +0000, Catalin Marinas wrote:
>> > diff --git a/arch/arm64/include/asm/cache.h b/arch/arm64/include/asm/cache.h
>> > index 5082b30bc2c0..4b5d7b27edaf 100644
>> > --- a/arch/arm64/include/asm/cache.h
>> > +++ b/arch/arm64/include/asm/cache.h
>> > @@ -18,17 +18,17 @@
>> >  
>> >  #include <asm/cachetype.h>
>> >  
>> > -#define L1_CACHE_SHIFT		7
>> > +#define L1_CACHE_SHIFT		6
>> >  #define L1_CACHE_BYTES		(1 << L1_CACHE_SHIFT)
>> >  
>> >  /*
>> >   * Memory returned by kmalloc() may be used for DMA, so we must make
>> > - * sure that all such allocations are cache aligned. Otherwise,
>> > - * unrelated code may cause parts of the buffer to be read into the
>> > - * cache before the transfer is done, causing old data to be seen by
>> > - * the CPU.
>> > + * sure that all such allocations are aligned to the maximum *known*
>> > + * cache line size on ARMv8 systems. Otherwise, unrelated code may cause
>> > + * parts of the buffer to be read into the cache before the transfer is
>> > + * done, causing old data to be seen by the CPU.
>> >   */
>> > -#define ARCH_DMA_MINALIGN	L1_CACHE_BYTES
>> > +#define ARCH_DMA_MINALIGN	(128)
>> 
>> Does this actually fix the reported iperf regression? My assumption was
>> that ARCH_DMA_MINALIGN is the problem, but I could be wrong.
>
>I can't tell. But since I haven't seen any better explanation in this
>thread yet, I hope that at least someone would try this patch and come
>back with numbers.
>
>For networking, SKB_DATA_ALIGN() uses SMP_CACHE_BYTES (== L1_CACHE_BYTES).
>I think (hope) this alignment is not meant for non-coherent DMA,
>otherwise using SMP_CACHE_BYTES wouldn't make sense.

As I see the problem, may be it is because of fewer number of buffers available because of this alignment requirement.
But that should be fixed by making slab alignment a run time thing. I may be totally wrong. 

We are still coming up with a decent benchmark that shows degradation.  
>
>-- 
>Catalin