Re: [PATCH] MIPS: CPU_LOONGSON2EF need software to maintain cache consistency

From: Lichao Liu <liulichao@loongson.cn>
To: Jiaxun Yang <jiaxun.yang@flygoat.com>
Cc: Thomas Bogendoerfer <tsbogend@alpha.franken.de>,
	Paul Burton <paulburton@kernel.org>,
	Robin Murphy <robin.murphy@arm.com>,
	Geert Uytterhoeven <geert@linux-m68k.org>,
	Max Filippov <jcmvbkbc@gmail.com>,
	yuanjunqing@loongson.cn, linux-mips@vger.kernel.org
Subject: Re: [PATCH] MIPS: CPU_LOONGSON2EF need software to maintain cache consistency
Date: Tue, 26 May 2020 22:50:42 +0800	[thread overview]
Message-ID: <678b6ae6-1396-1835-1145-faab7a44fbb2@loongson.cn> (raw)
In-Reply-To: <20200526212544.1737411f@halation.net.flygoat.com>

在 2020/5/26 下午9:25, Jiaxun Yang 写道:
> On Tue, 26 May 2020 20:40:28 +0800
> Lichao Liu <liulichao@loongson.cn> wrote:
>
>> On 2020/5/26 下午7:38, Jiaxun Yang wrote:
>>> On Tue, 26 May 2020 19:14:38 +0800
>>> Lichao Liu <liulichao@loongson.cn> wrote:
>>>  
>>>> CPU_LOONGSON2EF need software to maintain cache consistency,
>>>> so modify the 'cpu_needs_post_dma_flush' function to return true
>>>> when the cpu type is CPU_LOONGSON2EF.  
>>> Hi Lichao,
>>>
>>> I don't think that's required for Loongson-2EF,
>>>
>>> According to the comment in code:
>>>
>>> The affected CPUs below in 'cpu_needs_post_dma_flush()' can
>>> speculatively
>>> fill random cachelines with stale data at any time, requiring an
>>> extra flush post-DMA.
>>>
>>> And according to my understanding that's not going to happen on
>>> Loongson-2EF. We're always allocating coherent DMA memory in
>>> uncached range, Loongson-2EF's writeback policy will ensure it
>>> won't writeback random lines to the memory but only modified dirty
>>> lines.
>>>
>>> We've been fine without post flush for almost 10 years, there is no
>>> stability issue revealed.
>>>
>>> Btw: Please keep me CCed for Loongson-2EF patches. I'm not very
>>> active on 2EF development but I'll still review patches.
>>>
>>> Thanks.
>>>  
>> Hi Jiaxun,
>>
>> Loongson-2EF need software maintain cache consistency, So when using 
>> streaming DMA, software needs to maintain consistency.
>>
>> dma_map_single() is correct, but dma_unmap_single is wrong. 
>>
>> The function call path:
>> 'dma_unmap_single->dma_unmap_page_attrs->dma_direct_unmap_page->
>>  dma_direct_sync_single_for_cpu->arch_sync_dma_for_cpu->
>>  cpu_needs_post_dma_flush'
>>
>> In current version, 'cpu_needs_post_dma_flush' will return false 
>> at Loongon-2EF platform, and dma_unmap_single will not invalidate
>> cache, driver may access wrong dma data.
>>
>> I don't know what's the exact meaning of "fill random cachelines with 
>> stale data at any time". I always think 'cpu_needs_post_dma_flush()' 
>> means whether this platform needs software to maintain cache
>> consistency.
>>
>> I found this problem in 4.19.90 kernel's ethernet driver, 
>> and this patch can fix this problem.
>
> Which machine with which ethernet card?
>
> My Fuloong 2F is still serving as printer server and looks like there
> is no stability issue.
>
> Thanks.
> --
> Jiaxun Yang

I think a common intel igb or e1000 nic can reproduce the problem, 
I have encountered this problem on both 2.6 and 4.19 kernel.
Try use tftp to get a large file, such as 200M, then compare their md5.