Re: [PATCH 4/9] dmaengine: dw-edma: HDMA: Add memory barrier before starting the DMA transfer in remote setup

From: "Köry Maincent" <kory.maincent@bootlin.com>
To: Serge Semin <fancer.lancer@gmail.com>
Cc: Cai Huoqing <cai.huoqing@linux.dev>,
	Manivannan Sadhasivam <mani@kernel.org>,
	Vinod Koul <vkoul@kernel.org>,
	Gustavo Pimentel <Gustavo.Pimentel@synopsys.com>,
	dmaengine@vger.kernel.org, linux-kernel@vger.kernel.org,
	Thomas Petazzoni <thomas.petazzoni@bootlin.com>,
	Herve Codina <herve.codina@bootlin.com>
Subject: Re: [PATCH 4/9] dmaengine: dw-edma: HDMA: Add memory barrier before starting the DMA transfer in remote setup
Date: Wed, 21 Jun 2023 15:19:48 +0200	[thread overview]
Message-ID: <20230621151948.36125997@kmaincent-XPS-13-7390> (raw)
In-Reply-To: <qwkwtsjmfkmvsx4pmjetoxkjrpuwkndm6h6ntkpehxutz2h2jm@bmdzt7ywiuvs>

On Wed, 21 Jun 2023 12:45:35 +0300
Serge Semin <fancer.lancer@gmail.com> wrote:

> > I thought that using a read will solve the issue like the gpio_nand driver
> > (gpio_nand_dosync)   
> 
> AFAICS The io_sync dummy-read there is a workaround to fix the
> bus-reordering within the SoC bus. In this case we have a PCIe bus
> which is supposed to guarantee the strong order with the exception I
> described above or unless there is a bug someplace in the PCIe fabric.
> 
> > but I didn't thought of a cache that could return the value
> > of the read even if the write doesn't fully happen. In the case of a cache
> > how could we know that the write is done without using a delay?   
> 
> MMIO mapping is platform dependent and low-level driver dependent.
> That's why I asked many times about the platform you are using and the
> low-level driver that probes the eDMA engine. It would be also useful
> to know what PCIe host controller is utilized too.
> 
> Mainly MMIO spaces are mapped in a way to bypass the caching. But in
> some cases it might be useful to map an MMIO space with additional
> optimizations like Write-combining. For instance it could be
> effectively done for the eDMA linked-list BAR mapping. Indeed why
> would you need to send each linked-list byte/word/dword right away to
> the device while you can combine them and send all together, then
> flush the cache and only after that start the DMA transfer? Another
> possible reason of the writes reordering could be in a way the PCIe
> host outbound memory window (a memory region accesses to which are
> translated to the PCIe bus transfers) is configured. For instance DW
> PCIe Host controller outbound MW config CSR has a special flag which
> enables setting a custom PCIe bus TLPs (packets) attribute. As I
> mentioned above that attribute can affect the TLPs order: make it
> relaxed or ID-based.
> 
> Of course we can't reject a possibility of having some delays hidden
> inside your device which may cause writes to the internal memory
> landing after the writes to the CSRs. But that seems too exotic to be
> considered as the real one for sure until the alternatives are
> thoroughly checked.
> 
> What I was trying to say that your problem can be caused by some much
> more frequently met reason. If I were you I would have checked them
> first and only then considered a workaround like you suggest.

Thanks for you detailed answer, this was instructive.
I will come back with more information if TLP flags are set.
FYI the PCIe board I am currently working with is the one from Brainchip:
Here is the driver:
https://github.com/Brainchip-Inc/akida_dw_edma

Köry