All of lore.kernel.org
 help / color / mirror / Atom feed
From: Catalin Marinas <catalin.marinas@arm.com>
To: Frank Li <frank.li@nxp.com>
Cc: Will Deacon <will@kernel.org>,
	Shenwei Wang <shenwei.wang@nxp.com>, Han Xu <han.xu@nxp.com>,
	Nitin Garg <nitin.garg@nxp.com>,
	Jason Liu <jason.hui.liu@nxp.com>,
	"linux-arm-kernel@lists.infradead.org"
	<linux-arm-kernel@lists.infradead.org>,
	 Zhi Li <lznuaa@gmail.com>
Subject: Re: The problem about arm64: io: Relax implicit barriers in default I/O accessors
Date: Wed, 16 Jun 2021 19:40:23 +0100	[thread overview]
Message-ID: <20210616184023.GG22433@arm.com> (raw)
In-Reply-To: <AS8PR04MB850017FD8F07864BB98C07D488319@AS8PR04MB8500.eurprd04.prod.outlook.com>

On Mon, Jun 14, 2021 at 10:41:38PM +0000, Frank Li wrote:
> commit 22ec71615d824f4f11d38d0e55a88d8956b7e45f
> Author: Will Deacon <will@kernel.org>
> Date:   Fri Jun 7 15:48:58 2019 +0100
> 
>     arm64: io: Relax implicit barriers in default I/O accessors
> 
>     The arm64 implementation of the default I/O accessors requires barrier
>     instructions to satisfy the memory ordering requirements documented in
>     memory-barriers.txt [1], which are largely derived from the behaviour of
>     I/O accesses on x86.
[...]
> 	If I added wmb() before xhci_ring_ep_doorbell, the problem gone.
> 	Writel include io_wmb, which map into dma_wmb(). 
> 	
> 	1. write ddr
> 	2. writel
> 		2a. io_wmb(),   dmb(oshst)
> 		2b, write usb register
> 	3. usb dma read ddr.
> 
> 	
> 	Internal bus fabric only guarantee the order for the same AXID.
> 	1 write ddr may be slow.  USB register get data before 1 because
> 	GPU occupy ddr now.  So USB DMA start read from ddr and get old
> 	dma descriptor data and find not ready yet, then missed door
> 	bell. 

That's a complex topic, Will should have a better answer. I'll try some
thought exercise below introducing a hypothetical second CPU.

From Will's commit above w.r.t. other-multi-copy atomicity:

      1. A write arriving at an endpoint shared between multiple CPUs is
         visible to all CPUs

      2. A write that is visible to all CPUs is also visible to all other
         observers in the shareability domain

So (1) would be the write to the USB device which is also an observer in
the system (of the DDR writes). (2) refers to the write to the DDR.

If we have CPU0 writing to DDR, followed by DMB and the write to the USB
device, a CPU1 observing the write to the USB device would also observe
the write to DDR (with a DMB between them). Since the USB device is an
observer and the system is multi-copy atomic, the USB should also
observe the CPU0 write to the DDR if CPU1 observed it.

CPU1 can only observe the write to the USB device via an access to that
USB device (e.g. a register read). Such access probably goes through
some serialisation point and the DMB on CPU0 ensures that the prior
write to DDR is visible. Now, a CPU1 read from the USB device cannot
affect the DMA access that the USB device started to the DDR, so we can
take it out of the equation. However, this means that the hardware
should ensure such ordering USB DMA ordering otherwise it wouldn't be
multi-copy atomic (or our understanding of it).

Either the hardware doesn't match the memory model or our reasoning is
incorrect (both are possible ;)).

I wonder whether we can look at this in a different way: the USB device
doing a "speculative" access to the DDR before the write to USB is
globally observable. There isn't a way to fix it in the USB device since
it does not observe the write to its register, so we are left with
having to guarantee the completion of the write to the DDR before
informing the USB about it.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

  parent reply	other threads:[~2021-06-16 18:42 UTC|newest]

Thread overview: 8+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-06-14 22:41 Frank Li
2021-06-16 16:27 ` Frank Li
2021-06-16 16:29   ` Frank Li
2021-06-16 18:40 ` Catalin Marinas [this message]
2021-06-16 18:55   ` Will Deacon
     [not found] <AS8PR04MB850004639EE6CE9432BBF13E880F9@AS8PR04MB8500.eurprd04.prod.outlook.com>
     [not found] ` <CAHrpEqRsp2_bt=p5JgS5F-2F_LCwgT+VX7mSENzpEYTQiW1tjg@mail.gmail.com>
2021-06-17  9:27   ` Catalin Marinas
2021-06-17 17:25     ` Will Deacon
2021-06-17 17:41       ` Will Deacon

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210616184023.GG22433@arm.com \
    --to=catalin.marinas@arm.com \
    --cc=frank.li@nxp.com \
    --cc=han.xu@nxp.com \
    --cc=jason.hui.liu@nxp.com \
    --cc=linux-arm-kernel@lists.infradead.org \
    --cc=lznuaa@gmail.com \
    --cc=nitin.garg@nxp.com \
    --cc=shenwei.wang@nxp.com \
    --cc=will@kernel.org \
    --subject='Re: The problem about arm64: io: Relax implicit barriers in default I/O accessors' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.