All of lore.kernel.org
 help / color / mirror / Atom feed
From: Frank Li <frank.li@nxp.com>
To: Will Deacon <will@kernel.org>, Catalin Marinas <catalin.marinas@arm.com>
Cc: Zhi Li <lznuaa@gmail.com>, Shenwei Wang <shenwei.wang@nxp.com>,
	Han Xu <han.xu@nxp.com>, Nitin Garg <nitin.garg@nxp.com>,
	Jason Liu <jason.hui.liu@nxp.com>,
	"linux-arm-kernel@lists.infradead.org"
	<linux-arm-kernel@lists.infradead.org>
Subject: RE: [EXT] Re: The problem about arm64: io: Relax implicit barriers in default I/O accessors
Date: Thu, 17 Jun 2021 20:11:50 +0000	[thread overview]
Message-ID: <AS8PR04MB850070F3667938EEFE6F67C5880E9@AS8PR04MB8500.eurprd04.prod.outlook.com> (raw)
In-Reply-To: <20210617174131.GC24813@willie-the-truck>



> -----Original Message-----
> From: Will Deacon <will@kernel.org>
> Sent: Thursday, June 17, 2021 12:42 PM
> To: Catalin Marinas <catalin.marinas@arm.com>
> Cc: Zhi Li <lznuaa@gmail.com>; Frank Li <frank.li@nxp.com>; Shenwei Wang
> <shenwei.wang@nxp.com>; Han Xu <han.xu@nxp.com>; Nitin Garg
> <nitin.garg@nxp.com>; Jason Liu <jason.hui.liu@nxp.com>; linux-arm-
> kernel@lists.infradead.org
> Subject: [EXT] Re: The problem about arm64: io: Relax implicit barriers in
> default I/O accessors
> 
> Caution: EXT Email
> 
> On Thu, Jun 17, 2021 at 06:25:28PM +0100, Will Deacon wrote:
> > On Thu, Jun 17, 2021 at 10:27:44AM +0100, Catalin Marinas wrote:
> > > On Wed, Jun 16, 2021 at 02:24:39PM -0500, Zhi Li wrote:
> > > > On Wed, Jun 16, 2021 at 2:18 PM Frank Li <frank.li@nxp.com> wrote:
> > > > > Will Deacon wrote:
> > > > > > It would also be helpful to know a bit more about the hardware:
> > > > > >
> > > > > >   - What is the "internal bus fabric"?
> > > >
> > > > > Look like ARM call as "Interconnect",  Multi AXI master and multi
> AXI slave
> > > > > connected together.
> > > >
> > > > I  drawed simplified bus structure.
> > > >
> > > >         ┌──────┐ ┌────┐
> > > >         │ A53  │ │A72 │
> > > >         └───┬──┘ └─┬──┘
> > > >             │      │
> > > >         ┌───▼──────▼──┐
> > > >         │    CCI400   │
> > > >         └─────┬───────┘
> > > >               │   1 (a)write to ddr (normal uncached memory)
> > > >               │   DMB OSHST
> > > >               │   2 (b)write to usb register(device, nGnRE)
> > > >         ┌─────▼───────────────────────┐       ┌
> ───────────┐
> > > >         │                             ◄───────┤   GPU     │
> > > >         │     Bus fabric              │       │           │
> > > >         └────────────────────────────┬┘       └
> ───────────┘
> > > > 3 (b) reach usb   ▲ 4 usb read   ▲   │ 6.(a)reach
> > > >          │        │   ddr        │   │
> > > >       ┌──▼────────┴─┐            │   │
> > > >       │             │            │   │
> > > >       │  USB        │      5.usb │   │
> > > >       │             │      read  │   │
> > > >       └─────────────┘            │   │
> > > >                                ┌─┴───▼─┐
> > > >                                │       │
> > > >                                │ DDR   │
> > > >                                │       │
> > > >                                └───────┘
> > >
> > > Since you sent an HTML message, it was rejected by the list server. The
> > > above is a plain-text rendition by w3m (and changed barrier() to DMB
> > > OSHST).
> > >
> > > Is the DMB propagated to the bus fabric? IIUC, our logic is that if the
> > > write (b) to USB is observable by, let's say, the GPU, the same GPU
> > > should also observe the write (a) to DDR. Since the write (a) to DDR is
> > > globally observable, the USB device read at (4) should also observe it
> > > (well, we may be wrong).
> >
> > It's pretty rare for barriers to propagate onto the fabric -- usually the
> > CPU just orders everything based on acknowledgements. If the CCI gives
> the
> > write response for the non-cacheable write I could see that causing an
> issue
> > if the bus fabric can then reorder accesses, but then I would argue
> that's a
> > broken system because simple ring buffers in non-cacheable memory would
> fail

Bus fabric don't reorder the same axi master. 
https://elinux.org/images/7/73/Deacon-weak-to-weedy.pdf
Page 42 show race condition. I think above race condition happen at our system.
I am not sure if it is exist at Armv8 system.

> > for peripherals hooking into the bus fabric (i.e. dma_*mb() would be
> > broken). I think it would also mean that DSB doesn't necessarily fix the
> > issue, it probably just makes it less likely because it takes longer to
> > get the device write out after the acknowledgement -- ndelay() would
> achieve
> > the same effect :)

That's what I worried. 

> >
> > Frank -- what happens if you try either DMB SY, or DMB OSH (without the
> ST)
> > in writel()?

It works well for 2 hours! Normally, problem happen below 10min. So I think DMB SY
can fix it. 

diff --git a/arch/arm64/include/asm/barrier.h b/arch/arm64/include/asm/barrier.h
index c3009b0e52393..277c9d1c1a8fa 100644
--- a/arch/arm64/include/asm/barrier.h
+++ b/arch/arm64/include/asm/barrier.h
@@ -47,7 +47,7 @@

 #define dma_mb()       dmb(osh)
 #define dma_rmb()      dmb(oshld)
-#define dma_wmb()      dmb(oshst)
+#define dma_wmb()      dmb(sy)

> 
> Also, digging into the A72 TRM there are a bunch of configuration signals
> in this area; see SYSBARDISABLE and BROADCASTOUTER, for example.
> 
> Does the failure happen on both a53 and a72, or only on one CPU type?

Both A53, A72 have this problem. 

> 
> Will
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

  reply	other threads:[~2021-06-17 20:13 UTC|newest]

Thread overview: 22+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <AS8PR04MB850004639EE6CE9432BBF13E880F9@AS8PR04MB8500.eurprd04.prod.outlook.com>
     [not found] ` <CAHrpEqRsp2_bt=p5JgS5F-2F_LCwgT+VX7mSENzpEYTQiW1tjg@mail.gmail.com>
2021-06-17  9:27   ` Catalin Marinas
2021-06-17 17:25     ` Will Deacon
2021-06-17 17:41       ` Will Deacon
2021-06-17 20:11         ` Frank Li [this message]
2021-06-17 21:40           ` [EXT] " Will Deacon
2021-06-17 22:13             ` Frank Li
2021-06-18 14:56             ` Nitin Garg
2021-06-21 16:11             ` Frank Li
2021-06-21 16:26               ` Will Deacon
2021-06-21 16:59                 ` Will Deacon
2021-06-21 17:56                   ` Frank Li
2021-06-21 18:13                     ` Will Deacon
2021-06-21 21:32                       ` Frank Li
2021-06-22  9:11                         ` Will Deacon
2021-06-23 15:48                           ` Frank Li
2021-07-06 17:11                             ` Will Deacon
2021-07-15 15:53                               ` Frank Li
2021-07-22 19:14                                 ` Frank Li
2021-08-09 13:50                                   ` Will Deacon
2021-08-09 14:46                                     ` Frank Li
2021-08-09 15:26                                       ` Will Deacon
2021-08-10 18:50                                         ` Frank Li

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=AS8PR04MB850070F3667938EEFE6F67C5880E9@AS8PR04MB8500.eurprd04.prod.outlook.com \
    --to=frank.li@nxp.com \
    --cc=catalin.marinas@arm.com \
    --cc=han.xu@nxp.com \
    --cc=jason.hui.liu@nxp.com \
    --cc=linux-arm-kernel@lists.infradead.org \
    --cc=lznuaa@gmail.com \
    --cc=nitin.garg@nxp.com \
    --cc=shenwei.wang@nxp.com \
    --cc=will@kernel.org \
    --subject='RE: [EXT] Re: The problem about arm64: io: Relax implicit barriers in default I/O accessors' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.