From: Jakub Kicinski <kuba@kernel.org>
To: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: linux-arch@vger.kernel.org, BMC-SW <BMC-SW@aspeedtech.com>,
linux-aspeed <linux-aspeed@lists.ozlabs.org>,
Po-Yu Chuang <ratbert@faraday-tech.com>,
paulmck@kernel.org, netdev@vger.kernel.org,
OpenBMC Maillist <openbmc@lists.ozlabs.org>,
Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
Arnd Bergmann <arnd@arndb.de>,
Dylan Hung <dylan_hung@aspeedtech.com>,
"David S . Miller" <davem@davemloft.net>
Subject: Re: [PATCH] net: ftgmac100: Fix missing TX-poll issue
Date: Tue, 20 Oct 2020 10:24:15 -0700 [thread overview]
Message-ID: <20201020102415.52b51895@kicinski-fedora-pc1c0hjn.dhcp.thefacebook.com> (raw)
In-Reply-To: <3ebaa814fe21eb7b4b25a2c9455a34434e0207d6.camel@kernel.crashing.org>
On Tue, 20 Oct 2020 17:15:42 +1100 Benjamin Herrenschmidt wrote:
> On Mon, 2020-10-19 at 19:57 -0700, Jakub Kicinski wrote:
> > > I suspect the problem is that the HW (and yes this would be a HW bug)
> > > doesn't order the CPU -> memory and the CPU -> MMIO path.
> > >
> > > What I think happens is that the store to txde0 is potentially still in
> > > a buffer somewhere on its way to memory, gets bypassed by the store to
> > > MMIO, causing the MAC to try to read the descriptor, and getting the
> > > "old" data from memory.
> >
> > I see, but in general this sort of a problem should be resolved by
> > adding an appropriate memory barrier. And in fact such barrier should
> > (these days) be implied by a writel (I'm not 100% clear on why this
> > driver uses iowrite, and if it matters).
>
> No, a barrier won't solve this I think.
>
> This is a coherency problem at the fabric/interconnect level. I has to
> do with the way they implemented the DMA path from memory to the
> ethernet controller using a different "port" of the memory controller
> than the one used by the CPU, separately from the MMIO path, with no
> proper ordering between those busses. Old school design .... and
> broken.
>
> By doing a read back, they probably force the previous write to memory
> to get past the point where it will be visible to a subsequent DMA read
> by the ethernet controller.
Thanks for the explanation. How wonderful :/
It'd still be highly, highly preferable if the platform was conforming
to the Linux memory model. IO successors (iowrite32 / writel) must
ensure previous DRAM writes had completed. For performance sensitive
ops, which don't require ordering we have writel_relaxed etc.
I assume the DRAM controller queue is a straight FIFO and we don't have
to worry about hitting the same address, so how about we add a read
of some known uncached address in iowrite32 / writel?
next prev parent reply other threads:[~2020-10-20 17:26 UTC|newest]
Thread overview: 21+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-10-19 7:39 [PATCH] net: ftgmac100: Fix missing TX-poll issue Dylan Hung
2020-10-19 8:57 ` Joel Stanley
2020-10-19 9:19 ` Dylan Hung
2020-10-19 19:00 ` Jakub Kicinski
2020-10-19 23:23 ` Benjamin Herrenschmidt
2020-10-20 2:57 ` Jakub Kicinski
2020-10-20 6:15 ` Benjamin Herrenschmidt
2020-10-20 17:24 ` Jakub Kicinski [this message]
2020-10-20 6:14 ` Dylan Hung
2020-10-20 13:15 ` David Laight
2020-10-20 22:05 ` Benjamin Herrenschmidt
2020-10-20 19:49 ` Arnd Bergmann
2020-10-20 22:10 ` Benjamin Herrenschmidt
2020-10-20 22:25 ` Andrew Jeffery
2020-10-23 13:08 ` Dylan Hung
2020-10-26 22:21 ` Benjamin Herrenschmidt
2020-10-27 2:18 ` Joel Stanley
2020-10-21 7:16 ` Arnd Bergmann
2020-10-21 12:11 ` Arnd Bergmann
2020-10-22 7:40 ` Benjamin Herrenschmidt
2020-10-23 8:39 ` Arnd Bergmann
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20201020102415.52b51895@kicinski-fedora-pc1c0hjn.dhcp.thefacebook.com \
--to=kuba@kernel.org \
--cc=BMC-SW@aspeedtech.com \
--cc=arnd@arndb.de \
--cc=benh@kernel.crashing.org \
--cc=davem@davemloft.net \
--cc=dylan_hung@aspeedtech.com \
--cc=linux-arch@vger.kernel.org \
--cc=linux-aspeed@lists.ozlabs.org \
--cc=linux-kernel@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=openbmc@lists.ozlabs.org \
--cc=paulmck@kernel.org \
--cc=ratbert@faraday-tech.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).