From: Jeremy Linton <jeremy.linton@arm.com>
To: Florian Fainelli <f.fainelli@gmail.com>, netdev@vger.kernel.org
Cc: opendmb@gmail.com, davem@davemloft.net, kuba@kernel.org,
bcm-kernel-feedback-list@broadcom.com,
linux-kernel@vger.kernel.org,
Peter Robinson <pbrobinson@gmail.com>
Subject: Re: [PATCH] net: bcmgenet: Use stronger register read/writes to assure ordering
Date: Thu, 10 Mar 2022 19:09:12 -0600 [thread overview]
Message-ID: <de28c0ec-56a7-bfff-0c41-72aeef097ee3@arm.com> (raw)
In-Reply-To: <f831a4c6-58c9-20bd-94e8-e221369609e8@gmail.com>
On 3/10/22 12:59, Florian Fainelli wrote:
> On 3/9/22 8:53 PM, Jeremy Linton wrote:
>> GCC12 appears to be much smarter about its dependency tracking and is
>> aware that the relaxed variants are just normal loads and stores and
>> this is causing problems like:
>>
>> [ 210.074549] ------------[ cut here ]------------
>> [ 210.079223] NETDEV WATCHDOG: enabcm6e4ei0 (bcmgenet): transmit queue 1 timed out
>> [ 210.086717] WARNING: CPU: 1 PID: 0 at net/sched/sch_generic.c:529 dev_watchdog+0x234/0x240
>> [ 210.095044] Modules linked in: genet(E) nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat]
>> [ 210.146561] ACPI CPPC: PCC check channel failed for ss: 0. ret=-110
>> [ 210.146927] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G E 5.17.0-rc7G12+ #58
>> [ 210.153226] CPPC Cpufreq:cppc_scale_freq_workfn: failed to read perf counters
>> [ 210.161349] Hardware name: Raspberry Pi Foundation Raspberry Pi 4 Model B/Raspberry Pi 4 Model B, BIOS EDK2-DEV 02/08/2022
>> [ 210.161353] pstate: 80400005 (Nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
>> [ 210.161358] pc : dev_watchdog+0x234/0x240
>> [ 210.161364] lr : dev_watchdog+0x234/0x240
>> [ 210.161368] sp : ffff8000080a3a40
>> [ 210.161370] x29: ffff8000080a3a40 x28: ffffcd425af87000 x27: ffff8000080a3b20
>> [ 210.205150] x26: ffffcd425aa00000 x25: 0000000000000001 x24: ffffcd425af8ec08
>> [ 210.212321] x23: 0000000000000100 x22: ffffcd425af87000 x21: ffff55b142688000
>> [ 210.219491] x20: 0000000000000001 x19: ffff55b1426884c8 x18: ffffffffffffffff
>> [ 210.226661] x17: 64656d6974203120 x16: 0000000000000001 x15: 6d736e617274203a
>> [ 210.233831] x14: 2974656e65676d63 x13: ffffcd4259c300d8 x12: ffffcd425b07d5f0
>> [ 210.241001] x11: 00000000ffffffff x10: ffffcd425b07d5f0 x9 : ffffcd4258bdad9c
>> [ 210.248171] x8 : 00000000ffffdfff x7 : 000000000000003f x6 : 0000000000000000
>> [ 210.255341] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 0000000000001000
>> [ 210.262511] x2 : 0000000000001000 x1 : 0000000000000005 x0 : 0000000000000044
>> [ 210.269682] Call trace:
>> [ 210.272133] dev_watchdog+0x234/0x240
>> [ 210.275811] call_timer_fn+0x3c/0x15c
>> [ 210.279489] __run_timers.part.0+0x288/0x310
>> [ 210.283777] run_timer_softirq+0x48/0x80
>> [ 210.287716] __do_softirq+0x128/0x360
>> [ 210.291392] __irq_exit_rcu+0x138/0x140
>> [ 210.295243] irq_exit_rcu+0x1c/0x30
>> [ 210.298745] el1_interrupt+0x38/0x54
>> [ 210.302334] el1h_64_irq_handler+0x18/0x24
>> [ 210.306445] el1h_64_irq+0x7c/0x80
>> [ 210.309857] arch_cpu_idle+0x18/0x2c
>> [ 210.313445] default_idle_call+0x4c/0x140
>> [ 210.317470] cpuidle_idle_call+0x14c/0x1a0
>> [ 210.321584] do_idle+0xb0/0x100
>> [ 210.324737] cpu_startup_entry+0x30/0x8c
>> [ 210.328675] secondary_start_kernel+0xe4/0x110
>> [ 210.333138] __secondary_switched+0x94/0x98
>>
>> The assumption when these were relaxed seems to be that device memory
>> would be mapped non reordering, and that other constructs
>> (spinlocks/etc) would provide the barriers to assure that packet data
>> and in memory rings/queues were ordered with respect to device
>> register reads/writes. This itself seems a bit sketchy, but the real
>> problem with GCC12 is that it is moving the actual reads/writes around
>> at will as though they were independent operations when in truth they
>> are not, but the compiler can't know that. When looking at the
>> assembly dumps for many of these routines its possible to see very
>> clean, but not strictly in program order operations occurring as the
>> compiler would be free to do if these weren't actually register
>> reads/write operations.
>>
>> Its possible to suppress the timeout with a liberal bit of dma_mb()'s
>> sprinkled around but the device still seems unable to reliably
>> send/receive data. A better plan is to use the safer readl/writel
>> everywhere.
>>
>> Since this partially reverts an older commit, which notes the use of
>> the relaxed variants for performance reasons. I would suggest that
>> any performance problems with this commit are targeted at relaxing only
>> the performance critical code paths after assuring proper barriers.
>>
>> Fixes: 69d2ea9c79898 ("net: bcmgenet: Use correct I/O accessors")
>> Reported-by: Peter Robinson <pbrobinson@gmail.com>
>> Signed-off-by: Jeremy Linton <jeremy.linton@arm.com>
>
> I think this is the correct approach in that it favors correctness over
> speed, however there is an opportunity for maintaining the speed and
> correctness on non-2711 and non-7712 chips where the GENET core is
> interfaced to a system bus (GISB) that guarantees no re-ordering and no
> buffering. I suppose that until we prove that the extra barrier is
> harmful to performance on those chips, we should go with your patch.
>
> It seems like we missed the GENET_IO_MACRO() in bcmgenet.h, while most
> of them deal with the control path which likely does not have any
> re-ordering problem, there is an exception to that which are the
> intrl2_0 and intrl2_1 macros, which I believe *have* to be ordered as
> well in order to avoid spurious or missed interrupts, or maybe there is
> enough barriers in the interrupt processing code that this is moot?
Ok, so I spent some time and tracked down exactly which barrier "fixes"
this immediate problem on the rpi4.
static void bcmgenet_enable_dma(struct bcmgenet_priv *priv, u32 dma_ctrl)
{
u32 reg;
+ dma_mb(); //timeout fix
reg = bcmgenet_rdma_readl(priv, DMA_CTRL);
reg |= dma_ctrl;
fixes it as well, and keeps all the existing code. Although, granted I
didn't stress the adapter beyond a couple interactive ssh sessions. And
as you mention there are a fair number of other accessors that I didn't
touch which are still relaxed.
next prev parent reply other threads:[~2022-03-11 1:10 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-03-10 4:53 [PATCH] net: bcmgenet: Use stronger register read/writes to assure ordering Jeremy Linton
2022-03-10 17:29 ` Peter Robinson
2022-03-10 18:59 ` Florian Fainelli
2022-03-11 1:09 ` Jeremy Linton [this message]
2022-03-11 3:57 ` Florian Fainelli
2022-03-15 15:29 ` Peter Robinson
2022-03-18 19:01 ` Florian Fainelli
2022-03-18 19:20 ` Jakub Kicinski
2022-03-18 21:26 ` Florian Fainelli
2022-03-19 0:22 ` Jakub Kicinski
2022-03-18 19:04 ` Florian Fainelli
2022-03-21 21:26 ` Jakub Kicinski
2022-03-21 21:28 ` Florian Fainelli
2022-03-29 15:07 ` Peter Robinson
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=de28c0ec-56a7-bfff-0c41-72aeef097ee3@arm.com \
--to=jeremy.linton@arm.com \
--cc=bcm-kernel-feedback-list@broadcom.com \
--cc=davem@davemloft.net \
--cc=f.fainelli@gmail.com \
--cc=kuba@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=opendmb@gmail.com \
--cc=pbrobinson@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).