* [PATCH] net: sh_eth: fix mdio access in sh_eth_close() for some SoCs
@ 2019-05-15 5:36 Yoshihiro Shimoda
2019-05-15 15:41 ` Sergei Shtylyov
0 siblings, 1 reply; 5+ messages in thread
From: Yoshihiro Shimoda @ 2019-05-15 5:36 UTC (permalink / raw)
To: sergei.shtylyov, davem; +Cc: netdev, linux-renesas-soc, Yoshihiro Shimoda
The sh_eth_close() resets the MAC and then calls phy_stop()
so that mdio read access result is incorrect without any error
according to kernel trace like below:
ifconfig-216 [003] .n.. 109.133124: mdio_access: ee700000.ethernet-ffffffff read phy:0x01 reg:0x00 val:0xffff
To fix the issue, this patch adds a condition and set the RMII mode
regiseter in sh_eth_dev_exit() for some SoCs.
Note that when I have tried to move the sh_eth_dev_exit() calling
after phy_stop() on sh_eth_close(), but it gets worse.
Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
---
drivers/net/ethernet/renesas/sh_eth.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/drivers/net/ethernet/renesas/sh_eth.c b/drivers/net/ethernet/renesas/sh_eth.c
index e33af37..106ae90 100644
--- a/drivers/net/ethernet/renesas/sh_eth.c
+++ b/drivers/net/ethernet/renesas/sh_eth.c
@@ -1596,6 +1596,10 @@ static void sh_eth_dev_exit(struct net_device *ndev)
/* Set MAC address again */
update_mac_address(ndev);
+
+ /* Set the mode again if required */
+ if (mdp->cd->rmiimode)
+ sh_eth_write(ndev, 0x1, RMIIMODE);
}
static void sh_eth_rx_csum(struct sk_buff *skb)
--
2.7.4
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH] net: sh_eth: fix mdio access in sh_eth_close() for some SoCs
2019-05-15 5:36 [PATCH] net: sh_eth: fix mdio access in sh_eth_close() for some SoCs Yoshihiro Shimoda
@ 2019-05-15 15:41 ` Sergei Shtylyov
2019-05-20 7:42 ` Yoshihiro Shimoda
0 siblings, 1 reply; 5+ messages in thread
From: Sergei Shtylyov @ 2019-05-15 15:41 UTC (permalink / raw)
To: Yoshihiro Shimoda, davem; +Cc: netdev, linux-renesas-soc
Hello!
It's not "some SoCs", it's only R-Car gen2 and RZ/G1 SoCs.
On 05/15/2019 08:36 AM, Yoshihiro Shimoda wrote:
> The sh_eth_close() resets the MAC and then calls phy_stop()
> so that mdio read access result is incorrect without any error
> according to kernel trace like below:
>
> ifconfig-216 [003] .n.. 109.133124: mdio_access: ee700000.ethernet-ffffffff read phy:0x01 reg:0x00 val:0xffff
Not sure how RMII mode affects the MDIO transfers...
> To fix the issue, this patch adds a condition and set the RMII mode
> regiseter in sh_eth_dev_exit() for some SoCs.
>
> Note that when I have tried to move the sh_eth_dev_exit() calling
> after phy_stop() on sh_eth_close(), but it gets worse.
Ah, I was going to suggest changing the call order... what happens then?
> Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
> ---
> drivers/net/ethernet/renesas/sh_eth.c | 4 ++++
> 1 file changed, 4 insertions(+)
>
> diff --git a/drivers/net/ethernet/renesas/sh_eth.c b/drivers/net/ethernet/renesas/sh_eth.c
> index e33af37..106ae90 100644
> --- a/drivers/net/ethernet/renesas/sh_eth.c
> +++ b/drivers/net/ethernet/renesas/sh_eth.c
> @@ -1596,6 +1596,10 @@ static void sh_eth_dev_exit(struct net_device *ndev)
>
> /* Set MAC address again */
> update_mac_address(ndev);
> +
> + /* Set the mode again if required */
Should be "RMII mode", n ot just "Mode". We prolly need more detailed explanation...
> + if (mdp->cd->rmiimode)
> + sh_eth_write(ndev, 0x1, RMIIMODE);
> }
>
> static void sh_eth_rx_csum(struct sk_buff *skb)
MBR, Sergei
^ permalink raw reply [flat|nested] 5+ messages in thread
* RE: [PATCH] net: sh_eth: fix mdio access in sh_eth_close() for some SoCs
2019-05-15 15:41 ` Sergei Shtylyov
@ 2019-05-20 7:42 ` Yoshihiro Shimoda
2019-05-25 18:06 ` Sergei Shtylyov
0 siblings, 1 reply; 5+ messages in thread
From: Yoshihiro Shimoda @ 2019-05-20 7:42 UTC (permalink / raw)
To: Sergei Shtylyov, davem; +Cc: netdev, linux-renesas-soc
Hello Sergei-san,
Thank you for the reply! And, I'm sorry for the delayed response because I had days off in last week.
> From: Sergei Shtylyov, Sent: Thursday, May 16, 2019 12:41 AM
>
> Hello!
>
> It's not "some SoCs", it's only R-Car gen2 and RZ/G1 SoCs.
Do you prefer the subject "net: sh_eth: fix mdio access in sh_eth_close() for R-Car gen2 and RZ/G1 SoCs"
instead of "... some SoCs"?
> On 05/15/2019 08:36 AM, Yoshihiro Shimoda wrote:
>
> > The sh_eth_close() resets the MAC and then calls phy_stop()
> > so that mdio read access result is incorrect without any error
> > according to kernel trace like below:
> >
> > ifconfig-216 [003] .n.. 109.133124: mdio_access: ee700000.ethernet-ffffffff read phy:0x01 reg:0x00 val:0xffff
>
> Not sure how RMII mode affects the MDIO transfers...
I should have described what happen on the previous code.
According to the hardware manual, the RMII mode should be set to 1 before
operation the Ethernet MAC. The MDIO transferring is one of the Ethernet MAC operation.
So, I assumed this RMII mode affects the MDIO transfers.
By the way, I also should have described what happen on the previous code.
If I input the following command on non-NFS rootfs environment (e.g. USB memory rootfs),
sometimes, a lot of FCD interruption happened and then the following "rcu "error happened.
---
# ifconfig eth0 down# ifconfig eth0 up
[ 794.545823] Micrel KSZ8041RNLI ee700000.ethernet-ffffffff:01: attached PHY driver [Micrel KSZ8041RNLI] (mii_bus:phy_addr=ee700000.ethernet-ffffffff:01, irq=157)
# ifconfig eth0 up# ifconfig eth0 down
# ifconfig eth0 down# ifconfig eth0 up
[ 797.662138] Micrel KSZ8041RNLI ee700000.ethernet-ffffffff:01: attached PHY driver [Micrel KSZ8041RNLI] (mii_bus:phy_addr=ee700000.ethernet-ffffffff:01, irq=157)
[ 797.689158] sh-eth ee700000.ethernet eth0: Link is Up - 100Mbps/Full - flow control rx/tx
[ 797.702631] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
# [ 818.759628] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
[ 818.770827] rcu: 0-....: (2 GPs behind) idle=d12/0/0x3 softirq=388/388 fqs=1051
[ 818.783635] rcu: (detected by 1, t=2104 jiffies, g=377, q=18)
[ 818.794848] Sending NMI from CPU 1 to CPUs 0:
[ 818.804733] NMI backtrace for cpu 0
[ 818.804736] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.1.0-00002-g192664e-dirty #27
[ 818.804738] Hardware name: Generic R-Car Gen2 (Flattened Device Tree)
[ 818.804740] PC is at __do_softirq+0x7c/0x344
[ 818.804742] LR is at irq_exit+0x6c/0xcc
[ 818.804744] pc : [<c01021e4>] lr : [<c0126aa0>] psr: 60030113
[ 818.804746] sp : c0d01eb8 ip : 00200102 fp : c0d03080
[ 818.804748] r10: 00000001 r9 : 0000000a r8 : c0ca2940
[ 818.804750] r7 : 00000004 r6 : c0ca2934 r5 : 00000000 r4 : ffffe000
[ 818.804752] r3 : c0ca2940 r2 : 2aaa8000 r1 : 00000000 r0 : 2aaa8000
[ 818.804754] Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none
[ 818.804756] Control: 10c5387d Table: 6a5f006a DAC: 00000051
[ 818.804758] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.1.0-00002-g192664e-dirty #27
[ 818.804760] Hardware name: Generic R-Car Gen2 (Flattened Device Tree)
[ 818.804762] [<c011049c>] (unwind_backtrace) from [<c010bc0c>] (show_stack+0x18/0x1c)
[ 818.804764] [<c010bc0c>] (show_stack) from [<c07f7458>] (dump_stack+0x7c/0x9c)
[ 818.804766] [<c07f7458>] (dump_stack) from [<c07fd358>] (nmi_cpu_backtrace+0x98/0xb8)
[ 818.804768] [<c07fd358>] (nmi_cpu_backtrace) from [<c010e6bc>] (handle_IPI+0x2bc/0x33c)
[ 818.804770] [<c010e6bc>] (handle_IPI) from [<c03fa084>] (gic_handle_irq+0x8c/0x98)
[ 818.804773] [<c03fa084>] (gic_handle_irq) from [<c0101a8c>] (__irq_svc+0x6c/0x90)
[ 818.804774] Exception stack(0xc0d01e68 to 0xc0d01eb0)
[ 818.804777] 1e60: 2aaa8000 00000000 2aaa8000 c0ca2940 ffffe000 00000000
[ 818.804779] 1e80: c0ca2934 00000004 c0ca2940 0000000a 00000001 c0d03080 00200102 c0d01eb8
[ 818.804781] 1ea0: c0126aa0 c01021e4 60030113 ffffffff
[ 818.804783] [<c0101a8c>] (__irq_svc) from [<c01021e4>] (__do_softirq+0x7c/0x344)
[ 818.804785] [<c01021e4>] (__do_softirq) from [<c0126aa0>] (irq_exit+0x6c/0xcc)
[ 818.804787] [<c0126aa0>] (irq_exit) from [<c016c344>] (__handle_domain_irq+0x88/0xbc)
[ 818.804789] [<c016c344>] (__handle_domain_irq) from [<c03fa058>] (gic_handle_irq+0x60/0x98)
[ 818.804791] [<c03fa058>] (gic_handle_irq) from [<c0101a8c>] (__irq_svc+0x6c/0x90)
[ 818.804793] Exception stack(0xc0d01f38 to 0xc0d01f80)
[ 818.804795] 1f20: 00000000 0008bbb8
[ 818.804798] 1f40: eb74b520 c011aee0 00000001 ffffe000 c0d08c30 c0d8e5be c0d08c74 c0d980c0
[ 818.804800] 1f60: 00000001 c0c5ea40 00000000 c0d01f88 c01084ec c014c85c 60030013 ffffffff
[ 818.804802] [<c0101a8c>] (__irq_svc) from [<c014c85c>] (do_idle+0xf0/0x13c)
[ 818.804804] [<c014c85c>] (do_idle) from [<c014cb10>] (cpu_startup_entry+0x20/0x24)
[ 818.804806] [<c014cb10>] (cpu_startup_entry) from [<c0c00fac>] (start_kernel+0x3e0/0x488)
Note:
- After I input the second ifconfig eth0 up, I cannot input any command.
- If I disconnect an ethernet cable, I can input command again.
---
> > To fix the issue, this patch adds a condition and set the RMII mode
> > regiseter in sh_eth_dev_exit() for some SoCs.
> >
> > Note that when I have tried to move the sh_eth_dev_exit() calling
> > after phy_stop() on sh_eth_close(), but it gets worse.
>
> Ah, I was going to suggest changing the call order... what happens then?
Sometimes, the following error happened and then I cannot recover the system
even if I disconnected an ethernet cable. Do you prefer this way, instead of
adding RMII mode setting on sh_eth_dev_exit()?
---
# ifconfig eth0 up;
[ 64.896021] Micrel KSZ8041RNLI ee700000.ethernet-ffffffff:01: attached PHY driver [Micrel KSZ8041RNLI] (mii_bus:phy_addr=ee700000.ethernet-ffffffff:01, irq=157)
# [ 66.868241] sh-eth ee700000.ethernet eth0: Link is Up - 100Mbps/Full - flow control rx/tx
[ 66.881777] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
# ifconfig eth0 down
[ 71.164900] sh-eth ee700000.ethernet eth0: Link is Down
[ 92.169767] rcu: INFO: rcu_sched self-detected stall on CPU
[ 92.180654] rcu: 0-....: (2100 ticks this GP) idle=9d6/1/0x40000004 softirq=513/513 fqs=1050
[ 92.194720] rcu: (t=2102 jiffies g=-367 q=46)
[ 92.204723] NMI backtrace for cpu 0
[ 92.213705] CPU: 0 PID: 228 Comm: ifconfig Not tainted 5.1.0-00002-g192664e-dirty #28
[ 92.227149] Hardware name: Generic R-Car Gen2 (Flattened Device Tree)
[ 92.239258] [<c011049c>] (unwind_backtrace) from [<c010bc0c>] (show_stack+0x18/0x1c)
[ 92.252683] [<c010bc0c>] (show_stack) from [<c07f7458>] (dump_stack+0x7c/0x9c)
[ 92.265548] [<c07f7458>] (dump_stack) from [<c07fd360>] (nmi_cpu_backtrace+0xa0/0xb8)
[ 92.279039] [<c07fd360>] (nmi_cpu_backtrace) from [<c07fd3fc>] (nmi_trigger_cpumask_backtrace+0x84/0x114)
[ 92.294384] [<c07fd3fc>] (nmi_trigger_cpumask_backtrace) from [<c01827e4>] (rcu_dump_cpu_stacks+0xb4/0xd0)
[ 92.309893] [<c01827e4>] (rcu_dump_cpu_stacks) from [<c0180cc8>] (rcu_sched_clock_irq+0x2cc/0x6f0)
[ 92.324744] [<c0180cc8>] (rcu_sched_clock_irq) from [<c0189170>] (update_process_times+0x38/0x64)
[ 92.339528] [<c0189170>] (update_process_times) from [<c019a5d0>] (tick_nohz_handler+0xd4/0x128)
[ 92.354261] [<c019a5d0>] (tick_nohz_handler) from [<c06573a4>] (arch_timer_handler_virt+0x30/0x38)
[ 92.369207] [<c06573a4>] (arch_timer_handler_virt) from [<c0171548>] (handle_percpu_devid_irq+0xf0/0x224)
[ 92.384817] [<c0171548>] (handle_percpu_devid_irq) from [<c016c2ac>] (generic_handle_irq+0x20/0x30)
[ 92.399910] [<c016c2ac>] (generic_handle_irq) from [<c016c364>] (__handle_domain_irq+0xa8/0xbc)
[ 92.414678] [<c016c364>] (__handle_domain_irq) from [<c03fa058>] (gic_handle_irq+0x60/0x98)
[ 92.429128] [<c03fa058>] (gic_handle_irq) from [<c0101a8c>] (__irq_svc+0x6c/0x90)
[ 92.442748] Exception stack(0xea1f9b50 to 0xea1f9b98)
[ 92.453924] 9b40: 2aaa8000 00000000 2aaa8000 c0ca2940
[ 92.468345] 9b60: ffffe000 00000000 c0ca2934 00000002 c0ca2940 0000000a 00000000 c0d03080
[ 92.482764] 9b80: 00404100 ea1f9ba0 c0126aa0 c01021e4 60030113 ffffffff
[ 92.495519] [<c0101a8c>] (__irq_svc) from [<c01021e4>] (__do_softirq+0x7c/0x344)
[ 92.508973] [<c01021e4>] (__do_softirq) from [<c0126aa0>] (irq_exit+0x6c/0xcc)
[ 92.522215] [<c0126aa0>] (irq_exit) from [<c016c344>] (__handle_domain_irq+0x88/0xbc)
[ 92.536092] [<c016c344>] (__handle_domain_irq) from [<c03fa058>] (gic_handle_irq+0x60/0x98)
[ 92.550513] [<c03fa058>] (gic_handle_irq) from [<c0101a8c>] (__irq_svc+0x6c/0x90)
[ 92.564095] Exception stack(0xea1f9c20 to 0xea1f9c68)
[ 92.575129] 9c20: f0911320 00000001 00000000 f0911320 00000003 eabb7844 00000001 00000000
[ 92.589271] 9c40: 00000000 00003800 00000000 ea1f9e50 ea1f9d24 ea1f9c70 c0555d80 c055c590
[ 92.603319] 9c60: 60030013 ffffffff
[ 92.612529] [<c0101a8c>] (__irq_svc) from [<c055c590>] (sh_mdio_ctrl+0x3c/0x58)
[ 92.625531] [<c055c590>] (sh_mdio_ctrl) from [<c0555d80>] (mdiobb_send_num+0x28/0x34)
[ 92.638971] [<c0555d80>] (mdiobb_send_num) from [<c0555f10>] (mdiobb_write+0x70/0x94)
[ 92.652337] [<c0555f10>] (mdiobb_write) from [<c05555a4>] (__mdiobus_write+0x64/0x10c)
[ 92.665688] [<c05555a4>] (__mdiobus_write) from [<c05526e8>] (__phy_modify_changed+0x58/0x6c)
[ 92.679587] [<c05526e8>] (__phy_modify_changed) from [<c0552760>] (__phy_modify+0x10/0x18)
[ 92.693127] [<c0552760>] (__phy_modify) from [<c05527a4>] (phy_modify+0x3c/0x54)
[ 92.705749] [<c05527a4>] (phy_modify) from [<c0553b14>] (phy_suspend+0xb0/0xd0)
[ 92.718274] [<c0553b14>] (phy_suspend) from [<c0551118>] (phy_state_machine+0x1a4/0x1a8)
[ 92.731589] [<c0551118>] (phy_state_machine) from [<c05511a8>] (phy_stop+0x8c/0xb8)
[ 92.744411] [<c05511a8>] (phy_stop) from [<c055e2d4>] (sh_eth_close+0x58/0xa0)
[ 92.756668] [<c055e2d4>] (sh_eth_close) from [<c06ce740>] (__dev_close_many+0xc0/0xdc)
[ 92.769620] [<c06ce740>] (__dev_close_many) from [<c06d4f9c>] (__dev_change_flags+0xdc/0x1b0)
[ 92.783186] [<c06d4f9c>] (__dev_change_flags) from [<c06d5090>] (dev_change_flags+0x20/0x50)
[ 92.796660] [<c06d5090>] (dev_change_flags) from [<c074d740>] (devinet_ioctl+0x2d8/0x5dc)
[ 92.809892] [<c074d740>] (devinet_ioctl) from [<c074f8f8>] (inet_ioctl+0x260/0x324)
[ 92.822560] [<c074f8f8>] (inet_ioctl) from [<c06b2ea4>] (sock_ioctl+0x148/0x424)
[ 92.834968] [<c06b2ea4>] (sock_ioctl) from [<c0252e00>] (vfs_ioctl+0x28/0x3c)
[ 92.847110] [<c0252e00>] (vfs_ioctl) from [<c0253614>] (do_vfs_ioctl+0x98/0x844)
[ 92.859518] [<c0253614>] (do_vfs_ioctl) from [<c0253e00>] (ksys_ioctl+0x40/0x5c)
[ 92.871875] [<c0253e00>] (ksys_ioctl) from [<c0101000>] (ret_fast_syscall+0x0/0x54)
[ 92.884498] Exception stack(0xea1f9fa8 to 0xea1f9ff0)
[ 92.894477] 9fa0: 000ab853 bec4fc90 00000003 00008914 bec4fc90 000ab853
[ 92.907648] 9fc0: 000ab853 bec4fc90 bec4ff52 00000036 bec4fe48 000000b8 00096710 00000000
[ 92.920805] 9fe0: 000ca1e4 bec4fc2c 0001aa8c b6e72f3c
---
> > Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
> > ---
> > drivers/net/ethernet/renesas/sh_eth.c | 4 ++++
> > 1 file changed, 4 insertions(+)
> >
> > diff --git a/drivers/net/ethernet/renesas/sh_eth.c b/drivers/net/ethernet/renesas/sh_eth.c
> > index e33af37..106ae90 100644
> > --- a/drivers/net/ethernet/renesas/sh_eth.c
> > +++ b/drivers/net/ethernet/renesas/sh_eth.c
> > @@ -1596,6 +1596,10 @@ static void sh_eth_dev_exit(struct net_device *ndev)
> >
> > /* Set MAC address again */
> > update_mac_address(ndev);
> > +
> > + /* Set the mode again if required */
>
> Should be "RMII mode", n ot just "Mode". We prolly need more detailed explanation...
I got it.
Best regards,
Yoshihiro Shimoda
> > + if (mdp->cd->rmiimode)
> > + sh_eth_write(ndev, 0x1, RMIIMODE);
> > }
> >
> > static void sh_eth_rx_csum(struct sk_buff *skb)
>
> MBR, Sergei
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] net: sh_eth: fix mdio access in sh_eth_close() for some SoCs
2019-05-20 7:42 ` Yoshihiro Shimoda
@ 2019-05-25 18:06 ` Sergei Shtylyov
2019-05-28 2:33 ` Yoshihiro Shimoda
0 siblings, 1 reply; 5+ messages in thread
From: Sergei Shtylyov @ 2019-05-25 18:06 UTC (permalink / raw)
To: Yoshihiro Shimoda, davem; +Cc: netdev, linux-renesas-soc
Hello!
On 05/20/2019 10:42 AM, Yoshihiro Shimoda wrote:
> Thank you for the reply! And, I'm sorry for the delayed response because I had days off in last week.
And sorry for my delayed reply as well!
>> It's not "some SoCs", it's only R-Car gen2 and RZ/G1 SoCs.
>
> Do you prefer the subject "net: sh_eth: fix mdio access in sh_eth_close() for R-Car gen2 and RZ/G1 SoCs"
> instead of "... some SoCs"?
Yes, it's just about a single SoC family as the driver knows them, really: R-Car gen2;
RZ/G1 uses the same SoC data.
>>> The sh_eth_close() resets the MAC and then calls phy_stop()
>>> so that mdio read access result is incorrect without any error
>>> according to kernel trace like below:
>>>
>>> ifconfig-216 [003] .n.. 109.133124: mdio_access: ee700000.ethernet-ffffffff read phy:0x01 reg:0x00 val:0xffff
>>
>> Not sure how RMII mode affects the MDIO transfers...
>
> I should have described what happen on the previous code.
> According to the hardware manual, the RMII mode should be set to 1 before
> operation the Ethernet MAC. The MDIO transferring is one of the Ethernet MAC operation.
> So, I assumed this RMII mode affects the MDIO transfers.
>
> By the way, I also should have described what happen on the previous code.
> If I input the following command on non-NFS rootfs environment (e.g. USB memory rootfs),
> sometimes, a lot of FCD interruption happened and then the following "rcu "error happened.
> ---
> # ifconfig eth0 down# ifconfig eth0 up
> [ 794.545823] Micrel KSZ8041RNLI ee700000.ethernet-ffffffff:01: attached PHY driver [Micrel KSZ8041RNLI] (mii_bus:phy_addr=ee700000.ethernet-ffffffff:01, irq=157)
> # ifconfig eth0 up# ifconfig eth0 down
> # ifconfig eth0 down# ifconfig eth0 up
> [ 797.662138] Micrel KSZ8041RNLI ee700000.ethernet-ffffffff:01: attached PHY driver [Micrel KSZ8041RNLI] (mii_bus:phy_addr=ee700000.ethernet-ffffffff:01, irq=157)
> [ 797.689158] sh-eth ee700000.ethernet eth0: Link is Up - 100Mbps/Full - flow control rx/tx
> [ 797.702631] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
> # [ 818.759628] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
> [ 818.770827] rcu: 0-....: (2 GPs behind) idle=d12/0/0x3 softirq=388/388 fqs=1051
> [ 818.783635] rcu: (detected by 1, t=2104 jiffies, g=377, q=18)
> [ 818.794848] Sending NMI from CPU 1 to CPUs 0:
> [ 818.804733] NMI backtrace for cpu 0
> [ 818.804736] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.1.0-00002-g192664e-dirty #27
> [ 818.804738] Hardware name: Generic R-Car Gen2 (Flattened Device Tree)
> [ 818.804740] PC is at __do_softirq+0x7c/0x344
> [ 818.804742] LR is at irq_exit+0x6c/0xcc
> [ 818.804744] pc : [<c01021e4>] lr : [<c0126aa0>] psr: 60030113
> [ 818.804746] sp : c0d01eb8 ip : 00200102 fp : c0d03080
> [ 818.804748] r10: 00000001 r9 : 0000000a r8 : c0ca2940
> [ 818.804750] r7 : 00000004 r6 : c0ca2934 r5 : 00000000 r4 : ffffe000
> [ 818.804752] r3 : c0ca2940 r2 : 2aaa8000 r1 : 00000000 r0 : 2aaa8000
> [ 818.804754] Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none
> [ 818.804756] Control: 10c5387d Table: 6a5f006a DAC: 00000051
> [ 818.804758] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.1.0-00002-g192664e-dirty #27
> [ 818.804760] Hardware name: Generic R-Car Gen2 (Flattened Device Tree)
> [ 818.804762] [<c011049c>] (unwind_backtrace) from [<c010bc0c>] (show_stack+0x18/0x1c)
> [ 818.804764] [<c010bc0c>] (show_stack) from [<c07f7458>] (dump_stack+0x7c/0x9c)
> [ 818.804766] [<c07f7458>] (dump_stack) from [<c07fd358>] (nmi_cpu_backtrace+0x98/0xb8)
> [ 818.804768] [<c07fd358>] (nmi_cpu_backtrace) from [<c010e6bc>] (handle_IPI+0x2bc/0x33c)
> [ 818.804770] [<c010e6bc>] (handle_IPI) from [<c03fa084>] (gic_handle_irq+0x8c/0x98)
> [ 818.804773] [<c03fa084>] (gic_handle_irq) from [<c0101a8c>] (__irq_svc+0x6c/0x90)
> [ 818.804774] Exception stack(0xc0d01e68 to 0xc0d01eb0)
> [ 818.804777] 1e60: 2aaa8000 00000000 2aaa8000 c0ca2940 ffffe000 00000000
> [ 818.804779] 1e80: c0ca2934 00000004 c0ca2940 0000000a 00000001 c0d03080 00200102 c0d01eb8
> [ 818.804781] 1ea0: c0126aa0 c01021e4 60030113 ffffffff
> [ 818.804783] [<c0101a8c>] (__irq_svc) from [<c01021e4>] (__do_softirq+0x7c/0x344)
> [ 818.804785] [<c01021e4>] (__do_softirq) from [<c0126aa0>] (irq_exit+0x6c/0xcc)
> [ 818.804787] [<c0126aa0>] (irq_exit) from [<c016c344>] (__handle_domain_irq+0x88/0xbc)
> [ 818.804789] [<c016c344>] (__handle_domain_irq) from [<c03fa058>] (gic_handle_irq+0x60/0x98)
> [ 818.804791] [<c03fa058>] (gic_handle_irq) from [<c0101a8c>] (__irq_svc+0x6c/0x90)
> [ 818.804793] Exception stack(0xc0d01f38 to 0xc0d01f80)
> [ 818.804795] 1f20: 00000000 0008bbb8
> [ 818.804798] 1f40: eb74b520 c011aee0 00000001 ffffe000 c0d08c30 c0d8e5be c0d08c74 c0d980c0
> [ 818.804800] 1f60: 00000001 c0c5ea40 00000000 c0d01f88 c01084ec c014c85c 60030013 ffffffff
> [ 818.804802] [<c0101a8c>] (__irq_svc) from [<c014c85c>] (do_idle+0xf0/0x13c)
> [ 818.804804] [<c014c85c>] (do_idle) from [<c014cb10>] (cpu_startup_entry+0x20/0x24)
> [ 818.804806] [<c014cb10>] (cpu_startup_entry) from [<c0c00fac>] (start_kernel+0x3e0/0x488)
This *usually* means a register is accessed while the clock is off...
> Note:
> - After I input the second ifconfig eth0 up, I cannot input any command.
> - If I disconnect an ethernet cable, I can input command again.
Hm...
> ---
>
>>> To fix the issue, this patch adds a condition and set the RMII mode
>>> regiseter in sh_eth_dev_exit() for some SoCs.
Register.
>>> Note that when I have tried to move the sh_eth_dev_exit() calling
>>> after phy_stop() on sh_eth_close(), but it gets worse.
>>
>> Ah, I was going to suggest changing the call order... what happens then?
>
> Sometimes, the following error happened and then I cannot recover the system
> even if I disconnected an ethernet cable. Do you prefer this way, instead of
> adding RMII mode setting on sh_eth_dev_exit()?
> ---
> # ifconfig eth0 up;
> [ 64.896021] Micrel KSZ8041RNLI ee700000.ethernet-ffffffff:01: attached PHY driver [Micrel KSZ8041RNLI] (mii_bus:phy_addr=ee700000.ethernet-ffffffff:01, irq=157)
> # [ 66.868241] sh-eth ee700000.ethernet eth0: Link is Up - 100Mbps/Full - flow control rx/tx
> [ 66.881777] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
> # ifconfig eth0 down
> [ 71.164900] sh-eth ee700000.ethernet eth0: Link is Down
> [ 92.169767] rcu: INFO: rcu_sched self-detected stall on CPU
> [ 92.180654] rcu: 0-....: (2100 ticks this GP) idle=9d6/1/0x40000004 softirq=513/513 fqs=1050
> [ 92.194720] rcu: (t=2102 jiffies g=-367 q=46)
> [ 92.204723] NMI backtrace for cpu 0
> [ 92.213705] CPU: 0 PID: 228 Comm: ifconfig Not tainted 5.1.0-00002-g192664e-dirty #28
> [ 92.227149] Hardware name: Generic R-Car Gen2 (Flattened Device Tree)
> [ 92.239258] [<c011049c>] (unwind_backtrace) from [<c010bc0c>] (show_stack+0x18/0x1c)
> [ 92.252683] [<c010bc0c>] (show_stack) from [<c07f7458>] (dump_stack+0x7c/0x9c)
> [ 92.265548] [<c07f7458>] (dump_stack) from [<c07fd360>] (nmi_cpu_backtrace+0xa0/0xb8)
> [ 92.279039] [<c07fd360>] (nmi_cpu_backtrace) from [<c07fd3fc>] (nmi_trigger_cpumask_backtrace+0x84/0x114)
> [ 92.294384] [<c07fd3fc>] (nmi_trigger_cpumask_backtrace) from [<c01827e4>] (rcu_dump_cpu_stacks+0xb4/0xd0)
> [ 92.309893] [<c01827e4>] (rcu_dump_cpu_stacks) from [<c0180cc8>] (rcu_sched_clock_irq+0x2cc/0x6f0)
> [ 92.324744] [<c0180cc8>] (rcu_sched_clock_irq) from [<c0189170>] (update_process_times+0x38/0x64)
> [ 92.339528] [<c0189170>] (update_process_times) from [<c019a5d0>] (tick_nohz_handler+0xd4/0x128)
> [ 92.354261] [<c019a5d0>] (tick_nohz_handler) from [<c06573a4>] (arch_timer_handler_virt+0x30/0x38)
> [ 92.369207] [<c06573a4>] (arch_timer_handler_virt) from [<c0171548>] (handle_percpu_devid_irq+0xf0/0x224)
> [ 92.384817] [<c0171548>] (handle_percpu_devid_irq) from [<c016c2ac>] (generic_handle_irq+0x20/0x30)
> [ 92.399910] [<c016c2ac>] (generic_handle_irq) from [<c016c364>] (__handle_domain_irq+0xa8/0xbc)
> [ 92.414678] [<c016c364>] (__handle_domain_irq) from [<c03fa058>] (gic_handle_irq+0x60/0x98)
> [ 92.429128] [<c03fa058>] (gic_handle_irq) from [<c0101a8c>] (__irq_svc+0x6c/0x90)
> [ 92.442748] Exception stack(0xea1f9b50 to 0xea1f9b98)
> [ 92.453924] 9b40: 2aaa8000 00000000 2aaa8000 c0ca2940
> [ 92.468345] 9b60: ffffe000 00000000 c0ca2934 00000002 c0ca2940 0000000a 00000000 c0d03080
> [ 92.482764] 9b80: 00404100 ea1f9ba0 c0126aa0 c01021e4 60030113 ffffffff
> [ 92.495519] [<c0101a8c>] (__irq_svc) from [<c01021e4>] (__do_softirq+0x7c/0x344)
> [ 92.508973] [<c01021e4>] (__do_softirq) from [<c0126aa0>] (irq_exit+0x6c/0xcc)
> [ 92.522215] [<c0126aa0>] (irq_exit) from [<c016c344>] (__handle_domain_irq+0x88/0xbc)
> [ 92.536092] [<c016c344>] (__handle_domain_irq) from [<c03fa058>] (gic_handle_irq+0x60/0x98)
> [ 92.550513] [<c03fa058>] (gic_handle_irq) from [<c0101a8c>] (__irq_svc+0x6c/0x90)
> [ 92.564095] Exception stack(0xea1f9c20 to 0xea1f9c68)
> [ 92.575129] 9c20: f0911320 00000001 00000000 f0911320 00000003 eabb7844 00000001 00000000
> [ 92.589271] 9c40: 00000000 00003800 00000000 ea1f9e50 ea1f9d24 ea1f9c70 c0555d80 c055c590
> [ 92.603319] 9c60: 60030013 ffffffff
> [ 92.612529] [<c0101a8c>] (__irq_svc) from [<c055c590>] (sh_mdio_ctrl+0x3c/0x58)
> [ 92.625531] [<c055c590>] (sh_mdio_ctrl) from [<c0555d80>] (mdiobb_send_num+0x28/0x34)
> [ 92.638971] [<c0555d80>] (mdiobb_send_num) from [<c0555f10>] (mdiobb_write+0x70/0x94)
> [ 92.652337] [<c0555f10>] (mdiobb_write) from [<c05555a4>] (__mdiobus_write+0x64/0x10c)
> [ 92.665688] [<c05555a4>] (__mdiobus_write) from [<c05526e8>] (__phy_modify_changed+0x58/0x6c)
> [ 92.679587] [<c05526e8>] (__phy_modify_changed) from [<c0552760>] (__phy_modify+0x10/0x18)
> [ 92.693127] [<c0552760>] (__phy_modify) from [<c05527a4>] (phy_modify+0x3c/0x54)
> [ 92.705749] [<c05527a4>] (phy_modify) from [<c0553b14>] (phy_suspend+0xb0/0xd0)
> [ 92.718274] [<c0553b14>] (phy_suspend) from [<c0551118>] (phy_state_machine+0x1a4/0x1a8)
> [ 92.731589] [<c0551118>] (phy_state_machine) from [<c05511a8>] (phy_stop+0x8c/0xb8)
> [ 92.744411] [<c05511a8>] (phy_stop) from [<c055e2d4>] (sh_eth_close+0x58/0xa0)
> [ 92.756668] [<c055e2d4>] (sh_eth_close) from [<c06ce740>] (__dev_close_many+0xc0/0xdc)
> [ 92.769620] [<c06ce740>] (__dev_close_many) from [<c06d4f9c>] (__dev_change_flags+0xdc/0x1b0)
> [ 92.783186] [<c06d4f9c>] (__dev_change_flags) from [<c06d5090>] (dev_change_flags+0x20/0x50)
> [ 92.796660] [<c06d5090>] (dev_change_flags) from [<c074d740>] (devinet_ioctl+0x2d8/0x5dc)
> [ 92.809892] [<c074d740>] (devinet_ioctl) from [<c074f8f8>] (inet_ioctl+0x260/0x324)
> [ 92.822560] [<c074f8f8>] (inet_ioctl) from [<c06b2ea4>] (sock_ioctl+0x148/0x424)
> [ 92.834968] [<c06b2ea4>] (sock_ioctl) from [<c0252e00>] (vfs_ioctl+0x28/0x3c)
> [ 92.847110] [<c0252e00>] (vfs_ioctl) from [<c0253614>] (do_vfs_ioctl+0x98/0x844)
> [ 92.859518] [<c0253614>] (do_vfs_ioctl) from [<c0253e00>] (ksys_ioctl+0x40/0x5c)
> [ 92.871875] [<c0253e00>] (ksys_ioctl) from [<c0101000>] (ret_fast_syscall+0x0/0x54)
> [ 92.884498] Exception stack(0xea1f9fa8 to 0xea1f9ff0)
> [ 92.894477] 9fa0: 000ab853 bec4fc90 00000003 00008914 bec4fc90 000ab853
> [ 92.907648] 9fc0: 000ab853 bec4fc90 bec4ff52 00000036 bec4fe48 000000b8 00096710 00000000
> [ 92.920805] 9fe0: 000ca1e4 bec4fc2c 0001aa8c b6e72f3c
> ---
Again, usually means register access while the module clock is off...
>>> Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
>>> ---
>>> drivers/net/ethernet/renesas/sh_eth.c | 4 ++++
>>> 1 file changed, 4 insertions(+)
>>>
>>> diff --git a/drivers/net/ethernet/renesas/sh_eth.c b/drivers/net/ethernet/renesas/sh_eth.c
>>> index e33af37..106ae90 100644
>>> --- a/drivers/net/ethernet/renesas/sh_eth.c
>>> +++ b/drivers/net/ethernet/renesas/sh_eth.c
>>> @@ -1596,6 +1596,10 @@ static void sh_eth_dev_exit(struct net_device *ndev)
>>>
>>> /* Set MAC address again */
>>> update_mac_address(ndev);
>>> +
>>> + /* Set the mode again if required */
>>
>> Should be "RMII mode", n ot just "Mode". We prolly need more detailed explanation...
>
> I got it.
Your fix seems legit now, just misplaced a bit...
> Best regards,
> Yoshihiro Shimoda
>
>>> + if (mdp->cd->rmiimode)
>>> + sh_eth_write(ndev, 0x1, RMIIMODE);
According to the gen2 manual, this register should be written right after the soft reset
via EDMR.
>>> }
>>>
>>> static void sh_eth_rx_csum(struct sk_buff *skb)
MBR, Sergei
^ permalink raw reply [flat|nested] 5+ messages in thread
* RE: [PATCH] net: sh_eth: fix mdio access in sh_eth_close() for some SoCs
2019-05-25 18:06 ` Sergei Shtylyov
@ 2019-05-28 2:33 ` Yoshihiro Shimoda
0 siblings, 0 replies; 5+ messages in thread
From: Yoshihiro Shimoda @ 2019-05-28 2:33 UTC (permalink / raw)
To: Sergei Shtylyov; +Cc: netdev, linux-renesas-soc, davem
Hi Sergei-san,
> From: Sergei Shtylyov, Sent: Sunday, May 26, 2019 3:06 AM
>
> Hello!
>
> On 05/20/2019 10:42 AM, Yoshihiro Shimoda wrote:
>
> > Thank you for the reply! And, I'm sorry for the delayed response because I had days off in last week.
>
> And sorry for my delayed reply as well!
>
> >> It's not "some SoCs", it's only R-Car gen2 and RZ/G1 SoCs.
> >
> > Do you prefer the subject "net: sh_eth: fix mdio access in sh_eth_close() for R-Car gen2 and RZ/G1 SoCs"
> > instead of "... some SoCs"?
>
> Yes, it's just about a single SoC family as the driver knows them, really: R-Car gen2;
> RZ/G1 uses the same SoC data.
I got it.
> >>> The sh_eth_close() resets the MAC and then calls phy_stop()
> >>> so that mdio read access result is incorrect without any error
> >>> according to kernel trace like below:
> >>>
> >>> ifconfig-216 [003] .n.. 109.133124: mdio_access: ee700000.ethernet-ffffffff read phy:0x01 reg:0x00 val:0xffff
> >>
> >> Not sure how RMII mode affects the MDIO transfers...
> >
> > I should have described what happen on the previous code.
> > According to the hardware manual, the RMII mode should be set to 1 before
> > operation the Ethernet MAC. The MDIO transferring is one of the Ethernet MAC operation.
> > So, I assumed this RMII mode affects the MDIO transfers.
> >
> > By the way, I also should have described what happen on the previous code.
> > If I input the following command on non-NFS rootfs environment (e.g. USB memory rootfs),
> > sometimes, a lot of FCD interruption happened and then the following "rcu "error happened.
> > ---
> > # ifconfig eth0 down# ifconfig eth0 up
> > [ 794.545823] Micrel KSZ8041RNLI ee700000.ethernet-ffffffff:01: attached PHY driver [Micrel KSZ8041RNLI]
> (mii_bus:phy_addr=ee700000.ethernet-ffffffff:01, irq=157)
> > # ifconfig eth0 up# ifconfig eth0 down
> > # ifconfig eth0 down# ifconfig eth0 up
> > [ 797.662138] Micrel KSZ8041RNLI ee700000.ethernet-ffffffff:01: attached PHY driver [Micrel KSZ8041RNLI]
> (mii_bus:phy_addr=ee700000.ethernet-ffffffff:01, irq=157)
> > [ 797.689158] sh-eth ee700000.ethernet eth0: Link is Up - 100Mbps/Full - flow control rx/tx
> > [ 797.702631] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
> > # [ 818.759628] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
> > [ 818.770827] rcu: 0-....: (2 GPs behind) idle=d12/0/0x3 softirq=388/388 fqs=1051
> > [ 818.783635] rcu: (detected by 1, t=2104 jiffies, g=377, q=18)
> > [ 818.794848] Sending NMI from CPU 1 to CPUs 0:
> > [ 818.804733] NMI backtrace for cpu 0
> > [ 818.804736] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.1.0-00002-g192664e-dirty #27
> > [ 818.804738] Hardware name: Generic R-Car Gen2 (Flattened Device Tree)
> > [ 818.804740] PC is at __do_softirq+0x7c/0x344
> > [ 818.804742] LR is at irq_exit+0x6c/0xcc
> > [ 818.804744] pc : [<c01021e4>] lr : [<c0126aa0>] psr: 60030113
> > [ 818.804746] sp : c0d01eb8 ip : 00200102 fp : c0d03080
> > [ 818.804748] r10: 00000001 r9 : 0000000a r8 : c0ca2940
> > [ 818.804750] r7 : 00000004 r6 : c0ca2934 r5 : 00000000 r4 : ffffe000
> > [ 818.804752] r3 : c0ca2940 r2 : 2aaa8000 r1 : 00000000 r0 : 2aaa8000
> > [ 818.804754] Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none
> > [ 818.804756] Control: 10c5387d Table: 6a5f006a DAC: 00000051
> > [ 818.804758] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.1.0-00002-g192664e-dirty #27
> > [ 818.804760] Hardware name: Generic R-Car Gen2 (Flattened Device Tree)
> > [ 818.804762] [<c011049c>] (unwind_backtrace) from [<c010bc0c>] (show_stack+0x18/0x1c)
> > [ 818.804764] [<c010bc0c>] (show_stack) from [<c07f7458>] (dump_stack+0x7c/0x9c)
> > [ 818.804766] [<c07f7458>] (dump_stack) from [<c07fd358>] (nmi_cpu_backtrace+0x98/0xb8)
> > [ 818.804768] [<c07fd358>] (nmi_cpu_backtrace) from [<c010e6bc>] (handle_IPI+0x2bc/0x33c)
> > [ 818.804770] [<c010e6bc>] (handle_IPI) from [<c03fa084>] (gic_handle_irq+0x8c/0x98)
> > [ 818.804773] [<c03fa084>] (gic_handle_irq) from [<c0101a8c>] (__irq_svc+0x6c/0x90)
> > [ 818.804774] Exception stack(0xc0d01e68 to 0xc0d01eb0)
> > [ 818.804777] 1e60: 2aaa8000 00000000 2aaa8000 c0ca2940 ffffe000 00000000
> > [ 818.804779] 1e80: c0ca2934 00000004 c0ca2940 0000000a 00000001 c0d03080 00200102 c0d01eb8
> > [ 818.804781] 1ea0: c0126aa0 c01021e4 60030113 ffffffff
> > [ 818.804783] [<c0101a8c>] (__irq_svc) from [<c01021e4>] (__do_softirq+0x7c/0x344)
> > [ 818.804785] [<c01021e4>] (__do_softirq) from [<c0126aa0>] (irq_exit+0x6c/0xcc)
> > [ 818.804787] [<c0126aa0>] (irq_exit) from [<c016c344>] (__handle_domain_irq+0x88/0xbc)
> > [ 818.804789] [<c016c344>] (__handle_domain_irq) from [<c03fa058>] (gic_handle_irq+0x60/0x98)
> > [ 818.804791] [<c03fa058>] (gic_handle_irq) from [<c0101a8c>] (__irq_svc+0x6c/0x90)
> > [ 818.804793] Exception stack(0xc0d01f38 to 0xc0d01f80)
> > [ 818.804795] 1f20: 00000000 0008bbb8
> > [ 818.804798] 1f40: eb74b520 c011aee0 00000001 ffffe000 c0d08c30 c0d8e5be c0d08c74 c0d980c0
> > [ 818.804800] 1f60: 00000001 c0c5ea40 00000000 c0d01f88 c01084ec c014c85c 60030013 ffffffff
> > [ 818.804802] [<c0101a8c>] (__irq_svc) from [<c014c85c>] (do_idle+0xf0/0x13c)
> > [ 818.804804] [<c014c85c>] (do_idle) from [<c014cb10>] (cpu_startup_entry+0x20/0x24)
> > [ 818.804806] [<c014cb10>] (cpu_startup_entry) from [<c0c00fac>] (start_kernel+0x3e0/0x488)
>
> This *usually* means a register is accessed while the clock is off...
I see..
> > Note:
> > - After I input the second ifconfig eth0 up, I cannot input any command.
> > - If I disconnect an ethernet cable, I can input command again.
>
> Hm...
>
> > ---
> >
> >>> To fix the issue, this patch adds a condition and set the RMII mode
> >>> regiseter in sh_eth_dev_exit() for some SoCs.
>
> Register.
Oops. I'll fix it.
<snip>
> >>> diff --git a/drivers/net/ethernet/renesas/sh_eth.c b/drivers/net/ethernet/renesas/sh_eth.c
> >>> index e33af37..106ae90 100644
> >>> --- a/drivers/net/ethernet/renesas/sh_eth.c
> >>> +++ b/drivers/net/ethernet/renesas/sh_eth.c
> >>> @@ -1596,6 +1596,10 @@ static void sh_eth_dev_exit(struct net_device *ndev)
> >>>
> >>> /* Set MAC address again */
> >>> update_mac_address(ndev);
> >>> +
> >>> + /* Set the mode again if required */
> >>
> >> Should be "RMII mode", n ot just "Mode". We prolly need more detailed explanation...
> >
> > I got it.
>
> Your fix seems legit now, just misplaced a bit...
Thank you for your comment! I'll move the RMIIMODE setting before calls update_mac_address().
> >>> + if (mdp->cd->rmiimode)
> >>> + sh_eth_write(ndev, 0x1, RMIIMODE);
>
> According to the gen2 manual, this register should be written right after the soft reset
> via EDMR.
I got it.
Best regards,
Yoshihiro Shimoda
> >>> }
> >>>
> >>> static void sh_eth_rx_csum(struct sk_buff *skb)
>
> MBR, Sergei
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2019-05-28 2:34 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-05-15 5:36 [PATCH] net: sh_eth: fix mdio access in sh_eth_close() for some SoCs Yoshihiro Shimoda
2019-05-15 15:41 ` Sergei Shtylyov
2019-05-20 7:42 ` Yoshihiro Shimoda
2019-05-25 18:06 ` Sergei Shtylyov
2019-05-28 2:33 ` Yoshihiro Shimoda
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).