Re: Bug#927825: arm: mvneta driver used on Armada XP GP boards does not receive packets (regression from 4.9)

Re: Bug#927825: arm: mvneta driver used on Armada XP GP boards does not receive packets (regression from 4.9)

[Uwe Kleine-König]

[-- Attachment #1: Type: text/plain, Size: 2793 bytes --]

[Adding the mvebu guys and netdev to Cc]

Hello,

On Thu, Apr 25, 2019 at 09:17:32PM +0200, Aurelien Jarno wrote:
> On 2019-04-25 14:50, Aurelien Jarno wrote:
> > On 2019-04-23 22:16, Aurelien Jarno wrote:
> > > Source: linux
> > > Version: 4.19.28-2
> > > Severity: important
> > > 
> > > After upgrading hartmann.debian.org (an armhf buildd using an Armada XP
> > > GP board) from buster to stretch, the ethernet device is not working

"upgrading from buster to stretch" doesn't make sense. I think you meant
from stretch to buster.

> > 
> > More precisely the board is a "Marvell Armada XP Development Board
> > DB-MV784MP-GP"
> > 
> > > anymore. Using tcpdump on both the buildd and a remote host, it appears
> > > that the packets correctly leave the board and that the reception side
> > > fails.

If you can send to a remote host at least ARP (or ND) must be working,
so some reception still works, right?

> > > The module used for the ethernet device is mvneta. The corresponding DT
> > > compatible entry is "marvell,armada-xp-neta".
> > >
> > 
> > I have started a "bisection" with the kernels from snapshot. This is
> > what I have found so far:
> > 
> > This one works:
> > - linux-image-4.19.0-rc6-armmp-lpae_4.19~rc6-1~exp1_armhf.deb 
> > 
> > The following ones don't:
> > - linux-image-4.19.0-rc7-armmp-lpae_4.19~rc7-1~exp1_armhf.deb
> > - linux-image-5.0.0-trunk-armmp_5.0.2-1~exp1_armhf.deb
> > 
> > My guess (I don't have time to try more now) is that the issue is caused
> > by the following change:
> > 
> > |  [ Uwe Kleine-König ]
> > |  * [armhf] enable MVNETA_BM_ENABLE and CAN_FLEXCAN as a module
> > 
> 
> I confirm this is the issue. Disabling MVNETA_BM_ENABLE on kernel 
> 4.19.28-2 fixes the issue. Note that it breaks the ABI.

A colleague happens to work with an XP based machine with a (nearly)
vanilla kernel based on 5.1.0-rc6 and there enabling MVNETA_BM_ENABLE
doesn't render networking nonfunctional.

Looking through the changes to drivers/net/ethernet/marvell/mvneta*
between 5.0 and 5.1-rc6 there isn't something that would explain a fix
though. There doesn't seem to be a good explanation in the debian
specific patches either.

So this problem is either machine specific or it works with the mvneta
driver builtin. (I didn't double check, but guess that my colleague uses
=y and the Debian kernel =m). Well, or I missed something.

Is it possible to test a few things on hartmann? I'd suggest:

 - try (vanilla) 5.1-rc6 with MVNETA=y
 - try an older kernel (maybe 4.6 as the buffer manager stuff was
   introduced in dc35a10f68d3 ("net: mvneta: bm: add support for
   hardware buffer management") which made it into 4.6-rc1) with
   MVNETA_BM_ENABLE=[ym].

Best regards
Uwe

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Bug#927825: arm: mvneta driver used on Armada XP GP boards does not receive packets (regression from 4.9)

[Aurelien Jarno]

[-- Attachment #1: Type: text/plain, Size: 3750 bytes --]

On 2019-04-30 10:12, Uwe Kleine-König wrote:
> [Adding the mvebu guys and netdev to Cc]
> 
> Hello,
> 
> On Thu, Apr 25, 2019 at 09:17:32PM +0200, Aurelien Jarno wrote:
> > On 2019-04-25 14:50, Aurelien Jarno wrote:
> > > On 2019-04-23 22:16, Aurelien Jarno wrote:
> > > > Source: linux
> > > > Version: 4.19.28-2
> > > > Severity: important
> > > > 
> > > > After upgrading hartmann.debian.org (an armhf buildd using an Armada XP
> > > > GP board) from buster to stretch, the ethernet device is not working
> 
> "upgrading from buster to stretch" doesn't make sense. I think you meant
> from stretch to buster.

You're correct. To be more precise the kernel is there the issue, so
that should even be upgrading from the stretch kernel to the buster
kernel.

> > > More precisely the board is a "Marvell Armada XP Development Board
> > > DB-MV784MP-GP"
> > > 
> > > > anymore. Using tcpdump on both the buildd and a remote host, it appears
> > > > that the packets correctly leave the board and that the reception side
> > > > fails.
> 
> If you can send to a remote host at least ARP (or ND) must be working,
> so some reception still works, right?

I have to try again, but what i have seen is the ARP requests from
hartmann arriving to the other hosts on the subnet. Steve McIntyre
(added in Cc:) confirmed me on IRC being able to reproduce the issue on
another board.

> > > > The module used for the ethernet device is mvneta. The corresponding DT
> > > > compatible entry is "marvell,armada-xp-neta".
> > > >
> > > 
> > > I have started a "bisection" with the kernels from snapshot. This is
> > > what I have found so far:
> > > 
> > > This one works:
> > > - linux-image-4.19.0-rc6-armmp-lpae_4.19~rc6-1~exp1_armhf.deb 
> > > 
> > > The following ones don't:
> > > - linux-image-4.19.0-rc7-armmp-lpae_4.19~rc7-1~exp1_armhf.deb
> > > - linux-image-5.0.0-trunk-armmp_5.0.2-1~exp1_armhf.deb
> > > 
> > > My guess (I don't have time to try more now) is that the issue is caused
> > > by the following change:
> > > 
> > > |  [ Uwe Kleine-König ]
> > > |  * [armhf] enable MVNETA_BM_ENABLE and CAN_FLEXCAN as a module
> > > 
> > 
> > I confirm this is the issue. Disabling MVNETA_BM_ENABLE on kernel 
> > 4.19.28-2 fixes the issue. Note that it breaks the ABI.
> 
> A colleague happens to work with an XP based machine with a (nearly)
> vanilla kernel based on 5.1.0-rc6 and there enabling MVNETA_BM_ENABLE
> doesn't render networking nonfunctional.
> 
> Looking through the changes to drivers/net/ethernet/marvell/mvneta*
> between 5.0 and 5.1-rc6 there isn't something that would explain a fix
> though. There doesn't seem to be a good explanation in the debian
> specific patches either.
> 
> So this problem is either machine specific or it works with the mvneta
> driver builtin. (I didn't double check, but guess that my colleague uses
> =y and the Debian kernel =m). Well, or I missed something.

Yes the debian kernel uses:
CONFIG_MVNETA_BM_ENABLE=m
CONFIG_MVNETA=m
CONFIG_MVNETA_BM=m

> Is it possible to test a few things on hartmann? I'd suggest:
> 
>  - try (vanilla) 5.1-rc6 with MVNETA=y
>  - try an older kernel (maybe 4.6 as the buffer manager stuff was
>    introduced in dc35a10f68d3 ("net: mvneta: bm: add support for
>    hardware buffer management") which made it into 4.6-rc1) with
>    MVNETA_BM_ENABLE=[ym].

Ok, I'll try that in the next days. I just do not have remote power,
only serial console, I hope the kernels will boot enough to be able to
roll back to another kernel.

Best regards,
Aurelien

-- 
Aurelien Jarno                          GPG: 4096R/1DDD8C9B
aurelien@aurel32.net                 http://www.aurel32.net

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Bug#927825: arm: mvneta driver used on Armada XP GP boards does not receive packets (regression from 4.9)

[Aurelien Jarno]

[-- Attachment #1: Type: text/plain, Size: 1998 bytes --]

Hi,

On 2019-05-01 00:04, Aurelien Jarno wrote:
> On 2019-04-30 10:12, Uwe Kleine-König wrote:
> > > > More precisely the board is a "Marvell Armada XP Development Board
> > > > DB-MV784MP-GP"
> > > > 
> > > > > anymore. Using tcpdump on both the buildd and a remote host, it appears
> > > > > that the packets correctly leave the board and that the reception side
> > > > > fails.
> > 
> > If you can send to a remote host at least ARP (or ND) must be working,
> > so some reception still works, right?
> 
> I have to try again, but what i have seen is the ARP requests from
> hartmann arriving to the other hosts on the subnet. Steve McIntyre
> (added in Cc:) confirmed me on IRC being able to reproduce the issue on
> another board.

I confirm that. Basically on the other hosts of the same subnet, I can
see the ARP requests:

18:23:45.979860 ARP, Request who-has 172.28.17.1 tell 172.28.17.18, length 46
18:23:47.002990 ARP, Request who-has 172.28.17.1 tell 172.28.17.18, length 46
18:23:48.027262 ARP, Request who-has 172.28.17.1 tell 172.28.17.18, length 46
18:23:52.004248 ARP, Request who-has 172.28.17.1 tell 172.28.17.18, length 46
18:23:53.019252 ARP, Request who-has 172.28.17.1 tell 172.28.17.18, length 46
18:23:54.043276 ARP, Request who-has 172.28.17.1 tell 172.28.17.18, length 46
18:23:58.027937 ARP, Request who-has 172.28.17.1 tell 172.28.17.18, length 46

172.28.17.1 is the gateway, 172.28.17.18 is hartmann.d.o. 

> > Is it possible to test a few things on hartmann? I'd suggest:
> > 
> >  - try (vanilla) 5.1-rc6 with MVNETA=y

I have tried 5.1-rc7 with:

CONFIG_MVNETA_BM_ENABLE=y
CONFIG_MVNETA=y
CONFIG_MVNETA_BM=y

and also with

CONFIG_MVNETA_BM_ENABLE=m
CONFIG_MVNETA=m
CONFIG_MVNETA_BM=m

And the mvneta network driver is not able to receive data in both cases.

Best regards,
Aurelien

-- 
Aurelien Jarno                          GPG: 4096R/1DDD8C9B
aurelien@aurel32.net                 http://www.aurel32.net

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Bug#927825: arm: mvneta driver used on Armada XP GP boards does not receive packets (regression from 4.9)

[Uwe Kleine-König]

[-- Attachment #1: Type: text/plain, Size: 3395 bytes --]

Hello,

On Tue, Apr 30, 2019 at 10:12:27AM +0200, Uwe Kleine-König wrote:
> On Thu, Apr 25, 2019 at 09:17:32PM +0200, Aurelien Jarno wrote:
> > On 2019-04-25 14:50, Aurelien Jarno wrote:
> > > On 2019-04-23 22:16, Aurelien Jarno wrote:
> > > > Source: linux
> > > > Version: 4.19.28-2
> > > > Severity: important
> > > > 
> > > > After upgrading hartmann.debian.org (an armhf buildd using an Armada XP
> > > > GP board) from buster to stretch, the ethernet device is not working
> 
> "upgrading from buster to stretch" doesn't make sense. I think you meant
> from stretch to buster.
> 
> > > 
> > > More precisely the board is a "Marvell Armada XP Development Board
> > > DB-MV784MP-GP"
> > > 
> > > > anymore. Using tcpdump on both the buildd and a remote host, it appears
> > > > that the packets correctly leave the board and that the reception side
> > > > fails.
> 
> If you can send to a remote host at least ARP (or ND) must be working,
> so some reception still works, right?
> 
> > > > The module used for the ethernet device is mvneta. The corresponding DT
> > > > compatible entry is "marvell,armada-xp-neta".
> > > >
> > > 
> > > I have started a "bisection" with the kernels from snapshot. This is
> > > what I have found so far:
> > > 
> > > This one works:
> > > - linux-image-4.19.0-rc6-armmp-lpae_4.19~rc6-1~exp1_armhf.deb 
> > > 
> > > The following ones don't:
> > > - linux-image-4.19.0-rc7-armmp-lpae_4.19~rc7-1~exp1_armhf.deb
> > > - linux-image-5.0.0-trunk-armmp_5.0.2-1~exp1_armhf.deb
> > > 
> > > My guess (I don't have time to try more now) is that the issue is caused
> > > by the following change:
> > > 
> > > |  [ Uwe Kleine-König ]
> > > |  * [armhf] enable MVNETA_BM_ENABLE and CAN_FLEXCAN as a module
> > > 
> > 
> > I confirm this is the issue. Disabling MVNETA_BM_ENABLE on kernel 
> > 4.19.28-2 fixes the issue. Note that it breaks the ABI.
> 
> A colleague happens to work with an XP based machine with a (nearly)
> vanilla kernel based on 5.1.0-rc6 and there enabling MVNETA_BM_ENABLE
> doesn't render networking nonfunctional.
> 
> Looking through the changes to drivers/net/ethernet/marvell/mvneta*
> between 5.0 and 5.1-rc6 there isn't something that would explain a fix
> though. There doesn't seem to be a good explanation in the debian
> specific patches either.
> 
> So this problem is either machine specific or it works with the mvneta
> driver builtin. (I didn't double check, but guess that my colleague uses
> =y and the Debian kernel =m). Well, or I missed something.
> 
> Is it possible to test a few things on hartmann? I'd suggest:
> 
>  - try (vanilla) 5.1-rc6 with MVNETA=y
>  - try an older kernel (maybe 4.6 as the buffer manager stuff was
>    introduced in dc35a10f68d3 ("net: mvneta: bm: add support for
>    hardware buffer management") which made it into 4.6-rc1) with
>    MVNETA_BM_ENABLE=[ym].

Thanks to Steve McIntyre I got access to a DB-MV784MP-GP and did the
second test. I used mvebu_v7_defconfig and enabled CGROUPS and AUTOFS4
(to please systemd) and MVNETA_BM_ENABLE=y on 4.6. The latter breaks
networking similar to newer kernel versions.

So I guess the buffer management never worked on that board.

I don't have the time to debug this issue further, but will disable
buffer management for the Debian kernel again.

Best regards
Uwe

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]