m_can error/overrun frames on high speed

m_can error/overrun frames on high speed

[Belisko Marek]

Hi,

I have a beaglebone based board and I'm performing some tests. I
discovered that when set bitrate to 500k during replaying can file
from PC to board ip detect 4-5 error/overrun frames. When comparing
the original file with received one few lines in candump are missing.
When decreased can speed to 125KB replaying the same file no
error/overruns are detected and files are the same. I'm not can expert
thus I'm asking for some advice on how to debug such phenomena. I'm
using mainline 4.12 kernel which shows this symptom. I compared
changes with the latest mainline kernel and there are few patches only
which seems can influence can behavior (others are only cosmetical). I
took :

3cb3eaac52c0f145d895f4b6c22834d5f02b8569 - can: c_can: c_can_poll():
only read status register after status IRQ
23c5a9488f076bab336177cd1d1a366bd8ddf087 - can: c_can: D_CAN:
c_can_chip_config(): perform a sofware reset on open
6f12001ad5e79d0a0b08c599731d45c34cafd376 - can: c_can: C_CAN: add bus
recovery events

I know most of the answers for such issues is to try latest kernel
(i'm in process trying 5.10).

Thanks and BR,

marek

-- 
as simple and primitive as possible
-------------------------------------------------
Marek Belisko - OPEN-NANDRA
Freelance Developer

Ruska Nova Ves 219 | Presov, 08005 Slovak Republic
Tel: +421 915 052 184
skype: marekwhite
twitter: #opennandra
web: http://open-nandra.com

Re: m_can error/overrun frames on high speed

[Marc Kleine-Budde]

[-- Attachment #1: Type: text/plain, Size: 1811 bytes --]

On 28.03.2021 08:31:14, Belisko Marek wrote:
> I have a beaglebone based board and I'm performing some tests.

As far as I know the beagle bone boards all have d_can controllers, not
m_can.

> I discovered that when set bitrate to 500k during replaying can file
> from PC to board ip detect 4-5 error/overrun frames. When comparing
> the original file with received one few lines in candump are missing.
> When decreased can speed to 125KB replaying the same file no
> error/overruns are detected and files are the same. I'm not can expert
> thus I'm asking for some advice on how to debug such phenomena. I'm
> using mainline 4.12 kernel which shows this symptom. I compared
> changes with the latest mainline kernel and there are few patches only
> which seems can influence can behavior (others are only cosmetical). I
> took :
> 
> 3cb3eaac52c0f145d895f4b6c22834d5f02b8569 - can: c_can: c_can_poll():
> only read status register after status IRQ
> 23c5a9488f076bab336177cd1d1a366bd8ddf087 - can: c_can: D_CAN:
> c_can_chip_config(): perform a sofware reset on open
> 6f12001ad5e79d0a0b08c599731d45c34cafd376 - can: c_can: C_CAN: add bus
> recovery events
> 
> I know most of the answers for such issues is to try latest kernel
> (i'm in process trying 5.10).

That's going into the right direction. Please try the lastest
net-next/master, which includes this merge:

https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=9c0ee085c49c11381dcbd609ea85e902eab88a92

regards,
Marc

-- 
Pengutronix e.K.                 | Marc Kleine-Budde           |
Embedded Linux                   | https://www.pengutronix.de  |
Vertretung West/Dortmund         | Phone: +49-231-2826-924     |
Amtsgericht Hildesheim, HRA 2686 | Fax:   +49-5121-206917-5555 |

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: m_can error/overrun frames on high speed

[Belisko Marek]

Hi,
On Wed, Mar 31, 2021 at 10:37 AM Marc Kleine-Budde <mkl@pengutronix.de> wrote:
>
> On 28.03.2021 08:31:14, Belisko Marek wrote:
> > I have a beaglebone based board and I'm performing some tests.
>
> As far as I know the beagle bone boards all have d_can controllers, not
> m_can.
Yes sorry it was typo.
>
> > I discovered that when set bitrate to 500k during replaying can file
> > from PC to board ip detect 4-5 error/overrun frames. When comparing
> > the original file with received one few lines in candump are missing.
> > When decreased can speed to 125KB replaying the same file no
> > error/overruns are detected and files are the same. I'm not can expert
> > thus I'm asking for some advice on how to debug such phenomena. I'm
> > using mainline 4.12 kernel which shows this symptom. I compared
> > changes with the latest mainline kernel and there are few patches only
> > which seems can influence can behavior (others are only cosmetical). I
> > took :
> >
> > 3cb3eaac52c0f145d895f4b6c22834d5f02b8569 - can: c_can: c_can_poll():
> > only read status register after status IRQ
> > 23c5a9488f076bab336177cd1d1a366bd8ddf087 - can: c_can: D_CAN:
> > c_can_chip_config(): perform a sofware reset on open
> > 6f12001ad5e79d0a0b08c599731d45c34cafd376 - can: c_can: C_CAN: add bus
> > recovery events
> >
> > I know most of the answers for such issues is to try latest kernel
> > (i'm in process trying 5.10).
>
> That's going into the right direction. Please try the lastest
> net-next/master, which includes this merge:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=9c0ee085c49c11381dcbd609ea85e902eab88a92
I tried to build this kernel and when run on my target and run on
other side cangen can0 -g0 (at 500kb bitrate) after some time I see on
receiving side:
3: can0: <NOARP,UP,LOWER_UP,ECHO> mtu 16 qdisc pfifo_fast state UP
mode DEFAULT group default qlen 10
    link/can  promiscuity 0
    can state ERROR-ACTIVE (berr-counter tx 0 rx 0) restart-ms 0
          bitrate 500000 sample-point 0.875
          tq 125 prop-seg 6 phase-seg1 7 phase-seg2 2 sjw 1
          c_can: tseg1 2..16 tseg2 1..8 sjw 1..4 brp 1..1024 brp-inc 1
          clock 24000000
          re-started bus-errors arbit-lost error-warn error-pass bus-off
          0          0          0          0          0          0
    numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
    RX: bytes  packets  errors  dropped overrun mcast
    6300263    999976   4       0       4       0
    TX: bytes  packets  errors  dropped carrier collsns
    0          0        0       0       0       0

errors/overrun frames. My theory is that before napi handling of
received data we disable interrupts and when we process received
messages and re-enable irq again we can see overrun because reading of
data can be slow.
Is there anything I can tune to have it read faster? Thanks.
>
> regards,
> Marc
>
> --
> Pengutronix e.K.                 | Marc Kleine-Budde           |
> Embedded Linux                   | https://www.pengutronix.de  |
> Vertretung West/Dortmund         | Phone: +49-231-2826-924     |
> Amtsgericht Hildesheim, HRA 2686 | Fax:   +49-5121-206917-5555 |

BR,

marek

Re: m_can error/overrun frames on high speed

[Marc Kleine-Budde]

[-- Attachment #1: Type: text/plain, Size: 3894 bytes --]

On 01.04.2021 11:04:25, Belisko Marek wrote:
> > As far as I know the beagle bone boards all have d_can controllers, not
> > m_can.
> Yes sorry it was typo.

No problem, just wanted to be sure :)

> > > I discovered that when set bitrate to 500k during replaying can file
> > > from PC to board ip detect 4-5 error/overrun frames. When comparing
> > > the original file with received one few lines in candump are missing.
> > > When decreased can speed to 125KB replaying the same file no
> > > error/overruns are detected and files are the same. I'm not can expert
> > > thus I'm asking for some advice on how to debug such phenomena. I'm
> > > using mainline 4.12 kernel which shows this symptom. I compared
> > > changes with the latest mainline kernel and there are few patches only
> > > which seems can influence can behavior (others are only cosmetical). I
> > > took :
> > >
> > > 3cb3eaac52c0f145d895f4b6c22834d5f02b8569 - can: c_can: c_can_poll():
> > > only read status register after status IRQ
> > > 23c5a9488f076bab336177cd1d1a366bd8ddf087 - can: c_can: D_CAN:
> > > c_can_chip_config(): perform a sofware reset on open
> > > 6f12001ad5e79d0a0b08c599731d45c34cafd376 - can: c_can: C_CAN: add bus
> > > recovery events
> > >
> > > I know most of the answers for such issues is to try latest kernel
> > > (i'm in process trying 5.10).
> >
> > That's going into the right direction. Please try the lastest
> > net-next/master, which includes this merge:
> >
> > https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=9c0ee085c49c11381dcbd609ea85e902eab88a92

> I tried to build this kernel and when run on my target and run on
> other side cangen can0 -g0 (at 500kb bitrate) after some time I see on
> receiving side:

Does the current net-next lead to fewer lost frames than your original
kernel? I mean does it make the situation better?

> 3: can0: <NOARP,UP,LOWER_UP,ECHO> mtu 16 qdisc pfifo_fast state UP
> mode DEFAULT group default qlen 10
>     link/can  promiscuity 0
>     can state ERROR-ACTIVE (berr-counter tx 0 rx 0) restart-ms 0
>           bitrate 500000 sample-point 0.875
>           tq 125 prop-seg 6 phase-seg1 7 phase-seg2 2 sjw 1
>           c_can: tseg1 2..16 tseg2 1..8 sjw 1..4 brp 1..1024 brp-inc 1
>           clock 24000000
>           re-started bus-errors arbit-lost error-warn error-pass bus-off
>           0          0          0          0          0          0
>     numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
>     RX: bytes  packets  errors  dropped overrun mcast
>     6300263    999976   4       0       4       0
>     TX: bytes  packets  errors  dropped carrier collsns
>     0          0        0       0       0       0
> 
> errors/overrun frames. My theory is that before napi handling of
> received data we disable interrupts and when we process received
> messages and re-enable irq again we can see overrun because reading of
> data can be slow.

Yes, I assume the same problem.

> Is there anything I can tune to have it read faster? Thanks.

I don't think it can be done with tuning. To work around this problem,
you can convert the c_can driver to the rx-offload infrastructure. You
do the RX from the CAN HW in the IRQ handler, but pass it to the
networking stack in NAPI. This dance is needed, as otherwise the
networking stack messes up the order of received CAN frames.

There even is an old branch that implemented that, but was never merged:

https://git.kernel.org/pub/scm/linux/kernel/git/mkl/linux-can-next.git/log/?h=c_can

Marc

-- 
Pengutronix e.K.                 | Marc Kleine-Budde           |
Embedded Linux                   | https://www.pengutronix.de  |
Vertretung West/Dortmund         | Phone: +49-231-2826-924     |
Amtsgericht Hildesheim, HRA 2686 | Fax:   +49-5121-206917-5555 |

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: m_can error/overrun frames on high speed

[Belisko Marek]

Hi.

On Thu, Apr 1, 2021 at 11:23 AM Marc Kleine-Budde <mkl@pengutronix.de> wrote:
>
> On 01.04.2021 11:04:25, Belisko Marek wrote:
> > > As far as I know the beagle bone boards all have d_can controllers, not
> > > m_can.
> > Yes sorry it was typo.
>
> No problem, just wanted to be sure :)
>
> > > > I discovered that when set bitrate to 500k during replaying can file
> > > > from PC to board ip detect 4-5 error/overrun frames. When comparing
> > > > the original file with received one few lines in candump are missing.
> > > > When decreased can speed to 125KB replaying the same file no
> > > > error/overruns are detected and files are the same. I'm not can expert
> > > > thus I'm asking for some advice on how to debug such phenomena. I'm
> > > > using mainline 4.12 kernel which shows this symptom. I compared
> > > > changes with the latest mainline kernel and there are few patches only
> > > > which seems can influence can behavior (others are only cosmetical). I
> > > > took :
> > > >
> > > > 3cb3eaac52c0f145d895f4b6c22834d5f02b8569 - can: c_can: c_can_poll():
> > > > only read status register after status IRQ
> > > > 23c5a9488f076bab336177cd1d1a366bd8ddf087 - can: c_can: D_CAN:
> > > > c_can_chip_config(): perform a sofware reset on open
> > > > 6f12001ad5e79d0a0b08c599731d45c34cafd376 - can: c_can: C_CAN: add bus
> > > > recovery events
> > > >
> > > > I know most of the answers for such issues is to try latest kernel
> > > > (i'm in process trying 5.10).
> > >
> > > That's going into the right direction. Please try the lastest
> > > net-next/master, which includes this merge:
> > >
> > > https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=9c0ee085c49c11381dcbd609ea85e902eab88a92
>
> > I tried to build this kernel and when run on my target and run on
> > other side cangen can0 -g0 (at 500kb bitrate) after some time I see on
> > receiving side:
>
> Does the current net-next lead to fewer lost frames than your original
> kernel? I mean does it make the situation better?
Nope the situation is more less same. I'm just curious as this code is
there for years and nobody complains about missing frames ;).
I tried 4.12 mainline, 4.19 TI-SDK kernel, latest stable 5.10 and
linux-net-next.
>
> > 3: can0: <NOARP,UP,LOWER_UP,ECHO> mtu 16 qdisc pfifo_fast state UP
> > mode DEFAULT group default qlen 10
> >     link/can  promiscuity 0
> >     can state ERROR-ACTIVE (berr-counter tx 0 rx 0) restart-ms 0
> >           bitrate 500000 sample-point 0.875
> >           tq 125 prop-seg 6 phase-seg1 7 phase-seg2 2 sjw 1
> >           c_can: tseg1 2..16 tseg2 1..8 sjw 1..4 brp 1..1024 brp-inc 1
> >           clock 24000000
> >           re-started bus-errors arbit-lost error-warn error-pass bus-off
> >           0          0          0          0          0          0
> >     numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
> >     RX: bytes  packets  errors  dropped overrun mcast
> >     6300263    999976   4       0       4       0
> >     TX: bytes  packets  errors  dropped carrier collsns
> >     0          0        0       0       0       0
> >
> > errors/overrun frames. My theory is that before napi handling of
> > received data we disable interrupts and when we process received
> > messages and re-enable irq again we can see overrun because reading of
> > data can be slow.
>
> Yes, I assume the same problem.
>
> > Is there anything I can tune to have it read faster? Thanks.
>
> I don't think it can be done with tuning. To work around this problem,
> you can convert the c_can driver to the rx-offload infrastructure. You
> do the RX from the CAN HW in the IRQ handler, but pass it to the
> networking stack in NAPI. This dance is needed, as otherwise the
> networking stack messes up the order of received CAN frames.
OK thanks a lot for hint I'll investigate and report back.
>
> There even is an old branch that implemented that, but was never merged:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/mkl/linux-can-next.git/log/?h=c_can
>
> Marc
>
> --
> Pengutronix e.K.                 | Marc Kleine-Budde           |
> Embedded Linux                   | https://www.pengutronix.de  |
> Vertretung West/Dortmund         | Phone: +49-231-2826-924     |
> Amtsgericht Hildesheim, HRA 2686 | Fax:   +49-5121-206917-5555 |

BR,

marek

Re: m_can error/overrun frames on high speed

[Marc Kleine-Budde]

[-- Attachment #1: Type: text/plain, Size: 662 bytes --]

On 01.04.2021 11:04:25, Belisko Marek wrote:
> Is there anything I can tune to have it read faster? Thanks.

Indeed, with current net-next/master, you put the NAPI on a per
networking device from softirq to threaded mode.

| echo 1 | sudo tee /sys/class/net/can0/threaded

Then you a separate NAPI thread per networking device that can be tuned
with chrt, etc...

regards,
Marc

-- 
Pengutronix e.K.                 | Marc Kleine-Budde           |
Embedded Linux                   | https://www.pengutronix.de  |
Vertretung West/Dortmund         | Phone: +49-231-2826-924     |
Amtsgericht Hildesheim, HRA 2686 | Fax:   +49-5121-206917-5555 |

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: m_can error/overrun frames on high speed

[Belisko Marek]

On Fri, Apr 2, 2021 at 9:28 PM Marc Kleine-Budde <mkl@pengutronix.de> wrote:
>
> On 01.04.2021 11:04:25, Belisko Marek wrote:
> > Is there anything I can tune to have it read faster? Thanks.
>
> Indeed, with current net-next/master, you put the NAPI on a per
> networking device from softirq to threaded mode.
>
> | echo 1 | sudo tee /sys/class/net/can0/threaded
>
> Then you a separate NAPI thread per networking device that can be tuned
> with chrt, etc...
OK I've tse threaded mode for NAPI and for napi:can0 thread I set chrt
-p 99 <pid> but unfortunately there are still buffer overruns which is
kind of strange.

For rx-offload branch I was not able yet to run it on my platform
(have panic during boot which I need to resolve first).

Thanks and BR,

marek
>
> regards,
> Marc
>
> --
> Pengutronix e.K.                 | Marc Kleine-Budde           |
> Embedded Linux                   | https://www.pengutronix.de  |
> Vertretung West/Dortmund         | Phone: +49-231-2826-924     |
> Amtsgericht Hildesheim, HRA 2686 | Fax:   +49-5121-206917-5555 |

Re: m_can error/overrun frames on high speed

[Belisko Marek]

Hi Marc,

On Mon, Apr 5, 2021 at 7:31 PM Belisko Marek <marek.belisko@gmail.com> wrote:
>
> On Fri, Apr 2, 2021 at 9:28 PM Marc Kleine-Budde <mkl@pengutronix.de> wrote:
> >
> > On 01.04.2021 11:04:25, Belisko Marek wrote:
> > > Is there anything I can tune to have it read faster? Thanks.
> >
> > Indeed, with current net-next/master, you put the NAPI on a per
> > networking device from softirq to threaded mode.
> >
> > | echo 1 | sudo tee /sys/class/net/can0/threaded
> >
> > Then you a separate NAPI thread per networking device that can be tuned
> > with chrt, etc...
> OK I've tse threaded mode for NAPI and for napi:can0 thread I set chrt
> -p 99 <pid> but unfortunately there are still buffer overruns which is
> kind of strange.
>
> For rx-offload branch I was not able yet to run it on my platform
> (have panic during boot which I need to resolve first).
OK I resolved boot issues on my platform and tested using rx-offload
branch but again the issue is present.
It's enough to generate can frames using candeg can0 -g0 and I get
after some time error/overrun errors.
I'm trying now to bump rx-offload patches on top of linux-can-next
testing branch and test it.
>
> Thanks and BR,
>
> marek
> >
> > regards,
> > Marc
> >
> > --
> > Pengutronix e.K.                 | Marc Kleine-Budde           |
> > Embedded Linux                   | https://www.pengutronix.de  |
> > Vertretung West/Dortmund         | Phone: +49-231-2826-924     |
> > Amtsgericht Hildesheim, HRA 2686 | Fax:   +49-5121-206917-5555 |

Thanks and BR,

marek

Re: m_can error/overrun frames on high speed

[Belisko Marek]

Hi Marc,

On Mon, Apr 5, 2021 at 9:41 PM Belisko Marek <marek.belisko@gmail.com> wrote:
>
> Hi Marc,
>
> On Mon, Apr 5, 2021 at 7:31 PM Belisko Marek <marek.belisko@gmail.com> wrote:
> >
> > On Fri, Apr 2, 2021 at 9:28 PM Marc Kleine-Budde <mkl@pengutronix.de> wrote:
> > >
> > > On 01.04.2021 11:04:25, Belisko Marek wrote:
> > > > Is there anything I can tune to have it read faster? Thanks.
> > >
> > > Indeed, with current net-next/master, you put the NAPI on a per
> > > networking device from softirq to threaded mode.
> > >
> > > | echo 1 | sudo tee /sys/class/net/can0/threaded
> > >
> > > Then you a separate NAPI thread per networking device that can be tuned
> > > with chrt, etc...
> > OK I've tse threaded mode for NAPI and for napi:can0 thread I set chrt
> > -p 99 <pid> but unfortunately there are still buffer overruns which is
> > kind of strange.
> >
> > For rx-offload branch I was not able yet to run it on my platform
> > (have panic during boot which I need to resolve first).
> OK I resolved boot issues on my platform and tested using rx-offload
> branch but again the issue is present.
> It's enough to generate can frames using candeg can0 -g0 and I get
> after some time error/overrun errors.
> I'm trying now to bump rx-offload patches on top of linux-can-next
> testing branch and test it.
Was bit fast with response. With rx-offload feature receiving can
frames with high speed (small gaps) is working fine as expected.
So what are the next steps? I can port rx-offload for c_can to actual
master and verify and post patches for review?
> >


Thanks and BR,

marek

Re: m_can error/overrun frames on high speed

[Marc Kleine-Budde]

[-- Attachment #1: Type: text/plain, Size: 1106 bytes --]

On 28.04.2021 10:18:05, Belisko Marek wrote:
> > > For rx-offload branch I was not able yet to run it on my platform
> > > (have panic during boot which I need to resolve first).
> > OK I resolved boot issues on my platform and tested using rx-offload
> > branch but again the issue is present.
> > It's enough to generate can frames using candeg can0 -g0 and I get
> > after some time error/overrun errors.
> > I'm trying now to bump rx-offload patches on top of linux-can-next
> > testing branch and test it.
> Was bit fast with response. With rx-offload feature receiving can
> frames with high speed (small gaps) is working fine as expected.

Great!

> So what are the next steps? I can port rx-offload for c_can to actual
> master and verify and post patches for review?

Yes, please go ahead!

regards,
Marc

-- 
Pengutronix e.K.                 | Marc Kleine-Budde           |
Embedded Linux                   | https://www.pengutronix.de  |
Vertretung West/Dortmund         | Phone: +49-231-2826-924     |
Amtsgericht Hildesheim, HRA 2686 | Fax:   +49-5121-206917-5555 |

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]