* CAN messages being lost on i.MX25 with flexcan @ 2012-04-19 10:04 Martin Kozusky 2012-04-19 10:54 ` Wolfgang Grandegger 0 siblings, 1 reply; 12+ messages in thread From: Martin Kozusky @ 2012-04-19 10:04 UTC (permalink / raw) To: linux-can Hello, I'm using Voipac i.MX25 module with flexcan, kernel 2.6.38.9. I'm sending the data at 250kbps, around 1100 msgs/sec. When I enable the canbus interface (canconfig can0 start), CPU load is higher, that is understandable, there are many interrupts. I'm not doing anything else then using recvmsg (or recvmmsg which is little better), but some messages are still lost (around 1500 messages lost from 467 000 being send from another source). When I start doing something (like "cat /proc/interrupts", or write to file), many more messages are lost. Do you have any idea how to fix this? I need to make some CAN messages logger and I cannot lose any message (idealy :) So I made big buffer in my program so that I don't need to write the messages into the file while "recording" is enabled, after "recording" is switched off, I write the buffer into the file, but that is still not good enough. Is there any way how to write to some buffer directly in flexcan driver (the best would be in the IRQ routine) and then read messages from this buffer in my program? Or are just interrupts lost when doing something else in the system and I cannot fix this? Or can I somehow specify that "can rx interrupts" has highest prioroty? Thanks, Martin ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: CAN messages being lost on i.MX25 with flexcan 2012-04-19 10:04 CAN messages being lost on i.MX25 with flexcan Martin Kozusky @ 2012-04-19 10:54 ` Wolfgang Grandegger 2012-04-19 11:21 ` Martin Kozusky 0 siblings, 1 reply; 12+ messages in thread From: Wolfgang Grandegger @ 2012-04-19 10:54 UTC (permalink / raw) To: Martin Kozusky; +Cc: linux-can On 04/19/2012 12:04 PM, Martin Kozusky wrote: > Hello, > I'm using Voipac i.MX25 module with flexcan, kernel 2.6.38.9. > I'm sending the data at 250kbps, around 1100 msgs/sec. When I enable the > canbus interface (canconfig can0 start), CPU load is higher, that is > understandable, there are many interrupts. I'm not doing anything else > then using recvmsg (or recvmmsg which is little better), but some > messages are still lost (around 1500 messages lost from 467 000 being > send from another source). When I start doing something (like "cat > /proc/interrupts", or write to file), many more messages are lost. > > Do you have any idea how to fix this? I need to make some CAN messages > logger and I cannot lose any message (idealy :) So I made big buffer in > my program so that I don't need to write the messages into the file > while "recording" is enabled, after "recording" is switched off, I write > the buffer into the file, but that is still not good enough. > Is there any way how to write to some buffer directly in flexcan driver > (the best would be in the IRQ routine) and then read messages from this > buffer in my program? > Or are just interrupts lost when doing something else in the system and > I cannot fix this? Or can I somehow specify that "can rx interrupts" has > highest prioroty? Do you already know where the packets are dropped (get lost). Maybe your user space app is simply not faster enough to process them in time. You can check that with the candump option "-d". Wolfgang. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: CAN messages being lost on i.MX25 with flexcan 2012-04-19 10:54 ` Wolfgang Grandegger @ 2012-04-19 11:21 ` Martin Kozusky 2012-04-19 11:44 ` Wolfgang Grandegger 0 siblings, 1 reply; 12+ messages in thread From: Martin Kozusky @ 2012-04-19 11:21 UTC (permalink / raw) To: linux-can Dne 19.4.2012 12:54, Wolfgang Grandegger napsal(a): > On 04/19/2012 12:04 PM, Martin Kozusky wrote: >> Hello, >> I'm using Voipac i.MX25 module with flexcan, kernel 2.6.38.9. >> I'm sending the data at 250kbps, around 1100 msgs/sec. When I enable the >> canbus interface (canconfig can0 start), CPU load is higher, that is >> understandable, there are many interrupts. I'm not doing anything else >> then using recvmsg (or recvmmsg which is little better), but some >> messages are still lost (around 1500 messages lost from 467 000 being >> send from another source). When I start doing something (like "cat >> /proc/interrupts", or write to file), many more messages are lost. >> >> Do you have any idea how to fix this? I need to make some CAN messages >> logger and I cannot lose any message (idealy :) So I made big buffer in >> my program so that I don't need to write the messages into the file >> while "recording" is enabled, after "recording" is switched off, I write >> the buffer into the file, but that is still not good enough. >> Is there any way how to write to some buffer directly in flexcan driver >> (the best would be in the IRQ routine) and then read messages from this >> buffer in my program? >> Or are just interrupts lost when doing something else in the system and >> I cannot fix this? Or can I somehow specify that "can rx interrupts" has >> highest prioroty? > > Do you already know where the packets are dropped (get lost). Maybe your > user space app is simply not faster enough to process them in time. You > can check that with the candump option "-d". Hello Wolfgang, I don't know where it gets lost. But I'm just making select() and recvmmsg in my app now. Now when I'm trying candump -d, it shows messages and between that: can0 18FFC33D [8] 4A E6 00 00 03 00 00 F6 can0 1CEFB8F3 [8] 34 42 23 3D 8C 01 00 00 DROPCOUNT: dropped 245 CAN frames on 'can0' socket (total drops 30886) can0 18FF0AEF [8] 70 76 46 7C A7 9E 2A 6A can0 18FF0AF0 [8] B0 7E FB 7D A7 DC 27 69 can0 18FEBF0B [8] F5 03 79 81 69 74 6E 79 I've sent around 40 000 messages. Does this help somehow to identify where the problem is? I found out that if I (in my program) don't printf every frame, then dropcount is better, so I think when candump would only show dropps and not every frame then it would also get better. This is my /proc/interrupts root@vmx25 /opt$ cat /proc/interrupts CPU0 0: 0 - spi_imx 5: 32827 - IMX-uart 9: 129 - sdhci 13: 0 - spi_imx 25: 5 - imxdi_rtc 33: 12168 - mxc_nand 34: 1 - mxc-sdma 37: 109275 - ehci_hcd:usb1 43: 1624971 - can0 46: 0 - mxcadc 54: 195357 - i.MX Timer Tick 164: 0 - ESDHCI card 0 detect Err: 0 I have another problem - when I send messages and nobody is acking them (no other device on CANbus), system stops responding until I put another device onto CANbus, but I think I should start new thread with this. Martin > Wolfgang. > -- > To unsubscribe from this list: send the line "unsubscribe linux-can" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: CAN messages being lost on i.MX25 with flexcan 2012-04-19 11:21 ` Martin Kozusky @ 2012-04-19 11:44 ` Wolfgang Grandegger 2012-04-19 11:58 ` Martin Kozusky 0 siblings, 1 reply; 12+ messages in thread From: Wolfgang Grandegger @ 2012-04-19 11:44 UTC (permalink / raw) To: Martin Kozusky; +Cc: linux-can On 04/19/2012 01:21 PM, Martin Kozusky wrote: > Dne 19.4.2012 12:54, Wolfgang Grandegger napsal(a): >> On 04/19/2012 12:04 PM, Martin Kozusky wrote: >>> Hello, >>> I'm using Voipac i.MX25 module with flexcan, kernel 2.6.38.9. >>> I'm sending the data at 250kbps, around 1100 msgs/sec. When I enable the >>> canbus interface (canconfig can0 start), CPU load is higher, that is >>> understandable, there are many interrupts. I'm not doing anything else >>> then using recvmsg (or recvmmsg which is little better), but some >>> messages are still lost (around 1500 messages lost from 467 000 being >>> send from another source). When I start doing something (like "cat >>> /proc/interrupts", or write to file), many more messages are lost. >>> >>> Do you have any idea how to fix this? I need to make some CAN messages >>> logger and I cannot lose any message (idealy :) So I made big buffer in >>> my program so that I don't need to write the messages into the file >>> while "recording" is enabled, after "recording" is switched off, I write >>> the buffer into the file, but that is still not good enough. >>> Is there any way how to write to some buffer directly in flexcan driver >>> (the best would be in the IRQ routine) and then read messages from this >>> buffer in my program? >>> Or are just interrupts lost when doing something else in the system and >>> I cannot fix this? Or can I somehow specify that "can rx interrupts" has >>> highest prioroty? >> >> Do you already know where the packets are dropped (get lost). Maybe your >> user space app is simply not faster enough to process them in time. You >> can check that with the candump option "-d". > > Hello Wolfgang, > I don't know where it gets lost. But I'm just making select() and > recvmmsg in my app now. > Now when I'm trying candump -d, it shows messages and between that: > > can0 18FFC33D [8] 4A E6 00 00 03 00 00 F6 > can0 1CEFB8F3 [8] 34 42 23 3D 8C 01 00 00 > DROPCOUNT: dropped 245 CAN frames on 'can0' socket (total drops 30886) > can0 18FF0AEF [8] 70 76 46 7C A7 9E 2A 6A > can0 18FF0AF0 [8] B0 7E FB 7D A7 DC 27 69 > can0 18FEBF0B [8] F5 03 79 81 69 74 6E 79 > > I've sent around 40 000 messages. > Does this help somehow to identify where the problem is? Yes, you are loosing messages because your app is not faster enough. > I found out that if I (in my program) don't printf every frame, then > dropcount is better, so I think when candump would only show dropps and > not every frame then it would also get better. printf takes time, of course. Therefore everything which makes your app reading the messages faster will help including increasing the scheduling priority of the process/thread. Also increasing the size of the receive buffer may help. Try candump with the "-r" option. > This is my /proc/interrupts > > root@vmx25 /opt$ cat /proc/interrupts > CPU0 > 0: 0 - spi_imx > 5: 32827 - IMX-uart > 9: 129 - sdhci > 13: 0 - spi_imx > 25: 5 - imxdi_rtc > 33: 12168 - mxc_nand > 34: 1 - mxc-sdma > 37: 109275 - ehci_hcd:usb1 > 43: 1624971 - can0 > 46: 0 - mxcadc > 54: 195357 - i.MX Timer Tick > 164: 0 - ESDHCI card 0 detect > Err: 0 In a first place, I don't think your are loosing messages because they are not read-out faster enough from the CAN controller. > I have another problem - when I send messages and nobody is acking them > (no other device on CANbus), system stops responding until I put another > device onto CANbus, but I think I should start new thread with this. What kernel version do you use? Do you use the flexcan driver from that kernel? The problem is bus errors coming at high rate. Wolfgang. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: CAN messages being lost on i.MX25 with flexcan 2012-04-19 11:44 ` Wolfgang Grandegger @ 2012-04-19 11:58 ` Martin Kozusky 2012-04-19 13:37 ` Wolfgang Grandegger 2012-04-19 20:00 ` Jesper Pedersen 0 siblings, 2 replies; 12+ messages in thread From: Martin Kozusky @ 2012-04-19 11:58 UTC (permalink / raw) To: linux-can Dne 19.4.2012 13:44, Wolfgang Grandegger napsal(a): > On 04/19/2012 01:21 PM, Martin Kozusky wrote: >> Dne 19.4.2012 12:54, Wolfgang Grandegger napsal(a): >>> On 04/19/2012 12:04 PM, Martin Kozusky wrote: >>>> Hello, >>>> I'm using Voipac i.MX25 module with flexcan, kernel 2.6.38.9. >>>> I'm sending the data at 250kbps, around 1100 msgs/sec. When I enable the >>>> canbus interface (canconfig can0 start), CPU load is higher, that is >>>> understandable, there are many interrupts. I'm not doing anything else >>>> then using recvmsg (or recvmmsg which is little better), but some >>>> messages are still lost (around 1500 messages lost from 467 000 being >>>> send from another source). When I start doing something (like "cat >>>> /proc/interrupts", or write to file), many more messages are lost. >>>> >>>> Do you have any idea how to fix this? I need to make some CAN messages >>>> logger and I cannot lose any message (idealy :) So I made big buffer in >>>> my program so that I don't need to write the messages into the file >>>> while "recording" is enabled, after "recording" is switched off, I write >>>> the buffer into the file, but that is still not good enough. >>>> Is there any way how to write to some buffer directly in flexcan driver >>>> (the best would be in the IRQ routine) and then read messages from this >>>> buffer in my program? >>>> Or are just interrupts lost when doing something else in the system and >>>> I cannot fix this? Or can I somehow specify that "can rx interrupts" has >>>> highest prioroty? >>> >>> Do you already know where the packets are dropped (get lost). Maybe your >>> user space app is simply not faster enough to process them in time. You >>> can check that with the candump option "-d". >> >> Hello Wolfgang, >> I don't know where it gets lost. But I'm just making select() and >> recvmmsg in my app now. >> Now when I'm trying candump -d, it shows messages and between that: >> >> can0 18FFC33D [8] 4A E6 00 00 03 00 00 F6 >> can0 1CEFB8F3 [8] 34 42 23 3D 8C 01 00 00 >> DROPCOUNT: dropped 245 CAN frames on 'can0' socket (total drops 30886) >> can0 18FF0AEF [8] 70 76 46 7C A7 9E 2A 6A >> can0 18FF0AF0 [8] B0 7E FB 7D A7 DC 27 69 >> can0 18FEBF0B [8] F5 03 79 81 69 74 6E 79 >> >> I've sent around 40 000 messages. >> Does this help somehow to identify where the problem is? > > Yes, you are loosing messages because your app is not faster enough. > >> I found out that if I (in my program) don't printf every frame, then >> dropcount is better, so I think when candump would only show dropps and >> not every frame then it would also get better. > > printf takes time, of course. Therefore everything which makes your app > reading the messages faster will help including increasing the > scheduling priority of the process/thread. Also increasing the size of > the receive buffer may help. Try candump with the "-r" option. I tried -r 10000000 and the result was that dropping started some time later then it would normaly. From 467 000 packets, total drops was 103064. That is better ratio than before. But few times, my system was stucked for a while, then it recovered and continued to work. So I will try to use this buffer size in my program, without printf and I will see how many packets I lose. Is there a way how to get actual used size of the buffer? >> This is my /proc/interrupts >> >> root@vmx25 /opt$ cat /proc/interrupts >> CPU0 >> 0: 0 - spi_imx >> 5: 32827 - IMX-uart >> 9: 129 - sdhci >> 13: 0 - spi_imx >> 25: 5 - imxdi_rtc >> 33: 12168 - mxc_nand >> 34: 1 - mxc-sdma >> 37: 109275 - ehci_hcd:usb1 >> 43: 1624971 - can0 >> 46: 0 - mxcadc >> 54: 195357 - i.MX Timer Tick >> 164: 0 - ESDHCI card 0 detect >> Err: 0 > > In a first place, I don't think your are loosing messages because they > are not read-out faster enough from the CAN controller. > >> I have another problem - when I send messages and nobody is acking them >> (no other device on CANbus), system stops responding until I put another >> device onto CANbus, but I think I should start new thread with this. > > What kernel version do you use? Do you use the flexcan driver from that > kernel? The problem is bus errors coming at high rate. I'm using 2.6.35.9 Preempt, the flexcan.c driver is original. Martin > Wolfgang. > -- > To unsubscribe from this list: send the line "unsubscribe linux-can" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: CAN messages being lost on i.MX25 with flexcan 2012-04-19 11:58 ` Martin Kozusky @ 2012-04-19 13:37 ` Wolfgang Grandegger 2012-04-20 5:51 ` Martin Kozusky 2012-04-19 20:00 ` Jesper Pedersen 1 sibling, 1 reply; 12+ messages in thread From: Wolfgang Grandegger @ 2012-04-19 13:37 UTC (permalink / raw) To: Martin Kozusky; +Cc: linux-can On 04/19/2012 01:58 PM, Martin Kozusky wrote: > Dne 19.4.2012 13:44, Wolfgang Grandegger napsal(a): >> On 04/19/2012 01:21 PM, Martin Kozusky wrote: >>> Dne 19.4.2012 12:54, Wolfgang Grandegger napsal(a): >>>> On 04/19/2012 12:04 PM, Martin Kozusky wrote: >>>>> Hello, >>>>> I'm using Voipac i.MX25 module with flexcan, kernel 2.6.38.9. >>>>> I'm sending the data at 250kbps, around 1100 msgs/sec. When I >>>>> enable the >>>>> canbus interface (canconfig can0 start), CPU load is higher, that is >>>>> understandable, there are many interrupts. I'm not doing anything else >>>>> then using recvmsg (or recvmmsg which is little better), but some >>>>> messages are still lost (around 1500 messages lost from 467 000 being >>>>> send from another source). When I start doing something (like "cat >>>>> /proc/interrupts", or write to file), many more messages are lost. >>>>> >>>>> Do you have any idea how to fix this? I need to make some CAN messages >>>>> logger and I cannot lose any message (idealy :) So I made big >>>>> buffer in >>>>> my program so that I don't need to write the messages into the file >>>>> while "recording" is enabled, after "recording" is switched off, I >>>>> write >>>>> the buffer into the file, but that is still not good enough. >>>>> Is there any way how to write to some buffer directly in flexcan >>>>> driver >>>>> (the best would be in the IRQ routine) and then read messages from >>>>> this >>>>> buffer in my program? >>>>> Or are just interrupts lost when doing something else in the system >>>>> and >>>>> I cannot fix this? Or can I somehow specify that "can rx >>>>> interrupts" has >>>>> highest prioroty? >>>> >>>> Do you already know where the packets are dropped (get lost). Maybe >>>> your >>>> user space app is simply not faster enough to process them in time. You >>>> can check that with the candump option "-d". >>> >>> Hello Wolfgang, >>> I don't know where it gets lost. But I'm just making select() and >>> recvmmsg in my app now. >>> Now when I'm trying candump -d, it shows messages and between that: >>> >>> can0 18FFC33D [8] 4A E6 00 00 03 00 00 F6 >>> can0 1CEFB8F3 [8] 34 42 23 3D 8C 01 00 00 >>> DROPCOUNT: dropped 245 CAN frames on 'can0' socket (total drops 30886) >>> can0 18FF0AEF [8] 70 76 46 7C A7 9E 2A 6A >>> can0 18FF0AF0 [8] B0 7E FB 7D A7 DC 27 69 >>> can0 18FEBF0B [8] F5 03 79 81 69 74 6E 79 >>> >>> I've sent around 40 000 messages. >>> Does this help somehow to identify where the problem is? >> >> Yes, you are loosing messages because your app is not faster enough. >> >>> I found out that if I (in my program) don't printf every frame, then >>> dropcount is better, so I think when candump would only show dropps and >>> not every frame then it would also get better. >> >> printf takes time, of course. Therefore everything which makes your app >> reading the messages faster will help including increasing the >> scheduling priority of the process/thread. Also increasing the size of >> the receive buffer may help. Try candump with the "-r" option. > > I tried -r 10000000 > > and the result was that dropping started some time later then it would > normaly. From 467 000 packets, total drops was 103064. That is better > ratio than before. But few times, my system was stucked for a while, > then it recovered and continued to work. So I will try to use this > buffer size in my program, without printf and I will see how many > packets I lose. If other activities delay your app too much, your are in trouble. Proper thread priorities may help. > Is there a way how to get actual used size of the buffer? getsockopt? >>> This is my /proc/interrupts >>> >>> root@vmx25 /opt$ cat /proc/interrupts >>> CPU0 >>> 0: 0 - spi_imx >>> 5: 32827 - IMX-uart >>> 9: 129 - sdhci >>> 13: 0 - spi_imx >>> 25: 5 - imxdi_rtc >>> 33: 12168 - mxc_nand >>> 34: 1 - mxc-sdma >>> 37: 109275 - ehci_hcd:usb1 >>> 43: 1624971 - can0 >>> 46: 0 - mxcadc >>> 54: 195357 - i.MX Timer Tick >>> 164: 0 - ESDHCI card 0 detect >>> Err: 0 >> >> In a first place, I don't think your are loosing messages because they >> are not read-out faster enough from the CAN controller. >> >>> I have another problem - when I send messages and nobody is acking them >>> (no other device on CANbus), system stops responding until I put another >>> device onto CANbus, but I think I should start new thread with this. >> >> What kernel version do you use? Do you use the flexcan driver from that >> kernel? The problem is bus errors coming at high rate. > > I'm using 2.6.35.9 Preempt, the flexcan.c driver is original. IIRC, that kernel version did not include the flexcan driver. It entered with 2.6.36. Anyway, maybe the following patch helps: http://patchwork.ozlabs.org/patch/130761/ Wolfgang. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: CAN messages being lost on i.MX25 with flexcan 2012-04-19 13:37 ` Wolfgang Grandegger @ 2012-04-20 5:51 ` Martin Kozusky 2012-04-20 6:04 ` Wolfgang Grandegger 0 siblings, 1 reply; 12+ messages in thread From: Martin Kozusky @ 2012-04-20 5:51 UTC (permalink / raw) To: linux-can Dne 19.4.2012 15:37, Wolfgang Grandegger napsal(a): > On 04/19/2012 01:58 PM, Martin Kozusky wrote: >> Dne 19.4.2012 13:44, Wolfgang Grandegger napsal(a): >>> On 04/19/2012 01:21 PM, Martin Kozusky wrote: >>>> Dne 19.4.2012 12:54, Wolfgang Grandegger napsal(a): >>>>> On 04/19/2012 12:04 PM, Martin Kozusky wrote: >>>>>> Hello, >>>>>> I'm using Voipac i.MX25 module with flexcan, kernel 2.6.38.9. >>>>>> I'm sending the data at 250kbps, around 1100 msgs/sec. When I >>>>>> enable the >>>>>> canbus interface (canconfig can0 start), CPU load is higher, that is >>>>>> understandable, there are many interrupts. I'm not doing anything else >>>>>> then using recvmsg (or recvmmsg which is little better), but some >>>>>> messages are still lost (around 1500 messages lost from 467 000 being >>>>>> send from another source). When I start doing something (like "cat >>>>>> /proc/interrupts", or write to file), many more messages are lost. >>>>>> >>>>>> Do you have any idea how to fix this? I need to make some CAN messages >>>>>> logger and I cannot lose any message (idealy :) So I made big >>>>>> buffer in >>>>>> my program so that I don't need to write the messages into the file >>>>>> while "recording" is enabled, after "recording" is switched off, I >>>>>> write >>>>>> the buffer into the file, but that is still not good enough. >>>>>> Is there any way how to write to some buffer directly in flexcan >>>>>> driver >>>>>> (the best would be in the IRQ routine) and then read messages from >>>>>> this >>>>>> buffer in my program? >>>>>> Or are just interrupts lost when doing something else in the system >>>>>> and >>>>>> I cannot fix this? Or can I somehow specify that "can rx >>>>>> interrupts" has >>>>>> highest prioroty? >>>>> >>>>> Do you already know where the packets are dropped (get lost). Maybe >>>>> your >>>>> user space app is simply not faster enough to process them in time. You >>>>> can check that with the candump option "-d". >>>> >>>> Hello Wolfgang, >>>> I don't know where it gets lost. But I'm just making select() and >>>> recvmmsg in my app now. >>>> Now when I'm trying candump -d, it shows messages and between that: >>>> >>>> can0 18FFC33D [8] 4A E6 00 00 03 00 00 F6 >>>> can0 1CEFB8F3 [8] 34 42 23 3D 8C 01 00 00 >>>> DROPCOUNT: dropped 245 CAN frames on 'can0' socket (total drops 30886) >>>> can0 18FF0AEF [8] 70 76 46 7C A7 9E 2A 6A >>>> can0 18FF0AF0 [8] B0 7E FB 7D A7 DC 27 69 >>>> can0 18FEBF0B [8] F5 03 79 81 69 74 6E 79 >>>> >>>> I've sent around 40 000 messages. >>>> Does this help somehow to identify where the problem is? >>> >>> Yes, you are loosing messages because your app is not faster enough. >>> >>>> I found out that if I (in my program) don't printf every frame, then >>>> dropcount is better, so I think when candump would only show dropps and >>>> not every frame then it would also get better. >>> >>> printf takes time, of course. Therefore everything which makes your app >>> reading the messages faster will help including increasing the >>> scheduling priority of the process/thread. Also increasing the size of >>> the receive buffer may help. Try candump with the "-r" option. >> >> I tried -r 10000000 >> >> and the result was that dropping started some time later then it would >> normaly. From 467 000 packets, total drops was 103064. That is better >> ratio than before. But few times, my system was stucked for a while, >> then it recovered and continued to work. So I will try to use this >> buffer size in my program, without printf and I will see how many >> packets I lose. > > If other activities delay your app too much, your are in trouble. Proper > thread priorities may help. So I tried my app with bigger receive buffer and saving data into memory and it looks like no messages are lost now :) (but when I write them into file, they are lost, but I can use the temp memory storage, so I can live with that :) >> Is there a way how to get actual used size of the buffer? > > getsockopt? > >>>> This is my /proc/interrupts >>>> >>>> root@vmx25 /opt$ cat /proc/interrupts >>>> CPU0 >>>> 0: 0 - spi_imx >>>> 5: 32827 - IMX-uart >>>> 9: 129 - sdhci >>>> 13: 0 - spi_imx >>>> 25: 5 - imxdi_rtc >>>> 33: 12168 - mxc_nand >>>> 34: 1 - mxc-sdma >>>> 37: 109275 - ehci_hcd:usb1 >>>> 43: 1624971 - can0 >>>> 46: 0 - mxcadc >>>> 54: 195357 - i.MX Timer Tick >>>> 164: 0 - ESDHCI card 0 detect >>>> Err: 0 >>> >>> In a first place, I don't think your are loosing messages because they >>> are not read-out faster enough from the CAN controller. >>> >>>> I have another problem - when I send messages and nobody is acking them >>>> (no other device on CANbus), system stops responding until I put another >>>> device onto CANbus, but I think I should start new thread with this. >>> >>> What kernel version do you use? Do you use the flexcan driver from that >>> kernel? The problem is bus errors coming at high rate. >> >> I'm using 2.6.35.9 Preempt, the flexcan.c driver is original. > > IIRC, that kernel version did not include the flexcan driver. It entered > with 2.6.36. Anyway, maybe the following patch helps: > > http://patchwork.ozlabs.org/patch/130761/ It seems that it helped :) when nobody is ACKing, it doesn't hang. Thank you, Martin > Wolfgang. > -- > To unsubscribe from this list: send the line "unsubscribe linux-can" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: CAN messages being lost on i.MX25 with flexcan 2012-04-20 5:51 ` Martin Kozusky @ 2012-04-20 6:04 ` Wolfgang Grandegger 2012-04-20 6:17 ` Martin Kozusky 0 siblings, 1 reply; 12+ messages in thread From: Wolfgang Grandegger @ 2012-04-20 6:04 UTC (permalink / raw) To: Martin Kozusky; +Cc: linux-can On 04/20/2012 07:51 AM, Martin Kozusky wrote: > Dne 19.4.2012 15:37, Wolfgang Grandegger napsal(a): >> On 04/19/2012 01:58 PM, Martin Kozusky wrote: >>> Dne 19.4.2012 13:44, Wolfgang Grandegger napsal(a): >>>> On 04/19/2012 01:21 PM, Martin Kozusky wrote: >>>>> Dne 19.4.2012 12:54, Wolfgang Grandegger napsal(a): >>>>>> On 04/19/2012 12:04 PM, Martin Kozusky wrote: >>>>>>> Hello, >>>>>>> I'm using Voipac i.MX25 module with flexcan, kernel 2.6.38.9. >>>>>>> I'm sending the data at 250kbps, around 1100 msgs/sec. When I >>>>>>> enable the >>>>>>> canbus interface (canconfig can0 start), CPU load is higher, that is >>>>>>> understandable, there are many interrupts. I'm not doing anything >>>>>>> else >>>>>>> then using recvmsg (or recvmmsg which is little better), but some >>>>>>> messages are still lost (around 1500 messages lost from 467 000 >>>>>>> being >>>>>>> send from another source). When I start doing something (like "cat >>>>>>> /proc/interrupts", or write to file), many more messages are lost. >>>>>>> >>>>>>> Do you have any idea how to fix this? I need to make some CAN >>>>>>> messages >>>>>>> logger and I cannot lose any message (idealy :) So I made big >>>>>>> buffer in >>>>>>> my program so that I don't need to write the messages into the file >>>>>>> while "recording" is enabled, after "recording" is switched off, I >>>>>>> write >>>>>>> the buffer into the file, but that is still not good enough. >>>>>>> Is there any way how to write to some buffer directly in flexcan >>>>>>> driver >>>>>>> (the best would be in the IRQ routine) and then read messages from >>>>>>> this >>>>>>> buffer in my program? >>>>>>> Or are just interrupts lost when doing something else in the system >>>>>>> and >>>>>>> I cannot fix this? Or can I somehow specify that "can rx >>>>>>> interrupts" has >>>>>>> highest prioroty? >>>>>> >>>>>> Do you already know where the packets are dropped (get lost). Maybe >>>>>> your >>>>>> user space app is simply not faster enough to process them in >>>>>> time. You >>>>>> can check that with the candump option "-d". >>>>> >>>>> Hello Wolfgang, >>>>> I don't know where it gets lost. But I'm just making select() and >>>>> recvmmsg in my app now. >>>>> Now when I'm trying candump -d, it shows messages and between that: >>>>> >>>>> can0 18FFC33D [8] 4A E6 00 00 03 00 00 F6 >>>>> can0 1CEFB8F3 [8] 34 42 23 3D 8C 01 00 00 >>>>> DROPCOUNT: dropped 245 CAN frames on 'can0' socket (total drops 30886) >>>>> can0 18FF0AEF [8] 70 76 46 7C A7 9E 2A 6A >>>>> can0 18FF0AF0 [8] B0 7E FB 7D A7 DC 27 69 >>>>> can0 18FEBF0B [8] F5 03 79 81 69 74 6E 79 >>>>> >>>>> I've sent around 40 000 messages. >>>>> Does this help somehow to identify where the problem is? >>>> >>>> Yes, you are loosing messages because your app is not faster enough. >>>> >>>>> I found out that if I (in my program) don't printf every frame, then >>>>> dropcount is better, so I think when candump would only show dropps >>>>> and >>>>> not every frame then it would also get better. >>>> >>>> printf takes time, of course. Therefore everything which makes your app >>>> reading the messages faster will help including increasing the >>>> scheduling priority of the process/thread. Also increasing the size of >>>> the receive buffer may help. Try candump with the "-r" option. >>> >>> I tried -r 10000000 >>> >>> and the result was that dropping started some time later then it would >>> normaly. From 467 000 packets, total drops was 103064. That is better >>> ratio than before. But few times, my system was stucked for a while, >>> then it recovered and continued to work. So I will try to use this >>> buffer size in my program, without printf and I will see how many >>> packets I lose. >> >> If other activities delay your app too much, your are in trouble. Proper >> thread priorities may help. > > So I tried my app with bigger receive buffer and saving data into memory > and it looks like no messages are lost now :) (but when I write them > into file, they are lost, but I can use the temp memory storage, so I > can live with that :) You are obviously hitting the resource limits. Do you need -rt? It makes your system slower. >>> Is there a way how to get actual used size of the buffer? >> >> getsockopt? >> >>>>> This is my /proc/interrupts >>>>> >>>>> root@vmx25 /opt$ cat /proc/interrupts >>>>> CPU0 >>>>> 0: 0 - spi_imx >>>>> 5: 32827 - IMX-uart >>>>> 9: 129 - sdhci >>>>> 13: 0 - spi_imx >>>>> 25: 5 - imxdi_rtc >>>>> 33: 12168 - mxc_nand >>>>> 34: 1 - mxc-sdma >>>>> 37: 109275 - ehci_hcd:usb1 >>>>> 43: 1624971 - can0 >>>>> 46: 0 - mxcadc >>>>> 54: 195357 - i.MX Timer Tick >>>>> 164: 0 - ESDHCI card 0 detect >>>>> Err: 0 >>>> >>>> In a first place, I don't think your are loosing messages because they >>>> are not read-out faster enough from the CAN controller. >>>> >>>>> I have another problem - when I send messages and nobody is acking >>>>> them >>>>> (no other device on CANbus), system stops responding until I put >>>>> another >>>>> device onto CANbus, but I think I should start new thread with this. >>>> >>>> What kernel version do you use? Do you use the flexcan driver from that >>>> kernel? The problem is bus errors coming at high rate. >>> >>> I'm using 2.6.35.9 Preempt, the flexcan.c driver is original. >> >> IIRC, that kernel version did not include the flexcan driver. It entered >> with 2.6.36. Anyway, maybe the following patch helps: >> >> http://patchwork.ozlabs.org/patch/130761/ > > It seems that it helped :) when nobody is ACKing, it doesn't hang. Good. What does the CAN statistics show ($ ip -d -s link show can0)? Wolfgang. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: CAN messages being lost on i.MX25 with flexcan 2012-04-20 6:04 ` Wolfgang Grandegger @ 2012-04-20 6:17 ` Martin Kozusky 2012-04-20 7:59 ` Wolfgang Grandegger 0 siblings, 1 reply; 12+ messages in thread From: Martin Kozusky @ 2012-04-20 6:17 UTC (permalink / raw) To: linux-can Dne 20.4.2012 8:04, Wolfgang Grandegger napsal(a): > On 04/20/2012 07:51 AM, Martin Kozusky wrote: >> Dne 19.4.2012 15:37, Wolfgang Grandegger napsal(a): >>> On 04/19/2012 01:58 PM, Martin Kozusky wrote: >>>> Dne 19.4.2012 13:44, Wolfgang Grandegger napsal(a): >>>>> On 04/19/2012 01:21 PM, Martin Kozusky wrote: >>>>>> Dne 19.4.2012 12:54, Wolfgang Grandegger napsal(a): >>>>>>> On 04/19/2012 12:04 PM, Martin Kozusky wrote: >>>>>>>> Hello, >>>>>>>> I'm using Voipac i.MX25 module with flexcan, kernel 2.6.38.9. >>>>>>>> I'm sending the data at 250kbps, around 1100 msgs/sec. When I >>>>>>>> enable the >>>>>>>> canbus interface (canconfig can0 start), CPU load is higher, that is >>>>>>>> understandable, there are many interrupts. I'm not doing anything >>>>>>>> else >>>>>>>> then using recvmsg (or recvmmsg which is little better), but some >>>>>>>> messages are still lost (around 1500 messages lost from 467 000 >>>>>>>> being >>>>>>>> send from another source). When I start doing something (like "cat >>>>>>>> /proc/interrupts", or write to file), many more messages are lost. >>>>>>>> >>>>>>>> Do you have any idea how to fix this? I need to make some CAN >>>>>>>> messages >>>>>>>> logger and I cannot lose any message (idealy :) So I made big >>>>>>>> buffer in >>>>>>>> my program so that I don't need to write the messages into the file >>>>>>>> while "recording" is enabled, after "recording" is switched off, I >>>>>>>> write >>>>>>>> the buffer into the file, but that is still not good enough. >>>>>>>> Is there any way how to write to some buffer directly in flexcan >>>>>>>> driver >>>>>>>> (the best would be in the IRQ routine) and then read messages from >>>>>>>> this >>>>>>>> buffer in my program? >>>>>>>> Or are just interrupts lost when doing something else in the system >>>>>>>> and >>>>>>>> I cannot fix this? Or can I somehow specify that "can rx >>>>>>>> interrupts" has >>>>>>>> highest prioroty? >>>>>>> >>>>>>> Do you already know where the packets are dropped (get lost). Maybe >>>>>>> your >>>>>>> user space app is simply not faster enough to process them in >>>>>>> time. You >>>>>>> can check that with the candump option "-d". >>>>>> >>>>>> Hello Wolfgang, >>>>>> I don't know where it gets lost. But I'm just making select() and >>>>>> recvmmsg in my app now. >>>>>> Now when I'm trying candump -d, it shows messages and between that: >>>>>> >>>>>> can0 18FFC33D [8] 4A E6 00 00 03 00 00 F6 >>>>>> can0 1CEFB8F3 [8] 34 42 23 3D 8C 01 00 00 >>>>>> DROPCOUNT: dropped 245 CAN frames on 'can0' socket (total drops 30886) >>>>>> can0 18FF0AEF [8] 70 76 46 7C A7 9E 2A 6A >>>>>> can0 18FF0AF0 [8] B0 7E FB 7D A7 DC 27 69 >>>>>> can0 18FEBF0B [8] F5 03 79 81 69 74 6E 79 >>>>>> >>>>>> I've sent around 40 000 messages. >>>>>> Does this help somehow to identify where the problem is? >>>>> >>>>> Yes, you are loosing messages because your app is not faster enough. >>>>> >>>>>> I found out that if I (in my program) don't printf every frame, then >>>>>> dropcount is better, so I think when candump would only show dropps >>>>>> and >>>>>> not every frame then it would also get better. >>>>> >>>>> printf takes time, of course. Therefore everything which makes your app >>>>> reading the messages faster will help including increasing the >>>>> scheduling priority of the process/thread. Also increasing the size of >>>>> the receive buffer may help. Try candump with the "-r" option. >>>> >>>> I tried -r 10000000 >>>> >>>> and the result was that dropping started some time later then it would >>>> normaly. From 467 000 packets, total drops was 103064. That is better >>>> ratio than before. But few times, my system was stucked for a while, >>>> then it recovered and continued to work. So I will try to use this >>>> buffer size in my program, without printf and I will see how many >>>> packets I lose. >>> >>> If other activities delay your app too much, your are in trouble. Proper >>> thread priorities may help. >> >> So I tried my app with bigger receive buffer and saving data into memory >> and it looks like no messages are lost now :) (but when I write them >> into file, they are lost, but I can use the temp memory storage, so I >> can live with that :) > > You are obviously hitting the resource limits. Do you need -rt? It makes > your system slower. I don't know what steps are necessary to make the kernel (and can driver) -rt :( I will first try to finish my app and see how it behaves. > >>>> Is there a way how to get actual used size of the buffer? >>> >>> getsockopt? >>> >>>>>> This is my /proc/interrupts >>>>>> >>>>>> root@vmx25 /opt$ cat /proc/interrupts >>>>>> CPU0 >>>>>> 0: 0 - spi_imx >>>>>> 5: 32827 - IMX-uart >>>>>> 9: 129 - sdhci >>>>>> 13: 0 - spi_imx >>>>>> 25: 5 - imxdi_rtc >>>>>> 33: 12168 - mxc_nand >>>>>> 34: 1 - mxc-sdma >>>>>> 37: 109275 - ehci_hcd:usb1 >>>>>> 43: 1624971 - can0 >>>>>> 46: 0 - mxcadc >>>>>> 54: 195357 - i.MX Timer Tick >>>>>> 164: 0 - ESDHCI card 0 detect >>>>>> Err: 0 >>>>> >>>>> In a first place, I don't think your are loosing messages because they >>>>> are not read-out faster enough from the CAN controller. >>>>> >>>>>> I have another problem - when I send messages and nobody is acking >>>>>> them >>>>>> (no other device on CANbus), system stops responding until I put >>>>>> another >>>>>> device onto CANbus, but I think I should start new thread with this. >>>>> >>>>> What kernel version do you use? Do you use the flexcan driver from that >>>>> kernel? The problem is bus errors coming at high rate. >>>> >>>> I'm using 2.6.35.9 Preempt, the flexcan.c driver is original. >>> >>> IIRC, that kernel version did not include the flexcan driver. It entered >>> with 2.6.36. Anyway, maybe the following patch helps: >>> >>> http://patchwork.ozlabs.org/patch/130761/ >> >> It seems that it helped :) when nobody is ACKing, it doesn't hang. > > Good. What does the CAN statistics show ($ ip -d -s link show can0)? Now it shows (after some time of experimenting, it was disconnected from canbus for a while) 2: can0: <NOARP,ECHO> mtu 16 qdisc pfifo_fast state DOWN mode DEFAULT qlen 10 link/can can state STOPPED (berr-counter tx 0 rx 0) restart-ms 0 bitrate 250000 sample-point 0.857 tq 285 prop-seg 5 phase-seg1 6 phase-seg2 2 sjw 1 flexcan: tseg1 4..16 tseg2 2..8 sjw 1..4 brp 1..256 brp-inc 1 clock 66500000 re-started bus-errors arbit-lost error-warn error-pass bus-off 0 0 0 1 0 0 RX: bytes packets errors dropped overrun mcast 4758311 602831 212 0 212 0 TX: bytes packets errors dropped carrier collsns 73 73 0 0 0 0 Martin > Wolfgang. > > > -- > To unsubscribe from this list: send the line "unsubscribe linux-can" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: CAN messages being lost on i.MX25 with flexcan 2012-04-20 6:17 ` Martin Kozusky @ 2012-04-20 7:59 ` Wolfgang Grandegger 0 siblings, 0 replies; 12+ messages in thread From: Wolfgang Grandegger @ 2012-04-20 7:59 UTC (permalink / raw) To: Martin Kozusky; +Cc: linux-can On 04/20/2012 08:17 AM, Martin Kozusky wrote: > Dne 20.4.2012 8:04, Wolfgang Grandegger napsal(a): ... >> You are obviously hitting the resource limits. Do you need -rt? It makes >> your system slower. > > I don't know what steps are necessary to make the kernel (and can > driver) -rt :( I will first try to finish my app and see how it behaves. Forget my comment. I was thinking that you are already using a -rt kernel (with the real-time preemption patch), but that's obviously not the case. Sorry for the noise. ... >> Good. What does the CAN statistics show ($ ip -d -s link show can0)? > > Now it shows (after some time of experimenting, it was disconnected from > canbus for a while) > > 2: can0: <NOARP,ECHO> mtu 16 qdisc pfifo_fast state DOWN mode DEFAULT > qlen 10 > link/can > can state STOPPED (berr-counter tx 0 rx 0) restart-ms 0 > bitrate 250000 sample-point 0.857 > tq 285 prop-seg 5 phase-seg1 6 phase-seg2 2 sjw 1 > flexcan: tseg1 4..16 tseg2 2..8 sjw 1..4 brp 1..256 brp-inc 1 > clock 66500000 > re-started bus-errors arbit-lost error-warn error-pass bus-off > 0 0 0 1 0 0 > RX: bytes packets errors dropped overrun mcast > 4758311 602831 212 0 212 0 There have been some RX FIFO overflow errors, which means messages are lost by the hardware. Wolfgang. ^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: CAN messages being lost on i.MX25 with flexcan 2012-04-19 11:58 ` Martin Kozusky 2012-04-19 13:37 ` Wolfgang Grandegger @ 2012-04-19 20:00 ` Jesper Pedersen 2012-04-19 20:17 ` Wolfgang Grandegger 1 sibling, 1 reply; 12+ messages in thread From: Jesper Pedersen @ 2012-04-19 20:00 UTC (permalink / raw) To: Martin Kozusky, linux-can >> What kernel version do you use? Do you use the flexcan driver from >> that kernel? The problem is bus errors coming at high rate. > > I'm using 2.6.35.9 Preempt, the flexcan.c driver is original. We use flexcan on an iMX25 device. The vanilla flexcan driver (we use kernel 3.0, but same issue also seen on 2.6.37) has some serious bugs that occasionally caused high CPU utilization or even total crashes on our platform. If I remember correctly, I found some IRQs which were not serviced correctly, causing the ISR to be triggered continuously. We fixed it by pulling the flexcan.c driver from Wolfgangs linux-can-next tree. I don't recall, if you could just replace the original flexcan.c directly, or if some integration were required. Jesper Pedersen, Inntrasys ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: CAN messages being lost on i.MX25 with flexcan 2012-04-19 20:00 ` Jesper Pedersen @ 2012-04-19 20:17 ` Wolfgang Grandegger 0 siblings, 0 replies; 12+ messages in thread From: Wolfgang Grandegger @ 2012-04-19 20:17 UTC (permalink / raw) To: Jesper Pedersen; +Cc: Martin Kozusky, linux-can On 04/19/2012 10:00 PM, Jesper Pedersen wrote: >>> What kernel version do you use? Do you use the flexcan driver from >>> that kernel? The problem is bus errors coming at high rate. >> >> I'm using 2.6.35.9 Preempt, the flexcan.c driver is original. > > We use flexcan on an iMX25 device. The vanilla flexcan driver (we use kernel 3.0, but same issue also seen on 2.6.37) has some serious bugs that occasionally caused high CPU utilization or even total crashes on our platform. > If I remember correctly, I found some IRQs which were not serviced correctly, causing the ISR to be triggered continuously. > > We fixed it by pulling the flexcan.c driver from Wolfgangs linux-can-next tree. I don't recall, if you could just replace the original flexcan.c directly, or if some integration were required. This confirms that the patch http://patchwork.ozlabs.org/patch/130761/ is need for the i.MX25 as well. I already told Martin to give it a try. Wolfgang. ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2012-04-20 7:59 UTC | newest] Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2012-04-19 10:04 CAN messages being lost on i.MX25 with flexcan Martin Kozusky 2012-04-19 10:54 ` Wolfgang Grandegger 2012-04-19 11:21 ` Martin Kozusky 2012-04-19 11:44 ` Wolfgang Grandegger 2012-04-19 11:58 ` Martin Kozusky 2012-04-19 13:37 ` Wolfgang Grandegger 2012-04-20 5:51 ` Martin Kozusky 2012-04-20 6:04 ` Wolfgang Grandegger 2012-04-20 6:17 ` Martin Kozusky 2012-04-20 7:59 ` Wolfgang Grandegger 2012-04-19 20:00 ` Jesper Pedersen 2012-04-19 20:17 ` Wolfgang Grandegger
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.