Bus off

Bus off

[Austin Schuh]

Does anyone know if the TX queues are cleared on BUS-OFF?

I have a machine connected to a network where all the other CAN nodes
shut off periodically.  I monitor whether or not they are sending, and
don't send messages unless I think they are on.  Right after they shut
off, I occasionally send out a message when they are off, which
triggers a bus-off.  I'm seeing lots of bus-off/restarts due to
restart-ms=1000 after the nodes shut down, and I'm pretty certain that
I'm not sending anything at that time.  Any ideas on what is going on
would be appreciated.

Thanks,
    Austin

Re: Bus off

[Max S.]

When there is only one node. The lone node will continuously attempt to
transmit.

the tx error counter will increase and the node will move from
error_active to error_passive, and then to bus_off. At least one node I
know of will leave bus_off when it receives 128x11 consecutive dominant
bits and ten try to send again.

If the controller will wait for linux to restart it, it will probably
retry transmitting the frame upon being restart.

Restart does not mean reset as far as i know.

Regardless of what state the linux queue is in the frame in the
controllers outgoing message objects will continue to try to send,
unless explicitly told to cancel transmission. (at least on the
controllers i know. not that many :/)

Hence you will see the cycling error frames.

typically heartbeat frames are used to tell if a node exists or not.
each node send a predefined message at a desired interval, and you just
listen.

Regards,
Max S.

On Thu, 2014-06-26 at 10:13 -0700, Austin Schuh wrote:
> Does anyone know if the TX queues are cleared on BUS-OFF?
> 
> I have a machine connected to a network where all the other CAN nodes
> shut off periodically.  I monitor whether or not they are sending, and
> don't send messages unless I think they are on.  Right after they shut
> off, I occasionally send out a message when they are off, which
> triggers a bus-off.  I'm seeing lots of bus-off/restarts due to
> restart-ms=1000 after the nodes shut down, and I'm pretty certain that
> I'm not sending anything at that time.  Any ideas on what is going on
> would be appreciated.
> 
> Thanks,
>     Austin
> --
> To unsubscribe from this list: send the line "unsubscribe linux-can" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Bus off

[Kurt Van Dijck]

Hey,

On Thu, Jun 26, 2014 at 10:13:17AM -0700, Austin Schuh wrote:
> Does anyone know if the TX queues are cleared on BUS-OFF?

No queues are emptied, you will only loose the curent pending transmission.

> 
> I have a machine connected to a network where all the other CAN nodes
> shut off periodically.  I monitor whether or not they are sending, and
> don't send messages unless I think they are on.  Right after they shut
> off, I occasionally send out a message when they are off,

I assume this cannot be avoided due to timeouts.

> which triggers a bus-off.
> I'm seeing lots of bus-off/restarts due to
> restart-ms=1000 after the nodes shut down, and I'm pretty certain that
> I'm not sending anything at that time.  Any ideas on what is going on
> would be appreciated.

If you're not alone, a transmission is (normally) acked immediately
and the bus becomes idle again.

When you are the only active CAN chip, noone acks your message
and your transmission will be continuously repeated.
This makes you vulnerable for any bad bus condition.

If your bus conditions were good all the time, your would never
enter bus-off even if you transmit and are alone on the bus.

So, I _think_ that during shutdown or startup of your nodes, they
do (mayby shortly) bring a bad condition on the bus.
I would investigate the shutdown sequence of your nodes.
Some conditions I could think of/have experienced:
* after stopping the CAN chip, the CAN-TX pin returns being a GPIO
  and it's set as output.
* The CAN chip is stopped in the middle of transmission.
...

Kurt

Re: Bus off

[Marc Kleine-Budde]

[-- Attachment #1: Type: text/plain, Size: 809 bytes --]

On 06/27/2014 12:54 AM, Max S. wrote:
> When there is only one node. The lone node will continuously attempt to
> transmit.
> 
> the tx error counter will increase and the node will move from
> error_active to error_passive, and then to bus_off. At least one node I
> know of will leave bus_off when it receives 128x11 consecutive dominant
> bits and ten try to send again.

If the bus is properly terminated, a lone CAN node will not go into bus
off, though its error counters will increase until error passive.

Marc

-- 
Pengutronix e.K.                  | Marc Kleine-Budde           |
Industrial Linux Solutions        | Phone: +49-231-2826-924     |
Vertretung West/Dortmund          | Fax:   +49-5121-206917-5555 |
Amtsgericht Hildesheim, HRA 2686  | http://www.pengutronix.de   |


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 242 bytes --]

Re: Bus off

[Jason R1 White]

> 
> On Thu, Jun 26, 2014 at 10:13:17AM -0700, Austin Schuh wrote:
> > Does anyone know if the TX queues are cleared on BUS-OFF?
> 
> No queues are emptied, you will only loose the curent pending 
transmission.
> 
> > 

I thought the restart mechanism flushed the transmit queues (ip link set 
can0 type can restart).  Is this not the case?  That is what we are 
relying upon.  We used to close and open the interface explicitly. 

For controls its not a good practice to let stale data sit in the output 
buffer. 

Jason 

Re: Bus off

[Wolfgang Grandegger]

On 06/27/2014 05:22 PM, Jason R1 White wrote:
>>
>> On Thu, Jun 26, 2014 at 10:13:17AM -0700, Austin Schuh wrote:
>>> Does anyone know if the TX queues are cleared on BUS-OFF?
>>
>> No queues are emptied, you will only loose the curent pending 
> transmission.
>>
>>>
> 
> I thought the restart mechanism flushed the transmit queues (ip link set 
> can0 type can restart).  Is this not the case?  That is what we are 
> relying upon.  We used to close and open the interface explicitly. 


"restart" and "restart-ms" should trigger a bus-off recovery:

http://lxr.free-electrons.com/source/Documentation/networking/can.txt#L1087

> 
> For controls its not a good practice to let stale data sit in the output 
> buffer. 

The intention of restart(-ms) was to simply trigger a bus-off recovery.
The name is not well chosen, though.

Wolfgang.

Re: Bus off

[Jason R1 White]

So is it still the recommendation to close the interface to flush the 
queues?

Is there anything planned for an ip command or IOCTL to do this?

Jason


> On 06/27/2014 05:22 PM, Jason R1 White wrote:
> >>
> >> On Thu, Jun 26, 2014 at 10:13:17AM -0700, Austin Schuh wrote:
> >>> Does anyone know if the TX queues are cleared on BUS-OFF?
> >>
> >> No queues are emptied, you will only loose the curent pending 
> > transmission.
> >>
> >>>
> > 
> > I thought the restart mechanism flushed the transmit queues (ip link 
set 
> > can0 type can restart).  Is this not the case?  That is what we are 
> > relying upon.  We used to close and open the interface explicitly. 
> 
> 
> "restart" and "restart-ms" should trigger a bus-off recovery:
> 
> 
http://lxr.free-electrons.com/source/Documentation/networking/can.txt#L1087

> 
> > 
> > For controls its not a good practice to let stale data sit in the 
output 
> > buffer. 
> 
> The intention of restart(-ms) was to simply trigger a bus-off recovery.
> The name is not well chosen, though.
> 
> Wolfgang.

Re: Bus off

[Marc Kleine-Budde]

[-- Attachment #1: Type: text/plain, Size: 727 bytes --]

On 06/27/2014 05:45 PM, Jason R1 White wrote:
> So is it still the recommendation to close the interface to flush the 
> queues?
> 
> Is there anything planned for an ip command or IOCTL to do this?

I already thought some flushing/abort commands, abort the current CAN
frame and abort the tx queue in the controller and flush the whole tx
queue in linux. However we found no customer who actually wanted to
sponsor this development.

Marc
-- 
Pengutronix e.K.                  | Marc Kleine-Budde           |
Industrial Linux Solutions        | Phone: +49-231-2826-924     |
Vertretung West/Dortmund          | Fax:   +49-5121-206917-5555 |
Amtsgericht Hildesheim, HRA 2686  | http://www.pengutronix.de   |


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 242 bytes --]

Re: Bus off

[Wolfgang Grandegger]

On 06/27/2014 05:45 PM, Jason R1 White wrote:
> So is it still the recommendation to close the interface to flush the 
> queues?

As I said: the purpose of "restart" is not to do a fast if down->up.

> Is there anything planned for an ip command or IOCTL to do this?

It's on the wish list, let's say. Somebody needs to implement it. It's
also not straight-forward because there might be more than one socket
sending out CAN messages. For standard network devices there is no way
to flush all TX queues, IIRC.

Wolfgang.

>> On 06/27/2014 05:22 PM, Jason R1 White wrote:
>>>>
>>>> On Thu, Jun 26, 2014 at 10:13:17AM -0700, Austin Schuh wrote:
>>>>> Does anyone know if the TX queues are cleared on BUS-OFF?
>>>>
>>>> No queues are emptied, you will only loose the curent pending 
>>> transmission.
>>>>
>>>>>
>>>
>>> I thought the restart mechanism flushed the transmit queues (ip link 
> set 
>>> can0 type can restart).  Is this not the case?  That is what we are 
>>> relying upon.  We used to close and open the interface explicitly. 
>>
>>
>> "restart" and "restart-ms" should trigger a bus-off recovery:
>>
>>
> http://lxr.free-electrons.com/source/Documentation/networking/can.txt#L1087
> 
>>
>>>
>>> For controls its not a good practice to let stale data sit in the 
> output 
>>> buffer. 
>>
>> The intention of restart(-ms) was to simply trigger a bus-off recovery.
>> The name is not well chosen, though.
>>
>> Wolfgang.
> 
> 

Re: Bus off

[Oliver Hartkopp]

On 27.06.2014 18:28, Wolfgang Grandegger wrote:
> On 06/27/2014 05:45 PM, Jason R1 White wrote:
>> So is it still the recommendation to close the interface to flush the 
>> queues?
> 
> As I said: the purpose of "restart" is not to do a fast if down->up.
> 
>> Is there anything planned for an ip command or IOCTL to do this?
> 
> It's on the wish list, let's say. Somebody needs to implement it. It's
> also not straight-forward because there might be more than one socket
> sending out CAN messages. For standard network devices there is no way
> to flush all TX queues, IIRC.

Probably shutting down the netdevice at BUS_OFF would be an alternative to
restart-ms. When the netdevice is down all sockets are notified by -ENETDOWN.

We would need some kind of notification when frames got lost anyway. And after
-ENETDOWN the application can recover and set up it's communication again.

Regards,
Oliver

Re: Bus off

[Austin Schuh]

Thanks everyone!  Reading this thread has been very informative.  I
was hoping that the buffers would get flushed on BUS_OFF, but it
sounds like that is hard to do.

I'm not sure exactly what is going on with the machine in question,
and I don't really have physical access to sort it out.  The kernel
messages that I was getting now make a lot more sense.  I'll see if I
can change how I detect other nodes shutting down to fix that.

Austin

On Fri, Jun 27, 2014 at 10:34 AM, Oliver Hartkopp
<socketcan@hartkopp.net> wrote:
> On 27.06.2014 18:28, Wolfgang Grandegger wrote:
>> On 06/27/2014 05:45 PM, Jason R1 White wrote:
>>> So is it still the recommendation to close the interface to flush the
>>> queues?
>>
>> As I said: the purpose of "restart" is not to do a fast if down->up.
>>
>>> Is there anything planned for an ip command or IOCTL to do this?
>>
>> It's on the wish list, let's say. Somebody needs to implement it. It's
>> also not straight-forward because there might be more than one socket
>> sending out CAN messages. For standard network devices there is no way
>> to flush all TX queues, IIRC.
>
> Probably shutting down the netdevice at BUS_OFF would be an alternative to
> restart-ms. When the netdevice is down all sockets are notified by -ENETDOWN.
>
> We would need some kind of notification when frames got lost anyway. And after
> -ENETDOWN the application can recover and set up it's communication again.
>
> Regards,
> Oliver
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-can" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Bus off

[Wolfgang Grandegger]

On 06/28/2014 03:29 AM, Austin Schuh wrote:
> Thanks everyone!  Reading this thread has been very informative.  I
> was hoping that the buffers would get flushed on BUS_OFF, but it
> sounds like that is hard to do.
> 
> I'm not sure exactly what is going on with the machine in question,
> and I don't really have physical access to sort it out.  The kernel
> messages that I was getting now make a lot more sense.  I'll see if I
> can change how I detect other nodes shutting down to fix that.

You are welcome to show the kernel messages. After re-reading your first
mail of this thread I'm still puzzled what's going on. A node goes
bus-off in case of serious electrical problems on the bus, e.g. improper
termination, bitrate does not matches, etc. It does *not* go bus off
just because no other nodes are on the bus (ack slot error).

Wolfgang.

> On Fri, Jun 27, 2014 at 10:34 AM, Oliver Hartkopp
> <socketcan@hartkopp.net> wrote:
>> On 27.06.2014 18:28, Wolfgang Grandegger wrote:
>>> On 06/27/2014 05:45 PM, Jason R1 White wrote:
>>>> So is it still the recommendation to close the interface to flush the
>>>> queues?
>>>
>>> As I said: the purpose of "restart" is not to do a fast if down->up.
>>>
>>>> Is there anything planned for an ip command or IOCTL to do this?
>>>
>>> It's on the wish list, let's say. Somebody needs to implement it. It's
>>> also not straight-forward because there might be more than one socket
>>> sending out CAN messages. For standard network devices there is no way
>>> to flush all TX queues, IIRC.
>>
>> Probably shutting down the netdevice at BUS_OFF would be an alternative to
>> restart-ms. When the netdevice is down all sockets are notified by -ENETDOWN.
>>
>> We would need some kind of notification when frames got lost anyway. And after
>> -ENETDOWN the application can recover and set up it's communication again.
>>
>> Regards,
>> Oliver
>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-can" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-can" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: Bus off

[Austin Schuh]

On Sat, Jun 28, 2014 at 1:52 AM, Wolfgang Grandegger <wg@grandegger.com> wrote:
> On 06/28/2014 03:29 AM, Austin Schuh wrote:
>> Thanks everyone!  Reading this thread has been very informative.  I
>> was hoping that the buffers would get flushed on BUS_OFF, but it
>> sounds like that is hard to do.
>>
>> I'm not sure exactly what is going on with the machine in question,
>> and I don't really have physical access to sort it out.  The kernel
>> messages that I was getting now make a lot more sense.  I'll see if I
>> can change how I detect other nodes shutting down to fix that.
>
> You are welcome to show the kernel messages. After re-reading your first
> mail of this thread I'm still puzzled what's going on. A node goes
> bus-off in case of serious electrical problems on the bus, e.g. improper
> termination, bitrate does not matches, etc. It does *not* go bus off
> just because no other nodes are on the bus (ack slot error).
>
> Wolfgang.
>
>> On Fri, Jun 27, 2014 at 10:34 AM, Oliver Hartkopp
>> <socketcan@hartkopp.net> wrote:
>>> On 27.06.2014 18:28, Wolfgang Grandegger wrote:
>>>> On 06/27/2014 05:45 PM, Jason R1 White wrote:
>>>>> So is it still the recommendation to close the interface to flush the
>>>>> queues?
>>>>
>>>> As I said: the purpose of "restart" is not to do a fast if down->up.
>>>>
>>>>> Is there anything planned for an ip command or IOCTL to do this?
>>>>
>>>> It's on the wish list, let's say. Somebody needs to implement it. It's
>>>> also not straight-forward because there might be more than one socket
>>>> sending out CAN messages. For standard network devices there is no way
>>>> to flush all TX queues, IIRC.
>>>
>>> Probably shutting down the netdevice at BUS_OFF would be an alternative to
>>> restart-ms. When the netdevice is down all sockets are notified by -ENETDOWN.
>>>
>>> We would need some kind of notification when frames got lost anyway. And after
>>> -ENETDOWN the application can recover and set up it's communication again.
>>>
>>> Regards,
>>> Oliver
>>>
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-can" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-can" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
>

Here is what I see.

Jun 26 08:22:48 vpc6 kernel: [11535.284175] IPv6:
ADDRCONF(NETDEV_CHANGE): can0: link becomes ready
Jun 26 08:22:49 vpc6 kernel: [11536.262785] peak_pci 0000:04:00.0
can0: error warning interrupt
Jun 26 08:22:49 vpc6 kernel: [11536.265786] peak_pci 0000:04:00.0
can0: error passive interrupt
Jun 26 08:22:49 vpc6 kernel: [11536.266303] peak_pci 0000:04:00.0
can0: error passive interrupt
Jun 26 08:22:49 vpc6 kernel: [11536.270523] peak_pci 0000:04:00.0
can0: error passive interrupt
Jun 26 08:22:49 vpc6 kernel: [11536.271071] peak_pci 0000:04:00.0
can0: error passive interrupt
Jun 26 08:22:49 vpc6 kernel: [11536.271713] peak_pci 0000:04:00.0
can0: error passive interrupt
Jun 26 08:22:49 vpc6 kernel: [11536.272248] peak_pci 0000:04:00.0
can0: error passive interrupt
Jun 26 08:22:49 vpc6 kernel: [11536.279182] peak_pci 0000:04:00.0
can0: error passive interrupt
Jun 26 08:22:49 vpc6 kernel: [11536.281212] peak_pci 0000:04:00.0
can0: error warning interrupt
Jun 26 08:22:49 vpc6 kernel: [11536.281218] peak_pci 0000:04:00.0 can0: bus-off
Jun 26 08:22:50 vpc6 kernel: [11537.281885] peak_pci 0000:04:00.0
can0: restarted
Jun 26 08:22:50 vpc6 kernel: [11537.281948] IPv6:
ADDRCONF(NETDEV_CHANGE): can0: link becomes ready
Jun 26 08:22:50 vpc6 kernel: [11537.283897] peak_pci 0000:04:00.0
can0: error warning interrupt
Jun 26 08:22:51 vpc6 kernel: [11538.255450] peak_pci 0000:04:00.0
can0: error passive interrupt
Jun 26 08:22:51 vpc6 kernel: [11538.260195] peak_pci 0000:04:00.0
can0: error passive interrupt
Jun 26 08:22:51 vpc6 kernel: [11538.260365] peak_pci 0000:04:00.0
can0: error passive interrupt
Jun 26 08:22:51 vpc6 kernel: [11538.261225] peak_pci 0000:04:00.0
can0: error passive interrupt
Jun 26 08:22:51 vpc6 kernel: [11538.267145] peak_pci 0000:04:00.0
can0: error passive interrupt
Jun 26 08:22:51 vpc6 kernel: [11538.267685] peak_pci 0000:04:00.0
can0: error passive interrupt
Jun 26 08:22:51 vpc6 kernel: [11538.267926] peak_pci 0000:04:00.0
can0: error passive interrupt
Jun 26 08:22:51 vpc6 kernel: [11538.269352] peak_pci 0000:04:00.0
can0: error passive interrupt
Jun 26 08:22:51 vpc6 kernel: [11538.269565] peak_pci 0000:04:00.0
can0: error passive interrupt
Jun 26 08:22:51 vpc6 kernel: [11538.270125] peak_pci 0000:04:00.0
can0: error passive interrupt
Jun 26 08:22:51 vpc6 kernel: [11538.270342] peak_pci 0000:04:00.0
can0: error passive interrupt
Jun 26 08:22:51 vpc6 kernel: [11538.271370] peak_pci 0000:04:00.0
can0: error passive interrupt
Jun 26 08:22:51 vpc6 kernel: [11538.271997] peak_pci 0000:04:00.0
can0: error passive interrupt
Jun 26 08:22:51 vpc6 kernel: [11538.272505] peak_pci 0000:04:00.0
can0: error passive interrupt
Jun 26 08:22:51 vpc6 kernel: [11538.272720] peak_pci 0000:04:00.0
can0: error passive interrupt
Jun 26 08:22:51 vpc6 kernel: [11538.273271] peak_pci 0000:04:00.0
can0: error passive interrupt
Jun 26 08:22:51 vpc6 kernel: [11538.273498] peak_pci 0000:04:00.0
can0: error passive interrupt
Jun 26 08:22:51 vpc6 kernel: [11538.288521] peak_pci 0000:04:00.0
can0: error warning interrupt
Jun 26 08:22:51 vpc6 kernel: [11538.288529] peak_pci 0000:04:00.0 can0: bus-off
Jun 26 08:22:52 vpc6 kernel: [11539.287686] peak_pci 0000:04:00.0
can0: restarted

Austin