correct length of bcm message

correct length of bcm message

[Patrick Menschel]

Hi,

is anyone writing to BCMSocket on the Raspberry Pi from Python3 ?

I'm digging through an endianess / alignment issue on armhf platform.

My testcode [1] that I wrote years ago on works on X86_64 platform but
fails on armhf platform with OSERROR 22 "invalid argument".

Then I started concatenating bytes by hand instead of using ctypes.

What I came around is that frames[0] is somehow expected to be 8 bytes
length although it should be 16 bytes.

struct bcm_msg_head {
...
        struct can_frame frames[0];
};

I ended up inserting padding 8 bytes instead of frames[0] value and that
actually works.

That makes my overall BCMHead 40 bytes and the complete bcm message
including the can frame 56bytes.

[1] https://github.com/menschel/pysocketcan

Thanks,
Patrick

Re: correct length of bcm message

[Patrick Menschel]

Am 04.02.21 um 16:38 schrieb Patrick Menschel:
> Hi,
> 
> is anyone writing to BCMSocket on the Raspberry Pi from Python3 ?
> 
> I'm digging through an endianess / alignment issue on armhf platform.
> 
> My testcode [1] that I wrote years ago on works on X86_64 platform but
> fails on armhf platform with OSERROR 22 "invalid argument".

Some more details.

The length on X86_64 results in 72 bytes which are consumed by bcm.

On armhf it results in 52 bytes which cause OSError: [Errno 22] Invalid
argument.

> 
> Then I started concatenating bytes by hand instead of using ctypes.
> 
> What I came around is that frames[0] is somehow expected to be 8 bytes
> length although it should be 16 bytes.
> 
> struct bcm_msg_head {
> ...
>         struct can_frame frames[0];
> };
> 
> I ended up inserting padding 8 bytes instead of frames[0] value and that
> actually works.

I have to amend that, it's 4 padding bytes.

struct.pack("=IIIllllII4xIB3x8s",opcode,flags,count,ival1_sec,ival1_usec,ival2_sec,ival2_usec,can_id,nframes,can_id,can_dlc,data)
                       ^^ padding bytes

Sending the resulting 56bytes to bcmsocket gets me a cyclic message on
armhf.

> 
> That makes my overall BCMHead 40 bytes and the complete bcm message
> including the can frame 56bytes.
> 
> [1] https://github.com/menschel/pysocketcan
> 

Thanks,
Patrick

Re: correct length of bcm message

[Oliver Hartkopp]

Hello Patrick,

On 04.02.21 18:47, Patrick Menschel wrote:
> Am 04.02.21 um 16:38 schrieb Patrick Menschel:
>> Hi,
>>
>> is anyone writing to BCMSocket on the Raspberry Pi from Python3 ?
>>
>> I'm digging through an endianess / alignment issue on armhf platform.
>>
>> My testcode [1] that I wrote years ago on works on X86_64 platform but
>> fails on armhf platform with OSERROR 22 "invalid argument".
> 
> Some more details.
> 
> The length on X86_64 results in 72 bytes which are consumed by bcm.
> 
> On armhf it results in 52 bytes which cause OSError: [Errno 22] Invalid
> argument.
> 
>>
>> Then I started concatenating bytes by hand instead of using ctypes.
>>
>> What I came around is that frames[0] is somehow expected to be 8 bytes
>> length although it should be 16 bytes.

struct can_frames[0] is used in this struct definition to point out, 
that this struct is followed by a number (0..257) struct can_frame's.

This creates a padding at the end of struct bcm_head, so that the struct 
can_frame (which is always 64 bit aligned) can be directly concatenated.

Regards,
Oliver

>>
>> struct bcm_msg_head {
>> ...
>>          struct can_frame frames[0];
>> };
>>
>> I ended up inserting padding 8 bytes instead of frames[0] value and that
>> actually works.
> 
> I have to amend that, it's 4 padding bytes.
> 
> struct.pack("=IIIllllII4xIB3x8s",opcode,flags,count,ival1_sec,ival1_usec,ival2_sec,ival2_usec,can_id,nframes,can_id,can_dlc,data)
>                         ^^ padding bytes
> 
> Sending the resulting 56bytes to bcmsocket gets me a cyclic message on
> armhf.
> 
>>
>> That makes my overall BCMHead 40 bytes and the complete bcm message
>> including the can frame 56bytes.
>>
>> [1] https://github.com/menschel/pysocketcan
>>
> 
> Thanks,
> Patrick
> 

Re: correct length of bcm message

[Patrick Menschel]

Thanks Oliver,

Am 04.02.21 um 21:14 schrieb Oliver Hartkopp:
> struct can_frames[0] is used in this struct definition to point out,
> that this struct is followed by a number (0..257) struct can_frame's.
> 
> This creates a padding at the end of struct bcm_head, so that the struct
> can_frame (which is always 64 bit aligned) can be directly concatenated.

so this behavior is caused by this alignment

struct can_frame {
    ...
    __u8    data[8] __attribute__((aligned(8)));
}

and walks back upwards to

nframes (end at 36 bytes)
...
< 4 bytes gap >
...
frames (start at 40 bytes)

Should the alignment not be defined on structure instead?

struct can_frame __attribute__((aligned(8))) {
...
}


Interesting to know, apparently native alignment is 8 on X86_64 linux
and 4 on armhf linux. That's why it worked on X86_64.

Regarding python ctypes module, setting _pack_=8 has no remedy effect on
armhf platform, so I'll move to struct module.

Thanks again.

Regards,
Patrick

Re: correct length of bcm message

[Oliver Hartkopp]

On 05.02.21 09:27, Patrick Menschel wrote:

> Am 04.02.21 um 21:14 schrieb Oliver Hartkopp:
>> struct can_frames[0] is used in this struct definition to point out,
>> that this struct is followed by a number (0..257) struct can_frame's.
>>
>> This creates a padding at the end of struct bcm_head, so that the struct
>> can_frame (which is always 64 bit aligned) can be directly concatenated.
> 
> so this behavior is caused by this alignment
> 
> struct can_frame {
>      ...
>      __u8    data[8] __attribute__((aligned(8)));
> }
> 
> and walks back upwards to
> 
> nframes (end at 36 bytes)
> ...
> < 4 bytes gap >
> ...
> frames (start at 40 bytes)
> 
> Should the alignment not be defined on structure instead?
> 
> struct can_frame __attribute__((aligned(8))) {
> ...
> }

Hm, the original idea was to be able to access the data[] with a 64 bit 
aligned access, e.g. to initialize the content by a single assignment.

The can_id and the other stuff (len, flags, padding, whatever) before 
the data[] section is intentionally also 64 bit long.

So it should be no difference in functionality.

> Interesting to know, apparently native alignment is 8 on X86_64 linux
> and 4 on armhf linux. That's why it worked on X86_64.
> 
> Regarding python ctypes module, setting _pack_=8 has no remedy effect on
> armhf platform, so I'll move to struct module.

Regards,
Oliver

Re: correct length of bcm message

[Patrick Menschel]

>> Interesting to know, apparently native alignment is 8 on X86_64 linux
>> and 4 on armhf linux. That's why it worked on X86_64.
>>
>> Regarding python ctypes module, setting _pack_=8 has no remedy effect on
>> armhf platform, so I'll move to struct module.

Hi,

tying some loose ends here.
My final solution to force alignment after bcm_msg_head to the next 8
byte boundary in python3 is to use an alignment hack of struct module.
https://docs.python.org/3/library/struct.html#struct-examples

struct.pack("IIIllllII0q",opcode,flags,count,ival1_sec,ival1_usec,ival2_sec,ival2_usec,can_id,nframes)

The "0q" is a zero length long long which causes padding up to the next
8 byte boundary.

This works for ARMHF (40 bytes) and X86_64 (72 bytes).

Regards,
Patrick

Re: correct length of bcm message

[Oliver Hartkopp]

On 07.02.21 07:33, Patrick Menschel wrote:
>>> Interesting to know, apparently native alignment is 8 on X86_64 linux
>>> and 4 on armhf linux. That's why it worked on X86_64.
>>>
>>> Regarding python ctypes module, setting _pack_=8 has no remedy effect on
>>> armhf platform, so I'll move to struct module.
> 
> Hi,
> 
> tying some loose ends here.
> My final solution to force alignment after bcm_msg_head to the next 8
> byte boundary in python3 is to use an alignment hack of struct module.
> https://docs.python.org/3/library/struct.html#struct-examples
> 
> struct.pack("IIIllllII0q",opcode,flags,count,ival1_sec,ival1_usec,ival2_sec,ival2_usec,can_id,nframes)
> 
> The "0q" is a zero length long long which causes padding up to the next
> 8 byte boundary.

Excellent! Exactly what is needed here.

Thanks,
Oliver

> This works for ARMHF (40 bytes) and X86_64 (72 bytes).
> 
> Regards,
> Patrick
>