Strange switching [threadname] to secondary mode after exception #0 in kernel-space at 0x80272444

Strange switching [threadname] to secondary mode after exception #0 in kernel-space at 0x80272444

[François Legal]

Using RTNet with multiple interfaces, xenomai 3.1 with linux 4.4
Whenever a posix RT thread calls recvmsg (), I get this error.

I could track this down to line 422 in udp.c :     msg = rtnet_get_arg(fd, &_msg, u_msg, sizeof(_msg));

If I disassemble the kernel, the failing instruction (at 0x80272444) is a nop in arm_copy_from_user
Am I doing anything wrong ?

François

Re: Strange switching [threadname] to secondary mode after exception #0 in kernel-space at 0x80272444

[Jan Kiszka]

On 20.08.20 15:46, François Legal via Xenomai wrote:
> Using RTNet with multiple interfaces, xenomai 3.1 with linux 4.4
> Whenever a posix RT thread calls recvmsg (), I get this error.
> 
> I could track this down to line 422 in udp.c :     msg = rtnet_get_arg(fd, &_msg, u_msg, sizeof(_msg));
> 
> If I disassemble the kernel, the failing instruction (at 0x80272444) is a nop in arm_copy_from_user
> Am I doing anything wrong ?

Maybe you are passing an invalid pointer to recvmsg, directly or in its
message structures. That would cause an exception which is fixed up but
also reported. Do you get an error in return?

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: Strange switching [threadname] to secondary mode after exception #0 in kernel-space at 0x80272444

[François Legal]

Le Vendredi, Août 21, 2020 20:04 CEST, Jan Kiszka <jan.kiszka@siemens.com> a écrit:

> On 20.08.20 15:46, François Legal via Xenomai wrote:
> > Using RTNet with multiple interfaces, xenomai 3.1 with linux 4.4
> > Whenever a posix RT thread calls recvmsg (), I get this error.
> >
> > I could track this down to line 422 in udp.c :     msg = rtnet_get_arg(fd, &_msg, u_msg, sizeof(_msg));
> >
> > If I disassemble the kernel, the failing instruction (at 0x80272444) is a nop in arm_copy_from_user
> > Am I doing anything wrong ?
>
> Maybe you are passing an invalid pointer to recvmsg, directly or in its
> message structures. That would cause an exception which is fixed up but
> also reported. Do you get an error in return?
>

So, I checked, and the same happens on calls to sendto ().
I doubt the pointer passed is invalid, as I don't get that kind of problems with 3.0.9.
Upon return from recvmsg or sendto, I get -1 with errno set to 14.
Also, I checked in ipipe_base.h, exception #0 seem to be an instruction/data access exception, which seems very weird to me for a NOP instruction that does not appear to be on a line/page boundary.

François

> Jan
>
> --
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> Corporate Competence Center Embedded Linux

Re: Strange switching [threadname] to secondary mode after exception #0 in kernel-space at 0x80272444

[Jan Kiszka]

On 24.08.20 11:44, François Legal wrote:
> Le Vendredi, Août 21, 2020 20:04 CEST, Jan Kiszka <jan.kiszka@siemens.com> a écrit: 
>  
>> On 20.08.20 15:46, François Legal via Xenomai wrote:
>>> Using RTNet with multiple interfaces, xenomai 3.1 with linux 4.4
>>> Whenever a posix RT thread calls recvmsg (), I get this error.
>>>
>>> I could track this down to line 422 in udp.c :     msg = rtnet_get_arg(fd, &_msg, u_msg, sizeof(_msg));
>>>
>>> If I disassemble the kernel, the failing instruction (at 0x80272444) is a nop in arm_copy_from_user
>>> Am I doing anything wrong ?
>>
>> Maybe you are passing an invalid pointer to recvmsg, directly or in its
>> message structures. That would cause an exception which is fixed up but
>> also reported. Do you get an error in return?
>>
> 
> So, I checked, and the same happens on calls to sendto ().
> I doubt the pointer passed is invalid, as I don't get that kind of problems with 3.0.9.
> Upon return from recvmsg or sendto, I get -1 with errno set to 14.

14 is EFAULT. That at least is as expected.

> Also, I checked in ipipe_base.h, exception #0 seem to be an instruction/data access exception, which seems very weird to me for a NOP instruction that does not appear to be on a line/page boundary.
> 

Can you reproduce the issue with a simple test case? Is it
board-dependent, or does it even trigger in a qemu-arm environment like
xenomai-images builds?

Greg, any idea?

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: Strange switching [threadname] to secondary mode after exception #0 in kernel-space at 0x80272444

[François Legal]

Sorry to come back so late on that topic.
I took some time to write a simple test program, which is attached.

Config is as follows :
xenomai-3.1
linux-4.4.227

François

Le Vendredi, Août 21, 2020 20:04 CEST, Jan Kiszka <jan.kiszka@siemens.com> a écrit: 
 
> On 20.08.20 15:46, François Legal via Xenomai wrote:
> > Using RTNet with multiple interfaces, xenomai 3.1 with linux 4.4
> > Whenever a posix RT thread calls recvmsg (), I get this error.
> > 
> > I could track this down to line 422 in udp.c :     msg = rtnet_get_arg(fd, &_msg, u_msg, sizeof(_msg));
> > 
> > If I disassemble the kernel, the failing instruction (at 0x80272444) is a nop in arm_copy_from_user
> > Am I doing anything wrong ?
> 
> Maybe you are passing an invalid pointer to recvmsg, directly or in its
> message structures. That would cause an exception which is fixed up but
> also reported. Do you get an error in return?
> 
> Jan
> 
> -- 
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> Corporate Competence Center Embedded Linux
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test-net.c
Type: application/octet-stream
Size: 6141 bytes
Desc: not available
URL: <http://xenomai.org/pipermail/xenomai/attachments/20200925/f745e32f/attachment.obj>

Re: Strange switching [threadname] to secondary mode after exception #0 in kernel-space at 0x80272444

[François Legal]

So I decided to dig a little deeper in this problem, and found out that it is not xenomai3.1 specific.
I tried with xenomai-3.0.9 and linux 4.4.227 -> same problem.
I thought it might be a matter of ipipe patch not quite fit to 4.4.227, so I decided to try with latest ipipe-4.4.y-cip from git and xenomai 3.0.9, but the same problem happens.

François

Le Vendredi, Septembre 25, 2020 18:11 CEST, François Legal via Xenomai <xenomai@xenomai.org> a écrit:

> Sorry to come back so late on that topic.
> I took some time to write a simple test program, which is attached.
>
> Config is as follows :
> xenomai-3.1
> linux-4.4.227
>
> François
>
> Le Vendredi, Août 21, 2020 20:04 CEST, Jan Kiszka <jan.kiszka@siemens.com> a écrit:
>
> > On 20.08.20 15:46, François Legal via Xenomai wrote:
> > > Using RTNet with multiple interfaces, xenomai 3.1 with linux 4.4
> > > Whenever a posix RT thread calls recvmsg (), I get this error.
> > >
> > > I could track this down to line 422 in udp.c :     msg = rtnet_get_arg(fd, &_msg, u_msg, sizeof(_msg));
> > >
> > > If I disassemble the kernel, the failing instruction (at 0x80272444) is a nop in arm_copy_from_user
> > > Am I doing anything wrong ?
> >
> > Maybe you are passing an invalid pointer to recvmsg, directly or in its
> > message structures. That would cause an exception which is fixed up but
> > also reported. Do you get an error in return?
> >
> > Jan
> >
> > --
> > Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> > Corporate Competence Center Embedded Linux
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: test-net.c
> Type: application/octet-stream
> Size: 6141 bytes
> Desc: not available
> URL: <http://xenomai.org/pipermail/xenomai/attachments/20200925/f745e32f/attachment.obj>

Re: Strange switching [threadname] to secondary mode after exception #0 in kernel-space at 0x80272444

[François Legal]

I might have found what is causing this problem : CONFIG_CPU_SPECTRE

It was the noticeable difference between my last (4.4.189) kernel release not showing this problem, and the next I tried (4.4.227).
If I modify the Kconfig to disable CPU_SPECTRE, the problem does not show up anymore.

I'll dig a little deeper into that now.

François

Le Mercredi, Septembre 30, 2020 08:42 CEST, François Legal <francois.legal@thom.fr.eu.org> a écrit:

> So I decided to dig a little deeper in this problem, and found out that it is not xenomai3.1 specific.
> I tried with xenomai-3.0.9 and linux 4.4.227 -> same problem.
> I thought it might be a matter of ipipe patch not quite fit to 4.4.227, so I decided to try with latest ipipe-4.4.y-cip from git and xenomai 3.0.9, but the same problem happens.
>
> François
>
> Le Vendredi, Septembre 25, 2020 18:11 CEST, François Legal via Xenomai <xenomai@xenomai.org> a écrit:
>
> > Sorry to come back so late on that topic.
> > I took some time to write a simple test program, which is attached.
> >
> > Config is as follows :
> > xenomai-3.1
> > linux-4.4.227
> >
> > François
> >
> > Le Vendredi, Août 21, 2020 20:04 CEST, Jan Kiszka <jan.kiszka@siemens.com> a écrit:
> >
> > > On 20.08.20 15:46, François Legal via Xenomai wrote:
> > > > Using RTNet with multiple interfaces, xenomai 3.1 with linux 4.4
> > > > Whenever a posix RT thread calls recvmsg (), I get this error.
> > > >
> > > > I could track this down to line 422 in udp.c :     msg = rtnet_get_arg(fd, &_msg, u_msg, sizeof(_msg));
> > > >
> > > > If I disassemble the kernel, the failing instruction (at 0x80272444) is a nop in arm_copy_from_user
> > > > Am I doing anything wrong ?
> > >
> > > Maybe you are passing an invalid pointer to recvmsg, directly or in its
> > > message structures. That would cause an exception which is fixed up but
> > > also reported. Do you get an error in return?
> > >
> > > Jan
> > >
> > > --
> > > Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> > > Corporate Competence Center Embedded Linux
> > -------------- next part --------------
> > A non-text attachment was scrubbed...
> > Name: test-net.c
> > Type: application/octet-stream
> > Size: 6141 bytes
> > Desc: not available
> > URL: <http://xenomai.org/pipermail/xenomai/attachments/20200925/f745e32f/attachment.obj>
>

Re: Strange switching [threadname] to secondary mode after exception #0 in kernel-space at 0x80272444

[François Legal]

So the problem here lies in the calls to rtnet_get_arg(fd, &_msg, msg, sizeof(*msg)); in af_packet.c sendmasg/recvmsg

The only explanation I could find was that, for my architecture (32bit ARMv7), syscall use the COBALT_SYSCALL32emu variants for sendmsg/recvmsg calls, which in turn allocate a struct user_msghdr on the SVC stack, then calls rt_packet_sendmsg/rt_packet_recvmsg with the address of that struct.

Then, rtnet_get_arg would invoke arm_copy_from_user, which, with SPECTRE mitigation turned on (you normally can't disable this by configuration), would check that the source pointer is in the userland memory area, and hence fail in my case.

Going from there, except disabling SPECTRE mitigation, I'm not sure how I can fix this.

François

Le Mercredi, Septembre 30, 2020 10:48 CEST, François Legal via Xenomai <xenomai@xenomai.org> a écrit:

> I might have found what is causing this problem : CONFIG_CPU_SPECTRE
>
> It was the noticeable difference between my last (4.4.189) kernel release not showing this problem, and the next I tried (4.4.227).
> If I modify the Kconfig to disable CPU_SPECTRE, the problem does not show up anymore.
>
> I'll dig a little deeper into that now.
>
> François
>
> Le Mercredi, Septembre 30, 2020 08:42 CEST, François Legal <francois.legal@thom.fr.eu.org> a écrit:
>
> > So I decided to dig a little deeper in this problem, and found out that it is not xenomai3.1 specific.
> > I tried with xenomai-3.0.9 and linux 4.4.227 -> same problem.
> > I thought it might be a matter of ipipe patch not quite fit to 4.4.227, so I decided to try with latest ipipe-4.4.y-cip from git and xenomai 3.0.9, but the same problem happens.
> >
> > François
> >
> > Le Vendredi, Septembre 25, 2020 18:11 CEST, François Legal via Xenomai <xenomai@xenomai.org> a écrit:
> >
> > > Sorry to come back so late on that topic.
> > > I took some time to write a simple test program, which is attached.> >
> > > Config is as follows :
> > > xenomai-3.1
> > > linux-4.4.227
> > >
> > > François
> > >
> > > Le Vendredi, Août 21, 2020 20:04 CEST, Jan Kiszka <jan.kiszka@siemens.com> a écrit:
> > >
> > > > On 20.08.20 15:46, François Legal via Xenomai wrote:
> > > > > Using RTNet with multiple interfaces, xenomai 3.1 with linux 4.4
> > > > > Whenever a posix RT thread calls recvmsg (), I get this error.
> > > > >
> > > > > I could track this down to line 422 in udp.c :     msg = rtnet_get_arg(fd, &_msg, u_msg, sizeof(_msg));
> > > > >
> > > > > If I disassemble the kernel, the failing instruction (at 0x80272444) is a nop in arm_copy_from_user
> > > > > Am I doing anything wrong ?
> > > >
> > > > Maybe you are passing an invalid pointer to recvmsg, directly or in its
> > > > message structures. That would cause an exception which is fixed up but
> > > > also reported. Do you get an error in return?
> > > >
> > > > Jan
> > > >
> > > > --
> > > > Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> > > > Corporate Competence Center Embedded Linux
> > > -------------- next part --------------
> > > A non-text attachment was scrubbed...
> > > Name: test-net.c
> > > Type: application/octet-stream
> > > Size: 6141 bytes
> > > Desc: not available
> > > URL: <http://xenomai.org/pipermail/xenomai/attachments/20200925/f745e32f/attachment.obj>
> >
>
>

Re: Strange switching [threadname] to secondary mode after exception #0 in kernel-space at 0x80272444

[François Legal]

Anybody can help on this ?

François

Le Mercredi, Septembre 30, 2020 17:14 CEST, François Legal via Xenomai <xenomai@xenomai.org> a écrit:

> So the problem here lies in the calls to rtnet_get_arg(fd, &_msg, msg, sizeof(*msg)); in af_packet.c sendmasg/recvmsg
>
> The only explanation I could find was that, for my architecture (32bit ARMv7), syscall use the COBALT_SYSCALL32emu variants for sendmsg/recvmsg calls, which in turn allocate a struct user_msghdr on the SVC stack, then calls rt_packet_sendmsg/rt_packet_recvmsg with the address of that struct.
>
> Then, rtnet_get_arg would invoke arm_copy_from_user, which, with SPECTRE mitigation turned on (you normally can't disable this by configuration), would check that the source pointer is in the userland memory area, and hence fail in my case.
>
> Going from there, except disabling SPECTRE mitigation, I'm not sure how I can fix this.
>
> François
>
> Le Mercredi, Septembre 30, 2020 10:48 CEST, François Legal via Xenomai <xenomai@xenomai.org> a écrit:
>
> > I might have found what is causing this problem : CONFIG_CPU_SPECTRE
> >
> > It was the noticeable difference between my last (4.4.189) kernel release not showing this problem, and the next I tried (4.4.227).
> > If I modify the Kconfig to disable CPU_SPECTRE, the problem does not show up anymore.
> >
> > I'll dig a little deeper into that now.
> >
> > François
> >
> > Le Mercredi, Septembre 30, 2020 08:42 CEST, François Legal <francois.legal@thom.fr.eu.org> a écrit:
> >
> > > So I decided to dig a little deeper in this problem, and found out that it is not xenomai3.1 specific.
> > > I tried with xenomai-3.0.9 and linux 4.4.227 -> same problem.
> > > I thought it might be a matter of ipipe patch not quite fit to 4.4.227, so I decided to try with latest ipipe-4.4.y-cip from git and xenomai 3.0.9, but the same problem happens.
> > >
> > > François
> > >
> > > Le Vendredi, Septembre 25, 2020 18:11 CEST, François Legal via Xenomai <xenomai@xenomai.org> a écrit:
> > >
> > > > Sorry to come back so late on that topic.
> > > > I took some time to write a simple test program, which is attached.> >
> > > > Config is as follows :
> > > > xenomai-3.1
> > > > linux-4.4.227
> > > >
> > > > François
> > > >
> > > > Le Vendredi, Août 21, 2020 20:04 CEST, Jan Kiszka <jan.kiszka@siemens.com> a écrit:
> > > >
> > > > > On 20.08.20 15:46, François Legal via Xenomai wrote:
> > > > > > Using RTNet with multiple interfaces, xenomai 3.1 with linux 4.4
> > > > > > Whenever a posix RT thread calls recvmsg (), I get this error.
> > > > > >
> > > > > > I could track this down to line 422 in udp.c :     msg = rtnet_get_arg(fd, &_msg, u_msg, sizeof(_msg));
> > > > > >
> > > > > > If I disassemble the kernel, the failing instruction (at 0x80272444) is a nop in arm_copy_from_user
> > > > > > Am I doing anything wrong ?
> > > > >
> > > > > Maybe you are passing an invalid pointer to recvmsg, directly or in its
> > > > > message structures. That would cause an exception which is fixed up but
> > > > > also reported. Do you get an error in return?
> > > > >
> > > > > Jan
> > > > >
> > > > > --
> > > > > Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> > > > > Corporate Competence Center Embedded Linux
> > > > -------------- next part --------------
> > > > A non-text attachment was scrubbed...
> > > > Name: test-net.c
> > > > Type: application/octet-stream
> > > > Size: 6141 bytes
> > > > Desc: not available
> > > > URL: <http://xenomai.org/pipermail/xenomai/attachments/20200925/f745e32f/attachment.obj>
> > >
> >
> >
>
>

Re: Strange switching [threadname] to secondary mode after exception #0 in kernel-space at 0x80272444

[Jan Kiszka]

On 14.10.20 10:43, François Legal via Xenomai wrote:
> Anybody can help on this ?
> 

I'm unfortunately not familiar with the armv7 details of copy-from-user,
not too speak of how spectre contributed to it. Greg, Philippe, did you
come across this already?

Jan

> François
> 
> Le Mercredi, Septembre 30, 2020 17:14 CEST, François Legal via Xenomai <xenomai@xenomai.org> a écrit: 
>  
>> So the problem here lies in the calls to rtnet_get_arg(fd, &_msg, msg, sizeof(*msg)); in af_packet.c sendmasg/recvmsg
>>
>> The only explanation I could find was that, for my architecture (32bit ARMv7), syscall use the COBALT_SYSCALL32emu variants for sendmsg/recvmsg calls, which in turn allocate a struct user_msghdr on the SVC stack, then calls rt_packet_sendmsg/rt_packet_recvmsg with the address of that struct.
>>
>> Then, rtnet_get_arg would invoke arm_copy_from_user, which, with SPECTRE mitigation turned on (you normally can't disable this by configuration), would check that the source pointer is in the userland memory area, and hence fail in my case.
>>
>> Going from there, except disabling SPECTRE mitigation, I'm not sure how I can fix this.
>>
>> François
>>
>> Le Mercredi, Septembre 30, 2020 10:48 CEST, François Legal via Xenomai <xenomai@xenomai.org> a écrit: 
>>  
>>> I might have found what is causing this problem : CONFIG_CPU_SPECTRE
>>>
>>> It was the noticeable difference between my last (4.4.189) kernel release not showing this problem, and the next I tried (4.4.227).
>>> If I modify the Kconfig to disable CPU_SPECTRE, the problem does not show up anymore.
>>>
>>> I'll dig a little deeper into that now.
>>>
>>> François
>>>
>>> Le Mercredi, Septembre 30, 2020 08:42 CEST, François Legal <francois.legal@thom.fr.eu.org> a écrit: 
>>>  
>>>> So I decided to dig a little deeper in this problem, and found out that it is not xenomai3.1 specific.
>>>> I tried with xenomai-3.0.9 and linux 4.4.227 -> same problem.
>>>> I thought it might be a matter of ipipe patch not quite fit to 4.4.227, so I decided to try with latest ipipe-4.4.y-cip from git and xenomai 3.0.9, but the same problem happens.
>>>>
>>>> François
>>>>
>>>> Le Vendredi, Septembre 25, 2020 18:11 CEST, François Legal via Xenomai <xenomai@xenomai.org> a écrit: 
>>>>  
>>>>> Sorry to come back so late on that topic.
>>>>> I took some time to write a simple test program, which is attached.> > 
>>>>> Config is as follows :
>>>>> xenomai-3.1
>>>>> linux-4.4.227
>>>>>
>>>>> François
>>>>>
>>>>> Le Vendredi, Août 21, 2020 20:04 CEST, Jan Kiszka <jan.kiszka@siemens.com> a écrit: 
>>>>>  
>>>>>> On 20.08.20 15:46, François Legal via Xenomai wrote:
>>>>>>> Using RTNet with multiple interfaces, xenomai 3.1 with linux 4.4
>>>>>>> Whenever a posix RT thread calls recvmsg (), I get this error.
>>>>>>>
>>>>>>> I could track this down to line 422 in udp.c :     msg = rtnet_get_arg(fd, &_msg, u_msg, sizeof(_msg));
>>>>>>>
>>>>>>> If I disassemble the kernel, the failing instruction (at 0x80272444) is a nop in arm_copy_from_user
>>>>>>> Am I doing anything wrong ?
>>>>>>
>>>>>> Maybe you are passing an invalid pointer to recvmsg, directly or in its
>>>>>> message structures. That would cause an exception which is fixed up but
>>>>>> also reported. Do you get an error in return?
>>>>>>
>>>>>> Jan
>>>>>>
>>>>>> -- 
>>>>>> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
>>>>>> Corporate Competence Center Embedded Linux
>>>>> -------------- next part --------------
>>>>> A non-text attachment was scrubbed...
>>>>> Name: test-net.c
>>>>> Type: application/octet-stream
>>>>> Size: 6141 bytes
>>>>> Desc: not available
>>>>> URL: <http://xenomai.org/pipermail/xenomai/attachments/20200925/f745e32f/attachment.obj>
>>>>

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: Strange switching [threadname] to secondary mode after exception #0 in kernel-space at 0x80272444

[Greg Gallagher]

On Wed, Oct 14, 2020 at 5:37 AM Jan Kiszka <jan.kiszka@siemens.com> wrote:
>
> On 14.10.20 10:43, François Legal via Xenomai wrote:
> > Anybody can help on this ?
> >
>
> I'm unfortunately not familiar with the armv7 details of copy-from-user,
> not too speak of how spectre contributed to it. Greg, Philippe, did you
> come across this already?
>
> Jan
>
I'll take a look tonight but I haven't hit this in my testing.  This
was found on 4.4? Have you tried the 4.19 kernels?

-Greg

Re: Strange switching [threadname] to secondary mode after exception #0 in kernel-space at 0x80272444

[François Legal]

Le Mercredi, Octobre 14, 2020 16:16 CEST, Greg Gallagher <greg@embeddedgreg.com> a écrit:

> On Wed, Oct 14, 2020 at 5:37 AM Jan Kiszka <jan.kiszka@siemens.com> wrote:
> >
> > On 14.10.20 10:43, François Legal via Xenomai wrote:
> > > Anybody can help on this ?
> > >
> >
> > I'm unfortunately not familiar with the armv7 details of copy-from-user,
> > not too speak of how spectre contributed to it. Greg, Philippe, did you
> > come across this already?
> >
> > Jan
> >
> I'll take a look tonight but I haven't hit this in my testing.  This
> was found on 4.4? Have you tried the 4.19 kernels?
>
> -Greg

So I tried the same case on 4.19.121, with the same result, and I guess for the same reason.
Could anybody explain why, on ARMv7, we need to copy the message header at the syscall entry, and not let the xxxmsg routine do it on its own ?
Also, I could not find how those COBALT_SYSCALL32emu logic work. Can anybody point me to some documentation ?

Thanks

François

Re: Strange switching [threadname] to secondary mode after exception #0 in kernel-space at 0x80272444

[Philippe Gerum]

François Legal <devel@thom.fr.eu.org> writes:

> Le Mercredi, Octobre 14, 2020 16:16 CEST, Greg Gallagher <greg@embeddedgreg.com> a écrit: 
>  
>> On Wed, Oct 14, 2020 at 5:37 AM Jan Kiszka <jan.kiszka@siemens.com> wrote:
>> >
>> > On 14.10.20 10:43, François Legal via Xenomai wrote:
>> > > Anybody can help on this ?
>> > >
>> >
>> > I'm unfortunately not familiar with the armv7 details of copy-from-user,
>> > not too speak of how spectre contributed to it. Greg, Philippe, did you
>> > come across this already?
>> >
>> > Jan
>> >
>> I'll take a look tonight but I haven't hit this in my testing.  This
>> was found on 4.4? Have you tried the 4.19 kernels?
>> 
>> -Greg
>  
> So I tried the same case on 4.19.121, with the same result, and I guess for the same reason.
> Could anybody explain why, on ARMv7, we need to copy the message header at the syscall entry, and not let the xxxmsg routine do it on its own ?
> Also, I could not find how those COBALT_SYSCALL32emu logic work.

There is no way an armv7 system would run the sys32emu code in
Cobalt. This code path is specific to architectures which support mixed
ABI models. Only Cobalt/x86 supports this so far, issuing x86_32
syscalls to an x86_64 kernel. You may want to check
CONFIG_XENO_ARCH_SYS3264, it is set to "def_bool n" in the Kconfig
stuff.

Converting long/pointer data conforming to a 32bit ABI to their 64bit
representation is the reason why the sys32/compat wrappers exist and are
compiled in when CONFIG_XENO_ARCH_SYS3264 is set. Aggregates arguments
such as msg headers do contain such data.

> Can anybody point me to some documentation ?
>

There is none, I'm afraid.

-- 
Philippe.

Re: Strange switching [threadname] to secondary mode after exception #0 in kernel-space at 0x80272444

[François Legal]

Le Vendredi, Octobre 16, 2020 10:59 CEST, Philippe Gerum <rpm@xenomai.org> a écrit:

>
> François Legal <devel@thom.fr.eu.org> writes:
>
> > Le Mercredi, Octobre 14, 2020 16:16 CEST, Greg Gallagher <greg@embeddedgreg.com> a écrit:
> >
> >> On Wed, Oct 14, 2020 at 5:37 AM Jan Kiszka <jan.kiszka@siemens.com> wrote:
> >> >
> >> > On 14.10.20 10:43, François Legal via Xenomai wrote:
> >> > > Anybody can help on this ?
> >> > >
> >> >
> >> > I'm unfortunately not familiar with the armv7 details of copy-from-user,
> >> > not too speak of how spectre contributed to it. Greg, Philippe, did you
> >> > come across this already?
> >> >
> >> > Jan
> >> >
> >> I'll take a look tonight but I haven't hit this in my testing.  This
> >> was found on 4.4? Have you tried the 4.19 kernels?
> >>
> >> -Greg
> >
> > So I tried the same case on 4.19.121, with the same result, and I guess for the same reason.
> > Could anybody explain why, on ARMv7, we need to copy the message header at the syscall entry, and not let the xxxmsg routine do it on its own ?
> > Also, I could not find how those COBALT_SYSCALL32emu logic work.
>
> There is no way an armv7 system would run the sys32emu code in
> Cobalt. This code path is specific to architectures which support mixed
> ABI models. Only Cobalt/x86 supports this so far, issuing x86_32
> syscalls to an x86_64 kernel. You may want to check
> CONFIG_XENO_ARCH_SYS3264, it is set to "def_bool n" in the Kconfig
> stuff.
>

Maybe I don't use the right terms here, but what I can see from the code is (in linux tree kernel/xenomai/posix/syscall32.c)
COBALT_SYSCALL32emu(sendmsg, handover,
		    (int fd, struct compat_msghdr __user *umsg, int flags))
{
	struct user_msghdr m;
	int ret;

	ret = sys32_get_msghdr(&m, umsg);

	return ret ?: rtdm_fd_sendmsg(fd, &m, flags);
}

And the problem regarding SPECTRE mitigation is with the "ret = sys32_get_msghdr(&m, umsg);" line, as af_packet (in my case, but I believe the other handlers should do the same) will also call copy_from_user on the "msghdr" argument, and the SPECTRE mitigation will check that this pointer is in the userland MM area.

> Converting long/pointer data conforming to a 32bit ABI to their 64bit
> representation is the reason why the sys32/compat wrappers exist and are
> compiled in when CONFIG_XENO_ARCH_SYS3264 is set. Aggregates arguments
> such as msg headers do contain such data.
>
> > Can anybody point me to some documentation ?
> >
>
> There is none, I'm afraid.
>
> --
> Philippe.

Re: Strange switching [threadname] to secondary mode after exception #0 in kernel-space at 0x80272444

[Philippe Gerum]

François Legal <devel@thom.fr.eu.org> writes:

> Le Vendredi, Octobre 16, 2020 10:59 CEST, Philippe Gerum <rpm@xenomai.org> a écrit: 
>  
>> 
>> François Legal <devel@thom.fr.eu.org> writes:
>> 
>> > Le Mercredi, Octobre 14, 2020 16:16 CEST, Greg Gallagher <greg@embeddedgreg.com> a écrit: 
>> >  
>> >> On Wed, Oct 14, 2020 at 5:37 AM Jan Kiszka <jan.kiszka@siemens.com> wrote:
>> >> >
>> >> > On 14.10.20 10:43, François Legal via Xenomai wrote:
>> >> > > Anybody can help on this ?
>> >> > >
>> >> >
>> >> > I'm unfortunately not familiar with the armv7 details of copy-from-user,
>> >> > not too speak of how spectre contributed to it. Greg, Philippe, did you
>> >> > come across this already?
>> >> >
>> >> > Jan
>> >> >
>> >> I'll take a look tonight but I haven't hit this in my testing.  This
>> >> was found on 4.4? Have you tried the 4.19 kernels?
>> >> 
>> >> -Greg
>> >  
>> > So I tried the same case on 4.19.121, with the same result, and I guess for the same reason.
>> > Could anybody explain why, on ARMv7, we need to copy the message header at the syscall entry, and not let the xxxmsg routine do it on its own ?
>> > Also, I could not find how those COBALT_SYSCALL32emu logic work.
>> 
>> There is no way an armv7 system would run the sys32emu code in
>> Cobalt. This code path is specific to architectures which support mixed
>> ABI models. Only Cobalt/x86 supports this so far, issuing x86_32
>> syscalls to an x86_64 kernel. You may want to check
>> CONFIG_XENO_ARCH_SYS3264, it is set to "def_bool n" in the Kconfig
>> stuff.
>> 
>
> Maybe I don't use the right terms here, but what I can see from the code is (in linux tree kernel/xenomai/posix/syscall32.c)
> COBALT_SYSCALL32emu(sendmsg, handover,
> 		    (int fd, struct compat_msghdr __user *umsg, int flags))
> {
> 	struct user_msghdr m;
> 	int ret;
>
> 	ret = sys32_get_msghdr(&m, umsg);
>
> 	return ret ?: rtdm_fd_sendmsg(fd, &m, flags);
> }
>
> And the problem regarding SPECTRE mitigation is with the "ret =
> sys32_get_msghdr(&m, umsg);" line, as af_packet (in my case, but I
> believe the other handlers should do the same) will also call
> copy_from_user on the "msghdr" argument, and the SPECTRE mitigation
> will check that this pointer is in the userland MM area.

There is indeed a problem with this code passing the kernel memory
address of a bounce buffer to RTDM handlers which would expect __user
tagged memory instead. This ends up confusing any low-level
copy_to/from_user routine which includes attack
mitigation. rtnet_get_arg() does call such routine under the hood. This
is something some Xenomai contributor may want to address.

But, again, this sys32emu code cannot run for armv7 under the current
stock implementation. So what we are discussing is purely hypothetical
at this stage for this architecture, and should definitely never happen
by construction if you are running armv7 (which does not make the
original issue go away, that is granted).

-- 
Philippe.

Re: Strange switching [threadname] to secondary mode after exception #0 in kernel-space at 0x80272444

[François Legal]

Le Vendredi, Octobre 16, 2020 14:03 CEST, Philippe Gerum <rpm@xenomai.org> a écrit:

>
> François Legal <devel@thom.fr.eu.org> writes:
>
> > Le Vendredi, Octobre 16, 2020 10:59 CEST, Philippe Gerum <rpm@xenomai.org> a écrit:
> >
> >>
> >> François Legal <devel@thom.fr.eu.org> writes:
> >>
> >> > Le Mercredi, Octobre 14, 2020 16:16 CEST, Greg Gallagher <greg@embeddedgreg.com> a écrit:
> >> >
> >> >> On Wed, Oct 14, 2020 at 5:37 AM Jan Kiszka <jan.kiszka@siemens.com> wrote:
> >> >> >
> >> >> > On 14.10.20 10:43, François Legal via Xenomai wrote:
> >> >> > > Anybody can help on this ?
> >> >> > >
> >> >> >
> >> >> > I'm unfortunately not familiar with the armv7 details of copy-from-user,
> >> >> > not too speak of how spectre contributed to it. Greg, Philippe, did you
> >> >> > come across this already?
> >> >> >
> >> >> > Jan
> >> >> >
> >> >> I'll take a look tonight but I haven't hit this in my testing.  This
> >> >> was found on 4.4? Have you tried the 4.19 kernels?
> >> >>
> >> >> -Greg
> >> >
> >> > So I tried the same case on 4.19.121, with the same result, and I guess for the same reason.
> >> > Could anybody explain why, on ARMv7, we need to copy the message header at the syscall entry, and not let the xxxmsg routine do it on its own ?
> >> > Also, I could not find how those COBALT_SYSCALL32emu logic work.
> >>
> >> There is no way an armv7 system would run the sys32emu code in
> >> Cobalt. This code path is specific to architectures which support mixed
> >> ABI models. Only Cobalt/x86 supports this so far, issuing x86_32
> >> syscalls to an x86_64 kernel. You may want to check
> >> CONFIG_XENO_ARCH_SYS3264, it is set to "def_bool n" in the Kconfig
> >> stuff.
> >>
> >
> > Maybe I don't use the right terms here, but what I can see from the code is (in linux tree kernel/xenomai/posix/syscall32.c)
> > COBALT_SYSCALL32emu(sendmsg, handover,
> > 		    (int fd, struct compat_msghdr __user *umsg, int flags))
> > {
> > 	struct user_msghdr m;
> > 	int ret;
> >
> > 	ret = sys32_get_msghdr(&m, umsg);
> >
> > 	return ret ?: rtdm_fd_sendmsg(fd, &m, flags);
> > }
> >
> > And the problem regarding SPECTRE mitigation is with the "ret =
> > sys32_get_msghdr(&m, umsg);" line, as af_packet (in my case, but I
> > believe the other handlers should do the same) will also call
> > copy_from_user on the "msghdr" argument, and the SPECTRE mitigation
> > will check that this pointer is in the userland MM area.
>
> There is indeed a problem with this code passing the kernel memory
> address of a bounce buffer to RTDM handlers which would expect __user
> tagged memory instead. This ends up confusing any low-level
> copy_to/from_user routine which includes attack
> mitigation. rtnet_get_arg() does call such routine under the hood. This
> is something some Xenomai contributor may want to address.
>
> But, again, this sys32emu code cannot run for armv7 under the current
> stock implementation. So what we are discussing is purely hypothetical
> at this stage for this architecture, and should definitely never happen
> by construction if you are running armv7 (which does not make the
> original issue go away, that is granted).
>

I'm not sure I quite understand that point. The code reproduced above is well built in the kernel. Are you saying this code is not called whenever userland calls sendmsg on an rt socket ? Am I looking in the wrong direction ? In that case, where should I be looking ? I mean, I tracked the whole thing with a JTAG debugger, and it seemed to me that this was what was really happening, with the SPECTRE code setting the pointer to 0 which was later being caught by arm_copy_from_user.

François

> --
> Philippe.

Re: Strange switching [threadname] to secondary mode after exception #0 in kernel-space at 0x80272444

[Philippe Gerum]

François Legal <devel@thom.fr.eu.org> writes:

> Le Vendredi, Octobre 16, 2020 14:03 CEST, Philippe Gerum <rpm@xenomai.org> a écrit: 
>  
>> 
>> François Legal <devel@thom.fr.eu.org> writes:
>> 
>> > Le Vendredi, Octobre 16, 2020 10:59 CEST, Philippe Gerum <rpm@xenomai.org> a écrit: 
>> >  
>> >> 
>> >> François Legal <devel@thom.fr.eu.org> writes:
>> >> 
>> >> > Le Mercredi, Octobre 14, 2020 16:16 CEST, Greg Gallagher <greg@embeddedgreg.com> a écrit: 
>> >> >  
>> >> >> On Wed, Oct 14, 2020 at 5:37 AM Jan Kiszka <jan.kiszka@siemens.com> wrote:
>> >> >> >
>> >> >> > On 14.10.20 10:43, François Legal via Xenomai wrote:
>> >> >> > > Anybody can help on this ?
>> >> >> > >
> >> >> >
>> >> >> > I'm unfortunately not familiar with the armv7 details of copy-from-user,
>> >> >> > not too speak of how spectre contributed to it. Greg, Philippe, did you
>> >> >> > come across this already?
>> >> >> >
>> >> >> > Jan
>> >> >> >
>> >> >> I'll take a look tonight but I haven't hit this in my testing.  This
>> >> >> was found on 4.4? Have you tried the 4.19 kernels?
>> >> >> 
>> >> >> -Greg
>> >> >  
>> >> > So I tried the same case on 4.19.121, with the same result, and I guess for the same reason.
>> >> > Could anybody explain why, on ARMv7, we need to copy the message header at the syscall entry, and not let the xxxmsg routine do it on its own ?
>> >> > Also, I could not find how those COBALT_SYSCALL32emu logic work.
>> >> 
>> >> There is no way an armv7 system would run the sys32emu code in
>> >> Cobalt. This code path is specific to architectures which support mixed
>> >> ABI models. Only Cobalt/x86 supports this so far, issuing x86_32
>> >> syscalls to an x86_64 kernel. You may want to check
>> >> CONFIG_XENO_ARCH_SYS3264, it is set to "def_bool n" in the Kconfig
>> >> stuff.
>> >> 
>> >
>> > Maybe I don't use the right terms here, but what I can see from the code is (in linux tree kernel/xenomai/posix/syscall32.c)
>> > COBALT_SYSCALL32emu(sendmsg, handover,
>> > 		    (int fd, struct compat_msghdr __user *umsg, int flags))
>> > {
>> > 	struct user_msghdr m;
>> > 	int ret;
>> >
>> > 	ret = sys32_get_msghdr(&m, umsg);
>> >
>> > 	return ret ?: rtdm_fd_sendmsg(fd, &m, flags);
>> > }
>> >
>> > And the problem regarding SPECTRE mitigation is with the "ret =
>> > sys32_get_msghdr(&m, umsg);" line, as af_packet (in my case, but I
>> > believe the other handlers should do the same) will also call
>> > copy_from_user on the "msghdr" argument, and the SPECTRE mitigation
>> > will check that this pointer is in the userland MM area.
>> 
>> There is indeed a problem with this code passing the kernel memory
>> address of a bounce buffer to RTDM handlers which would expect __user
>> tagged memory instead. This ends up confusing any low-level
>> copy_to/from_user routine which includes attack
>> mitigation. rtnet_get_arg() does call such routine under the hood. This
>> is something some Xenomai contributor may want to address.
>> 
>> But, again, this sys32emu code cannot run for armv7 under the current
>> stock implementation. So what we are discussing is purely hypothetical
>> at this stage for this architecture, and should definitely never happen
>> by construction if you are running armv7 (which does not make the
>> original issue go away, that is granted).
>> 
>
> I'm not sure I quite understand that point. The code reproduced above is well built in the kernel. Are you saying this code is not called whenever userland calls sendmsg on an rt socket ? Am I looking in the wrong direction ? In that case, where should I be looking ? I mean, I tracked the whole thing with a JTAG debugger, and it seemed to me that this was what was really happening, with the SPECTRE code setting the pointer to 0 which was later being caught by arm_copy_from_user.
>

How could syscall32.c and compat.c be built into the kernel with
CONFIG_XENO_ARCH_SYS3264 forcibly unset in the Kconfig, which is always
the case when building for anything else than x86?

Checking kernel/cobalt/posix/Makefile may help in understanding why it
is simply not possible. arm_copy_from_user is built in, no question,
and your analysis regarding SVC context memory being spuriously fed into
arm_copy_from_user is likely right.

But the sys32 wrappers are neither for armv7, armv8 nor ppc32. So yes,
you are certainly following the wrong path when looking at
kernel/cobalt/posix/syscall32.c. This 32-to-64bit syscall support is NOT
built into a kernel targeting armv7, at least when it comes to the
vanilla Xenomai code.

You may want to double-check which call site actually invokes
arm_copy_from_user.

-- 
Philippe.

Re: Strange switching [threadname] to secondary mode after exception #0 in kernel-space at 0x80272444

[François Legal]

Le Vendredi, Octobre 16, 2020 18:24 CEST, Philippe Gerum <rpm@xenomai.org> a écrit:

>
> François Legal <devel@thom.fr.eu.org> writes:
>
> > Le Vendredi, Octobre 16, 2020 14:03 CEST, Philippe Gerum <rpm@xenomai.org> a écrit:
> >
> >>
> >> François Legal <devel@thom.fr.eu.org> writes:
> >>
> >> > Le Vendredi, Octobre 16, 2020 10:59 CEST, Philippe Gerum <rpm@xenomai.org> a écrit:
> >> >
> >> >>
> >> >> François Legal <devel@thom.fr.eu.org> writes:
> >> >>
> >> >> > Le Mercredi, Octobre 14, 2020 16:16 CEST, Greg Gallagher <greg@embeddedgreg.com> a écrit:
> >> >> >
> >> >> >> On Wed, Oct 14, 2020 at 5:37 AM Jan Kiszka <jan.kiszka@siemens.com> wrote:
> >> >> >> >
> >> >> >> > On 14.10.20 10:43, François Legal via Xenomai wrote:
> >> >> >> > > Anybody can help on this ?
> >> >> >> > >
> > >> >> >
> >> >> >> > I'm unfortunately not familiar with the armv7 details of copy-from-user,
> >> >> >> > not too speak of how spectre contributed to it. Greg, Philippe, did you
> >> >> >> > come across this already?
> >> >> >> >
> >> >> >> > Jan
> >> >> >> >
> >> >> >> I'll take a look tonight but I haven't hit this in my testing.  This
> >> >> >> was found on 4.4? Have you tried the 4.19 kernels?
> >> >> >>
> >> >> >> -Greg
> >> >> >
> >> >> > So I tried the same case on 4.19.121, with the same result, and I guess for the same reason.
> >> >> > Could anybody explain why, on ARMv7, we need to copy the message header at the syscall entry, and not let the xxxmsg routine do it on its own ?
> >> >> > Also, I could not find how those COBALT_SYSCALL32emu logic work.
> >> >>
> >> >> There is no way an armv7 system would run the sys32emu code in
> >> >> Cobalt. This code path is specific to architectures which support mixed
> >> >> ABI models. Only Cobalt/x86 supports this so far, issuing x86_32
> >> >> syscalls to an x86_64 kernel. You may want to check
> >> >> CONFIG_XENO_ARCH_SYS3264, it is set to "def_bool n" in the Kconfig
> >> >> stuff.
> >> >>
> >> >
> >> > Maybe I don't use the right terms here, but what I can see from the code is (in linux tree kernel/xenomai/posix/syscall32.c)
> >> > COBALT_SYSCALL32emu(sendmsg, handover,
> >> > 		    (int fd, struct compat_msghdr __user *umsg, int flags))
> >> > {
> >> > 	struct user_msghdr m;
> >> > 	int ret;
> >> >
> >> > 	ret = sys32_get_msghdr(&m, umsg);
> >> >
> >> > 	return ret ?: rtdm_fd_sendmsg(fd, &m, flags);
> >> > }
> >> >
> >> > And the problem regarding SPECTRE mitigation is with the "ret =
> >> > sys32_get_msghdr(&m, umsg);" line, as af_packet (in my case, but I
> >> > believe the other handlers should do the same) will also call
> >> > copy_from_user on the "msghdr" argument, and the SPECTRE mitigation
> >> > will check that this pointer is in the userland MM area.
> >>
> >> There is indeed a problem with this code passing the kernel memory
> >> address of a bounce buffer to RTDM handlers which would expect __user
> >> tagged memory instead. This ends up confusing any low-level
> >> copy_to/from_user routine which includes attack
> >> mitigation. rtnet_get_arg() does call such routine under the hood. This
> >> is something some Xenomai contributor may want to address.
> >>
> >> But, again, this sys32emu code cannot run for armv7 under the current
> >> stock implementation. So what we are discussing is purely hypothetical
> >> at this stage for this architecture, and should definitely never happen
> >> by construction if you are running armv7 (which does not make the
> >> original issue go away, that is granted).
> >>
> >
> > I'm not sure I quite understand that point. The code reproduced above is well built in the kernel. Are you saying this code is not called whenever userland calls sendmsg on an rt socket ? Am I looking in the wrong direction ? In that case, where should I be looking ? I mean, I tracked the whole thing with a JTAG debugger, and it seemed to me that this was what was really happening, with the SPECTRE code setting the pointer to 0 which was later being caught by arm_copy_from_user.
> >
>
> How could syscall32.c and compat.c be built into the kernel with
> CONFIG_XENO_ARCH_SYS3264 forcibly unset in the Kconfig, which is always
> the case when building for anything else than x86?
>
> Checking kernel/cobalt/posix/Makefile may help in understanding why it
> is simply not possible. arm_copy_from_user is built in, no question,
> and your analysis regarding SVC context memory being spuriously fed into
> arm_copy_from_user is likely right.
>
> But the sys32 wrappers are neither for armv7, armv8 nor ppc32. So yes,
> you are certainly following the wrong path when looking at
> kernel/cobalt/posix/syscall32.c. This 32-to-64bit syscall support is NOT
> built into a kernel targeting armv7, at least when it comes to the
> vanilla Xenomai code.
>
> You may want to double-check which call site actually invokes
> arm_copy_from_user.
>
> --
> Philippe.

Thanks Philippe for pointing me in the right direction.
So, if I'm correct this time, the problem is about the same, but in posix/io.c.
In

COBALT_SYSCALL(sendmsg, handover,
	       (int fd, struct user_msghdr __user *umsg, int flags))
{
	struct user_msghdr m;
	int ret;

	ret = cobalt_copy_from_user(&m, umsg, sizeof(m));

	return ret ?: rtdm_fd_sendmsg(fd, &m, flags);
}

Same thing, the user_msghdr is allocated on the SVC stack, then copied from user, then handed over to the sendmsg handling function pertaining to that fd, and whenever that handling function call copy_from_user for the same user_msghdr pointer, it triggers the SPECTRE mitigation protection.

What is still unclear to me is why this user_msghdr struct is copied here, and not left to the handling function, as in the sendmmsg syscalls, the struct mmsghdr is not copied from user in syscall entry.

Thanks

François

Re: Strange switching [threadname] to secondary mode after exception #0 in kernel-space at 0x80272444

[Philippe Gerum]

François Legal <devel@thom.fr.eu.org> writes:

> Le Vendredi, Octobre 16, 2020 18:24 CEST, Philippe Gerum <rpm@xenomai.org> a écrit: 
>  
>> 
>> François Legal <devel@thom.fr.eu.org> writes:
>> 
>> > Le Vendredi, Octobre 16, 2020 14:03 CEST, Philippe Gerum <rpm@xenomai.org> a écrit: 
>> >  
>> >> 
>> >> François Legal <devel@thom.fr.eu.org> writes:
>> >> 
>> >> > Le Vendredi, Octobre 16, 2020 10:59 CEST, Philippe Gerum <rpm@xenomai.org> a écrit: 
>> >> >  
>> >> >> 
>> >> >> François Legal <devel@thom.fr.eu.org> writes:
>> >> >> 
>> >> >> > Le Mercredi, Octobre 14, 2020 16:16 CEST, Greg Gallagher <greg@embeddedgreg.com> a écrit: 
>> >> >> >  
>> >> >> >> On Wed, Oct 14, 2020 at 5:37 AM Jan Kiszka <jan.kiszka@siemens.com> wrote:
>> >> >> >> >
>> >> >> >> > On 14.10.20 10:43, François Legal via Xenomai wrote:
>> >> >> >> > > Anybody can help on this ?
>> >> >> >> > >
>> > >> >> >
>> >> >> >> > I'm unfortunately not familiar with the armv7 details of copy-from-user,
>> >> >> >> > not too speak of how spectre contributed to it. Greg, Philippe, did you
>> >> >> >> > come across this already?
>> >> >> >> >
>> >> >> >> > Jan
>> >> >> >> >
>> >> >> >> I'll take a look tonight but I haven't hit this in my testing.  This
>> >> >> >> was found on 4.4? Have you tried the 4.19 kernels?
>> >> >> >> 
>> >> >> >> -Greg
>> >> >> >  
>> >> >> > So I tried the same case on 4.19.121, with the same result, and I guess for the same reason.
>> >> >> > Could anybody explain why, on ARMv7, we need to copy the message header at the syscall entry, and not let the xxxmsg routine do it on its own ?
>> >> >> > Also, I could not find how those COBALT_SYSCALL32emu logic work.
>> >> >> 
>> >> >> There is no way an armv7 system would run the sys32emu code in
>> >> >> Cobalt. This code path is specific to architectures which support mixed
>> >> >> ABI models. Only Cobalt/x86 supports this so far, issuing x86_32
>> >> >> syscalls to an x86_64 kernel. You may want to check
>> >> >> CONFIG_XENO_ARCH_SYS3264, it is set to "def_bool n" in the Kconfig
>> >> >> stuff.
>> >> >> 
>> >> >
>> >> > Maybe I don't use the right terms here, but what I can see from the code is (in linux tree kernel/xenomai/posix/syscall32.c)
>> >> > COBALT_SYSCALL32emu(sendmsg, handover,
>> >> > 		    (int fd, struct compat_msghdr __user *umsg, int flags))
>> >> > {
>> >> > 	struct user_msghdr m;
>> >> > 	int ret;
>> >> >
>> >> > 	ret = sys32_get_msghdr(&m, umsg);
>> >> >
>> >> > 	return ret ?: rtdm_fd_sendmsg(fd, &m, flags);
>> >> > }
>> >> >
>> >> > And the problem regarding SPECTRE mitigation is with the "ret =
>> >> > sys32_get_msghdr(&m, umsg);" line, as af_packet (in my case, but I
>> >> > believe the other handlers should do the same) will also call
>> >> > copy_from_user on the "msghdr" argument, and the SPECTRE mitigation
>> >> > will check that this pointer is in the userland MM area.
>> >> 
>> >> There is indeed a problem with this code passing the kernel memory
>> >> address of a bounce buffer to RTDM handlers which would expect __user
>> >> tagged memory instead. This ends up confusing any low-level
>> >> copy_to/from_user routine which includes attack
>> >> mitigation. rtnet_get_arg() does call such routine under the hood. This
>> >> is something some Xenomai contributor may want to address.
>> >> 
>> >> But, again, this sys32emu code cannot run for armv7 under the current
>> >> stock implementation. So what we are discussing is purely hypothetical
>> >> at this stage for this architecture, and should definitely never happen
>> >> by construction if you are running armv7 (which does not make the
>> >> original issue go away, that is granted).
>> >> 
>> >
>> > I'm not sure I quite understand that point. The code reproduced above is well built in the kernel. Are you saying this code is not called whenever userland calls sendmsg on an rt socket ? Am I looking in the wrong direction ? In that case, where should I be looking ? I mean, I tracked the whole thing with a JTAG debugger, and it seemed to me that this was what was really happening, with the SPECTRE code setting the pointer to 0 which was later being caught by arm_copy_from_user.
>> >
>> 
>> How could syscall32.c and compat.c be built into the kernel with
>> CONFIG_XENO_ARCH_SYS3264 forcibly unset in the Kconfig, which is always
>> the case when building for anything else than x86?
>> 
>> Checking kernel/cobalt/posix/Makefile may help in understanding why it
>> is simply not possible. arm_copy_from_user is built in, no question,
>> and your analysis regarding SVC context memory being spuriously fed into
>> arm_copy_from_user is likely right.
>> 
>> But the sys32 wrappers are neither for armv7, armv8 nor ppc32. So yes,
>> you are certainly following the wrong path when looking at
>> kernel/cobalt/posix/syscall32.c. This 32-to-64bit syscall support is NOT
>> built into a kernel targeting armv7, at least when it comes to the
>> vanilla Xenomai code.
>> 
>> You may want to double-check which call site actually invokes
>> arm_copy_from_user.
>> 
>> -- 
>> Philippe.
>  
> Thanks Philippe for pointing me in the right direction.
> So, if I'm correct this time, the problem is about the same, but in posix/io.c.
> In 
>  
> COBALT_SYSCALL(sendmsg, handover,
> 	       (int fd, struct user_msghdr __user *umsg, int flags))
> {
> 	struct user_msghdr m;
> 	int ret;
>
> 	ret = cobalt_copy_from_user(&m, umsg, sizeof(m));
>
> 	return ret ?: rtdm_fd_sendmsg(fd, &m, flags);
> }
>
> Same thing, the user_msghdr is allocated on the SVC stack, then copied from user, then handed over to the sendmsg handling function pertaining to that fd, and whenever that handling function call copy_from_user for the same user_msghdr pointer, it triggers the SPECTRE mitigation protection.
>
> What is still unclear to me is why this user_msghdr struct is copied here, and not left to the handling function, as in the sendmmsg syscalls, the struct mmsghdr is not copied from user in syscall entry.
>

Unlike sendmsg(), sendmmsg() has to deal with 32bit compat mode, so
__rtdm_fd_sendmmsg() receives a couple of helpers along with the user
pointer in order to do the right thing, depending on the 32/64 syscall
entry. sendmsg() was implemented much earlier too, when SPECTRE
mitigation was not there: passing SVC memory to copy_from/to_user
services was still wrong, but went unnoticed.

-- 
Philippe.

Re: Strange switching [threadname] to secondary mode after exception #0 in kernel-space at 0x80272444

[François Legal]

Le Samedi, Octobre 31, 2020 17:48 CET, Philippe Gerum <rpm@xenomai.org> a écrit:

>
> François Legal <devel@thom.fr.eu.org> writes:
>
> > Le Vendredi, Octobre 16, 2020 18:24 CEST, Philippe Gerum <rpm@xenomai.org> a écrit:
> >
> >>
> >> François Legal <devel@thom.fr.eu.org> writes:
> >>
> >> > Le Vendredi, Octobre 16, 2020 14:03 CEST, Philippe Gerum <rpm@xenomai.org> a écrit:
> >> >
> >> >>
> >> >> François Legal <devel@thom.fr.eu.org> writes:
> >> >>
> >> >> > Le Vendredi, Octobre 16, 2020 10:59 CEST, Philippe Gerum <rpm@xenomai.org> a écrit:
> >> >> >
> >> >> >>
> >> >> >> François Legal <devel@thom.fr.eu.org> writes:
> >> >> >>
> >> >> >> > Le Mercredi, Octobre 14, 2020 16:16 CEST, Greg Gallagher <greg@embeddedgreg.com> a écrit:
> >> >> >> >
> >> >> >> >> On Wed, Oct 14, 2020 at 5:37 AM Jan Kiszka <jan.kiszka@siemens.com> wrote:
> >> >> >> >> >
> >> >> >> >> > On 14.10.20 10:43, François Legal via Xenomai wrote:
> >> >> >> >> > > Anybody can help on this ?
> >> >> >> >> > >
> >> > >> >> >
> >> >> >> >> > I'm unfortunately not familiar with the armv7 details of copy-from-user,
> >> >> >> >> > not too speak of how spectre contributed to it. Greg, Philippe, did you
> >> >> >> >> > come across this already?
> >> >> >> >> >
> >> >> >> >> > Jan
> >> >> >> >> >
> >> >> >> >> I'll take a look tonight but I haven't hit this in my testing.  This
> >> >> >> >> was found on 4.4? Have you tried the 4.19 kernels?
> >> >> >> >>
> >> >> >> >> -Greg
> >> >> >> >
> >> >> >> > So I tried the same case on 4.19.121, with the same result, and I guess for the same reason.
> >> >> >> > Could anybody explain why, on ARMv7, we need to copy the message header at the syscall entry, and not let the xxxmsg routine do it on its own ?
> >> >> >> > Also, I could not find how those COBALT_SYSCALL32emu logic work.
> >> >> >>
> >> >> >> There is no way an armv7 system would run the sys32emu code in
> >> >> >> Cobalt. This code path is specific to architectures which support mixed
> >> >> >> ABI models. Only Cobalt/x86 supports this so far, issuing x86_32
> >> >> >> syscalls to an x86_64 kernel. You may want to check
> >> >> >> CONFIG_XENO_ARCH_SYS3264, it is set to "def_bool n" in the Kconfig
> >> >> >> stuff.
> >> >> >>
> >> >> >
> >> >> > Maybe I don't use the right terms here, but what I can see from the code is (in linux tree kernel/xenomai/posix/syscall32.c)
> >> >> > COBALT_SYSCALL32emu(sendmsg, handover,
> >> >> > 		    (int fd, struct compat_msghdr __user *umsg, int flags))
> >> >> > {
> >> >> > 	struct user_msghdr m;
> >> >> > 	int ret;
> >> >> >
> >> >> > 	ret = sys32_get_msghdr(&m, umsg);
> >> >> >
> >> >> > 	return ret ?: rtdm_fd_sendmsg(fd, &m, flags);
> >> >> > }
> >> >> >
> >> >> > And the problem regarding SPECTRE mitigation is with the "ret =
> >> >> > sys32_get_msghdr(&m, umsg);" line, as af_packet (in my case, but I
> >> >> > believe the other handlers should do the same) will also call
> >> >> > copy_from_user on the "msghdr" argument, and the SPECTRE mitigation
> >> >> > will check that this pointer is in the userland MM area.
> >> >>
> >> >> There is indeed a problem with this code passing the kernel memory
> >> >> address of a bounce buffer to RTDM handlers which would expect __user
> >> >> tagged memory instead. This ends up confusing any low-level
> >> >> copy_to/from_user routine which includes attack
> >> >> mitigation. rtnet_get_arg() does call such routine under the hood. This
> >> >> is something some Xenomai contributor may want to address.
> >> >>
> >> >> But, again, this sys32emu code cannot run for armv7 under the current
> >> >> stock implementation. So what we are discussing is purely hypothetical
> >> >> at this stage for this architecture, and should definitely never happen
> >> >> by construction if you are running armv7 (which does not make the
> >> >> original issue go away, that is granted).
> >> >>
> >> >
> >> > I'm not sure I quite understand that point. The code reproduced above is well built in the kernel. Are you saying this code is not called whenever userland calls sendmsg on an rt socket ? Am I looking in the wrong direction ? In that case, where should I be looking ? I mean, I tracked the whole thing with a JTAG debugger, and it seemed to me that this was what was really happening, with the SPECTRE code setting the pointer to 0 which was later being caught by arm_copy_from_user.
> >> >
> >>
> >> How could syscall32.c and compat.c be built into the kernel with
> >> CONFIG_XENO_ARCH_SYS3264 forcibly unset in the Kconfig, which is always
> >> the case when building for anything else than x86?
> >>
> >> Checking kernel/cobalt/posix/Makefile may help in understanding why it
> >> is simply not possible. arm_copy_from_user is built in, no question,
> >> and your analysis regarding SVC context memory being spuriously fed into
> >> arm_copy_from_user is likely right.
> >>
> >> But the sys32 wrappers are neither for armv7, armv8 nor ppc32. So yes,
> >> you are certainly following the wrong path when looking at
> >> kernel/cobalt/posix/syscall32.c. This 32-to-64bit syscall support is NOT
> >> built into a kernel targeting armv7, at least when it comes to the
> >> vanilla Xenomai code.
> >>
> >> You may want to double-check which call site actually invokes
> >> arm_copy_from_user.
> >>
> >> --
> >> Philippe.
> >
> > Thanks Philippe for pointing me in the right direction.
> > So, if I'm correct this time, the problem is about the same, but in posix/io.c.
> > In
> >
> > COBALT_SYSCALL(sendmsg, handover,
> > 	       (int fd, struct user_msghdr __user *umsg, int flags))
> > {
> > 	struct user_msghdr m;
> > 	int ret;
> >
> > 	ret = cobalt_copy_from_user(&m, umsg, sizeof(m));
> >
> > 	return ret ?: rtdm_fd_sendmsg(fd, &m, flags);
> > }
> >
> > Same thing, the user_msghdr is allocated on the SVC stack, then copied from user, then handed over to the sendmsg handling function pertaining to that fd, and whenever that handling function call copy_from_user for the same user_msghdr pointer, it triggers the SPECTRE mitigation protection.
> >
> > What is still unclear to me is why this user_msghdr struct is copied here, and not left to the handling function, as in the sendmmsg syscalls, the struct mmsghdr is not copied from user in syscall entry.
> >
>
> Unlike sendmsg(), sendmmsg() has to deal with 32bit compat mode, so
> __rtdm_fd_sendmmsg() receives a couple of helpers along with the user
> pointer in order to do the right thing, depending on the 32/64 syscall
> entry. sendmsg() was implemented much earlier too, when SPECTRE
> mitigation was not there: passing SVC memory to copy_from/to_user
> services was still wrong, but went unnoticed.
>
> --
> Philippe.

 Alright, so shall I submit a patch removing those copy_from/to_user calls at system call entry in sendmsg and recvmsg ?
Is this the way to go ?

Thanks

François

Re: Strange switching [threadname] to secondary mode after exception #0 in kernel-space at 0x80272444

[François Legal]

Le Lundi, Novembre 02, 2020 09:08 CET, François Legal via Xenomai <xenomai@xenomai.org> a écrit:

> Le Samedi, Octobre 31, 2020 17:48 CET, Philippe Gerum <rpm@xenomai.org> a écrit:
>
> >
> > François Legal <devel@thom.fr.eu.org> writes:
> >
> > > Le Vendredi, Octobre 16, 2020 18:24 CEST, Philippe Gerum <rpm@xenomai.org> a écrit:
> > >
> > >>
> > >> François Legal <devel@thom.fr.eu.org> writes:
> > >>
> > >> > Le Vendredi, Octobre 16, 2020 14:03 CEST, Philippe Gerum <rpm@xenomai.org> a écrit:
> > >> >
> > >> >>
> > >> >> François Legal <devel@thom.fr.eu.org> writes:
> > >> >>
> > >> >> > Le Vendredi, Octobre 16, 2020 10:59 CEST, Philippe Gerum <rpm@xenomai.org> a écrit:
> > >> >> >
> > >> >> >>
> > >> >> >> François Legal <devel@thom.fr.eu.org> writes:
> > >> >> >>
> > >> >> >> > Le Mercredi, Octobre 14, 2020 16:16 CEST, Greg Gallagher <greg@embeddedgreg.com> a écrit:
> > >> >> >> >
> > >> >> >> >> On Wed, Oct 14, 2020 at 5:37 AM Jan Kiszka <jan.kiszka@siemens.com> wrote:
> > >> >> >> >> >
> > >> >> >> >> > On 14.10.20 10:43, François Legal via Xenomai wrote:
> > >> >> >> >> > > Anybody can help on this ?
> > >> >> >> >> > >
> > >> > >> >> >
> > >> >> >> >> > I'm unfortunately not familiar with the armv7 details of copy-from-user,
> > >> >> >> >> > not too speak of how spectre contributed to it. Greg, Philippe, did you
> > >> >> >> >> > come across this already?
> > >> >> >> >> >
> > >> >> >> >> > Jan
> > >> >> >> >> >
> > >> >> >> >> I'll take a look tonight but I haven't hit this in my testing.  This
> > >> >> >> >> was found on 4.4? Have you tried the 4.19 kernels?
> > >> >> >> >>
> > >> >> >> >> -Greg
> > >> >> >> >
> > >> >> >> > So I tried the same case on 4.19.121, with the same result, and I guess for the same reason.
> > >> >> >> > Could anybody explain why, on ARMv7, we need to copy the message header at the syscall entry, and not let the xxxmsg routine do it on its own ?
> > >> >> >> > Also, I could not find how those COBALT_SYSCALL32emu logic work.
> > >> >> >>
> > >> >> >> There is no way an armv7 system would run the sys32emu code in
> > >> >> >> Cobalt. This code path is specific to architectures which support mixed
> > >> >> >> ABI models. Only Cobalt/x86 supports this so far, issuing x86_32
> > >> >> >> syscalls to an x86_64 kernel. You may want to check
> > >> >> >> CONFIG_XENO_ARCH_SYS3264, it is set to "def_bool n" in the Kconfig
> > >> >> >> stuff.
> > >> >> >>
> > >> >> >
> > >> >> > Maybe I don't use the right terms here, but what I can see from the code is (in linux tree kernel/xenomai/posix/syscall32.c)
> > >> >> > COBALT_SYSCALL32emu(sendmsg, handover,
> > >> >> > 		    (int fd, struct compat_msghdr __user *umsg, int flags))
> > >> >> > {
> > >> >> > 	struct user_msghdr m;
> > >> >> > 	int ret;
> > >> >> >
> > >> >> > 	ret = sys32_get_msghdr(&m, umsg);
> > >> >> >
> > >> >> > 	return ret ?: rtdm_fd_sendmsg(fd, &m, flags);
> > >> >> > }
> > >> >> >
> > >> >> > And the problem regarding SPECTRE mitigation is with the "ret =
> > >> >> > sys32_get_msghdr(&m, umsg);" line, as af_packet (in my case, but I
> > >> >> > believe the other handlers should do the same) will also call> >> >> > copy_from_user on the "msghdr" argument, and the SPECTRE mitigation
> > >> >> > will check that this pointer is in the userland MM area.
> > >> >>
> > >> >> There is indeed a problem with this code passing the kernel memory
> > >> >> address of a bounce buffer to RTDM handlers which would expect __user
> > >> >> tagged memory instead. This ends up confusing any low-level
> > >> >> copy_to/from_user routine which includes attack
> > >> >> mitigation. rtnet_get_arg() does call such routine under the hood. This
> > >> >> is something some Xenomai contributor may want to address.
> > >> >>
> > >> >> But, again, this sys32emu code cannot run for armv7 under the current
> > >> >> stock implementation. So what we are discussing is purely hypothetical
> > >> >> at this stage for this architecture, and should definitely never happen
> > >> >> by construction if you are running armv7 (which does not make the
> > >> >> original issue go away, that is granted).
> > >> >>
> > >> >
> > >> > I'm not sure I quite understand that point. The code reproduced above is well built in the kernel. Are you saying this code is not called whenever userland calls sendmsg on an rt socket ? Am I looking in the wrong direction ? In that case, where should I be looking ? I mean, I tracked the whole thing with a JTAG debugger, and it seemed to me that this was what was really happening, with the SPECTRE code setting the pointer to 0 which was later being caught by arm_copy_from_user.
> > >> >
> > >>
> > >> How could syscall32.c and compat.c be built into the kernel with
> > >> CONFIG_XENO_ARCH_SYS3264 forcibly unset in the Kconfig, which is always
> > >> the case when building for anything else than x86?
> > >>
> > >> Checking kernel/cobalt/posix/Makefile may help in understanding why it
> > >> is simply not possible. arm_copy_from_user is built in, no question,
> > >> and your analysis regarding SVC context memory being spuriously fed into
> > >> arm_copy_from_user is likely right.
> > >>
> > >> But the sys32 wrappers are neither for armv7, armv8 nor ppc32. So yes,
> > >> you are certainly following the wrong path when looking at
> > >> kernel/cobalt/posix/syscall32.c. This 32-to-64bit syscall support is NOT
> > >> built into a kernel targeting armv7, at least when it comes to the> >> vanilla Xenomai code.
> > >>
> > >> You may want to double-check which call site actually invokes
> > >> arm_copy_from_user.
> > >>
> > >> --
> > >> Philippe.
> > >
> > > Thanks Philippe for pointing me in the right direction.
> > > So, if I'm correct this time, the problem is about the same, but in posix/io.c.
> > > In
> > >
> > > COBALT_SYSCALL(sendmsg, handover,
> > > 	       (int fd, struct user_msghdr __user *umsg, int flags))> > {
> > > 	struct user_msghdr m;
> > > 	int ret;
> > >
> > > 	ret = cobalt_copy_from_user(&m, umsg, sizeof(m));
> > >
> > > 	return ret ?: rtdm_fd_sendmsg(fd, &m, flags);
> > > }
> > >
> > > Same thing, the user_msghdr is allocated on the SVC stack, then copied from user, then handed over to the sendmsg handling function pertaining to that fd, and whenever that handling function call copy_from_user for the same user_msghdr pointer, it triggers the SPECTRE mitigation protection.
> > >
> > > What is still unclear to me is why this user_msghdr struct is copied here, and not left to the handling function, as in the sendmmsg syscalls, the struct mmsghdr is not copied from user in syscall entry.
> > >
> >
> > Unlike sendmsg(), sendmmsg() has to deal with 32bit compat mode, so
> > __rtdm_fd_sendmmsg() receives a couple of helpers along with the user
> > pointer in order to do the right thing, depending on the 32/64 syscall
> > entry. sendmsg() was implemented much earlier too, when SPECTRE
> > mitigation was not there: passing SVC memory to copy_from/to_user> services was still wrong, but went unnoticed.
> >
> > --
> > Philippe.
>
>  Alright, so shall I submit a patch removing those copy_from/to_user calls at system call entry in sendmsg and recvmsg ?
> Is this the way to go ?
>
> Thanks
>
> François
>
>

So for the records, I tried to patch posix/io.c to remove the copy_to/from_user from there to try to avoid the SPECTRE problem, and while that seem to work perfectly well for the rt_packet endpoint, I come across another problem in xddp sockets which do expect the struct user_msghdr to be present in kernel space.

So I still wonder which is the way to go ? Removing the copy_from/to_user in posix/io.c  and modifying xddp (and possibly other protocols relying on that) to to add the copy_to/from_user there, or modify only af_packet (and possibly/probably all sockets type in RTNET) ?

Thanks

François

Re: Strange switching [threadname] to secondary mode after exception #0 in kernel-space at 0x80272444

[Philippe Gerum]

François Legal <devel@thom.fr.eu.org> writes:

> Le Lundi, Novembre 02, 2020 09:08 CET, François Legal via Xenomai <xenomai@xenomai.org> a écrit: 
>  
>> Le Samedi, Octobre 31, 2020 17:48 CET, Philippe Gerum <rpm@xenomai.org> a écrit: 
>>  
>> > 
>> > François Legal <devel@thom.fr.eu.org> writes:
>> > 
>> > > Le Vendredi, Octobre 16, 2020 18:24 CEST, Philippe Gerum <rpm@xenomai.org> a écrit: 
>> > >  
>> > >> 
>> > >> François Legal <devel@thom.fr.eu.org> writes:
>> > >> 
>> > >> > Le Vendredi, Octobre 16, 2020 14:03 CEST, Philippe Gerum <rpm@xenomai.org> a écrit: 
>> > >> >  
>> > >> >> 
>> > >> >> François Legal <devel@thom.fr.eu.org> writes:
>> > >> >> 
>> > >> >> > Le Vendredi, Octobre 16, 2020 10:59 CEST, Philippe Gerum <rpm@xenomai.org> a écrit: 
>> > >> >> >  
>> > >> >> >> 
>> > >> >> >> François Legal <devel@thom.fr.eu.org> writes:
>> > >> >> >> 
>> > >> >> >> > Le Mercredi, Octobre 14, 2020 16:16 CEST, Greg Gallagher <greg@embeddedgreg.com> a écrit: 
>> > >> >> >> >  
>> > >> >> >> >> On Wed, Oct 14, 2020 at 5:37 AM Jan Kiszka <jan.kiszka@siemens.com> wrote:
>> > >> >> >> >> >
>> > >> >> >> >> > On 14.10.20 10:43, François Legal via Xenomai wrote:
>> > >> >> >> >> > > Anybody can help on this ?
>> > >> >> >> >> > >
>> > >> > >> >> >
>> > >> >> >> >> > I'm unfortunately not familiar with the armv7 details of copy-from-user,
>> > >> >> >> >> > not too speak of how spectre contributed to it. Greg, Philippe, did you
>> > >> >> >> >> > come across this already?
>> > >> >> >> >> >
>> > >> >> >> >> > Jan
>> > >> >> >> >> >
>> > >> >> >> >> I'll take a look tonight but I haven't hit this in my testing.  This
>> > >> >> >> >> was found on 4.4? Have you tried the 4.19 kernels?
>> > >> >> >> >> 
>> > >> >> >> >> -Greg
>> > >> >> >> >  
>> > >> >> >> > So I tried the same case on 4.19.121, with the same result, and I guess for the same reason.
>> > >> >> >> > Could anybody explain why, on ARMv7, we need to copy the message header at the syscall entry, and not let the xxxmsg routine do it on its own ?
>> > >> >> >> > Also, I could not find how those COBALT_SYSCALL32emu logic work.
>> > >> >> >> 
>> > >> >> >> There is no way an armv7 system would run the sys32emu code in
>> > >> >> >> Cobalt. This code path is specific to architectures which support mixed
>> > >> >> >> ABI models. Only Cobalt/x86 supports this so far, issuing x86_32
>> > >> >> >> syscalls to an x86_64 kernel. You may want to check
>> > >> >> >> CONFIG_XENO_ARCH_SYS3264, it is set to "def_bool n" in the Kconfig
>> > >> >> >> stuff.
>> > >> >> >> 
>> > >> >> >
>> > >> >> > Maybe I don't use the right terms here, but what I can see from the code is (in linux tree kernel/xenomai/posix/syscall32.c)
>> > >> >> > COBALT_SYSCALL32emu(sendmsg, handover,
>> > >> >> > 		    (int fd, struct compat_msghdr __user *umsg, int flags))
>> > >> >> > {
>> > >> >> > 	struct user_msghdr m;
>> > >> >> > 	int ret;
>> > >> >> >
>> > >> >> > 	ret = sys32_get_msghdr(&m, umsg);
>> > >> >> >
>> > >> >> > 	return ret ?: rtdm_fd_sendmsg(fd, &m, flags);
>> > >> >> > }
>> > >> >> >
>> > >> >> > And the problem regarding SPECTRE mitigation is with the "ret =
>> > >> >> > sys32_get_msghdr(&m, umsg);" line, as af_packet (in my case, but I
>> > >> >> > believe the other handlers should do the same) will also call> >> >> > copy_from_user on the "msghdr" argument, and the SPECTRE mitigation
>> > >> >> > will check that this pointer is in the userland MM area.
>> > >> >> 
>> > >> >> There is indeed a problem with this code passing the kernel memory
>> > >> >> address of a bounce buffer to RTDM handlers which would expect __user
>> > >> >> tagged memory instead. This ends up confusing any low-level
>> > >> >> copy_to/from_user routine which includes attack
>> > >> >> mitigation. rtnet_get_arg() does call such routine under the hood. This
>> > >> >> is something some Xenomai contributor may want to address.
>> > >> >> 
>> > >> >> But, again, this sys32emu code cannot run for armv7 under the current
>> > >> >> stock implementation. So what we are discussing is purely hypothetical
>> > >> >> at this stage for this architecture, and should definitely never happen
>> > >> >> by construction if you are running armv7 (which does not make the
>> > >> >> original issue go away, that is granted).
>> > >> >> 
>> > >> >
>> > >> > I'm not sure I quite understand that point. The code reproduced above is well built in the kernel. Are you saying this code is not called whenever userland calls sendmsg on an rt socket ? Am I looking in the wrong direction ? In that case, where should I be looking ? I mean, I tracked the whole thing with a JTAG debugger, and it seemed to me that this was what was really happening, with the SPECTRE code setting the pointer to 0 which was later being caught by arm_copy_from_user.
>> > >> >
>> > >> 
>> > >> How could syscall32.c and compat.c be built into the kernel with
>> > >> CONFIG_XENO_ARCH_SYS3264 forcibly unset in the Kconfig, which is always
>> > >> the case when building for anything else than x86?
>> > >> 
>> > >> Checking kernel/cobalt/posix/Makefile may help in understanding why it
>> > >> is simply not possible. arm_copy_from_user is built in, no question,
>> > >> and your analysis regarding SVC context memory being spuriously fed into
>> > >> arm_copy_from_user is likely right.
>> > >> 
>> > >> But the sys32 wrappers are neither for armv7, armv8 nor ppc32. So yes,
>> > >> you are certainly following the wrong path when looking at
>> > >> kernel/cobalt/posix/syscall32.c. This 32-to-64bit syscall support is NOT
>> > >> built into a kernel targeting armv7, at least when it comes to the> >> vanilla Xenomai code.
>> > >> 
>> > >> You may want to double-check which call site actually invokes
>> > >> arm_copy_from_user.
>> > >> 
>> > >> -- 
>> > >> Philippe.
>> > >  
>> > > Thanks Philippe for pointing me in the right direction.
>> > > So, if I'm correct this time, the problem is about the same, but in posix/io.c.
>> > > In 
>> > >  
>> > > COBALT_SYSCALL(sendmsg, handover,
>> > > 	       (int fd, struct user_msghdr __user *umsg, int flags))> > {
>> > > 	struct user_msghdr m;
>> > > 	int ret;
>> > >
>> > > 	ret = cobalt_copy_from_user(&m, umsg, sizeof(m));
>> > >
>> > > 	return ret ?: rtdm_fd_sendmsg(fd, &m, flags);
>> > > }
>> > >
>> > > Same thing, the user_msghdr is allocated on the SVC stack, then copied from user, then handed over to the sendmsg handling function pertaining to that fd, and whenever that handling function call copy_from_user for the same user_msghdr pointer, it triggers the SPECTRE mitigation protection.
>> > >
>> > > What is still unclear to me is why this user_msghdr struct is copied here, and not left to the handling function, as in the sendmmsg syscalls, the struct mmsghdr is not copied from user in syscall entry.
>> > >
>> > 
>> > Unlike sendmsg(), sendmmsg() has to deal with 32bit compat mode, so
>> > __rtdm_fd_sendmmsg() receives a couple of helpers along with the user
>> > pointer in order to do the right thing, depending on the 32/64 syscall
>> > entry. sendmsg() was implemented much earlier too, when SPECTRE
>> > mitigation was not there: passing SVC memory to copy_from/to_user> services was still wrong, but went unnoticed.
>> > 
>> > -- 
>> > Philippe.
>>  
>>  Alright, so shall I submit a patch removing those copy_from/to_user calls at system call entry in sendmsg and recvmsg ?
>> Is this the way to go ?
>> 
>> Thanks
>> 
>> François
>> 
>> 
>  
> So for the records, I tried to patch posix/io.c to remove the copy_to/from_user from there to try to avoid the SPECTRE problem, and while that seem to work perfectly well for the rt_packet endpoint, I come across another problem in xddp sockets which do expect the struct user_msghdr to be present in kernel space.
>
> So I still wonder which is the way to go ? Removing the copy_from/to_user in posix/io.c  and modifying xddp (and possibly other protocols relying on that) to to add the copy_to/from_user there, or modify only af_packet (and possibly/probably all sockets type in RTNET) ?
>
> Thanks
>
> François


The IPC module implementing XDDP is independent from RTnet, so you would
have to fix all offending callers separately, including XDDP.

-- 
Philippe.

Re: Strange switching [threadname] to secondary mode after exception #0 in kernel-space at 0x80272444

[François Legal]

Le Vendredi, Octobre 16, 2020 18:24 CEST, Philippe Gerum <rpm@xenomai.org> a écrit:

>
> François Legal <devel@thom.fr.eu.org> writes:
>
> > Le Vendredi, Octobre 16, 2020 14:03 CEST, Philippe Gerum <rpm@xenomai.org> a écrit:
> >
> >>
> >> François Legal <devel@thom.fr.eu.org> writes:
> >>
> >> > Le Vendredi, Octobre 16, 2020 10:59 CEST, Philippe Gerum <rpm@xenomai.org> a écrit:
> >> >
> >> >>
> >> >> François Legal <devel@thom.fr.eu.org> writes:
> >> >>
> >> >> > Le Mercredi, Octobre 14, 2020 16:16 CEST, Greg Gallagher <greg@embeddedgreg.com> a écrit:
> >> >> >
> >> >> >> On Wed, Oct 14, 2020 at 5:37 AM Jan Kiszka <jan.kiszka@siemens.com> wrote:
> >> >> >> >
> >> >> >> > On 14.10.20 10:43, François Legal via Xenomai wrote:
> >> >> >> > > Anybody can help on this ?
> >> >> >> > >
> > >> >> >
> >> >> >> > I'm unfortunately not familiar with the armv7 details of copy-from-user,
> >> >> >> > not too speak of how spectre contributed to it. Greg, Philippe, did you
> >> >> >> > come across this already?
> >> >> >> >
> >> >> >> > Jan
> >> >> >> >
> >> >> >> I'll take a look tonight but I haven't hit this in my testing.  This
> >> >> >> was found on 4.4? Have you tried the 4.19 kernels?
> >> >> >>
> >> >> >> -Greg
> >> >> >
> >> >> > So I tried the same case on 4.19.121, with the same result, and I guess for the same reason.
> >> >> > Could anybody explain why, on ARMv7, we need to copy the message header at the syscall entry, and not let the xxxmsg routine do it on its own ?
> >> >> > Also, I could not find how those COBALT_SYSCALL32emu logic work.
> >> >>
> >> >> There is no way an armv7 system would run the sys32emu code in
> >> >> Cobalt. This code path is specific to architectures which support mixed
> >> >> ABI models. Only Cobalt/x86 supports this so far, issuing x86_32
> >> >> syscalls to an x86_64 kernel. You may want to check
> >> >> CONFIG_XENO_ARCH_SYS3264, it is set to "def_bool n" in the Kconfig
> >> >> stuff.
> >> >>
> >> >
> >> > Maybe I don't use the right terms here, but what I can see from the code is (in linux tree kernel/xenomai/posix/syscall32.c)
> >> > COBALT_SYSCALL32emu(sendmsg, handover,
> >> > 		    (int fd, struct compat_msghdr __user *umsg, int flags))
> >> > {
> >> > 	struct user_msghdr m;
> >> > 	int ret;
> >> >
> >> > 	ret = sys32_get_msghdr(&m, umsg);
> >> >
> >> > 	return ret ?: rtdm_fd_sendmsg(fd, &m, flags);
> >> > }
> >> >
> >> > And the problem regarding SPECTRE mitigation is with the "ret =
> >> > sys32_get_msghdr(&m, umsg);" line, as af_packet (in my case, but I
> >> > believe the other handlers should do the same) will also call
> >> > copy_from_user on the "msghdr" argument, and the SPECTRE mitigation
> >> > will check that this pointer is in the userland MM area.
> >>
> >> There is indeed a problem with this code passing the kernel memory
> >> address of a bounce buffer to RTDM handlers which would expect __user
> >> tagged memory instead. This ends up confusing any low-level
> >> copy_to/from_user routine which includes attack
> >> mitigation. rtnet_get_arg() does call such routine under the hood. This
> >> is something some Xenomai contributor may want to address.
> >>
> >> But, again, this sys32emu code cannot run for armv7 under the current
> >> stock implementation. So what we are discussing is purely hypothetical
> >> at this stage for this architecture, and should definitely never happen
> >> by construction if you are running armv7 (which does not make the
> >> original issue go away, that is granted).
> >>
> >
> > I'm not sure I quite understand that point. The code reproduced above is well built in the kernel. Are you saying this code is not called whenever userland calls sendmsg on an rt socket ? Am I looking in the wrong direction ? In that case, where should I be looking ? I mean, I tracked the whole thing with a JTAG debugger, and it seemed to me that this was what was really happening, with the SPECTRE code setting the pointer to 0 which was later being caught by arm_copy_from_user.
> >
>
> How could syscall32.c and compat.c be built into the kernel with
> CONFIG_XENO_ARCH_SYS3264 forcibly unset in the Kconfig, which is always
> the case when building for anything else than x86?
>
> Checking kernel/cobalt/posix/Makefile may help in understanding why it
> is simply not possible. arm_copy_from_user is built in, no question,
> and your analysis regarding SVC context memory being spuriously fed into
> arm_copy_from_user is likely right.
>
> But the sys32 wrappers are neither for armv7, armv8 nor ppc32. So yes,
> you are certainly following the wrong path when looking at
> kernel/cobalt/posix/syscall32.c. This 32-to-64bit syscall support is NOT
> built into a kernel targeting armv7, at least when it comes to the
> vanilla Xenomai code.
>
> You may want to double-check which call site actually invokes
> arm_copy_from_user.
>
> --
> Philippe.

Just to follow up on that. I tried to modify cobalt system calls (sendmsg/recvmsg) to remove the copy_from_user as follows and run with SPECTRE mitigation built in :

 COBALT_SYSCALL(recvmsg, handover,
	       (int fd, struct user_msghdr __user *umsg, int flags))
{
//	struct user_msghdr m;
	ssize_t ret;
//
//	ret = cobalt_copy_from_user(&m, umsg, sizeof(m));
//	if (ret)
//		return ret;
//
//    ret = rtdm_fd_recvmsg(fd, &m, flags);
    ret = rtdm_fd_recvmsg(fd, umsg, flags);
//	if (ret < 0)
		return ret;

//	return cobalt_copy_to_user(umsg, &m, sizeof(*umsg)) ?: ret;
}

COBALT_SYSCALL(sendmsg, handover,
	       (int fd, struct user_msghdr __user *umsg, int flags))
{
//	struct user_msghdr m;
//	int ret;
//
//	ret = cobalt_copy_from_user(&m, umsg, sizeof(m));
//
//  return ret ?: rtdm_fd_sendmsg(fd, &m, flags);
  return rtdm_fd_sendmsg(fd, umsg, flags);
}

I don't get no more exceptions, and still the RTNET network seems functionnal.
So now, I wonder why these copy_from/to_user calls need to be there at the syscall entry ?

François