[Xenomai] [PULL] ipipe: fixes for 3.14

[Xenomai] [PULL] ipipe: fixes for 3.14

[Jan Kiszka]

The following changes since commit 00d8a6f2e95453f61ae97b6587dba03bc91c7e1a:

  arm/ipipe: Resolve trival merge conflicts (2015-01-14 17:00:22 +0100)

are available in the git repository at:

  git://git.xenomai.org/ipipe-jki 

for you to fetch changes up to f982e48ce99e1bc9e71eac21830adac2db265c98:

  x86/ipipe: Restore invocation of ipipe_unlock_irq from startup_ioapic_irq (2015-02-16 18:22:21 +0100)

All patches apply to 3.16 as well but weren't tested there yet. Still
analyzing strange

traps: (systemd)[1420] trap invalid opcode ip:7f4d138f612b sp:7fff4fd8f568 error:0 in libpthread-2.19.so[7f4d138e5000+18000]

on the target under 3.14...

----------------------------------------------------------------
Jan Kiszka (3):
      arm/ipipe: Fix ret_from_exception for !CONFIG_IPIPE
      ipipe: Disable PROVE_RCU when I-pipe is on
      x86/ipipe: Restore invocation of ipipe_unlock_irq from startup_ioapic_irq

 arch/arm/kernel/entry-armv.S   | 4 +++-
 arch/x86/kernel/apic/io_apic.c | 1 +
 lib/Kconfig.debug              | 2 +-
 3 files changed, 5 insertions(+), 2 deletions(-)

Re: [Xenomai] [PULL] ipipe: fixes for 3.14

[Philippe Gerum]

On 02/16/2015 06:35 PM, Jan Kiszka wrote:
> The following changes since commit 00d8a6f2e95453f61ae97b6587dba03bc91c7e1a:
> 
>   arm/ipipe: Resolve trival merge conflicts (2015-01-14 17:00:22 +0100)
> 
> are available in the git repository at:
> 
>   git://git.xenomai.org/ipipe-jki 
> 
> for you to fetch changes up to f982e48ce99e1bc9e71eac21830adac2db265c98:
> 
>   x86/ipipe: Restore invocation of ipipe_unlock_irq from startup_ioapic_irq (2015-02-16 18:22:21 +0100)
> 
> All patches apply to 3.16 as well but weren't tested there yet. Still
> analyzing strange
> 
> traps: (systemd)[1420] trap invalid opcode ip:7f4d138f612b sp:7fff4fd8f568 error:0 in libpthread-2.19.so[7f4d138e5000+18000]
> 
> on the target under 3.14...
> 

Register trashing on the fast call path (system_call_after_gs) maybe.
Forcing CONFIG_IPIPE_LEGACY there might help ruling this out.

-- 
Philippe.

Re: [Xenomai] [PULL] ipipe: fixes for 3.14

[Jan Kiszka]

On 2015-02-16 18:48, Philippe Gerum wrote:
> On 02/16/2015 06:35 PM, Jan Kiszka wrote:
>> The following changes since commit 00d8a6f2e95453f61ae97b6587dba03bc91c7e1a:
>>
>>   arm/ipipe: Resolve trival merge conflicts (2015-01-14 17:00:22 +0100)
>>
>> are available in the git repository at:
>>
>>   git://git.xenomai.org/ipipe-jki 
>>
>> for you to fetch changes up to f982e48ce99e1bc9e71eac21830adac2db265c98:
>>
>>   x86/ipipe: Restore invocation of ipipe_unlock_irq from startup_ioapic_irq (2015-02-16 18:22:21 +0100)
>>
>> All patches apply to 3.16 as well but weren't tested there yet. Still
>> analyzing strange
>>
>> traps: (systemd)[1420] trap invalid opcode ip:7f4d138f612b sp:7fff4fd8f568 error:0 in libpthread-2.19.so[7f4d138e5000+18000]
>>
>> on the target under 3.14...
>>
> 
> Register trashing on the fast call path (system_call_after_gs) maybe.
> Forcing CONFIG_IPIPE_LEGACY there might help ruling this out.

That's without any Xenomai configured. But I will check.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] [PULL] ipipe: fixes for 3.14

[Jan Kiszka]

On 2015-02-16 18:52, Jan Kiszka wrote:
> On 2015-02-16 18:48, Philippe Gerum wrote:
>> On 02/16/2015 06:35 PM, Jan Kiszka wrote:
>>> The following changes since commit 00d8a6f2e95453f61ae97b6587dba03bc91c7e1a:
>>>
>>>   arm/ipipe: Resolve trival merge conflicts (2015-01-14 17:00:22 +0100)
>>>
>>> are available in the git repository at:
>>>
>>>   git://git.xenomai.org/ipipe-jki 
>>>
>>> for you to fetch changes up to f982e48ce99e1bc9e71eac21830adac2db265c98:
>>>
>>>   x86/ipipe: Restore invocation of ipipe_unlock_irq from startup_ioapic_irq (2015-02-16 18:22:21 +0100)
>>>
>>> All patches apply to 3.16 as well but weren't tested there yet. Still
>>> analyzing strange
>>>
>>> traps: (systemd)[1420] trap invalid opcode ip:7f4d138f612b sp:7fff4fd8f568 error:0 in libpthread-2.19.so[7f4d138e5000+18000]
>>>
>>> on the target under 3.14...
>>>
>>
>> Register trashing on the fast call path (system_call_after_gs) maybe.
>> Forcing CONFIG_IPIPE_LEGACY there might help ruling this out.
> 
> That's without any Xenomai configured. But I will check.

This is where it traps in libpthread:

   1112b:       c7 f8 00 00 00 00       xbeginq 11131 <__lll_lock_elision+0x51>

For some reason, glibc starts to believe it could use RTM on that
machine - although this is clearly not available (Haswell i7, thus a
buggy CPU). Need to understand which feature check goes wrong, and why
only on I-pipe enabled kernels.

The good news is that I still have a similar and unresolved report from
early customer tests. That could solve their issues as well (they
caused massive domain migrations).

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] [PULL] ipipe: fixes for 3.14

[Jan Kiszka]

On 2015-02-16 19:52, Jan Kiszka wrote:
> On 2015-02-16 18:52, Jan Kiszka wrote:
>> On 2015-02-16 18:48, Philippe Gerum wrote:
>>> On 02/16/2015 06:35 PM, Jan Kiszka wrote:
>>>> The following changes since commit 00d8a6f2e95453f61ae97b6587dba03bc91c7e1a:
>>>>
>>>>   arm/ipipe: Resolve trival merge conflicts (2015-01-14 17:00:22 +0100)
>>>>
>>>> are available in the git repository at:
>>>>
>>>>   git://git.xenomai.org/ipipe-jki 
>>>>
>>>> for you to fetch changes up to f982e48ce99e1bc9e71eac21830adac2db265c98:
>>>>
>>>>   x86/ipipe: Restore invocation of ipipe_unlock_irq from startup_ioapic_irq (2015-02-16 18:22:21 +0100)
>>>>
>>>> All patches apply to 3.16 as well but weren't tested there yet. Still
>>>> analyzing strange
>>>>
>>>> traps: (systemd)[1420] trap invalid opcode ip:7f4d138f612b sp:7fff4fd8f568 error:0 in libpthread-2.19.so[7f4d138e5000+18000]
>>>>
>>>> on the target under 3.14...
>>>>
>>>
>>> Register trashing on the fast call path (system_call_after_gs) maybe.
>>> Forcing CONFIG_IPIPE_LEGACY there might help ruling this out.
>>
>> That's without any Xenomai configured. But I will check.
> 
> This is where it traps in libpthread:
> 
>    1112b:       c7 f8 00 00 00 00       xbeginq 11131 <__lll_lock_elision+0x51>
> 
> For some reason, glibc starts to believe it could use RTM on that
> machine - although this is clearly not available (Haswell i7, thus a
> buggy CPU). Need to understand which feature check goes wrong, and why
> only on I-pipe enabled kernels.
> 
> The good news is that I still have a similar and unresolved report from
> early customer tests. That could solve their issues as well (they
> caused massive domain migrations).

Wait - we were already one step further: a potential trigger for this
behaviour is a nullified pthread mutex. The decision to take the TSX
path is based on field in that struct, and 0 means transactional. Guess
I need a better reproduction case than some systemd service.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] [PULL] ipipe: fixes for 3.14

[Philippe Gerum]

On 02/16/2015 08:11 PM, Jan Kiszka wrote:
> On 2015-02-16 19:52, Jan Kiszka wrote:
>> On 2015-02-16 18:52, Jan Kiszka wrote:
>>> On 2015-02-16 18:48, Philippe Gerum wrote:
>>>> On 02/16/2015 06:35 PM, Jan Kiszka wrote:
>>>>> The following changes since commit 00d8a6f2e95453f61ae97b6587dba03bc91c7e1a:
>>>>>
>>>>>   arm/ipipe: Resolve trival merge conflicts (2015-01-14 17:00:22 +0100)
>>>>>
>>>>> are available in the git repository at:
>>>>>
>>>>>   git://git.xenomai.org/ipipe-jki 
>>>>>
>>>>> for you to fetch changes up to f982e48ce99e1bc9e71eac21830adac2db265c98:
>>>>>
>>>>>   x86/ipipe: Restore invocation of ipipe_unlock_irq from startup_ioapic_irq (2015-02-16 18:22:21 +0100)
>>>>>
>>>>> All patches apply to 3.16 as well but weren't tested there yet. Still
>>>>> analyzing strange
>>>>>
>>>>> traps: (systemd)[1420] trap invalid opcode ip:7f4d138f612b sp:7fff4fd8f568 error:0 in libpthread-2.19.so[7f4d138e5000+18000]
>>>>>
>>>>> on the target under 3.14...
>>>>>
>>>>
>>>> Register trashing on the fast call path (system_call_after_gs) maybe.
>>>> Forcing CONFIG_IPIPE_LEGACY there might help ruling this out.
>>>
>>> That's without any Xenomai configured. But I will check.
>>
>> This is where it traps in libpthread:
>>
>>    1112b:       c7 f8 00 00 00 00       xbeginq 11131 <__lll_lock_elision+0x51>
>>
>> For some reason, glibc starts to believe it could use RTM on that
>> machine - although this is clearly not available (Haswell i7, thus a
>> buggy CPU). Need to understand which feature check goes wrong, and why
>> only on I-pipe enabled kernels.
>>
>> The good news is that I still have a similar and unresolved report from
>> early customer tests. That could solve their issues as well (they
>> caused massive domain migrations).
> 
> Wait - we were already one step further: a potential trigger for this
> behaviour is a nullified pthread mutex. The decision to take the TSX
> path is based on field in that struct, and 0 means transactional. Guess
> I need a better reproduction case than some systemd service.

Which glibc release are you running?

-- 
Philippe.

Re: [Xenomai] [PULL] ipipe: fixes for 3.14

[Jan Kiszka]

On 2015-02-16 20:35, Philippe Gerum wrote:
> On 02/16/2015 08:11 PM, Jan Kiszka wrote:
>> On 2015-02-16 19:52, Jan Kiszka wrote:
>>> On 2015-02-16 18:52, Jan Kiszka wrote:
>>>> On 2015-02-16 18:48, Philippe Gerum wrote:
>>>>> On 02/16/2015 06:35 PM, Jan Kiszka wrote:
>>>>>> The following changes since commit 00d8a6f2e95453f61ae97b6587dba03bc91c7e1a:
>>>>>>
>>>>>>   arm/ipipe: Resolve trival merge conflicts (2015-01-14 17:00:22 +0100)
>>>>>>
>>>>>> are available in the git repository at:
>>>>>>
>>>>>>   git://git.xenomai.org/ipipe-jki 
>>>>>>
>>>>>> for you to fetch changes up to f982e48ce99e1bc9e71eac21830adac2db265c98:
>>>>>>
>>>>>>   x86/ipipe: Restore invocation of ipipe_unlock_irq from startup_ioapic_irq (2015-02-16 18:22:21 +0100)
>>>>>>
>>>>>> All patches apply to 3.16 as well but weren't tested there yet. Still
>>>>>> analyzing strange
>>>>>>
>>>>>> traps: (systemd)[1420] trap invalid opcode ip:7f4d138f612b sp:7fff4fd8f568 error:0 in libpthread-2.19.so[7f4d138e5000+18000]
>>>>>>
>>>>>> on the target under 3.14...
>>>>>>
>>>>>
>>>>> Register trashing on the fast call path (system_call_after_gs) maybe.
>>>>> Forcing CONFIG_IPIPE_LEGACY there might help ruling this out.
>>>>
>>>> That's without any Xenomai configured. But I will check.
>>>
>>> This is where it traps in libpthread:
>>>
>>>    1112b:       c7 f8 00 00 00 00       xbeginq 11131 <__lll_lock_elision+0x51>
>>>
>>> For some reason, glibc starts to believe it could use RTM on that
>>> machine - although this is clearly not available (Haswell i7, thus a
>>> buggy CPU). Need to understand which feature check goes wrong, and why
>>> only on I-pipe enabled kernels.
>>>
>>> The good news is that I still have a similar and unresolved report from
>>> early customer tests. That could solve their issues as well (they
>>> caused massive domain migrations).
>>
>> Wait - we were already one step further: a potential trigger for this
>> behaviour is a nullified pthread mutex. The decision to take the TSX
>> path is based on field in that struct, and 0 means transactional. Guess
>> I need a better reproduction case than some systemd service.
> 
> Which glibc release are you running?

2.19.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] [PULL] ipipe: fixes for 3.14

[Jan Kiszka]

On 2015-02-16 21:15, Jan Kiszka wrote:
> On 2015-02-16 20:35, Philippe Gerum wrote:
>> On 02/16/2015 08:11 PM, Jan Kiszka wrote:
>>> On 2015-02-16 19:52, Jan Kiszka wrote:
>>>> On 2015-02-16 18:52, Jan Kiszka wrote:
>>>>> On 2015-02-16 18:48, Philippe Gerum wrote:
>>>>>> On 02/16/2015 06:35 PM, Jan Kiszka wrote:
>>>>>>> The following changes since commit 00d8a6f2e95453f61ae97b6587dba03bc91c7e1a:
>>>>>>>
>>>>>>>   arm/ipipe: Resolve trival merge conflicts (2015-01-14 17:00:22 +0100)
>>>>>>>
>>>>>>> are available in the git repository at:
>>>>>>>
>>>>>>>   git://git.xenomai.org/ipipe-jki 
>>>>>>>
>>>>>>> for you to fetch changes up to f982e48ce99e1bc9e71eac21830adac2db265c98:
>>>>>>>
>>>>>>>   x86/ipipe: Restore invocation of ipipe_unlock_irq from startup_ioapic_irq (2015-02-16 18:22:21 +0100)
>>>>>>>
>>>>>>> All patches apply to 3.16 as well but weren't tested there yet. Still
>>>>>>> analyzing strange
>>>>>>>
>>>>>>> traps: (systemd)[1420] trap invalid opcode ip:7f4d138f612b sp:7fff4fd8f568 error:0 in libpthread-2.19.so[7f4d138e5000+18000]
>>>>>>>
>>>>>>> on the target under 3.14...
>>>>>>>
>>>>>>
>>>>>> Register trashing on the fast call path (system_call_after_gs) maybe.
>>>>>> Forcing CONFIG_IPIPE_LEGACY there might help ruling this out.
>>>>>
>>>>> That's without any Xenomai configured. But I will check.
>>>>
>>>> This is where it traps in libpthread:
>>>>
>>>>    1112b:       c7 f8 00 00 00 00       xbeginq 11131 <__lll_lock_elision+0x51>
>>>>
>>>> For some reason, glibc starts to believe it could use RTM on that
>>>> machine - although this is clearly not available (Haswell i7, thus a
>>>> buggy CPU). Need to understand which feature check goes wrong, and why
>>>> only on I-pipe enabled kernels.
>>>>
>>>> The good news is that I still have a similar and unresolved report from
>>>> early customer tests. That could solve their issues as well (they
>>>> caused massive domain migrations).
>>>
>>> Wait - we were already one step further: a potential trigger for this
>>> behaviour is a nullified pthread mutex. The decision to take the TSX
>>> path is based on field in that struct, and 0 means transactional. Guess
>>> I need a better reproduction case than some systemd service.
>>
>> Which glibc release are you running?
> 
> 2.19.

Retested and, although I could have sworn that !CONFIG_IPIPE doesn't
cause this effect, it does on 3.14.28. And checking CPU features again,
there is both "hle" and "rtm" - in contrast to a newer 3.18 kernel where
those are gone (same distro, same hardware).

So this is not an I-pipe issue, it's something related to the kernel or
its configuration. Need to study again where the microcode comes from. I
thought it was pulled from /lib/firmware/somewhere, thus would be shared
between all kernels on the same rootfs.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] [PULL] ipipe: fixes for 3.14

[Jan Kiszka]

On 2015-02-17 15:49, Jan Kiszka wrote:
> Retested and, although I could have sworn that !CONFIG_IPIPE doesn't
> cause this effect, it does on 3.14.28. And checking CPU features again,
> there is both "hle" and "rtm" - in contrast to a newer 3.18 kernel where
> those are gone (same distro, same hardware).
> 
> So this is not an I-pipe issue, it's something related to the kernel or
> its configuration. Need to study again where the microcode comes from. I
> thought it was pulled from /lib/firmware/somewhere, thus would be shared
> between all kernels on the same rootfs.

Just to close the topic: We had early microcode loading disabled in the
kernel, likely due to some historic reasons (there was a crash once).
That caused systemd to load libpthread before the microcode fix that
disables RTM was loaded. Once the microcode came in, the RTM
instructions suddenly start to cause #UD. Nice. Not sure yet if this
explains the issues in the customer application - we will see later.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux