[I-PIPE] ipipe-core-4.19.165-cip41-arm64-09 released

[I-PIPE] ipipe-core-4.19.165-cip41-arm64-09 released

[xenomai]

Download URL: https://xenomai.org/downloads/ipipe/v4.x/arm64/ipipe-core-4.19.165-cip41-arm64-09.patch

Repository: https://git.xenomai.org/ipipe-arm64
Release tag: ipipe-core-4.19.165-cip41-arm64-09

Re: [I-PIPE] ipipe-core-4.19.165-cip41-arm64-09 released

[Jan Kiszka]

On 10.02.21 07:22, xenomai--- via Xenomai wrote:
> Download URL: https://xenomai.org/downloads/ipipe/v4.x/arm64/ipipe-core-4.19.165-cip41-arm64-09.patch
> 
> Repository: https://git.xenomai.org/ipipe-arm64
> Release tag: ipipe-core-4.19.165-cip41-arm64-09
> 

Hmm, now we have the 5.4-arm64 issue also on 4.19:
https://gitlab.denx.de/Xenomai/xenomai-images/-/jobs/219984

Guess we need to understand this next now, specifically as it "spreads".

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: [I-PIPE] ipipe-core-4.19.165-cip41-arm64-09 released

[florian.bezdeka]

On Wed, 2021-02-10 at 09:15 +0100, Jan Kiszka via Xenomai wrote:
> On 10.02.21 07:22, xenomai--- via Xenomai wrote:
> > Download URL: https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fxenomai.org%2Fdownloads%2Fipipe%2Fv4.x%2Farm64%2Fipipe-core-4.19.165-cip41-arm64-09.patch&data=04%7C01%7Cflorian.bezdeka%40siemens.com%7C1e006deb5de148cc9c3208d8cd9c19df%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637485417625973221%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Tdne3lbEtMRgAB0frpqOvaCOUVY7d4vrMd8nZY9je%2FU%3D&reserved=0
> > 
> > Repository: https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.xenomai.org%2Fipipe-arm64&data=04%7C01%7Cflorian.bezdeka%40siemens.com%7C1e006deb5de148cc9c3208d8cd9c19df%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637485417625973221%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=egVS00o0j02j8aqfeeAjajAJVPnFHyMckXrZ5NzNggE%3D&reserved=0
> > Release tag: ipipe-core-4.19.165-cip41-arm64-09
> > 
> 
> Hmm, now we have the 5.4-arm64 issue also on 4.19:
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.denx.de%2FXenomai%2Fxenomai-images%2F-%2Fjobs%2F219984&data=04%7C01%7Cflorian.bezdeka%40siemens.com%7C1e006deb5de148cc9c3208d8cd9c19df%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637485417625983209%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Dg8thZ6hjA2e8KRyUyuA2wxCZI0A6l8uyra2VqlB5J0%3D&reserved=0
> 

I don't know much about the things going on here, but found this line
in the log. Maybe a starting point...

2021-02-10T07:51:47 setsched.c:120, assertion failed: stats.msw == msw


> Guess we need to understand this next now, specifically as it "spreads".
> 
> Jan
> 

Re: [I-PIPE] ipipe-core-4.19.165-cip41-arm64-09 released

[Jan Kiszka]

On 10.02.21 11:07, Bezdeka, Florian (T RDA IOT SES-DE) wrote:
> On Wed, 2021-02-10 at 09:15 +0100, Jan Kiszka via Xenomai wrote:
>> On 10.02.21 07:22, xenomai--- via Xenomai wrote:
>>> Download URL: https://xenomai.org/downloads/ipipe/v4.x/arm64/ipipe-core-4.19.165-cip41-arm64-09.patch
>>>
>>> Repository: https://git.xenomai.org/ipipe-arm64
>>> Release tag: ipipe-core-4.19.165-cip41-arm64-09
>>>
>>
>> Hmm, now we have the 5.4-arm64 issue also on 4.19:
>> https://gitlab.denx.de/Xenomai/xenomai-images/-/jobs/219984
>>
> 
> I don't know much about the things going on here, but found this line
> in the log. Maybe a starting point...
> 
> 2021-02-10T07:51:47 setsched.c:120, assertion failed: stats.msw == msw
> 

Exactly, that is causing the overall failure. And it was first seen with
the newly added 5.4 kernel.

Jan

> 
>> Guess we need to understand this next now, specifically as it "spreads".
>>
>> Jan
>>
> 

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: [I-PIPE] ipipe-core-4.19.165-cip41-arm64-09 released

[Greg Gallagher]

On Wed, Feb 10, 2021 at 6:08 AM Jan Kiszka <jan.kiszka@siemens.com> wrote:

> On 10.02.21 11:07, Bezdeka, Florian (T RDA IOT SES-DE) wrote:
> > On Wed, 2021-02-10 at 09:15 +0100, Jan Kiszka via Xenomai wrote:
> >> On 10.02.21 07:22, xenomai--- via Xenomai wrote:
> >>> Download URL:
> https://xenomai.org/downloads/ipipe/v4.x/arm64/ipipe-core-4.19.165-cip41-arm64-09.patch
> >>>
> >>> Repository: https://git.xenomai.org/ipipe-arm64
> >>> Release tag: ipipe-core-4.19.165-cip41-arm64-09
> >>>
> >>
> >> Hmm, now we have the 5.4-arm64 issue also on 4.19:
> >> https://gitlab.denx.de/Xenomai/xenomai-images/-/jobs/219984
> >>
> >
> > I don't know much about the things going on here, but found this line
> > in the log. Maybe a starting point...
> >
> > 2021-02-10T07:51:47 setsched.c:120, assertion failed: stats.msw == msw
> >
>
> Exactly, that is causing the overall failure. And it was first seen with
> the newly added 5.4 kernel.
>
> Jan
>
> >
> >> Guess we need to understand this next now, specifically as it "spreads".
> >>
> >> Jan
> >>
> >
>
> --
> Siemens AG, T RDA IOT
> Corporate Competence Center Embedded Linux


I’ll look into this today, seems to be the same issue I’m looking at for 5.4

Greg

>
>

Re: [I-PIPE] ipipe-core-4.19.165-cip41-arm64-09 released

[Greg Gallagher]

On Wed, Feb 10, 2021 at 7:41 AM Greg Gallagher <greg@embeddedgreg.com>
wrote:

>
>
> On Wed, Feb 10, 2021 at 6:08 AM Jan Kiszka <jan.kiszka@siemens.com> wrote:
>
>> On 10.02.21 11:07, Bezdeka, Florian (T RDA IOT SES-DE) wrote:
>> > On Wed, 2021-02-10 at 09:15 +0100, Jan Kiszka via Xenomai wrote:
>> >> On 10.02.21 07:22, xenomai--- via Xenomai wrote:
>> >>> Download URL:
>> https://xenomai.org/downloads/ipipe/v4.x/arm64/ipipe-core-4.19.165-cip41-arm64-09.patch
>> >>>
>> >>> Repository: https://git.xenomai.org/ipipe-arm64
>> >>> Release tag: ipipe-core-4.19.165-cip41-arm64-09
>> >>>
>> >>
>> >> Hmm, now we have the 5.4-arm64 issue also on 4.19:
>> >> https://gitlab.denx.de/Xenomai/xenomai-images/-/jobs/219984
>> >>
>> >
>> > I don't know much about the things going on here, but found this line
>> > in the log. Maybe a starting point...
>> >
>> > 2021-02-10T07:51:47 setsched.c:120, assertion failed: stats.msw == msw
>> >
>>
>> Exactly, that is causing the overall failure. And it was first seen with
>> the newly added 5.4 kernel.
>>
>> Jan
>>
>> >
>> >> Guess we need to understand this next now, specifically as it
>> "spreads".
>> >>
>> >> Jan
>> >>
>> >
>>
>> --
>> Siemens AG, T RDA IOT
>> Corporate Competence Center Embedded Linux
>
>
> I’ll look into this today, seems to be the same issue I’m looking at for
> 5.4
>
> Greg
>
>>
>>
Based on the log I assume this is coming from the setsched test.  This test
passes on my ultra96 board and zynq 7000, I'll build beagleboard black and
qemu to continue to investigate.

-Greg

Re: [I-PIPE] ipipe-core-4.19.165-cip41-arm64-09 released

[Jan Kiszka]

On 10.02.21 15:58, Greg Gallagher wrote:
> 
> 
> On Wed, Feb 10, 2021 at 7:41 AM Greg Gallagher <greg@embeddedgreg.com
> <mailto:greg@embeddedgreg.com>> wrote:
> 
> 
> 
>     On Wed, Feb 10, 2021 at 6:08 AM Jan Kiszka <jan.kiszka@siemens.com
>     <mailto:jan.kiszka@siemens.com>> wrote:
> 
>         On 10.02.21 11:07, Bezdeka, Florian (T RDA IOT SES-DE) wrote:
>         > On Wed, 2021-02-10 at 09:15 +0100, Jan Kiszka via Xenomai wrote:
>         >> On 10.02.21 07:22, xenomai--- via Xenomai wrote:
>         >>> Download URL:
>         https://xenomai.org/downloads/ipipe/v4.x/arm64/ipipe-core-4.19.165-cip41-arm64-09.patch
>         <https://xenomai.org/downloads/ipipe/v4.x/arm64/ipipe-core-4.19.165-cip41-arm64-09.patch>
>         >>>
>         >>> Repository: https://git.xenomai.org/ipipe-arm64
>         <https://git.xenomai.org/ipipe-arm64>
>         >>> Release tag: ipipe-core-4.19.165-cip41-arm64-09
>         >>>
>         >>
>         >> Hmm, now we have the 5.4-arm64 issue also on 4.19:
>         >> https://gitlab.denx.de/Xenomai/xenomai-images/-/jobs/219984
>         <https://gitlab.denx.de/Xenomai/xenomai-images/-/jobs/219984>
>         >>
>         >
>         > I don't know much about the things going on here, but found
>         this line
>         > in the log. Maybe a starting point...
>         >
>         > 2021-02-10T07:51:47 setsched.c:120, assertion failed:
>         stats.msw == msw
>         >
> 
>         Exactly, that is causing the overall failure. And it was first
>         seen with
>         the newly added 5.4 kernel.
> 
>         Jan
> 
>         >
>         >> Guess we need to understand this next now, specifically as it
>         "spreads".
>         >>
>         >> Jan
>         >>
>         >
> 
>         -- 
>         Siemens AG, T RDA IOT
>         Corporate Competence Center Embedded Linux
> 
> 
>     I’ll look into this today, seems to be the same issue I’m looking at
>     for 5.4
> 
>     Greg
> 
> 
> 
> Based on the log I assume this is coming from the setsched test.  This
> test passes on my ultra96 board and zynq 7000, I'll build beagleboard
> black and qemu to continue to investigate. 
> 

Please check if our kernel configs deviate in any suspicious way - or
try them instead [1].

Jan

[1]
https://gitlab.denx.de/Xenomai/xenomai-images/-/tree/master/recipes-kernel/linux/files

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: [I-PIPE] ipipe-core-4.19.165-cip41-arm64-09 released

[Greg Gallagher]

On Wed, Feb 10, 2021 at 10:00 AM Jan Kiszka <jan.kiszka@siemens.com> wrote:

> On 10.02.21 15:58, Greg Gallagher wrote:
> >
> >
> > On Wed, Feb 10, 2021 at 7:41 AM Greg Gallagher <greg@embeddedgreg.com
> > <mailto:greg@embeddedgreg.com>> wrote:
> >
> >
> >
> >     On Wed, Feb 10, 2021 at 6:08 AM Jan Kiszka <jan.kiszka@siemens.com
> >     <mailto:jan.kiszka@siemens.com>> wrote:
> >
> >         On 10.02.21 11:07, Bezdeka, Florian (T RDA IOT SES-DE) wrote:
> >         > On Wed, 2021-02-10 at 09:15 +0100, Jan Kiszka via Xenomai
> wrote:
> >         >> On 10.02.21 07:22, xenomai--- via Xenomai wrote:
> >         >>> Download URL:
> >
> https://xenomai.org/downloads/ipipe/v4.x/arm64/ipipe-core-4.19.165-cip41-arm64-09.patch
> >         <
> https://xenomai.org/downloads/ipipe/v4.x/arm64/ipipe-core-4.19.165-cip41-arm64-09.patch
> >
> >         >>>
> >         >>> Repository: https://git.xenomai.org/ipipe-arm64
> >         <https://git.xenomai.org/ipipe-arm64>
> >         >>> Release tag: ipipe-core-4.19.165-cip41-arm64-09
> >         >>>
> >         >>
> >         >> Hmm, now we have the 5.4-arm64 issue also on 4.19:
> >         >> https://gitlab.denx.de/Xenomai/xenomai-images/-/jobs/219984
> >         <https://gitlab.denx.de/Xenomai/xenomai-images/-/jobs/219984>
> >         >>
> >         >
> >         > I don't know much about the things going on here, but found
> >         this line
> >         > in the log. Maybe a starting point...
> >         >
> >         > 2021-02-10T07:51:47 setsched.c:120, assertion failed:
> >         stats.msw == msw
> >         >
> >
> >         Exactly, that is causing the overall failure. And it was first
> >         seen with
> >         the newly added 5.4 kernel.
> >
> >         Jan
> >
> >         >
> >         >> Guess we need to understand this next now, specifically as it
> >         "spreads".
> >         >>
> >         >> Jan
> >         >>
> >         >
> >
> >         --
> >         Siemens AG, T RDA IOT
> >         Corporate Competence Center Embedded Linux
> >
> >
> >     I’ll look into this today, seems to be the same issue I’m looking at
> >     for 5.4
> >
> >     Greg
> >
> >
> >
> > Based on the log I assume this is coming from the setsched test.  This
> > test passes on my ultra96 board and zynq 7000, I'll build beagleboard
> > black and qemu to continue to investigate.
> >
>
> Please check if our kernel configs deviate in any suspicious way - or
> try them instead [1].
>
> Jan
>
> [1]
>
> https://gitlab.denx.de/Xenomai/xenomai-images/-/tree/master/recipes-kernel/linux/files
>
> --
> Siemens AG, T RDA IOT
> Corporate Competence Center Embedded Linux
>

I used the arm64_defconfig and I still don't see the error.  I tried with
gcc 10 and gcc 7.3.  I'll try reproducing on the beaglebone black. I pushed
up a fix for the ipipe-arm for a build error I found and hopefully I'll
have more luck when I test this tomorrow.

-Greg

Re: [I-PIPE] ipipe-core-4.19.165-cip41-arm64-09 released

[Henning Schild]

Am Wed, 10 Feb 2021 12:08:43 +0100
schrieb Jan Kiszka via Xenomai <xenomai@xenomai.org>:

> On 10.02.21 11:07, Bezdeka, Florian (T RDA IOT SES-DE) wrote:
> > On Wed, 2021-02-10 at 09:15 +0100, Jan Kiszka via Xenomai wrote:  
> >> On 10.02.21 07:22, xenomai--- via Xenomai wrote:  
> >>> Download URL:
> >>> https://xenomai.org/downloads/ipipe/v4.x/arm64/ipipe-core-4.19.165-cip41-arm64-09.patch
> >>>
> >>> Repository: https://git.xenomai.org/ipipe-arm64
> >>> Release tag: ipipe-core-4.19.165-cip41-arm64-09
> >>>  
> >>
> >> Hmm, now we have the 5.4-arm64 issue also on 4.19:
> >> https://gitlab.denx.de/Xenomai/xenomai-images/-/jobs/219984
> >>  
> > 
> > I don't know much about the things going on here, but found this
> > line in the log. Maybe a starting point...
> > 
> > 2021-02-10T07:51:47 setsched.c:120, assertion failed: stats.msw ==
> > msw 
> 
> Exactly, that is causing the overall failure. And it was first seen
> with the newly added 5.4 kernel.

Seing the same on amd64 when testing on qemu, real HW is fine.

Managed to bisect it down to 4.19.147-cip (good) 4.19.150-cip (bad)

Which also means that ipipe-core-4.19.152-cip37-x86-15 is affected.

https://gitlab.denx.de/Xenomai/xenomai-images/-/jobs/200646
did not find it, so maybe our config differs

Henning

> Jan
> 
> >   
> >> Guess we need to understand this next now, specifically as it
> >> "spreads".
> >>
> >> Jan
> >>  
> >   
> 

Re: [I-PIPE] ipipe-core-4.19.165-cip41-arm64-09 released

[Henning Schild]

Am Wed, 24 Feb 2021 11:24:55 +0100
schrieb Henning Schild via Xenomai <xenomai@xenomai.org>:

> Am Wed, 10 Feb 2021 12:08:43 +0100
> schrieb Jan Kiszka via Xenomai <xenomai@xenomai.org>:
> 
> > On 10.02.21 11:07, Bezdeka, Florian (T RDA IOT SES-DE) wrote:  
> > > On Wed, 2021-02-10 at 09:15 +0100, Jan Kiszka via Xenomai wrote:
> > >   
> > >> On 10.02.21 07:22, xenomai--- via Xenomai wrote:    
> > >>> Download URL:
> > >>> https://xenomai.org/downloads/ipipe/v4.x/arm64/ipipe-core-4.19.165-cip41-arm64-09.patch
> > >>>
> > >>> Repository: https://git.xenomai.org/ipipe-arm64
> > >>> Release tag: ipipe-core-4.19.165-cip41-arm64-09
> > >>>    
> > >>
> > >> Hmm, now we have the 5.4-arm64 issue also on 4.19:
> > >> https://gitlab.denx.de/Xenomai/xenomai-images/-/jobs/219984
> > >>    
> > > 
> > > I don't know much about the things going on here, but found this
> > > line in the log. Maybe a starting point...
> > > 
> > > 2021-02-10T07:51:47 setsched.c:120, assertion failed: stats.msw ==
> > > msw   
> > 
> > Exactly, that is causing the overall failure. And it was first seen
> > with the newly added 5.4 kernel.  
> 
> Seing the same on amd64 when testing on qemu, real HW is fine.
> 
> Managed to bisect it down to 4.19.147-cip (good) 4.19.150-cip (bad)
> 
> Which also means that ipipe-core-4.19.152-cip37-x86-15 is affected.
> 
> https://gitlab.denx.de/Xenomai/xenomai-images/-/jobs/200646
> did not find it, so maybe our config differs

Digging further i found 0f0b6099c45ff3e06d2487816cf1ff30d21835f6 likely
causing the problem.

ipipe-core-4.19.152-cip37-x86-15 <- bad
revert 2b294ac325c7ce3f36854b74d0d1d89dc1d1d8b8
revert 8579a0440381353e0a71dd6a4d4371be8457eac4 <- bad
revert 0f0b6099c45ff3e06d2487816cf1ff30d <- good

I think here Jan or Phillipe should take over.

Henning

> Henning
> 
> > Jan
> >   
> > >     
> > >> Guess we need to understand this next now, specifically as it
> > >> "spreads".
> > >>
> > >> Jan
> > >>    
> > >     
> >   
> 
> 

Re: [I-PIPE] ipipe-core-4.19.165-cip41-arm64-09 released

[Jan Kiszka]

On 24.02.21 12:35, Henning Schild via Xenomai wrote:
> Am Wed, 24 Feb 2021 11:24:55 +0100
> schrieb Henning Schild via Xenomai <xenomai@xenomai.org>:
> 
>> Am Wed, 10 Feb 2021 12:08:43 +0100
>> schrieb Jan Kiszka via Xenomai <xenomai@xenomai.org>:
>>
>>> On 10.02.21 11:07, Bezdeka, Florian (T RDA IOT SES-DE) wrote:  
>>>> On Wed, 2021-02-10 at 09:15 +0100, Jan Kiszka via Xenomai wrote:
>>>>   
>>>>> On 10.02.21 07:22, xenomai--- via Xenomai wrote:    
>>>>>> Download URL:
>>>>>> https://xenomai.org/downloads/ipipe/v4.x/arm64/ipipe-core-4.19.165-cip41-arm64-09.patch
>>>>>>
>>>>>> Repository: https://git.xenomai.org/ipipe-arm64
>>>>>> Release tag: ipipe-core-4.19.165-cip41-arm64-09
>>>>>>    
>>>>>
>>>>> Hmm, now we have the 5.4-arm64 issue also on 4.19:
>>>>> https://gitlab.denx.de/Xenomai/xenomai-images/-/jobs/219984
>>>>>    
>>>>
>>>> I don't know much about the things going on here, but found this
>>>> line in the log. Maybe a starting point...
>>>>
>>>> 2021-02-10T07:51:47 setsched.c:120, assertion failed: stats.msw ==
>>>> msw   
>>>
>>> Exactly, that is causing the overall failure. And it was first seen
>>> with the newly added 5.4 kernel.  
>>
>> Seing the same on amd64 when testing on qemu, real HW is fine.
>>
>> Managed to bisect it down to 4.19.147-cip (good) 4.19.150-cip (bad)
>>
>> Which also means that ipipe-core-4.19.152-cip37-x86-15 is affected.
>>
>> https://gitlab.denx.de/Xenomai/xenomai-images/-/jobs/200646
>> did not find it, so maybe our config differs

Already compared yours against the one in xenomai-images? That would be
useful.

> 
> Digging further i found 0f0b6099c45ff3e06d2487816cf1ff30d21835f6 likely
> causing the problem.
> 
> ipipe-core-4.19.152-cip37-x86-15 <- bad
> revert 2b294ac325c7ce3f36854b74d0d1d89dc1d1d8b8
> revert 8579a0440381353e0a71dd6a4d4371be8457eac4 <- bad
> revert 0f0b6099c45ff3e06d2487816cf1ff30d <- good
> 
> I think here Jan or Phillipe should take over.

Thanks for bisecting, this is helpful!

Philippe, any immediate idea why all that is failing now?

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: [I-PIPE] ipipe-core-4.19.165-cip41-arm64-09 released

[Philippe Gerum]

Jan Kiszka <jan.kiszka@siemens.com> writes:

> On 24.02.21 12:35, Henning Schild via Xenomai wrote:
>> Am Wed, 24 Feb 2021 11:24:55 +0100
>> schrieb Henning Schild via Xenomai <xenomai@xenomai.org>:
>> 
>>> Am Wed, 10 Feb 2021 12:08:43 +0100
>>> schrieb Jan Kiszka via Xenomai <xenomai@xenomai.org>:
>>>
>>>> On 10.02.21 11:07, Bezdeka, Florian (T RDA IOT SES-DE) wrote:  
>>>>> On Wed, 2021-02-10 at 09:15 +0100, Jan Kiszka via Xenomai wrote:
>>>>>   
>>>>>> On 10.02.21 07:22, xenomai--- via Xenomai wrote:    
>>>>>>> Download URL:
>>>>>>> https://xenomai.org/downloads/ipipe/v4.x/arm64/ipipe-core-4.19.165-cip41-arm64-09.patch
>>>>>>>
>>>>>>> Repository: https://git.xenomai.org/ipipe-arm64
>>>>>>> Release tag: ipipe-core-4.19.165-cip41-arm64-09
>>>>>>>    
>>>>>>
>>>>>> Hmm, now we have the 5.4-arm64 issue also on 4.19:
>>>>>> https://gitlab.denx.de/Xenomai/xenomai-images/-/jobs/219984
>>>>>>    
>>>>>
>>>>> I don't know much about the things going on here, but found this
>>>>> line in the log. Maybe a starting point...
>>>>>
>>>>> 2021-02-10T07:51:47 setsched.c:120, assertion failed: stats.msw ==
>>>>> msw   
>>>>
>>>> Exactly, that is causing the overall failure. And it was first seen
>>>> with the newly added 5.4 kernel.  
>>>
>>> Seing the same on amd64 when testing on qemu, real HW is fine.
>>>
>>> Managed to bisect it down to 4.19.147-cip (good) 4.19.150-cip (bad)
>>>
>>> Which also means that ipipe-core-4.19.152-cip37-x86-15 is affected.
>>>
>>> https://gitlab.denx.de/Xenomai/xenomai-images/-/jobs/200646
>>> did not find it, so maybe our config differs
>
> Already compared yours against the one in xenomai-images? That would be
> useful.
>
>> 
>> Digging further i found 0f0b6099c45ff3e06d2487816cf1ff30d21835f6 likely
>> causing the problem.
>> 
>> ipipe-core-4.19.152-cip37-x86-15 <- bad
>> revert 2b294ac325c7ce3f36854b74d0d1d89dc1d1d8b8
>> revert 8579a0440381353e0a71dd6a4d4371be8457eac4 <- bad
>> revert 0f0b6099c45ff3e06d2487816cf1ff30d <- good
>> 
>> I think here Jan or Phillipe should take over.
>
> Thanks for bisecting, this is helpful!
>
> Philippe, any immediate idea why all that is failing now?

Something may be going wrong with MAP_SHARED mappings wrt commit_vma()
in Dovetail. I'm adding this to my debug queue.

-- 
Philippe.

Re: [I-PIPE] ipipe-core-4.19.165-cip41-arm64-09 released

[Jan Kiszka]

On 25.02.21 14:54, Philippe Gerum wrote:
> 
> Jan Kiszka <jan.kiszka@siemens.com> writes:
> 
>> On 24.02.21 12:35, Henning Schild via Xenomai wrote:
>>> Am Wed, 24 Feb 2021 11:24:55 +0100
>>> schrieb Henning Schild via Xenomai <xenomai@xenomai.org>:
>>>
>>>> Am Wed, 10 Feb 2021 12:08:43 +0100
>>>> schrieb Jan Kiszka via Xenomai <xenomai@xenomai.org>:
>>>>
>>>>> On 10.02.21 11:07, Bezdeka, Florian (T RDA IOT SES-DE) wrote:  
>>>>>> On Wed, 2021-02-10 at 09:15 +0100, Jan Kiszka via Xenomai wrote:
>>>>>>   
>>>>>>> On 10.02.21 07:22, xenomai--- via Xenomai wrote:    
>>>>>>>> Download URL:
>>>>>>>> https://xenomai.org/downloads/ipipe/v4.x/arm64/ipipe-core-4.19.165-cip41-arm64-09.patch
>>>>>>>>
>>>>>>>> Repository: https://git.xenomai.org/ipipe-arm64
>>>>>>>> Release tag: ipipe-core-4.19.165-cip41-arm64-09
>>>>>>>>    
>>>>>>>
>>>>>>> Hmm, now we have the 5.4-arm64 issue also on 4.19:
>>>>>>> https://gitlab.denx.de/Xenomai/xenomai-images/-/jobs/219984
>>>>>>>    
>>>>>>
>>>>>> I don't know much about the things going on here, but found this
>>>>>> line in the log. Maybe a starting point...
>>>>>>
>>>>>> 2021-02-10T07:51:47 setsched.c:120, assertion failed: stats.msw ==
>>>>>> msw   
>>>>>
>>>>> Exactly, that is causing the overall failure. And it was first seen
>>>>> with the newly added 5.4 kernel.  
>>>>
>>>> Seing the same on amd64 when testing on qemu, real HW is fine.
>>>>
>>>> Managed to bisect it down to 4.19.147-cip (good) 4.19.150-cip (bad)
>>>>
>>>> Which also means that ipipe-core-4.19.152-cip37-x86-15 is affected.
>>>>
>>>> https://gitlab.denx.de/Xenomai/xenomai-images/-/jobs/200646
>>>> did not find it, so maybe our config differs
>>
>> Already compared yours against the one in xenomai-images? That would be
>> useful.
>>
>>>
>>> Digging further i found 0f0b6099c45ff3e06d2487816cf1ff30d21835f6 likely
>>> causing the problem.
>>>
>>> ipipe-core-4.19.152-cip37-x86-15 <- bad
>>> revert 2b294ac325c7ce3f36854b74d0d1d89dc1d1d8b8
>>> revert 8579a0440381353e0a71dd6a4d4371be8457eac4 <- bad
>>> revert 0f0b6099c45ff3e06d2487816cf1ff30d <- good
>>>
>>> I think here Jan or Phillipe should take over.
>>
>> Thanks for bisecting, this is helpful!
>>
>> Philippe, any immediate idea why all that is failing now?
> 
> Something may be going wrong with MAP_SHARED mappings wrt commit_vma()
> in Dovetail. I'm adding this to my debug queue.
> 

This is still I-pipe, not a dovetail-related issues.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: [I-PIPE] ipipe-core-4.19.165-cip41-arm64-09 released

[Philippe Gerum]

Jan Kiszka <jan.kiszka@siemens.com> writes:

> On 25.02.21 14:54, Philippe Gerum wrote:
>> 
>> Jan Kiszka <jan.kiszka@siemens.com> writes:
>> 
>>> On 24.02.21 12:35, Henning Schild via Xenomai wrote:
>>>> Am Wed, 24 Feb 2021 11:24:55 +0100
>>>> schrieb Henning Schild via Xenomai <xenomai@xenomai.org>:
>>>>
>>>>> Am Wed, 10 Feb 2021 12:08:43 +0100
>>>>> schrieb Jan Kiszka via Xenomai <xenomai@xenomai.org>:
>>>>>
>>>>>> On 10.02.21 11:07, Bezdeka, Florian (T RDA IOT SES-DE) wrote:  
>>>>>>> On Wed, 2021-02-10 at 09:15 +0100, Jan Kiszka via Xenomai wrote:
>>>>>>>   
>>>>>>>> On 10.02.21 07:22, xenomai--- via Xenomai wrote:    
>>>>>>>>> Download URL:
>>>>>>>>> https://xenomai.org/downloads/ipipe/v4.x/arm64/ipipe-core-4.19.165-cip41-arm64-09.patch
>>>>>>>>>
>>>>>>>>> Repository: https://git.xenomai.org/ipipe-arm64
>>>>>>>>> Release tag: ipipe-core-4.19.165-cip41-arm64-09
>>>>>>>>>    
>>>>>>>>
>>>>>>>> Hmm, now we have the 5.4-arm64 issue also on 4.19:
>>>>>>>> https://gitlab.denx.de/Xenomai/xenomai-images/-/jobs/219984
>>>>>>>>    
>>>>>>>
>>>>>>> I don't know much about the things going on here, but found this
>>>>>>> line in the log. Maybe a starting point...
>>>>>>>
>>>>>>> 2021-02-10T07:51:47 setsched.c:120, assertion failed: stats.msw ==
>>>>>>> msw   
>>>>>>
>>>>>> Exactly, that is causing the overall failure. And it was first seen
>>>>>> with the newly added 5.4 kernel.  
>>>>>
>>>>> Seing the same on amd64 when testing on qemu, real HW is fine.
>>>>>
>>>>> Managed to bisect it down to 4.19.147-cip (good) 4.19.150-cip (bad)
>>>>>
>>>>> Which also means that ipipe-core-4.19.152-cip37-x86-15 is affected.
>>>>>
>>>>> https://gitlab.denx.de/Xenomai/xenomai-images/-/jobs/200646
>>>>> did not find it, so maybe our config differs
>>>
>>> Already compared yours against the one in xenomai-images? That would be
>>> useful.
>>>
>>>>
>>>> Digging further i found 0f0b6099c45ff3e06d2487816cf1ff30d21835f6 likely
>>>> causing the problem.
>>>>
>>>> ipipe-core-4.19.152-cip37-x86-15 <- bad
>>>> revert 2b294ac325c7ce3f36854b74d0d1d89dc1d1d8b8
>>>> revert 8579a0440381353e0a71dd6a4d4371be8457eac4 <- bad
>>>> revert 0f0b6099c45ff3e06d2487816cf1ff30d <- good
>>>>
>>>> I think here Jan or Phillipe should take over.
>>>
>>> Thanks for bisecting, this is helpful!
>>>
>>> Philippe, any immediate idea why all that is failing now?
>> 
>> Something may be going wrong with MAP_SHARED mappings wrt commit_vma()
>> in Dovetail. I'm adding this to my debug queue.
>> 
>
> This is still I-pipe, not a dovetail-related issues.

This I-pipe release mimics what Dovetail does wrt mm pinning.

-- 
Philippe.

Re: [I-PIPE] ipipe-core-4.19.165-cip41-arm64-09 released

[Greg Gallagher]

On Thu, Feb 25, 2021 at 9:19 AM Philippe Gerum via Xenomai <
xenomai@xenomai.org> wrote:

>
> Jan Kiszka <jan.kiszka@siemens.com> writes:
>
> > On 25.02.21 14:54, Philippe Gerum wrote:
> >>
> >> Jan Kiszka <jan.kiszka@siemens.com> writes:
> >>
> >>> On 24.02.21 12:35, Henning Schild via Xenomai wrote:
> >>>> Am Wed, 24 Feb 2021 11:24:55 +0100
> >>>> schrieb Henning Schild via Xenomai <xenomai@xenomai.org>:
> >>>>
> >>>>> Am Wed, 10 Feb 2021 12:08:43 +0100
> >>>>> schrieb Jan Kiszka via Xenomai <xenomai@xenomai.org>:
> >>>>>
> >>>>>> On 10.02.21 11:07, Bezdeka, Florian (T RDA IOT SES-DE) wrote:
> >>>>>>> On Wed, 2021-02-10 at 09:15 +0100, Jan Kiszka via Xenomai wrote:
> >>>>>>>
> >>>>>>>> On 10.02.21 07:22, xenomai--- via Xenomai wrote:
> >>>>>>>>> Download URL:
> >>>>>>>>>
> https://xenomai.org/downloads/ipipe/v4.x/arm64/ipipe-core-4.19.165-cip41-arm64-09.patch
> >>>>>>>>>
> >>>>>>>>> Repository: https://git.xenomai.org/ipipe-arm64
> >>>>>>>>> Release tag: ipipe-core-4.19.165-cip41-arm64-09
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> Hmm, now we have the 5.4-arm64 issue also on 4.19:
> >>>>>>>> https://gitlab.denx.de/Xenomai/xenomai-images/-/jobs/219984
> >>>>>>>>
> >>>>>>>
> >>>>>>> I don't know much about the things going on here, but found this
> >>>>>>> line in the log. Maybe a starting point...
> >>>>>>>
> >>>>>>> 2021-02-10T07:51:47 setsched.c:120, assertion failed: stats.msw ==
> >>>>>>> msw
> >>>>>>
> >>>>>> Exactly, that is causing the overall failure. And it was first seen
> >>>>>> with the newly added 5.4 kernel.
> >>>>>
> >>>>> Seing the same on amd64 when testing on qemu, real HW is fine.
> >>>>>
> >>>>> Managed to bisect it down to 4.19.147-cip (good) 4.19.150-cip (bad)
> >>>>>
> >>>>> Which also means that ipipe-core-4.19.152-cip37-x86-15 is affected.
> >>>>>
> >>>>> https://gitlab.denx.de/Xenomai/xenomai-images/-/jobs/200646
> >>>>> did not find it, so maybe our config differs
> >>>
> >>> Already compared yours against the one in xenomai-images? That would be
> >>> useful.
> >>>
> >>>>
> >>>> Digging further i found 0f0b6099c45ff3e06d2487816cf1ff30d21835f6
> likely
> >>>> causing the problem.
> >>>>
> >>>> ipipe-core-4.19.152-cip37-x86-15 <- bad
> >>>> revert 2b294ac325c7ce3f36854b74d0d1d89dc1d1d8b8
> >>>> revert 8579a0440381353e0a71dd6a4d4371be8457eac4 <- bad
> >>>> revert 0f0b6099c45ff3e06d2487816cf1ff30d <- good
> >>>>
> >>>> I think here Jan or Phillipe should take over.
> >>>
> >>> Thanks for bisecting, this is helpful!
> >>>
> >>> Philippe, any immediate idea why all that is failing now?
> >>
> >> Something may be going wrong with MAP_SHARED mappings wrt commit_vma()
> >> in Dovetail. I'm adding this to my debug queue.
> >>
> >
> > This is still I-pipe, not a dovetail-related issues.
>
> This I-pipe release mimics what Dovetail does wrt mm pinning.
>
> --
> Philippe.
>
> I can start to look into this also if that would help and ping Philippe
when I need some help.  I'm already doing a couple ipipe fixes.  I'll use a
qemu ARM64 xenomai image to debug.

Thanks

Greg

Re: [I-PIPE] ipipe-core-4.19.165-cip41-arm64-09 released

[Jan Kiszka]

On 25.02.21 15:18, Philippe Gerum wrote:
> 
> Jan Kiszka <jan.kiszka@siemens.com> writes:
> 
>> On 25.02.21 14:54, Philippe Gerum wrote:
>>>
>>> Jan Kiszka <jan.kiszka@siemens.com> writes:
>>>
>>>> On 24.02.21 12:35, Henning Schild via Xenomai wrote:
>>>>> Am Wed, 24 Feb 2021 11:24:55 +0100
>>>>> schrieb Henning Schild via Xenomai <xenomai@xenomai.org>:
>>>>>
>>>>>> Am Wed, 10 Feb 2021 12:08:43 +0100
>>>>>> schrieb Jan Kiszka via Xenomai <xenomai@xenomai.org>:
>>>>>>
>>>>>>> On 10.02.21 11:07, Bezdeka, Florian (T RDA IOT SES-DE) wrote:  
>>>>>>>> On Wed, 2021-02-10 at 09:15 +0100, Jan Kiszka via Xenomai wrote:
>>>>>>>>   
>>>>>>>>> On 10.02.21 07:22, xenomai--- via Xenomai wrote:    
>>>>>>>>>> Download URL:
>>>>>>>>>> https://xenomai.org/downloads/ipipe/v4.x/arm64/ipipe-core-4.19.165-cip41-arm64-09.patch
>>>>>>>>>>
>>>>>>>>>> Repository: https://git.xenomai.org/ipipe-arm64
>>>>>>>>>> Release tag: ipipe-core-4.19.165-cip41-arm64-09
>>>>>>>>>>    
>>>>>>>>>
>>>>>>>>> Hmm, now we have the 5.4-arm64 issue also on 4.19:
>>>>>>>>> https://gitlab.denx.de/Xenomai/xenomai-images/-/jobs/219984
>>>>>>>>>    
>>>>>>>>
>>>>>>>> I don't know much about the things going on here, but found this
>>>>>>>> line in the log. Maybe a starting point...
>>>>>>>>
>>>>>>>> 2021-02-10T07:51:47 setsched.c:120, assertion failed: stats.msw ==
>>>>>>>> msw   
>>>>>>>
>>>>>>> Exactly, that is causing the overall failure. And it was first seen
>>>>>>> with the newly added 5.4 kernel.  
>>>>>>
>>>>>> Seing the same on amd64 when testing on qemu, real HW is fine.
>>>>>>
>>>>>> Managed to bisect it down to 4.19.147-cip (good) 4.19.150-cip (bad)
>>>>>>
>>>>>> Which also means that ipipe-core-4.19.152-cip37-x86-15 is affected.
>>>>>>
>>>>>> https://gitlab.denx.de/Xenomai/xenomai-images/-/jobs/200646
>>>>>> did not find it, so maybe our config differs
>>>>
>>>> Already compared yours against the one in xenomai-images? That would be
>>>> useful.
>>>>
>>>>>
>>>>> Digging further i found 0f0b6099c45ff3e06d2487816cf1ff30d21835f6 likely
>>>>> causing the problem.
>>>>>
>>>>> ipipe-core-4.19.152-cip37-x86-15 <- bad
>>>>> revert 2b294ac325c7ce3f36854b74d0d1d89dc1d1d8b8
>>>>> revert 8579a0440381353e0a71dd6a4d4371be8457eac4 <- bad
>>>>> revert 0f0b6099c45ff3e06d2487816cf1ff30d <- good
>>>>>
>>>>> I think here Jan or Phillipe should take over.
>>>>
>>>> Thanks for bisecting, this is helpful!
>>>>
>>>> Philippe, any immediate idea why all that is failing now?
>>>
>>> Something may be going wrong with MAP_SHARED mappings wrt commit_vma()
>>> in Dovetail. I'm adding this to my debug queue.
>>>
>>
>> This is still I-pipe, not a dovetail-related issues.
> 
> This I-pipe release mimics what Dovetail does wrt mm pinning.
> 

Any news on this from your side?

Florian took a trace from the system where this was observed on x86. It 
seems to confirm that we have an unexpected minor fault here:

smokey-568   [000]    74.233945: cobalt_head_sysentry: syscall=thread_getschedparam_ex
smokey-568   [000]    74.233950: cobalt_pthread_getschedparam: pth=0x7f2a18ebf700 policy=fifo param={ priority=3 }
smokey-568   [000]    74.233950: cobalt_head_sysexit:  result=0
smokey-568   [000]    74.233952: cobalt_head_sysentry: syscall=thread_getstat
smokey-568   [000]    74.233952: cobalt_pthread_stat:  pid=568
smokey-568   [000]    74.233953: cobalt_head_sysexit:  result=0
smokey-568   [000]    74.233962: cobalt_thread_fault:  ip=0x7f2a19cd6b46 type=e
smokey-568   [000]    74.233962: cobalt_shadow_gorelax: reason=fault
smokey-568   [000]    74.233963: cobalt_lostage_request: request=ffffffffbcd85992 pid=568 comm=smokey
...
smokey-568   [000]    74.234005: cobalt_shadow_relaxed: state=0x480c0 info=0x0
...
smokey-568   [000]    74.235027: cobalt_head_sysentry: syscall=ftrace_puts
smokey-568   [000]    74.235028: cobalt_root_sysentry: syscall=ftrace_puts
smokey-568   [000]    74.235028: print:                CoBaLt_ftrace_puts: Second assertion failed
(that's around line 120 in smokey/setsched/setsched.c)

Now, before I dig into the code you pointed to, I just wanted to sync.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: [I-PIPE] ipipe-core-4.19.165-cip41-arm64-09 released

[Greg Gallagher]

On Mon, Mar 1, 2021 at 11:20 AM Jan Kiszka via Xenomai <xenomai@xenomai.org>
wrote:

> On 25.02.21 15:18, Philippe Gerum wrote:
> >
> > Jan Kiszka <jan.kiszka@siemens.com> writes:
> >
> >> On 25.02.21 14:54, Philippe Gerum wrote:
> >>>
> >>> Jan Kiszka <jan.kiszka@siemens.com> writes:
> >>>
> >>>> On 24.02.21 12:35, Henning Schild via Xenomai wrote:
> >>>>> Am Wed, 24 Feb 2021 11:24:55 +0100
> >>>>> schrieb Henning Schild via Xenomai <xenomai@xenomai.org>:
> >>>>>
> >>>>>> Am Wed, 10 Feb 2021 12:08:43 +0100
> >>>>>> schrieb Jan Kiszka via Xenomai <xenomai@xenomai.org>:
> >>>>>>
> >>>>>>> On 10.02.21 11:07, Bezdeka, Florian (T RDA IOT SES-DE) wrote:
> >>>>>>>> On Wed, 2021-02-10 at 09:15 +0100, Jan Kiszka via Xenomai wrote:
> >>>>>>>>
> >>>>>>>>> On 10.02.21 07:22, xenomai--- via Xenomai wrote:
> >>>>>>>>>> Download URL:
> >>>>>>>>>>
> https://xenomai.org/downloads/ipipe/v4.x/arm64/ipipe-core-4.19.165-cip41-arm64-09.patch
> >>>>>>>>>>
> >>>>>>>>>> Repository: https://git.xenomai.org/ipipe-arm64
> >>>>>>>>>> Release tag: ipipe-core-4.19.165-cip41-arm64-09
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Hmm, now we have the 5.4-arm64 issue also on 4.19:
> >>>>>>>>> https://gitlab.denx.de/Xenomai/xenomai-images/-/jobs/219984
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> I don't know much about the things going on here, but found this
> >>>>>>>> line in the log. Maybe a starting point...
> >>>>>>>>
> >>>>>>>> 2021-02-10T07:51:47 setsched.c:120, assertion failed: stats.msw ==
> >>>>>>>> msw
> >>>>>>>
> >>>>>>> Exactly, that is causing the overall failure. And it was first seen
> >>>>>>> with the newly added 5.4 kernel.
> >>>>>>
> >>>>>> Seing the same on amd64 when testing on qemu, real HW is fine.
> >>>>>>
> >>>>>> Managed to bisect it down to 4.19.147-cip (good) 4.19.150-cip (bad)
> >>>>>>
> >>>>>> Which also means that ipipe-core-4.19.152-cip37-x86-15 is affected.
> >>>>>>
> >>>>>> https://gitlab.denx.de/Xenomai/xenomai-images/-/jobs/200646
> >>>>>> did not find it, so maybe our config differs
> >>>>
> >>>> Already compared yours against the one in xenomai-images? That would
> be
> >>>> useful.
> >>>>
> >>>>>
> >>>>> Digging further i found 0f0b6099c45ff3e06d2487816cf1ff30d21835f6
> likely
> >>>>> causing the problem.
> >>>>>
> >>>>> ipipe-core-4.19.152-cip37-x86-15 <- bad
> >>>>> revert 2b294ac325c7ce3f36854b74d0d1d89dc1d1d8b8
> >>>>> revert 8579a0440381353e0a71dd6a4d4371be8457eac4 <- bad
> >>>>> revert 0f0b6099c45ff3e06d2487816cf1ff30d <- good
> >>>>>
> >>>>> I think here Jan or Phillipe should take over.
> >>>>
> >>>> Thanks for bisecting, this is helpful!
> >>>>
> >>>> Philippe, any immediate idea why all that is failing now?
> >>>
> >>> Something may be going wrong with MAP_SHARED mappings wrt commit_vma()
> >>> in Dovetail. I'm adding this to my debug queue.
> >>>
> >>
> >> This is still I-pipe, not a dovetail-related issues.
> >
> > This I-pipe release mimics what Dovetail does wrt mm pinning.
> >
>
> Any news on this from your side?
>
> Florian took a trace from the system where this was observed on x86. It
> seems to confirm that we have an unexpected minor fault here:
>
> smokey-568   [000]    74.233945: cobalt_head_sysentry:
> syscall=thread_getschedparam_ex
> smokey-568   [000]    74.233950: cobalt_pthread_getschedparam:
> pth=0x7f2a18ebf700 policy=fifo param={ priority=3 }
> smokey-568   [000]    74.233950: cobalt_head_sysexit:  result=0
> smokey-568   [000]    74.233952: cobalt_head_sysentry:
> syscall=thread_getstat
> smokey-568   [000]    74.233952: cobalt_pthread_stat:  pid=568
> smokey-568   [000]    74.233953: cobalt_head_sysexit:  result=0
> smokey-568   [000]    74.233962: cobalt_thread_fault:  ip=0x7f2a19cd6b46
> type=e
> smokey-568   [000]    74.233962: cobalt_shadow_gorelax: reason=fault
> smokey-568   [000]    74.233963: cobalt_lostage_request:
> request=ffffffffbcd85992 pid=568 comm=smokey
> ...
> smokey-568   [000]    74.234005: cobalt_shadow_relaxed: state=0x480c0
> info=0x0
> ...
> smokey-568   [000]    74.235027: cobalt_head_sysentry: syscall=ftrace_puts
> smokey-568   [000]    74.235028: cobalt_root_sysentry: syscall=ftrace_puts
> smokey-568   [000]    74.235028: print:                CoBaLt_ftrace_puts:
> Second assertion failed
> (that's around line 120 in smokey/setsched/setsched.c)
>
> Now, before I dig into the code you pointed to, I just wanted to sync.
>
> Jan
>
> --
> Siemens AG, T RDA IOT
> Corporate Competence Center Embedded Linux
>
>
On my end I spent the weekend trying to reproduce the issue on ARM64.  I've
been trying to reproduce on qemu using the default defconfig and the
defconfig from the CI machines.  To save time I run the setsched test in
smokey by passing in the test id.  I still haven't had any luck reproducing
it.  I'm currently running it in a loop hoping it will cause the failure.
When you guys reproduce the issue, do all the tests get run?

-Greg

Re: [I-PIPE] ipipe-core-4.19.165-cip41-arm64-09 released

[Jan Kiszka]

On 01.03.21 17:34, Greg Gallagher wrote:
> 
> 
> On Mon, Mar 1, 2021 at 11:20 AM Jan Kiszka via Xenomai
> <xenomai@xenomai.org <mailto:xenomai@xenomai.org>> wrote:
> 
>     On 25.02.21 15:18, Philippe Gerum wrote:
>     >
>     > Jan Kiszka <jan.kiszka@siemens.com
>     <mailto:jan.kiszka@siemens.com>> writes:
>     >
>     >> On 25.02.21 14:54, Philippe Gerum wrote:
>     >>>
>     >>> Jan Kiszka <jan.kiszka@siemens.com
>     <mailto:jan.kiszka@siemens.com>> writes:
>     >>>
>     >>>> On 24.02.21 12:35, Henning Schild via Xenomai wrote:
>     >>>>> Am Wed, 24 Feb 2021 11:24:55 +0100
>     >>>>> schrieb Henning Schild via Xenomai <xenomai@xenomai.org
>     <mailto:xenomai@xenomai.org>>:
>     >>>>>
>     >>>>>> Am Wed, 10 Feb 2021 12:08:43 +0100
>     >>>>>> schrieb Jan Kiszka via Xenomai <xenomai@xenomai.org
>     <mailto:xenomai@xenomai.org>>:
>     >>>>>>
>     >>>>>>> On 10.02.21 11:07, Bezdeka, Florian (T RDA IOT SES-DE) wrote: 
>     >>>>>>>> On Wed, 2021-02-10 at 09:15 +0100, Jan Kiszka via Xenomai
>     wrote:
>     >>>>>>>>   
>     >>>>>>>>> On 10.02.21 07:22, xenomai--- via Xenomai wrote:   
>     >>>>>>>>>> Download URL:
>     >>>>>>>>>>
>     https://xenomai.org/downloads/ipipe/v4.x/arm64/ipipe-core-4.19.165-cip41-arm64-09.patch
>     <https://xenomai.org/downloads/ipipe/v4.x/arm64/ipipe-core-4.19.165-cip41-arm64-09.patch>
>     >>>>>>>>>>
>     >>>>>>>>>> Repository: https://git.xenomai.org/ipipe-arm64
>     <https://git.xenomai.org/ipipe-arm64>
>     >>>>>>>>>> Release tag: ipipe-core-4.19.165-cip41-arm64-09
>     >>>>>>>>>>   
>     >>>>>>>>>
>     >>>>>>>>> Hmm, now we have the 5.4-arm64 issue also on 4.19:
>     >>>>>>>>>
>     https://gitlab.denx.de/Xenomai/xenomai-images/-/jobs/219984
>     <https://gitlab.denx.de/Xenomai/xenomai-images/-/jobs/219984>
>     >>>>>>>>>   
>     >>>>>>>>
>     >>>>>>>> I don't know much about the things going on here, but found
>     this
>     >>>>>>>> line in the log. Maybe a starting point...
>     >>>>>>>>
>     >>>>>>>> 2021-02-10T07:51:47 setsched.c:120, assertion failed:
>     stats.msw ==
>     >>>>>>>> msw   
>     >>>>>>>
>     >>>>>>> Exactly, that is causing the overall failure. And it was
>     first seen
>     >>>>>>> with the newly added 5.4 kernel. 
>     >>>>>>
>     >>>>>> Seing the same on amd64 when testing on qemu, real HW is fine.
>     >>>>>>
>     >>>>>> Managed to bisect it down to 4.19.147-cip (good) 4.19.150-cip
>     (bad)
>     >>>>>>
>     >>>>>> Which also means that ipipe-core-4.19.152-cip37-x86-15 is
>     affected.
>     >>>>>>
>     >>>>>> https://gitlab.denx.de/Xenomai/xenomai-images/-/jobs/200646
>     <https://gitlab.denx.de/Xenomai/xenomai-images/-/jobs/200646>
>     >>>>>> did not find it, so maybe our config differs
>     >>>>
>     >>>> Already compared yours against the one in xenomai-images? That
>     would be
>     >>>> useful.
>     >>>>
>     >>>>>
>     >>>>> Digging further i found
>     0f0b6099c45ff3e06d2487816cf1ff30d21835f6 likely
>     >>>>> causing the problem.
>     >>>>>
>     >>>>> ipipe-core-4.19.152-cip37-x86-15 <- bad
>     >>>>> revert 2b294ac325c7ce3f36854b74d0d1d89dc1d1d8b8
>     >>>>> revert 8579a0440381353e0a71dd6a4d4371be8457eac4 <- bad
>     >>>>> revert 0f0b6099c45ff3e06d2487816cf1ff30d <- good
>     >>>>>
>     >>>>> I think here Jan or Phillipe should take over.
>     >>>>
>     >>>> Thanks for bisecting, this is helpful!
>     >>>>
>     >>>> Philippe, any immediate idea why all that is failing now?
>     >>>
>     >>> Something may be going wrong with MAP_SHARED mappings wrt
>     commit_vma()
>     >>> in Dovetail. I'm adding this to my debug queue.
>     >>>
>     >>
>     >> This is still I-pipe, not a dovetail-related issues.
>     >
>     > This I-pipe release mimics what Dovetail does wrt mm pinning.
>     >
> 
>     Any news on this from your side?
> 
>     Florian took a trace from the system where this was observed on x86. It
>     seems to confirm that we have an unexpected minor fault here:
> 
>     smokey-568   [000]    74.233945: cobalt_head_sysentry:
>     syscall=thread_getschedparam_ex
>     smokey-568   [000]    74.233950: cobalt_pthread_getschedparam:
>     pth=0x7f2a18ebf700 policy=fifo param={ priority=3 }
>     smokey-568   [000]    74.233950: cobalt_head_sysexit:  result=0
>     smokey-568   [000]    74.233952: cobalt_head_sysentry:
>     syscall=thread_getstat
>     smokey-568   [000]    74.233952: cobalt_pthread_stat:  pid=568
>     smokey-568   [000]    74.233953: cobalt_head_sysexit:  result=0
>     smokey-568   [000]    74.233962: cobalt_thread_fault: 
>     ip=0x7f2a19cd6b46 type=e
>     smokey-568   [000]    74.233962: cobalt_shadow_gorelax: reason=fault
>     smokey-568   [000]    74.233963: cobalt_lostage_request:
>     request=ffffffffbcd85992 pid=568 comm=smokey
>     ...
>     smokey-568   [000]    74.234005: cobalt_shadow_relaxed:
>     state=0x480c0 info=0x0
>     ...
>     smokey-568   [000]    74.235027: cobalt_head_sysentry:
>     syscall=ftrace_puts
>     smokey-568   [000]    74.235028: cobalt_root_sysentry:
>     syscall=ftrace_puts
>     smokey-568   [000]    74.235028: print:               
>     CoBaLt_ftrace_puts: Second assertion failed
>     (that's around line 120 in smokey/setsched/setsched.c)
> 
>     Now, before I dig into the code you pointed to, I just wanted to sync.
> 
>     Jan
> 
>     -- 
>     Siemens AG, T RDA IOT
>     Corporate Competence Center Embedded Linux
> 
> 
> On my end I spent the weekend trying to reproduce the issue on ARM64. 
> I've been trying to reproduce on qemu using the default defconfig and
> the defconfig from the CI machines.  To save time I run the setsched
> test in smokey by passing in the test id.  I still haven't had any luck
> reproducing it.  I'm currently running it in a loop hoping it will cause
> the failure.  When you guys reproduce the issue, do all the tests get run?
> 

I had the same situation with the issue on x86/qemu: Same config
locally, same VM setup, but nothing triggered. So I asked Florian to
collect data from the lab setup where it can be seen reliably. It seems
that running the full testsuite via xeno-test also plays a role.

We likely miss some page pinning now, and whether those pages are
actually not mapped at the point of the test seems like luck. That makes
it hard to reproduce reliably.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: [I-PIPE] ipipe-core-4.19.165-cip41-arm64-09 released

[Philippe Gerum]

Jan Kiszka <jan.kiszka@siemens.com> writes:

> On 25.02.21 15:18, Philippe Gerum wrote:
>> 
>> Jan Kiszka <jan.kiszka@siemens.com> writes:
>> 
>>> On 25.02.21 14:54, Philippe Gerum wrote:
>>>>
>>>> Jan Kiszka <jan.kiszka@siemens.com> writes:
>>>>
>>>>> On 24.02.21 12:35, Henning Schild via Xenomai wrote:
>>>>>> Am Wed, 24 Feb 2021 11:24:55 +0100
>>>>>> schrieb Henning Schild via Xenomai <xenomai@xenomai.org>:
>>>>>>
>>>>>>> Am Wed, 10 Feb 2021 12:08:43 +0100
>>>>>>> schrieb Jan Kiszka via Xenomai <xenomai@xenomai.org>:
>>>>>>>
>>>>>>>> On 10.02.21 11:07, Bezdeka, Florian (T RDA IOT SES-DE) wrote:  
>>>>>>>>> On Wed, 2021-02-10 at 09:15 +0100, Jan Kiszka via Xenomai wrote:
>>>>>>>>>   
>>>>>>>>>> On 10.02.21 07:22, xenomai--- via Xenomai wrote:    
>>>>>>>>>>> Download URL:
>>>>>>>>>>> https://xenomai.org/downloads/ipipe/v4.x/arm64/ipipe-core-4.19.165-cip41-arm64-09.patch
>>>>>>>>>>>
>>>>>>>>>>> Repository: https://git.xenomai.org/ipipe-arm64
>>>>>>>>>>> Release tag: ipipe-core-4.19.165-cip41-arm64-09
>>>>>>>>>>>    
>>>>>>>>>>
>>>>>>>>>> Hmm, now we have the 5.4-arm64 issue also on 4.19:
>>>>>>>>>> https://gitlab.denx.de/Xenomai/xenomai-images/-/jobs/219984
>>>>>>>>>>    
>>>>>>>>>
>>>>>>>>> I don't know much about the things going on here, but found this
>>>>>>>>> line in the log. Maybe a starting point...
>>>>>>>>>
>>>>>>>>> 2021-02-10T07:51:47 setsched.c:120, assertion failed: stats.msw ==
>>>>>>>>> msw   
>>>>>>>>
>>>>>>>> Exactly, that is causing the overall failure. And it was first seen
>>>>>>>> with the newly added 5.4 kernel.  
>>>>>>>
>>>>>>> Seing the same on amd64 when testing on qemu, real HW is fine.
>>>>>>>
>>>>>>> Managed to bisect it down to 4.19.147-cip (good) 4.19.150-cip (bad)
>>>>>>>
>>>>>>> Which also means that ipipe-core-4.19.152-cip37-x86-15 is affected.
>>>>>>>
>>>>>>> https://gitlab.denx.de/Xenomai/xenomai-images/-/jobs/200646
>>>>>>> did not find it, so maybe our config differs
>>>>>
>>>>> Already compared yours against the one in xenomai-images? That would be
>>>>> useful.
>>>>>
>>>>>>
>>>>>> Digging further i found 0f0b6099c45ff3e06d2487816cf1ff30d21835f6 likely
>>>>>> causing the problem.
>>>>>>
>>>>>> ipipe-core-4.19.152-cip37-x86-15 <- bad
>>>>>> revert 2b294ac325c7ce3f36854b74d0d1d89dc1d1d8b8
>>>>>> revert 8579a0440381353e0a71dd6a4d4371be8457eac4 <- bad
>>>>>> revert 0f0b6099c45ff3e06d2487816cf1ff30d <- good
>>>>>>
>>>>>> I think here Jan or Phillipe should take over.
>>>>>
>>>>> Thanks for bisecting, this is helpful!
>>>>>
>>>>> Philippe, any immediate idea why all that is failing now?
>>>>
>>>> Something may be going wrong with MAP_SHARED mappings wrt commit_vma()
>>>> in Dovetail. I'm adding this to my debug queue.
>>>>
>>>
>>> This is still I-pipe, not a dovetail-related issues.
>> 
>> This I-pipe release mimics what Dovetail does wrt mm pinning.
>> 
>
> Any news on this from your side?
>

No time slot for working on this yet. High multiplexing rate ATM.

-- 
Philippe.

Re: [I-PIPE] ipipe-core-4.19.165-cip41-arm64-09 released

[Jan Kiszka]

On 01.03.21 17:53, Philippe Gerum wrote:
> 
> Jan Kiszka <jan.kiszka@siemens.com> writes:
> 
>> On 25.02.21 15:18, Philippe Gerum wrote:
>>>
>>> Jan Kiszka <jan.kiszka@siemens.com> writes:
>>>
>>>> On 25.02.21 14:54, Philippe Gerum wrote:
>>>>>
>>>>> Jan Kiszka <jan.kiszka@siemens.com> writes:
>>>>>
>>>>>> On 24.02.21 12:35, Henning Schild via Xenomai wrote:
>>>>>>> Am Wed, 24 Feb 2021 11:24:55 +0100
>>>>>>> schrieb Henning Schild via Xenomai <xenomai@xenomai.org>:
>>>>>>>
>>>>>>>> Am Wed, 10 Feb 2021 12:08:43 +0100
>>>>>>>> schrieb Jan Kiszka via Xenomai <xenomai@xenomai.org>:
>>>>>>>>
>>>>>>>>> On 10.02.21 11:07, Bezdeka, Florian (T RDA IOT SES-DE) wrote:  
>>>>>>>>>> On Wed, 2021-02-10 at 09:15 +0100, Jan Kiszka via Xenomai wrote:
>>>>>>>>>>   
>>>>>>>>>>> On 10.02.21 07:22, xenomai--- via Xenomai wrote:    
>>>>>>>>>>>> Download URL:
>>>>>>>>>>>> https://xenomai.org/downloads/ipipe/v4.x/arm64/ipipe-core-4.19.165-cip41-arm64-09.patch
>>>>>>>>>>>>
>>>>>>>>>>>> Repository: https://git.xenomai.org/ipipe-arm64
>>>>>>>>>>>> Release tag: ipipe-core-4.19.165-cip41-arm64-09
>>>>>>>>>>>>    
>>>>>>>>>>>
>>>>>>>>>>> Hmm, now we have the 5.4-arm64 issue also on 4.19:
>>>>>>>>>>> https://gitlab.denx.de/Xenomai/xenomai-images/-/jobs/219984
>>>>>>>>>>>    
>>>>>>>>>>
>>>>>>>>>> I don't know much about the things going on here, but found this
>>>>>>>>>> line in the log. Maybe a starting point...
>>>>>>>>>>
>>>>>>>>>> 2021-02-10T07:51:47 setsched.c:120, assertion failed: stats.msw ==
>>>>>>>>>> msw   
>>>>>>>>>
>>>>>>>>> Exactly, that is causing the overall failure. And it was first seen
>>>>>>>>> with the newly added 5.4 kernel.  
>>>>>>>>
>>>>>>>> Seing the same on amd64 when testing on qemu, real HW is fine.
>>>>>>>>
>>>>>>>> Managed to bisect it down to 4.19.147-cip (good) 4.19.150-cip (bad)
>>>>>>>>
>>>>>>>> Which also means that ipipe-core-4.19.152-cip37-x86-15 is affected.
>>>>>>>>
>>>>>>>> https://gitlab.denx.de/Xenomai/xenomai-images/-/jobs/200646
>>>>>>>> did not find it, so maybe our config differs
>>>>>>
>>>>>> Already compared yours against the one in xenomai-images? That would be
>>>>>> useful.
>>>>>>
>>>>>>>
>>>>>>> Digging further i found 0f0b6099c45ff3e06d2487816cf1ff30d21835f6 likely
>>>>>>> causing the problem.
>>>>>>>
>>>>>>> ipipe-core-4.19.152-cip37-x86-15 <- bad
>>>>>>> revert 2b294ac325c7ce3f36854b74d0d1d89dc1d1d8b8
>>>>>>> revert 8579a0440381353e0a71dd6a4d4371be8457eac4 <- bad
>>>>>>> revert 0f0b6099c45ff3e06d2487816cf1ff30d <- good
>>>>>>>
>>>>>>> I think here Jan or Phillipe should take over.
>>>>>>
>>>>>> Thanks for bisecting, this is helpful!
>>>>>>
>>>>>> Philippe, any immediate idea why all that is failing now?
>>>>>
>>>>> Something may be going wrong with MAP_SHARED mappings wrt commit_vma()
>>>>> in Dovetail. I'm adding this to my debug queue.
>>>>>
>>>>
>>>> This is still I-pipe, not a dovetail-related issues.
>>>
>>> This I-pipe release mimics what Dovetail does wrt mm pinning.
>>>
>>
>> Any news on this from your side?
>>
> 
> No time slot for working on this yet. High multiplexing rate ATM.
> 

I reproduced the issue on qemu-arm64 (xenomai-images exposes it
directly), and I'm testing a fix.

Brief summary:
Removal of un-COW support was a mistake. We will continue to require it
because it not only affects the child (where the removal argumentation
was targeting), but it also prevents that shared pages - even if locked
- on a RT parent suddenly become read-only.

Expect some patches later today.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: [I-PIPE] ipipe-core-4.19.165-cip41-arm64-09 released

[Philippe Gerum]

Jan Kiszka <jan.kiszka@siemens.com> writes:

> On 01.03.21 17:53, Philippe Gerum wrote:
>> 
>> Jan Kiszka <jan.kiszka@siemens.com> writes:
>> 
>>> On 25.02.21 15:18, Philippe Gerum wrote:
>>>>
>>>> Jan Kiszka <jan.kiszka@siemens.com> writes:
>>>>
>>>>> On 25.02.21 14:54, Philippe Gerum wrote:
>>>>>>
>>>>>> Jan Kiszka <jan.kiszka@siemens.com> writes:
>>>>>>
>>>>>>> On 24.02.21 12:35, Henning Schild via Xenomai wrote:
>>>>>>>> Am Wed, 24 Feb 2021 11:24:55 +0100
>>>>>>>> schrieb Henning Schild via Xenomai <xenomai@xenomai.org>:
>>>>>>>>
>>>>>>>>> Am Wed, 10 Feb 2021 12:08:43 +0100
>>>>>>>>> schrieb Jan Kiszka via Xenomai <xenomai@xenomai.org>:
>>>>>>>>>
>>>>>>>>>> On 10.02.21 11:07, Bezdeka, Florian (T RDA IOT SES-DE) wrote:  
>>>>>>>>>>> On Wed, 2021-02-10 at 09:15 +0100, Jan Kiszka via Xenomai wrote:
>>>>>>>>>>>   
>>>>>>>>>>>> On 10.02.21 07:22, xenomai--- via Xenomai wrote:    
>>>>>>>>>>>>> Download URL:
>>>>>>>>>>>>> https://xenomai.org/downloads/ipipe/v4.x/arm64/ipipe-core-4.19.165-cip41-arm64-09.patch
>>>>>>>>>>>>>
>>>>>>>>>>>>> Repository: https://git.xenomai.org/ipipe-arm64
>>>>>>>>>>>>> Release tag: ipipe-core-4.19.165-cip41-arm64-09
>>>>>>>>>>>>>    
>>>>>>>>>>>>
>>>>>>>>>>>> Hmm, now we have the 5.4-arm64 issue also on 4.19:
>>>>>>>>>>>> https://gitlab.denx.de/Xenomai/xenomai-images/-/jobs/219984
>>>>>>>>>>>>    
>>>>>>>>>>>
>>>>>>>>>>> I don't know much about the things going on here, but found this
>>>>>>>>>>> line in the log. Maybe a starting point...
>>>>>>>>>>>
>>>>>>>>>>> 2021-02-10T07:51:47 setsched.c:120, assertion failed: stats.msw ==
>>>>>>>>>>> msw   
>>>>>>>>>>
>>>>>>>>>> Exactly, that is causing the overall failure. And it was first seen
>>>>>>>>>> with the newly added 5.4 kernel.  
>>>>>>>>>
>>>>>>>>> Seing the same on amd64 when testing on qemu, real HW is fine.
>>>>>>>>>
>>>>>>>>> Managed to bisect it down to 4.19.147-cip (good) 4.19.150-cip (bad)
>>>>>>>>>
>>>>>>>>> Which also means that ipipe-core-4.19.152-cip37-x86-15 is affected.
>>>>>>>>>
>>>>>>>>> https://gitlab.denx.de/Xenomai/xenomai-images/-/jobs/200646
>>>>>>>>> did not find it, so maybe our config differs
>>>>>>>
>>>>>>> Already compared yours against the one in xenomai-images? That would be
>>>>>>> useful.
>>>>>>>
>>>>>>>>
>>>>>>>> Digging further i found 0f0b6099c45ff3e06d2487816cf1ff30d21835f6 likely
>>>>>>>> causing the problem.
>>>>>>>>
>>>>>>>> ipipe-core-4.19.152-cip37-x86-15 <- bad
>>>>>>>> revert 2b294ac325c7ce3f36854b74d0d1d89dc1d1d8b8
>>>>>>>> revert 8579a0440381353e0a71dd6a4d4371be8457eac4 <- bad
>>>>>>>> revert 0f0b6099c45ff3e06d2487816cf1ff30d <- good
>>>>>>>>
>>>>>>>> I think here Jan or Phillipe should take over.
>>>>>>>
>>>>>>> Thanks for bisecting, this is helpful!
>>>>>>>
>>>>>>> Philippe, any immediate idea why all that is failing now?
>>>>>>
>>>>>> Something may be going wrong with MAP_SHARED mappings wrt commit_vma()
>>>>>> in Dovetail. I'm adding this to my debug queue.
>>>>>>
>>>>>
>>>>> This is still I-pipe, not a dovetail-related issues.
>>>>
>>>> This I-pipe release mimics what Dovetail does wrt mm pinning.
>>>>
>>>
>>> Any news on this from your side?
>>>
>> 
>> No time slot for working on this yet. High multiplexing rate ATM.
>> 
>
> I reproduced the issue on qemu-arm64 (xenomai-images exposes it
> directly), and I'm testing a fix.
>
> Brief summary:
> Removal of un-COW support was a mistake. We will continue to require it
> because it not only affects the child (where the removal argumentation
> was targeting), but it also prevents that shared pages - even if locked
> - on a RT parent suddenly become read-only.
>
> Expect some patches later today.

The best fix is not add that ugly code back, but rather make VMA commit
code work with shared mappings.

-- 
Philippe.

Re: [I-PIPE] ipipe-core-4.19.165-cip41-arm64-09 released

[Jan Kiszka]

On 05.03.21 10:34, Philippe Gerum wrote:
> 
> Jan Kiszka <jan.kiszka@siemens.com> writes:
> 
>> On 01.03.21 17:53, Philippe Gerum wrote:
>>>
>>> Jan Kiszka <jan.kiszka@siemens.com> writes:
>>>
>>>> On 25.02.21 15:18, Philippe Gerum wrote:
>>>>>
>>>>> Jan Kiszka <jan.kiszka@siemens.com> writes:
>>>>>
>>>>>> On 25.02.21 14:54, Philippe Gerum wrote:
>>>>>>>
>>>>>>> Jan Kiszka <jan.kiszka@siemens.com> writes:
>>>>>>>
>>>>>>>> On 24.02.21 12:35, Henning Schild via Xenomai wrote:
>>>>>>>>> Am Wed, 24 Feb 2021 11:24:55 +0100
>>>>>>>>> schrieb Henning Schild via Xenomai <xenomai@xenomai.org>:
>>>>>>>>>
>>>>>>>>>> Am Wed, 10 Feb 2021 12:08:43 +0100
>>>>>>>>>> schrieb Jan Kiszka via Xenomai <xenomai@xenomai.org>:
>>>>>>>>>>
>>>>>>>>>>> On 10.02.21 11:07, Bezdeka, Florian (T RDA IOT SES-DE) wrote:  
>>>>>>>>>>>> On Wed, 2021-02-10 at 09:15 +0100, Jan Kiszka via Xenomai wrote:
>>>>>>>>>>>>   
>>>>>>>>>>>>> On 10.02.21 07:22, xenomai--- via Xenomai wrote:    
>>>>>>>>>>>>>> Download URL:
>>>>>>>>>>>>>> https://xenomai.org/downloads/ipipe/v4.x/arm64/ipipe-core-4.19.165-cip41-arm64-09.patch
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Repository: https://git.xenomai.org/ipipe-arm64
>>>>>>>>>>>>>> Release tag: ipipe-core-4.19.165-cip41-arm64-09
>>>>>>>>>>>>>>    
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hmm, now we have the 5.4-arm64 issue also on 4.19:
>>>>>>>>>>>>> https://gitlab.denx.de/Xenomai/xenomai-images/-/jobs/219984
>>>>>>>>>>>>>    
>>>>>>>>>>>>
>>>>>>>>>>>> I don't know much about the things going on here, but found this
>>>>>>>>>>>> line in the log. Maybe a starting point...
>>>>>>>>>>>>
>>>>>>>>>>>> 2021-02-10T07:51:47 setsched.c:120, assertion failed: stats.msw ==
>>>>>>>>>>>> msw   
>>>>>>>>>>>
>>>>>>>>>>> Exactly, that is causing the overall failure. And it was first seen
>>>>>>>>>>> with the newly added 5.4 kernel.  
>>>>>>>>>>
>>>>>>>>>> Seing the same on amd64 when testing on qemu, real HW is fine.
>>>>>>>>>>
>>>>>>>>>> Managed to bisect it down to 4.19.147-cip (good) 4.19.150-cip (bad)
>>>>>>>>>>
>>>>>>>>>> Which also means that ipipe-core-4.19.152-cip37-x86-15 is affected.
>>>>>>>>>>
>>>>>>>>>> https://gitlab.denx.de/Xenomai/xenomai-images/-/jobs/200646
>>>>>>>>>> did not find it, so maybe our config differs
>>>>>>>>
>>>>>>>> Already compared yours against the one in xenomai-images? That would be
>>>>>>>> useful.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Digging further i found 0f0b6099c45ff3e06d2487816cf1ff30d21835f6 likely
>>>>>>>>> causing the problem.
>>>>>>>>>
>>>>>>>>> ipipe-core-4.19.152-cip37-x86-15 <- bad
>>>>>>>>> revert 2b294ac325c7ce3f36854b74d0d1d89dc1d1d8b8
>>>>>>>>> revert 8579a0440381353e0a71dd6a4d4371be8457eac4 <- bad
>>>>>>>>> revert 0f0b6099c45ff3e06d2487816cf1ff30d <- good
>>>>>>>>>
>>>>>>>>> I think here Jan or Phillipe should take over.
>>>>>>>>
>>>>>>>> Thanks for bisecting, this is helpful!
>>>>>>>>
>>>>>>>> Philippe, any immediate idea why all that is failing now?
>>>>>>>
>>>>>>> Something may be going wrong with MAP_SHARED mappings wrt commit_vma()
>>>>>>> in Dovetail. I'm adding this to my debug queue.
>>>>>>>
>>>>>>
>>>>>> This is still I-pipe, not a dovetail-related issues.
>>>>>
>>>>> This I-pipe release mimics what Dovetail does wrt mm pinning.
>>>>>
>>>>
>>>> Any news on this from your side?
>>>>
>>>
>>> No time slot for working on this yet. High multiplexing rate ATM.
>>>
>>
>> I reproduced the issue on qemu-arm64 (xenomai-images exposes it
>> directly), and I'm testing a fix.
>>
>> Brief summary:
>> Removal of un-COW support was a mistake. We will continue to require it
>> because it not only affects the child (where the removal argumentation
>> was targeting), but it also prevents that shared pages - even if locked
>> - on a RT parent suddenly become read-only.
>>
>> Expect some patches later today.
> 
> The best fix is not add that ugly code back, but rather make VMA commit
> code work with shared mappings.
> 

What exactly do you mean?

We must avoid that shared pages (with the child) become read-only on the
parent. How to do that other than un-COWing?

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: [I-PIPE] ipipe-core-4.19.165-cip41-arm64-09 released

[Philippe Gerum]

Jan Kiszka <jan.kiszka@siemens.com> writes:

> On 05.03.21 10:34, Philippe Gerum wrote:
>> 
>> Jan Kiszka <jan.kiszka@siemens.com> writes:
>> 
>>> On 01.03.21 17:53, Philippe Gerum wrote:
>>>>
>>>> Jan Kiszka <jan.kiszka@siemens.com> writes:
>>>>
>>>>> On 25.02.21 15:18, Philippe Gerum wrote:
>>>>>>
>>>>>> Jan Kiszka <jan.kiszka@siemens.com> writes:
>>>>>>
>>>>>>> On 25.02.21 14:54, Philippe Gerum wrote:
>>>>>>>>
>>>>>>>> Jan Kiszka <jan.kiszka@siemens.com> writes:
>>>>>>>>
>>>>>>>>> On 24.02.21 12:35, Henning Schild via Xenomai wrote:
>>>>>>>>>> Am Wed, 24 Feb 2021 11:24:55 +0100
>>>>>>>>>> schrieb Henning Schild via Xenomai <xenomai@xenomai.org>:
>>>>>>>>>>
>>>>>>>>>>> Am Wed, 10 Feb 2021 12:08:43 +0100
>>>>>>>>>>> schrieb Jan Kiszka via Xenomai <xenomai@xenomai.org>:
>>>>>>>>>>>
>>>>>>>>>>>> On 10.02.21 11:07, Bezdeka, Florian (T RDA IOT SES-DE) wrote:  
>>>>>>>>>>>>> On Wed, 2021-02-10 at 09:15 +0100, Jan Kiszka via Xenomai wrote:
>>>>>>>>>>>>>   
>>>>>>>>>>>>>> On 10.02.21 07:22, xenomai--- via Xenomai wrote:    
>>>>>>>>>>>>>>> Download URL:
>>>>>>>>>>>>>>> https://xenomai.org/downloads/ipipe/v4.x/arm64/ipipe-core-4.19.165-cip41-arm64-09.patch
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Repository: https://git.xenomai.org/ipipe-arm64
>>>>>>>>>>>>>>> Release tag: ipipe-core-4.19.165-cip41-arm64-09
>>>>>>>>>>>>>>>    
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hmm, now we have the 5.4-arm64 issue also on 4.19:
>>>>>>>>>>>>>> https://gitlab.denx.de/Xenomai/xenomai-images/-/jobs/219984
>>>>>>>>>>>>>>    
>>>>>>>>>>>>>
>>>>>>>>>>>>> I don't know much about the things going on here, but found this
>>>>>>>>>>>>> line in the log. Maybe a starting point...
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2021-02-10T07:51:47 setsched.c:120, assertion failed: stats.msw ==
>>>>>>>>>>>>> msw   
>>>>>>>>>>>>
>>>>>>>>>>>> Exactly, that is causing the overall failure. And it was first seen
>>>>>>>>>>>> with the newly added 5.4 kernel.  
>>>>>>>>>>>
>>>>>>>>>>> Seing the same on amd64 when testing on qemu, real HW is fine.
>>>>>>>>>>>
>>>>>>>>>>> Managed to bisect it down to 4.19.147-cip (good) 4.19.150-cip (bad)
>>>>>>>>>>>
>>>>>>>>>>> Which also means that ipipe-core-4.19.152-cip37-x86-15 is affected.
>>>>>>>>>>>
>>>>>>>>>>> https://gitlab.denx.de/Xenomai/xenomai-images/-/jobs/200646
>>>>>>>>>>> did not find it, so maybe our config differs
>>>>>>>>>
>>>>>>>>> Already compared yours against the one in xenomai-images? That would be
>>>>>>>>> useful.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Digging further i found 0f0b6099c45ff3e06d2487816cf1ff30d21835f6 likely
>>>>>>>>>> causing the problem.
>>>>>>>>>>
>>>>>>>>>> ipipe-core-4.19.152-cip37-x86-15 <- bad
>>>>>>>>>> revert 2b294ac325c7ce3f36854b74d0d1d89dc1d1d8b8
>>>>>>>>>> revert 8579a0440381353e0a71dd6a4d4371be8457eac4 <- bad
>>>>>>>>>> revert 0f0b6099c45ff3e06d2487816cf1ff30d <- good
>>>>>>>>>>
>>>>>>>>>> I think here Jan or Phillipe should take over.
>>>>>>>>>
>>>>>>>>> Thanks for bisecting, this is helpful!
>>>>>>>>>
>>>>>>>>> Philippe, any immediate idea why all that is failing now?
>>>>>>>>
>>>>>>>> Something may be going wrong with MAP_SHARED mappings wrt commit_vma()
>>>>>>>> in Dovetail. I'm adding this to my debug queue.
>>>>>>>>
>>>>>>>
>>>>>>> This is still I-pipe, not a dovetail-related issues.
>>>>>>
>>>>>> This I-pipe release mimics what Dovetail does wrt mm pinning.
>>>>>>
>>>>>
>>>>> Any news on this from your side?
>>>>>
>>>>
>>>> No time slot for working on this yet. High multiplexing rate ATM.
>>>>
>>>
>>> I reproduced the issue on qemu-arm64 (xenomai-images exposes it
>>> directly), and I'm testing a fix.
>>>
>>> Brief summary:
>>> Removal of un-COW support was a mistake. We will continue to require it
>>> because it not only affects the child (where the removal argumentation
>>> was targeting), but it also prevents that shared pages - even if locked
>>> - on a RT parent suddenly become read-only.
>>>
>>> Expect some patches later today.
>> 
>> The best fix is not add that ugly code back, but rather make VMA commit
>> code work with shared mappings.
>> 
>
> What exactly do you mean?
>
> We must avoid that shared pages (with the child) become read-only on the
> parent. How to do that other than un-COWing?
>

The issue is not with un-COW is obviously the only thing to do, but
rather with how and where this is done. The way it used to be done when
copying the PTEs led to several conflicts and subtle breakages due to
upstream changes over time. Hopefully a better implementation is
possible.

-- 
Philippe.

Re: [I-PIPE] ipipe-core-4.19.165-cip41-arm64-09 released

[Jan Kiszka]

On 05.03.21 12:29, Philippe Gerum wrote:
> 
> Jan Kiszka <jan.kiszka@siemens.com> writes:
> 
>> On 05.03.21 10:34, Philippe Gerum wrote:
>>>
>>> Jan Kiszka <jan.kiszka@siemens.com> writes:
>>>
>>>> On 01.03.21 17:53, Philippe Gerum wrote:
>>>>>
>>>>> Jan Kiszka <jan.kiszka@siemens.com> writes:
>>>>>
>>>>>> On 25.02.21 15:18, Philippe Gerum wrote:
>>>>>>>
>>>>>>> Jan Kiszka <jan.kiszka@siemens.com> writes:
>>>>>>>
>>>>>>>> On 25.02.21 14:54, Philippe Gerum wrote:
>>>>>>>>>
>>>>>>>>> Jan Kiszka <jan.kiszka@siemens.com> writes:
>>>>>>>>>
>>>>>>>>>> On 24.02.21 12:35, Henning Schild via Xenomai wrote:
>>>>>>>>>>> Am Wed, 24 Feb 2021 11:24:55 +0100
>>>>>>>>>>> schrieb Henning Schild via Xenomai <xenomai@xenomai.org>:
>>>>>>>>>>>
>>>>>>>>>>>> Am Wed, 10 Feb 2021 12:08:43 +0100
>>>>>>>>>>>> schrieb Jan Kiszka via Xenomai <xenomai@xenomai.org>:
>>>>>>>>>>>>
>>>>>>>>>>>>> On 10.02.21 11:07, Bezdeka, Florian (T RDA IOT SES-DE) wrote:  
>>>>>>>>>>>>>> On Wed, 2021-02-10 at 09:15 +0100, Jan Kiszka via Xenomai wrote:
>>>>>>>>>>>>>>   
>>>>>>>>>>>>>>> On 10.02.21 07:22, xenomai--- via Xenomai wrote:    
>>>>>>>>>>>>>>>> Download URL:
>>>>>>>>>>>>>>>> https://xenomai.org/downloads/ipipe/v4.x/arm64/ipipe-core-4.19.165-cip41-arm64-09.patch
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Repository: https://git.xenomai.org/ipipe-arm64
>>>>>>>>>>>>>>>> Release tag: ipipe-core-4.19.165-cip41-arm64-09
>>>>>>>>>>>>>>>>    
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hmm, now we have the 5.4-arm64 issue also on 4.19:
>>>>>>>>>>>>>>> https://gitlab.denx.de/Xenomai/xenomai-images/-/jobs/219984
>>>>>>>>>>>>>>>    
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I don't know much about the things going on here, but found this
>>>>>>>>>>>>>> line in the log. Maybe a starting point...
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2021-02-10T07:51:47 setsched.c:120, assertion failed: stats.msw ==
>>>>>>>>>>>>>> msw   
>>>>>>>>>>>>>
>>>>>>>>>>>>> Exactly, that is causing the overall failure. And it was first seen
>>>>>>>>>>>>> with the newly added 5.4 kernel.  
>>>>>>>>>>>>
>>>>>>>>>>>> Seing the same on amd64 when testing on qemu, real HW is fine.
>>>>>>>>>>>>
>>>>>>>>>>>> Managed to bisect it down to 4.19.147-cip (good) 4.19.150-cip (bad)
>>>>>>>>>>>>
>>>>>>>>>>>> Which also means that ipipe-core-4.19.152-cip37-x86-15 is affected.
>>>>>>>>>>>>
>>>>>>>>>>>> https://gitlab.denx.de/Xenomai/xenomai-images/-/jobs/200646
>>>>>>>>>>>> did not find it, so maybe our config differs
>>>>>>>>>>
>>>>>>>>>> Already compared yours against the one in xenomai-images? That would be
>>>>>>>>>> useful.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Digging further i found 0f0b6099c45ff3e06d2487816cf1ff30d21835f6 likely
>>>>>>>>>>> causing the problem.
>>>>>>>>>>>
>>>>>>>>>>> ipipe-core-4.19.152-cip37-x86-15 <- bad
>>>>>>>>>>> revert 2b294ac325c7ce3f36854b74d0d1d89dc1d1d8b8
>>>>>>>>>>> revert 8579a0440381353e0a71dd6a4d4371be8457eac4 <- bad
>>>>>>>>>>> revert 0f0b6099c45ff3e06d2487816cf1ff30d <- good
>>>>>>>>>>>
>>>>>>>>>>> I think here Jan or Phillipe should take over.
>>>>>>>>>>
>>>>>>>>>> Thanks for bisecting, this is helpful!
>>>>>>>>>>
>>>>>>>>>> Philippe, any immediate idea why all that is failing now?
>>>>>>>>>
>>>>>>>>> Something may be going wrong with MAP_SHARED mappings wrt commit_vma()
>>>>>>>>> in Dovetail. I'm adding this to my debug queue.
>>>>>>>>>
>>>>>>>>
>>>>>>>> This is still I-pipe, not a dovetail-related issues.
>>>>>>>
>>>>>>> This I-pipe release mimics what Dovetail does wrt mm pinning.
>>>>>>>
>>>>>>
>>>>>> Any news on this from your side?
>>>>>>
>>>>>
>>>>> No time slot for working on this yet. High multiplexing rate ATM.
>>>>>
>>>>
>>>> I reproduced the issue on qemu-arm64 (xenomai-images exposes it
>>>> directly), and I'm testing a fix.
>>>>
>>>> Brief summary:
>>>> Removal of un-COW support was a mistake. We will continue to require it
>>>> because it not only affects the child (where the removal argumentation
>>>> was targeting), but it also prevents that shared pages - even if locked
>>>> - on a RT parent suddenly become read-only.
>>>>
>>>> Expect some patches later today.
>>>
>>> The best fix is not add that ugly code back, but rather make VMA commit
>>> code work with shared mappings.
>>>
>>
>> What exactly do you mean?
>>
>> We must avoid that shared pages (with the child) become read-only on the
>> parent. How to do that other than un-COWing?
>>
> 
> The issue is not with un-COW is obviously the only thing to do, but
> rather with how and where this is done. The way it used to be done when
> copying the PTEs led to several conflicts and subtle breakages due to
> upstream changes over time. Hopefully a better implementation is
> possible.
> 

Do you have one at hand or can guide how to write that?

Otherwise, I would suggest to restore the code to fix the regression and
clean up later. FWIW, I'll throw my current 4.19 fix on the list in a
minute.

Jan