[Dovetail] x86 test version available (kernel v5.10)

[Dovetail] x86 test version available (kernel v5.10)

[Philippe Gerum]

The initial port of the Cobalt core to Dovetail/x86 is available from
[1]. Ports to Dovetail/ARM and Dovetail/arm64 should follow within a
couple of weeks.

So far, latency and switchtest run flawlessly. Most of the smokey test
suite passes successfully, except the GDB test at the moment.

How to test this:

- clone Xenomai from [1], switch to branch for-upstream/dovetail
- clone the Dovetail tree (v5.10) from [2], switch to branch dovetail/master
- run scripts/prepare-kernel.sh available from [1] into [2] (usual procedure)
  for x86, that would be: .../scripts/prepare-kernel.sh --arch=x86
- build your kernel using the sources from [2]

There is no user-visible Kconfig change compared to an I-pipe based
version.

Alternatively, the Xenomai code base in [1] can also run on top of the
I-pipe. prepare-kernel.sh detects which pipeline flavour is there, and
prepares the source tree accordingly.

This code is being gradually merged into Xenomai's -next branch, and
will be at the core of Xenomai 3.2. Testing and feedback appreciated.

Thanks,

[1] https://lab.xenomai.org/xenomai-rpm.git, "for-upstream/dovetail" branch
[2] https://git.evlproject.org/linux-evl.git, "dovetail/master" branch

-- 
Philippe.

Re: [Dovetail] x86 test version available (kernel v5.10)

[Jan Kiszka]

On 31.01.21 17:06, Philippe Gerum wrote:
> 
> The initial port of the Cobalt core to Dovetail/x86 is available from
> [1]. Ports to Dovetail/ARM and Dovetail/arm64 should follow within a
> couple of weeks.
> 
> So far, latency and switchtest run flawlessly. Most of the smokey test
> suite passes successfully, except the GDB test at the moment.
> 
> How to test this:
> 
> - clone Xenomai from [1], switch to branch for-upstream/dovetail
> - clone the Dovetail tree (v5.10) from [2], switch to branch dovetail/master
> - run scripts/prepare-kernel.sh available from [1] into [2] (usual procedure)
>   for x86, that would be: .../scripts/prepare-kernel.sh --arch=x86
> - build your kernel using the sources from [2]
> 
> There is no user-visible Kconfig change compared to an I-pipe based
> version.
> 
> Alternatively, the Xenomai code base in [1] can also run on top of the
> I-pipe. prepare-kernel.sh detects which pipeline flavour is there, and
> prepares the source tree accordingly.
> 
> This code is being gradually merged into Xenomai's -next branch, and
> will be at the core of Xenomai 3.2. Testing and feedback appreciated.
> 
> Thanks,
> 
> [1] https://lab.xenomai.org/xenomai-rpm.git, "for-upstream/dovetail" branch
> [2] https://git.evlproject.org/linux-evl.git, "dovetail/master" branch
> 

First try on my end looked good as well - great news, well done!

Would it help to stick the wip branch into our CI/CT?

Will have a look at the next round of pending patches.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: [Dovetail] x86 test version available (kernel v5.10)

[Philippe Gerum]

Jan Kiszka <jan.kiszka@siemens.com> writes:

> On 31.01.21 17:06, Philippe Gerum wrote:
>> 
>> The initial port of the Cobalt core to Dovetail/x86 is available from
>> [1]. Ports to Dovetail/ARM and Dovetail/arm64 should follow within a
>> couple of weeks.
>> 
>> So far, latency and switchtest run flawlessly. Most of the smokey test
>> suite passes successfully, except the GDB test at the moment.
>> 
>> How to test this:
>> 
>> - clone Xenomai from [1], switch to branch for-upstream/dovetail
>> - clone the Dovetail tree (v5.10) from [2], switch to branch dovetail/master
>> - run scripts/prepare-kernel.sh available from [1] into [2] (usual procedure)
>>   for x86, that would be: .../scripts/prepare-kernel.sh --arch=x86
>> - build your kernel using the sources from [2]
>> 
>> There is no user-visible Kconfig change compared to an I-pipe based
>> version.
>> 
>> Alternatively, the Xenomai code base in [1] can also run on top of the
>> I-pipe. prepare-kernel.sh detects which pipeline flavour is there, and
>> prepares the source tree accordingly.
>> 
>> This code is being gradually merged into Xenomai's -next branch, and
>> will be at the core of Xenomai 3.2. Testing and feedback appreciated.
>> 
>> Thanks,
>> 
>> [1] https://lab.xenomai.org/xenomai-rpm.git, "for-upstream/dovetail" branch
>> [2] https://git.evlproject.org/linux-evl.git, "dovetail/master" branch
>> 
>
> First try on my end looked good as well - great news, well done!
>
> Would it help to stick the wip branch into our CI/CT?
>

Yep, it would. TIA,

-- 
Philippe.

Re: [Dovetail] x86 test version available (kernel v5.10)

[Jan Kiszka]

On 03.02.21 18:35, Philippe Gerum wrote:
> 
> Jan Kiszka <jan.kiszka@siemens.com> writes:
> 
>> On 31.01.21 17:06, Philippe Gerum wrote:
>>>
>>> The initial port of the Cobalt core to Dovetail/x86 is available from
>>> [1]. Ports to Dovetail/ARM and Dovetail/arm64 should follow within a
>>> couple of weeks.
>>>
>>> So far, latency and switchtest run flawlessly. Most of the smokey test
>>> suite passes successfully, except the GDB test at the moment.
>>>
>>> How to test this:
>>>
>>> - clone Xenomai from [1], switch to branch for-upstream/dovetail
>>> - clone the Dovetail tree (v5.10) from [2], switch to branch dovetail/master
>>> - run scripts/prepare-kernel.sh available from [1] into [2] (usual procedure)
>>>   for x86, that would be: .../scripts/prepare-kernel.sh --arch=x86
>>> - build your kernel using the sources from [2]
>>>
>>> There is no user-visible Kconfig change compared to an I-pipe based
>>> version.
>>>
>>> Alternatively, the Xenomai code base in [1] can also run on top of the
>>> I-pipe. prepare-kernel.sh detects which pipeline flavour is there, and
>>> prepares the source tree accordingly.
>>>
>>> This code is being gradually merged into Xenomai's -next branch, and
>>> will be at the core of Xenomai 3.2. Testing and feedback appreciated.
>>>
>>> Thanks,
>>>
>>> [1] https://lab.xenomai.org/xenomai-rpm.git, "for-upstream/dovetail" branch
>>> [2] https://git.evlproject.org/linux-evl.git, "dovetail/master" branch
>>>
>>
>> First try on my end looked good as well - great news, well done!
>>
>> Would it help to stick the wip branch into our CI/CT?
>>
> 
> Yep, it would. TIA,
> 

CI build is passing now, see

https://gitlab.denx.de/Xenomai/xenomai/-/pipelines/6258

I'm carrying two fixes and a could of tweaks:

https://gitlab.denx.de/Xenomai/xenomai/-/commits/wip/dovetail

Specifically
https://gitlab.denx.de/Xenomai/xenomai/-/commit/4196216e821abea3107284a8e31e13bdf05141d5
would a second look as the timer ipi abstraction became obsolete with
the code moving to pipeline-specific code.

I've just started also target runs, see
https://gitlab.denx.de/Xenomai/xenomai-images/-/pipelines/6263. There
will be several known breakages, though, namely the gdb test but also
the unrelated arm[64]-5.4 regressions.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: [Dovetail] x86 test version available (kernel v5.10)

[Philippe Gerum]

Jan Kiszka <jan.kiszka@siemens.com> writes:

> On 03.02.21 18:35, Philippe Gerum wrote:
>> 
>> Jan Kiszka <jan.kiszka@siemens.com> writes:
>> 
>>> On 31.01.21 17:06, Philippe Gerum wrote:
>>>>
>>>> The initial port of the Cobalt core to Dovetail/x86 is available from
>>>> [1]. Ports to Dovetail/ARM and Dovetail/arm64 should follow within a
>>>> couple of weeks.
>>>>
>>>> So far, latency and switchtest run flawlessly. Most of the smokey test
>>>> suite passes successfully, except the GDB test at the moment.
>>>>
>>>> How to test this:
>>>>
>>>> - clone Xenomai from [1], switch to branch for-upstream/dovetail
>>>> - clone the Dovetail tree (v5.10) from [2], switch to branch dovetail/master
>>>> - run scripts/prepare-kernel.sh available from [1] into [2] (usual procedure)
>>>>   for x86, that would be: .../scripts/prepare-kernel.sh --arch=x86
>>>> - build your kernel using the sources from [2]
>>>>
>>>> There is no user-visible Kconfig change compared to an I-pipe based
>>>> version.
>>>>
>>>> Alternatively, the Xenomai code base in [1] can also run on top of the
>>>> I-pipe. prepare-kernel.sh detects which pipeline flavour is there, and
>>>> prepares the source tree accordingly.
>>>>
>>>> This code is being gradually merged into Xenomai's -next branch, and
>>>> will be at the core of Xenomai 3.2. Testing and feedback appreciated.
>>>>
>>>> Thanks,
>>>>
>>>> [1] https://lab.xenomai.org/xenomai-rpm.git, "for-upstream/dovetail" branch
>>>> [2] https://git.evlproject.org/linux-evl.git, "dovetail/master" branch
>>>>
>>>
>>> First try on my end looked good as well - great news, well done!
>>>
>>> Would it help to stick the wip branch into our CI/CT?
>>>
>> 
>> Yep, it would. TIA,
>> 
>
> CI build is passing now, see
>
> https://gitlab.denx.de/Xenomai/xenomai/-/pipelines/6258
>
> I'm carrying two fixes and a could of tweaks:
>
> https://gitlab.denx.de/Xenomai/xenomai/-/commits/wip/dovetail
>

I'll pick the trace-related fixes in my queue.

> Specifically
> https://gitlab.denx.de/Xenomai/xenomai/-/commit/4196216e821abea3107284a8e31e13bdf05141d5
> would a second look as the timer ipi abstraction became obsolete with
> the code moving to pipeline-specific code.
>

Yes, moving this to the pipeline-specific section, so that we reduce the
level of visual clutter in the generic code.

> I've just started also target runs, see
> https://gitlab.denx.de/Xenomai/xenomai-images/-/pipelines/6263. There
> will be several known breakages, though, namely the gdb test but also
> the unrelated arm[64]-5.4 regressions.
>

I'm fixing the RTnet build issues ATM.

-- 
Philippe.

Re: [Dovetail] x86 test version available (kernel v5.10)

[Jan Kiszka]

On 06.02.21 13:29, Jan Kiszka via Xenomai wrote:
> On 03.02.21 18:35, Philippe Gerum wrote:
>>
>> Jan Kiszka <jan.kiszka@siemens.com> writes:
>>
>>> On 31.01.21 17:06, Philippe Gerum wrote:
>>>>
>>>> The initial port of the Cobalt core to Dovetail/x86 is available from
>>>> [1]. Ports to Dovetail/ARM and Dovetail/arm64 should follow within a
>>>> couple of weeks.
>>>>
>>>> So far, latency and switchtest run flawlessly. Most of the smokey test
>>>> suite passes successfully, except the GDB test at the moment.
>>>>
>>>> How to test this:
>>>>
>>>> - clone Xenomai from [1], switch to branch for-upstream/dovetail
>>>> - clone the Dovetail tree (v5.10) from [2], switch to branch dovetail/master
>>>> - run scripts/prepare-kernel.sh available from [1] into [2] (usual procedure)
>>>>   for x86, that would be: .../scripts/prepare-kernel.sh --arch=x86
>>>> - build your kernel using the sources from [2]
>>>>
>>>> There is no user-visible Kconfig change compared to an I-pipe based
>>>> version.
>>>>
>>>> Alternatively, the Xenomai code base in [1] can also run on top of the
>>>> I-pipe. prepare-kernel.sh detects which pipeline flavour is there, and
>>>> prepares the source tree accordingly.
>>>>
>>>> This code is being gradually merged into Xenomai's -next branch, and
>>>> will be at the core of Xenomai 3.2. Testing and feedback appreciated.
>>>>
>>>> Thanks,
>>>>
>>>> [1] https://lab.xenomai.org/xenomai-rpm.git, "for-upstream/dovetail" branch
>>>> [2] https://git.evlproject.org/linux-evl.git, "dovetail/master" branch
>>>>
>>>
>>> First try on my end looked good as well - great news, well done!
>>>
>>> Would it help to stick the wip branch into our CI/CT?
>>>
>>
>> Yep, it would. TIA,
>>
> 
> CI build is passing now, see
> 
> https://gitlab.denx.de/Xenomai/xenomai/-/pipelines/6258
> 
> I'm carrying two fixes and a could of tweaks:
> 
> https://gitlab.denx.de/Xenomai/xenomai/-/commits/wip/dovetail
> 
> Specifically
> https://gitlab.denx.de/Xenomai/xenomai/-/commit/4196216e821abea3107284a8e31e13bdf05141d5
> would a second look as the timer ipi abstraction became obsolete with
> the code moving to pipeline-specific code.
> 
> I've just started also target runs, see
> https://gitlab.denx.de/Xenomai/xenomai-images/-/pipelines/6263. There
> will be several known breakages, though, namely the gdb test but also
> the unrelated arm[64]-5.4 regressions.
> 

I think I (widely) understood the issue on x86 qemu:

posix-clock.c:285, assertion failed: diff >= 5500000000LL && diff <=
6500000000LL
/usr/lib/xenomai/testsuite/smokey: test posix_clock failed: Invalid argument

It should have been visible on a real machine as well - as long as
systemd-timesyncd is running. Stopping that service makes the test work.

As we are now sharing CLOCK_REALTIME with Linux, any ongoing tweak of
the system time by ntp and similar services will defeat our temporal
set-back in smokey's test case. We can address that in xenomai-images,
but I'm unsure how to resolve that more generically for smokey. Any ideas?

Now the next issue is

/usr/lib/xenomai/testsuite/smokey: test sched_quota failed: Protocol error
sched-quota.c:332, out of quota: 13.7%
(on a real target, an Atom-class box).

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: [Dovetail] x86 test version available (kernel v5.10)

[Jan Kiszka]

On 15.02.21 14:28, Jan Kiszka wrote:
> Now the next issue is
> 
> /usr/lib/xenomai/testsuite/smokey: test sched_quota failed: Protocol error
> sched-quota.c:332, out of quota: 13.7%
> (on a real target, an Atom-class box).
> 

Testsuite issue, long pending:

https://gitlab.denx.de/Xenomai/xenomai/-/blob/master/testsuite/smokey/sched-quota/sched-quota.c#L204

Those pthread_kill() in the loop will kick the caller into non-rt, while
it is not yet done with collecting the loop counters from the other
workers. Result is pure random, sometimes still within the defined
limits, but sometimes reporting false overruns.

No idea what those signal submissions supposed to do - test runs and
terminates fine without them as well.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: [Dovetail] x86 test version available (kernel v5.10)

[Jan Kiszka]

On 15.02.21 16:11, Jan Kiszka via Xenomai wrote:
> On 15.02.21 14:28, Jan Kiszka wrote:
>> Now the next issue is
>>
>> /usr/lib/xenomai/testsuite/smokey: test sched_quota failed: Protocol error
>> sched-quota.c:332, out of quota: 13.7%
>> (on a real target, an Atom-class box).
>>
> 
> Testsuite issue, long pending:
> 
> https://gitlab.denx.de/Xenomai/xenomai/-/blob/master/testsuite/smokey/sched-quota/sched-quota.c#L204
> 
> Those pthread_kill() in the loop will kick the caller into non-rt, while
> it is not yet done with collecting the loop counters from the other
> workers. Result is pure random, sometimes still within the defined
> limits, but sometimes reporting false overruns.
> 
> No idea what those signal submissions supposed to do - test runs and
> terminates fine without them as well.
> 

Correction: The pthread_kill stay in primary and do not cause the issue
(they are still not needed IMHO, though).

It must be something compiler (setting?) related: smokey built with my
gcc-7 from SUSE is fine, the one build with Debian's gcc-8 is not,
causing noteworthy variations on the measured workload. Examining further...

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: [Dovetail] x86 test version available (kernel v5.10)

[Philippe Gerum]

Jan Kiszka <jan.kiszka@siemens.com> writes:

> On 15.02.21 14:28, Jan Kiszka wrote:
>> Now the next issue is
>> 
>> /usr/lib/xenomai/testsuite/smokey: test sched_quota failed: Protocol error
>> sched-quota.c:332, out of quota: 13.7%
>> (on a real target, an Atom-class box).
>> 
>
> Testsuite issue, long pending:
>
> https://gitlab.denx.de/Xenomai/xenomai/-/blob/master/testsuite/smokey/sched-quota/sched-quota.c#L204
>
> Those pthread_kill() in the loop will kick the caller into non-rt, while
> it is not yet done with collecting the loop counters from the other
> workers. Result is pure random, sometimes still within the defined
> limits, but sometimes reporting false overruns.

Make sense, we need to collect the counters independently then.

>
> No idea what those signal submissions supposed to do - test runs and
> terminates fine without them as well.
>

On SMP probably, on UP I don't think so, because if you don't demote the
workers, then the following call to pthread_cancel() will switch the
main context out of primary mode, leaving one of the workers preempt
back, then go spinning full speed, locking the CPU (there is no
condition for exiting the work loop).

Using the throttling hack like the calibration loop does may help (I
guess I overlooked this back then).


-- 
Philippe.

Re: [Dovetail] x86 test version available (kernel v5.10)

[Jan Kiszka]

On 15.02.21 17:53, Jan Kiszka via Xenomai wrote:
> On 15.02.21 16:11, Jan Kiszka via Xenomai wrote:
>> On 15.02.21 14:28, Jan Kiszka wrote:
>>> Now the next issue is
>>>
>>> /usr/lib/xenomai/testsuite/smokey: test sched_quota failed: Protocol error
>>> sched-quota.c:332, out of quota: 13.7%
>>> (on a real target, an Atom-class box).
>>>
>>
>> Testsuite issue, long pending:
>>
>> https://gitlab.denx.de/Xenomai/xenomai/-/blob/master/testsuite/smokey/sched-quota/sched-quota.c#L204
>>
>> Those pthread_kill() in the loop will kick the caller into non-rt, while
>> it is not yet done with collecting the loop counters from the other
>> workers. Result is pure random, sometimes still within the defined
>> limits, but sometimes reporting false overruns.
>>
>> No idea what those signal submissions supposed to do - test runs and
>> terminates fine without them as well.
>>
> 
> Correction: The pthread_kill stay in primary and do not cause the issue
> (they are still not needed IMHO, though).
> 
> It must be something compiler (setting?) related: smokey built with my
> gcc-7 from SUSE is fine, the one build with Debian's gcc-8 is not,
> causing noteworthy variations on the measured workload. Examining further...
> 

It was gcc-10 in fact on my machine - but the actual difference was
--enable-debug=full here vs. no debug in xenomai-images. Without full
debugging, the loop calibration becomes too instable, likely because the
inner __do_work is reduced to a simple

  417960:       48 8d 47 01             lea    0x1(%rdi),%rax
  417964:       c3                      retq

This can be mitigated by extending the calibration loop (crunch_loops =
100000).

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: [Dovetail] x86 test version available (kernel v5.10)

[Philippe Gerum]

Philippe Gerum <rpm@xenomai.org> writes:

> Jan Kiszka <jan.kiszka@siemens.com> writes:
>
>> On 15.02.21 14:28, Jan Kiszka wrote:
>>> Now the next issue is
>>> 
>>> /usr/lib/xenomai/testsuite/smokey: test sched_quota failed: Protocol error
>>> sched-quota.c:332, out of quota: 13.7%
>>> (on a real target, an Atom-class box).
>>> 
>>
>> Testsuite issue, long pending:
>>
>> https://gitlab.denx.de/Xenomai/xenomai/-/blob/master/testsuite/smokey/sched-quota/sched-quota.c#L204
>>
>> Those pthread_kill() in the loop will kick the caller into non-rt, while
>> it is not yet done with collecting the loop counters from the other
>> workers. Result is pure random, sometimes still within the defined
>> limits, but sometimes reporting false overruns.
>
> Make sense, we need to collect the counters independently then.
>

Strike that, all threads are pinned on the same CPU (0), so there is no
way pthread_kill() would cause the worker threads to keep running since
the main context always runs with higher priority.

-- 
Philippe.

Re: [Dovetail] x86 test version available (kernel v5.10)

[Henning Schild]

Am Sun, 31 Jan 2021 17:06:21 +0100
schrieb Philippe Gerum via Xenomai <xenomai@xenomai.org>:

> The initial port of the Cobalt core to Dovetail/x86 is available from
> [1]. Ports to Dovetail/ARM and Dovetail/arm64 should follow within a
> couple of weeks.
> 
> So far, latency and switchtest run flawlessly. Most of the smokey test
> suite passes successfully, except the GDB test at the moment.
> 
> How to test this:
> 
> - clone Xenomai from [1], switch to branch for-upstream/dovetail
> - clone the Dovetail tree (v5.10) from [2], switch to branch
> dovetail/master
> - run scripts/prepare-kernel.sh available from [1] into [2] (usual
> procedure) for x86, that would be: .../scripts/prepare-kernel.sh
> --arch=x86
> - build your kernel using the sources from [2]

I followed this and got

0625b829450dab22b2eea860eef4fb94f64af4fd
61769e49d82a6775f6eb259bdc16c2f84df79505


> There is no user-visible Kconfig change compared to an I-pipe based
> version.

took a 4.19 x86_64 savedefconfig in via olddefconfig

> Alternatively, the Xenomai code base in [1] can also run on top of the
> I-pipe. prepare-kernel.sh detects which pipeline flavour is there, and
> prepares the source tree accordingly.
> 
> This code is being gradually merged into Xenomai's -next branch, and
> will be at the core of Xenomai 3.2. Testing and feedback appreciated.

Now i get a BUG pretty early on during boot. Since my config still
seems off i could not yet enter my rootfs, probably some
drivers/filesystems/raid switches got lost ...

Its a null pointer deref in kmem_cache_alloc, coming somewhere from
mempool_init_node

First guess was that "numa=off" would hide the problem, seems it does.
So my next guess is that even a qemu with numa could reproduce this ...
probably almost on a random config ... as long as it has smp+numa

I do not have more at the moment, but there seem to be issues around
NUMA.

regards,
Henning

> Thanks,
> 
> [1] https://lab.xenomai.org/xenomai-rpm.git, "for-upstream/dovetail"
> branch [2] https://git.evlproject.org/linux-evl.git,
> "dovetail/master" branch
> 

Re: [Dovetail] x86 test version available (kernel v5.10)

[Henning Schild]

Am Mon, 22 Feb 2021 16:35:30 +0100
schrieb Henning Schild via Xenomai <xenomai@xenomai.org>:

> Am Sun, 31 Jan 2021 17:06:21 +0100
> schrieb Philippe Gerum via Xenomai <xenomai@xenomai.org>:
> 
> > The initial port of the Cobalt core to Dovetail/x86 is available
> > from [1]. Ports to Dovetail/ARM and Dovetail/arm64 should follow
> > within a couple of weeks.
> > 
> > So far, latency and switchtest run flawlessly. Most of the smokey
> > test suite passes successfully, except the GDB test at the moment.
> > 
> > How to test this:
> > 
> > - clone Xenomai from [1], switch to branch for-upstream/dovetail
> > - clone the Dovetail tree (v5.10) from [2], switch to branch
> > dovetail/master
> > - run scripts/prepare-kernel.sh available from [1] into [2] (usual
> > procedure) for x86, that would be: .../scripts/prepare-kernel.sh
> > --arch=x86
> > - build your kernel using the sources from [2]  
> 
> I followed this and got
> 
> 0625b829450dab22b2eea860eef4fb94f64af4fd
> 61769e49d82a6775f6eb259bdc16c2f84df79505
> 
> 
> > There is no user-visible Kconfig change compared to an I-pipe based
> > version.  
> 
> took a 4.19 x86_64 savedefconfig in via olddefconfig
> 
> > Alternatively, the Xenomai code base in [1] can also run on top of
> > the I-pipe. prepare-kernel.sh detects which pipeline flavour is
> > there, and prepares the source tree accordingly.
> > 
> > This code is being gradually merged into Xenomai's -next branch, and
> > will be at the core of Xenomai 3.2. Testing and feedback
> > appreciated.  
> 
> Now i get a BUG pretty early on during boot. Since my config still
> seems off i could not yet enter my rootfs, probably some
> drivers/filesystems/raid switches got lost ...
> 
> Its a null pointer deref in kmem_cache_alloc, coming somewhere from
> mempool_init_node
> 
> First guess was that "numa=off" would hide the problem, seems it does.
> So my next guess is that even a qemu with numa could reproduce this
> ... probably almost on a random config ... as long as it has smp+numa

Tried that again, it is not 100% reproducible ... "numa=off" does not
solve the issue.

I did try the dovetail bits of xenomai-images 5.10 together with a
numa-qemu ... that works.

qemu-system-x86_64 -drive
file=./build/tmp/deploy/images/qemu-amd64/demo-image-xenomai-demo-qemu-amd64.ext4.img,discard=unmap,if=none,id=disk,format=raw
-serial mon:stdio -netdev user,id=net -kernel
./build/tmp/deploy/images/qemu-amd64/demo-image-xenomai-demo-qemu-amd64-vmlinuz
-append ' root=/dev/sda vga=0x305' -initrd
./build/tmp/deploy/images/qemu-amd64/demo-image-xenomai-demo-qemu-amd64-initrd.img
-cpu host -smp 4 -enable-kvm -machine q35 -device ide-hd,drive=disk
-device virtio-net-pci,netdev=net -object
memory-backend-ram,id=mem0,size=1G -object
memory-backend-ram,id=mem1,size=1G -numa
node,cpus=0-1,nodeid=0,memdev=mem0 -numa
node,cpus=2-3,nodeid=1,memdev=mem1 -m 2G

> I do not have more at the moment, but there seem to be issues around
> NUMA.

So maybe it is not NUMA, will keep digging.

Henning

> regards,
> Henning
> 
> > Thanks,
> > 
> > [1] https://lab.xenomai.org/xenomai-rpm.git, "for-upstream/dovetail"
> > branch [2] https://git.evlproject.org/linux-evl.git,
> > "dovetail/master" branch
> >   
> 
> 

Re: [Dovetail] x86 test version available (kernel v5.10)

[Henning Schild]

Am Mon, 22 Feb 2021 16:35:30 +0100
schrieb Henning Schild via Xenomai <xenomai@xenomai.org>:

> Am Sun, 31 Jan 2021 17:06:21 +0100
> schrieb Philippe Gerum via Xenomai <xenomai@xenomai.org>:
> 
> > The initial port of the Cobalt core to Dovetail/x86 is available
> > from [1]. Ports to Dovetail/ARM and Dovetail/arm64 should follow
> > within a couple of weeks.
> > 
> > So far, latency and switchtest run flawlessly. Most of the smokey
> > test suite passes successfully, except the GDB test at the moment.
> > 
> > How to test this:
> > 
> > - clone Xenomai from [1], switch to branch for-upstream/dovetail
> > - clone the Dovetail tree (v5.10) from [2], switch to branch
> > dovetail/master
> > - run scripts/prepare-kernel.sh available from [1] into [2] (usual
> > procedure) for x86, that would be: .../scripts/prepare-kernel.sh
> > --arch=x86
> > - build your kernel using the sources from [2]  
> 
> I followed this and got
> 
> 0625b829450dab22b2eea860eef4fb94f64af4fd
> 61769e49d82a6775f6eb259bdc16c2f84df79505
> 
> 
> > There is no user-visible Kconfig change compared to an I-pipe based
> > version.  
> 
> took a 4.19 x86_64 savedefconfig in via olddefconfig
> 
> > Alternatively, the Xenomai code base in [1] can also run on top of
> > the I-pipe. prepare-kernel.sh detects which pipeline flavour is
> > there, and prepares the source tree accordingly.
> > 
> > This code is being gradually merged into Xenomai's -next branch, and
> > will be at the core of Xenomai 3.2. Testing and feedback
> > appreciated.  
> 
> Now i get a BUG pretty early on during boot. Since my config still
> seems off i could not yet enter my rootfs, probably some
> drivers/filesystems/raid switches got lost ...
> 
> Its a null pointer deref in kmem_cache_alloc, coming somewhere from
> mempool_init_node
> 
> First guess was that "numa=off" would hide the problem, seems it does.
> So my next guess is that even a qemu with numa could reproduce this
> ... probably almost on a random config ... as long as it has smp+numa

It is not about numa, now also see that on qemu. Jan looks into it at
the moment.

Henning

> I do not have more at the moment, but there seem to be issues around
> NUMA.
> 
> regards,
> Henning
> 
> > Thanks,
> > 
> > [1] https://lab.xenomai.org/xenomai-rpm.git, "for-upstream/dovetail"
> > branch [2] https://git.evlproject.org/linux-evl.git,
> > "dovetail/master" branch
> >   
> 
> 

Re: [Dovetail] x86 test version available (kernel v5.10)

[Jan Kiszka]

On 11.03.21 17:35, Henning Schild wrote:
> Am Mon, 22 Feb 2021 16:35:30 +0100
> schrieb Henning Schild via Xenomai <xenomai@xenomai.org>:
> 
>> Am Sun, 31 Jan 2021 17:06:21 +0100
>> schrieb Philippe Gerum via Xenomai <xenomai@xenomai.org>:
>>
>>> The initial port of the Cobalt core to Dovetail/x86 is available
>>> from [1]. Ports to Dovetail/ARM and Dovetail/arm64 should follow
>>> within a couple of weeks.
>>>
>>> So far, latency and switchtest run flawlessly. Most of the smokey
>>> test suite passes successfully, except the GDB test at the moment.
>>>
>>> How to test this:
>>>
>>> - clone Xenomai from [1], switch to branch for-upstream/dovetail
>>> - clone the Dovetail tree (v5.10) from [2], switch to branch
>>> dovetail/master
>>> - run scripts/prepare-kernel.sh available from [1] into [2] (usual
>>> procedure) for x86, that would be: .../scripts/prepare-kernel.sh
>>> --arch=x86
>>> - build your kernel using the sources from [2]  
>>
>> I followed this and got
>>
>> 0625b829450dab22b2eea860eef4fb94f64af4fd
>> 61769e49d82a6775f6eb259bdc16c2f84df79505
>>
>>
>>> There is no user-visible Kconfig change compared to an I-pipe based
>>> version.  
>>
>> took a 4.19 x86_64 savedefconfig in via olddefconfig
>>
>>> Alternatively, the Xenomai code base in [1] can also run on top of
>>> the I-pipe. prepare-kernel.sh detects which pipeline flavour is
>>> there, and prepares the source tree accordingly.
>>>
>>> This code is being gradually merged into Xenomai's -next branch, and
>>> will be at the core of Xenomai 3.2. Testing and feedback
>>> appreciated.  
>>
>> Now i get a BUG pretty early on during boot. Since my config still
>> seems off i could not yet enter my rootfs, probably some
>> drivers/filesystems/raid switches got lost ...
>>
>> Its a null pointer deref in kmem_cache_alloc, coming somewhere from
>> mempool_init_node
>>
>> First guess was that "numa=off" would hide the problem, seems it does.
>> So my next guess is that even a qemu with numa could reproduce this
>> ... probably almost on a random config ... as long as it has smp+numa
> 
> It is not about numa, now also see that on qemu. Jan looks into it at
> the moment.
> 

I'm suspecting an upstream issue. The bug is some freelist corruption,
possibly only surfaced by dovetail. After digging longer, I enabled
CONFIG_SLAB_FREELIST_HARDENED, and that gave

[    0.159611] ACPI Error: Could not remove SCI handler
(20210105/evmisc-251)
[    0.160188] ------------[ cut here ]------------
[    0.160566] cache_from_obj: Wrong slab cache. ftrace_event_field but
object is from kmalloc-64

and more. However: You get those bugs also from running latest 5.10.23
and even Linus' master. Will debug that further and then possibly report
upstream.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: [Dovetail] x86 test version available (kernel v5.10)

[Jan Kiszka]

On 12.03.21 08:22, Jan Kiszka wrote:
> On 11.03.21 17:35, Henning Schild wrote:
>> Am Mon, 22 Feb 2021 16:35:30 +0100
>> schrieb Henning Schild via Xenomai <xenomai@xenomai.org>:
>>
>>> Am Sun, 31 Jan 2021 17:06:21 +0100
>>> schrieb Philippe Gerum via Xenomai <xenomai@xenomai.org>:
>>>
>>>> The initial port of the Cobalt core to Dovetail/x86 is available
>>>> from [1]. Ports to Dovetail/ARM and Dovetail/arm64 should follow
>>>> within a couple of weeks.
>>>>
>>>> So far, latency and switchtest run flawlessly. Most of the smokey
>>>> test suite passes successfully, except the GDB test at the moment.
>>>>
>>>> How to test this:
>>>>
>>>> - clone Xenomai from [1], switch to branch for-upstream/dovetail
>>>> - clone the Dovetail tree (v5.10) from [2], switch to branch
>>>> dovetail/master
>>>> - run scripts/prepare-kernel.sh available from [1] into [2] (usual
>>>> procedure) for x86, that would be: .../scripts/prepare-kernel.sh
>>>> --arch=x86
>>>> - build your kernel using the sources from [2]  
>>>
>>> I followed this and got
>>>
>>> 0625b829450dab22b2eea860eef4fb94f64af4fd
>>> 61769e49d82a6775f6eb259bdc16c2f84df79505
>>>
>>>
>>>> There is no user-visible Kconfig change compared to an I-pipe based
>>>> version.  
>>>
>>> took a 4.19 x86_64 savedefconfig in via olddefconfig
>>>
>>>> Alternatively, the Xenomai code base in [1] can also run on top of
>>>> the I-pipe. prepare-kernel.sh detects which pipeline flavour is
>>>> there, and prepares the source tree accordingly.
>>>>
>>>> This code is being gradually merged into Xenomai's -next branch, and
>>>> will be at the core of Xenomai 3.2. Testing and feedback
>>>> appreciated.  
>>>
>>> Now i get a BUG pretty early on during boot. Since my config still
>>> seems off i could not yet enter my rootfs, probably some
>>> drivers/filesystems/raid switches got lost ...
>>>
>>> Its a null pointer deref in kmem_cache_alloc, coming somewhere from
>>> mempool_init_node
>>>
>>> First guess was that "numa=off" would hide the problem, seems it does.
>>> So my next guess is that even a qemu with numa could reproduce this
>>> ... probably almost on a random config ... as long as it has smp+numa
>>
>> It is not about numa, now also see that on qemu. Jan looks into it at
>> the moment.
>>
> 
> I'm suspecting an upstream issue. The bug is some freelist corruption,
> possibly only surfaced by dovetail. After digging longer, I enabled
> CONFIG_SLAB_FREELIST_HARDENED, and that gave
> 
> [    0.159611] ACPI Error: Could not remove SCI handler
> (20210105/evmisc-251)
> [    0.160188] ------------[ cut here ]------------
> [    0.160566] cache_from_obj: Wrong slab cache. ftrace_event_field but
> object is from kmalloc-64
> 
> and more. However: You get those bugs also from running latest 5.10.23
> and even Linus' master. Will debug that further and then possibly report
> upstream.

This is a corner case of upstream (but still a bug there):

Your config lost CONFIG_PCI, likely because it is no longer default y.
That not only generates and unbootable systems, it also sends the ACPI
subsystem into an error path. There it seems to release objects to the
wrong caches, leading to corruptions later on. Enabling PCI resolves all
this.

I'll dump the config to upstream ACPI folks, should be their business.

Jan

PS: Your config has more issues as it does not even boot in QEMU, even
with PCI enabled. You should (re-)derive from x86_64-defconfig.

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: [Dovetail] x86 test version available (kernel v5.10)

[Henning Schild]

Am Fri, 12 Mar 2021 12:32:54 +0100
schrieb Jan Kiszka <jan.kiszka@siemens.com>:

> On 12.03.21 08:22, Jan Kiszka wrote:
> > On 11.03.21 17:35, Henning Schild wrote:  
> >> Am Mon, 22 Feb 2021 16:35:30 +0100
> >> schrieb Henning Schild via Xenomai <xenomai@xenomai.org>:
> >>  
> >>> Am Sun, 31 Jan 2021 17:06:21 +0100
> >>> schrieb Philippe Gerum via Xenomai <xenomai@xenomai.org>:
> >>>  
> >>>> The initial port of the Cobalt core to Dovetail/x86 is available
> >>>> from [1]. Ports to Dovetail/ARM and Dovetail/arm64 should follow
> >>>> within a couple of weeks.
> >>>>
> >>>> So far, latency and switchtest run flawlessly. Most of the smokey
> >>>> test suite passes successfully, except the GDB test at the
> >>>> moment.
> >>>>
> >>>> How to test this:
> >>>>
> >>>> - clone Xenomai from [1], switch to branch for-upstream/dovetail
> >>>> - clone the Dovetail tree (v5.10) from [2], switch to branch
> >>>> dovetail/master
> >>>> - run scripts/prepare-kernel.sh available from [1] into [2]
> >>>> (usual procedure) for x86, that would be:
> >>>> .../scripts/prepare-kernel.sh --arch=x86
> >>>> - build your kernel using the sources from [2]    
> >>>
> >>> I followed this and got
> >>>
> >>> 0625b829450dab22b2eea860eef4fb94f64af4fd
> >>> 61769e49d82a6775f6eb259bdc16c2f84df79505
> >>>
> >>>  
> >>>> There is no user-visible Kconfig change compared to an I-pipe
> >>>> based version.    
> >>>
> >>> took a 4.19 x86_64 savedefconfig in via olddefconfig
> >>>  
> >>>> Alternatively, the Xenomai code base in [1] can also run on top
> >>>> of the I-pipe. prepare-kernel.sh detects which pipeline flavour
> >>>> is there, and prepares the source tree accordingly.
> >>>>
> >>>> This code is being gradually merged into Xenomai's -next branch,
> >>>> and will be at the core of Xenomai 3.2. Testing and feedback
> >>>> appreciated.    
> >>>
> >>> Now i get a BUG pretty early on during boot. Since my config still
> >>> seems off i could not yet enter my rootfs, probably some
> >>> drivers/filesystems/raid switches got lost ...
> >>>
> >>> Its a null pointer deref in kmem_cache_alloc, coming somewhere
> >>> from mempool_init_node
> >>>
> >>> First guess was that "numa=off" would hide the problem, seems it
> >>> does. So my next guess is that even a qemu with numa could
> >>> reproduce this ... probably almost on a random config ... as long
> >>> as it has smp+numa  
> >>
> >> It is not about numa, now also see that on qemu. Jan looks into it
> >> at the moment.
> >>  
> > 
> > I'm suspecting an upstream issue. The bug is some freelist
> > corruption, possibly only surfaced by dovetail. After digging
> > longer, I enabled CONFIG_SLAB_FREELIST_HARDENED, and that gave
> > 
> > [    0.159611] ACPI Error: Could not remove SCI handler
> > (20210105/evmisc-251)
> > [    0.160188] ------------[ cut here ]------------
> > [    0.160566] cache_from_obj: Wrong slab cache. ftrace_event_field
> > but object is from kmalloc-64
> > 
> > and more. However: You get those bugs also from running latest
> > 5.10.23 and even Linus' master. Will debug that further and then
> > possibly report upstream.  
> 
> This is a corner case of upstream (but still a bug there):
> 
> Your config lost CONFIG_PCI, likely because it is no longer default y.
> That not only generates and unbootable systems, it also sends the ACPI
> subsystem into an error path. There it seems to release objects to the
> wrong caches, leading to corruptions later on. Enabling PCI resolves
> all this.
> 
> I'll dump the config to upstream ACPI folks, should be their business.

Thanks Jan!

> Jan
> 
> PS: Your config has more issues as it does not even boot in QEMU, even
> with PCI enabled. You should (re-)derive from x86_64-defconfig.

Yes that was CI progressing with "olddefconfig" from a 4.19
"savedefconfig". No human took a closer look at the result, before you.
But hey that might fix an upstream bug so it was a good exercise
anyhow.

Henning 

Re: [Dovetail] x86 test version available (kernel v5.10)

[Henning Schild]

Am Fri, 12 Mar 2021 19:18:14 +0100
schrieb Henning Schild via Xenomai <xenomai@xenomai.org>:

> Am Fri, 12 Mar 2021 12:32:54 +0100
> schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> 
> > On 12.03.21 08:22, Jan Kiszka wrote:  
> > > On 11.03.21 17:35, Henning Schild wrote:    
> > >> Am Mon, 22 Feb 2021 16:35:30 +0100
> > >> schrieb Henning Schild via Xenomai <xenomai@xenomai.org>:
> > >>    
> > >>> Am Sun, 31 Jan 2021 17:06:21 +0100
> > >>> schrieb Philippe Gerum via Xenomai <xenomai@xenomai.org>:
> > >>>    
> > >>>> The initial port of the Cobalt core to Dovetail/x86 is
> > >>>> available from [1]. Ports to Dovetail/ARM and Dovetail/arm64
> > >>>> should follow within a couple of weeks.
> > >>>>
> > >>>> So far, latency and switchtest run flawlessly. Most of the
> > >>>> smokey test suite passes successfully, except the GDB test at
> > >>>> the moment.
> > >>>>
> > >>>> How to test this:
> > >>>>
> > >>>> - clone Xenomai from [1], switch to branch
> > >>>> for-upstream/dovetail
> > >>>> - clone the Dovetail tree (v5.10) from [2], switch to branch
> > >>>> dovetail/master
> > >>>> - run scripts/prepare-kernel.sh available from [1] into [2]
> > >>>> (usual procedure) for x86, that would be:
> > >>>> .../scripts/prepare-kernel.sh --arch=x86
> > >>>> - build your kernel using the sources from [2]      
> > >>>
> > >>> I followed this and got
> > >>>
> > >>> 0625b829450dab22b2eea860eef4fb94f64af4fd
> > >>> 61769e49d82a6775f6eb259bdc16c2f84df79505
> > >>>
> > >>>    
> > >>>> There is no user-visible Kconfig change compared to an I-pipe
> > >>>> based version.      
> > >>>
> > >>> took a 4.19 x86_64 savedefconfig in via olddefconfig
> > >>>    
> > >>>> Alternatively, the Xenomai code base in [1] can also run on top
> > >>>> of the I-pipe. prepare-kernel.sh detects which pipeline flavour
> > >>>> is there, and prepares the source tree accordingly.
> > >>>>
> > >>>> This code is being gradually merged into Xenomai's -next
> > >>>> branch, and will be at the core of Xenomai 3.2. Testing and
> > >>>> feedback appreciated.      
> > >>>
> > >>> Now i get a BUG pretty early on during boot. Since my config
> > >>> still seems off i could not yet enter my rootfs, probably some
> > >>> drivers/filesystems/raid switches got lost ...
> > >>>
> > >>> Its a null pointer deref in kmem_cache_alloc, coming somewhere
> > >>> from mempool_init_node
> > >>>
> > >>> First guess was that "numa=off" would hide the problem, seems it
> > >>> does. So my next guess is that even a qemu with numa could
> > >>> reproduce this ... probably almost on a random config ... as
> > >>> long as it has smp+numa    
> > >>
> > >> It is not about numa, now also see that on qemu. Jan looks into
> > >> it at the moment.
> > >>    
> > > 
> > > I'm suspecting an upstream issue. The bug is some freelist
> > > corruption, possibly only surfaced by dovetail. After digging
> > > longer, I enabled CONFIG_SLAB_FREELIST_HARDENED, and that gave
> > > 
> > > [    0.159611] ACPI Error: Could not remove SCI handler
> > > (20210105/evmisc-251)
> > > [    0.160188] ------------[ cut here ]------------
> > > [    0.160566] cache_from_obj: Wrong slab cache.
> > > ftrace_event_field but object is from kmalloc-64
> > > 
> > > and more. However: You get those bugs also from running latest
> > > 5.10.23 and even Linus' master. Will debug that further and then
> > > possibly report upstream.    
> > 
> > This is a corner case of upstream (but still a bug there):
> > 
> > Your config lost CONFIG_PCI, likely because it is no longer default
> > y. That not only generates and unbootable systems, it also sends
> > the ACPI subsystem into an error path. There it seems to release
> > objects to the wrong caches, leading to corruptions later on.
> > Enabling PCI resolves all this.
> > 
> > I'll dump the config to upstream ACPI folks, should be their
> > business.  
> 
> Thanks Jan!
> 
> > Jan
> > 
> > PS: Your config has more issues as it does not even boot in QEMU,
> > even with PCI enabled. You should (re-)derive from
> > x86_64-defconfig.  
> 
> Yes that was CI progressing with "olddefconfig" from a 4.19
> "savedefconfig". No human took a closer look at the result, before
> you. But hey that might fix an upstream bug so it was a good exercise
> anyhow.

Looking much better with a changed config. AFAIK the gdb test from
smokey is known to fail at the moment. Should that be marked as "skip"
mainline to reduce downstream skipping fixups?

Henning

> Henning 
> 
> 

Re: [Dovetail] x86 test version available (kernel v5.10)

[Henning Schild]

Am Fri, 12 Mar 2021 20:46:30 +0100
schrieb Henning Schild <henning.schild@siemens.com>:

> Am Fri, 12 Mar 2021 19:18:14 +0100
> schrieb Henning Schild via Xenomai <xenomai@xenomai.org>:
> 
> > Am Fri, 12 Mar 2021 12:32:54 +0100
> > schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> >   
> > > On 12.03.21 08:22, Jan Kiszka wrote:    
> > > > On 11.03.21 17:35, Henning Schild wrote:      
> > > >> Am Mon, 22 Feb 2021 16:35:30 +0100
> > > >> schrieb Henning Schild via Xenomai <xenomai@xenomai.org>:
> > > >>      
> > > >>> Am Sun, 31 Jan 2021 17:06:21 +0100
> > > >>> schrieb Philippe Gerum via Xenomai <xenomai@xenomai.org>:
> > > >>>      
> > > >>>> The initial port of the Cobalt core to Dovetail/x86 is
> > > >>>> available from [1]. Ports to Dovetail/ARM and Dovetail/arm64
> > > >>>> should follow within a couple of weeks.
> > > >>>>
> > > >>>> So far, latency and switchtest run flawlessly. Most of the
> > > >>>> smokey test suite passes successfully, except the GDB test at
> > > >>>> the moment.
> > > >>>>
> > > >>>> How to test this:
> > > >>>>
> > > >>>> - clone Xenomai from [1], switch to branch
> > > >>>> for-upstream/dovetail
> > > >>>> - clone the Dovetail tree (v5.10) from [2], switch to branch
> > > >>>> dovetail/master
> > > >>>> - run scripts/prepare-kernel.sh available from [1] into [2]
> > > >>>> (usual procedure) for x86, that would be:
> > > >>>> .../scripts/prepare-kernel.sh --arch=x86
> > > >>>> - build your kernel using the sources from [2]        
> > > >>>
> > > >>> I followed this and got
> > > >>>
> > > >>> 0625b829450dab22b2eea860eef4fb94f64af4fd
> > > >>> 61769e49d82a6775f6eb259bdc16c2f84df79505
> > > >>>
> > > >>>      
> > > >>>> There is no user-visible Kconfig change compared to an I-pipe
> > > >>>> based version.        
> > > >>>
> > > >>> took a 4.19 x86_64 savedefconfig in via olddefconfig
> > > >>>      
> > > >>>> Alternatively, the Xenomai code base in [1] can also run on
> > > >>>> top of the I-pipe. prepare-kernel.sh detects which pipeline
> > > >>>> flavour is there, and prepares the source tree accordingly.
> > > >>>>
> > > >>>> This code is being gradually merged into Xenomai's -next
> > > >>>> branch, and will be at the core of Xenomai 3.2. Testing and
> > > >>>> feedback appreciated.        
> > > >>>
> > > >>> Now i get a BUG pretty early on during boot. Since my config
> > > >>> still seems off i could not yet enter my rootfs, probably some
> > > >>> drivers/filesystems/raid switches got lost ...
> > > >>>
> > > >>> Its a null pointer deref in kmem_cache_alloc, coming somewhere
> > > >>> from mempool_init_node
> > > >>>
> > > >>> First guess was that "numa=off" would hide the problem, seems
> > > >>> it does. So my next guess is that even a qemu with numa could
> > > >>> reproduce this ... probably almost on a random config ... as
> > > >>> long as it has smp+numa      
> > > >>
> > > >> It is not about numa, now also see that on qemu. Jan looks into
> > > >> it at the moment.
> > > >>      
> > > > 
> > > > I'm suspecting an upstream issue. The bug is some freelist
> > > > corruption, possibly only surfaced by dovetail. After digging
> > > > longer, I enabled CONFIG_SLAB_FREELIST_HARDENED, and that gave
> > > > 
> > > > [    0.159611] ACPI Error: Could not remove SCI handler
> > > > (20210105/evmisc-251)
> > > > [    0.160188] ------------[ cut here ]------------
> > > > [    0.160566] cache_from_obj: Wrong slab cache.
> > > > ftrace_event_field but object is from kmalloc-64
> > > > 
> > > > and more. However: You get those bugs also from running latest
> > > > 5.10.23 and even Linus' master. Will debug that further and then
> > > > possibly report upstream.      
> > > 
> > > This is a corner case of upstream (but still a bug there):
> > > 
> > > Your config lost CONFIG_PCI, likely because it is no longer
> > > default y. That not only generates and unbootable systems, it
> > > also sends the ACPI subsystem into an error path. There it seems
> > > to release objects to the wrong caches, leading to corruptions
> > > later on. Enabling PCI resolves all this.
> > > 
> > > I'll dump the config to upstream ACPI folks, should be their
> > > business.    
> > 
> > Thanks Jan!
> >   
> > > Jan
> > > 
> > > PS: Your config has more issues as it does not even boot in QEMU,
> > > even with PCI enabled. You should (re-)derive from
> > > x86_64-defconfig.    
> > 
> > Yes that was CI progressing with "olddefconfig" from a 4.19
> > "savedefconfig". No human took a closer look at the result, before
> > you. But hey that might fix an upstream bug so it was a good
> > exercise anyhow.  
> 
> Looking much better with a changed config. AFAIK the gdb test from
> smokey is known to fail at the moment. Should that be marked as "skip"
> mainline to reduce downstream skipping fixups?

Related to that ...

smokey --run --keep-going

does not keep-going on gdb, one needs

smokey --run --exclude gdb --keep-going

Henning

> Henning
> 
> > Henning 
> > 
> >   
> 

Re: [Dovetail] x86 test version available (kernel v5.10)

[Jan Kiszka]

On 12.03.21 20:53, Henning Schild wrote:
> Am Fri, 12 Mar 2021 20:46:30 +0100
> schrieb Henning Schild <henning.schild@siemens.com>:
> 
>> Am Fri, 12 Mar 2021 19:18:14 +0100
>> schrieb Henning Schild via Xenomai <xenomai@xenomai.org>:
>>
>>> Am Fri, 12 Mar 2021 12:32:54 +0100
>>> schrieb Jan Kiszka <jan.kiszka@siemens.com>:
>>>   
>>>> On 12.03.21 08:22, Jan Kiszka wrote:    
>>>>> On 11.03.21 17:35, Henning Schild wrote:      
>>>>>> Am Mon, 22 Feb 2021 16:35:30 +0100
>>>>>> schrieb Henning Schild via Xenomai <xenomai@xenomai.org>:
>>>>>>      
>>>>>>> Am Sun, 31 Jan 2021 17:06:21 +0100
>>>>>>> schrieb Philippe Gerum via Xenomai <xenomai@xenomai.org>:
>>>>>>>      
>>>>>>>> The initial port of the Cobalt core to Dovetail/x86 is
>>>>>>>> available from [1]. Ports to Dovetail/ARM and Dovetail/arm64
>>>>>>>> should follow within a couple of weeks.
>>>>>>>>
>>>>>>>> So far, latency and switchtest run flawlessly. Most of the
>>>>>>>> smokey test suite passes successfully, except the GDB test at
>>>>>>>> the moment.
>>>>>>>>
>>>>>>>> How to test this:
>>>>>>>>
>>>>>>>> - clone Xenomai from [1], switch to branch
>>>>>>>> for-upstream/dovetail
>>>>>>>> - clone the Dovetail tree (v5.10) from [2], switch to branch
>>>>>>>> dovetail/master
>>>>>>>> - run scripts/prepare-kernel.sh available from [1] into [2]
>>>>>>>> (usual procedure) for x86, that would be:
>>>>>>>> .../scripts/prepare-kernel.sh --arch=x86
>>>>>>>> - build your kernel using the sources from [2]        
>>>>>>>
>>>>>>> I followed this and got
>>>>>>>
>>>>>>> 0625b829450dab22b2eea860eef4fb94f64af4fd
>>>>>>> 61769e49d82a6775f6eb259bdc16c2f84df79505
>>>>>>>
>>>>>>>      
>>>>>>>> There is no user-visible Kconfig change compared to an I-pipe
>>>>>>>> based version.        
>>>>>>>
>>>>>>> took a 4.19 x86_64 savedefconfig in via olddefconfig
>>>>>>>      
>>>>>>>> Alternatively, the Xenomai code base in [1] can also run on
>>>>>>>> top of the I-pipe. prepare-kernel.sh detects which pipeline
>>>>>>>> flavour is there, and prepares the source tree accordingly.
>>>>>>>>
>>>>>>>> This code is being gradually merged into Xenomai's -next
>>>>>>>> branch, and will be at the core of Xenomai 3.2. Testing and
>>>>>>>> feedback appreciated.        
>>>>>>>
>>>>>>> Now i get a BUG pretty early on during boot. Since my config
>>>>>>> still seems off i could not yet enter my rootfs, probably some
>>>>>>> drivers/filesystems/raid switches got lost ...
>>>>>>>
>>>>>>> Its a null pointer deref in kmem_cache_alloc, coming somewhere
>>>>>>> from mempool_init_node
>>>>>>>
>>>>>>> First guess was that "numa=off" would hide the problem, seems
>>>>>>> it does. So my next guess is that even a qemu with numa could
>>>>>>> reproduce this ... probably almost on a random config ... as
>>>>>>> long as it has smp+numa      
>>>>>>
>>>>>> It is not about numa, now also see that on qemu. Jan looks into
>>>>>> it at the moment.
>>>>>>      
>>>>>
>>>>> I'm suspecting an upstream issue. The bug is some freelist
>>>>> corruption, possibly only surfaced by dovetail. After digging
>>>>> longer, I enabled CONFIG_SLAB_FREELIST_HARDENED, and that gave
>>>>>
>>>>> [    0.159611] ACPI Error: Could not remove SCI handler
>>>>> (20210105/evmisc-251)
>>>>> [    0.160188] ------------[ cut here ]------------
>>>>> [    0.160566] cache_from_obj: Wrong slab cache.
>>>>> ftrace_event_field but object is from kmalloc-64
>>>>>
>>>>> and more. However: You get those bugs also from running latest
>>>>> 5.10.23 and even Linus' master. Will debug that further and then
>>>>> possibly report upstream.      
>>>>
>>>> This is a corner case of upstream (but still a bug there):
>>>>
>>>> Your config lost CONFIG_PCI, likely because it is no longer
>>>> default y. That not only generates and unbootable systems, it
>>>> also sends the ACPI subsystem into an error path. There it seems
>>>> to release objects to the wrong caches, leading to corruptions
>>>> later on. Enabling PCI resolves all this.
>>>>
>>>> I'll dump the config to upstream ACPI folks, should be their
>>>> business.    
>>>
>>> Thanks Jan!
>>>   
>>>> Jan
>>>>
>>>> PS: Your config has more issues as it does not even boot in QEMU,
>>>> even with PCI enabled. You should (re-)derive from
>>>> x86_64-defconfig.    
>>>
>>> Yes that was CI progressing with "olddefconfig" from a 4.19
>>> "savedefconfig". No human took a closer look at the result, before
>>> you. But hey that might fix an upstream bug so it was a good
>>> exercise anyhow.  
>>
>> Looking much better with a changed config. AFAIK the gdb test from
>> smokey is known to fail at the moment. Should that be marked as "skip"
>> mainline to reduce downstream skipping fixups?
> 
> Related to that ...
> 
> smokey --run --keep-going
> 
> does not keep-going on gdb, one needs
> 
> smokey --run --exclude gdb --keep-going
> 

--keep-going might be valuable on its own, the get a more complete
picture. Excluding gdb test for now would be another hack on top of
wip/dovetail, yes.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux