* [Dovetail] x86 test version available (kernel v5.10) @ 2021-01-31 16:06 Philippe Gerum 2021-02-01 17:32 ` Jan Kiszka 2021-02-22 15:35 ` Henning Schild 0 siblings, 2 replies; 20+ messages in thread From: Philippe Gerum @ 2021-01-31 16:06 UTC (permalink / raw) To: xenomai The initial port of the Cobalt core to Dovetail/x86 is available from [1]. Ports to Dovetail/ARM and Dovetail/arm64 should follow within a couple of weeks. So far, latency and switchtest run flawlessly. Most of the smokey test suite passes successfully, except the GDB test at the moment. How to test this: - clone Xenomai from [1], switch to branch for-upstream/dovetail - clone the Dovetail tree (v5.10) from [2], switch to branch dovetail/master - run scripts/prepare-kernel.sh available from [1] into [2] (usual procedure) for x86, that would be: .../scripts/prepare-kernel.sh --arch=x86 - build your kernel using the sources from [2] There is no user-visible Kconfig change compared to an I-pipe based version. Alternatively, the Xenomai code base in [1] can also run on top of the I-pipe. prepare-kernel.sh detects which pipeline flavour is there, and prepares the source tree accordingly. This code is being gradually merged into Xenomai's -next branch, and will be at the core of Xenomai 3.2. Testing and feedback appreciated. Thanks, [1] https://lab.xenomai.org/xenomai-rpm.git, "for-upstream/dovetail" branch [2] https://git.evlproject.org/linux-evl.git, "dovetail/master" branch -- Philippe. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Dovetail] x86 test version available (kernel v5.10) 2021-01-31 16:06 [Dovetail] x86 test version available (kernel v5.10) Philippe Gerum @ 2021-02-01 17:32 ` Jan Kiszka 2021-02-03 17:35 ` Philippe Gerum 2021-02-22 15:35 ` Henning Schild 1 sibling, 1 reply; 20+ messages in thread From: Jan Kiszka @ 2021-02-01 17:32 UTC (permalink / raw) To: Philippe Gerum, xenomai, hongzha1 On 31.01.21 17:06, Philippe Gerum wrote: > > The initial port of the Cobalt core to Dovetail/x86 is available from > [1]. Ports to Dovetail/ARM and Dovetail/arm64 should follow within a > couple of weeks. > > So far, latency and switchtest run flawlessly. Most of the smokey test > suite passes successfully, except the GDB test at the moment. > > How to test this: > > - clone Xenomai from [1], switch to branch for-upstream/dovetail > - clone the Dovetail tree (v5.10) from [2], switch to branch dovetail/master > - run scripts/prepare-kernel.sh available from [1] into [2] (usual procedure) > for x86, that would be: .../scripts/prepare-kernel.sh --arch=x86 > - build your kernel using the sources from [2] > > There is no user-visible Kconfig change compared to an I-pipe based > version. > > Alternatively, the Xenomai code base in [1] can also run on top of the > I-pipe. prepare-kernel.sh detects which pipeline flavour is there, and > prepares the source tree accordingly. > > This code is being gradually merged into Xenomai's -next branch, and > will be at the core of Xenomai 3.2. Testing and feedback appreciated. > > Thanks, > > [1] https://lab.xenomai.org/xenomai-rpm.git, "for-upstream/dovetail" branch > [2] https://git.evlproject.org/linux-evl.git, "dovetail/master" branch > First try on my end looked good as well - great news, well done! Would it help to stick the wip branch into our CI/CT? Will have a look at the next round of pending patches. Jan -- Siemens AG, T RDA IOT Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Dovetail] x86 test version available (kernel v5.10) 2021-02-01 17:32 ` Jan Kiszka @ 2021-02-03 17:35 ` Philippe Gerum 2021-02-06 12:29 ` Jan Kiszka 0 siblings, 1 reply; 20+ messages in thread From: Philippe Gerum @ 2021-02-03 17:35 UTC (permalink / raw) To: Jan Kiszka; +Cc: xenomai, hongzha1 Jan Kiszka <jan.kiszka@siemens.com> writes: > On 31.01.21 17:06, Philippe Gerum wrote: >> >> The initial port of the Cobalt core to Dovetail/x86 is available from >> [1]. Ports to Dovetail/ARM and Dovetail/arm64 should follow within a >> couple of weeks. >> >> So far, latency and switchtest run flawlessly. Most of the smokey test >> suite passes successfully, except the GDB test at the moment. >> >> How to test this: >> >> - clone Xenomai from [1], switch to branch for-upstream/dovetail >> - clone the Dovetail tree (v5.10) from [2], switch to branch dovetail/master >> - run scripts/prepare-kernel.sh available from [1] into [2] (usual procedure) >> for x86, that would be: .../scripts/prepare-kernel.sh --arch=x86 >> - build your kernel using the sources from [2] >> >> There is no user-visible Kconfig change compared to an I-pipe based >> version. >> >> Alternatively, the Xenomai code base in [1] can also run on top of the >> I-pipe. prepare-kernel.sh detects which pipeline flavour is there, and >> prepares the source tree accordingly. >> >> This code is being gradually merged into Xenomai's -next branch, and >> will be at the core of Xenomai 3.2. Testing and feedback appreciated. >> >> Thanks, >> >> [1] https://lab.xenomai.org/xenomai-rpm.git, "for-upstream/dovetail" branch >> [2] https://git.evlproject.org/linux-evl.git, "dovetail/master" branch >> > > First try on my end looked good as well - great news, well done! > > Would it help to stick the wip branch into our CI/CT? > Yep, it would. TIA, -- Philippe. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Dovetail] x86 test version available (kernel v5.10) 2021-02-03 17:35 ` Philippe Gerum @ 2021-02-06 12:29 ` Jan Kiszka 2021-02-06 17:23 ` Philippe Gerum 2021-02-15 13:28 ` Jan Kiszka 0 siblings, 2 replies; 20+ messages in thread From: Jan Kiszka @ 2021-02-06 12:29 UTC (permalink / raw) To: Philippe Gerum; +Cc: xenomai, hongzha1 On 03.02.21 18:35, Philippe Gerum wrote: > > Jan Kiszka <jan.kiszka@siemens.com> writes: > >> On 31.01.21 17:06, Philippe Gerum wrote: >>> >>> The initial port of the Cobalt core to Dovetail/x86 is available from >>> [1]. Ports to Dovetail/ARM and Dovetail/arm64 should follow within a >>> couple of weeks. >>> >>> So far, latency and switchtest run flawlessly. Most of the smokey test >>> suite passes successfully, except the GDB test at the moment. >>> >>> How to test this: >>> >>> - clone Xenomai from [1], switch to branch for-upstream/dovetail >>> - clone the Dovetail tree (v5.10) from [2], switch to branch dovetail/master >>> - run scripts/prepare-kernel.sh available from [1] into [2] (usual procedure) >>> for x86, that would be: .../scripts/prepare-kernel.sh --arch=x86 >>> - build your kernel using the sources from [2] >>> >>> There is no user-visible Kconfig change compared to an I-pipe based >>> version. >>> >>> Alternatively, the Xenomai code base in [1] can also run on top of the >>> I-pipe. prepare-kernel.sh detects which pipeline flavour is there, and >>> prepares the source tree accordingly. >>> >>> This code is being gradually merged into Xenomai's -next branch, and >>> will be at the core of Xenomai 3.2. Testing and feedback appreciated. >>> >>> Thanks, >>> >>> [1] https://lab.xenomai.org/xenomai-rpm.git, "for-upstream/dovetail" branch >>> [2] https://git.evlproject.org/linux-evl.git, "dovetail/master" branch >>> >> >> First try on my end looked good as well - great news, well done! >> >> Would it help to stick the wip branch into our CI/CT? >> > > Yep, it would. TIA, > CI build is passing now, see https://gitlab.denx.de/Xenomai/xenomai/-/pipelines/6258 I'm carrying two fixes and a could of tweaks: https://gitlab.denx.de/Xenomai/xenomai/-/commits/wip/dovetail Specifically https://gitlab.denx.de/Xenomai/xenomai/-/commit/4196216e821abea3107284a8e31e13bdf05141d5 would a second look as the timer ipi abstraction became obsolete with the code moving to pipeline-specific code. I've just started also target runs, see https://gitlab.denx.de/Xenomai/xenomai-images/-/pipelines/6263. There will be several known breakages, though, namely the gdb test but also the unrelated arm[64]-5.4 regressions. Jan -- Siemens AG, T RDA IOT Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Dovetail] x86 test version available (kernel v5.10) 2021-02-06 12:29 ` Jan Kiszka @ 2021-02-06 17:23 ` Philippe Gerum 2021-02-15 13:28 ` Jan Kiszka 1 sibling, 0 replies; 20+ messages in thread From: Philippe Gerum @ 2021-02-06 17:23 UTC (permalink / raw) To: Jan Kiszka; +Cc: xenomai, hongzha1 Jan Kiszka <jan.kiszka@siemens.com> writes: > On 03.02.21 18:35, Philippe Gerum wrote: >> >> Jan Kiszka <jan.kiszka@siemens.com> writes: >> >>> On 31.01.21 17:06, Philippe Gerum wrote: >>>> >>>> The initial port of the Cobalt core to Dovetail/x86 is available from >>>> [1]. Ports to Dovetail/ARM and Dovetail/arm64 should follow within a >>>> couple of weeks. >>>> >>>> So far, latency and switchtest run flawlessly. Most of the smokey test >>>> suite passes successfully, except the GDB test at the moment. >>>> >>>> How to test this: >>>> >>>> - clone Xenomai from [1], switch to branch for-upstream/dovetail >>>> - clone the Dovetail tree (v5.10) from [2], switch to branch dovetail/master >>>> - run scripts/prepare-kernel.sh available from [1] into [2] (usual procedure) >>>> for x86, that would be: .../scripts/prepare-kernel.sh --arch=x86 >>>> - build your kernel using the sources from [2] >>>> >>>> There is no user-visible Kconfig change compared to an I-pipe based >>>> version. >>>> >>>> Alternatively, the Xenomai code base in [1] can also run on top of the >>>> I-pipe. prepare-kernel.sh detects which pipeline flavour is there, and >>>> prepares the source tree accordingly. >>>> >>>> This code is being gradually merged into Xenomai's -next branch, and >>>> will be at the core of Xenomai 3.2. Testing and feedback appreciated. >>>> >>>> Thanks, >>>> >>>> [1] https://lab.xenomai.org/xenomai-rpm.git, "for-upstream/dovetail" branch >>>> [2] https://git.evlproject.org/linux-evl.git, "dovetail/master" branch >>>> >>> >>> First try on my end looked good as well - great news, well done! >>> >>> Would it help to stick the wip branch into our CI/CT? >>> >> >> Yep, it would. TIA, >> > > CI build is passing now, see > > https://gitlab.denx.de/Xenomai/xenomai/-/pipelines/6258 > > I'm carrying two fixes and a could of tweaks: > > https://gitlab.denx.de/Xenomai/xenomai/-/commits/wip/dovetail > I'll pick the trace-related fixes in my queue. > Specifically > https://gitlab.denx.de/Xenomai/xenomai/-/commit/4196216e821abea3107284a8e31e13bdf05141d5 > would a second look as the timer ipi abstraction became obsolete with > the code moving to pipeline-specific code. > Yes, moving this to the pipeline-specific section, so that we reduce the level of visual clutter in the generic code. > I've just started also target runs, see > https://gitlab.denx.de/Xenomai/xenomai-images/-/pipelines/6263. There > will be several known breakages, though, namely the gdb test but also > the unrelated arm[64]-5.4 regressions. > I'm fixing the RTnet build issues ATM. -- Philippe. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Dovetail] x86 test version available (kernel v5.10) 2021-02-06 12:29 ` Jan Kiszka 2021-02-06 17:23 ` Philippe Gerum @ 2021-02-15 13:28 ` Jan Kiszka 2021-02-15 15:11 ` Jan Kiszka 1 sibling, 1 reply; 20+ messages in thread From: Jan Kiszka @ 2021-02-15 13:28 UTC (permalink / raw) To: Philippe Gerum; +Cc: xenomai On 06.02.21 13:29, Jan Kiszka via Xenomai wrote: > On 03.02.21 18:35, Philippe Gerum wrote: >> >> Jan Kiszka <jan.kiszka@siemens.com> writes: >> >>> On 31.01.21 17:06, Philippe Gerum wrote: >>>> >>>> The initial port of the Cobalt core to Dovetail/x86 is available from >>>> [1]. Ports to Dovetail/ARM and Dovetail/arm64 should follow within a >>>> couple of weeks. >>>> >>>> So far, latency and switchtest run flawlessly. Most of the smokey test >>>> suite passes successfully, except the GDB test at the moment. >>>> >>>> How to test this: >>>> >>>> - clone Xenomai from [1], switch to branch for-upstream/dovetail >>>> - clone the Dovetail tree (v5.10) from [2], switch to branch dovetail/master >>>> - run scripts/prepare-kernel.sh available from [1] into [2] (usual procedure) >>>> for x86, that would be: .../scripts/prepare-kernel.sh --arch=x86 >>>> - build your kernel using the sources from [2] >>>> >>>> There is no user-visible Kconfig change compared to an I-pipe based >>>> version. >>>> >>>> Alternatively, the Xenomai code base in [1] can also run on top of the >>>> I-pipe. prepare-kernel.sh detects which pipeline flavour is there, and >>>> prepares the source tree accordingly. >>>> >>>> This code is being gradually merged into Xenomai's -next branch, and >>>> will be at the core of Xenomai 3.2. Testing and feedback appreciated. >>>> >>>> Thanks, >>>> >>>> [1] https://lab.xenomai.org/xenomai-rpm.git, "for-upstream/dovetail" branch >>>> [2] https://git.evlproject.org/linux-evl.git, "dovetail/master" branch >>>> >>> >>> First try on my end looked good as well - great news, well done! >>> >>> Would it help to stick the wip branch into our CI/CT? >>> >> >> Yep, it would. TIA, >> > > CI build is passing now, see > > https://gitlab.denx.de/Xenomai/xenomai/-/pipelines/6258 > > I'm carrying two fixes and a could of tweaks: > > https://gitlab.denx.de/Xenomai/xenomai/-/commits/wip/dovetail > > Specifically > https://gitlab.denx.de/Xenomai/xenomai/-/commit/4196216e821abea3107284a8e31e13bdf05141d5 > would a second look as the timer ipi abstraction became obsolete with > the code moving to pipeline-specific code. > > I've just started also target runs, see > https://gitlab.denx.de/Xenomai/xenomai-images/-/pipelines/6263. There > will be several known breakages, though, namely the gdb test but also > the unrelated arm[64]-5.4 regressions. > I think I (widely) understood the issue on x86 qemu: posix-clock.c:285, assertion failed: diff >= 5500000000LL && diff <= 6500000000LL /usr/lib/xenomai/testsuite/smokey: test posix_clock failed: Invalid argument It should have been visible on a real machine as well - as long as systemd-timesyncd is running. Stopping that service makes the test work. As we are now sharing CLOCK_REALTIME with Linux, any ongoing tweak of the system time by ntp and similar services will defeat our temporal set-back in smokey's test case. We can address that in xenomai-images, but I'm unsure how to resolve that more generically for smokey. Any ideas? Now the next issue is /usr/lib/xenomai/testsuite/smokey: test sched_quota failed: Protocol error sched-quota.c:332, out of quota: 13.7% (on a real target, an Atom-class box). Jan -- Siemens AG, T RDA IOT Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Dovetail] x86 test version available (kernel v5.10) 2021-02-15 13:28 ` Jan Kiszka @ 2021-02-15 15:11 ` Jan Kiszka 2021-02-15 16:53 ` Jan Kiszka 2021-02-15 17:07 ` Philippe Gerum 0 siblings, 2 replies; 20+ messages in thread From: Jan Kiszka @ 2021-02-15 15:11 UTC (permalink / raw) To: Philippe Gerum; +Cc: xenomai On 15.02.21 14:28, Jan Kiszka wrote: > Now the next issue is > > /usr/lib/xenomai/testsuite/smokey: test sched_quota failed: Protocol error > sched-quota.c:332, out of quota: 13.7% > (on a real target, an Atom-class box). > Testsuite issue, long pending: https://gitlab.denx.de/Xenomai/xenomai/-/blob/master/testsuite/smokey/sched-quota/sched-quota.c#L204 Those pthread_kill() in the loop will kick the caller into non-rt, while it is not yet done with collecting the loop counters from the other workers. Result is pure random, sometimes still within the defined limits, but sometimes reporting false overruns. No idea what those signal submissions supposed to do - test runs and terminates fine without them as well. Jan -- Siemens AG, T RDA IOT Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Dovetail] x86 test version available (kernel v5.10) 2021-02-15 15:11 ` Jan Kiszka @ 2021-02-15 16:53 ` Jan Kiszka 2021-02-15 17:19 ` Jan Kiszka 2021-02-15 17:07 ` Philippe Gerum 1 sibling, 1 reply; 20+ messages in thread From: Jan Kiszka @ 2021-02-15 16:53 UTC (permalink / raw) To: Philippe Gerum; +Cc: xenomai On 15.02.21 16:11, Jan Kiszka via Xenomai wrote: > On 15.02.21 14:28, Jan Kiszka wrote: >> Now the next issue is >> >> /usr/lib/xenomai/testsuite/smokey: test sched_quota failed: Protocol error >> sched-quota.c:332, out of quota: 13.7% >> (on a real target, an Atom-class box). >> > > Testsuite issue, long pending: > > https://gitlab.denx.de/Xenomai/xenomai/-/blob/master/testsuite/smokey/sched-quota/sched-quota.c#L204 > > Those pthread_kill() in the loop will kick the caller into non-rt, while > it is not yet done with collecting the loop counters from the other > workers. Result is pure random, sometimes still within the defined > limits, but sometimes reporting false overruns. > > No idea what those signal submissions supposed to do - test runs and > terminates fine without them as well. > Correction: The pthread_kill stay in primary and do not cause the issue (they are still not needed IMHO, though). It must be something compiler (setting?) related: smokey built with my gcc-7 from SUSE is fine, the one build with Debian's gcc-8 is not, causing noteworthy variations on the measured workload. Examining further... Jan -- Siemens AG, T RDA IOT Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Dovetail] x86 test version available (kernel v5.10) 2021-02-15 16:53 ` Jan Kiszka @ 2021-02-15 17:19 ` Jan Kiszka 0 siblings, 0 replies; 20+ messages in thread From: Jan Kiszka @ 2021-02-15 17:19 UTC (permalink / raw) To: Philippe Gerum; +Cc: xenomai On 15.02.21 17:53, Jan Kiszka via Xenomai wrote: > On 15.02.21 16:11, Jan Kiszka via Xenomai wrote: >> On 15.02.21 14:28, Jan Kiszka wrote: >>> Now the next issue is >>> >>> /usr/lib/xenomai/testsuite/smokey: test sched_quota failed: Protocol error >>> sched-quota.c:332, out of quota: 13.7% >>> (on a real target, an Atom-class box). >>> >> >> Testsuite issue, long pending: >> >> https://gitlab.denx.de/Xenomai/xenomai/-/blob/master/testsuite/smokey/sched-quota/sched-quota.c#L204 >> >> Those pthread_kill() in the loop will kick the caller into non-rt, while >> it is not yet done with collecting the loop counters from the other >> workers. Result is pure random, sometimes still within the defined >> limits, but sometimes reporting false overruns. >> >> No idea what those signal submissions supposed to do - test runs and >> terminates fine without them as well. >> > > Correction: The pthread_kill stay in primary and do not cause the issue > (they are still not needed IMHO, though). > > It must be something compiler (setting?) related: smokey built with my > gcc-7 from SUSE is fine, the one build with Debian's gcc-8 is not, > causing noteworthy variations on the measured workload. Examining further... > It was gcc-10 in fact on my machine - but the actual difference was --enable-debug=full here vs. no debug in xenomai-images. Without full debugging, the loop calibration becomes too instable, likely because the inner __do_work is reduced to a simple 417960: 48 8d 47 01 lea 0x1(%rdi),%rax 417964: c3 retq This can be mitigated by extending the calibration loop (crunch_loops = 100000). Jan -- Siemens AG, T RDA IOT Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Dovetail] x86 test version available (kernel v5.10) 2021-02-15 15:11 ` Jan Kiszka 2021-02-15 16:53 ` Jan Kiszka @ 2021-02-15 17:07 ` Philippe Gerum 2021-02-15 17:21 ` Philippe Gerum 1 sibling, 1 reply; 20+ messages in thread From: Philippe Gerum @ 2021-02-15 17:07 UTC (permalink / raw) To: Jan Kiszka; +Cc: xenomai Jan Kiszka <jan.kiszka@siemens.com> writes: > On 15.02.21 14:28, Jan Kiszka wrote: >> Now the next issue is >> >> /usr/lib/xenomai/testsuite/smokey: test sched_quota failed: Protocol error >> sched-quota.c:332, out of quota: 13.7% >> (on a real target, an Atom-class box). >> > > Testsuite issue, long pending: > > https://gitlab.denx.de/Xenomai/xenomai/-/blob/master/testsuite/smokey/sched-quota/sched-quota.c#L204 > > Those pthread_kill() in the loop will kick the caller into non-rt, while > it is not yet done with collecting the loop counters from the other > workers. Result is pure random, sometimes still within the defined > limits, but sometimes reporting false overruns. Make sense, we need to collect the counters independently then. > > No idea what those signal submissions supposed to do - test runs and > terminates fine without them as well. > On SMP probably, on UP I don't think so, because if you don't demote the workers, then the following call to pthread_cancel() will switch the main context out of primary mode, leaving one of the workers preempt back, then go spinning full speed, locking the CPU (there is no condition for exiting the work loop). Using the throttling hack like the calibration loop does may help (I guess I overlooked this back then). -- Philippe. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Dovetail] x86 test version available (kernel v5.10) 2021-02-15 17:07 ` Philippe Gerum @ 2021-02-15 17:21 ` Philippe Gerum 0 siblings, 0 replies; 20+ messages in thread From: Philippe Gerum @ 2021-02-15 17:21 UTC (permalink / raw) To: Jan Kiszka; +Cc: xenomai Philippe Gerum <rpm@xenomai.org> writes: > Jan Kiszka <jan.kiszka@siemens.com> writes: > >> On 15.02.21 14:28, Jan Kiszka wrote: >>> Now the next issue is >>> >>> /usr/lib/xenomai/testsuite/smokey: test sched_quota failed: Protocol error >>> sched-quota.c:332, out of quota: 13.7% >>> (on a real target, an Atom-class box). >>> >> >> Testsuite issue, long pending: >> >> https://gitlab.denx.de/Xenomai/xenomai/-/blob/master/testsuite/smokey/sched-quota/sched-quota.c#L204 >> >> Those pthread_kill() in the loop will kick the caller into non-rt, while >> it is not yet done with collecting the loop counters from the other >> workers. Result is pure random, sometimes still within the defined >> limits, but sometimes reporting false overruns. > > Make sense, we need to collect the counters independently then. > Strike that, all threads are pinned on the same CPU (0), so there is no way pthread_kill() would cause the worker threads to keep running since the main context always runs with higher priority. -- Philippe. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Dovetail] x86 test version available (kernel v5.10) 2021-01-31 16:06 [Dovetail] x86 test version available (kernel v5.10) Philippe Gerum 2021-02-01 17:32 ` Jan Kiszka @ 2021-02-22 15:35 ` Henning Schild 2021-02-22 17:59 ` Henning Schild 2021-03-11 16:35 ` Henning Schild 1 sibling, 2 replies; 20+ messages in thread From: Henning Schild @ 2021-02-22 15:35 UTC (permalink / raw) To: Philippe Gerum via Xenomai Am Sun, 31 Jan 2021 17:06:21 +0100 schrieb Philippe Gerum via Xenomai <xenomai@xenomai.org>: > The initial port of the Cobalt core to Dovetail/x86 is available from > [1]. Ports to Dovetail/ARM and Dovetail/arm64 should follow within a > couple of weeks. > > So far, latency and switchtest run flawlessly. Most of the smokey test > suite passes successfully, except the GDB test at the moment. > > How to test this: > > - clone Xenomai from [1], switch to branch for-upstream/dovetail > - clone the Dovetail tree (v5.10) from [2], switch to branch > dovetail/master > - run scripts/prepare-kernel.sh available from [1] into [2] (usual > procedure) for x86, that would be: .../scripts/prepare-kernel.sh > --arch=x86 > - build your kernel using the sources from [2] I followed this and got 0625b829450dab22b2eea860eef4fb94f64af4fd 61769e49d82a6775f6eb259bdc16c2f84df79505 > There is no user-visible Kconfig change compared to an I-pipe based > version. took a 4.19 x86_64 savedefconfig in via olddefconfig > Alternatively, the Xenomai code base in [1] can also run on top of the > I-pipe. prepare-kernel.sh detects which pipeline flavour is there, and > prepares the source tree accordingly. > > This code is being gradually merged into Xenomai's -next branch, and > will be at the core of Xenomai 3.2. Testing and feedback appreciated. Now i get a BUG pretty early on during boot. Since my config still seems off i could not yet enter my rootfs, probably some drivers/filesystems/raid switches got lost ... Its a null pointer deref in kmem_cache_alloc, coming somewhere from mempool_init_node First guess was that "numa=off" would hide the problem, seems it does. So my next guess is that even a qemu with numa could reproduce this ... probably almost on a random config ... as long as it has smp+numa I do not have more at the moment, but there seem to be issues around NUMA. regards, Henning > Thanks, > > [1] https://lab.xenomai.org/xenomai-rpm.git, "for-upstream/dovetail" > branch [2] https://git.evlproject.org/linux-evl.git, > "dovetail/master" branch > ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Dovetail] x86 test version available (kernel v5.10) 2021-02-22 15:35 ` Henning Schild @ 2021-02-22 17:59 ` Henning Schild 2021-03-11 16:35 ` Henning Schild 1 sibling, 0 replies; 20+ messages in thread From: Henning Schild @ 2021-02-22 17:59 UTC (permalink / raw) To: Henning Schild via Xenomai Am Mon, 22 Feb 2021 16:35:30 +0100 schrieb Henning Schild via Xenomai <xenomai@xenomai.org>: > Am Sun, 31 Jan 2021 17:06:21 +0100 > schrieb Philippe Gerum via Xenomai <xenomai@xenomai.org>: > > > The initial port of the Cobalt core to Dovetail/x86 is available > > from [1]. Ports to Dovetail/ARM and Dovetail/arm64 should follow > > within a couple of weeks. > > > > So far, latency and switchtest run flawlessly. Most of the smokey > > test suite passes successfully, except the GDB test at the moment. > > > > How to test this: > > > > - clone Xenomai from [1], switch to branch for-upstream/dovetail > > - clone the Dovetail tree (v5.10) from [2], switch to branch > > dovetail/master > > - run scripts/prepare-kernel.sh available from [1] into [2] (usual > > procedure) for x86, that would be: .../scripts/prepare-kernel.sh > > --arch=x86 > > - build your kernel using the sources from [2] > > I followed this and got > > 0625b829450dab22b2eea860eef4fb94f64af4fd > 61769e49d82a6775f6eb259bdc16c2f84df79505 > > > > There is no user-visible Kconfig change compared to an I-pipe based > > version. > > took a 4.19 x86_64 savedefconfig in via olddefconfig > > > Alternatively, the Xenomai code base in [1] can also run on top of > > the I-pipe. prepare-kernel.sh detects which pipeline flavour is > > there, and prepares the source tree accordingly. > > > > This code is being gradually merged into Xenomai's -next branch, and > > will be at the core of Xenomai 3.2. Testing and feedback > > appreciated. > > Now i get a BUG pretty early on during boot. Since my config still > seems off i could not yet enter my rootfs, probably some > drivers/filesystems/raid switches got lost ... > > Its a null pointer deref in kmem_cache_alloc, coming somewhere from > mempool_init_node > > First guess was that "numa=off" would hide the problem, seems it does. > So my next guess is that even a qemu with numa could reproduce this > ... probably almost on a random config ... as long as it has smp+numa Tried that again, it is not 100% reproducible ... "numa=off" does not solve the issue. I did try the dovetail bits of xenomai-images 5.10 together with a numa-qemu ... that works. qemu-system-x86_64 -drive file=./build/tmp/deploy/images/qemu-amd64/demo-image-xenomai-demo-qemu-amd64.ext4.img,discard=unmap,if=none,id=disk,format=raw -serial mon:stdio -netdev user,id=net -kernel ./build/tmp/deploy/images/qemu-amd64/demo-image-xenomai-demo-qemu-amd64-vmlinuz -append ' root=/dev/sda vga=0x305' -initrd ./build/tmp/deploy/images/qemu-amd64/demo-image-xenomai-demo-qemu-amd64-initrd.img -cpu host -smp 4 -enable-kvm -machine q35 -device ide-hd,drive=disk -device virtio-net-pci,netdev=net -object memory-backend-ram,id=mem0,size=1G -object memory-backend-ram,id=mem1,size=1G -numa node,cpus=0-1,nodeid=0,memdev=mem0 -numa node,cpus=2-3,nodeid=1,memdev=mem1 -m 2G > I do not have more at the moment, but there seem to be issues around > NUMA. So maybe it is not NUMA, will keep digging. Henning > regards, > Henning > > > Thanks, > > > > [1] https://lab.xenomai.org/xenomai-rpm.git, "for-upstream/dovetail" > > branch [2] https://git.evlproject.org/linux-evl.git, > > "dovetail/master" branch > > > > ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Dovetail] x86 test version available (kernel v5.10) 2021-02-22 15:35 ` Henning Schild 2021-02-22 17:59 ` Henning Schild @ 2021-03-11 16:35 ` Henning Schild 2021-03-12 7:22 ` Jan Kiszka 1 sibling, 1 reply; 20+ messages in thread From: Henning Schild @ 2021-03-11 16:35 UTC (permalink / raw) To: Henning Schild via Xenomai Am Mon, 22 Feb 2021 16:35:30 +0100 schrieb Henning Schild via Xenomai <xenomai@xenomai.org>: > Am Sun, 31 Jan 2021 17:06:21 +0100 > schrieb Philippe Gerum via Xenomai <xenomai@xenomai.org>: > > > The initial port of the Cobalt core to Dovetail/x86 is available > > from [1]. Ports to Dovetail/ARM and Dovetail/arm64 should follow > > within a couple of weeks. > > > > So far, latency and switchtest run flawlessly. Most of the smokey > > test suite passes successfully, except the GDB test at the moment. > > > > How to test this: > > > > - clone Xenomai from [1], switch to branch for-upstream/dovetail > > - clone the Dovetail tree (v5.10) from [2], switch to branch > > dovetail/master > > - run scripts/prepare-kernel.sh available from [1] into [2] (usual > > procedure) for x86, that would be: .../scripts/prepare-kernel.sh > > --arch=x86 > > - build your kernel using the sources from [2] > > I followed this and got > > 0625b829450dab22b2eea860eef4fb94f64af4fd > 61769e49d82a6775f6eb259bdc16c2f84df79505 > > > > There is no user-visible Kconfig change compared to an I-pipe based > > version. > > took a 4.19 x86_64 savedefconfig in via olddefconfig > > > Alternatively, the Xenomai code base in [1] can also run on top of > > the I-pipe. prepare-kernel.sh detects which pipeline flavour is > > there, and prepares the source tree accordingly. > > > > This code is being gradually merged into Xenomai's -next branch, and > > will be at the core of Xenomai 3.2. Testing and feedback > > appreciated. > > Now i get a BUG pretty early on during boot. Since my config still > seems off i could not yet enter my rootfs, probably some > drivers/filesystems/raid switches got lost ... > > Its a null pointer deref in kmem_cache_alloc, coming somewhere from > mempool_init_node > > First guess was that "numa=off" would hide the problem, seems it does. > So my next guess is that even a qemu with numa could reproduce this > ... probably almost on a random config ... as long as it has smp+numa It is not about numa, now also see that on qemu. Jan looks into it at the moment. Henning > I do not have more at the moment, but there seem to be issues around > NUMA. > > regards, > Henning > > > Thanks, > > > > [1] https://lab.xenomai.org/xenomai-rpm.git, "for-upstream/dovetail" > > branch [2] https://git.evlproject.org/linux-evl.git, > > "dovetail/master" branch > > > > ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Dovetail] x86 test version available (kernel v5.10) 2021-03-11 16:35 ` Henning Schild @ 2021-03-12 7:22 ` Jan Kiszka 2021-03-12 11:32 ` Jan Kiszka 0 siblings, 1 reply; 20+ messages in thread From: Jan Kiszka @ 2021-03-12 7:22 UTC (permalink / raw) To: Henning Schild, Henning Schild via Xenomai On 11.03.21 17:35, Henning Schild wrote: > Am Mon, 22 Feb 2021 16:35:30 +0100 > schrieb Henning Schild via Xenomai <xenomai@xenomai.org>: > >> Am Sun, 31 Jan 2021 17:06:21 +0100 >> schrieb Philippe Gerum via Xenomai <xenomai@xenomai.org>: >> >>> The initial port of the Cobalt core to Dovetail/x86 is available >>> from [1]. Ports to Dovetail/ARM and Dovetail/arm64 should follow >>> within a couple of weeks. >>> >>> So far, latency and switchtest run flawlessly. Most of the smokey >>> test suite passes successfully, except the GDB test at the moment. >>> >>> How to test this: >>> >>> - clone Xenomai from [1], switch to branch for-upstream/dovetail >>> - clone the Dovetail tree (v5.10) from [2], switch to branch >>> dovetail/master >>> - run scripts/prepare-kernel.sh available from [1] into [2] (usual >>> procedure) for x86, that would be: .../scripts/prepare-kernel.sh >>> --arch=x86 >>> - build your kernel using the sources from [2] >> >> I followed this and got >> >> 0625b829450dab22b2eea860eef4fb94f64af4fd >> 61769e49d82a6775f6eb259bdc16c2f84df79505 >> >> >>> There is no user-visible Kconfig change compared to an I-pipe based >>> version. >> >> took a 4.19 x86_64 savedefconfig in via olddefconfig >> >>> Alternatively, the Xenomai code base in [1] can also run on top of >>> the I-pipe. prepare-kernel.sh detects which pipeline flavour is >>> there, and prepares the source tree accordingly. >>> >>> This code is being gradually merged into Xenomai's -next branch, and >>> will be at the core of Xenomai 3.2. Testing and feedback >>> appreciated. >> >> Now i get a BUG pretty early on during boot. Since my config still >> seems off i could not yet enter my rootfs, probably some >> drivers/filesystems/raid switches got lost ... >> >> Its a null pointer deref in kmem_cache_alloc, coming somewhere from >> mempool_init_node >> >> First guess was that "numa=off" would hide the problem, seems it does. >> So my next guess is that even a qemu with numa could reproduce this >> ... probably almost on a random config ... as long as it has smp+numa > > It is not about numa, now also see that on qemu. Jan looks into it at > the moment. > I'm suspecting an upstream issue. The bug is some freelist corruption, possibly only surfaced by dovetail. After digging longer, I enabled CONFIG_SLAB_FREELIST_HARDENED, and that gave [ 0.159611] ACPI Error: Could not remove SCI handler (20210105/evmisc-251) [ 0.160188] ------------[ cut here ]------------ [ 0.160566] cache_from_obj: Wrong slab cache. ftrace_event_field but object is from kmalloc-64 and more. However: You get those bugs also from running latest 5.10.23 and even Linus' master. Will debug that further and then possibly report upstream. Jan -- Siemens AG, T RDA IOT Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Dovetail] x86 test version available (kernel v5.10) 2021-03-12 7:22 ` Jan Kiszka @ 2021-03-12 11:32 ` Jan Kiszka 2021-03-12 18:18 ` Henning Schild 0 siblings, 1 reply; 20+ messages in thread From: Jan Kiszka @ 2021-03-12 11:32 UTC (permalink / raw) To: Henning Schild, Henning Schild via Xenomai, Bezdeka, Florian (T RDA IOT SES-DE) On 12.03.21 08:22, Jan Kiszka wrote: > On 11.03.21 17:35, Henning Schild wrote: >> Am Mon, 22 Feb 2021 16:35:30 +0100 >> schrieb Henning Schild via Xenomai <xenomai@xenomai.org>: >> >>> Am Sun, 31 Jan 2021 17:06:21 +0100 >>> schrieb Philippe Gerum via Xenomai <xenomai@xenomai.org>: >>> >>>> The initial port of the Cobalt core to Dovetail/x86 is available >>>> from [1]. Ports to Dovetail/ARM and Dovetail/arm64 should follow >>>> within a couple of weeks. >>>> >>>> So far, latency and switchtest run flawlessly. Most of the smokey >>>> test suite passes successfully, except the GDB test at the moment. >>>> >>>> How to test this: >>>> >>>> - clone Xenomai from [1], switch to branch for-upstream/dovetail >>>> - clone the Dovetail tree (v5.10) from [2], switch to branch >>>> dovetail/master >>>> - run scripts/prepare-kernel.sh available from [1] into [2] (usual >>>> procedure) for x86, that would be: .../scripts/prepare-kernel.sh >>>> --arch=x86 >>>> - build your kernel using the sources from [2] >>> >>> I followed this and got >>> >>> 0625b829450dab22b2eea860eef4fb94f64af4fd >>> 61769e49d82a6775f6eb259bdc16c2f84df79505 >>> >>> >>>> There is no user-visible Kconfig change compared to an I-pipe based >>>> version. >>> >>> took a 4.19 x86_64 savedefconfig in via olddefconfig >>> >>>> Alternatively, the Xenomai code base in [1] can also run on top of >>>> the I-pipe. prepare-kernel.sh detects which pipeline flavour is >>>> there, and prepares the source tree accordingly. >>>> >>>> This code is being gradually merged into Xenomai's -next branch, and >>>> will be at the core of Xenomai 3.2. Testing and feedback >>>> appreciated. >>> >>> Now i get a BUG pretty early on during boot. Since my config still >>> seems off i could not yet enter my rootfs, probably some >>> drivers/filesystems/raid switches got lost ... >>> >>> Its a null pointer deref in kmem_cache_alloc, coming somewhere from >>> mempool_init_node >>> >>> First guess was that "numa=off" would hide the problem, seems it does. >>> So my next guess is that even a qemu with numa could reproduce this >>> ... probably almost on a random config ... as long as it has smp+numa >> >> It is not about numa, now also see that on qemu. Jan looks into it at >> the moment. >> > > I'm suspecting an upstream issue. The bug is some freelist corruption, > possibly only surfaced by dovetail. After digging longer, I enabled > CONFIG_SLAB_FREELIST_HARDENED, and that gave > > [ 0.159611] ACPI Error: Could not remove SCI handler > (20210105/evmisc-251) > [ 0.160188] ------------[ cut here ]------------ > [ 0.160566] cache_from_obj: Wrong slab cache. ftrace_event_field but > object is from kmalloc-64 > > and more. However: You get those bugs also from running latest 5.10.23 > and even Linus' master. Will debug that further and then possibly report > upstream. This is a corner case of upstream (but still a bug there): Your config lost CONFIG_PCI, likely because it is no longer default y. That not only generates and unbootable systems, it also sends the ACPI subsystem into an error path. There it seems to release objects to the wrong caches, leading to corruptions later on. Enabling PCI resolves all this. I'll dump the config to upstream ACPI folks, should be their business. Jan PS: Your config has more issues as it does not even boot in QEMU, even with PCI enabled. You should (re-)derive from x86_64-defconfig. -- Siemens AG, T RDA IOT Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Dovetail] x86 test version available (kernel v5.10) 2021-03-12 11:32 ` Jan Kiszka @ 2021-03-12 18:18 ` Henning Schild 2021-03-12 19:46 ` Henning Schild 0 siblings, 1 reply; 20+ messages in thread From: Henning Schild @ 2021-03-12 18:18 UTC (permalink / raw) To: Jan Kiszka Cc: Henning Schild via Xenomai, Bezdeka, Florian (T RDA IOT SES-DE) Am Fri, 12 Mar 2021 12:32:54 +0100 schrieb Jan Kiszka <jan.kiszka@siemens.com>: > On 12.03.21 08:22, Jan Kiszka wrote: > > On 11.03.21 17:35, Henning Schild wrote: > >> Am Mon, 22 Feb 2021 16:35:30 +0100 > >> schrieb Henning Schild via Xenomai <xenomai@xenomai.org>: > >> > >>> Am Sun, 31 Jan 2021 17:06:21 +0100 > >>> schrieb Philippe Gerum via Xenomai <xenomai@xenomai.org>: > >>> > >>>> The initial port of the Cobalt core to Dovetail/x86 is available > >>>> from [1]. Ports to Dovetail/ARM and Dovetail/arm64 should follow > >>>> within a couple of weeks. > >>>> > >>>> So far, latency and switchtest run flawlessly. Most of the smokey > >>>> test suite passes successfully, except the GDB test at the > >>>> moment. > >>>> > >>>> How to test this: > >>>> > >>>> - clone Xenomai from [1], switch to branch for-upstream/dovetail > >>>> - clone the Dovetail tree (v5.10) from [2], switch to branch > >>>> dovetail/master > >>>> - run scripts/prepare-kernel.sh available from [1] into [2] > >>>> (usual procedure) for x86, that would be: > >>>> .../scripts/prepare-kernel.sh --arch=x86 > >>>> - build your kernel using the sources from [2] > >>> > >>> I followed this and got > >>> > >>> 0625b829450dab22b2eea860eef4fb94f64af4fd > >>> 61769e49d82a6775f6eb259bdc16c2f84df79505 > >>> > >>> > >>>> There is no user-visible Kconfig change compared to an I-pipe > >>>> based version. > >>> > >>> took a 4.19 x86_64 savedefconfig in via olddefconfig > >>> > >>>> Alternatively, the Xenomai code base in [1] can also run on top > >>>> of the I-pipe. prepare-kernel.sh detects which pipeline flavour > >>>> is there, and prepares the source tree accordingly. > >>>> > >>>> This code is being gradually merged into Xenomai's -next branch, > >>>> and will be at the core of Xenomai 3.2. Testing and feedback > >>>> appreciated. > >>> > >>> Now i get a BUG pretty early on during boot. Since my config still > >>> seems off i could not yet enter my rootfs, probably some > >>> drivers/filesystems/raid switches got lost ... > >>> > >>> Its a null pointer deref in kmem_cache_alloc, coming somewhere > >>> from mempool_init_node > >>> > >>> First guess was that "numa=off" would hide the problem, seems it > >>> does. So my next guess is that even a qemu with numa could > >>> reproduce this ... probably almost on a random config ... as long > >>> as it has smp+numa > >> > >> It is not about numa, now also see that on qemu. Jan looks into it > >> at the moment. > >> > > > > I'm suspecting an upstream issue. The bug is some freelist > > corruption, possibly only surfaced by dovetail. After digging > > longer, I enabled CONFIG_SLAB_FREELIST_HARDENED, and that gave > > > > [ 0.159611] ACPI Error: Could not remove SCI handler > > (20210105/evmisc-251) > > [ 0.160188] ------------[ cut here ]------------ > > [ 0.160566] cache_from_obj: Wrong slab cache. ftrace_event_field > > but object is from kmalloc-64 > > > > and more. However: You get those bugs also from running latest > > 5.10.23 and even Linus' master. Will debug that further and then > > possibly report upstream. > > This is a corner case of upstream (but still a bug there): > > Your config lost CONFIG_PCI, likely because it is no longer default y. > That not only generates and unbootable systems, it also sends the ACPI > subsystem into an error path. There it seems to release objects to the > wrong caches, leading to corruptions later on. Enabling PCI resolves > all this. > > I'll dump the config to upstream ACPI folks, should be their business. Thanks Jan! > Jan > > PS: Your config has more issues as it does not even boot in QEMU, even > with PCI enabled. You should (re-)derive from x86_64-defconfig. Yes that was CI progressing with "olddefconfig" from a 4.19 "savedefconfig". No human took a closer look at the result, before you. But hey that might fix an upstream bug so it was a good exercise anyhow. Henning ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Dovetail] x86 test version available (kernel v5.10) 2021-03-12 18:18 ` Henning Schild @ 2021-03-12 19:46 ` Henning Schild 2021-03-12 19:53 ` Henning Schild 0 siblings, 1 reply; 20+ messages in thread From: Henning Schild @ 2021-03-12 19:46 UTC (permalink / raw) To: Henning Schild via Xenomai Am Fri, 12 Mar 2021 19:18:14 +0100 schrieb Henning Schild via Xenomai <xenomai@xenomai.org>: > Am Fri, 12 Mar 2021 12:32:54 +0100 > schrieb Jan Kiszka <jan.kiszka@siemens.com>: > > > On 12.03.21 08:22, Jan Kiszka wrote: > > > On 11.03.21 17:35, Henning Schild wrote: > > >> Am Mon, 22 Feb 2021 16:35:30 +0100 > > >> schrieb Henning Schild via Xenomai <xenomai@xenomai.org>: > > >> > > >>> Am Sun, 31 Jan 2021 17:06:21 +0100 > > >>> schrieb Philippe Gerum via Xenomai <xenomai@xenomai.org>: > > >>> > > >>>> The initial port of the Cobalt core to Dovetail/x86 is > > >>>> available from [1]. Ports to Dovetail/ARM and Dovetail/arm64 > > >>>> should follow within a couple of weeks. > > >>>> > > >>>> So far, latency and switchtest run flawlessly. Most of the > > >>>> smokey test suite passes successfully, except the GDB test at > > >>>> the moment. > > >>>> > > >>>> How to test this: > > >>>> > > >>>> - clone Xenomai from [1], switch to branch > > >>>> for-upstream/dovetail > > >>>> - clone the Dovetail tree (v5.10) from [2], switch to branch > > >>>> dovetail/master > > >>>> - run scripts/prepare-kernel.sh available from [1] into [2] > > >>>> (usual procedure) for x86, that would be: > > >>>> .../scripts/prepare-kernel.sh --arch=x86 > > >>>> - build your kernel using the sources from [2] > > >>> > > >>> I followed this and got > > >>> > > >>> 0625b829450dab22b2eea860eef4fb94f64af4fd > > >>> 61769e49d82a6775f6eb259bdc16c2f84df79505 > > >>> > > >>> > > >>>> There is no user-visible Kconfig change compared to an I-pipe > > >>>> based version. > > >>> > > >>> took a 4.19 x86_64 savedefconfig in via olddefconfig > > >>> > > >>>> Alternatively, the Xenomai code base in [1] can also run on top > > >>>> of the I-pipe. prepare-kernel.sh detects which pipeline flavour > > >>>> is there, and prepares the source tree accordingly. > > >>>> > > >>>> This code is being gradually merged into Xenomai's -next > > >>>> branch, and will be at the core of Xenomai 3.2. Testing and > > >>>> feedback appreciated. > > >>> > > >>> Now i get a BUG pretty early on during boot. Since my config > > >>> still seems off i could not yet enter my rootfs, probably some > > >>> drivers/filesystems/raid switches got lost ... > > >>> > > >>> Its a null pointer deref in kmem_cache_alloc, coming somewhere > > >>> from mempool_init_node > > >>> > > >>> First guess was that "numa=off" would hide the problem, seems it > > >>> does. So my next guess is that even a qemu with numa could > > >>> reproduce this ... probably almost on a random config ... as > > >>> long as it has smp+numa > > >> > > >> It is not about numa, now also see that on qemu. Jan looks into > > >> it at the moment. > > >> > > > > > > I'm suspecting an upstream issue. The bug is some freelist > > > corruption, possibly only surfaced by dovetail. After digging > > > longer, I enabled CONFIG_SLAB_FREELIST_HARDENED, and that gave > > > > > > [ 0.159611] ACPI Error: Could not remove SCI handler > > > (20210105/evmisc-251) > > > [ 0.160188] ------------[ cut here ]------------ > > > [ 0.160566] cache_from_obj: Wrong slab cache. > > > ftrace_event_field but object is from kmalloc-64 > > > > > > and more. However: You get those bugs also from running latest > > > 5.10.23 and even Linus' master. Will debug that further and then > > > possibly report upstream. > > > > This is a corner case of upstream (but still a bug there): > > > > Your config lost CONFIG_PCI, likely because it is no longer default > > y. That not only generates and unbootable systems, it also sends > > the ACPI subsystem into an error path. There it seems to release > > objects to the wrong caches, leading to corruptions later on. > > Enabling PCI resolves all this. > > > > I'll dump the config to upstream ACPI folks, should be their > > business. > > Thanks Jan! > > > Jan > > > > PS: Your config has more issues as it does not even boot in QEMU, > > even with PCI enabled. You should (re-)derive from > > x86_64-defconfig. > > Yes that was CI progressing with "olddefconfig" from a 4.19 > "savedefconfig". No human took a closer look at the result, before > you. But hey that might fix an upstream bug so it was a good exercise > anyhow. Looking much better with a changed config. AFAIK the gdb test from smokey is known to fail at the moment. Should that be marked as "skip" mainline to reduce downstream skipping fixups? Henning > Henning > > ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Dovetail] x86 test version available (kernel v5.10) 2021-03-12 19:46 ` Henning Schild @ 2021-03-12 19:53 ` Henning Schild 2021-03-15 7:57 ` Jan Kiszka 0 siblings, 1 reply; 20+ messages in thread From: Henning Schild @ 2021-03-12 19:53 UTC (permalink / raw) To: Henning Schild via Xenomai Am Fri, 12 Mar 2021 20:46:30 +0100 schrieb Henning Schild <henning.schild@siemens.com>: > Am Fri, 12 Mar 2021 19:18:14 +0100 > schrieb Henning Schild via Xenomai <xenomai@xenomai.org>: > > > Am Fri, 12 Mar 2021 12:32:54 +0100 > > schrieb Jan Kiszka <jan.kiszka@siemens.com>: > > > > > On 12.03.21 08:22, Jan Kiszka wrote: > > > > On 11.03.21 17:35, Henning Schild wrote: > > > >> Am Mon, 22 Feb 2021 16:35:30 +0100 > > > >> schrieb Henning Schild via Xenomai <xenomai@xenomai.org>: > > > >> > > > >>> Am Sun, 31 Jan 2021 17:06:21 +0100 > > > >>> schrieb Philippe Gerum via Xenomai <xenomai@xenomai.org>: > > > >>> > > > >>>> The initial port of the Cobalt core to Dovetail/x86 is > > > >>>> available from [1]. Ports to Dovetail/ARM and Dovetail/arm64 > > > >>>> should follow within a couple of weeks. > > > >>>> > > > >>>> So far, latency and switchtest run flawlessly. Most of the > > > >>>> smokey test suite passes successfully, except the GDB test at > > > >>>> the moment. > > > >>>> > > > >>>> How to test this: > > > >>>> > > > >>>> - clone Xenomai from [1], switch to branch > > > >>>> for-upstream/dovetail > > > >>>> - clone the Dovetail tree (v5.10) from [2], switch to branch > > > >>>> dovetail/master > > > >>>> - run scripts/prepare-kernel.sh available from [1] into [2] > > > >>>> (usual procedure) for x86, that would be: > > > >>>> .../scripts/prepare-kernel.sh --arch=x86 > > > >>>> - build your kernel using the sources from [2] > > > >>> > > > >>> I followed this and got > > > >>> > > > >>> 0625b829450dab22b2eea860eef4fb94f64af4fd > > > >>> 61769e49d82a6775f6eb259bdc16c2f84df79505 > > > >>> > > > >>> > > > >>>> There is no user-visible Kconfig change compared to an I-pipe > > > >>>> based version. > > > >>> > > > >>> took a 4.19 x86_64 savedefconfig in via olddefconfig > > > >>> > > > >>>> Alternatively, the Xenomai code base in [1] can also run on > > > >>>> top of the I-pipe. prepare-kernel.sh detects which pipeline > > > >>>> flavour is there, and prepares the source tree accordingly. > > > >>>> > > > >>>> This code is being gradually merged into Xenomai's -next > > > >>>> branch, and will be at the core of Xenomai 3.2. Testing and > > > >>>> feedback appreciated. > > > >>> > > > >>> Now i get a BUG pretty early on during boot. Since my config > > > >>> still seems off i could not yet enter my rootfs, probably some > > > >>> drivers/filesystems/raid switches got lost ... > > > >>> > > > >>> Its a null pointer deref in kmem_cache_alloc, coming somewhere > > > >>> from mempool_init_node > > > >>> > > > >>> First guess was that "numa=off" would hide the problem, seems > > > >>> it does. So my next guess is that even a qemu with numa could > > > >>> reproduce this ... probably almost on a random config ... as > > > >>> long as it has smp+numa > > > >> > > > >> It is not about numa, now also see that on qemu. Jan looks into > > > >> it at the moment. > > > >> > > > > > > > > I'm suspecting an upstream issue. The bug is some freelist > > > > corruption, possibly only surfaced by dovetail. After digging > > > > longer, I enabled CONFIG_SLAB_FREELIST_HARDENED, and that gave > > > > > > > > [ 0.159611] ACPI Error: Could not remove SCI handler > > > > (20210105/evmisc-251) > > > > [ 0.160188] ------------[ cut here ]------------ > > > > [ 0.160566] cache_from_obj: Wrong slab cache. > > > > ftrace_event_field but object is from kmalloc-64 > > > > > > > > and more. However: You get those bugs also from running latest > > > > 5.10.23 and even Linus' master. Will debug that further and then > > > > possibly report upstream. > > > > > > This is a corner case of upstream (but still a bug there): > > > > > > Your config lost CONFIG_PCI, likely because it is no longer > > > default y. That not only generates and unbootable systems, it > > > also sends the ACPI subsystem into an error path. There it seems > > > to release objects to the wrong caches, leading to corruptions > > > later on. Enabling PCI resolves all this. > > > > > > I'll dump the config to upstream ACPI folks, should be their > > > business. > > > > Thanks Jan! > > > > > Jan > > > > > > PS: Your config has more issues as it does not even boot in QEMU, > > > even with PCI enabled. You should (re-)derive from > > > x86_64-defconfig. > > > > Yes that was CI progressing with "olddefconfig" from a 4.19 > > "savedefconfig". No human took a closer look at the result, before > > you. But hey that might fix an upstream bug so it was a good > > exercise anyhow. > > Looking much better with a changed config. AFAIK the gdb test from > smokey is known to fail at the moment. Should that be marked as "skip" > mainline to reduce downstream skipping fixups? Related to that ... smokey --run --keep-going does not keep-going on gdb, one needs smokey --run --exclude gdb --keep-going Henning > Henning > > > Henning > > > > > ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Dovetail] x86 test version available (kernel v5.10) 2021-03-12 19:53 ` Henning Schild @ 2021-03-15 7:57 ` Jan Kiszka 0 siblings, 0 replies; 20+ messages in thread From: Jan Kiszka @ 2021-03-15 7:57 UTC (permalink / raw) To: Henning Schild, Henning Schild via Xenomai On 12.03.21 20:53, Henning Schild wrote: > Am Fri, 12 Mar 2021 20:46:30 +0100 > schrieb Henning Schild <henning.schild@siemens.com>: > >> Am Fri, 12 Mar 2021 19:18:14 +0100 >> schrieb Henning Schild via Xenomai <xenomai@xenomai.org>: >> >>> Am Fri, 12 Mar 2021 12:32:54 +0100 >>> schrieb Jan Kiszka <jan.kiszka@siemens.com>: >>> >>>> On 12.03.21 08:22, Jan Kiszka wrote: >>>>> On 11.03.21 17:35, Henning Schild wrote: >>>>>> Am Mon, 22 Feb 2021 16:35:30 +0100 >>>>>> schrieb Henning Schild via Xenomai <xenomai@xenomai.org>: >>>>>> >>>>>>> Am Sun, 31 Jan 2021 17:06:21 +0100 >>>>>>> schrieb Philippe Gerum via Xenomai <xenomai@xenomai.org>: >>>>>>> >>>>>>>> The initial port of the Cobalt core to Dovetail/x86 is >>>>>>>> available from [1]. Ports to Dovetail/ARM and Dovetail/arm64 >>>>>>>> should follow within a couple of weeks. >>>>>>>> >>>>>>>> So far, latency and switchtest run flawlessly. Most of the >>>>>>>> smokey test suite passes successfully, except the GDB test at >>>>>>>> the moment. >>>>>>>> >>>>>>>> How to test this: >>>>>>>> >>>>>>>> - clone Xenomai from [1], switch to branch >>>>>>>> for-upstream/dovetail >>>>>>>> - clone the Dovetail tree (v5.10) from [2], switch to branch >>>>>>>> dovetail/master >>>>>>>> - run scripts/prepare-kernel.sh available from [1] into [2] >>>>>>>> (usual procedure) for x86, that would be: >>>>>>>> .../scripts/prepare-kernel.sh --arch=x86 >>>>>>>> - build your kernel using the sources from [2] >>>>>>> >>>>>>> I followed this and got >>>>>>> >>>>>>> 0625b829450dab22b2eea860eef4fb94f64af4fd >>>>>>> 61769e49d82a6775f6eb259bdc16c2f84df79505 >>>>>>> >>>>>>> >>>>>>>> There is no user-visible Kconfig change compared to an I-pipe >>>>>>>> based version. >>>>>>> >>>>>>> took a 4.19 x86_64 savedefconfig in via olddefconfig >>>>>>> >>>>>>>> Alternatively, the Xenomai code base in [1] can also run on >>>>>>>> top of the I-pipe. prepare-kernel.sh detects which pipeline >>>>>>>> flavour is there, and prepares the source tree accordingly. >>>>>>>> >>>>>>>> This code is being gradually merged into Xenomai's -next >>>>>>>> branch, and will be at the core of Xenomai 3.2. Testing and >>>>>>>> feedback appreciated. >>>>>>> >>>>>>> Now i get a BUG pretty early on during boot. Since my config >>>>>>> still seems off i could not yet enter my rootfs, probably some >>>>>>> drivers/filesystems/raid switches got lost ... >>>>>>> >>>>>>> Its a null pointer deref in kmem_cache_alloc, coming somewhere >>>>>>> from mempool_init_node >>>>>>> >>>>>>> First guess was that "numa=off" would hide the problem, seems >>>>>>> it does. So my next guess is that even a qemu with numa could >>>>>>> reproduce this ... probably almost on a random config ... as >>>>>>> long as it has smp+numa >>>>>> >>>>>> It is not about numa, now also see that on qemu. Jan looks into >>>>>> it at the moment. >>>>>> >>>>> >>>>> I'm suspecting an upstream issue. The bug is some freelist >>>>> corruption, possibly only surfaced by dovetail. After digging >>>>> longer, I enabled CONFIG_SLAB_FREELIST_HARDENED, and that gave >>>>> >>>>> [ 0.159611] ACPI Error: Could not remove SCI handler >>>>> (20210105/evmisc-251) >>>>> [ 0.160188] ------------[ cut here ]------------ >>>>> [ 0.160566] cache_from_obj: Wrong slab cache. >>>>> ftrace_event_field but object is from kmalloc-64 >>>>> >>>>> and more. However: You get those bugs also from running latest >>>>> 5.10.23 and even Linus' master. Will debug that further and then >>>>> possibly report upstream. >>>> >>>> This is a corner case of upstream (but still a bug there): >>>> >>>> Your config lost CONFIG_PCI, likely because it is no longer >>>> default y. That not only generates and unbootable systems, it >>>> also sends the ACPI subsystem into an error path. There it seems >>>> to release objects to the wrong caches, leading to corruptions >>>> later on. Enabling PCI resolves all this. >>>> >>>> I'll dump the config to upstream ACPI folks, should be their >>>> business. >>> >>> Thanks Jan! >>> >>>> Jan >>>> >>>> PS: Your config has more issues as it does not even boot in QEMU, >>>> even with PCI enabled. You should (re-)derive from >>>> x86_64-defconfig. >>> >>> Yes that was CI progressing with "olddefconfig" from a 4.19 >>> "savedefconfig". No human took a closer look at the result, before >>> you. But hey that might fix an upstream bug so it was a good >>> exercise anyhow. >> >> Looking much better with a changed config. AFAIK the gdb test from >> smokey is known to fail at the moment. Should that be marked as "skip" >> mainline to reduce downstream skipping fixups? > > Related to that ... > > smokey --run --keep-going > > does not keep-going on gdb, one needs > > smokey --run --exclude gdb --keep-going > --keep-going might be valuable on its own, the get a more complete picture. Excluding gdb test for now would be another hack on top of wip/dovetail, yes. Jan -- Siemens AG, T RDA IOT Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2021-03-15 7:57 UTC | newest] Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-01-31 16:06 [Dovetail] x86 test version available (kernel v5.10) Philippe Gerum 2021-02-01 17:32 ` Jan Kiszka 2021-02-03 17:35 ` Philippe Gerum 2021-02-06 12:29 ` Jan Kiszka 2021-02-06 17:23 ` Philippe Gerum 2021-02-15 13:28 ` Jan Kiszka 2021-02-15 15:11 ` Jan Kiszka 2021-02-15 16:53 ` Jan Kiszka 2021-02-15 17:19 ` Jan Kiszka 2021-02-15 17:07 ` Philippe Gerum 2021-02-15 17:21 ` Philippe Gerum 2021-02-22 15:35 ` Henning Schild 2021-02-22 17:59 ` Henning Schild 2021-03-11 16:35 ` Henning Schild 2021-03-12 7:22 ` Jan Kiszka 2021-03-12 11:32 ` Jan Kiszka 2021-03-12 18:18 ` Henning Schild 2021-03-12 19:46 ` Henning Schild 2021-03-12 19:53 ` Henning Schild 2021-03-15 7:57 ` Jan Kiszka
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.