* [LTP] [PATCH 00/36] Remove UCLINUX from LTP @ 2024-01-03 1:52 Petr Vorel 2024-01-03 7:58 ` Cyril Hrubis 2024-01-03 9:46 ` Geert Uytterhoeven 0 siblings, 2 replies; 40+ messages in thread From: Petr Vorel @ 2024-01-03 1:52 UTC (permalink / raw) To: ltp Cc: Jonathan Corbet, Christophe Lyon, Randy Dunlap, linux-kernel, linux-m68k, Geert Uytterhoeven, John Paul Adrian Glaubitz, Greg Ungerer Hi all, UCLINUX is broken in LTP and nobody really cares. Actually I dare to say UCLINUX is dead. Therefore I prepared patchset to remove UCLINUX from LTP. We have been actively removing UCLINUX from LTP during rewrite tests to new LTP API. This removes the rest from the old tests (which will be sooner or later rewritten to new API). Because the patchset is quite big, I did not want to send it to mailing lists (but I can do it if you want). Can you please have look at my fork on gitlab, branch: remove-UCLINUX https://gitlab.com/pevik/ltp/-/commits/remove-UCLINUX?ref_type=heads Build test: https://github.com/pevik/ltp/actions/runs/7392470215 Kind regards, Petr Petr Vorel (36): m4: Remove UCLINUX detection make: Remove WITH_POWER_MANAGEMENT_TESTSUITE make: Remove UCLINUX test.h: Remove MAP_PRIVATE_EXCEPT_UCLINUX tree: Remove FORK_OR_VFORK and tst_vfork() lib/parse_opts.c: Remove UCLINUX tlibio.c: Remove UCLINUX clone02: Remove UCLINUX connect01: Remove UCLINUX creat06: Remove UCLINUX fcntl: Remove UCLINUX getpeername01: Remove UCLINUX getsockname01: Remove UCLINUX getsockopt01: Remove UCLINUX semctl06: Remove UCLINUX kill: Remove UCLINUX madvise02: Remove UCLINUX mlockall: Remove UCLINUX waitpid: Remove UCLINUX munmap: Remove UCLINUX writev05: Remove UCLINUX pipe: Remove UCLINUX pause: Remove UCLINUX recv*: Remove UCLINUX send*: Remove UCLINUX sock*: Remove UCLINUX munlockall01: Remove UCLINUX read02: Remove UCLINUX setgroups04: Remove UCLINUX setsid01: Remove UCLINUX sigrelse01: Remove UCLINUX sysinfo02: Remove UCLINUX ustat02: Remove UCLINUX lib: Remove -C option and self_exec.c Remove doc/nommu-notes.txt doc: UCLINUX has been removed Makefile | 7 - configure.ac | 1 - ...Maintainer-Patch-Review-Checklist.asciidoc | 3 - doc/nommu-notes.txt | 171 ------------- include/mk/env_post.mk | 4 - include/mk/features.mk.in | 11 - include/old/test.h | 21 -- lib/parse_opts.c | 23 +- lib/self_exec.c | 225 ------------------ lib/tlibio.c | 2 +- lib/tst_res.c | 8 - lib/tst_test.c | 15 -- m4/ltp-nommu-linux.m4 | 14 -- runtest/Makefile | 4 - testcases/kernel/Makefile | 7 +- testcases/kernel/syscalls/Makefile | 5 - testcases/kernel/syscalls/access/Makefile | 4 - testcases/kernel/syscalls/clone/clone02.c | 5 - testcases/kernel/syscalls/connect/connect01.c | 17 +- testcases/kernel/syscalls/creat/creat06.c | 6 - testcases/kernel/syscalls/epoll/epoll-ltp.c | 4 +- testcases/kernel/syscalls/exit/exit01.c | 2 +- testcases/kernel/syscalls/fcntl/fcntl07.c | 2 +- testcases/kernel/syscalls/fcntl/fcntl11.c | 16 +- testcases/kernel/syscalls/fcntl/fcntl14.c | 52 +--- testcases/kernel/syscalls/fcntl/fcntl16.c | 29 +-- testcases/kernel/syscalls/fcntl/fcntl17.c | 59 +---- testcases/kernel/syscalls/fcntl/fcntl18.c | 12 +- testcases/kernel/syscalls/fcntl/fcntl19.c | 15 +- testcases/kernel/syscalls/fcntl/fcntl20.c | 16 +- testcases/kernel/syscalls/fcntl/fcntl21.c | 18 +- testcases/kernel/syscalls/fcntl/fcntl22.c | 2 +- .../syscalls/getpeername/getpeername01.c | 2 - .../syscalls/getsockname/getsockname01.c | 3 - .../kernel/syscalls/getsockopt/getsockopt01.c | 2 - testcases/kernel/syscalls/ipc/msgsnd/Makefile | 4 - .../syscalls/ipc/msgstress/msgstress01.c | 4 +- .../syscalls/ipc/msgstress/msgstress02.c | 6 +- .../syscalls/ipc/msgstress/msgstress03.c | 4 +- .../syscalls/ipc/msgstress/msgstress04.c | 6 +- .../kernel/syscalls/ipc/semctl/semctl06.c | 9 +- testcases/kernel/syscalls/kill/kill02.c | 101 +------- testcases/kernel/syscalls/kill/kill07.c | 12 +- testcases/kernel/syscalls/kill/kill08.c | 15 +- testcases/kernel/syscalls/kill/kill09.c | 13 +- testcases/kernel/syscalls/kill/kill12.c | 2 +- testcases/kernel/syscalls/madvise/madvise02.c | 25 +- .../kernel/syscalls/mlockall/mlockall01.c | 12 - .../kernel/syscalls/mlockall/mlockall02.c | 12 - .../kernel/syscalls/mlockall/mlockall03.c | 12 - .../kernel/syscalls/modify_ldt/modify_ldt02.c | 2 +- .../kernel/syscalls/mprotect/mprotect02.c | 4 +- .../kernel/syscalls/mprotect/mprotect03.c | 2 +- .../kernel/syscalls/munlockall/munlockall01.c | 12 - testcases/kernel/syscalls/munmap/munmap01.c | 18 +- testcases/kernel/syscalls/munmap/munmap02.c | 18 -- testcases/kernel/syscalls/munmap/munmap03.c | 3 +- testcases/kernel/syscalls/pause/pause02.c | 11 +- testcases/kernel/syscalls/pause/pause03.c | 13 +- testcases/kernel/syscalls/pipe/pipe02.c | 9 - testcases/kernel/syscalls/pipe/pipe04.c | 23 +- testcases/kernel/syscalls/pipe/pipe09.c | 4 +- testcases/kernel/syscalls/read/read02.c | 4 - testcases/kernel/syscalls/recv/recv01.c | 19 +- .../kernel/syscalls/recvfrom/recvfrom01.c | 17 +- testcases/kernel/syscalls/rename/rename14.c | 4 +- testcases/kernel/syscalls/send/send01.c | 23 +- testcases/kernel/syscalls/sendmsg/sendmsg01.c | 16 +- testcases/kernel/syscalls/sendto/sendto01.c | 23 +- .../kernel/syscalls/setfsuid/setfsuid04.c | 4 +- .../kernel/syscalls/setgroups/setgroups04.c | 12 - testcases/kernel/syscalls/setpgid/setpgid01.c | 2 +- testcases/kernel/syscalls/setpgrp/setpgrp01.c | 2 +- testcases/kernel/syscalls/setpgrp/setpgrp02.c | 2 +- .../kernel/syscalls/setrlimit/setrlimit01.c | 6 +- testcases/kernel/syscalls/setsid/setsid01.c | 29 +-- .../kernel/syscalls/sigrelse/sigrelse01.c | 20 +- .../kernel/syscalls/socketpair/socketpair01.c | 2 - .../kernel/syscalls/sockioctl/sockioctl01.c | 2 - testcases/kernel/syscalls/sysinfo/sysinfo02.c | 12 - testcases/kernel/syscalls/ustat/ustat02.c | 2 - testcases/kernel/syscalls/waitpid/waitpid02.c | 10 +- testcases/kernel/syscalls/waitpid/waitpid03.c | 28 +-- testcases/kernel/syscalls/waitpid/waitpid04.c | 2 +- testcases/kernel/syscalls/waitpid/waitpid05.c | 28 +-- testcases/kernel/syscalls/writev/Makefile | 4 - testcases/kernel/syscalls/writev/writev02.c | 3 +- testcases/kernel/syscalls/writev/writev05.c | 15 +- testcases/kernel/syscalls/writev/writev06.c | 8 +- 89 files changed, 105 insertions(+), 1337 deletions(-) delete mode 100644 doc/nommu-notes.txt delete mode 100644 lib/self_exec.c delete mode 100644 m4/ltp-nommu-linux.m4 -- 2.43.0 -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [LTP] [PATCH 00/36] Remove UCLINUX from LTP 2024-01-03 1:52 [LTP] [PATCH 00/36] Remove UCLINUX from LTP Petr Vorel @ 2024-01-03 7:58 ` Cyril Hrubis 2024-01-03 8:04 ` Cyril Hrubis 2024-01-03 9:46 ` Geert Uytterhoeven 1 sibling, 1 reply; 40+ messages in thread From: Cyril Hrubis @ 2024-01-03 7:58 UTC (permalink / raw) To: Petr Vorel Cc: Jonathan Corbet, Christophe Lyon, Randy Dunlap, linux-kernel, linux-m68k, Geert Uytterhoeven, John Paul Adrian Glaubitz, Greg Ungerer, ltp Hi! > UCLINUX is broken in LTP and nobody really cares. Actually I dare to > say UCLINUX is dead. Therefore I prepared patchset to remove UCLINUX > from LTP. We have been actively removing UCLINUX from LTP during rewrite > tests to new LTP API. This removes the rest from the old tests (which > will be sooner or later rewritten to new API). > > Because the patchset is quite big, I did not want to send it to mailing > lists (but I can do it if you want). I agree that this should be done, but I'm not sure if we want to get this in before the January release. I guess that such change would be safer to merge just after the release so that we have a few months to actually catch possible problems. -- Cyril Hrubis chrubis@suse.cz -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [LTP] [PATCH 00/36] Remove UCLINUX from LTP 2024-01-03 7:58 ` Cyril Hrubis @ 2024-01-03 8:04 ` Cyril Hrubis 2024-01-03 8:39 ` Petr Vorel 0 siblings, 1 reply; 40+ messages in thread From: Cyril Hrubis @ 2024-01-03 8:04 UTC (permalink / raw) To: Petr Vorel Cc: Jonathan Corbet, Christophe Lyon, Randy Dunlap, linux-kernel, linux-m68k, Geert Uytterhoeven, John Paul Adrian Glaubitz, Greg Ungerer, ltp Hi! > > UCLINUX is broken in LTP and nobody really cares. Actually I dare to > > say UCLINUX is dead. Therefore I prepared patchset to remove UCLINUX > > from LTP. We have been actively removing UCLINUX from LTP during rewrite > > tests to new LTP API. This removes the rest from the old tests (which > > will be sooner or later rewritten to new API). > > > > Because the patchset is quite big, I did not want to send it to mailing > > lists (but I can do it if you want). > > I agree that this should be done, but I'm not sure if we want to get > this in before the January release. I guess that such change would be > safer to merge just after the release so that we have a few months to > actually catch possible problems. Looking at the actuall changes it does not look awfuly complex, so maybe we can try to merge it before the pre-release testing and hopefully things will not break badly. -- Cyril Hrubis chrubis@suse.cz -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [LTP] [PATCH 00/36] Remove UCLINUX from LTP 2024-01-03 8:04 ` Cyril Hrubis @ 2024-01-03 8:39 ` Petr Vorel 0 siblings, 0 replies; 40+ messages in thread From: Petr Vorel @ 2024-01-03 8:39 UTC (permalink / raw) To: Cyril Hrubis Cc: Jonathan Corbet, Christophe Lyon, Randy Dunlap, linux-kernel, linux-m68k, Geert Uytterhoeven, John Paul Adrian Glaubitz, Greg Ungerer, ltp Hi Cyril, > Hi! > > > UCLINUX is broken in LTP and nobody really cares. Actually I dare to > > > say UCLINUX is dead. Therefore I prepared patchset to remove UCLINUX > > > from LTP. We have been actively removing UCLINUX from LTP during rewrite > > > tests to new LTP API. This removes the rest from the old tests (which > > > will be sooner or later rewritten to new API). > > > Because the patchset is quite big, I did not want to send it to mailing > > > lists (but I can do it if you want). > > I agree that this should be done, but I'm not sure if we want to get > > this in before the January release. I guess that such change would be > > safer to merge just after the release so that we have a few months to > > actually catch possible problems. > Looking at the actuall changes it does not look awfuly complex, so maybe > we can try to merge it before the pre-release testing and hopefully > things will not break badly. Thanks for a quick look. Both ways would work for me, depends on you and others. Obviously fewer rebasing is better :). Kind regards, Petr -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [LTP] [PATCH 00/36] Remove UCLINUX from LTP 2024-01-03 1:52 [LTP] [PATCH 00/36] Remove UCLINUX from LTP Petr Vorel 2024-01-03 7:58 ` Cyril Hrubis @ 2024-01-03 9:46 ` Geert Uytterhoeven 2024-01-03 11:49 ` Petr Vorel 2024-01-05 3:50 ` Rob Landley 1 sibling, 2 replies; 40+ messages in thread From: Geert Uytterhoeven @ 2024-01-03 9:46 UTC (permalink / raw) To: Petr Vorel Cc: Jonathan Corbet, Linux-sh list, Christophe Lyon, Randy Dunlap, linux-kernel, linux-m68k, Linux ARM, John Paul Adrian Glaubitz, linux-riscv, Greg Ungerer, ltp Hi Petr, CC other uClinux arch lists On Wed, Jan 3, 2024 at 2:52 AM Petr Vorel <pvorel@suse.cz> wrote: > UCLINUX is broken in LTP and nobody really cares. Actually I dare to > say UCLINUX is dead. Therefore I prepared patchset to remove UCLINUX > from LTP. We have been actively removing UCLINUX from LTP during rewrite > tests to new LTP API. This removes the rest from the old tests (which > will be sooner or later rewritten to new API). > > Because the patchset is quite big, I did not want to send it to mailing > lists (but I can do it if you want). > > Can you please have look at my fork on gitlab, branch: remove-UCLINUX > https://gitlab.com/pevik/ltp/-/commits/remove-UCLINUX?ref_type=heads > > Build test: > https://github.com/pevik/ltp/actions/runs/7392470215 Thanks for your series! I see you only CCed linux-m68k, but AFAIK, uClinux is not restricted to m68k/coldfire, but also available on arm32, riscv, sh, and xtensa. Gr{oetje,eeting}s, Geert -- Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [LTP] [PATCH 00/36] Remove UCLINUX from LTP 2024-01-03 9:46 ` Geert Uytterhoeven @ 2024-01-03 11:49 ` Petr Vorel 2024-01-03 11:54 ` Geert Uytterhoeven 2024-01-05 3:50 ` Rob Landley 1 sibling, 1 reply; 40+ messages in thread From: Petr Vorel @ 2024-01-03 11:49 UTC (permalink / raw) To: Geert Uytterhoeven Cc: Jonathan Corbet, Linux-sh list, Christophe Lyon, Randy Dunlap, linux-kernel, linux-m68k, Linux ARM, John Paul Adrian Glaubitz, linux-riscv, Greg Ungerer, ltp Hi Geert, > Hi Petr, > CC other uClinux arch lists > On Wed, Jan 3, 2024 at 2:52 AM Petr Vorel <pvorel@suse.cz> wrote: > > UCLINUX is broken in LTP and nobody really cares. Actually I dare to > > say UCLINUX is dead. Therefore I prepared patchset to remove UCLINUX > > from LTP. We have been actively removing UCLINUX from LTP during rewrite > > tests to new LTP API. This removes the rest from the old tests (which > > will be sooner or later rewritten to new API). > > Because the patchset is quite big, I did not want to send it to mailing > > lists (but I can do it if you want). > > Can you please have look at my fork on gitlab, branch: remove-UCLINUX > > https://gitlab.com/pevik/ltp/-/commits/remove-UCLINUX?ref_type=heads > > Build test: > > https://github.com/pevik/ltp/actions/runs/7392470215 > Thanks for your series! Thank you for your feedback. May I add your Acked-by: tag to the series when we agree to merge? > I see you only CCed linux-m68k, but AFAIK, uClinux is not restricted > to m68k/coldfire, but also available on arm32, riscv, sh, and xtensa. Good point, I'll reply to their lists as well. Kind regards, Petr > Gr{oetje,eeting}s, > Geert -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [LTP] [PATCH 00/36] Remove UCLINUX from LTP 2024-01-03 11:49 ` Petr Vorel @ 2024-01-03 11:54 ` Geert Uytterhoeven 2024-01-03 12:09 ` Cyril Hrubis 0 siblings, 1 reply; 40+ messages in thread From: Geert Uytterhoeven @ 2024-01-03 11:54 UTC (permalink / raw) To: Petr Vorel Cc: Jonathan Corbet, Linux-sh list, Christophe Lyon, Randy Dunlap, linux-kernel, linux-m68k, Linux ARM, John Paul Adrian Glaubitz, linux-riscv, Greg Ungerer, ltp Hi Petr, On Wed, Jan 3, 2024 at 12:50 PM Petr Vorel <pvorel@suse.cz> wrote: > > On Wed, Jan 3, 2024 at 2:52 AM Petr Vorel <pvorel@suse.cz> wrote: > > > UCLINUX is broken in LTP and nobody really cares. Actually I dare to > > > say UCLINUX is dead. Therefore I prepared patchset to remove UCLINUX > > > from LTP. We have been actively removing UCLINUX from LTP during rewrite > > > tests to new LTP API. This removes the rest from the old tests (which > > > will be sooner or later rewritten to new API). > > > > Because the patchset is quite big, I did not want to send it to mailing > > > lists (but I can do it if you want). > > > > Can you please have look at my fork on gitlab, branch: remove-UCLINUX > > > https://gitlab.com/pevik/ltp/-/commits/remove-UCLINUX?ref_type=heads > > > > Build test: > > > https://github.com/pevik/ltp/actions/runs/7392470215 > > > Thanks for your series! > > Thank you for your feedback. May I add your Acked-by: tag to the series when we > agree to merge? I am not sure I agree with this series. Removing support for UCLINUX from LTP is almost a guarantee for not noticing when more breakage is introduced. How exactly is UCLINUX broken in LTP? Thanks! Gr{oetje,eeting}s, Geert -- Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [LTP] [PATCH 00/36] Remove UCLINUX from LTP 2024-01-03 11:54 ` Geert Uytterhoeven @ 2024-01-03 12:09 ` Cyril Hrubis 2024-01-03 12:40 ` Petr Vorel 2024-01-05 3:52 ` Rob Landley 0 siblings, 2 replies; 40+ messages in thread From: Cyril Hrubis @ 2024-01-03 12:09 UTC (permalink / raw) To: Geert Uytterhoeven Cc: Jonathan Corbet, Linux-sh list, Christophe Lyon, Randy Dunlap, linux-kernel, linux-m68k, Linux ARM, John Paul Adrian Glaubitz, linux-riscv, Greg Ungerer, ltp Hi! > I am not sure I agree with this series. > Removing support for UCLINUX from LTP is almost a guarantee for > not noticing when more breakage is introduced. > > How exactly is UCLINUX broken in LTP? As far as we know noone is using it and nobody is maintaing it for a decade, so it's bitrotting and we do not have manpower to fix it, or rather we do not want to invest the scarcely limited resources we have into something that is niche at best. We asked repeatedly if anyone want to invest time into keeping it alive, but nobody answered the call so far and I doubt that it will happen at this point. -- Cyril Hrubis chrubis@suse.cz -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [LTP] [PATCH 00/36] Remove UCLINUX from LTP 2024-01-03 12:09 ` Cyril Hrubis @ 2024-01-03 12:40 ` Petr Vorel 2024-01-05 3:52 ` Rob Landley 1 sibling, 0 replies; 40+ messages in thread From: Petr Vorel @ 2024-01-03 12:40 UTC (permalink / raw) To: Cyril Hrubis Cc: Jonathan Corbet, Linux-sh list, Christophe Lyon, Randy Dunlap, linux-kernel, linux-m68k, Geert Uytterhoeven, Linux ARM, John Paul Adrian Glaubitz, linux-riscv, Greg Ungerer, ltp Hi Geert, Cyril, all, Geert, first, thank you for Cc all the other lists. For anybody from those lists, we talk about: https://lore.kernel.org/ltp/20240103015240.1065284-1-pvorel@suse.cz/ > Hi! > > I am not sure I agree with this series. > > Removing support for UCLINUX from LTP is almost a guarantee for > > not noticing when more breakage is introduced. > > How exactly is UCLINUX broken in LTP? > As far as we know noone is using it and nobody is maintaing it for a > decade, so it's bitrotting and we do not have manpower to fix it, or > rather we do not want to invest the scarcely limited resources we have > into something that is niche at best. We asked repeatedly if anyone want > to invest time into keeping it alive, but nobody answered the call so > far and I doubt that it will happen at this point. Also, UCLINUX was used in tests which used the legacy LTP API, which was buggy. We slowly rewrite tests into new API [1], which is more reliable and do cleanup and bug fixes during test rewrites. But because nobody stand to maintain UCLINUX support, it's not in the new API. Thus we have actively deleted it's support during the rewrite in past years. I wonder myself if anybody is even using LTP on UCLINUX platforms. Nearly 25% of the syscalls tests use fork(), thus will not work on UCLINUX. First tests were rewritten in 2016 (first release in 20160510) and nobody complained. All tests C based tests (both new and legacy API): $ git grep -l -e 'include .tst_test.h' -e 'include .test.h' testcases/ |wc -l 1494 Tests, which use fork(), i.e. not working in UCLINUX: $ git grep -l '\.forks_child.*1' testcases/ |wc -l 334 Kind regards, Petr [1] https://github.com/linux-test-project/ltp/wiki/C-Test-API -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [LTP] [PATCH 00/36] Remove UCLINUX from LTP 2024-01-03 12:09 ` Cyril Hrubis 2024-01-03 12:40 ` Petr Vorel @ 2024-01-05 3:52 ` Rob Landley 2024-01-05 13:11 ` [LTP] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP] Petr Vorel ` (2 more replies) 1 sibling, 3 replies; 40+ messages in thread From: Rob Landley @ 2024-01-05 3:52 UTC (permalink / raw) To: Cyril Hrubis, Geert Uytterhoeven Cc: Jonathan Corbet, Linux-sh list, Christophe Lyon, Randy Dunlap, linux-kernel, linux-m68k, Linux ARM, John Paul Adrian Glaubitz, linux-riscv, Greg Ungerer, ltp On 1/3/24 06:09, Cyril Hrubis wrote: > Hi! >> I am not sure I agree with this series. >> Removing support for UCLINUX from LTP is almost a guarantee for >> not noticing when more breakage is introduced. >> >> How exactly is UCLINUX broken in LTP? > > As far as we know noone is using it and nobody is maintaing it for a > decade, Nobody is maintaining "uclinux" because that was a distro, but you can build nommu support in buildroot and such, and people do. Rob -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 40+ messages in thread
* [LTP] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP] 2024-01-05 3:52 ` Rob Landley @ 2024-01-05 13:11 ` Petr Vorel 2024-01-06 3:58 ` Rob Landley 2024-01-08 8:33 ` [LTP] [PATCH 00/36] Remove UCLINUX from LTP Andrea Cervesato via ltp 2024-01-08 8:34 ` Andrea Cervesato via ltp 2 siblings, 1 reply; 40+ messages in thread From: Petr Vorel @ 2024-01-05 13:11 UTC (permalink / raw) To: Rob Landley Cc: Jonathan Corbet, Linux-sh list, Christophe Lyon, Randy Dunlap, linux-kernel, linux-m68k, Geert Uytterhoeven, Linux ARM, John Paul Adrian Glaubitz, linux-riscv, buildroot, Greg Ungerer, ltp, automated-testing Hi all, [ Cc also automated-testing and buildroot ML FYI thread started here: https://lore.kernel.org/ltp/20240103015240.1065284-1-pvorel@suse.cz/ ] > On 1/3/24 06:09, Cyril Hrubis wrote: > > Hi! > >> I am not sure I agree with this series. > >> Removing support for UCLINUX from LTP is almost a guarantee for > >> not noticing when more breakage is introduced. > >> How exactly is UCLINUX broken in LTP? > > As far as we know noone is using it and nobody is maintaing it for a > > decade, > Nobody is maintaining "uclinux" because that was a distro, but you can build > nommu support in buildroot and such, and people do. Right, there are nommu users. Will anybody join LTP development to maintain nommu support in LTP? The needed work is to add this support to LTP new C API [1] and use it in the relevant test. There is some implementation in the old API, I have no idea how well it worked. If nobody stands for maintaing nommu, we will have to delete it. There is nobody from the current maintainers who is using LTP on nommu HW (that is the reason why nommu support have not been implemented in the new API). Kind regards, Petr > Rob [1] https://github.com/linux-test-project/ltp/wiki/C-Test-API -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [LTP] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP] 2024-01-05 13:11 ` [LTP] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP] Petr Vorel @ 2024-01-06 3:58 ` Rob Landley 2024-01-08 9:03 ` Petr Vorel 0 siblings, 1 reply; 40+ messages in thread From: Rob Landley @ 2024-01-06 3:58 UTC (permalink / raw) To: Petr Vorel Cc: Jonathan Corbet, Linux-sh list, Christophe Lyon, Randy Dunlap, linux-kernel, linux-m68k, Geert Uytterhoeven, Linux ARM, John Paul Adrian Glaubitz, linux-riscv, buildroot, Greg Ungerer, ltp, automated-testing On 1/5/24 07:11, Petr Vorel wrote: >> Nobody is maintaining "uclinux" because that was a distro, but you can build >> nommu support in buildroot and such, and people do. > > Right, there are nommu users. Will anybody join LTP development to maintain > nommu support in LTP? The needed work is to add this support to LTP new C API > [1] and use it in the relevant test. There is some implementation in the old > API, I have no idea how well it worked. > > If nobody stands for maintaing nommu, we will have to delete it. There is nobody > from the current maintainers who is using LTP on nommu HW (that is the reason why > nommu support have not been implemented in the new API). I'm interested, but overwhelmed. Not sure I've got the spoons to come up to speed on a new project and give it regular attention just now. I see you cc'd buildroot (although the message might not go through if you aren't subscribed, dunno how clogged their moderation queue is these days, and the cc: list is long enough it might twig anyway). They had a nommu fix go in earlier this week (commit 98684ba7885b). That said, qemu supports several nommu platforms and buildroot has defconfigs to build systems for them: $ git clone git://buildroot.org/buildroot $ make help $ make list-defconfigs | grep qemu $ make qemu_ppc_bamboo_defconfig $ make (time passes...) >>> host-gettext-tiny 0.3.2 Extracting gzip -d -c /home/landley/buildroot/buildroot/dl/gettext-tiny/gettext-tiny-0.3.2.tar.gz | tar --strip-components=1 -C /home/landley/buildroot/buildroot/output/build/host-gettext-tiny-0.3.2 -xf - mkdir -p /home/landley/buildroot/buildroot/output/build/host-gettext-tiny-0.3.2/gettext-gnu xzcat /home/landley/buildroot/buildroot/dl/gettext-tiny/gettext-0.22.4.tar.xz | tar --strip-components=1 -C /home/landley/buildroot/buildroot/output/build/host-gettext-tiny-0.3.2/gettext-gnu -xf - xzcat: /home/landley/buildroot/buildroot/dl/gettext-tiny/gettext-0.22.4.tar.xz: No such file or directory tar: This does not look like a tar archive tar: Exiting with failure status due to previous errors make: *** [package/pkg-generic.mk:209: /home/landley/buildroot/buildroot/output/build/host-gettext-tiny-0.3.2/.stamp_extracted] Error 2 Sigh, never build git pull du jour of anything, buildroot's having glitch du jour. But the point is: $ grep -rl bamboo board/ board/qemu/ppc-bamboo/readme.txt $ cat board/qemu/ppc-bamboo/readme.txt Run the emulation with: qemu-system-ppc -nographic -M bamboo -kernel output/images/vmlinux -net nic,model=virtio-net-pci -net user # qemu_ppc_bamboo_defconfig The login prompt will appear in the terminal that started Qemu ------------------- In THEORY, once it builds an image (presumably using a tagged release version rather than expecting "continuous integration" to ever mean anything) you should be able to launch it with qemu. Assuming the instructions aren't also bit-rotted. (Or using one of the other nommu boards, I haven't gone through the whole list to see what they've got. I used to use a nommu arm board, but the linux kernel broke it when converting everything to device tree and not regression testing it.) Buildroot also apparently has an LTP package selectable in menuconfig: https://github.com/buildroot/buildroot/tree/master/package/ltp-testsuite But I haven't tried it... Rob P.S. I automate qemu testing all the time over in toybox, see testroot.sh under https://github.com/landley/toybox/tree/master/mkroot for an example. -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [LTP] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP] 2024-01-06 3:58 ` Rob Landley @ 2024-01-08 9:03 ` Petr Vorel 2024-01-08 10:07 ` Cyril Hrubis 2024-01-09 20:24 ` [LTP] " Rob Landley 0 siblings, 2 replies; 40+ messages in thread From: Petr Vorel @ 2024-01-08 9:03 UTC (permalink / raw) To: Rob Landley Cc: Niklas Cassel, Jonathan Corbet, Linux-sh list, Christophe Lyon, Randy Dunlap, linux-kernel, linux-m68k, Geert Uytterhoeven, Linux ARM, John Paul Adrian Glaubitz, linux-riscv, buildroot, Greg Ungerer, ltp, automated-testing Hi Rob, all, [ Added Niklas Cassel, who is maintainer of qemu_riscv64_nommu_virt_defconfig in buildroot ] > On 1/5/24 07:11, Petr Vorel wrote: > >> Nobody is maintaining "uclinux" because that was a distro, but you can build > >> nommu support in buildroot and such, and people do. > > Right, there are nommu users. Will anybody join LTP development to maintain > > nommu support in LTP? The needed work is to add this support to LTP new C API > > [1] and use it in the relevant test. There is some implementation in the old > > API, I have no idea how well it worked. > > If nobody stands for maintaing nommu, we will have to delete it. There is nobody > > from the current maintainers who is using LTP on nommu HW (that is the reason why > > nommu support have not been implemented in the new API). > I'm interested, but overwhelmed. Not sure I've got the spoons to come up to > speed on a new project and give it regular attention just now. > I see you cc'd buildroot (although the message might not go through if you > aren't subscribed, dunno how clogged their moderation queue is these days, and > the cc: list is long enough it might twig anyway). They had a nommu fix go in > earlier this week (commit 98684ba7885b). > That said, qemu supports several nommu platforms and buildroot has defconfigs to > build systems for them: > $ git clone git://buildroot.org/buildroot > $ make help > $ make list-defconfigs | grep qemu > $ make qemu_ppc_bamboo_defconfig > $ make > (time passes...) > >>> host-gettext-tiny 0.3.2 Extracting > gzip -d -c > /home/landley/buildroot/buildroot/dl/gettext-tiny/gettext-tiny-0.3.2.tar.gz | > tar --strip-components=1 -C > /home/landley/buildroot/buildroot/output/build/host-gettext-tiny-0.3.2 -xf - > mkdir -p > /home/landley/buildroot/buildroot/output/build/host-gettext-tiny-0.3.2/gettext-gnu > xzcat /home/landley/buildroot/buildroot/dl/gettext-tiny/gettext-0.22.4.tar.xz | > tar --strip-components=1 -C > /home/landley/buildroot/buildroot/output/build/host-gettext-tiny-0.3.2/gettext-gnu > -xf - > xzcat: /home/landley/buildroot/buildroot/dl/gettext-tiny/gettext-0.22.4.tar.xz: > No such file or directory > tar: This does not look like a tar archive > tar: Exiting with failure status due to previous errors > make: *** [package/pkg-generic.mk:209: > /home/landley/buildroot/buildroot/output/build/host-gettext-tiny-0.3.2/.stamp_extracted] > Error 2 > Sigh, never build git pull du jour of anything, buildroot's having glitch du > jour. But the point is: > $ grep -rl bamboo board/ > board/qemu/ppc-bamboo/readme.txt > $ cat board/qemu/ppc-bamboo/readme.txt > Run the emulation with: > qemu-system-ppc -nographic -M bamboo -kernel output/images/vmlinux -net > nic,model=virtio-net-pci -net user # qemu_ppc_bamboo_defconfig > The login prompt will appear in the terminal that started Qemu > ------------------- > In THEORY, once it builds an image (presumably using a tagged release version > rather than expecting "continuous integration" to ever mean anything) you should > be able to launch it with qemu. Assuming the instructions aren't also > bit-rotted. (Or using one of the other nommu boards, I haven't gone through the > whole list to see what they've got. I used to use a nommu arm board, but the > linux kernel broke it when converting everything to device tree and not > regression testing it.) > Buildroot also apparently has an LTP package selectable in menuconfig: > https://github.com/buildroot/buildroot/tree/master/package/ltp-testsuite > But I haven't tried it... I'm the maintainer of the LTP package in buildroot in my private time. BTW I spent quite a lot of time fixing LTP (and some other system packages, e.g. nfs-utils) compilation on some old legacy architectures reported via http://autobuild.buildroot.net/ I've never used in the reality. But I certainly don't have time to drive nommu support in my private time. I don't even have an interest, I don't use any nommu device. And I would not justify to work on nommu in my working paid by SUSE, because that's not a platform SUSE uses. Lack of resources means that there is a vast majority of new kernel functionality not being tested. Also with very small resources it's hard to even fix existing tests broken by changed functionality in each mainline kernel release. Therefore nobody who is not involved in nommu will not find a time to support it in LTP (support does not mean just to add the functionality to the new C API, but run tests on nommu and fix failing bugs). I suppose nobody is paid to work on nommu platforms, it would have to be a hobby project, right? But as I said, if anybody from nommu decides to maintain it in LTP, I'll try to support him in my free time (review patches, give advices). And if nobody stands, this patchset which removes the support in the old API will be merged after next LTP release (in the end of January). Kind regards, Petr > Rob > P.S. I automate qemu testing all the time over in toybox, see testroot.sh under > https://github.com/landley/toybox/tree/master/mkroot for an example. -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [LTP] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP] 2024-01-08 9:03 ` Petr Vorel @ 2024-01-08 10:07 ` Cyril Hrubis 2024-01-09 22:37 ` [LTP] [Automated-testing] " Bird, Tim 2024-01-09 20:24 ` [LTP] " Rob Landley 1 sibling, 1 reply; 40+ messages in thread From: Cyril Hrubis @ 2024-01-08 10:07 UTC (permalink / raw) To: Petr Vorel Cc: Niklas Cassel, Jonathan Corbet, Linux-sh list, Christophe Lyon, Randy Dunlap, linux-kernel, linux-m68k, Geert Uytterhoeven, Linux ARM, Rob Landley, John Paul Adrian Glaubitz, linux-riscv, buildroot, Greg Ungerer, ltp, automated-testing Hi! > But as I said, if anybody from nommu decides to maintain it in LTP, I'll try to > support him in my free time (review patches, give advices). And if nobody > stands, this patchset which removes the support in the old API will be merged > after next LTP release (in the end of January). Let me highlight this part, we are eager to help anybody who is willing to pick the nommu work, but we do not have resources to drive it. -- Cyril Hrubis chrubis@suse.cz -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [LTP] [Automated-testing] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP] 2024-01-08 10:07 ` Cyril Hrubis @ 2024-01-09 22:37 ` Bird, Tim 2024-01-10 5:01 ` Rob Landley 2024-01-10 14:14 ` Petr Vorel 0 siblings, 2 replies; 40+ messages in thread From: Bird, Tim @ 2024-01-09 22:37 UTC (permalink / raw) To: Cyril Hrubis, Petr Vorel Cc: Niklas Cassel, Jonathan Corbet, Linux-sh list, Christophe Lyon, Randy Dunlap, linux-kernel, linux-m68k, Geert Uytterhoeven, Linux ARM, Rob Landley, John Paul Adrian Glaubitz, linux-riscv, buildroot, Greg Ungerer, ltp, automated-testing > -----Original Message----- > From: automated-testing@lists.yoctoproject.org <automated-testing@lists.yoctoproject.org> On Behalf Of Cyril Hrubis > Hi! > > But as I said, if anybody from nommu decides to maintain it in LTP, I'll try to > > support him in my free time (review patches, give advices). And if nobody > > stands, this patchset which removes the support in the old API will be merged > > after next LTP release (in the end of January). > > Let me highlight this part, we are eager to help anybody who is willing > to pick the nommu work, but we do not have resources to drive it. I have a couple of comments here. I think it would be good to give a little bit more time to try to find a helper/maintainer for this. As Rob pointed out, a lot of embedded Linux developers are using very old kernels (and, if they are using LTP, likely very old versions of LTP). They are also notorious for not being active on the mailing lists. So this might take some active outreach to find helpers. (I realize that this thread is part of this outreach effort). For this reason, I'd like a few more weeks to try to advertise this need within the embedded Linux community. I am not using nommu systems myself, so I'm in a similar position as Petr in terms of it not making much sense for me to be the maintainer. However, having said that, I have had for a few years now an idea for a background project related to LTP that might make this a more interesting fit for me. Sony uses NuttX, and is considering using Zephyr in some of our low-end processor systems. This includes some nommu systems. For some time now, I have wanted to experiment with using LTP to test the compatibility of those systems with the Linux system APIs. In full disclosure, I have no idea if this is a feasible or useful idea or not. But it's something I'd like to investigate. I realize that testing non-Linux RTOSes is out-of-scope for LTP. But given that that is something I would like to do, and that it might be relevant to the Linux nommu tests, I would humbly request a few weeks to investigate this before the nommu code is removed. This delay would be to see if it would make sense for me to volunteer to help out with maintaining this otherwise abandoned code. I can't promise anything, but I'd like to find out more about: 1) what parts of the current LTP are not supporting nommu (what's currently broken), 2) how much code we're talking about, and 3) what the desired roadmap going forward would be, to continue to support this code. Thanks, -- Tim -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [LTP] [Automated-testing] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP] 2024-01-09 22:37 ` [LTP] [Automated-testing] " Bird, Tim @ 2024-01-10 5:01 ` Rob Landley 2024-01-10 14:14 ` Petr Vorel 1 sibling, 0 replies; 40+ messages in thread From: Rob Landley @ 2024-01-10 5:01 UTC (permalink / raw) To: Bird, Tim, Cyril Hrubis, Petr Vorel Cc: Niklas Cassel, Jonathan Corbet, Linux-sh list, Christophe Lyon, Randy Dunlap, linux-kernel, linux-m68k, Geert Uytterhoeven, Linux ARM, John Paul Adrian Glaubitz, linux-riscv, buildroot, Greg Ungerer, ltp, automated-testing On 1/9/24 16:37, Bird, Tim wrote: >> -----Original Message----- >> From: automated-testing@lists.yoctoproject.org <automated-testing@lists.yoctoproject.org> On Behalf Of Cyril Hrubis >> Hi! >> > But as I said, if anybody from nommu decides to maintain it in LTP, I'll try to >> > support him in my free time (review patches, give advices). And if nobody >> > stands, this patchset which removes the support in the old API will be merged >> > after next LTP release (in the end of January). >> >> Let me highlight this part, we are eager to help anybody who is willing >> to pick the nommu work, but we do not have resources to drive it. > > I have a couple of comments here. > > I think it would be good to give a little bit more time to try to find a helper/maintainer > for this. As Rob pointed out, a lot of embedded Linux developers are using very old > kernels (and, if they are using LTP, likely very old versions of LTP). They are also > notorious for not being active on the mailing lists. So this might take some active > outreach to find helpers. (I realize that this thread is part of this > outreach effort). For this reason, I'd like a few more weeks to try to advertise this > need within the embedded Linux community. I'd like to point out I have the _interest_ in doing this, and might have some ability, but what I don't have is the spare bandwidth. I maintain toybox, which Android is using as the command line for both its installed systems and its build system (which they call "hermetic" builds, shipping their own build dependencies). I'm trying to get Android to build under android, which is a slightly heavier lift than turning busybox into a development environment capable of building Linux From Scratch starting from just 7 packages (Linux, busybox, uclibc, make, gcc, binutils, and bash) was. Which I _succeeded_ at some years ago, by the way: https://landley.net/aboriginal/about.html The development work I was doing on busybox was why I wound up maintaining that project for a bit, and after I got it to work (the project's 1.0 release) other projects like Alpine and Adelie Linux took it from there. Unfortunately, this time due to the FSF's spectacular stupidity with GPLv3, the Android trademark licensing guidelines do not allow adding any GPL code in userspace beyond what was grandfathered in circa 2007. (Busybox predates android, yet android does not ship busybox. There's a reason for that.) Even those grandfathered in ones they've been steadily replacing (rewriting the bluetooth daemon with an apache licensed version, switching the build from gcc to llvm, replacing gnu make with kati, and so on...) So I couldn't use ANY existing gpl code (like busybox could) and largely had to write a new one of everything. (Android was using a lot of bsd implementations, but have you ever tried to build the linux kernel with bsd sed or bsd make? Doesn't quite fit.) Anyway, I'm most of the way done now (see http://landley.net/toybox/status.html and https://landley.net/toybox/roadmap.html#dev_env) and almost all my toybox stuff already supports nommu. I'm even writing a bash replacement shell (handling all the <(bash/{weirdness}/{1..7}) I can manage) that has full nommu support. To support subshells it does a vfork() and exec of itself, then marshalls all the local variable and function state across a pipe to the child instance. (The sending side is at https://github.com/landley/toybox/blob/master/toys/pending/sh.c#L1360 and the receiving side at https://github.com/landley/toybox/blob/master/toys/pending/sh.c#L4146 .) Speaking of which, I'm still sad that the kernel never implemented a "re-exec self" that isn't dependent on /proc and doesn't get confused by chroots, but this is another aspect of "linux-kernel does not care when we bring this stuff up". The topic comes up from time to time, and some patches have been proposed, but it has yet to result in a way to do it that I am aware of: https://lkml.iu.edu/hypermail/linux/kernel/0612.3/0238.html https://lkml.iu.edu/hypermail/linux/kernel/1709.1/03186.html https://lkml.iu.edu/hypermail/linux/kernel/2005.2/07206.html Also, ext4 eventually fixed the ext2/ext3 split, but binfmt_fdpic.c is still a separate file and not a couple of if() statements in binfmt_elf.c. Sigh... Anyway... > I am not using nommu systems myself, so I'm in a similar position as Petr in terms > of it not making much sense for me to be the maintainer. However, having said that, > I have had for a few years now an idea for a background project related to LTP > that might make this a more interesting fit for me. Sony uses NuttX, and is considering > using Zephyr in some of our low-end processor systems. This includes some nommu > systems. For some time now, I have wanted to experiment with using LTP to test > the compatibility of those systems with the Linux system APIs. In full disclosure, > I have no idea if this is a feasible or useful idea or not. But it's something I'd like > to investigate. I've been talking with Rich Felker on IRC about what's involved in porting musl on top of RTOS du jour. There was a long list of things he said in IRC that I could try to scrape out of the log if you're interested. The midipix guy also pointed me at https://midipix.org/sys_sysapi.h and https://git.midipix.org/mmglue from where he ported musl to Windows. I also got pointed at http://lists.landley.net/pipermail/toybox-landley.net/2024-January/029967.html I.E. https://github.com/apexrtos/apex/blob/master/sys/kern/syscall_table.c from somebody ELSE who did it for a different RTOS... > I realize that testing non-Linux RTOSes is out-of-scope for LTP. What we've basically been discussing is that "the Linux API" is a de-facto standard somewhere beyond posix, which has been implemented a bunch of different times now. FreeBSD's Linux emulation layer, Windows Subsystem For Linux, Google proposed a Linux layer for Fuchsia (https://9to5google.com/2021/02/12/google-fuchsia-os-android-linux-programs-starnix/), here's a guy who wrote his own kernel that runs the Linux binaries (https://github.com/vvaltchev/tilck I.E. https://www.youtube.com/watch?v=Ce1pMlZO_mI )... I dunno what subset of the API other operating systems want to support, but given that Linus is an empty nester now (all three daughters off to college), idle discussions about his eventual retirement have been quietly going on for a while now... > But given that that is > something I would like to do, and that it might be relevant to the Linux nommu tests, > I would humbly request a few weeks to investigate this before the nommu code is removed. > This delay would be to see if it would make sense for me to volunteer to help out with > maintaining this otherwise abandoned code. I am interested in helping, but I am overstretched as it is. My mkroot script builds bootable linux systems and regression tests them under qemu for a dozen architectures, all in about 350 lines of bash. To make that work I created an even _more_ compressed kernel config format, microconfig, so adding support for a new target is... well I recently added or1k support and everything the script knows about that target is the 3 lines starting at: https://github.com/landley/toybox/blob/master/mkroot/mkroot.sh#L233 That build script is already mostly self-hosting, in that it first builds a directory of toybox binaries (line 62) and points the $PATH at those, and then builds toybox and a Linux kernel under that. Which works, but cheats: https://github.com/landley/toybox/blob/master/scripts/install.sh#L105 The PENDING= command list is the binaries it symlinks out of the host $PATH, the ones before the multi-space break are theones I have partial implementations of in "pending" (but which aren't in defconfig yet because they're not finished), and the ones after the break are the ones I haven't started writing yet. I.E. I still need to finish "expr git tr bash sh gzip", and I need to start "awk bison flex make". But once I've written all of those, I should be able to run mkroot.sh in a mkroot image. And THEN I need to get the automated regression test script that makes sure all the targets still work under qemu (https://github.com/landley/toybox/blob/master/mkroot/testroot.sh about 100 lines) to also run the toybox test suite in qemu, but that still requires actual bash to run the test suite, toysh isn't quite finished yet (and I refuse to trim it because I want toysh to implement all those bash features). And THEN I need to get it to build Linux From Scratch, which I did back under aboriginal linux back in the day: https://landley.net/aboriginal/control-images/ (I have to start over with the current LFS 12.0 stuff because none of those old packages know what musl is and autoconf is just craptacular. Alas, LFS is full of gnu packages, and gnu is brittle navel-gazing crap. But that's the best stress test for my command line stuff handling all the weired evil corner cases it throws at them. And it's also why both busybox and toybox seds reply to --version with "this is not gnu sed 9.0" because of STUPID autoconf regexes...) mkroot's already got the plumbing to add arbitrary extra behavior to the images as build packages, the https://github.com/landley/toybox/blob/master/mkroot/packages/tests package is just a stub for now but I know what I want to have that do once the shell's ready... And once I've got it building Linux From Scratch, THEN I need to tackle AOSP, which is its own rant, but luckily its maintainer (Elliott Hughes, the "Android base OS" maintainer) is the #2 developer on toybox, and I've been discussing these plans with him for years: http://lists.landley.net/pipermail/toybox-landley.net/2016-July/024590.html But he's waiting for me to get through my todo list before actually trying to add anything like a posix container to Android. It's mostly a security thing, the ability to create arbitrary code and then execute it is like someone asking to bring an open flame onto an airplane: http://lists.landley.net/pipermail/toybox-landley.net/2019-September/026992.html The other problem is that for historical reasons each app installs as a different UID and a "posix container" would need to be able to install a UID/GID _range_ which the container could then remap using container plumbing (and also lock the hell DOWN using container plumbing, so its ability to trojan the phone and do evil maid attacks and 37 other bad things was NOT ALLOWED, which is some design work they seem to be deferring until I wave something otherwise usable at them)... (Ok, and he thinks phones are too slow to build android, that nobody does development on anything less powerful than a 32 processor machine with at least that many gigs of ram and a fiber connection to the net because nobody at GOOGLE has less than that. He has boggled at me about this cultural difference before, skip to about 20:30 in http://androidbackstage.blogspot.com/2016/07/episode-53-adb-on-adb.html for example. Anyway, at some point I need to STRIP DOWN the AOSP build so it takes less than forever. Haven't even opened that can of worms yet, but from a capability standpoint it's gotta build the whole thing _first_...) Anyway, the point is this is its own whole ecosystem, which I'd need a staff of a dozen people to navigate properly, and there's just me. And now that toybox has users, I get support requests coming in... > I can't promise anything, but I'd like to find out more about: > 1) what parts of the current LTP are not supporting nommu (what's currently broken), > 2) how much code we're talking about, and > 3) what the desired roadmap going forward would be, to continue to support this code. I am very interested in these things as well. I would like to help, and am happy to answer all the questions I can, but caffeine only takes you so far when it comes to regular commitments... Rob -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [LTP] [Automated-testing] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP] 2024-01-09 22:37 ` [LTP] [Automated-testing] " Bird, Tim 2024-01-10 5:01 ` Rob Landley @ 2024-01-10 14:14 ` Petr Vorel 2024-01-10 19:23 ` Rob Landley 1 sibling, 1 reply; 40+ messages in thread From: Petr Vorel @ 2024-01-10 14:14 UTC (permalink / raw) To: Tim Bird Cc: Niklas Cassel, Jonathan Corbet, Linux-sh list, Christophe Lyon, Randy Dunlap, linux-kernel, linux-m68k, Geert Uytterhoeven, Linux ARM, Rob Landley, John Paul Adrian Glaubitz, linux-riscv, buildroot, Greg Ungerer, ltp, automated-testing Hi Tim, all, > > -----Original Message----- > > From: automated-testing@lists.yoctoproject.org <automated-testing@lists.yoctoproject.org> On Behalf Of Cyril Hrubis > > Hi! > > > But as I said, if anybody from nommu decides to maintain it in LTP, I'll try to > > > support him in my free time (review patches, give advices). And if nobody > > > stands, this patchset which removes the support in the old API will be merged > > > after next LTP release (in the end of January). > > Let me highlight this part, we are eager to help anybody who is willing > > to pick the nommu work, but we do not have resources to drive it. > I have a couple of comments here. > I think it would be good to give a little bit more time to try to find a helper/maintainer > for this. As Rob pointed out, a lot of embedded Linux developers are using very old > kernels (and, if they are using LTP, likely very old versions of LTP). They are also > notorious for not being active on the mailing lists. So this might take some active > outreach to find helpers. (I realize that this thread is part of this > outreach effort). For this reason, I'd like a few more weeks to try to advertise this > need within the embedded Linux community. Thank you. > I am not using nommu systems myself, so I'm in a similar position as Petr in terms > of it not making much sense for me to be the maintainer. However, having said that, > I have had for a few years now an idea for a background project related to LTP > that might make this a more interesting fit for me. Sony uses NuttX, and is considering > using Zephyr in some of our low-end processor systems. This includes some nommu > systems. For some time now, I have wanted to experiment with using LTP to test > the compatibility of those systems with the Linux system APIs. In full disclosure, > I have no idea if this is a feasible or useful idea or not. But it's something I'd like > to investigate. > I realize that testing non-Linux RTOSes is out-of-scope for LTP. But given that that is > something I would like to do, and that it might be relevant to the Linux nommu tests, > I would humbly request a few weeks to investigate this before the nommu code is removed. > This delay would be to see if it would make sense for me to volunteer to help out with > maintaining this otherwise abandoned code. > I can't promise anything, but I'd like to find out more about: > 1) what parts of the current LTP are not supporting nommu (what's currently broken), The new C API, I described it in my reply to Rob: https://lore.kernel.org/ltp/20240110133358.GB1698252@pevik/ But I don't know whether the code in the old API was even working, whole old API suffered with random failures, that was one of the reasons to write a new one from the scratch. > 2) how much code we're talking about, and There was FORK_OR_VFORK(), which would probably in the new API call vfork() for nommu targets (tst_old_flush() is probably not needed in the new API). There is a special handling of getopts in lib/parse_opts.c + -C param for it. One would have to integrate these two functions from lib/self_exec.c to the new API (and port them to use new API via tst_test.h with #define TST_NO_DEFAULT_MAIN): void maybe_run_child(void (*child)(), const char *fmt, ...); int self_exec(const char *argv0, const char *fmt, ...); char *child_args is somehow integrated to lib/tst_test.c via -C arg, I haven't found what uses that option. There is m4, that would be usable (m4/ltp-nommu-linux.m4). Various tests and testsuites were not compiled for nommu (e.g. capget). There is MAP_PRIVATE_EXCEPT_UCLINUX constant to avoid using MAP_PRIVATE on uClinux, who knows if this is relevant on nommu? > 3) what the desired roadmap going forward would be, to continue to support this code. All LTP tests are being rewritten to use new API since 2016 (new API was introduced in 20160510), thus we are loosing the support with old API going away. Sure, I can hold on this patchset and we continue removing the functionality tests manually. But sooner or later it's gone. One can check files which had special handling in the old API: $ git grep -l UCLINUX 20160126 -- testcases/ | wc -l 173 What is supported now: $ git grep -l UCLINUX -- testcases/ |wc -l 55 => We have now removed nearly 2/3 of it (this means we're arguing about 1/3 of the tests which initially somehow supported nommu). Kind regards, Petr > Thanks, > -- Tim > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#1271): https://lists.yoctoproject.org/g/automated-testing/message/1271 > Mute This Topic: https://lists.yoctoproject.org/mt/103541824/3616762 > Group Owner: automated-testing+owner@lists.yoctoproject.org > Unsubscribe: https://lists.yoctoproject.org/g/automated-testing/unsub [pvorel@suse.cz] > -=-=-=-=-=-=-=-=-=-=-=- -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [LTP] [Automated-testing] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP] 2024-01-10 14:14 ` Petr Vorel @ 2024-01-10 19:23 ` Rob Landley 2024-01-10 21:17 ` Niklas Cassel via ltp ` (2 more replies) 0 siblings, 3 replies; 40+ messages in thread From: Rob Landley @ 2024-01-10 19:23 UTC (permalink / raw) To: Petr Vorel, Tim Bird Cc: Niklas Cassel, Jonathan Corbet, Linux-sh list, Christophe Lyon, Randy Dunlap, linux-kernel, linux-m68k, Geert Uytterhoeven, Linux ARM, John Paul Adrian Glaubitz, linux-riscv, buildroot, Greg Ungerer, ltp, automated-testing On 1/10/24 08:14, Petr Vorel wrote: > There is MAP_PRIVATE_EXCEPT_UCLINUX constant to avoid using MAP_PRIVATE on > uClinux, who knows if this is relevant on nommu? MAP_PRIVATE creates a copy-on-write mapping, and doing copy-on-write requires an MMU. (You mark it read only in the page table, take a fault when they try to write, the fault handler allocates a new physical page, copies the old contents to it, marks it writeable, and returns allowing the write to complete to the new page.) On NOMMU you can MAP_SHARED and MAP_ANON, but not MAP_PRIVATE. Swap is implemented kind of similarly, except when you recycle pages you mark them as neither readable nor writeable in the page table, schedule the page's contents to be written to disk, suspend the process so the scheduler can go run something else, and then when you get the I/O completion interrupt you mark the page free so whatever else needed a page can use it. And then when the process tries to access the page the fault handler reverses the process, allocating a new physical page and load in the contents back in while the process is suspended waiting for that to finish. Can't do that without an MMU either. >> 3) what the desired roadmap going forward would be, to continue to support this code. > > All LTP tests are being rewritten to use new API since 2016 (new API was > introduced in 20160510), thus we are loosing the support with old API going > away. Sure, I can hold on this patchset and we continue removing the > functionality tests manually. But sooner or later it's gone. You can't fork() on nommu because copies of the mappings have different addresses, meaning any pointers in the copied mappings would point into the OLD mappings (belonging to the parent process), and fixing them up is 100% equivalent to the "garbage collection in C" problem. (It's AI-complete. Of the C3PO kind, not the "autocorrect with syntax checking" kind.) People get hung up on the "it would be very inefficient to do that because no copy-on-write" problem and miss the "the child couldn't FUNCTION because its pointer variables all contain parent addresses" problem. So instead vfork() creates a child with all the same memory mappings (a bit like a thread) and freezes the parent process until that child discards those mappings, either by calling exec() or _exit(). (At which point the parent gets un-suspended.) The child can do quite a lot of setup before calling exec, it already has its own filehandle table for example, but it has to be really careful about MEMORY. Anything it writes to global variables the parent will see, any changes to the heap persist in the parent, and anything it writes to local variables the parent MAY see. (Systems have historically differed about whether the vfork() child gets a new stack like a thread would, or keeps using the parent's mapping since the new stack would be quickly discarded anyway. If you call into a new setup() function after vfork() it doesn't matter much either way, but do NOT return from the function that called vfork(): either your new stack hasn't got anything to return to or you'll corrupt the parent's stack by overwriting its return address so when the parent exits from its current function it jumps to la-la land.) The OTHER fun thing about nommu is you can't run conventional ELF binaries, because everything is linked at fixed address. So you might be able to run ONE instance of the program as your init task, assuming those addresses were available even then, but as soon as you try to run a second one it's a conflict. The quick and dirty work around is to make PIE binaries, which can relocate everything into available space, which works but doesn't scale. The problem with ELF PIE is that everything is linked contiguously from a single base pointer, meaning your text, rodata, data, and bss segments are all one linear blob. So if you run two instances of bash, you've loaded two copies of the test and the rodoata. This fills up your memory fast. AND PIE requires contiguous memory, which nommu is bad at providing because it has no page tables to remap stuff. With an mmu it can coalesce scattered physical pages into a virtually contiguous range, but without an mmu you can have plenty of memory free but in tiny chunks, none big enough to satisfy an allocation request. So they invented FDPIC, which is ELF with FOUR base pointers. Each major section (rodata, text, data, and bss) has its own base pointer, so you need to find smaller chunks of memory to load them into (and thus it can work on a more fragmented system), AND it means that two instances of the same program can share the read-only sections (rodata and text) so you only need new copies of the writeable segments (data and bss. And the heap. And the stack.) (The earlier binflt format is an a.out variant with 4 base pointers. FDPIC is the ELF version of the same idea. Since a.out went bye-bye binflt is obsolete, but not everybody's moved off it yet because so many nommu people are still using 2.6 or even earlier, and also using gcc 3.x or 2.x toolchains.) Oh, the OTHER thing is none of this is deferred allocation, it's all up front. On systems with mmu you can populate large empty virtual mappings that read as zeroed but it's actually redundant copy-on-write instances of the zero page, and when you write to them it takes a soft fault and the fault handler allocates the page you dirtied when you dirty it. On nommu, if you want a 4 megabyte mapping you have to find 4 contiguous megabyte and allocate it immediately, or else the mmap() or malloc() returns failure. (On systems with mmu malloc() almost never returns NULL, because you've got virtual address space coming out of your ears and if you ACTUALLY run out of memory that's happens way later, the OOM killer triggers long after malloc() returned success. But on a nommu system, malloc() returns NULL all the time, even if you THINK you have enough memory, because what's left is too fragmented to contain a free chunk THAT BIG...) This impacts the stack. On MMU Linux, the default stack size is 8 megs but it's seldom all used. On nommu linux, that would be RIDICULOUS because A) it would always be allocated to its full size right up front, B) you'd need contiguous memory for it. So instead you set the default stack size when building the linker (you can also set it on the ld command line), and common sizes range from 8k to maybe 256k depending on what you expect to be running. Toybox tries not to use more than 16k stack, although I usually test it with 32k on nommu. (There's no guard page to tell you when you went off the edge, because no MMU so no page faults, but you can check that the stack page at end-16k is still clean at exit if you like. Some nommu hardware has range registers, but Linux has never supported them that I'm aware of.) There's not THAT much to learn about NOMMU. It could all be covered in an hour presentation at a conference, I expect? > One can check files which had special handling in the old API: > > $ git grep -l UCLINUX 20160126 -- testcases/ | wc -l > 173 > > What is supported now: > > $ git grep -l UCLINUX -- testcases/ |wc -l > 55 UCLINUX is a long-dead distro. Linaro died in the dot-com crash and its founder Jeff Dionne moved to Japan for his next gig and never came back. On the way out he handed uclinux off to someone else, who didn't do a lot of work maintaining it. Most of the actual support went "upstream" into various packages (linux and busybox and gcc and so on) before the handoff, so you didn't NEED uclinux anymore. The real nail in the coffin is the inheritor of uclinux never migrated it off CVS, and then the disk holding the CVS archive crashed with no backup. He came out with a couple more releases after that by monkey-patching the last release's filesystem, but the writing was on the wall and it rolled to a stop. I did a triage of its last release (from 2014) as part of my toybox roadmap: https://landley.net/toybox/roadmap.html#uclinux > => We have now removed nearly 2/3 of it (this means we're arguing about 1/3 of > the tests which initially somehow supported nommu). I'd like to get more tests supporting nommu. Possibly the approach here is just get ANYTHING working with the new api, and then whack-a-mole more tests in as we go. Other than lacking fork(), restricted mmap(), different executable packaging, smaller stack size, having to actually test the return from malloc(), no page faults if you follow a wild pointer, and the complete lack of swap space... Unless I missed something, it's otherwise normal Linux? (People comfortable with threads can still do all their thread tricks on nommu systems. And the Turtle board I'm using is an SMP nommu system, they do exist. :) Rob -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [LTP] [Automated-testing] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP] 2024-01-10 19:23 ` Rob Landley @ 2024-01-10 21:17 ` Niklas Cassel via ltp 2024-01-11 0:00 ` Greg Ungerer 2024-01-11 2:25 ` Greg Ungerer 2024-01-11 13:11 ` [LTP] " Geert Uytterhoeven 2 siblings, 1 reply; 40+ messages in thread From: Niklas Cassel via ltp @ 2024-01-10 21:17 UTC (permalink / raw) To: Rob Landley Cc: Linux ARM, Jonathan Corbet, Linux-sh list, Randy Dunlap, linux-kernel, linux-m68k, Geert Uytterhoeven, Christophe Lyon, John Paul Adrian Glaubitz, linux-riscv, buildroot, Greg Ungerer, ltp, automated-testing On Wed, Jan 10, 2024 at 01:23:51PM -0600, Rob Landley wrote: > UCLINUX is a long-dead distro. Linaro died in the dot-com crash and its founder > Jeff Dionne moved to Japan for his next gig and never came back. On the way out > he handed uclinux off to someone else, who didn't do a lot of work maintaining > it. Most of the actual support went "upstream" into various packages (linux and > busybox and gcc and so on) before the handoff, so you didn't NEED uclinux anymore. s/Linaro/Lineo/ Kind regards, Niklas -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [LTP] [Automated-testing] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP] 2024-01-10 21:17 ` Niklas Cassel via ltp @ 2024-01-11 0:00 ` Greg Ungerer 2024-01-11 9:21 ` Niklas Cassel via ltp 0 siblings, 1 reply; 40+ messages in thread From: Greg Ungerer @ 2024-01-11 0:00 UTC (permalink / raw) To: Niklas Cassel, Rob Landley Cc: Jonathan Corbet, Linux-sh list, Randy Dunlap, linux-kernel, linux-m68k, Geert Uytterhoeven, Linux ARM, John Paul Adrian Glaubitz, linux-riscv, buildroot, Christophe Lyon, ltp, automated-testing On 11/1/24 07:17, Niklas Cassel wrote: > On Wed, Jan 10, 2024 at 01:23:51PM -0600, Rob Landley wrote: >> UCLINUX is a long-dead distro. Linaro died in the dot-com crash and its founder >> Jeff Dionne moved to Japan for his next gig and never came back. On the way out >> he handed uclinux off to someone else, who didn't do a lot of work maintaining >> it. Most of the actual support went "upstream" into various packages (linux and >> busybox and gcc and so on) before the handoff, so you didn't NEED uclinux anymore. > > s/Linaro/Lineo/ Lineo was not founded by Jeff Dionne, see https://en.wikipedia.org/wiki/Lineo for its genisys. Maybe you are thinking of RT-Control. Greg -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [LTP] [Automated-testing] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP] 2024-01-11 0:00 ` Greg Ungerer @ 2024-01-11 9:21 ` Niklas Cassel via ltp 2024-01-12 20:18 ` Rob Landley 0 siblings, 1 reply; 40+ messages in thread From: Niklas Cassel via ltp @ 2024-01-11 9:21 UTC (permalink / raw) To: Greg Ungerer Cc: Rob Landley, Jonathan Corbet, Linux-sh list, Randy Dunlap, linux-kernel, linux-m68k, Geert Uytterhoeven, Linux ARM, John Paul Adrian Glaubitz, linux-riscv, buildroot, Christophe Lyon, ltp, automated-testing On Thu, Jan 11, 2024 at 10:00:11AM +1000, Greg Ungerer wrote: > > On 11/1/24 07:17, Niklas Cassel wrote: > > On Wed, Jan 10, 2024 at 01:23:51PM -0600, Rob Landley wrote: > > > UCLINUX is a long-dead distro. Linaro died in the dot-com crash and its founder > > > Jeff Dionne moved to Japan for his next gig and never came back. On the way out > > > he handed uclinux off to someone else, who didn't do a lot of work maintaining > > > it. Most of the actual support went "upstream" into various packages (linux and > > > busybox and gcc and so on) before the handoff, so you didn't NEED uclinux anymore. > > > > s/Linaro/Lineo/ > > Lineo was not founded by Jeff Dionne, see https://en.wikipedia.org/wiki/Lineo > for its genisys. Maybe you are thinking of RT-Control. Yes, Jeff cofounded Rt-Control which later merged with Lineo. My main point is that Linaro is a completely different company, which did not "die in the dot-com crash", like stated above. Kind regards, Niklas -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [LTP] [Automated-testing] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP] 2024-01-11 9:21 ` Niklas Cassel via ltp @ 2024-01-12 20:18 ` Rob Landley 0 siblings, 0 replies; 40+ messages in thread From: Rob Landley @ 2024-01-12 20:18 UTC (permalink / raw) To: Niklas Cassel, Greg Ungerer Cc: Jonathan Corbet, Linux-sh list, Randy Dunlap, linux-kernel, linux-m68k, Geert Uytterhoeven, Linux ARM, John Paul Adrian Glaubitz, linux-riscv, buildroot, Christophe Lyon, ltp, automated-testing On 1/11/24 03:21, Niklas Cassel wrote: >> > s/Linaro/Lineo/ >> >> Lineo was not founded by Jeff Dionne, see https://en.wikipedia.org/wiki/Lineo >> for its genisys. Maybe you are thinking of RT-Control. > > Yes, Jeff cofounded Rt-Control which later merged with Lineo. > > My main point is that Linaro is a completely different company, > which did not "die in the dot-com crash", like stated above. Sorry, I typo uclibc/uclinux all the time too. (In extreme cases I swap busybox/toybox, which is personally embarassing...) Rob -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [LTP] [Automated-testing] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP] 2024-01-10 19:23 ` Rob Landley 2024-01-10 21:17 ` Niklas Cassel via ltp @ 2024-01-11 2:25 ` Greg Ungerer 2024-01-12 20:16 ` Rob Landley 2024-01-11 13:11 ` [LTP] " Geert Uytterhoeven 2 siblings, 1 reply; 40+ messages in thread From: Greg Ungerer @ 2024-01-11 2:25 UTC (permalink / raw) To: Rob Landley, Petr Vorel, Tim Bird Cc: Niklas Cassel, Jonathan Corbet, Linux-sh list, Christophe Lyon, Randy Dunlap, linux-kernel, linux-m68k, Geert Uytterhoeven, Linux ARM, John Paul Adrian Glaubitz, linux-riscv, buildroot, ltp, automated-testing Sorry Rob, but I think you are wrong about a number of things here. I know, I was there at the coal face so to speak during the early years of uClinux and right up today. I feel I have to correct some of the things you claim. On 11/1/24 05:23, Rob Landley wrote: > On 1/10/24 08:14, Petr Vorel wrote: >> There is MAP_PRIVATE_EXCEPT_UCLINUX constant to avoid using MAP_PRIVATE on >> uClinux, who knows if this is relevant on nommu? > > MAP_PRIVATE creates a copy-on-write mapping, and doing copy-on-write requires an > MMU. (You mark it read only in the page table, take a fault when they try to > write, the fault handler allocates a new physical page, copies the old contents > to it, marks it writeable, and returns allowing the write to complete to the new > page.) > > On NOMMU you can MAP_SHARED and MAP_ANON, but not MAP_PRIVATE. > > Swap is implemented kind of similarly, except when you recycle pages you mark > them as neither readable nor writeable in the page table, schedule the page's > contents to be written to disk, suspend the process so the scheduler can go run > something else, and then when you get the I/O completion interrupt you mark the > page free so whatever else needed a page can use it. And then when the process > tries to access the page the fault handler reverses the process, allocating a > new physical page and load in the contents back in while the process is > suspended waiting for that to finish. Can't do that without an MMU either. > >>> 3) what the desired roadmap going forward would be, to continue to support this code. >> >> All LTP tests are being rewritten to use new API since 2016 (new API was >> introduced in 20160510), thus we are loosing the support with old API going >> away. Sure, I can hold on this patchset and we continue removing the >> functionality tests manually. But sooner or later it's gone. > > You can't fork() on nommu because copies of the mappings have different > addresses, meaning any pointers in the copied mappings would point into the OLD > mappings (belonging to the parent process), and fixing them up is 100% > equivalent to the "garbage collection in C" problem. (It's AI-complete. Of the > C3PO kind, not the "autocorrect with syntax checking" kind.) People get hung up > on the "it would be very inefficient to do that because no copy-on-write" > problem and miss the "the child couldn't FUNCTION because its pointer variables > all contain parent addresses" problem. > > So instead vfork() creates a child with all the same memory mappings (a bit like > a thread) and freezes the parent process until that child discards those > mappings, either by calling exec() or _exit(). (At which point the parent gets > un-suspended.) Just to be clear, vfork is not a uClinux invention. It dates way back to the early BSD UNIX days. It just so happens that it fits in very nicely with the no-MMU model of the world. > The child can do quite a lot of setup before calling exec, it already has its > own filehandle table for example, but it has to be really careful about MEMORY. > Anything it writes to global variables the parent will see, any changes to the > heap persist in the parent, and anything it writes to local variables the parent > MAY see. (Systems have historically differed about whether the vfork() child > gets a new stack like a thread would, or keeps using the parent's mapping since > the new stack would be quickly discarded anyway. If you call into a new setup() > function after vfork() it doesn't matter much either way, but do NOT return from > the function that called vfork(): either your new stack hasn't got anything to > return to or you'll corrupt the parent's stack by overwriting its return address > so when the parent exits from its current function it jumps to la-la land.) > > The OTHER fun thing about nommu is you can't run conventional ELF binaries, > because everything is linked at fixed address. So you might be able to run ONE > instance of the program as your init task, assuming those addresses were > available even then, but as soon as you try to run a second one it's a conflict. > > The quick and dirty work around is to make PIE binaries, which can relocate > everything into available space, which works but doesn't scale. The problem with > ELF PIE is that everything is linked contiguously from a single base pointer, > meaning your text, rodata, data, and bss segments are all one linear blob. So if > you run two instances of bash, you've loaded two copies of the test and the > rodoata. This fills up your memory fast. > > AND PIE requires contiguous memory, which nommu is bad at providing because it > has no page tables to remap stuff. With an mmu it can coalesce scattered > physical pages into a virtually contiguous range, but without an mmu you can > have plenty of memory free but in tiny chunks, none big enough to satisfy an > allocation request. > > So they invented FDPIC, which is ELF with FOUR base pointers. Each major section > (rodata, text, data, and bss) has its own base pointer, so you need to find > smaller chunks of memory to load them into (and thus it can work on a more > fragmented system), AND it means that two instances of the same program can > share the read-only sections (rodata and text) so you only need new copies of > the writeable segments (data and bss. And the heap. And the stack.) It is worth noting that to run PIE style ELF binaries on no-MMU systems you actually use the FDPIC ELF loader - not the traditional linux ELF loader (binfmt_elf). Using this setup is going to become much easier now that uClibc has native support for generating a relevant library and ld.so for noMMU linux (certainly for m68k, arm and riscv at least) - as of version 1.0.45. > (The earlier binflt format is an a.out variant with 4 base pointers. FDPIC is > the ELF version of the same idea. Since a.out went bye-bye binflt is obsolete, > but not everybody's moved off it yet because so many nommu people are still > using 2.6 or even earlier, and also using gcc 3.x or 2.x toolchains.) Flat format (binfmt_flat) is in no way related to a.out. It is not derived from it and shares nothing with it. A.out being removed from the kernel in no way affects flat format. It doesn't magically make it obsolete. It isn't going anywhere. I don't see how this is related to versions of kernel or gcc or toolchains either. Flat format works on every kernel up to todays v6.7. The conversion tool works with the latest gcc and binutils (gcc-13.2 and bintuils-2.41 as of today) and many versions going back for the best part of 20 years. Sure that elf2flt tool has been forked a lot it has a spotty history of getting updated. But, hey, as of today it looks pretty up to date across most architectures. > Oh, the OTHER thing is none of this is deferred allocation, it's all up front. > On systems with mmu you can populate large empty virtual mappings that read as > zeroed but it's actually redundant copy-on-write instances of the zero page, and > when you write to them it takes a soft fault and the fault handler allocates the > page you dirtied when you dirty it. On nommu, if you want a 4 megabyte mapping > you have to find 4 contiguous megabyte and allocate it immediately, or else the > mmap() or malloc() returns failure. (On systems with mmu malloc() almost never > returns NULL, because you've got virtual address space coming out of your ears > and if you ACTUALLY run out of memory that's happens way later, the OOM killer > triggers long after malloc() returned success. But on a nommu system, malloc() > returns NULL all the time, even if you THINK you have enough memory, because > what's left is too fragmented to contain a free chunk THAT BIG...) > > This impacts the stack. On MMU Linux, the default stack size is 8 megs but it's > seldom all used. On nommu linux, that would be RIDICULOUS because A) it would > always be allocated to its full size right up front, B) you'd need contiguous > memory for it. So instead you set the default stack size when building the > linker (you can also set it on the ld command line), and common sizes range from > 8k to maybe 256k depending on what you expect to be running. Toybox tries not to > use more than 16k stack, although I usually test it with 32k on nommu. (There's > no guard page to tell you when you went off the edge, because no MMU so no page > faults, but you can check that the stack page at end-16k is still clean at exit > if you like. Some nommu hardware has range registers, but Linux has never > supported them that I'm aware of.) > > There's not THAT much to learn about NOMMU. It could all be covered in an hour > presentation at a conference, I expect? > >> One can check files which had special handling in the old API: >> >> $ git grep -l UCLINUX 20160126 -- testcases/ | wc -l >> 173 >> >> What is supported now: >> >> $ git grep -l UCLINUX -- testcases/ |wc -l >> 55 > > UCLINUX is a long-dead distro. Linaro died in the dot-com crash and its founder > Jeff Dionne moved to Japan for his next gig and never came back. On the way out > he handed uclinux off to someone else, who didn't do a lot of work maintaining > it. Most of the actual support went "upstream" into various packages (linux and > busybox and gcc and so on) before the handoff, so you didn't NEED uclinux anymore. Why do you keep claiming that "UCLINUX" (why all caps?) is a distro? That is just not the case. uClinux has been used as an umbrella term for patches, tools, packages relating to running Linux on a CPU with no MMU. There was a build package called uclinux-dist that you might consider a distro. But it has always been cleanly named as "uclinux-dist". For a long time that was the goto starting point for anyone wanting to use Linux with no-MMU. The collection of patches to the toolchains, kernel, libraries and application packages was a pretty high mountain to climb to get started in those early days (late 90's and early 2000's). But through the work of a _lot_ of people much of that change has made its way into relevant packages across the board (from gcc to kernel to creation of uClibc, etc). But lets face it once no-MMU support was integrated into mainline linux as off v2.5.46 it is really now just "Linux". The no-MMU support outgrew that trade marked term. But the name has stubbornly stuck around. > The real nail in the coffin is the inheritor of uclinux never migrated it off > CVS, and then the disk holding the CVS archive crashed with no backup. He came > out with a couple more releases after that by monkey-patching the last release's > filesystem, but the writing was on the wall and it rolled to a stop. No, the public facing CVS was never the master sources for the uclinux-dist. The public facing CVS was only ever a copy of the tar ball releases. Its loss was annoying to some who used it but had no real impact on the development. Its development model was kind of "internal" and published at random points in time. Today that is what probably most people would call the "vendor" distro model. That work was slowing down due to fact that it just wasn't really needed any more. There are way more popular build systems (eg build-root) for building complete firmware images from scratch. > I did a triage of its last release (from 2014) as part of my toybox roadmap: No, the last release was in 2016, see it archived at: https://sourceforge.net/projects/uclinux/files/uClinux%20Stable/ But that is all kind of archeological now, relegated to history. Regards Greg -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [LTP] [Automated-testing] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP] 2024-01-11 2:25 ` Greg Ungerer @ 2024-01-12 20:16 ` Rob Landley 2024-01-14 13:01 ` Greg Ungerer 2024-01-15 13:41 ` [LTP] [Buildroot] " Waldemar Brodkorb 0 siblings, 2 replies; 40+ messages in thread From: Rob Landley @ 2024-01-12 20:16 UTC (permalink / raw) To: Greg Ungerer, Petr Vorel, Tim Bird Cc: Niklas Cassel, Jonathan Corbet, Linux-sh list, Christophe Lyon, Randy Dunlap, linux-kernel, linux-m68k, Geert Uytterhoeven, Linux ARM, John Paul Adrian Glaubitz, linux-riscv, buildroot, ltp, automated-testing On 1/10/24 20:25, Greg Ungerer wrote: > Sorry Rob, but I think you are wrong about a number of things here. Happy to be corrected. I learned most of this stuff by people pointing things out I didn't know. But I _have_ been trying to collect this info for about 15 years now... > I know, I was there at the coal face so to speak during the early > years of uClinux and right up today. I feel I have to correct some of > the things you claim. I only heard about the politics long after the fact, and the stories have a lot of elisions because what happens in Utah sadly does NOT stay in Utah. I didn't get involved in busybox until either 2002 or 2003 depending how you want to count it. After the dot com crash, around the time of the SCO trial. Erik Andersen having once worked there was about as relevant as Linus' day job at Transmeta, or busybox having started life years earlier as Debian's answer to Red Hat Nash before being abandoned for 3 years before Erik picked it up. I heard that this stuff was used in uClinux, but its use in linksys routers was far more immediately relevant. Shortly after the dot-com crash Ray Noorda went senile and got elder abused into allowing the SCO lawsuits run by the canopy group (his personal money managers, gone rogue), by which time Erik Andersen and Jeff Dionne and Tim Bird and so on had all gone on to other employers. Tim heading to Sony, Jeff moved to Japan to do hardware and leaving uClinux behind, and Erik continued busybox and uclibc as personal projects from a server connected to the DSL line in his basement. By the time I showed up uclinux never came up on the mailing list, the IRC channel, or in private email. I was introduced to busybox+uclibc by tomsrtbt, and the highest profile consumer of Erik's output was Linksys routers. My goal with busybox was making it work in a development environment, initially to replace most of the the 110 megabytes of gnu bloatware in Linux From Scratch with the 1.7 megabytes of tomsrtbt to free up some of the 700 megabyte budget of knoppix CDs. Making that actually _work_ took long enough to accomplish that "knoppix" stopped being "the bootable CD" and every distro had one now (usually as their install CD: try it live then install from the desktop you'd booted into if you want to keep it). After I finally got it all working (https://landley.net/aboriginal/news.html either the 1.0 release rebuilding itself under itself on September 5, 2010 or the 1.1 release building Linux From Scratch 6.7 under the result on October 2, 2011 depending how you want to measure), I moved on to focusing on toybox development instead (trying to get it into Android so THAT could become self-hosting and provide a real development environment with a USB keyboard+mouse and chromecast to a big TV), and other people took over creating Adelie Linux and Alpine Linux and so on based on busybox instead of gnu tools. I didn't get into nommu development full-time until 2014 when Jeff Dionne hired me to work on his j-core project. I'd heard the name, but never spoken to him before he reached out to me by email, then phone interview, then flew me to Japan. While working for Jeff I asked various computer history questions because it's a hobby of mine, but there's been a pandemic between then and now. I wasn't there, the details of the answers I _did_ get are fuzzy by now, and there were some... stories. Let's just say a canadian who moved to Utah and then moved to LITERALLY THE OTHER SIDE OF THE PLANET may may have seen some stuff. I've also bumped into Tim Bird on and off since 2006 (through CELF/ELC and the Linux Jamboree at Nakano Sun Plaza in Tokyo and https://elinux.org/CELF_Project_Proposal/Combine_tcg_with_tcc and so on) and he's never spoken of Lineo once. The work I did helped kill the old uclinux distro, although I didn't realize it at the time. The various package patches it had carried were already pushed upstream into linux and gcc and so on, and busybox worked as a nommu userspace. By making "build a busybox+uclibc system with a vanilla kernel" easier to do and more powerful by itself, uclinux became less relevant. Various people sent me nommu fixes when I was maintaining busybox, which was like sending me fixes for running busybox on linksys (or openwrt) routers. I didn't have that hardware, but I knew the _general_ theory and tried not to break it. Don't do unaligned access, be careful of endianness, and here's what's necessary for nommu: https://git.busybox.net/busybox/commit/?id=b1b3cee831bc https://git.busybox.net/busybox/commit/?id=03628c8f75ba https://git.busybox.net/busybox/commit/?id=b21837714a37 When buildroot metastasized from uclibc's test suite (what packages build under uclibc? Let's regression test them and slot them together into a testable root filesystem) into a new distro (the first post to its mailing list in 2006 is from me because I abused my busybox maintainer root access to the shared server to set up a third list, because buildroot discussion was smothering uclibc development on that list), buildroot became the logical base point for nommu systems. If you wanted a busybox+uclibc root filesystem, buildroot provided an up to date actively developed one with build recipes for hundreds of packages, and a busy community where you could ask questions. Meaning there was already no reason to install uclinux in 2006. It had been replaced by either a simple busybox+uclibc root filesystem, or buildroot. So "me noticing this area in 2002", "me accidentally helping finish off uclinux in passing in 2006", and "me getting involved in nommu development in 2014 because Jeff Dionne flew me to Japan and personally explained it to me" does not give me any direct experience with what went on at lineo before that, true. Just like I never visited transmeta. >> You can't fork() on nommu because copies of the mappings have different >> addresses, meaning any pointers in the copied mappings would point into the OLD >> mappings (belonging to the parent process), and fixing them up is 100% >> equivalent to the "garbage collection in C" problem. (It's AI-complete. Of the >> C3PO kind, not the "autocorrect with syntax checking" kind.) People get hung up >> on the "it would be very inefficient to do that because no copy-on-write" >> problem and miss the "the child couldn't FUNCTION because its pointer variables >> all contain parent addresses" problem. >> >> So instead vfork() creates a child with all the same memory mappings (a bit like >> a thread) and freezes the parent process until that child discards those >> mappings, either by calling exec() or _exit(). (At which point the parent gets >> un-suspended.) > > Just to be clear, vfork is not a uClinux invention. I never said vfork() was a uClinux invention. I'm describing how Linux does nommu. Doing nommu on linux hasn't had anything to do with uclinux for 20 years. Confusing "uclinux" with "nommu" is like confusing "knoppix" with "Linux Live CD". We've moved on a bit since one magic distro pioneered the technique literal decades ago, there are other options now. I first looked at uClinux itself in 2015 because Jeff Dionne asked me to see if there was anything left to salvage out of it, resulting in https://github.com/landley/toybox/commit/469d7f11b66d and the answer "no, there isn't" nine years ago. > It dates way back to the > early BSD UNIX days. I know. > It just so happens that it fits in very nicely with > the no-MMU model of the world. Ken and Dennis' ported Unix from a PDP-7 to a PDP-11/20, which didn't have an MMU. Dennis wrote about it here: https://www.bell-labs.com/usr/dmr/www/odd.html The PDP-11/45 the upgraded to did, and the system call sematics changed quite a bit in those early years before stabilizing with Unix v7 (largely because AT&T stopped allowing Bell Labs releases to go out to the public after that one, so v8, v9, and v10 saw almost no use, and then they started over with Plan 9.) So yeah I knew about BSD coming up with the name, but I thought "vfork()" was just "what fork() used to do circa Unix v3 or similar"? I was trying to explain why vfork() is still in use NOW. It's not just nommu either, it's also useful when forking from heavily threaded processes, because normal fork() will freeze all the threads in the process causing a latency spike and potentially quite large allocation because in that case it breaks all copy-on-write up front rather than trying to untangle the layers of sharing. But vfork() will only freeze the one thread that called it, and doesn't allocate new memory before the exec() or _exit() of the new child process. (You may hit a vma lock if other threads try to tear down the mappings, I haven't tried. But if you DON'T do something stupid, you don't get a latency spike on the other threads.) >> So they invented FDPIC, "They" being the blackfin devs, I think. >> which is ELF with FOUR base pointers. Each major section >> (rodata, text, data, and bss) has its own base pointer, so you need to find >> smaller chunks of memory to load them into (and thus it can work on a more >> fragmented system), AND it means that two instances of the same program can >> share the read-only sections (rodata and text) so you only need new copies of >> the writeable segments (data and bss. And the heap. And the stack.) > > It is worth noting that to run PIE style ELF binaries on no-MMU systems you > actually use the FDPIC ELF loader - not the traditional linux ELF loader (binfmt_elf). Yup. Although that's mostly because binfmt_elf refuses to build without CONFIG_MMU due to some stupid #ifdefs and assumptions. My understanding (mostly from making puppy eyes at Rich Felker) is that the bimfmt_fdpic loader can load conventional ELF binaries just fine, the way the ext4 driver can mount ext3 or ext2. Unfortunately, my attempts to build bimfmt_fdpic on i386 and arm64 and mips and powerpc and so on to TRY that hit the same "kconfig and the #ifdefs really dowanna allow this" nonsense. I note that qemu's application emulation handles conventional ELF, static PIE, and fdpic pretty much interchangeably. THEY didn't see the need to have two redundant loaders doing the same thing, with only one understanding some fairly minor extensions. But the linux-kernel community seems to segregate out nommu support and quarrantine it, for no obvious reason. (And then make the nommu stuff build ONLY in one mode and the with-mmu stuff build ONLY in the other mode and NEVER THE TWAIN SHALL MEET, again for no obvious reason.) > Using this setup is going to become much easier now that uClibc has native > support for generating a relevant library and ld.so for noMMU linux (certainly > for m68k, arm and riscv at least) - as of version 1.0.45. I haven't followed buildroot's uclibc fork since uclibc.org died, I switched to musl instead. >> (The earlier binflt format is an a.out variant with 4 base pointers. FDPIC is >> the ELF version of the same idea. Since a.out went bye-bye binflt is obsolete, >> but not everybody's moved off it yet because so many nommu people are still >> using 2.6 or even earlier, and also using gcc 3.x or 2.x toolchains.) > > Flat format (binfmt_flat) is in no way related to a.out. It is not derived from > it and shares nothing with it. A.out being removed from the kernel in no way affects > flat format. It doesn't magically make it obsolete. It isn't going anywhere. *shrug* I'll take your word for it. Back when I tried to make it work circa https://landley.net/notes-2014.html#07-12-2014 the binflt plumbing was so bit-rotted that A) there was no official repository for it, B) it was implemented as a weird post-processing step where you ran an "elf2flt" binary but it could not actually digest a normal ELF file, instead you made a SPECIAL elf file with various compiler flags, which the postprocessing tool then converted to a flat file. (This is another variant of "embedded devs get something to work and then keep using the old version FOREVER". I could download 10 year old binflt toolchains that produced binaries, but nobody seemed to have built a NEW one in years.) So I shoehorned basic elf2flt support into my build system: https://github.com/landley/aboriginal/commit/50d139530dd4 And then I asked Yoshinori Sato what version _he'd_ used when adding h8300 support to Linux (since that was nommu) and switched to his version, which he'd fixed a lot of stuff in and which worked with newer compiler versions and had commits less than a year old in the tree: https://github.com/landley/aboriginal/commit/247ee063028f And around that time I emailed various people who had once been interested in this, and they started a new binflt tree (merging the various trees I'd found) and switched buildroot over to use it. But I didn't track it much beyond that because instead Jeff paid Rich Felker to add fdpic support to musl and we switched j-core over to that instead. What little I know about binflt comes from reading the code and from Jeff Dionne trying to explain it to me back in 2014 and 2015, and mostly he was concerned that you had to use older versions of gcc (despite sato-san's upgrades) because current versions of gcc could do some pointer range thing that broke design assumptions in elf2flt, which apparently weren't fixable because information had already been discarded when the .o file was generated, so the thing consuming that output didn't have all the information it needed to link. (You'd have to ask Jeff, I never quite wrapped my head around what the specific issue was and it's a decade ago now. My impression was you'd _mostly_ get lucky and it would usually work with modern gcc, but you rolled the dice with every code change...) > I don't see how this is related to versions of kernel or gcc or toolchains either. > Flat format works on every kernel up to todays v6.7. The conversion tool works > with the latest gcc and binutils (gcc-13.2 and bintuils-2.41 as of today) Again, I'll take your word for it. > and many > versions going back for the best part of 20 years. Sure that elf2flt tool has been > forked a lot it has a spotty history of getting updated. But, hey, as of today it > looks pretty up to date across most architectures. Um, yes. Check your back email for a message from me September 4, 2015, title "elf2flt upstream repo". You were cc'd on it. I created a website that tried to act as a uclinux.org replacement for the social aspect of "where devs can learn about nommu development for linux, and come together to talk about it": https://web.archive.org/web/20160425072007/http://nommu.org/ Alas, I was busy with a half-dozen other things and handed it off to "official web people" when the company got some, and I wasn't supposed to do "community outreach" anymore because we had specific staff for that now. The discussion on the mailing list about elf2flt petered out, but it DID result in people setting up a new repo and merging some forks and buildroot switching over to the new one. Sadly, archive.org only spidered the TOP page, not the actual months: https://web.archive.org/web/20160416102445/http://lists.nommu.org/pipermail/nommu/ I can probably fish it out of my old mbox files if anybody cares. But the point is, elf2flt got maintained again (not by me, I didn't have the expertise), and I switched over to fdpic anyway. Alas reliance of fdpic means that trying to do cortex-m in mkroot, I was stuck using https://github.com/mickael-guene/fdpic_manifest and waiting for it to go upstream, which is _why_ I was doing the static pie toolchain for that architecture until such time as gcc+binutils+linux caught up with the group. (I think I saw arm fdpic support go into the kernel. Rich said that musl should work, last I asked him. I haven't checked upstream gcc or binutils recently, mostly because Rich won't update musl-cross-make and redoing my https://github.com/landley/toybox/blob/master/scripts/mcm-buildall.sh to build without it is nontrivial. For one thing I don't think https://github.com/richfelker/musl-cross-make/blob/master/patches/binutils-2.33.1/0001-j2.diff ever went upstream either...) >> UCLINUX is a long-dead distro. Linaro died in the dot-com crash and its founder >> Jeff Dionne moved to Japan for his next gig and never came back. On the way out >> he handed uclinux off to someone else, who didn't do a lot of work maintaining >> it. Most of the actual support went "upstream" into various packages (linux and >> busybox and gcc and so on) before the handoff, so you didn't NEED uclinux anymore. > > Why do you keep claiming that "UCLINUX" (why all caps?) is a distro? Because it was. Full of obsolete crap, which had its last release in 2014. > That is just not the case. uClinux has been used as an umbrella term for > patches, tools, packages relating to running Linux on a CPU with no MMU. Which is why so many people are convinced nommu linux is dead because http://uclinux.org/ was long-dead before it went down. Because of exactly that confusion. > There was a build package called uclinux-dist that you might consider a distro. > But it has always been cleanly named as "uclinux-dist". Cleanly. Uh-huh: https://web.archive.org/web/20150219205915/http://www.uclinux.org/ https://web.archive.org/web/20181202164812/http://www.uclinux.org/description/ Here are two entries from cut and pasted from https://web.archive.org/web/20181204132729/http://www.uclinux.org/status/ 12 April 2000 The release of uClinux version 2.0.38.1pre7 announced today. Download the .diff NOW! 10 January 2000 The latest release of uClinux, 2.0.38.1pre5, is announced today. The uClinux CD contains the popular uClinux System Builder Kit. This new version supports a broad spectrum of chip sets. Order the CD! > For a long time that > was the goto starting point for anyone wanting to use Linux with no-MMU. > The collection of patches to the toolchains, kernel, libraries and application > packages was a pretty high mountain to climb to get started in those early > days (late 90's and early 2000's). But through the work of a _lot_ of people > much of that change has made its way into relevant packages across the board > (from gcc to kernel to creation of uClibc, etc). It was introduced to me as a distro at the first ELC I attended in 2006. (I attended as the new busybox maintainer, there were two "uclinux developers" who wanted to say hi, we spoke for about 30 seconds. One of them wore a t-shirt with "uclinux" on it?) As I said, I never used it. I just heard about it. You're the first person to tell me it _didn't_ consider itself a distro. > But lets face it once no-MMU support was integrated into mainline linux as off > v2.5.46 it is really now just "Linux". The no-MMU support outgrew that trade marked > term. But the name has stubbornly stuck around. My successor as busybox maintainer was already ripping _uclinux_ #defines out of that codebase a dozen years ago. https://git.busybox.net/busybox/commit/?id=153fcaa6c171 >> The real nail in the coffin is the inheritor of uclinux never migrated it off >> CVS, and then the disk holding the CVS archive crashed with no backup. He came >> out with a couple more releases after that by monkey-patching the last release's >> filesystem, but the writing was on the wall and it rolled to a stop. > > No, the public facing CVS was never the master sources for the uclinux-dist. > The public facing CVS was only ever a copy of the tar ball releases. All I know is I emailed Michael Durrant in February 2015 to see if I could get a copy of the repo (preparatory for doing my triage of what was actually IN uclinux for Jeff) and he replied: > Are you are talking about the dead cvs.uclinux.org (CVS) machine or the > the uClibc website? (http://www.uclibc.org/) > > If so .. that machine is very very dead .. nothing from the harddrive was > salvageable. I will look and see if we have a CD or DVD backup but it is very > very doubtful. > > I did replace the http://mailman.uclinux.org machine a few years ago after it > had a catastrophic fail. I sent a follow-up email but didn't get a further reply. Thank you for clarifying that the stuff I was interested in was apparently never tracked under source control in the first place. > That work was slowing down due to fact that it just wasn't really > needed any more. There are way more popular build systems (eg build-root) > for building complete firmware images from scratch. Agreed, yes. >> I did a triage of its last release (from 2014) as part of my toybox roadmap: > > No, the last release was in 2016, see it archived at: > https://sourceforge.net/projects/uclinux/files/uClinux%20Stable/ They did another one after my triage then. I hadn't noticed. In early 2015 I went to https://web.archive.org/web/20150219205915/http://www.uclinux.org/ which had a cvs archive link on the left (which was dead, hence the email exchange), a page on bflt in the same nav bar (first link on that page went to the dead CVS archive, the beyondlogic page was 404, and vapier's stuff was still there but several years old, I tracked down the newer forks later), and then the "http download" link at the end of the list led to https://web.archive.org/web/20150206052300/http://www.uclinux.org/pub/uClinux/ the newest date in which was 2004. The resulting first impression was "this project is VERY DEAD". And then my email exchange with the contact info link got back "cvs archive died with no backup" and no reply to a follow-up email. > But that is all kind of archeological now, relegated to history. Agreed. > Regards > Greg Rob -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [LTP] [Automated-testing] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP] 2024-01-12 20:16 ` Rob Landley @ 2024-01-14 13:01 ` Greg Ungerer 2024-01-15 13:41 ` [LTP] [Buildroot] " Waldemar Brodkorb 1 sibling, 0 replies; 40+ messages in thread From: Greg Ungerer @ 2024-01-14 13:01 UTC (permalink / raw) To: Rob Landley, Petr Vorel, Tim Bird Cc: Niklas Cassel, Jonathan Corbet, Linux-sh list, Christophe Lyon, Randy Dunlap, linux-kernel, linux-m68k, Geert Uytterhoeven, Linux ARM, John Paul Adrian Glaubitz, linux-riscv, buildroot, ltp, automated-testing Hey Rob, Reading your emails can be exhausting :-) On 13/1/24 06:16, Rob Landley wrote: > On 1/10/24 20:25, Greg Ungerer wrote: >> Sorry Rob, but I think you are wrong about a number of things here. > > Happy to be corrected. I learned most of this stuff by people pointing things > out I didn't know. But I _have_ been trying to collect this info for about 15 > years now... > >> I know, I was there at the coal face so to speak during the early >> years of uClinux and right up today. I feel I have to correct some of >> the things you claim. > > I only heard about the politics long after the fact, and the stories have a lot > of elisions because what happens in Utah sadly does NOT stay in Utah. I came to Lineo through the Moreton Bay acquisition, and I worked with Jeff, Tim and Erik. Actually I had been collaborating with Jeff when still at RT-control before that as well. Lineo was a wild ride, to say the least. > I didn't get involved in busybox until either 2002 or 2003 depending how you > want to count it. After the dot com crash, around the time of the SCO trial. > Erik Andersen having once worked there was about as relevant as Linus' day job > at Transmeta, or busybox having started life years earlier as Debian's answer to > Red Hat Nash before being abandoned for 3 years before Erik picked it up. I > heard that this stuff was used in uClinux, but its use in linksys routers was > far more immediately relevant. > > Shortly after the dot-com crash Ray Noorda went senile and got elder abused into > allowing the SCO lawsuits run by the canopy group (his personal money managers, > gone rogue), by which time Erik Andersen and Jeff Dionne and Tim Bird and so on > had all gone on to other employers. Tim heading to Sony, Jeff moved to Japan to > do hardware and leaving uClinux behind, and Erik continued busybox and uclibc as > personal projects from a server connected to the DSL line in his basement. > > By the time I showed up uclinux never came up on the mailing list, the IRC > channel, or in private email. I was introduced to busybox+uclibc by tomsrtbt, > and the highest profile consumer of Erik's output was Linksys routers. > > My goal with busybox was making it work in a development environment, initially > to replace most of the the 110 megabytes of gnu bloatware in Linux From Scratch > with the 1.7 megabytes of tomsrtbt to free up some of the 700 megabyte budget of > knoppix CDs. Making that actually _work_ took long enough to accomplish that > "knoppix" stopped being "the bootable CD" and every distro had one now (usually > as their install CD: try it live then install from the desktop you'd booted into > if you want to keep it). > > After I finally got it all working (https://landley.net/aboriginal/news.html > either the 1.0 release rebuilding itself under itself on September 5, 2010 or > the 1.1 release building Linux From Scratch 6.7 under the result on October 2, > 2011 depending how you want to measure), I moved on to focusing on toybox > development instead (trying to get it into Android so THAT could become > self-hosting and provide a real development environment with a USB > keyboard+mouse and chromecast to a big TV), and other people took over creating > Adelie Linux and Alpine Linux and so on based on busybox instead of gnu tools. > > I didn't get into nommu development full-time until 2014 when Jeff Dionne hired > me to work on his j-core project. I'd heard the name, but never spoken to him > before he reached out to me by email, then phone interview, then flew me to Japan. > > While working for Jeff I asked various computer history questions because it's a > hobby of mine, but there's been a pandemic between then and now. I wasn't there, > the details of the answers I _did_ get are fuzzy by now, and there were some... > stories. Let's just say a canadian who moved to Utah and then moved to LITERALLY > THE OTHER SIDE OF THE PLANET may may have seen some stuff. I've also bumped into > Tim Bird on and off since 2006 (through CELF/ELC and the Linux Jamboree at > Nakano Sun Plaza in Tokyo and > https://elinux.org/CELF_Project_Proposal/Combine_tcg_with_tcc and so on) and > he's never spoken of Lineo once. > > The work I did helped kill the old uclinux distro, although I didn't realize it I think you are giving yourself a little bit too much credit here... > at the time. The various package patches it had carried were already pushed > upstream into linux and gcc and so on, and busybox worked as a nommu userspace. > By making "build a busybox+uclibc system with a vanilla kernel" easier to do and > more powerful by itself, uclinux became less relevant. Various people sent me > nommu fixes when I was maintaining busybox, which was like sending me fixes for > running busybox on linksys (or openwrt) routers. I didn't have that hardware, > but I knew the _general_ theory and tried not to break it. Don't do unaligned > access, be careful of endianness, and here's what's necessary for nommu: Surely this is the end goal though? Build systems should not have to carry patches in a perfect world. > https://git.busybox.net/busybox/commit/?id=b1b3cee831bc > https://git.busybox.net/busybox/commit/?id=03628c8f75ba > https://git.busybox.net/busybox/commit/?id=b21837714a37 > > When buildroot metastasized from uclibc's test suite (what packages build under > uclibc? Let's regression test them and slot them together into a testable root > filesystem) into a new distro (the first post to its mailing list in 2006 is > from me because I abused my busybox maintainer root access to the shared server > to set up a third list, because buildroot discussion was smothering uclibc > development on that list), buildroot became the logical base point for nommu > systems. If you wanted a busybox+uclibc root filesystem, buildroot provided an > up to date actively developed one with build recipes for hundreds of packages, > and a busy community where you could ask questions. > > Meaning there was already no reason to install uclinux in 2006. It had been > replaced by either a simple busybox+uclibc root filesystem, or buildroot. > > So "me noticing this area in 2002", "me accidentally helping finish off uclinux > in passing in 2006", and "me getting involved in nommu development in 2014 > because Jeff Dionne flew me to Japan and personally explained it to me" does not > give me any direct experience with what went on at lineo before that, true. Just > like I never visited transmeta. > >>> You can't fork() on nommu because copies of the mappings have different >>> addresses, meaning any pointers in the copied mappings would point into the OLD >>> mappings (belonging to the parent process), and fixing them up is 100% >>> equivalent to the "garbage collection in C" problem. (It's AI-complete. Of the >>> C3PO kind, not the "autocorrect with syntax checking" kind.) People get hung up >>> on the "it would be very inefficient to do that because no copy-on-write" >>> problem and miss the "the child couldn't FUNCTION because its pointer variables >>> all contain parent addresses" problem. >>> >>> So instead vfork() creates a child with all the same memory mappings (a bit like >>> a thread) and freezes the parent process until that child discards those >>> mappings, either by calling exec() or _exit(). (At which point the parent gets >>> un-suspended.) >> >> Just to be clear, vfork is not a uClinux invention. > > I never said vfork() was a uClinux invention. I'm describing how Linux does > nommu. Doing nommu on linux hasn't had anything to do with uclinux for 20 years. > > Confusing "uclinux" with "nommu" is like confusing "knoppix" with "Linux Live > CD". We've moved on a bit since one magic distro pioneered the technique literal > decades ago, there are other options now. > > I first looked at uClinux itself in 2015 because Jeff Dionne asked me to see if > there was anything left to salvage out of it, resulting in > https://github.com/landley/toybox/commit/469d7f11b66d and the answer "no, there > isn't" nine years ago. > >> It dates way back to the >> early BSD UNIX days. > > I know. > >> It just so happens that it fits in very nicely with >> the no-MMU model of the world. > > Ken and Dennis' ported Unix from a PDP-7 to a PDP-11/20, which didn't have an > MMU. Dennis wrote about it here: > > https://www.bell-labs.com/usr/dmr/www/odd.html > > The PDP-11/45 the upgraded to did, and the system call sematics changed quite a > bit in those early years before stabilizing with Unix v7 (largely because AT&T > stopped allowing Bell Labs releases to go out to the public after that one, so > v8, v9, and v10 saw almost no use, and then they started over with Plan 9.) > > So yeah I knew about BSD coming up with the name, but I thought "vfork()" was > just "what fork() used to do circa Unix v3 or similar"? > > I was trying to explain why vfork() is still in use NOW. It's not just nommu > either, it's also useful when forking from heavily threaded processes, because > normal fork() will freeze all the threads in the process causing a latency spike > and potentially quite large allocation because in that case it breaks all > copy-on-write up front rather than trying to untangle the layers of sharing. But > vfork() will only freeze the one thread that called it, and doesn't allocate new > memory before the exec() or _exit() of the new child process. (You may hit a vma > lock if other threads try to tear down the mappings, I haven't tried. But if you > DON'T do something stupid, you don't get a latency spike on the other threads.) > >>> So they invented FDPIC, > > "They" being the blackfin devs, I think. No, I believe it was David Howells that did it originally for the Fujitsu FRV architecture. >>> which is ELF with FOUR base pointers. Each major section >>> (rodata, text, data, and bss) has its own base pointer, so you need to find >>> smaller chunks of memory to load them into (and thus it can work on a more >>> fragmented system), AND it means that two instances of the same program can >>> share the read-only sections (rodata and text) so you only need new copies of >>> the writeable segments (data and bss. And the heap. And the stack.) >> >> It is worth noting that to run PIE style ELF binaries on no-MMU systems you >> actually use the FDPIC ELF loader - not the traditional linux ELF loader (binfmt_elf). > > Yup. Although that's mostly because binfmt_elf refuses to build without > CONFIG_MMU due to some stupid #ifdefs and assumptions. > > My understanding (mostly from making puppy eyes at Rich Felker) is that the > bimfmt_fdpic loader can load conventional ELF binaries just fine, the way the > ext4 driver can mount ext3 or ext2. Unfortunately, my attempts to build > bimfmt_fdpic on i386 and arm64 and mips and powerpc and so on to TRY that hit > the same "kconfig and the #ifdefs really dowanna allow this" nonsense. > > I note that qemu's application emulation handles conventional ELF, static PIE, > and fdpic pretty much interchangeably. THEY didn't see the need to have two > redundant loaders doing the same thing, with only one understanding some fairly > minor extensions. But the linux-kernel community seems to segregate out nommu > support and quarrantine it, for no obvious reason. (And then make the nommu > stuff build ONLY in one mode and the with-mmu stuff build ONLY in the other mode > and NEVER THE TWAIN SHALL MEET, again for no obvious reason.)> >> Using this setup is going to become much easier now that uClibc has native >> support for generating a relevant library and ld.so for noMMU linux (certainly >> for m68k, arm and riscv at least) - as of version 1.0.45. > > I haven't followed buildroot's uclibc fork since uclibc.org died, I switched to > musl instead. > >>> (The earlier binflt format is an a.out variant with 4 base pointers. FDPIC is >>> the ELF version of the same idea. Since a.out went bye-bye binflt is obsolete, >>> but not everybody's moved off it yet because so many nommu people are still >>> using 2.6 or even earlier, and also using gcc 3.x or 2.x toolchains.) >> >> Flat format (binfmt_flat) is in no way related to a.out. It is not derived from >> it and shares nothing with it. A.out being removed from the kernel in no way affects >> flat format. It doesn't magically make it obsolete. It isn't going anywhere. > > *shrug* I'll take your word for it. > > Back when I tried to make it work circa > https://landley.net/notes-2014.html#07-12-2014 the binflt plumbing was so > bit-rotted that A) there was no official repository for it, B) it was > implemented as a weird post-processing step where you ran an "elf2flt" binary > but it could not actually digest a normal ELF file, instead you made a SPECIAL > elf file with various compiler flags, which the postprocessing tool then > converted to a flat file. > > (This is another variant of "embedded devs get something to work and then keep > using the old version FOREVER". I could download 10 year old binflt toolchains > that produced binaries, but nobody seemed to have built a NEW one in years.) > > So I shoehorned basic elf2flt support into my build system: > > https://github.com/landley/aboriginal/commit/50d139530dd4 > > And then I asked Yoshinori Sato what version _he'd_ used when adding h8300 > support to Linux (since that was nommu) and switched to his version, which he'd > fixed a lot of stuff in and which worked with newer compiler versions and had > commits less than a year old in the tree: > > https://github.com/landley/aboriginal/commit/247ee063028f > > And around that time I emailed various people who had once been interested in > this, and they started a new binflt tree (merging the various trees I'd found) > and switched buildroot over to use it. But I didn't track it much beyond that > because instead Jeff paid Rich Felker to add fdpic support to musl and we > switched j-core over to that instead. > > What little I know about binflt comes from reading the code and from Jeff Dionne > trying to explain it to me back in 2014 and 2015, and mostly he was concerned > that you had to use older versions of gcc (despite sato-san's upgrades) because > current versions of gcc could do some pointer range thing that broke design > assumptions in elf2flt, which apparently weren't fixable because information had > already been discarded when the .o file was generated, so the thing consuming > that output didn't have all the information it needed to link. (You'd have to > ask Jeff, I never quite wrapped my head around what the specific issue was and > it's a decade ago now. My impression was you'd _mostly_ get lucky and it would > usually work with modern gcc, but you rolled the dice with every code change...) > >> I don't see how this is related to versions of kernel or gcc or toolchains either. >> Flat format works on every kernel up to todays v6.7. The conversion tool works >> with the latest gcc and binutils (gcc-13.2 and bintuils-2.41 as of today) > > Again, I'll take your word for it. > >> and many >> versions going back for the best part of 20 years. Sure that elf2flt tool has been >> forked a lot it has a spotty history of getting updated. But, hey, as of today it >> looks pretty up to date across most architectures. > > Um, yes. Check your back email for a message from me September 4, 2015, title > "elf2flt upstream repo". You were cc'd on it. > > I created a website that tried to act as a uclinux.org replacement for the > social aspect of "where devs can learn about nommu development for linux, and > come together to talk about it": > > https://web.archive.org/web/20160425072007/http://nommu.org/ > > Alas, I was busy with a half-dozen other things and handed it off to "official > web people" when the company got some, and I wasn't supposed to do "community > outreach" anymore because we had specific staff for that now. The discussion on > the mailing list about elf2flt petered out, but it DID result in people setting > up a new repo and merging some forks and buildroot switching over to the new one. > > Sadly, archive.org only spidered the TOP page, not the actual months: > > https://web.archive.org/web/20160416102445/http://lists.nommu.org/pipermail/nommu/ > > I can probably fish it out of my old mbox files if anybody cares. But the point > is, elf2flt got maintained again (not by me, I didn't have the expertise), and I > switched over to fdpic anyway. My point is that the elf2flt tool was very widely forked. At the very least pretty much every architecture port has its own, and in many cases many vendors had their own as well. So, yeah, its development and progress got messy. But I did get involved in that effort to get elf2flt centered and restored to some semblance of being maintained. The git hub repository is now the main focus, https://github.com/uclinux-dev/elf2flt. It is maintained and pretty much up to date. So I think we can notch that up as a win. Of course the flat format itself has issues, it is far from perfect. But for the most part it works fine for its intended purpose. elf-fdpic is superior in many respects, but requires much more invasive compiler support - that takes a lot more effort to create for a new architecture. > Alas reliance of fdpic means that trying to do cortex-m in mkroot, I was stuck > using https://github.com/mickael-guene/fdpic_manifest and waiting for it to go > upstream, which is _why_ I was doing the static pie toolchain for that > architecture until such time as gcc+binutils+linux caught up with the group. (I > think I saw arm fdpic support go into the kernel. Rich said that musl should > work, last I asked him. I haven't checked upstream gcc or binutils recently, > mostly because Rich won't update musl-cross-make and redoing my > https://github.com/landley/toybox/blob/master/scripts/mcm-buildall.sh to build > without it is nontrivial. For one thing I don't think > https://github.com/richfelker/musl-cross-make/blob/master/patches/binutils-2.33.1/0001-j2.diff > ever went upstream either...) Elf-fdpic on ARM works fine, I test it regularly. You don't need any patches for current versions of binutils, gcc, or linux kernel. Oh, I test with uClibc-ng, don't know about other libraries. And PIE style ELF works at least on ARM, M68K, RISCV and XTENSA nommu configurations. Again with uClibc-ng at least. >>> UCLINUX is a long-dead distro. Linaro died in the dot-com crash and its founder >>> Jeff Dionne moved to Japan for his next gig and never came back. On the way out >>> he handed uclinux off to someone else, who didn't do a lot of work maintaining >>> it. Most of the actual support went "upstream" into various packages (linux and >>> busybox and gcc and so on) before the handoff, so you didn't NEED uclinux anymore. >> >> Why do you keep claiming that "UCLINUX" (why all caps?) is a distro? > > Because it was. Full of obsolete crap, which had its last release in 2014. > >> That is just not the case. uClinux has been used as an umbrella term for >> patches, tools, packages relating to running Linux on a CPU with no MMU. > > Which is why so many people are convinced nommu linux is dead because > http://uclinux.org/ was long-dead before it went down. Because of exactly that > confusion. I am very confused by this. Who thinks nommu linux is dead? I am continually surprised by how many people still use it. >> There was a build package called uclinux-dist that you might consider a distro. >> But it has always been cleanly named as "uclinux-dist". > > Cleanly. Uh-huh: > > https://web.archive.org/web/20150219205915/http://www.uclinux.org/ > > https://web.archive.org/web/20181202164812/http://www.uclinux.org/description/ > > Here are two entries from cut and pasted from > https://web.archive.org/web/20181204132729/http://www.uclinux.org/status/ > > 12 April 2000 > The release of uClinux version 2.0.38.1pre7 announced today. Download the .diff NOW! > > 10 January 2000 > The latest release of uClinux, 2.0.38.1pre5, is announced today. The uClinux CD > contains the popular uClinux System Builder Kit. This new version supports a > broad spectrum of chip sets. Order the CD! More than anything they point out the continual overloading of the term and general confusion on what the term actually covers. It is trivial to find examples of its use that are clearly not related to a distribution. Grep for it in the Linux kernel sources one, there is plenty, eg: mm/nommu.c: * handle mapping creation for uClinux arch/m68k/include/asm/flat.h: * flat.h -- uClinux flat-format executables fs/Kconfig.binfmt: Support uClinux FLAT format binaries. fs/binfmt_elf_fdpic.c: * - on uClinux the holes between may actually be filled with system ... Despite what you think my experience is that most people simply equate uClinux with no-mmu linux. And I for one am pretty comfortable with that. >> For a long time that >> was the goto starting point for anyone wanting to use Linux with no-MMU. >> The collection of patches to the toolchains, kernel, libraries and application >> packages was a pretty high mountain to climb to get started in those early >> days (late 90's and early 2000's). But through the work of a _lot_ of people >> much of that change has made its way into relevant packages across the board >> (from gcc to kernel to creation of uClibc, etc). > > It was introduced to me as a distro at the first ELC I attended in 2006. (I > attended as the new busybox maintainer, there were two "uclinux developers" who > wanted to say hi, we spoke for about 30 seconds. One of them wore a t-shirt with > "uclinux" on it?) > > As I said, I never used it. I just heard about it. You're the first person to > tell me it _didn't_ consider itself a distro. > >> But lets face it once no-MMU support was integrated into mainline linux as off >> v2.5.46 it is really now just "Linux". The no-MMU support outgrew that trade marked >> term. But the name has stubbornly stuck around. > > My successor as busybox maintainer was already ripping _uclinux_ #defines out of > that codebase a dozen years ago. > > https://git.busybox.net/busybox/commit/?id=153fcaa6c171 > >>> The real nail in the coffin is the inheritor of uclinux never migrated it off >>> CVS, and then the disk holding the CVS archive crashed with no backup. He came >>> out with a couple more releases after that by monkey-patching the last release's >>> filesystem, but the writing was on the wall and it rolled to a stop. >> >> No, the public facing CVS was never the master sources for the uclinux-dist. >> The public facing CVS was only ever a copy of the tar ball releases. > > All I know is I emailed Michael Durrant in February 2015 to see if I could get a > copy of the repo (preparatory for doing my triage of what was actually IN > uclinux for Jeff) and he replied: > >> Are you are talking about the dead cvs.uclinux.org (CVS) machine or the >> the uClibc website? (http://www.uclibc.org/) >> >> If so .. that machine is very very dead .. nothing from the harddrive was >> salvageable. I will look and see if we have a CD or DVD backup but it is very >> very doubtful. >> >> I did replace the http://mailman.uclinux.org machine a few years ago after it >> had a catastrophic fail. > > I sent a follow-up email but didn't get a further reply. > > Thank you for clarifying that the stuff I was interested in was apparently never > tracked under source control in the first place. I didn't say it was not tracked under source control, it was. But that was an internal company vendor tree. The external public facing CVS tree on uclinux.org was simply not a copy of that internal working tree. It was not much more than a dump of the tar ball releases (and occasionally direct fix updates). >> That work was slowing down due to fact that it just wasn't really >> needed any more. There are way more popular build systems (eg build-root) >> for building complete firmware images from scratch. > > Agreed, yes. > >>> I did a triage of its last release (from 2014) as part of my toybox roadmap: >> >> No, the last release was in 2016, see it archived at: >> https://sourceforge.net/projects/uclinux/files/uClinux%20Stable/ > > They did another one after my triage then. I hadn't noticed. > > In early 2015 I went to > https://web.archive.org/web/20150219205915/http://www.uclinux.org/ which had a > cvs archive link on the left (which was dead, hence the email exchange), a page > on bflt in the same nav bar (first link on that page went to the dead CVS > archive, the beyondlogic page was 404, and vapier's stuff was still there but > several years old, I tracked down the newer forks later), and then the "http > download" link at the end of the list led to > https://web.archive.org/web/20150206052300/http://www.uclinux.org/pub/uClinux/ > the newest date in which was 2004. > > The resulting first impression was "this project is VERY DEAD". And then my > email exchange with the contact info link got back "cvs archive died with no > backup" and no reply to a follow-up email. > >> But that is all kind of archeological now, relegated to history. > > Agreed. > >> Regards >> Greg > > Rob Regards Greg -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [LTP] [Buildroot] [Automated-testing] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP] 2024-01-12 20:16 ` Rob Landley 2024-01-14 13:01 ` Greg Ungerer @ 2024-01-15 13:41 ` Waldemar Brodkorb 2024-01-15 14:22 ` Cyril Hrubis 1 sibling, 1 reply; 40+ messages in thread From: Waldemar Brodkorb @ 2024-01-15 13:41 UTC (permalink / raw) To: Petr Vorel Cc: Niklas Cassel, Jonathan Corbet, Linux-sh list, ltp, linux-kernel, linux-m68k, Randy Dunlap, Christophe Lyon, Rob Landley, John Paul Adrian Glaubitz, Geert Uytterhoeven, linux-riscv, buildroot, Greg Ungerer, Linux ARM, automated-testing Hi, I want to clarify some things of the point of view of uClibc-ng support. There is support for following configurations for noMMU targets: ARM: - FLAT with Linuxthreads supported (for Qemu you need a Linux patch) - FDPIC with NPTL supported (NPTL works only on real hardware not in Qemu) - ELF with Thread support not working M68k: - FLAT with Linuxthreads supported - ELF with Thread support not working RISCV64: - FLAT without Thread support - ELF with Thread support not working RISCV32: - FLAT without Thread support, needs a small Linux Kernel patch SH2/J2: - FLAT with Linuxthreads supported Xtensa: - FLAT with Linuxthreads supported There are some obsolete architectures supported by uClibc-ng, but no longer supported by Linux: Blackfin: - FLAT with Linuxthreads supported - FDPIC H8300: - FLAT with Linuxthreads supported C6X: - DSBT LM32: - FLAT LTP requires NPTL to work, so the only testable platform is ARM with FDPIC right now. Unfortunately LTP 20230929 needs fork for some files: RANLIB libltp.a /home/wbx/arm/toolchain_st-stm32f746g_uclibc-ng_cortex_m7_soft_eabi_thumb_nommu/usr/lib/gcc/arm-openadk-uclinuxfdpiceabi/13.2.0/../../../../arm-openadk-uclinuxfdpiceabi/bin/ld: ../../lib/libltp.a(tst_res.o): in function `tst_fork': /home/wbx/arm/build_st-stm32f746g_uclibc-ng_cortex_m7_soft_eabi_thumb_nommu/w-ltp-20230929-1/ltp-full-20230929/lib/tst_res.c:430:(.text+0x952): undefined reference to `fork' /home/wbx/arm/toolchain_st-stm32f746g_uclibc-ng_cortex_m7_soft_eabi_thumb_nommu/usr/lib/gcc/arm-openadk-uclinuxfdpiceabi/13.2.0/../../../../arm-openadk-uclinuxfdpiceabi/bin/ld: ../../lib/libltp.a(tst_test.o): in function `fork_testrun': /home/wbx/arm/build_st-stm32f746g_uclibc-ng_cortex_m7_soft_eabi_thumb_nommu/w-ltp-20230929-1/ltp-full-20230929/lib/tst_test.c:1597:(.text+0xf4e): undefined reference to `fork' /home/wbx/arm/toolchain_st-stm32f746g_uclibc-ng_cortex_m7_soft_eabi_thumb_nommu/usr/lib/gcc/arm-openadk-uclinuxfdpiceabi/13.2.0/../../../../arm-openadk-uclinuxfdpiceabi/bin/ld: ../../lib/libltp.a(tst_test.o): in function `safe_fork': /home/wbx/arm/build_st-stm32f746g_uclibc-ng_cortex_m7_soft_eabi_thumb_nommu/w-ltp-20230929-1/ltp-full-20230929/lib/tst_test.c:435:(.text+0x345c): undefined reference to `fork' collect2: error: ld returned 1 exit status gmake[8]: *** [../../include/mk/rules.mk:45: test01] Error 1 gmake[7]: *** [../include/mk/generic_trunk_target.inc:108: all] Error 2 gmake[6]: *** [Makefile:94: lib-all] Error 2 gmake[5]: *** [/home/wbx/arm/mk/pkg-bottom.mk:141: /home/wbx/arm/build_st-stm32f746g_uclibc-ng_cortex_m7_soft_eabi_thumb_nommu/w-ltp-20230929-1/ltp-full-20230929/.build_done] Error 2 gmake[4]: *** [Makefile:61: ltp-compile] Error 2 gmake[3]: *** [mk/build.mk:221: package/compile] Error 2 gmake[2]: *** [/home/wbx/arm/mk/build.mk:176: world] Error 2 So there is really work to be done to make the existing code work on noMMU. best regards Waldemar -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [LTP] [Buildroot] [Automated-testing] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP] 2024-01-15 13:41 ` [LTP] [Buildroot] " Waldemar Brodkorb @ 2024-01-15 14:22 ` Cyril Hrubis 0 siblings, 0 replies; 40+ messages in thread From: Cyril Hrubis @ 2024-01-15 14:22 UTC (permalink / raw) To: Waldemar Brodkorb Cc: Niklas Cassel, Jonathan Corbet, Linux-sh list, ltp, linux-kernel, linux-m68k, Randy Dunlap, Christophe Lyon, Rob Landley, John Paul Adrian Glaubitz, Geert Uytterhoeven, linux-riscv, buildroot, Greg Ungerer, Linux ARM, automated-testing Hi! > RANLIB libltp.a > /home/wbx/arm/toolchain_st-stm32f746g_uclibc-ng_cortex_m7_soft_eabi_thumb_nommu/usr/lib/gcc/arm-openadk-uclinuxfdpiceabi/13.2.0/../../../../arm-openadk-uclinuxfdpiceabi/bin/ld: ../../lib/libltp.a(tst_res.o): in function `tst_fork': > /home/wbx/arm/build_st-stm32f746g_uclibc-ng_cortex_m7_soft_eabi_thumb_nommu/w-ltp-20230929-1/ltp-full-20230929/lib/tst_res.c:430:(.text+0x952): undefined reference to `fork' > /home/wbx/arm/toolchain_st-stm32f746g_uclibc-ng_cortex_m7_soft_eabi_thumb_nommu/usr/lib/gcc/arm-openadk-uclinuxfdpiceabi/13.2.0/../../../../arm-openadk-uclinuxfdpiceabi/bin/ld: ../../lib/libltp.a(tst_test.o): in function `fork_testrun': > /home/wbx/arm/build_st-stm32f746g_uclibc-ng_cortex_m7_soft_eabi_thumb_nommu/w-ltp-20230929-1/ltp-full-20230929/lib/tst_test.c:1597:(.text+0xf4e): undefined reference to `fork' > /home/wbx/arm/toolchain_st-stm32f746g_uclibc-ng_cortex_m7_soft_eabi_thumb_nommu/usr/lib/gcc/arm-openadk-uclinuxfdpiceabi/13.2.0/../../../../arm-openadk-uclinuxfdpiceabi/bin/ld: ../../lib/libltp.a(tst_test.o): in function `safe_fork': > /home/wbx/arm/build_st-stm32f746g_uclibc-ng_cortex_m7_soft_eabi_thumb_nommu/w-ltp-20230929-1/ltp-full-20230929/lib/tst_test.c:435:(.text+0x345c): undefined reference to `fork' > collect2: error: ld returned 1 exit status > gmake[8]: *** [../../include/mk/rules.mk:45: test01] Error 1 > gmake[7]: *** [../include/mk/generic_trunk_target.inc:108: all] Error 2 > gmake[6]: *** [Makefile:94: lib-all] Error 2 > gmake[5]: *** [/home/wbx/arm/mk/pkg-bottom.mk:141: /home/wbx/arm/build_st-stm32f746g_uclibc-ng_cortex_m7_soft_eabi_thumb_nommu/w-ltp-20230929-1/ltp-full-20230929/.build_done] Error 2 > gmake[4]: *** [Makefile:61: ltp-compile] Error 2 > gmake[3]: *** [mk/build.mk:221: package/compile] Error 2 > gmake[2]: *** [/home/wbx/arm/mk/build.mk:176: world] Error 2 > > So there is really work to be done to make the existing code work on noMMU. The new test library in LTP runs the actuall test in a child process, which provides all kinds of benefits, most notably isolation of the setup/cleanup/result reporting from actuall test code that may crash. This is of course useless on nommu targets, so I suppose that we would need a test library tailored for nommu first. However the testcases themselve fork quite often too. Which means that some kind of parameter marshaling into a string needs to be done for such tests as well each test needs to be adjusted to use that in a case of nommu. All in all getting into a state where majority of tests runs on nommu would be a major effort. -- Cyril Hrubis chrubis@suse.cz -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [LTP] [Automated-testing] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP] 2024-01-10 19:23 ` Rob Landley 2024-01-10 21:17 ` Niklas Cassel via ltp 2024-01-11 2:25 ` Greg Ungerer @ 2024-01-11 13:11 ` Geert Uytterhoeven 2024-01-11 13:19 ` Greg Ungerer 2 siblings, 1 reply; 40+ messages in thread From: Geert Uytterhoeven @ 2024-01-11 13:11 UTC (permalink / raw) To: Rob Landley Cc: Linux ARM, Niklas Cassel, Jonathan Corbet, Linux-sh list, Randy Dunlap, linux-kernel, linux-m68k, Christophe Lyon, John Paul Adrian Glaubitz, linux-riscv, buildroot, Greg Ungerer, ltp, automated-testing Hi Rob, On Wed, Jan 10, 2024 at 8:17 PM Rob Landley <rob@landley.net> wrote: > You can't fork() on nommu because copies of the mappings have different > addresses, meaning any pointers in the copied mappings would point into the OLD > mappings (belonging to the parent process), and fixing them up is 100% > equivalent to the "garbage collection in C" problem. (It's AI-complete. Of the > C3PO kind, not the "autocorrect with syntax checking" kind.) People get hung up > on the "it would be very inefficient to do that because no copy-on-write" > problem and miss the "the child couldn't FUNCTION because its pointer variables > all contain parent addresses" problem. Actually you can implement fork(), if you teach the compiler to use separate stacks for return addresses and data: - The first stack would contain only absolute addresses, to be relocated after copying, - The second stack would contain integers and relative pointers (see FDPIC below), which do not need relocation after copying. > The OTHER fun thing about nommu is you can't run conventional ELF binaries, > because everything is linked at fixed address. So you might be able to run ONE > instance of the program as your init task, assuming those addresses were > available even then, but as soon as you try to run a second one it's a conflict. > > The quick and dirty work around is to make PIE binaries, which can relocate > everything into available space, which works but doesn't scale. The problem with > ELF PIE is that everything is linked contiguously from a single base pointer, > meaning your text, rodata, data, and bss segments are all one linear blob. So if > you run two instances of bash, you've loaded two copies of the test and the > rodoata. This fills up your memory fast. > > AND PIE requires contiguous memory, which nommu is bad at providing because it > has no page tables to remap stuff. With an mmu it can coalesce scattered > physical pages into a virtually contiguous range, but without an mmu you can > have plenty of memory free but in tiny chunks, none big enough to satisfy an > allocation request. > > So they invented FDPIC, which is ELF with FOUR base pointers. Each major section > (rodata, text, data, and bss) has its own base pointer, so you need to find > smaller chunks of memory to load them into (and thus it can work on a more > fragmented system), AND it means that two instances of the same program can > share the read-only sections (rodata and text) so you only need new copies of > the writeable segments (data and bss. And the heap. And the stack.) Or Amiga LoadSeg() relocatable binaries and shared libraries ;-) As this supported splitting code, data, and bss in lots of smaller hunks, it could counter fragmented memory quite well. BTW, can't you run and thus test nommu-binaries under normal Linux, too? Gr{oetje,eeting}s, Geert -- Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [LTP] [Automated-testing] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP] 2024-01-11 13:11 ` [LTP] " Geert Uytterhoeven @ 2024-01-11 13:19 ` Greg Ungerer 0 siblings, 0 replies; 40+ messages in thread From: Greg Ungerer @ 2024-01-11 13:19 UTC (permalink / raw) To: Geert Uytterhoeven, Rob Landley Cc: Niklas Cassel, Jonathan Corbet, Linux-sh list, Randy Dunlap, linux-kernel, linux-m68k, Linux ARM, John Paul Adrian Glaubitz, linux-riscv, buildroot, Christophe Lyon, ltp, automated-testing On 11/1/24 23:11, Geert Uytterhoeven wrote: > Hi Rob, > > On Wed, Jan 10, 2024 at 8:17 PM Rob Landley <rob@landley.net> wrote: >> You can't fork() on nommu because copies of the mappings have different >> addresses, meaning any pointers in the copied mappings would point into the OLD >> mappings (belonging to the parent process), and fixing them up is 100% >> equivalent to the "garbage collection in C" problem. (It's AI-complete. Of the >> C3PO kind, not the "autocorrect with syntax checking" kind.) People get hung up >> on the "it would be very inefficient to do that because no copy-on-write" >> problem and miss the "the child couldn't FUNCTION because its pointer variables >> all contain parent addresses" problem. > > Actually you can implement fork(), if you teach the compiler to use > separate stacks for return addresses and data: > - The first stack would contain only absolute addresses, to be > relocated after copying, > - The second stack would contain integers and relative pointers > (see FDPIC below), which do not need relocation after copying. > >> The OTHER fun thing about nommu is you can't run conventional ELF binaries, >> because everything is linked at fixed address. So you might be able to run ONE >> instance of the program as your init task, assuming those addresses were >> available even then, but as soon as you try to run a second one it's a conflict. >> >> The quick and dirty work around is to make PIE binaries, which can relocate >> everything into available space, which works but doesn't scale. The problem with >> ELF PIE is that everything is linked contiguously from a single base pointer, >> meaning your text, rodata, data, and bss segments are all one linear blob. So if >> you run two instances of bash, you've loaded two copies of the test and the >> rodoata. This fills up your memory fast. >> >> AND PIE requires contiguous memory, which nommu is bad at providing because it >> has no page tables to remap stuff. With an mmu it can coalesce scattered >> physical pages into a virtually contiguous range, but without an mmu you can >> have plenty of memory free but in tiny chunks, none big enough to satisfy an >> allocation request. >> >> So they invented FDPIC, which is ELF with FOUR base pointers. Each major section >> (rodata, text, data, and bss) has its own base pointer, so you need to find >> smaller chunks of memory to load them into (and thus it can work on a more >> fragmented system), AND it means that two instances of the same program can >> share the read-only sections (rodata and text) so you only need new copies of >> the writeable segments (data and bss. And the heap. And the stack.) > > Or Amiga LoadSeg() relocatable binaries and shared libraries ;-) > As this supported splitting code, data, and bss in lots of smaller > hunks, it could counter fragmented memory quite well. > > BTW, can't you run and thus test nommu-binaries under normal Linux, too? Yes, you can. The flat format loader can be built for MMU arm and m68k Linux. It will happily load and run flat format binaries on normal VM Linux. I test that often on m68k (on ColdFire platforms). Regards Greg -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [LTP] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP] 2024-01-08 9:03 ` Petr Vorel 2024-01-08 10:07 ` Cyril Hrubis @ 2024-01-09 20:24 ` Rob Landley 2024-01-09 23:17 ` Greg Ungerer 2024-01-10 13:33 ` Petr Vorel 1 sibling, 2 replies; 40+ messages in thread From: Rob Landley @ 2024-01-09 20:24 UTC (permalink / raw) To: Petr Vorel Cc: Niklas Cassel, Jonathan Corbet, Linux-sh list, Christophe Lyon, Randy Dunlap, linux-kernel, linux-m68k, Geert Uytterhoeven, Linux ARM, John Paul Adrian Glaubitz, linux-riscv, buildroot, Greg Ungerer, ltp, automated-testing On 1/8/24 03:03, Petr Vorel wrote: > Hi Rob, all, > > [ Added Niklas Cassel, who is maintainer of qemu_riscv64_nommu_virt_defconfig in > buildroot ] Hi Niklas! >> Buildroot also apparently has an LTP package selectable in menuconfig: > >> https://github.com/buildroot/buildroot/tree/master/package/ltp-testsuite > >> But I haven't tried it... > > I'm the maintainer of the LTP package in buildroot in my private time. > BTW I spent quite a lot of time fixing LTP (and some other system packages, > e.g. nfs-utils) compilation on some old legacy architectures reported via > http://autobuild.buildroot.net/ I've never used in the reality. > But I certainly don't have time to drive nommu support in my private time. > I don't even have an interest, I don't use any nommu device. I do, but I've never done much with LTP, and I have my hands full with toybox and mkroot already. > Therefore nobody who is not involved in nommu will not find a time to support it > in LTP (support does not mean just to add the functionality to the new C API, > but run tests on nommu and fix failing bugs). I suppose nobody is paid to work > on nommu platforms, it would have to be a hobby project, right? A bunch of people are paid to work on nommu platforms, and I've worked with them a bunch, but none of them talk to linux-kernel. They find the culture toxic, insular, and categorically dismissive of their interests. For example, cortex-m is a large nommu platform on which vendors support Linux BSPs, but notice how page 8 of https://www.microsemi.com/document-portal/doc_view/132181-linux-cortex-m-users-manual points at a cross compiler toolchain from _2010_ and page 4 says they're booting a 2.6.33 kernel? I'm a bit weird in that I try to get CURRENT stuff to work on nommu, and a lot of people have been happy to consume my work, but getting any of them to post directly to linux-kernel is like pulling teeth. > But as I said, if anybody from nommu decides to maintain it in LTP, I'll try to > support him in my free time (review patches, give advices). And if nobody > stands, this patchset which removes the support in the old API will be merged > after next LTP release (in the end of January). What does the API migration do? Is there a page on it ala OABI vs EABI in arm or something? Rob -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [LTP] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP] 2024-01-09 20:24 ` [LTP] " Rob Landley @ 2024-01-09 23:17 ` Greg Ungerer 2024-01-10 5:47 ` Rob Landley 2024-01-10 13:33 ` Petr Vorel 1 sibling, 1 reply; 40+ messages in thread From: Greg Ungerer @ 2024-01-09 23:17 UTC (permalink / raw) To: Rob Landley, Petr Vorel Cc: Niklas Cassel, Jonathan Corbet, Linux-sh list, Christophe Lyon, Randy Dunlap, linux-kernel, linux-m68k, Geert Uytterhoeven, Linux ARM, John Paul Adrian Glaubitz, linux-riscv, buildroot, ltp, automated-testing On 10/1/24 06:24, Rob Landley wrote: > On 1/8/24 03:03, Petr Vorel wrote: >> Hi Rob, all, >> >> [ Added Niklas Cassel, who is maintainer of qemu_riscv64_nommu_virt_defconfig in >> buildroot ] > > Hi Niklas! > >>> Buildroot also apparently has an LTP package selectable in menuconfig: >> >>> https://github.com/buildroot/buildroot/tree/master/package/ltp-testsuite >> >>> But I haven't tried it... >> >> I'm the maintainer of the LTP package in buildroot in my private time. >> BTW I spent quite a lot of time fixing LTP (and some other system packages, >> e.g. nfs-utils) compilation on some old legacy architectures reported via >> http://autobuild.buildroot.net/ I've never used in the reality. >> But I certainly don't have time to drive nommu support in my private time. >> I don't even have an interest, I don't use any nommu device. > > I do, but I've never done much with LTP, and I have my hands full with toybox > and mkroot already. > >> Therefore nobody who is not involved in nommu will not find a time to support it >> in LTP (support does not mean just to add the functionality to the new C API, >> but run tests on nommu and fix failing bugs). I suppose nobody is paid to work >> on nommu platforms, it would have to be a hobby project, right? > > A bunch of people are paid to work on nommu platforms, and I've worked with them > a bunch, but none of them talk to linux-kernel. They find the culture toxic, > insular, and categorically dismissive of their interests. I have been involved in the kernel nommu space for 20 years, and sure, there is some of that. But mostly spending some time and effort to get involved pays off. I have seen potential contributors show up with some arrogant attitudes too, so it cuts both ways here. The m68k community I have been part of has been nothing but welcoming. The mm people have tried hard to keep nommu support up-to-date where almost none of them actually have a vested interest in doing so. What I have seen is that many companies working in this space just don't want to spend the time and effort to go mainline. That is a business decision they make, and that is fine. Heck my work in actual mainline has never really been paid for by any company and I have sunk a _lot_ of time into it. (Full disclosure I did get paid to work on early porting and support - just not geting it into mainline and maintain it there). > For example, cortex-m is a large nommu platform on which vendors support Linux > BSPs, but notice how page 8 of > https://www.microsemi.com/document-portal/doc_view/132181-linux-cortex-m-users-manual > points at a cross compiler toolchain from _2010_ and page 4 says they're booting > a 2.6.33 kernel? Any company/person who follows the route of not working with the linux kernel community to get their work included is going to inevitably get stuck on older versions of everything. > I'm a bit weird in that I try to get CURRENT stuff to work on nommu, and a lot > of people have been happy to consume my work, but getting any of them to post > directly to linux-kernel is like pulling teeth. I regularly test nommu configurations (as in every kernel rc and release) on m68k and at least every release on other architectures like arm(*) and recently on riscv as well. (*) somewhat annoyingly needing a minor patch to run the versatile qemu platform I like to test with. But hey, that is on me :-) Regards Greg >> But as I said, if anybody from nommu decides to maintain it in LTP, I'll try to >> support him in my free time (review patches, give advices). And if nobody >> stands, this patchset which removes the support in the old API will be merged >> after next LTP release (in the end of January). > > What does the API migration do? Is there a page on it ala OABI vs EABI in arm or > something? > > Rob -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [LTP] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP] 2024-01-09 23:17 ` Greg Ungerer @ 2024-01-10 5:47 ` Rob Landley 2024-01-10 14:46 ` Greg Ungerer 0 siblings, 1 reply; 40+ messages in thread From: Rob Landley @ 2024-01-10 5:47 UTC (permalink / raw) To: Greg Ungerer, Petr Vorel Cc: Niklas Cassel, Jonathan Corbet, Linux-sh list, Christophe Lyon, Randy Dunlap, linux-kernel, linux-m68k, Geert Uytterhoeven, Linux ARM, John Paul Adrian Glaubitz, linux-riscv, buildroot, ltp, automated-testing On 1/9/24 17:17, Greg Ungerer wrote: > > On 10/1/24 06:24, Rob Landley wrote: >> On 1/8/24 03:03, Petr Vorel wrote: >>> Hi Rob, all, >>> >>> [ Added Niklas Cassel, who is maintainer of qemu_riscv64_nommu_virt_defconfig in >>> buildroot ] >> >> Hi Niklas! >> >>>> Buildroot also apparently has an LTP package selectable in menuconfig: >>> >>>> https://github.com/buildroot/buildroot/tree/master/package/ltp-testsuite >>> >>>> But I haven't tried it... >>> >>> I'm the maintainer of the LTP package in buildroot in my private time. >>> BTW I spent quite a lot of time fixing LTP (and some other system packages, >>> e.g. nfs-utils) compilation on some old legacy architectures reported via >>> http://autobuild.buildroot.net/ I've never used in the reality. >>> But I certainly don't have time to drive nommu support in my private time. >>> I don't even have an interest, I don't use any nommu device. >> >> I do, but I've never done much with LTP, and I have my hands full with toybox >> and mkroot already. >> >>> Therefore nobody who is not involved in nommu will not find a time to support it >>> in LTP (support does not mean just to add the functionality to the new C API, >>> but run tests on nommu and fix failing bugs). I suppose nobody is paid to work >>> on nommu platforms, it would have to be a hobby project, right? >> >> A bunch of people are paid to work on nommu platforms, and I've worked with them >> a bunch, but none of them talk to linux-kernel. They find the culture toxic, >> insular, and categorically dismissive of their interests. > > I have been involved in the kernel nommu space for 20 years, and sure, there is > some of that. But mostly spending some time and effort to get involved pays off. > I have seen potential contributors show up with some arrogant attitudes too, > so it cuts both ways here. > > The m68k community I have been part of has been nothing but welcoming. The mm > people have tried hard to keep nommu support up-to-date where almost none of them > actually have a vested interest in doing so. > > What I have seen is that many companies working in this space just don't want > to spend the time and effort to go mainline. Sometimes they don't bother. Sometimes there's a language barrier. Sometimes they can't get anything newer than 2.6 working because that's the BSP they were given so what's the point of trying to engage upstream? Sometimes they think it's their upstream vendor's responsibility. Sometimes they poke their head up and get it bit off ala http://landley.net/notes-2008.html#11-12-2008 and then that serves as a warning to others for generations. Sometimes the company's legal department thinks it's a terrible idea to attract attention from people like the SFLC. And sometimes... The SmartFusion 2 project I was doing on cortex-m was for a project that was to be launched into space on a NASA rocket, and thus fell under ITAR export regulations (as the entire US space program has since 1996 due to https://en.wikipedia.org/wiki/International_Traffic_in_Arms_Regulations#:~:text=Intelsat ) and my manager explained it to me as: "If I buy a screwdriver from Home Depot, it's just a screwdriver. If I use it to turn a screw on a spacecraft it is now a munition and cannot be discussed with non-US persons". The stupid linked above was: 1) cryptography was categorized as a munition until Bill Clinton relaxed it via executive order, because 56 bit https was preventing anybody from trusting the web with their credit card data. 2) Before that, in 1996, china wanted to launch a satellite into space with crypto stuff so they negotiated with the USA to get some cryptographic hardware which was delivered/installed under armed guard and escorted to the launch pad. 3) The rocket blew up on launch, scattering debris over a chinese city (becuase of COURSE china's rocket launches go over large population centers). The crypto hardware was never recovered. 4) It became a scandal. Congress freaked. And somehow in the scuffle ITAR export regulations were extended from cryptography to the entire US space program. 5) The US space program dried up and blew away, as engineers had to choose between "I can work on space stuff" and "I can have any sort of professional network online". (Because who online is a "non-US person"? That includes canada. You can't discuss ITAR subjects with canadians. Or foreign nationals living in the USA. You couldn't ask Alan Cox a question about an ITAR project.) So that's a whole category that stays very quiet about what they do, and whose legal analysis of the GPL is "we're making 3 of these and shooting them into space, if you retrieve one from space and demand source code from us we will forward you to the relevant federal agencies, and there's a nonzero chance you'll be on a black site in Diego Garcia within 24 hours where they will figure out how you did that. Or maybe you'll get the code. Who knows?" Me, I try to avoid that kind of contract... > That is a business decision they > make, and that is fine. Heck my work in actual mainline has never really been > paid for by any company and I have sunk a _lot_ of time into it. (Full disclosure > I did get paid to work on early porting and support - just not geting it into > mainline and maintain it there). The thing is if you post something _once_ it gets ignored, and if you follow-up long enough for it to go in (which often takes years), it will then get ripped out again a few years later because "we never hear from anybody who uses this". Engaging with the community is signing up for an ongoing commitment. >> For example, cortex-m is a large nommu platform on which vendors support Linux >> BSPs, but notice how page 8 of >> https://www.microsemi.com/document-portal/doc_view/132181-linux-cortex-m-users-manual >> points at a cross compiler toolchain from _2010_ and page 4 says they're booting >> a 2.6.33 kernel? > > Any company/person who follows the route of not working with the linux kernel > community to get their work included is going to inevitably get stuck on older > versions of everything. I fight hard to get current versions of everything to work on all my supported targets. This requires regular regression testing, and I maintain a pile of patches that I post here periodically but I fully admit will probably never go in: https://lkml.iu.edu/hypermail/linux/kernel/2302.2/05594.html (Sigh, now that 6.7 is out I should post the new round...) People who want to use my kernels as a source are welcome to do so (and I've seen my patches quietly show up in other projects), but getting upstream to actually _fix_ anything? Every one of those patches had a link to the previous time it was posted to the list and ignored. I mean literally, the first of those patches teaches the makefile to autodetect whether $PREFIX-cc is gcc or llvm and just do the right thing, and I was told that they actively didn't want it to: https://lkml.iu.edu/hypermail/linux/kernel/2302.2/07184.html That is modern linux-kernel development. >> I'm a bit weird in that I try to get CURRENT stuff to work on nommu, and a lot >> of people have been happy to consume my work, but getting any of them to post >> directly to linux-kernel is like pulling teeth. > > I regularly test nommu configurations (as in every kernel rc and release) on m68k > and at least every release on other architectures like arm(*) and recently on > riscv as well. Sigh, I should start caring about riscv. I added or1k support, I should do riscv. (Except I did or1k because I found it in actual hardware, the Orange Pi 3b's power controller is an or1k asic so I needed an or1k toolchain to build some of u-boot's firmware or else the board couldn't reboot, and there was a qemu-system-or1k already, which turned into adding it to mkroot via a long https://lore.kernel.org/openrisc/ZX1xbs_AGdgLgcx7@antec/ thread with its developers. Alas I still can't get qemu to exit (I.E. virtually reboot or power off), apparently I need to reinstall my laptop to have a new enough version of python 3 to build a newer qemu with. It's on the todo list...) I still have a hard time considering riscv anything other than open source's version of Itanium. Promises of ubiquity, but even a 28 nanometer mask is still 6 figures before you run any wafers and your mask build process is sucking in all the black box libraries the fab can sell you, so what does "open" really get you here? Cortex-m got cheap when the superh patents expired so Arm didn't have to pay royalties to hitachi (renesas?) for the thumb instruction set anymore, and they belt those suckers en masse amortizing the up-front costs over ENORMOUS volume. And yes, j-core was trying to fix the closed source library and toolchain issues back when I was still working with them. Among other things fishing Google/skywater's openlane toolchain build out of their magic docker and reproducing it under a vanilla debootstrap, ala https://github.com/j-core/openlane-vhdl-build (As with most corporate clusterfscks, once you dig far enough it turns out you can throw over 90% of it out...) But these days I'm trying to get toybox to 1.0... > (*) somewhat annoyingly needing a minor patch to run the versatile qemu platform > I like to test with. But hey, that is on me :-) I would very much like to add more nommu targets to mkroot, can I get your build/config info offline? (I tried fishing configs out of buildroot a couple years ago, but after the THIRD one where the secret was "use very old versions of packages, the current stuff is broken"... And the problems were things like "the conversion to device tree deleted a huge chunk of this infrastructure", not simple fixes.) > Regards > Greg Rob -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [LTP] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP] 2024-01-10 5:47 ` Rob Landley @ 2024-01-10 14:46 ` Greg Ungerer 0 siblings, 0 replies; 40+ messages in thread From: Greg Ungerer @ 2024-01-10 14:46 UTC (permalink / raw) To: Rob Landley, Petr Vorel Cc: Niklas Cassel, Jonathan Corbet, Linux-sh list, Christophe Lyon, Randy Dunlap, linux-kernel, linux-m68k, Geert Uytterhoeven, Linux ARM, John Paul Adrian Glaubitz, linux-riscv, buildroot, ltp, automated-testing On 10/1/24 15:47, Rob Landley wrote: > On 1/9/24 17:17, Greg Ungerer wrote: >> On 10/1/24 06:24, Rob Landley wrote: >>> I'm a bit weird in that I try to get CURRENT stuff to work on nommu, and a lot >>> of people have been happy to consume my work, but getting any of them to post >>> directly to linux-kernel is like pulling teeth. >> >> I regularly test nommu configurations (as in every kernel rc and release) on m68k >> and at least every release on other architectures like arm(*) and recently on >> riscv as well. > > Sigh, I should start caring about riscv. I added or1k support, I should do > riscv. (Except I did or1k because I found it in actual hardware, the Orange Pi > 3b's power controller is an or1k asic so I needed an or1k toolchain to build > some of u-boot's firmware or else the board couldn't reboot, and there was a > qemu-system-or1k already, which turned into adding it to mkroot via a long > https://lore.kernel.org/openrisc/ZX1xbs_AGdgLgcx7@antec/ thread with its > developers. Alas I still can't get qemu to exit (I.E. virtually reboot or power > off), apparently I need to reinstall my laptop to have a new enough version of > python 3 to build a newer qemu with. It's on the todo list...) > > I still have a hard time considering riscv anything other than open source's > version of Itanium. Promises of ubiquity, but even a 28 nanometer mask is still > 6 figures before you run any wafers and your mask build process is sucking in > all the black box libraries the fab can sell you, so what does "open" really get > you here? Cortex-m got cheap when the superh patents expired so Arm didn't have > to pay royalties to hitachi (renesas?) for the thumb instruction set anymore, > and they belt those suckers en masse amortizing the up-front costs over ENORMOUS > volume. > > And yes, j-core was trying to fix the closed source library and toolchain issues > back when I was still working with them. Among other things fishing > Google/skywater's openlane toolchain build out of their magic docker and > reproducing it under a vanilla debootstrap, ala > https://github.com/j-core/openlane-vhdl-build (As with most corporate > clusterfscks, once you dig far enough it turns out you can throw over 90% of it > out...) > > But these days I'm trying to get toybox to 1.0... > >> (*) somewhat annoyingly needing a minor patch to run the versatile qemu platform >> I like to test with. But hey, that is on me :-) > > I would very much like to add more nommu targets to mkroot, can I get your > build/config info offline? (I tried fishing configs out of buildroot a couple > years ago, but after the THIRD one where the secret was "use very old versions > of packages, the current stuff is broken"... And the problems were things like > "the conversion to device tree deleted a huge chunk of this infrastructure", not > simple fixes.) Maybe getting a little off-topic here, but I'll just send links here. Who knows it might be useful to others. Recently I have been experimenting with minimal builds, this is a bunch of scripts, configs and a couple of patches I currently have: https://github.com/gregungerer/simple-linux Mostly the kernel builds use the architecture defconfigs, but for armnommu versatile it was easier to use a dedicated config and patch: https://github.com/gregungerer/simple-linux/blob/master/configs/linux-6.6-armnommu-versatile.config https://github.com/gregungerer/simple-linux/blob/master/patches/linux-6.6-armnommu-versatile.patch Anyway the scripting uses the newest package versions of everything (binutils, gcc, linux, uClibc, busybox). Regards Greg -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [LTP] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP] 2024-01-09 20:24 ` [LTP] " Rob Landley 2024-01-09 23:17 ` Greg Ungerer @ 2024-01-10 13:33 ` Petr Vorel 2024-01-10 18:23 ` Rob Landley 1 sibling, 1 reply; 40+ messages in thread From: Petr Vorel @ 2024-01-10 13:33 UTC (permalink / raw) To: Rob Landley Cc: Niklas Cassel, Jonathan Corbet, Linux-sh list, Christophe Lyon, Randy Dunlap, linux-kernel, linux-m68k, Geert Uytterhoeven, Linux ARM, John Paul Adrian Glaubitz, linux-riscv, buildroot, Greg Ungerer, ltp, automated-testing Hi Rob, all, > On 1/8/24 03:03, Petr Vorel wrote: > > Hi Rob, all, > > [ Added Niklas Cassel, who is maintainer of qemu_riscv64_nommu_virt_defconfig in > > buildroot ] > Hi Niklas! > >> Buildroot also apparently has an LTP package selectable in menuconfig: > >> https://github.com/buildroot/buildroot/tree/master/package/ltp-testsuite > >> But I haven't tried it... > > I'm the maintainer of the LTP package in buildroot in my private time. > > BTW I spent quite a lot of time fixing LTP (and some other system packages, > > e.g. nfs-utils) compilation on some old legacy architectures reported via > > http://autobuild.buildroot.net/ I've never used in the reality. > > But I certainly don't have time to drive nommu support in my private time. > > I don't even have an interest, I don't use any nommu device. > I do, but I've never done much with LTP, and I have my hands full with toybox > and mkroot already. Understand. > > Therefore nobody who is not involved in nommu will not find a time to support it > > in LTP (support does not mean just to add the functionality to the new C API, > > but run tests on nommu and fix failing bugs). I suppose nobody is paid to work > > on nommu platforms, it would have to be a hobby project, right? > A bunch of people are paid to work on nommu platforms, and I've worked with them > a bunch, but none of them talk to linux-kernel. They find the culture toxic, > insular, and categorically dismissive of their interests. > For example, cortex-m is a large nommu platform on which vendors support Linux > BSPs, but notice how page 8 of > https://www.microsemi.com/document-portal/doc_view/132181-linux-cortex-m-users-manual > points at a cross compiler toolchain from _2010_ and page 4 says they're booting > a 2.6.33 kernel? > I'm a bit weird in that I try to get CURRENT stuff to work on nommu, and a lot > of people have been happy to consume my work, but getting any of them to post > directly to linux-kernel is like pulling teeth. Interesting, thanks for sharing this. BTW I'm not saying anybody is using nommu, but I wonder if anybody really test it with LTP. And if yes, I wonder why we don't have reports about tests broken in new API. > > But as I said, if anybody from nommu decides to maintain it in LTP, I'll try to > > support him in my free time (review patches, give advices). And if nobody > > stands, this patchset which removes the support in the old API will be merged > > after next LTP release (in the end of January). > What does the API migration do? Is there a page on it ala OABI vs EABI in arm or > something? New C API is documented at our wiki: the API for using in the tests [1] and the library itself [2]. (We also have shell API, but we can ignore it for nommu.) All files in lib/ directory which include tst_test.h are part of new C API. Main file is lib/tst_test.c. LTP tests, which has been rewritten to new API include tst_test.h, they are in testcases/ directory. Library has it's own tests (for testing regression in in lib/newlib_tests/*.c. The reason why Cyril wrote in 2016 new C API was that the old API was buggy (tests randomly fails). Tests which are still using the old API (there is ongoing rewrite) include test.h. The old API is not much documented. Feel free to ask any more question. Kind regards, Petr [1] https://github.com/linux-test-project/ltp/wiki/C-Test-API [2] https://github.com/linux-test-project/ltp/tree/master/lib > Rob -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [LTP] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP] 2024-01-10 13:33 ` Petr Vorel @ 2024-01-10 18:23 ` Rob Landley 2024-01-10 22:33 ` Petr Vorel 0 siblings, 1 reply; 40+ messages in thread From: Rob Landley @ 2024-01-10 18:23 UTC (permalink / raw) To: Petr Vorel Cc: Niklas Cassel, Jonathan Corbet, Linux-sh list, Christophe Lyon, Randy Dunlap, linux-kernel, linux-m68k, Geert Uytterhoeven, Linux ARM, John Paul Adrian Glaubitz, linux-riscv, buildroot, Greg Ungerer, ltp, automated-testing On 1/10/24 07:33, Petr Vorel wrote: >> I'm a bit weird in that I try to get CURRENT stuff to work on nommu, and a lot >> of people have been happy to consume my work, but getting any of them to post >> directly to linux-kernel is like pulling teeth. > > Interesting, thanks for sharing this. BTW I'm not saying anybody is using nommu, > but I wonder if anybody really test it with LTP. And if yes, I wonder why we > don't have reports about tests broken in new API. I don't expect a lot of nommu users are aware you ever _could_ run LTP on nommu. But I'd like to get nommu more regularly supported. You _should_ be able to build a musl-linux userspace with busybox or toybox and be able to build a recognizable system (even an alpine-alike) which could then get the basic plumbing regression tested on qemu even without access to nommu hardware. >> > But as I said, if anybody from nommu decides to maintain it in LTP, I'll try to >> > support him in my free time (review patches, give advices). And if nobody >> > stands, this patchset which removes the support in the old API will be merged >> > after next LTP release (in the end of January). > >> What does the API migration do? Is there a page on it ala OABI vs EABI in arm or >> something? > > New C API is documented at our wiki: the API for using in the tests [1] > and the library itself [2]. (We also have shell API, but we can ignore it for > nommu.) I'm writing a bash-compatible shell, which (thanks to Elliott forwarding questions) has involved surprisingly long threads with the bash maintainer about weird corner cases neither the man page nor my testing made clear: http://lists.landley.net/pipermail/toybox-landley.net/2023-July/029631.html (Alas I try NOT to involve him because when I bring stuff up he keeps FIXING BASH which from my point of view just makes it a moving target...) Anyway, running the shell API on nommu doesn't seem out of the question, but probably not any time soon. (The fact the shell isn't finished yet is one of the big REASONS I haven't got enough time to take on LTP. That and I haven't started writing "awk" and "make" yet". And I need to cycle back to https://landley.net/notes-2023.html#12-10-2023 . And after that debian, ala https://peertube.debian.social/w/chzkKrMvEczG7qQyjbMKPr and https://peertube.debian.social/w/45XroN9CnbYLNLKQH3GD9F . And follow up on https://lists.gnu.org/archive/html/coreutils/2023-08/msg00009.html . And...) > All files in lib/ directory which include tst_test.h are part of new C API. Main > file is lib/tst_test.c. safe_fork(), safe_clone(), fork_testrun()... > LTP tests, which has been rewritten to new API include > tst_test.h, they are in testcases/ directory. Library has it's own tests (for > testing regression in in lib/newlib_tests/*.c. Library meaning... libc? Or does LTP have a library? > The reason why Cyril wrote in 2016 new C API was that the old API was buggy > (tests randomly fails). Tests which are still using the old API (there is > ongoing rewrite) include test.h. The old API is not much documented. > > Feel free to ask any more question. My standard questions are "what does success look like" and "how do I reproduce the problem". For the first: if there previously was nommu support in LTP, what's the last version that's known to work? Is there an existing build/test setup that can be reproduced? For the second... If I try to run LTP on sh2eb (my current nommu test board) with the current LTP... do I get a build break? Additional test failures at runtime? You talk about "removing nommu support", but... what's the current status? (A subset of tests still use the old api...?) Yes I need to read https://github.com/linux-test-project/ltp/wiki/C-Test-API but I also need to know how to build LTP from source. I'm looking at the README's list of "autoconf, automake, m4, pkgconf / pkg-config" and wincing significantly. (What does gnu/autoconf DO here? Disable tests? I never understand why anybody uses that giant hairball of complexity. Half of cross compiling is figuring out how to lie to autoconf, and my normal workaround for that is to bootstrap a target system and build natively, but while I've gotten gcc to run natively on nommu systems, I never _tried_ gnu/autoconf. Bootstrapping some subset of LFS on a nommu system so it has the dependencies LFS needs to natively build seems like the long way 'round... (I am not the right guy for "make it work the easy way". I am the guy who will step on every land mine between here and there. I code by debugging an empty screen. If I don't start from "known working" setup... it would take a while.) Rob -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [LTP] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP] 2024-01-10 18:23 ` Rob Landley @ 2024-01-10 22:33 ` Petr Vorel 0 siblings, 0 replies; 40+ messages in thread From: Petr Vorel @ 2024-01-10 22:33 UTC (permalink / raw) To: Rob Landley Cc: Niklas Cassel, Jonathan Corbet, Linux-sh list, Christophe Lyon, Randy Dunlap, linux-kernel, linux-m68k, Geert Uytterhoeven, Linux ARM, John Paul Adrian Glaubitz, linux-riscv, buildroot, Greg Ungerer, ltp, automated-testing > On 1/10/24 07:33, Petr Vorel wrote: > >> I'm a bit weird in that I try to get CURRENT stuff to work on nommu, and a lot > >> of people have been happy to consume my work, but getting any of them to post > >> directly to linux-kernel is like pulling teeth. > > Interesting, thanks for sharing this. BTW I'm not saying anybody is using nommu, > > but I wonder if anybody really test it with LTP. And if yes, I wonder why we > > don't have reports about tests broken in new API. > I don't expect a lot of nommu users are aware you ever _could_ run LTP on nommu. > But I'd like to get nommu more regularly supported. You _should_ be able to > build a musl-linux userspace with busybox or toybox and be able to build a > recognizable system (even an alpine-alike) which could then get the basic > plumbing regression tested on qemu even without access to nommu hardware. > >> > But as I said, if anybody from nommu decides to maintain it in LTP, I'll try to > >> > support him in my free time (review patches, give advices). And if nobody > >> > stands, this patchset which removes the support in the old API will be merged > >> > after next LTP release (in the end of January). > >> What does the API migration do? Is there a page on it ala OABI vs EABI in arm or > >> something? > > New C API is documented at our wiki: the API for using in the tests [1] > > and the library itself [2]. (We also have shell API, but we can ignore it for > > nommu.) > I'm writing a bash-compatible shell, which (thanks to Elliott forwarding > questions) has involved surprisingly long threads with the bash maintainer about > weird corner cases neither the man page nor my testing made clear: > http://lists.landley.net/pipermail/toybox-landley.net/2023-July/029631.html > (Alas I try NOT to involve him because when I bring stuff up he keeps FIXING > BASH which from my point of view just makes it a moving target...) > Anyway, running the shell API on nommu doesn't seem out of the question, but > probably not any time soon. (The fact the shell isn't finished yet is one of the > big REASONS I haven't got enough time to take on LTP. That and I haven't started > writing "awk" and "make" yet". And I need to cycle back to > https://landley.net/notes-2023.html#12-10-2023 . And after that debian, ala > https://peertube.debian.social/w/chzkKrMvEczG7qQyjbMKPr and > https://peertube.debian.social/w/45XroN9CnbYLNLKQH3GD9F . And follow up on > https://lists.gnu.org/archive/html/coreutils/2023-08/msg00009.html . And...) > > All files in lib/ directory which include tst_test.h are part of new C API. Main > > file is lib/tst_test.c. > safe_fork(), safe_clone(), fork_testrun()... > > LTP tests, which has been rewritten to new API include > > tst_test.h, they are in testcases/ directory. Library has it's own tests (for > > testing regression in in lib/newlib_tests/*.c. > Library meaning... libc? Or does LTP have a library? Yes, LTP has a library (lib/libltp.a). That's what I meant here and in all my text. So far I did not mention anything libc specific. > > The reason why Cyril wrote in 2016 new C API was that the old API was buggy > > (tests randomly fails). Tests which are still using the old API (there is > > ongoing rewrite) include test.h. The old API is not much documented. > > Feel free to ask any more question. > My standard questions are "what does success look like" and "how do I reproduce > the problem". > For the first: if there previously was nommu support in LTP, what's the last > version that's known to work? Is there an existing build/test setup that can be > reproduced? I have no idea whether it worked. Best would be to ask Mike Frysinger (the author of m4/ltp-nommu-linux.m4). The code was added 14 years ago, even before all of the current maintainers were involved. > For the second... If I try to run LTP on sh2eb (my current nommu test board) > with the current LTP... do I get a build break? Additional test failures at > runtime? You talk about "removing nommu support", but... what's the current > status? (A subset of tests still use the old api...?) Yes, subset of the tests which use the old API (git grep UCLINUX). > Yes I need to read https://github.com/linux-test-project/ltp/wiki/C-Test-API but > I also need to know how to build LTP from source. I'm looking at the README's > list of "autoconf, automake, m4, pkgconf / pkg-config" and wincing > significantly. (What does gnu/autoconf DO here? Disable tests? I never > understand why anybody uses that giant hairball of complexity. Half of cross > compiling is figuring out how to lie to autoconf, and my normal workaround for > that is to bootstrap a target system and build natively, but while I've gotten > gcc to run natively on nommu systems, I never _tried_ gnu/autoconf. > Bootstrapping some subset of LFS on a nommu system so it has the dependencies > LFS needs to natively build seems like the long way 'round... Well, one day we might migrate to use something else (meson?), but until then autoconf + m4 + pkgconf is used (instead of automake there is LTP custom system). This was written in 2009 and nobody plans to change it (well, Andrea played with meson [1] [2]). But we got far away from the original topic :). Kind regards, Petr [1] https://github.com/acerv/ltp-core [2] https://github.com/acerv/ltp-testcases > (I am not the right guy for "make it work the easy way". I am the guy who will > step on every land mine between here and there. I code by debugging an empty > screen. If I don't start from "known working" setup... it would take a while.) > Rob -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [LTP] [PATCH 00/36] Remove UCLINUX from LTP 2024-01-05 3:52 ` Rob Landley 2024-01-05 13:11 ` [LTP] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP] Petr Vorel @ 2024-01-08 8:33 ` Andrea Cervesato via ltp 2024-01-08 8:34 ` Andrea Cervesato via ltp 2 siblings, 0 replies; 40+ messages in thread From: Andrea Cervesato via ltp @ 2024-01-08 8:33 UTC (permalink / raw) To: Rob Landley, Cyril Hrubis, Geert Uytterhoeven Cc: Jonathan Corbet, Linux-sh list, Christophe Lyon, Randy Dunlap, linux-kernel, linux-m68k, Linux ARM, John Paul Adrian Glaubitz, linux-riscv, Greg Ungerer, ltp Hi! My 2 cents. I'm working on refactoring growfiles test which uses UCLINUX flag. During its development I had occasion to check UCLINUX support and (indeed) it seems pretty broken for LTP, because nobody is maintaining it for a while and such tests use old API that will be replaced in any case sooner or later. I agree with other people about removing it, unless there's a valid reason to keep it. Just in case we want to keep it, someone should take care about UCLINUX support, testing LTP releases for it as well, but it doesn't seem like something we can do inside the LTP devs team due to the lack of resources. Regards, Andrea On 1/5/24 04:52, Rob Landley wrote: > On 1/3/24 06:09, Cyril Hrubis wrote: >> Hi! >>> I am not sure I agree with this series. >>> Removing support for UCLINUX from LTP is almost a guarantee for >>> not noticing when more breakage is introduced. >>> >>> How exactly is UCLINUX broken in LTP? >> As far as we know noone is using it and nobody is maintaing it for a >> decade, > Nobody is maintaining "uclinux" because that was a distro, but you can build > nommu support in buildroot and such, and people do. > > Rob -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [LTP] [PATCH 00/36] Remove UCLINUX from LTP 2024-01-05 3:52 ` Rob Landley 2024-01-05 13:11 ` [LTP] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP] Petr Vorel 2024-01-08 8:33 ` [LTP] [PATCH 00/36] Remove UCLINUX from LTP Andrea Cervesato via ltp @ 2024-01-08 8:34 ` Andrea Cervesato via ltp 2 siblings, 0 replies; 40+ messages in thread From: Andrea Cervesato via ltp @ 2024-01-08 8:34 UTC (permalink / raw) To: Rob Landley, Cyril Hrubis, Geert Uytterhoeven Cc: Jonathan Corbet, Linux-sh list, Christophe Lyon, Randy Dunlap, linux-kernel, linux-m68k, Linux ARM, John Paul Adrian Glaubitz, linux-riscv, Greg Ungerer, ltp Hi! My 2 cents. I'm working on refactoring growfiles test which uses UCLINUX flag. During its development I had occasion to check UCLINUX support and (indeed) it seems pretty broken for LTP, because nobody is maintaining it for a while and such tests use old API that will be replaced in any case sooner or later. I agree with other people about removing it, unless there's a valid reason to keep it. Just in case we want to keep it, someone should take care about UCLINUX support, testing LTP releases for it as well, but it doesn't seem like something we can do inside the LTP devs team due to the lack of resources. Regards, Andrea On 1/5/24 04:52, Rob Landley wrote: > On 1/3/24 06:09, Cyril Hrubis wrote: >> Hi! >>> I am not sure I agree with this series. >>> Removing support for UCLINUX from LTP is almost a guarantee for >>> not noticing when more breakage is introduced. >>> >>> How exactly is UCLINUX broken in LTP? >> As far as we know noone is using it and nobody is maintaing it for a >> decade, > Nobody is maintaining "uclinux" because that was a distro, but you can build > nommu support in buildroot and such, and people do. > > Rob -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [LTP] [PATCH 00/36] Remove UCLINUX from LTP 2024-01-03 9:46 ` Geert Uytterhoeven 2024-01-03 11:49 ` Petr Vorel @ 2024-01-05 3:50 ` Rob Landley 1 sibling, 0 replies; 40+ messages in thread From: Rob Landley @ 2024-01-05 3:50 UTC (permalink / raw) To: Geert Uytterhoeven, Petr Vorel Cc: Jonathan Corbet, Linux-sh list, Christophe Lyon, Randy Dunlap, linux-kernel, linux-m68k, Linux ARM, John Paul Adrian Glaubitz, linux-riscv, Greg Ungerer, ltp On 1/3/24 03:46, Geert Uytterhoeven wrote: > Hi Petr, > > CC other uClinux arch lists > > On Wed, Jan 3, 2024 at 2:52 AM Petr Vorel <pvorel@suse.cz> wrote: >> UCLINUX is broken in LTP and nobody really cares. Actually I dare to >> say UCLINUX is dead. Therefore I prepared patchset to remove UCLINUX >> from LTP. We have been actively removing UCLINUX from LTP during rewrite >> tests to new LTP API. This removes the rest from the old tests (which >> will be sooner or later rewritten to new API). >> >> Because the patchset is quite big, I did not want to send it to mailing >> lists (but I can do it if you want). >> >> Can you please have look at my fork on gitlab, branch: remove-UCLINUX >> https://gitlab.com/pevik/ltp/-/commits/remove-UCLINUX?ref_type=heads >> >> Build test: >> https://github.com/pevik/ltp/actions/runs/7392470215 > > Thanks for your series! > > I see you only CCed linux-m68k, but AFAIK, uClinux is not restricted > to m68k/coldfire, but also available on arm32, riscv, sh, and xtensa. Do you mean "nommu support", or do you mean the ancient distro Jeff Dionne stopped maintaining in 2003? Because I've been doing nommu musl-libc systems for a few years now. Works for me... Rob -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [LTP] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP] @ 2024-01-31 0:05 Giovanni Lostumbo 0 siblings, 0 replies; 40+ messages in thread From: Giovanni Lostumbo @ 2024-01-31 0:05 UTC (permalink / raw) To: ltp On 1/10/24 20:25, Greg Ungerer wrote: Sorry Rob, but I think you are wrong about a number of things here. Happy to be corrected. I learned most of this stuff by people pointing things out I didn't know. But I _have_ been trying to collect this info for about 15 years now... I know, I was there at the coal face so to speak during the early years of uClinux and right up today. I feel I have to correct some of the things you claim. I only heard about the politics long after the fact, and the stories have a lot of elisions because what happens in Utah sadly does NOT stay in Utah. | I came to Lineo through the Moreton Bay acquisition, and I worked with | Jeff, Tim and Erik. Actually I had been collaborating with Jeff when still at | RT-control before that as well. Lineo was a wild ride, to say the least. I am really impressed by Lineo's RT-Control and Embedix. I have been searching for "archeological" software for porting to new thermal power envelopes. https://en.wikipedia.org/wiki/Lineo "In April 1999, Caldera Thin Clients released the no longer needed sources to GEM <https://en.wikipedia.org/wiki/GEM_(desktop_environment)> and ViewMAX <https://en.wikipedia.org/wiki/ViewMAX> under the GNU General Public License <https://en.wikipedia.org/wiki/GNU_General_Public_License> (GPL).[9] <https://en.wikipedia.org/wiki/Lineo#cite_note-Jemmett_1999-9>" - Rt-Control provided μClinux <https://en.wikipedia.org/wiki/%CE%9CClinux> - a version of Linux for microcontrollers <https://en.wikipedia.org/wiki/Microcontroller>, such as the Motorola 68k <https://en.wikipedia.org/wiki/Motorola_68000_series>/ColdFire <https://en.wikipedia.org/wiki/NXP_ColdFire> line, i960 <https://en.wikipedia.org/wiki/Intel_i960>, ARM7 <https://en.wikipedia.org/wiki/ARM_architecture_family>, and ETRAX CRIS <https://en.wikipedia.org/wiki/ETRAX_CRIS> chips. With these chips <https://en.wikipedia.org/wiki/Microprocessor> lacking MMU <https://en.wikipedia.org/wiki/Memory_management_unit> and thus unable to provide multi-tasking <https://en.wikipedia.org/wiki/Computer_multitasking> capabilities, uClinux was able to run full-featured in as little as 150 KB <https://en.wikipedia.org/wiki/Kilobyte> of RAM <https://en.wikipedia.org/wiki/Random-access_memory> with a 1 MB <https://en.wikipedia.org/wiki/Megabyte> ROM <https://en.wikipedia.org/wiki/Read-only_memory> chip. - Embedix - Lineo's flagship <https://en.wikipedia.org/wiki/Flagship#Flagship_in_language> product that ran a complete multitasking, networked Linux operating system in 2 MB of ROM/flash <https://en.wikipedia.org/wiki/Flash_memory> and 4 MB of RAM. A programmer recently sent me this about Caldera: https://rant.gulbrandsen.priv.no/linux/openlinux-lizard Another programmer in 2021 began describing some of the earliest GUIs that provided "Basic interactive computation" : https://github.com/kragen/dernocua/blob/master/text/energy-autonomous-computing.md#computation-needs (defines efficiency according to glyphs/DMIPS) From the SDS 940, to Wordstar, to GEM, GEOS and Apple's System 1. Most STM32 microcontrollers today such as the Cortex M4 exceed the Dhrystone marks that these early PCs used. Some also have more memory than the thin clients of Embedix. Sun Microsystems also developed stateless clients around that time: https://en.wikipedia.org/wiki/Sun_Ray These systems will be very useful for running remote desktops/cloud software in portable devices. I think that the concept of obsolescence is becoming less universal, while "vintage" or "retro" projects are becoming less about nostalgia and more about practical restoration projects: https://github.com/readme/featured/vintage-computing "Thanks to Open Source, Nothing is Ever Obsolete," The headline reads. Projects like Psion ROM restorations SIBO/EPOC16 SDK(https://zedstarr.com/2023/02/10/back-to-the-source-taking-an-emulated-mc200-back-to-psions-old-hq-in-harcourt-st/) are very practical reuse of very dense-code, which today microcontrollers are able to emulate or even get a native port for: Examples: https://www.freethegameboy.info/ and Commodore on an STM32: https://www.youtube.com/watch?v=uI4o9XAFLt0 https://hackaday.com/2021/07/28/emulating-the-ibm-pc-on-an-esp32/ An emerging design idea is that by lowering power consumption, one can allow energy autarkic devices: https://semiengineering.com/a-power-first-approach/ "Even today, when I interview people and ask if power is a primary optimization consideration, the answer is typically, “We have to meet performance metrics first and then we can worry about reducing power.” Over the next 10 years, I expect that mentality will have changed to a “power first” approach. The real question is, within a defined power budget, how much performance on a given task can I attain? You don’t get to that answer by looking at performance first." With microcontrollers already able to be powered by solar energy like a pocket calculator, the "instant-on" design is the next focus of mobile computing. I have had this interest since 2011: https://github.com/EI2030/Low-power-E-Paper-OS Furthermore, there are several chip designers: Ambiq Micro (https://ambiq.com/apollo4-plus/), and RISC-V IP designs (Andes http://www.andestech.com/en/products-solutions/andescore-processors/v3-e8/) that develop lowest power cores. As in 4uA/Mhz, 2uW/Mhz TDPs. The question is, how much linux (or any other OS) can fit in 4-8MB of RAM? A credit card-sized solar panel produces up to 1-2mA indoors, and several mA outdoors, allowing for nearly instant on power. As portable electronics sold today rarely allow for removable and standardized batteries, their obsolescence arises out of the hard-to-replace aspect of their anti-repair design. Also, the most power efficient RAM consumes several milliamps for just 4MB. It may not be until 2030 that Angstrom sized RAM in amounts rivaling the earliest Android phones (128MB in the 1st iPhone, 192MB of RAM in the HTC Dream) will be able to be powered by 1% of the current consumption. Just last week, TSMC and Industrial Technology Research Institute claimed SOT MRAM to consume 1% of the power of STT MRAM. 128MB today consumes tens or hundreds of milliwatts. And the developers of such memory chips typically develop for data centers that see the benefits of these technologies first, rather than mobile phones and laptops. When RAM was still expensive in the 1990s, many coders, including Symbian S60 developers into the early 00s relied on extremely efficient code to save as much power consumption as possible. These benefits have not fully been realized since uClinux (as an umbrella term, and a toolkit, rather than necessarily as one instance of a distro), has been splintered into less mainline projects and applications. That said, there are processors such as the Hitachi-originated J2 and Andes E8 that appear to have optimized power consumption with modern, code-dense architecture. A 16MB memory address could allow for a variety of applications such as lwIP, wolfSSL, and processing audio such as Codec2. While I am not a programmer and have no capability of being a maintainer for such a project, I continue to see the value of projects such as nommu for these reasons. Maybe not in LTP, at least at this time. |"In early 2015 I went to |https://web.archive.org/web/20150219205915/http://www.uclinux.org/ which had a |cvs archive link on the left (which was dead, hence the email exchange), a page |on bflt in the same nav bar (first link on that page went to the dead CVS |archive, the beyondlogic page was 404, and vapier's stuff was still there but |several years old, I tracked down the newer forks later), and then the "http |download" link at the end of the list led to |https://web.archive.org/web/20150206052300/http://www.uclinux.org/pub/uClinux/ |the newest date in which was 2004. | |The resulting first impression was "this project is VERY DEAD". And then my |email exchange with the contact info link got back "cvs archive died with no |backup" and no reply to a follow-up email. | But that is all kind of archeological now, relegated to history. Agreed. | Well, some things belong in a museum. Other things, like Shakespeare, get renewed every year... Another way I see this codebase is as a physics experiment- where the goal is developing a self-sustaining solar power plant on a single board computer that runs linux. Ernest Rutherford in 1933 said of splitting lithium: "The energy produced by the breaking down of the atom is a very poor kind of thing. Any one who expects a source of power from the transformation of these atoms is talking moonshine." (p.9/18- 136: https://lukemuehlhauser.com/wp-content/uploads/Jenkin-Atomic-energy-is-moonshine-what-did-Rutherford-really-mean.pdf) But of course, once he realized the interest, atomic energy was given more scrutiny, including by Rutherford (p.14/18-141). Since the same has been said of solar powered phones, I expect new discoveries to prove us wrong. That said, near-threshold voltage has a long history, and hasn't been an easy technology to harness for most of the past 10 years: https://semiengineering.com/near-threshold-issues-widen/ It's only been in the past few years where advances have been made: https://semiengineering.com/near-threshold-computing-gets-a-boost/ (and trickle down to older nodes too) There might be more 16-32MB linux systems that benefit from nommu, perhaps with less overhead in 3-5 years. But like many long-term projects, it is not easy to see the utility in the short term if there is not a hardware platform that can clearly provide a use-case. Lastly, a driving force is hyperscaling: https://semiengineering.com/challenges-with-adaptive-control/ "But if you look at something like an embedded system, especially with edge-based intelligence, there is a very different cost in terms of battery life. This trickle-down of technology, tools, and methodology means they are able to leverage those exact same things that the hyperscale customers are using, but they’re doing it in a much smaller footprint and much more fine-grain method and achieving their own benefits.” -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 40+ messages in thread
end of thread, other threads:[~2024-01-31 9:59 UTC | newest] Thread overview: 40+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2024-01-03 1:52 [LTP] [PATCH 00/36] Remove UCLINUX from LTP Petr Vorel 2024-01-03 7:58 ` Cyril Hrubis 2024-01-03 8:04 ` Cyril Hrubis 2024-01-03 8:39 ` Petr Vorel 2024-01-03 9:46 ` Geert Uytterhoeven 2024-01-03 11:49 ` Petr Vorel 2024-01-03 11:54 ` Geert Uytterhoeven 2024-01-03 12:09 ` Cyril Hrubis 2024-01-03 12:40 ` Petr Vorel 2024-01-05 3:52 ` Rob Landley 2024-01-05 13:11 ` [LTP] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP] Petr Vorel 2024-01-06 3:58 ` Rob Landley 2024-01-08 9:03 ` Petr Vorel 2024-01-08 10:07 ` Cyril Hrubis 2024-01-09 22:37 ` [LTP] [Automated-testing] " Bird, Tim 2024-01-10 5:01 ` Rob Landley 2024-01-10 14:14 ` Petr Vorel 2024-01-10 19:23 ` Rob Landley 2024-01-10 21:17 ` Niklas Cassel via ltp 2024-01-11 0:00 ` Greg Ungerer 2024-01-11 9:21 ` Niklas Cassel via ltp 2024-01-12 20:18 ` Rob Landley 2024-01-11 2:25 ` Greg Ungerer 2024-01-12 20:16 ` Rob Landley 2024-01-14 13:01 ` Greg Ungerer 2024-01-15 13:41 ` [LTP] [Buildroot] " Waldemar Brodkorb 2024-01-15 14:22 ` Cyril Hrubis 2024-01-11 13:11 ` [LTP] " Geert Uytterhoeven 2024-01-11 13:19 ` Greg Ungerer 2024-01-09 20:24 ` [LTP] " Rob Landley 2024-01-09 23:17 ` Greg Ungerer 2024-01-10 5:47 ` Rob Landley 2024-01-10 14:46 ` Greg Ungerer 2024-01-10 13:33 ` Petr Vorel 2024-01-10 18:23 ` Rob Landley 2024-01-10 22:33 ` Petr Vorel 2024-01-08 8:33 ` [LTP] [PATCH 00/36] Remove UCLINUX from LTP Andrea Cervesato via ltp 2024-01-08 8:34 ` Andrea Cervesato via ltp 2024-01-05 3:50 ` Rob Landley 2024-01-31 0:05 [LTP] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP] Giovanni Lostumbo
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).