Re: Adding or1k support to mkroot.

[Rob Landley <rob@landley.net>]

On 12/16/23 03:44, Stafford Horne wrote:
> On Fri, Dec 15, 2023 at 07:33:42AM -0600, Rob Landley wrote:
>> I've more or less gotten it to work. The trick was statically linking in the
>> initramfs image:
>> 
>>   https://github.com/landley/toybox/commit/5647741f6687
>> 
>> I haven't added any of the OPENRISC_HAVE_INST_* symbols because it booted
>> without them, and presumably the toolchain will automatically do the right thing
>> for userspace:
>> 
>>   https://landley.net/notes-2023.html#10-12-2023
>> 
>> (I just specify an i486, i586, or i686 toolchain, or tell x86-64
>> "CFLAGS=-mtune=nocona", and so on. I don't need to micromanage stuff in
>> menuconfig to build for minor variations of a given target, I have no idea why
>> openrisc would work differently.)
>> 
>> Nor did I add JUMP_UPON_UNHANDLED_EXCEPTION because I honestly don't know what
>> that modifies or why you would ever NOT do it if it was at all useful? I err on
>> the side of fewer symbols when the benefit's not clear...
> 
> I just use the defaults.

Because of the way I'm doing it, the "default" for any symbol I don't select is
switched off. My config lists all the symbols I need that aren't in allnoconfig.
(Although "need" means a dependency won't switch it on when the other symbol
gets switched on.)

Conceptually it's the same as running "make allnoconfig" and then going through
the list of symbols and switching them on by hand in menuconfig. "This is what's
enabled in this kernel."

> Those instruction specific switches will only be
> helpful if you really want tune things.  Which may be fun for some people.
> 
> The OpenRISC kernel is not yet sophisticated enough to detect and replace,
> emulate the missing instructions.  So we have these switches.  Some soft CPUs
> may have these instructions enabled/or disabled and these switches help if you
> know what you are doing.  Luckily QEMU has all the instruction enabled.

Isn't that a toolchain issue, though? Your userspace gets built with the same
toolchain, "we have a multiply instruction" is kind of something it needs to
know. Which means:

$ :| or1k-linux-musl-cc -E -dM -
#define __DBL_MIN_EXP__ (-1021)
#define __UINT_LEAST16_MAX__ 0xffff
#define __ATOMIC_ACQUIRE 2
...
#define __UINT_FAST8_TYPE__ unsigned char
#define __ATOMIC_ACQ_REL 4
#define __ATOMIC_RELEASE 3

Can't you have #defines in your toolchain saying what architecture variant it
supports? And thus the code can #ifdef those without kconfig having to care?

> In terms of JUMP_UPON_UNHANDLED_EXCEPTION, this was added before my time.  We
> probably could get rid of it and do a more graceful power based halt.

I'm still not 100% clear on what it does, even after reading the help text. :)

>> >> I built an or1k musl+gcc toolchain with my wrapper script around Rich Felker's
>> >> musl-cross-make project by adding "or1k::" to the TARGETS list in
>> >> https://github.com/landley/toybox/blob/master/scripts/mcm-buildall.sh#L34 and
>> >> that seems to have worked-ish. (Well, the kernel headers didn't install because
>> >> I have to add or1k->openrisc to musl-cross-make's TARGET_ARCH_MANGLED in
>> >> litecross/Makefile, but eh, close enough for the moment.)
>> 
>> I fixed that too, by the way:
>> 
>>   https://github.com/landley/toybox/commit/ab046139f9d8
> 
> OK.

Some glorious day Rich might apply the backlog of patches I've sent him to
musl-cross-make (which last updated April 2022). Or I might scrape up the spoons
to write a new linux-from-scratch style toolchain builder script so my
mcm-buildall.sh isn't a wrapper around an external project. It's a toss-up which
happens first...

(Mostly when I have spare toolchain brain I wrestle with llvm stuff instead...)

>> > This are my qemu arguments:
>> > 
>> > qemu-system-or1k -cpu or1200 -machine virt \
>> >   -no-reboot -kernel /home/shorne/work/linux/vmlinux \
>> >   -device virtio-net-pci,netdev=user \
>> >   -netdev user,id=user,net=10.9.0.1/24,host=10.9.0.100,hostfwd=tcp::2222-:22 \
>> >   -serial mon:stdio -nographic
>> 
>> If you're putting the monitor on stdio, where does the console go? (Is this
>> going to pop up a graphics window or something? Or do you talk to it entirely
>> through the virtual network?)
> 
> Console goes to stdio, I use ctrl-a C to drop into monitor which is also in
> stdio.

Huh, I didn't know that was an option.

>> > -device virtio-blk-device,drive=d0 \
>> >   -drive file=/home/shorne/work/openrisc/or1k-utils/buildroot/output/qemu-fs-or1k.qcow2,id=d0,if=none,format=qcow2 \
>> >   -gdb tcp::10001 \
>> >   -accel tcg,thread=multi \
>> >   -smp cpus=4 -m 768 \
>> 
>> Do I need to micromanage -accel?
> 
> Probably not for your use cases.  I can not recall the defaults but I think it
> was not multi threaded.  But I cannot be sure unless I go look.  You probably
> can run with a single core and no SMP.

What I did years ago was set up distcc so qemu called out to the cross compiler
running on loopback through the 10.0.2.2 emulated interface, thus moving the
heavy lifting of compilation outside the emulator to where CPU was cheap without
reintroducing all the problems of cross compiling (two compilers to keep
straight, two sets of headers and two sets of libraries leaking into each other,
can't run the binaries you generate, configure keeps asking questions about the
host and applying the results to the target...)

It still maxed out around -j 3 because of the preprocessing, linking, network
transfer, and make figuring out what to do next, but was way better than purely
native single CPU builds.

No clue what the sweet spot is at today...

>> >   -append rootwait \
>> >   -append boot=/dev/vda2
>> 
>> Did they ever fix qemu's problem that "append" didn't and you could only have
>> one -append and it in fact completely replaced the existing command line on most
>> architectures?
>> 
>> https://lists.gnu.org/archive/html/qemu-devel/2017-05/msg03214.html
> 
> After I sent this message I noticed this looked strange.  It doesn't work, so my
> command line ends up being: 'boot=/dev/vda2'.  Which was enough to get it to
> work.

It's a misleading name.

>> Hmmm, I've got:
>> 
>> qemu-system-or1k -M or1k-sim -m 256 "$@" -nographic -no-reboot -kernel
>> "$DIR"/linux-kernel -append "HOST=or1k console=FIXME $KARGS"
>> 
>> And it looks like the answer to "how do you enable a halt/reboot switch that can
>> exit qemu" is "don't use the or1k-sim board, use the virt board".
> 
> Yeah, virt board is the only one that supports halt/reboot.  The 'or1ksim'
> platform that we used as the model of qemu 'or1k-sim' supported reboot and halt
> via special 'nop' instructions.  The QEMU maintainers did not feel these special
> nop instructions should be supported in QEMU.

Sigh. Ok, I'll start the process over digesting the other config and trying to
get that one to work instead...

>> It provides boot messages without any initrd at all. It eventually panics
>> because there's no init, but early in board bringup I'm just looking for proof
>> of life.
>> 
>> > Do you have a valid initrd that maybe I can try out?  I recently use qcow2
>> > images to build, but I did used to use initrd images and it did work fine.
>> 
>> Sure, just remove the BUILTIN=1 from the if or1k stanza, and... it's 582k, I'll
>> send it _not_ cc-ing the mailing list. :)
> 
> OK, I will try this out. But sorry busy this weekend I will probably not get
> time.

No rush, I think you've answered my question above. It would be nice to get the
bug fixed, but unless I can -device,poweroff or similar on the qemu command line
to insert whatever it's poking to halt (and then tell the kernel about it), the
sim board isn't useful to me for automation.

>> >> Any idea what's going on here? That last one's kind of a blocker...
>> > 
>> > Have you tried to debug where it's getting hung up?  Gdb remote debug should
>> > be working for or1k.
>> 
>> I have not yet tried to rustle up a gdb build that understands or1k machine
>> code, no. I'm more likely to try to either get EARLY_PRINTK less broken or
>> dredge up the 2 line spin-to-write-register version of 8250 output and cut and
>> paste it in various places if I need to debug this myself.
> 
> That's OK, I usually use the console too to help debug too.

If I can reproduce it and stick a printf in it, I can generally debug it.

>> But I was hoping it was a known limitation or at least easily recognized...
> 
> OK, are things working for you now?  Or is the problem that the initrd still
> does not work?

I can use initrd built-in to the kernel. I cannot supply it externally via
qemu's -initrd. Which means I need to rebuild the kernel (7 minute clean
rebuild) to tweak the filesystem (30 second clean rebuild).

That said, if I can supply an external -initrd to the virt board, then it works
like the other targets. (It's a matter of workflow, I'm trying to regression
test stuff against as many architectures as I can, which means I want them to
work the same.)

One script I have doing this is my smoketest script at
https://github.com/landley/toybox/blob/master/mkroot/testroot.sh which runs all
the targets under qemu and makes sure they can get their clocks set (via ntp if
necessary) and access a block device and do basic networking.

Eventually I want to get that running the toybox test suite inside qemu on each
architecture, but that's still a ways off. (The shell isn't capable enough yet,
working on it...)

I also want to get an automated Linux From Scratch build to compile and run on
each target. I did that before with aboriginal linux years ago, but that was
making busybox into a development environment. (Other projects like Alpine Linux
and Adelie Linux took over from there when I stopped.) Now I'm trying to do it
again with toybox, with the extra difficulty that I can't use any existing GPL
code for anything because toybox is under a public domain equivalent license
(Zero Clause BSD).

(Yes, this would mean building Linux From Scratch on or1k, under qemu. That's
why I need I need a board with a working block device and at least 256 megs of
ram, although I can use a network block device and a swap file in a pinch on
systems with an MMU. I would someday LIKE to do a from-source debootstrap and
populate a musl-backed package repository via automated cluster build, but
there's still a certain amount of undocumented black magic in getting that to
work. The old "people who know have known so long they've forgotten how to
explain it" problem...)

> -Stafford

Rob