Running m68k on qemu with external initramfs?

Running m68k on qemu with external initramfs?

[Rob Landley]

I'm adding m68k support to toybox's "make root" target and I have a kernel and a
rootfs.cpio.gz, but qemu's -initrd option doesn't put the rootfs image somewhere
the m68k kernel can find it.

Is this a "modify qemu" problem or a "modify kernel" problem?

Rob

Re: Running m68k on qemu with external initramfs?

[Finn Thain]

On Fri, 10 Apr 2020, Rob Landley wrote:

> I'm adding m68k support to toybox's "make root" target and I have a 
> kernel and a rootfs.cpio.gz, but qemu's -initrd option doesn't put the 
> rootfs image somewhere the m68k kernel can find it.
> 
> Is this a "modify qemu" problem or a "modify kernel" problem?
> 

It sounds more like a "modify kernel .config" problem. But it could be a 
regression; what versions of qemu and kernel did you try?

> Rob
> 

Re: Running m68k on qemu with external initramfs?

[John Paul Adrian Glaubitz]

Hi Rob!

On 4/11/20 2:50 AM, Rob Landley wrote:
> I'm adding m68k support to toybox's "make root" target and I have a kernel and a
> rootfs.cpio.gz, but qemu's -initrd option doesn't put the rootfs image somewhere
> the m68k kernel can find it.

Can you try whether this guide works for you with your version of qemu?

> https://wiki.debian.org/M68k/QemuSystemM68k

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: Running m68k on qemu with external initramfs?

[Rob Landley]

On 4/11/20 1:12 AM, Finn Thain wrote:
> On Fri, 10 Apr 2020, Rob Landley wrote:
> 
>> I'm adding m68k support to toybox's "make root" target and I have a 
>> kernel and a rootfs.cpio.gz, but qemu's -initrd option doesn't put the 
>> rootfs image somewhere the m68k kernel can find it.
>>
>> Is this a "modify qemu" problem or a "modify kernel" problem?
> 
> It sounds more like a "modify kernel .config" problem. But it could be a 
> regression; what versions of qemu and kernel did you try?

"Works for me", got it. I'll keep digging.

QEMU is the one Laurent Vivier maintains that's been out of tree for something
like 12 years now, the default branch claims to be "q800-dev" and the last
commit was in december. It still runs my old aboriginal linux system image from
2014 (boots to a shell prompt).

Kernel is vanilla v5.6 with the arch configured with:

QEMU="m68k -M q800" KARCH=m68k KARGS=ttyS0 VMLINUX=vmlinux
KCONF=MMU,M68040,M68KFPU_EMU,MAC,SCSI_MAC_ESP,MACINTOSH_DRIVERS,ADB,ADB_MACII,NET_CORE,MACSONIC,SERIAL_PMACZILOG,SERIAL_PMACZILOG_TTYS,SERIAL_PMACZILOG_CONSOLE

Which is top of:

BASE=EARLY_PRINTK,BINFMT_ELF,BINFMT_SCRIPT,NO_HZ,HIGH_RES_TIMERS,BLK_DEV,BLK_DEV_INITRD,RD_GZIP,BLK_DEV_LOOP,EXT4_FS,EXT4_USE_FOR_EXT2,VFAT_FS,FAT_DEFAULT_UTF8,MISC_FILESYSTEMS,SQUASHFS,SQUASHFS_XATTR,SQUASHFS_ZLIB,DEVTMPFS,DEVTMPFS_MOUNT,TMPFS,TMPFS_POSIX_ACL,NET,PACKET,UNIX,INET,IPV6,NETDEVICES,NET_CORE,NETCONSOLE,ETHERNET

As interpreted by:

  sed -E '/^$/d;s/([^,]*)($|,)/CONFIG_\1=y\n/g' <<< "$BASE,$KCONF > mini.conf
  make ARCH=$KARCH allnoconfig KCONFIG_ALLCONFIG=mini.conf
  make ARCH=$KARCH CROSS_COMPILE="$CROSS_COMPILE" -j $(nproc)

Plus of course:

  echo "qemu-system-$QEMU" '"$@"' -nographic -no-reboot -m 256 \
       "-kernel $(basename "$VMLINUX") -initrd root.cpio.gz" \
       "-append \"quiet panic=1 HOST=$TARGET console=$KARGS \$KARGS\"" \
       ${DTB:+-dtb "$(basename "$DTB")"} > "$OUTPUT/qemu-$TARGET.sh"

I.E. the usual. (I'm trying to add m68k support to
https://github.com/landley/toybox/blob/master/scripts/mkroot.sh which already
boots armv5l, armv7l, aarch64, i486, i686, x86_64, mips, mipsel, powerpc, s390x,
and sh4 to a shell prompt under qemu, although you still have to
KARGS=rdinit=/bin/sh because I haven't QUITE got my new shell making it through
the end of the init script. Should do in another couple days.)

Rob

Re: Running m68k on qemu with external initramfs?

[Finn Thain]

On Sat, 11 Apr 2020, Rob Landley wrote:

> On 4/11/20 1:12 AM, Finn Thain wrote:
> > On Fri, 10 Apr 2020, Rob Landley wrote:
> > 
> >> I'm adding m68k support to toybox's "make root" target and I have a 
> >> kernel and a rootfs.cpio.gz, but qemu's -initrd option doesn't put the 
> >> rootfs image somewhere the m68k kernel can find it.
> >>
> >> Is this a "modify qemu" problem or a "modify kernel" problem?
> > 
> > It sounds more like a "modify kernel .config" problem. But it could be a 
> > regression; what versions of qemu and kernel did you try?
> 
> "Works for me", got it. I'll keep digging.
> 

If you want to try a kernel binary that works for me, so as to narrow down 
the problem, you can get one from here:
https://sourceforge.net/projects/linux-mac68k/files/

> QEMU is the one Laurent Vivier maintains that's been out of tree for 
> something like 12 years now, the default branch claims to be "q800-dev" 
> and the last commit was in december.

Should be fine. For the last few months I've used mainline QEMU instead.

> It still runs my old aboriginal linux system image from 2014 (boots to a 
> shell prompt).
> 
> Kernel is vanilla v5.6 with the arch configured with:
> 
> QEMU="m68k -M q800" KARCH=m68k KARGS=ttyS0 VMLINUX=vmlinux
> KCONF=MMU,M68040,M68KFPU_EMU,MAC,SCSI_MAC_ESP,MACINTOSH_DRIVERS,ADB,ADB_MACII,NET_CORE,MACSONIC,SERIAL_PMACZILOG,SERIAL_PMACZILOG_TTYS,SERIAL_PMACZILOG_CONSOLE
> 
> Which is top of:
> 
> BASE=EARLY_PRINTK,BINFMT_ELF,BINFMT_SCRIPT,NO_HZ,HIGH_RES_TIMERS,BLK_DEV,BLK_DEV_INITRD,RD_GZIP,BLK_DEV_LOOP,EXT4_FS,EXT4_USE_FOR_EXT2,VFAT_FS,FAT_DEFAULT_UTF8,MISC_FILESYSTEMS,SQUASHFS,SQUASHFS_XATTR,SQUASHFS_ZLIB,DEVTMPFS,DEVTMPFS_MOUNT,TMPFS,TMPFS_POSIX_ACL,NET,PACKET,UNIX,INET,IPV6,NETDEVICES,NET_CORE,NETCONSOLE,ETHERNET
> 
> As interpreted by:
> 
>   sed -E '/^$/d;s/([^,]*)($|,)/CONFIG_\1=y\n/g' <<< "$BASE,$KCONF > mini.conf
>   make ARCH=$KARCH allnoconfig KCONFIG_ALLCONFIG=mini.conf
>   make ARCH=$KARCH CROSS_COMPILE="$CROSS_COMPILE" -j $(nproc)
> 
> Plus of course:
> 
>   echo "qemu-system-$QEMU" '"$@"' -nographic -no-reboot -m 256 \
>        "-kernel $(basename "$VMLINUX") -initrd root.cpio.gz" \
>        "-append \"quiet panic=1 HOST=$TARGET console=$KARGS \$KARGS\"" \
>        ${DTB:+-dtb "$(basename "$DTB")"} > "$OUTPUT/qemu-$TARGET.sh"
> 

That works for me, with a few tweaks:

qemu-system-m68k -M q800 -serial none -serial mon:stdio -nographic 
-no-reboot -m 256 -kernel vmlinux-4.14.167-mac-backport+ 
-initrd sysroot.cpio.gz -append console=ttyS0

> I.E. the usual. 

This is my usual invocation:

qemu-system-m68k -M q800 -m 992M
-serial none -serial mon:stdio -g 800x600x4
-drive file=guest-root,format=raw
-drive file=guest-swap,format=raw
-append "fbcon=font:ProFont6x11 console=tty0 console=ttyS0 root=/dev/sda rw ignore_loglevel"
-kernel vmlinux
-net nic,model=dp83932,addr=00:00:00:01:02:03 -net bridge
-rtc base=localtime

> (I'm trying to add m68k support to 
> https://github.com/landley/toybox/blob/master/scripts/mkroot.sh which 
> already boots armv5l, armv7l, aarch64, i486, i686, x86_64, mips, mipsel, 
> powerpc, s390x, and sh4 to a shell prompt under qemu, although you still 
> have to KARGS=rdinit=/bin/sh because I haven't QUITE got my new shell 
> making it through the end of the init script. Should do in another 
> couple days.)
> 

Nice!

> Rob
> 

Re: Running m68k on qemu with external initramfs?

[John Paul Adrian Glaubitz]

Hi!

> On Apr 12, 2020, at 5:31 AM, Rob Landley <rob@landley.net> wrote:
> 
> On 4/11/20 1:12 AM, Finn Thain wrote:
>>> On Fri, 10 Apr 2020, Rob Landley wrote:
>>> 
>>> I'm adding m68k support to toybox's "make root" target and I have a 
>>> kernel and a rootfs.cpio.gz, but qemu's -initrd option doesn't put the 
>>> rootfs image somewhere the m68k kernel can find it.
>>> 
>>> Is this a "modify qemu" problem or a "modify kernel" problem?
>> 
>> It sounds more like a "modify kernel .config" problem. But it could be a 
>> regression; what versions of qemu and kernel did you try?
> 
> "Works for me", got it. I'll keep digging.
> 
> QEMU is the one Laurent Vivier maintains that's been out of tree for something
> like 12 years now, the default branch claims to be "q800-dev" and the last
> commit was in december. It still runs my old aboriginal linux system image from
> 2014 (boots to a shell prompt).

I think most of these things should be merged upstream these days but Laurent probably needs to correct me on what’s missing.

> Kernel is vanilla v5.6 with the arch configured with:
> 
> QEMU="m68k -M q800" KARCH=m68k KARGS=ttyS0 VMLINUX=vmlinux
> KCONF=MMU,M68040,M68KFPU_EMU,MAC,SCSI_MAC_ESP,MACINTOSH_DRIVERS,ADB,ADB_MACII,NET_CORE,MACSONIC,SERIAL_PMACZILOG,SERIAL_PMACZILOG_TTYS,SERIAL_PMACZILOG_CONSOLE
> 
> Which is top of:
> 
> BASE=EARLY_PRINTK,BINFMT_ELF,BINFMT_SCRIPT,NO_HZ,HIGH_RES_TIMERS,BLK_DEV,BLK_DEV_INITRD,RD_GZIP,BLK_DEV_LOOP,EXT4_FS,EXT4_USE_FOR_EXT2,VFAT_FS,FAT_DEFAULT_UTF8,MISC_FILESYSTEMS,SQUASHFS,SQUASHFS_XATTR,SQUASHFS_ZLIB,DEVTMPFS,DEVTMPFS_MOUNT,TMPFS,TMPFS_POSIX_ACL,NET,PACKET,UNIX,INET,IPV6,NETDEVICES,NET_CORE,NETCONSOLE,ETHERNET
> 
> As interpreted by:
> 
>  sed -E '/^$/d;s/([^,]*)($|,)/CONFIG_\1=y\n/g' <<< "$BASE,$KCONF > mini.conf
>  make ARCH=$KARCH allnoconfig KCONFIG_ALLCONFIG=mini.conf
>  make ARCH=$KARCH CROSS_COMPILE="$CROSS_COMPILE" -j $(nproc)
> 
> Plus of course:
> 
>  echo "qemu-system-$QEMU" '"$@"' -nographic -no-reboot -m 256 \
>       "-kernel $(basename "$VMLINUX") -initrd root.cpio.gz" \
>       "-append \"quiet panic=1 HOST=$TARGET console=$KARGS \$KARGS\"" \
>       ${DTB:+-dtb "$(basename "$DTB")"} > "$OUTPUT/qemu-$TARGET.sh"
> 
> I.E. the usual. (I'm trying to add m68k support to
> https://github.com/landley/toybox/blob/master/scripts/mkroot.sh which already
> boots armv5l, armv7l, aarch64, i486, i686, x86_64, mips, mipsel, powerpc, s390x,
> and sh4 to a shell prompt under qemu, although you still have to
> KARGS=rdinit=/bin/sh because I haven't QUITE got my new shell making it through
> the end of the init script. Should do in another couple days.)

Very cool. Thanks a lot for doing this!

Adrian

Re: Running m68k on qemu with external initramfs?

[Laurent Vivier]

Le 12/04/2020 à 10:27, John Paul Adrian Glaubitz a écrit :
> Hi!
> 
>> On Apr 12, 2020, at 5:31 AM, Rob Landley <rob@landley.net> wrote:
>>
>> On 4/11/20 1:12 AM, Finn Thain wrote:
>>>> On Fri, 10 Apr 2020, Rob Landley wrote:
>>>>
>>>> I'm adding m68k support to toybox's "make root" target and I have a 
>>>> kernel and a rootfs.cpio.gz, but qemu's -initrd option doesn't put the 
>>>> rootfs image somewhere the m68k kernel can find it.
>>>>
>>>> Is this a "modify qemu" problem or a "modify kernel" problem?
>>>
>>> It sounds more like a "modify kernel .config" problem. But it could be a 
>>> regression; what versions of qemu and kernel did you try?
>>
>> "Works for me", got it. I'll keep digging.
>>
>> QEMU is the one Laurent Vivier maintains that's been out of tree for something
>> like 12 years now, the default branch claims to be "q800-dev" and the last
>> commit was in december. It still runs my old aboriginal linux system image from
>> 2014 (boots to a shell prompt).
> 
> I think most of these things should be merged upstream these days but Laurent probably needs to correct me on what’s missing.

Yes, all is merged in QEMU master for Q800 emulation.

What stays in q800-dev are dev patches needed to run MacOS ROM but
nothing works for the moment.

Thanks,
Laurent

Re: Running m68k on qemu with external initramfs?

[Rob Landley]

On 4/12/20 12:29 AM, Finn Thain wrote:
>>> It sounds more like a "modify kernel .config" problem. But it could be a 
>>> regression; what versions of qemu and kernel did you try?
>>
>> "Works for me", got it. I'll keep digging.
>>
> 
> If you want to try a kernel binary that works for me, so as to narrow down 
> the problem, you can get one from here:
> https://sourceforge.net/projects/linux-mac68k/files/

You have 4.1 and 4.14 kernels. The last one I had in my old build system (which
works but was uClibc based using gcc 4.2.1) was 4.3:
http://landley.net/aboriginal/downloads/binaries/system-image-m68k.tar.gz

But that was a static initramfs, not an external intitramfs.

>> QEMU is the one Laurent Vivier maintains that's been out of tree for 
>> something like 12 years now, the default branch claims to be "q800-dev" 
>> and the last commit was in december.
> 
> Should be fine. For the last few months I've used mainline QEMU instead.

Oh cool. (My last qemu build apparently just predated the merge, and hadn't had it.)

> That works for me, with a few tweaks:
>
> qemu-system-m68k -M q800 -serial none -serial mon:stdio -nographic 
> -no-reboot -m 256 -kernel vmlinux-4.14.167-mac-backport+ 
> -initrd sysroot.cpio.gz -append console=ttyS0 

Yeah, it's looking like it's something about my userspace build. (A problem
which is unique to m68k because all the other targets are working. Sigh.)

I'll keep digging...

And this makes the segfault go away:

--- a/toys/pending/sh.c
+++ b/toys/pending/sh.c
@@ -2575,7 +2588,7 @@ void cd_main(void)
     if (to-dd>1 && to[-1]=='/') to--;
     *to = 0;
   }
-
+write(1, "", 0);
   if (bad || chdir(dd)) perror_msg("chdir '%s'", dd);
   else {
     if (pwd) {

So that's reassuring. I should run it under gdb to see what the segfault that
heisenbugs acutally _is_ but given it doesn't show up on any other architecture
(32 or 64 bit, big or little endian, sh4 cares about alignment, and I've built
it under glibc musl and bionic), I'm going to throw m68k back on the todo heap
for now.

Rob

Re: Running m68k on qemu with external initramfs?

[Rob Landley]

On 4/11/20 7:12 AM, John Paul Adrian Glaubitz wrote:
> Hi Rob!

Hello.

> On 4/11/20 2:50 AM, Rob Landley wrote:
>> I'm adding m68k support to toybox's "make root" target and I have a kernel and a
>> rootfs.cpio.gz, but qemu's -initrd option doesn't put the rootfs image somewhere
>> the m68k kernel can find it.
> 
> Can you try whether this guide works for you with your version of qemu?
> 
>> https://wiki.debian.org/M68k/QemuSystemM68k

Install debian to a block device to see why initramfs isn't working?

I already downloaded and ran
http://landley.net/aboriginal/downloads/binaries/system-image-m68k.tar.gz to
make sure laurent's tip of tree qemu-m68k is working, and that booted to a shell
prompt fine, but that's using a static initramfs instead of an external once,
which is why I thought that was the issue. (It used to be back when I first made
that work, which was something like 2014.)

I also ran my toybox binary under qemu-m68k to smoketest that the compiler was
making binaries that ran, but it turns out that I had two problems:

1) I'd run "make defconfig in the wrong directory and toybox switched the shell
back off (because it's still in "pending" until I finish it), hence the not
finding init. I stuck enough printk(KERN_ERR) into init/initramfs.c to list all
the filenames it was extracting and figured that one out.

2) THEN the problem was that gcc is doing something in cd_main so that the
"chdir ." at the start of the shell setup (to set $PWD and friends) is
segfaulting, but it's a segfault that went away when I got near it with printf()
calls, and even write(1, "", 0); right before it made it go away. Given that
this code has run on 32 and 64 bit, big and little endian, glibc musl and
bionic, and on a target that's both nommu and cares about alignment?

Someday I might build gdb for m68k, but until then I should check it next time I
upgrade compiler versions. (Or go through disabling every optimization one by
one until I find the one that's making the registers go nuts the way sh4 had
that https://landley.net/notes-2020.html#29-01-2020 thing.)

Rob

P.S: For context, I built a musl-cross-make toolchain and symlinked it into
toybox, and then used "make root CROSS=m68k LINUX=~/linux" to build a system:

git clone https://github.com/landley/musl-cross-make
git clone https://github.com/landley/toybox
cd musl-cross-make
sed -Ei 's/(BINUTILS_VER =).*/\1 2.32/;s/(LINUX_VER =).*/\1 4.19.90/' Makefile
../toybox/scripts/mcm-buildall.sh i686:: m68k::
cd ../toybox
ln -s ccc ../musl-cross-make/ccc
make root CROSS=i686 LINUX=~/linux
make root CROSS=m68k LINUX=~/linux

If you (cd root/i686; ./qemu-*.sh) it boots to a shell prompt.... well, if you
KARGS=rdinit=/bin/sh it does, running the init script still says:

ln: cannot create symbolic link from '/proc/self/fdi/,*/}' to 'dev/i/*,/}': No y
ln: cannot create symbolic link from '/proc/self/fdi/,*/}' to 'dev/i/*,/}': No y
ln: cannot create symbolic link from '/proc/self/fdi/,*/}' to 'dev/i/*,/}': No y
ln: cannot create symbolic link from '/proc/self/fdi/,*/}' to 'dev/i/*,/}': No y
sh: exec /etc/rc/*: No such file or directory
Type exit when done.
oneit: /dev/CONSOLE:-console}: No such file or directory

Because I'm still halfway through teaching toysh about all the ${} constructs.
(Also, remind me to stick echo -e '\e[?7h' at the start of my init script to
undo the STUPID wordwrap disable escape that qemu's bios outputs, which screws
up bash's command line history too!)

Anyway, at least it's known behavior. But in root/m68k I'm getting:

Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
CPU: 0 PID: 1 Comm: init Not tainted 5.6.0 #1
Stack from 0f81bdfc:

But I just went over that in the previous email I sent (downside of having two
reply windows open at once), and am clicking "send" on this now because the
sun's coming up and I should have been in bed a while ago. Sorry if it's
duplicative.

P.P.S. Zzzzzzz....

Re: Running m68k on qemu with external initramfs?

[John Paul Adrian Glaubitz]

On 4/12/20 2:48 PM, Rob Landley wrote:
> Install debian to a block device to see why initramfs isn't working?

d-i uses a ramdisk so I assume that should be similar to what you are
using.

> I already downloaded and ran
> http://landley.net/aboriginal/downloads/binaries/system-image-m68k.tar.gz to
> make sure laurent's tip of tree qemu-m68k is working, and that booted to a shell
> prompt fine, but that's using a static initramfs instead of an external once,
> which is why I thought that was the issue. (It used to be back when I first made
> that work, which was something like 2014.)

Not sure I understand what your particular configuration is but I have never
run into issues with an initramfs and the root filesystem in RAM on m68k,
native or with qemu.

> I also ran my toybox binary under qemu-m68k to smoketest that the compiler was
> making binaries that ran, but it turns out that I had two problems:
> 
> 1) I'd run "make defconfig in the wrong directory and toybox switched the shell
> back off (because it's still in "pending" until I finish it), hence the not
> finding init. I stuck enough printk(KERN_ERR) into init/initramfs.c to list all
> the filenames it was extracting and figured that one out.
> 
> 2) THEN the problem was that gcc is doing something in cd_main so that the
> "chdir ." at the start of the shell setup (to set $PWD and friends) is
> segfaulting, but it's a segfault that went away when I got near it with printf()
> calls, and even write(1, "", 0); right before it made it go away. Given that
> this code has run on 32 and 64 bit, big and little endian, glibc musl and
> bionic, and on a target that's both nommu and cares about alignment?
> 
> Someday I might build gdb for m68k, but until then I should check it next time I
> upgrade compiler versions. (Or go through disabling every optimization one by
> one until I find the one that's making the registers go nuts the way sh4 had
> that https://landley.net/notes-2020.html#29-01-2020 thing.)
> 
> Rob
> 
> P.S: For context, I built a musl-cross-make toolchain and symlinked it into
> toybox, and then used "make root CROSS=m68k LINUX=~/linux" to build a system:
> 
> git clone https://github.com/landley/musl-cross-make
> git clone https://github.com/landley/toybox
> cd musl-cross-make
> sed -Ei 's/(BINUTILS_VER =).*/\1 2.32/;s/(LINUX_VER =).*/\1 4.19.90/' Makefile
> ../toybox/scripts/mcm-buildall.sh i686:: m68k::
> cd ../toybox
> ln -s ccc ../musl-cross-make/ccc
> make root CROSS=i686 LINUX=~/linux
> make root CROSS=m68k LINUX=~/linux
> 
> If you (cd root/i686; ./qemu-*.sh) it boots to a shell prompt.... well, if you
> KARGS=rdinit=/bin/sh it does, running the init script still says:
> 
> ln: cannot create symbolic link from '/proc/self/fdi/,*/}' to 'dev/i/*,/}': No y
> ln: cannot create symbolic link from '/proc/self/fdi/,*/}' to 'dev/i/*,/}': No y
> ln: cannot create symbolic link from '/proc/self/fdi/,*/}' to 'dev/i/*,/}': No y
> ln: cannot create symbolic link from '/proc/self/fdi/,*/}' to 'dev/i/*,/}': No y
> sh: exec /etc/rc/*: No such file or directory
> Type exit when done.
> oneit: /dev/CONSOLE:-console}: No such file or directory
> 
> Because I'm still halfway through teaching toysh about all the ${} constructs.
> (Also, remind me to stick echo -e '\e[?7h' at the start of my init script to
> undo the STUPID wordwrap disable escape that qemu's bios outputs, which screws
> up bash's command line history too!)
> 
> Anyway, at least it's known behavior. But in root/m68k I'm getting:
> 
> Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
> CPU: 0 PID: 1 Comm: init Not tainted 5.6.0 #1
> Stack from 0f81bdfc:

So, do I understand correctly that this is buildroot with musl?

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: Running m68k on qemu with external initramfs?

[Rob Landley]

On 4/12/20 3:31 AM, Laurent Vivier wrote:
>>> QEMU is the one Laurent Vivier maintains that's been out of tree for something
>>> like 12 years now, the default branch claims to be "q800-dev" and the last
>>> commit was in december. It still runs my old aboriginal linux system image from
>>> 2014 (boots to a shell prompt).
>>
>> I think most of these things should be merged upstream these days but Laurent probably needs to correct me on what’s missing.
> 
> Yes, all is merged in QEMU master for Q800 emulation.
> 
> What stays in q800-dev are dev patches needed to run MacOS ROM but
> nothing works for the moment.

I confirmed that upstream qemu is Doing The Thing. Thanks and congratulations.
(It had been so long I stopped checking -git when my last vanilla build wasn't
doing it.)

What I've hit now is either a compiler bug or a subtle enough stack corruption
it doesn't manifest on any other platform and I haven't spotted it. I can fix it
by adding a NOP function call right before the segfault, but for the moment I'm
just punting worrying about it until after the next toybox release.

I checked in what I've got though. It builds a kernel and userspace for m68k,
and the _kernel_ seems to work.

Rob

P.S. I should poke Rich Felker again about fixing his kernel headers. He's
switched the default in musl-cross-make to use an out of tree kernel header
package from the sabotage project, which doesn't have s390x or m68k directories.
I'm just going "oh goddess, I fought this war 15 years ago with Mariusz Mazur's
headers package and I REALLY dowanna go back". He acknowledged the breakage last
month, but it seems to have fallen through the cracks. That's what the sed LINUX
is for in the toolchain build part of the reproduction sequence. The binutils
version change is because https://marc.info/?l=linux-sh&m=158544749818664&w=2
which ALSO still hasn't made it upstream, but hey, at least the kernel guys
added an SPDX license identifier to the .gitignore file, so they've got their
priorities.

Re: Running m68k on qemu with external initramfs?

[Rob Landley]

On 4/12/20 8:02 AM, John Paul Adrian Glaubitz wrote:
> On 4/12/20 2:48 PM, Rob Landley wrote:
>> Install debian to a block device to see why initramfs isn't working?
> 
> d-i uses a ramdisk so I assume that should be similar to what you are
> using.

Ah, I'd missed that part. (I did the debian cross-setup for arm and powerpc a
couple years back, but hadn't tried it for m68k.)

> So, do I understand correctly that this is buildroot with musl?

It's mkroot, not buildroot. My current system builder is a single bash script,
currently 231 lines long:

  https://github.com/landley/toybox/blob/master/scripts/mkroot.sh

Adding m68k support to it was 3 lines, which thunderbird does NOT want to indent
properly:

+elif [ "$TARGET" == m68k ]; then
+QEMU="m68k -M q800" KARCH=m68k KARGS=ttyS0 VMLINUX=vmlinux
+KCONF=MMU,M68040,M68KFPU_EMU,MAC,SCSI_MAC_ESP,MACINTOSH_DRIVERS,ADB,ADB_MACII,NET_CORE,MACSONIC,SERIAL_PMACZILOG,SERIAL_PMACZILOG_TTYS,SERIAL_PMACZILOG_CONSOLE

It uses an externally supplied cross compiler, and I'm currently trying to
support every target musl supports using musl-cross-make. (It also works with
the devuan host toolchain (glibc). I made a separate script to build all the
musl-cross-make targets although in theory you could just use the prebuilt
binary ones on https://musl.cc/ and at one point I had it working with the
Android NDK (bionic) although that was several releases back and I haven't
regression tested recently. Toybox gets regression tested against llvm by the
android guys, but building the kernel with llvm was fiddly.)

To build all supported targets, you do something like:

  git clone https://github.com/landley/musl-cross-make
  git clone https://github.com/landley/toybox
  cd musl-cross-make
  sed -Ei 's/(BINUTILS_VER =).*/\1 2.32/;s/(LINUX_VER =).*/\1 4.19.90/' Makefile
  ../toybox/scripts/mcm-buildall.sh
  cd ../toybox
  ln -s ccc ../musl-cross-make/ccc
  make defconfig
  make menuconfig # switch on sh and route in pending/
  make root CROSS=all LINUX=~/linux ALL=1

(That last ALL=1 tells it to continue to the next architecture instead of
stopping for build breaks. I haven't got kernel configs and QEMU invocations for
the armv7r and such.)

Then booting one under qemu is just:

  (cd root/sh4; ./qemu-*.sh)

Except I'm still implementing the shell so it doesn't QUITE make it through the
init script, so to get a prompt you need to:

  (cd root/sh4; KARGS=rdinit=/bin/sh ./qemu-*.sh)

*shrug* Working on it...

Rob

Re: Running m68k on qemu with external initramfs?

[Finn Thain]

On Sun, 12 Apr 2020, Rob Landley wrote:

> 
> What I've hit now is either a compiler bug or a subtle enough stack 
> corruption it doesn't manifest on any other platform and I haven't 
> spotted it. I can fix it by adding a NOP function call right before the 
> segfault, but for the moment I'm just punting worrying about it until 
> after the next toybox release.
> 

Can you provide us with an initramfs/initrd that would reproduce this?

I can try it on Aranym and Laurent or Stan may be willing to test on 
Motorola hardware.

GCC?, was Re: Running m68k on qemu with external initramfs?

[Finn Thain]

On Sun, 12 Apr 2020, Rob Landley wrote:

> 
> 2) THEN the problem was that gcc is doing something in cd_main so that 
> the "chdir ." at the start of the shell setup (to set $PWD and friends) 
> is segfaulting, but it's a segfault that went away when I got near it 
> with printf() calls, and even write(1, "", 0); right before it made it 
> go away. Given that this code has run on 32 and 64 bit, big and little 
> endian, glibc musl and bionic, and on a target that's both nommu and 
> cares about alignment?

Are you sure that "gcc is doing something [wrong]"? Do you think it could 
be a QEMU bug? Can you provide some information about your tool chain?

> 
> Because I'm still halfway through teaching toysh about all the ${} 
> constructs. (Also, remind me to stick echo -e '\e[?7h' at the start of 
> my init script to undo the STUPID wordwrap disable escape that qemu's 
> bios outputs, which screws up bash's command line history too!)

How awful. It baffles me every time bash manages to mishandle unprintable 
output from a command such that it somehow becomes command line input 
which bash then attempts to execute! I wish I knew how to prevent that.

Re: GCC?, was Re: Running m68k on qemu with external initramfs?

[Rob Landley]

On 4/12/20 6:30 PM, Finn Thain wrote:
> On Sun, 12 Apr 2020, Rob Landley wrote:
> 
>>
>> 2) THEN the problem was that gcc is doing something in cd_main so that 
>> the "chdir ." at the start of the shell setup (to set $PWD and friends) 
>> is segfaulting, but it's a segfault that went away when I got near it 
>> with printf() calls, and even write(1, "", 0); right before it made it 
>> go away. Given that this code has run on 32 and 64 bit, big and little 
>> endian, glibc musl and bionic, and on a target that's both nommu and 
>> cares about alignment?
> 
> Are you sure that "gcc is doing something [wrong]"? Do you think it could 
> be a QEMU bug? Can you provide some information about your tool chain?

I provided the reproduction sequence to build the toolchain from source a post
or two back as part of the general reproduction sequence?

Here's the version and config stuff:

m68k-linux-musl-cc --version
m68k-linux-musl-cc (GCC) 9.2.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

m68k-linux-musl-ld --version
GNU ld (GNU Binutils) 2.32
Copyright (C) 2019 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) a later version.
This program has absolutely no warranty.

$ ./m68k-linux-musl-cc -v
Using built-in specs.
COLLECT_GCC=./m68k-linux-musl-cc
COLLECT_LTO_WRAPPER=/home/landley/mcm/clean/ccc/m68k-linux-musl-cross/bin/../libexec/gcc/m68k-linux-musl/9.2.0/lto-wrapper
Target: m68k-linux-musl
Configured with: ../src_gcc/configure --enable-languages=c,c++ CFLAGS=' -g0 -Os'
CXXFLAGS=' -g0 -Os' LDFLAGS=' -s' CC='i686-linux-musl-gcc -static --static'
CXX='i686-linux-musl-g++ -static --static' --disable-nls --disable-libquadmath
--disable-decimal-float --disable-multilib --enable-languages=c,c++
--disable-bootstrap --disable-assembly --disable-werror --target=m68k-linux-musl
--prefix= --libdir=/lib --disable-multilib --with-sysroot=/m68k-linux-musl
--enable-tls --disable-libmudflap --disable-libsanitizer
--disable-gnu-indirect-function --disable-libmpx --enable-libstdcxx-time=rt
--with-build-sysroot=/home/landley/mcm/clean/build/local/m68k-linux-musl/obj_sysroot
AR_FOR_TARGET=/home/landley/mcm/clean/build/local/m68k-linux-musl/obj_binutils/binutils/ar
AS_FOR_TARGET=/home/landley/mcm/clean/build/local/m68k-linux-musl/obj_binutils/gas/as-new
LD_FOR_TARGET=/home/landley/mcm/clean/build/local/m68k-linux-musl/obj_binutils/ld/ld-new
NM_FOR_TARGET=/home/landley/mcm/clean/build/local/m68k-linux-musl/obj_binutils/binutils/nm-new
OBJCOPY_FOR_TARGET=/home/landley/mcm/clean/build/local/m68k-linux-musl/obj_binutils/binutils/objcopy
OBJDUMP_FOR_TARGET=/home/landley/mcm/clean/build/local/m68k-linux-musl/obj_binutils/binutils/objdump
RANLIB_FOR_TARGET=/home/landley/mcm/clean/build/local/m68k-linux-musl/obj_binutils/binutils/ranlib
READELF_FOR_TARGET=/home/landley/mcm/clean/build/local/m68k-linux-musl/obj_binutils/binutils/readelf
STRIP_FOR_TARGET=/home/landley/mcm/clean/build/local/m68k-linux-musl/obj_binutils/binutils/strip-new
--build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu
Thread model: posix
gcc version 9.2.0 (GCC)

(I build toolchains statically linked on against musl on the host so they're
portable and you can just extract the tarball of binaries and run it). And it's
built with i686 instead of x86-64 because that runs faster due to cache
locality: 90% of the reason for x32 was the 32 bit time_t and that got fixed
anyway, so just "statically link 32 bit binary" and you get all the benefits of
that these days...)

I have a todo item to pare my 2700 line toysh source file (plus however much of
main.c and lib/*.c it's pulling in, "LDFLAGS=--static
CROSS_COMPILE=ccc/m68k-linux-musl-cross/bin/m68k-linux-musl- make sh" makes a
99k m68k binary) down into a minimal reproduction thingy, but I'm trying to get
the other targets to finish parsing the init script so I can cut a toybox release.

>> Because I'm still halfway through teaching toysh about all the ${} 
>> constructs. (Also, remind me to stick echo -e '\e[?7h' at the start of 
>> my init script to undo the STUPID wordwrap disable escape that qemu's 
>> bios outputs, which screws up bash's command line history too!)
> 
> How awful. It baffles me every time bash manages to mishandle unprintable 
> output from a command such that it somehow becomes command line input 
> which bash then attempts to execute! I wish I knew how to prevent that.

This isn't bash's fault, this is qemu using a deeply obscure escape sequence
("man 4 console_codes" search for DECAWM) to leave the terminal in a weird state.

Bash is assembling the command line fine, you just can't see the output because
the cursor alignment's off (and it keeps cursoring up one line every time you
edit stuff because bash thinks the cursor went to the next line and it didn't
because line wrap disabled). When it does this I hit home # enter to comment out
the line I'm editing, and then run "reset", then cursor up twice and resume
editing the line.

I just stuck the "fix it" escape sequence at the end of my qemu wrapper script.
I'd submit a patch to qemu, but the last few I've posted to that list got lost
in the noise.

Rob

Re: GCC?, was Re: Running m68k on qemu with external initramfs?

[Finn Thain]

On Sun, 12 Apr 2020, Rob Landley wrote:

> Target: m68k-linux-musl
> Configured with: ../src_gcc/configure --enable-languages=c,c++ CFLAGS=' -g0 -Os'

Was toysh also built with -Os in CFLAGS? If so, I suggest you try -O2. The 
reason being that my Gentoo/m68k system failed to build Python with -Os.
Like your toysh seg fault, this python seg fault also happened on 
qemu-system-m68k. This toolchain uses glibc 2.31, gcc 9.3 and binutils 
2.34.

building '_ctypes' extension
...
/bin/sh: line 9: 11966 Segmentation fault      
LD_LIBRARY_PATH=/var/tmp/portage/dev-lang/python-3.7.7-r1/work/Python-3.7.7 
CC='m68k-unknown-linux-gnu-gcc -pthread' 
LDSHARED='m68k-unknown-linux-gnu-gcc -pthread -shared -Wl,-O1 
-Wl,--as-needed -L.   ' CFLAGS='-Wno-unused-result -Wsign-compare -DNDEBUG  
-Os -fwrapv  ' _TCLTK_INCLUDES='' _TCLTK_LIBS='' ./python -E ./setup.py 
$quiet build
make: *** [Makefile:628: sharedmods] Error 139

> I have a todo item to pare my 2700 line toysh source file (plus however 
> muchof main.c and lib/*.c it's pulling in, "LDFLAGS=--static 
> CROSS_COMPILE=ccc/m68k-linux-musl-cross/bin/m68k-linux-musl- make sh" 
> makes a 99k m68k binary) down into a minimal reproduction thingy, but 
> I'm trying to get the other targets to finish parsing the init script so 
> I can cut a toybox release.

It would be great to have a minimal reproducer from toysh. Getting that 
from the python build looks hard, even with creduce.
https://embed.cs.utah.edu/creduce/

Re: GCC?, was Re: Running m68k on qemu with external initramfs?

[Rob Landley]

On 4/13/20 12:17 AM, Finn Thain wrote:
> On Sun, 12 Apr 2020, Rob Landley wrote:
> 
>> Target: m68k-linux-musl
>> Configured with: ../src_gcc/configure --enable-languages=c,c++ CFLAGS=' -g0 -Os'
> 
> Was toysh also built with -Os in CFLAGS? If so, I suggest you try -O2. The 

It was. -O2 still segfaults the same way, but -O1 runs fine.

Sigh, tracking this down,
https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html says -O2 adds:

-falign-functions  -falign-jumps -falign-labels  -falign-loops -fcaller-saves
-fcode-hoisting -fcrossjumping -fcse-follow-jumps  -fcse-skip-blocks
-fdelete-null-pointer-checks -fdevirtualize  -fdevirtualize-speculatively
-fexpensive-optimizations -ffinite-loops -fgcse  -fgcse-lm
-fhoist-adjacent-loads -finline-functions -finline-small-functions
-findirect-inlining -fipa-bit-cp  -fipa-cp  -fipa-icf -fipa-ra  -fipa-sra
-fipa-vrp -fisolate-erroneous-paths-dereference -flra-remat
-foptimize-sibling-calls -foptimize-strlen -fpartial-inlining -fpeephole2
-freorder-blocks-algorithm=stc -freorder-blocks-and-partition
-freorder-functions -frerun-cse-after-loop  -fschedule-insns  -fschedule-insns2
-fsched-interblock  -fsched-spec -fstore-merging -fstrict-aliasing
-fthread-jumps -ftree-builtin-call-dce -ftree-pre -ftree-switch-conversion
-ftree-tail-merge -ftree-vrp

So let's try...

unrecognized command line option '-ffinite-loops'; did you mean '-fsplit-loops'?

Did I mention I'm really done with gcc development ever since they decided to
rewrite everything in C++? Sigh. [grind grind grind...]

CFLAGS='-fgcse'

That's the one.

>> I have a todo item to pare my 2700 line toysh source file (plus however 
>> muchof main.c and lib/*.c it's pulling in, "LDFLAGS=--static 
>> CROSS_COMPILE=ccc/m68k-linux-musl-cross/bin/m68k-linux-musl- make sh" 
>> makes a 99k m68k binary) down into a minimal reproduction thingy, but 
>> I'm trying to get the other targets to finish parsing the init script so 
>> I can cut a toybox release.
> 
> It would be great to have a minimal reproducer from toysh. Getting that 
> from the python build looks hard, even with creduce.
> https://embed.cs.utah.edu/creduce/

Oh it's not hard, it's just tedious.

Rob

Re: GCC?, was Re: Running m68k on qemu with external initramfs?

[John Paul Adrian Glaubitz]

On 4/13/20 9:07 AM, Rob Landley wrote:
> It was. -O2 still segfaults the same way, but -O1 runs fine.
> (...)
> Did I mention I'm really done with gcc development ever since they decided to
> rewrite everything in C++? Sigh. [grind grind grind...]
> 
> CFLAGS='-fgcse'
> 
> That's the one.
You should try gcc-10 where the m68k backend has seen a large overhaul.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: GCC?, was Re: Running m68k on qemu with external initramfs?

[Rob Landley]

On 4/13/20 2:07 AM, Rob Landley wrote:
>> It would be great to have a minimal reproducer from toysh. Getting that 
>> from the python build looks hard, even with creduce.
>> https://embed.cs.utah.edu/creduce/
> 
> Oh it's not hard, it's just tedious.

Did you know that if you disable optimizations you can get _more_ warnings?

_disabling_ the gcse optimization triggered one of those "may be used
uninitialized but is a false positive 99% of the time" which in this case,
turned out to have a path that could trigger in a function I added last week,
which which was called in an else case 5 lines down. (Yes, when it DIDN'T
segfault, it gave me the warning.)

My bad,

Rob

Re: GCC?, was Re: Running m68k on qemu with external initramfs?

[Geert Uytterhoeven]

Hi Rob,

On Mon, Apr 13, 2020 at 11:37 AM Rob Landley <rob@landley.net> wrote:
> On 4/13/20 2:07 AM, Rob Landley wrote:
> >> It would be great to have a minimal reproducer from toysh. Getting that
> >> from the python build looks hard, even with creduce.
> >> https://embed.cs.utah.edu/creduce/
> >
> > Oh it's not hard, it's just tedious.
>
> Did you know that if you disable optimizations you can get _more_ warnings?
>
> _disabling_ the gcse optimization triggered one of those "may be used
> uninitialized but is a false positive 99% of the time" which in this case,
> turned out to have a path that could trigger in a function I added last week,
> which which was called in an else case 5 lines down. (Yes, when it DIDN'T
> segfault, it gave me the warning.)

It's indeed a pity.  I looked into each and every one of them when I could
still compile the kernel with gcc 4.1, to find the few cases that were real
bugs...

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: GCC?, was Re: Running m68k on qemu with external initramfs?

[Finn Thain]

On Mon, 13 Apr 2020, Geert Uytterhoeven wrote:

> On Mon, Apr 13, 2020 at 11:37 AM Rob Landley <rob@landley.net> wrote:
> >
> > Did you know that if you disable optimizations you can get _more_ 
> > warnings?
> >
> > _disabling_ the gcse optimization triggered one of those "may be used 
> > uninitialized but is a false positive 99% of the time" which in this 
> > case, turned out to have a path that could trigger in a function I 
> > added last week, which which was called in an else case 5 lines down. 
> > (Yes, when it DIDN'T segfault, it gave me the warning.)
> 
> It's indeed a pity.  I looked into each and every one of them when I 
> could still compile the kernel with gcc 4.1, to find the few cases that 
> were real bugs...
> 

It appears code generation and static analysis really are separate 
problems. Code gen could be done faster when the user just wants an 
executable and has no use for the analysis (up to 'git bisect'). And 
static analysis works better when you have proper coverage, as you guys 
have shown.