Re: stack smashing detected

Re: stack smashing detected

[Geert Uytterhoeven]

CC linux-m68k

On Mon, Jan 30, 2023 at 5:01 AM Stan Johnson <userm57@yahoo.com> wrote:
>
> Hello,
>
> I am seeing anywhere from zero to four of the following errors while
> booting Linux on 68030 systems and using sysvinit startup scripts:
>
> *** stack smashing detected ***: terminated
> Aborted
>
> I usually (but not always) see three of the errors while init is running
> the rcS.d scripts, and one while running the rc2.d scripts. The stack
> smashing messages appear only on the system console (nothing is logged
> in an error log or dmesg). Despite the errors, the system continues
> booting to multiuser mode without any obvious additional problems. I
> haven't tested systemd, which is too slow to be useful on my m68k
> systems (though I have a Debian SID with systemd that I can restore for
> testing if necessary).
>
> I'm using the current Debian SID and Debian kernel, and I've confirmed
> the errors on a Mac IIci and SE/30. I haven't seen the errors on any
> 68040 system (I only tested on a Centris 650 and PowerBook 550c). I also
> notice the errors on 68030 systems using custom kernels that I have
> cross-compiled using GCC 12 or GCC 10 on a x86_64 system running Debian
> SID; however, I do not see the errors as often if I cross-compile using
> GCC 8.3.0 on a 686 system (running Debian 10.7 Buster) -- I saw the
> errors a few weeks ago with an earlier kernel, but none today using
> Linux 6.1.8 cross-compiled with GCC 8.3.0.
>
> I'll be happy to help debug or troubleshoot, though at this point, since
> the "stack smashing detected" errors aren't reporting which processes
> are being terminated/aborted, I'm not sure where to start.
>
> thanks for any suggestions
>
> -Stan Johnson   userm57@yahoo.com

Re: stack smashing detected

[Michael Schmitz]

Hi Stan,

Am 30.01.2023 um 17:00 schrieb Stan Johnson:
> Hello,
>
> I am seeing anywhere from zero to four of the following errors while
> booting Linux on 68030 systems and using sysvinit startup scripts:
>
> *** stack smashing detected ***: terminated
> Aborted
>
> I usually (but not always) see three of the errors while init is running
> the rcS.d scripts, and one while running the rc2.d scripts. The stack
> smashing messages appear only on the system console (nothing is logged
> in an error log or dmesg). Despite the errors, the system continues
> booting to multiuser mode without any obvious additional problems. I
> haven't tested systemd, which is too slow to be useful on my m68k
> systems (though I have a Debian SID with systemd that I can restore for
> testing if necessary).
>
> I'm using the current Debian SID and Debian kernel, and I've confirmed
> the errors on a Mac IIci and SE/30. I haven't seen the errors on any
> 68040 system (I only tested on a Centris 650 and PowerBook 550c). I also
> notice the errors on 68030 systems using custom kernels that I have
> cross-compiled using GCC 12 or GCC 10 on a x86_64 system running Debian
> SID; however, I do not see the errors as often if I cross-compile using
> GCC 8.3.0 on a 686 system (running Debian 10.7 Buster) -- I saw the
> errors a few weeks ago with an earlier kernel, but none today using
> Linux 6.1.8 cross-compiled with GCC 8.3.0.
>
> I'll be happy to help debug or troubleshoot, though at this point, since
> the "stack smashing detected" errors aren't reporting which processes
> are being terminated/aborted, I'm not sure where to start.

The man page of init states that init logs process and reason for 
termination in /var/run/utmp and /var/log/wtmp each time a child process 
terminates. You're looking for processed terminated by SIGABRT as far as 
I can see.

There does not appear to be any tool to extract that information from 
utmp/wtmp files though - utmpdump only shows login process information 
for me, nothing on init processes.

Another way may be logging the start of each of the rcS.d or rc2.d 
scripts until you know what scripts to look at in more detail, then 
adding 'set -v' at the start of those to log every command in the 
offending script.

Once the offending binary is known (and the crash can be reproduced 
after system boot), gdb can be used to find the function that overwrote 
its local stack guard.

That's a lot of work on a 030 Mac - have you tried to reproduce this on 
any kind of emulator?

I suppose one difference between your 030 and 040 Macs might be the 
amount of RAM available. I wonder if this bug results from a combination 
of 030 MMU and memory pressure, or 030 MMU only.

Cheers,

	Michael

>
> thanks for any suggestions
>
> -Stan Johnson   userm57@yahoo.com
>

Re: stack smashing detected

[Michael Schmitz]

Hi Stan,

On 2/02/23 05:38, Stan Johnson wrote:
> On 1/30/23 8:05 PM, Michael Schmitz wrote:
>> ...
>> Am 30.01.2023 um 17:00 schrieb Stan Johnson:
>>> Hello,
>>>
>>> I am seeing anywhere from zero to four of the following errors while
>>> booting Linux on 68030 systems and using sysvinit startup scripts:
>>>
>>> *** stack smashing detected ***: terminated
>>> Aborted
>>>
>>> I usually (but not always) see three of the errors while init is running
>>> the rcS.d scripts, and one while running the rc2.d scripts. The stack
>>> smashing messages appear only on the system console (nothing is logged
>>> in an error log or dmesg). Despite the errors, the system continues
>>> booting to multiuser mode without any obvious additional problems. I
>>> haven't tested systemd, which is too slow to be useful on my m68k
>>> systems (though I have a Debian SID with systemd that I can restore for
>>> testing if necessary).
>>>
>>> ...
>> Another way may be logging the start of each of the rcS.d or rc2.d
>> scripts until you know what scripts to look at in more detail, then
>> adding 'set -v' at the start of those to log every command in the
>> offending script.
> Hi Michael,
>
> Thanks for your reply.
>
> After logging the start and end of each script, I see that the "stack
> smashing detected" error often happens while running
> "/etc/rcS.d/S01mountkernfs.sh" (/etc/init.d/mountkernfs.sh). I'll try to
> isolate it to a particular command.
>
> This may be a coincidence, but the error seems to happen (up to about 4
> times) after a warm boot into Mac OS 7.5.5 and a subsequent boot into
> Linux that when starting with a cold boot into Mac OS 7.5.5, but it
> doesn't seem that that should make any difference for Linux. This
> morning, after a cold boot, I saw two of the errors, while after a warm
> boot, I saw four.
Hmm - that might well indicate a hardware issue rather than software. 
Bits flipping at random in RAM (and getting picked up because the stack 
canary changes).
>
>> Once the offending binary is known (and the crash can be reproduced
>> after system boot), gdb can be used to find the function that overwrote
>> its local stack guard.
> Is there a way to configure the kernel to use the stack guard for every
> function, and then log every resulting abort? I realize that that would
> be very slow, but it might also be useful for debugging.

The stack canary mechanism pushes a token on the stack at function 
entry, and compares against that token's value at function exit. This is 
all code generated by gcc in the user binary.

The kernel is not involved in function calls other than syscalls. For 
syscalls, we could try to find the user mode stack, and do a similar 
canary trick, but I don't think that would be necessary for all 
syscalls. Might be easier to instrument copy_to_user() instead if you're 
worried about a syscall receiving result data that way to a variable on 
the stack.

But since we're touching on copy_to_user() here - the 'remove set_fs' 
patch set by Christoph Hellwig refactored the m68k inline helpers around 
July 2021. Can you test a kernel prior to those patches (5.15-rc2)?

>
>> That's a lot of work on a 030 Mac - have you tried to reproduce this on
>> any kind of emulator?
> I haven't seen the error in QEMU.
>
>> I suppose one difference between your 030 and 040 Macs might be the
>> amount of RAM available. I wonder if this bug results from a combination
>> of 030 MMU and memory pressure, or 030 MMU only.
> For some reason, the error seems to happen only with 68030 systems,
> regardless of processor speed or memory:
>
> PB 170      : 68030, 25 MHz, 8 MiB, external SCSI2SD
> Mac IIci    : 68030, 25 MHz, 80 MiB, internal SCSI2SD
> SE/30       : 68030, 16 MHz, 128 MiB, external SCSI2SD
> PB 550c     : 68040, 33 MHz, 36 MiB, external SCSI2SD
> Centris 650 : 68040, 25 MHz, 136 MiB, internal SCSI2SD

Exception handling in copy_to_user() and the related bits in 030 page 
fault handling might need another look in then...

Cheers,

     Michael



>
> -Stan

Re: stack smashing detected

[Geert Uytterhoeven]

Hi Michael et al,

On Wed, Feb 1, 2023 at 7:52 PM Michael Schmitz <schmitzmic@gmail.com> wrote:
> On 2/02/23 05:38, Stan Johnson wrote:
> > On 1/30/23 8:05 PM, Michael Schmitz wrote:
> >> ...
> >> Am 30.01.2023 um 17:00 schrieb Stan Johnson:
> >>> I am seeing anywhere from zero to four of the following errors while
> >>> booting Linux on 68030 systems and using sysvinit startup scripts:
> >>>
> >>> *** stack smashing detected ***: terminated
> >>> Aborted
> >>>
> >>> I usually (but not always) see three of the errors while init is running
> >>> the rcS.d scripts, and one while running the rc2.d scripts. The stack
> >>> smashing messages appear only on the system console (nothing is logged
> >>> in an error log or dmesg). Despite the errors, the system continues
> >>> booting to multiuser mode without any obvious additional problems. I
> >>> haven't tested systemd, which is too slow to be useful on my m68k
> >>> systems (though I have a Debian SID with systemd that I can restore for
> >>> testing if necessary).
> >>>
> >>> ...
> >> Another way may be logging the start of each of the rcS.d or rc2.d
> >> scripts until you know what scripts to look at in more detail, then
> >> adding 'set -v' at the start of those to log every command in the
> >> offending script.
> > Hi Michael,
> >
> > Thanks for your reply.
> >
> > After logging the start and end of each script, I see that the "stack
> > smashing detected" error often happens while running
> > "/etc/rcS.d/S01mountkernfs.sh" (/etc/init.d/mountkernfs.sh). I'll try to
> > isolate it to a particular command.
> >
> > This may be a coincidence, but the error seems to happen (up to about 4
> > times) after a warm boot into Mac OS 7.5.5 and a subsequent boot into
> > Linux that when starting with a cold boot into Mac OS 7.5.5, but it
> > doesn't seem that that should make any difference for Linux. This
> > morning, after a cold boot, I saw two of the errors, while after a warm
> > boot, I saw four.
> Hmm - that might well indicate a hardware issue rather than software.
> Bits flipping at random in RAM (and getting picked up because the stack
> canary changes).

You can enable extra debugging options in the kernel, which might help
detecting memory corruption, like CONFIG_DEBUG_LIST and DEBUG_SLAB.
It will slow down your kernel, and make it grow too large, though.

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: stack smashing detected

[Michael Schmitz]

Hi Stan,

Am 03.02.2023 um 12:16 schrieb Stan Johnson:
> On 2/1/23 11:51 AM, Michael Schmitz wrote:
>> ...
>>
>> The stack canary mechanism pushes a token on the stack at function
>> entry, and compares against that token's value at function exit. This is
>> all code generated by gcc in the user binary.
>>
>> The kernel is not involved in function calls other than syscalls. For
>> syscalls, we could try to find the user mode stack, and do a similar
>> canary trick, but I don't think that would be necessary for all
>> syscalls. Might be easier to instrument copy_to_user() instead if you're
>> worried about a syscall receiving result data that way to a variable on
>> the stack.
>>
>> But since we're touching on copy_to_user() here - the 'remove set_fs'
>> patch set by Christoph Hellwig refactored the m68k inline helpers around
>> July 2021. Can you test a kernel prior to those patches (5.15-rc2)?
>> ...
> I updated my m68k rootfs in QEMU to the latest Debian SID and installed
> the updated rootfs on my Mac IIci. And it's now restoring to my SE/30,
> so I'll be able to test on that system again eventually.
>
> I tested 5.15.0-rc3, 5.15.0-rc2 and 5.15.0-rc1, with inconclusive
> results on the IIci (3 tests per kernel; e.g. "0, 1, 5" in
> "Stack-Smashing" indicates that test 1 had zero smashing errors, test 2
> had 1, and test 3 had 5):
>
> Kernel          Stack-Smashing
> 5.15.0-rc3	    0, 1, 5
> 5.15.0-rc2          2, 3, 2
> 5.15.0-rc1          7, 1, 3

I think we can rule out those changes to usermode copy routines now.

Good idea to test on some other hardware. I'd also consider Geert's 
suggestion to enable kernel level debugging for the kernel's memory 
allocators. That does look a lot easier than debugging usermode copy.

Cheers,

	Michael


> I saved the serial console logs, but they're probably not too useful at
> this point. Next, I'll confirm a similar failure on the SE/30, then I'll
> check 4.0 and 5.0 (on the IIci) to see whether the issue started in that
> range, then use git bisect in an attempt to isolate the issue further.
> I'll also do a spot check on a different IIci to lessen the chance that
> the issue is being caused by a hardware problem. If it's not caused by a
> kernel bug, the next step will be to isolate the specific offending
> executable(s) in the sysvinit scripts.
>
> Or I could test using systemd instead of sysvinit; that would take
> longer but it might be worthwhile if it doesn't look like a kernel bug.
>
> -Stan
>

Re: stack smashing detected

[Michael Schmitz]

Hi Stan,

Am 02.02.2023 um 07:51 schrieb Michael Schmitz:
>
> But since we're touching on copy_to_user() here - the 'remove set_fs'
> patch set by Christoph Hellwig refactored the m68k inline helpers around
> July 2021. Can you test a kernel prior to those patches (5.15-rc2)?
>
>>
>>> That's a lot of work on a 030 Mac - have you tried to reproduce this on
>>> any kind of emulator?
>> I haven't seen the error in QEMU.
>>
>>> I suppose one difference between your 030 and 040 Macs might be the
>>> amount of RAM available. I wonder if this bug results from a combination
>>> of 030 MMU and memory pressure, or 030 MMU only.
>> For some reason, the error seems to happen only with 68030 systems,
>> regardless of processor speed or memory:
>>
>> PB 170      : 68030, 25 MHz, 8 MiB, external SCSI2SD
>> Mac IIci    : 68030, 25 MHz, 80 MiB, internal SCSI2SD
>> SE/30       : 68030, 16 MHz, 128 MiB, external SCSI2SD
>> PB 550c     : 68040, 33 MHz, 36 MiB, external SCSI2SD
>> Centris 650 : 68040, 25 MHz, 136 MiB, internal SCSI2SD
>
> Exception handling in copy_to_user() and the related bits in 030 page
> fault handling might need another look in then...

Seeing Finn's report that Al Viro's VM_FAULT_RETRY fix may have solved 
his task corruption troubles on 040, I just noticed that I probably 
misunderstood how Al's patch works.

Botching up a fault retry and carrying on may well leave the page tables 
in a state where some later access could go to the wrong page and 
manifest as user space corruption. Could you try Al's patch 4 (m68k: fix 
livelock in uaccess) to see if this helps?

Cheers,

	Michael

Re: stack smashing detected

[Michael Schmitz]

Thanks Stan,

On 8/02/23 08:37, Stan Johnson wrote:
> Hi Michael,
>
> On 2/5/23 3:19 PM, Michael Schmitz wrote:
>> ...
>>
>> Seeing Finn's report that Al Viro's VM_FAULT_RETRY fix may have solved
>> his task corruption troubles on 040, I just noticed that I probably
>> misunderstood how Al's patch works.
>>
>> Botching up a fault retry and carrying on may well leave the page tables
>> in a state where some later access could go to the wrong page and
>> manifest as user space corruption. Could you try Al's patch 4 (m68k: fix
>> livelock in uaccess) to see if this helps?
>> ...
> ok, this appears to be the patch:
>
> Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
> ---
>   arch/m68k/mm/fault.c | 5 ++++-
>   1 file changed, 4 insertions(+), 1 deletion(-)
>
> diff --git a/arch/m68k/mm/fault.c b/arch/m68k/mm/fault.c
> index 4d2837eb3e2a..228128e45c67 100644
> --- a/arch/m68k/mm/fault.c
> +++ b/arch/m68k/mm/fault.c
> @@ -138,8 +138,11 @@ int do_page_fault(struct pt_regs *regs, unsigned
> long address,
>   	fault = handle_mm_fault(vma, address, flags, regs);
>   	pr_debug("handle_mm_fault returns %x\n", fault);
>
> -	if (fault_signal_pending(fault, regs))
> +	if (fault_signal_pending(fault, regs)) {
> +		if (!user_mode(regs))
> +			goto no_context;
>   		return 0;
> +	}
>
>   	/* The fault is fully completed (including releasing mmap lock) */
>   	if (fault & VM_FAULT_COMPLETED)

That's correct.

Your results show improvement but the problem does not entirely go away.

Looking at differences between 030 and 040/040 fault handling, it 
appears only 030 handles faults corrected by exception tables (such as 
used in uaccess macros) special, i.e. aborting bus error processing 
while 040 and 060 carry on in the fault handler.

I wonder if that's the main difference between 030 and 040 behaviour?

I'll try and log such accesses caught by exception tables on 030 to see 
if they are rare enough to allow adding a kernel log message...

Cheers,

     Michael

Re: stack smashing detected

[Michael Schmitz]

[-- Attachment #1: Type: text/plain, Size: 2536 bytes --]

Hi Stan,

Am 08.02.2023 um 11:58 schrieb Michael Schmitz:
> Thanks Stan,
>
> On 8/02/23 08:37, Stan Johnson wrote:
>> Hi Michael,
>>
>> On 2/5/23 3:19 PM, Michael Schmitz wrote:
>>> ...
>>>
>>> Seeing Finn's report that Al Viro's VM_FAULT_RETRY fix may have solved
>>> his task corruption troubles on 040, I just noticed that I probably
>>> misunderstood how Al's patch works.
>>>
>>> Botching up a fault retry and carrying on may well leave the page tables
>>> in a state where some later access could go to the wrong page and
>>> manifest as user space corruption. Could you try Al's patch 4 (m68k: fix
>>> livelock in uaccess) to see if this helps?
>>> ...
>> ok, this appears to be the patch:
>>
>> Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
>> ---
>>   arch/m68k/mm/fault.c | 5 ++++-
>>   1 file changed, 4 insertions(+), 1 deletion(-)
>>
>> diff --git a/arch/m68k/mm/fault.c b/arch/m68k/mm/fault.c
>> index 4d2837eb3e2a..228128e45c67 100644
>> --- a/arch/m68k/mm/fault.c
>> +++ b/arch/m68k/mm/fault.c
>> @@ -138,8 +138,11 @@ int do_page_fault(struct pt_regs *regs, unsigned
>> long address,
>>       fault = handle_mm_fault(vma, address, flags, regs);
>>       pr_debug("handle_mm_fault returns %x\n", fault);
>>
>> -    if (fault_signal_pending(fault, regs))
>> +    if (fault_signal_pending(fault, regs)) {
>> +        if (!user_mode(regs))
>> +            goto no_context;
>>           return 0;
>> +    }
>>
>>       /* The fault is fully completed (including releasing mmap lock) */
>>       if (fault & VM_FAULT_COMPLETED)
>
> That's correct.
>
> Your results show improvement but the problem does not entirely go away.
>
> Looking at differences between 030 and 040/040 fault handling, it
> appears only 030 handles faults corrected by exception tables (such as
> used in uaccess macros) special, i.e. aborting bus error processing
> while 040 and 060 carry on in the fault handler.
>
> I wonder if that's the main difference between 030 and 040 behaviour?

Following the 040 code a bit further, I suspect that happens in the 040 
writeback handler, so this may be a red herring.

> I'll try and log such accesses caught by exception tables on 030 to see
> if they are rare enough to allow adding a kernel log message...

Looks like this kind of event is rare enough to not trigger in a normal 
boot on my 030. Please give the attached patch a try so we can confirm 
(or rule out) that user space access faults from kernel mode are to 
blame for your stack smashes.

Cheers,

	Michael


> Cheers,
>
>     Michael
>
>

[-- Attachment #2: 0001-m68k-debug-exception-handling-data-faults-on-030.patch --]
[-- Type: text/x-diff, Size: 1333 bytes --]

From a55467a02b66addca6f74fc32b473bc077cb34b2 Mon Sep 17 00:00:00 2001
From: Michael Schmitz <schmitzmic@gmail.com>
Date: Thu, 9 Feb 2023 14:39:35 +1300
Subject: [PATCH] m68k: debug exception handling data faults on 030

030 faults handled by exception tables are just silently ignored - see how
many of these do happen in practice, and if they are related to 'stack
smashing' faults.

Signed-off-by: Michael Schmitz <schmitzmic@gmail.com>
---
 arch/m68k/kernel/traps.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/arch/m68k/kernel/traps.c b/arch/m68k/kernel/traps.c
index 5c8cba0efc63..b3cef760f7e8 100644
--- a/arch/m68k/kernel/traps.c
+++ b/arch/m68k/kernel/traps.c
@@ -554,8 +554,13 @@ static inline void bus_error030 (struct frame *fp)
 			}
 			/* Don't try to do anything further if an exception was
 			   handled. */
-			if (do_page_fault (&fp->ptregs, addr, errorcode) < 0)
+			if (do_page_fault (&fp->ptregs, addr, errorcode) < 0) {
+				pr_err("Exception handled for data %s fault at %#010lx in %s (pc=%#lx)\n",
+				       ssw & RW ? "read" : "write",
+				       fp->un.fmtb.daddr,
+				       space_names[ssw & DFC], fp->ptregs.pc);
 				return;
+			}
 		} else if (!(mmusr & MMU_I)) {
 			/* probably a 020 cas fault */
 			if (!(ssw & RM) && send_fault_sig(&fp->ptregs) > 0)
-- 
2.17.1

core dump analysis, was Re: stack smashing detected

[Finn Thain]

On Sat, 18 Feb 2023, I wrote:

> On Fri, 17 Feb 2023, Stan Johnson wrote:
> 
> > 
> > That's not to say a SIGABRT is ignored, it just doesn't kill PID 1.
> > 
> 
> I doubt that /sbin/init is generating the "stack smashing detected" 
> error but you may need to modify it to find out. If you can't figure out 
> which userland binary is involved, you'll have to focus on your custom 
> kernel binary, just as I proposed in my message dated 8 Feb 2023.
> 

Using the core dump I generated on my Mac LC III, and using a workaround 
for the gdb regression, I was able to get the backtrace below.

root@(none):/root# gdb
GNU gdb (Debian 13.1-2) 13.1
Copyright (C) 2023 Free Software Foundation, Inc.
...
(gdb) set osabi GNU/Linux
(gdb) exec /bin/dash
(gdb) core /root/core.0
warning: core file may not match specified executable file.
[New LWP 366]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/m68k-linux-gnu/libthread_db.so.1".
Core was generated by `/bin/sh /etc/init.d/mountkernfs.sh reload'.
Program terminated with signal SIGABRT, Aborted.
#0  __pthread_kill_implementation (threadid=3222954656, signo=6, no_tid=0)
    at pthread_kill.c:44
44      pthread_kill.c: No such file or directory.
(gdb) bt
#0  __pthread_kill_implementation (threadid=3222954656, signo=6, no_tid=0)
    at pthread_kill.c:44
#1  0xc00a7080 in __pthread_kill_internal (signo=6, threadid=3222954656)
    at pthread_kill.c:78
#2  __GI___pthread_kill (threadid=3222954656, signo=6) at pthread_kill.c:89
#3  0xc0064c22 in __GI_raise (sig=6) at ../sysdeps/posix/raise.c:26
#4  0xc0052faa in __GI_abort () at abort.c:79
#5  0xc009b328 in __libc_message (action=<optimized out>, fmt=<optimized out>)
    at ../sysdeps/posix/libc_fatal.c:155
#6  0xc012a3c2 in __GI___fortify_fail (
    msg=0xc0182c5e "stack smashing detected") at fortify_fail.c:26
#7  0xc012a3a0 in __stack_chk_fail () at stack_chk_fail.c:24
#8  0xc00e0172 in __wait3 (stat_loc=<optimized out>, options=<optimized out>, 
    usage=<optimized out>) at ../sysdeps/unix/sysv/linux/wait3.c:41
#9  0xd000c38e in ?? ()
#10 0xefee111e in ?? ()
#11 0x00000000 in ?? ()
(gdb) 

It appears that the failure was in glibc (though I guess the root cause 
may lie elsewhere). I have two more core files generated by dash (actually 
by `/bin/sh /etc/rcS.d/S08mountall.sh start') that give the same 
backtrace. So even though the failure is intermittent, the site of the 
buffer overrun seems to be consistent.

Looking at sysdeps/unix/sysv/linux/wait3.c, I guess the only possible 
place for a buffer overrun would be struct __rusage64 usage64.
https://sources.debian.org/src/glibc/2.36-8/sysdeps/unix/sysv/linux/wait3.c/?hl=41#L41

(gdb) select-frame 8
(gdb) print usage64
$3 = {ru_utime = {tv_sec = 6481621047248640, tv_usec = 91671782025504}, 
  ru_stime = {tv_sec = 25769811968, tv_usec = 8591449888}, {
    ru_maxrss = 1515296, __ru_maxrss_word = 1515296}, {ru_ixrss = 1515296, 
    __ru_ixrss_word = 1515296}, {ru_idrss = 224, __ru_idrss_word = 224}, {
    ru_isrss = 224, __ru_isrss_word = 224}, {ru_minflt = 6, 
    __ru_minflt_word = 6}, {ru_majflt = 4, __ru_majflt_word = 4}, {
    ru_nswap = 4, __ru_nswap_word = 4}, {ru_inblock = 372, 
    __ru_inblock_word = 372}, {ru_oublock = 0, __ru_oublock_word = 0}, {
    ru_msgsnd = 0, __ru_msgsnd_word = 0}, {ru_msgrcv = 8, 
    __ru_msgrcv_word = 8}, {ru_nsignals = 367, __ru_nsignals_word = 367}, {
    ru_nvcsw = 10, __ru_nvcsw_word = 10}, {ru_nivcsw = 0, 
    __ru_nivcsw_word = 0}}
(gdb)

Of course, at this point the damage has already been done and the culprit 
has gone. I guess there was a buffer overrun during the call to 
__wait4_time64(). 
https://sources.debian.org/src/glibc/2.36-8/sysdeps/unix/sysv/linux/wait4.c/?hl=26#L26

It's hard to read glibc source code without knowing what all the macros 
were set to (such as __KERNEL_OLD_TIMEVAL_MATCHES_TIMEVAL64 and 
__TIMESIZE).

It would be disappointing if rusage64_to_rusage() in __wait3() was being 
applied to the result of rusage32_to_rusage64() from __wait4_time64(). 
Perhaps the ifdefs are arranged in such a way that it doesn't happen...

Anyway, does anyone know how to get a hex dump of the whole stack frame 
including the canary, in case there is something to be learned from that?

Re: core dump analysis, was Re: stack smashing detected

[Finn Thain]

On 28 March 2023, I wrote,

> 
> Looking at sysdeps/unix/sysv/linux/wait3.c, I guess the only possible 
> place for a buffer overrun would be struct __rusage64 usage64. 
> https://sources.debian.org/src/glibc/2.36-8/sysdeps/unix/sysv/linux/wait3.c/?hl=41#L41
> 

... but now I see the usage64 variable is not involved at all because 
__wait3() was passed a NULL pointer: 
https://sources.debian.org/src/dash/0.5.12-2/src/jobs.c/?hl=1179#L1179

So NULL (rather than &usage64) was passed to the wait4() syscall which 
means the kernel didn't invoke copy_to_user() at all. AFAICS there's no 
possible buffer overflow in __wait3(), __wait4_time64() etc.

That suggests to a problem with GCC's SSP detector.

Here's a more complete backtrace and some disassembly.

# gdb
GNU gdb (Debian 13.1-2) 13.1
...
(gdb) set osabi GNU/Linux
(gdb) file /bin/dash
Reading symbols from /bin/dash...
Reading symbols from /usr/lib/debug/.build-id/aa/4160f84f3eeee809c554cb9f3e1ef0686b8dcc.debug...
(gdb) 
(gdb) core /root/core.0
warning: core file may not match specified executable file.
[New LWP 366]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/m68k-linux-gnu/libthread_db.so.1".
Core was generated by `/bin/sh /etc/init.d/mountkernfs.sh reload'.
Program terminated with signal SIGABRT, Aborted.
#0  __pthread_kill_implementation (threadid=3222954656, signo=6, no_tid=0)
    at pthread_kill.c:44
44      pthread_kill.c: No such file or directory.
(gdb) bt
#0  __pthread_kill_implementation (threadid=3222954656, signo=6, no_tid=0)
    at pthread_kill.c:44
#1  0xc00a7080 in __pthread_kill_internal (signo=6, threadid=3222954656)
    at pthread_kill.c:78
#2  __GI___pthread_kill (threadid=3222954656, signo=6) at pthread_kill.c:89
#3  0xc0064c22 in __GI_raise (sig=6) at ../sysdeps/posix/raise.c:26
#4  0xc0052faa in __GI_abort () at abort.c:79
#5  0xc009b328 in __libc_message (action=<optimized out>, fmt=<optimized out>)
    at ../sysdeps/posix/libc_fatal.c:155
#6  0xc012a3c2 in __GI___fortify_fail (
    msg=0xc0182c5e "stack smashing detected") at fortify_fail.c:26
#7  0xc012a3a0 in __stack_chk_fail () at stack_chk_fail.c:24
#8  0xc00e0172 in __wait3 (stat_loc=<optimized out>, options=<optimized out>, 
    usage=<optimized out>) at ../sysdeps/unix/sysv/linux/wait3.c:41
#9  0xd000c38e in waitproc (status=0xefee110e, block=1) at jobs.c:1179
#10 waitone (block=1, job=0xd0021930) at jobs.c:1055
#11 0xd000c5b8 in dowait (block=1, jp=0xd0021930) at jobs.c:1137
#12 0xd000ddb0 in waitforjob (jp=0xd0021930) at jobs.c:1014
#13 0xd000aade in expbackq (flag=324, cmd=0xd00222c4) at expand.c:520
#14 argstr (p=<optimized out>, flag=68) at expand.c:335
#15 0xd000b5ce in expandarg (arg=0xd00222ac, arglist=0xefee13bc, flag=4)
    at expand.c:192
#16 0xd0007e2a in evalcommand (cmd=<optimized out>, flags=<optimized out>)
    at eval.c:855
#17 0xd0006ffc in evaltree (n=0xd0022294, flags=0) at eval.c:300
#18 0xd0006e96 in evaltree (n=0xd0022294, flags=0) at eval.c:300
#19 0xd0006e6a in evaltree (n=0xd002224c, flags=0) at eval.c:292
#20 0xd0006e6a in evaltree (n=0xd00220d4, flags=0) at eval.c:292
#21 0xd0006e6a in evaltree (n=0xd002208c, flags=0) at eval.c:292
#22 0xd000746a in evalfun (func=0xd0022078, argc=<optimized out>, 
    argv=0xd001e61c <stackbase+376>, flags=<optimized out>) at eval.c:1009
#23 0xd0008176 in evalcommand (cmd=<optimized out>, flags=<optimized out>)
    at eval.c:921
#24 0xd0006ffc in evaltree (n=0xd001e588 <stackbase+228>, flags=1)
    at eval.c:300
#25 0xd00084c8 in evaltreenr (flags=1, n=0xd001e588 <stackbase+228>)
    at eval.c:347
#26 evalbackcmd (n=<optimized out>, result=0xefee17d4) at eval.c:650
#27 0xd000a984 in expbackq (flag=324, cmd=0xd001e588 <stackbase+228>)
    at expand.c:495
#28 argstr (p=<optimized out>, flag=68) at expand.c:335
#29 0xd000b5ce in expandarg (arg=0xd001e5b0 <stackbase+268>, 
    arglist=0xefee191c, flag=4) at expand.c:192
#30 0xd0007e2a in evalcommand (cmd=<optimized out>, flags=<optimized out>)
    at eval.c:855
#31 0xd0006ffc in evaltree (n=0xd001e5c0 <stackbase+284>, flags=0)
    at eval.c:300
#32 0xd000e3c0 in cmdloop (top=0) at main.c:246
#33 0xd000e588 in dotcmd (argc=2, argv=<optimized out>) at main.c:341
#34 0xd0007a12 in evalbltin (cmd=0xd001b598 <builtincmd>, 
    argc=<optimized out>, argv=<optimized out>, flags=<optimized out>)
    at eval.c:967
#35 0xd00080ca in evalcommand (cmd=<optimized out>, flags=<optimized out>)
    at eval.c:910
#36 0xd0006ffc in evaltree (n=0xd001e4e8 <stackbase+68>, flags=0) at eval.c:300
#37 0xd000e3c0 in cmdloop (top=1) at main.c:246
#38 0xd0005018 in main (argc=<optimized out>, argv=<optimized out>)
    at main.c:181
(gdb) frame 8
#8  0xc00e0172 in __wait3 (stat_loc=<optimized out>, options=<optimized out>, 
    usage=<optimized out>) at ../sysdeps/unix/sysv/linux/wait3.c:41
41      ../sysdeps/unix/sysv/linux/wait3.c: No such file or directory.
(gdb) info frame
Stack level 8, frame at 0xefee10e0:
 pc = 0xc00e0172 in __wait3 (../sysdeps/unix/sysv/linux/wait3.c:41); 
    saved pc = 0xd000c38e
 called by frame at 0xefee11dc, caller of frame at 0xefee106c
 source language c.
 Arglist at 0xefee10d8, args: stat_loc=<optimized out>, 
    options=<optimized out>, usage=<optimized out>
 Locals at 0xefee10d8, Previous frame's sp is 0xefee10e0
 Saved registers:
  a2 at 0xefee106c, a3 at 0xefee1070, a5 at 0xefee1074, fp at 0xefee10d8,
  pc at 0xefee10dc
(gdb) x/32z 0xefee1060
0xefee1060:     0xc0182c5e      0xc0198000      0xc00e0172      0xd001e718
0xefee1070:     0xd001e498      0xd001b874      0x00170700      0x00170700
0xefee1080:     0x00170700      0x00005360      0x0000e920      0x00000006
0xefee1090:     0x00002000      0x00000002      0x00171f20      0x00171f20
0xefee10a0:     0x00171f20      0x000000e0      0x000000e0      0x00000006
0xefee10b0:     0x00000004      0x00000004      0x00000174      0x00000000
0xefee10c0:     0x00000000      0x00000008      0x0000016f      0x0000000a
0xefee10d0:     0x00000000      0x00ac3dbe      0xd001c1ec      0xd000c38e
(gdb)
0xefee10e0:     0xefee111e      0x00000000      0x00000000      0x00000001
0xefee10f0:     0x00000001      0xefee1284      0x00000044      0xd0017714
0xefee1100:     0x00000100      0xd0021930      0xd001c1ec      0xd001e498
0xefee1110:     0xd001b874      0xefee1308      0xc0023e8c      0xefee0000
0xefee1120:     0x00000044      0xd0017714      0x00000100      0xefee1274
0xefee1130:     0xc0023e8c      0xd001c028      0xd001b874      0xefee1208
0xefee1140:     0x00000000      0xc0023e8c      0x00000000      0x00000000
0xefee1150:     0x00000000      0x00000000      0x00000000      0x00000000
(gdb) print &usage64
$1 = (struct __rusage64 *) 0xefee107c
(gdb) disass
Dump of assembler code for function __wait3:
   0xc00e0070 <+0>:     linkw %fp,#-96
   0xc00e0074 <+4>:     moveml %a2-%a3/%a5,%sp@-
   0xc00e0078 <+8>:     lea %pc@(0xc0198000),%a5
   0xc00e0080 <+16>:    movel %fp@(8),%d0
   0xc00e0084 <+20>:    moveal %fp@(16),%a2
   0xc00e0088 <+24>:    moveal %a5@(108),%a3
   0xc00e008c <+28>:    movel %a3@,%fp@(-4)
   0xc00e0090 <+32>:    tstl %a2
   0xc00e0092 <+34>:    beqw 0xc00e0152 <__wait3+226>
   0xc00e0096 <+38>:    pea %fp@(-92)
   0xc00e009a <+42>:    movel %fp@(12),%sp@-
   0xc00e009e <+46>:    movel %d0,%sp@-
   0xc00e00a0 <+48>:    pea 0xffffffff
   0xc00e00a4 <+52>:    bsrl 0xc00e0174 <__GI___wait4_time64>
   0xc00e00aa <+58>:    lea %sp@(16),%sp
   0xc00e00ae <+62>:    tstl %d0
   0xc00e00b0 <+64>:    bgts 0xc00e00c8 <__wait3+88>
   0xc00e00b2 <+66>:    moveal %fp@(-4),%a0
   0xc00e00b6 <+70>:    movel %a3@,%d1
   0xc00e00b8 <+72>:    cmpl %a0,%d1
   0xc00e00ba <+74>:    bnew 0xc00e016c <__wait3+252>
   0xc00e00be <+78>:    moveml %fp@(-108),%a2-%a3/%a5
   0xc00e00c4 <+84>:    unlk %fp
   0xc00e00c6 <+86>:    rts
   0xc00e00c8 <+88>:    pea 0x44
   0xc00e00cc <+92>:    clrl %sp@-
   0xc00e00ce <+94>:    pea %a2@(4)
   0xc00e00d2 <+98>:    movel %d0,%fp@(-96)
   0xc00e00d6 <+102>:   bsrl 0xc00b8850 <__GI_memset>
   0xc00e00dc <+108>:   movel %fp@(-88),%a2@
   0xc00e00e0 <+112>:   movel %fp@(-80),%a2@(4)
   0xc00e00e6 <+118>:   movel %fp@(-72),%a2@(8)
   0xc00e00ec <+124>:   movel %fp@(-64),%a2@(12)
   0xc00e00f2 <+130>:   movel %fp@(-60),%a2@(16)
   0xc00e00f8 <+136>:   movel %fp@(-56),%a2@(20)
   0xc00e00fe <+142>:   movel %fp@(-52),%a2@(24)
   0xc00e0104 <+148>:   movel %fp@(-48),%a2@(28)
   0xc00e010a <+154>:   movel %fp@(-44),%a2@(32)
   0xc00e0110 <+160>:   movel %fp@(-40),%a2@(36)
   0xc00e0116 <+166>:   movel %fp@(-36),%a2@(40)
   0xc00e011c <+172>:   movel %fp@(-32),%a2@(44)
   0xc00e0122 <+178>:   movel %fp@(-28),%a2@(48)
   0xc00e0128 <+184>:   movel %fp@(-24),%a2@(52)
   0xc00e012e <+190>:   movel %fp@(-20),%a2@(56)
   0xc00e0134 <+196>:   movel %fp@(-16),%a2@(60)
   0xc00e013a <+202>:   movel %fp@(-12),%a2@(64)
   0xc00e0140 <+208>:   movel %fp@(-8),%a2@(68)
   0xc00e0146 <+214>:   lea %sp@(12),%sp
   0xc00e014a <+218>:   movel %fp@(-96),%d0
   0xc00e014e <+222>:   braw 0xc00e00b2 <__wait3+66>
   0xc00e0152 <+226>:   clrl %sp@-
   0xc00e0154 <+228>:   movel %fp@(12),%sp@-
   0xc00e0158 <+232>:   movel %d0,%sp@-
   0xc00e015a <+234>:   pea 0xffffffff
   0xc00e015e <+238>:   bsrl 0xc00e0174 <__GI___wait4_time64>
   0xc00e0164 <+244>:   lea %sp@(16),%sp
   0xc00e0168 <+248>:   braw 0xc00e00b2 <__wait3+66>
   0xc00e016c <+252>:   bsrl 0xc012a38c <__stack_chk_fail>
End of assembler dump.
(gdb) 

Note that __wait3(stat_loc, options, NULL) reduces to,

	return __wait4_time64(-1, stat_loc, options, NULL);

So I think the branch at __wait3+34 was taken, and after bsr 
__GI___wait4_time64, the branch at __wait3+248 would have been taken. Then 
the canary located at %fp@(-4) was compared with %a3@. From the hex dump 
above, %fp@(-4) is 0xd000c38e.

As for %a3, we know its value when SIGABRT was caught, and if I'm not 
mistaken, %a3 was not altered by __stack_chk_fail or 
__GI___fortify_fail...

(gdb) frame 7
#7  0xc012a3a0 in __stack_chk_fail () at stack_chk_fail.c:24
24      stack_chk_fail.c: No such file or directory.
(gdb) info frame
Stack level 7, frame at 0xefee106c:
 pc = 0xc012a3a0 in __stack_chk_fail (stack_chk_fail.c:24); 
    saved pc = 0xc00e0172
 called by frame at 0xefee10e0, caller of frame at 0xefee1060
 source language c.
 Arglist at 0xefee105c, args: 
 Locals at 0xefee105c, Previous frame's sp is 0xefee106c
 Saved registers:
  a5 at 0xefee1064, pc at 0xefee1068
(gdb) disass
Dump of assembler code for function __stack_chk_fail:
   0xc012a38c <+0>:     movel %a5,%sp@-
   0xc012a38e <+2>:     lea %pc@(0xc0198000),%a5
   0xc012a396 <+10>:    movel %a5@(10696),%sp@-
   0xc012a39a <+14>:    bsrl 0xc012a3a0 <__GI___fortify_fail>
End of assembler dump.
(gdb) frame 6
#6  0xc012a3c2 in __GI___fortify_fail (
    msg=0xc0182c5e "stack smashing detected") at fortify_fail.c:26
26      fortify_fail.c: No such file or directory.
(gdb) info frame
Stack level 6, frame at 0xefee1060:
 pc = 0xc012a3c2 in __GI___fortify_fail (fortify_fail.c:26); 
    saved pc = 0xc012a3a0
 called by frame at 0xefee106c, caller of frame at 0xefee1044
 source language c.
 Arglist at 0xefee1040, args: msg=0xc0182c5e "stack smashing detected"
 Locals at 0xefee1040, Previous frame's sp is 0xefee1060
 Saved registers:
  d2 at 0xefee1050, d3 at 0xefee1054, a5 at 0xefee1058, pc at 0xefee105c
(gdb) disass
Dump of assembler code for function __GI___fortify_fail:
   0xc012a3a0 <+0>:     moveml %d2-%d3/%a5,%sp@-
   0xc012a3a4 <+4>:     lea %pc@(0xc0198000),%a5
   0xc012a3ac <+12>:    movel %sp@(16),%d3
   0xc012a3b0 <+16>:    movel %a5@(10700),%d2
   0xc012a3b4 <+20>:    movel %d3,%sp@-
   0xc012a3b6 <+22>:    movel %d2,%sp@-
   0xc012a3b8 <+24>:    pea 0x1
   0xc012a3bc <+28>:    bsrl 0xc009b1d8 <__libc_message>
=> 0xc012a3c2 <+34>:    lea %sp@(12),%sp
   0xc012a3c6 <+38>:    movel %d3,%sp@-
   0xc012a3c8 <+40>:    movel %d2,%sp@-
   0xc012a3ca <+42>:    pea 0x1
   0xc012a3ce <+46>:    bsrl 0xc009b1d8 <__libc_message>
   0xc012a3d4 <+52>:    lea %sp@(12),%sp
   0xc012a3d8 <+56>:    bras 0xc012a3b4 <__GI___fortify_fail+20>
End of assembler dump.
(gdb) info reg
d0             0x0                 0
d1             0x16e               366
d2             0xc0182c76          -1072157578
d3             0xc0182c5e          -1072157602
d4             0xefee1122          -269610718
d5             0x1                 1
d6             0xd0021930          -805168848
d7             0x100               256
a0             0xc01a62a0          0xc01a62a0
a1             0xffffffe6          0xffffffe6
a2             0x0                 0x0
a3             0xefee1068          0xefee1068
a4             0xd001e71c          0xd001e71c <pending_sig>
a5             0xc0198000          0xc0198000
fp             0xefee10d8          0xefee10d8
sp             0xefee1044          0xefee1044
ps             0x10                [ X ]
pc             0xc012a3c2          0xc012a3c2 <__GI___fortify_fail+34>
fpcontrol      0x0                 0
fpstatus       0x0                 0
fpiaddr        0x0                 0x0

So %a3 was a pointer into stack frame 6??

(gdb) x/z $a3
0xefee1068:     0xc00e0172

Clearly 0xd000c38e != 0xc00e0172 (that is, %fp@(-4) != %a3@) but did the 
canary value change? It rather looks like the canary pointer is wrong...

Another way to find the value of %a3 during __wait3() execution is to look 
at its initialization: moveal %a5@(108),%a3. And we can see from 'info 
frame' above that __stack_chk_fail() saved %a5 at 0xefee1064.

(gdb) x/4z 0xefee1060
0xefee1060:     0xc0182c5e      0xc0198000      0xc00e0172      0xd001e718
(gdb) x/z *0xefee1064+108
0xc019806c:     Cannot access memory at address 0xc019806c

Anyway, if the analysis is right (hopefully someone can confirm that) 
this looks like a GCC bug.

I'm not sure why it only shows up during (sysvinit) init script execution. 
The canary value is derived from /dev/urandom so I guess the failure is 
intermittent because it is connected to kernel PRNG state during early 
startup.

Re: core dump analysis, was Re: stack smashing detected

[Finn Thain]

On 31 Mar 2023, I wrote,

> ...
> 
> So %a3 was a pointer into stack frame 6??
> 
> (gdb) x/z $a3
> 0xefee1068:     0xc00e0172
> 
> Clearly 0xd000c38e != 0xc00e0172 (that is, %fp@(-4) != %a3@) but did the 
> canary value change? It rather looks like the canary pointer is wrong...
> 

So the question is, why is %a3 wrong? We know it was wrong because %a3 and 
%fp got saved on the stack when __libc_message was called:

(gdb) info frame 5
Stack frame at 0xefee1044:
 pc = 0xc009b328 in __libc_message (../sysdeps/posix/libc_fatal.c:155); 
    saved pc = 0xc012a3c2
 called by frame at 0xefee1060, caller of frame at 0xefee0fd0
 source language c.
 Arglist at 0xefee103c, args: action=<optimized out>, fmt=<optimized out>
 Locals at 0xefee103c, Previous frame's sp is 0xefee1044
 Saved registers:
  d2 at 0xefee100c, d3 at 0xefee1010, d4 at 0xefee1014, d5 at 0xefee1018,
  d6 at 0xefee101c, d7 at 0xefee1020, a2 at 0xefee1024, a3 at 0xefee1028,
  a4 at 0xefee102c, a5 at 0xefee1030, fp at 0xefee103c, pc at 0xefee1040
(gdb) x/z 0xefee1028
0xefee1028:     0xefee1068
(gdb) x/z 0xefee1068
0xefee1068:     0xc00e0172
(gdb) x/z 0xefee103c
0xefee103c:     0xefee10d8
(gdb) x/z 0xefee10d8-4
0xefee10d4:     0x00ac3dbe

That value for %fp@(-4) is the correct canary value (see below) showing 
that there's no buffer overrun in __wait3, and also showing that %a3 was 
wrong when the canary was to be validated.

> Another way to find the value of %a3 during __wait3() execution is to look 
> at its initialization: moveal %a5@(108),%a3. And we can see from 'info 
> frame' above that __stack_chk_fail() saved %a5 at 0xefee1064.
> 
> (gdb) x/4z 0xefee1060
> 0xefee1060:     0xc0182c5e      0xc0198000      0xc00e0172      0xd001e718
> (gdb) x/z *0xefee1064+108
> 0xc019806c:     Cannot access memory at address 0xc019806c
> 

That was a mistake on my part. It should have been,

(gdb) x/z 0xefee1064
0xefee1064:     0xc0198000
(gdb) x/z 0xc0198000+108
0xc019806c:     0xc0023e8c

So that's the value %a3 was given at __wait3+24, and it looks fine:

(gdb) x/z 0xc0023e8c
0xc0023e8c <__stack_chk_guard>: 0x00ac3dbe

... the expected value for %a3 at __wait3+70 is <stack_chk_guard> as this 
is used to validate the canary at %fp-4.

So, in summary, the canary validation failed in this case not because the 
canary got clobbered but because %a3 got clobbered, somewhere between 
__wait3+24 and __wait3+70 (below).

The call to __GI___wait4_time64 causes %a3 to be saved to and restored 
from the stack, so stack corruption seems to be a strong possibility to 
explain the change in %a3.

But if that's what happened, I'd expect __GI___wait4_time64 to report 
stack smashing, not __wait3... And it just begs the question, what then 
caused the corruption? Was it the wait4 syscall? Was it another thread? 
And why is it so rare?

(gdb) disass __wait3
Dump of assembler code for function __wait3:
   0xc00e0070 <+0>:     linkw %fp,#-96
   0xc00e0074 <+4>:     moveml %a2-%a3/%a5,%sp@-
   0xc00e0078 <+8>:     lea %pc@(0xc0198000),%a5
   0xc00e0080 <+16>:    movel %fp@(8),%d0
   0xc00e0084 <+20>:    moveal %fp@(16),%a2
   0xc00e0088 <+24>:    moveal %a5@(108),%a3
   0xc00e008c <+28>:    movel %a3@,%fp@(-4)
   0xc00e0090 <+32>:    tstl %a2
   0xc00e0092 <+34>:    beqw 0xc00e0152 <__wait3+226>
   0xc00e0096 <+38>:    pea %fp@(-92)
   0xc00e009a <+42>:    movel %fp@(12),%sp@-
   0xc00e009e <+46>:    movel %d0,%sp@-
   0xc00e00a0 <+48>:    pea 0xffffffff
   0xc00e00a4 <+52>:    bsrl 0xc00e0174 <__GI___wait4_time64>
   0xc00e00aa <+58>:    lea %sp@(16),%sp
   0xc00e00ae <+62>:    tstl %d0
   0xc00e00b0 <+64>:    bgts 0xc00e00c8 <__wait3+88>
   0xc00e00b2 <+66>:    moveal %fp@(-4),%a0
   0xc00e00b6 <+70>:    movel %a3@,%d1
   0xc00e00b8 <+72>:    cmpl %a0,%d1
   0xc00e00ba <+74>:    bnew 0xc00e016c <__wait3+252>
   0xc00e00be <+78>:    moveml %fp@(-108),%a2-%a3/%a5
   0xc00e00c4 <+84>:    unlk %fp
   0xc00e00c6 <+86>:    rts
   0xc00e00c8 <+88>:    pea 0x44
   0xc00e00cc <+92>:    clrl %sp@-
   0xc00e00ce <+94>:    pea %a2@(4)
   0xc00e00d2 <+98>:    movel %d0,%fp@(-96)
   0xc00e00d6 <+102>:   bsrl 0xc00b8850 <__GI_memset>
   0xc00e00dc <+108>:   movel %fp@(-88),%a2@
   0xc00e00e0 <+112>:   movel %fp@(-80),%a2@(4)
   0xc00e00e6 <+118>:   movel %fp@(-72),%a2@(8)
   0xc00e00ec <+124>:   movel %fp@(-64),%a2@(12)
   0xc00e00f2 <+130>:   movel %fp@(-60),%a2@(16)
   0xc00e00f8 <+136>:   movel %fp@(-56),%a2@(20)
   0xc00e00fe <+142>:   movel %fp@(-52),%a2@(24)
   0xc00e0104 <+148>:   movel %fp@(-48),%a2@(28)
   0xc00e010a <+154>:   movel %fp@(-44),%a2@(32)
   0xc00e0110 <+160>:   movel %fp@(-40),%a2@(36)
   0xc00e0116 <+166>:   movel %fp@(-36),%a2@(40)
   0xc00e011c <+172>:   movel %fp@(-32),%a2@(44)
   0xc00e0122 <+178>:   movel %fp@(-28),%a2@(48)
   0xc00e0128 <+184>:   movel %fp@(-24),%a2@(52)
   0xc00e012e <+190>:   movel %fp@(-20),%a2@(56)
   0xc00e0134 <+196>:   movel %fp@(-16),%a2@(60)
   0xc00e013a <+202>:   movel %fp@(-12),%a2@(64)
   0xc00e0140 <+208>:   movel %fp@(-8),%a2@(68)
   0xc00e0146 <+214>:   lea %sp@(12),%sp
   0xc00e014a <+218>:   movel %fp@(-96),%d0
   0xc00e014e <+222>:   braw 0xc00e00b2 <__wait3+66>
   0xc00e0152 <+226>:   clrl %sp@-
   0xc00e0154 <+228>:   movel %fp@(12),%sp@-
   0xc00e0158 <+232>:   movel %d0,%sp@-
   0xc00e015a <+234>:   pea 0xffffffff
   0xc00e015e <+238>:   bsrl 0xc00e0174 <__GI___wait4_time64>
   0xc00e0164 <+244>:   lea %sp@(16),%sp
   0xc00e0168 <+248>:   braw 0xc00e00b2 <__wait3+66>
   0xc00e016c <+252>:   bsrl 0xc012a38c <__stack_chk_fail>
End of assembler dump.
(gdb) disass __GI___wait4_time64
Dump of assembler code for function __GI___wait4_time64:
   0xc00e0174 <+0>:     lea %sp@(-80),%sp
   0xc00e0178 <+4>:     moveml %d2-%d5/%a2-%a3/%a5,%sp@-
   0xc00e017c <+8>:     lea %pc@(0xc0198000),%a5
   0xc00e0184 <+16>:    movel %sp@(116),%d2
   0xc00e0188 <+20>:    moveal %sp@(124),%a2
   0xc00e018c <+24>:    moveal %a5@(108),%a3
   0xc00e0190 <+28>:    movel %a3@,%sp@(104)
   0xc00e0194 <+32>:    bsrl 0xc0052e2c <__m68k_read_tp@plt>
   0xc00e019a <+38>:    movel %a0@(-29920),%d4
   0xc00e019e <+42>:    bnew 0xc00e026c <__GI___wait4_time64+248>
   0xc00e01a2 <+46>:    tstl %a2
   0xc00e01a4 <+48>:    beqs 0xc00e01aa <__GI___wait4_time64+54>
   0xc00e01a6 <+50>:    moveq #32,%d4
   0xc00e01a8 <+52>:    addl %sp,%d4
   0xc00e01aa <+54>:    movel %sp@(120),%d3
   0xc00e01ae <+58>:    movel %sp@(112),%d1
   0xc00e01b2 <+62>:    moveq #114,%d0
   0xc00e01b4 <+64>:    trap #0
   0xc00e01b6 <+66>:    cmpil #-4096,%d0
   0xc00e01bc <+72>:    bhiw 0xc00e02a6 <__GI___wait4_time64+306>
   0xc00e01c0 <+76>:    tstl %d0
   0xc00e01c2 <+78>:    blew 0xc00e0256 <__GI___wait4_time64+226>
   0xc00e01c6 <+82>:    tstl %a2
   0xc00e01c8 <+84>:    beqw 0xc00e0256 <__GI___wait4_time64+226>
   0xc00e01cc <+88>:    movel %sp@(32),%a2@(4)
   0xc00e01d2 <+94>:    smi %d1
   0xc00e01d4 <+96>:    extbl %d1
   0xc00e01d6 <+98>:    movel %d1,%a2@
   0xc00e01d8 <+100>:   movel %sp@(36),%a2@(12)
   0xc00e01de <+106>:   smi %d1
   0xc00e01e0 <+108>:   extbl %d1
   0xc00e01e2 <+110>:   movel %d1,%a2@(8)
   0xc00e01e6 <+114>:   movel %sp@(40),%a2@(20)
   0xc00e01ec <+120>:   smi %d1
   0xc00e01ee <+122>:   extbl %d1
   0xc00e01f0 <+124>:   movel %d1,%a2@(16)
   0xc00e01f4 <+128>:   movel %sp@(44),%a2@(28)
   0xc00e01fa <+134>:   smi %d1
   0xc00e01fc <+136>:   extbl %d1
   0xc00e01fe <+138>:   movel %d1,%a2@(24)
   0xc00e0202 <+142>:   movel %sp@(48),%a2@(32)
   0xc00e0208 <+148>:   movel %sp@(52),%a2@(36)
   0xc00e020e <+154>:   movel %sp@(56),%a2@(40)
   0xc00e0214 <+160>:   movel %sp@(60),%a2@(44)
   0xc00e021a <+166>:   movel %sp@(64),%a2@(48)
   0xc00e0220 <+172>:   movel %sp@(68),%a2@(52)
   0xc00e0226 <+178>:   movel %sp@(72),%a2@(56)
   0xc00e022c <+184>:   movel %sp@(76),%a2@(60)
   0xc00e0232 <+190>:   movel %sp@(80),%a2@(64)
   0xc00e0238 <+196>:   movel %sp@(84),%a2@(68)
   0xc00e023e <+202>:   movel %sp@(88),%a2@(72)
   0xc00e0244 <+208>:   movel %sp@(92),%a2@(76)
   0xc00e024a <+214>:   movel %sp@(96),%a2@(80)
   0xc00e0250 <+220>:   movel %sp@(100),%a2@(84)
   0xc00e0256 <+226>:   moveal %sp@(104),%a0
   0xc00e025a <+230>:   movel %a3@,%d1
   0xc00e025c <+232>:   cmpl %a0,%d1
   0xc00e025e <+234>:   bnew 0xc00e02f2 <__GI___wait4_time64+382>
   0xc00e0262 <+238>:   moveml %sp@+,%d2-%d5/%a2-%a3/%a5
   0xc00e0266 <+242>:   lea %sp@(80),%sp
   0xc00e026a <+246>:   rts
   0xc00e026c <+248>:   bsrl 0xc00a1b88 <__GI___pthread_enable_asynccancel>
   0xc00e0272 <+254>:   movel %d0,%d5
   0xc00e0274 <+256>:   tstl %a2
   0xc00e0276 <+258>:   beqs 0xc00e02c4 <__GI___wait4_time64+336>
   0xc00e0278 <+260>:   moveq #32,%d4
   0xc00e027a <+262>:   addl %sp,%d4
   0xc00e027c <+264>:   movel %sp@(120),%d3
   0xc00e0280 <+268>:   movel %sp@(112),%d1
   0xc00e0284 <+272>:   moveq #114,%d0
   0xc00e0286 <+274>:   trap #0
   0xc00e0288 <+276>:   cmpil #-4096,%d0
   0xc00e028e <+282>:   bhis 0xc00e02c8 <__GI___wait4_time64+340>
   0xc00e0290 <+284>:   movel %d5,%sp@-
   0xc00e0292 <+286>:   movel %d0,%sp@(32)
   0xc00e0296 <+290>:   bsrl 0xc00a1bea <__GI___pthread_disable_asynccancel>
   0xc00e029c <+296>:   addql #4,%sp
   0xc00e029e <+298>:   movel %sp@(28),%d0
   0xc00e02a2 <+302>:   braw 0xc00e01c0 <__GI___wait4_time64+76>
   0xc00e02a6 <+306>:   movel %d0,%sp@(28)
   0xc00e02aa <+310>:   bsrl 0xc0052e2c <__m68k_read_tp@plt>
   0xc00e02b0 <+316>:   addal %a5@(2cf8),%a0
   0xc00e02b8 <+324>:   movel %sp@(28),%d0
   0xc00e02bc <+328>:   negl %d0
   0xc00e02be <+330>:   movel %d0,%a0@
   0xc00e02c0 <+332>:   moveq #-1,%d0
   0xc00e02c2 <+334>:   bras 0xc00e0256 <__GI___wait4_time64+226>
   0xc00e02c4 <+336>:   clrl %d4
   0xc00e02c6 <+338>:   bras 0xc00e027c <__GI___wait4_time64+264>
   0xc00e02c8 <+340>:   movel %d0,%sp@(28)
   0xc00e02cc <+344>:   bsrl 0xc0052e2c <__m68k_read_tp@plt>
   0xc00e02d2 <+350>:   addal %a5@(2cf8),%a0
   0xc00e02da <+358>:   movel %sp@(28),%d0
   0xc00e02de <+362>:   negl %d0
   0xc00e02e0 <+364>:   movel %d0,%a0@
   0xc00e02e2 <+366>:   movel %d5,%sp@-
   0xc00e02e4 <+368>:   bsrl 0xc00a1bea <__GI___pthread_disable_asynccancel>
   0xc00e02ea <+374>:   addql #4,%sp
   0xc00e02ec <+376>:   moveq #-1,%d0
   0xc00e02ee <+378>:   braw 0xc00e0256 <__GI___wait4_time64+226>
   0xc00e02f2 <+382>:   bsrl 0xc012a38c <__stack_chk_fail>
End of assembler dump.

Re: core dump analysis, was Re: stack smashing detected

[Andreas Schwab]

On Apr 01 2023, Finn Thain wrote:

> So, in summary, the canary validation failed in this case not because the 
> canary got clobbered but because %a3 got clobbered, somewhere between 
> __wait3+24 and __wait3+70 (below).
>
> The call to __GI___wait4_time64 causes %a3 to be saved to and restored 
> from the stack, so stack corruption seems to be a strong possibility to 
> explain the change in %a3.
>
> But if that's what happened, I'd expect __GI___wait4_time64 to report 
> stack smashing, not __wait3...

The stask smashing probably didn't fire in __wait4_time64, because it
hit the saved register area, not the canary (which reside on the
opposite ends of the stack frame).

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: core dump analysis, was Re: stack smashing detected

[Finn Thain]

On Sat, 1 Apr 2023, Andreas Schwab wrote:

> On Apr 01 2023, Finn Thain wrote:
> 
> > So, in summary, the canary validation failed in this case not because 
> > the canary got clobbered but because %a3 got clobbered, somewhere 
> > between __wait3+24 and __wait3+70 (below).
> >
> > The call to __GI___wait4_time64 causes %a3 to be saved to and restored 
> > from the stack, so stack corruption seems to be a strong possibility 
> > to explain the change in %a3.
> >
> > But if that's what happened, I'd expect __GI___wait4_time64 to report 
> > stack smashing, not __wait3...
> 
> The stask smashing probably didn't fire in __wait4_time64, because it 
> hit the saved register area, not the canary (which reside on the 
> opposite ends of the stack frame).
> 

OK.

This is odd:

https://sources.debian.org/src/dash/0.5.12-2/src/jobs.c/?hl=1165#L1165

  1176          do {
  1177                  gotsigchld = 0;
  1178                  do
  1179                          err = wait3(status, flags, NULL);
  1180                  while (err < 0 && errno == EINTR);
  1181
  1182                  if (err || (err = -!block))
  1183                          break;
  1184                  
  1185                  sigblockall(&oldmask);
  1186
  1187                  while (!gotsigchld && !pending_sig)
  1188                          sigsuspend(&oldmask);
  1189
  1190                  sigclearmask();
  1191          } while (gotsigchld);
  1192
  1193  return err;

Execution of dash under gdb doesn't seem to agree with the source code 
above.

If wait3() returns the child pid then the break should execute. And it 
does return the pid (4107) but the while loop was not terminated. Hence 
wait3() was called again and the same breakpoint was hit again. Also, the 
while loop should have ended after the first iteration because gotsigchild 
should have been set by the signal handler which executed before wait3() 
even returned...

...
(gdb) c
Continuing.
# 
# 
# x=$(:)
[Detaching after fork from child process 4107]

Program received signal SIGCHLD, Child status changed.
0xc00e81b6 in __GI___wait4_time64 (pid=-1, stat_loc=0xeffff87a, options=2, 
    usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:35
35      ../sysdeps/unix/sysv/linux/wait4.c: No such file or directory.
(gdb) c
Continuing.

Breakpoint 3, waitproc (status=0xeffff86a, block=1) at jobs.c:1180
1180    jobs.c: No such file or directory.
(gdb) info locals
oldmask = {__val = {1101825, 3844132865, 2072969216, 192511, 4190371840, 
    4509697, 3836788738, 1049415681, 3837317121, 3094671359, 4184080384, 
    536870943, 717475840, 3485913089, 3836792833, 2072969216, 184321, 
    3844141055, 4190425089, 4127248385, 3094659084, 597610497, 4137734145, 
    3844079616, 131072, 269156352, 184320, 3878473729, 3844132865, 3094663168, 
    3549089793, 3844132865}}
flags = 2
err = 4107
oldmask = <optimized out>
flags = <optimized out>
err = <optimized out>
(gdb) print errno
$6 = 2
(gdb) c
Continuing.

Breakpoint 3, waitproc (status=0xeffff86a, block=0) at jobs.c:1180
1180    in jobs.c
(gdb) info locals
oldmask = {__val = {1101825, 3844132865, 2072969216, 192511, 4190371840, 
    4509697, 3836788738, 1049415681, 3837317121, 3094671359, 4184080384, 
    536870943, 717475840, 3485913089, 3836792833, 2072969216, 184321, 
    3844141055, 4190425089, 4127248385, 3094659084, 597610497, 4137734145, 
    3844079616, 131072, 269156352, 184320, 3878473729, 3844132865, 3094663168, 
    3549089793, 3844132865}}
flags = 3
err = -1
oldmask = <optimized out>
flags = <optimized out>
err = <optimized out>
(gdb) print errno
$7 = 10
(gdb) 

Re: core dump analysis, was Re: stack smashing detected

[Michael Schmitz]

Hi Finn,

On 2/04/23 22:46, Finn Thain wrote:
> On Sat, 1 Apr 2023, Andreas Schwab wrote:
>
>> On Apr 01 2023, Finn Thain wrote:
>>
>>> So, in summary, the canary validation failed in this case not because
>>> the canary got clobbered but because %a3 got clobbered, somewhere
>>> between __wait3+24 and __wait3+70 (below).
>>>
>>> The call to __GI___wait4_time64 causes %a3 to be saved to and restored
>>> from the stack, so stack corruption seems to be a strong possibility
>>> to explain the change in %a3.
>>>
>>> But if that's what happened, I'd expect __GI___wait4_time64 to report
>>> stack smashing, not __wait3...
>> The stask smashing probably didn't fire in __wait4_time64, because it
>> hit the saved register area, not the canary (which reside on the
>> opposite ends of the stack frame).
>>
> OK.
>
> This is odd:
>
> https://sources.debian.org/src/dash/0.5.12-2/src/jobs.c/?hl=1165#L1165
>
>    1176          do {
>    1177                  gotsigchld = 0;
>    1178                  do
>    1179                          err = wait3(status, flags, NULL);
>    1180                  while (err < 0 && errno == EINTR);
>    1181
>    1182                  if (err || (err = -!block))
>    1183                          break;
>    1184
>    1185                  sigblockall(&oldmask);
>    1186
>    1187                  while (!gotsigchld && !pending_sig)
>    1188                          sigsuspend(&oldmask);
>    1189
>    1190                  sigclearmask();
>    1191          } while (gotsigchld);
>    1192
>    1193  return err;
>
> Execution of dash under gdb doesn't seem to agree with the source code
> above.
>
> If wait3() returns the child pid then the break should execute. And it
> does return the pid (4107) but the while loop was not terminated. Hence
> wait3() was called again and the same breakpoint was hit again. Also, the

I wonder whether line 1182 got miscompiled by gcc. As err == 4107 it's > 
0 and the break clearly ought to have been taken, and the second 
condition (which changes err) does not need to be examined.  Do the same 
ordering constraints apply to '||' as to '&&' ?

What does the disassembly of this section look like?

> while loop should have ended after the first iteration because gotsigchild
> should have been set by the signal handler which executed before wait3()
> even returned...

Setting gotsigchild > 0 would cause the while loop to continue, no?

The second wait3 call then returns an error (errno ECHILD), leaves 
gotsigchild clear and exits the loop (returning from waitproc with 
error, which it shouldn't have).

>
> ...
> (gdb) c
> Continuing.
> #
> #
> # x=$(:)
> [Detaching after fork from child process 4107]
>
> Program received signal SIGCHLD, Child status changed.
> 0xc00e81b6 in __GI___wait4_time64 (pid=-1, stat_loc=0xeffff87a, options=2,
>      usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:35
> 35      ../sysdeps/unix/sysv/linux/wait4.c: No such file or directory.
> (gdb) c
> Continuing.
>
> Breakpoint 3, waitproc (status=0xeffff86a, block=1) at jobs.c:1180
> 1180    jobs.c: No such file or directory.
> (gdb) info locals
> oldmask = {__val = {1101825, 3844132865, 2072969216, 192511, 4190371840,
>      4509697, 3836788738, 1049415681, 3837317121, 3094671359, 4184080384,
>      536870943, 717475840, 3485913089, 3836792833, 2072969216, 184321,
>      3844141055, 4190425089, 4127248385, 3094659084, 597610497, 4137734145,
>      3844079616, 131072, 269156352, 184320, 3878473729, 3844132865, 3094663168,
>      3549089793, 3844132865}}
> flags = 2
> err = 4107
> oldmask = <optimized out>
> flags = <optimized out>
> err = <optimized out>
> (gdb) print errno
> $6 = 2

Is that ENOENT the returned status from the child process?

Cheers,

     Michael

> (gdb) c
> Continuing.
>
> Breakpoint 3, waitproc (status=0xeffff86a, block=0) at jobs.c:1180
> 1180    in jobs.c
> (gdb) info locals
> oldmask = {__val = {1101825, 3844132865, 2072969216, 192511, 4190371840,
>      4509697, 3836788738, 1049415681, 3837317121, 3094671359, 4184080384,
>      536870943, 717475840, 3485913089, 3836792833, 2072969216, 184321,
>      3844141055, 4190425089, 4127248385, 3094659084, 597610497, 4137734145,
>      3844079616, 131072, 269156352, 184320, 3878473729, 3844132865, 3094663168,
>      3549089793, 3844132865}}
> flags = 3
> err = -1
> oldmask = <optimized out>
> flags = <optimized out>
> err = <optimized out>
> (gdb) print errno
> $7 = 10
> (gdb)

Re: core dump analysis, was Re: stack smashing detected

[Finn Thain]

On Mon, 3 Apr 2023, Michael Schmitz wrote:

> On 2/04/23 22:46, Finn Thain wrote:
> 
> > This is odd:
> >
> > https://sources.debian.org/src/dash/0.5.12-2/src/jobs.c/?hl=1165#L1165
> >
> >    1176          do {
> >    1177                  gotsigchld = 0;
> >    1178                  do
> >    1179                          err = wait3(status, flags, NULL);
> >    1180                  while (err < 0 && errno == EINTR);
> >    1181
> >    1182                  if (err || (err = -!block))
> >    1183                          break;
> >    1184
> >    1185                  sigblockall(&oldmask);
> >    1186
> >    1187                  while (!gotsigchld && !pending_sig)
> >    1188                          sigsuspend(&oldmask);
> >    1189
> >    1190                  sigclearmask();
> >    1191          } while (gotsigchld);
> >    1192
> >    1193  return err;
> >
> > Execution of dash under gdb doesn't seem to agree with the source code
> > above.
> >
> > If wait3() returns the child pid then the break should execute. And it
> > does return the pid (4107) but the while loop was not terminated. Hence
> > wait3() was called again and the same breakpoint was hit again. Also, the
> 
> I wonder whether line 1182 got miscompiled by gcc. As err == 4107 it's > 
> 0 and the break clearly ought to have been taken, and the second 
> condition (which changes err) does not need to be examined.  Do the same 
> ordering constraints apply to '||' as to '&&' ?
> 

AFAICT, the source code is valid. This article has some information: 
https://stackoverflow.com/questions/628526/is-short-circuiting-logical-operators-mandated-and-evaluation-order

It looks like I messed up. waitproc() appears to have been invoked
twice, which is why wait3 was invoked twice...

GNU gdb (Debian 13.1-2) 13.1
...
(gdb) set osabi GNU/Linux
(gdb) file /bin/dash
Reading symbols from /bin/dash...
Reading symbols from /usr/lib/debug/.build-id/aa/4160f84f3eeee809c554cb9f3e1ef0686b8dcc.debug...
(gdb) b waitproc
Breakpoint 1 at 0xc346: file jobs.c, line 1168.
(gdb) b jobs.c:1180
Breakpoint 2 at 0xc390: file jobs.c, line 1180.
(gdb) run
Starting program: /usr/bin/dash 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/m68k-linux-gnu/libthread_db.so.1".
# x=$(:)
[Detaching after fork from child process 570]

Breakpoint 1, waitproc (status=0xeffff86a, block=1) at jobs.c:1168
1168    jobs.c: No such file or directory.
(gdb) c
Continuing.

Breakpoint 2, waitproc (status=0xeffff86a, block=1) at jobs.c:1180
1180    in jobs.c
(gdb) info locals
oldmask = {__val = {1997799424, 49154, 396623872, 184321, 3223896090, 53249, 
    3836788738, 1049411610, 867225601, 3094609920, 0, 1048580, 2857693183, 
    4184129547, 3435708442, 863764480, 184321, 3844141055, 4190425089, 
    4127248385, 3094659084, 597610497, 4135112705, 3844079616, 131072, 
    37355520, 184320, 3878473729, 3844132865, 3094663168, 3549089793, 
    3844132865}}
flags = 2
err = 570
oldmask = <optimized out>
flags = <optimized out>
err = <optimized out>
(gdb) c
Continuing.

Breakpoint 1, waitproc (status=0xeffff86a, block=0) at jobs.c:1168
1168    in jobs.c
(gdb) c
Continuing.

Breakpoint 2, waitproc (status=0xeffff86a, block=0) at jobs.c:1180
1180    in jobs.c
(gdb) info locals
oldmask = {__val = {1997799424, 49154, 396623872, 184321, 3223896090, 53249, 
    3836788738, 1049411610, 867225601, 3094609920, 0, 1048580, 2857693183, 
    4184129547, 3435708442, 863764480, 184321, 3844141055, 4190425089, 
    4127248385, 3094659084, 597610497, 4135112705, 3844079616, 131072, 
    37355520, 184320, 3878473729, 3844132865, 3094663168, 3549089793, 
    3844132865}}
flags = 3
err = -1
oldmask = <optimized out>
flags = <optimized out>
err = <optimized out>
(gdb) c
Continuing.
# 

> What does the disassembly of this section look like?
> 
> > while loop should have ended after the first iteration because gotsigchild
> > should have been set by the signal handler which executed before wait3()
> > even returned...
> 
> Setting gotsigchild > 0 would cause the while loop to continue, no?
> 

Right. Sorry for the noise.

Re: core dump analysis, was Re: stack smashing detected

[Michael Schmitz]

Hi Finn,

On 4/04/23 12:13, Finn Thain wrote:
> It looks like I messed up. waitproc() appears to have been invoked
> twice, which is why wait3 was invoked twice...
>
> GNU gdb (Debian 13.1-2) 13.1
> ...
> (gdb) set osabi GNU/Linux
> (gdb) file /bin/dash
> Reading symbols from /bin/dash...
> Reading symbols from /usr/lib/debug/.build-id/aa/4160f84f3eeee809c554cb9f3e1ef0686b8dcc.debug...
> (gdb) b waitproc
> Breakpoint 1 at 0xc346: file jobs.c, line 1168.
> (gdb) b jobs.c:1180
> Breakpoint 2 at 0xc390: file jobs.c, line 1180.
> (gdb) run
> Starting program: /usr/bin/dash
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/m68k-linux-gnu/libthread_db.so.1".
> # x=$(:)
> [Detaching after fork from child process 570]
>
> Breakpoint 1, waitproc (status=0xeffff86a, block=1) at jobs.c:1168
> 1168    jobs.c: No such file or directory.
> (gdb) c
> Continuing.
>
> Breakpoint 2, waitproc (status=0xeffff86a, block=1) at jobs.c:1180
> 1180    in jobs.c
> (gdb) info locals
> oldmask = {__val = {1997799424, 49154, 396623872, 184321, 3223896090, 53249,
>      3836788738, 1049411610, 867225601, 3094609920, 0, 1048580, 2857693183,
>      4184129547, 3435708442, 863764480, 184321, 3844141055, 4190425089,
>      4127248385, 3094659084, 597610497, 4135112705, 3844079616, 131072,
>      37355520, 184320, 3878473729, 3844132865, 3094663168, 3549089793,
>      3844132865}}
> flags = 2
> err = 570
> oldmask = <optimized out>
> flags = <optimized out>
> err = <optimized out>
> (gdb) c
> Continuing.
>
> Breakpoint 1, waitproc (status=0xeffff86a, block=0) at jobs.c:1168
> 1168    in jobs.c
> (gdb) c
> Continuing.
>
> Breakpoint 2, waitproc (status=0xeffff86a, block=0) at jobs.c:1180
> 1180    in jobs.c
> (gdb) info locals
> oldmask = {__val = {1997799424, 49154, 396623872, 184321, 3223896090, 53249,
>      3836788738, 1049411610, 867225601, 3094609920, 0, 1048580, 2857693183,
>      4184129547, 3435708442, 863764480, 184321, 3844141055, 4190425089,
>      4127248385, 3094659084, 597610497, 4135112705, 3844079616, 131072,
>      37355520, 184320, 3878473729, 3844132865, 3094663168, 3549089793,
>      3844132865}}
> flags = 3
> err = -1
> oldmask = <optimized out>
> flags = <optimized out>
> err = <optimized out>
> (gdb) c
> Continuing.
> #
>
That means we may well see both signals delivered at the same time if 
the parent shell wasn't scheduled to run until the second subshell 
terminated (answering the question I was about to ask on your other 
mail, the one about the crashy script with multiple subshells).

Now does waitproc() handle that case correctly? The first signal 
delivered results in err == child PID so the break is taken, causing 
exit from waitproc(). Does waitproc() get called repeatedly until an 
error is returned?

Cheers,

     Michael

Re: core dump analysis, was Re: stack smashing detected

[Finn Thain]

On Wed, 5 Apr 2023, Michael Schmitz wrote:

> On 4/04/23 12:13, Finn Thain wrote:
> > It looks like I messed up. waitproc() appears to have been invoked
> > twice, which is why wait3 was invoked twice...
> >
> > GNU gdb (Debian 13.1-2) 13.1
> > ...
> > (gdb) set osabi GNU/Linux
> > (gdb) file /bin/dash
> > Reading symbols from /bin/dash...
> > Reading symbols from
> > /usr/lib/debug/.build-id/aa/4160f84f3eeee809c554cb9f3e1ef0686b8dcc.debug...
> > (gdb) b waitproc
> > Breakpoint 1 at 0xc346: file jobs.c, line 1168.
> > (gdb) b jobs.c:1180
> > Breakpoint 2 at 0xc390: file jobs.c, line 1180.
> > (gdb) run
> > Starting program: /usr/bin/dash
> > [Thread debugging using libthread_db enabled]
> > Using host libthread_db library "/lib/m68k-linux-gnu/libthread_db.so.1".
> > # x=$(:)
> > [Detaching after fork from child process 570]
> >
> > Breakpoint 1, waitproc (status=0xeffff86a, block=1) at jobs.c:1168
> > 1168    jobs.c: No such file or directory.
> > (gdb) c
> > Continuing.
> >
> > Breakpoint 2, waitproc (status=0xeffff86a, block=1) at jobs.c:1180
> > 1180    in jobs.c
> > (gdb) info locals
> > oldmask = {__val = {1997799424, 49154, 396623872, 184321, 3223896090, 53249,
> >      3836788738, 1049411610, 867225601, 3094609920, 0, 1048580, 2857693183,
> >      4184129547, 3435708442, 863764480, 184321, 3844141055, 4190425089,
> >      4127248385, 3094659084, 597610497, 4135112705, 3844079616, 131072,
> >      37355520, 184320, 3878473729, 3844132865, 3094663168, 3549089793,
> >      3844132865}}
> > flags = 2
> > err = 570
> > oldmask = <optimized out>
> > flags = <optimized out>
> > err = <optimized out>
> > (gdb) c
> > Continuing.
> >
> > Breakpoint 1, waitproc (status=0xeffff86a, block=0) at jobs.c:1168
> > 1168    in jobs.c
> > (gdb) c
> > Continuing.
> >
> > Breakpoint 2, waitproc (status=0xeffff86a, block=0) at jobs.c:1180
> > 1180    in jobs.c
> > (gdb) info locals
> > oldmask = {__val = {1997799424, 49154, 396623872, 184321, 3223896090, 53249,
> >      3836788738, 1049411610, 867225601, 3094609920, 0, 1048580, 2857693183,
> >      4184129547, 3435708442, 863764480, 184321, 3844141055, 4190425089,
> >      4127248385, 3094659084, 597610497, 4135112705, 3844079616, 131072,
> >      37355520, 184320, 3878473729, 3844132865, 3094663168, 3549089793,
> >      3844132865}}
> > flags = 3
> > err = -1
> > oldmask = <optimized out>
> > flags = <optimized out>
> > err = <optimized out>
> > (gdb) c
> > Continuing.
> > #
> >
> That means we may well see both signals delivered at the same time if the
> parent shell wasn't scheduled to run until the second subshell terminated
> (answering the question I was about to ask on your other mail, the one about
> the crashy script with multiple subshells).
> 

How is that possible? If the parent does not get scheduled, the second 
fork will not take place.

> Now does waitproc() handle that case correctly? The first signal 
> delivered results in err == child PID so the break is taken, causing 
> exit from waitproc().

I don't follow. Can you rephrase that perhaps?

For a single subshell, the SIGCHLD signal can be delivered before wait4 is 
called or after it returns. For example, $(sleep 5) seems to produce the 
latter whereas $(:) tends to produce the former.

> Does waitproc() get called repeatedly until an error is returned?
> 

It's complicated...
https://sources.debian.org/src/dash/0.5.12-2/src/jobs.c/?hl=1122#L1122

I don't care that much what dash does as long as it isn't corrupting it's 
own stack, which is a real possibility, and one which gdb's data watch 
point would normally resolve. And yet I have no way to tackle that.

I've been running gdb under QEMU, where the failure is not reproducible. 
Running dash under gdb on real hardware is doable (RAM permitting). But 
the failure is intermittent even then -- it only happens during execution 
of certain init scripts, and I can't reproduce it by manually running 
those scripts.

(Even if I could reproduce the failure under gdb, instrumenting execution 
in gdb can alter timing in undesirable ways...)

So, again, the best avenue I can think of for such experiments to modify 
the kernel to either keep track of the times of the wait4 syscalls and 
signal delivery and/or push the timing one way or the other e.g. by 
delaying signal delivery, altering scheduler behaviour, etc. But I don't 
have code for that. I did try adding random delays around kernel_wait4() 
but it didn't have any effect...

Re: core dump analysis, was Re: stack smashing detected

[Michael Schmitz]

Hi Finn,

Am 05.04.2023 um 14:00 schrieb Finn Thain:
> On Wed, 5 Apr 2023, Michael Schmitz wrote:
>
>> On 4/04/23 12:13, Finn Thain wrote:
>>> It looks like I messed up. waitproc() appears to have been invoked
>>> twice, which is why wait3 was invoked twice...
>>>
>>> GNU gdb (Debian 13.1-2) 13.1
>>> ...
>>> (gdb) set osabi GNU/Linux
>>> (gdb) file /bin/dash
>>> Reading symbols from /bin/dash...
>>> Reading symbols from
>>> /usr/lib/debug/.build-id/aa/4160f84f3eeee809c554cb9f3e1ef0686b8dcc.debug...
>>> (gdb) b waitproc
>>> Breakpoint 1 at 0xc346: file jobs.c, line 1168.
>>> (gdb) b jobs.c:1180
>>> Breakpoint 2 at 0xc390: file jobs.c, line 1180.
>>> (gdb) run
>>> Starting program: /usr/bin/dash
>>> [Thread debugging using libthread_db enabled]
>>> Using host libthread_db library "/lib/m68k-linux-gnu/libthread_db.so.1".
>>> # x=$(:)
>>> [Detaching after fork from child process 570]
>>>
>>> Breakpoint 1, waitproc (status=0xeffff86a, block=1) at jobs.c:1168
>>> 1168    jobs.c: No such file or directory.
>>> (gdb) c
>>> Continuing.
>>>
>>> Breakpoint 2, waitproc (status=0xeffff86a, block=1) at jobs.c:1180
>>> 1180    in jobs.c
>>> (gdb) info locals
>>> oldmask = {__val = {1997799424, 49154, 396623872, 184321, 3223896090, 53249,
>>>      3836788738, 1049411610, 867225601, 3094609920, 0, 1048580, 2857693183,
>>>      4184129547, 3435708442, 863764480, 184321, 3844141055, 4190425089,
>>>      4127248385, 3094659084, 597610497, 4135112705, 3844079616, 131072,
>>>      37355520, 184320, 3878473729, 3844132865, 3094663168, 3549089793,
>>>      3844132865}}
>>> flags = 2
>>> err = 570
>>> oldmask = <optimized out>
>>> flags = <optimized out>
>>> err = <optimized out>
>>> (gdb) c
>>> Continuing.
>>>
>>> Breakpoint 1, waitproc (status=0xeffff86a, block=0) at jobs.c:1168
>>> 1168    in jobs.c
>>> (gdb) c
>>> Continuing.
>>>
>>> Breakpoint 2, waitproc (status=0xeffff86a, block=0) at jobs.c:1180
>>> 1180    in jobs.c
>>> (gdb) info locals
>>> oldmask = {__val = {1997799424, 49154, 396623872, 184321, 3223896090, 53249,
>>>      3836788738, 1049411610, 867225601, 3094609920, 0, 1048580, 2857693183,
>>>      4184129547, 3435708442, 863764480, 184321, 3844141055, 4190425089,
>>>      4127248385, 3094659084, 597610497, 4135112705, 3844079616, 131072,
>>>      37355520, 184320, 3878473729, 3844132865, 3094663168, 3549089793,
>>>      3844132865}}
>>> flags = 3
>>> err = -1
>>> oldmask = <optimized out>
>>> flags = <optimized out>
>>> err = <optimized out>
>>> (gdb) c
>>> Continuing.
>>> #
>>>
>> That means we may well see both signals delivered at the same time if the
>> parent shell wasn't scheduled to run until the second subshell terminated
>> (answering the question I was about to ask on your other mail, the one about
>> the crashy script with multiple subshells).
>>
>
> How is that possible? If the parent does not get scheduled, the second
> fork will not take place.

I assumed subshells could run asynchronously, and the parent shell 
continue until it hits a statement that needs the result of one of the 
subshells.

What is the point of subshells, if not to allow this?

>
>> Now does waitproc() handle that case correctly? The first signal
>> delivered results in err == child PID so the break is taken, causing
>> exit from waitproc().
>
> I don't follow. Can you rephrase that perhaps?

The first subshell exiting causes the wait3() to return, and the return 
code is > 0 (child PID). The break statement executes and waitproc() 
returns that PID.

I was wondering how multiple child processes exiting would be handled, 
but I now see that repeatedly calling waitproc() until all outstanding 
jobs have completed is the only way. dash implements this method, so it 
must expect multiple jobs / subshells to run concurrently.

> For a single subshell, the SIGCHLD signal can be delivered before wait4 is
> called or after it returns. For example, $(sleep 5) seems to produce the
> latter whereas $(:) tends to produce the former.

I don't think wait4 can return with success before SIGCHLD has been 
delivered. Delivery of SIGCHLD is what makes the wait syscalls unblock 
as far as I understand.

>> Does waitproc() get called repeatedly until an error is returned?
>>
>
> It's complicated...

Yep, but I'm now satisfied that is the only way ... sorry for the noise.

> https://sources.debian.org/src/dash/0.5.12-2/src/jobs.c/?hl=1122#L1122
>
> I don't care that much what dash does as long as it isn't corrupting it's
> own stack, which is a real possibility, and one which gdb's data watch
> point would normally resolve. And yet I have no way to tackle that.
>
> I've been running gdb under QEMU, where the failure is not reproducible.
> Running dash under gdb on real hardware is doable (RAM permitting). But
> the failure is intermittent even then -- it only happens during execution
> of certain init scripts, and I can't reproduce it by manually running
> those scripts.
>
> (Even if I could reproduce the failure under gdb, instrumenting execution
> in gdb can alter timing in undesirable ways...)
>
> So, again, the best avenue I can think of for such experiments to modify
> the kernel to either keep track of the times of the wait4 syscalls and

The easiest way to do that is to log all wait and signal syscalls, as 
well as process exit. That might alter timing if these log messages go 
to the serial console though. Is that what you have in mind?

> signal delivery and/or push the timing one way or the other e.g. by
> delaying signal delivery, altering scheduler behaviour, etc. But I don't
> have code for that. I did try adding random delays around kernel_wait4()
> but it didn't have any effect...
>

I wonder whether it's possible to delay process exit (and parent process 
signaling) by placing the exit syscall on a timer workqueue. But the 
same effect could be had by inserting a sleep before subshell exit ...

And causing a half-dead task to schedule in order to delay signaling 
doesn't seem safe to me ...

Cheers,

	Michael

dash behaviour, was Re: core dump analysis

[Finn Thain]

On Fri, 7 Apr 2023, Michael Schmitz wrote:

> Am 05.04.2023 um 14:00 schrieb Finn Thain:
> > On Wed, 5 Apr 2023, Michael Schmitz wrote:
> >
> >> That means we may well see both signals delivered at the same time if 
> >> the parent shell wasn't scheduled to run until the second subshell 
> >> terminated (answering the question I was about to ask on your other 
> >> mail, the one about the crashy script with multiple subshells).
> >>
> >
> > How is that possible? If the parent does not get scheduled, the second 
> > fork will not take place.
> 
> I assumed subshells could run asynchronously, and the parent shell 
> continue until it hits a statement that needs the result of one of the 
> subshells.
> 

That would be nice but I don't think dash is so sophisticated as to keep 
track of data dependencies between the various expressions and commands in 
a shell script.

> What is the point of subshells, if not to allow this?
> 

$ x=$(exit 123)
$ echo $?
123

$ set -e
$ x=$(false)

Anyway, my gut says that we're barking up the wrong tree. My recent tests 
show that the failure is not uniformly random. Either the script fails 
often or it fails not at all. It's as if there was some unknown variable 
that caused dash to corrupt its own stack.

> > Running dash under gdb on real hardware is doable (RAM permitting). 
> > But the failure is intermittent even then -- it only happens during 
> > execution of certain init scripts, and I can't reproduce it by 
> > manually running those scripts.
> >
> > (Even if I could reproduce the failure under gdb, instrumenting 
> > execution in gdb can alter timing in undesirable ways...)
> >
> > So, again, the best avenue I can think of for such experiments to 
> > modify the kernel to either keep track of the times of the wait4 
> > syscalls and
> 
> The easiest way to do that is to log all wait and signal syscalls, as 
> well as process exit. That might alter timing if these log messages go 
> to the serial console though. Is that what you have in mind?
> 

What I had in mind was collecting measurements in such way that would not 
impact timing, perhaps by storing them somewhere they could be retrieved 
from the process core dump.

But that's probably not realistic and it's probably pointless anyway -- I 
don't expect to find an old bug in common code like kernel/exit.c, or in a 
hot path like those in arch/m68k/kernel/entry.S.

More likely is that some kind of bug in dash causes it to corrupt its own 
stack when conditions are just right. I just need to figure out how to 
recreate those conditions. :-/

When dash is feeling crashy, you can get results like this:

root@debian:~# sh /etc/init.d/mountdevsubfs.sh 
*** stack smashing detected ***: terminated
Aborted (core dumped)
Warning: mountdevsubfs should be called with the 'start' argument.
root@debian:~# sh /etc/init.d/mountdevsubfs.sh 
*** stack smashing detected ***: terminated
Aborted (core dumped)
Warning: mountdevsubfs should be called with the 'start' argument.
root@debian:~# sh /etc/init.d/mountdevsubfs.sh 
*** stack smashing detected ***: terminated
Aborted (core dumped)
Warning: mountdevsubfs should be called with the 'start' argument.
root@debian:~# sh /etc/init.d/mountdevsubfs.sh 
*** stack smashing detected ***: terminated
Aborted (core dumped)
Warning: mountdevsubfs should be called with the 'start' argument.
root@debian:~# 

But when it's not feeling crashy, you can't:

root@debian:~# sh /etc/init.d/mountdevsubfs.sh 
Warning: mountdevsubfs should be called with the 'start' argument.
root@debian:~# sh /etc/init.d/mountdevsubfs.sh 
Warning: mountdevsubfs should be called with the 'start' argument.
root@debian:~# sh /etc/init.d/mountdevsubfs.sh 
Warning: mountdevsubfs should be called with the 'start' argument.
root@debian:~# sh /etc/init.d/mountdevsubfs.sh 
Warning: mountdevsubfs should be called with the 'start' argument.

The only way I have found to alter dash's inclination to crash is to 
reboot. (I said previously I was unable to reproduce this in a single user 
mode shell but it turned out to be more subtle.)

> > signal delivery and/or push the timing one way or the other e.g. by 
> > delaying signal delivery, altering scheduler behaviour, etc. But I 
> > don't have code for that. I did try adding random delays around 
> > kernel_wait4() but it didn't have any effect...
> >
> 
> I wonder whether it's possible to delay process exit (and parent process 
> signaling) by placing the exit syscall on a timer workqueue. But the 
> same effect could be had by inserting a sleep before subshell exit ...
> 
> And causing a half-dead task to schedule in order to delay signaling 
> doesn't seem safe to me ...
> 

Re: dash behaviour, was Re: core dump analysis

[Geert Uytterhoeven]

Hi Finn,

On Fri, Apr 7, 2023 at 6:08 AM Finn Thain <fthain@linux-m68k.org> wrote:
> When dash is feeling crashy, you can get results like this:
>
> root@debian:~# sh /etc/init.d/mountdevsubfs.sh
> *** stack smashing detected ***: terminated
> Aborted (core dumped)
> Warning: mountdevsubfs should be called with the 'start' argument.
> root@debian:~# sh /etc/init.d/mountdevsubfs.sh
> *** stack smashing detected ***: terminated
> Aborted (core dumped)
> Warning: mountdevsubfs should be called with the 'start' argument.
> root@debian:~# sh /etc/init.d/mountdevsubfs.sh
> *** stack smashing detected ***: terminated
> Aborted (core dumped)
> Warning: mountdevsubfs should be called with the 'start' argument.
> root@debian:~# sh /etc/init.d/mountdevsubfs.sh
> *** stack smashing detected ***: terminated
> Aborted (core dumped)
> Warning: mountdevsubfs should be called with the 'start' argument.
> root@debian:~#
>
> But when it's not feeling crashy, you can't:
>
> root@debian:~# sh /etc/init.d/mountdevsubfs.sh
> Warning: mountdevsubfs should be called with the 'start' argument.
> root@debian:~# sh /etc/init.d/mountdevsubfs.sh
> Warning: mountdevsubfs should be called with the 'start' argument.
> root@debian:~# sh /etc/init.d/mountdevsubfs.sh
> Warning: mountdevsubfs should be called with the 'start' argument.
> root@debian:~# sh /etc/init.d/mountdevsubfs.sh
> Warning: mountdevsubfs should be called with the 'start' argument.
>
> The only way I have found to alter dash's inclination to crash is to
> reboot. (I said previously I was unable to reproduce this in a single user
> mode shell but it turned out to be more subtle.)

That sounds like memory corruption somewhere else, e.g.
in the buffer cache...

Can you reproduce with CONFIG_DEBUG_SLAB=y?

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: dash behaviour, was Re: core dump analysis

[Finn Thain]

On Fri, 7 Apr 2023, Geert Uytterhoeven wrote:

> >
> > The only way I have found to alter dash's inclination to crash is to 
> > reboot. (I said previously I was unable to reproduce this in a single 
> > user mode shell but it turned out to be more subtle.)
> 
> That sounds like memory corruption somewhere else, e.g. in the buffer 
> cache...
> 

If so, once the corruption showed up, you would expect the same crash next 
time...

root@debian:~# sh /etc/init.d/mountdevsubfs.sh start
*** stack smashing detected ***: terminated
Aborted (core dumped)
root@debian:~# sh /etc/init.d/mountdevsubfs.sh start
*** stack smashing detected ***: terminated
Aborted (core dumped)
*** stack smashing detected ***: terminated
Aborted (core dumped)
root@debian:~# sh /etc/init.d/mountdevsubfs.sh start
*** stack smashing detected ***: terminated
Aborted (core dumped)
root@debian:~# echo 3 > /proc/sys/vm/drop_caches 
[  937.250000] bash (717): drop_caches: 3
root@debian:~# sh /etc/init.d/mountdevsubfs.sh start
root@debian:~# sh /etc/init.d/mountdevsubfs.sh start
*** stack smashing detected ***: terminated
Aborted (core dumped)
root@debian:~# sh /etc/init.d/mountdevsubfs.sh start
*** stack smashing detected ***: terminated
Aborted (core dumped)
*** stack smashing detected ***: terminated
Aborted (core dumped)

I'd say it's probably not buffer cache corruption causing this because we 
can see two subshells fail, then just one.

For that build I enabled SLUB_DEBUG but forgot to enable SLUB_DEBUG_ON --

CONFIG_DEBUG_PAGEALLOC=y
CONFIG_DEBUG_PAGEALLOC_ENABLE_DEFAULT=y
CONFIG_SLUB_DEBUG=y
# CONFIG_SLUB_DEBUG_ON is not set
CONFIG_PAGE_POISONING=y
CONFIG_SCHED_STACK_END_CHECK=y

Below you can see the console log for a build with CONFIG_SLUB_DEBUG_ON=y.

BTW, these builds have a hacked RNG which produces only zeros, to try to 
get more consistent behaviour. It seems to have made no difference. None 
of the kernel changes I've made seem to make much difference, besides 
perturbing timing slightly. I tend to think this is a dash bug that's 
confined to slow machines only because it's timing sensitive.

ABCFGHIJK
[    0.000000] Linux version 6.3.0-rc5-mac-00001-gfaf1ca712d79 (fthain@nippy) (m68k-unknown-linux-musl-gcc (Gentoo 12.2.1_p20221224 p7) 12.2.1 20221224, GNU ld (Gentoo 2.39 p5) 2.39.0) #4 Sat Apr  8 14:25:16 AEST 2023
[    0.000000] Saving 402 bytes of bootinfo
[    0.000000] printk: debug: ignoring loglevel setting.
[    0.000000] printk: bootconsole [debug0] enabled
[    0.000000] Detected Macintosh model: 27
[    0.000000]  Penguin bootinfo data:
[    0.000000]  Video: addr 0x60b00000 row 0x50 depth 1 dimensions 640 x 480
[    0.000000]  Videological 0xf0300000 phys. 0x60b00000, SCC at 0x50f04000
[    0.000000]  Boottime: 0x73e GMTBias: 0x0
[    0.000000]  Machine ID: 27 CPUid: 0x1 memory size: 0x24
[    0.000000] Apple Macintosh LC III
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000000000-0x00000023ffffffff]
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x00000000023fffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x00000000023fffff]
[    0.000000] pcpu-alloc: s0 r0 d32768 u32768 alloc=1*32768
[    0.000000] pcpu-alloc: [0] 0 
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 9135
[    0.000000] Kernel command line: console=ttyS0 ignore_loglevel ro earlyprintk fbcon=font:ProFont6x11 coherent_pool=4k root=/dev/sda6 single 
[    0.000000] Unknown kernel command line parameters "single", will be passed to user space.
[    0.000000] Dentry cache hash table entries: 8192 (order: 3, 32768 bytes, linear)
[    0.000000] Inode-cache hash table entries: 4096 (order: 2, 16384 bytes, linear)
[    0.000000] Sorting __ex_table...
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 31008K/36864K available (4193K kernel code, 283K rwdata, 808K rodata, 132K init, 109K bss, 5856K reserved, 0K cma-reserved)
[    0.000000] **********************************************************
[    0.000000] **   NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE   **
[    0.000000] **                                                      **
[    0.000000] ** This system shows unhashed kernel memory addresses   **
[    0.000000] ** via the console, logs, and other interfaces. This    **
[    0.000000] ** might reduce the security of your system.            **
[    0.000000] **                                                      **
[    0.000000] ** If you see this message and you are not debugging    **
[    0.000000] ** the kernel, report this immediately to your system   **
[    0.000000] ** administrator!                                       **
[    0.000000] **                                                      **
[    0.000000] **   NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE   **
[    0.000000] **********************************************************
[    0.000000] SLUB: HWalign=16, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] NR_IRQS: 72
[    0.000000] clocksource: via1: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 2439823894983 ns
[    0.050000] Console: colour dummy device 80x25
[    0.060000] printk: console [ttyS0] enabled
[    0.060000] printk: console [ttyS0] enabled
[    0.080000] printk: bootconsole [debug0] disabled
[    0.080000] printk: bootconsole [debug0] disabled
[    0.110000] Calibrating delay loop... 5.83 BogoMIPS (lpj=29184)
[    0.270000] pid_max: default: 32768 minimum: 301
[    0.320000] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.340000] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.730000] devtmpfs: initialized
[    1.060000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    1.070000] futex hash table entries: 256 (order: -1, 3072 bytes, linear)
[    1.280000] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    1.340000] DMA: preallocated 4 KiB GFP_KERNEL pool for atomic allocations
[    1.360000] DMA: preallocated 4 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    1.940000] NuBus: Scanning NuBus slots.
[    2.180000] SCSI subsystem initialized
[    2.250000] libata version 3.00 loaded.
[    2.510000] clocksource: Switched to clocksource via1
[    5.540000] NET: Registered PF_INET protocol family
[    5.590000] IP idents hash table entries: 2048 (order: 2, 16384 bytes, linear)
[    5.780000] tcp_listen_portaddr_hash hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    5.790000] Table-perturb hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    5.810000] TCP established hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    5.820000] TCP bind hash table entries: 1024 (order: 1, 8192 bytes, linear)
[    5.830000] TCP: Hash tables configured (established 1024 bind 1024)
[    5.880000] UDP hash table entries: 256 (order: 0, 4096 bytes, linear)
[    5.890000] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes, linear)
[    5.930000] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    6.090000] RPC: Registered named UNIX socket transport module.
[    6.110000] RPC: Registered udp transport module.
[    6.120000] RPC: Registered tcp transport module.
[    6.130000] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    6.430000] workingset: timestamp_bits=30 max_order=13 bucket_order=0
[    6.910000] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[    6.920000] io scheduler mq-deadline registered
[    6.930000] macfb: framebuffer at 0x60b00000, mapped to 0xd0000000, size 37k
[    6.940000] macfb: mode is 640x480x1, linelength=80
[    8.510000] Console: switching to mono frame buffer device 106x43
[    9.670000] fb0: Sonora frame buffer device
[    9.750000] scc.0: ttyS0 at MMIO 0x50f04002 (irq = 4, base_baud = 230400) is a Z85c30 ESCC - Serial port
[    9.820000] scc.1: ttyS1 at MMIO 0x50f04000 (irq = 4, base_baud = 230400) is a Z85c30 ESCC - Serial port
[    9.950000] Non-volatile memory driver v1.3
[    9.980000] SWIM floppy driver Version 0.2 (2008-10-30)
[   10.670000] brd: module loaded
[   10.690000] cuda_input: 01 00 03 7c 25 b7 d6
[   10.710000] Macintosh Cuda and Egret driver.
[   11.270000] scsi host0: Macintosh NCR5380 SCSI, irq 19, io_port 0x0, base 0x50f10000, can_queue 16, cmd_per_lun 2, sg_tablesize 1, this_id 7, flags { }
[   11.420000] scsi 0:0:0:0: Direct-Access      codesrc          SCSI2SD  4.2 PQ: 0 ANSI: 2
[   11.800000] adb device [2]: 2 0x2
[   11.830000] adb device [3]: 3 0x1
[   11.880000] ADB keyboard at 2 has handler 0x3
[   11.900000] Detected ADB keyboard, type ANSI.
[   11.960000] input: ADB keyboard as /devices/virtual/input/input0
[   12.180000] ADB mouse (standard) at 3 has handler 0x2
[   12.240000] input: ADB mouse as /devices/virtual/input/input1
[   13.720000] sd 0:0:0:0: [sda] 4194303 512-byte logical blocks: (2.15 GB/2.00 GiB)
[   13.750000] sd 0:0:0:0: Attached scsi generic sg0 type 0
[   13.800000] mac8390 slot.E: Memory length resource not found, probing
[   13.810000] mac8390 slot.E: Ethernet A-Series (type farallon)
[   13.820000] mac8390 slot.E: MAC 00:40:10:07:60:c8, IRQ 61, 32 KB shared memory at 0xfeed0000, 32-bit access.
[   14.150000] sd 0:0:0:0: [sda] Write Protect is off
[   14.160000] sd 0:0:0:0: [sda] Mode Sense: 8d 00 00 08
[   14.180000] sd 0:0:0:0: [sda] Write cache: disabled, read cache: disabled, doesn't support DPO or FUA
[   14.480000]  sda: [mac] sda1 sda2 sda3 sda4 sda5 sda6 sda7
[   14.980000] sd 0:0:0:0: [sda] Attached SCSI disk
[   15.050000] aoe: AoE v85 initialised.
[   15.190000] mousedev: PS/2 mouse device common for all mice
[   15.320000] rtc-generic rtc-generic: registered as rtc0
[   15.410000] NET: Registered PF_PACKET protocol family
[   15.440000] printk: bootconsole [debug0] disabled
[   21.960000] EXT4-fs (sda6): mounting ext2 file system using the ext4 subsystem
[   22.620000] EXT4-fs (sda6): mounted filesystem d9738810-a43e-43d5-8eea-fd50d23f69b4 without journal. Quota mode: disabled.
[   22.650000] VFS: Mounted root (ext2 filesystem) readonly on device 8:6.
[   22.690000] Freeing unused kernel image (initmem) memory: 132K
[   22.700000] This architecture does not have kernel memory protection.
[   22.710000] Run /sbin/init as init process
[   22.720000]   with arguments:
[   22.730000]     /sbin/init
[   22.740000]     single
[   22.750000]   with environment:
[   22.760000]     HOME=/
[   22.770000]     TERM=linux
INIT: version 3.06 booting
INIT: No inittab.d directory found
[   40.210000] random: crng init done
Setting hostname to 'debian'...done.
findfs: unable to resolve 'UUID=d9738810-a43e-43d5-8eea-fd50d23f69b4'
Files under mount point '/run' will be hidden. ... (warning).
Starting hotplug events dispatcher: systemd-udevd.
Synthesizing the initial hotplug events (subsystems)...done.
Synthesizing the initial hotplug events (devices)...done.
Waiting for /dev to be fully populated...Failed to wait for daemon to reply: Connection timed out
done (timeout).
Press Enter for maintenance
(or press Control-D to continue): 
root@debian:~# sh /etc/init.d/mountdevsubfs.sh start
root@debian:~# sh /etc/init.d/mountdevsubfs.sh start
root@debian:~# sh /etc/init.d/mountdevsubfs.sh start
root@debian:~# sh /etc/init.d/mountdevsubfs.sh start
root@debian:~# 
exit
Activating swap:swapon: /dev/sda5: found signature [pagesize=4096, signature=swap]
swapon: /dev/sda5: pagesize=4096, swapsize=268435456, devsize=268435456
swapon /dev/sda5
[ 1768.650000] Adding 262140k swap on /dev/sda5.  Priority:-2 extents:1 across:262140k 
.
[ 1778.640000] EXT4-fs (sda6): re-mounted d9738810-a43e-43d5-8eea-fd50d23f69b4. Quota mode: disabled.
Will now check root file system:[/sbin/fsck.ext2 (1) -- /] fsck.ext2 -a -C0 /dev/sda6 
/dev/sda6: clean, 24004/425152 files, 234859/424872 blocks
.
[ 1797.940000] EXT4-fs (sda6): re-mounted d9738810-a43e-43d5-8eea-fd50d23f69b4. Quota mode: disabled.
*** stack smashing detected ***: terminated
Aborted (core dumped)
*** stack smashing detected ***: terminated
Aborted (core dumped)
Will now check all file systems.
Checking all file systems.
UUID=d9738810-a43e-43d5-8eea-fd50d23f69b4 is mounted
Done checking file systems.
Log is being saved in /var/log/fsck/checkfs if that location is writable.
Cleaning up temporary files...Cleaning /tmp...done.
 /tmp.
Will now mount local filesystems:.
Will now activate swapfile swap, if any:swapon: /dev/sda5: already active -- ignored
done.
Checking minimum space in /tmp...done.
Cleaning up temporary files....
Starting Setting kernel variables: sysctl.
Initializing random number generator...done.
Configuring network interfaces...done.
Starting nftables: nonemnl.c:60: Unable to initialize Netlink socket: Protocol not supported
 failed!
Cleaning up temporary files....
Press Enter for maintenance
(or press Control-D to continue): 
root@debian:~# sh /etc/init.d/mountdevsubfs.sh start
*** stack smashing detected ***: terminated
Aborted (core dumped)
root@debian:~# sh /etc/init.d/mountdevsubfs.sh start
root@debian:~# sh /etc/init.d/mountdevsubfs.sh start
*** stack smashing detected ***: terminated
Aborted (core dumped)
*** stack smashing detected ***: terminated
Aborted (core dumped)
root@debian:~# sh /etc/init.d/mountdevsubfs.sh start
root@debian:~# reboot
INIT: Switching to runlevel: 6
INIT: No inittab.d directory found
INIT: Sending processes configured via /etc/inittab the TERM signal
INIT: Sending processes configured via /etc/inittab the KILL signal
INIT: Pid 1006 [id ~~] seems to hang
Stopping Flexible I/O Tester as service: fio is not running.
Stopping nftables: nonemnl.c:60: Unable to initialize Netlink socket: Protocol not supported
 failed!
Asking all remaining processes to terminate...done.
All processes ended within 1 seconds...done.
Stopping hotplug events dispatcher: systemd-udevd.
Saving random seed...done.
Stopping enhanced syslogd: rsyslogd already stopped.
Saving the system clock to /dev/rtc0.
Hardware Clock updated to Thu Jan  1 11:13:47 AEST 1970.
Deconfiguring network interfaces...done.
Will now deactivate swap:swapoff /dev/sda5
.
Mounting root filesystem read-only...[ 2590.610000] EXT4-fs (sda6): re-mounted d9738810-a43e-43d5-8eea-fd50d23f69b4. Quota mode: disabled.
done.
Will now restart.
[ 2597.720000] reboot: Restarting system

Re: dash behaviour, was Re: core dump analysis

[Geert Uytterhoeven]

Hi Finn,

On Sat, Apr 8, 2023 at 7:26 AM Finn Thain <fthain@linux-m68k.org> wrote:
> On Fri, 7 Apr 2023, Geert Uytterhoeven wrote:
> > > The only way I have found to alter dash's inclination to crash is to
> > > reboot. (I said previously I was unable to reproduce this in a single
> > > user mode shell but it turned out to be more subtle.)
> >
> > That sounds like memory corruption somewhere else, e.g. in the buffer
> > cache...
> >
>
> If so, once the corruption showed up, you would expect the same crash next
> time...
>
> root@debian:~# sh /etc/init.d/mountdevsubfs.sh start
> *** stack smashing detected ***: terminated
> Aborted (core dumped)
> root@debian:~# sh /etc/init.d/mountdevsubfs.sh start
> *** stack smashing detected ***: terminated
> Aborted (core dumped)
> *** stack smashing detected ***: terminated
> Aborted (core dumped)
> root@debian:~# sh /etc/init.d/mountdevsubfs.sh start
> *** stack smashing detected ***: terminated
> Aborted (core dumped)
> root@debian:~# echo 3 > /proc/sys/vm/drop_caches
> [  937.250000] bash (717): drop_caches: 3
> root@debian:~# sh /etc/init.d/mountdevsubfs.sh start
> root@debian:~# sh /etc/init.d/mountdevsubfs.sh start
> *** stack smashing detected ***: terminated
> Aborted (core dumped)
> root@debian:~# sh /etc/init.d/mountdevsubfs.sh start
> *** stack smashing detected ***: terminated
> Aborted (core dumped)
> *** stack smashing detected ***: terminated
> Aborted (core dumped)
>
> I'd say it's probably not buffer cache corruption causing this because we
> can see two subshells fail, then just one.

OK.

> For that build I enabled SLUB_DEBUG but forgot to enable SLUB_DEBUG_ON --

FWIW, unfortunately not all SL*B_DEBUG are equal. I've seen
memory corruptions that were identified by SLAB_DEBUG, but not by
the other allocators.

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: dash behaviour, was Re: core dump analysis

[Michael Schmitz]

Hi Finn,

Am 07.04.2023 um 16:03 schrieb Finn Thain:
>>> So, again, the best avenue I can think of for such experiments to
>>> modify the kernel to either keep track of the times of the wait4
>>> syscalls and
>>
>> The easiest way to do that is to log all wait and signal syscalls, as
>> well as process exit. That might alter timing if these log messages go
>> to the serial console though. Is that what you have in mind?
>>
>
> What I had in mind was collecting measurements in such way that would not
> impact timing, perhaps by storing them somewhere they could be retrieved
> from the process core dump.

Storing such information to the process core dump would need 
modifications to dash (allocate log buffer and pass the address to the 
kernel) in addition to the kernel. Taking Geert's suggestions to have 
the kernel allocate a log buffer (similar to the kernel message log) and 
read that out through procfs after the tests would be a lot easier.

>
> But that's probably not realistic and it's probably pointless anyway -- I
> don't expect to find an old bug in common code like kernel/exit.c, or in a
> hot path like those in arch/m68k/kernel/entry.S.
>
> More likely is that some kind of bug in dash causes it to corrupt its own
> stack when conditions are just right. I just need to figure out how to
> recreate those conditions. :-/
>
> When dash is feeling crashy, you can get results like this:
>
> root@debian:~# sh /etc/init.d/mountdevsubfs.sh
> *** stack smashing detected ***: terminated
> Aborted (core dumped)
> Warning: mountdevsubfs should be called with the 'start' argument.
> root@debian:~# sh /etc/init.d/mountdevsubfs.sh
> *** stack smashing detected ***: terminated
> Aborted (core dumped)
> Warning: mountdevsubfs should be called with the 'start' argument.
> root@debian:~# sh /etc/init.d/mountdevsubfs.sh
> *** stack smashing detected ***: terminated
> Aborted (core dumped)
> Warning: mountdevsubfs should be called with the 'start' argument.
> root@debian:~# sh /etc/init.d/mountdevsubfs.sh
> *** stack smashing detected ***: terminated
> Aborted (core dumped)
> Warning: mountdevsubfs should be called with the 'start' argument.
> root@debian:~#
>
> But when it's not feeling crashy, you can't:
>
> root@debian:~# sh /etc/init.d/mountdevsubfs.sh
> Warning: mountdevsubfs should be called with the 'start' argument.
> root@debian:~# sh /etc/init.d/mountdevsubfs.sh
> Warning: mountdevsubfs should be called with the 'start' argument.
> root@debian:~# sh /etc/init.d/mountdevsubfs.sh
> Warning: mountdevsubfs should be called with the 'start' argument.
> root@debian:~# sh /etc/init.d/mountdevsubfs.sh
> Warning: mountdevsubfs should be called with the 'start' argument.
>
> The only way I have found to alter dash's inclination to crash is to
> reboot. (I said previously I was unable to reproduce this in a single user
> mode shell but it turned out to be more subtle.)

I wonder what could change from one boot to another - can you have dash 
(and its subshells) dump /proc/self/maps and see whether there's any 
variation in that? But what we really need is the physical mappings. How 
can we find those?

With the kernel RNG disabled, I would expect neither of these mappings 
to change between boots?

Cheers,

	Michael

Re: dash behaviour, was Re: core dump analysis

[Finn Thain]

On Sun, 9 Apr 2023, Michael Schmitz wrote:

> >
> > The only way I have found to alter dash's inclination to crash is to 
> > reboot. (I said previously I was unable to reproduce this in a single 
> > user mode shell but it turned out to be more subtle.)
> 
> I wonder what could change from one boot to another - can you have dash 
> (and its subshells) dump /proc/self/maps and see whether there's any 
> variation in that? But what we really need is the physical mappings. How 
> can we find those?
> 
> With the kernel RNG disabled, I would expect neither of these mappings 
> to change between boots?
> 

It looks like the stack area still changes across invocations:

# sh
# cat < /proc/self/maps
c0000000-c0021000 r-xp 00000000 08:06 38780      /usr/lib/m68k-linux-gnu/ld.so.1
c0021000-c0023000 rw-p 00000000 00:00 0 
c0023000-c0024000 r--p 00021000 08:06 38780      /usr/lib/m68k-linux-gnu/ld.so.1
c0024000-c0026000 rw-p 00022000 08:06 38780      /usr/lib/m68k-linux-gnu/ld.so.1
c002a000-c0199000 r-xp 00000000 08:06 38786      /usr/lib/m68k-linux-gnu/libc.so.6
c0199000-c019a000 ---p 0016f000 08:06 38786      /usr/lib/m68k-linux-gnu/libc.so.6
c019a000-c019c000 r--p 00170000 08:06 38786      /usr/lib/m68k-linux-gnu/libc.so.6
c019c000-c01a0000 rw-p 00172000 08:06 38786      /usr/lib/m68k-linux-gnu/libc.so.6
c01a0000-c01aa000 rw-p 00000000 00:00 0 
d0000000-d0019000 r-xp 00000000 08:06 32713      /usr/bin/dash
d001b000-d001c000 r--p 00019000 08:06 32713      /usr/bin/dash
d001c000-d001d000 rw-p 0001a000 08:06 32713      /usr/bin/dash
d001d000-d001f000 rwxp 00000000 00:00 0          [heap]
d001f000-d0040000 rwxp 00000000 00:00 0          [heap]
eff9f000-effc0000 rw-p 00000000 00:00 0          [stack]
# sh
# cat < /proc/self/maps
c0000000-c0021000 r-xp 00000000 08:06 38780      /usr/lib/m68k-linux-gnu/ld.so.1
c0021000-c0023000 rw-p 00000000 00:00 0 
c0023000-c0024000 r--p 00021000 08:06 38780      /usr/lib/m68k-linux-gnu/ld.so.1
c0024000-c0026000 rw-p 00022000 08:06 38780      /usr/lib/m68k-linux-gnu/ld.so.1
c002a000-c0199000 r-xp 00000000 08:06 38786      /usr/lib/m68k-linux-gnu/libc.so.6
c0199000-c019a000 ---p 0016f000 08:06 38786      /usr/lib/m68k-linux-gnu/libc.so.6
c019a000-c019c000 r--p 00170000 08:06 38786      /usr/lib/m68k-linux-gnu/libc.so.6
c019c000-c01a0000 rw-p 00172000 08:06 38786      /usr/lib/m68k-linux-gnu/libc.so.6
c01a0000-c01aa000 rw-p 00000000 00:00 0 
d0000000-d0019000 r-xp 00000000 08:06 32713      /usr/bin/dash
d001b000-d001c000 r--p 00019000 08:06 32713      /usr/bin/dash
d001c000-d001d000 rw-p 0001a000 08:06 32713      /usr/bin/dash
d001d000-d001f000 rwxp 00000000 00:00 0          [heap]
d001f000-d0040000 rwxp 00000000 00:00 0          [heap]
effd8000-efff9000 rw-p 00000000 00:00 0          [stack]
# sh
# cat < /proc/self/maps
c0000000-c0021000 r-xp 00000000 08:06 38780      /usr/lib/m68k-linux-gnu/ld.so.1
c0021000-c0023000 rw-p 00000000 00:00 0 
c0023000-c0024000 r--p 00021000 08:06 38780      /usr/lib/m68k-linux-gnu/ld.so.1
c0024000-c0026000 rw-p 00022000 08:06 38780      /usr/lib/m68k-linux-gnu/ld.so.1
c002a000-c0199000 r-xp 00000000 08:06 38786      /usr/lib/m68k-linux-gnu/libc.so.6
c0199000-c019a000 ---p 0016f000 08:06 38786      /usr/lib/m68k-linux-gnu/libc.so.6
c019a000-c019c000 r--p 00170000 08:06 38786      /usr/lib/m68k-linux-gnu/libc.so.6
c019c000-c01a0000 rw-p 00172000 08:06 38786      /usr/lib/m68k-linux-gnu/libc.so.6
c01a0000-c01aa000 rw-p 00000000 00:00 0 
d0000000-d0019000 r-xp 00000000 08:06 32713      /usr/bin/dash
d001b000-d001c000 r--p 00019000 08:06 32713      /usr/bin/dash
d001c000-d001d000 rw-p 0001a000 08:06 32713      /usr/bin/dash
d001d000-d001f000 rwxp 00000000 00:00 0          [heap]
d001f000-d0040000 rwxp 00000000 00:00 0          [heap]
effdf000-f0000000 rw-p 00000000 00:00 0          [stack]
#

That can be disabled easily though (see below). I'll have to modify some 
init scripts to find out what effect it has.

# setarch -R sh
# sh
# cat < /proc/self/maps
c0000000-c0021000 r-xp 00000000 08:06 38780      /usr/lib/m68k-linux-gnu/ld.so.1
c0021000-c0023000 rw-p 00000000 00:00 0 
c0023000-c0024000 r--p 00021000 08:06 38780      /usr/lib/m68k-linux-gnu/ld.so.1
c0024000-c0026000 rw-p 00022000 08:06 38780      /usr/lib/m68k-linux-gnu/ld.so.1
c002a000-c0199000 r-xp 00000000 08:06 38786      /usr/lib/m68k-linux-gnu/libc.so.6
c0199000-c019a000 ---p 0016f000 08:06 38786      /usr/lib/m68k-linux-gnu/libc.so.6
c019a000-c019c000 r--p 00170000 08:06 38786      /usr/lib/m68k-linux-gnu/libc.so.6
c019c000-c01a0000 rw-p 00172000 08:06 38786      /usr/lib/m68k-linux-gnu/libc.so.6
c01a0000-c01aa000 rw-p 00000000 00:00 0 
d0000000-d0019000 r-xp 00000000 08:06 32713      /usr/bin/dash
d001b000-d001c000 r--p 00019000 08:06 32713      /usr/bin/dash
d001c000-d001d000 rw-p 0001a000 08:06 32713      /usr/bin/dash
d001d000-d001f000 rwxp 00000000 00:00 0          [heap]
d001f000-d0040000 rwxp 00000000 00:00 0          [heap]
effdf000-f0000000 rw-p 00000000 00:00 0          [stack]
# md5sum < /proc/self/maps
baacbaf944fb01d3200d924da7f7a815  -
# sh
# md5sum < /proc/self/maps
baacbaf944fb01d3200d924da7f7a815  -
# sh
# md5sum < /proc/self/maps
baacbaf944fb01d3200d924da7f7a815  -

Re: dash behaviour, was Re: core dump analysis

[Michael Schmitz]

Hi Finn,

Am 09.04.2023 um 16:42 schrieb Finn Thain:
> On Sun, 9 Apr 2023, Michael Schmitz wrote:
>
>>>
>>> The only way I have found to alter dash's inclination to crash is to
>>> reboot. (I said previously I was unable to reproduce this in a single
>>> user mode shell but it turned out to be more subtle.)
>>
>> I wonder what could change from one boot to another - can you have dash
>> (and its subshells) dump /proc/self/maps and see whether there's any
>> variation in that? But what we really need is the physical mappings. How
>> can we find those?
>>
>> With the kernel RNG disabled, I would expect neither of these mappings
>> to change between boots?
>>
>
> It looks like the stack area still changes across invocations:

Yep, but running the same commands in the same order across different 
boots, does it still change?

(I'm making a huge assumption here - that timing of the boot process and 
hence evolution of the kernel RNG is sufficiently deterministic. And 
this might apply only to the shells run from sysvinit, since that does 
require no keyboard input ...)

Looks like cat < /proc/self/maps | grep stack would give us enough 
information without overwhelming the serial console?

OTOH - if you can show the error is gone without stack address 
randomization, that would be a hint maybe?

Cheers,

	Michael
>
> # sh
> # cat < /proc/self/maps
> c0000000-c0021000 r-xp 00000000 08:06 38780      /usr/lib/m68k-linux-gnu/ld.so.1
> c0021000-c0023000 rw-p 00000000 00:00 0
> c0023000-c0024000 r--p 00021000 08:06 38780      /usr/lib/m68k-linux-gnu/ld.so.1
> c0024000-c0026000 rw-p 00022000 08:06 38780      /usr/lib/m68k-linux-gnu/ld.so.1
> c002a000-c0199000 r-xp 00000000 08:06 38786      /usr/lib/m68k-linux-gnu/libc.so.6
> c0199000-c019a000 ---p 0016f000 08:06 38786      /usr/lib/m68k-linux-gnu/libc.so.6
> c019a000-c019c000 r--p 00170000 08:06 38786      /usr/lib/m68k-linux-gnu/libc.so.6
> c019c000-c01a0000 rw-p 00172000 08:06 38786      /usr/lib/m68k-linux-gnu/libc.so.6
> c01a0000-c01aa000 rw-p 00000000 00:00 0
> d0000000-d0019000 r-xp 00000000 08:06 32713      /usr/bin/dash
> d001b000-d001c000 r--p 00019000 08:06 32713      /usr/bin/dash
> d001c000-d001d000 rw-p 0001a000 08:06 32713      /usr/bin/dash
> d001d000-d001f000 rwxp 00000000 00:00 0          [heap]
> d001f000-d0040000 rwxp 00000000 00:00 0          [heap]
> eff9f000-effc0000 rw-p 00000000 00:00 0          [stack]
> # sh
> # cat < /proc/self/maps
> c0000000-c0021000 r-xp 00000000 08:06 38780      /usr/lib/m68k-linux-gnu/ld.so.1
> c0021000-c0023000 rw-p 00000000 00:00 0
> c0023000-c0024000 r--p 00021000 08:06 38780      /usr/lib/m68k-linux-gnu/ld.so.1
> c0024000-c0026000 rw-p 00022000 08:06 38780      /usr/lib/m68k-linux-gnu/ld.so.1
> c002a000-c0199000 r-xp 00000000 08:06 38786      /usr/lib/m68k-linux-gnu/libc.so.6
> c0199000-c019a000 ---p 0016f000 08:06 38786      /usr/lib/m68k-linux-gnu/libc.so.6
> c019a000-c019c000 r--p 00170000 08:06 38786      /usr/lib/m68k-linux-gnu/libc.so.6
> c019c000-c01a0000 rw-p 00172000 08:06 38786      /usr/lib/m68k-linux-gnu/libc.so.6
> c01a0000-c01aa000 rw-p 00000000 00:00 0
> d0000000-d0019000 r-xp 00000000 08:06 32713      /usr/bin/dash
> d001b000-d001c000 r--p 00019000 08:06 32713      /usr/bin/dash
> d001c000-d001d000 rw-p 0001a000 08:06 32713      /usr/bin/dash
> d001d000-d001f000 rwxp 00000000 00:00 0          [heap]
> d001f000-d0040000 rwxp 00000000 00:00 0          [heap]
> effd8000-efff9000 rw-p 00000000 00:00 0          [stack]
> # sh
> # cat < /proc/self/maps
> c0000000-c0021000 r-xp 00000000 08:06 38780      /usr/lib/m68k-linux-gnu/ld.so.1
> c0021000-c0023000 rw-p 00000000 00:00 0
> c0023000-c0024000 r--p 00021000 08:06 38780      /usr/lib/m68k-linux-gnu/ld.so.1
> c0024000-c0026000 rw-p 00022000 08:06 38780      /usr/lib/m68k-linux-gnu/ld.so.1
> c002a000-c0199000 r-xp 00000000 08:06 38786      /usr/lib/m68k-linux-gnu/libc.so.6
> c0199000-c019a000 ---p 0016f000 08:06 38786      /usr/lib/m68k-linux-gnu/libc.so.6
> c019a000-c019c000 r--p 00170000 08:06 38786      /usr/lib/m68k-linux-gnu/libc.so.6
> c019c000-c01a0000 rw-p 00172000 08:06 38786      /usr/lib/m68k-linux-gnu/libc.so.6
> c01a0000-c01aa000 rw-p 00000000 00:00 0
> d0000000-d0019000 r-xp 00000000 08:06 32713      /usr/bin/dash
> d001b000-d001c000 r--p 00019000 08:06 32713      /usr/bin/dash
> d001c000-d001d000 rw-p 0001a000 08:06 32713      /usr/bin/dash
> d001d000-d001f000 rwxp 00000000 00:00 0          [heap]
> d001f000-d0040000 rwxp 00000000 00:00 0          [heap]
> effdf000-f0000000 rw-p 00000000 00:00 0          [stack]
> #
>
> That can be disabled easily though (see below). I'll have to modify some
> init scripts to find out what effect it has.
>
> # setarch -R sh
> # sh
> # cat < /proc/self/maps
> c0000000-c0021000 r-xp 00000000 08:06 38780      /usr/lib/m68k-linux-gnu/ld.so.1
> c0021000-c0023000 rw-p 00000000 00:00 0
> c0023000-c0024000 r--p 00021000 08:06 38780      /usr/lib/m68k-linux-gnu/ld.so.1
> c0024000-c0026000 rw-p 00022000 08:06 38780      /usr/lib/m68k-linux-gnu/ld.so.1
> c002a000-c0199000 r-xp 00000000 08:06 38786      /usr/lib/m68k-linux-gnu/libc.so.6
> c0199000-c019a000 ---p 0016f000 08:06 38786      /usr/lib/m68k-linux-gnu/libc.so.6
> c019a000-c019c000 r--p 00170000 08:06 38786      /usr/lib/m68k-linux-gnu/libc.so.6
> c019c000-c01a0000 rw-p 00172000 08:06 38786      /usr/lib/m68k-linux-gnu/libc.so.6
> c01a0000-c01aa000 rw-p 00000000 00:00 0
> d0000000-d0019000 r-xp 00000000 08:06 32713      /usr/bin/dash
> d001b000-d001c000 r--p 00019000 08:06 32713      /usr/bin/dash
> d001c000-d001d000 rw-p 0001a000 08:06 32713      /usr/bin/dash
> d001d000-d001f000 rwxp 00000000 00:00 0          [heap]
> d001f000-d0040000 rwxp 00000000 00:00 0          [heap]
> effdf000-f0000000 rw-p 00000000 00:00 0          [stack]
> # md5sum < /proc/self/maps
> baacbaf944fb01d3200d924da7f7a815  -
> # sh
> # md5sum < /proc/self/maps
> baacbaf944fb01d3200d924da7f7a815  -
> # sh
> # md5sum < /proc/self/maps
> baacbaf944fb01d3200d924da7f7a815  -
>

Re: dash behaviour, was Re: core dump analysis

[Finn Thain]

On Sun, 9 Apr 2023, Michael Schmitz wrote:

> Am 09.04.2023 um 16:42 schrieb Finn Thain:
> > On Sun, 9 Apr 2023, Michael Schmitz wrote:
> >
> >>>
> >>> The only way I have found to alter dash's inclination to crash is to 
> >>> reboot. (I said previously I was unable to reproduce this in a 
> >>> single user mode shell but it turned out to be more subtle.)
> >>
> >> I wonder what could change from one boot to another - can you have 
> >> dash (and its subshells) dump /proc/self/maps and see whether there's 
> >> any variation in that? But what we really need is the physical 
> >> mappings. How can we find those?
> >>
> >> With the kernel RNG disabled, I would expect neither of these 
> >> mappings to change between boots?
> >>
> >
> > It looks like the stack area still changes across invocations:
> 
> Yep, but running the same commands in the same order across different 
> boots, does it still change?
> 
> (I'm making a huge assumption here - that timing of the boot process and 
> hence evolution of the kernel RNG is sufficiently deterministic. And 
> this might apply only to the shells run from sysvinit, since that does 
> require no keyboard input ...)
> 
> Looks like cat < /proc/self/maps | grep stack would give us enough 
> information without overwhelming the serial console?
> 
> OTOH - if you can show the error is gone without stack address 
> randomization, that would be a hint maybe?
> 

The results below were produced with 'norandmaps' added to the kernel 
parameters to avoid ASLR.

root@debian:~# sh /etc/init.d/mountdevsubfs.sh start
root@debian:~# sh /etc/init.d/mountdevsubfs.sh start
root@debian:~# sh /etc/init.d/mountdevsubfs.sh start
root@debian:~# sh /etc/init.d/mountdevsubfs.sh start
root@debian:~# echo 3 > /proc/sys/vm/drop_caches 
[  913.560000] bash (1024): drop_caches: 3
root@debian:~# sh /etc/init.d/mountdevsubfs.sh start
root@debian:~# sh /etc/init.d/mountdevsubfs.sh start
root@debian:~# sh /etc/init.d/mountdevsubfs.sh start
root@debian:~# sh /etc/init.d/mountdevsubfs.sh start
*** stack smashing detected ***: terminated
Aborted (core dumped)
root@debian:~# sh -c "md5sum < /proc/self/maps"
baacbaf944fb01d3200d924da7f7a815  -
root@debian:~# sh -c "md5sum < /proc/self/maps"
baacbaf944fb01d3200d924da7f7a815  -
root@debian:~# sh -c "md5sum < /proc/self/maps"
baacbaf944fb01d3200d924da7f7a815  -
root@debian:~# sh -c "md5sum < /proc/self/maps"
baacbaf944fb01d3200d924da7f7a815  -
root@debian:~# sh -c "cat < /proc/self/maps"
c0000000-c0021000 r-xp 00000000 08:06 38780      /usr/lib/m68k-linux-gnu/ld.so.1
c0021000-c0023000 rw-p 00000000 00:00 0 
c0023000-c0024000 r--p 00021000 08:06 38780      /usr/lib/m68k-linux-gnu/ld.so.1
c0024000-c0026000 rw-p 00022000 08:06 38780      /usr/lib/m68k-linux-gnu/ld.so.1
c002a000-c0199000 r-xp 00000000 08:06 38786      /usr/lib/m68k-linux-gnu/libc.so.6
c0199000-c019a000 ---p 0016f000 08:06 38786      /usr/lib/m68k-linux-gnu/libc.so.6
c019a000-c019c000 r--p 00170000 08:06 38786      /usr/lib/m68k-linux-gnu/libc.so.6
c019c000-c01a0000 rw-p 00172000 08:06 38786      /usr/lib/m68k-linux-gnu/libc.so.6
c01a0000-c01aa000 rw-p 00000000 00:00 0 
d0000000-d0019000 r-xp 00000000 08:06 32713      /usr/bin/dash
d001b000-d001c000 r--p 00019000 08:06 32713      /usr/bin/dash
d001c000-d001d000 rw-p 0001a000 08:06 32713      /usr/bin/dash
d001d000-d001f000 rwxp 00000000 00:00 0          [heap]
d001f000-d0040000 rwxp 00000000 00:00 0          [heap]
effdf000-f0000000 rw-p 00000000 00:00 0          [stack]


So I guess this bug has more to do with timing and little to do with 
state, contrary to my guesswork above. And no doubt I will have to 
contradict myself again if/when it turns out that uninitialized memory is 
a factor :-/

Re: dash behaviour, was Re: core dump analysis

[Michael Schmitz]

Hi Finn,

Am 09.04.2023 um 19:32 schrieb Finn Thain:
>> Looks like cat < /proc/self/maps | grep stack would give us enough
>> information without overwhelming the serial console?
>>
>> OTOH - if you can show the error is gone without stack address
>> randomization, that would be a hint maybe?
>>
>
> The results below were produced with 'norandmaps' added to the kernel
> parameters to avoid ASLR.
>
> root@debian:~# sh /etc/init.d/mountdevsubfs.sh start
> root@debian:~# sh /etc/init.d/mountdevsubfs.sh start
> root@debian:~# sh /etc/init.d/mountdevsubfs.sh start
> root@debian:~# sh /etc/init.d/mountdevsubfs.sh start
> root@debian:~# echo 3 > /proc/sys/vm/drop_caches
> [  913.560000] bash (1024): drop_caches: 3
> root@debian:~# sh /etc/init.d/mountdevsubfs.sh start
> root@debian:~# sh /etc/init.d/mountdevsubfs.sh start
> root@debian:~# sh /etc/init.d/mountdevsubfs.sh start
> root@debian:~# sh /etc/init.d/mountdevsubfs.sh start
> *** stack smashing detected ***: terminated
> Aborted (core dumped)
> root@debian:~# sh -c "md5sum < /proc/self/maps"
> baacbaf944fb01d3200d924da7f7a815  -
> root@debian:~# sh -c "md5sum < /proc/self/maps"
> baacbaf944fb01d3200d924da7f7a815  -
> root@debian:~# sh -c "md5sum < /proc/self/maps"
> baacbaf944fb01d3200d924da7f7a815  -
> root@debian:~# sh -c "md5sum < /proc/self/maps"
> baacbaf944fb01d3200d924da7f7a815  -
> root@debian:~# sh -c "cat < /proc/self/maps"
> c0000000-c0021000 r-xp 00000000 08:06 38780      /usr/lib/m68k-linux-gnu/ld.so.1
> c0021000-c0023000 rw-p 00000000 00:00 0
> c0023000-c0024000 r--p 00021000 08:06 38780      /usr/lib/m68k-linux-gnu/ld.so.1
> c0024000-c0026000 rw-p 00022000 08:06 38780      /usr/lib/m68k-linux-gnu/ld.so.1
> c002a000-c0199000 r-xp 00000000 08:06 38786      /usr/lib/m68k-linux-gnu/libc.so.6
> c0199000-c019a000 ---p 0016f000 08:06 38786      /usr/lib/m68k-linux-gnu/libc.so.6
> c019a000-c019c000 r--p 00170000 08:06 38786      /usr/lib/m68k-linux-gnu/libc.so.6
> c019c000-c01a0000 rw-p 00172000 08:06 38786      /usr/lib/m68k-linux-gnu/libc.so.6
> c01a0000-c01aa000 rw-p 00000000 00:00 0
> d0000000-d0019000 r-xp 00000000 08:06 32713      /usr/bin/dash
> d001b000-d001c000 r--p 00019000 08:06 32713      /usr/bin/dash
> d001c000-d001d000 rw-p 0001a000 08:06 32713      /usr/bin/dash
> d001d000-d001f000 rwxp 00000000 00:00 0          [heap]
> d001f000-d0040000 rwxp 00000000 00:00 0          [heap]
> effdf000-f0000000 rw-p 00000000 00:00 0          [stack]
>
>
> So I guess this bug has more to do with timing and little to do with
> state, contrary to my guesswork above. And no doubt I will have to

What may still vary is physical mapping - I remember you had used some 
tool before to parse proc/<pid>/pagemap to determine the physical 
addresses for task stack areas? Or am I misremembering that from some 
other bug?

> contradict myself again if/when it turns out that uninitialized memory is
> a factor :-/

I haven't found a config option to initialize memory returned by the 
kernel page allocators, so not sure how to test that ...

Cheers,

	Michael

kernel behaviour, was Re: dash behaviour

[Finn Thain]

On Mon, 10 Apr 2023, Michael Schmitz wrote:

> >
> > So I guess this bug has more to do with timing and little to do with 
> > state, contrary to my guesswork above. And no doubt I will have to
> 
> What may still vary is physical mapping - I remember you had used some 
> tool before to parse proc/<pid>/pagemap to determine the physical 
> addresses for task stack areas? Or am I misremembering that from some 
> other bug?
> 

You're right, back in September 2021 when I was chasing a different bug we 
did discuss tools to look at physical mappings. I don't think that would 
help here though. We know the failure is not bad RAM because multiple Macs 
fail in the same way. Also, there's no DMA taking place on these 
particular machines.

> > contradict myself again if/when it turns out that uninitialized memory 
> > is a factor :-/
> 
> I haven't found a config option to initialize memory returned by the 
> kernel page allocators, so not sure how to test that ...
> 

I was able to find some command line options (init_on_alloc, init_on_free) 
and the related Kconfig symbols (CONFIG_INIT_ON_ALLOC_DEFAULT_ON, 
CONFIG_INIT_ON_FREE_DEFAULT_ON).

Given the compiler supports -fzero-call-used-regs=used-gpr there's also 
CONFIG_ZERO_CALL_USED_REGS. Also CONFIG_INIT_STACK_ALL_ZERO 
(-ftrivial-auto-var-init=zero).

The problem with these options is that they may produce a large effect on 
the timing of events but they should still have no effect on the behaviour 
of a correct userspace program.

Since we are dealing with a suspect userspace program, what could we learn 
from such a test? E.g. if the crashing stopped one could simply attribute 
that to the timing change. I suppose, if the crashing became more 
frequent, perhaps that would help debug the userspace program. So maybe 
it's worth a try...

Re: kernel behaviour, was Re: dash behaviour

[Michael Schmitz]

Hi Finn,

Am 10.04.2023 um 21:39 schrieb Finn Thain:
> On Mon, 10 Apr 2023, Michael Schmitz wrote:
>
>>>
>>> So I guess this bug has more to do with timing and little to do with
>>> state, contrary to my guesswork above. And no doubt I will have to
>>
>> What may still vary is physical mapping - I remember you had used some
>> tool before to parse proc/<pid>/pagemap to determine the physical
>> addresses for task stack areas? Or am I misremembering that from some
>> other bug?
>>
>
> You're right, back in September 2021 when I was chasing a different bug we
> did discuss tools to look at physical mappings. I don't think that would
> help here though. We know the failure is not bad RAM because multiple Macs
> fail in the same way. Also, there's no DMA taking place on these
> particular machines.
>
>>> contradict myself again if/when it turns out that uninitialized memory
>>> is a factor :-/
>>
>> I haven't found a config option to initialize memory returned by the
>> kernel page allocators, so not sure how to test that ...
>>
>
> I was able to find some command line options (init_on_alloc, init_on_free)
> and the related Kconfig symbols (CONFIG_INIT_ON_ALLOC_DEFAULT_ON,
> CONFIG_INIT_ON_FREE_DEFAULT_ON).

Right - not sure how I managed to miss those.

init_on_free might delay the boot process a while! But I would guesss 
init_on_alloc should be OK in the first instance.

>
> Given the compiler supports -fzero-call-used-regs=used-gpr there's also
> CONFIG_ZERO_CALL_USED_REGS. Also CONFIG_INIT_STACK_ALL_ZERO
> (-ftrivial-auto-var-init=zero).
>
> The problem with these options is that they may produce a large effect on
> the timing of events but they should still have no effect on the behaviour
> of a correct userspace program.
>
> Since we are dealing with a suspect userspace program, what could we learn
> from such a test? E.g. if the crashing stopped one could simply attribute

We don't know for definite that we deal with a suspect user space 
program - it might just be a change in a previously fine program that 
now exposes a subtle kernel bug (undetected for quite a long time, but 
we've seen a few of those now...)?

> that to the timing change. I suppose, if the crashing became more
> frequent, perhaps that would help debug the userspace program. So maybe
> it's worth a try...

We'd then have to try and minimize the impact on timing, by instead 
initializing a 'shadow' page reserved for that purpose. Though I suspect 
the loop over the pages might be optimized away in that case. See 
include/linux/highmem.h:clear_highpage_kasan_tagged() and 
mm/page_alloc.c:kernel_init_pages() ...

Cheers,

	Michael

Re: kernel behaviour, was Re: dash behaviour

[Finn Thain]

On Tue, 11 Apr 2023, Michael Schmitz wrote:

> >
> > I was able to find some command line options (init_on_alloc, 
> > init_on_free) and the related Kconfig symbols 
> > (CONFIG_INIT_ON_ALLOC_DEFAULT_ON, CONFIG_INIT_ON_FREE_DEFAULT_ON).
> 
> Right - not sure how I managed to miss those.
> 
> init_on_free might delay the boot process a while! But I would guesss 
> init_on_alloc should be OK in the first instance.
> 
> >
> > Given the compiler supports -fzero-call-used-regs=used-gpr there's 
> > also CONFIG_ZERO_CALL_USED_REGS. Also CONFIG_INIT_STACK_ALL_ZERO 
> > (-ftrivial-auto-var-init=zero).
> >

With all of those options enabled I still see dash crash sometimes. I 
don't think I've learned anything new about the bug from that test.

> > The problem with these options is that they may produce a large effect 
> > on the timing of events but they should still have no effect on the 
> > behaviour of a correct userspace program.
> >
> > Since we are dealing with a suspect userspace program, what could we 
> > learn from such a test? E.g. if the crashing stopped one could simply 
> > attribute
> 
> We don't know for definite that we deal with a suspect user space 
> program - it might just be a change in a previously fine program that 
> now exposes a subtle kernel bug (undetected for quite a long time, but 
> we've seen a few of those now...)?
> 

That's right -- the kernel is also suspect. As is glibc. I will keep 
looking for some way to narrow down the search.

Re: kernel behaviour, was Re: dash behaviour

[Geert Uytterhoeven]

Hi Finn,

On Tue, Apr 11, 2023 at 6:59 AM Finn Thain <fthain@linux-m68k.org> wrote:
> On Tue, 11 Apr 2023, Michael Schmitz wrote:
> > We don't know for definite that we deal with a suspect user space
> > program - it might just be a change in a previously fine program that
> > now exposes a subtle kernel bug (undetected for quite a long time, but
> > we've seen a few of those now...)?
> >
>
> That's right -- the kernel is also suspect. As is glibc. I will keep
> looking for some way to narrow down the search.

Or the compiler...

https://lore.kernel.org/all/CABVgOSmgpkktiLkU-ic0xGitDOhep+3sb5X91hb8RNEzFauhAA@mail.gmail.com

Gr{oetje,eeting}s,

                        Geert


--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: core dump analysis, was Re: stack smashing detected

[Geert Uytterhoeven]

Hi Michael,

On Fri, Apr 7, 2023 at 3:58 AM Michael Schmitz <schmitzmic@gmail.com> wrote:
> The easiest way to do that is to log all wait and signal syscalls, as
> well as process exit. That might alter timing if these log messages go
> to the serial console though. Is that what you have in mind?

Store to RAM, retrieve through a new /proc file?

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: core dump analysis, was Re: stack smashing detected

[Michael Schmitz]

Hi Geert,

Am 08.04.2023 um 00:06 schrieb Geert Uytterhoeven:
> Hi Michael,
>
> On Fri, Apr 7, 2023 at 3:58 AM Michael Schmitz <schmitzmic@gmail.com> wrote:
>> The easiest way to do that is to log all wait and signal syscalls, as
>> well as process exit. That might alter timing if these log messages go
>> to the serial console though. Is that what you have in mind?
>
> Store to RAM, retrieve through a new /proc file?

Yes, that could be done, though I'd rather avoid duplicating a lot of 
the generic message formatting code (printk and friends).

I'll have a look around ...

Cheers,

	Michael

>
> Gr{oetje,eeting}s,
>
>                         Geert
>

instrumentation, was Re: core dump analysis

[Finn Thain]

[-- Attachment #1: Type: text/plain, Size: 1098 bytes --]

On Sun, 9 Apr 2023, Michael Schmitz wrote:

> Am 08.04.2023 um 00:06 schrieb Geert Uytterhoeven:
> >
> > On Fri, Apr 7, 2023 at 3:58 AM Michael Schmitz <wrote:
> >> The easiest way to do that is to log all wait and signal syscalls, as 
> >> well as process exit. That might alter timing if these log messages 
> >> go to the serial console though. Is that what you have in mind?
> >
> > Store to RAM, retrieve through a new /proc file?
> 
> Yes, that could be done, though I'd rather avoid duplicating a lot of 
> the generic message formatting code (printk and friends).
> 
> I'll have a look around ...
> 

A better solution might be be to port the existing instrumentation like 
ftrace, kprobes, uprobes etc. Might be a lot of work though. I wonder 
how portable that stuff is.

If you use printk, you could probably avoid most of the delays by enabling 
the dummy console. Then the kernel messages would be collected with dmesg, 
given a sufficiently large CONFIG_LOG_BUF_SHIFT. But it would be 
inconvenient to have no serial console available for the usual purposes.

Re: instrumentation, was Re: core dump analysis

[Michael Schmitz]

Hi Finn,

Am 11.04.2023 um 12:20 schrieb Finn Thain:
> On Sun, 9 Apr 2023, Michael Schmitz wrote:
>
>> Am 08.04.2023 um 00:06 schrieb Geert Uytterhoeven:
>>>
>>> On Fri, Apr 7, 2023 at 3:58 AM Michael Schmitz <wrote:
>>>> The easiest way to do that is to log all wait and signal syscalls, as
>>>> well as process exit. That might alter timing if these log messages
>>>> go to the serial console though. Is that what you have in mind?
>>>
>>> Store to RAM, retrieve through a new /proc file?
>>
>> Yes, that could be done, though I'd rather avoid duplicating a lot of
>> the generic message formatting code (printk and friends).
>>
>> I'll have a look around ...
>>
>
> A better solution might be be to port the existing instrumentation like
> ftrace, kprobes, uprobes etc. Might be a lot of work though. I wonder
> how portable that stuff is.
>
> If you use printk, you could probably avoid most of the delays by enabling
> the dummy console. Then the kernel messages would be collected with dmesg,
> given a sufficiently large CONFIG_LOG_BUF_SHIFT. But it would be
> inconvenient to have no serial console available for the usual purposes.

Can we disable the serial console after boot, by registering the dummy 
console? Or will that just log messages to both?

Cheers,

	Michael

Re: instrumentation, was Re: core dump analysis

[Finn Thain]

[-- Attachment #1: Type: text/plain, Size: 1461 bytes --]

On Tue, 11 Apr 2023, Michael Schmitz wrote:

> Am 11.04.2023 um 12:20 schrieb Finn Thain:
> > On Sun, 9 Apr 2023, Michael Schmitz wrote:
> >> Am 08.04.2023 um 00:06 schrieb Geert Uytterhoeven:
> >>> On Fri, Apr 7, 2023 at 3:58 AM Michael Schmitz <wrote:
> >>>> The easiest way to do that is to log all wait and signal syscalls, 
> >>>> as well as process exit. That might alter timing if these log 
> >>>> messages go to the serial console though. Is that what you have in 
> >>>> mind?
> >>>
> >>> Store to RAM, retrieve through a new /proc file?
> >>
> >> Yes, that could be done, though I'd rather avoid duplicating a lot of 
> >> the generic message formatting code (printk and friends).
> >>
> >> I'll have a look around ...
> >>
> >
> > A better solution might be be to port the existing instrumentation 
> > like ftrace, kprobes, uprobes etc. Might be a lot of work though. I 
> > wonder how portable that stuff is.
> >
> > If you use printk, you could probably avoid most of the delays by 
> > enabling the dummy console. Then the kernel messages would be 
> > collected with dmesg, given a sufficiently large CONFIG_LOG_BUF_SHIFT. 
> > But it would be inconvenient to have no serial console available for 
> > the usual purposes.
> 
> Can we disable the serial console after boot, by registering the dummy 
> console? Or will that just log messages to both?
> 

I don't know. I'd have to experiment in QEMU or Aranym.

Re: instrumentation, was Re: core dump analysis

[Geert Uytterhoeven]

Hi Michael,

On Tue, Apr 11, 2023 at 6:56 AM Michael Schmitz <schmitzmic@gmail.com> wrote:
> Am 11.04.2023 um 12:20 schrieb Finn Thain:
> > On Sun, 9 Apr 2023, Michael Schmitz wrote:
> >> Am 08.04.2023 um 00:06 schrieb Geert Uytterhoeven:
> >>> On Fri, Apr 7, 2023 at 3:58 AM Michael Schmitz <wrote:
> >>>> The easiest way to do that is to log all wait and signal syscalls, as
> >>>> well as process exit. That might alter timing if these log messages
> >>>> go to the serial console though. Is that what you have in mind?
> >>>
> >>> Store to RAM, retrieve through a new /proc file?
> >>
> >> Yes, that could be done, though I'd rather avoid duplicating a lot of
> >> the generic message formatting code (printk and friends).
> >>
> >> I'll have a look around ...
> >>
> >
> > A better solution might be be to port the existing instrumentation like
> > ftrace, kprobes, uprobes etc. Might be a lot of work though. I wonder
> > how portable that stuff is.
> >
> > If you use printk, you could probably avoid most of the delays by enabling
> > the dummy console. Then the kernel messages would be collected with dmesg,
> > given a sufficiently large CONFIG_LOG_BUF_SHIFT. But it would be
> > inconvenient to have no serial console available for the usual purposes.
>
> Can we disable the serial console after boot, by registering the dummy
> console? Or will that just log messages to both?

You can increase loglevel.

Gr{oetje,eeting}s,

                        Geert


--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: instrumentation, was Re: core dump analysis

[Michael Schmitz]

Hi Geert,

Am 11.04.2023 um 19:19 schrieb Geert Uytterhoeven:
> Hi Michael,
>
> On Tue, Apr 11, 2023 at 6:56 AM Michael Schmitz <schmitzmic@gmail.com> wrote:
>> Am 11.04.2023 um 12:20 schrieb Finn Thain:
>>> On Sun, 9 Apr 2023, Michael Schmitz wrote:
>>>> Am 08.04.2023 um 00:06 schrieb Geert Uytterhoeven:
>>>>> On Fri, Apr 7, 2023 at 3:58 AM Michael Schmitz <wrote:
>>>>>> The easiest way to do that is to log all wait and signal syscalls, as
>>>>>> well as process exit. That might alter timing if these log messages
>>>>>> go to the serial console though. Is that what you have in mind?
>>>>>
>>>>> Store to RAM, retrieve through a new /proc file?
>>>>
>>>> Yes, that could be done, though I'd rather avoid duplicating a lot of
>>>> the generic message formatting code (printk and friends).
>>>>
>>>> I'll have a look around ...
>>>>
>>>
>>> A better solution might be be to port the existing instrumentation like
>>> ftrace, kprobes, uprobes etc. Might be a lot of work though. I wonder
>>> how portable that stuff is.

Looking at a few arch implementations, I'm utterly confused. Wouldn't 
know where to start.

>>>
>>> If you use printk, you could probably avoid most of the delays by enabling
>>> the dummy console. Then the kernel messages would be collected with dmesg,
>>> given a sufficiently large CONFIG_LOG_BUF_SHIFT. But it would be
>>> inconvenient to have no serial console available for the usual purposes.
>>
>> Can we disable the serial console after boot, by registering the dummy
>> console? Or will that just log messages to both?
>
> You can increase loglevel.

Yes, we could do that. I thought /proc has to be mounted for the 
loglevel to be changed, but I might be wrong.

OTOH, if we log wait and signal actions at level debug, and set loglevel 
to info or notice through the command line, we won't miss much of the 
usual boot messages on the serial console ...

Cheers,

	Michael


>
> Gr{oetje,eeting}s,
>
>                         Geert
>
>
> --
> Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
>
> In personal conversations with technical people, I call myself a hacker. But
> when I'm talking to journalists I just say "programmer" or something like that.
>                                 -- Linus Torvalds
>

Re: instrumentation, was Re: core dump analysis

[Eero Tamminen]

Hi Michael,

On 11.4.2023 11.24, Michael Schmitz wrote:
> Am 11.04.2023 um 19:19 schrieb Geert Uytterhoeven:
>> On Tue, Apr 11, 2023 at 6:56 AM Michael Schmitz <schmitzmic@gmail.com> 
>>> Am 11.04.2023 um 12:20 schrieb Finn Thain:
>>>> A better solution might be be to port the existing instrumentation like
>>>> ftrace, kprobes, uprobes etc. Might be a lot of work though. I wonder
>>>> how portable that stuff is.

Another possibility is just using manually added kernel tracepoints:
https://www.kernel.org/doc/html/latest/trace/tracepoints.html

"git grep TRACE_EVENT" did not given any hits for arch/m68k/ though.

(I don't think those need any particular architecture support. One just 
adds them to locations where it makes sense to trace activity that is 
important, but which does not happen too often for the very small 
overhead from disabled static trace point to be significant.)


> Looking at a few arch implementations, I'm utterly confused. Wouldn't 
> know where to start.

Good article: https://lwn.net/Articles/132196/

(Ignore jprobes stuff as that's deprecated.)


Because probe replaces instruction at the probe point with a breakpoint:
https://www.kernel.org/doc/html/latest/trace/kprobes.html#how-does-a-kprobe-work

It needs architecture specific code to decode and simulate the replaced 
instruction (its side-effects) when it is later executed "out-of-line".

This also causes architecture specific limitations on where the probe 
can be set.  Architecture may not support simulating all possible 
instructions, just the most common ones (e.g. ones typically used on 
function entry and exit points).

Besides that, AFAIK plain kprobes support should not have much 
architecture specific stuff.


(I have no experience with kernel code, I just remember some LWN 
articles about kprobes during past 2 decades, and I was involved with a 
tool that did something similar in user-space, before Linux got uprobes 
support.)



As to what architecture to use as an example, I think it would be best 
to ask tracing maintainers for advice, as there seems to be a lot of 
code which has been copied from each other at different points in time, 
that might be nowadays better as shared rather than under each arch...

These architectures support kprobes:
------------------------------------
$ grep " ok " Documentation/features/debug/kprobes/arch-support.txt
     |         arc: |  ok  |
     |         arm: |  ok  |
     |       arm64: |  ok  |
     |        csky: |  ok  |
     |        ia64: |  ok  |
     |        mips: |  ok  |
     |      parisc: |  ok  |
     |     powerpc: |  ok  |
     |       riscv: |  ok  |
     |        s390: |  ok  |
     |          sh: |  ok  |
     |       sparc: |  ok  |
     |         x86: |  ok  |
------------------------------------

But only these seem to have probing functionality isolated nicely to its 
own directory:
------------------------------------
$ ls arch/*/kernel/probes/

arch/arm64/kernel/probes/:
decode-insn.c  decode-insn.h  kprobes.c  kprobes_trampoline.S  Makefile 
simulate-insn.c  simulate-insn.h  uprobes.c

arch/csky/kernel/probes/:
decode-insn.c  decode-insn.h  ftrace.c  kprobes.c  kprobes_trampoline.S 
Makefile  simulate-insn.c  simulate-insn.h  uprobes.c

arch/riscv/kernel/probes/:
decode-insn.c  decode-insn.h  ftrace.c  kprobes.c  Makefile  rethook.c 
rethook.h  rethook_trampoline.S  simulate-insn.c  simulate-insn.h  uprobes.c
------------------------------------


As to testing whether kprobes work, that could be easier with ftrace 
support also present:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/testing/selftests/ftrace/README


	- Eero

PS. Looking at the kernel features list, m68k arch seems to be missing a 
lot of other features too:
https://www.kernel.org/doc/html/latest/m68k/features.html

Re: instrumentation, was Re: core dump analysis

[Eero Tamminen]

Hi,

On 13.4.2023 2.22, Eero Tamminen wrote:
> On 11.4.2023 11.24, Michael Schmitz wrote:
>> Am 11.04.2023 um 19:19 schrieb Geert Uytterhoeven:
>>> On Tue, Apr 11, 2023 at 6:56 AM Michael Schmitz <schmitzmic@gmail.com>
>>>> Am 11.04.2023 um 12:20 schrieb Finn Thain:
>>>>> A better solution might be be to port the existing instrumentation 
>>>>> like
>>>>> ftrace, kprobes, uprobes etc. Might be a lot of work though. I wonder
>>>>> how portable that stuff is.
> 
> Another possibility is just using manually added kernel tracepoints:
> https://www.kernel.org/doc/html/latest/trace/tracepoints.html
> 
> "git grep TRACE_EVENT" did not given any hits for arch/m68k/ though.
> 
> (I don't think those need any particular architecture support. One just 
> adds them to locations where it makes sense to trace activity that is 
> important, but which does not happen too often for the very small 
> overhead from disabled static trace point to be significant.)
> 
> 
>> Looking at a few arch implementations, I'm utterly confused. Wouldn't 
>> know where to start.
> 
> Good article: https://lwn.net/Articles/132196/
> 
> (Ignore jprobes stuff as that's deprecated.)
> 
> 
> Because probe replaces instruction at the probe point with a breakpoint:
> https://www.kernel.org/doc/html/latest/trace/kprobes.html#how-does-a-kprobe-work
> 
> It needs architecture specific code to decode and simulate the replaced 
> instruction (its side-effects) when it is later executed "out-of-line".
> 
> This also causes architecture specific limitations on where the probe 
> can be set.  Architecture may not support simulating all possible 
> instructions, just the most common ones (e.g. ones typically used on 
> function entry and exit points).

These are the kernel symbol entry point instructions that get executed 
when booting Atari Falcon from IDE (using emulator LILO) to Busybox init:
    4524 link.w
       4 movea.l
       3 move.l
       3 clr.l
       2 movem.l
       2 dc.w
       2 btst.b
       1 subq.l
       1 ori.w
       1 move.w
       1 lea.l
       1 jsr
       1 bra.b

For Linus' 6.2 kernel built with this config:
https://git.tuxfamily.org/hatari/hatari.git/tree/tools/linux/kernel.config

Using "gcc-m68k-linux-gnu" (4:10.2.1-1) in Debian Stable.


Count is for how many of the executed kernel symbol addresses had given 
instruction.

I'm a bit surprised how many functions (symbol addresses) start with 
link.w instruction, instead of e.g. move* instruction.

(dc.w is check for emulator NatFeats API.)


	- Eero

PS. Above info is produced by building & booting Linux under Hatari 
emulator as documented here:
https://hatari.tuxfamily.org/doc/m68k-linux.txt

And using:
-------------------
hatari ... --parse sym-breakpoint.ini > instr.txt
-------------------

Where "sym-breakpoint.ini" is:
-------------------
b PConSymbol > 0 :trace :quiet :file pc-disasm.ini
-------------------

And "pc-disasm.ini" file is:
-------------------
d pc-"pc+2"
-------------------

Then to get counts from the output:
-------------------
$ cat  instr.txt | grep -E -e '^\$' | sort -u |\
   cut -b46-52 | sort | uniq -c | sort -nr
-------------------

(Match disassembly lines, drop duplicate function calls, count uniq 
entry point instructions.)


> Besides that, AFAIK plain kprobes support should not have much 
> architecture specific stuff.
> 
> 
> (I have no experience with kernel code, I just remember some LWN 
> articles about kprobes during past 2 decades, and I was involved with a 
> tool that did something similar in user-space, before Linux got uprobes 
> support.)
> 
> 
> 
> As to what architecture to use as an example, I think it would be best 
> to ask tracing maintainers for advice, as there seems to be a lot of 
> code which has been copied from each other at different points in time, 
> that might be nowadays better as shared rather than under each arch...
> 
> These architectures support kprobes:
> ------------------------------------
> $ grep " ok " Documentation/features/debug/kprobes/arch-support.txt
>      |         arc: |  ok  |
>      |         arm: |  ok  |
>      |       arm64: |  ok  |
>      |        csky: |  ok  |
>      |        ia64: |  ok  |
>      |        mips: |  ok  |
>      |      parisc: |  ok  |
>      |     powerpc: |  ok  |
>      |       riscv: |  ok  |
>      |        s390: |  ok  |
>      |          sh: |  ok  |
>      |       sparc: |  ok  |
>      |         x86: |  ok  |
> ------------------------------------
> 
> But only these seem to have probing functionality isolated nicely to its 
> own directory:
> ------------------------------------
> $ ls arch/*/kernel/probes/
> 
> arch/arm64/kernel/probes/:
> decode-insn.c  decode-insn.h  kprobes.c  kprobes_trampoline.S  Makefile 
> simulate-insn.c  simulate-insn.h  uprobes.c
> 
> arch/csky/kernel/probes/:
> decode-insn.c  decode-insn.h  ftrace.c  kprobes.c  kprobes_trampoline.S 
> Makefile  simulate-insn.c  simulate-insn.h  uprobes.c
> 
> arch/riscv/kernel/probes/:
> decode-insn.c  decode-insn.h  ftrace.c  kprobes.c  Makefile  rethook.c 
> rethook.h  rethook_trampoline.S  simulate-insn.c  simulate-insn.h  
> uprobes.c
> ------------------------------------
> 
> 
> As to testing whether kprobes work, that could be easier with ftrace 
> support also present:
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/testing/selftests/ftrace/README
> 
> 
>      - Eero
> 
> PS. Looking at the kernel features list, m68k arch seems to be missing a 
> lot of other features too:
> https://www.kernel.org/doc/html/latest/m68k/features.html
> 

Re: instrumentation, was Re: core dump analysis

[Brad Boyer]

On Sat, Apr 15, 2023 at 06:50:08PM +0300, Eero Tamminen wrote:
> I'm a bit surprised how many functions (symbol addresses) start with link.w
> instruction, instead of e.g. move* instruction.

That seems pretty normal to me. The LINK instruction sets up a stack
frame for a function. It takes an address register and size (the .w
suffix indicates a 16-bit size, while .l would indicate a 32-bit value
of size). The old value of the register is pushed to the stack, the
current stack pointer is saved into the given register, then the stack
is increased by the given size. There is a corresponding UNLK
instruction to tear down the stack frame which will frequently be the
last instruction before returning to the caller. The compiler will use
these in the function prologue and epilogue in all but the most
trivial functions.

	Brad Boyer
	flar@allandria.com

Re: core dump analysis, was Re: stack smashing detected

[Finn Thain]

On Wed, 5 Apr 2023, I wrote:

> 
> I don't care that much what dash does as long as it isn't corrupting 
> it's own stack, which is a real possibility, and one which gdb's data 
> watch point would normally resolve. And yet I have no way to tackle 
> that.
> 
> I've been running gdb under QEMU, where the failure is not reproducible. 
> Running dash under gdb on real hardware is doable (RAM permitting). But 
> the failure is intermittent even then -- it only happens during 
> execution of certain init scripts, and I can't reproduce it by manually 
> running those scripts.
> 
> (Even if I could reproduce the failure under gdb, instrumenting 
> execution in gdb can alter timing in undesirable ways...)
> 

Somewhat optimistically, I upgraded the RAM on this system to 36 MB so I 
can run dash under gdb (20 MB was not enough). But, as expected, the crash 
went away when I did so.

Outside of gdb, I was able to reproduce the same failure with a clean 
build from the dash repo (commit b00288f). I can get a crash with 
optimization level -O1 and -O though it becomes even more rare. So it's 
easier to use Debian's build (-O2).

One of the difficulties with the core dump is that it happens too late. 
After the canary check fails, __stack_chk_fail() is called, which then 
calls a bunch of other stuff until finally abort() is called. This 
obliterates whatever was below the stack pointer at the time of the 
failure.

So I modified libc.so.6 and now it just crashes with an illegal 
instruction in __wait3 rather than branching to __stack_chk_fail. This let 
me see whatever was left behind in stack memory by __wait4_time64() etc.

__wait4_time64() calls __m68k_read_tp(), and the return address from 
__m68k_read_tp() can still be seen in stack memory, which suggests that 
the stack never grew after that call. (So __m68k_read_tp() is implicated.)

Would signal delivery erase any of the memory immediately below the USP? 
If so, it would erase those old stack frames, which would give some 
indication of the timing of signal delivery.

If I run dash under gdb under QEMU, I can break on entry to onsig() and 
find the signal frame on the stack. But when I examine stack memory from 
the core dump, I can't find 0x70774e40 (i.e. moveq __NR_sigreturn,%d0 ; 
trap #0) which the kernel puts on the stack in my QEMU experiments.

That suggests that no signal was delivered... and yet gotsigchld == 1 at 
the time of the coredump, after having been initialized by waitproc() 
prior to calling __wait3(). So the signal handler onsig() must have 
executed during __wait3() or __wait4_time64(). I can't explain this.

Re: core dump analysis, was Re: stack smashing detected

[Michael Schmitz]

Hi Finn,

Am 14.04.2023 um 21:30 schrieb Finn Thain:
> Would signal delivery erase any of the memory immediately below the USP?
> If so, it would erase those old stack frames, which would give some
> indication of the timing of signal delivery.

The signal stack is set up immediately below USP, from my reading of 
signal.c:setup_frame(). Old stack frames will be overwritten.

>
> If I run dash under gdb under QEMU, I can break on entry to onsig() and
> find the signal frame on the stack. But when I examine stack memory from
> the core dump, I can't find 0x70774e40 (i.e. moveq __NR_sigreturn,%d0 ;
> trap #0) which the kernel puts on the stack in my QEMU experiments.

As I understand this, the call to sys_sigreturn() removes both this code 
(signal trampoline IIRC) and the signal stack. At that point, the kernel 
signal handler (which in turn calls onsig()) has already been run.

Again as far as I understand, the core dump happens on process exit. 
Stack smashing is detected and process exit is forced only at exit from 
__wait3() or __wait4_time64(), and those would have only been woken by 
receiving a signal. Both signal delivery and removal of signal stack by 
sys_sigreturn() must have happened before return from the wait syscalls.

Cheers,

	Michael


> That suggests that no signal was delivered... and yet gotsigchld == 1 at
> the time of the coredump, after having been initialized by waitproc()
> prior to calling __wait3(). So the signal handler onsig() must have
> executed during __wait3() or __wait4_time64(). I can't explain this.

Re: core dump analysis, was Re: stack smashing detected

[Finn Thain]

On Sun, 16 Apr 2023, Michael Schmitz wrote:

> Am 14.04.2023 um 21:30 schrieb Finn Thain:
> > Would signal delivery erase any of the memory immediately below the 
> > USP? If so, it would erase those old stack frames, which would give 
> > some indication of the timing of signal delivery.
> 
> The signal stack is set up immediately below USP, from my reading of 
> signal.c:setup_frame(). Old stack frames will be overwritten.
> 

OK.

> >
> > If I run dash under gdb under QEMU, I can break on entry to onsig() 
> > and find the signal frame on the stack. But when I examine stack 
> > memory from the core dump, I can't find 0x70774e40 (i.e. moveq 
> > __NR_sigreturn,%d0 ; trap #0) which the kernel puts on the stack in my 
> > QEMU experiments.
> 
> As I understand this, the call to sys_sigreturn() removes both this code 
> (signal trampoline IIRC) and the signal stack...

I don't see that stuff getting removed when I run dash under gdb under 
QEMU. With breakpoints at the head of onsig() and the tail of __wait3(), 
the memory under USP is the same when examined at either juncture.

The backtrace confirms that this signal was delivered during execution of 
__wait3(). (Delivery can happen during execution of __libc_fork() but I 
just repeat the test until I get these ducks in a row.)

(gdb) c
Continuing.
# x=$(:)
[Detaching after fork from child process 1055]

Breakpoint 6.1, onsig (signo=17) at trap.c:286
286     trap.c: No such file or directory.
(gdb) bt
#0  onsig (signo=17) at trap.c:286
#1  <signal handler called>
#2  0xc00e81b6 in __GI___wait4_time64 (pid=-1, stat_loc=0xeffff86a, options=2, 
    usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:35
#3  0xc00e8164 in __GI___wait3_time64 (usage=0x0, options=<optimized out>, 
    stat_loc=<optimized out>) at ../sysdeps/unix/sysv/linux/wait3.c:26
#4  __wait3 (stat_loc=<optimized out>, options=<optimized out>, 
    usage=<optimized out>) at ../sysdeps/unix/sysv/linux/wait3.c:35
#5  0xd000c38e in waitproc (status=0xeffff85a, block=1) at jobs.c:1179
#6  waitone (block=1, job=0xd001f618) at jobs.c:1055
#7  0xd000c5b8 in dowait (block=1, jp=0xd001f618) at jobs.c:1137
#8  0xd000ddb0 in waitforjob (jp=0xd001f618) at jobs.c:1014
#9  0xd000aade in expbackq (flag=68, cmd=0xd001e4c8 <stackbase+36>)
    at expand.c:520
#10 argstr (p=<optimized out>, flag=68) at expand.c:335
#11 0xd000b5ce in expandarg (arg=0xd001e4e8 <stackbase+68>, 
    arglist=0xeffffb08, flag=4) at expand.c:192
#12 0xd0007e2a in evalcommand (cmd=<optimized out>, flags=<optimized out>)
    at eval.c:855
#13 0xd0006ffc in evaltree (n=0xd001e4f8 <stackbase+84>, flags=0) at eval.c:300
#14 0xd000e3c0 in cmdloop (top=1) at main.c:246
#15 0xd0005018 in main (argc=<optimized out>, argv=<optimized out>)
    at main.c:181


0xeffff750:     0xc01a0000                      saved $a5 == libc .got
0xeffff74c:     0xc0023e8c                      saved $a3 == &__stack_chk_guard
0xeffff748:     0x00000000                      saved $a2
0xeffff744:     0x00000001                      saved $d5
0xeffff740:     0xeffff86e                      saved $d4
0xeffff73c:     0xeffff86a                      saved $d3
0xeffff738:     0x00000002                      saved $d2 
0xeffff734:     0x00000000
0xeffff730:     0x00000000
0xeffff72c:     0x00000000
0xeffff728:     0x00000000
0xeffff724:     0x00000000
0xeffff720:     0x00000000
0xeffff71c:     0x00000000
0xeffff718:     0x00000000
0xeffff714:     0x00000000
0xeffff710:     0x00000000
0xeffff70c:     0x00000000
0xeffff708:     0x00000000
0xeffff704:     0x00000000
0xeffff700:     0x00000000
0xeffff6fc:     0x00000000
0xeffff6f8:     0x00000000
0xeffff6f4:     0x00000000
0xeffff6f0:     0x00000000
0xeffff6ec:     0x00000000
0xeffff6e8:     0x00000000
0xeffff6e4:     0x00000000
0xeffff6e0:     0x00000000
0xeffff6dc:     0x00000000
0xeffff6d8:     0x00000000
0xeffff6d4:     0x00000000
0xeffff6d0:     0x00000000
0xeffff6cc:     0x00000000
0xeffff6c8:     0x00000000
0xeffff6c4:     0x00000000
0xeffff6c0:     0x00000000
0xeffff6bc:     0x00000000
0xeffff6b8:     0x00000000
0xeffff6b4:     0x00000000
0xeffff6b0:     0x00000000
0xeffff6ac:     0x00000000
0xeffff6a8:     0x00000000
0xeffff6a4:     0x00000000
0xeffff6a0:     0x00000000
0xeffff69c:     0x00000000
0xeffff698:     0x00000000
0xeffff694:     0x00000000
0xeffff690:     0x00000000
0xeffff68c:     0x00000000
0xeffff688:     0x00000000
0xeffff684:     0x00000000
0xeffff680:     0x00000000
0xeffff67c:     0x00000000
0xeffff678:     0x00000000
0xeffff674:     0x00000000
0xeffff670:     0x00000000
0xeffff66c:     0x00000000
0xeffff668:     0x00000000
0xeffff664:     0x00000000
0xeffff660:     0x41000000
0xeffff65c:     0x00000000
0xeffff658:     0x00000000
0xeffff654:     0x00000000
0xeffff650:     0x00000000
0xeffff64c:     0x80000000
0xeffff648:     0x3fff0000
0xeffff644:     0x00000000
0xeffff640:     0xd0000000
0xeffff63c:     0x40020000
0xeffff638:     0x81b60080      
0xeffff634:     0x0000c00e
0xeffff630:     0xd001e4e3                               saved a1?
0xeffff62c:     0xc0028780                               saved a0?
0xeffff628:     0xffffffff                               saved d1?
0xeffff624:     0x0000041f                               saved d0?
0xeffff620:     0xeffff738                               saved sp?
0xeffff61c:     0x00000000      
0xeffff618:     0x00000000
0xeffff614:     0x00000000
0xeffff610:     0x70774e40      moveq #119,%d0 ; trap #0
0xeffff60c:     0xeffff61c
0xeffff608:     0x00000080
0xeffff604:     0x00000011
0xeffff600:     0xeffff610      return address

The above comes from dash running under gdb under qemu, which does not 
exhibit the failure but is convenient for that kind of experiment.

> Again as far as I understand, the core dump happens on process exit. 
> Stack smashing is detected and process exit is forced only at exit from 
> __wait3() or __wait4_time64(), 

I placed an illegal instruction in __wait3. This executes instead of the 
call to __stack_chk_fail because that obliterates stack memory of 
interest.

Consequently the latest core dump still contains dead stack frames (see 
below) of subroutines that returned before __wait3() dumped core. You can 
see the return address for the branch to __wait4_time64() and below that 
you can see the return address for the branch to __m68k_read_tp().

(gdb) disas __wait4_time64
Dump of assembler code for function __GI___wait4_time64:
   0xc00e4174 <+0>:     lea %sp@(-80),%sp
   0xc00e4178 <+4>:     moveml %d2-%d5/%a2-%a3/%a5,%sp@-
   0xc00e417c <+8>:     lea %pc@(0xc019c000),%a5
   0xc00e4184 <+16>:    movel %sp@(116),%d2
   0xc00e4188 <+20>:    moveal %sp@(124),%a2
   0xc00e418c <+24>:    moveal %a5@(108),%a3
   0xc00e4190 <+28>:    movel %a3@,%sp@(104)
   0xc00e4194 <+32>:    bsrl 0xc0056e2c <__m68k_read_tp@plt>

I gather the signal was delivered before __wait4_time64+38, otherwise the 
return address 0xc00e419a (which appears below) would have been 
overwritten by the signal frame. The signal must have been delivered after 
waitproc() initialized gotsigchld = 0 since gotsigchld is 1 at the time of 
the coredump.

I assume the %a3 corruption happened after __wait4_time64+8 because that's 
when %a3 first appears on the stack. And the corruption must have happened 
before __wait4_time64+238, which is when %a3 was restored.

If it was the signal which somehow corrupted the saved %a3, there's only a 
small window for that. The only syscall in that window is get_thread_area.

Here's some stack memory from the core dump.

0xeffff0dc:     0xd000c38e      return address waitproc+124
0xeffff0d8:     0xd001c1ec      frame 0 $fp                   == &suppressint
0xeffff0d4:     0x00add14b      canary
0xeffff0d0:     0x00000000
0xeffff0cc:     0x0000000a
0xeffff0c8:     0x00000202
0xeffff0c4:     0x00000008
0xeffff0c0:     0x00000000
0xeffff0bc:     0x00000000
0xeffff0b8:     0x00000174
0xeffff0b4:     0x00000004
0xeffff0b0:     0x00000004
0xeffff0ac:     0x00000006
0xeffff0a8:     0x000000e0
0xeffff0a4:     0x000000e0
0xeffff0a0:     0x00171f20
0xeffff09c:     0x00171f20
0xeffff098:     0x00171f20
0xeffff094:     0x00000002
0xeffff090:     0x00002000
0xeffff08c:     0x00000006
0xeffff088:     0x0000e920
0xeffff084:     0x00005360
0xeffff080:     0x00170700
0xeffff07c:     0x00170700
0xeffff078:     0x00170700      frame 0 $fp - 96
0xeffff074:     0xd001b874                         saved $a5 == dash .got
0xeffff070:     0xd001e498                         saved $a3 == &dash_errno
0xeffff06c:     0xd001e718      frame 0 $sp        saved $a2 == &gotsigchld
0xeffff068:     0x00000000
0xeffff064:     0x00000000
0xeffff060:     0xeffff11e    
0xeffff05c:     0xffffffff    
0xeffff058:     0xc00e4164      return address __wait3+244
0xeffff054:     0x00add14b      canary
0xeffff050:     0x00000001
0xeffff04c:     0x00000004
0xeffff048:     0x0000000d
0xeffff044:     0x0000000d
0xeffff040:     0x0015ef82
0xeffff03c:     0x0015ef82
0xeffff038:     0x0015ef82
0xeffff034:     0x00000003
0xeffff030:     0x00000004
0xeffff02c:     0x00000004
0xeffff028:     0x00000140
0xeffff024:     0x00000140
0xeffff020:     0x00000034
0xeffff01c:     0x00000034
0xeffff018:     0x00000034
0xeffff014:     0x00000006
0xeffff010:     0x003b003a
0xeffff00c:     0x000a0028    
0xeffff008:     0x00340020      
0xeffff004:     0xc019c000                      saved $a5 == libc .got
0xeffff000:     0xeffff068                      saved $a3 (corrupted)
0xefffeffc:     0x00000000                      saved $a2
0xefffeff8:     0x00000001                      saved $d5
0xefffeff4:     0xeffff122                      saved $d4
0xefffeff0:     0xeffff11e                      saved $d3
0xefffefec:     0x00000000                      saved $d2
0xefffefe8:     0xc00e419a      return address __GI___wait4_time64+38
0xefffefe4:     0xc0028780
0xefffefe0:     0x3c344bfb
0xefffefdc:     0x000af353
0xefffefd8:     0x3c340170
0xefffefd4:     0x00000000
0xefffefd0:     0xc00e417c
0xefffefcc:     0xc00e417e
0xefffefc8:     0xc00e4180
0xefffefc4:     0x48e73c34
0xefffefc0:     0x00000000
0xefffefbc:     0xefffeff8
0xefffefb8:     0xefffeffc
0xefffefb4:     0x4bfb0170
0xefffefb0:     0x0eee0709
0xefffefac:     0x00000000
0xefffefa8:     0x00000000
0xefffefa4:     0x00000000
0xefffefa0:     0x00000000
0xefffef9c:     0x00000000
0xefffef98:     0x00000000
0xefffef94:     0x00000000
0xefffef90:     0x00000000
0xefffef8c:     0x00000000
0xefffef88:     0x00000000
0xefffef84:     0x00000000
0xefffef80:     0x00000000
0xefffef7c:     0x00000000
0xefffef78:     0x00000000
0xefffef74:     0x00000000
0xefffef70:     0x00000000
0xefffef6c:     0x00000000
0xefffef68:     0x00000000
0xefffef64:     0x00000000
0xefffef60:     0x00000000
0xefffef5c:     0x00000000
0xefffef58:     0x00000000
0xefffef54:     0x00000000
0xefffef50:     0x00000000
0xefffef4c:     0x00000000
0xefffef48:     0x00000000
0xefffef44:     0x00000000
0xefffef40:     0x00000000
0xefffef3c:     0x00000000
0xefffef38:     0x00000000
0xefffef34:     0x00000000
0xefffef30:     0x00000000
0xefffef2c:     0x00000000
0xefffef28:     0x00000000
0xefffef24:     0x00000000
0xefffef20:     0x00000000
0xefffef1c:     0x00000000
0xefffef18:     0x00000000
0xefffef14:     0x00000000
0xefffef10:     0x7c0effff
0xefffef0c:     0xffffffff
0xefffef08:     0xaaaaaaaa
0xefffef04:     0xaf54eaaa
0xefffef00:     0x40040000
0xefffeefc:     0x40040000
0xefffeef8:     0x2b000000
0xefffeef4:     0x00000000
0xefffeef0:     0x00000000
0xefffeeec:     0x408ece9a
0xefffeee8:     0x00000000
0xefffeee4:     0xf0ff0000
0xefffeee0:     0x0f800000
0xefffeedc:     0xf0fff0ff
0xefffeed8:     0x1f380000
0xefffeed4:     0x00000000
0xefffeed0:     0x00000000
0xefffeecc:     0x00000000
0xefffeec8:     0xffffffff
0xefffeec4:     0xffffffff
0xefffeec0:     0x7fff0000
0xefffeebc:     0xffffffff
0xefffeeb8:     0xffffffff
0xefffeeb4:     0x7fff0000

The signal frame is not readily apparent (to me).

Also, stack memory in the core dump and stack memory as observed upon 
signal handler breakpoint do not agree very well... I'd expect the values 
immediately below the stack pointer to be zeros once the signal frame was 
put there.

I can't explain all of that unless it's discontiguous stack corruption.

Re: core dump analysis, was Re: stack smashing detected

[Michael Schmitz]

Hi Finn,

Am 16.04.2023 um 18:44 schrieb Finn Thain:
>> As I understand this, the call to sys_sigreturn() removes both this code
>> (signal trampoline IIRC) and the signal stack...
>
> I don't see that stuff getting removed when I run dash under gdb under
> QEMU. With breakpoints at the head of onsig() and the tail of __wait3(),
> the memory under USP is the same when examined at either juncture.

OK - with usp restored to what it was before the signal stack was added, 
the stack below original usp can be left as is (will be overwritten 
eventually).

>
> The backtrace confirms that this signal was delivered during execution of
> __wait3(). (Delivery can happen during execution of __libc_fork() but I
> just repeat the test until I get these ducks in a row.)
>
> (gdb) c
> Continuing.
> # x=$(:)
> [Detaching after fork from child process 1055]
>
> Breakpoint 6.1, onsig (signo=17) at trap.c:286
> 286     trap.c: No such file or directory.
> (gdb) bt
> #0  onsig (signo=17) at trap.c:286
> #1  <signal handler called>
> #2  0xc00e81b6 in __GI___wait4_time64 (pid=-1, stat_loc=0xeffff86a, options=2,
>     usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:35
> #3  0xc00e8164 in __GI___wait3_time64 (usage=0x0, options=<optimized out>,
>     stat_loc=<optimized out>) at ../sysdeps/unix/sysv/linux/wait3.c:26

Where did that one come from? I don't think we saw __GI___wait3_time64 
called in your disassembly of __wait3 ...

> #4  __wait3 (stat_loc=<optimized out>, options=<optimized out>,
>     usage=<optimized out>) at ../sysdeps/unix/sysv/linux/wait3.c:35
> #5  0xd000c38e in waitproc (status=0xeffff85a, block=1) at jobs.c:1179
> #6  waitone (block=1, job=0xd001f618) at jobs.c:1055
> #7  0xd000c5b8 in dowait (block=1, jp=0xd001f618) at jobs.c:1137
> #8  0xd000ddb0 in waitforjob (jp=0xd001f618) at jobs.c:1014
> #9  0xd000aade in expbackq (flag=68, cmd=0xd001e4c8 <stackbase+36>)
>     at expand.c:520
> #10 argstr (p=<optimized out>, flag=68) at expand.c:335
> #11 0xd000b5ce in expandarg (arg=0xd001e4e8 <stackbase+68>,
>     arglist=0xeffffb08, flag=4) at expand.c:192
> #12 0xd0007e2a in evalcommand (cmd=<optimized out>, flags=<optimized out>)
>     at eval.c:855
> #13 0xd0006ffc in evaltree (n=0xd001e4f8 <stackbase+84>, flags=0) at eval.c:300
> #14 0xd000e3c0 in cmdloop (top=1) at main.c:246
> #15 0xd0005018 in main (argc=<optimized out>, argv=<optimized out>)
>     at main.c:181
>
>
> 0xeffff750:     0xc01a0000                      saved $a5 == libc .got
> 0xeffff74c:     0xc0023e8c                      saved $a3 == &__stack_chk_guard
> 0xeffff748:     0x00000000                      saved $a2
> 0xeffff744:     0x00000001                      saved $d5
> 0xeffff740:     0xeffff86e                      saved $d4
> 0xeffff73c:     0xeffff86a                      saved $d3
> 0xeffff738:     0x00000002                      saved $d2
> 0xeffff734:     0x00000000
> 0xeffff730:     0x00000000
> 0xeffff72c:     0x00000000
> 0xeffff728:     0x00000000
> 0xeffff724:     0x00000000
> 0xeffff720:     0x00000000
> 0xeffff71c:     0x00000000
> 0xeffff718:     0x00000000
> 0xeffff714:     0x00000000
> 0xeffff710:     0x00000000
> 0xeffff70c:     0x00000000
> 0xeffff708:     0x00000000
> 0xeffff704:     0x00000000
> 0xeffff700:     0x00000000
> 0xeffff6fc:     0x00000000
> 0xeffff6f8:     0x00000000
> 0xeffff6f4:     0x00000000
> 0xeffff6f0:     0x00000000
> 0xeffff6ec:     0x00000000
> 0xeffff6e8:     0x00000000
> 0xeffff6e4:     0x00000000
> 0xeffff6e0:     0x00000000
> 0xeffff6dc:     0x00000000
> 0xeffff6d8:     0x00000000
> 0xeffff6d4:     0x00000000
> 0xeffff6d0:     0x00000000
> 0xeffff6cc:     0x00000000
> 0xeffff6c8:     0x00000000
> 0xeffff6c4:     0x00000000
> 0xeffff6c0:     0x00000000
> 0xeffff6bc:     0x00000000
> 0xeffff6b8:     0x00000000
> 0xeffff6b4:     0x00000000
> 0xeffff6b0:     0x00000000
> 0xeffff6ac:     0x00000000
> 0xeffff6a8:     0x00000000
> 0xeffff6a4:     0x00000000
> 0xeffff6a0:     0x00000000
> 0xeffff69c:     0x00000000
> 0xeffff698:     0x00000000
> 0xeffff694:     0x00000000
> 0xeffff690:     0x00000000
> 0xeffff68c:     0x00000000
> 0xeffff688:     0x00000000
> 0xeffff684:     0x00000000
> 0xeffff680:     0x00000000
> 0xeffff67c:     0x00000000
> 0xeffff678:     0x00000000
> 0xeffff674:     0x00000000
> 0xeffff670:     0x00000000
> 0xeffff66c:     0x00000000
> 0xeffff668:     0x00000000
> 0xeffff664:     0x00000000
> 0xeffff660:     0x41000000
> 0xeffff65c:     0x00000000
> 0xeffff658:     0x00000000
> 0xeffff654:     0x00000000
> 0xeffff650:     0x00000000
> 0xeffff64c:     0x80000000
> 0xeffff648:     0x3fff0000
> 0xeffff644:     0x00000000
> 0xeffff640:     0xd0000000
> 0xeffff63c:     0x40020000				<= (sc.formatvec& 0xffff) << 16; fpregs from here on
> 0xeffff638:     0x81b60080      			<= (sc.pc & 0xffff) << 16 | sc.formatvec >> 16
> 0xeffff634:     0x0000c00e				<= sc.sr << 16  sc.pc >> 16
> 0xeffff630:     0xd001e4e3                               saved a1?	<= sc.a1
> 0xeffff62c:     0xc0028780                               saved a0?	<= sc.a0
> 0xeffff628:     0xffffffff                               saved d1?	<= sc.d1
> 0xeffff624:     0x0000041f                               saved d0?	<= sc.d0
> 0xeffff620:     0xeffff738                               saved sp?	<= sc.usp
> 0xeffff61c:     0x00000000      			<= sc.mask
> 0xeffff618:     0x00000000				<= extramask
> 0xeffff614:     0x00000000				<= frame.retcode[1]
> 0xeffff610:     0x70774e40      moveq #119,%d0 ; trap #0
> 0xeffff60c:     0xeffff61c				<= frame->sc
> 0xeffff608:     0x00000080				<= tregs->vector
> 0xeffff604:     0x00000011				<= signal no.
> 0xeffff600:     0xeffff610      return address
>
> The above comes from dash running under gdb under qemu, which does not
> exhibit the failure but is convenient for that kind of experiment.

I would have expected to see a different signal trampoline (for 
sys_rt_sigreturn) ... But anyway:

The saved pc is 0xc00e81b6 which does match the backtrace above. Vector 
offset 80 matches trap 0 which suggests 0xc00e81b6 should be the 
instruction after a trap 0 instruction. d0 is 1055 which is not a signal 
number I recognize.

>
>> Again as far as I understand, the core dump happens on process exit.
>> Stack smashing is detected and process exit is forced only at exit from
>> __wait3() or __wait4_time64(),
>
> I placed an illegal instruction in __wait3. This executes instead of the
> call to __stack_chk_fail because that obliterates stack memory of
> interest.

OK.

>
> Consequently the latest core dump still contains dead stack frames (see
> below) of subroutines that returned before __wait3() dumped core. You can
> see the return address for the branch to __wait4_time64() and below that
> you can see the return address for the branch to __m68k_read_tp().
>
> (gdb) disas __wait4_time64
> Dump of assembler code for function __GI___wait4_time64:
>    0xc00e4174 <+0>:     lea %sp@(-80),%sp
>    0xc00e4178 <+4>:     moveml %d2-%d5/%a2-%a3/%a5,%sp@-
>    0xc00e417c <+8>:     lea %pc@(0xc019c000),%a5
>    0xc00e4184 <+16>:    movel %sp@(116),%d2
>    0xc00e4188 <+20>:    moveal %sp@(124),%a2
>    0xc00e418c <+24>:    moveal %a5@(108),%a3
>    0xc00e4190 <+28>:    movel %a3@,%sp@(104)
>    0xc00e4194 <+32>:    bsrl 0xc0056e2c <__m68k_read_tp@plt>
>
> I gather the signal was delivered before __wait4_time64+38, otherwise the
> return address 0xc00e419a (which appears below) would have been
> overwritten by the signal frame. The signal must have been delivered after
> waitproc() initialized gotsigchld = 0 since gotsigchld is 1 at the time of
> the coredump.
>
> I assume the %a3 corruption happened after __wait4_time64+8 because that's
> when %a3 first appears on the stack. And the corruption must have happened
> before __wait4_time64+238, which is when %a3 was restored.
>
> If it was the signal which somehow corrupted the saved %a3, there's only a
> small window for that. The only syscall in that window is get_thread_area.

I see sys_wait4 called in two places (0xc00e01b4, and then 0xc00e0286 
depending on the return code of the first). The second one again would 
have called  __m68k_read_tp so would have left a return address on the 
stack (0xc00e02d2). Leaves the first.

>
> Here's some stack memory from the core dump.
>
> 0xeffff0dc:     0xd000c38e      return address waitproc+124
> 0xeffff0d8:     0xd001c1ec      frame 0 $fp                   == &suppressint
> 0xeffff0d4:     0x00add14b      canary
> 0xeffff0d0:     0x00000000
> 0xeffff0cc:     0x0000000a
> 0xeffff0c8:     0x00000202
> 0xeffff0c4:     0x00000008
> 0xeffff0c0:     0x00000000
> 0xeffff0bc:     0x00000000
> 0xeffff0b8:     0x00000174
> 0xeffff0b4:     0x00000004
> 0xeffff0b0:     0x00000004
> 0xeffff0ac:     0x00000006
> 0xeffff0a8:     0x000000e0
> 0xeffff0a4:     0x000000e0
> 0xeffff0a0:     0x00171f20
> 0xeffff09c:     0x00171f20
> 0xeffff098:     0x00171f20
> 0xeffff094:     0x00000002
> 0xeffff090:     0x00002000
> 0xeffff08c:     0x00000006
> 0xeffff088:     0x0000e920
> 0xeffff084:     0x00005360
> 0xeffff080:     0x00170700
> 0xeffff07c:     0x00170700
> 0xeffff078:     0x00170700      frame 0 $fp - 96
> 0xeffff074:     0xd001b874                         saved $a5 == dash .got
> 0xeffff070:     0xd001e498                         saved $a3 == &dash_errno
> 0xeffff06c:     0xd001e718      frame 0 $sp        saved $a2 == &gotsigchld
> 0xeffff068:     0x00000000
> 0xeffff064:     0x00000000
> 0xeffff060:     0xeffff11e
> 0xeffff05c:     0xffffffff
> 0xeffff058:     0xc00e4164      return address __wait3+244
> 0xeffff054:     0x00add14b      canary
> 0xeffff050:     0x00000001
> 0xeffff04c:     0x00000004
> 0xeffff048:     0x0000000d
> 0xeffff044:     0x0000000d
> 0xeffff040:     0x0015ef82
> 0xeffff03c:     0x0015ef82
> 0xeffff038:     0x0015ef82
> 0xeffff034:     0x00000003
> 0xeffff030:     0x00000004
> 0xeffff02c:     0x00000004
> 0xeffff028:     0x00000140
> 0xeffff024:     0x00000140
> 0xeffff020:     0x00000034
> 0xeffff01c:     0x00000034
> 0xeffff018:     0x00000034
> 0xeffff014:     0x00000006
> 0xeffff010:     0x003b003a
> 0xeffff00c:     0x000a0028
> 0xeffff008:     0x00340020
> 0xeffff004:     0xc019c000                      saved $a5 == libc .got
> 0xeffff000:     0xeffff068                      saved $a3 (corrupted)
> 0xefffeffc:     0x00000000                      saved $a2
> 0xefffeff8:     0x00000001                      saved $d5
> 0xefffeff4:     0xeffff122                      saved $d4
> 0xefffeff0:     0xeffff11e                      saved $d3
> 0xefffefec:     0x00000000                      saved $d2
> 0xefffefe8:     0xc00e419a      return address __GI___wait4_time64+38
> 0xefffefe4:     0xc0028780
> 0xefffefe0:     0x3c344bfb
> 0xefffefdc:     0x000af353
> 0xefffefd8:     0x3c340170
> 0xefffefd4:     0x00000000
> 0xefffefd0:     0xc00e417c
> 0xefffefcc:     0xc00e417e
> 0xefffefc8:     0xc00e4180
> 0xefffefc4:     0x48e73c34
> 0xefffefc0:     0x00000000
> 0xefffefbc:     0xefffeff8
> 0xefffefb8:     0xefffeffc
> 0xefffefb4:     0x4bfb0170
> 0xefffefb0:     0x0eee0709
> 0xefffefac:     0x00000000
> 0xefffefa8:     0x00000000
> 0xefffefa4:     0x00000000
> 0xefffefa0:     0x00000000
> 0xefffef9c:     0x00000000
> 0xefffef98:     0x00000000
> 0xefffef94:     0x00000000
> 0xefffef90:     0x00000000
> 0xefffef8c:     0x00000000
> 0xefffef88:     0x00000000
> 0xefffef84:     0x00000000
> 0xefffef80:     0x00000000
> 0xefffef7c:     0x00000000
> 0xefffef78:     0x00000000
> 0xefffef74:     0x00000000
> 0xefffef70:     0x00000000
> 0xefffef6c:     0x00000000
> 0xefffef68:     0x00000000
> 0xefffef64:     0x00000000
> 0xefffef60:     0x00000000
> 0xefffef5c:     0x00000000
> 0xefffef58:     0x00000000
> 0xefffef54:     0x00000000
> 0xefffef50:     0x00000000
> 0xefffef4c:     0x00000000
> 0xefffef48:     0x00000000
> 0xefffef44:     0x00000000
> 0xefffef40:     0x00000000
> 0xefffef3c:     0x00000000
> 0xefffef38:     0x00000000
> 0xefffef34:     0x00000000
> 0xefffef30:     0x00000000
> 0xefffef2c:     0x00000000
> 0xefffef28:     0x00000000
> 0xefffef24:     0x00000000
> 0xefffef20:     0x00000000
> 0xefffef1c:     0x00000000
> 0xefffef18:     0x00000000
> 0xefffef14:     0x00000000
> 0xefffef10:     0x7c0effff
> 0xefffef0c:     0xffffffff
> 0xefffef08:     0xaaaaaaaa
> 0xefffef04:     0xaf54eaaa
> 0xefffef00:     0x40040000
> 0xefffeefc:     0x40040000
> 0xefffeef8:     0x2b000000
> 0xefffeef4:     0x00000000
> 0xefffeef0:     0x00000000
> 0xefffeeec:     0x408ece9a
> 0xefffeee8:     0x00000000
> 0xefffeee4:     0xf0ff0000
> 0xefffeee0:     0x0f800000
> 0xefffeedc:     0xf0fff0ff
> 0xefffeed8:     0x1f380000
> 0xefffeed4:     0x00000000
> 0xefffeed0:     0x00000000
> 0xefffeecc:     0x00000000
> 0xefffeec8:     0xffffffff
> 0xefffeec4:     0xffffffff
> 0xefffeec0:     0x7fff0000
> 0xefffeebc:     0xffffffff
> 0xefffeeb8:     0xffffffff
> 0xefffeeb4:     0x7fff0000
>
> The signal frame is not readily apparent (to me).

 From looking at the above stack dump, sc ought to start at 0xefffee90, 
and the trampoline would be three words below that. The last address you 
show corresponds to 0xeffff640 in first dump above, which is at the 
start of the saved fpregs. I'd say we just miss the beginning of the 
signal frame?

(My reasoning is that copy_siginfo_to_user clears the end of the signal 
stack, which is what we can see in both cases.)

Can't explain the 14 words below the saved return address though.

>
> Also, stack memory in the core dump and stack memory as observed upon
> signal handler breakpoint do not agree very well... I'd expect the values
> immediately below the stack pointer to be zeros once the signal frame was
> put there.

Yes, but what you get to see in the core dump is the stack after return 
from __GI___wait4_time64 ... and differences in timing may have resulted 
in different code executed (assuming the gdb trace is from the first 
subshell signaling, and the dump from one of the later ones ...).

>
> I can't explain all of that unless it's discontiguous stack corruption.
>

Me neither, sorry.

Cheers,

	Michael

Re: core dump analysis, was Re: stack smashing detected

[Finn Thain]

On Tue, 18 Apr 2023, Michael Schmitz wrote:

> Am 16.04.2023 um 18:44 schrieb Finn Thain:
> 
> >
> > The backtrace confirms that this signal was delivered during execution 
> > of __wait3(). (Delivery can happen during execution of __libc_fork() 
> > but I just repeat the test until I get these ducks in a row.)
> >
> > (gdb) c
> > Continuing.
> > # x=$(:)
> > [Detaching after fork from child process 1055]
> >
> > Breakpoint 6.1, onsig (signo=17) at trap.c:286
> > 286     trap.c: No such file or directory.
> > (gdb) bt
> > #0  onsig (signo=17) at trap.c:286
> > #1  <signal handler called>
> > #2  0xc00e81b6 in __GI___wait4_time64 (pid=-1, stat_loc=0xeffff86a,
> > options=2,
> >     usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:35
> > #3  0xc00e8164 in __GI___wait3_time64 (usage=0x0, options=<optimized out>,
> >     stat_loc=<optimized out>) at ../sysdeps/unix/sysv/linux/wait3.c:26
> 
> Where did that one come from? I don't think we saw __GI___wait3_time64 
> called in your disassembly of __wait3 ...
> 
> > #4  __wait3 (stat_loc=<optimized out>, options=<optimized out>,
> >     usage=<optimized out>) at ../sysdeps/unix/sysv/linux/wait3.c:35


Well spotted. However, it turns out there is a good explanation for that:

(gdb) print __GI___wait3_time64
$3 = {pid_t (int *, int, struct __rusage64 *)} 0xc00e4054 <__GI___wait3_time64>
(gdb) disass __GI___wait3_time64
Dump of assembler code for function __GI___wait3_time64:
   0xc00e4054 <+0>:     movel %sp@(12),%sp@-
   0xc00e4058 <+4>:     movel %sp@(12),%sp@-
   0xc00e405c <+8>:     movel %sp@(12),%sp@-
   0xc00e4060 <+12>:    pea 0xffffffff
   0xc00e4064 <+16>:    bsrl 0xc00e4174 <__GI___wait4_time64>
   0xc00e406a <+22>:    lea %sp@(16),%sp
   0xc00e406e <+26>:    rts
End of assembler dump.
(gdb) print __wait3
$2 = {pid_t (int *, int, struct rusage *)} 0xc00e4070 <__wait3>
(gdb) disass __wait3
Dump of assembler code for function __wait3:
   0xc00e4070 <+0>:     linkw %fp,#-96
   0xc00e4074 <+4>:     moveml %a2-%a3/%a5,%sp@-
   0xc00e4078 <+8>:     lea %pc@(0xc019c000),%a5
   0xc00e4080 <+16>:    movel %fp@(8),%d0
   0xc00e4084 <+20>:    moveal %fp@(16),%a2
   0xc00e4088 <+24>:    moveal %a5@(108),%a3
   0xc00e408c <+28>:    movel %a3@,%fp@(-4)
   0xc00e4090 <+32>:    tstl %a2
   0xc00e4092 <+34>:    beqw 0xc00e4152 <__wait3+226>
   0xc00e4096 <+38>:    pea %fp@(-92)
   0xc00e409a <+42>:    movel %fp@(12),%sp@-
   0xc00e409e <+46>:    movel %d0,%sp@-
   0xc00e40a0 <+48>:    pea 0xffffffff
   0xc00e40a4 <+52>:    bsrl 0xc00e4174 <__GI___wait4_time64>
   0xc00e40aa <+58>:    lea %sp@(16),%sp
   0xc00e40ae <+62>:    tstl %d0
   0xc00e40b0 <+64>:    bgts 0xc00e40c8 <__wait3+88>
   0xc00e40b2 <+66>:    moveal %fp@(-4),%a0
   0xc00e40b6 <+70>:    movel %a3@,%d1
   0xc00e40b8 <+72>:    cmpl %a0,%d1
   0xc00e40ba <+74>:    bnew 0xc00e416c <__wait3+252>
   0xc00e40be <+78>:    moveml %fp@(-108),%a2-%a3/%a5
   0xc00e40c4 <+84>:    unlk %fp
   0xc00e40c6 <+86>:    rts
   0xc00e40c8 <+88>:    pea 0x44
   0xc00e40cc <+92>:    clrl %sp@-
   0xc00e40ce <+94>:    pea %a2@(4)
   0xc00e40d2 <+98>:    movel %d0,%fp@(-96)
   0xc00e40d6 <+102>:   bsrl 0xc00bc850 <__GI_memset>
   0xc00e40dc <+108>:   movel %fp@(-88),%a2@
   0xc00e40e0 <+112>:   movel %fp@(-80),%a2@(4)
   0xc00e40e6 <+118>:   movel %fp@(-72),%a2@(8)
   0xc00e40ec <+124>:   movel %fp@(-64),%a2@(12)
   0xc00e40f2 <+130>:   movel %fp@(-60),%a2@(16)
   0xc00e40f8 <+136>:   movel %fp@(-56),%a2@(20)
   0xc00e40fe <+142>:   movel %fp@(-52),%a2@(24)
   0xc00e4104 <+148>:   movel %fp@(-48),%a2@(28)
   0xc00e410a <+154>:   movel %fp@(-44),%a2@(32)
   0xc00e4110 <+160>:   movel %fp@(-40),%a2@(36)
   0xc00e4116 <+166>:   movel %fp@(-36),%a2@(40)
   0xc00e411c <+172>:   movel %fp@(-32),%a2@(44)
   0xc00e4122 <+178>:   movel %fp@(-28),%a2@(48)
   0xc00e4128 <+184>:   movel %fp@(-24),%a2@(52)
   0xc00e412e <+190>:   movel %fp@(-20),%a2@(56)
   0xc00e4134 <+196>:   movel %fp@(-16),%a2@(60)
   0xc00e413a <+202>:   movel %fp@(-12),%a2@(64)
   0xc00e4140 <+208>:   movel %fp@(-8),%a2@(68)
   0xc00e4146 <+214>:   lea %sp@(12),%sp
   0xc00e414a <+218>:   movel %fp@(-96),%d0
   0xc00e414e <+222>:   braw 0xc00e40b2 <__wait3+66>
   0xc00e4152 <+226>:   clrl %sp@-
   0xc00e4154 <+228>:   movel %fp@(12),%sp@-
   0xc00e4158 <+232>:   movel %d0,%sp@-
   0xc00e415a <+234>:   pea 0xffffffff
   0xc00e415e <+238>:   bsrl 0xc00e4174 <__GI___wait4_time64>
   0xc00e4164 <+244>:   lea %sp@(16),%sp
   0xc00e4168 <+248>:   braw 0xc00e40b2 <__wait3+66>
=> 0xc00e416c <+252>:   illegal
End of assembler dump.


(gdb) info frame
Stack level 3, frame at 0xeffff82c:
 pc = 0xc00e8164 in __GI___wait3_time64
    (../sysdeps/unix/sysv/linux/wait3.c:26); saved pc = 0xd000c38e
 inlined into frame 4, caller of frame at 0xeffff7a8
 source language c.
 Arglist at unknown address.
 Locals at unknown address, Previous frame's sp is 0xeffff7a8
 Saved registers:
  d2 at 0xeffff738, d3 at 0xeffff73c, d4 at 0xeffff740, d5 at 0xeffff744,
  a2 at 0xeffff748, a3 at 0xeffff74c, a5 at 0xeffff750, pc at 0xeffff7a4


(gdb) up
#4  __wait3 (stat_loc=<optimized out>, options=<optimized out>, 
    usage=<optimized out>) at ../sysdeps/unix/sysv/linux/wait3.c:35
35      in ../sysdeps/unix/sysv/linux/wait3.c
(gdb) info frame
Stack level 4, frame at 0xeffff82c:
 pc = 0xc00e8164 in __wait3 (../sysdeps/unix/sysv/linux/wait3.c:35); 
    saved pc = 0xd000c38e
 called by frame at 0xeffff928, caller of frame at 0xeffff82c
 source language c.
 Arglist at 0xeffff824, args: stat_loc=<optimized out>, 
    options=<optimized out>, usage=<optimized out>
 Locals at 0xeffff824, Previous frame's sp is 0xeffff82c
 Saved registers:
  a2 at 0xeffff7b8, a3 at 0xeffff7bc, a5 at 0xeffff7c0, fp at 0xeffff824,
  pc at 0xeffff828


Note that frame 3 was "inlined into frame 4". The inlined code can be seen 
above at 0xc00e4154. So the backtrace is misleading inasmuchas it 
represents the source code rather than the disassembly.

> > #5  0xd000c38e in waitproc (status=0xeffff85a, block=1) at jobs.c:1179
> > #6  waitone (block=1, job=0xd001f618) at jobs.c:1055
> > #7  0xd000c5b8 in dowait (block=1, jp=0xd001f618) at jobs.c:1137
> > #8  0xd000ddb0 in waitforjob (jp=0xd001f618) at jobs.c:1014
> > #9  0xd000aade in expbackq (flag=68, cmd=0xd001e4c8 <stackbase+36>)
> >     at expand.c:520
> > #10 argstr (p=<optimized out>, flag=68) at expand.c:335
> > #11 0xd000b5ce in expandarg (arg=0xd001e4e8 <stackbase+68>,
> >     arglist=0xeffffb08, flag=4) at expand.c:192
> > #12 0xd0007e2a in evalcommand (cmd=<optimized out>, flags=<optimized out>)
> >     at eval.c:855
> > #13 0xd0006ffc in evaltree (n=0xd001e4f8 <stackbase+84>, flags=0) at
> > eval.c:300
> > #14 0xd000e3c0 in cmdloop (top=1) at main.c:246
> > #15 0xd0005018 in main (argc=<optimized out>, argv=<optimized out>)
> >     at main.c:181
> >
> >
> > 0xeffff750:     0xc01a0000                      saved $a5 == libc .got
> > 0xeffff74c:     0xc0023e8c                      saved $a3 == &__stack_chk_guard
> > 0xeffff748:     0x00000000                      saved $a2
> > 0xeffff744:     0x00000001                      saved $d5
> > 0xeffff740:     0xeffff86e                      saved $d4
> > 0xeffff73c:     0xeffff86a                      saved $d3
> > 0xeffff738:     0x00000002                      saved $d2
> > 0xeffff734:     0x00000000
> > 0xeffff730:     0x00000000
> > 0xeffff72c:     0x00000000
> > 0xeffff728:     0x00000000
> > 0xeffff724:     0x00000000
> > 0xeffff720:     0x00000000
> > 0xeffff71c:     0x00000000
> > 0xeffff718:     0x00000000
> > 0xeffff714:     0x00000000
> > 0xeffff710:     0x00000000
> > 0xeffff70c:     0x00000000
> > 0xeffff708:     0x00000000
> > 0xeffff704:     0x00000000
> > 0xeffff700:     0x00000000
> > 0xeffff6fc:     0x00000000
> > 0xeffff6f8:     0x00000000
> > 0xeffff6f4:     0x00000000
> > 0xeffff6f0:     0x00000000
> > 0xeffff6ec:     0x00000000
> > 0xeffff6e8:     0x00000000
> > 0xeffff6e4:     0x00000000
> > 0xeffff6e0:     0x00000000
> > 0xeffff6dc:     0x00000000
> > 0xeffff6d8:     0x00000000
> > 0xeffff6d4:     0x00000000
> > 0xeffff6d0:     0x00000000
> > 0xeffff6cc:     0x00000000
> > 0xeffff6c8:     0x00000000
> > 0xeffff6c4:     0x00000000
> > 0xeffff6c0:     0x00000000
> > 0xeffff6bc:     0x00000000
> > 0xeffff6b8:     0x00000000
> > 0xeffff6b4:     0x00000000
> > 0xeffff6b0:     0x00000000
> > 0xeffff6ac:     0x00000000
> > 0xeffff6a8:     0x00000000
> > 0xeffff6a4:     0x00000000
> > 0xeffff6a0:     0x00000000
> > 0xeffff69c:     0x00000000
> > 0xeffff698:     0x00000000
> > 0xeffff694:     0x00000000
> > 0xeffff690:     0x00000000
> > 0xeffff68c:     0x00000000
> > 0xeffff688:     0x00000000
> > 0xeffff684:     0x00000000
> > 0xeffff680:     0x00000000
> > 0xeffff67c:     0x00000000
> > 0xeffff678:     0x00000000
> > 0xeffff674:     0x00000000
> > 0xeffff670:     0x00000000
> > 0xeffff66c:     0x00000000
> > 0xeffff668:     0x00000000
> > 0xeffff664:     0x00000000
> > 0xeffff660:     0x41000000
> > 0xeffff65c:     0x00000000
> > 0xeffff658:     0x00000000
> > 0xeffff654:     0x00000000
> > 0xeffff650:     0x00000000
> > 0xeffff64c:     0x80000000
> > 0xeffff648:     0x3fff0000
> > 0xeffff644:     0x00000000
> > 0xeffff640:     0xd0000000
> > 0xeffff63c:     0x40020000				<= (sc.formatvec & 0xffff) << 16; fpregs from here on
> > 0xeffff638:     0x81b60080      			<= (sc.pc & 0xffff) << 16 | sc.formatvec >> 16
> > 0xeffff634:     0x0000c00e				<= sc.sr << 16  sc.pc >> 16
> > 0xeffff630:     0xd001e4e3                          <= sc.a1
> > 0xeffff62c:     0xc0028780                          <= sc.a0
> > 0xeffff628:     0xffffffff                          <= sc.d1
> > 0xeffff624:     0x0000041f                          <= sc.d0
> > 0xeffff620:     0xeffff738                          <= sc.usp
> > 0xeffff61c:     0x00000000      			<= sc.mask
> > 0xeffff618:     0x00000000				<= extramask
> > 0xeffff614:     0x00000000				<= frame.retcode[1]
> > 0xeffff610:     0x70774e40      moveq #119,%d0 ; trap #0
> > 0xeffff60c:     0xeffff61c				<= frame->sc
> > 0xeffff608:     0x00000080				<= tregs->vector
> > 0xeffff604:     0x00000011				<= signal no.
> > 0xeffff600:     0xeffff610      return address
> >
> > The above comes from dash running under gdb under qemu, which does not 
> > exhibit the failure but is convenient for that kind of experiment.
> 
> I would have expected to see a different signal trampoline (for 
> sys_rt_sigreturn) ... 

Well, this seems to be the trampoline from setup_frame() and not 
setup_rt_frame().

> But anyway:
> 
> The saved pc is 0xc00e81b6 which does match the backtrace above. Vector 
> offset 80 matches trap 0 which suggests 0xc00e81b6 should be the 
> instruction after a trap 0 instruction. d0 is 1055 which is not a signal 
> number I recognize.
> 

I don't know what d0 represents here. But &frame->sig == 0x11 is correct 
(SIGCHLD).

> >
> >> Again as far as I understand, the core dump happens on process exit. 
> >> Stack smashing is detected and process exit is forced only at exit 
> >> from __wait3() or __wait4_time64(),
> >
> > I placed an illegal instruction in __wait3. This executes instead of 
> > the call to __stack_chk_fail because that obliterates stack memory of 
> > interest.
> 
> OK.
> 
> >
> > Consequently the latest core dump still contains dead stack frames 
> > (see below) of subroutines that returned before __wait3() dumped core. 
> > You can see the return address for the branch to __wait4_time64() and 
> > below that you can see the return address for the branch to 
> > __m68k_read_tp().
> >
> > (gdb) disas __wait4_time64
> > Dump of assembler code for function __GI___wait4_time64:
> >    0xc00e4174 <+0>:     lea %sp@(-80),%sp
> >    0xc00e4178 <+4>:     moveml %d2-%d5/%a2-%a3/%a5,%sp@-
> >    0xc00e417c <+8>:     lea %pc@(0xc019c000),%a5
> >    0xc00e4184 <+16>:    movel %sp@(116),%d2
> >    0xc00e4188 <+20>:    moveal %sp@(124),%a2
> >    0xc00e418c <+24>:    moveal %a5@(108),%a3
> >    0xc00e4190 <+28>:    movel %a3@,%sp@(104)
> >    0xc00e4194 <+32>:    bsrl 0xc0056e2c <__m68k_read_tp@plt>
> >
> > I gather the signal was delivered before __wait4_time64+38, otherwise 
> > the return address 0xc00e419a (which appears below) would have been 
> > overwritten by the signal frame. The signal must have been delivered 
> > after waitproc() initialized gotsigchld = 0 since gotsigchld is 1 at 
> > the time of the coredump.
> >
> > I assume the %a3 corruption happened after __wait4_time64+8 because 
> > that's when %a3 first appears on the stack. And the corruption must 
> > have happened before __wait4_time64+238, which is when %a3 was 
> > restored.
> >
> > If it was the signal which somehow corrupted the saved %a3, there's 
> > only a small window for that. The only syscall in that window is 
> > get_thread_area.
> 
> I see sys_wait4 called in two places (0xc00e01b4, and then 0xc00e0286 
> depending on the return code of the first). The second one again would 
> have called __m68k_read_tp so would have left a return address on the 
> stack (0xc00e02d2). Leaves the first.
> 

That's why my analysis stopped at __wait4_time64+38: the rest of 
__wait4_time64 is not relevant to the dead stack contents. (It would have 
left a different return address in that memory location.)

> >
> > Here's some stack memory from the core dump.
> >
> > 0xeffff0dc:     0xd000c38e      return address waitproc+124
> > 0xeffff0d8:     0xd001c1ec      frame 0 $fp                   ==
> > &suppressint
> > 0xeffff0d4:     0x00add14b      canary
> > 0xeffff0d0:     0x00000000
> > 0xeffff0cc:     0x0000000a
> > 0xeffff0c8:     0x00000202
> > 0xeffff0c4:     0x00000008
> > 0xeffff0c0:     0x00000000
> > 0xeffff0bc:     0x00000000
> > 0xeffff0b8:     0x00000174
> > 0xeffff0b4:     0x00000004
> > 0xeffff0b0:     0x00000004
> > 0xeffff0ac:     0x00000006
> > 0xeffff0a8:     0x000000e0
> > 0xeffff0a4:     0x000000e0
> > 0xeffff0a0:     0x00171f20
> > 0xeffff09c:     0x00171f20
> > 0xeffff098:     0x00171f20
> > 0xeffff094:     0x00000002
> > 0xeffff090:     0x00002000
> > 0xeffff08c:     0x00000006
> > 0xeffff088:     0x0000e920
> > 0xeffff084:     0x00005360
> > 0xeffff080:     0x00170700
> > 0xeffff07c:     0x00170700
> > 0xeffff078:     0x00170700      frame 0 $fp - 96
> > 0xeffff074:     0xd001b874                         saved $a5 == dash .got
> > 0xeffff070:     0xd001e498                         saved $a3 == &dash_errno
> > 0xeffff06c:     0xd001e718      frame 0 $sp        saved $a2 == &gotsigchld
> > 0xeffff068:     0x00000000
> > 0xeffff064:     0x00000000
> > 0xeffff060:     0xeffff11e
> > 0xeffff05c:     0xffffffff
> > 0xeffff058:     0xc00e4164      return address __wait3+244
> > 0xeffff054:     0x00add14b      canary
> > 0xeffff050:     0x00000001
> > 0xeffff04c:     0x00000004
> > 0xeffff048:     0x0000000d
> > 0xeffff044:     0x0000000d
> > 0xeffff040:     0x0015ef82
> > 0xeffff03c:     0x0015ef82
> > 0xeffff038:     0x0015ef82
> > 0xeffff034:     0x00000003
> > 0xeffff030:     0x00000004
> > 0xeffff02c:     0x00000004
> > 0xeffff028:     0x00000140
> > 0xeffff024:     0x00000140
> > 0xeffff020:     0x00000034
> > 0xeffff01c:     0x00000034
> > 0xeffff018:     0x00000034
> > 0xeffff014:     0x00000006
> > 0xeffff010:     0x003b003a
> > 0xeffff00c:     0x000a0028
> > 0xeffff008:     0x00340020
> > 0xeffff004:     0xc019c000                      saved $a5 == libc .got
> > 0xeffff000:     0xeffff068                      saved $a3 (corrupted)
> > 0xefffeffc:     0x00000000                      saved $a2
> > 0xefffeff8:     0x00000001                      saved $d5
> > 0xefffeff4:     0xeffff122                      saved $d4
> > 0xefffeff0:     0xeffff11e                      saved $d3
> > 0xefffefec:     0x00000000                      saved $d2
> > 0xefffefe8:     0xc00e419a      return address __GI___wait4_time64+38
> > 0xefffefe4:     0xc0028780
> > 0xefffefe0:     0x3c344bfb
> > 0xefffefdc:     0x000af353
> > 0xefffefd8:     0x3c340170
> > 0xefffefd4:     0x00000000
> > 0xefffefd0:     0xc00e417c
> > 0xefffefcc:     0xc00e417e
> > 0xefffefc8:     0xc00e4180
> > 0xefffefc4:     0x48e73c34
> > 0xefffefc0:     0x00000000
> > 0xefffefbc:     0xefffeff8
> > 0xefffefb8:     0xefffeffc
> > 0xefffefb4:     0x4bfb0170
> > 0xefffefb0:     0x0eee0709
> > 0xefffefac:     0x00000000
> > 0xefffefa8:     0x00000000
> > 0xefffefa4:     0x00000000
> > 0xefffefa0:     0x00000000
> > 0xefffef9c:     0x00000000
> > 0xefffef98:     0x00000000
> > 0xefffef94:     0x00000000
> > 0xefffef90:     0x00000000
> > 0xefffef8c:     0x00000000
> > 0xefffef88:     0x00000000
> > 0xefffef84:     0x00000000
> > 0xefffef80:     0x00000000
> > 0xefffef7c:     0x00000000
> > 0xefffef78:     0x00000000
> > 0xefffef74:     0x00000000
> > 0xefffef70:     0x00000000
> > 0xefffef6c:     0x00000000
> > 0xefffef68:     0x00000000
> > 0xefffef64:     0x00000000
> > 0xefffef60:     0x00000000
> > 0xefffef5c:     0x00000000
> > 0xefffef58:     0x00000000
> > 0xefffef54:     0x00000000
> > 0xefffef50:     0x00000000
> > 0xefffef4c:     0x00000000
> > 0xefffef48:     0x00000000
> > 0xefffef44:     0x00000000
> > 0xefffef40:     0x00000000
> > 0xefffef3c:     0x00000000
> > 0xefffef38:     0x00000000
> > 0xefffef34:     0x00000000
> > 0xefffef30:     0x00000000
> > 0xefffef2c:     0x00000000
> > 0xefffef28:     0x00000000
> > 0xefffef24:     0x00000000
> > 0xefffef20:     0x00000000
> > 0xefffef1c:     0x00000000
> > 0xefffef18:     0x00000000
> > 0xefffef14:     0x00000000
> > 0xefffef10:     0x7c0effff
> > 0xefffef0c:     0xffffffff
> > 0xefffef08:     0xaaaaaaaa
> > 0xefffef04:     0xaf54eaaa
> > 0xefffef00:     0x40040000
> > 0xefffeefc:     0x40040000
> > 0xefffeef8:     0x2b000000
> > 0xefffeef4:     0x00000000
> > 0xefffeef0:     0x00000000
> > 0xefffeeec:     0x408ece9a
> > 0xefffeee8:     0x00000000
> > 0xefffeee4:     0xf0ff0000
> > 0xefffeee0:     0x0f800000
> > 0xefffeedc:     0xf0fff0ff
> > 0xefffeed8:     0x1f380000
> > 0xefffeed4:     0x00000000
> > 0xefffeed0:     0x00000000
> > 0xefffeecc:     0x00000000	
> > 0xefffeec8:     0xffffffff	
> > 0xefffeec4:     0xffffffff	
> > 0xefffeec0:     0x7fff0000	
> > 0xefffeebc:     0xffffffff	
> > 0xefffeeb8:     0xffffffff
> > 0xefffeeb4:     0x7fff0000	sc_formatvec
> >
> > The signal frame is not readily apparent (to me).
> 
> From looking at the above stack dump, sc ought to start at 0xefffee90, 
> and the trampoline would be three words below that.

0xefffeeb0:     0x4178b008	sc_pc, sc_formatvec
0xefffeeac:     0x0008c00e	sc_sr, sc_pc
0xefffeea8:     0xd00223bb	sc_a1
0xefffeea4:     0xd001e32c	sc_a0
0xefffeea0:     0x00000003	sc_d1
0xefffee9c:     0xeffff11e	sc_d0
0xefffee98:     0xeffff004	sc_usp
0xefffee94:     0x00000000	sc_mask
0xefffee90:     0x00000000	extramask
0xefffee8c:     0xc0024a90	retcode[1]
0xefffee88:     0x70774e40	retcode[0]
0xefffee84:     0xefffee94	psc
0xefffee80:     0x00000008	code
0xefffee7c:     0x00000011	sig
0xefffee78:     0xefffee88	pretcode
0xefffee74:     0xc019c000
0xefffee70:     0x00000000
0xefffee6c:     0xc0025878
0xefffee68:     0xc0007ed4
0xefffee64:     0xc0024000
0xefffee60:     0xefffef50
0xefffee5c:     0xc0024000
0xefffee58:     0xc002a034
0xefffee54:     0xc0024a90
0xefffee50:     0xc0025878
0xefffee4c:     0x00000001
0xefffee48:     0x0017f020
0xefffee44:     0x0000002c
0xefffee40:     0x0000000f
0xefffee3c:     0x00000000
0xefffee38:     0xfffff7fa
0xefffee34:     0xffffffff
0xefffee30:     0x00009782
0xefffee2c:     0x00000000
0xefffee28:     0x0000001e
0xefffee24:     0xc0025858
0xefffee20:     0xc0025af8
0xefffee1c:     0xc000b376
0xefffee18:     0xc0024000
0xefffee14:     0xc0025878
0xefffee10:     0x0000001d
0xefffee0c:     0xd0001b60
0xefffee08:     0x0000002f
0xefffee04:     0xc002563e
0xefffee00:     0xc0025490

> The last address you show corresponds to 0xeffff640 in first dump above, 
> which is at the start of the saved fpregs. I'd say we just miss the 
> beginning of the signal frame?
> 

It looks like you're right. I'm not sure how I missed that.

So when the signal was delivered, PC == 0xc00e4178 and USP == 0xc00e4178. 
Those addresses can be found in the disassembly and the stack contents I 
sent previously (quoted above) and it all seems to line up.

> (My reasoning is that copy_siginfo_to_user clears the end of the signal 
> stack, which is what we can see in both cases.)
> 
> Can't explain the 14 words below the saved return address though.
> 

Right. Is it sc_fpstate? Perhaps we should expect QEMU to differ here.

Bottom line is, the corrupted %a3 register would have been saved by the 
MOVEM instruction at 0xc00e4178, which turns out to be the PC in the 
signal frame. So it certainly looks like the kernel was the culprit here.

Re: core dump analysis, was Re: stack smashing detected

[Michael Schmitz]

Hi Finn,

Am 18.04.2023 um 14:04 schrieb Finn Thain:
> On Tue, 18 Apr 2023, Michael Schmitz wrote:
>
>> Am 16.04.2023 um 18:44 schrieb Finn Thain:
>>
>>>
>>> The backtrace confirms that this signal was delivered during execution
>>> of __wait3(). (Delivery can happen during execution of __libc_fork()
>>> but I just repeat the test until I get these ducks in a row.)
>>>
>>> (gdb) c
>>> Continuing.
>>> # x=$(:)
>>> [Detaching after fork from child process 1055]
>>>
>>> Breakpoint 6.1, onsig (signo=17) at trap.c:286
>>> 286     trap.c: No such file or directory.
>>> (gdb) bt
>>> #0  onsig (signo=17) at trap.c:286
>>> #1  <signal handler called>
>>> #2  0xc00e81b6 in __GI___wait4_time64 (pid=-1, stat_loc=0xeffff86a,
>>> options=2,
>>>     usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:35
>>> #3  0xc00e8164 in __GI___wait3_time64 (usage=0x0, options=<optimized out>,
>>>     stat_loc=<optimized out>) at ../sysdeps/unix/sysv/linux/wait3.c:26
>>
>> Where did that one come from? I don't think we saw __GI___wait3_time64
>> called in your disassembly of __wait3 ...
>>
>>> #4  __wait3 (stat_loc=<optimized out>, options=<optimized out>,
>>>     usage=<optimized out>) at ../sysdeps/unix/sysv/linux/wait3.c:35
>
>
> Well spotted. However, it turns out there is a good explanation for that:
>
> (gdb) print __GI___wait3_time64
> $3 = {pid_t (int *, int, struct __rusage64 *)} 0xc00e4054 <__GI___wait3_time64>
> (gdb) disass __GI___wait3_time64
> Dump of assembler code for function __GI___wait3_time64:
>    0xc00e4054 <+0>:     movel %sp@(12),%sp@-
>    0xc00e4058 <+4>:     movel %sp@(12),%sp@-
>    0xc00e405c <+8>:     movel %sp@(12),%sp@-
>    0xc00e4060 <+12>:    pea 0xffffffff
>    0xc00e4064 <+16>:    bsrl 0xc00e4174 <__GI___wait4_time64>
>    0xc00e406a <+22>:    lea %sp@(16),%sp
>    0xc00e406e <+26>:    rts
> End of assembler dump.
> (gdb) print __wait3
> $2 = {pid_t (int *, int, struct rusage *)} 0xc00e4070 <__wait3>
> (gdb) disass __wait3
> Dump of assembler code for function __wait3:
>    0xc00e4070 <+0>:     linkw %fp,#-96
>    0xc00e4074 <+4>:     moveml %a2-%a3/%a5,%sp@-
>    0xc00e4078 <+8>:     lea %pc@(0xc019c000),%a5
>    0xc00e4080 <+16>:    movel %fp@(8),%d0
>    0xc00e4084 <+20>:    moveal %fp@(16),%a2
>    0xc00e4088 <+24>:    moveal %a5@(108),%a3
>    0xc00e408c <+28>:    movel %a3@,%fp@(-4)
>    0xc00e4090 <+32>:    tstl %a2
>    0xc00e4092 <+34>:    beqw 0xc00e4152 <__wait3+226>
>    0xc00e4096 <+38>:    pea %fp@(-92)
>    0xc00e409a <+42>:    movel %fp@(12),%sp@-
>    0xc00e409e <+46>:    movel %d0,%sp@-
>    0xc00e40a0 <+48>:    pea 0xffffffff
>    0xc00e40a4 <+52>:    bsrl 0xc00e4174 <__GI___wait4_time64>
>    0xc00e40aa <+58>:    lea %sp@(16),%sp
>    0xc00e40ae <+62>:    tstl %d0
>    0xc00e40b0 <+64>:    bgts 0xc00e40c8 <__wait3+88>
>    0xc00e40b2 <+66>:    moveal %fp@(-4),%a0
>    0xc00e40b6 <+70>:    movel %a3@,%d1
>    0xc00e40b8 <+72>:    cmpl %a0,%d1
>    0xc00e40ba <+74>:    bnew 0xc00e416c <__wait3+252>
>    0xc00e40be <+78>:    moveml %fp@(-108),%a2-%a3/%a5
>    0xc00e40c4 <+84>:    unlk %fp
>    0xc00e40c6 <+86>:    rts
>    0xc00e40c8 <+88>:    pea 0x44
>    0xc00e40cc <+92>:    clrl %sp@-
>    0xc00e40ce <+94>:    pea %a2@(4)
>    0xc00e40d2 <+98>:    movel %d0,%fp@(-96)
>    0xc00e40d6 <+102>:   bsrl 0xc00bc850 <__GI_memset>
>    0xc00e40dc <+108>:   movel %fp@(-88),%a2@
>    0xc00e40e0 <+112>:   movel %fp@(-80),%a2@(4)
>    0xc00e40e6 <+118>:   movel %fp@(-72),%a2@(8)
>    0xc00e40ec <+124>:   movel %fp@(-64),%a2@(12)
>    0xc00e40f2 <+130>:   movel %fp@(-60),%a2@(16)
>    0xc00e40f8 <+136>:   movel %fp@(-56),%a2@(20)
>    0xc00e40fe <+142>:   movel %fp@(-52),%a2@(24)
>    0xc00e4104 <+148>:   movel %fp@(-48),%a2@(28)
>    0xc00e410a <+154>:   movel %fp@(-44),%a2@(32)
>    0xc00e4110 <+160>:   movel %fp@(-40),%a2@(36)
>    0xc00e4116 <+166>:   movel %fp@(-36),%a2@(40)
>    0xc00e411c <+172>:   movel %fp@(-32),%a2@(44)
>    0xc00e4122 <+178>:   movel %fp@(-28),%a2@(48)
>    0xc00e4128 <+184>:   movel %fp@(-24),%a2@(52)
>    0xc00e412e <+190>:   movel %fp@(-20),%a2@(56)
>    0xc00e4134 <+196>:   movel %fp@(-16),%a2@(60)
>    0xc00e413a <+202>:   movel %fp@(-12),%a2@(64)
>    0xc00e4140 <+208>:   movel %fp@(-8),%a2@(68)
>    0xc00e4146 <+214>:   lea %sp@(12),%sp
>    0xc00e414a <+218>:   movel %fp@(-96),%d0
>    0xc00e414e <+222>:   braw 0xc00e40b2 <__wait3+66>
>    0xc00e4152 <+226>:   clrl %sp@-
>    0xc00e4154 <+228>:   movel %fp@(12),%sp@-
>    0xc00e4158 <+232>:   movel %d0,%sp@-
>    0xc00e415a <+234>:   pea 0xffffffff
>    0xc00e415e <+238>:   bsrl 0xc00e4174 <__GI___wait4_time64>
>    0xc00e4164 <+244>:   lea %sp@(16),%sp
>    0xc00e4168 <+248>:   braw 0xc00e40b2 <__wait3+66>
> => 0xc00e416c <+252>:   illegal
> End of assembler dump.
>
>
> (gdb) info frame
> Stack level 3, frame at 0xeffff82c:
>  pc = 0xc00e8164 in __GI___wait3_time64
>     (../sysdeps/unix/sysv/linux/wait3.c:26); saved pc = 0xd000c38e
>  inlined into frame 4, caller of frame at 0xeffff7a8
>  source language c.
>  Arglist at unknown address.
>  Locals at unknown address, Previous frame's sp is 0xeffff7a8
>  Saved registers:
>   d2 at 0xeffff738, d3 at 0xeffff73c, d4 at 0xeffff740, d5 at 0xeffff744,
>   a2 at 0xeffff748, a3 at 0xeffff74c, a5 at 0xeffff750, pc at 0xeffff7a4
>
>
> (gdb) up
> #4  __wait3 (stat_loc=<optimized out>, options=<optimized out>,
>     usage=<optimized out>) at ../sysdeps/unix/sysv/linux/wait3.c:35
> 35      in ../sysdeps/unix/sysv/linux/wait3.c
> (gdb) info frame
> Stack level 4, frame at 0xeffff82c:
>  pc = 0xc00e8164 in __wait3 (../sysdeps/unix/sysv/linux/wait3.c:35);
>     saved pc = 0xd000c38e
>  called by frame at 0xeffff928, caller of frame at 0xeffff82c
>  source language c.
>  Arglist at 0xeffff824, args: stat_loc=<optimized out>,
>     options=<optimized out>, usage=<optimized out>
>  Locals at 0xeffff824, Previous frame's sp is 0xeffff82c
>  Saved registers:
>   a2 at 0xeffff7b8, a3 at 0xeffff7bc, a5 at 0xeffff7c0, fp at 0xeffff824,
>   pc at 0xeffff828
>
>
> Note that frame 3 was "inlined into frame 4". The inlined code can be seen
> above at 0xc00e4154. So the backtrace is misleading inasmuchas it
> represents the source code rather than the disassembly.

OK then ...

>
>>> #5  0xd000c38e in waitproc (status=0xeffff85a, block=1) at jobs.c:1179
>>> #6  waitone (block=1, job=0xd001f618) at jobs.c:1055
>>> #7  0xd000c5b8 in dowait (block=1, jp=0xd001f618) at jobs.c:1137
>>> #8  0xd000ddb0 in waitforjob (jp=0xd001f618) at jobs.c:1014
>>> #9  0xd000aade in expbackq (flag=68, cmd=0xd001e4c8 <stackbase+36>)
>>>     at expand.c:520
>>> #10 argstr (p=<optimized out>, flag=68) at expand.c:335
>>> #11 0xd000b5ce in expandarg (arg=0xd001e4e8 <stackbase+68>,
>>>     arglist=0xeffffb08, flag=4) at expand.c:192
>>> #12 0xd0007e2a in evalcommand (cmd=<optimized out>, flags=<optimized out>)
>>>     at eval.c:855
>>> #13 0xd0006ffc in evaltree (n=0xd001e4f8 <stackbase+84>, flags=0) at
>>> eval.c:300
>>> #14 0xd000e3c0 in cmdloop (top=1) at main.c:246
>>> #15 0xd0005018 in main (argc=<optimized out>, argv=<optimized out>)
>>>     at main.c:181
>>>
>>>
>>> 0xeffff750:     0xc01a0000                      saved $a5 == libc .got
>>> 0xeffff74c:     0xc0023e8c                      saved $a3 == &__stack_chk_guard
>>> 0xeffff748:     0x00000000                      saved $a2
>>> 0xeffff744:     0x00000001                      saved $d5
>>> 0xeffff740:     0xeffff86e                      saved $d4
>>> 0xeffff73c:     0xeffff86a                      saved $d3
>>> 0xeffff738:     0x00000002                      saved $d2
>>> 0xeffff734:     0x00000000
>>> 0xeffff730:     0x00000000
>>> 0xeffff72c:     0x00000000
>>> 0xeffff728:     0x00000000
>>> 0xeffff724:     0x00000000
>>> 0xeffff720:     0x00000000
>>> 0xeffff71c:     0x00000000
>>> 0xeffff718:     0x00000000
>>> 0xeffff714:     0x00000000
>>> 0xeffff710:     0x00000000
>>> 0xeffff70c:     0x00000000
>>> 0xeffff708:     0x00000000
>>> 0xeffff704:     0x00000000
>>> 0xeffff700:     0x00000000
>>> 0xeffff6fc:     0x00000000
>>> 0xeffff6f8:     0x00000000
>>> 0xeffff6f4:     0x00000000
>>> 0xeffff6f0:     0x00000000
>>> 0xeffff6ec:     0x00000000
>>> 0xeffff6e8:     0x00000000
>>> 0xeffff6e4:     0x00000000
>>> 0xeffff6e0:     0x00000000
>>> 0xeffff6dc:     0x00000000
>>> 0xeffff6d8:     0x00000000
>>> 0xeffff6d4:     0x00000000
>>> 0xeffff6d0:     0x00000000
>>> 0xeffff6cc:     0x00000000
>>> 0xeffff6c8:     0x00000000
>>> 0xeffff6c4:     0x00000000
>>> 0xeffff6c0:     0x00000000
>>> 0xeffff6bc:     0x00000000
>>> 0xeffff6b8:     0x00000000
>>> 0xeffff6b4:     0x00000000
>>> 0xeffff6b0:     0x00000000
>>> 0xeffff6ac:     0x00000000
>>> 0xeffff6a8:     0x00000000
>>> 0xeffff6a4:     0x00000000
>>> 0xeffff6a0:     0x00000000
>>> 0xeffff69c:     0x00000000
>>> 0xeffff698:     0x00000000
>>> 0xeffff694:     0x00000000
>>> 0xeffff690:     0x00000000
>>> 0xeffff68c:     0x00000000
>>> 0xeffff688:     0x00000000
>>> 0xeffff684:     0x00000000
>>> 0xeffff680:     0x00000000
>>> 0xeffff67c:     0x00000000
>>> 0xeffff678:     0x00000000
>>> 0xeffff674:     0x00000000
>>> 0xeffff670:     0x00000000
>>> 0xeffff66c:     0x00000000
>>> 0xeffff668:     0x00000000
>>> 0xeffff664:     0x00000000
>>> 0xeffff660:     0x41000000
>>> 0xeffff65c:     0x00000000
>>> 0xeffff658:     0x00000000
>>> 0xeffff654:     0x00000000
>>> 0xeffff650:     0x00000000
>>> 0xeffff64c:     0x80000000
>>> 0xeffff648:     0x3fff0000
>>> 0xeffff644:     0x00000000
>>> 0xeffff640:     0xd0000000
>>> 0xeffff63c:     0x40020000				<= (sc.formatvec & 0xffff) << 16; fpregs from here on
>>> 0xeffff638:     0x81b60080      			<= (sc.pc & 0xffff) << 16 | sc.formatvec >> 16
>>> 0xeffff634:     0x0000c00e				<= sc.sr << 16  sc.pc >> 16
>>> 0xeffff630:     0xd001e4e3                          <= sc.a1
>>> 0xeffff62c:     0xc0028780                          <= sc.a0
>>> 0xeffff628:     0xffffffff                          <= sc.d1
>>> 0xeffff624:     0x0000041f                          <= sc.d0
>>> 0xeffff620:     0xeffff738                          <= sc.usp
>>> 0xeffff61c:     0x00000000      			<= sc.mask
>>> 0xeffff618:     0x00000000				<= extramask
>>> 0xeffff614:     0x00000000				<= frame.retcode[1]
>>> 0xeffff610:     0x70774e40      moveq #119,%d0 ; trap #0
>>> 0xeffff60c:     0xeffff61c				<= frame->sc
>>> 0xeffff608:     0x00000080				<= tregs->vector
>>> 0xeffff604:     0x00000011				<= signal no.
>>> 0xeffff600:     0xeffff610      return address
>>>
>>> The above comes from dash running under gdb under qemu, which does not
>>> exhibit the failure but is convenient for that kind of experiment.
>>
>> I would have expected to see a different signal trampoline (for
>> sys_rt_sigreturn) ...
>
> Well, this seems to be the trampoline from setup_frame() and not
> setup_rt_frame().

According to the manpages I've seen, glibc ought to pick rt signals if 
the kernel supports those (which I suppose it does).

>
>> But anyway:
>>
>> The saved pc is 0xc00e81b6 which does match the backtrace above. Vector
>> offset 80 matches trap 0 which suggests 0xc00e81b6 should be the
>> instruction after a trap 0 instruction. d0 is 1055 which is not a signal
>> number I recognize.
>>
>
> I don't know what d0 represents here. But &frame->sig == 0x11 is correct
> (SIGCHLD).

Correct - that all works out. But d0 holds the syscall number when we 
enter the kernel via trap 0, and that one is odd.

>>>
>>>> Again as far as I understand, the core dump happens on process exit.
>>>> Stack smashing is detected and process exit is forced only at exit
>>>> from __wait3() or __wait4_time64(),
>>>
>>> I placed an illegal instruction in __wait3. This executes instead of
>>> the call to __stack_chk_fail because that obliterates stack memory of
>>> interest.
>>
>> OK.
>>
>>>
>>> Consequently the latest core dump still contains dead stack frames
>>> (see below) of subroutines that returned before __wait3() dumped core.
>>> You can see the return address for the branch to __wait4_time64() and
>>> below that you can see the return address for the branch to
>>> __m68k_read_tp().
>>>
>>> (gdb) disas __wait4_time64
>>> Dump of assembler code for function __GI___wait4_time64:
>>>    0xc00e4174 <+0>:     lea %sp@(-80),%sp
>>>    0xc00e4178 <+4>:     moveml %d2-%d5/%a2-%a3/%a5,%sp@-
>>>    0xc00e417c <+8>:     lea %pc@(0xc019c000),%a5
>>>    0xc00e4184 <+16>:    movel %sp@(116),%d2
>>>    0xc00e4188 <+20>:    moveal %sp@(124),%a2
>>>    0xc00e418c <+24>:    moveal %a5@(108),%a3
>>>    0xc00e4190 <+28>:    movel %a3@,%sp@(104)
>>>    0xc00e4194 <+32>:    bsrl 0xc0056e2c <__m68k_read_tp@plt>
>>>
>>> I gather the signal was delivered before __wait4_time64+38, otherwise
>>> the return address 0xc00e419a (which appears below) would have been
>>> overwritten by the signal frame. The signal must have been delivered
>>> after waitproc() initialized gotsigchld = 0 since gotsigchld is 1 at
>>> the time of the coredump.
>>>
>>> I assume the %a3 corruption happened after __wait4_time64+8 because
>>> that's when %a3 first appears on the stack. And the corruption must
>>> have happened before __wait4_time64+238, which is when %a3 was
>>> restored.
>>>
>>> If it was the signal which somehow corrupted the saved %a3, there's
>>> only a small window for that. The only syscall in that window is
>>> get_thread_area.
>>
>> I see sys_wait4 called in two places (0xc00e01b4, and then 0xc00e0286
>> depending on the return code of the first). The second one again would
>> have called __m68k_read_tp so would have left a return address on the
>> stack (0xc00e02d2). Leaves the first.
>>
>
> That's why my analysis stopped at __wait4_time64+38: the rest of
> __wait4_time64 is not relevant to the dead stack contents. (It would have
> left a different return address in that memory location.)
>
>>>
>>> Here's some stack memory from the core dump.
>>>
>>> 0xeffff0dc:     0xd000c38e      return address waitproc+124
>>> 0xeffff0d8:     0xd001c1ec      frame 0 $fp                   ==
>>> &suppressint
>>> 0xeffff0d4:     0x00add14b      canary
>>> 0xeffff0d0:     0x00000000
>>> 0xeffff0cc:     0x0000000a
>>> 0xeffff0c8:     0x00000202
>>> 0xeffff0c4:     0x00000008
>>> 0xeffff0c0:     0x00000000
>>> 0xeffff0bc:     0x00000000
>>> 0xeffff0b8:     0x00000174
>>> 0xeffff0b4:     0x00000004
>>> 0xeffff0b0:     0x00000004
>>> 0xeffff0ac:     0x00000006
>>> 0xeffff0a8:     0x000000e0
>>> 0xeffff0a4:     0x000000e0
>>> 0xeffff0a0:     0x00171f20
>>> 0xeffff09c:     0x00171f20
>>> 0xeffff098:     0x00171f20
>>> 0xeffff094:     0x00000002
>>> 0xeffff090:     0x00002000
>>> 0xeffff08c:     0x00000006
>>> 0xeffff088:     0x0000e920
>>> 0xeffff084:     0x00005360
>>> 0xeffff080:     0x00170700
>>> 0xeffff07c:     0x00170700
>>> 0xeffff078:     0x00170700      frame 0 $fp - 96
>>> 0xeffff074:     0xd001b874                         saved $a5 == dash .got
>>> 0xeffff070:     0xd001e498                         saved $a3 == &dash_errno
>>> 0xeffff06c:     0xd001e718      frame 0 $sp        saved $a2 == &gotsigchld
>>> 0xeffff068:     0x00000000
>>> 0xeffff064:     0x00000000
>>> 0xeffff060:     0xeffff11e
>>> 0xeffff05c:     0xffffffff
>>> 0xeffff058:     0xc00e4164      return address __wait3+244
>>> 0xeffff054:     0x00add14b      canary
>>> 0xeffff050:     0x00000001
>>> 0xeffff04c:     0x00000004
>>> 0xeffff048:     0x0000000d
>>> 0xeffff044:     0x0000000d
>>> 0xeffff040:     0x0015ef82
>>> 0xeffff03c:     0x0015ef82
>>> 0xeffff038:     0x0015ef82
>>> 0xeffff034:     0x00000003
>>> 0xeffff030:     0x00000004
>>> 0xeffff02c:     0x00000004
>>> 0xeffff028:     0x00000140
>>> 0xeffff024:     0x00000140
>>> 0xeffff020:     0x00000034
>>> 0xeffff01c:     0x00000034
>>> 0xeffff018:     0x00000034
>>> 0xeffff014:     0x00000006
>>> 0xeffff010:     0x003b003a
>>> 0xeffff00c:     0x000a0028
>>> 0xeffff008:     0x00340020
>>> 0xeffff004:     0xc019c000                      saved $a5 == libc .got
>>> 0xeffff000:     0xeffff068                      saved $a3 (corrupted)
>>> 0xefffeffc:     0x00000000                      saved $a2
>>> 0xefffeff8:     0x00000001                      saved $d5
>>> 0xefffeff4:     0xeffff122                      saved $d4
>>> 0xefffeff0:     0xeffff11e                      saved $d3
>>> 0xefffefec:     0x00000000                      saved $d2
>>> 0xefffefe8:     0xc00e419a      return address __GI___wait4_time64+38
>>> 0xefffefe4:     0xc0028780
>>> 0xefffefe0:     0x3c344bfb
>>> 0xefffefdc:     0x000af353
>>> 0xefffefd8:     0x3c340170
>>> 0xefffefd4:     0x00000000
>>> 0xefffefd0:     0xc00e417c
>>> 0xefffefcc:     0xc00e417e
>>> 0xefffefc8:     0xc00e4180
>>> 0xefffefc4:     0x48e73c34
>>> 0xefffefc0:     0x00000000
>>> 0xefffefbc:     0xefffeff8
>>> 0xefffefb8:     0xefffeffc
>>> 0xefffefb4:     0x4bfb0170
>>> 0xefffefb0:     0x0eee0709
>>> 0xefffefac:     0x00000000
>>> 0xefffefa8:     0x00000000
>>> 0xefffefa4:     0x00000000
>>> 0xefffefa0:     0x00000000
>>> 0xefffef9c:     0x00000000
>>> 0xefffef98:     0x00000000
>>> 0xefffef94:     0x00000000
>>> 0xefffef90:     0x00000000
>>> 0xefffef8c:     0x00000000
>>> 0xefffef88:     0x00000000
>>> 0xefffef84:     0x00000000
>>> 0xefffef80:     0x00000000
>>> 0xefffef7c:     0x00000000
>>> 0xefffef78:     0x00000000
>>> 0xefffef74:     0x00000000
>>> 0xefffef70:     0x00000000
>>> 0xefffef6c:     0x00000000
>>> 0xefffef68:     0x00000000
>>> 0xefffef64:     0x00000000
>>> 0xefffef60:     0x00000000
>>> 0xefffef5c:     0x00000000
>>> 0xefffef58:     0x00000000
>>> 0xefffef54:     0x00000000
>>> 0xefffef50:     0x00000000
>>> 0xefffef4c:     0x00000000
>>> 0xefffef48:     0x00000000
>>> 0xefffef44:     0x00000000
>>> 0xefffef40:     0x00000000
>>> 0xefffef3c:     0x00000000
>>> 0xefffef38:     0x00000000
>>> 0xefffef34:     0x00000000
>>> 0xefffef30:     0x00000000
>>> 0xefffef2c:     0x00000000
>>> 0xefffef28:     0x00000000
>>> 0xefffef24:     0x00000000
>>> 0xefffef20:     0x00000000
>>> 0xefffef1c:     0x00000000
>>> 0xefffef18:     0x00000000
>>> 0xefffef14:     0x00000000
>>> 0xefffef10:     0x7c0effff
>>> 0xefffef0c:     0xffffffff
>>> 0xefffef08:     0xaaaaaaaa
>>> 0xefffef04:     0xaf54eaaa
>>> 0xefffef00:     0x40040000
>>> 0xefffeefc:     0x40040000
>>> 0xefffeef8:     0x2b000000
>>> 0xefffeef4:     0x00000000
>>> 0xefffeef0:     0x00000000
>>> 0xefffeeec:     0x408ece9a
>>> 0xefffeee8:     0x00000000
>>> 0xefffeee4:     0xf0ff0000
>>> 0xefffeee0:     0x0f800000
>>> 0xefffeedc:     0xf0fff0ff
>>> 0xefffeed8:     0x1f380000
>>> 0xefffeed4:     0x00000000
>>> 0xefffeed0:     0x00000000
>>> 0xefffeecc:     0x00000000	
>>> 0xefffeec8:     0xffffffff	
>>> 0xefffeec4:     0xffffffff	
>>> 0xefffeec0:     0x7fff0000	
>>> 0xefffeebc:     0xffffffff	
>>> 0xefffeeb8:     0xffffffff
>>> 0xefffeeb4:     0x7fff0000	sc_formatvec
>>>
>>> The signal frame is not readily apparent (to me).
>>
>> From looking at the above stack dump, sc ought to start at 0xefffee90,
>> and the trampoline would be three words below that.
>
> 0xefffeeb0:     0x4178b008	sc_pc, sc_formatvec
> 0xefffeeac:     0x0008c00e	sc_sr, sc_pc
> 0xefffeea8:     0xd00223bb	sc_a1
> 0xefffeea4:     0xd001e32c	sc_a0
> 0xefffeea0:     0x00000003	sc_d1
> 0xefffee9c:     0xeffff11e	sc_d0
> 0xefffee98:     0xeffff004	sc_usp
> 0xefffee94:     0x00000000	sc_mask
> 0xefffee90:     0x00000000	extramask
> 0xefffee8c:     0xc0024a90	retcode[1]
> 0xefffee88:     0x70774e40	retcode[0]
> 0xefffee84:     0xefffee94	psc
> 0xefffee80:     0x00000008	code
> 0xefffee7c:     0x00000011	sig
> 0xefffee78:     0xefffee88	pretcode

OK, that's our  SIGCHLD. But the signal frame format is odd ...

Frame format b, vector offset 008. That's a bus error? How does that get 
on the user mode stack?

> 0xefffee74:     0xc019c000
> 0xefffee70:     0x00000000
> 0xefffee6c:     0xc0025878
> 0xefffee68:     0xc0007ed4
> 0xefffee64:     0xc0024000
> 0xefffee60:     0xefffef50
> 0xefffee5c:     0xc0024000
> 0xefffee58:     0xc002a034
> 0xefffee54:     0xc0024a90
> 0xefffee50:     0xc0025878
> 0xefffee4c:     0x00000001
> 0xefffee48:     0x0017f020
> 0xefffee44:     0x0000002c
> 0xefffee40:     0x0000000f
> 0xefffee3c:     0x00000000
> 0xefffee38:     0xfffff7fa
> 0xefffee34:     0xffffffff
> 0xefffee30:     0x00009782
> 0xefffee2c:     0x00000000
> 0xefffee28:     0x0000001e
> 0xefffee24:     0xc0025858
> 0xefffee20:     0xc0025af8
> 0xefffee1c:     0xc000b376
> 0xefffee18:     0xc0024000
> 0xefffee14:     0xc0025878
> 0xefffee10:     0x0000001d
> 0xefffee0c:     0xd0001b60
> 0xefffee08:     0x0000002f
> 0xefffee04:     0xc002563e
> 0xefffee00:     0xc0025490
>
>> The last address you show corresponds to 0xeffff640 in first dump above,
>> which is at the start of the saved fpregs. I'd say we just miss the
>> beginning of the signal frame?
>>
>
> It looks like you're right. I'm not sure how I missed that.
>
> So when the signal was delivered, PC == 0xc00e4178 and USP == 0xc00e4178.

USP is 0xeffff004 AFAICS. That's the location 15 was saved to above 
(holding libc .got according to your interpretation).

The saved PC is that from the exception frame, in this case a long bus 
error sequence fault frame. The PC is that of the instruction executing 
when the fault occurred. As you say, that's the moveml saving registers 
to the stack.

I don't believe the whole fault frame is on the signal stack in one 
contiguous piece, just the first four words, then we have struct 
sigcontext. But after that, the extra contents follows, and that nicely 
explains the extra bits right below the return address from the 
__m68k_read_tp call.

> Those addresses can be found in the disassembly and the stack contents I
> sent previously (quoted above) and it all seems to line up.
>
>> (My reasoning is that copy_siginfo_to_user clears the end of the signal
>> stack, which is what we can see in both cases.)
>>
>> Can't explain the 14 words below the saved return address though.
>>
>
> Right. Is it sc_fpstate? Perhaps we should expect QEMU to differ here.

See above - I think what's stored there is the extra frame content for a 
format b bus error frame. But that extra frame is incomplete at best 
(should be 22 longwords, only a4 are seen). Probably overwritten by the 
stack frame from __GI___wait4_time64.

Let's parse what's left:
<=
 >>> 0xefffefe4:     0xc0028780		<= internal registers (6x)
 >>> 0xefffefe0:     0x3c344bfb		<=
 >>> 0xefffefdc:     0x000af353		<=
 >>> 0xefffefd8:     0x3c340170		<= internal reg; version no.
 >>> 0xefffefd4:     0x00000000		<= data input buffer
 >>> 0xefffefd0:     0xc00e417c		<= internal registers (2x)
 >>> 0xefffefcc:     0xc00e417e		<= stage b address
 >>> 0xefffefc8:     0xc00e4180		<= internal registers (4x)
 >>> 0xefffefc4:     0x48e73c34		<=
 >>> 0xefffefc0:     0x00000000		<= data output buffer
 >>> 0xefffefbc:     0xefffeff8		<= internal registers (2x)
 >>> 0xefffefb8:     0xefffeffc		<= data fault address
 >>> 0xefffefb4:     0x4bfb0170		<= ins stage c, stage b
 >>> 0xefffefb0:     0x0eee0709		<= internal register; ssw

The fault address is the location on the stack where a2 is saved. That 
does match the data output buffer contents BTW. fc, fb, rc, rb bits 
clear means the fault didn't occur in stage b or c instructions. ssw bit 
8 set indicates a data fault - the data cycle should be rerun on rte. rm 
and rw bits clear tell us it's a write fault. If the moveml instruction 
copies registers to the stack in descending order, the fault address 
makes sense - the stack pointer just crossed a page boundary.

>
> Bottom line is, the corrupted %a3 register would have been saved by the
> MOVEM instruction at 0xc00e4178, which turns out to be the PC in the
> signal frame. So it certainly looks like the kernel was the culprit here.

I think the moveml instruction did cause a bus error, and on return from 
that exception the signal got delivered.

On entering the buserror handler, only a1 and a2 are saved, but the 
comment in entry.h states that a3-a6 and d6, d7 are preserved by C code. 
After buserr_c returns, a3 should be restored to what it was when taking 
the bus error. All registers restored before rte, the moveml instruction 
ought to be able to resume normally.

Unless that register use constraint has changed, I don't see how a3 
could have changed midway during return from the bus error exception. 
But maybe a disassembly of buserr_c from your kernel could confirm that?

Cheers,

	Michael

Re: core dump analysis, was Re: stack smashing detected

[Finn Thain]

On Tue, 18 Apr 2023, Michael Schmitz wrote:

> Am 18.04.2023 um 14:04 schrieb Finn Thain:
> > On Tue, 18 Apr 2023, Michael Schmitz wrote:
> >> Am 16.04.2023 um 18:44 schrieb Finn Thain:
> >>
> >>> 0xeffff750:     0xc01a0000                      saved $a5 == libc .got
> >>> 0xeffff74c:     0xc0023e8c                      saved $a3 == &__stack_chk_guard
> >>> 0xeffff748:     0x00000000                      saved $a2
> >>> 0xeffff744:     0x00000001                      saved $d5
> >>> 0xeffff740:     0xeffff86e                      saved $d4
> >>> 0xeffff73c:     0xeffff86a                      saved $d3
> >>> 0xeffff738:     0x00000002                      saved $d2
> >>> 0xeffff734:     0x00000000
> >>> 0xeffff730:     0x00000000
> >>> 0xeffff72c:     0x00000000
> >>> 0xeffff728:     0x00000000
> >>> 0xeffff724:     0x00000000
> >>> 0xeffff720:     0x00000000
> >>> 0xeffff71c:     0x00000000
> >>> 0xeffff718:     0x00000000
> >>> 0xeffff714:     0x00000000
> >>> 0xeffff710:     0x00000000
> >>> 0xeffff70c:     0x00000000
> >>> 0xeffff708:     0x00000000
> >>> 0xeffff704:     0x00000000
> >>> 0xeffff700:     0x00000000
> >>> 0xeffff6fc:     0x00000000
> >>> 0xeffff6f8:     0x00000000
> >>> 0xeffff6f4:     0x00000000
> >>> 0xeffff6f0:     0x00000000
> >>> 0xeffff6ec:     0x00000000
> >>> 0xeffff6e8:     0x00000000
> >>> 0xeffff6e4:     0x00000000
> >>> 0xeffff6e0:     0x00000000
> >>> 0xeffff6dc:     0x00000000
> >>> 0xeffff6d8:     0x00000000
> >>> 0xeffff6d4:     0x00000000
> >>> 0xeffff6d0:     0x00000000
> >>> 0xeffff6cc:     0x00000000
> >>> 0xeffff6c8:     0x00000000
> >>> 0xeffff6c4:     0x00000000
> >>> 0xeffff6c0:     0x00000000
> >>> 0xeffff6bc:     0x00000000
> >>> 0xeffff6b8:     0x00000000
> >>> 0xeffff6b4:     0x00000000
> >>> 0xeffff6b0:     0x00000000
> >>> 0xeffff6ac:     0x00000000
> >>> 0xeffff6a8:     0x00000000
> >>> 0xeffff6a4:     0x00000000
> >>> 0xeffff6a0:     0x00000000
> >>> 0xeffff69c:     0x00000000
> >>> 0xeffff698:     0x00000000
> >>> 0xeffff694:     0x00000000
> >>> 0xeffff690:     0x00000000
> >>> 0xeffff68c:     0x00000000
> >>> 0xeffff688:     0x00000000
> >>> 0xeffff684:     0x00000000
> >>> 0xeffff680:     0x00000000
> >>> 0xeffff67c:     0x00000000
> >>> 0xeffff678:     0x00000000
> >>> 0xeffff674:     0x00000000
> >>> 0xeffff670:     0x00000000
> >>> 0xeffff66c:     0x00000000
> >>> 0xeffff668:     0x00000000
> >>> 0xeffff664:     0x00000000
> >>> 0xeffff660:     0x41000000
> >>> 0xeffff65c:     0x00000000
> >>> 0xeffff658:     0x00000000
> >>> 0xeffff654:     0x00000000
> >>> 0xeffff650:     0x00000000
> >>> 0xeffff64c:     0x80000000
> >>> 0xeffff648:     0x3fff0000
> >>> 0xeffff644:     0x00000000
> >>> 0xeffff640:     0xd0000000
> >>> 0xeffff63c:     0x40020000                <= (sc.formatvec & 0xffff) << 16; fpregs from here on
> >>> 0xeffff638:     0x81b60080                <= (sc.pc & 0xffff) << 16 | sc.formatvec >> 16
> >>> 0xeffff634:     0x0000c00e                <= sc.sr << 16  sc.pc >> 16
> >>> 0xeffff630:     0xd001e4e3                <= sc.a1
> >>> 0xeffff62c:     0xc0028780                <= sc.a0
> >>> 0xeffff628:     0xffffffff                <= sc.d1
> >>> 0xeffff624:     0x0000041f                <= sc.d0
> >>> 0xeffff620:     0xeffff738                <= sc.usp
> >>> 0xeffff61c:     0x00000000                <= sc.mask
> >>> 0xeffff618:     0x00000000                <= extramask
> >>> 0xeffff614:     0x00000000                <= frame.retcode[1]
> >>> 0xeffff610:     0x70774e40      moveq #119,%d0 ; trap #0
> >>> 0xeffff60c:     0xeffff61c                <= frame->sc
> >>> 0xeffff608:     0x00000080                <= tregs->vector
> >>> 0xeffff604:     0x00000011                <= signal no.
> >>> 0xeffff600:     0xeffff610      return address
> >>>
> >>> The above comes from dash running under gdb under qemu, which does 
> >>> not exhibit the failure but is convenient for that kind of 
> >>> experiment.
> >>
> >> I would have expected to see a different signal trampoline (for 
> >> sys_rt_sigreturn) ...
> >
> > Well, this seems to be the trampoline from setup_frame() and not 
> > setup_rt_frame().
> 
> According to the manpages I've seen, glibc ought to pick rt signals if 
> the kernel supports those (which I suppose it does).
> 

It's got to be the trampoline from setup_frame() because dash did this:

        act.sa_flags = 0;
        sigfillset(&act.sa_mask);
        sigaction(signo, &act, 0);

and the kernel did this:

        /* set up the stack frame */
        if (ksig->ka.sa.sa_flags & SA_SIGINFO)
                err = setup_rt_frame(ksig, oldset, regs);
        else
                err = setup_frame(ksig, oldset, regs);

> >
> >> But anyway:
> >>
> >> The saved pc is 0xc00e81b6 which does match the backtrace above. 
> >> Vector offset 80 matches trap 0 which suggests 0xc00e81b6 should be 
> >> the instruction after a trap 0 instruction. d0 is 1055 which is not a 
> >> signal number I recognize.
> >>
> >
> > I don't know what d0 represents here. But &frame->sig == 0x11 is 
> > correct (SIGCHLD).
> 
> Correct - that all works out. But d0 holds the syscall number when we 
> enter the kernel via trap 0, and that one is odd.
> 

Well, you showed subsequently that the kernel was probably entered via a 
page fault and not the get_thread_area trap. Would that explain the d0 
value?

> >>> ...
> >>>
> >>> Here's some stack memory from the core dump.
> >>>
> >>> 0xeffff0dc:     0xd000c38e      return address waitproc+124
> >>> 0xeffff0d8:     0xd001c1ec      frame 0 $fp               == &suppressint
> >>> 0xeffff0d4:     0x00add14b      canary
> >>> 0xeffff0d0:     0x00000000
> >>> 0xeffff0cc:     0x0000000a
> >>> 0xeffff0c8:     0x00000202
> >>> 0xeffff0c4:     0x00000008
> >>> 0xeffff0c0:     0x00000000
> >>> 0xeffff0bc:     0x00000000
> >>> 0xeffff0b8:     0x00000174
> >>> 0xeffff0b4:     0x00000004
> >>> 0xeffff0b0:     0x00000004
> >>> 0xeffff0ac:     0x00000006
> >>> 0xeffff0a8:     0x000000e0
> >>> 0xeffff0a4:     0x000000e0
> >>> 0xeffff0a0:     0x00171f20
> >>> 0xeffff09c:     0x00171f20
> >>> 0xeffff098:     0x00171f20
> >>> 0xeffff094:     0x00000002
> >>> 0xeffff090:     0x00002000
> >>> 0xeffff08c:     0x00000006
> >>> 0xeffff088:     0x0000e920
> >>> 0xeffff084:     0x00005360
> >>> 0xeffff080:     0x00170700
> >>> 0xeffff07c:     0x00170700
> >>> 0xeffff078:     0x00170700      frame 0 $fp - 96
> >>> 0xeffff074:     0xd001b874                         saved $a5 == dash .got
> >>> 0xeffff070:     0xd001e498                         saved $a3 == &dash_errno
> >>> 0xeffff06c:     0xd001e718      frame 0 $sp        saved $a2 == &gotsigchld
> >>> 0xeffff068:     0x00000000
> >>> 0xeffff064:     0x00000000
> >>> 0xeffff060:     0xeffff11e
> >>> 0xeffff05c:     0xffffffff
> >>> 0xeffff058:     0xc00e4164      return address __wait3+244
> >>> 0xeffff054:     0x00add14b      canary
> >>> 0xeffff050:     0x00000001
> >>> 0xeffff04c:     0x00000004
> >>> 0xeffff048:     0x0000000d
> >>> 0xeffff044:     0x0000000d
> >>> 0xeffff040:     0x0015ef82
> >>> 0xeffff03c:     0x0015ef82
> >>> 0xeffff038:     0x0015ef82
> >>> 0xeffff034:     0x00000003
> >>> 0xeffff030:     0x00000004
> >>> 0xeffff02c:     0x00000004
> >>> 0xeffff028:     0x00000140
> >>> 0xeffff024:     0x00000140
> >>> 0xeffff020:     0x00000034
> >>> 0xeffff01c:     0x00000034
> >>> 0xeffff018:     0x00000034
> >>> 0xeffff014:     0x00000006
> >>> 0xeffff010:     0x003b003a
> >>> 0xeffff00c:     0x000a0028
> >>> 0xeffff008:     0x00340020
> >>> 0xeffff004:     0xc019c000                      saved $a5 == libc .got
> >>> 0xeffff000:     0xeffff068                      saved $a3 (corrupted)
> >>> 0xefffeffc:     0x00000000                      saved $a2
> >>> 0xefffeff8:     0x00000001                      saved $d5
> >>> 0xefffeff4:     0xeffff122                      saved $d4
> >>> 0xefffeff0:     0xeffff11e                      saved $d3
> >>> 0xefffefec:     0x00000000                      saved $d2
> >>> 0xefffefe8:     0xc00e419a      return address __GI___wait4_time64+38
> >>> 0xefffefe4:     0xc0028780
> >>> 0xefffefe0:     0x3c344bfb
> >>> 0xefffefdc:     0x000af353
> >>> 0xefffefd8:     0x3c340170
> >>> 0xefffefd4:     0x00000000
> >>> 0xefffefd0:     0xc00e417c
> >>> 0xefffefcc:     0xc00e417e
> >>> 0xefffefc8:     0xc00e4180
> >>> 0xefffefc4:     0x48e73c34
> >>> 0xefffefc0:     0x00000000
> >>> 0xefffefbc:     0xefffeff8
> >>> 0xefffefb8:     0xefffeffc
> >>> 0xefffefb4:     0x4bfb0170
> >>> 0xefffefb0:     0x0eee0709
> >>> 0xefffefac:     0x00000000
> >>> 0xefffefa8:     0x00000000
> >>> 0xefffefa4:     0x00000000
> >>> 0xefffefa0:     0x00000000
> >>> 0xefffef9c:     0x00000000
> >>> 0xefffef98:     0x00000000
> >>> 0xefffef94:     0x00000000
> >>> 0xefffef90:     0x00000000
> >>> 0xefffef8c:     0x00000000
> >>> 0xefffef88:     0x00000000
> >>> 0xefffef84:     0x00000000
> >>> 0xefffef80:     0x00000000
> >>> 0xefffef7c:     0x00000000
> >>> 0xefffef78:     0x00000000
> >>> 0xefffef74:     0x00000000
> >>> 0xefffef70:     0x00000000
> >>> 0xefffef6c:     0x00000000
> >>> 0xefffef68:     0x00000000
> >>> 0xefffef64:     0x00000000
> >>> 0xefffef60:     0x00000000
> >>> 0xefffef5c:     0x00000000
> >>> 0xefffef58:     0x00000000
> >>> 0xefffef54:     0x00000000
> >>> 0xefffef50:     0x00000000
> >>> 0xefffef4c:     0x00000000
> >>> 0xefffef48:     0x00000000
> >>> 0xefffef44:     0x00000000
> >>> 0xefffef40:     0x00000000
> >>> 0xefffef3c:     0x00000000
> >>> 0xefffef38:     0x00000000
> >>> 0xefffef34:     0x00000000
> >>> 0xefffef30:     0x00000000
> >>> 0xefffef2c:     0x00000000
> >>> 0xefffef28:     0x00000000
> >>> 0xefffef24:     0x00000000
> >>> 0xefffef20:     0x00000000
> >>> 0xefffef1c:     0x00000000
> >>> 0xefffef18:     0x00000000
> >>> 0xefffef14:     0x00000000
> >>> 0xefffef10:     0x7c0effff
> >>> 0xefffef0c:     0xffffffff
> >>> 0xefffef08:     0xaaaaaaaa
> >>> 0xefffef04:     0xaf54eaaa
> >>> 0xefffef00:     0x40040000
> >>> 0xefffeefc:     0x40040000
> >>> 0xefffeef8:     0x2b000000
> >>> 0xefffeef4:     0x00000000
> >>> 0xefffeef0:     0x00000000
> >>> 0xefffeeec:     0x408ece9a
> >>> 0xefffeee8:     0x00000000
> >>> 0xefffeee4:     0xf0ff0000
> >>> 0xefffeee0:     0x0f800000
> >>> 0xefffeedc:     0xf0fff0ff
> >>> 0xefffeed8:     0x1f380000
> >>> 0xefffeed4:     0x00000000
> >>> 0xefffeed0:     0x00000000
> >>> 0xefffeecc:     0x00000000    
> >>> 0xefffeec8:     0xffffffff    
> >>> 0xefffeec4:     0xffffffff    
> >>> 0xefffeec0:     0x7fff0000    
> >>> 0xefffeebc:     0xffffffff    
> >>> 0xefffeeb8:     0xffffffff
> >>> 0xefffeeb4:     0x7fff0000    sc_formatvec
> >>>
> >>> The signal frame is not readily apparent (to me).
> >>
> >> From looking at the above stack dump, sc ought to start at 0xefffee90,
> >> and the trampoline would be three words below that.
> >
> > 0xefffeeb0:     0x4178b008    sc_pc, sc_formatvec
> > 0xefffeeac:     0x0008c00e    sc_sr, sc_pc
> > 0xefffeea8:     0xd00223bb    sc_a1
> > 0xefffeea4:     0xd001e32c    sc_a0
> > 0xefffeea0:     0x00000003    sc_d1
> > 0xefffee9c:     0xeffff11e    sc_d0
> > 0xefffee98:     0xeffff004    sc_usp
> > 0xefffee94:     0x00000000    sc_mask
> > 0xefffee90:     0x00000000    extramask
> > 0xefffee8c:     0xc0024a90    retcode[1]
> > 0xefffee88:     0x70774e40    retcode[0]
> > 0xefffee84:     0xefffee94    psc
> > 0xefffee80:     0x00000008    code
> > 0xefffee7c:     0x00000011    sig
> > 0xefffee78:     0xefffee88    pretcode
> 
> OK, that's our  SIGCHLD. But the signal frame format is odd ...
> 
> Frame format b, vector offset 008. That's a bus error? 
> How does that get on the user mode stack?
> 
> > 0xefffee74:     0xc019c000
> > 0xefffee70:     0x00000000
> > 0xefffee6c:     0xc0025878
> > 0xefffee68:     0xc0007ed4
> > 0xefffee64:     0xc0024000
> > 0xefffee60:     0xefffef50
> > 0xefffee5c:     0xc0024000
> > 0xefffee58:     0xc002a034
> > 0xefffee54:     0xc0024a90
> > 0xefffee50:     0xc0025878
> > 0xefffee4c:     0x00000001
> > 0xefffee48:     0x0017f020
> > 0xefffee44:     0x0000002c
> > 0xefffee40:     0x0000000f
> > 0xefffee3c:     0x00000000
> > 0xefffee38:     0xfffff7fa
> > 0xefffee34:     0xffffffff
> > 0xefffee30:     0x00009782
> > 0xefffee2c:     0x00000000
> > 0xefffee28:     0x0000001e
> > 0xefffee24:     0xc0025858
> > 0xefffee20:     0xc0025af8
> > 0xefffee1c:     0xc000b376
> > 0xefffee18:     0xc0024000
> > 0xefffee14:     0xc0025878
> > 0xefffee10:     0x0000001d
> > 0xefffee0c:     0xd0001b60
> > 0xefffee08:     0x0000002f
> > 0xefffee04:     0xc002563e
> > 0xefffee00:     0xc0025490
> >
> >> The last address you show corresponds to 0xeffff640 in first dump 
> >> above, which is at the start of the saved fpregs. I'd say we just 
> >> miss the beginning of the signal frame?
> >>
> >
> > It looks like you're right. I'm not sure how I missed that.
> >
> > So when the signal was delivered, PC == 0xc00e4178 and USP == 
> > 0xc00e4178.
> 
> USP is 0xeffff004 AFAICS. That's the location 15 was saved to above 
> (holding libc .got according to your interpretation).
> 

Right, it was a typo. USP is 0xeffff004, where a5 is to be saved.

> The saved PC is that from the exception frame, in this case a long bus 
> error sequence fault frame. The PC is that of the instruction executing 
> when the fault occurred. As you say, that's the moveml saving registers 
> to the stack.
> 
> I don't believe the whole fault frame is on the signal stack in one 
> contiguous piece, just the first four words, then we have struct 
> sigcontext. But after that, the extra contents follows, and that nicely 
> explains the extra bits right below the return address from the 
> __m68k_read_tp call.
> 
> > Those addresses can be found in the disassembly and the stack contents 
> > I sent previously (quoted above) and it all seems to line up.
> >
> >> (My reasoning is that copy_siginfo_to_user clears the end of the 
> >> signal stack, which is what we can see in both cases.)
> >>
> >> Can't explain the 14 words below the saved return address though.
> >>
> >
> > Right. Is it sc_fpstate? Perhaps we should expect QEMU to differ here.
> 
> See above - I think what's stored there is the extra frame content for a 
> format b bus error frame. But that extra frame is incomplete at best 
> (should be 22 longwords, only a4 are seen). Probably overwritten by the 
> stack frame from __GI___wait4_time64.
> 

Maybe the exception frame leaked onto the user stack via setup_frame()?

> Let's parse what's left:
> <=
> >>> 0xefffefe4:     0xc0028780        <= internal registers (6x)
> >>> 0xefffefe0:     0x3c344bfb        <=
> >>> 0xefffefdc:     0x000af353        <=
> >>> 0xefffefd8:     0x3c340170        <= internal reg; version no.
> >>> 0xefffefd4:     0x00000000        <= data input buffer
> >>> 0xefffefd0:     0xc00e417c        <= internal registers (2x)
> >>> 0xefffefcc:     0xc00e417e        <= stage b address
> >>> 0xefffefc8:     0xc00e4180        <= internal registers (4x)
> >>> 0xefffefc4:     0x48e73c34        <=
> >>> 0xefffefc0:     0x00000000        <= data output buffer
> >>> 0xefffefbc:     0xefffeff8        <= internal registers (2x)
> >>> 0xefffefb8:     0xefffeffc        <= data fault address
> >>> 0xefffefb4:     0x4bfb0170        <= ins stage c, stage b
> >>> 0xefffefb0:     0x0eee0709        <= internal register; ssw
> 
> The fault address is the location on the stack where a2 is saved. That 
> does match the data output buffer contents BTW. fc, fb, rc, rb bits 
> clear means the fault didn't occur in stage b or c instructions. ssw bit 
> 8 set indicates a data fault - the data cycle should be rerun on rte. rm 
> and rw bits clear tell us it's a write fault. If the moveml instruction 
> copies registers to the stack in descending order, the fault address 
> makes sense - the stack pointer just crossed a page boundary.
> 

Well spotted! 

> >
> > Bottom line is, the corrupted %a3 register would have been saved by 
> > the MOVEM instruction at 0xc00e4178, which turns out to be the PC in 
> > the signal frame. So it certainly looks like the kernel was the 
> > culprit here.
> 
> I think the moveml instruction did cause a bus error, and on return from 
> that exception the signal got delivered.
> 

Maybe the signal frame was partially overwritten by the resumed MOVEM?

I wonder what we'd see if we patched the kernel to log every user data 
write fault caused by a MOVEM instruction. I'll try to code that up.

> On entering the buserror handler, only a1 and a2 are saved, but the 
> comment in entry.h states that a3-a6 and d6, d7 are preserved by C code. 
> After buserr_c returns, a3 should be restored to what it was when taking 
> the bus error. All registers restored before rte, the moveml instruction 
> ought to be able to resume normally.
> 
> Unless that register use constraint has changed, I don't see how a3 
> could have changed midway during return from the bus error exception. 
> But maybe a disassembly of buserr_c from your kernel could confirm that?
> 

I disassembled the relevant build. AFAICT, buserr_c() saves and restores 
those registers in the right places.

BTW, I've reproduced the failures with kernels built with both GCC 12 and 
GCC 6.

Re: core dump analysis, was Re: stack smashing detected

[Michael Schmitz]

Hi Finn,

Am 19.04.2023 um 13:50 schrieb Finn Thain:
>>>> I would have expected to see a different signal trampoline (for
>>>> sys_rt_sigreturn) ...
>>>
>>> Well, this seems to be the trampoline from setup_frame() and not
>>> setup_rt_frame().
>>
>> According to the manpages I've seen, glibc ought to pick rt signals if
>> the kernel supports those (which I suppose it does).
>>
>
> It's got to be the trampoline from setup_frame() because dash did this:
>
>         act.sa_flags = 0;
>         sigfillset(&act.sa_mask);
>         sigaction(signo, &act, 0);

Ah - dash explicitly requests the old format. Make sense then.
>
> and the kernel did this:
>
>         /* set up the stack frame */
>         if (ksig->ka.sa.sa_flags & SA_SIGINFO)
>                 err = setup_rt_frame(ksig, oldset, regs);
>         else
>                 err = setup_frame(ksig, oldset, regs);
>
>>>
>>>> But anyway:
>>>>
>>>> The saved pc is 0xc00e81b6 which does match the backtrace above.
>>>> Vector offset 80 matches trap 0 which suggests 0xc00e81b6 should be
>>>> the instruction after a trap 0 instruction. d0 is 1055 which is not a
>>>> signal number I recognize.
>>>>
>>>
>>> I don't know what d0 represents here. But &frame->sig == 0x11 is
>>> correct (SIGCHLD).
>>
>> Correct - that all works out. But d0 holds the syscall number when we
>> enter the kernel via trap 0, and that one is odd.
>>
>
> Well, you showed subsequently that the kernel was probably entered via a
> page fault and not the get_thread_area trap. Would that explain the d0
> value?

That d0 was from the dash under gdb run. But I got my signal delivery 
mixed up - d0 is only expected to hold the syscall number when we issue 
a syscall. That would be in the child process, not the parent which we 
debug.

d0 is just whatever the parent had in its register when it started 
do_signal_return after exception or syscall. On return after syscall, d0 
holds the task info flags, maybe that's what we see here.

>> See above - I think what's stored there is the extra frame content for a
>> format b bus error frame. But that extra frame is incomplete at best
>> (should be 22 longwords, only a4 are seen). Probably overwritten by the
>> stack frame from __GI___wait4_time64.
>>
>
> Maybe the exception frame leaked onto the user stack via setup_frame()?

Yes, for exception frames larger than four words the excess is copied 
after the end of the sigcontext block.

>
>> Let's parse what's left:
>> <=
>>>>> 0xefffefe4:     0xc0028780        <= internal registers (6x)
>>>>> 0xefffefe0:     0x3c344bfb        <=
>>>>> 0xefffefdc:     0x000af353        <=
>>>>> 0xefffefd8:     0x3c340170        <= internal reg; version no.
>>>>> 0xefffefd4:     0x00000000        <= data input buffer
>>>>> 0xefffefd0:     0xc00e417c        <= internal registers (2x)
>>>>> 0xefffefcc:     0xc00e417e        <= stage b address
>>>>> 0xefffefc8:     0xc00e4180        <= internal registers (4x)
>>>>> 0xefffefc4:     0x48e73c34        <=
>>>>> 0xefffefc0:     0x00000000        <= data output buffer
>>>>> 0xefffefbc:     0xefffeff8        <= internal registers (2x)
>>>>> 0xefffefb8:     0xefffeffc        <= data fault address
>>>>> 0xefffefb4:     0x4bfb0170        <= ins stage c, stage b
>>>>> 0xefffefb0:     0x0eee0709        <= internal register; ssw
>>
>> The fault address is the location on the stack where a2 is saved. That
>> does match the data output buffer contents BTW. fc, fb, rc, rb bits
>> clear means the fault didn't occur in stage b or c instructions. ssw bit
>> 8 set indicates a data fault - the data cycle should be rerun on rte. rm
>> and rw bits clear tell us it's a write fault. If the moveml instruction
>> copies registers to the stack in descending order, the fault address
>> makes sense - the stack pointer just crossed a page boundary.
>>
>
> Well spotted!
>
>>>
>>> Bottom line is, the corrupted %a3 register would have been saved by
>>> the MOVEM instruction at 0xc00e4178, which turns out to be the PC in
>>> the signal frame. So it certainly looks like the kernel was the
>>> culprit here.
>>
>> I think the moveml instruction did cause a bus error, and on return from
>> that exception the signal got delivered.
>>
>
> Maybe the signal frame was partially overwritten by the resumed MOVEM?

That's possible - the saved usp in the signal frame is that of the first 
register saved to the stack (before the page fault).

> I wonder what we'd see if we patched the kernel to log every user data
> write fault caused by a MOVEM instruction. I'll try to code that up.

If these instructions did always cause stack corruption on 030, I think 
we would have noticed long ago?

>
>> On entering the buserror handler, only a1 and a2 are saved, but the
>> comment in entry.h states that a3-a6 and d6, d7 are preserved by C code.
>> After buserr_c returns, a3 should be restored to what it was when taking
>> the bus error. All registers restored before rte, the moveml instruction
>> ought to be able to resume normally.
>>
>> Unless that register use constraint has changed, I don't see how a3
>> could have changed midway during return from the bus error exception.
>> But maybe a disassembly of buserr_c from your kernel could confirm that?
>>
>
> I disassembled the relevant build. AFAICT, buserr_c() saves and restores
> those registers in the right places.
>
> BTW, I've reproduced the failures with kernels built with both GCC 12 and
> GCC 6.

Thanks - that was highly unlikely but had to be checked.

Leaves the possibility that some kernel bug did corrupt the saved a3 
copy in struct switch_stack... but that is not used in bus error 
exceptions. And the only other use of a3 is in ret_from_kernel_thread 
which is called only from copy_thread() ...

Still baffled...

Cheers,

	Michael

Re: core dump analysis, was Re: stack smashing detected

[Finn Thain]

On Wed, 19 Apr 2023, Michael Schmitz wrote:

> > I wonder what we'd see if we patched the kernel to log every user data 
> > write fault caused by a MOVEM instruction. I'll try to code that up.
> 
> If these instructions did always cause stack corruption on 030, I think 
> we would have noticed long ago?
> 

I think it probably was noticed long ago, in the form of rare userland 
crashes on 68030. But it was probably never reported because the actual 
culprit is too distant from the symptoms.

But I take your point -- signal delivery seems to be crucial. Would it be 
difficult to skip signal delivery following a bus error? Perhaps there's 
no need to try that experiment, as we know what would happen.

I will take a look at your modified test program and try to use the output 
to figure out the stack gymnastics.

IIUC, there are two RTEs following the page fault. The first one runs the 
signal handler, the second one resumes the MOVEM that faulted. Maybe we'll 
have to intercept the latter (at do_sigreturn() perhaps?) and examine that 
exception frame.

Re: core dump analysis, was Re: stack smashing detected

[Michael Schmitz]

Hi Finn,

On 23/04/23 19:46, Finn Thain wrote:
> On Wed, 19 Apr 2023, Michael Schmitz wrote:
>
>>> I wonder what we'd see if we patched the kernel to log every user data
>>> write fault caused by a MOVEM instruction. I'll try to code that up.
>> If these instructions did always cause stack corruption on 030, I think
>> we would have noticed long ago?
>>
> I think it probably was noticed long ago, in the form of rare userland
> crashes on 68030. But it was probably never reported because the actual
> culprit is too distant from the symptoms.
>
> But I take your point -- signal delivery seems to be crucial. Would it be
> difficult to skip signal delivery following a bus error? Perhaps there's
> no need to try that experiment, as we know what would happen.
Shouldn't be too hard, see my other mail.
> I will take a look at your modified test program and try to use the output
> to figure out the stack gymnastics.
>
> IIUC, there are two RTEs following the page fault. The first one runs the
> signal handler, the second one resumes the MOVEM that faulted. Maybe we'll
> have to intercept the latter (at do_sigreturn() perhaps?) and examine that
> exception frame.

There's no second RTE as far as I can see - upon return from buserr_c, 
the asm buserr handler jumps to ret_from_exception. Seeing as the bus 
error was taken from user space, ret_from_exception proceeds to 
resume_userspace, and seeing the task info flags field non-zero, jumps 
to exit_work where with signal pending, a jump to do_signal_return is 
taken and the signal handler is set up (frame setup to return through 
the sigreturn trampoline, pc set to hander etc). No rte anywhere on that 
path. After setting up for the signal handler, we return to 
resume_userspace and no further signals pending, hit RESTORE_ALL which 
restores registes from the pt_regs struct on the kernel stack, and has 
the rte instruction at the end. We had earlier set usp to the signal 
frame and pc to the signal handler, so that is now run after resuming 
user mode after the rte instruction.

Exiting from the signal handler, sys_sigreturn runs and cleans up the 
user stack, then returns to the instruction at the pc from the saved 
exception frame that got us into kernel mode in the first instance. This 
is the moment the moveml instruction resumes.

There should be no difference between ret_from_exception (after buserr) 
jumping to RESTORE_ALL directly (with exception frame still on the 
kernel stack from the bus error exception) and doing so after the detour 
through signal hander setup, signal handler and sys_sigreturn cleanup. 
If the exception frame on the stack was any different from what it ought 
to be, rte would fail and raise a format error exception.

If the frame was different from that needed to complete the bus error 
exception, f.e. one from a trap exception, we'd fail to resume that 
moveml instruction and do something else instead. Hmmm - that's an 
interesting fault mode... might explain why a3 wasn't saved as it ought 
to have been? Can we 'poison' the user stack area that will be used for 
register save upon rec() entry with some other patterns to prove that 
moveml sometimes does not complete after the bus error?

Cheers,

     Michael

reliable reproducer, was Re: core dump analysis

[Finn Thain]

On Tue, 18 Apr 2023, Michael Schmitz wrote:

> 
> ... I think what's stored there is the extra frame content for a format 
> b bus error frame. But that extra frame is incomplete at best (should be 
> 22 longwords, only a4 are seen). Probably overwritten by the stack frame 
> from __GI___wait4_time64.
> 
> Let's parse what's left:
> <=
> >>> 0xefffefe4:     0xc0028780		<= internal registers (6x)
> >>> 0xefffefe0:     0x3c344bfb		<=
> >>> 0xefffefdc:     0x000af353		<=
> >>> 0xefffefd8:     0x3c340170		<= internal reg; version no.
> >>> 0xefffefd4:     0x00000000		<= data input buffer
> >>> 0xefffefd0:     0xc00e417c		<= internal registers (2x)
> >>> 0xefffefcc:     0xc00e417e		<= stage b address
> >>> 0xefffefc8:     0xc00e4180		<= internal registers (4x)
> >>> 0xefffefc4:     0x48e73c34		<=
> >>> 0xefffefc0:     0x00000000		<= data output buffer
> >>> 0xefffefbc:     0xefffeff8		<= internal registers (2x)
> >>> 0xefffefb8:     0xefffeffc		<= data fault address
> >>> 0xefffefb4:     0x4bfb0170		<= ins stage c, stage b
> >>> 0xefffefb0:     0x0eee0709		<= internal register; ssw
> 
> The fault address is the location on the stack where a2 is saved. That 
> does match the data output buffer contents BTW. fc, fb, rc, rb bits 
> clear means the fault didn't occur in stage b or c instructions. ssw bit 
> 8 set indicates a data fault - the data cycle should be rerun on rte. rm 
> and rw bits clear tell us it's a write fault. If the moveml instruction 
> copies registers to the stack in descending order, the fault address 
> makes sense - the stack pointer just crossed a page boundary.
> 

Inspired by your observation about the page fault and stack growth, I 
wrote a small test program (given below) that just pushes registers onto 
the stack recursively while forking processes and collecting the SIGCHLD 
signals.

On a Motorola '030 the stack grows to about 7 MiB before it gets 
corrupted. The program detects the stack corruption and terminates 
immediately with an illegal instruction. Oddly, the program never detects 
any stack corruption when run on the QEMU '040.

root@debian:~# ./movem 
Illegal instruction
root@debian:~# ulimit -a
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) 0
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 242
max locked memory           (kbytes, -l) 8192
max memory size             (kbytes, -m) unlimited
open files                          (-n) 1024
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) 8192
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 242
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited
root@debian:~# ulimit -s 7200 
root@debian:~# ./movem 
Illegal instruction
root@debian:~# ulimit -s 7000
root@debian:~# ./movem 
Segmentation fault
root@debian:~# ulimit -s 16384
root@debian:~# ./movem 
Illegal instruction
root@debian:~# 

Looking at the core dump in gdb, the backtrace has 189869 frames. The dead 
stack frames confirm the recursion depth reached the limit I set at 200000 
before the stack began to reduce again. This was also confirmed by the 
lowest page fault address that was logged by the custom kernel.

That means validation succeeded 200000 - 189869 == 10131 times before it 
encountered corruption (I should try to figure out whether this varies).

The registers %a2, %a3 and %a4 below should contain 0x91929394, 0xa1a2a3a4 
and 0xb1b2b3b4 respectively. But they don't. Their values were restored 
from a corrupted stack by the returning rec() function call.

(gdb) info reg
d0             0x91929394          -1852664940
d1             0xf3                243
d2             0xd1d2d3d4          -774712364
d3             0xe1e2e3e4          -505224220
d4             0xf1f2f3f4          -235736076
d5             0x80003f0c          -2147467508
d6             0xd014c528          -803945176
d7             0x0                 0
a0             0xc0021708          0xc0021708
a1             0xc0023e8c          0xc0023e8c <__stack_chk_guard>
a2             0xf3                0xf3
a3             0x1464000           0x1464000
a4             0xef97bf44          0xef97bf44
a5             0xc1c2c3c4          0xc1c2c3c4
fp             0xef97b034          0xef97b034
sp             0xef97b018          0xef97b018
ps             0x8                 [ N ]
pc             0x800005f6          0x800005f6 <rec+262>
fpcontrol      0x0                 0
fpstatus       0x0                 0
fpiaddr        0x0                 0x0

(gdb) x/z $sp - 36
0xef97aff4:     0xd1d2d3d4
(gdb) 
0xef97aff8:     0xe1e2e3e4
(gdb) 
0xef97affc:     0xf1f2f3f4
(gdb) 
0xef97b000:     0x000000f3
(gdb) 
0xef97b004:     0x01464000
(gdb) 
0xef97b008:     0xef97bf44
(gdb) 
0xef97b00c:     0xc1c2c3c4
(gdb) 
0xef97b010:     0xef97b034
(gdb) 
0xef97b014:     0x8000055c

As with dash, the corruption lies the page boundary.

Any signal frames or exception frames have been completely overwritten 
because the recursion continued after the corruption took place. So 
there's not much to see in the core dump.

(gdb) disass rec
Dump of assembler code for function rec:
   0x800004f0 <+0>:     linkw %fp,#0
   0x800004f4 <+4>:     moveml %d2-%d4/%a2-%a5,%sp@-
   0x800004f8 <+8>:     moveal 0x80000672 <i0>,%a2
   0x800004fe <+14>:    moveal 0x80000676 <i1>,%a3
   0x80000504 <+20>:    moveal 0x8000067a <i2>,%a4
   0x8000050a <+26>:    moveal 0x8000067e <i3>,%a5
   0x80000510 <+32>:    movel 0x80000682 <i4>,%d2
   0x80000516 <+38>:    movel 0x80000686 <i5>,%d3
   0x8000051c <+44>:    movel 0x8000068a <i6>,%d4
   0x80000522 <+50>:    movel 0x80004034 <depth>,%d0
   0x80000528 <+56>:    andil #2047,%d0
   0x8000052e <+62>:    bnes 0x80000542 <rec+82>
   0x80000530 <+64>:    jsr 0x8000042c <fork@plt>
   0x80000536 <+70>:    tstl %d0
   0x80000538 <+72>:    bnes 0x80000542 <rec+82>
   0x8000053a <+74>:    clrl %sp@-
   0x8000053c <+76>:    jsr 0x80000404 <exit@plt>
   0x80000542 <+82>:    movel 0x80004034 <depth>,%d0
   0x80000548 <+88>:    subql #1,%d0
   0x8000054a <+90>:    movel %d0,0x80004034 <depth>
   0x80000550 <+96>:    movel 0x80004034 <depth>,%d0
   0x80000556 <+102>:   beqs 0x8000055c <rec+108>
   0x80000558 <+104>:   jsr %pc@(0x800004f0 <rec>)
   0x8000055c <+108>:   movel %a2,0x8000403c <o0>
   0x80000562 <+114>:   movel %a3,0x80004040 <o1>
   0x80000568 <+120>:   movel %a4,0x80004044 <o2>
   0x8000056e <+126>:   movel %a5,0x80004048 <o3>
   0x80000574 <+132>:   movel %d2,0x8000404c <o4>
   0x8000057a <+138>:   movel %d3,0x80004050 <o5>
   0x80000580 <+144>:   movel %d4,0x80004054 <o6>
   0x80000586 <+150>:   movel 0x8000403c <o0>,%d1
   0x8000058c <+156>:   movel #-1852664940,%d0
   0x80000592 <+162>:   cmpl %d1,%d0
   0x80000594 <+164>:   bnes 0x800005f6 <rec+262>
   0x80000596 <+166>:   movel 0x80004040 <o1>,%d1
   0x8000059c <+172>:   movel #-1583176796,%d0
   0x800005a2 <+178>:   cmpl %d1,%d0
   0x800005a4 <+180>:   bnes 0x800005f6 <rec+262>
   0x800005a6 <+182>:   movel 0x80004044 <o2>,%d1
   0x800005ac <+188>:   movel #-1313688652,%d0
   0x800005b2 <+194>:   cmpl %d1,%d0
   0x800005b4 <+196>:   bnes 0x800005f6 <rec+262>
   0x800005b6 <+198>:   movel 0x80004048 <o3>,%d1
   0x800005bc <+204>:   movel #-1044200508,%d0
   0x800005c2 <+210>:   cmpl %d1,%d0
   0x800005c4 <+212>:   bnes 0x800005f6 <rec+262>
   0x800005c6 <+214>:   movel 0x8000404c <o4>,%d1
   0x800005cc <+220>:   movel #-774712364,%d0
   0x800005d2 <+226>:   cmpl %d1,%d0
   0x800005d4 <+228>:   bnes 0x800005f6 <rec+262>
   0x800005d6 <+230>:   movel 0x80004050 <o5>,%d1
   0x800005dc <+236>:   movel #-505224220,%d0
   0x800005e2 <+242>:   cmpl %d1,%d0
   0x800005e4 <+244>:   bnes 0x800005f6 <rec+262>
   0x800005e6 <+246>:   movel 0x80004054 <o6>,%d1
   0x800005ec <+252>:   movel #-235736076,%d0
   0x800005f2 <+258>:   cmpl %d1,%d0
   0x800005f4 <+260>:   beqs 0x800005f8 <rec+264>
=> 0x800005f6 <+262>:   illegal
   0x800005f8 <+264>:   nop
   0x800005fa <+266>:   moveml %fp@(-28),%d2-%d4/%a2-%a5
   0x80000600 <+272>:   unlk %fp
   0x80000602 <+274>:   rts
End of assembler dump.

---

#include <stdlib.h>
#include <unistd.h>
#include <stdio.h>
#include <signal.h>
#include <string.h>

int depth = 200000;

const unsigned long i0 = 0x91929394;
const unsigned long i1 = 0xa1a2a3a4;
const unsigned long i2 = 0xb1b2b3b4;
const unsigned long i3 = 0xc1c2c3c4;
const unsigned long i4 = 0xd1d2d3d4;
const unsigned long i5 = 0xe1e2e3e4;
const unsigned long i6 = 0xf1f2f3f4;

unsigned long o0;
unsigned long o1;
unsigned long o2;
unsigned long o3;
unsigned long o4;
unsigned long o5;
unsigned long o6;

static void rec(void)
{
	// initialize registers
	asm(	"	move.l %0, %%a2\n" 
		"	move.l %1, %%a3\n"
		"	move.l %2, %%a4\n"
		"	move.l %3, %%a5\n"
		"	move.l %4, %%d2\n"
		"	move.l %5, %%d3\n"
		"	move.l %6, %%d4\n"
		:
		: "m" (i0), "m" (i1), "m" (i2),
		  "m" (i3), "m" (i4), "m" (i5), "m" (i6)
		: "a2", "a3", "a4", "a5", "d2", "d3", "d4"
	);

	// maybe fork a short-lived process
	if ((depth & 0x7ff) == 0)
		if (fork() == 0)
			exit(0);

	if (--depth)
		rec();	// callee to save & restore registers

	// compare register contents
	asm(	"	move.l %%a2, %0\n" 
		"	move.l %%a3, %1\n"
		"	move.l %%a4, %2\n"
		"	move.l %%a5, %3\n"
		"	move.l %%d2, %4\n"
		"	move.l %%d3, %5\n"
		"	move.l %%d4, %6\n"
		: "=m" (o0), "=m" (o1), "=m" (o2),
		  "=m" (o3), "=m" (o4), "=m" (o5), "=m" (o6)
		:
		:
	);
	if (o0 != i0 || o1 != i1 || o2 != i2 ||
	    o3 != i3 || o4 != i4 || o5 != i5 || o6 != i6)
		asm("illegal");
}

static void handler(int)
{
}

int main(void)
{
	struct sigaction act;

	memset(&act, 0, sizeof(act));
	act.sa_handler = handler;
	sigaction(SIGCHLD, &act, NULL);

	rec();
}

Re: reliable reproducer, was Re: core dump analysis

[Geert Uytterhoeven]

Hi Finn,

On Wed, Apr 19, 2023 at 12:53 PM Finn Thain <fthain@linux-m68k.org> wrote:
> Inspired by your observation about the page fault and stack growth, I
> wrote a small test program (given below) that just pushes registers onto
> the stack recursively while forking processes and collecting the SIGCHLD
> signals.
>
> On a Motorola '030 the stack grows to about 7 MiB before it gets
> corrupted. The program detects the stack corruption and terminates
> immediately with an illegal instruction. Oddly, the program never detects
> any stack corruption when run on the QEMU '040.

Nice! (the reproducer, not the bug ;-)

Does it also fail on a very old kernel image you still have lying around?
Just to rule out a recent kernel bug.

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: reliable reproducer, was Re: core dump analysis

[Finn Thain]

On Wed, 19 Apr 2023, Geert Uytterhoeven wrote:

> Does it also fail on a very old kernel image you still have lying 
> around? Just to rule out a recent kernel bug.
> 

These are two mainline builds I've tried (among others):

[    0.000000] Linux version 4.14.0-mac (fthain@nippy) (gcc version 6.4.0 
(btc)) #1 Mon May 23 15:03:18 AEST 2022

[    0.000000] Linux version 6.3.0-rc5-mac (fthain@nippy) 
(m68k-unknown-linux-musl-gcc (Gentoo 12.2.1_p20221224 p7) 12.2.1 20221224, 
GNU ld (Gentoo 2.39 p5) 2.39.0) #1 Thu Apr 20 10:10:37 AEST 2023

The results are the same. 

Re: reliable reproducer, was Re: core dump analysis

[Michael Schmitz]

Hi Finn,

On 19/04/23 22:50, Finn Thain wrote:
> On Tue, 18 Apr 2023, Michael Schmitz wrote:
>
>> ... I think what's stored there is the extra frame content for a format
>> b bus error frame. But that extra frame is incomplete at best (should be
>> 22 longwords, only a4 are seen). Probably overwritten by the stack frame
>> from __GI___wait4_time64.
>>
>> Let's parse what's left:
>> <=
>>>>> 0xefffefe4:     0xc0028780		<= internal registers (6x)
>>>>> 0xefffefe0:     0x3c344bfb		<=
>>>>> 0xefffefdc:     0x000af353		<=
>>>>> 0xefffefd8:     0x3c340170		<= internal reg; version no.
>>>>> 0xefffefd4:     0x00000000		<= data input buffer
>>>>> 0xefffefd0:     0xc00e417c		<= internal registers (2x)
>>>>> 0xefffefcc:     0xc00e417e		<= stage b address
>>>>> 0xefffefc8:     0xc00e4180		<= internal registers (4x)
>>>>> 0xefffefc4:     0x48e73c34		<=
>>>>> 0xefffefc0:     0x00000000		<= data output buffer
>>>>> 0xefffefbc:     0xefffeff8		<= internal registers (2x)
>>>>> 0xefffefb8:     0xefffeffc		<= data fault address
>>>>> 0xefffefb4:     0x4bfb0170		<= ins stage c, stage b
>>>>> 0xefffefb0:     0x0eee0709		<= internal register; ssw
>> The fault address is the location on the stack where a2 is saved. That
>> does match the data output buffer contents BTW. fc, fb, rc, rb bits
>> clear means the fault didn't occur in stage b or c instructions. ssw bit
>> 8 set indicates a data fault - the data cycle should be rerun on rte. rm
>> and rw bits clear tell us it's a write fault. If the moveml instruction
>> copies registers to the stack in descending order, the fault address
>> makes sense - the stack pointer just crossed a page boundary.
>>
> Inspired by your observation about the page fault and stack growth, I
> wrote a small test program (given below) that just pushes registers onto
> the stack recursively while forking processes and collecting the SIGCHLD
> signals.
>
> On a Motorola '030 the stack grows to about 7 MiB before it gets
> corrupted. The program detects the stack corruption and terminates
> immediately with an illegal instruction. Oddly, the program never detects
> any stack corruption when run on the QEMU '040.

That's great - finally irrefutable confirmation that we're onto a kernel 
bug.

>
> root@debian:~# ./movem
> Illegal instruction
> root@debian:~# ulimit -a
> real-time non-blocking time  (microseconds, -R) unlimited
> core file size              (blocks, -c) 0
> data seg size               (kbytes, -d) unlimited
> scheduling priority                 (-e) 0
> file size                   (blocks, -f) unlimited
> pending signals                     (-i) 242
> max locked memory           (kbytes, -l) 8192
> max memory size             (kbytes, -m) unlimited
> open files                          (-n) 1024
> pipe size                (512 bytes, -p) 8
> POSIX message queues         (bytes, -q) 819200
> real-time priority                  (-r) 0
> stack size                  (kbytes, -s) 8192
> cpu time                   (seconds, -t) unlimited
> max user processes                  (-u) 242
> virtual memory              (kbytes, -v) unlimited
> file locks                          (-x) unlimited
> root@debian:~# ulimit -s 7200
> root@debian:~# ./movem
> Illegal instruction
> root@debian:~# ulimit -s 7000
> root@debian:~# ./movem
> Segmentation fault
> root@debian:~# ulimit -s 16384
> root@debian:~# ./movem
> Illegal instruction
> root@debian:~#
>
> Looking at the core dump in gdb, the backtrace has 189869 frames. The dead
> stack frames confirm the recursion depth reached the limit I set at 200000
> before the stack began to reduce again. This was also confirmed by the
> lowest page fault address that was logged by the custom kernel.
>
> That means validation succeeded 200000 - 189869 == 10131 times before it
> encountered corruption (I should try to figure out whether this varies).
>
> The registers %a2, %a3 and %a4 below should contain 0x91929394, 0xa1a2a3a4
> and 0xb1b2b3b4 respectively. But they don't. Their values were restored
> from a corrupted stack by the returning rec() function call.
>
> (gdb) info reg
> d0             0x91929394          -1852664940
> d1             0xf3                243
> d2             0xd1d2d3d4          -774712364
> d3             0xe1e2e3e4          -505224220
> d4             0xf1f2f3f4          -235736076
> d5             0x80003f0c          -2147467508
> d6             0xd014c528          -803945176
> d7             0x0                 0
> a0             0xc0021708          0xc0021708
> a1             0xc0023e8c          0xc0023e8c <__stack_chk_guard>
> a2             0xf3                0xf3
> a3             0x1464000           0x1464000
> a4             0xef97bf44          0xef97bf44
> a5             0xc1c2c3c4          0xc1c2c3c4
> fp             0xef97b034          0xef97b034
> sp             0xef97b018          0xef97b018
> ps             0x8                 [ N ]
> pc             0x800005f6          0x800005f6 <rec+262>
> fpcontrol      0x0                 0
> fpstatus       0x0                 0
> fpiaddr        0x0                 0x0
>
> (gdb) x/z $sp - 36
> 0xef97aff4:     0xd1d2d3d4
> (gdb)
> 0xef97aff8:     0xe1e2e3e4
> (gdb)
> 0xef97affc:     0xf1f2f3f4
> (gdb)
> 0xef97b000:     0x000000f3
> (gdb)
> 0xef97b004:     0x01464000
> (gdb)
> 0xef97b008:     0xef97bf44
> (gdb)
> 0xef97b00c:     0xc1c2c3c4
> (gdb)
> 0xef97b010:     0xef97b034
> (gdb)
> 0xef97b014:     0x8000055c
>
> As with dash, the corruption lies the page boundary.

Hence implies a page fault handled at the page boundary.

Can you try and fault in as many of these stack pages as possible, ahead 
of filling the stack? (Depending on how much RAM you have ...). Maybe we 
would need to lock those pages into memory? Just to show that with no 
page faults (but still signals) there is no corruption?

> Any signal frames or exception frames have been completely overwritten
> because the recursion continued after the corruption took place. So
> there's not much to see in the core dump.

We'd need a way to stop recursion once the first corruption has taken 
place. If the 'safe' recursion depth of 10131 is constant, the dump 
taken at that point should look similar to what you saw in dash 
(assuming it is the page fault and subsequent signal return that causes 
the corruption).

I'll give your test case a spin on my Falcon. I should also have a range 
of kernels to test and answer Geert's question...

Cheers,

     Michael



>
> (gdb) disass rec
> Dump of assembler code for function rec:
>     0x800004f0 <+0>:     linkw %fp,#0
>     0x800004f4 <+4>:     moveml %d2-%d4/%a2-%a5,%sp@-
>     0x800004f8 <+8>:     moveal 0x80000672 <i0>,%a2
>     0x800004fe <+14>:    moveal 0x80000676 <i1>,%a3
>     0x80000504 <+20>:    moveal 0x8000067a <i2>,%a4
>     0x8000050a <+26>:    moveal 0x8000067e <i3>,%a5
>     0x80000510 <+32>:    movel 0x80000682 <i4>,%d2
>     0x80000516 <+38>:    movel 0x80000686 <i5>,%d3
>     0x8000051c <+44>:    movel 0x8000068a <i6>,%d4
>     0x80000522 <+50>:    movel 0x80004034 <depth>,%d0
>     0x80000528 <+56>:    andil #2047,%d0
>     0x8000052e <+62>:    bnes 0x80000542 <rec+82>
>     0x80000530 <+64>:    jsr 0x8000042c <fork@plt>
>     0x80000536 <+70>:    tstl %d0
>     0x80000538 <+72>:    bnes 0x80000542 <rec+82>
>     0x8000053a <+74>:    clrl %sp@-
>     0x8000053c <+76>:    jsr 0x80000404 <exit@plt>
>     0x80000542 <+82>:    movel 0x80004034 <depth>,%d0
>     0x80000548 <+88>:    subql #1,%d0
>     0x8000054a <+90>:    movel %d0,0x80004034 <depth>
>     0x80000550 <+96>:    movel 0x80004034 <depth>,%d0
>     0x80000556 <+102>:   beqs 0x8000055c <rec+108>
>     0x80000558 <+104>:   jsr %pc@(0x800004f0 <rec>)
>     0x8000055c <+108>:   movel %a2,0x8000403c <o0>
>     0x80000562 <+114>:   movel %a3,0x80004040 <o1>
>     0x80000568 <+120>:   movel %a4,0x80004044 <o2>
>     0x8000056e <+126>:   movel %a5,0x80004048 <o3>
>     0x80000574 <+132>:   movel %d2,0x8000404c <o4>
>     0x8000057a <+138>:   movel %d3,0x80004050 <o5>
>     0x80000580 <+144>:   movel %d4,0x80004054 <o6>
>     0x80000586 <+150>:   movel 0x8000403c <o0>,%d1
>     0x8000058c <+156>:   movel #-1852664940,%d0
>     0x80000592 <+162>:   cmpl %d1,%d0
>     0x80000594 <+164>:   bnes 0x800005f6 <rec+262>
>     0x80000596 <+166>:   movel 0x80004040 <o1>,%d1
>     0x8000059c <+172>:   movel #-1583176796,%d0
>     0x800005a2 <+178>:   cmpl %d1,%d0
>     0x800005a4 <+180>:   bnes 0x800005f6 <rec+262>
>     0x800005a6 <+182>:   movel 0x80004044 <o2>,%d1
>     0x800005ac <+188>:   movel #-1313688652,%d0
>     0x800005b2 <+194>:   cmpl %d1,%d0
>     0x800005b4 <+196>:   bnes 0x800005f6 <rec+262>
>     0x800005b6 <+198>:   movel 0x80004048 <o3>,%d1
>     0x800005bc <+204>:   movel #-1044200508,%d0
>     0x800005c2 <+210>:   cmpl %d1,%d0
>     0x800005c4 <+212>:   bnes 0x800005f6 <rec+262>
>     0x800005c6 <+214>:   movel 0x8000404c <o4>,%d1
>     0x800005cc <+220>:   movel #-774712364,%d0
>     0x800005d2 <+226>:   cmpl %d1,%d0
>     0x800005d4 <+228>:   bnes 0x800005f6 <rec+262>
>     0x800005d6 <+230>:   movel 0x80004050 <o5>,%d1
>     0x800005dc <+236>:   movel #-505224220,%d0
>     0x800005e2 <+242>:   cmpl %d1,%d0
>     0x800005e4 <+244>:   bnes 0x800005f6 <rec+262>
>     0x800005e6 <+246>:   movel 0x80004054 <o6>,%d1
>     0x800005ec <+252>:   movel #-235736076,%d0
>     0x800005f2 <+258>:   cmpl %d1,%d0
>     0x800005f4 <+260>:   beqs 0x800005f8 <rec+264>
> => 0x800005f6 <+262>:   illegal
>     0x800005f8 <+264>:   nop
>     0x800005fa <+266>:   moveml %fp@(-28),%d2-%d4/%a2-%a5
>     0x80000600 <+272>:   unlk %fp
>     0x80000602 <+274>:   rts
> End of assembler dump.
>
> ---
>
> #include <stdlib.h>
> #include <unistd.h>
> #include <stdio.h>
> #include <signal.h>
> #include <string.h>
>
> int depth = 200000;
>
> const unsigned long i0 = 0x91929394;
> const unsigned long i1 = 0xa1a2a3a4;
> const unsigned long i2 = 0xb1b2b3b4;
> const unsigned long i3 = 0xc1c2c3c4;
> const unsigned long i4 = 0xd1d2d3d4;
> const unsigned long i5 = 0xe1e2e3e4;
> const unsigned long i6 = 0xf1f2f3f4;
>
> unsigned long o0;
> unsigned long o1;
> unsigned long o2;
> unsigned long o3;
> unsigned long o4;
> unsigned long o5;
> unsigned long o6;
>
> static void rec(void)
> {
> 	// initialize registers
> 	asm(	"	move.l %0, %%a2\n"
> 		"	move.l %1, %%a3\n"
> 		"	move.l %2, %%a4\n"
> 		"	move.l %3, %%a5\n"
> 		"	move.l %4, %%d2\n"
> 		"	move.l %5, %%d3\n"
> 		"	move.l %6, %%d4\n"
> 		:
> 		: "m" (i0), "m" (i1), "m" (i2),
> 		  "m" (i3), "m" (i4), "m" (i5), "m" (i6)
> 		: "a2", "a3", "a4", "a5", "d2", "d3", "d4"
> 	);
>
> 	// maybe fork a short-lived process
> 	if ((depth & 0x7ff) == 0)
> 		if (fork() == 0)
> 			exit(0);
>
> 	if (--depth)
> 		rec();	// callee to save & restore registers
>
> 	// compare register contents
> 	asm(	"	move.l %%a2, %0\n"
> 		"	move.l %%a3, %1\n"
> 		"	move.l %%a4, %2\n"
> 		"	move.l %%a5, %3\n"
> 		"	move.l %%d2, %4\n"
> 		"	move.l %%d3, %5\n"
> 		"	move.l %%d4, %6\n"
> 		: "=m" (o0), "=m" (o1), "=m" (o2),
> 		  "=m" (o3), "=m" (o4), "=m" (o5), "=m" (o6)
> 		:
> 		:
> 	);
> 	if (o0 != i0 || o1 != i1 || o2 != i2 ||
> 	    o3 != i3 || o4 != i4 || o5 != i5 || o6 != i6)
> 		asm("illegal");
> }
>
> static void handler(int)
> {
> }
>
> int main(void)
> {
> 	struct sigaction act;
>
> 	memset(&act, 0, sizeof(act));
> 	act.sa_handler = handler;
> 	sigaction(SIGCHLD, &act, NULL);
>
> 	rec();
> }

Re: reliable reproducer, was Re: core dump analysis

[Finn Thain]

On Thu, 20 Apr 2023, Michael Schmitz wrote:

> Can you try and fault in as many of these stack pages as possible, ahead 
> of filling the stack? (Depending on how much RAM you have ...). Maybe we 
> would need to lock those pages into memory? Just to show that with no 
> page faults (but still signals) there is no corruption?
> 

OK.

> > Any signal frames or exception frames have been completely overwritten 
> > because the recursion continued after the corruption took place. So 
> > there's not much to see in the core dump.
> 
> We'd need a way to stop recursion once the first corruption has taken 
> place. If the 'safe' recursion depth of 10131 is constant, the dump 
> taken at that point should look similar to what you saw in dash 
> (assuming it is the page fault and subsequent signal return that causes 
> the corruption).
> 

It turns out that the recursion depth can be set a lot lower than the 
200000 that I chose in that test program. (I used that value as it kept 
the stack size just below the default 8192 kB limit.)

At depth = 2500, a failure is around 95% certain. At depth = 2048 I can 
still get an intermittent failure. This only required 21 stack pagefaults 
and one fork.

I suspect that the location of the corruption is probably somewhat random, 
and the larger the stack happens to be when the signal comes in, the 
better the odds of detection.

Re: reliable reproducer, was Re: core dump analysis

[Michael Schmitz]

Hi Finn,

reproduced on my Falcon (with minor mods to the C source - my version of 
gcc didn't like asm with no clobbers, so I added "memory" as clobber in 
the second asm block). In this case it's a4 that is corrupted, but that 
varies.

depth of 4096 gets me two core dumps on 20 attempts so this isn't quite 
as fast on my Falcon. With 8192, it's nine.

Example:

Core was generated by `./moveml'.
Program terminated with signal 4, Illegal instruction.
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld.so.1...done.
Loaded symbols for /lib/ld.so.1
#0  0x8000060e in rec ()
(gdb) info reg
d0             0x8000057c	-2147482244
d1             0xc0017000	-1073647616
d2             0xd1d2d3d4	-774712364
d3             0xe1e2e3e4	-505224220
d4             0xf1f2f3f4	-235736076
d5             0x80096168	-2146868888
d6             0x80093108	-2146881272
d7             0x0	0
a0             0x0	0x0
a1             0xefdadbdc	0xefdadbdc
a2             0x91929394	0x91929394
a3             0xa1a2a3a4	0xa1a2a3a4
a4             0x8000057c	0x8000057c
a5             0xc1c2c3c4	0xc1c2c3c4
fp             0xef87402c	0xef87402c
sp             0xef874010	0xef874010
ps             0x209	521
pc             0x8000060e	0x8000060e <rec+242>
fpcontrol      0x0	0
fpstatus       0x0	0
fpiaddr        0x0	0
(gdb)


Am 20.04.2023 um 14:57 schrieb Finn Thain:
> On Thu, 20 Apr 2023, Michael Schmitz wrote:
>
>> Can you try and fault in as many of these stack pages as possible, ahead
>> of filling the stack? (Depending on how much RAM you have ...). Maybe we
>> would need to lock those pages into memory? Just to show that with no
>> page faults (but still signals) there is no corruption?
>>
>
> OK.
>
>>> Any signal frames or exception frames have been completely overwritten
>>> because the recursion continued after the corruption took place. So
>>> there's not much to see in the core dump.
>>
>> We'd need a way to stop recursion once the first corruption has taken
>> place. If the 'safe' recursion depth of 10131 is constant, the dump
>> taken at that point should look similar to what you saw in dash
>> (assuming it is the page fault and subsequent signal return that causes
>> the corruption).
>>
>
> It turns out that the recursion depth can be set a lot lower than the
> 200000 that I chose in that test program. (I used that value as it kept
> the stack size just below the default 8192 kB limit.)

And it does keep the core a lot smaller. Still not hard to work with on 
my 14 MB RAM Falcon...

>
> At depth = 2500, a failure is around 95% certain. At depth = 2048 I can
> still get an intermittent failure. This only required 21 stack pagefaults
> and one fork.
>
> I suspect that the location of the corruption is probably somewhat random,
> and the larger the stack happens to be when the signal comes in, the
> better the odds of detection.

Yep, but there must me some more to that. Timing of page faults due to 
swap bandwidth, perhaps?

Cheers,

	Michael

>

Re: reliable reproducer, was Re: core dump analysis

[Finn Thain]

On Thu, 20 Apr 2023, Michael Schmitz wrote:

> >
> > As with dash, the corruption lies the page boundary.
> 
> Hence implies a page fault handled at the page boundary.
> 
> Can you try and fault in as many of these stack pages as possible, ahead 
> of filling the stack? (Depending on how much RAM you have ...). Maybe we 
> would need to lock those pages into memory? Just to show that with no 
> page faults (but still signals) there is no corruption?
> 

I modified the test program to execute rec() to full depth with no 
forking, then do it again with forking.

root@(none):/root# while ./stack-test 5000 ; do : ; done
starting recursion
done.
starting recursion with fork
done.
starting recursion
done.
starting recursion with fork
Illegal instruction
root@(none):/root# 

I can't get this to crash during the first descent. The second descent 
always crashes, given sufficient depth:

root@(none):/root# while ./stack-test 50000 ; do : ; done
starting recursion
done.
starting recursion with fork
Illegal instruction

So all the stack pages would have been faulted in well before the failure 
shows up. It appears to be the signal that's the problem and not the page 
fault. That's not surprising considering the PC in the signal frame in the 
dash crash was a MOVEM saving registers onto the stack.

It's worth noting that the test program never crashes with a corrupted 
return address. Random corruption would have clobbered that address about 
10% of the time, since the entire rec() stack frame is 9 long words. So it 
must be that a MOVEM went awry when a signal got delivered.

Re: reliable reproducer, was Re: core dump analysis

[Finn Thain]

On Thu, 20 Apr 2023, I wrote:

> 
> So it must be that a MOVEM went awry when a signal got delivered.

Or signal delivery went awry after a MOVEM got resumed?

Re: reliable reproducer, was Re: core dump analysis

[Michael Schmitz]

Hi Finn,

Am 20.04.2023 um 17:17 schrieb Finn Thain:
> On Thu, 20 Apr 2023, Michael Schmitz wrote:
>
>>>
>>> As with dash, the corruption lies the page boundary.
>>
>> Hence implies a page fault handled at the page boundary.
>>
>> Can you try and fault in as many of these stack pages as possible, ahead
>> of filling the stack? (Depending on how much RAM you have ...). Maybe we
>> would need to lock those pages into memory? Just to show that with no
>> page faults (but still signals) there is no corruption?
>>
>
> I modified the test program to execute rec() to full depth with no
> forking, then do it again with forking.
>
> root@(none):/root# while ./stack-test 5000 ; do : ; done
> starting recursion
> done.
> starting recursion with fork
> done.
> starting recursion
> done.
> starting recursion with fork
> Illegal instruction
> root@(none):/root#
>
> I can't get this to crash during the first descent. The second descent
> always crashes, given sufficient depth:
>
> root@(none):/root# while ./stack-test 50000 ; do : ; done
> starting recursion
> done.
> starting recursion with fork
> Illegal instruction
>
> So all the stack pages would have been faulted in well before the failure
> shows up. It appears to be the signal that's the problem and not the page
> fault. That's not surprising considering the PC in the signal frame in the
> dash crash was a MOVEM saving registers onto the stack.

Well. without locking the faulted in pages in memory we can't be sure 
they were not swapped back out. Unless I misunderstand what's involved 
in that ...

In my tests, increasing the depth does not cause a monotonous increase 
in fault probability. 16k depth only has four crashes, 8k had nine. I'll 
stick with 200000 for now.

Best try and narrow down how long this bug has been present - my kernel 
builds on the current system go back a little over ten years.

Cheers,

	Michael


>
> It's worth noting that the test program never crashes with a corrupted
> return address. Random corruption would have clobbered that address about
> 10% of the time, since the entire rec() stack frame is 9 long words. So it
> must be that a MOVEM went awry when a signal got delivered.
>

Re: reliable reproducer, was Re: core dump analysis

[Finn Thain]

On Thu, 20 Apr 2023, Michael Schmitz wrote:

> Am 20.04.2023 um 17:17 schrieb Finn Thain:
> > On Thu, 20 Apr 2023, Michael Schmitz wrote:
> >
> >>>
> >>> As with dash, the corruption lies the page boundary.
> >>
> >> Hence implies a page fault handled at the page boundary.
> >>
> >> Can you try and fault in as many of these stack pages as possible, ahead
> >> of filling the stack? (Depending on how much RAM you have ...). Maybe we
> >> would need to lock those pages into memory? Just to show that with no
> >> page faults (but still signals) there is no corruption?
> >>
> >
> > I modified the test program to execute rec() to full depth with no
> > forking, then do it again with forking.
> >
> > root@(none):/root# while ./stack-test 5000 ; do : ; done
> > starting recursion
> > done.
> > starting recursion with fork
> > done.
> > starting recursion
> > done.
> > starting recursion with fork
> > Illegal instruction
> > root@(none):/root#
> >
> > I can't get this to crash during the first descent. The second descent
> > always crashes, given sufficient depth:
> >
> > root@(none):/root# while ./stack-test 50000 ; do : ; done
> > starting recursion
> > done.
> > starting recursion with fork
> > Illegal instruction
> >
> > So all the stack pages would have been faulted in well before the 
> > failure shows up. It appears to be the signal that's the problem and 
> > not the page fault. That's not surprising considering the PC in the 
> > signal frame in the dash crash was a MOVEM saving registers onto the 
> > stack.
> 
> Well. without locking the faulted in pages in memory we can't be sure 
> they were not swapped back out. Unless I misunderstand what's involved 
> in that ...
> 

There was no swap enabled.

50000 frames * 36 bytes per frame == 1.8 MB

Re: reliable reproducer, was Re: core dump analysis

[Michael Schmitz]

Hi Finn,

Am 20.04.2023 um 19:47 schrieb Finn Thain:
>>> So all the stack pages would have been faulted in well before the
>>> failure shows up. It appears to be the signal that's the problem and
>>> not the page fault. That's not surprising considering the PC in the
>>> signal frame in the dash crash was a MOVEM saving registers onto the
>>> stack.
>>
>> Well. without locking the faulted in pages in memory we can't be sure
>> they were not swapped back out. Unless I misunderstand what's involved
>> in that ...
>>
>
> There was no swap enabled.
>
> 50000 frames * 36 bytes per frame == 1.8 MB

OK - swap is enabled in my case. That may explain the different fault 
rates.

But in any case, it looks like we can eliminate the bus error code. Same 
fault on both 030 and 040 with very different bus error handlers is 
highly unlikely.

Cheers.

	Michael

Re: reliable reproducer, was Re: core dump analysis

[Finn Thain]

On Thu, 20 Apr 2023, Michael Schmitz wrote:

> Am 20.04.2023 um 19:47 schrieb Finn Thain:
> >>> So all the stack pages would have been faulted in well before the 
> >>> failure shows up. It appears to be the signal that's the problem and 
> >>> not the page fault. That's not surprising considering the PC in the 
> >>> signal frame in the dash crash was a MOVEM saving registers onto the 
> >>> stack.
> >>
> >> Well. without locking the faulted in pages in memory we can't be sure 
> >> they were not swapped back out. Unless I misunderstand what's 
> >> involved in that ...
> >>
> >
> > There was no swap enabled.
> >
> > 50000 frames * 36 bytes per frame == 1.8 MB
> 
> OK - swap is enabled in my case. That may explain the different fault 
> rates.
> 
> But in any case, it looks like we can eliminate the bus error code. Same 
> fault on both 030 and 040 with very different bus error handlers is 
> highly unlikely.
> 

There's no failure on '040. QEMU and Motorola '040 gave the same result.

Re: reliable reproducer, was Re: core dump analysis

[Michael Schmitz]

Hi Finn,

On 20/04/23 20:55, Finn Thain wrote:
>
>> But in any case, it looks like we can eliminate the bus error code. Same
>> fault on both 030 and 040 with very different bus error handlers is
>> highly unlikely.
>>
> There's no failure on '040. QEMU and Motorola '040 gave the same result.

Sorry, my fault - I interpreted your mail as saying 030 and 040 gave the 
same result.

Back to the drawing board ... I've got kernel images back to 2.4.30 and 
2.6.37 to try and test. I'm also trying with rt signals and alternate 
signal stack (rt signals show the same behaviour).

Cheers,

     Michael

Re: reliable reproducer, was Re: core dump analysis

[Michael Schmitz]

Hi Finn,

one more datapoint: using sigaltstack() to set up a separate signal 
stack in main() (code lifted straight from the sigaltstack man page) 
does avoid the stack corruption.

That might be useful as a workaround for dash, while we're analyzing 
this bug.

I wonder whether recursively growing the stack while taking signals on 
the same stack is meant to be safe. Need to look at usp inside the 
signal frame (i.e. at the time the program was preempted for signal 
delivery) and usp at the lowest recursion depth at the time of the 
signal handler running ...

Cheers,

     Michael


On 21/04/23 09:58, Michael Schmitz wrote:
> Hi Finn,
>
> On 20/04/23 20:55, Finn Thain wrote:
>>
>>> But in any case, it looks like we can eliminate the bus error code. 
>>> Same
>>> fault on both 030 and 040 with very different bus error handlers is
>>> highly unlikely.
>>>
>> There's no failure on '040. QEMU and Motorola '040 gave the same result.
>
> Sorry, my fault - I interpreted your mail as saying 030 and 040 gave 
> the same result.
>
> Back to the drawing board ... I've got kernel images back to 2.4.30 
> and 2.6.37 to try and test. I'm also trying with rt signals and 
> alternate signal stack (rt signals show the same behaviour).
>
> Cheers,
>
>     Michael
>
>
>
>

Re: reliable reproducer, was Re: core dump analysis

[Michael Schmitz]

Hi Finn,

Am 21.04.2023 um 09:58 schrieb Michael Schmitz:
> Hi Finn,
>
> On 20/04/23 20:55, Finn Thain wrote:
>>
>>> But in any case, it looks like we can eliminate the bus error code. Same
>>> fault on both 030 and 040 with very different bus error handlers is
>>> highly unlikely.
>>>
>> There's no failure on '040. QEMU and Motorola '040 gave the same result.
>
> Sorry, my fault - I interpreted your mail as saying 030 and 040 gave the
> same result.
>
> Back to the drawing board ... I've got kernel images back to 2.4.30 and
> 2.6.37 to try and test. I'm also trying with rt signals and alternate
> signal stack (rt signals show the same behaviour).

2.4.30 behaves the same as recent kernels - stack corruption unless 
signal handlers use the alternate signal stack.

We already knew the 030 bus error code hasn't seen changes since 2.6, so 
this isn't much of a surprise.

I'll add some logging to the signal frame setup to print the user stack 
pointer, and perhaps do the same in the bus error handler (when hitting 
a moveml instruction).

How often did a page fault happen when executing moveml, in other programs?

Cheers,

	Michael

Re: reliable reproducer, was Re: core dump analysis

[Finn Thain]

On Fri, 21 Apr 2023, Michael Schmitz wrote:

> 
> How often did a page fault happen when executing moveml, in other 
> programs?
> 

The printk() I placed in bus_error030() was conditional on the short word 
at the instruction pointer. It didn't consider all forms of movem, just 
0x48e7 which is the start of "moveml X,%sp@-". This matched page faults in 
many of the programs that executed while booting to single user mode. I 
suppose most of them don't use signal handlers in the same way dash does 
otherwise they would probably be unreliable too.

Re: reliable reproducer, was Re: core dump analysis

[Michael Schmitz]

Hi Finn,

Am 21.04.2023 um 20:30 schrieb Finn Thain:
> On Fri, 21 Apr 2023, Michael Schmitz wrote:
>
>>
>> How often did a page fault happen when executing moveml, in other
>> programs?
>>
>
> The printk() I placed in bus_error030() was conditional on the short word
> at the instruction pointer. It didn't consider all forms of movem, just
> 0x48e7 which is the start of "moveml X,%sp@-". This matched page faults in
> many of the programs that executed while booting to single user mode. I
> suppose most of them don't use signal handlers in the same way dash does
> otherwise they would probably be unreliable too.

OK; so too much noise unless filtered on the command name...

I'll try first to get register state at the time of signal delivery from 
the sa_sigaction handler's ucontext parameter to see where the signal 
stack falls in relation to the call frames from your rec() function on 
the stack (and what the register contents were). Hope that won't be too 
noisy ... Then see how that changes with bus error handling forcing 
signal delivery.

Shame we don't have similar code to find the MMU descriptors on 040.

Cheers,

	Michael

Re: reliable reproducer, was Re: core dump analysis

[Michael Schmitz]

[-- Attachment #1: Type: text/plain, Size: 4804 bytes --]

Hi Finn,

Am 21.04.2023 um 21:18 schrieb Michael Schmitz:
> Hi Finn,
>
> Am 21.04.2023 um 20:30 schrieb Finn Thain:
>> On Fri, 21 Apr 2023, Michael Schmitz wrote:
>>
>>>
>>> How often did a page fault happen when executing moveml, in other
>>> programs?
>>>
>>
>> The printk() I placed in bus_error030() was conditional on the short word
>> at the instruction pointer. It didn't consider all forms of movem, just
>> 0x48e7 which is the start of "moveml X,%sp@-". This matched page
>> faults in
>> many of the programs that executed while booting to single user mode. I
>> suppose most of them don't use signal handlers in the same way dash does
>> otherwise they would probably be unreliable too.
>
> OK; so too much noise unless filtered on the command name...
>
> I'll try first to get register state at the time of signal delivery from
> the sa_sigaction handler's ucontext parameter to see where the signal
> stack falls in relation to the call frames from your rec() function on
> the stack (and what the register contents were). Hope that won't be too

Took a little while to figure out that the ucontext format changed in 
the decade or two since my userland's libc headers were generated. With 
the correct format, the information stored on th signal frame made a lot 
more sense.

Log of your test program (attached), instrumented to keep track of user 
stack pointer in the parent process, user stack pointer in the signal 
handler, and stack pointer, pc and exceptiopn frame format from the 
signal stack (only the last few signals shown):

parent usp  : 0xef97beb8
handler tos : 0xef97bdc4
handler usp : 0xef97bbe0
signal usp  : 0xef97bea8
signal pc   : 0xc009f37a
signal fmtv : 0x800006ca

parent usp  : 0xef969eb8
handler tos : 0xef969dc4
handler usp : 0xef969be0
signal usp  : 0xef969ea8
signal pc   : 0xc009f37a
signal fmtv : 0x800006ca

parent usp  : 0xef9530d0
handler tos : 0xef952fec
handler usp : 0xef952e08
signal usp  : 0xef9530d0
signal pc   : 0x800006dc
signal fmtv : 0x91929394

parent usp  : 0xef945eb8
handler tos : 0xef945dc4
handler usp : 0xef945be0
signal usp  : 0xef945ea8
signal pc   : 0xc009f37a
signal fmtv : 0x800006ca

parent usp  : 0xef933eb8
handler tos : 0xef933dc4
handler usp : 0xef933be0
signal usp  : 0xef933ea8
signal pc   : 0xc009f37a
signal fmtv : 0x800006ca

parent usp  : 0xef921edc
handler tos : 0xef997984
handler stack overwrote usp!
handler usp : 0xef9977a0
signal usp  : 0xef997a64
signal pc   : 0x80000768
signal fmtv : 0xa1a2a3a4

Illegal instruction (core dumped)

usp from the signal stack is below that of the parent process (before 
calling fork()).

usp from the signal handler is below both of those. So far, so good.

The top of the signal frame, however, is getting quite close to these 
stack pointers. In the last log, it has grown above the user stack pointer.

Two things to note:

- pc in the signal frame (from struct uc_mcontext) is either the return 
pc from the stack stuffing function, or something else I cannot work 
out. That part of ucontext appears valid.

- what ought to be the frame format and vector offset does in fact hold 
varying longwords from the user stack. This information is not from 
struct uc_mcontext, but from extra information copied after struct 
ucontext ends. That wouldn't be there if at time of signal delivery, 
nothing had yet written to the area where the signal frame is stored.

It appears that neither format/vector nor other exception frame 
information is stored there by the kernel signal code, so the registers 
that rec() had saved to the stack remain there. At time of signal 
delivery, the user stack appears to have grown into the area used for 
the signal stack, in spite of the user stack pointer saved after setting 
up the registers saying otherwise. I can't figure that out.

At any rate, the constant part of the extra signal stack information 
does not take into account that we may end up storing additional 
information beyond that, in the case that the exception frame that got 
us into kernel mode is larger than four words. Maybe that is why the 
last stack frames logged do have the end of the signal stack go beyond 
the user stack pointer.

The FPU state and exception frame part stored above the ucontext struct 
may well have a different size in your libc headers, or I may have made 
a mistake calculating the size of the extra information stored on the 
stack. You may want to undo my hack to use a local copy of the struct 
declarations and compile with the correct headers, maybe that does 
change the overall picture.

> noisy ... Then see how that changes with bus error handling forcing
> signal delivery.

Haven't got to that bit yet ...

Cheers,

	Michael


> Shame we don't have similar code to find the MMU descriptors on 040.
>
> Cheers,
>
>     Michael
>
>

[-- Attachment #2: movemlrtas.c --]
[-- Type: text/x-csrc, Size: 4554 bytes --]

#include <stdlib.h>
#include <unistd.h>
#include <stdio.h>
#include <signal.h>
#include <string.h>
#include <ucontext.h>

/* copies of ucontext struct matching 6.x kernels */
/* no extra stack contents */
typedef struct k_ucontext {
        unsigned long     uc_flags;
	struct ucontext  *uc_link;
	stack_t           uc_stack;
	mcontext_t	  uc_mcontext;
	sigset_t          uc_sigmask;   /* mask last for extensibility */
} k_ucontext_t;
/* extra stack contents (fpu state, exception frame part (> 4 words) */                                        
typedef struct k_ucontextf {
        unsigned long     uc_flags;
	struct ucontext  *uc_link;
	stack_t           uc_stack;
	mcontext_t	  uc_mcontext;
	sigset_t          uc_sigmask;   /* mask last for extensibility */
	long int uc_filler[174];
} k_ucontextf_t;
                                        
int depth = 200000 /* 32768 ; 200000 */;

const unsigned long i0 = 0x91929394;
const unsigned long i1 = 0xa1a2a3a4;
const unsigned long i2 = 0xb1b2b3b4;
const unsigned long i3 = 0xc1c2c3c4;
const unsigned long i4 = 0xd1d2d3d4;
const unsigned long i5 = 0xe1e2e3e4;
const unsigned long i6 = 0xf1f2f3f4;

unsigned long o0;
unsigned long o1;
unsigned long o2;
unsigned long o3;
unsigned long o4;
unsigned long o5;
unsigned long o6;

unsigned long parent_usp, child_usp;

static void rec(void)
{
	// initialize registers
	asm(	"	move.l %0, %%a2\n" 
		"	move.l %1, %%a3\n"
		"	move.l %2, %%a4\n"
		"	move.l %3, %%a5\n"
		"	move.l %4, %%d2\n"
		"	move.l %5, %%d3\n"
		"	move.l %6, %%d4\n"
		:
		: "m" (i0), "m" (i1), "m" (i2),
		  "m" (i3), "m" (i4), "m" (i5), "m" (i6)
		: "a2", "a3", "a4", "a5", "d2", "d3", "d4"
	);

	// note current usp
	asm(	"	move.l %%sp, %0\n" 
		: "=m" (parent_usp)
		:
		: "memory" 
	);

	// maybe fork a short-lived process
	if ((depth & 0x7ff) == 0)
		if (fork() == 0) {
			// note current usp
			asm(	"	move.l %%sp, %0\n" 
				: "=m" (child_usp)
				:
				: "memory" 
			);
			exit(0);
		}

	if (--depth)
		rec();	// callee to save & restore registers

	// compare register contents
	asm(	"	move.l %%a2, %0\n" 
		"	move.l %%a3, %1\n"
		"	move.l %%a4, %2\n"
		"	move.l %%a5, %3\n"
		"	move.l %%d2, %4\n"
		"	move.l %%d3, %5\n"
		"	move.l %%d4, %6\n"
		: "=m" (o0), "=m" (o1), "=m" (o2),
		  "=m" (o3), "=m" (o4), "=m" (o5), "=m" (o6)
		:
		: "memory" /* "a2", "a3", "a4", "a5", "d2", "d3", "d4" */
	);
	if (o0 != i0 || o1 != i1 || o2 != i2 ||
	    o3 != i3 || o4 != i4 || o5 != i5 || o6 != i6)
		asm("illegal");
}

static void handler(int nevermind)
{
}
            
static void rthandler(int signo, siginfo_t *info, void *ucontext)
{
	/* use struct ucontext from libc headers */
	// ucontext_t *ctx = (ucontext_t *) ucontext;
	/* use struct ucontext from kernel source */
	k_ucontext_t *ctx = (k_ucontext_t *) ucontext;
	k_ucontextf_t *ctxf = (k_ucontextf_t *) ucontext;

	if (signo == SIGCHLD) {
		unsigned long usp, pc;
		int i;

		asm(	"	move.l %%sp, %0\n" 
			: "=m" (usp)
			:
			: "memory" 
		);
		fprintf(stderr, "parent usp  : 0x%lx\n", parent_usp);
		// fprintf(stderr, "child usp   : 0x%lx\n", child_usp);
		/* top of signal stack - using ucontext w/o uc_filler
		 * Adding filler (which will take additional exception
		 * frame contents for longer tha four word frames),
		 * top of stack may end up above parent usp! */
		fprintf(stderr, "handler tos : 0x%lx\n", usp + sizeof(struct siginfo) + sizeof(struct k_ucontext) + 24);
                if (usp + sizeof(struct siginfo) + sizeof(struct k_ucontext) + 24 > parent_usp)
                        fprintf(stderr, "handler stack overwrote usp!\n");
		fprintf(stderr, "handler usp : 0x%lx\n", usp);
		fprintf(stderr, "signal usp  : 0x%lx\n", ctxf->uc_mcontext.gregs[15]);
		fprintf(stderr, "signal pc   : 0x%lx\n", ctxf->uc_mcontext.gregs[16]);
		fprintf(stderr, "signal fmtv : 0x%lx\n", ctxf->uc_filler[54]);
		fprintf(stderr, "\n");
	}
}

int main(void)

{
	struct sigaction act;
	stack_t ss;

	ss.ss_sp = malloc(SIGSTKSZ);
	if (ss.ss_sp == NULL) {
		perror("malloc");
		exit(EXIT_FAILURE);
	}
	
	ss.ss_size = SIGSTKSZ;
	ss.ss_flags = 0;
	if (sigaltstack(&ss, NULL) == -1) {
		perror("sigaltstack");
		exit(EXIT_FAILURE);
	}
	
#ifdef USE_ALTSTACK
	fprintf(stderr, "alt sig stack: 0x%lx - 0x%lx\n", ss.ss_sp, ss.ss_sp + ss.ss_size);

	act.sa_flags = SA_ONSTACK;
	act.sa_flags |= SA_SIGINFO;
	act.sa_sigaction = rthandler;
#else
	act.sa_flags = SA_SIGINFO;
	act.sa_handler = handler;
#endif
	sigemptyset(&act.sa_mask);
	if (sigaction(SIGCHLD, &act, NULL) == -1) {
		perror("sigaction");
		exit(EXIT_FAILURE);
	}

	rec();
}

Re: reliable reproducer, was Re: core dump analysis

[Andreas Schwab]

On Apr 22 2023, Michael Schmitz wrote:

> Took a little while to figure out that the ucontext format changed in the
> decade or two since my userland's libc headers were generated.

In which way did it change?

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: reliable reproducer, was Re: core dump analysis

[Michael Schmitz]

Hi Andreas,

Am 22.04.2023 um 20:07 schrieb Andreas Schwab:
> On Apr 22 2023, Michael Schmitz wrote:
>
>> Took a little while to figure out that the ucontext format changed in the
>> decade or two since my userland's libc headers were generated.
>
> In which way did it change?

This is the definition from the kernel's 
include/uapi/asm-generic/ucontext.h:

struct ucontext {
         unsigned long     uc_flags;
         struct ucontext  *uc_link;
         stack_t           uc_stack;
         struct sigcontext uc_mcontext;
         sigset_t          uc_sigmask;   /* mask last for extensibility */
};

And this is /usr/include/sys/ucontext.h:

/* Userlevel context.  */
typedef struct ucontext
{
   unsigned long int uc_flags;
   struct ucontext *uc_link;
   __sigset_t uc_sigmask;
   stack_t uc_stack;
   mcontext_t uc_mcontext;
   long int uc_filler[174];
} ucontext_t;

uc_sigmask appears before uc_stack and uc_mcontext.

I'm assuming libc just passes on what the kernel set, without reordering?

Cheers,

	Michael

Re: reliable reproducer, was Re: core dump analysis

[Andreas Schwab]

On Apr 22 2023, Michael Schmitz wrote:

> This is the definition from the kernel's
> include/uapi/asm-generic/ucontext.h:

That's not actually used by m68k, it uses
arch/m68k/include/asm/ucontext.h, which confusingly isn't an uapi
header.

> And this is /usr/include/sys/ucontext.h:
>
> /* Userlevel context.  */
> typedef struct ucontext
> {
>   unsigned long int uc_flags;
>   struct ucontext *uc_link;
>   __sigset_t uc_sigmask;
>   stack_t uc_stack;
>   mcontext_t uc_mcontext;
>   long int uc_filler[174];
> } ucontext_t;
>
> uc_sigmask appears before uc_stack and uc_mcontext.

Yes, that got fixed as part of commit 9c986f878a back in 2006.
 
> I'm assuming libc just passes on what the kernel set, without reordering?

Trying to rewrite the signal context would be prohibitive, yes.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: reliable reproducer, was Re: core dump analysis

[Michael Schmitz]

Hi Andreas,

Am 22.04.2023 um 22:12 schrieb Andreas Schwab:
> On Apr 22 2023, Michael Schmitz wrote:
>
>> This is the definition from the kernel's
>> include/uapi/asm-generic/ucontext.h:
>
> That's not actually used by m68k, it uses
> arch/m68k/include/asm/ucontext.h, which confusingly isn't an uapi
> header.

Argh - not sure how I missed that one.

>> And this is /usr/include/sys/ucontext.h:
>>
>> /* Userlevel context.  */
>> typedef struct ucontext
>> {
>>   unsigned long int uc_flags;
>>   struct ucontext *uc_link;
>>   __sigset_t uc_sigmask;
>>   stack_t uc_stack;
>>   mcontext_t uc_mcontext;
>>   long int uc_filler[174];
>> } ucontext_t;
>>
>> uc_sigmask appears before uc_stack and uc_mcontext.
>
> Yes, that got fixed as part of commit 9c986f878a back in 2006.

With the header corrected as per arch/m68k/include/asm/ucontext.h, I get 
the correct information in uc_filler[54] aka fmtvec now. Good. Every 
sigreturn possibly overwrites the user stack now (if the filler is used 
in full). Not so good.

>> I'm assuming libc just passes on what the kernel set, without reordering?
>
> Trying to rewrite the signal context would be prohibitive, yes.

Now I wonder who adds sigmask ... and whether that's also ending up on 
the user stack.

Cheers,

	Michael


>

Re: reliable reproducer, was Re: core dump analysis

[Andreas Schwab]

On Apr 23 2023, Michael Schmitz wrote:

> Now I wonder who adds sigmask ... and whether that's also ending up on the
> user stack.

The kernel only writes the first 64 bits of the signal mask, as it does
for all signal mask related syscalls.  The kernel version of the context
ends after that; since the user-space version is larger it actually
extends into the next stack frame.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: reliable reproducer, was Re: core dump analysis

[Michael Schmitz]

Hi Andreas,

Am 23.04.2023 um 06:38 schrieb Andreas Schwab:
> On Apr 23 2023, Michael Schmitz wrote:
>
>> Now I wonder who adds sigmask ... and whether that's also ending up on the
>> user stack.
>
> The kernel only writes the first 64 bits of the signal mask, as it does
> for all signal mask related syscalls.  The kernel version of the context
> ends after that; since the user-space version is larger it actually
> extends into the next stack frame.

Yep, seen that now, but only with rt signals this gets copied at the 
very end of the signal context.

I'll have to double check my calculations to see where this will 
actually end up on the stack in either case.

I'll see whether the signal context is available on the stack even if 
the handler isn't passed that parameter.

Cheers,

	Michael

Re: reliable reproducer, was Re: core dump analysis

[Andreas Schwab]

On Apr 23 2023, Michael Schmitz wrote:

> I'll see whether the signal context is available on the stack even if the
> handler isn't passed that parameter.

The signal context is always on the stack, and used by the
(rt_)sigreturn syscall.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: reliable reproducer, was Re: core dump analysis

[Michael Schmitz]

Hi Andreas,

Am 23.04.2023 um 08:46 schrieb Andreas Schwab:
> On Apr 23 2023, Michael Schmitz wrote:
>
>> I'll see whether the signal context is available on the stack even if the
>> handler isn't passed that parameter.
>
> The signal context is always on the stack, and used by the
> (rt_)sigreturn syscall.

True, but at the time the signal handler (sa_handler type) runs, all I 
have is the user stack pointer upon entry to the handler. I'll have to 
calculate back from that address, if that is possible.

Cheers,

	Michael

Re: reliable reproducer, was Re: core dump analysis

[Andreas Schwab]

On Apr 23 2023, Michael Schmitz wrote:

> Hi Andreas,
>
> Am 23.04.2023 um 08:46 schrieb Andreas Schwab:
>> On Apr 23 2023, Michael Schmitz wrote:
>>
>>> I'll see whether the signal context is available on the stack even if the
>>> handler isn't passed that parameter.
>>
>> The signal context is always on the stack, and used by the
>> (rt_)sigreturn syscall.
>
> True, but at the time the signal handler (sa_handler type) runs, all I
> have is the user stack pointer upon entry to the handler. I'll have to
> calculate back from that address, if that is possible.

The signal handler always receives all three arguments.
-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: reliable reproducer, was Re: core dump analysis

[Michael Schmitz]

Hi Finn, Andreas,

Am 23.04.2023 um 13:41 schrieb Michael Schmitz:
> Hi Andreas,
>
> Am 23.04.2023 um 08:46 schrieb Andreas Schwab:
>> On Apr 23 2023, Michael Schmitz wrote:
>>
>>> I'll see whether the signal context is available on the stack even if
>>> the
>>> handler isn't passed that parameter.
>>
>> The signal context is always on the stack, and used by the
>> (rt_)sigreturn syscall.
>
> True, but at the time the signal handler (sa_handler type) runs, all I
> have is the user stack pointer upon entry to the handler. I'll have to
> calculate back from that address, if that is possible.

Wasn't too hard actually. The signo parameter passed to the handler 
turns out to be passed by reference, and signo is located 4 bytes into 
the kernel sigframe. The address of the sigcontext struct follows from 
that.

Result for the original test code that Finn posted (using sa_handler):

parent usp  : 0xefa2f010
handler tos : 0xefa2efa0
frame end   : 0xefa2efb4
frame start : 0xefa2ee78
handler usp : 0xefa2ee60
signal usp  : 0xefa2f004
signal pc   : 0x80000588
signal fmtv : 0xb008
parent usp  : 0xef957ed4
handler tos : 0xef957eb0
frame end   : 0xef957ec4
frame start : 0xef957d88
handler usp : 0xef957d70
signal usp  : 0xef957ec4
signal pc   : 0xc009f37a
signal fmtv : 0x80
parent usp  : 0xef957ed4
handler tos : 0xef957eb0
frame end   : 0xef957ec4
frame start : 0xef957d88
handler usp : 0xef957d70
signal usp  : 0xef957ec4
signal pc   : 0xc009f37a
signal fmtv : 0x80
parent usp  : 0xef945ed4
handler tos : 0xef945eb0
frame end   : 0xef945ec4
frame start : 0xef945d88
handler usp : 0xef945d70
signal usp  : 0xef945ec4
signal pc   : 0xc009f37a
signal fmtv : 0x80
parent usp  : 0xef921ef8
handler tos : 0xef9b0f88
handler stack overwrote usp!
frame end   : 0xef9b0f9c
frame start : 0xef9b0e60
handler usp : 0xef9b0e48
signal usp  : 0xef9b0fec
signal pc   : 0x8000067e
signal fmtv : 0xb008
Illegal instruction (core dumped)

In this case, only on the last recursion before the core dump does the 
handler warn that the end of the kernel sigframe (shown as frame end) 
lies above the current stack pointer from the main program.

That nicely explains the overwriting of the saved register a3 (which did 
point to the stack canary) and subsequent stack corruption abort.

Though the question remains - is this expected behaviour for programs 
that do deep recursion on the stack while taking signals (and the reason 
for the option to run signal handlers on an alternate stack)?

And why does this almost always appear to happen after bus error 
exceptions (frame format b)? The extra exception stack information isn't 
even accounted for in the above frame end address!

Result with sa_sigaction handler:

parent usp  : 0xef969e28
handler tos : 0xef969e6c
handler stack overwrote usp!
frame end   : 0xef969e7c
frame start : 0xef969b58
handler usp : 0xef969b40
signal usp  : 0xef969e04
signal pc   : 0x80000696
signal fmtv : 0x114

parent usp  : 0xef955008
handler tos : 0xef955064
handler stack overwrote usp!
frame end   : 0xef955074
frame start : 0xef954d50
handler usp : 0xef954d38
signal usp  : 0xef954ffc
signal pc   : 0x80000680
signal fmtv : 0xb008

parent usp  : 0xef945eb8
handler tos : 0xef945f0c
handler stack overwrote usp!
frame end   : 0xef945f1c
frame start : 0xef945bf8
handler usp : 0xef945be0
signal usp  : 0xef945ea8
signal pc   : 0xc009f37a
signal fmtv : 0x80

parent usp  : 0xef933eb8
handler tos : 0xef933f0c
handler stack overwrote usp!
frame end   : 0xef933f1c
frame start : 0xef933bf8
handler usp : 0xef933be0
signal usp  : 0xef933ea8
signal pc   : 0xc009f37a
signal fmtv : 0x80

parent usp  : 0xef921edc
handler tos : 0xef9aaca4
handler stack overwrote usp!
frame end   : 0xef9aacb4
frame start : 0xef9aa990
handler usp : 0xef9aa978
signal usp  : 0xef9aac40
signal pc   : 0x80000782
signal fmtv : 0x114

Illegal instruction (core dumped)

Exception right before crash was an interrupt in this case (only seen 
that once in this context, though I've seen lots of those in the course 
of the test runs). Frame start calculated from siginfo pointer value in 
this case.

I'll clean up the test code and combine all signal use cases into a 
single source file so Stan and Finn can give this a spin on their hardware.

Cheers,

	Michael

Re: reliable reproducer, was Re: core dump analysis

[Andreas Schwab]

On Apr 23 2023, Michael Schmitz wrote:

> Wasn't too hard actually. The signo parameter passed to the handler turns
> out to be passed by reference, and signo is located 4 bytes into the
> kernel sigframe.

That's not "passed by reference".  Function arguments are always passed
on the stack.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: reliable reproducer, was Re: core dump analysis

[Michael Schmitz]

Hi Andreas,

On 23/04/23 20:23, Andreas Schwab wrote:
> On Apr 23 2023, Michael Schmitz wrote:
>
>> Wasn't too hard actually. The signo parameter passed to the handler turns
>> out to be passed by reference, and signo is located 4 bytes into the
>> kernel sigframe.
> That's not "passed by reference".  Function arguments are always passed
> on the stack.

You're right of course, I should have phrased that different.

signo is passed by value as the first argument on the stack, but I can 
take a reference to signo to find it's address on the stack. That in 
turn will tell me the address of the complete signal frame on the stack.

This works because at the time the handler is entered, the stack points 
to the return address which has been set up by setup_frame, and that's 
the first longword in struct sigframe (frame.pretcode; points to 
frame.retcode which holds the trampoline code that will call 
sys_sigreturn upon exit from the handler). This in turn works because 
usp is loaded with the address of the sigframe right before calling the 
handler.

In effect, at least indirectly the signal handler receives all two 
arguments that matter (signo and sigcontext) even though sigcontext is 
not explicitly passed on the stack (the third longword is frame.code 
which holds the vector number AFAICS). Not sure what third argument you 
referred to in another mail.

With the rt_sigframe, pointers to siginfo and ucontext are stored after 
signo so no stack arithmetics is needed there to access all three 
arguments.

Anyway, the whole point of the exercise was to see whether I can peek at 
the registers in the signal frame and their saved copies on the stack, 
to perhaps spot when the stack corruption begins to happen. I know - 
should have used a debugger in the first place instead instrumenting a 
test case.

Cheers,

     Michael

Re: reliable reproducer, was Re: core dump analysis

[Andreas Schwab]

On Apr 24 2023, Michael Schmitz wrote:

> Not sure what third argument you referred to in another mail.

See struct sigframe and struct rt_sigframe.  The non-rt signal handler
gets signal number, vector number and sigcontext*.  The rt signal
handler gets signal number, siginfo* and ucontext*.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: reliable reproducer, was Re: core dump analysis

[Michael Schmitz]

[-- Attachment #1: Type: text/plain, Size: 1549 bytes --]

Hi Andreas,

On 24/04/23 09:48, Andreas Schwab wrote:
> On Apr 24 2023, Michael Schmitz wrote:
>
>> Not sure what third argument you referred to in another mail.
> See struct sigframe and struct rt_sigframe.  The non-rt signal handler
> gets signal number, vector number and sigcontext*.  The rt signal
> handler gets signal number, siginfo* and ucontext*.

Thanks, I see now. Got confused by the sigaction man page (despite 
working out that it's all there on the stack before...). Might need a 
comment in the code (or an update to the man pages).

I've rewritten my test program to make the non-rt handler take three 
arguments, just to simplify things. Also fixed the end of signal frame 
calculation for the non-rt case where the exception places additional 
data on the stack.

Running with the non-rt handler, the crash appears to happen right at 
the end of the recursion (or at least, I take no further SIGCHLD on the 
way back up the stack). With the rt handler, I see the stack depth 
decreased on the last signal taken before the crash.

When I enable dumping the extra exception frame contents (which will 
show prior stack contents when the exception only used a four-word 
frame) in the rt handler case, I only see saved register data placed 
there at the very end. That's different from previous tests where I saw 
the saved register patterns all the time. (but might have had the 
offsets wrong).

I'll see what peeking at the registers shows (now that I can be 
confident I have got the offsets correct).

Cheers,

     Michael




[-- Attachment #2: movemlrt.c --]
[-- Type: text/x-csrc, Size: 8286 bytes --]

#include <stdlib.h>
#include <unistd.h>
#include <stdio.h>
#include <signal.h>
#include <string.h>
#include <ucontext.h>
#include <asm/ptrace.h>

/* how deep at recursion */
int depth = 200000;
const int max_depth = 200000; 
/* how deep still on the stack while returning */
int rdepth = 0;

const unsigned long i0 = 0x91929394;
const unsigned long i1 = 0xa1a2a3a4;
const unsigned long i2 = 0xb1b2b3b4;
const unsigned long i3 = 0xc1c2c3c4;
const unsigned long i4 = 0xd1d2d3d4;
const unsigned long i5 = 0xe1e2e3e4;
const unsigned long i6 = 0xf1f2f3f4;

unsigned long o0;
unsigned long o1;
unsigned long o2;
unsigned long o3;
unsigned long o4;
unsigned long o5;
unsigned long o6;

unsigned long parent_usp;

/* extra frame size used for sa_handler case */
static const int frame_size_change[16] = {
  [1]   = -1, /* sizeof_field(struct frame, un.fmt1), */
  [2]   = 4, /* sizeof_field(struct frame, un.fmt2), */
  [3]   = 4, /* sizeof_field(struct frame, un.fmt3), */
  [4]   = 8, /* sizeof_field(struct frame, un.fmt4), */
  [5]   = -1, /* sizeof_field(struct frame, un.fmt5), */
  [6]   = -1, /* sizeof_field(struct frame, un.fmt6), */
  [7]   = 52, /* sizeof_field(struct frame, un.fmt7), */
  [8]   = -1, /* sizeof_field(struct frame, un.fmt8), */
  [9]   = 12, /* sizeof_field(struct frame, un.fmt9), */
  [10]  = 20, /* sizeof_field(struct frame, un.fmta), */
  [11]  = 84, /* sizeof_field(struct frame, un.fmtb), */
  [12]  = -1, /* sizeof_field(struct frame, un.fmtc), */
  [13]  = -1, /* sizeof_field(struct frame, un.fmtd), */
  [14]  = -1, /* sizeof_field(struct frame, un.fmte), */
  [15]  = -1, /* sizeof_field(struct frame, un.fmtf), */
};

static inline int frame_extra_sizes(int f)
{
        return frame_size_change[f];
}

static void rec(void)
{
	// initialize registers
	asm(	"	move.l %0, %%a2\n" 
		"	move.l %1, %%a3\n"
		"	move.l %2, %%a4\n"
		"	move.l %3, %%a5\n"
		"	move.l %4, %%d2\n"
		"	move.l %5, %%d3\n"
		"	move.l %6, %%d4\n"
		:
		: "m" (i0), "m" (i1), "m" (i2),
		  "m" (i3), "m" (i4), "m" (i5), "m" (i6)
		: "a2", "a3", "a4", "a5", "d2", "d3", "d4"
	);

	// note current usp
	asm(	"	move.l %%sp, %0\n" 
		: "=m" (parent_usp)
		:
		: "memory" 
	);

	// maybe fork a short-lived process
	if ((depth & 0x7ff) == 0)
		if (fork() == 0)
			exit(0);

	if (--depth)
		rec();	// callee to save & restore registers

	// compare register contents
	asm(	"	move.l %%a2, %0\n" 
		"	move.l %%a3, %1\n"
		"	move.l %%a4, %2\n"
		"	move.l %%a5, %3\n"
		"	move.l %%d2, %4\n"
		"	move.l %%d3, %5\n"
		"	move.l %%d4, %6\n"
		: "=m" (o0), "=m" (o1), "=m" (o2),
		  "=m" (o3), "=m" (o4), "=m" (o5), "=m" (o6)
		:
		: "memory" /* "a2", "a3", "a4", "a5", "d2", "d3", "d4" */
	);
	if (o0 != i0 || o1 != i1 || o2 != i2 ||
	    o3 != i3 || o4 != i4 || o5 != i5 || o6 != i6)
		asm("illegal");

	rdepth++;
}

static void handler(int signo, int vector, struct sigcontext *ctx)
{
	// use if only taking signo argument  
	// struct sigcontext *ctx = (struct sigcontext *) ((unsigned long) &signo + 24);
	unsigned long *frame = (unsigned long *)((unsigned long) &signo - 4);

	if (signo == SIGCHLD) {
		unsigned long usp, end_frame, extra;
		int i;

		asm(	"	move.l %%sp, %0\n" 
			: "=m" (usp)
			:
			: "memory" 
		);

		fprintf(stderr, "stack depth : %d\n", max_depth - depth);
		fprintf(stderr, "retn. depth : %d\n", max_depth - rdepth);
		fprintf(stderr, "parent usp  : 0x%lx\n", parent_usp);
		/* top of signal stack - including extra size for
		 * exception frames longer than four words
		 */
		// end_frame = (unsigned long) frame + 32 + sizeof(struct sigcontext);
		extra = frame_extra_sizes(ctx->sc_formatvec >> 12);
		end_frame = (unsigned long) ctx + sizeof(struct sigcontext) + extra;
                if (end_frame  > parent_usp)
                        fprintf(stderr, "signal frame overwrote parent usp!\n");
		fprintf(stderr, "frame end   : 0x%lx\n", end_frame);
		fprintf(stderr, "frame start : 0x%lx\n", frame);
		fprintf(stderr, "handler usp : 0x%lx\n", usp);	
		// fprintf(stderr, "signal vec  : 0x%lx\n", vector);
		fprintf(stderr, "signal usp  : 0x%lx\n", ctx->sc_usp);
		fprintf(stderr, "signal pc   : 0x%lx\n", ctx->sc_pc);
		fprintf(stderr, "signal fmtv : 0x%lx\n", ctx->sc_formatvec);
		fprintf(stderr, "\n");
	}
}
            
static void rthandler(int signo, siginfo_t *info, void *ucontext)
{
	ucontext_t *ctx = (ucontext_t *) ucontext;
	unsigned long *frame = (unsigned long *)((unsigned long) &signo - 4);

	if (signo == SIGCHLD) {
		unsigned long usp, end_frame;
		int i;

		asm(	"	move.l %%sp, %0\n" 
			: "=m" (usp)
			:
			: "memory" 
		);
		fprintf(stderr, "stack depth : %d\n", max_depth - depth);
		fprintf(stderr, "retn. depth : %d\n", max_depth - rdepth);
		fprintf(stderr, "parent usp  : 0x%lx\n", parent_usp);
		/* top of signal stack - extra size for exception
		 * frames longer than four words follows uc_filler[54]
		 */
		// end_frame = (unsigned long) info + sizeof(struct siginfo) + sizeof(struct ucontext);
		end_frame = (unsigned long) ucontext + sizeof(struct ucontext);
		if (end_frame > parent_usp)
			fprintf(stderr, "signal stack overwrote parent usp!\n");
		fprintf(stderr, "frame end   : 0x%lx\n", end_frame);
		fprintf(stderr, "frame start : 0x%lx\n", frame);
		fprintf(stderr, "handler usp : 0x%lx\n", usp);
		fprintf(stderr, "signal usp  : 0x%lx\n", ctx->uc_mcontext.gregs[15]);
		fprintf(stderr, "signal pc   : 0x%lx\n", ctx->uc_mcontext.gregs[16]);
		fprintf(stderr, "signal fmtv : 0x%lx\n", ctx->uc_filler[54]);
#if 0
		/* dump frame contents not used for exception frame extra */
		if (frame_extra_sizes(ctx->uc_filler[54] >> 12) <= 0)
			for (i=0; i < 26; i++) {
				if ( !(i & 3) ) fprintf(stderr, "\nextra %d\t", i);
				fprintf(stderr, "0x%lx\t", ctx->uc_filler[55+i]);
			}
#endif
		fprintf(stderr, "\n");
#if 0
		/* dump stored registers */
		fprintf(stderr, "signal stack: 0x%lx\n", ctx->uc_stack.ss_sp);
		fprintf(stderr, "signal usp  : 0x%lx\n", ctx->uc_mcontext.gregs[15]);
		fprintf(stderr, "signal pc   : 0x%lx\n", ctx->uc_mcontext.gregs[16]);
		fprintf(stderr, "signal f.d1 : 0x%lx\n", ctx->uc_filler[41]);
		fprintf(stderr, "signal f.d2 : 0x%lx\n", ctx->uc_filler[42]);
		fprintf(stderr, "signal f.d3 : 0x%lx\n", ctx->uc_filler[43]);
		fprintf(stderr, "signal f.d4 : 0x%lx\n", ctx->uc_filler[44]);
		fprintf(stderr, "signal f.d5 : 0x%lx\n", ctx->uc_filler[45]);
		fprintf(stderr, "signal f.a0 : 0x%lx\n", ctx->uc_filler[46]);
		fprintf(stderr, "signal f.a1 : 0x%lx\n", ctx->uc_filler[47]);
		fprintf(stderr, "signal f.a2 : 0x%lx\n", ctx->uc_filler[48]);
		fprintf(stderr, "signal f.d0 : 0x%lx\n", ctx->uc_filler[49]);
		fprintf(stderr, "signal f.od0: 0x%lx\n", ctx->uc_filler[50]);
		fprintf(stderr, "signal f.sad: 0x%lx\n", ctx->uc_filler[51]);
		fprintf(stderr, "signal f.sr : 0x%lx\n", ctx->uc_filler[52]);
		fprintf(stderr, "signal f.pc : 0x%lx\n", ctx->uc_filler[53]);
		fprintf(stderr, "signal a2   : 0x%lx\n", ctx->uc_mcontext.gregs[10]);
		fprintf(stderr, "signal a3   : 0x%lx\n", ctx->uc_mcontext.gregs[11]);
		fprintf(stderr, "signal a4   : 0x%lx\n", ctx->uc_mcontext.gregs[12]);
		fprintf(stderr, "signal a5   : 0x%lx\n", ctx->uc_mcontext.gregs[13]);
		fprintf(stderr, "signal d2   : 0x%lx\n", ctx->uc_mcontext.gregs[2]);
		fprintf(stderr, "signal d3   : 0x%lx\n", ctx->uc_mcontext.gregs[3]);
		fprintf(stderr, "signal d4   : 0x%lx\n\n", ctx->uc_mcontext.gregs[4]);
#endif
	}
}

int main(int use_rt)

{
	struct sigaction act;
	stack_t ss;

	use_rt--;
	/* poor man's getopt - use none, one or two args */ 
	fprintf(stderr, "use_rt = %d\n", use_rt);

	ss.ss_sp = malloc(SIGSTKSZ);
	if (ss.ss_sp == NULL) {
		perror("malloc");
		exit(EXIT_FAILURE);
	}
	
	ss.ss_size = SIGSTKSZ;
	ss.ss_flags = 0;

	if (use_rt & 2)
		fprintf(stderr, "alt stack 0x%lx - 0x%lx\n", ss.ss_sp, ss.ss_sp + ss.ss_size); 

	if (sigaltstack(&ss, NULL) == -1) {
		perror("sigaltstack");
		exit(EXIT_FAILURE);
	}

	memset(&act, 0, sizeof(act));
	sigemptyset(&act.sa_mask);

	if (use_rt) { 
		act.sa_flags = SA_SIGINFO;
		if (use_rt & 2) 
			act.sa_flags |= SA_ONSTACK;
		fprintf(stderr, "using sa_sigaction\n");
		act.sa_sigaction = rthandler;
	} else {
		fprintf(stderr, "using sa_handler\n");
		act.sa_handler = handler;
	}

	if (sigaction(SIGCHLD, &act, NULL) == -1) {
		perror("sigaction");
		exit(EXIT_FAILURE);
	}

	rec();
}

Re: reliable reproducer, was Re: core dump analysis

[Michael Schmitz]

Hi Finn,

Am 24.04.2023 um 15:51 schrieb Michael Schmitz:
> Hi Andreas,
>
> On 24/04/23 09:48, Andreas Schwab wrote:
>> On Apr 24 2023, Michael Schmitz wrote:
>>
>>> Not sure what third argument you referred to in another mail.
>> See struct sigframe and struct rt_sigframe.  The non-rt signal handler
>> gets signal number, vector number and sigcontext*.  The rt signal
>> handler gets signal number, siginfo* and ucontext*.
>
> Thanks, I see now. Got confused by the sigaction man page (despite
> working out that it's all there on the stack before...). Might need a
> comment in the code (or an update to the man pages).
>
> I've rewritten my test program to make the non-rt handler take three
> arguments, just to simplify things. Also fixed the end of signal frame
> calculation for the non-rt case where the exception places additional
> data on the stack.
>
> Running with the non-rt handler, the crash appears to happen right at
> the end of the recursion (or at least, I take no further SIGCHLD on the
> way back up the stack). With the rt handler, I see the stack depth
> decreased on the last signal taken before the crash.
>
> When I enable dumping the extra exception frame contents (which will
> show prior stack contents when the exception only used a four-word
> frame) in the rt handler case, I only see saved register data placed
> there at the very end. That's different from previous tests where I saw
> the saved register patterns all the time. (but might have had the
> offsets wrong).
>
> I'll see what peeking at the registers shows (now that I can be
> confident I have got the offsets correct).

Last two signals received before core dump below:

stack depth : 197952
retn. depth : 200000
parent usp  : 0xef933eb8
signal stack overwrote parent usp!
frame end   : 0xef933f1c
frame start : 0xef933bf8
handler usp : 0xef933be0
signal usp  : 0xef933ea8
signal pc   : 0xc009f37a

signal stack: 0x80004050
signal usp  : 0xef933ea8
signal pc   : 0xc009f37a
signal f.d1 : 0x0
signal f.d2 : 0x0
signal f.d3 : 0x0
signal f.d4 : 0x0
signal f.d5 : 0x0
signal f.a0 : 0x0
signal f.a1 : 0x0
signal f.a2 : 0x0
signal f.d0 : 0x0
signal f.od0: 0x0
signal f.sad: 0x0
signal f.sr : 0x0
signal f.pc : 0x0
signal fmtv : 0x80
signal a2   : 0x91929394
signal a3   : 0xa1a2a3a4
signal a4   : 0xb1b2b3b4
signal a5   : 0xc0104e0c
signal d2   : 0xd1d2d3d4
signal d3   : 0xe1e2e3e4
signal d4   : 0xf1f2f3f4

stack depth : 200000
retn. depth : 193942
parent usp  : 0xef921edc
signal stack overwrote parent usp!
frame end   : 0xef95733c
frame start : 0xef957018
handler usp : 0xef957000
signal usp  : 0xef9572c4
signal pc   : 0x8000073a

signal stack: 0x80004050
signal usp  : 0xef9572c4
signal pc   : 0x8000073a
signal f.d1 : 0xa1a2a3a4
signal f.d2 : 0xb1b2b3b4
signal f.d3 : 0xc1c2c3c4
signal f.d4 : 0xef957250
signal f.d5 : 0x80000722
signal f.a0 : 0xd1d2d3d4
signal f.a1 : 0xe1e2e3e4
signal f.a2 : 0xf1f2f3f4
signal f.d0 : 0x91929394
signal f.od0: 0xa1a2a3a4
signal f.sad: 0xb1b2b3b4
signal f.sr : 0xc1c2c3c4
signal f.pc : 0xef957274
signal fmtv : 0x114
signal a2   : 0x91929394
signal a3   : 0xa1a2a3a4
signal a4   : 0xb1b2b3b4
signal a5   : 0xc1c2c3c4
signal d2   : 0xd1d2d3d4
signal d3   : 0xe1e2e3e4
signal d4   : 0xf1f2f3f4

Illegal instruction (core dumped)

What I thought would hold the copies of pt_regs and switch_stack remains 
empty while on the way down the stack, and is filled with saved 
registers when on the way back up.

The registers as saved on exception, and copied to the mcontext struct, 
are correct in all cases, except for a5 which often contains 0xc0104e0c. 
Whenever that happens, a trap exception did get us into kernel mode. On 
both page faults and interrupts, a5 is as stored there (0xc1c2c3c4).

usp stored in the signal frame (i.e. usp when the signal was taken) is 
0x10 below that saved at the start of rec(). That's not enough to store 
7 registers plus return pc...

I still seem to have the sigframe end address wrong - it's larger than 
both parent usp and usp stored in the sigframe.

Still baffled.

Cheers,

	Michael

Re: reliable reproducer, was Re: core dump analysis

[Finn Thain]

On Sun, 23 Apr 2023, Michael Schmitz wrote:

> Am 23.04.2023 um 13:41 schrieb Michael Schmitz:
> 
> Though the question remains - is this expected behaviour for programs 
> that do deep recursion on the stack while taking signals (and the reason 
> for the option to run signal handlers on an alternate stack)?
> 

I don't understand how "deep recursion" can be used to explain this. We've 
seen crashes with only 1.8 MB of stack usage.

The best reason I can think of for having a signal stack would be that it 
may be better for signal delivery to fail than for the target process to 
fail. But I've no idea whether the kernel makes that kind of defensive 
programming possible (?)

> And why does this almost always appear to happen after bus error exceptions
> (frame format b)? The extra exception stack information isn't even accounted
> for in the above frame end address!
> 
> Result with sa_sigaction handler:
> 
> parent usp  : 0xef969e28
> handler tos : 0xef969e6c
> handler stack overwrote usp!
> frame end   : 0xef969e7c
> frame start : 0xef969b58
> handler usp : 0xef969b40
> signal usp  : 0xef969e04
> signal pc   : 0x80000696
> signal fmtv : 0x114
> 
> parent usp  : 0xef955008
> handler tos : 0xef955064
> handler stack overwrote usp!
> frame end   : 0xef955074
> frame start : 0xef954d50
> handler usp : 0xef954d38
> signal usp  : 0xef954ffc
> signal pc   : 0x80000680
> signal fmtv : 0xb008
> 
> parent usp  : 0xef945eb8
> handler tos : 0xef945f0c
> handler stack overwrote usp!
> frame end   : 0xef945f1c
> frame start : 0xef945bf8
> handler usp : 0xef945be0
> signal usp  : 0xef945ea8
> signal pc   : 0xc009f37a
> signal fmtv : 0x80
> 
> parent usp  : 0xef933eb8
> handler tos : 0xef933f0c
> handler stack overwrote usp!
> frame end   : 0xef933f1c
> frame start : 0xef933bf8
> handler usp : 0xef933be0
> signal usp  : 0xef933ea8
> signal pc   : 0xc009f37a
> signal fmtv : 0x80
> 
> parent usp  : 0xef921edc
> handler tos : 0xef9aaca4
> handler stack overwrote usp!
> frame end   : 0xef9aacb4
> frame start : 0xef9aa990
> handler usp : 0xef9aa978
> signal usp  : 0xef9aac40
> signal pc   : 0x80000782
> signal fmtv : 0x114
> 
> Illegal instruction (core dumped)
> 

I don't understand these results. If usp was really overwritten, the 
program would have crashed early, no?

> Exception right before crash was an interrupt in this case (only seen 
> that once in this context, though I've seen lots of those in the course 
> of the test runs). Frame start calculated from siginfo pointer value in 
> this case.
> 

I didn't realize that you could get a crash from a signal delivered 
following an interrupt. I'll try to modify the kernel such that signals 
are not delivered after page faults.

Re: reliable reproducer, was Re: core dump analysis

[Michael Schmitz]

Hi Finn,

On 23/04/23 21:23, Finn Thain wrote:
> On Sun, 23 Apr 2023, Michael Schmitz wrote:
>
>> Am 23.04.2023 um 13:41 schrieb Michael Schmitz:
>>
>> Though the question remains - is this expected behaviour for programs
>> that do deep recursion on the stack while taking signals (and the reason
>> for the option to run signal handlers on an alternate stack)?
>>
> I don't understand how "deep recursion" can be used to explain this. We've
> seen crashes with only 1.8 MB of stack usage.
OK, it's not really deep (though I've managed to get the test case 
aborted by the oom killer once on my rather puny RAM). But it's putting 
lots of frames on the stack in a short span while also utilizing the 
stack for signal delivery.
> The best reason I can think of for having a signal stack would be that it
> may be better for signal delivery to fail than for the target process to
> fail. But I've no idea whether the kernel makes that kind of defensive
> programming possible (?)

I don't think there's any provision for signal delivery to fail - the 
signal handler is started from the return-to-userspace code in entry.S, 
and upon return from the handler, a sigreturn syscall is automatically 
executed to clean up the stack. As long as the handler returns, all's fine.

Not sure what happens if the process context that the handler runs in is 
killed by the kernel - I suppose the entire process is killed and the 
context removed, so the issue of parent process survival is moot. But 
I'm sure we can place an illegal instruction in the handler as soon as a 
stack overflow is spotted, get a dump and look at that.

>> And why does this almost always appear to happen after bus error exceptions
>> (frame format b)? The extra exception stack information isn't even accounted
>> for in the above frame end address!
>>
>> Result with sa_sigaction handler:
>>
>> parent usp  : 0xef969e28
>> handler tos : 0xef969e6c
>> handler stack overwrote usp!
>> frame end   : 0xef969e7c
>> frame start : 0xef969b58
>> handler usp : 0xef969b40
>> signal usp  : 0xef969e04
>> signal pc   : 0x80000696
>> signal fmtv : 0x114
>>
>> parent usp  : 0xef955008
>> handler tos : 0xef955064
>> handler stack overwrote usp!
>> frame end   : 0xef955074
>> frame start : 0xef954d50
>> handler usp : 0xef954d38
>> signal usp  : 0xef954ffc
>> signal pc   : 0x80000680
>> signal fmtv : 0xb008
>>
>> parent usp  : 0xef945eb8
>> handler tos : 0xef945f0c
>> handler stack overwrote usp!
>> frame end   : 0xef945f1c
>> frame start : 0xef945bf8
>> handler usp : 0xef945be0
>> signal usp  : 0xef945ea8
>> signal pc   : 0xc009f37a
>> signal fmtv : 0x80
>>
>> parent usp  : 0xef933eb8
>> handler tos : 0xef933f0c
>> handler stack overwrote usp!
>> frame end   : 0xef933f1c
>> frame start : 0xef933bf8
>> handler usp : 0xef933be0
>> signal usp  : 0xef933ea8
>> signal pc   : 0xc009f37a
>> signal fmtv : 0x80
>>
>> parent usp  : 0xef921edc
>> handler tos : 0xef9aaca4
>> handler stack overwrote usp!
>> frame end   : 0xef9aacb4
>> frame start : 0xef9aa990
>> handler usp : 0xef9aa978
>> signal usp  : 0xef9aac40
>> signal pc   : 0x80000782
>> signal fmtv : 0x114
>>
>> Illegal instruction (core dumped)
>>
> I don't understand these results. If usp was really overwritten, the
> program would have crashed early, no?
I think we're still at the point where rec() is called recursively, 
before any returns.
>> Exception right before crash was an interrupt in this case (only seen
>> that once in this context, though I've seen lots of those in the course
>> of the test runs). Frame start calculated from siginfo pointer value in
>> this case.
>>
> I didn't realize that you could get a crash from a signal delivered
> following an interrupt. I'll try to modify the kernel such that signals
> are not delivered after page faults.

Yes, that was news to me, too. I've got swap enabled and probably see a 
lot more disk I/O than on your machines.

Delaying signal return until the next syscall or interrupt after page 
fault ought not be too hard - just replace the 'jra ret_from_exception' 
by 'RESTORE_ALL' (though that would also defer rescheduling until the 
next interrupt). For a proper solution, replicate exit_work without a 
call to do_signal_return ...

Cheers,

     Michael

Re: reliable reproducer, was Re: core dump analysis

[Finn Thain]

On Mon, 24 Apr 2023, Michael Schmitz wrote:

> > I don't understand these results. If usp was really overwritten, the 
> > program would have crashed early, no?
> 
> I think we're still at the point where rec() is called recursively, 
> before any returns.

Right. I wasn't thinking.

I'll try to confirm that each "overwrote usp" error from movemlrt.c 
corresponds to visible corruption at the given address in the core dump.

> >> Exception right before crash was an interrupt in this case (only seen 
> >> that once in this context, though I've seen lots of those in the 
> >> course of the test runs). Frame start calculated from siginfo pointer 
> >> value in this case.
> >
> > I didn't realize that you could get a crash from a signal delivered 
> > following an interrupt. I'll try to modify the kernel such that 
> > signals are not delivered after page faults.
> 
> Yes, that was news to me, too.
> 

That seems to be a mistake (?) I didn't see any failures when I patched 
the kernel to skip signal delivery after a page fault.

Re: signal delivery, was Re: reliable reproducer

[Finn Thain]

On Tue, 25 Apr 2023, Finn Thain wrote:

> On Tue, 25 Apr 2023, Michael Schmitz wrote:
> 
> > As to a cause for the corruption: all the calculations in setup_frame 
> > and sys_sigreturn use fsize, but get_sigframe() masks off the result 
> > of usp - sizeof(sigframe) - fsize to place the entire frame at a 
> > quadword boundary.
> ...
> 
> I wonder if we are seeing some fallout from the issue described in 
> do_page_fault() i.e. usp is unreliable.
> 
>                 /* Accessing the stack below usp is always a bug.  The
>                    "+ 256" is there due to some instructions doing
>                    pre-decrement on the stack and that doesn't show up
>                    until later.  */
>                 if (address + 256 < rdusp())
>                         goto map_err;
> 
> Maybe we should try modifying get_sigframe() to increase the gap between 
> the signal and exception frames from 0-1 long words up to 64-65 long 
> words.
> 

It turns out that doing so (patch below) does make the problem go away. 
Was the exception frame getting clobbered?

diff --git a/arch/m68k/kernel/signal.c b/arch/m68k/kernel/signal.c
index b9f6908a31bc..94104699f5a8 100644
--- a/arch/m68k/kernel/signal.c
+++ b/arch/m68k/kernel/signal.c
@@ -862,7 +862,7 @@ get_sigframe(struct ksignal *ksig, size_t frame_size)
 {
 	unsigned long usp = sigsp(rdusp(), ksig);
 
-	return (void __user *)((usp - frame_size) & -8UL);
+	return (void __user *)((usp - 256 - frame_size) & -8UL);
 }
 
 static int setup_frame(struct ksignal *ksig, sigset_t *set,

Re: signal delivery, was Re: reliable reproducer

[Michael Schmitz]

Hi Finn,

Am 25.04.2023 um 13:55 schrieb Finn Thain:
> On Tue, 25 Apr 2023, Finn Thain wrote:
>
>> On Tue, 25 Apr 2023, Michael Schmitz wrote:
>>
>>> As to a cause for the corruption: all the calculations in setup_frame
>>> and sys_sigreturn use fsize, but get_sigframe() masks off the result
>>> of usp - sizeof(sigframe) - fsize to place the entire frame at a
>>> quadword boundary.
>> ...
>>
>> I wonder if we are seeing some fallout from the issue described in
>> do_page_fault() i.e. usp is unreliable.
>>
>>                 /* Accessing the stack below usp is always a bug.  The
>>                    "+ 256" is there due to some instructions doing
>>                    pre-decrement on the stack and that doesn't show up
>>                    until later.  */
>>                 if (address + 256 < rdusp())
>>                         goto map_err;
>>
>> Maybe we should try modifying get_sigframe() to increase the gap between
>> the signal and exception frames from 0-1 long words up to 64-65 long
>> words.
>>
>
> It turns out that doing so (patch below) does make the problem go away.
> Was the exception frame getting clobbered?

Might happen, if the frame gap isn't actually equal to the exception 
frame extra size anymore? Aligning the start of the signal frame to the 
next lower quadword boundary increases the gap size.

When setting up the sigframe, the extra is copied to the correct 
location (right past struct sigframe, or into uc_filler). When moving 
that exception frame into place, the assumption is that the gap is the 
extra size, not more.

I'll try dropping the quadword alignment constraint - the return 
trampoline still ought to remain longword aligned.

Cheers,

	Michael


>
> diff --git a/arch/m68k/kernel/signal.c b/arch/m68k/kernel/signal.c
> index b9f6908a31bc..94104699f5a8 100644
> --- a/arch/m68k/kernel/signal.c
> +++ b/arch/m68k/kernel/signal.c
> @@ -862,7 +862,7 @@ get_sigframe(struct ksignal *ksig, size_t frame_size)
>  {
>  	unsigned long usp = sigsp(rdusp(), ksig);
>
> -	return (void __user *)((usp - frame_size) & -8UL);
> +	return (void __user *)((usp - 256 - frame_size) & -8UL);
>  }
>
>  static int setup_frame(struct ksignal *ksig, sigset_t *set,
>

Re: signal delivery, was Re: reliable reproducer

[Michael Schmitz]

Hi Finn,

Am 25.04.2023 um 14:32 schrieb Michael Schmitz:
>>> Maybe we should try modifying get_sigframe() to increase the gap between
>>> the signal and exception frames from 0-1 long words up to 64-65 long
>>> words.
>>>
>>
>> It turns out that doing so (patch below) does make the problem go away.
>> Was the exception frame getting clobbered?
>
> Might happen, if the frame gap isn't actually equal to the exception
> frame extra size anymore? Aligning the start of the signal frame to the
> next lower quadword boundary increases the gap size.
>
> When setting up the sigframe, the extra is copied to the correct
> location (right past struct sigframe, or into uc_filler). When moving
> that exception frame into place, the assumption is that the gap is the
> extra size, not more.
>
> I'll try dropping the quadword alignment constraint - the return
> trampoline still ought to remain longword aligned.

No luck - still stack corruption.

Cheers,

	Michael

Re: signal delivery, was Re: reliable reproducer

[Michael Schmitz]

Hi Finn,

On 25/04/23 14:32, Michael Schmitz wrote:
> Hi Finn,
>
> Am 25.04.2023 um 13:55 schrieb Finn Thain:
>> On Tue, 25 Apr 2023, Finn Thain wrote:
>>
>>> On Tue, 25 Apr 2023, Michael Schmitz wrote:
>>>
>>>> As to a cause for the corruption: all the calculations in setup_frame
>>>> and sys_sigreturn use fsize, but get_sigframe() masks off the result
>>>> of usp - sizeof(sigframe) - fsize to place the entire frame at a
>>>> quadword boundary.
>>> ...
>>>
>>> I wonder if we are seeing some fallout from the issue described in
>>> do_page_fault() i.e. usp is unreliable.
>>>
>>>                 /* Accessing the stack below usp is always a bug.  The
>>>                    "+ 256" is there due to some instructions doing
>>>                    pre-decrement on the stack and that doesn't show up
>>>                    until later.  */
>>>                 if (address + 256 < rdusp())
>>>                         goto map_err;
>>>
Minimal gap that avoids the corruption for me (in your patch below) is 
20 bytes. Still can't explain that (and I won't have time to throw more 
confusion into the discussion until maybe the weekend).

Cheers,

     Michael


>>> Maybe we should try modifying get_sigframe() to increase the gap 
>>> between
>>> the signal and exception frames from 0-1 long words up to 64-65 long
>>> words.
>>>
>>
>> It turns out that doing so (patch below) does make the problem go away.
>> Was the exception frame getting clobbered?
>
> Might happen, if the frame gap isn't actually equal to the exception 
> frame extra size anymore? Aligning the start of the signal frame to 
> the next lower quadword boundary increases the gap size.
>
> When setting up the sigframe, the extra is copied to the correct 
> location (right past struct sigframe, or into uc_filler). When moving 
> that exception frame into place, the assumption is that the gap is the 
> extra size, not more.
>
> I'll try dropping the quadword alignment constraint - the return 
> trampoline still ought to remain longword aligned.
>
> Cheers,
>
>     Michael
>
>
>>
>> diff --git a/arch/m68k/kernel/signal.c b/arch/m68k/kernel/signal.c
>> index b9f6908a31bc..94104699f5a8 100644
>> --- a/arch/m68k/kernel/signal.c
>> +++ b/arch/m68k/kernel/signal.c
>> @@ -862,7 +862,7 @@ get_sigframe(struct ksignal *ksig, size_t 
>> frame_size)
>>  {
>>      unsigned long usp = sigsp(rdusp(), ksig);
>>
>> -    return (void __user *)((usp - frame_size) & -8UL);
>> +    return (void __user *)((usp - 256 - frame_size) & -8UL);
>>  }
>>
>>  static int setup_frame(struct ksignal *ksig, sigset_t *set,
>>

Re: signal delivery, was Re: reliable reproducer

[Finn Thain]

[Cc kernel mailing list]

On Sat, 29 Apr 2023, Michael Schmitz wrote:

> The address right past the end of the signal frame is equal to the fault 
> address. Any smaller gap size (as evidenced by the fact that 16 causes 
> corruption) will see the signal frame go past the fault address.
> 
> Using the user stack pointer to calculate the address for the signal 
> frame is incorrect - the user stack pointer still reflects the situation 
> before the faulting instruction.
> 

Well, to quote do_page_fault(), USP is not reliable "due to some 
instructions doing pre-decrement on the stack and that doesn't show up 
until later".

> I think what we need to do in setup_(rt_)_signal is to check if the 
> frame format is format b, and use the fault pc address from the 
> exception frame (if it's lower) instead of the usp. I'll test a patch 
> for that shortly.
> 

Right. If we fix this in the signal handling code, we take care of address 
errors as well, which was my concern with Andreas' patch. We can do what 
do_page_fault() does and assume the worst (256 bytes?).

I believe we can use USP to get a worst case estimate for the future 
extent of the user stack. Using the fault address seems like an 
unnecessary complication.

I agree that this should be contingent on frame format 0xB and also on 
CPU_IS_020_OR_030 so the code is omitted on other builds. (Much like 
Andreas' patch.)

Here's what I learned from running gdb on the LC III. The transcript below 
shows a printk from handle_signal() following a printk from 
bus_error030(). So this is a page fault at the top of stack page 
efffe000..efffefff which also delivered a signal.

(gdb) file /root/stack-test
Reading symbols from /root/stack-test...
(No debugging symbols found in /root/stack-test)

(gdb) run 2050
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /root/stack-test 2050
[ 2958.890000] bus_error030: page fault: pc 80038d14, usp efffee28, addr efffee28
starting recursion
[ 2958.940000] bus_error030: page fault: pc 8000038c, usp efffe00c, addr efffdffc
[ 2958.960000] bus_error030: page fault: pc 8000038c, usp efffd004, addr efffcffc
[ 2958.970000] bus_error030: page fault: pc 8000038c, usp efffbffc, addr efffbffc
[ 2958.980000] bus_error030: page fault: pc 8000038c, usp efffa010, addr efff9ffc
[ 2958.990000] bus_error030: page fault: pc 8000038c, usp efff9008, addr efff8ffc
[ 2959.010000] bus_error030: page fault: pc 8000038c, usp efff8000, addr efff7ffc
[ 2959.030000] bus_error030: page fault: pc 8000038c, usp efff6014, addr efff5ffc
[ 2959.050000] bus_error030: page fault: pc 8000038c, usp efff500c, addr efff4ffc
[ 2959.060000] bus_error030: page fault: pc 8000038c, usp efff4004, addr efff3ffc
[ 2959.080000] bus_error030: page fault: pc 8000038c, usp efff2ffc, addr efff2ffc
[ 2959.100000] bus_error030: page fault: pc 8000038c, usp efff1010, addr efff0ffc
[ 2959.110000] bus_error030: page fault: pc 8000038c, usp efff0008, addr effefffc
[ 2959.130000] bus_error030: page fault: pc 8000038c, usp effef000, addr effeeffc
done.
starting recursion with fork
[Detaching after fork from child process 166]
[ 2959.310000] bus_error030: page fault: pc 8000038c, usp effff014, addr efffeffc
[ 2959.350000] handle_signal: fmtvec b008, pc 8000038c, usp efffee88


With a breakpoint set in the userspace signal handler, we can examine 
stack memory from the USP as it presently is, going back up to USP as it 
was at the time of the page fault (both addresses are given in printk 
output above).

The difference is 396 bytes, and pahole confirms that sizeof(struct 
sigframe) + sizeof(struct frame.un.fmtb) == 312 + 84 == 396.

(Note that this kernel was patched so that it doesn't truncate the result 
of that pointer arithmetic. Hence there's no gap between the bottom of the 
user stack and the start of the signal frame, which tends to confuse the 
issue, as the gap would normally come and go unpredictably.)

Breakpoint 1, 0x800004a4 in handler ()
(gdb) x/110z 0xeffff014 - 396
0xefffee88:     0xefffee98      0x00000011      0x00000008      0xefffeea4
0xefffee98:     0x70774e40      0xf1f2f3f4      0x00000000      0x00000000
0xefffeea8:     0xeffff014      0x000007a7      0x000007a6      0x800744ec
0xefffeeb8:     0x8006d2b0      0x00008000      0x038cb008      0x7fff0000
0xefffeec8:     0xffffffff      0xffffffff      0x7fff0000      0xffffffff
0xefffeed8:     0xffffffff      0x00000000      0x00000000      0x00000000
0xefffeee8:     0x1f386980      0xf0fff0ff      0x0f800000      0xf0ff0000
0xefffeef8:     0x00000000      0x408ece9a      0x00000000      0x00000000
0xefffef08:     0x00000000      0x00000000      0x3ffe0000      0x00000000
0xefffef18:     0x00000000      0xffffffff      0x7c0effff      0x014ee530
0xefffef28:     0x08010000      0x00000004      0x00994000      0x00995e94
0xefffef38:     0x014ed540      0x00994000      0x0040ee24      0x08010000
0xefffef48:     0x00000004      0x004fa794      0x00995f18      0x00995e34
0xefffef58:     0x00079a72      0x004fa794      0x000270ce      0x00995e54
0xefffef68:     0x00027370      0x00000001      0x014f1e60      0x00995f2c
0xefffef78:     0x00994000      0x00000011      0x00995f18      0x00995e98
0xefffef88:     0x00027d08      0x00000011      0x00000004      0x00000000
0xefffef98:     0x00995f2c      0xd1d2d3d4      0xe1e2e3e4      0xf1f2f3f4
0xefffefa8:     0xeffffed0      0x00000001      0x00000001      0x00995f78
0xefffefb8:     0xb1b2b3b4      0xc1c2c3c4      0x0eee0709      0x24798004
0xefffefc8:     0xefffeffc      0xefffeff8      0xd1d2d3d4      0x48e7383c
0xefffefd8:     0x80000394      0x80000392      0x80000390      0x000007a7
0xefffefe8:     0x383cff0d      0x000ff153      0x383c2479      0x8007b4e0
0xefffeff8:     0x0000383c      0x00000000      0x80100000      0x800003f8
0xeffff008:     0x000000e0      0x014ed500      0xeffffcc4      0xc1c2c3c4
0xeffff018:     0xeffff03c      0x800003f8      0xd1d2d3d4      0xe1e2e3e4
0xeffff028:     0xf1f2f3f4      0x91929394      0xa1a2a3a4      0xb1b2b3b4
0xeffff038:     0xc1c2c3c4      0xeffff060


Here's the same hex dump with struct member names added, much like the 
output from your test program:

efffee88:	efffee98	sigframe.pretcode
efffee8c:	00000011	sigframe.sig
efffee90:	00000008	sigframe.code
efffee94:	efffeea4	sigframe.psc
efffee98:	70774e40	sigframe.retcode
efffee9c:	f1f2f3f4	sigframe.retcode
efffeea0:	00000000	sigframe.extramask
efffeea4:	00000000	sigframe.sc.mask
efffeea8:	effff014	sigframe.sc.sc_usp
efffeeac:	000007a7	sigframe.sc.sc_d0
efffeeb0:	000007a6	sigframe.sc.sc_d1
efffeeb4:	800744ec	sigframe.sc.sc_a0
efffeeb8:	8006d2b0	sigframe.sc.sc_a1
efffeebc:	00008000	sigframe.sc.sc_sr, sc_pc
efffeec0:	038cb008	sigframe.sc.sc_pc, sc_formatvec
efffeec4:	7fff0000	sigframe.sc.sc_fpregs
efffeec8:	ffffffff	sigframe.sc.sc_fpregs
efffeecc:	ffffffff	sigframe.sc.sc_fpregs
efffeed0:	7fff0000	sigframe.sc.sc_fpregs
efffeed4:	ffffffff	sigframe.sc.sc_fpregs
efffeed8:	ffffffff	sigframe.sc.sc_fpregs
efffeedc:	00000000	sigframe.sc.sc_fpcntl
efffeee0:	00000000	sigframe.sc.sc_fpcntl
efffeee4:	00000000	sigframe.sc.sc_fpcntl
efffeee8:	1f386980	sigframe.sc.sc_fpstate
efffeeec:	f0fff0ff	sigframe.sc.sc_fpstate
efffeef0:	0f800000	sigframe.sc.sc_fpstate
efffeef4:	f0ff0000	sigframe.sc.sc_fpstate
efffeef8:	00000000	sigframe.sc.sc_fpstate
efffeefc:	408ece9a	sigframe.sc.sc_fpstate
efffef00:	00000000	sigframe.sc.sc_fpstate
efffef04:	00000000	sigframe.sc.sc_fpstate
efffef08:	00000000	sigframe.sc.sc_fpstate
efffef0c:	00000000	sigframe.sc.sc_fpstate
efffef10:	3ffe0000	sigframe.sc.sc_fpstate
efffef14:	00000000	sigframe.sc.sc_fpstate
efffef18:	00000000	sigframe.sc.sc_fpstate
efffef1c:	ffffffff	sigframe.sc.sc_fpstate
efffef20:	7c0effff	sigframe.sc.sc_fpstate
efffef24:	014ee530	sigframe.sc.sc_fpstate
efffef28:	08010000	sigframe.sc.sc_fpstate
efffef2c:	00000004	sigframe.sc.sc_fpstate
efffef30:	00994000	sigframe.sc.sc_fpstate
efffef34:	00995e94	sigframe.sc.sc_fpstate
efffef38:	014ed540	sigframe.sc.sc_fpstate
efffef3c:	00994000	sigframe.sc.sc_fpstate
efffef40:	0040ee24	sigframe.sc.sc_fpstate
efffef44:	08010000	sigframe.sc.sc_fpstate
efffef48:	00000004	sigframe.sc.sc_fpstate
efffef4c:	004fa794	sigframe.sc.sc_fpstate
efffef50:	00995f18	sigframe.sc.sc_fpstate
efffef54:	00995e34	sigframe.sc.sc_fpstate
efffef58:	00079a72	sigframe.sc.sc_fpstate
efffef5c:	004fa794	sigframe.sc.sc_fpstate
efffef60:	000270ce	sigframe.sc.sc_fpstate
efffef64:	00995e54	sigframe.sc.sc_fpstate
efffef68:	00027370	sigframe.sc.sc_fpstate
efffef6c:	00000001	sigframe.sc.sc_fpstate
efffef70:	014f1e60	sigframe.sc.sc_fpstate
efffef74:	00995f2c	sigframe.sc.sc_fpstate
efffef78:	00994000	sigframe.sc.sc_fpstate
efffef7c:	00000011	sigframe.sc.sc_fpstate
efffef80:	00995f18	sigframe.sc.sc_fpstate
efffef84:	00995e98	sigframe.sc.sc_fpstate
efffef88:	00027d08	sigframe.sc.sc_fpstate
efffef8c:	00000011	sigframe.sc.sc_fpstate
efffef90:	00000004	sigframe.sc.sc_fpstate
efffef94:	00000000	sigframe.sc.sc_fpstate
efffef98:	00995f2c	sigframe.sc.sc_fpstate
efffef9c:	d1d2d3d4	sigframe.sc.sc_fpstate
efffefa0:	e1e2e3e4	sigframe.sc.sc_fpstate
efffefa4:	f1f2f3f4	sigframe.sc.sc_fpstate
efffefa8:	effffed0	sigframe.sc.sc_fpstate
efffefac:	00000001	sigframe.sc.sc_fpstate
efffefb0:	00000001	sigframe.sc.sc_fpstate
efffefb4:	00995f78	sigframe.sc.sc_fpstate
efffefb8:	b1b2b3b4	sigframe.sc.sc_fpstate
efffefbc:	c1c2c3c4	sigframe.sc.sc_fpstate
efffefc0:	0eee0709	fmtb.int1, ssw
efffefc4:	24798004	fmtb.isc, isb
efffefc8:	efffeffc	fmtb.daddr
efffefcc:	efffeff8	fmtb.int2
efffefd0:	d1d2d3d4	fmtb.dobuf
efffefd4:	48e7383c	fmtb.int3
efffefd8:	80000394	fmtb.int3
efffefdc:	80000392	fmtb.baddr
efffefe0:	80000390	fmtb.int4
efffefe4:	000007a7	fmtb.dibuf
efffefe8:	383cff0d	fmtb.int5
efffefec:	000ff153	fmtb.int5, ver, int6
efffeff0:	383c2479	fmtb.int7
efffeff4:	8007b4e0	fmtb.int7
efffeff8:	0000383c	fmtb.int7
efffeffc:	00000000	fmtb.int7
effff000: *	80100000	fmtb.int7
effff004: *	800003f8	fmtb.int7
effff008: *	000000e0	fmtb.int7
effff00c: *	014ed500	fmtb.int7
effff010: *	effffcc4	fmtb.int7
effff014:	c1c2c3c4	user stack
effff018:	effff03c	user stack
effff01c:	800003f8	user stack
effff020:	d1d2d3d4	user stack
effff024:	e1e2e3e4	user stack
effff028:	f1f2f3f4	user stack
effff02c:	91929394	user stack
effff030:	a1a2a3a4	user stack
effff034:	b1b2b3b4	user stack
effff038:	c1c2c3c4	user stack
effff03c:	effff060	user stack

The portion marked with an * is the part that damages the user stack, as 
these locations are all higher than USP. This just confirms by other means 
what your test program already demonstrated.


(gdb) c
Continuing.
[ 3334.340000] bus_error030: page fault: pc 8000038c, usp efffe00c, addr efffdffc
[ 3334.360000] bus_error030: page fault: pc 8000038c, usp efffd004, addr efffcffc
[ 3334.380000] bus_error030: page fault: pc 8000038c, usp efffbffc, addr efffbffc
[ 3334.390000] bus_error030: page fault: pc 8000038c, usp efffa010, addr efff9ffc
[ 3334.410000] bus_error030: page fault: pc 8000038c, usp efff9008, addr efff8ffc
[ 3334.430000] bus_error030: page fault: pc 8000038c, usp efff8000, addr efff7ffc
[ 3334.440000] bus_error030: page fault: pc 8000038c, usp efff6014, addr efff5ffc
[ 3334.450000] bus_error030: page fault: pc 8000038c, usp efff500c, addr efff4ffc
[ 3334.460000] bus_error030: page fault: pc 8000038c, usp efff4004, addr efff3ffc
[ 3334.480000] bus_error030: page fault: pc 8000038c, usp efff2ffc, addr efff2ffc
[ 3334.500000] bus_error030: page fault: pc 8000038c, usp efff1010, addr efff0ffc
[ 3334.510000] bus_error030: page fault: pc 8000038c, usp efff0008, addr effefffc
[ 3334.530000] bus_error030: page fault: pc 8000038c, usp effef000, addr effeeffc
[Detaching after fork from child process 167]

Breakpoint 1, 0x800004a4 in handler ()
(gdb) c
Continuing.

Program received signal SIGILL, Illegal instruction.
0x80000492 in rec ()


My test program terminates having detected the corruption visible at 
0xeffff000 but gdb allows us to examine that memory afterwards:

(gdb) x/110z 0xeffff014 - 396
0xefffee88:     0xc1c2c3c4      0xefffeeb0      0x800003f8      0xd1d2d3d4
0xefffee98:     0xe1e2e3e4      0xf1f2f3f4      0x91929394      0xa1a2a3a4
0xefffeea8:     0xb1b2b3b4      0xc1c2c3c4      0xefffeed4      0x800003f8
0xefffeeb8:     0xd1d2d3d4      0xe1e2e3e4      0xf1f2f3f4      0x91929394
0xefffeec8:     0xa1a2a3a4      0xb1b2b3b4      0xc1c2c3c4      0xefffeef8
0xefffeed8:     0x800003f8      0xd1d2d3d4      0xe1e2e3e4      0xf1f2f3f4
0xefffeee8:     0x91929394      0xa1a2a3a4      0xb1b2b3b4      0xc1c2c3c4
0xefffeef8:     0xefffef1c      0x800003f8      0xd1d2d3d4      0xe1e2e3e4
0xefffef08:     0xf1f2f3f4      0x91929394      0xa1a2a3a4      0xb1b2b3b4
0xefffef18:     0xc1c2c3c4      0xefffef40      0x800003f8      0xd1d2d3d4
0xefffef28:     0xe1e2e3e4      0xf1f2f3f4      0x91929394      0xa1a2a3a4
0xefffef38:     0xb1b2b3b4      0xc1c2c3c4      0xefffef64      0x800003f8
0xefffef48:     0xd1d2d3d4      0xe1e2e3e4      0xf1f2f3f4      0x91929394
0xefffef58:     0xa1a2a3a4      0xb1b2b3b4      0xc1c2c3c4      0xefffef88
0xefffef68:     0x800003f8      0xd1d2d3d4      0xe1e2e3e4      0xf1f2f3f4
0xefffef78:     0x91929394      0xa1a2a3a4      0xb1b2b3b4      0xc1c2c3c4
0xefffef88:     0xefffefac      0x800003f8      0xd1d2d3d4      0xe1e2e3e4
0xefffef98:     0xf1f2f3f4      0x91929394      0xa1a2a3a4      0xb1b2b3b4
0xefffefa8:     0xc1c2c3c4      0xefffefd0      0x800003f8      0xd1d2d3d4
0xefffefb8:     0xe1e2e3e4      0xf1f2f3f4      0x91929394      0xa1a2a3a4
0xefffefc8:     0xb1b2b3b4      0xc1c2c3c4      0xefffeff4      0x800003f8
0xefffefd8:     0xd1d2d3d4      0xe1e2e3e4      0xf1f2f3f4      0x91929394
0xefffefe8:     0xa1a2a3a4      0xb1b2b3b4      0xc1c2c3c4      0xeffff018
0xefffeff8:     0x800003f8      0xd1d2d3d4      0x80100000      0x800003f8
0xeffff008:     0x000000e0      0x014ed500      0xeffffcc4      0xc1c2c3c4
0xeffff018:     0xeffff03c      0x800003f8      0xd1d2d3d4      0xe1e2e3e4
0xeffff028:     0xf1f2f3f4      0x91929394      0xa1a2a3a4      0xb1b2b3b4
0xeffff038:     0xc1c2c3c4      0xeffff060
(gdb) 


So the page that caused the fault never shows any corruption. It turns out 
that the MOVEM resumes correctly after the signal is handled, which 
overwrites the stack memory used for the signal frame, starting at the top 
of the new stack page. I had incorrectly assumed the MOVEM had gone awry 
but it actually resumed just fine.

Re: signal delivery, was Re: reliable reproducer

[Finn Thain]

On Sat, 29 Apr 2023, Finn Thain wrote:

> efffefc0:	0eee0709	fmtb.int1, ssw
> efffefc4:	24798004	fmtb.isc, isb
> efffefc8:	efffeffc	fmtb.daddr
> efffefcc:	efffeff8	fmtb.int2
> efffefd0:	d1d2d3d4	fmtb.dobuf
> efffefd4:	48e7383c	fmtb.int3
> efffefd8:	80000394	fmtb.int3
> efffefdc:	80000392	fmtb.baddr
> efffefe0:	80000390	fmtb.int4
> efffefe4:	000007a7	fmtb.dibuf
> efffefe8:	383cff0d	fmtb.int5
> efffefec:	000ff153	fmtb.int5, ver, int6
> efffeff0:	383c2479	fmtb.int7
> efffeff4:	8007b4e0	fmtb.int7
> efffeff8:	0000383c	fmtb.int7
> efffeffc:	00000000	fmtb.int7
> effff000: *	80100000	fmtb.int7
> effff004: *	800003f8	fmtb.int7
> effff008: *	000000e0	fmtb.int7
> effff00c: *	014ed500	fmtb.int7
> effff010: *	effffcc4	fmtb.int7
> effff014:	c1c2c3c4	user stack
> effff018:	effff03c	user stack
> effff01c:	800003f8	user stack
> effff020:	d1d2d3d4	user stack
> effff024:	e1e2e3e4	user stack
> effff028:	f1f2f3f4	user stack
> effff02c:	91929394	user stack
> effff030:	a1a2a3a4	user stack
> effff034:	b1b2b3b4	user stack
> effff038:	c1c2c3c4	user stack
> effff03c:	effff060	user stack
> 
> The portion marked with an * is the part that damages the user stack, as 
> these locations are all higher than USP. 

Well, the nominal "USP" is 0xeffff014. But at this point MOVEM had already 
used up 5 long words below that, and will use up 2 more once resumed. So 
the real USP is 0xefffeff8 (which is found in fmtb.int2, FWIW).

Re: signal delivery, was Re: reliable reproducer

[Michael Schmitz]

Hi Finn,

Am 29.04.2023 um 12:28 schrieb Finn Thain:
> [Cc kernel mailing list]
>
> On Sat, 29 Apr 2023, Michael Schmitz wrote:
>
>> The address right past the end of the signal frame is equal to the fault
>> address. Any smaller gap size (as evidenced by the fact that 16 causes
>> corruption) will see the signal frame go past the fault address.
>>
>> Using the user stack pointer to calculate the address for the signal
>> frame is incorrect - the user stack pointer still reflects the situation
>> before the faulting instruction.
>>
>
> Well, to quote do_page_fault(), USP is not reliable "due to some
> instructions doing pre-decrement on the stack and that doesn't show up
> until later".
>
>> I think what we need to do in setup_(rt_)_signal is to check if the
>> frame format is format b, and use the fault pc address from the
>> exception frame (if it's lower) instead of the usp. I'll test a patch
>> for that shortly.
>>
>
> Right. If we fix this in the signal handling code, we take care of address
> errors as well, which was my concern with Andreas' patch. We can do what
> do_page_fault() does and assume the worst (256 bytes?).

Well, we could do that if we could be certain this does not cause a 
memory leak in some way. The reason I bring this up is that I've just 
seen the kernel that I'd used to run the latest test cases (which 
inserts a 20 byte gap only!) run amok terminating pretty much my entire 
user space because it ran out of memory. Never seen the like of that.

> I believe we can use USP to get a worst case estimate for the future
> extent of the user stack. Using the fault address seems like an
> unnecessary complication.

Yes, and it's not guaranteed to be a good estimate even in the general 
case. Our intended use case is where the instruction was a movem, and 
the fault address and USP are on adjacent pages. While that happens a 
lot during function calls, it's by no means the only place it will 
happen. We need to make sure the start of the frame ends up on the same 
page as the fault address, if at all possible.

What is the most data a moveml <...>,sp@- can take? If that's not too 
much, a constant offset for the signal stack in case of format b frames 
on 020/030 might be easiest.

>
> I agree that this should be contingent on frame format 0xB and also on
> CPU_IS_020_OR_030 so the code is omitted on other builds. (Much like
> Andreas' patch.)
>
> Here's what I learned from running gdb on the LC III. The transcript below
> shows a printk from handle_signal() following a printk from
> bus_error030(). So this is a page fault at the top of stack page
> efffe000..efffefff which also delivered a signal.
>
> (gdb) file /root/stack-test
> Reading symbols from /root/stack-test...
> (No debugging symbols found in /root/stack-test)
>
> (gdb) run 2050
> The program being debugged has been started already.
> Start it from the beginning? (y or n) y
> Starting program: /root/stack-test 2050
> [ 2958.890000] bus_error030: page fault: pc 80038d14, usp efffee28, addr efffee28
> starting recursion
> [ 2958.940000] bus_error030: page fault: pc 8000038c, usp efffe00c, addr efffdffc
> [ 2958.960000] bus_error030: page fault: pc 8000038c, usp efffd004, addr efffcffc
> [ 2958.970000] bus_error030: page fault: pc 8000038c, usp efffbffc, addr efffbffc
> [ 2958.980000] bus_error030: page fault: pc 8000038c, usp efffa010, addr efff9ffc
> [ 2958.990000] bus_error030: page fault: pc 8000038c, usp efff9008, addr efff8ffc
> [ 2959.010000] bus_error030: page fault: pc 8000038c, usp efff8000, addr efff7ffc
> [ 2959.030000] bus_error030: page fault: pc 8000038c, usp efff6014, addr efff5ffc
> [ 2959.050000] bus_error030: page fault: pc 8000038c, usp efff500c, addr efff4ffc
> [ 2959.060000] bus_error030: page fault: pc 8000038c, usp efff4004, addr efff3ffc
> [ 2959.080000] bus_error030: page fault: pc 8000038c, usp efff2ffc, addr efff2ffc
> [ 2959.100000] bus_error030: page fault: pc 8000038c, usp efff1010, addr efff0ffc
> [ 2959.110000] bus_error030: page fault: pc 8000038c, usp efff0008, addr effefffc
> [ 2959.130000] bus_error030: page fault: pc 8000038c, usp effef000, addr effeeffc
> done.
> starting recursion with fork
> [Detaching after fork from child process 166]
> [ 2959.310000] bus_error030: page fault: pc 8000038c, usp effff014, addr efffeffc
> [ 2959.350000] handle_signal: fmtvec b008, pc 8000038c, usp efffee88
>
>
> With a breakpoint set in the userspace signal handler, we can examine
> stack memory from the USP as it presently is, going back up to USP as it
> was at the time of the page fault (both addresses are given in printk
> output above).
>
> The difference is 396 bytes, and pahole confirms that sizeof(struct
> sigframe) + sizeof(struct frame.un.fmtb) == 312 + 84 == 396.
>
> (Note that this kernel was patched so that it doesn't truncate the result
> of that pointer arithmetic. Hence there's no gap between the bottom of the
> user stack and the start of the signal frame, which tends to confuse the
> issue, as the gap would normally come and go unpredictably.)
>
> Breakpoint 1, 0x800004a4 in handler ()
> (gdb) x/110z 0xeffff014 - 396
> 0xefffee88:     0xefffee98      0x00000011      0x00000008      0xefffeea4
> 0xefffee98:     0x70774e40      0xf1f2f3f4      0x00000000      0x00000000
> 0xefffeea8:     0xeffff014      0x000007a7      0x000007a6      0x800744ec
> 0xefffeeb8:     0x8006d2b0      0x00008000      0x038cb008      0x7fff0000
> 0xefffeec8:     0xffffffff      0xffffffff      0x7fff0000      0xffffffff
> 0xefffeed8:     0xffffffff      0x00000000      0x00000000      0x00000000
> 0xefffeee8:     0x1f386980      0xf0fff0ff      0x0f800000      0xf0ff0000
> 0xefffeef8:     0x00000000      0x408ece9a      0x00000000      0x00000000
> 0xefffef08:     0x00000000      0x00000000      0x3ffe0000      0x00000000
> 0xefffef18:     0x00000000      0xffffffff      0x7c0effff      0x014ee530
> 0xefffef28:     0x08010000      0x00000004      0x00994000      0x00995e94
> 0xefffef38:     0x014ed540      0x00994000      0x0040ee24      0x08010000
> 0xefffef48:     0x00000004      0x004fa794      0x00995f18      0x00995e34
> 0xefffef58:     0x00079a72      0x004fa794      0x000270ce      0x00995e54
> 0xefffef68:     0x00027370      0x00000001      0x014f1e60      0x00995f2c
> 0xefffef78:     0x00994000      0x00000011      0x00995f18      0x00995e98
> 0xefffef88:     0x00027d08      0x00000011      0x00000004      0x00000000
> 0xefffef98:     0x00995f2c      0xd1d2d3d4      0xe1e2e3e4      0xf1f2f3f4
> 0xefffefa8:     0xeffffed0      0x00000001      0x00000001      0x00995f78
> 0xefffefb8:     0xb1b2b3b4      0xc1c2c3c4      0x0eee0709      0x24798004
> 0xefffefc8:     0xefffeffc      0xefffeff8      0xd1d2d3d4      0x48e7383c
> 0xefffefd8:     0x80000394      0x80000392      0x80000390      0x000007a7
> 0xefffefe8:     0x383cff0d      0x000ff153      0x383c2479      0x8007b4e0
> 0xefffeff8:     0x0000383c      0x00000000      0x80100000      0x800003f8
> 0xeffff008:     0x000000e0      0x014ed500      0xeffffcc4      0xc1c2c3c4
> 0xeffff018:     0xeffff03c      0x800003f8      0xd1d2d3d4      0xe1e2e3e4
> 0xeffff028:     0xf1f2f3f4      0x91929394      0xa1a2a3a4      0xb1b2b3b4
> 0xeffff038:     0xc1c2c3c4      0xeffff060
>
>
> Here's the same hex dump with struct member names added, much like the
> output from your test program:
>
> efffee88:	efffee98	sigframe.pretcode
> efffee8c:	00000011	sigframe.sig
> efffee90:	00000008	sigframe.code
> efffee94:	efffeea4	sigframe.psc
> efffee98:	70774e40	sigframe.retcode
> efffee9c:	f1f2f3f4	sigframe.retcode
> efffeea0:	00000000	sigframe.extramask
> efffeea4:	00000000	sigframe.sc.mask
> efffeea8:	effff014	sigframe.sc.sc_usp
> efffeeac:	000007a7	sigframe.sc.sc_d0
> efffeeb0:	000007a6	sigframe.sc.sc_d1
> efffeeb4:	800744ec	sigframe.sc.sc_a0
> efffeeb8:	8006d2b0	sigframe.sc.sc_a1
> efffeebc:	00008000	sigframe.sc.sc_sr, sc_pc
> efffeec0:	038cb008	sigframe.sc.sc_pc, sc_formatvec
> efffeec4:	7fff0000	sigframe.sc.sc_fpregs
> efffeec8:	ffffffff	sigframe.sc.sc_fpregs
> efffeecc:	ffffffff	sigframe.sc.sc_fpregs
> efffeed0:	7fff0000	sigframe.sc.sc_fpregs
> efffeed4:	ffffffff	sigframe.sc.sc_fpregs
> efffeed8:	ffffffff	sigframe.sc.sc_fpregs
> efffeedc:	00000000	sigframe.sc.sc_fpcntl
> efffeee0:	00000000	sigframe.sc.sc_fpcntl
> efffeee4:	00000000	sigframe.sc.sc_fpcntl
> efffeee8:	1f386980	sigframe.sc.sc_fpstate
> efffeeec:	f0fff0ff	sigframe.sc.sc_fpstate
> efffeef0:	0f800000	sigframe.sc.sc_fpstate
> efffeef4:	f0ff0000	sigframe.sc.sc_fpstate
> efffeef8:	00000000	sigframe.sc.sc_fpstate
> efffeefc:	408ece9a	sigframe.sc.sc_fpstate
> efffef00:	00000000	sigframe.sc.sc_fpstate
> efffef04:	00000000	sigframe.sc.sc_fpstate
> efffef08:	00000000	sigframe.sc.sc_fpstate
> efffef0c:	00000000	sigframe.sc.sc_fpstate
> efffef10:	3ffe0000	sigframe.sc.sc_fpstate
> efffef14:	00000000	sigframe.sc.sc_fpstate
> efffef18:	00000000	sigframe.sc.sc_fpstate
> efffef1c:	ffffffff	sigframe.sc.sc_fpstate
> efffef20:	7c0effff	sigframe.sc.sc_fpstate
> efffef24:	014ee530	sigframe.sc.sc_fpstate
> efffef28:	08010000	sigframe.sc.sc_fpstate
> efffef2c:	00000004	sigframe.sc.sc_fpstate
> efffef30:	00994000	sigframe.sc.sc_fpstate
> efffef34:	00995e94	sigframe.sc.sc_fpstate
> efffef38:	014ed540	sigframe.sc.sc_fpstate
> efffef3c:	00994000	sigframe.sc.sc_fpstate
> efffef40:	0040ee24	sigframe.sc.sc_fpstate
> efffef44:	08010000	sigframe.sc.sc_fpstate
> efffef48:	00000004	sigframe.sc.sc_fpstate
> efffef4c:	004fa794	sigframe.sc.sc_fpstate
> efffef50:	00995f18	sigframe.sc.sc_fpstate
> efffef54:	00995e34	sigframe.sc.sc_fpstate
> efffef58:	00079a72	sigframe.sc.sc_fpstate
> efffef5c:	004fa794	sigframe.sc.sc_fpstate
> efffef60:	000270ce	sigframe.sc.sc_fpstate
> efffef64:	00995e54	sigframe.sc.sc_fpstate
> efffef68:	00027370	sigframe.sc.sc_fpstate
> efffef6c:	00000001	sigframe.sc.sc_fpstate
> efffef70:	014f1e60	sigframe.sc.sc_fpstate
> efffef74:	00995f2c	sigframe.sc.sc_fpstate
> efffef78:	00994000	sigframe.sc.sc_fpstate
> efffef7c:	00000011	sigframe.sc.sc_fpstate
> efffef80:	00995f18	sigframe.sc.sc_fpstate
> efffef84:	00995e98	sigframe.sc.sc_fpstate
> efffef88:	00027d08	sigframe.sc.sc_fpstate
> efffef8c:	00000011	sigframe.sc.sc_fpstate
> efffef90:	00000004	sigframe.sc.sc_fpstate
> efffef94:	00000000	sigframe.sc.sc_fpstate
> efffef98:	00995f2c	sigframe.sc.sc_fpstate
> efffef9c:	d1d2d3d4	sigframe.sc.sc_fpstate
> efffefa0:	e1e2e3e4	sigframe.sc.sc_fpstate
> efffefa4:	f1f2f3f4	sigframe.sc.sc_fpstate
> efffefa8:	effffed0	sigframe.sc.sc_fpstate
> efffefac:	00000001	sigframe.sc.sc_fpstate
> efffefb0:	00000001	sigframe.sc.sc_fpstate
> efffefb4:	00995f78	sigframe.sc.sc_fpstate
> efffefb8:	b1b2b3b4	sigframe.sc.sc_fpstate
> efffefbc:	c1c2c3c4	sigframe.sc.sc_fpstate
> efffefc0:	0eee0709	fmtb.int1, ssw
> efffefc4:	24798004	fmtb.isc, isb
> efffefc8:	efffeffc	fmtb.daddr
> efffefcc:	efffeff8	fmtb.int2
> efffefd0:	d1d2d3d4	fmtb.dobuf
> efffefd4:	48e7383c	fmtb.int3
> efffefd8:	80000394	fmtb.int3
> efffefdc:	80000392	fmtb.baddr
> efffefe0:	80000390	fmtb.int4
> efffefe4:	000007a7	fmtb.dibuf
> efffefe8:	383cff0d	fmtb.int5
> efffefec:	000ff153	fmtb.int5, ver, int6
> efffeff0:	383c2479	fmtb.int7
> efffeff4:	8007b4e0	fmtb.int7
> efffeff8:	0000383c	fmtb.int7
> efffeffc:	00000000	fmtb.int7
> effff000: *	80100000	fmtb.int7
> effff004: *	800003f8	fmtb.int7
> effff008: *	000000e0	fmtb.int7
> effff00c: *	014ed500	fmtb.int7
> effff010: *	effffcc4	fmtb.int7
> effff014:	c1c2c3c4	user stack
> effff018:	effff03c	user stack
> effff01c:	800003f8	user stack
> effff020:	d1d2d3d4	user stack
> effff024:	e1e2e3e4	user stack
> effff028:	f1f2f3f4	user stack
> effff02c:	91929394	user stack
> effff030:	a1a2a3a4	user stack
> effff034:	b1b2b3b4	user stack
> effff038:	c1c2c3c4	user stack
> effff03c:	effff060	user stack
>
> The portion marked with an * is the part that damages the user stack, as
> these locations are all higher than USP. This just confirms by other means
> what your test program already demonstrated.

Good - as I said, I ought to have learned gdb properly when I was young ...

>
> (gdb) c
> Continuing.
> [ 3334.340000] bus_error030: page fault: pc 8000038c, usp efffe00c, addr efffdffc
> [ 3334.360000] bus_error030: page fault: pc 8000038c, usp efffd004, addr efffcffc
> [ 3334.380000] bus_error030: page fault: pc 8000038c, usp efffbffc, addr efffbffc
> [ 3334.390000] bus_error030: page fault: pc 8000038c, usp efffa010, addr efff9ffc
> [ 3334.410000] bus_error030: page fault: pc 8000038c, usp efff9008, addr efff8ffc
> [ 3334.430000] bus_error030: page fault: pc 8000038c, usp efff8000, addr efff7ffc
> [ 3334.440000] bus_error030: page fault: pc 8000038c, usp efff6014, addr efff5ffc
> [ 3334.450000] bus_error030: page fault: pc 8000038c, usp efff500c, addr efff4ffc
> [ 3334.460000] bus_error030: page fault: pc 8000038c, usp efff4004, addr efff3ffc
> [ 3334.480000] bus_error030: page fault: pc 8000038c, usp efff2ffc, addr efff2ffc
> [ 3334.500000] bus_error030: page fault: pc 8000038c, usp efff1010, addr efff0ffc
> [ 3334.510000] bus_error030: page fault: pc 8000038c, usp efff0008, addr effefffc
> [ 3334.530000] bus_error030: page fault: pc 8000038c, usp effef000, addr effeeffc
> [Detaching after fork from child process 167]
>
> Breakpoint 1, 0x800004a4 in handler ()
> (gdb) c
> Continuing.
>
> Program received signal SIGILL, Illegal instruction.
> 0x80000492 in rec ()
>
>
> My test program terminates having detected the corruption visible at
> 0xeffff000 but gdb allows us to examine that memory afterwards:
>
> (gdb) x/110z 0xeffff014 - 396
> 0xefffee88:     0xc1c2c3c4      0xefffeeb0      0x800003f8      0xd1d2d3d4
> 0xefffee98:     0xe1e2e3e4      0xf1f2f3f4      0x91929394      0xa1a2a3a4
> 0xefffeea8:     0xb1b2b3b4      0xc1c2c3c4      0xefffeed4      0x800003f8
> 0xefffeeb8:     0xd1d2d3d4      0xe1e2e3e4      0xf1f2f3f4      0x91929394
> 0xefffeec8:     0xa1a2a3a4      0xb1b2b3b4      0xc1c2c3c4      0xefffeef8
> 0xefffeed8:     0x800003f8      0xd1d2d3d4      0xe1e2e3e4      0xf1f2f3f4
> 0xefffeee8:     0x91929394      0xa1a2a3a4      0xb1b2b3b4      0xc1c2c3c4
> 0xefffeef8:     0xefffef1c      0x800003f8      0xd1d2d3d4      0xe1e2e3e4
> 0xefffef08:     0xf1f2f3f4      0x91929394      0xa1a2a3a4      0xb1b2b3b4
> 0xefffef18:     0xc1c2c3c4      0xefffef40      0x800003f8      0xd1d2d3d4
> 0xefffef28:     0xe1e2e3e4      0xf1f2f3f4      0x91929394      0xa1a2a3a4
> 0xefffef38:     0xb1b2b3b4      0xc1c2c3c4      0xefffef64      0x800003f8
> 0xefffef48:     0xd1d2d3d4      0xe1e2e3e4      0xf1f2f3f4      0x91929394
> 0xefffef58:     0xa1a2a3a4      0xb1b2b3b4      0xc1c2c3c4      0xefffef88
> 0xefffef68:     0x800003f8      0xd1d2d3d4      0xe1e2e3e4      0xf1f2f3f4
> 0xefffef78:     0x91929394      0xa1a2a3a4      0xb1b2b3b4      0xc1c2c3c4
> 0xefffef88:     0xefffefac      0x800003f8      0xd1d2d3d4      0xe1e2e3e4
> 0xefffef98:     0xf1f2f3f4      0x91929394      0xa1a2a3a4      0xb1b2b3b4
> 0xefffefa8:     0xc1c2c3c4      0xefffefd0      0x800003f8      0xd1d2d3d4
> 0xefffefb8:     0xe1e2e3e4      0xf1f2f3f4      0x91929394      0xa1a2a3a4
> 0xefffefc8:     0xb1b2b3b4      0xc1c2c3c4      0xefffeff4      0x800003f8
> 0xefffefd8:     0xd1d2d3d4      0xe1e2e3e4      0xf1f2f3f4      0x91929394
> 0xefffefe8:     0xa1a2a3a4      0xb1b2b3b4      0xc1c2c3c4      0xeffff018
> 0xefffeff8:     0x800003f8      0xd1d2d3d4      0x80100000      0x800003f8
> 0xeffff008:     0x000000e0      0x014ed500      0xeffffcc4      0xc1c2c3c4
> 0xeffff018:     0xeffff03c      0x800003f8      0xd1d2d3d4      0xe1e2e3e4
> 0xeffff028:     0xf1f2f3f4      0x91929394      0xa1a2a3a4      0xb1b2b3b4
> 0xeffff038:     0xc1c2c3c4      0xeffff060
> (gdb)
>
>
> So the page that caused the fault never shows any corruption. It turns out
> that the MOVEM resumes correctly after the signal is handled, which
> overwrites the stack memory used for the signal frame, starting at the top
> of the new stack page. I had incorrectly assumed the MOVEM had gone awry
> but it actually resumed just fine.

Yes, it's always the page where the movem started on that gets 
clobbered, because we use usp to figure our where our signal stack may 
end. fmtb.int2 or fmtb.daddr is the correct end location in this case. 
But we need to find something that works in the general case (and then 
analyze the performance impact it might have in stack and signal heavy 
applications - I might have mentioned that before, but your equivalent 
to Andreas' patch seemed quite a bit slower in the test case than when 
signals were allowed after format b bus faults. Interrupt latency, most 
likely).

Cheers,

	Michael

Re: signal delivery, was Re: reliable reproducer

[Finn Thain]

On Sat, 29 Apr 2023, Michael Schmitz wrote:

> Am 29.04.2023 um 12:28 schrieb Finn Thain:
> >
> > Right. If we fix this in the signal handling code, we take care of 
> > address errors as well, which was my concern with Andreas' patch. We 
> > can do what do_page_fault() does and assume the worst (256 bytes?).
> 
> Well, we could do that if we could be certain this does not cause a 
> memory leak in some way. The reason I bring this up is that I've just 
> seen the kernel that I'd used to run the latest test cases (which 
> inserts a 20 byte gap only!) run amok terminating pretty much my entire 
> user space because it ran out of memory. Never seen the like of that.
> 

If the test program ran out of stack space it would not trigger the OOM 
killer. So that incident probably has something to do with upgrading your 
kernel (?)

Anyway, I agree that stkadj would need to account for the gap, as you 
pointed out earlier.

> > I believe we can use USP to get a worst case estimate for the future 
> > extent of the user stack. ...
> 
> What is the most data a moveml <...>,sp@- can take? If that's not too 
> much, a constant offset for the signal stack in case of format b frames 
> on 020/030 might be easiest.
> 

I think it's 64 bytes (16 registers). But we also have to consider all of 
the other instructions that may write to the stack. There's probably a 
reason why do_page_fault() picked a 256 byte gap (?)

> But we need to find something that works in the general case (and then 
> analyze the performance impact it might have in stack and signal heavy 
> applications - I might have mentioned that before, but your equivalent 
> to Andreas' patch seemed quite a bit slower in the test case than when 
> signals were allowed after format b bus faults. Interrupt latency, most 
> likely).
> 

The alternative is to use more stack memory, which means marginally more 
paging. Choose your poison...

Re: signal delivery, was Re: reliable reproducer

[Michael Schmitz]

Hi Finn,

Am 29.04.2023 um 17:03 schrieb Finn Thain:
> On Sat, 29 Apr 2023, Michael Schmitz wrote:
>
>> Am 29.04.2023 um 12:28 schrieb Finn Thain:
>>>
>>> Right. If we fix this in the signal handling code, we take care of
>>> address errors as well, which was my concern with Andreas' patch. We
>>> can do what do_page_fault() does and assume the worst (256 bytes?).
>>
>> Well, we could do that if we could be certain this does not cause a
>> memory leak in some way. The reason I bring this up is that I've just
>> seen the kernel that I'd used to run the latest test cases (which
>> inserts a 20 byte gap only!) run amok terminating pretty much my entire
>> user space because it ran out of memory. Never seen the like of that.
>>
>
> If the test program ran out of stack space it would not trigger the OOM
> killer. So that incident probably has something to do with upgrading your
> kernel (?)

Might be, but this has all been on m68k v6.3-rc7, and I hadn't seen the 
memory squeeze before there. I'll have to run a few hundred of the test 
case on an unpatched v6.3-rc7 and on the one with the minimal frame gap 
to be sure though.

> Anyway, I agree that stkadj would need to account for the gap, as you
> pointed out earlier.

Not sure about that anymore - mangle_kernel_stack() does not even use 
stkadj to shift contents on the kernel stack (after restoring the 
exception frame from the signal stack, but it uses the start address of 
the frame for that copy operation, and uses a local buffer to move it 
from user space to kernel space). It uses the extra frame size from the 
exception frame directly.

stkadj is the offset of the replacement exception frame on the kernel 
stack. The replacement frame gets us into the user space signal handler 
instead of completing the exception right away. stkadj is used to skip 
that replacement exception frame used for the signal handler on the 
final rte (after a trip through sys_sigreturn to copy the original 
exception frame back on the kernel stack).

The offset we use for he signal stack on the user stack does not matter 
here at all.

Or so my limited understanding...

>
>>> I believe we can use USP to get a worst case estimate for the future
>>> extent of the user stack. ...
>>
>> What is the most data a moveml <...>,sp@- can take? If that's not too
>> much, a constant offset for the signal stack in case of format b frames
>> on 020/030 might be easiest.
>>
>
> I think it's 64 bytes (16 registers). But we also have to consider all of
> the other instructions that may write to the stack. There's probably a
> reason why do_page_fault() picked a 256 byte gap (?)

That's not used as a gap, just to catch any user access below the user 
stack pointer.

>
>> But we need to find something that works in the general case (and then
>> analyze the performance impact it might have in stack and signal heavy
>> applications - I might have mentioned that before, but your equivalent
>> to Andreas' patch seemed quite a bit slower in the test case than when
>> signals were allowed after format b bus faults. Interrupt latency, most
>> likely).
>>
>
> The alternative is to use more stack memory, which means marginally more
> paging. Choose your poison...

Yes - I'll have to run a few benchmarks to see which I'd prefer.

In the meantime, I'll send what I have at present as RFC.

Cheers,

	Michael

Re: signal delivery, was Re: reliable reproducer

[Finn Thain]

On Sat, 29 Apr 2023, Michael Schmitz wrote:

> 
> Might be, but this has all been on m68k v6.3-rc7, and I hadn't seen the 
> memory squeeze before there. I'll have to run a few hundred of the test 
> case on an unpatched v6.3-rc7 and on the one with the minimal frame gap 
> to be sure though.
> 

You could use a stable kernel for this.

> > Anyway, I agree that stkadj would need to account for the gap, as you 
> > pointed out earlier.
> 
> Not sure about that anymore - mangle_kernel_stack() does not even use 
> stkadj to shift contents on the kernel stack (after restoring the 
> exception frame from the signal stack, but it uses the start address of 
> the frame for that copy operation, and uses a local buffer to move it 
> from user space to kernel space). It uses the extra frame size from the 
> exception frame directly.
> 
> stkadj is the offset of the replacement exception frame on the kernel 
> stack. The replacement frame gets us into the user space signal handler 
> instead of completing the exception right away. stkadj is used to skip 
> that replacement exception frame used for the signal handler on the 
> final rte (after a trip through sys_sigreturn to copy the original 
> exception frame back on the kernel stack).
> 
> The offset we use for he signal stack on the user stack does not matter 
> here at all.
> 
> Or so my limited understanding...

Thanks for passing it on. I sure wish this stuff was documented somewhere.

> 
> >
> >>> I believe we can use USP to get a worst case estimate for the future 
> >>> extent of the user stack. ...
> >>
> >> What is the most data a moveml <...>,sp@- can take? If that's not too 
> >> much, a constant offset for the signal stack in case of format b 
> >> frames on 020/030 might be easiest.
> >
> > I think it's 64 bytes (16 registers). But we also have to consider all 
> > of the other instructions that may write to the stack. There's 
> > probably a reason why do_page_fault() picked a 256 byte gap (?)
> 
> That's not used as a gap, just to catch any user access below the user 
> stack pointer.
> 

MOVEM _is_ accessing the stack below the stack pointer. That's why the 256 
is relevant here.

Re: signal delivery, was Re: reliable reproducer

[Michael Schmitz]

Hi Finn,

Am 29.04.2023 um 18:13 schrieb Finn Thain:
> On Sat, 29 Apr 2023, Michael Schmitz wrote:
>
>>
>> Might be, but this has all been on m68k v6.3-rc7, and I hadn't seen the
>> memory squeeze before there. I'll have to run a few hundred of the test
>> case on an unpatched v6.3-rc7 and on the one with the minimal frame gap
>> to be sure though.
>>
>
> You could use a stable kernel for this.
>
>>> Anyway, I agree that stkadj would need to account for the gap, as you
>>> pointed out earlier.
>>
>> Not sure about that anymore - mangle_kernel_stack() does not even use
>> stkadj to shift contents on the kernel stack (after restoring the
>> exception frame from the signal stack, but it uses the start address of
>> the frame for that copy operation, and uses a local buffer to move it
>> from user space to kernel space). It uses the extra frame size from the
>> exception frame directly.
>>
>> stkadj is the offset of the replacement exception frame on the kernel
>> stack. The replacement frame gets us into the user space signal handler
>> instead of completing the exception right away. stkadj is used to skip
>> that replacement exception frame used for the signal handler on the
>> final rte (after a trip through sys_sigreturn to copy the original
>> exception frame back on the kernel stack).
>>
>> The offset we use for he signal stack on the user stack does not matter
>> here at all.
>>
>> Or so my limited understanding...
>
> Thanks for passing it on. I sure wish this stuff was documented somewhere.

I hope this one works:

https://lore.kernel.org/r/YP2c1xk9LJ0zE3KW@zeniv-ca.linux.org.uk

Al Viro's patch series fixing our signal code - as I recall, he 
explained the principles behind the code in the discussion on that series.

>
>>
>>>
>>>>> I believe we can use USP to get a worst case estimate for the future
>>>>> extent of the user stack. ...
>>>>
>>>> What is the most data a moveml <...>,sp@- can take? If that's not too
>>>> much, a constant offset for the signal stack in case of format b
>>>> frames on 020/030 might be easiest.
>>>
>>> I think it's 64 bytes (16 registers). But we also have to consider all
>>> of the other instructions that may write to the stack. There's
>>> probably a reason why do_page_fault() picked a 256 byte gap (?)
>>
>> That's not used as a gap, just to catch any user access below the user
>> stack pointer.
>>
>
> MOVEM _is_ accessing the stack below the stack pointer. That's why the 256
> is relevant here.

Indirectly, as far as I can see (we should not expect do_page_fault() to 
fix up a mapping if we try to access an unmapped page that's 256 bytes 
below USP). Might need to add that constraint to my patch.

But it's not reserving memory below USP, or shifting something we want 
placed just below USP a little further down.

I'm making not much sense here, I fear ...

Cheers,

	Michael

Re: signal delivery, was Re: reliable reproducer

[Finn Thain]

On Sat, 29 Apr 2023, Finn Thain wrote:

> 
> Anyway, I agree that stkadj would need to account for the gap, as you 
> pointed out earlier.
> 

OTOH, there is no allowance made for the gap already created here due to 
masking off the least significant bits of the address:

static inline void __user *
get_sigframe(struct ksignal *ksig, size_t frame_size)
{
        unsigned long usp = sigsp(rdusp(), ksig);

        return (void __user *)((usp - frame_size) & -8UL);
}

Is that a bug? Shouldn't that gap be added to stkadj?

Re: signal delivery, was Re: reliable reproducer

[Andreas Schwab]

On Apr 29 2023, Finn Thain wrote:

> There's probably a reason why do_page_fault() picked a 256 byte gap

The biggest size of an operand a (coprocessor) instruction can transfer
is 255 bytes.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: signal delivery, was Re: reliable reproducer

[Michael Schmitz]

Hi Andreas,

Am 29.04.2023 um 19:11 schrieb Andreas Schwab:
> On Apr 29 2023, Finn Thain wrote:
>
>> There's probably a reason why do_page_fault() picked a 256 byte gap
>
> The biggest size of an operand a (coprocessor) instruction can transfer
> is 255 bytes.

Thanks, that makes more sense now.

Cheers,

	Michael

Re: signal delivery, was Re: reliable reproducer

[Eero Tamminen]

Hi,

On 25.4.2023 4.55, Finn Thain wrote:
> On Tue, 25 Apr 2023, Finn Thain wrote:
...
>> I wonder if we are seeing some fallout from the issue described in
>> do_page_fault() i.e. usp is unreliable.
>>
>>                  /* Accessing the stack below usp is always a bug.  The
>>                     "+ 256" is there due to some instructions doing
>>                     pre-decrement on the stack and that doesn't show up
>>                     until later.  */
>>                  if (address + 256 < rdusp())
>>                          goto map_err;
>>
>> Maybe we should try modifying get_sigframe() to increase the gap between
>> the signal and exception frames from 0-1 long words up to 64-65 long
>> words.
> 
> It turns out that doing so (patch below) does make the problem go away.
> Was the exception frame getting clobbered?
> 
> diff --git a/arch/m68k/kernel/signal.c b/arch/m68k/kernel/signal.c
> index b9f6908a31bc..94104699f5a8 100644
> --- a/arch/m68k/kernel/signal.c
> +++ b/arch/m68k/kernel/signal.c
> @@ -862,7 +862,7 @@ get_sigframe(struct ksignal *ksig, size_t frame_size)
>   {
>   	unsigned long usp = sigsp(rdusp(), ksig);
>   
> -	return (void __user *)((usp - frame_size) & -8UL);
> +	return (void __user *)((usp - 256 - frame_size) & -8UL);
>   }
>   
>   static int setup_frame(struct ksignal *ksig, sigset_t *set,

While this is most likely Hatari emulation [1] issue, it has some of the 
same triggering conditions, so I thought to mention it...

Above patch does not fix kernel panic I'm seen on booting Linux under 
Hatari emulated Atari Falcon, to a small IDE root fs with just (old 
Debian) Busybox and a shell script acting as init:
----------------------------------------
Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
CPU: 0 PID: 1 Comm: sh Not tainted 6.2.0hatari-00006-g01793428cbc5-dirty #4
Stack from 00819dc8:
         00819dc8 0032b646 0032b646 00027800 00000001 00819de8 00292878 
0032b646
         00819e0c 0028d60e 00027848 0000000b 0081caff 000300d6 00815a90 
00000000
         00819f2c 00819e50 00028878 003243fd 0000000b 0000000b 00818007 
0081ca30
         000300d6 0081caf8 00000001 00819f18 0081bbd0 00819f2c 00000000 
00000000
         00000000 00000000 00819e60 00028ea0 0000000b 0000000b 00819e98 
000303c8
         0000000b 00818000 00000002 00000000 00000000 00000000 8017705e 
00819f78
Call Trace: [<00027800>] set_cpu_online+0x1c/0x3e
  [<00292878>] dump_stack+0x10/0x16
  [<0028d60e>] panic+0xc4/0x22a
  [<00027848>] arch_local_irq_enable+0x0/0x22
  [<000300d6>] do_signal_stop+0x0/0x152
  [<00028878>] do_exit+0x138/0x642
  [<000300d6>] do_signal_stop+0x0/0x152
  [<00028ea0>] do_group_exit+0x22/0x62
  [<000303c8>] get_signal+0xf8/0x4ba
  [<00003508>] test_ti_thread_flag+0x0/0x1a
  [<00003f4a>] do_notify_resume+0x36/0x488
  [<00005706>] send_fault_sig+0x28/0x8c
  [<00005888>] do_page_fault+0x11e/0x242
  [<00005814>] do_page_fault+0xaa/0x242
  [<00002814>] do_signal_return+0x10/0x1a
  [<00020007>] _I_CALL_TOP+0xd83/0x1900
  [<0000b280>] sp_over+0x2c/0x3c
  [<00007201>] atari_irq_enable+0x3/0x2a
  [<000066f6>] atari_get_hardware_list+0x33a/0x3e8
----------------------------------------

(Only way to get rid of the panic is disabling both CPU cache and 
prefetch emulation.)


Is it possible that in your case there's also IRQ (exception) happening 
at the same time with page fault and signal?


	- Eero

[1] 030 MMU vs. cache/prefetch vs. exception handling vs. IDE emulation.

Re: signal delivery, was Re: reliable reproducer

[Andreas Schwab]

On Apr 25 2023, Finn Thain wrote:

> It turns out that doing so (patch below) does make the problem go away. 
> Was the exception frame getting clobbered?
>
> diff --git a/arch/m68k/kernel/signal.c b/arch/m68k/kernel/signal.c
> index b9f6908a31bc..94104699f5a8 100644
> --- a/arch/m68k/kernel/signal.c
> +++ b/arch/m68k/kernel/signal.c
> @@ -862,7 +862,7 @@ get_sigframe(struct ksignal *ksig, size_t frame_size)
>  {
>  	unsigned long usp = sigsp(rdusp(), ksig);
>  
> -	return (void __user *)((usp - frame_size) & -8UL);
> +	return (void __user *)((usp - 256 - frame_size) & -8UL);

Probably the issue is that a bus error exception should never start
signal delivery when returning to user space.  On the 030 returning from
a bus error resumes the execution of the faulting insn (unlike the
040/060 which restart it), and the saved USP may have the original value
from before the insn started (modified registers may not be updated
until the insn is complete or just before the final bus cycle).  Signal
delivery should only ever happen at insn boundaries.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: signal delivery, was Re: reliable reproducer

[Michael Schmitz]

Hi Andreas,

On 25/04/23 23:25, Andreas Schwab wrote:
> On Apr 25 2023, Finn Thain wrote:
>
>> It turns out that doing so (patch below) does make the problem go away.
>> Was the exception frame getting clobbered?
>>
>> diff --git a/arch/m68k/kernel/signal.c b/arch/m68k/kernel/signal.c
>> index b9f6908a31bc..94104699f5a8 100644
>> --- a/arch/m68k/kernel/signal.c
>> +++ b/arch/m68k/kernel/signal.c
>> @@ -862,7 +862,7 @@ get_sigframe(struct ksignal *ksig, size_t frame_size)
>>   {
>>   	unsigned long usp = sigsp(rdusp(), ksig);
>>   
>> -	return (void __user *)((usp - frame_size) & -8UL);
>> +	return (void __user *)((usp - 256 - frame_size) & -8UL);
> Probably the issue is that a bus error exception should never start
> signal delivery when returning to user space.  On the 030 returning from
> a bus error resumes the execution of the faulting insn (unlike the
> 040/060 which restart it), and the saved USP may have the original value
> from before the insn started (modified registers may not be updated
> until the insn is complete or just before the final bus cycle).  Signal
> delivery should only ever happen at insn boundaries.

Thanks - we had seen evidence that a bus error generated mid-instruction 
did leave the USP at the address where the bus fault happened (not 
before the instruction started, neither what it would have been once the 
instruction completed), and the operation did not complete normally 
after the bus error (at least the value/address seen in the exception 
frame not stored). Finn had also demonstrated that skipping signal 
delivery on bus errors abolishes the stack corruption.  Your patch 
achieves the same objective in a different way, so I'm sure this will 
work as well.

I had thought the 030 could resume the interrupted instruction using the 
information from the exception frame - and that does appear to work in 
all other cases except where signal delivery gets in the way, and it 
also works if moving the exception frame a little bit further down the 
stack. So our treatment of the bus error exception frame during signal 
delivery appears to be incorrect. Wouldn't you agree?

Cheers,

     Michael

Re: signal delivery, was Re: reliable reproducer

[Michael Schmitz]

Hi Andreas,

On 26/04/23 07:46, Michael Schmitz wrote:
> Hi Andreas,
>
> On 25/04/23 23:25, Andreas Schwab wrote:
>> On Apr 25 2023, Finn Thain wrote:
>>
>>> It turns out that doing so (patch below) does make the problem go away.
>>> Was the exception frame getting clobbered?
>>>
>>> diff --git a/arch/m68k/kernel/signal.c b/arch/m68k/kernel/signal.c
>>> index b9f6908a31bc..94104699f5a8 100644
>>> --- a/arch/m68k/kernel/signal.c
>>> +++ b/arch/m68k/kernel/signal.c
>>> @@ -862,7 +862,7 @@ get_sigframe(struct ksignal *ksig, size_t 
>>> frame_size)
>>>   {
>>>       unsigned long usp = sigsp(rdusp(), ksig);
>>>   -    return (void __user *)((usp - frame_size) & -8UL);
>>> +    return (void __user *)((usp - 256 - frame_size) & -8UL);
>> Probably the issue is that a bus error exception should never start
>> signal delivery when returning to user space.  On the 030 returning from
>> a bus error resumes the execution of the faulting insn (unlike the
>> 040/060 which restart it), and the saved USP may have the original value
>> from before the insn started (modified registers may not be updated
>> until the insn is complete or just before the final bus cycle). Signal
>> delivery should only ever happen at insn boundaries.
>
> Thanks - we had seen evidence that a bus error generated 
> mid-instruction did leave the USP at the address where the bus fault 
> happened (not before the instruction started, neither what it would 
> have been once the instruction completed), and the operation did not 
> complete normally after the bus error (at least the value/address seen 
> in the exception frame not stored). Finn had also demonstrated that 
> skipping signal delivery on bus errors abolishes the stack 
> corruption.  Your patch achieves the same objective in a different 
> way, so I'm sure this will work as well.
>
> I had thought the 030 could resume the interrupted instruction using 
> the information from the exception frame - and that does appear to 
> work in all other cases except where signal delivery gets in the way, 
> and it also works if moving the exception frame a little bit further 
> down the stack. So our treatment of the bus error exception frame 
> during signal delivery appears to be incorrect. Wouldn't you agree?

Inspection of the format b frame placed in the signal frame in both rt 
and non-rt cases (at the time the signal handler runs) shows the 
expected contents in the data output buffer, data fault address and ssw. 
At that time, returning to user space with rte would correctly resume 
the instruction execution.  I had previously confirmed that the register 
contents saved in the rt signal frame is correct also.

That is with a kernel patched similar to above patch by Finn (using an 
offset of 128 or 64 instead of  256).

Cheers,

     Michael

Re: signal delivery, was Re: reliable reproducer

[Finn Thain]

On Wed, 26 Apr 2023, Michael Schmitz wrote:

> On 26/04/23 07:46, Michael Schmitz wrote:
> >
> > I had thought the 030 could resume the interrupted instruction using 
> > the information from the exception frame - and that does appear to 
> > work in all other cases except where signal delivery gets in the way, 
> > and it also works if moving the exception frame a little bit further 
> > down the stack. So our treatment of the bus error exception frame 
> > during signal delivery appears to be incorrect. Wouldn't you agree?
> 
> Inspection of the format b frame placed in the signal frame in both rt 
> and non-rt cases (at the time the signal handler runs) shows the 
> expected contents in the data output buffer, data fault address and ssw. 
> At that time, returning to user space with rte would correctly resume 
> the instruction execution.  I had previously confirmed that the register 
> contents saved in the rt signal frame is correct also.
> 
> That is with a kernel patched similar to above patch by Finn (using an 
> offset of 128 or 64 instead of 256).

That means things go awry during sys_sigreturn or sys_rt_sigreturn. I'm 
not sure what happens to the exception frame:

1:
        | stack contents now:
        |   [original pt_regs address] [original switch_stack address]
        |   [unused part of the gap] [moved switch_stack] [moved pt_regs]
        |   [replacement exception frame]
        | return value of do_{rt_,}sigreturn() points to moved switch_stack.

        movel   %d0,%sp                         | discard the leftover junk
        RESTORE_SWITCH_STACK
        | stack contents now is just [syscall return address] [pt_regs] [frame]
        | return pt_regs.d0
        movel   %sp@(PT_OFF_D0+4),%d0
        rts

... but I noticed that the sys_rt_sigreturn entry point does 
SAVE_SWITCH_STACK while the sys_sigreturn entry point does not. Yet both 
jump to label 1 above, so both syscalls do RESTORE_SWITCH_STACK. Hmmm.

.macro SAVE_SWITCH_STACK
        moveml  %a3-%a6/%d6-%d7,%sp@-
.endm

.macro RESTORE_SWITCH_STACK
        moveml  %sp@+,%a3-%a6/%d6-%d7
.endm

Well, that has to corrupt %a3, right?

Re: signal delivery, was Re: reliable reproducer

[Michael Schmitz]

Hi Finn,

Am 26.04.2023 um 15:27 schrieb Finn Thain:
> On Wed, 26 Apr 2023, Michael Schmitz wrote:
>
>> On 26/04/23 07:46, Michael Schmitz wrote:
>>>
>>> I had thought the 030 could resume the interrupted instruction using
>>> the information from the exception frame - and that does appear to
>>> work in all other cases except where signal delivery gets in the way,
>>> and it also works if moving the exception frame a little bit further
>>> down the stack. So our treatment of the bus error exception frame
>>> during signal delivery appears to be incorrect. Wouldn't you agree?
>>
>> Inspection of the format b frame placed in the signal frame in both rt
>> and non-rt cases (at the time the signal handler runs) shows the
>> expected contents in the data output buffer, data fault address and ssw.
>> At that time, returning to user space with rte would correctly resume
>> the instruction execution.  I had previously confirmed that the register
>> contents saved in the rt signal frame is correct also.
>>
>> That is with a kernel patched similar to above patch by Finn (using an
>> offset of 128 or 64 instead of 256).
>
> That means things go awry during sys_sigreturn or sys_rt_sigreturn. I'm
> not sure what happens to the exception frame:
>
> 1:
>         | stack contents now:
>         |   [original pt_regs address] [original switch_stack address]
>         |   [unused part of the gap] [moved switch_stack] [moved pt_regs]
>         |   [replacement exception frame]
>         | return value of do_{rt_,}sigreturn() points to moved switch_stack.
>
>         movel   %d0,%sp                         | discard the leftover junk
>         RESTORE_SWITCH_STACK
>         | stack contents now is just [syscall return address] [pt_regs] [frame]
>         | return pt_regs.d0
>         movel   %sp@(PT_OFF_D0+4),%d0
>         rts
>
> ... but I noticed that the sys_rt_sigreturn entry point does
> SAVE_SWITCH_STACK while the sys_sigreturn entry point does not. Yet both
> jump to label 1 above, so both syscalls do RESTORE_SWITCH_STACK. Hmmm.

In my copy of entry.S, sys_sigreturn has SAVE_SWITCH_STACK, just as 
sys_rt_sigreturn does.

Cheers,

	Michael


> .macro SAVE_SWITCH_STACK
>         moveml  %a3-%a6/%d6-%d7,%sp@-
> .endm
>
> .macro RESTORE_SWITCH_STACK
>         moveml  %sp@+,%a3-%a6/%d6-%d7
> .endm
>
> Well, that has to corrupt %a3, right?
>

Re: signal delivery, was Re: reliable reproducer

[Finn Thain]

On Wed, 26 Apr 2023, Michael Schmitz wrote:

> In my copy of entry.S, sys_sigreturn has SAVE_SWITCH_STACK, just as
> sys_rt_sigreturn does.
> 

You're right.

I think I've been at this too long.

Re: signal delivery, was Re: reliable reproducer

[Finn Thain]

On Wed, 26 Apr 2023, Michael Schmitz wrote:

> Thanks - we had seen evidence that a bus error generated mid-instruction 
> did leave the USP at the address where the bus fault happened (not 
> before the instruction started, neither what it would have been once the 
> instruction completed), and the operation did not complete normally 
> after the bus error (at least the value/address seen in the exception 
> frame not stored).

I'm afraid I still don't fully understand how and why the user stack 
(rather than the supervisor stack) gets used for processing the exception 
frame.

> Finn had also demonstrated that skipping signal delivery on bus errors 
> abolishes the stack corruption.  Your patch achieves the same objective 
> in a different way, so I'm sure this will work as well.
> 
> I had thought the 030 could resume the interrupted instruction using the 
> information from the exception frame - and that does appear to work in 
> all other cases except where signal delivery gets in the way, and it 
> also works if moving the exception frame a little bit further down the 
> stack. So our treatment of the bus error exception frame during signal 
> delivery appears to be incorrect.

Wouldn't that depend on the exception frame format? Perhaps it is unsafe 
to treat any format 0xB exception frame in the way we do. If so, what do 
we do about address error exceptions, which are to produce SIGBUS? The 
Programmers Reference Manual says "a long bus fault stack frame may be 
generated" in this case.

Re: signal delivery, was Re: reliable reproducer

[Michael Schmitz]

Hi Finn,

Am 26.04.2023 um 14:02 schrieb Finn Thain:
> On Wed, 26 Apr 2023, Michael Schmitz wrote:
>
>> Thanks - we had seen evidence that a bus error generated mid-instruction
>> did leave the USP at the address where the bus fault happened (not
>> before the instruction started, neither what it would have been once the
>> instruction completed), and the operation did not complete normally
>> after the bus error (at least the value/address seen in the exception
>> frame not stored).
>
> I'm afraid I still don't fully understand how and why the user stack
> (rather than the supervisor stack) gets used for processing the exception
> frame.

The kernel stack would not be accessible to the signal handler which 
must run in process context (i.e. user space).

The exception frame is copied to the signal frame for informational 
purposes only (such as examination of processor state when the signal 
was taken - not too useful for SIGCHLD but could be used to interpret 
SIGSEGV).

>
>> Finn had also demonstrated that skipping signal delivery on bus errors
>> abolishes the stack corruption.  Your patch achieves the same objective
>> in a different way, so I'm sure this will work as well.
>>
>> I had thought the 030 could resume the interrupted instruction using the
>> information from the exception frame - and that does appear to work in
>> all other cases except where signal delivery gets in the way, and it
>> also works if moving the exception frame a little bit further down the
>> stack. So our treatment of the bus error exception frame during signal
>> delivery appears to be incorrect.

It seems I got confused about user and kernel stack there myself. And 
managed to confuse almost everyone else about this bug. Apologies for 
the incessant noise.

What matters for the return from exception is an intact frame on the 
kernel stack. Anything we do on the user stack (mucking around with the 
offset the sigframe is set up at, copying siginfo, ucontext or 
sigcontext plus exception frame extra) does not change the kernel stack 
one whit.

The mangle_kernel_stack stuff is needed because sys_sigreturn will place 
another exception frame on the kernel stack (a four word frame) that 
needs to be replaced by the bus error exception frame (or any other 
frame that caused the kernel mode entry prior to signal delivery) before 
finally returning from the bus error exception.

Only at that time will the movel instruction that took the bus fault 
resume (and complete its writes correctly, I hope).

Our problem may be that, if we take the signal too late and our main 
process inspects the stack that has been left partially saved only (due 
to the bus error processing still in-flight), we appear to be in 
trouble. After completing sys_sigreturn, everything will be OK.

I can see this cause the stack error in the test case. Not sure it also 
applies to the dash case ...

> Wouldn't that depend on the exception frame format? Perhaps it is unsafe
> to treat any format 0xB exception frame in the way we do. If so, what do
> we do about address error exceptions, which are to produce SIGBUS? The
> Programmers Reference Manual says "a long bus fault stack frame may be
> generated" in this case.

We don't handle access errors (beyond terminating the offending process).

I hope this makes a little more sense now...

Cheers,

	Michael

Re: signal delivery, was Re: reliable reproducer

[Finn Thain]

On Wed, 26 Apr 2023, Michael Schmitz wrote:

> It seems I got confused about user and kernel stack there myself. And 
> managed to confuse almost everyone else about this bug. Apologies for 
> the incessant noise.
> 

Likewise... I think it amuses some readers and annoys others ;-)

> What matters for the return from exception is an intact frame on the 
> kernel stack. Anything we do on the user stack (mucking around with the 
> offset the sigframe is set up at, copying siginfo, ucontext or 
> sigcontext plus exception frame extra) does not change the kernel stack 
> one whit.
> 
> The mangle_kernel_stack stuff is needed because sys_sigreturn will place 
> another exception frame on the kernel stack (a four word frame) that 
> needs to be replaced by the bus error exception frame (or any other 
> frame that caused the kernel mode entry prior to signal delivery) before 
> finally returning from the bus error exception.
> 
> Only at that time will the movel instruction that took the bus fault 
> resume (and complete its writes correctly, I hope).
> 

If the long format frame was corrupted while on the user stack, the 
partially completed MOVEM won't be resumed correctly. That's why I was 
concerned about a bug in sys_sigreturn.

> Our problem may be that, if we take the signal too late and our main 
> process inspects the stack that has been left partially saved only (due 
> to the bus error processing still in-flight), we appear to be in 
> trouble. After completing sys_sigreturn, everything will be OK.
> 
> I can see this cause the stack error in the test case. Not sure it also 
> applies to the dash case ...
> 
> > Wouldn't that depend on the exception frame format? Perhaps it is 
> > unsafe to treat any format 0xB exception frame in the way we do. If 
> > so, what do we do about address error exceptions, which are to produce 
> > SIGBUS? The Programmers Reference Manual says "a long bus fault stack 
> > frame may be generated" in this case.
> 
> We don't handle access errors (beyond terminating the offending 
> process).
> 

SIGBUS could be caught and handled, perhaps followed by a setcontext() or 
siglongjmp() rather than return. So I don't think we get to disallow frame 
format 0xB.

Re: signal delivery, was Re: reliable reproducer

[Michael Schmitz]

Hi Finn,

Am 26.04.2023 um 16:42 schrieb Finn Thain:
> If the long format frame was corrupted while on the user stack, the
> partially completed MOVEM won't be resumed correctly. That's why I was
> concerned about a bug in sys_sigreturn.

Yes, it turns out I hadn't read mangle_kernel_stack() carefully enough. 
I thought the exception frame had remained on the kernel stack to be 
restored, but I'd missed that it is actually being restored from the 
user stack copy to the kernel stack.

Need to go back and check whether the sigmask (located after uc_filler 
in rt sigframes) ever got clobbered.

>>> Wouldn't that depend on the exception frame format? Perhaps it is
>>> unsafe to treat any format 0xB exception frame in the way we do. If
>>> so, what do we do about address error exceptions, which are to produce
>>> SIGBUS? The Programmers Reference Manual says "a long bus fault stack
>>> frame may be generated" in this case.
>>
>> We don't handle access errors (beyond terminating the offending

Meant to say 'address errors' there. These are misaligned instruction 
accesses only, and we take them to indicate corruption of the binary or 
stack, so abort the process.

>> process).
>>
>
> SIGBUS could be caught and handled, perhaps followed by a setcontext() or
> siglongjmp() rather than return. So I don't think we get to disallow frame
> format 0xB.

Yes, we may still need to take these signals. But we don't have to 
deliver the signals to the user process while exception processing is 
still in progress. For bus faults mid-instruction, it isn't enough to 
map in a page so writes can succeed, we also need to resume the 
instruction to carry out the write.

Come to think of it, the same argument applies for faults taken at an 
instruction boundary. These need to be resumed as well, they just take 
less information to load back into the processor.

Now I just wonder why 040 bus errors (exception restart instead of 
resume) aren't a problem. Pending writebacks are handled during signal 
return there, too. But the format 7 frame is only 15 longwords, the 
format b one is 23 longwords. We know increasing the apparent length of 
the signal frame helps ...

Anyway, Andreas' patch addresses the issue without incurring too much 
delay in signal processing. If we find out what causes corruption of the 
exception frame, Andreas' patch can be dropped once a fix is in place.

Cheers,

	Michael

Re: signal delivery, was Re: reliable reproducer

[Brad Boyer]

On Wed, Apr 26, 2023 at 09:10:50PM +1200, Michael Schmitz wrote:
> Am 26.04.2023 um 16:42 schrieb Finn Thain:
> >If the long format frame was corrupted while on the user stack, the
> >partially completed MOVEM won't be resumed correctly. That's why I was
> >concerned about a bug in sys_sigreturn.
> 
> Yes, it turns out I hadn't read mangle_kernel_stack() carefully enough. I
> thought the exception frame had remained on the kernel stack to be restored,
> but I'd missed that it is actually being restored from the user stack copy
> to the kernel stack.

Isn't that a security hole? If we restore the exception frame from
user memory, doesn't that allow a malicious program to affect the
internal state of the CPU just by handling a signal?

	Brad Boyer
	flar@allandria.com

Re: signal delivery, was Re: reliable reproducer

[Andreas Schwab]

On Apr 26 2023, Michael Schmitz wrote:

> The exception frame is copied to the signal frame for informational
> purposes only

This is not for informational purpose.  It is the interrupted context
that is restored by the kernel in the sigreturn syscall.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: signal delivery, was Re: reliable reproducer

[Andreas Schwab]

Please try this patch.  Signal delivery should only happen at insn
boundaries, but due to the way the 030 handles return from bus error
exceptions (the insn is resumed, not restarted like on the 040/060) the
kernel may do it in the middle of the faulting insn.

diff --git a/arch/m68k/kernel/entry.S b/arch/m68k/kernel/entry.S
index 4dd2fd7acba9..6c09a5710728 100644
--- a/arch/m68k/kernel/entry.S
+++ b/arch/m68k/kernel/entry.S
@@ -117,7 +117,11 @@ ENTRY(buserr)
 	movel	%sp,%sp@-		| stack frame pointer argument
 	jbsr	buserr_c
 	addql	#4,%sp
-	jra	ret_from_exception
+	| don't do signal delivery when interrupted while insn is in progress
+	| (on the 020/030)
+	tstl	%d0
+	jeq	ret_from_exception
+	RESTORE_ALL
 
 ENTRY(trap)
 	SAVE_ALL_INT
diff --git a/arch/m68k/kernel/traps.c b/arch/m68k/kernel/traps.c
index a700807c9b6d..40fc408d1333 100644
--- a/arch/m68k/kernel/traps.c
+++ b/arch/m68k/kernel/traps.c
@@ -751,8 +751,10 @@ static inline void access_errorcf(unsigned int fs, struct frame *fp)
 }
 #endif /* CONFIG_COLDFIRE CONFIG_MMU */
 
-asmlinkage void buserr_c(struct frame *fp)
+asmlinkage int buserr_c(struct frame *fp)
 {
+	int not_insn_boundary = 0;
+
 	/* Only set esp0 if coming from user mode */
 	if (user_mode(&fp->ptregs))
 		current->thread.esp0 = (unsigned long) fp;
@@ -793,8 +795,9 @@ asmlinkage void buserr_c(struct frame *fp)
 	  break;
 #endif
 #if defined (CPU_M68020_OR_M68030)
-	case 0xa:
 	case 0xb:
+	  not_insn_boundary = 1;
+	case 0xa:
 	  bus_error030 (fp);
 	  break;
 #endif
@@ -803,6 +806,8 @@ asmlinkage void buserr_c(struct frame *fp)
 	  pr_debug("Unknown SIGSEGV - 4\n");
 	  force_sig(SIGSEGV);
 	}
+
+	return not_insn_boundary;
 }
 
 

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: signal delivery, was Re: reliable reproducer

[Finn Thain]

On Tue, 25 Apr 2023, Andreas Schwab wrote:

> Please try this patch. 

Thanks for looking into this. Your patch solves the problem for me and 
doesn't seem to cause any regression.

Tested-by: Finn Thain <fthain@linux-m68k.org>

> Signal delivery should only happen at insn boundaries, but due to the 
> way the 030 handles return from bus error

The Programmer's Reference Manual says that the format 0xB frame is used 
by '020 as well which is consistent with the ifdef below, so it probably 
should be mentioned in the commit log too. (And I assume that '020 is also 
affected by this bug.)

> exceptions (the insn is resumed, not restarted like on the 040/060) the 
> kernel may do it in the middle of the faulting insn.
> 
> diff --git a/arch/m68k/kernel/entry.S b/arch/m68k/kernel/entry.S
> index 4dd2fd7acba9..6c09a5710728 100644
> --- a/arch/m68k/kernel/entry.S
> +++ b/arch/m68k/kernel/entry.S
> @@ -117,7 +117,11 @@ ENTRY(buserr)
>  	movel	%sp,%sp@-		| stack frame pointer argument
>  	jbsr	buserr_c
>  	addql	#4,%sp
> -	jra	ret_from_exception
> +	| don't do signal delivery when interrupted while insn is in progress

We might avoid the word "interrupted" like so:

	| deliver no signal if the fault occurred while insn was in progress

> +	| (on the 020/030)
> +	tstl	%d0
> +	jeq	ret_from_exception
> +	RESTORE_ALL
>  
>  ENTRY(trap)
>  	SAVE_ALL_INT
> diff --git a/arch/m68k/kernel/traps.c b/arch/m68k/kernel/traps.c
> index a700807c9b6d..40fc408d1333 100644
> --- a/arch/m68k/kernel/traps.c
> +++ b/arch/m68k/kernel/traps.c
> @@ -751,8 +751,10 @@ static inline void access_errorcf(unsigned int fs, struct frame *fp)
>  }
>  #endif /* CONFIG_COLDFIRE CONFIG_MMU */
>  
> -asmlinkage void buserr_c(struct frame *fp)
> +asmlinkage int buserr_c(struct frame *fp)
>  {
> +	int not_insn_boundary = 0;
> +
>  	/* Only set esp0 if coming from user mode */
>  	if (user_mode(&fp->ptregs))
>  		current->thread.esp0 = (unsigned long) fp;
> @@ -793,8 +795,9 @@ asmlinkage void buserr_c(struct frame *fp)
>  	  break;
>  #endif
>  #if defined (CPU_M68020_OR_M68030)
> -	case 0xa:
>  	case 0xb:
> +	  not_insn_boundary = 1;

The build uses -Wimplicit-fallthrough so I added this:

	  fallthrough;

> +	case 0xa:
>  	  bus_error030 (fp);
>  	  break;
>  #endif
> @@ -803,6 +806,8 @@ asmlinkage void buserr_c(struct frame *fp)
>  	  pr_debug("Unknown SIGSEGV - 4\n");
>  	  force_sig(SIGSEGV);
>  	}
> +
> +	return not_insn_boundary;
>  }
>  

Re: reliable reproducer, was Re: core dump analysis

[Michael Schmitz]

Hi Andreas,

Am 22.04.2023 um 22:12 schrieb Andreas Schwab:
> On Apr 22 2023, Michael Schmitz wrote:
>
>> This is the definition from the kernel's
>> include/uapi/asm-generic/ucontext.h:
>
> That's not actually used by m68k, it uses
> arch/m68k/include/asm/ucontext.h, which confusingly isn't an uapi
> header.

As Stan and Finn report, it is actually used by recent m68k gcc version 
12.2.0 (Debian 12.2.0-12):

> Debian/sid has this struct in asm-generic/ucontext.h --
>
> struct ucontext {
>         unsigned long     uc_flags;
>         struct ucontext  *uc_link;
>         stack_t           uc_stack;
>         struct sigcontext uc_mcontext;
>         sigset_t          uc_sigmask;   /* mask last for extensibility */
> };
>
> but that lacks uc_filler[] so it's not ideal. I adapted the definitions
> from the kernel source.

Time to move this into an uapi header, or is that more hassle than it's 
worth?

Cheers,

	Michael

Re: reliable reproducer, was Re: core dump analysis

[Michael Schmitz]

Hi Finn,

Am 22.04.2023 um 19:54 schrieb Michael Schmitz:
> Hi Finn,
>
> Am 21.04.2023 um 21:18 schrieb Michael Schmitz:
>> Hi Finn,
>>
>> Am 21.04.2023 um 20:30 schrieb Finn Thain:
>>> On Fri, 21 Apr 2023, Michael Schmitz wrote:
>>>
>>>>
>>>> How often did a page fault happen when executing moveml, in other
>>>> programs?
>>>>
>>>
>>> The printk() I placed in bus_error030() was conditional on the short
>>> word
>>> at the instruction pointer. It didn't consider all forms of movem, just
>>> 0x48e7 which is the start of "moveml X,%sp@-". This matched page
>>> faults in
>>> many of the programs that executed while booting to single user mode. I
>>> suppose most of them don't use signal handlers in the same way dash does
>>> otherwise they would probably be unreliable too.
>>
>> OK; so too much noise unless filtered on the command name...
>>
>> I'll try first to get register state at the time of signal delivery from
>> the sa_sigaction handler's ucontext parameter to see where the signal
>> stack falls in relation to the call frames from your rec() function on
>> the stack (and what the register contents were). Hope that won't be too
>
> Took a little while to figure out that the ucontext format changed in
> the decade or two since my userland's libc headers were generated. With
> the correct format, the information stored on th signal frame made a lot
> more sense.
>
> Log of your test program (attached), instrumented to keep track of user
> stack pointer in the parent process, user stack pointer in the signal
> handler, and stack pointer, pc and exceptiopn frame format from the
> signal stack (only the last few signals shown):
>
> parent usp  : 0xef97beb8
> handler tos : 0xef97bdc4
> handler usp : 0xef97bbe0
> signal usp  : 0xef97bea8
> signal pc   : 0xc009f37a
> signal fmtv : 0x800006ca
>
> parent usp  : 0xef969eb8
> handler tos : 0xef969dc4
> handler usp : 0xef969be0
> signal usp  : 0xef969ea8
> signal pc   : 0xc009f37a
> signal fmtv : 0x800006ca
>
> parent usp  : 0xef9530d0
> handler tos : 0xef952fec
> handler usp : 0xef952e08
> signal usp  : 0xef9530d0
> signal pc   : 0x800006dc
> signal fmtv : 0x91929394
>
> parent usp  : 0xef945eb8
> handler tos : 0xef945dc4
> handler usp : 0xef945be0
> signal usp  : 0xef945ea8
> signal pc   : 0xc009f37a
> signal fmtv : 0x800006ca
>
> parent usp  : 0xef933eb8
> handler tos : 0xef933dc4
> handler usp : 0xef933be0
> signal usp  : 0xef933ea8
> signal pc   : 0xc009f37a
> signal fmtv : 0x800006ca
>
> parent usp  : 0xef921edc
> handler tos : 0xef997984
> handler stack overwrote usp!
> handler usp : 0xef9977a0
> signal usp  : 0xef997a64
> signal pc   : 0x80000768
> signal fmtv : 0xa1a2a3a4
>
> Illegal instruction (core dumped)
>
> usp from the signal stack is below that of the parent process (before
> calling fork()).
>
> usp from the signal handler is below both of those. So far, so good.
>
> The top of the signal frame, however, is getting quite close to these
> stack pointers. In the last log, it has grown above the user stack pointer.
>
> Two things to note:
>
> - pc in the signal frame (from struct uc_mcontext) is either the return
> pc from the stack stuffing function, or something else I cannot work
> out. That part of ucontext appears valid.
>
> - what ought to be the frame format and vector offset does in fact hold
> varying longwords from the user stack. This information is not from
> struct uc_mcontext, but from extra information copied after struct
> ucontext ends. That wouldn't be there if at time of signal delivery,
> nothing had yet written to the area where the signal frame is stored.

Actually, 0x800006ca (should be format/vector but more likely a return 
address saved on the stack by rec()) is the instruction after fork() in 
rec(), and 0xc009f37a is the instuction after trap 0 in fork(). The 
signal frame pc seems to be anywhere in rec() ...

Cheers,

	Michael

Re: reliable reproducer, was Re: core dump analysis

[Finn Thain]

[-- Attachment #1: Type: text/plain, Size: 428 bytes --]

Fri, 21 Apr 2023 11:15:22 +1000 (AEST)n Thu, 20 Apr 2023, Michael Schmitz wrote:

> 
> In my tests, increasing the depth does not cause a monotonous increase 
> in fault probability. 16k depth only has four crashes, 8k had nine. I'll 
> stick with 200000 for now.
> 

My tests used 'norandmaps' in the kernel parameters. With the attached 
.config (which supports Mac and Atari) I saw 12 failures out of 16 tests 
at depth 5000.

[-- Attachment #2: Type: text/plain, Size: 49862 bytes --]

#
# Automatically generated file; DO NOT EDIT.
# Linux/m68k 6.3.0-rc5 Kernel Configuration
#
CONFIG_CC_VERSION_TEXT="m68k-unknown-linux-musl-gcc (Gentoo 12.2.1_p20221224 p7) 12.2.1 20221224"
CONFIG_CC_IS_GCC=y
CONFIG_GCC_VERSION=120201
CONFIG_CLANG_VERSION=0
CONFIG_AS_IS_GNU=y
CONFIG_AS_VERSION=23900
CONFIG_LD_IS_BFD=y
CONFIG_LD_VERSION=23900
CONFIG_LLD_VERSION=0
CONFIG_CC_HAS_ASM_INLINE=y
CONFIG_CC_HAS_NO_PROFILE_FN_ATTR=y
CONFIG_PAHOLE_VERSION=124
CONFIG_IRQ_WORK=y

#
# General setup
#
CONFIG_BROKEN_ON_SMP=y
CONFIG_INIT_ENV_ARG_LIMIT=32
# CONFIG_COMPILE_TEST is not set
# CONFIG_WERROR is not set
CONFIG_LOCALVERSION="-multi"
CONFIG_LOCALVERSION_AUTO=y
CONFIG_BUILD_SALT=""
CONFIG_DEFAULT_INIT=""
CONFIG_DEFAULT_HOSTNAME="(none)"
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
CONFIG_POSIX_MQUEUE=y
CONFIG_POSIX_MQUEUE_SYSCTL=y
# CONFIG_WATCH_QUEUE is not set
# CONFIG_CROSS_MEMORY_ATTACH is not set
# CONFIG_USELIB is not set
# CONFIG_AUDIT is not set

#
# IRQ subsystem
#
CONFIG_GENERIC_IRQ_SHOW=y
# end of IRQ subsystem

CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_LEGACY_TIMER_TICK=y

#
# Timers subsystem
#
CONFIG_TICK_ONESHOT=y
CONFIG_HZ_PERIODIC=y
# CONFIG_NO_HZ_IDLE is not set
# CONFIG_NO_HZ is not set
CONFIG_HIGH_RES_TIMERS=y
# end of Timers subsystem

CONFIG_BPF=y

#
# BPF subsystem
#
# CONFIG_BPF_SYSCALL is not set
# end of BPF subsystem

CONFIG_PREEMPT_NONE_BUILD=y
CONFIG_PREEMPT_NONE=y

#
# CPU/Task time and stats accounting
#
CONFIG_TICK_CPU_ACCOUNTING=y
# CONFIG_BSD_PROCESS_ACCT is not set
# CONFIG_TASKSTATS is not set
# CONFIG_PSI is not set
# end of CPU/Task time and stats accounting

#
# RCU Subsystem
#
CONFIG_TINY_RCU=y
# CONFIG_RCU_EXPERT is not set
CONFIG_SRCU=y
CONFIG_TINY_SRCU=y
# end of RCU Subsystem

# CONFIG_IKCONFIG is not set
# CONFIG_IKHEADERS is not set
CONFIG_LOG_BUF_SHIFT=14
CONFIG_PRINTK_SAFE_LOG_BUF_SHIFT=13

#
# Scheduler features
#
# end of Scheduler features

CONFIG_CC_IMPLICIT_FALLTHROUGH="-Wimplicit-fallthrough=5"
CONFIG_GCC11_NO_ARRAY_BOUNDS=y
CONFIG_GCC12_NO_ARRAY_BOUNDS=y
CONFIG_CC_NO_ARRAY_BOUNDS=y
CONFIG_CGROUPS=y
# CONFIG_CGROUP_FAVOR_DYNMODS is not set
# CONFIG_MEMCG is not set
# CONFIG_BLK_CGROUP is not set
# CONFIG_CGROUP_SCHED is not set
# CONFIG_CGROUP_PIDS is not set
# CONFIG_CGROUP_RDMA is not set
# CONFIG_CGROUP_FREEZER is not set
# CONFIG_CGROUP_DEVICE is not set
# CONFIG_CGROUP_CPUACCT is not set
# CONFIG_CGROUP_MISC is not set
# CONFIG_CGROUP_DEBUG is not set
CONFIG_NAMESPACES=y
CONFIG_UTS_NS=y
CONFIG_IPC_NS=y
# CONFIG_USER_NS is not set
CONFIG_PID_NS=y
CONFIG_NET_NS=y
# CONFIG_CHECKPOINT_RESTORE is not set
# CONFIG_SCHED_AUTOGROUP is not set
# CONFIG_SYSFS_DEPRECATED is not set
# CONFIG_RELAY is not set
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
CONFIG_RD_GZIP=y
# CONFIG_RD_BZIP2 is not set
# CONFIG_RD_LZMA is not set
# CONFIG_RD_XZ is not set
# CONFIG_RD_LZO is not set
# CONFIG_RD_LZ4 is not set
# CONFIG_RD_ZSTD is not set
# CONFIG_BOOT_CONFIG is not set
CONFIG_INITRAMFS_PRESERVE_MTIME=y
CONFIG_CC_OPTIMIZE_FOR_PERFORMANCE=y
# CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
CONFIG_SYSCTL=y
CONFIG_HAVE_UID16=y
CONFIG_EXPERT=y
CONFIG_UID16=y
CONFIG_MULTIUSER=y
# CONFIG_SGETMASK_SYSCALL is not set
CONFIG_SYSFS_SYSCALL=y
CONFIG_FHANDLE=y
CONFIG_POSIX_TIMERS=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_FUTEX_PI=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_AIO=y
# CONFIG_IO_URING is not set
CONFIG_ADVISE_SYSCALLS=y
# CONFIG_MEMBARRIER is not set
CONFIG_KALLSYMS=y
# CONFIG_KALLSYMS_SELFTEST is not set
# CONFIG_KALLSYMS_ALL is not set
CONFIG_KALLSYMS_BASE_RELATIVE=y
# CONFIG_KCMP is not set
CONFIG_EMBEDDED=y
# CONFIG_PC104 is not set

#
# Kernel Performance Events And Counters
#
# end of Kernel Performance Events And Counters

# CONFIG_PROFILING is not set
# end of General setup

CONFIG_M68K=y
CONFIG_CPU_BIG_ENDIAN=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_TIME_LOW_RES=y
CONFIG_NO_IOPORT_MAP=y
CONFIG_HZ=100
CONFIG_PGTABLE_LEVELS=3
CONFIG_MMU=y
CONFIG_MMU_MOTOROLA=y
CONFIG_KEXEC=y
CONFIG_BOOTINFO_PROC=y

#
# Platform setup
#

#
# Processor Type
#
CONFIG_M68KCLASSIC=y
# CONFIG_COLDFIRE is not set
CONFIG_M68020=y
CONFIG_M68030=y
CONFIG_M68040=y
CONFIG_M68060=y

#
# Processor Specific Options
#
CONFIG_M68KFPU_EMU=y
# CONFIG_M68KFPU_EMU_EXTRAPREC is not set
# CONFIG_M68KFPU_EMU_ONLY is not set
# CONFIG_ADVANCED is not set
CONFIG_ARCH_FORCE_MAX_ORDER=11
CONFIG_M68K_L2_CACHE=y
CONFIG_CPU_HAS_ADDRESS_SPACES=y
CONFIG_FPU=y

#
# Machine Types
#
# CONFIG_AMIGA is not set
CONFIG_ATARI=y
CONFIG_ATARI_KBD_CORE=y
CONFIG_MAC=y
# CONFIG_APOLLO is not set
CONFIG_VME=y
CONFIG_MVME147=y
CONFIG_MVME16x=y
CONFIG_BVME6000=y
# CONFIG_HP300 is not set
# CONFIG_SUN3X is not set
# CONFIG_Q40 is not set
CONFIG_VIRT=y

#
# Bus Support
#
CONFIG_NUBUS=y
CONFIG_ATARI_ROM_ISA=y
# end of Platform setup

#
# Kernel Features
#
# end of Kernel Features

#
# Platform devices
#
# CONFIG_HEARTBEAT is not set
CONFIG_PROC_HARDWARE=y
CONFIG_NATFEAT=y
CONFIG_NFBLOCK=y
CONFIG_NFCON=y
CONFIG_NFETH=y
CONFIG_ATARI_ETHERNAT=y
# CONFIG_ATARI_ETHERNEC is not set
# end of Platform devices

#
# Character devices
#
# CONFIG_ATARI_DSP56K is not set
# end of Character devices

#
# General architecture-dependent options
#
CONFIG_CRASH_CORE=y
CONFIG_KEXEC_CORE=y
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
CONFIG_ARCH_32BIT_OFF_T=y
CONFIG_HAVE_ASM_MODVERSIONS=y
CONFIG_MMU_GATHER_NO_RANGE=y
CONFIG_MMU_GATHER_MERGE_VMAS=y
CONFIG_ARCH_WANT_IPC_PARSE_VERSION=y
CONFIG_HAVE_ARCH_SECCOMP=y
CONFIG_HAVE_ARCH_SECCOMP_FILTER=y
CONFIG_SECCOMP=y
CONFIG_SECCOMP_FILTER=y
# CONFIG_SECCOMP_CACHE_DEBUG is not set
CONFIG_LTO_NONE=y
CONFIG_HAVE_MOD_ARCH_SPECIFIC=y
CONFIG_MODULES_USE_ELF_RELA=y
CONFIG_MODULES_USE_ELF_REL=y
CONFIG_ALTERNATE_USER_ADDRESS_SPACE=y
CONFIG_PAGE_SIZE_LESS_THAN_64KB=y
CONFIG_PAGE_SIZE_LESS_THAN_256KB=y
CONFIG_HAVE_ARCH_NVRAM_OPS=y
CONFIG_OLD_SIGSUSPEND3=y
CONFIG_OLD_SIGACTION=y
CONFIG_COMPAT_32BIT_TIME=y
CONFIG_ARCH_NO_PREEMPT=y
CONFIG_HAVE_ARCH_PFN_VALID=y

#
# GCOV-based kernel profiling
#
# end of GCOV-based kernel profiling

CONFIG_FUNCTION_ALIGNMENT=0
# end of General architecture-dependent options

CONFIG_RT_MUTEXES=y
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
# CONFIG_MODULE_FORCE_LOAD is not set
CONFIG_MODULE_UNLOAD=y
# CONFIG_MODULE_FORCE_UNLOAD is not set
# CONFIG_MODULE_UNLOAD_TAINT_TRACKING is not set
# CONFIG_MODVERSIONS is not set
# CONFIG_MODULE_SRCVERSION_ALL is not set
# CONFIG_MODULE_SIG is not set
CONFIG_MODULE_COMPRESS_NONE=y
# CONFIG_MODULE_COMPRESS_GZIP is not set
# CONFIG_MODULE_COMPRESS_XZ is not set
# CONFIG_MODULE_COMPRESS_ZSTD is not set
# CONFIG_MODULE_ALLOW_MISSING_NAMESPACE_IMPORTS is not set
CONFIG_MODPROBE_PATH="/sbin/modprobe"
CONFIG_TRIM_UNUSED_KSYMS=y
CONFIG_UNUSED_KSYMS_WHITELIST=""
CONFIG_BLOCK=y
CONFIG_BLOCK_LEGACY_AUTOLOAD=y
CONFIG_BLK_DEV_BSG_COMMON=y
CONFIG_BLK_ICQ=y
# CONFIG_BLK_DEV_BSGLIB is not set
# CONFIG_BLK_DEV_INTEGRITY is not set
# CONFIG_BLK_DEV_ZONED is not set
# CONFIG_BLK_WBT is not set
# CONFIG_BLK_SED_OPAL is not set
# CONFIG_BLK_INLINE_ENCRYPTION is not set

#
# Partition Types
#
CONFIG_PARTITION_ADVANCED=y
# CONFIG_ACORN_PARTITION is not set
# CONFIG_AIX_PARTITION is not set
# CONFIG_OSF_PARTITION is not set
CONFIG_AMIGA_PARTITION=y
CONFIG_ATARI_PARTITION=y
CONFIG_MAC_PARTITION=y
CONFIG_MSDOS_PARTITION=y
# CONFIG_BSD_DISKLABEL is not set
# CONFIG_MINIX_SUBPARTITION is not set
# CONFIG_SOLARIS_X86_PARTITION is not set
# CONFIG_UNIXWARE_DISKLABEL is not set
# CONFIG_LDM_PARTITION is not set
# CONFIG_SGI_PARTITION is not set
# CONFIG_ULTRIX_PARTITION is not set
# CONFIG_SUN_PARTITION is not set
# CONFIG_KARMA_PARTITION is not set
# CONFIG_EFI_PARTITION is not set
# CONFIG_SYSV68_PARTITION is not set
# CONFIG_CMDLINE_PARTITION is not set
# end of Partition Types

CONFIG_BLK_MQ_VIRTIO=y

#
# IO Schedulers
#
CONFIG_MQ_IOSCHED_DEADLINE=y
# CONFIG_MQ_IOSCHED_KYBER is not set
CONFIG_IOSCHED_BFQ=m
# end of IO Schedulers

CONFIG_INLINE_SPIN_UNLOCK_IRQ=y
CONFIG_INLINE_READ_UNLOCK=y
CONFIG_INLINE_READ_UNLOCK_IRQ=y
CONFIG_INLINE_WRITE_UNLOCK=y
CONFIG_INLINE_WRITE_UNLOCK_IRQ=y

#
# Executable file formats
#
CONFIG_BINFMT_ELF=y
CONFIG_ELFCORE=y
# CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS is not set
CONFIG_BINFMT_SCRIPT=y
CONFIG_ARCH_HAS_BINFMT_FLAT=y
# CONFIG_BINFMT_FLAT is not set
CONFIG_BINFMT_FLAT_ARGVP_ENVP_ON_STACK=y
# CONFIG_BINFMT_MISC is not set
CONFIG_COREDUMP=y
# end of Executable file formats

#
# Memory Management options
#
CONFIG_SWAP=y
# CONFIG_ZSWAP is not set

#
# SLAB allocator options
#
# CONFIG_SLAB is not set
CONFIG_SLUB=y
# CONFIG_SLOB_DEPRECATED is not set
# CONFIG_SLUB_TINY is not set
CONFIG_SLAB_MERGE_DEFAULT=y
# CONFIG_SLAB_FREELIST_RANDOM is not set
# CONFIG_SLAB_FREELIST_HARDENED is not set
# CONFIG_SLUB_STATS is not set
# end of SLAB allocator options

# CONFIG_SHUFFLE_PAGE_ALLOCATOR is not set
# CONFIG_COMPAT_BRK is not set
CONFIG_FLATMEM=y
CONFIG_SPLIT_PTLOCK_CPUS=4
# CONFIG_COMPACTION is not set
# CONFIG_PAGE_REPORTING is not set
# CONFIG_KSM is not set
CONFIG_DEFAULT_MMAP_MIN_ADDR=4096
CONFIG_NEED_PER_CPU_KM=y
# CONFIG_CMA is not set
# CONFIG_IDLE_PAGE_TRACKING is not set
CONFIG_ARCH_HAS_CURRENT_STACK_POINTER=y
CONFIG_ZONE_DMA=y
CONFIG_VM_EVENT_COUNTERS=y
# CONFIG_PERCPU_STATS is not set

#
# GUP_TEST needs to have DEBUG_FS enabled
#
# CONFIG_ANON_VMA_NAME is not set
# CONFIG_USERFAULTFD is not set
CONFIG_LRU_GEN=y
CONFIG_LRU_GEN_ENABLED=y
# CONFIG_LRU_GEN_STATS is not set

#
# Data Access Monitoring
#
# CONFIG_DAMON is not set
# end of Data Access Monitoring
# end of Memory Management options

CONFIG_NET=y

#
# Networking options
#
CONFIG_PACKET=y
# CONFIG_PACKET_DIAG is not set
CONFIG_UNIX=y
CONFIG_UNIX_SCM=y
CONFIG_AF_UNIX_OOB=y
# CONFIG_UNIX_DIAG is not set
# CONFIG_TLS is not set
# CONFIG_XFRM_USER is not set
# CONFIG_NET_KEY is not set
CONFIG_INET=y
# CONFIG_IP_MULTICAST is not set
# CONFIG_IP_ADVANCED_ROUTER is not set
CONFIG_IP_PNP=y
CONFIG_IP_PNP_DHCP=y
CONFIG_IP_PNP_BOOTP=y
CONFIG_IP_PNP_RARP=y
# CONFIG_NET_IPIP is not set
# CONFIG_NET_IPGRE_DEMUX is not set
CONFIG_NET_IP_TUNNEL=m
CONFIG_SYN_COOKIES=y
# CONFIG_NET_IPVTI is not set
# CONFIG_NET_FOU is not set
# CONFIG_NET_FOU_IP_TUNNELS is not set
# CONFIG_INET_AH is not set
# CONFIG_INET_ESP is not set
# CONFIG_INET_IPCOMP is not set
CONFIG_INET_TABLE_PERTURB_ORDER=16
CONFIG_INET_TUNNEL=m
# CONFIG_INET_DIAG is not set
# CONFIG_TCP_CONG_ADVANCED is not set
CONFIG_TCP_CONG_CUBIC=y
CONFIG_DEFAULT_TCP_CONG="cubic"
# CONFIG_TCP_MD5SIG is not set
CONFIG_IPV6=m
# CONFIG_IPV6_ROUTER_PREF is not set
# CONFIG_IPV6_OPTIMISTIC_DAD is not set
# CONFIG_INET6_AH is not set
# CONFIG_INET6_ESP is not set
# CONFIG_INET6_IPCOMP is not set
# CONFIG_IPV6_MIP6 is not set
# CONFIG_IPV6_VTI is not set
CONFIG_IPV6_SIT=m
CONFIG_IPV6_SIT_6RD=y
CONFIG_IPV6_NDISC_NODETYPE=y
# CONFIG_IPV6_TUNNEL is not set
# CONFIG_IPV6_MULTIPLE_TABLES is not set
# CONFIG_IPV6_MROUTE is not set
# CONFIG_IPV6_SEG6_LWTUNNEL is not set
# CONFIG_IPV6_SEG6_HMAC is not set
# CONFIG_IPV6_RPL_LWTUNNEL is not set
# CONFIG_IPV6_IOAM6_LWTUNNEL is not set
# CONFIG_MPTCP is not set
# CONFIG_NETWORK_SECMARK is not set
# CONFIG_NETWORK_PHY_TIMESTAMPING is not set
# CONFIG_NETFILTER is not set
# CONFIG_BPFILTER is not set
# CONFIG_IP_DCCP is not set
# CONFIG_IP_SCTP is not set
# CONFIG_RDS is not set
# CONFIG_TIPC is not set
# CONFIG_ATM is not set
# CONFIG_L2TP is not set
# CONFIG_BRIDGE is not set
# CONFIG_NET_DSA is not set
# CONFIG_VLAN_8021Q is not set
CONFIG_LLC=m
# CONFIG_LLC2 is not set
CONFIG_ATALK=m
CONFIG_DEV_APPLETALK=m
CONFIG_IPDDP=m
CONFIG_IPDDP_ENCAP=y
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_PHONET is not set
# CONFIG_6LOWPAN is not set
# CONFIG_IEEE802154 is not set
# CONFIG_NET_SCHED is not set
# CONFIG_DCB is not set
# CONFIG_BATMAN_ADV is not set
# CONFIG_OPENVSWITCH is not set
# CONFIG_VSOCKETS is not set
# CONFIG_NETLINK_DIAG is not set
# CONFIG_MPLS is not set
# CONFIG_NET_NSH is not set
# CONFIG_HSR is not set
# CONFIG_NET_SWITCHDEV is not set
# CONFIG_NET_L3_MASTER_DEV is not set
# CONFIG_QRTR is not set
# CONFIG_NET_NCSI is not set
# CONFIG_CGROUP_NET_PRIO is not set
# CONFIG_CGROUP_NET_CLASSID is not set
CONFIG_NET_RX_BUSY_POLL=y
CONFIG_BQL=y

#
# Network testing
#
# CONFIG_NET_PKTGEN is not set
# end of Network testing
# end of Networking options

# CONFIG_HAMRADIO is not set
# CONFIG_CAN is not set
# CONFIG_BT is not set
# CONFIG_AF_RXRPC is not set
# CONFIG_AF_KCM is not set
# CONFIG_MCTP is not set
# CONFIG_WIRELESS is not set
# CONFIG_RFKILL is not set
# CONFIG_NET_9P is not set
# CONFIG_CAIF is not set
# CONFIG_CEPH_LIB is not set
# CONFIG_NFC is not set
# CONFIG_PSAMPLE is not set
# CONFIG_NET_IFE is not set
# CONFIG_LWTUNNEL is not set
CONFIG_DST_CACHE=y
CONFIG_GRO_CELLS=y
CONFIG_NET_SELFTESTS=y
# CONFIG_FAILOVER is not set
# CONFIG_ETHTOOL_NETLINK is not set

#
# Device Drivers
#
# CONFIG_PCCARD is not set

#
# Generic Driver Options
#
# CONFIG_UEVENT_HELPER is not set
CONFIG_DEVTMPFS=y
# CONFIG_DEVTMPFS_MOUNT is not set
# CONFIG_DEVTMPFS_SAFE is not set
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y

#
# Firmware loader
#
# CONFIG_FW_LOADER is not set
# end of Firmware loader

# CONFIG_ALLOW_DEV_COREDUMP is not set
# CONFIG_DEBUG_DRIVER is not set
# CONFIG_DEBUG_DEVRES is not set
# CONFIG_DEBUG_TEST_DRIVER_REMOVE is not set
# CONFIG_TEST_ASYNC_DRIVER_PROBE is not set
CONFIG_GENERIC_CPU_DEVICES=y
# end of Generic Driver Options

#
# Bus devices
#
# CONFIG_MHI_BUS is not set
# CONFIG_MHI_BUS_EP is not set
# end of Bus devices

# CONFIG_CONNECTOR is not set

#
# Firmware Drivers
#

#
# ARM System Control and Management Interface Protocol
#
# end of ARM System Control and Management Interface Protocol

# CONFIG_FIRMWARE_MEMMAP is not set
# CONFIG_GOOGLE_FIRMWARE is not set

#
# Tegra firmware driver
#
# end of Tegra firmware driver
# end of Firmware Drivers

# CONFIG_GNSS is not set
# CONFIG_MTD is not set
# CONFIG_OF is not set
# CONFIG_PARPORT is not set
CONFIG_BLK_DEV=y
# CONFIG_BLK_DEV_NULL_BLK is not set
# CONFIG_ATARI_FLOPPY is not set
CONFIG_BLK_DEV_SWIM=m
CONFIG_CDROM=y
CONFIG_BLK_DEV_LOOP=m
CONFIG_BLK_DEV_LOOP_MIN_COUNT=0
# CONFIG_BLK_DEV_DRBD is not set
# CONFIG_BLK_DEV_NBD is not set
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_COUNT=4
CONFIG_BLK_DEV_RAM_SIZE=8192
# CONFIG_CDROM_PKTCDVD is not set
CONFIG_ATA_OVER_ETH=y
CONFIG_VIRTIO_BLK=y
# CONFIG_BLK_DEV_RBD is not set
# CONFIG_BLK_DEV_UBLK is not set

#
# NVME Support
#
# CONFIG_NVME_FC is not set
# CONFIG_NVME_TCP is not set
# end of NVME Support

#
# Misc devices
#
# CONFIG_DUMMY_IRQ is not set
# CONFIG_ENCLOSURE_SERVICES is not set
# CONFIG_SRAM is not set
# CONFIG_XILINX_SDFEC is not set
# CONFIG_C2PORT is not set

#
# EEPROM support
#
# CONFIG_EEPROM_93CX6 is not set
# end of EEPROM support

#
# Texas Instruments shared transport line discipline
#
# end of Texas Instruments shared transport line discipline

#
# Altera FPGA firmware download module (requires I2C)
#
# CONFIG_ECHO is not set
# CONFIG_PVPANIC is not set
# end of Misc devices

#
# SCSI device support
#
CONFIG_SCSI_MOD=y
# CONFIG_RAID_ATTRS is not set
CONFIG_SCSI_COMMON=y
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y
CONFIG_SCSI_ESP_PIO=y
CONFIG_SCSI_PROC_FS=y

#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=y
# CONFIG_CHR_DEV_ST is not set
CONFIG_BLK_DEV_SR=y
CONFIG_CHR_DEV_SG=y
CONFIG_BLK_DEV_BSG=y
# CONFIG_CHR_DEV_SCH is not set
CONFIG_SCSI_CONSTANTS=y
CONFIG_SCSI_LOGGING=y
# CONFIG_SCSI_SCAN_ASYNC is not set

#
# SCSI Transports
#
CONFIG_SCSI_SPI_ATTRS=y
# CONFIG_SCSI_FC_ATTRS is not set
# CONFIG_SCSI_ISCSI_ATTRS is not set
# CONFIG_SCSI_SAS_ATTRS is not set
# CONFIG_SCSI_SAS_LIBSAS is not set
# CONFIG_SCSI_SRP_ATTRS is not set
# end of SCSI Transports

CONFIG_SCSI_LOWLEVEL=y
# CONFIG_ISCSI_TCP is not set
# CONFIG_ISCSI_BOOT_SYSFS is not set
CONFIG_53C700_BE_BUS=y
# CONFIG_SCSI_DEBUG is not set
CONFIG_ATARI_SCSI=y
CONFIG_MAC_SCSI=y
CONFIG_SCSI_MAC_ESP=y
CONFIG_MVME147_SCSI=y
CONFIG_MVME16x_SCSI=y
CONFIG_BVME6000_SCSI=y
CONFIG_SCSI_VIRTIO=y
# CONFIG_SCSI_DH is not set
# end of SCSI device support

CONFIG_HAVE_PATA_PLATFORM=y
CONFIG_ATA=y
CONFIG_ATA_VERBOSE_ERROR=y
CONFIG_ATA_FORCE=y

#
# Controllers with non-SFF native interface
#
# CONFIG_SATA_AHCI_PLATFORM is not set
# CONFIG_AHCI_DWC is not set
CONFIG_ATA_SFF=y

#
# SFF controllers with custom DMA interface
#
CONFIG_ATA_BMDMA=y

#
# SATA SFF controllers with BMDMA
#

#
# PATA SFF controllers with BMDMA
#

#
# PIO-only SFF controllers
#
CONFIG_PATA_FALCON=y
CONFIG_PATA_PLATFORM=y

#
# Generic fallback / legacy drivers
#
# CONFIG_MD is not set
# CONFIG_TARGET_CORE is not set
CONFIG_MACINTOSH_DRIVERS=y
CONFIG_ADB=y
CONFIG_ADB_MACII=y
CONFIG_ADB_IOP=y
CONFIG_ADB_CUDA=y
CONFIG_ADB_PMU=y
CONFIG_ADB_PMU_EVENT=y
CONFIG_INPUT_ADBHID=y
CONFIG_MAC_EMUMOUSEBTN=y
CONFIG_NETDEVICES=y
CONFIG_MII=m
# CONFIG_NET_CORE is not set
CONFIG_ETHERNET=y
# CONFIG_NET_VENDOR_ALACRITECH is not set
# CONFIG_ALTERA_TSE is not set
# CONFIG_NET_VENDOR_AMAZON is not set
CONFIG_NET_VENDOR_AMD=y
CONFIG_ATARILANCE=y
CONFIG_MVME147_NET=y
CONFIG_NET_VENDOR_APPLE=y
CONFIG_MACMACE=y
# CONFIG_NET_VENDOR_AQUANTIA is not set
# CONFIG_NET_VENDOR_ARC is not set
# CONFIG_NET_VENDOR_ASIX is not set
# CONFIG_NET_VENDOR_BROADCOM is not set
# CONFIG_NET_VENDOR_CADENCE is not set
# CONFIG_NET_VENDOR_CAVIUM is not set
CONFIG_NET_VENDOR_CIRRUS=y
CONFIG_MAC89x0=y
# CONFIG_NET_VENDOR_CORTINA is not set
# CONFIG_NET_VENDOR_DAVICOM is not set
# CONFIG_DNET is not set
# CONFIG_NET_VENDOR_ENGLEDER is not set
# CONFIG_NET_VENDOR_EZCHIP is not set
# CONFIG_NET_VENDOR_FUNGIBLE is not set
# CONFIG_NET_VENDOR_GOOGLE is not set
# CONFIG_NET_VENDOR_HUAWEI is not set
# CONFIG_NET_VENDOR_INTEL is not set
# CONFIG_NET_VENDOR_WANGXUN is not set
# CONFIG_NET_VENDOR_LITEX is not set
# CONFIG_NET_VENDOR_MARVELL is not set
# CONFIG_NET_VENDOR_MICREL is not set
# CONFIG_NET_VENDOR_MICROCHIP is not set
# CONFIG_NET_VENDOR_MICROSEMI is not set
# CONFIG_NET_VENDOR_MICROSOFT is not set
# CONFIG_NET_VENDOR_NI is not set
CONFIG_NET_VENDOR_NATSEMI=y
CONFIG_MACSONIC=y
# CONFIG_NET_VENDOR_NETRONOME is not set
CONFIG_NET_VENDOR_8390=y
CONFIG_MAC8390=y
# CONFIG_ETHOC is not set
# CONFIG_NET_VENDOR_PENSANDO is not set
# CONFIG_NET_VENDOR_QUALCOMM is not set
# CONFIG_NET_VENDOR_RENESAS is not set
# CONFIG_NET_VENDOR_ROCKER is not set
# CONFIG_NET_VENDOR_SAMSUNG is not set
# CONFIG_NET_VENDOR_SEEQ is not set
# CONFIG_NET_VENDOR_SOLARFLARE is not set
CONFIG_NET_VENDOR_SMSC=y
CONFIG_SMC91X=m
# CONFIG_SMSC911X is not set
# CONFIG_NET_VENDOR_SOCIONEXT is not set
# CONFIG_NET_VENDOR_STMICRO is not set
# CONFIG_NET_VENDOR_SYNOPSYS is not set
# CONFIG_NET_VENDOR_VERTEXCOM is not set
# CONFIG_NET_VENDOR_VIA is not set
# CONFIG_NET_VENDOR_WIZNET is not set
# CONFIG_NET_VENDOR_XILINX is not set
CONFIG_PHYLIB=y
# CONFIG_FIXED_PHY is not set

#
# MII PHY device drivers
#
# CONFIG_AMD_PHY is not set
# CONFIG_ADIN_PHY is not set
# CONFIG_ADIN1100_PHY is not set
# CONFIG_AQUANTIA_PHY is not set
# CONFIG_AX88796B_PHY is not set
# CONFIG_BROADCOM_PHY is not set
# CONFIG_BCM54140_PHY is not set
# CONFIG_BCM7XXX_PHY is not set
# CONFIG_BCM84881_PHY is not set
# CONFIG_BCM87XX_PHY is not set
# CONFIG_CICADA_PHY is not set
# CONFIG_CORTINA_PHY is not set
# CONFIG_DAVICOM_PHY is not set
# CONFIG_ICPLUS_PHY is not set
# CONFIG_LXT_PHY is not set
# CONFIG_INTEL_XWAY_PHY is not set
# CONFIG_LSI_ET1011C_PHY is not set
# CONFIG_MARVELL_PHY is not set
# CONFIG_MARVELL_10G_PHY is not set
# CONFIG_MARVELL_88X2222_PHY is not set
# CONFIG_MAXLINEAR_GPHY is not set
# CONFIG_MEDIATEK_GE_PHY is not set
# CONFIG_MICREL_PHY is not set
# CONFIG_MICROCHIP_PHY is not set
# CONFIG_MICROCHIP_T1_PHY is not set
# CONFIG_MICROSEMI_PHY is not set
# CONFIG_MOTORCOMM_PHY is not set
# CONFIG_NATIONAL_PHY is not set
# CONFIG_NXP_C45_TJA11XX_PHY is not set
# CONFIG_NCN26000_PHY is not set
# CONFIG_QSEMI_PHY is not set
# CONFIG_REALTEK_PHY is not set
# CONFIG_RENESAS_PHY is not set
# CONFIG_ROCKCHIP_PHY is not set
# CONFIG_SMSC_PHY is not set
# CONFIG_STE10XP is not set
# CONFIG_TERANETICS_PHY is not set
# CONFIG_DP83822_PHY is not set
# CONFIG_DP83TC811_PHY is not set
# CONFIG_DP83848_PHY is not set
# CONFIG_DP83867_PHY is not set
# CONFIG_DP83869_PHY is not set
# CONFIG_DP83TD510_PHY is not set
# CONFIG_VITESSE_PHY is not set
# CONFIG_XILINX_GMII2RGMII is not set
# CONFIG_PSE_CONTROLLER is not set
CONFIG_MDIO_DEVICE=y
CONFIG_MDIO_BUS=y
CONFIG_MDIO_DEVRES=y
# CONFIG_MDIO_BITBANG is not set
# CONFIG_MDIO_BCM_UNIMAC is not set

#
# MDIO Multiplexers
#

#
# PCS device drivers
#
# end of PCS device drivers

CONFIG_PPP=m
CONFIG_PPP_BSDCOMP=m
CONFIG_PPP_DEFLATE=m
# CONFIG_PPP_FILTER is not set
# CONFIG_PPP_MPPE is not set
# CONFIG_PPP_MULTILINK is not set
# CONFIG_PPPOE is not set
CONFIG_PPP_ASYNC=m
# CONFIG_PPP_SYNC_TTY is not set
# CONFIG_SLIP is not set
CONFIG_SLHC=m

#
# Host-side USB support is needed for USB Network Adapter support
#
# CONFIG_WLAN is not set
# CONFIG_WAN is not set

#
# Wireless WAN
#
# CONFIG_WWAN is not set
# end of Wireless WAN

# CONFIG_NET_FAILOVER is not set
# CONFIG_ISDN is not set

#
# Input device support
#
CONFIG_INPUT=y
# CONFIG_INPUT_FF_MEMLESS is not set
# CONFIG_INPUT_SPARSEKMAP is not set
# CONFIG_INPUT_MATRIXKMAP is not set

#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=800
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=600
# CONFIG_INPUT_JOYDEV is not set
CONFIG_INPUT_EVDEV=y
# CONFIG_INPUT_EVBUG is not set

#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATARI=y
# CONFIG_KEYBOARD_ATKBD is not set
# CONFIG_KEYBOARD_LKKBD is not set
# CONFIG_KEYBOARD_NEWTON is not set
# CONFIG_KEYBOARD_OPENCORES is not set
CONFIG_KEYBOARD_GOLDFISH_EVENTS=y
# CONFIG_KEYBOARD_STOWAWAY is not set
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_XTKBD is not set
CONFIG_INPUT_MOUSE=y
# CONFIG_MOUSE_PS2 is not set
# CONFIG_MOUSE_SERIAL is not set
CONFIG_MOUSE_ATARI=m
# CONFIG_MOUSE_VSXXXAA is not set
# CONFIG_INPUT_JOYSTICK is not set
# CONFIG_INPUT_TABLET is not set
# CONFIG_INPUT_TOUCHSCREEN is not set
# CONFIG_INPUT_MISC is not set
# CONFIG_RMI4_CORE is not set

#
# Hardware I/O ports
#
# CONFIG_SERIO is not set
# CONFIG_GAMEPORT is not set
# end of Hardware I/O ports
# end of Input device support

#
# Character devices
#
CONFIG_TTY=y
CONFIG_VT=y
CONFIG_CONSOLE_TRANSLATIONS=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
CONFIG_VT_HW_CONSOLE_BINDING=y
CONFIG_UNIX98_PTYS=y
# CONFIG_LEGACY_PTYS is not set
# CONFIG_LEGACY_TIOCSTI is not set
CONFIG_LDISC_AUTOLOAD=y

#
# Serial drivers
#
CONFIG_SERIAL_EARLYCON=y
# CONFIG_SERIAL_8250 is not set

#
# Non-8250 serial port support
#
# CONFIG_SERIAL_UARTLITE is not set
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
CONFIG_SERIAL_PMACZILOG=y
CONFIG_SERIAL_PMACZILOG_TTYS=y
CONFIG_SERIAL_PMACZILOG_CONSOLE=y
# CONFIG_SERIAL_SCCNXP is not set
# CONFIG_SERIAL_ALTERA_JTAGUART is not set
# CONFIG_SERIAL_ALTERA_UART is not set
# CONFIG_SERIAL_ARC is not set
# CONFIG_SERIAL_FSL_LPUART is not set
# CONFIG_SERIAL_FSL_LINFLEXUART is not set
# end of Serial drivers

# CONFIG_SERIAL_NONSTANDARD is not set
CONFIG_GOLDFISH_TTY=y
CONFIG_GOLDFISH_TTY_EARLY_CONSOLE=y
# CONFIG_N_GSM is not set
# CONFIG_NULL_TTY is not set
# CONFIG_SERIAL_DEV_BUS is not set
# CONFIG_TTY_PRINTK is not set
# CONFIG_VIRTIO_CONSOLE is not set
# CONFIG_IPMI_HANDLER is not set
# CONFIG_HW_RANDOM is not set
CONFIG_DEVMEM=y
CONFIG_NVRAM=y
# CONFIG_TCG_TPM is not set
# end of Character devices

#
# I2C support
#
# CONFIG_I2C is not set
# end of I2C support

# CONFIG_I3C is not set
# CONFIG_SPI is not set
# CONFIG_SPMI is not set
# CONFIG_HSI is not set
# CONFIG_PPS is not set

#
# PTP clock support
#
# CONFIG_PTP_1588_CLOCK is not set
CONFIG_PTP_1588_CLOCK_OPTIONAL=y

#
# Enable PHYLIB and NETWORK_PHY_TIMESTAMPING to see the additional clocks.
#
# end of PTP clock support

# CONFIG_PINCTRL is not set
# CONFIG_GPIOLIB is not set
# CONFIG_W1 is not set
# CONFIG_POWER_RESET is not set
# CONFIG_POWER_SUPPLY is not set
# CONFIG_HWMON is not set
# CONFIG_THERMAL is not set
# CONFIG_WATCHDOG is not set
CONFIG_SSB_POSSIBLE=y
# CONFIG_SSB is not set
CONFIG_BCMA_POSSIBLE=y
# CONFIG_BCMA is not set

#
# Multifunction device drivers
#
# CONFIG_MFD_MADERA is not set
# CONFIG_MFD_KEMPLD is not set
# CONFIG_MFD_MT6397 is not set
# CONFIG_MFD_SM501 is not set
# CONFIG_MFD_SYSCON is not set
# CONFIG_MFD_TI_AM335X_TSCADC is not set
# CONFIG_MFD_TQMX86 is not set
# end of Multifunction device drivers

# CONFIG_REGULATOR is not set
# CONFIG_RC_CORE is not set

#
# CEC support
#
# CONFIG_MEDIA_CEC_SUPPORT is not set
# end of CEC support

# CONFIG_MEDIA_SUPPORT is not set

#
# Graphics support
#
# CONFIG_DRM is not set

#
# ARM devices
#
# end of ARM devices

#
# Frame buffer Devices
#
CONFIG_FB_CMDLINE=y
CONFIG_FB_NOTIFY=y
CONFIG_FB=y
# CONFIG_FIRMWARE_EDID is not set
CONFIG_FB_CFB_FILLRECT=y
CONFIG_FB_CFB_COPYAREA=y
CONFIG_FB_CFB_IMAGEBLIT=y
# CONFIG_FB_FOREIGN_ENDIAN is not set
CONFIG_FB_MACMODES=y
# CONFIG_FB_MODE_HELPERS is not set
# CONFIG_FB_TILEBLITTING is not set

#
# Frame buffer hardware drivers
#
CONFIG_FB_ATARI=y
CONFIG_FB_VALKYRIE=y
CONFIG_FB_MAC=y
# CONFIG_FB_OPENCORES is not set
# CONFIG_FB_S1D13XXX is not set
# CONFIG_FB_ATY is not set
# CONFIG_FB_IBM_GXT4500 is not set
CONFIG_FB_GOLDFISH=y
# CONFIG_FB_VIRTUAL is not set
# CONFIG_FB_METRONOME is not set
# CONFIG_FB_SIMPLE is not set
# end of Frame buffer Devices

#
# Backlight & LCD device support
#
# CONFIG_LCD_CLASS_DEVICE is not set
# CONFIG_BACKLIGHT_CLASS_DEVICE is not set
# end of Backlight & LCD device support

#
# Console display driver support
#
CONFIG_DUMMY_CONSOLE=y
CONFIG_DUMMY_CONSOLE_COLUMNS=80
CONFIG_DUMMY_CONSOLE_ROWS=25
CONFIG_FRAMEBUFFER_CONSOLE=y
# CONFIG_FRAMEBUFFER_CONSOLE_LEGACY_ACCELERATION is not set
# CONFIG_FRAMEBUFFER_CONSOLE_DETECT_PRIMARY is not set
# CONFIG_FRAMEBUFFER_CONSOLE_ROTATION is not set
# CONFIG_FRAMEBUFFER_CONSOLE_DEFERRED_TAKEOVER is not set
# end of Console display driver support

# CONFIG_LOGO is not set
# end of Graphics support

# CONFIG_SOUND is not set
CONFIG_HID_SUPPORT=y
CONFIG_HID=m
# CONFIG_HID_BATTERY_STRENGTH is not set
CONFIG_HIDRAW=y
# CONFIG_UHID is not set
CONFIG_HID_GENERIC=m

#
# Special HID drivers
#
# CONFIG_HID_A4TECH is not set
# CONFIG_HID_ACRUX is not set
# CONFIG_HID_AUREAL is not set
# CONFIG_HID_BELKIN is not set
# CONFIG_HID_CHERRY is not set
# CONFIG_HID_COUGAR is not set
# CONFIG_HID_MACALLY is not set
# CONFIG_HID_CMEDIA is not set
# CONFIG_HID_CYPRESS is not set
# CONFIG_HID_DRAGONRISE is not set
# CONFIG_HID_EMS_FF is not set
# CONFIG_HID_ELECOM is not set
# CONFIG_HID_EVISION is not set
# CONFIG_HID_EZKEY is not set
# CONFIG_HID_GEMBIRD is not set
# CONFIG_HID_GFRM is not set
# CONFIG_HID_GLORIOUS is not set
# CONFIG_HID_VIVALDI is not set
# CONFIG_HID_KEYTOUCH is not set
# CONFIG_HID_KYE is not set
# CONFIG_HID_WALTOP is not set
# CONFIG_HID_VIEWSONIC is not set
# CONFIG_HID_VRC2 is not set
# CONFIG_HID_XIAOMI is not set
# CONFIG_HID_GYRATION is not set
# CONFIG_HID_ICADE is not set
# CONFIG_HID_ITE is not set
# CONFIG_HID_JABRA is not set
# CONFIG_HID_TWINHAN is not set
# CONFIG_HID_KENSINGTON is not set
# CONFIG_HID_LCPOWER is not set
# CONFIG_HID_LENOVO is not set
# CONFIG_HID_MAGICMOUSE is not set
# CONFIG_HID_MALTRON is not set
# CONFIG_HID_MAYFLASH is not set
# CONFIG_HID_REDRAGON is not set
# CONFIG_HID_MICROSOFT is not set
# CONFIG_HID_MONTEREY is not set
# CONFIG_HID_MULTITOUCH is not set
# CONFIG_HID_NTI is not set
# CONFIG_HID_ORTEK is not set
# CONFIG_HID_PANTHERLORD is not set
# CONFIG_HID_PETALYNX is not set
# CONFIG_HID_PICOLCD is not set
# CONFIG_HID_PLANTRONICS is not set
# CONFIG_HID_PXRC is not set
# CONFIG_HID_RAZER is not set
# CONFIG_HID_PRIMAX is not set
# CONFIG_HID_SAITEK is not set
# CONFIG_HID_SEMITEK is not set
# CONFIG_HID_SPEEDLINK is not set
# CONFIG_HID_STEAM is not set
# CONFIG_HID_STEELSERIES is not set
# CONFIG_HID_SUNPLUS is not set
# CONFIG_HID_RMI is not set
# CONFIG_HID_GREENASIA is not set
# CONFIG_HID_SMARTJOYPLUS is not set
# CONFIG_HID_TIVO is not set
# CONFIG_HID_TOPSEED is not set
# CONFIG_HID_TOPRE is not set
# CONFIG_HID_UDRAW_PS3 is not set
# CONFIG_HID_XINMO is not set
# CONFIG_HID_ZEROPLUS is not set
# CONFIG_HID_ZYDACRON is not set
# CONFIG_HID_SENSOR_HUB is not set
# CONFIG_HID_ALPS is not set
# end of Special HID drivers

#
# HID-BPF support
#
# end of HID-BPF support

CONFIG_USB_OHCI_LITTLE_ENDIAN=y
# CONFIG_USB_SUPPORT is not set
# CONFIG_MMC is not set
# CONFIG_SCSI_UFSHCD is not set
# CONFIG_MEMSTICK is not set
# CONFIG_NEW_LEDS is not set
# CONFIG_ACCESSIBILITY is not set
# CONFIG_INFINIBAND is not set
CONFIG_RTC_LIB=y
CONFIG_RTC_CLASS=y
# CONFIG_RTC_HCTOSYS is not set
# CONFIG_RTC_SYSTOHC is not set
# CONFIG_RTC_DEBUG is not set
# CONFIG_RTC_NVMEM is not set

#
# RTC interfaces
#
# CONFIG_RTC_INTF_SYSFS is not set
# CONFIG_RTC_INTF_PROC is not set
CONFIG_RTC_INTF_DEV=y
# CONFIG_RTC_INTF_DEV_UIE_EMUL is not set
# CONFIG_RTC_DRV_TEST is not set

#
# I2C RTC drivers
#

#
# SPI RTC drivers
#

#
# SPI and I2C RTC drivers
#

#
# Platform RTC drivers
#
# CONFIG_RTC_DRV_DS1286 is not set
# CONFIG_RTC_DRV_DS1511 is not set
# CONFIG_RTC_DRV_DS1553 is not set
# CONFIG_RTC_DRV_DS1685_FAMILY is not set
# CONFIG_RTC_DRV_DS1742 is not set
# CONFIG_RTC_DRV_DS2404 is not set
# CONFIG_RTC_DRV_STK17TA8 is not set
# CONFIG_RTC_DRV_M48T86 is not set
# CONFIG_RTC_DRV_M48T35 is not set
CONFIG_RTC_DRV_M48T59=y
# CONFIG_RTC_DRV_MSM6242 is not set
# CONFIG_RTC_DRV_BQ4802 is not set
# CONFIG_RTC_DRV_RP5C01 is not set

#
# on-CPU RTC drivers
#
CONFIG_RTC_DRV_GENERIC=y
# CONFIG_RTC_DRV_FTRTC010 is not set

#
# HID Sensor RTC drivers
#
CONFIG_RTC_DRV_GOLDFISH=y
# CONFIG_DMADEVICES is not set

#
# DMABUF options
#
# CONFIG_SYNC_FILE is not set
# CONFIG_DMABUF_HEAPS is not set
# end of DMABUF options

# CONFIG_AUXDISPLAY is not set
# CONFIG_UIO is not set
# CONFIG_VFIO is not set
# CONFIG_VIRT_DRIVERS is not set
CONFIG_VIRTIO_ANCHOR=y
CONFIG_VIRTIO=y
CONFIG_VIRTIO_MENU=y
# CONFIG_VIRTIO_BALLOON is not set
CONFIG_VIRTIO_INPUT=y
CONFIG_VIRTIO_MMIO=y
# CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES is not set
# CONFIG_VDPA is not set
# CONFIG_VHOST_MENU is not set

#
# Microsoft Hyper-V guest support
#
# end of Microsoft Hyper-V guest support

# CONFIG_GREYBUS is not set
# CONFIG_COMEDI is not set
# CONFIG_STAGING is not set
CONFIG_GOLDFISH=y
CONFIG_GOLDFISH_PIPE=y
# CONFIG_COMMON_CLK is not set
# CONFIG_HWSPINLOCK is not set

#
# Clock Source drivers
#
CONFIG_GOLDFISH_TIMER=y
# end of Clock Source drivers

# CONFIG_MAILBOX is not set
# CONFIG_IOMMU_SUPPORT is not set

#
# Remoteproc drivers
#
# CONFIG_REMOTEPROC is not set
# end of Remoteproc drivers

#
# Rpmsg drivers
#
# CONFIG_RPMSG_VIRTIO is not set
# end of Rpmsg drivers

#
# SOC (System On Chip) specific Drivers
#

#
# Amlogic SoC drivers
#
# end of Amlogic SoC drivers

#
# Broadcom SoC drivers
#
# end of Broadcom SoC drivers

#
# NXP/Freescale QorIQ SoC drivers
#
# end of NXP/Freescale QorIQ SoC drivers

#
# fujitsu SoC drivers
#
# end of fujitsu SoC drivers

#
# i.MX SoC drivers
#
# end of i.MX SoC drivers

#
# Enable LiteX SoC Builder specific drivers
#
# end of Enable LiteX SoC Builder specific drivers

# CONFIG_WPCM450_SOC is not set

#
# Qualcomm SoC drivers
#
# end of Qualcomm SoC drivers

# CONFIG_SOC_TI is not set

#
# Xilinx SoC drivers
#
# end of Xilinx SoC drivers
# end of SOC (System On Chip) specific Drivers

# CONFIG_PM_DEVFREQ is not set
# CONFIG_EXTCON is not set
# CONFIG_MEMORY is not set
# CONFIG_IIO is not set
# CONFIG_PWM is not set

#
# IRQ chip support
#
# end of IRQ chip support

# CONFIG_IPACK_BUS is not set
# CONFIG_RESET_CONTROLLER is not set

#
# PHY Subsystem
#
# CONFIG_GENERIC_PHY is not set
# CONFIG_PHY_CAN_TRANSCEIVER is not set

#
# PHY drivers for Broadcom platforms
#
# CONFIG_BCM_KONA_USB2_PHY is not set
# end of PHY drivers for Broadcom platforms

# CONFIG_PHY_PXA_28NM_HSIC is not set
# CONFIG_PHY_PXA_28NM_USB2 is not set
# end of PHY Subsystem

# CONFIG_POWERCAP is not set
# CONFIG_MCB is not set
# CONFIG_RAS is not set

#
# Android
#
# CONFIG_ANDROID_BINDER_IPC is not set
# end of Android

# CONFIG_DAX is not set
# CONFIG_NVMEM is not set

#
# HW tracing support
#
# CONFIG_STM is not set
# CONFIG_INTEL_TH is not set
# end of HW tracing support

# CONFIG_FPGA is not set
# CONFIG_SIOX is not set
# CONFIG_SLIMBUS is not set
# CONFIG_INTERCONNECT is not set
# CONFIG_COUNTER is not set
# CONFIG_PECI is not set
# CONFIG_HTE is not set
# end of Device Drivers

#
# File systems
#
# CONFIG_VALIDATE_FS_PARSER is not set
CONFIG_FS_IOMAP=y
CONFIG_LEGACY_DIRECT_IO=y
# CONFIG_EXT2_FS is not set
# CONFIG_EXT3_FS is not set
CONFIG_EXT4_FS=y
CONFIG_EXT4_USE_FOR_EXT2=y
CONFIG_EXT4_FS_POSIX_ACL=y
# CONFIG_EXT4_FS_SECURITY is not set
# CONFIG_EXT4_DEBUG is not set
CONFIG_JBD2=y
# CONFIG_JBD2_DEBUG is not set
CONFIG_FS_MBCACHE=y
# CONFIG_REISERFS_FS is not set
# CONFIG_JFS_FS is not set
# CONFIG_XFS_FS is not set
# CONFIG_GFS2_FS is not set
# CONFIG_BTRFS_FS is not set
# CONFIG_NILFS2_FS is not set
# CONFIG_F2FS_FS is not set
CONFIG_FS_POSIX_ACL=y
CONFIG_EXPORTFS=y
# CONFIG_EXPORTFS_BLOCK_OPS is not set
CONFIG_FILE_LOCKING=y
# CONFIG_FS_ENCRYPTION is not set
# CONFIG_FS_VERITY is not set
CONFIG_FSNOTIFY=y
# CONFIG_DNOTIFY is not set
CONFIG_INOTIFY_USER=y
CONFIG_FANOTIFY=y
# CONFIG_QUOTA is not set
CONFIG_AUTOFS4_FS=m
CONFIG_AUTOFS_FS=m
# CONFIG_FUSE_FS is not set
# CONFIG_OVERLAY_FS is not set

#
# Caches
#
# CONFIG_FSCACHE is not set
# end of Caches

#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=y
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_UDF_FS=m
# end of CD-ROM/DVD Filesystems

#
# DOS/FAT/EXFAT/NT Filesystems
#
CONFIG_FAT_FS=m
CONFIG_MSDOS_FS=m
CONFIG_VFAT_FS=m
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
# CONFIG_FAT_DEFAULT_UTF8 is not set
CONFIG_EXFAT_FS=m
CONFIG_EXFAT_DEFAULT_IOCHARSET="utf8"
# CONFIG_NTFS_FS is not set
CONFIG_NTFS3_FS=m
# CONFIG_NTFS3_LZX_XPRESS is not set
CONFIG_NTFS3_FS_POSIX_ACL=y
# end of DOS/FAT/EXFAT/NT Filesystems

#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
# CONFIG_PROC_KCORE is not set
CONFIG_PROC_SYSCTL=y
CONFIG_PROC_PAGE_MONITOR=y
# CONFIG_PROC_CHILDREN is not set
CONFIG_KERNFS=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
CONFIG_TMPFS_POSIX_ACL=y
CONFIG_TMPFS_XATTR=y
CONFIG_MEMFD_CREATE=y
# CONFIG_CONFIGFS_FS is not set
# end of Pseudo filesystems

CONFIG_MISC_FILESYSTEMS=y
# CONFIG_ORANGEFS_FS is not set
# CONFIG_ADFS_FS is not set
# CONFIG_AFFS_FS is not set
CONFIG_HFS_FS=y
CONFIG_HFSPLUS_FS=m
# CONFIG_BEFS_FS is not set
# CONFIG_BFS_FS is not set
# CONFIG_EFS_FS is not set
# CONFIG_CRAMFS is not set
# CONFIG_SQUASHFS is not set
# CONFIG_VXFS_FS is not set
# CONFIG_MINIX_FS is not set
# CONFIG_OMFS_FS is not set
# CONFIG_HPFS_FS is not set
# CONFIG_QNX4FS_FS is not set
# CONFIG_QNX6FS_FS is not set
# CONFIG_ROMFS_FS is not set
# CONFIG_PSTORE is not set
# CONFIG_SYSV_FS is not set
CONFIG_UFS_FS=m
CONFIG_UFS_FS_WRITE=y
# CONFIG_UFS_DEBUG is not set
# CONFIG_EROFS_FS is not set
CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_NFS_FS=y
CONFIG_NFS_V2=y
CONFIG_NFS_V3=y
# CONFIG_NFS_V3_ACL is not set
# CONFIG_NFS_V4 is not set
# CONFIG_NFS_SWAP is not set
CONFIG_ROOT_NFS=y
# CONFIG_NFS_DISABLE_UDP_SUPPORT is not set
# CONFIG_NFSD is not set
CONFIG_GRACE_PERIOD=y
CONFIG_LOCKD=y
CONFIG_LOCKD_V4=y
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=y
# CONFIG_RPCSEC_GSS_KRB5 is not set
# CONFIG_SUNRPC_DEBUG is not set
# CONFIG_CEPH_FS is not set
# CONFIG_CIFS is not set
# CONFIG_SMB_SERVER is not set
# CONFIG_CODA_FS is not set
# CONFIG_AFS_FS is not set
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-1"
CONFIG_NLS_CODEPAGE_437=y
CONFIG_NLS_CODEPAGE_737=m
CONFIG_NLS_CODEPAGE_775=m
CONFIG_NLS_CODEPAGE_850=m
CONFIG_NLS_CODEPAGE_852=m
CONFIG_NLS_CODEPAGE_855=m
CONFIG_NLS_CODEPAGE_857=m
CONFIG_NLS_CODEPAGE_860=m
CONFIG_NLS_CODEPAGE_861=m
CONFIG_NLS_CODEPAGE_862=m
CONFIG_NLS_CODEPAGE_863=m
CONFIG_NLS_CODEPAGE_864=m
CONFIG_NLS_CODEPAGE_865=m
CONFIG_NLS_CODEPAGE_866=m
CONFIG_NLS_CODEPAGE_869=m
CONFIG_NLS_CODEPAGE_936=m
CONFIG_NLS_CODEPAGE_950=m
CONFIG_NLS_CODEPAGE_932=m
CONFIG_NLS_CODEPAGE_949=m
CONFIG_NLS_CODEPAGE_874=m
CONFIG_NLS_ISO8859_8=m
CONFIG_NLS_CODEPAGE_1250=m
CONFIG_NLS_CODEPAGE_1251=m
CONFIG_NLS_ASCII=y
CONFIG_NLS_ISO8859_1=y
CONFIG_NLS_ISO8859_2=m
CONFIG_NLS_ISO8859_3=m
CONFIG_NLS_ISO8859_4=m
CONFIG_NLS_ISO8859_5=m
CONFIG_NLS_ISO8859_6=m
CONFIG_NLS_ISO8859_7=m
CONFIG_NLS_ISO8859_9=m
CONFIG_NLS_ISO8859_13=m
CONFIG_NLS_ISO8859_14=m
CONFIG_NLS_ISO8859_15=m
CONFIG_NLS_KOI8_R=m
CONFIG_NLS_KOI8_U=m
CONFIG_NLS_MAC_ROMAN=y
CONFIG_NLS_MAC_CELTIC=m
CONFIG_NLS_MAC_CENTEURO=m
CONFIG_NLS_MAC_CROATIAN=m
CONFIG_NLS_MAC_CYRILLIC=m
CONFIG_NLS_MAC_GAELIC=m
CONFIG_NLS_MAC_GREEK=m
CONFIG_NLS_MAC_ICELAND=m
CONFIG_NLS_MAC_INUIT=m
CONFIG_NLS_MAC_ROMANIAN=m
CONFIG_NLS_MAC_TURKISH=m
CONFIG_NLS_UTF8=y
# CONFIG_UNICODE is not set
# end of File systems

#
# Security options
#
# CONFIG_KEYS is not set
# CONFIG_SECURITY_DMESG_RESTRICT is not set
# CONFIG_SECURITY is not set
# CONFIG_SECURITYFS is not set
CONFIG_HAVE_HARDENED_USERCOPY_ALLOCATOR=y
# CONFIG_HARDENED_USERCOPY is not set
# CONFIG_STATIC_USERMODEHELPER is not set
CONFIG_DEFAULT_SECURITY_DAC=y
CONFIG_LSM="yama,loadpin,safesetid,integrity"

#
# Kernel hardening options
#

#
# Memory initialization
#
CONFIG_CC_HAS_AUTO_VAR_INIT_PATTERN=y
CONFIG_CC_HAS_AUTO_VAR_INIT_ZERO_BARE=y
CONFIG_CC_HAS_AUTO_VAR_INIT_ZERO=y
CONFIG_INIT_STACK_NONE=y
# CONFIG_INIT_STACK_ALL_PATTERN is not set
# CONFIG_INIT_STACK_ALL_ZERO is not set
# CONFIG_INIT_ON_ALLOC_DEFAULT_ON is not set
# CONFIG_INIT_ON_FREE_DEFAULT_ON is not set
CONFIG_CC_HAS_ZERO_CALL_USED_REGS=y
# CONFIG_ZERO_CALL_USED_REGS is not set
# end of Memory initialization

CONFIG_RANDSTRUCT_NONE=y
# end of Kernel hardening options
# end of Security options

CONFIG_CRYPTO=y

#
# Crypto core or helper
#
CONFIG_CRYPTO_ALGAPI=y
CONFIG_CRYPTO_ALGAPI2=y
CONFIG_CRYPTO_HASH=y
CONFIG_CRYPTO_HASH2=y
CONFIG_CRYPTO_RNG=m
CONFIG_CRYPTO_RNG2=m
# CONFIG_CRYPTO_MANAGER is not set
# CONFIG_CRYPTO_USER is not set
CONFIG_CRYPTO_MANAGER_DISABLE_TESTS=y
# CONFIG_CRYPTO_NULL is not set
# CONFIG_CRYPTO_CRYPTD is not set
# CONFIG_CRYPTO_AUTHENC is not set
# CONFIG_CRYPTO_TEST is not set
# end of Crypto core or helper

#
# Public-key cryptography
#
# CONFIG_CRYPTO_RSA is not set
# CONFIG_CRYPTO_DH is not set
# CONFIG_CRYPTO_ECDH is not set
# CONFIG_CRYPTO_ECDSA is not set
# CONFIG_CRYPTO_ECRDSA is not set
# CONFIG_CRYPTO_SM2 is not set
# CONFIG_CRYPTO_CURVE25519 is not set
# end of Public-key cryptography

#
# Block ciphers
#
CONFIG_CRYPTO_AES=y
# CONFIG_CRYPTO_AES_TI is not set
# CONFIG_CRYPTO_ARIA is not set
# CONFIG_CRYPTO_BLOWFISH is not set
# CONFIG_CRYPTO_CAMELLIA is not set
# CONFIG_CRYPTO_CAST5 is not set
# CONFIG_CRYPTO_CAST6 is not set
# CONFIG_CRYPTO_DES is not set
# CONFIG_CRYPTO_FCRYPT is not set
# CONFIG_CRYPTO_SERPENT is not set
# CONFIG_CRYPTO_SM4_GENERIC is not set
# CONFIG_CRYPTO_TWOFISH is not set
# end of Block ciphers

#
# Length-preserving ciphers and modes
#
# CONFIG_CRYPTO_ADIANTUM is not set
# CONFIG_CRYPTO_CHACHA20 is not set
# CONFIG_CRYPTO_CBC is not set
# CONFIG_CRYPTO_CFB is not set
# CONFIG_CRYPTO_CTR is not set
# CONFIG_CRYPTO_CTS is not set
# CONFIG_CRYPTO_ECB is not set
# CONFIG_CRYPTO_HCTR2 is not set
# CONFIG_CRYPTO_KEYWRAP is not set
# CONFIG_CRYPTO_LRW is not set
# CONFIG_CRYPTO_OFB is not set
# CONFIG_CRYPTO_PCBC is not set
# CONFIG_CRYPTO_XTS is not set
# end of Length-preserving ciphers and modes

#
# AEAD (authenticated encryption with associated data) ciphers
#
# CONFIG_CRYPTO_AEGIS128 is not set
# CONFIG_CRYPTO_CHACHA20POLY1305 is not set
# CONFIG_CRYPTO_CCM is not set
# CONFIG_CRYPTO_GCM is not set
# CONFIG_CRYPTO_SEQIV is not set
# CONFIG_CRYPTO_ECHAINIV is not set
# CONFIG_CRYPTO_ESSIV is not set
# end of AEAD (authenticated encryption with associated data) ciphers

#
# Hashes, digests, and MACs
#
# CONFIG_CRYPTO_BLAKE2B is not set
# CONFIG_CRYPTO_CMAC is not set
# CONFIG_CRYPTO_GHASH is not set
# CONFIG_CRYPTO_HMAC is not set
# CONFIG_CRYPTO_MD4 is not set
# CONFIG_CRYPTO_MD5 is not set
# CONFIG_CRYPTO_MICHAEL_MIC is not set
# CONFIG_CRYPTO_POLY1305 is not set
# CONFIG_CRYPTO_RMD160 is not set
# CONFIG_CRYPTO_SHA1 is not set
# CONFIG_CRYPTO_SHA256 is not set
# CONFIG_CRYPTO_SHA512 is not set
# CONFIG_CRYPTO_SHA3 is not set
# CONFIG_CRYPTO_SM3_GENERIC is not set
# CONFIG_CRYPTO_STREEBOG is not set
# CONFIG_CRYPTO_VMAC is not set
# CONFIG_CRYPTO_WP512 is not set
# CONFIG_CRYPTO_XCBC is not set
# CONFIG_CRYPTO_XXHASH is not set
# end of Hashes, digests, and MACs

#
# CRCs (cyclic redundancy checks)
#
CONFIG_CRYPTO_CRC32C=y
# CONFIG_CRYPTO_CRC32 is not set
# CONFIG_CRYPTO_CRCT10DIF is not set
# end of CRCs (cyclic redundancy checks)

#
# Compression
#
# CONFIG_CRYPTO_DEFLATE is not set
# CONFIG_CRYPTO_LZO is not set
# CONFIG_CRYPTO_842 is not set
# CONFIG_CRYPTO_LZ4 is not set
# CONFIG_CRYPTO_LZ4HC is not set
# CONFIG_CRYPTO_ZSTD is not set
# end of Compression

#
# Random number generation
#
# CONFIG_CRYPTO_ANSI_CPRNG is not set
# CONFIG_CRYPTO_DRBG_MENU is not set
CONFIG_CRYPTO_JITTERENTROPY=m
# end of Random number generation

#
# Userspace interface
#
# CONFIG_CRYPTO_USER_API_HASH is not set
# CONFIG_CRYPTO_USER_API_SKCIPHER is not set
# CONFIG_CRYPTO_USER_API_RNG is not set
# CONFIG_CRYPTO_USER_API_AEAD is not set
# end of Userspace interface

# CONFIG_CRYPTO_HW is not set

#
# Certificates for signature checking
#
# end of Certificates for signature checking

#
# Library routines
#
# CONFIG_PACKING is not set
CONFIG_BITREVERSE=y
CONFIG_GENERIC_STRNCPY_FROM_USER=y
CONFIG_GENERIC_STRNLEN_USER=y
CONFIG_GENERIC_NET_UTILS=y
# CONFIG_CORDIC is not set
# CONFIG_PRIME_NUMBERS is not set
CONFIG_GENERIC_PCI_IOMAP=y
CONFIG_GENERIC_IOMAP=y

#
# Crypto library routines
#
CONFIG_CRYPTO_LIB_UTILS=y
CONFIG_CRYPTO_LIB_AES=y
CONFIG_CRYPTO_LIB_BLAKE2S_GENERIC=y
# CONFIG_CRYPTO_LIB_CHACHA is not set
# CONFIG_CRYPTO_LIB_CURVE25519 is not set
CONFIG_CRYPTO_LIB_POLY1305_RSIZE=1
# CONFIG_CRYPTO_LIB_POLY1305 is not set
# CONFIG_CRYPTO_LIB_CHACHA20POLY1305 is not set
CONFIG_CRYPTO_LIB_SHA1=y
# end of Crypto library routines

CONFIG_CRC_CCITT=m
CONFIG_CRC16=y
# CONFIG_CRC_T10DIF is not set
# CONFIG_CRC64_ROCKSOFT is not set
CONFIG_CRC_ITU_T=m
CONFIG_CRC32=y
# CONFIG_CRC32_SELFTEST is not set
CONFIG_CRC32_SLICEBY8=y
# CONFIG_CRC32_SLICEBY4 is not set
# CONFIG_CRC32_SARWATE is not set
# CONFIG_CRC32_BIT is not set
# CONFIG_CRC64 is not set
# CONFIG_CRC4 is not set
# CONFIG_CRC7 is not set
# CONFIG_LIBCRC32C is not set
# CONFIG_CRC8 is not set
# CONFIG_RANDOM32_SELFTEST is not set
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=m
# CONFIG_XZ_DEC is not set
CONFIG_DECOMPRESS_GZIP=y
CONFIG_GENERIC_ALLOCATOR=y
CONFIG_HAS_IOMEM=y
CONFIG_HAS_DMA=y
CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE=y
CONFIG_ARCH_HAS_DMA_PREP_COHERENT=y
CONFIG_DMA_NONCOHERENT_MMAP=y
CONFIG_DMA_COHERENT_POOL=y
CONFIG_DMA_DIRECT_REMAP=y
# CONFIG_DMA_API_DEBUG is not set
CONFIG_DQL=y
CONFIG_GLOB=y
# CONFIG_GLOB_SELFTEST is not set
CONFIG_NLATTR=y
CONFIG_GENERIC_ATOMIC64=y
# CONFIG_IRQ_POLL is not set
CONFIG_FONT_SUPPORT=y
CONFIG_FONTS=y
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y
CONFIG_FONT_6x11=y
# CONFIG_FONT_7x14 is not set
# CONFIG_FONT_PEARL_8x8 is not set
# CONFIG_FONT_ACORN_8x8 is not set
# CONFIG_FONT_MINI_4x6 is not set
# CONFIG_FONT_6x10 is not set
# CONFIG_FONT_10x18 is not set
# CONFIG_FONT_SUN8x16 is not set
# CONFIG_FONT_SUN12x22 is not set
# CONFIG_FONT_TER16x32 is not set
# CONFIG_FONT_6x8 is not set
CONFIG_SG_POOL=y
CONFIG_SBITMAP=y
# end of Library routines

#
# Kernel hacking
#

#
# printk and dmesg options
#
CONFIG_PRINTK_TIME=y
# CONFIG_PRINTK_CALLER is not set
# CONFIG_STACKTRACE_BUILD_ID is not set
CONFIG_CONSOLE_LOGLEVEL_DEFAULT=7
CONFIG_CONSOLE_LOGLEVEL_QUIET=4
CONFIG_MESSAGE_LOGLEVEL_DEFAULT=5
# CONFIG_BOOT_PRINTK_DELAY is not set
# CONFIG_DYNAMIC_DEBUG is not set
# CONFIG_DYNAMIC_DEBUG_CORE is not set
# CONFIG_SYMBOLIC_ERRNAME is not set
CONFIG_DEBUG_BUGVERBOSE=y
# end of printk and dmesg options

CONFIG_DEBUG_KERNEL=y
# CONFIG_DEBUG_MISC is not set

#
# Compile-time checks and compiler options
#
CONFIG_AS_HAS_NON_CONST_LEB128=y
CONFIG_DEBUG_INFO_NONE=y
# CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT is not set
# CONFIG_DEBUG_INFO_DWARF4 is not set
# CONFIG_DEBUG_INFO_DWARF5 is not set
CONFIG_FRAME_WARN=1024
CONFIG_STRIP_ASM_SYMS=y
# CONFIG_READABLE_ASM is not set
# CONFIG_HEADERS_INSTALL is not set
# CONFIG_DEBUG_SECTION_MISMATCH is not set
CONFIG_SECTION_MISMATCH_WARN_ONLY=y
CONFIG_FRAME_POINTER=y
CONFIG_VMLINUX_MAP=y
# CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set
# end of Compile-time checks and compiler options

#
# Generic Kernel Debugging Instruments
#
CONFIG_MAGIC_SYSRQ=y
CONFIG_MAGIC_SYSRQ_DEFAULT_ENABLE=0x1
CONFIG_MAGIC_SYSRQ_SERIAL=y
CONFIG_MAGIC_SYSRQ_SERIAL_SEQUENCE=""
# CONFIG_DEBUG_FS is not set
# CONFIG_UBSAN is not set
CONFIG_HAVE_KCSAN_COMPILER=y
# end of Generic Kernel Debugging Instruments

#
# Networking Debugging
#
# CONFIG_DEBUG_NET is not set
# end of Networking Debugging

#
# Memory Debugging
#
# CONFIG_PAGE_EXTENSION is not set
# CONFIG_DEBUG_PAGEALLOC is not set
# CONFIG_SLUB_DEBUG is not set
# CONFIG_PAGE_POISONING is not set
# CONFIG_DEBUG_OBJECTS is not set
# CONFIG_DEBUG_STACK_USAGE is not set
# CONFIG_SCHED_STACK_END_CHECK is not set
# CONFIG_DEBUG_VM is not set
# CONFIG_DEBUG_MEMORY_INIT is not set
CONFIG_CC_HAS_WORKING_NOSANITIZE_ADDRESS=y
# end of Memory Debugging

# CONFIG_DEBUG_SHIRQ is not set

#
# Debug Oops, Lockups and Hangs
#
CONFIG_PANIC_ON_OOPS=y
CONFIG_PANIC_ON_OOPS_VALUE=1
CONFIG_PANIC_TIMEOUT=0
CONFIG_LOCKUP_DETECTOR=y
CONFIG_SOFTLOCKUP_DETECTOR=y
# CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC is not set
CONFIG_DETECT_HUNG_TASK=y
CONFIG_DEFAULT_HUNG_TASK_TIMEOUT=60
# CONFIG_BOOTPARAM_HUNG_TASK_PANIC is not set
# CONFIG_WQ_WATCHDOG is not set
# CONFIG_TEST_LOCKUP is not set
# end of Debug Oops, Lockups and Hangs

#
# Scheduler Debugging
#
# CONFIG_SCHED_DEBUG is not set
# CONFIG_SCHEDSTATS is not set
# end of Scheduler Debugging

# CONFIG_DEBUG_TIMEKEEPING is not set

#
# Lock Debugging (spinlocks, mutexes, etc...)
#
# CONFIG_DEBUG_RT_MUTEXES is not set
# CONFIG_DEBUG_SPINLOCK is not set
# CONFIG_DEBUG_MUTEXES is not set
# CONFIG_DEBUG_RWSEMS is not set
# CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
# CONFIG_LOCK_TORTURE_TEST is not set
# CONFIG_WW_MUTEX_SELFTEST is not set
# CONFIG_SCF_TORTURE_TEST is not set
# end of Lock Debugging (spinlocks, mutexes, etc...)

# CONFIG_DEBUG_IRQFLAGS is not set
# CONFIG_WARN_ALL_UNSEEDED_RANDOM is not set
# CONFIG_DEBUG_KOBJECT is not set
CONFIG_HAVE_DEBUG_BUGVERBOSE=y

#
# Debug kernel data structures
#
# CONFIG_DEBUG_LIST is not set
# CONFIG_DEBUG_PLIST is not set
# CONFIG_DEBUG_SG is not set
# CONFIG_DEBUG_NOTIFIERS is not set
# CONFIG_BUG_ON_DATA_CORRUPTION is not set
# CONFIG_DEBUG_MAPLE_TREE is not set
# end of Debug kernel data structures

# CONFIG_DEBUG_CREDENTIALS is not set

#
# RCU Debugging
#
# CONFIG_RCU_SCALE_TEST is not set
# CONFIG_RCU_TORTURE_TEST is not set
# CONFIG_RCU_REF_SCALE_TEST is not set
# CONFIG_RCU_TRACE is not set
# CONFIG_RCU_EQS_DEBUG is not set
# end of RCU Debugging

# CONFIG_DEBUG_WQ_FORCE_RR_CPU is not set
# CONFIG_SAMPLES is not set

#
# m68k Debugging
#
# CONFIG_BOOTPARAM is not set
CONFIG_EARLY_PRINTK=y
# end of m68k Debugging

#
# Kernel Testing and Coverage
#
# CONFIG_KUNIT is not set
# CONFIG_NOTIFIER_ERROR_INJECTION is not set
# CONFIG_FAULT_INJECTION is not set
CONFIG_CC_HAS_SANCOV_TRACE_PC=y
# CONFIG_RUNTIME_TESTING_MENU is not set
CONFIG_ARCH_USE_MEMTEST=y
CONFIG_MEMTEST=y
# end of Kernel Testing and Coverage

#
# Rust hacking
#
# end of Rust hacking
# end of Kernel hacking

Re: reliable reproducer, was Re: core dump analysis

[Michael Schmitz]

Hi Finn,

On 21/04/23 13:15, Finn Thain wrote:
> Fri, 21 Apr 2023 11:15:22 +1000 (AEST)n Thu, 20 Apr 2023, Michael Schmitz wrote:
>
>> In my tests, increasing the depth does not cause a monotonous increase
>> in fault probability. 16k depth only has four crashes, 8k had nine. I'll
>> stick with 200000 for now.
>>
> My tests used 'norandmaps' in the kernel parameters. With the attached
> .config (which supports Mac and Atari) I saw 12 failures out of 16 tests
> at depth 5000.

Right - I' make sure to add that in future tests.

I'll also add your patch to bus_error030 in my builds.

Cheers,

     Michael

Re: reliable reproducer, was Re: core dump analysis

[Finn Thain]

On Thu, 20 Apr 2023, Finn Thain wrote:

> 
> I modified the test program to execute rec() to full depth with no 
> forking, then do it again with forking.
> 
> root@(none):/root# while ./stack-test 5000 ; do : ; done
> starting recursion
> done.
> starting recursion with fork
> done.
> starting recursion
> done.
> starting recursion with fork
> Illegal instruction
> root@(none):/root# 
> 
> I can't get this to crash during the first descent. The second descent 
> always crashes, given sufficient depth:
> 
> root@(none):/root# while ./stack-test 50000 ; do : ; done
> starting recursion
> done.
> starting recursion with fork
> Illegal instruction
> 
> So all the stack pages would have been faulted in well before the 
> failure shows up. It appears to be the signal that's the problem and not 
> the page fault. 

Sorry, I was mistaken when I infered the absence of bus errors. I should 
have enabled ignore_loglevel and checked the printk output from my patched 
bus_error030():

# ./stack-test 5000
[   47.430000] bus_error030: page fault: pc 80000642, addr 80072fe0
[   47.440000] bus_error030: descriptor address 005a3dc8, contents 00000000
[   47.460000] bus_error030: page fault: pc 8000064a, addr 8006ddc8
[   47.470000] bus_error030: descriptor address 005a3db4, contents 011b300d
[   47.750000] bus_error030: page fault: pc 8001a952, addr 80073068
[   47.760000] bus_error030: descriptor address 005a3dcc, contents 00000000
[   47.790000] bus_error030: page fault: pc 8001a99c, addr 8006fb14
[   47.800000] bus_error030: descriptor address 005a3dbc, contents 011b800d
[   47.880000] bus_error030: page fault: pc 80013e9e, addr 800744e0
[   47.890000] bus_error030: descriptor address 005a3dd0, contents 00000000
[   48.150000] bus_error030: page fault: pc 80038d14, addr efffee38
[   48.160000] bus_error030: descriptor address 005a3bf8, contents 00000000
[   48.600000] bus_error030: page fault: pc 80011d08, addr 80075adc
[   48.620000] bus_error030: descriptor address 005a3dd4, contents 00000000
starting recursion
[   48.650000] bus_error030: page fault: pc 8000038c, addr efffdffc
[   48.660000] bus_error030: descriptor address 005a3bf4, contents 00000000
[   48.670000] bus_error030: page fault: pc 8000038c, addr efffcffc
[   48.690000] bus_error030: descriptor address 005a3bf0, contents 00000000
[   48.710000] bus_error030: page fault: pc 8000038c, addr efffbffc
[   48.720000] bus_error030: descriptor address 005a3bec, contents 00000000
[   48.730000] bus_error030: page fault: pc 8000038c, addr efffaffc
[   48.760000] bus_error030: descriptor address 005a3be8, contents 00000000
[   48.770000] bus_error030: page fault: pc 8000038c, addr efff9ffc
[   48.780000] bus_error030: descriptor address 005a3be4, contents 00000000
[   48.790000] bus_error030: page fault: pc 80000388, addr efff8ffc
[   48.820000] bus_error030: descriptor address 005a3be0, contents 00000000
[   48.830000] bus_error030: page fault: pc 8000038c, addr efff7ffc
[   48.840000] bus_error030: descriptor address 005a3bdc, contents 00000000
[   48.850000] bus_error030: page fault: pc 8000038c, addr efff6ffc
[   48.880000] bus_error030: descriptor address 005a3bd8, contents 00000000
[   48.890000] bus_error030: page fault: pc 8000038c, addr efff5ffc
[   48.900000] bus_error030: descriptor address 005a3bd4, contents 00000000
[   48.920000] bus_error030: page fault: pc 8000038c, addr efff4ffc
[   48.930000] bus_error030: descriptor address 005a3bd0, contents 00000000
[   48.940000] bus_error030: page fault: pc 8000038c, addr efff3ffc
[   48.950000] bus_error030: descriptor address 005a3bcc, contents 00000000
[   48.970000] bus_error030: page fault: pc 8000038c, addr efff2ffc
[   48.980000] bus_error030: descriptor address 005a3bc8, contents 00000000
[   48.990000] bus_error030: page fault: pc 8000038c, addr efff1ffc
[   49.000000] bus_error030: descriptor address 005a3bc4, contents 00000000
[   49.020000] bus_error030: page fault: pc 8000038c, addr efff0ffc
[   49.030000] bus_error030: descriptor address 005a3bc0, contents 00000000
[   49.040000] bus_error030: page fault: pc 80000388, addr effefffc
[   49.050000] bus_error030: descriptor address 005a3bbc, contents 00000000
[   49.070000] bus_error030: page fault: pc 8000038c, addr effeeffc
[   49.080000] bus_error030: descriptor address 005a3bb8, contents 00000000
[   49.090000] bus_error030: page fault: pc 8000038c, addr effedffc
[   49.100000] bus_error030: descriptor address 005a3bb4, contents 00000000
[   49.120000] bus_error030: page fault: pc 8000038c, addr effecffc
[   49.130000] bus_error030: descriptor address 005a3bb0, contents 00000000
[   49.140000] bus_error030: page fault: pc 8000038c, addr effebffc
[   49.150000] bus_error030: descriptor address 005a3bac, contents 00000000
[   49.170000] bus_error030: page fault: pc 8000038c, addr effeaffc
[   49.180000] bus_error030: descriptor address 005a3ba8, contents 00000000
[   49.190000] bus_error030: page fault: pc 8000038c, addr effe9ffc
[   49.200000] bus_error030: descriptor address 005a3ba4, contents 00000000
[   49.220000] bus_error030: page fault: pc 8000038c, addr effe8ffc
[   49.230000] bus_error030: descriptor address 005a3ba0, contents 00000000
[   49.240000] bus_error030: page fault: pc 8000038c, addr effe7ffc
[   49.250000] bus_error030: descriptor address 005a3b9c, contents 00000000
[   49.270000] bus_error030: page fault: pc 80000388, addr effe6ffc
[   49.280000] bus_error030: descriptor address 005a3b98, contents 00000000
[   49.290000] bus_error030: page fault: pc 8000038c, addr effe5ffc
[   49.300000] bus_error030: descriptor address 005a3b94, contents 00000000
[   49.320000] bus_error030: page fault: pc 8000038c, addr effe4ffc
[   49.330000] bus_error030: descriptor address 005a3b90, contents 00000000
[   49.340000] bus_error030: page fault: pc 8000038c, addr effe3ffc
[   49.360000] bus_error030: descriptor address 005a3b8c, contents 00000000
[   49.380000] bus_error030: page fault: pc 8000038c, addr effe2ffc
[   49.390000] bus_error030: descriptor address 005a3b88, contents 00000000
[   49.400000] bus_error030: page fault: pc 8000038c, addr effe1ffc
[   49.420000] bus_error030: descriptor address 005a3b84, contents 00000000
[   49.440000] bus_error030: page fault: pc 8000038c, addr effe0ffc
[   49.450000] bus_error030: descriptor address 005a3b80, contents 00000000
[   49.460000] bus_error030: page fault: pc 8000038c, addr effdfffc
[   49.490000] bus_error030: descriptor address 005a3b7c, contents 00000000
[   49.500000] bus_error030: page fault: pc 8000038c, addr effdeffc
[   49.510000] bus_error030: descriptor address 005a3b78, contents 00000000
[   49.530000] bus_error030: page fault: pc 80000388, addr effddffc
[   49.540000] bus_error030: descriptor address 005a3b74, contents 00000000
[   49.550000] bus_error030: page fault: pc 8000038c, addr effdcffc
[   49.560000] bus_error030: descriptor address 005a3b70, contents 00000000
[   49.580000] bus_error030: page fault: pc 8000038c, addr effdbffc
[   49.590000] bus_error030: descriptor address 005a3b6c, contents 00000000
[   49.600000] bus_error030: page fault: pc 8000038c, addr effdaffc
[   49.610000] bus_error030: descriptor address 005a3b68, contents 00000000
[   49.630000] bus_error030: page fault: pc 8000038c, addr effd9ffc
[   49.640000] bus_error030: descriptor address 005a3b64, contents 00000000
[   49.650000] bus_error030: page fault: pc 8000038c, addr effd8ffc
[   49.680000] bus_error030: descriptor address 005a3b60, contents 00000000
[   49.690000] bus_error030: page fault: pc 8000038c, addr effd7ffc
[   49.700000] bus_error030: descriptor address 005a3b5c, contents 00000000
[   49.720000] bus_error030: page fault: pc 8000038c, addr effd6ffc
[   49.730000] bus_error030: descriptor address 005a3b58, contents 00000000
[   49.740000] bus_error030: page fault: pc 8000038c, addr effd5ffc
[   49.750000] bus_error030: descriptor address 005a3b54, contents 00000000
[   49.770000] bus_error030: page fault: pc 80000388, addr effd4ffc
[   49.780000] bus_error030: descriptor address 005a3b50, contents 00000000
[   49.800000] bus_error030: page fault: pc 8000038c, addr effd3ffc
[   49.820000] bus_error030: descriptor address 005a3b4c, contents 00000000
done.
starting recursion with fork
[   49.910000] bus_error030: page fault: pc 8001599c, addr efff7d3c
[   49.930000] bus_error030: descriptor address 005a3bdc, contents 0182e01d
[   49.940000] bus_error030: page fault: pc 80000b3c, addr efff7d24
[   49.970000] bus_error030: descriptor address 014280dc, contents 0182e01d
[   49.980000] bus_error030: page fault: pc 800003ea, addr 800703a0
[   49.990000] bus_error030: descriptor address 005a3dc0, contents 011aa01d
[   50.000000] bus_error030: page fault: pc 80015730, addr 800731d0
[   50.030000] bus_error030: descriptor address 005a3fcc, contents 0132201d
[   50.040000] bus_error030: page fault: pc 8000038c, addr efff6ffc
[   50.050000] bus_error030: descriptor address 005a3bd8, contents 0182f01d
[   50.070000] bus_error030: page fault: pc 8000038c, addr efff5ffc
[   50.080000] bus_error030: descriptor address 005a3bd4, contents 0183001d
[   50.090000] bus_error030: page fault: pc 80015736, addr 8006faf0
[   50.100000] bus_error030: descriptor address 005a3fbc, contents 0132301d
[   50.120000] bus_error030: page fault: pc 8000038c, addr efff4ffc
[   50.130000] bus_error030: descriptor address 005a3bd0, contents 0183101d
[   50.150000] bus_error030: page fault: pc 8000038c, addr efff3ffc
[   50.160000] bus_error030: descriptor address 005a3bcc, contents 0183201d
[   50.190000] bus_error030: page fault: pc 8000038c, addr efff2ffc
[   50.200000] bus_error030: descriptor address 005a3bc8, contents 0183301d
[   50.210000] bus_error030: page fault: pc 8000038c, addr efff1ffc
[   50.240000] bus_error030: descriptor address 005a3bc4, contents 0183401d
[   50.250000] bus_error030: page fault: pc 8000038c, addr efff0ffc
[   50.260000] bus_error030: descriptor address 005a3bc0, contents 0183501d
[   50.270000] bus_error030: page fault: pc 80000388, addr effefffc
[   50.290000] bus_error030: descriptor address 005a3bbc, contents 0183601d
[   50.310000] bus_error030: page fault: pc 8000038c, addr effeeffc
[   50.320000] bus_error030: descriptor address 005a3bb8, contents 0183701d
[   50.330000] bus_error030: page fault: pc 8000038c, addr effedffc
[   50.360000] bus_error030: descriptor address 005a3bb4, contents 0183801d
[   50.370000] bus_error030: page fault: pc 8000038c, addr effecffc
[   50.380000] bus_error030: descriptor address 005a3bb0, contents 0183901d
[   50.390000] bus_error030: page fault: pc 8000038c, addr effebffc
[   50.420000] bus_error030: descriptor address 005a3bac, contents 0183a01d
[   50.430000] bus_error030: page fault: pc 8000038c, addr effeaffc
[   50.440000] bus_error030: descriptor address 005a3ba8, contents 0183b01d
[   50.450000] bus_error030: page fault: pc 8000038c, addr effe9ffc
[   50.480000] bus_error030: descriptor address 005a3ba4, contents 0183c01d
[   50.490000] bus_error030: page fault: pc 8000038c, addr effe8ffc
[   50.500000] bus_error030: descriptor address 005a3ba0, contents 0183d01d
[   50.510000] bus_error030: page fault: pc 8000038c, addr effe7ffc
[   50.530000] bus_error030: descriptor address 005a3b9c, contents 0183e01d
[   50.550000] bus_error030: page fault: pc 80000388, addr effe6ffc
[   50.560000] bus_error030: descriptor address 005a3b98, contents 0183f01d
[   50.570000] bus_error030: page fault: pc 8000038c, addr effe5ffc
[   50.590000] bus_error030: descriptor address 005a3b94, contents 0184001d
[   50.620000] bus_error030: page fault: pc 8001599c, addr effe5d3c
[   50.630000] bus_error030: descriptor address 005a3b94, contents 0184001d
[   50.650000] bus_error030: page fault: pc 80000b3c, addr effe5d24
[   50.660000] bus_error030: descriptor address 01429094, contents 0184001d
[   50.670000] bus_error030: page fault: pc 800003ea, addr 800703a0
[   50.680000] bus_error030: descriptor address 005a3dc0, contents 0185501d
[   50.700000] bus_error030: page fault: pc 80015730, addr 800731d0
[   50.710000] bus_error030: descriptor address 005a3fcc, contents 0132201d
[   50.720000] bus_error030: page fault: pc 8000038c, addr effe4ffc
[   50.730000] bus_error030: descriptor address 005a3b90, contents 0184101d
[   50.750000] bus_error030: page fault: pc 80015736, addr 8006faf0
[   50.760000] bus_error030: descriptor address 005a3fbc, contents 0132301d
[   50.780000] bus_error030: page fault: pc 8000038c, addr effe3ffc
[   50.790000] bus_error030: descriptor address 005a3b8c, contents 0184201d
[   50.820000] bus_error030: page fault: pc 8000038c, addr effe2ffc
[   50.830000] bus_error030: descriptor address 005a3b88, contents 0184301d
[   50.840000] bus_error030: page fault: pc 8000038c, addr effe1ffc
[   50.860000] bus_error030: descriptor address 005a3b84, contents 0184401d
[   50.880000] bus_error030: page fault: pc 8000038c, addr effe0ffc
[   50.890000] bus_error030: descriptor address 005a3b80, contents 0184501d
[   50.900000] bus_error030: page fault: pc 8000038c, addr effdfffc
[   50.920000] bus_error030: descriptor address 005a3b7c, contents 0184601d
[   50.940000] bus_error030: page fault: pc 8000038c, addr effdeffc
[   50.950000] bus_error030: descriptor address 005a3b78, contents 0184701d
[   50.960000] bus_error030: page fault: pc 80000388, addr effddffc
[   50.990000] bus_error030: descriptor address 005a3b74, contents 0184801d
[   51.000000] bus_error030: page fault: pc 8000038c, addr effdcffc
[   51.010000] bus_error030: descriptor address 005a3b70, contents 0184901d
[   51.030000] bus_error030: page fault: pc 8000038c, addr effdbffc
[   51.040000] bus_error030: descriptor address 005a3b6c, contents 0184a01d
[   51.050000] bus_error030: page fault: pc 8000038c, addr effdaffc
[   51.060000] bus_error030: descriptor address 005a3b68, contents 0184b01d
[   51.070000] bus_error030: page fault: pc 8000038c, addr effd9ffc
[   51.100000] bus_error030: descriptor address 005a3b64, contents 0184c01d
[   51.110000] bus_error030: page fault: pc 8000038c, addr effd8ffc
[   51.120000] bus_error030: descriptor address 005a3b60, contents 0184d01d
[   51.130000] bus_error030: page fault: pc 8000038c, addr effd7ffc
[   51.160000] bus_error030: descriptor address 005a3b5c, contents 0184e01d
[   51.170000] bus_error030: page fault: pc 8000038c, addr effd6ffc
[   51.180000] bus_error030: descriptor address 005a3b58, contents 0184f01d
[   51.190000] bus_error030: page fault: pc 8000038c, addr effd5ffc
[   51.220000] bus_error030: descriptor address 005a3b54, contents 0185001d
[   51.230000] bus_error030: page fault: pc 80000388, addr effd4ffc
[   51.240000] bus_error030: descriptor address 005a3b50, contents 0185101d
[   51.260000] bus_error030: page fault: pc 8000038c, addr effd3ffc
[   51.270000] bus_error030: descriptor address 005a3b4c, contents 0185201d
[   51.290000] bus_error030: page fault: pc 8001599c, addr effd3d3c
[   51.300000] bus_error030: descriptor address 005a3b4c, contents 0185201d
[   51.320000] bus_error030: page fault: pc 80000b3c, addr effd3d24
[   51.330000] bus_error030: descriptor address 0142804c, contents 0185201d
[   51.340000] bus_error030: page fault: pc 800003ea, addr 800703a0
[   51.360000] bus_error030: descriptor address 005a3dc0, contents 0185601d
[   51.380000] bus_error030: page fault: pc 80015730, addr 800731d0
[   51.390000] bus_error030: descriptor address 005a3fcc, contents 0132201d
[   51.400000] bus_error030: page fault: pc 80015736, addr 8006faf0
[   51.430000] bus_error030: descriptor address 005a3fbc, contents 0132301d
Illegal instruction
# 


diff --git a/arch/m68k/kernel/traps.c b/arch/m68k/kernel/traps.c
index a700807c9b6d..4820d5e7dd8d 100644
--- a/arch/m68k/kernel/traps.c
+++ b/arch/m68k/kernel/traps.c
@@ -500,9 +500,7 @@ static inline void bus_error030 (struct frame *fp)
 	unsigned short mmusr;
 	unsigned long addr, errorcode;
 	unsigned short ssw = fp->un.fmtb.ssw;
-#ifdef DEBUG
 	unsigned long desc;
-#endif
 
 	pr_debug("pid = %x  ", current->pid);
 	pr_debug("SSW=%#06x  ", ssw);
@@ -526,25 +524,24 @@ static inline void bus_error030 (struct frame *fp)
 	if (ssw & DF) {
 		addr = fp->un.fmtb.daddr;
 
-#ifdef DEBUG
 		asm volatile ("ptestr %3,%2@,#7,%0\n\t"
 			      "pmove %%psr,%1"
 			      : "=a&" (desc), "=m" (temp)
 			      : "a" (addr), "d" (ssw));
-		pr_debug("mmusr is %#x for addr %#lx in task %p\n",
-			 temp, addr, current);
-		pr_debug("descriptor address is 0x%p, contents %#lx\n",
-			 __va(desc), *(unsigned long *)__va(desc));
-#else
-		asm volatile ("ptestr %2,%1@,#7\n\t"
-			      "pmove %%psr,%0"
-			      : "=m" (temp) : "a" (addr), "d" (ssw));
-#endif
 		mmusr = temp;
 		errorcode = (mmusr & MMU_I) ? 0 : 1;
 		if (!(ssw & RW) || (ssw & RM))
 			errorcode |= 2;
 
+		if ((ssw & RW) == 0) {
+			u16 inst;
+			if (get_user(inst, (u16 *)fp->ptregs.pc) == 0 &&
+			    memcmp(current->comm, "stack-test", 10) == 0) {
+				printk(KERN_DEBUG "%s: page fault: pc %08lx, addr %08lx\n", __func__, fp->ptregs.pc, addr);
+				printk(KERN_DEBUG "%s: descriptor address %08lx, contents %08lx\n", __func__, (unsigned long)__va(desc), *(unsigned long *)__va(desc));
+			}
+		}
+
 		if (mmusr & (MMU_I | MMU_WP)) {
 			/* We might have an exception table for this PC */
 			if (ssw & 4 && !search_exception_tables(fp->ptregs.pc)) {

Re: reliable reproducer, was Re: core dump analysis

[Finn Thain]

On Wed, 19 Apr 2023, I wrote:

> Oddly, the program never detects any stack corruption when run on the 
> QEMU '040.
> 

I tested a Motorola '040 and got the same result.

Re: reliable reproducer, was Re: core dump analysis

[Michael Schmitz]

Hi Finn,

Am 20.04.2023 um 18:04 schrieb Finn Thain:
> On Wed, 19 Apr 2023, I wrote:
>
>> Oddly, the program never detects any stack corruption when run on the
>> QEMU '040.
>>
>
> I tested a Motorola '040 and got the same result.

OK, that would mean the bus error was just the most reliable way to get 
do_signal_return run after child process termination, and the signal 
delivery itself may be responsible for stack corruption.

Cheers,

	Michael

Re: reliable reproducer, was Re: core dump analysis

[Finn Thain]

On Thu, 20 Apr 2023, Michael Schmitz wrote:

> Am 20.04.2023 um 18:04 schrieb Finn Thain:
> > On Wed, 19 Apr 2023, I wrote:
> >
> >> Oddly, the program never detects any stack corruption when run on the
> >> QEMU '040.
> >>
> >
> > I tested a Motorola '040 and got the same result.
> 
> OK, that would mean the bus error was just the most reliable way to get 
> do_signal_return run after child process termination, and the signal 
> delivery itself may be responsible for stack corruption.
> 

I agree (I assume you're replying to my other message about the absence of 
swap).

Re: core dump analysis, was Re: stack smashing detected

[Michael Schmitz]

Hi Finn,

nice work!

Am 01.04.2023 um 22:27 schrieb Finn Thain:
> So, in summary, the canary validation failed in this case not because the
> canary got clobbered but because %a3 got clobbered, somewhere between
> __wait3+24 and __wait3+70 (below).
>
> The call to __GI___wait4_time64 causes %a3 to be saved to and restored
> from the stack, so stack corruption seems to be a strong possibility to
> explain the change in %a3.
>
> But if that's what happened, I'd expect __GI___wait4_time64 to report
> stack smashing, not __wait3... And it just begs the question, what then
> caused the corruption? Was it the wait4 syscall? Was it another thread?

Saved registers are restored from the stack before return from 
__GI___wait4_time64 but we don't know which of the two wait4 call sites 
was used, do we?

What registers does __m68k_read_tp@plt clobber?

> And why is it so rare?

Maybe an interaction between (multiple?) signals and syscall return... 
depends on how long we sleep in wait4, and whether a signal happens just 
during that time.

%a3 is the first register saved to the switch stack BTW.

That kernel does contain Al Viro's patch that corrected our switch stack 
handling in the signal return path? I wonder whether there's a potential 
race lurking in there?

And I just notice that we had had trouble with a copy_to_user in 
setup_frame() earlier (reason for my buserr handler patch). I wonder 
whether something's gone wrong there. Do you get a segfault instead of 
the abort signal if you drop my patch?

Cheers,

	Michael



>
> (gdb) disass __wait3
> Dump of assembler code for function __wait3:
>    0xc00e0070 <+0>:     linkw %fp,#-96
>    0xc00e0074 <+4>:     moveml %a2-%a3/%a5,%sp@-
>    0xc00e0078 <+8>:     lea %pc@(0xc0198000),%a5
>    0xc00e0080 <+16>:    movel %fp@(8),%d0
>    0xc00e0084 <+20>:    moveal %fp@(16),%a2
>    0xc00e0088 <+24>:    moveal %a5@(108),%a3
>    0xc00e008c <+28>:    movel %a3@,%fp@(-4)
>    0xc00e0090 <+32>:    tstl %a2
>    0xc00e0092 <+34>:    beqw 0xc00e0152 <__wait3+226>
>    0xc00e0096 <+38>:    pea %fp@(-92)
>    0xc00e009a <+42>:    movel %fp@(12),%sp@-
>    0xc00e009e <+46>:    movel %d0,%sp@-
>    0xc00e00a0 <+48>:    pea 0xffffffff
>    0xc00e00a4 <+52>:    bsrl 0xc00e0174 <__GI___wait4_time64>
>    0xc00e00aa <+58>:    lea %sp@(16),%sp
>    0xc00e00ae <+62>:    tstl %d0
>    0xc00e00b0 <+64>:    bgts 0xc00e00c8 <__wait3+88>
>    0xc00e00b2 <+66>:    moveal %fp@(-4),%a0
>    0xc00e00b6 <+70>:    movel %a3@,%d1
>    0xc00e00b8 <+72>:    cmpl %a0,%d1
>    0xc00e00ba <+74>:    bnew 0xc00e016c <__wait3+252>
>    0xc00e00be <+78>:    moveml %fp@(-108),%a2-%a3/%a5
>    0xc00e00c4 <+84>:    unlk %fp
>    0xc00e00c6 <+86>:    rts
>    0xc00e00c8 <+88>:    pea 0x44
>    0xc00e00cc <+92>:    clrl %sp@-
>    0xc00e00ce <+94>:    pea %a2@(4)
>    0xc00e00d2 <+98>:    movel %d0,%fp@(-96)
>    0xc00e00d6 <+102>:   bsrl 0xc00b8850 <__GI_memset>
>    0xc00e00dc <+108>:   movel %fp@(-88),%a2@
>    0xc00e00e0 <+112>:   movel %fp@(-80),%a2@(4)
>    0xc00e00e6 <+118>:   movel %fp@(-72),%a2@(8)
>    0xc00e00ec <+124>:   movel %fp@(-64),%a2@(12)
>    0xc00e00f2 <+130>:   movel %fp@(-60),%a2@(16)
>    0xc00e00f8 <+136>:   movel %fp@(-56),%a2@(20)
>    0xc00e00fe <+142>:   movel %fp@(-52),%a2@(24)
>    0xc00e0104 <+148>:   movel %fp@(-48),%a2@(28)
>    0xc00e010a <+154>:   movel %fp@(-44),%a2@(32)
>    0xc00e0110 <+160>:   movel %fp@(-40),%a2@(36)
>    0xc00e0116 <+166>:   movel %fp@(-36),%a2@(40)
>    0xc00e011c <+172>:   movel %fp@(-32),%a2@(44)
>    0xc00e0122 <+178>:   movel %fp@(-28),%a2@(48)
>    0xc00e0128 <+184>:   movel %fp@(-24),%a2@(52)
>    0xc00e012e <+190>:   movel %fp@(-20),%a2@(56)
>    0xc00e0134 <+196>:   movel %fp@(-16),%a2@(60)
>    0xc00e013a <+202>:   movel %fp@(-12),%a2@(64)
>    0xc00e0140 <+208>:   movel %fp@(-8),%a2@(68)
>    0xc00e0146 <+214>:   lea %sp@(12),%sp
>    0xc00e014a <+218>:   movel %fp@(-96),%d0
>    0xc00e014e <+222>:   braw 0xc00e00b2 <__wait3+66>
>    0xc00e0152 <+226>:   clrl %sp@-
>    0xc00e0154 <+228>:   movel %fp@(12),%sp@-
>    0xc00e0158 <+232>:   movel %d0,%sp@-
>    0xc00e015a <+234>:   pea 0xffffffff
>    0xc00e015e <+238>:   bsrl 0xc00e0174 <__GI___wait4_time64>
>    0xc00e0164 <+244>:   lea %sp@(16),%sp
>    0xc00e0168 <+248>:   braw 0xc00e00b2 <__wait3+66>
>    0xc00e016c <+252>:   bsrl 0xc012a38c <__stack_chk_fail>
> End of assembler dump.
> (gdb) disass __GI___wait4_time64
> Dump of assembler code for function __GI___wait4_time64:
>    0xc00e0174 <+0>:     lea %sp@(-80),%sp
>    0xc00e0178 <+4>:     moveml %d2-%d5/%a2-%a3/%a5,%sp@-
>    0xc00e017c <+8>:     lea %pc@(0xc0198000),%a5
>    0xc00e0184 <+16>:    movel %sp@(116),%d2
>    0xc00e0188 <+20>:    moveal %sp@(124),%a2
>    0xc00e018c <+24>:    moveal %a5@(108),%a3
>    0xc00e0190 <+28>:    movel %a3@,%sp@(104)
>    0xc00e0194 <+32>:    bsrl 0xc0052e2c <__m68k_read_tp@plt>
>    0xc00e019a <+38>:    movel %a0@(-29920),%d4
>    0xc00e019e <+42>:    bnew 0xc00e026c <__GI___wait4_time64+248>
>    0xc00e01a2 <+46>:    tstl %a2
>    0xc00e01a4 <+48>:    beqs 0xc00e01aa <__GI___wait4_time64+54>
>    0xc00e01a6 <+50>:    moveq #32,%d4
>    0xc00e01a8 <+52>:    addl %sp,%d4
>    0xc00e01aa <+54>:    movel %sp@(120),%d3
>    0xc00e01ae <+58>:    movel %sp@(112),%d1
>    0xc00e01b2 <+62>:    moveq #114,%d0
>    0xc00e01b4 <+64>:    trap #0
>    0xc00e01b6 <+66>:    cmpil #-4096,%d0
>    0xc00e01bc <+72>:    bhiw 0xc00e02a6 <__GI___wait4_time64+306>
>    0xc00e01c0 <+76>:    tstl %d0
>    0xc00e01c2 <+78>:    blew 0xc00e0256 <__GI___wait4_time64+226>
>    0xc00e01c6 <+82>:    tstl %a2
>    0xc00e01c8 <+84>:    beqw 0xc00e0256 <__GI___wait4_time64+226>
>    0xc00e01cc <+88>:    movel %sp@(32),%a2@(4)
>    0xc00e01d2 <+94>:    smi %d1
>    0xc00e01d4 <+96>:    extbl %d1
>    0xc00e01d6 <+98>:    movel %d1,%a2@
>    0xc00e01d8 <+100>:   movel %sp@(36),%a2@(12)
>    0xc00e01de <+106>:   smi %d1
>    0xc00e01e0 <+108>:   extbl %d1
>    0xc00e01e2 <+110>:   movel %d1,%a2@(8)
>    0xc00e01e6 <+114>:   movel %sp@(40),%a2@(20)
>    0xc00e01ec <+120>:   smi %d1
>    0xc00e01ee <+122>:   extbl %d1
>    0xc00e01f0 <+124>:   movel %d1,%a2@(16)
>    0xc00e01f4 <+128>:   movel %sp@(44),%a2@(28)
>    0xc00e01fa <+134>:   smi %d1
>    0xc00e01fc <+136>:   extbl %d1
>    0xc00e01fe <+138>:   movel %d1,%a2@(24)
>    0xc00e0202 <+142>:   movel %sp@(48),%a2@(32)
>    0xc00e0208 <+148>:   movel %sp@(52),%a2@(36)
>    0xc00e020e <+154>:   movel %sp@(56),%a2@(40)
>    0xc00e0214 <+160>:   movel %sp@(60),%a2@(44)
>    0xc00e021a <+166>:   movel %sp@(64),%a2@(48)
>    0xc00e0220 <+172>:   movel %sp@(68),%a2@(52)
>    0xc00e0226 <+178>:   movel %sp@(72),%a2@(56)
>    0xc00e022c <+184>:   movel %sp@(76),%a2@(60)
>    0xc00e0232 <+190>:   movel %sp@(80),%a2@(64)
>    0xc00e0238 <+196>:   movel %sp@(84),%a2@(68)
>    0xc00e023e <+202>:   movel %sp@(88),%a2@(72)
>    0xc00e0244 <+208>:   movel %sp@(92),%a2@(76)
>    0xc00e024a <+214>:   movel %sp@(96),%a2@(80)
>    0xc00e0250 <+220>:   movel %sp@(100),%a2@(84)
>    0xc00e0256 <+226>:   moveal %sp@(104),%a0
>    0xc00e025a <+230>:   movel %a3@,%d1
>    0xc00e025c <+232>:   cmpl %a0,%d1
>    0xc00e025e <+234>:   bnew 0xc00e02f2 <__GI___wait4_time64+382>
>    0xc00e0262 <+238>:   moveml %sp@+,%d2-%d5/%a2-%a3/%a5
>    0xc00e0266 <+242>:   lea %sp@(80),%sp
>    0xc00e026a <+246>:   rts
>    0xc00e026c <+248>:   bsrl 0xc00a1b88 <__GI___pthread_enable_asynccancel>
>    0xc00e0272 <+254>:   movel %d0,%d5
>    0xc00e0274 <+256>:   tstl %a2
>    0xc00e0276 <+258>:   beqs 0xc00e02c4 <__GI___wait4_time64+336>
>    0xc00e0278 <+260>:   moveq #32,%d4
>    0xc00e027a <+262>:   addl %sp,%d4
>    0xc00e027c <+264>:   movel %sp@(120),%d3
>    0xc00e0280 <+268>:   movel %sp@(112),%d1
>    0xc00e0284 <+272>:   moveq #114,%d0
>    0xc00e0286 <+274>:   trap #0
>    0xc00e0288 <+276>:   cmpil #-4096,%d0
>    0xc00e028e <+282>:   bhis 0xc00e02c8 <__GI___wait4_time64+340>
>    0xc00e0290 <+284>:   movel %d5,%sp@-
>    0xc00e0292 <+286>:   movel %d0,%sp@(32)
>    0xc00e0296 <+290>:   bsrl 0xc00a1bea <__GI___pthread_disable_asynccancel>
>    0xc00e029c <+296>:   addql #4,%sp
>    0xc00e029e <+298>:   movel %sp@(28),%d0
>    0xc00e02a2 <+302>:   braw 0xc00e01c0 <__GI___wait4_time64+76>
>    0xc00e02a6 <+306>:   movel %d0,%sp@(28)
>    0xc00e02aa <+310>:   bsrl 0xc0052e2c <__m68k_read_tp@plt>
>    0xc00e02b0 <+316>:   addal %a5@(2cf8),%a0
>    0xc00e02b8 <+324>:   movel %sp@(28),%d0
>    0xc00e02bc <+328>:   negl %d0
>    0xc00e02be <+330>:   movel %d0,%a0@
>    0xc00e02c0 <+332>:   moveq #-1,%d0
>    0xc00e02c2 <+334>:   bras 0xc00e0256 <__GI___wait4_time64+226>
>    0xc00e02c4 <+336>:   clrl %d4
>    0xc00e02c6 <+338>:   bras 0xc00e027c <__GI___wait4_time64+264>
>    0xc00e02c8 <+340>:   movel %d0,%sp@(28)
>    0xc00e02cc <+344>:   bsrl 0xc0052e2c <__m68k_read_tp@plt>
>    0xc00e02d2 <+350>:   addal %a5@(2cf8),%a0
>    0xc00e02da <+358>:   movel %sp@(28),%d0
>    0xc00e02de <+362>:   negl %d0
>    0xc00e02e0 <+364>:   movel %d0,%a0@
>    0xc00e02e2 <+366>:   movel %d5,%sp@-
>    0xc00e02e4 <+368>:   bsrl 0xc00a1bea <__GI___pthread_disable_asynccancel>
>    0xc00e02ea <+374>:   addql #4,%sp
>    0xc00e02ec <+376>:   moveq #-1,%d0
>    0xc00e02ee <+378>:   braw 0xc00e0256 <__GI___wait4_time64+226>
>    0xc00e02f2 <+382>:   bsrl 0xc012a38c <__stack_chk_fail>
> End of assembler dump.
>

Re: core dump analysis, was Re: stack smashing detected

[Finn Thain]

On Sun, 2 Apr 2023, Michael Schmitz wrote:

> Saved registers are restored from the stack before return from 
> __GI___wait4_time64 but we don't know which of the two wait4 call sites 
> was used, do we?
> 
> What registers does __m68k_read_tp@plt clobber?
> 

But that won't matter to the caller, __wait3, right? 

I did check that %a3 was saved on entry, before any wait4 syscall or 
__m68k_read_tp call etc. I also looked at the rts and %a3 did get restored 
there. Is it worth the effort to trace every branch, in case there's some 
way to reach an rts without having first restored the saved registers?

> 
> Maybe an interaction between (multiple?) signals and syscall return...

When running dash from gdb in QEMU, there's only one signal (SIGCHLD) and 
it gets handled before __wait3() returns. (Of course, the "stack smashing 
detected" failure never shows up in QEMU.)

> depends on how long we sleep in wait4, and whether a signal happens just 
> during that time.
> 

I agree, there seems to be a race condition there. (And dash's waitproc() 
seems to take pains to reap the child and handle the signal in any order.) 
I wouldn't be surprised if this race somehow makes the failure rare.

I don't want to recompile any userland binaries at this stage, so it would 
be nice if we could modify the kernel to keep track of exactly how that 
race gets won and lost. Or perhaps there's an easy way to rig the outcome 
one way or the other.

> %a3 is the first register saved to the switch stack BTW.
> 
> That kernel does contain Al Viro's patch that corrected our switch stack 
> handling in the signal return path? I wonder whether there's a potential 
> race lurking in there?
> 

I'm not sure which patch you're referring to, but I think Al's signal 
handling work appeared in v5.15-rc4. I have reproduced the "stack smashing 
detected" failure with v5.14.0 and with recent mainline (62bad54b26db from 
March 30th).

> And I just notice that we had had trouble with a copy_to_user in 
> setup_frame() earlier (reason for my buserr handler patch). I wonder 
> whether something's gone wrong there. Do you get a segfault instead of 
> the abort signal if you drop my patch?
> 

Are you referring to e36a82bebbf7? I doubt that it's related. I believe 
that copy_to_user is not involved here for the reason already given i.e. 
wait3(status, flags, NULL) means wait4 gets a NULL pointer for the struct 
rusage * parameter. Also, Stan first reported this failure in December 
with v6.0.9.

Re: core dump analysis, was Re: stack smashing detected

[Michael Schmitz]

Hi Finn,

Am 02.04.2023 um 21:31 schrieb Finn Thain:
> On Sun, 2 Apr 2023, Michael Schmitz wrote:
>
>> Saved registers are restored from the stack before return from
>> __GI___wait4_time64 but we don't know which of the two wait4 call sites
>> was used, do we?
>>
>> What registers does __m68k_read_tp@plt clobber?
>>
>
> But that won't matter to the caller, __wait3, right?

Not if they are properly restored ... best not go there.

> I did check that %a3 was saved on entry, before any wait4 syscall or
> __m68k_read_tp call etc. I also looked at the rts and %a3 did get restored
> there. Is it worth the effort to trace every branch, in case there's some
> way to reach an rts without having first restored the saved registers?

No, I dont think that's possible - from inspection, I now see 
__GI___wait4_time64 does not allow that, and I think the same is true 
for wait3 (haven't spent quite long enough on that).

>
>>
>> Maybe an interaction between (multiple?) signals and syscall return...
>
> When running dash from gdb in QEMU, there's only one signal (SIGCHLD) and
> it gets handled before __wait3() returns. (Of course, the "stack smashing
> detected" failure never shows up in QEMU.)

Might be a clue that we need multiple signals to force the stack 
smashing error. And we might not get that in QEMU, due to the faster 
execution in emulating on a modern processor.

Thinking a bit more about interactions between signal delivery and 
syscall return, it turns out that we don't check for pending signals 
when returning from a syscall. That's OK on SMP systems, because we 
don't have another process running while we execute the syscall (and we 
_do_ run signal handling when scheduling, i.e. when wait4 sleeps or is 
woken up)?

Seems we can forget about that interaction then.

>
>> depends on how long we sleep in wait4, and whether a signal happens just
>> during that time.
>>
>
> I agree, there seems to be a race condition there. (And dash's waitproc()
> seems to take pains to reap the child and handle the signal in any order.)

Yes, it makes sure the SIGCHLD is seen no matter in what order the 
signals are delivered ...

> I wouldn't be surprised if this race somehow makes the failure rare.
>
> I don't want to recompile any userland binaries at this stage, so it would
> be nice if we could modify the kernel to keep track of exactly how that
> race gets won and lost. Or perhaps there's an easy way to rig the outcome
> one way or the other.

A race between syscall return due to child exit and signal delivery 
seems unlikely, but maybe there is a race between syscall return due to 
a timer firing and signal delivery. Are there any timers set to 
periodically interrupt wait3?

>
>> %a3 is the first register saved to the switch stack BTW.
>>
>> That kernel does contain Al Viro's patch that corrected our switch stack
>> handling in the signal return path? I wonder whether there's a potential
>> race lurking in there?
>>
>
> I'm not sure which patch you're referring to, but I think Al's signal
> handling work appeared in v5.15-rc4. I have reproduced the "stack smashing

I have it in 5.15-rc2 in my tree but that's probably from my running 
tests on that patch series.

> detected" failure with v5.14.0 and with recent mainline (62bad54b26db from
> March 30th).

OK, so it's not related (or the patch did not fix all the problems with 
multiple signals, but a) that's unlikely and b) signals during wait4 
should not matter, see above).

So the fact that %a3 is involved here is probably just coincidence.

>
>> And I just notice that we had had trouble with a copy_to_user in
>> setup_frame() earlier (reason for my buserr handler patch). I wonder
>> whether something's gone wrong there. Do you get a segfault instead of
>> the abort signal if you drop my patch?
>>
>
> Are you referring to e36a82bebbf7? I doubt that it's related. I believe
> that copy_to_user is not involved here for the reason already given i.e.
> wait3(status, flags, NULL) means wait4 gets a NULL pointer for the struct
> rusage * parameter. Also, Stan first reported this failure in December
> with v6.0.9.

Can't be related then.

Still no nearer to a solution - something smashes the stack near %sp, 
causes the %a3 register restore after __GI___wait4_time64 to return a 
wrong pointer to the stack canary, and triggers a stack smashing warning 
in this indirect way. But what??

Cheers,

	Michael

Re: core dump analysis, was Re: stack smashing detected

[Finn Thain]

On Mon, 3 Apr 2023, Michael Schmitz wrote:

> Am 02.04.2023 um 21:31 schrieb Finn Thain:
> >
> >>
> >> Maybe an interaction between (multiple?) signals and syscall 
> >> return...
> >
> > When running dash from gdb in QEMU, there's only one signal (SIGCHLD) 
> > and it gets handled before __wait3() returns. (Of course, the "stack 
> > smashing detected" failure never shows up in QEMU.)
> 
> Might be a clue that we need multiple signals to force the stack 
> smashing error. And we might not get that in QEMU, due to the faster 
> execution in emulating on a modern processor.
> 

Right -- being that the failure is intermittent on real hardware, it's not 
surprising that I can't make it show up in QEMU or Aranym.

But no-one has reproduced it on Atari or Amiga hardware yet so I guess it 
could be a driver issue...

I wonder whether anyone else is actually running recent Debian/SID with 
sysvinit and without a Debian initrd on a Motorola 68030 system.

> Thinking a bit more about interactions between signal delivery and 
> syscall return, it turns out that we don't check for pending signals 
> when returning from a syscall. That's OK on SMP systems, because we 
> don't have another process running while we execute the syscall (and we 
> _do_ run signal handling when scheduling, i.e. when wait4 sleeps or is 
> woken up)?
> 
> Seems we can forget about that interaction then.
> 
> >
> >> depends on how long we sleep in wait4, and whether a signal happens 
> >> just during that time.
> >>
> >
> > I agree, there seems to be a race condition there. (And dash's 
> > waitproc() seems to take pains to reap the child and handle the signal 
> > in any order.)
> 
> Yes, it makes sure the SIGCHLD is seen no matter in what order the 
> signals are delivered ...
> 
> > I wouldn't be surprised if this race somehow makes the failure rare.
> >
> > I don't want to recompile any userland binaries at this stage, so it 
> > would be nice if we could modify the kernel to keep track of exactly 
> > how that race gets won and lost. Or perhaps there's an easy way to rig 
> > the outcome one way or the other.
> 
> A race between syscall return due to child exit and signal delivery 
> seems unlikely, but maybe there is a race between syscall return due to 
> a timer firing and signal delivery. Are there any timers set to 
> periodically interrupt wait3?
> 

I searched the source code and SIGALRM appears to be unused by dash. And 
'timeout' is not a dash builtin. But that doesn't mean we don't get 
multiple signals. One crashy script looks like this:

TMPFS_SIZE="$(tmpfs_size_vm "$TMPFS_SIZE")"
RUN_SIZE="$(tmpfs_size_vm "$RUN_SIZE")"
LOCK_SIZE="$(tmpfs_size_vm "$LOCK_SIZE")"
SHM_SIZE="$(tmpfs_size_vm "$SHM_SIZE")"
TMP_SIZE="$(tmpfs_size_vm "$TMP_SIZE")"

Is it possible that the SIGCHLD from the first sub-shell got delayed?

> 
> Still no nearer to a solution - something smashes the stack near %sp, 
> causes the %a3 register restore after __GI___wait4_time64 to return a 
> wrong pointer to the stack canary, and triggers a stack smashing warning 
> in this indirect way. But what??
> 

I've no idea.

The actual corruption might offer a clue here. I believe the saved %a3 was 
clobbered with the value 0xefee1068 which seems to be a pointer into some 
stack frame that would have come into existence shortly after 
__GI___wait4_time64 was called. That stack frame is gone by the time the 
core dump was made. Was it dash's signal handler, onsig(), or some libc 
subroutine called by __GI___wait4_time64(), or was it something that the 
kernel put there?

Dash's SIGCHLD handler looks safe enough -- I don't see how it could 
corrupt the saved registers in the __GI___wait4_time64 stack frame (it's 
not like 1 was stored in the wrong place). 
https://sources.debian.org/src/dash/0.5.12-2/src/trap.c/?hl=285#L285

Re: core dump analysis, was Re: stack smashing detected

[Finn Thain]

On Tue, 4 Apr 2023, I wrote:

> 
> The actual corruption might offer a clue here. I believe the saved %a3 
> was clobbered with the value 0xefee1068 which seems to be a pointer into 
> some stack frame that would have come into existence shortly after 
> __GI___wait4_time64 was called.

Wrong... it is a pointer to the location below the __wait3 stack frame.

(gdb) info frame
Stack level 8, frame at 0xefee10e0:
 pc = 0xc00e0172 in __wait3 (../sysdeps/unix/sysv/linux/wait3.c:41); 
    saved pc = 0xd000c38e
 called by frame at 0xefee11dc, caller of frame at 0xefee106c
 source language c.
 Arglist at 0xefee10d8, args: stat_loc=<optimized out>, 
    options=<optimized out>, usage=<optimized out>
 Locals at 0xefee10d8, Previous frame's sp is 0xefee10e0
 Saved registers:
  a2 at 0xefee106c, a3 at 0xefee1070, a5 at 0xefee1074, fp at 0xefee10d8,
  pc at 0xefee10dc

That shows %a2 was saved at 0xefee106c, and the address of interest is the 
stack location immediately below that. But it has no particular 
significance: it holds a NULL pointer when the struct __rusage64 *usage 
argument to __wait4_time64() gets pushed there:

   0xc00e8152 <__wait3+226>:   clrl %sp@-
   0xc00e8154 <__wait3+228>:   movel %fp@(12),%sp@-
   0xc00e8158 <__wait3+232>:   movel %d0,%sp@-
   0xc00e815a <__wait3+234>:   pea 0xffffffff
   0xc00e815e <__wait3+238>:   bsrl 0xc00e8174 <__GI___wait4_time64>

But it's no longer a NULL pointer at the time of the crash, though it 
should be, since that stack frame is still active.

(gdb) x/16z 0xefee1068
0xefee1068:     0xc00e0172      0xd001e718      0xd001e498      0xd001b874
0xefee1078:     0x00170700      0x00170700      0x00170700      0x00005360
0xefee1088:     0x0000e920      0x00000006      0x00002000      0x00000002
0xefee1098:     0x00171f20      0x00171f20      0x00171f20      0x000000e0

Beats me.

Re: core dump analysis, was Re: stack smashing detected

[Finn Thain]

On Tue, 4 Apr 2023, I wrote:

> On Tue, 4 Apr 2023, I wrote:
> 
> > 
> > The actual corruption might offer a clue here. I believe the saved %a3 
> > was clobbered with the value 0xefee1068 which seems to be a pointer into 
> > some stack frame that would have come into existence shortly after 
> > __GI___wait4_time64 was called.
> 
> Wrong... it is a pointer to the location below the __wait3 stack frame.
> 
> (gdb) info frame
> Stack level 8, frame at 0xefee10e0:
>  pc = 0xc00e0172 in __wait3 (../sysdeps/unix/sysv/linux/wait3.c:41); 
>     saved pc = 0xd000c38e
>  called by frame at 0xefee11dc, caller of frame at 0xefee106c
>  source language c.
>  Arglist at 0xefee10d8, args: stat_loc=<optimized out>, 
>     options=<optimized out>, usage=<optimized out>
>  Locals at 0xefee10d8, Previous frame's sp is 0xefee10e0
>  Saved registers:
>   a2 at 0xefee106c, a3 at 0xefee1070, a5 at 0xefee1074, fp at 0xefee10d8,
>   pc at 0xefee10dc
> 
> That shows %a2 was saved at 0xefee106c, and the address of interest is the 
> stack location immediately below that. But it has no particular 
> significance: it holds a NULL pointer when the struct __rusage64 *usage 
> argument to __wait4_time64() gets pushed there:
> 
>    0xc00e8152 <__wait3+226>:   clrl %sp@-
>    0xc00e8154 <__wait3+228>:   movel %fp@(12),%sp@-
>    0xc00e8158 <__wait3+232>:   movel %d0,%sp@-
>    0xc00e815a <__wait3+234>:   pea 0xffffffff
>    0xc00e815e <__wait3+238>:   bsrl 0xc00e8174 <__GI___wait4_time64>
> 
> But it's no longer a NULL pointer at the time of the crash, though it 
> should be, since that stack frame is still active.
> 
> (gdb) x/16z 0xefee1068
> 0xefee1068:     0xc00e0172      0xd001e718      0xd001e498      0xd001b874
> 0xefee1078:     0x00170700      0x00170700      0x00170700      0x00005360
> 0xefee1088:     0x0000e920      0x00000006      0x00002000      0x00000002
> 0xefee1098:     0x00171f20      0x00171f20      0x00171f20      0x000000e0
> 
> Beats me.
> 

At the time of the crash, the corrupted %a3 was a pointer to location in 
__wait3's stack. That location was a NULL pointer (the *usage parameter) 
when __GI___wait4_time64 was called but now points to 0xc00e0172, which is 
just after the __wait3 text and just before __GI___wait4_time64 text.

(gdb) disass __wait3
Dump of assembler code for function __wait3:
...
   0xc00e015e <+238>:   bsrl 0xc00e0174 <__GI___wait4_time64>
   0xc00e0164 <+244>:   lea %sp@(16),%sp
   0xc00e0168 <+248>:   braw 0xc00e00b2 <__wait3+66>
   0xc00e016c <+252>:   bsrl 0xc012a38c <__stack_chk_fail>
End of assembler dump.
(gdb) disass __GI___wait4_time64
Dump of assembler code for function __GI___wait4_time64:
   0xc00e0174 <+0>:     lea %sp@(-80),%sp
   0xc00e0178 <+4>:     moveml %d2-%d5/%a2-%a3/%a5,%sp@-
   0xc00e017c <+8>:     lea %pc@(0xc0198000),%a5
   0xc00e0184 <+16>:    movel %sp@(116),%d2
...

But I realize now that this stack location gets overwritten with the 
return address for bsrl __stack_chk_fail, so there's nothing wrong there.

Perhaps its just a coincidence that the saved %a3, once corrupted, ended 
up pointing to the *usage parameter... I don't know what to make of that.

Re: stack smashing detected

[Finn Thain]

On Wed, 1 Feb 2023, Stan Johnson wrote:

> 
> After logging the start and end of each script, I see that the "stack 
> smashing detected" error often happens while running 
> "/etc/rcS.d/S01mountkernfs.sh" (/etc/init.d/mountkernfs.sh). I'll try to 
> isolate it to a particular command.
> 

That brings to mind some other unresolved initscript failures, also 
involving 68030, which were accompanied by "page allocation failure".

But it's hard to blame "stack smashing detected" on a page allocation 
failure since the latter always produces a very noisy splat in the kernel 
messages.

Have you reproduced the error with Debian's kernel package?

If not, please refer to private correspondence from me dated 10 December 
2022 regarding setting up /etc/initramfs-tools so as to produce a suitably 
small initramfs.

To save time, I recommend using QEMU and an up-to-date Debian/m68k SID 
virtual machine to produce the vmlinux and initrd files needed for use 
with Penguin on your slower machines.

Debian initramfs/initrd, was Re: stack smashing detected

[Finn Thain]

On Sun, 5 Feb 2023, Stan Johnson wrote:

> > 
> > To save time, I recommend using QEMU and an up-to-date Debian/m68k SID 
> > virtual machine to produce the vmlinux and initrd files needed for use 
> > with Penguin on your slower machines.
> > 
> 
> AFAIK, the initrd must be created on the system that is using the initrd 
> (or an identical system, at least that was the response I received a 
> while back when I was unsuccessfully trying to use the initrd that ships 
> on the Debian install CD).

If that was true, how could Debian ship a single initrd that works on 
Atari, Amiga, Mac etc.? They could not.

If you boot a random Debian/m68k rootfs with a random Debian/m68k 
vmlinux/initrd combination, and if you need a kernal module at some point, 
then your random rootfs must contain the modules that match your random 
kernel binary (probably not going to work).

If you never need to load a module, perhaps because the important ones all 
got loaded from the initrd, then all you need is a valid vmlinux/initrd 
combination and the rootfs is not relevant.

When you need to generate a valid Debian/m68k vmlinux/initrd combination 
that is also current, then you'll need a Debian/m68k system that is 
current. The quickest way to get it is an emulator.

When you need a small initrd, because of RAM limitations, you'll need to 
customize your initrd for your hardware using /etc/initramfs-tools (as 
mentioned).

If you succeed, you'll get an initrd that is missing all the modules for 
all the hardware that you don't own, which saves RAM. It doesn't matter 
what system generates that initrd (could be a build farm).

Re: Debian initramfs/initrd, was Re: stack smashing detected

[John Paul Adrian Glaubitz]

On Mon, 2023-02-06 at 09:29 +1100, Finn Thain wrote:
> On Sun, 5 Feb 2023, Stan Johnson wrote:
> 
> > > 
> > > To save time, I recommend using QEMU and an up-to-date Debian/m68k SID 
> > > virtual machine to produce the vmlinux and initrd files needed for use 
> > > with Penguin on your slower machines.
> > > 
> > 
> > AFAIK, the initrd must be created on the system that is using the initrd 
> > (or an identical system, at least that was the response I received a 
> > while back when I was unsuccessfully trying to use the initrd that ships 
> > on the Debian install CD).
> 
> If that was true, how could Debian ship a single initrd that works on 
> Atari, Amiga, Mac etc.? They could not.

You cannot use the initrd that's on the debian-installer medium because it
boots directly into the installer. It's not a regular initrd.

That's completely independent of the architecture/sub-architecture in use.

> When you need to generate a valid Debian/m68k vmlinux/initrd combination 
> that is also current, then you'll need a Debian/m68k system that is 
> current. The quickest way to get it is an emulator.

Correct.

Adrian


-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer
`. `'   Physicist
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: Debian initramfs/initrd, was Re: stack smashing detected

[Geert Uytterhoeven]

Hi Stan,

On Mon, Feb 6, 2023 at 4:42 AM Stan Johnson <userm57@yahoo.com> wrote:
> On an SE/30 with 128 MiB memory, the latest Debian SID kernel
> (vmlinux-6.1.0-2-m68k), using Debian SID modules, and with
> initrd-6.1.0-2-m68k built on the SE/30, hangs after the initial
> "ABCFGHIJK" (I tried it twice).

If you get to "K", you're almost at the end of arch/m68k/kernel/head.S,
and it is very likely the kernel C-code actually started.
Do you get any output using "debug earlyprintk" on the kernel command
line?

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds