* [PATCH RFC] m68k: skip kernel premption if interrupts were disabled @ 2024-03-22 1:48 Michael Schmitz [not found] ` <fcabda73-4b16-ee2c-e993-2bfcf392a26a@linux-m68k.org> 0 siblings, 1 reply; 21+ messages in thread From: Michael Schmitz @ 2024-03-22 1:48 UTC (permalink / raw) To: linux-m68k, geert; +Cc: schmitzmic, Finn Thain Comparing m68k with mips, we are missing the tests for disabled interrupts present in the low level kernel preemption code for mips. Adding that test results in no more 'table already freed' messages when running stress-ng --zombie (on ARAnyM). Needs testing on qemu / Mac as this test result may be Atari specific due to the hsync interrupt disable scheme there. Signed-off-by: Michael Schmitz <schmitzmic@gmail.com> Link: https://lore.kernel.org/all/7858a184cda66e0991fd295c711dfed7e4d1248c.1696603287.git.geert@linux-m68k.org/ Cc: Geert Uytterhoeven <geert@linux-m68k.org> Cc: Finn Thain <fthain@linux-m68k.org> --- arch/m68k/kernel/entry.S | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/arch/m68k/kernel/entry.S b/arch/m68k/kernel/entry.S index 9933679ea28b..f2cdea3cbc93 100644 --- a/arch/m68k/kernel/entry.S +++ b/arch/m68k/kernel/entry.S @@ -249,6 +249,10 @@ ENTRY(ret_from_exception) jpl 1f movel %a0@(TINFO_PREEMPT),%d1 + jne 1f + + | check if interrupts disabled + bfextu %sp@(PT_OFF_SR){#0,#3},%d1 jne 1f pea 1f -- 2.17.1 ^ permalink raw reply related [flat|nested] 21+ messages in thread
[parent not found: <fcabda73-4b16-ee2c-e993-2bfcf392a26a@linux-m68k.org>]
[parent not found: <e89bf618-2fa8-7cbd-16ed-d021a9a7f740@gmail.com>]
[parent not found: <948c12e4-e85f-a86c-ae95-a1eb03ca026d@gmail.com>]
[parent not found: <40928e38-0de7-75a6-d5f7-8c913155da03@linux-m68k.org>]
* Re: [PATCH RFC] m68k: skip kernel premption if interrupts were disabled [not found] ` <40928e38-0de7-75a6-d5f7-8c913155da03@linux-m68k.org> @ 2024-03-23 6:32 ` Michael Schmitz 2024-03-25 6:41 ` Michael Schmitz 0 siblings, 1 reply; 21+ messages in thread From: Michael Schmitz @ 2024-03-23 6:32 UTC (permalink / raw) To: Finn Thain, Geert Uytterhoeven; +Cc: linux-m68k Hi Finn, Am 23.03.2024 um 17:35 schrieb Finn Thain: > On Fri, 22 Mar 2024, Michael Schmitz wrote: > >> Am 22.03.2024 um 17:53 schrieb Michael Schmitz: >>> Am 22.03.2024 um 17:39 schrieb Finn Thain: >>> >>>> I find that patch description to be a bit confusing, since >>>> preempt_schedule_irq() requires that interrupts were disabled: >>>> >>>> BUG_ON(preempt_count() || !irqs_disabled()); >>>> >>>> Looking more closely I see that you're testing the IPL bits from the >>>> stack not the status register... >>> >>> Yes - the problem appears to be that if we enter preemption when about >>> to return to kernel code that had interrupts disabled, bad things may >>> happen [...] >>> That's independent from having interrupts disabled in the currently >>> active exception. >>> > > Is it? If interrupts were disabled by the code we're returning to, surely > they must be disabled still (in the Status reg, for the BUG_ON above). They would still be disabled, but by calling preempt_schedule_irq() they will be reenabled before calling schedule(). At that time, another interrupt may come in, and change things from under the piece of code that had interrupts disabled. That won't happen without preemption. What's more, I suspect schedule() may cause another process to exit that would otherwise first take a signal, and signal delivery then repeats the vma teardown on process exit. At least that is how I read those 'table already freed' stack traces. > >>>> >>>> It's not clear to me which pre-emption opportunities remain in >>>> effect. E.g. is full preemption (in effect) the same as voluntary >>>> preemption? >>> >>> Good question - I'll have to instrument preempt_schedule_irq() and see >>> if it gets called at all with my patch. >> >> Can't see preempt_schedule_irq() entered with my patch ... >> > > Neither can I: A quick "stress-ng --zombie -1 -t 100" test passed without > ever calling preempt_schedule_irq() in QEMU. Right. Whenever I allow preemption with IPL (in the saved frame's SR) > 0, I eventually get the 'table already free' panic. Trying to save the whole frame for later printing in free_pointer_table() hasn't been too helpful though. The saved frame's PC does not appear anywhere in the trace. > >> In general, full preemption is not the same as voluntary preemption. >> Full preemption attempts to preempt on every return from interrupt or >> syscall (given certain constraints are fulfilled such as we're not >> currently in any interrupt or already preempting, and there's other work >> that needs scheduling). >> >> The constraint that interrupts must not be disabled in the saved stack >> frame apparently is one constraint too many... >> > > I dropped the mailing list from the recipients in my previous message. > This might a good time to follow up on the list in case Geert can shed > some light. Adding Geert and list back in now ... Cheers, Michael > ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH RFC] m68k: skip kernel premption if interrupts were disabled 2024-03-23 6:32 ` Michael Schmitz @ 2024-03-25 6:41 ` Michael Schmitz 2024-03-25 22:28 ` Finn Thain 0 siblings, 1 reply; 21+ messages in thread From: Michael Schmitz @ 2024-03-25 6:41 UTC (permalink / raw) To: Finn Thain, Geert Uytterhoeven; +Cc: linux-m68k Hi Finn, > What's more, I suspect schedule() may cause another process to exit that > would otherwise first take a signal, and signal delivery then repeats > the vma teardown on process exit. At least that is how I read those > 'table already freed' stack traces. In the backtrace below, I do actually see free_pointer_table() reentered after preemption. (Ignore the 2200, 2200, 6480 in the call trace - that's just the arguments to a pr_err() that prints the first line in the excerpt below.) To get this snapshot of what exception context preempt_schedule_irq() was last called from, I save the stack pointer, the status register in the exception stack frame, and the vector number before calling preempt_schedule_irq(). The saved values are printed before calling panic() in free_pointer_table(). Unfortunately, when I add preempt_disable()/ sched_preempt_enable_noresched() in free_pointer_table(), I don't get that particular panic() anymore, but the bad kernel buserror with the warning from set_fc() that you reported. So no real improvement ... Cheers, Michael saved from last preempt: nr 2d sr 2200 2200 pc 6480 sp dcddc4 Kernel panic - not syncing: table already free! CPU: 0 PID: 1629 Comm: udevd Not tainted 6.8.0-rc7-atari-fpuemu-spinlocktest+ #1759 Stack from 00f7fd9c: 00f7fd9c 003e75ee 003e75ee 00000000 00000001 003354cc 003e75ee 0032f208 00000001 00000001 00e14000 004f0000 00f8b40c 0000643c 00f7ff0e 000064d4 003d9128 003d90ee 0000002d 00002200 00002200 00006480 00dcddc4 c0000000 c017c000 efd00000 c00c0000 01e14800 ee000000 00f8b184 000cdee8 00e14800 00000001 ffffffff c0000000 000d367c 0031d67e 000d9916 000d367c 00dc00cc 000cdd12 0031d67e 000d9916 c0080000 00f8b180 c017bfff c017bfff edffffff Call Trace: [<003354cc>] dump_stack+0xc/0x10 [<0032f208>] panic+0xd8/0x258 [<0000643c>] free_pointer_table+0x0/0x160 [<000064d4>] free_pointer_table+0x98/0x160 [<00002200>] do_one_initcall+0xa4/0x188 [<00002200>] do_one_initcall+0xa4/0x188 [<00006480>] free_pointer_table+0x44/0x160 [<000cdee8>] free_pgd_range+0x1d6/0x1f2 [<000d367c>] unlink_file_vma+0x0/0x40 [<0031d67e>] mas_find+0x0/0x102 [<000d9916>] unlink_anon_vmas+0x0/0x106 [<000d367c>] unlink_file_vma+0x0/0x40 [<000cdd12>] free_pgd_range+0x0/0x1f2 [<0031d67e>] mas_find+0x0/0x102 [<000d9916>] unlink_anon_vmas+0x0/0x106 [<000cdfbe>] free_pgtables+0xba/0xcc [<0031d67e>] mas_find+0x0/0x102 [<000d2950>] mas_set+0x0/0x22 [<00039f02>] up_read+0x0/0x64 [<000d41d0>] exit_mmap+0x13c/0x1de [<0000efd3>] NODD+0xab/0x104 [<001900f9>] ext2_get_folio.constprop.11+0x1a7/0x274 [<0000ffff>] EXPTBL+0x3bf/0x400 [<00016782>] __mmput+0x16/0x94 [<0001a158>] do_exit+0x250/0x714 [<00339f12>] _raw_spin_unlock_irq+0x10/0x38 [<0001a79a>] sys_exit_group+0x0/0x14 [<0001a7ae>] pid_child_should_wake+0x0/0x52 [<0000269a>] syscall+0x8/0xc [<0008c00b>] do_check_common+0x325/0x2946 ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH RFC] m68k: skip kernel premption if interrupts were disabled 2024-03-25 6:41 ` Michael Schmitz @ 2024-03-25 22:28 ` Finn Thain 2024-03-25 22:31 ` Michael Schmitz 0 siblings, 1 reply; 21+ messages in thread From: Finn Thain @ 2024-03-25 22:28 UTC (permalink / raw) To: Michael Schmitz; +Cc: Geert Uytterhoeven, linux-m68k On Mon, 25 Mar 2024, Michael Schmitz wrote: > Unfortunately, when I add preempt_disable()/ > sched_preempt_enable_noresched() in free_pointer_table(), I don't get > that particular panic() anymore, I'd like to try that patch. Would you send it please? > but the bad kernel buserror with the warning from set_fc() that you > reported... I think those set_fc() calls are confined to 68040 fault handling. Maybe we should try running stress tests on a 68030? ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH RFC] m68k: skip kernel premption if interrupts were disabled 2024-03-25 22:28 ` Finn Thain @ 2024-03-25 22:31 ` Michael Schmitz 2024-03-26 0:41 ` Finn Thain 2024-03-26 9:58 ` Finn Thain 0 siblings, 2 replies; 21+ messages in thread From: Michael Schmitz @ 2024-03-25 22:31 UTC (permalink / raw) To: Finn Thain; +Cc: Geert Uytterhoeven, linux-m68k Hi Finn, see below (hopefully not whitespace-damaged). The set_fc warning is specific to 040 but the bad kernel buserror (null pointer) causing it isn't. But testing on 030 would be a good idea if you have one with enough RAM. Haven't tried to boot my Falcon with a preempt enabled kernel yet. Cheers, Michael diff --git a/arch/m68k/mm/motorola.c b/arch/m68k/mm/motorola.c index 4d4d390a2df2..96cdcacb94ea 100644 --- a/arch/m68k/mm/motorola.c +++ b/arch/m68k/mm/motorola.c @@ -15,6 +15,7 @@ #include <linux/mm.h> #include <linux/swap.h> #include <linux/kernel.h> +#include <linux/preempt.h> #include <linux/string.h> #include <linux/types.h> #include <linux/init.h> @@ -195,6 +196,8 @@ int free_pointer_table(void *table, int type) unsigned long page = ptable & PAGE_MASK; unsigned int mask = 1U << ((ptable - page)/ptable_size(type)); + preempt_disable(); + dp = PD_PTABLE(page); if (PD_MARKBITS (dp) & mask) { pr_err("saved from last preempt: nr %x sr %lx %x pc %lx sp %lx\n", @@ -213,6 +216,7 @@ int free_pointer_table(void *table, int type) if (type == TABLE_PTE) pagetable_pte_dtor(virt_to_ptdesc((void *)page)); free_page (page); + sched_preempt_enable_no_resched(); return 1; } else if (ptable_list[type].next != dp) { /* @@ -221,6 +225,7 @@ int free_pointer_table(void *table, int type) */ list_move(dp, &ptable_list[type]); } + sched_preempt_enable_no_resched(); return 0; } On 26/03/24 11:28, Finn Thain wrote: > On Mon, 25 Mar 2024, Michael Schmitz wrote: > >> Unfortunately, when I add preempt_disable()/ >> sched_preempt_enable_noresched() in free_pointer_table(), I don't get >> that particular panic() anymore, > I'd like to try that patch. Would you send it please? > >> but the bad kernel buserror with the warning from set_fc() that you >> reported... > I think those set_fc() calls are confined to 68040 fault handling. Maybe > we should try running stress tests on a 68030? ^ permalink raw reply related [flat|nested] 21+ messages in thread
* Re: [PATCH RFC] m68k: skip kernel premption if interrupts were disabled 2024-03-25 22:31 ` Michael Schmitz @ 2024-03-26 0:41 ` Finn Thain 2024-03-26 9:58 ` Finn Thain 1 sibling, 0 replies; 21+ messages in thread From: Finn Thain @ 2024-03-26 0:41 UTC (permalink / raw) To: Michael Schmitz; +Cc: Geert Uytterhoeven, linux-m68k On Tue, 26 Mar 2024, Michael Schmitz wrote: > The set_fc warning is specific to 040 but the bad kernel buserror (null > pointer) causing it isn't. I'm curious as to what happens if we just ignore the error and continue stress testing. You're right though -- we don't need an '030 for that (it's a non-fatal error). > But testing on 030 would be a good idea if you have one with enough RAM. My LC III has 36 MB which is sufficient for stress-ng. I will try it. ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH RFC] m68k: skip kernel premption if interrupts were disabled 2024-03-25 22:31 ` Michael Schmitz 2024-03-26 0:41 ` Finn Thain @ 2024-03-26 9:58 ` Finn Thain 2024-03-26 19:59 ` Michael Schmitz 1 sibling, 1 reply; 21+ messages in thread From: Finn Thain @ 2024-03-26 9:58 UTC (permalink / raw) To: Michael Schmitz; +Cc: Geert Uytterhoeven, linux-m68k On Tue, 26 Mar 2024, Michael Schmitz wrote: > see below (hopefully not whitespace-damaged). > Following your approach, I added mutual exclusion to the other candidate in the same file, namely get_pointer_table(). Together with your patch, that seems to be sufficient to make stress-ng happy. I don't know the real extent of the data race here so I've probably used too much locking in this experiment (?) diff --git a/arch/m68k/mm/motorola.c b/arch/m68k/mm/motorola.c index 068ad7c9bb5c..d1ad55ee1b60 100644 --- a/arch/m68k/mm/motorola.c +++ b/arch/m68k/mm/motorola.c @@ -140,10 +140,14 @@ void __init init_pointer_table(void *table, int type) void *get_pointer_table(int type) { - ptable_desc *dp = ptable_list[type].next; - unsigned int mask = list_empty(&ptable_list[type]) ? 0 : PD_MARKBITS(dp); + ptable_desc *dp; + unsigned int mask; unsigned int tmp, off; + preempt_disable(); + + dp = ptable_list[type].next; + mask = list_empty(&ptable_list[type]) ? 0 : PD_MARKBITS(dp); /* * For a pointer table for a user process address space, a * table is taken from a page allocated for the purpose. Each @@ -154,8 +158,13 @@ void *get_pointer_table(int type) void *page; ptable_desc *new; - if (!(page = (void *)get_zeroed_page(GFP_KERNEL))) + sched_preempt_enable_no_resched(); + + if (!(page = (void *)get_zeroed_page(GFP_KERNEL))) { return NULL; + } + + preempt_disable(); if (type == TABLE_PTE) { /* @@ -171,6 +180,7 @@ void *get_pointer_table(int type) PD_MARKBITS(new) = ptable_mask(type) - 1; list_add_tail(new, dp); + sched_preempt_enable_no_resched(); return (pmd_t *)page; } @@ -181,6 +191,7 @@ void *get_pointer_table(int type) /* move to end of list */ list_move_tail(dp, &ptable_list[type]); } + sched_preempt_enable_no_resched(); return page_address(PD_PAGE(dp)) + off; } ^ permalink raw reply related [flat|nested] 21+ messages in thread
* Re: [PATCH RFC] m68k: skip kernel premption if interrupts were disabled 2024-03-26 9:58 ` Finn Thain @ 2024-03-26 19:59 ` Michael Schmitz 2024-03-26 22:03 ` Finn Thain 2024-03-27 8:22 ` Geert Uytterhoeven 0 siblings, 2 replies; 21+ messages in thread From: Michael Schmitz @ 2024-03-26 19:59 UTC (permalink / raw) To: Finn Thain; +Cc: Geert Uytterhoeven, linux-m68k Hi Finn, thanks - I'm pretty sure tried that early on but botched it by excessive locking (i.e., keeping preemption disabled when calling get_zeroed_page()!). So I don't think you do use too much locking here. I'm running stress tests for a while now, without any trouble so far. Need to add a few other stressors back in, and repeat all that on a slower ARAnyM instance but I'm quite confident you found the solution. Geert: with this data race fixed, it does appear my RFC patch is no longer needed. Finn or I probably ought to prepare a new RFC patch to go on top of your preemption patch. There is no commit ID to use in a Fixes: tag for that one, correct? Cheers, Michael On 26/03/24 22:58, Finn Thain wrote: > On Tue, 26 Mar 2024, Michael Schmitz wrote: > >> see below (hopefully not whitespace-damaged). >> > Following your approach, I added mutual exclusion to the other candidate > in the same file, namely get_pointer_table(). Together with your patch, > that seems to be sufficient to make stress-ng happy. I don't know the real > extent of the data race here so I've probably used too much locking in > this experiment (?) > > diff --git a/arch/m68k/mm/motorola.c b/arch/m68k/mm/motorola.c > index 068ad7c9bb5c..d1ad55ee1b60 100644 > --- a/arch/m68k/mm/motorola.c > +++ b/arch/m68k/mm/motorola.c > @@ -140,10 +140,14 @@ void __init init_pointer_table(void *table, int type) > > void *get_pointer_table(int type) > { > - ptable_desc *dp = ptable_list[type].next; > - unsigned int mask = list_empty(&ptable_list[type]) ? 0 : PD_MARKBITS(dp); > + ptable_desc *dp; > + unsigned int mask; > unsigned int tmp, off; > > + preempt_disable(); > + > + dp = ptable_list[type].next; > + mask = list_empty(&ptable_list[type]) ? 0 : PD_MARKBITS(dp); > /* > * For a pointer table for a user process address space, a > * table is taken from a page allocated for the purpose. Each > @@ -154,8 +158,13 @@ void *get_pointer_table(int type) > void *page; > ptable_desc *new; > > - if (!(page = (void *)get_zeroed_page(GFP_KERNEL))) > + sched_preempt_enable_no_resched(); > + > + if (!(page = (void *)get_zeroed_page(GFP_KERNEL))) { > return NULL; > + } > + > + preempt_disable(); > > if (type == TABLE_PTE) { > /* > @@ -171,6 +180,7 @@ void *get_pointer_table(int type) > PD_MARKBITS(new) = ptable_mask(type) - 1; > list_add_tail(new, dp); > > + sched_preempt_enable_no_resched(); > return (pmd_t *)page; > } > > @@ -181,6 +191,7 @@ void *get_pointer_table(int type) > /* move to end of list */ > list_move_tail(dp, &ptable_list[type]); > } > + sched_preempt_enable_no_resched(); > return page_address(PD_PAGE(dp)) + off; > } > ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH RFC] m68k: skip kernel premption if interrupts were disabled 2024-03-26 19:59 ` Michael Schmitz @ 2024-03-26 22:03 ` Finn Thain 2024-03-26 22:35 ` Michael Schmitz 2024-03-27 8:22 ` Geert Uytterhoeven 1 sibling, 1 reply; 21+ messages in thread From: Finn Thain @ 2024-03-26 22:03 UTC (permalink / raw) To: Michael Schmitz; +Cc: Geert Uytterhoeven, linux-m68k On Wed, 27 Mar 2024, Michael Schmitz wrote: > thanks - I'm pretty sure tried that early on but botched it by excessive > locking (i.e., keeping preemption disabled when calling > get_zeroed_page()!). > > So I don't think you do use too much locking here. > > I'm running stress tests for a while now, without any trouble so far. > Need to add a few other stressors back in, and repeat all that on a > slower ARAnyM instance but I'm quite confident you found the solution. > > Geert: with this data race fixed, it does appear my RFC patch is no > longer needed. Finn or I probably ought to prepare a new RFC patch to go > on top of your preemption patch. There is no commit ID to use in a > Fixes: tag for that one, correct? > I don't think we need a formal fix. I'd be content for Geert to simply roll your free_pointer_table() changes and my get_pointer_table() changes into his own commit. That way, bisection won't produce an unstable build. ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH RFC] m68k: skip kernel premption if interrupts were disabled 2024-03-26 22:03 ` Finn Thain @ 2024-03-26 22:35 ` Michael Schmitz 2024-03-27 3:50 ` Finn Thain 0 siblings, 1 reply; 21+ messages in thread From: Michael Schmitz @ 2024-03-26 22:35 UTC (permalink / raw) To: Finn Thain; +Cc: Geert Uytterhoeven, linux-m68k Hi Finn, yes, that would be the end goal, but it might be easier for Geert to merge the two patches (or a single one) on top of his preempt branch if they come in the usual git format. Cheers, Michael On 27/03/24 11:03, Finn Thain wrote: > On Wed, 27 Mar 2024, Michael Schmitz wrote: > >> thanks - I'm pretty sure tried that early on but botched it by excessive >> locking (i.e., keeping preemption disabled when calling >> get_zeroed_page()!). >> >> So I don't think you do use too much locking here. >> >> I'm running stress tests for a while now, without any trouble so far. >> Need to add a few other stressors back in, and repeat all that on a >> slower ARAnyM instance but I'm quite confident you found the solution. >> >> Geert: with this data race fixed, it does appear my RFC patch is no >> longer needed. Finn or I probably ought to prepare a new RFC patch to go >> on top of your preemption patch. There is no commit ID to use in a >> Fixes: tag for that one, correct? >> > I don't think we need a formal fix. I'd be content for Geert to simply > roll your free_pointer_table() changes and my get_pointer_table() changes > into his own commit. That way, bisection won't produce an unstable build. ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH RFC] m68k: skip kernel premption if interrupts were disabled 2024-03-26 22:35 ` Michael Schmitz @ 2024-03-27 3:50 ` Finn Thain 2024-03-27 6:56 ` Michael Schmitz 2024-03-27 8:38 ` Finn Thain 0 siblings, 2 replies; 21+ messages in thread From: Finn Thain @ 2024-03-27 3:50 UTC (permalink / raw) To: Michael Schmitz; +Cc: Geert Uytterhoeven, linux-m68k On Wed, 27 Mar 2024, Michael Schmitz wrote: > it might be easier for Geert to merge the two patches (or a single one) > on top of his preempt branch if they come in the usual git format. I suppose that would be easier, assuming it doesn't need a lot of rework. Anyway, here's the diff. diff --git a/arch/m68k/mm/motorola.c b/arch/m68k/mm/motorola.c index c1761d309fc6..d0092a3127c0 100644 --- a/arch/m68k/mm/motorola.c +++ b/arch/m68k/mm/motorola.c @@ -15,6 +15,7 @@ #include <linux/mm.h> #include <linux/swap.h> #include <linux/kernel.h> +#include <linux/preempt.h> #include <linux/string.h> #include <linux/types.h> #include <linux/init.h> @@ -139,10 +140,14 @@ void __init init_pointer_table(void *table, int type) void *get_pointer_table(int type) { - ptable_desc *dp = ptable_list[type].next; - unsigned int mask = list_empty(&ptable_list[type]) ? 0 : PD_MARKBITS(dp); + ptable_desc *dp; + unsigned int mask; unsigned int tmp, off; + preempt_disable(); + + dp = ptable_list[type].next; + mask = list_empty(&ptable_list[type]) ? 0 : PD_MARKBITS(dp); /* * For a pointer table for a user process address space, a * table is taken from a page allocated for the purpose. Each @@ -153,9 +158,13 @@ void *get_pointer_table(int type) void *page; ptable_desc *new; + sched_preempt_enable_no_resched(); + if (!(page = (void *)get_zeroed_page(GFP_KERNEL))) return NULL; + preempt_disable(); + if (type == TABLE_PTE) { /* * m68k doesn't have SPLIT_PTE_PTLOCKS for not having @@ -170,6 +179,7 @@ void *get_pointer_table(int type) PD_MARKBITS(new) = ptable_mask(type) - 1; list_add_tail(new, dp); + sched_preempt_enable_no_resched(); return (pmd_t *)page; } @@ -180,6 +190,7 @@ void *get_pointer_table(int type) /* move to end of list */ list_move_tail(dp, &ptable_list[type]); } + sched_preempt_enable_no_resched(); return page_address(PD_PAGE(dp)) + off; } @@ -190,6 +201,8 @@ int free_pointer_table(void *table, int type) unsigned long page = ptable & PAGE_MASK; unsigned int mask = 1U << ((ptable - page)/ptable_size(type)); + preempt_disable(); + dp = PD_PTABLE(page); if (PD_MARKBITS (dp) & mask) panic ("table already free!"); @@ -203,6 +216,7 @@ int free_pointer_table(void *table, int type) if (type == TABLE_PTE) pagetable_pte_dtor(virt_to_ptdesc((void *)page)); free_page (page); + sched_preempt_enable_no_resched(); return 1; } else if (ptable_list[type].next != dp) { /* @@ -211,6 +225,7 @@ int free_pointer_table(void *table, int type) */ list_move(dp, &ptable_list[type]); } + sched_preempt_enable_no_resched(); return 0; } ^ permalink raw reply related [flat|nested] 21+ messages in thread
* Re: [PATCH RFC] m68k: skip kernel premption if interrupts were disabled 2024-03-27 3:50 ` Finn Thain @ 2024-03-27 6:56 ` Michael Schmitz 2024-03-27 23:29 ` Spinlock debug, was " Finn Thain 2024-03-27 8:38 ` Finn Thain 1 sibling, 1 reply; 21+ messages in thread From: Michael Schmitz @ 2024-03-27 6:56 UTC (permalink / raw) To: Finn Thain; +Cc: Geert Uytterhoeven, linux-m68k Hi Finn, thanks - that should suffice. Tests on the slower ARAnyM instance also passed OK. Leaves test on 030 - I'll see whether I can find an earlier kernel version with a smaller memory footprint to test these patches on. Now none of this has got us any closer to a solution for the spinlock recursion bug... Cheers, Michael Am 27.03.2024 um 16:50 schrieb Finn Thain: > > On Wed, 27 Mar 2024, Michael Schmitz wrote: > >> it might be easier for Geert to merge the two patches (or a single one) >> on top of his preempt branch if they come in the usual git format. > > I suppose that would be easier, assuming it doesn't need a lot of rework. > Anyway, here's the diff. > > diff --git a/arch/m68k/mm/motorola.c b/arch/m68k/mm/motorola.c > index c1761d309fc6..d0092a3127c0 100644 > --- a/arch/m68k/mm/motorola.c > +++ b/arch/m68k/mm/motorola.c > @@ -15,6 +15,7 @@ > #include <linux/mm.h> > #include <linux/swap.h> > #include <linux/kernel.h> > +#include <linux/preempt.h> > #include <linux/string.h> > #include <linux/types.h> > #include <linux/init.h> > @@ -139,10 +140,14 @@ void __init init_pointer_table(void *table, int type) > > void *get_pointer_table(int type) > { > - ptable_desc *dp = ptable_list[type].next; > - unsigned int mask = list_empty(&ptable_list[type]) ? 0 : PD_MARKBITS(dp); > + ptable_desc *dp; > + unsigned int mask; > unsigned int tmp, off; > > + preempt_disable(); > + > + dp = ptable_list[type].next; > + mask = list_empty(&ptable_list[type]) ? 0 : PD_MARKBITS(dp); > /* > * For a pointer table for a user process address space, a > * table is taken from a page allocated for the purpose. Each > @@ -153,9 +158,13 @@ void *get_pointer_table(int type) > void *page; > ptable_desc *new; > > + sched_preempt_enable_no_resched(); > + > if (!(page = (void *)get_zeroed_page(GFP_KERNEL))) > return NULL; > > + preempt_disable(); > + > if (type == TABLE_PTE) { > /* > * m68k doesn't have SPLIT_PTE_PTLOCKS for not having > @@ -170,6 +179,7 @@ void *get_pointer_table(int type) > PD_MARKBITS(new) = ptable_mask(type) - 1; > list_add_tail(new, dp); > > + sched_preempt_enable_no_resched(); > return (pmd_t *)page; > } > > @@ -180,6 +190,7 @@ void *get_pointer_table(int type) > /* move to end of list */ > list_move_tail(dp, &ptable_list[type]); > } > + sched_preempt_enable_no_resched(); > return page_address(PD_PAGE(dp)) + off; > } > > @@ -190,6 +201,8 @@ int free_pointer_table(void *table, int type) > unsigned long page = ptable & PAGE_MASK; > unsigned int mask = 1U << ((ptable - page)/ptable_size(type)); > > + preempt_disable(); > + > dp = PD_PTABLE(page); > if (PD_MARKBITS (dp) & mask) > panic ("table already free!"); > @@ -203,6 +216,7 @@ int free_pointer_table(void *table, int type) > if (type == TABLE_PTE) > pagetable_pte_dtor(virt_to_ptdesc((void *)page)); > free_page (page); > + sched_preempt_enable_no_resched(); > return 1; > } else if (ptable_list[type].next != dp) { > /* > @@ -211,6 +225,7 @@ int free_pointer_table(void *table, int type) > */ > list_move(dp, &ptable_list[type]); > } > + sched_preempt_enable_no_resched(); > return 0; > } > > ^ permalink raw reply [flat|nested] 21+ messages in thread
* Spinlock debug, was Re: [PATCH RFC] m68k: skip kernel premption if interrupts were disabled 2024-03-27 6:56 ` Michael Schmitz @ 2024-03-27 23:29 ` Finn Thain [not found] ` <7aee9fd9-1c0e-4874-bb86-e512a7c0caa6@gmail.com> 0 siblings, 1 reply; 21+ messages in thread From: Finn Thain @ 2024-03-27 23:29 UTC (permalink / raw) To: Michael Schmitz; +Cc: Geert Uytterhoeven, linux-m68k On Wed, 27 Mar 2024, Michael Schmitz wrote: > Now none of this has got us any closer to a solution for the spinlock > recursion bug... > It appears that the spinlock recursion check really works: I am able to reproduce the following failure every time I paste too much text into the qemu console. This looks like a straight-forward driver bug, whereby the "irq flood" printk() results in pmz_console_write() attempting to lock the spinlock already locked by pmz_interrupt(). The spinlock at 0x62e140 is the one in struct uart_port. I'll see if I can come up with a driver patch. [ 14.560000] ttyPZ0: pmz: rx irq flood ! [ 14.560000] BUG: spinlock recursion on CPU#0, swapper/0 [ 14.560000] lock: 0x62e140, .magic: dead4ead, .owner: swapper/0, .owner_cpu: 0 [ 14.560000] CPU: 0 PID: 0 Comm: swapper Not tainted 6.8.0-mac-dbg-preempt-00004-g4143b7b9144a #1 [ 14.560000] Stack from 0059bcc4: [ 14.560000] 0059bcc4 0056316f 0056316f 00002700 004b6444 0059bce4 004ad8c6 0056316f [ 14.560000] 0059bd10 004a6546 00556759 0062e140 dead4ead 0059f892 00000000 00000000 [ 14.560000] 0062e140 0059bde8 005c03d0 0059bd24 0004daf6 0062e140 005567bf 0062e140 [ 14.560000] 0059bd34 004b64c2 0062e140 00000001 0059bd50 002e15ea 0062e140 00000001 [ 14.560000] 0059bde7 0059bde8 005c03d0 0059bdac 0005124e 005c03d0 005cdc00 0000002b [ 14.560000] 005a3caa 005a3caa 00000000 0059bde8 0004ff00 0059be8b 00038200 000529ba [ 14.560000] Call Trace: [<00002700>] ret_from_kernel_thread+0xc/0x14 [ 14.560000] [<004b6444>] _raw_spin_lock+0x0/0x28 [ 14.560000] [<004ad8c6>] dump_stack+0x10/0x16 [ 14.560000] [<004a6546>] spin_dump+0x6e/0x7c [ 14.560000] [<0004daf6>] do_raw_spin_lock+0x9c/0xa6 [ 14.560000] [<004b64c2>] _raw_spin_lock_irqsave+0x2a/0x34 [ 14.560000] [<002e15ea>] pmz_console_write+0x32/0x9a [ 14.560000] [<0005124e>] console_flush_all+0x112/0x3a2 [ 14.560000] [<0004ff00>] console_trylock+0x0/0x7a [ 14.560000] [<00038200>] parameq+0x48/0x6e [ 14.560000] [<000529ba>] __printk_safe_enter+0x0/0x36 [ 14.560000] [<0005113c>] console_flush_all+0x0/0x3a2 [ 14.560000] [<000542c4>] prb_read_valid+0x0/0x1a [ 14.560000] [<004b65a4>] _raw_spin_unlock+0x0/0x38 [ 14.560000] [<0005151e>] console_unlock+0x40/0xb8 [ 14.560000] [<00038200>] parameq+0x48/0x6e [ 14.560000] [<002c778c>] __tty_insert_flip_string_flags+0x0/0x14e [ 14.560000] [<00051798>] vprintk_emit+0x156/0x238 [ 14.560000] [<00051894>] vprintk_default+0x1a/0x1e [ 14.560000] [<000529a8>] vprintk+0x74/0x86 [ 14.560000] [<004a6596>] _printk+0x12/0x16 [ 14.560000] [<002e23be>] pmz_receive_chars+0x1cc/0x394 [ 14.560000] [<004b6444>] _raw_spin_lock+0x0/0x28 [ 14.560000] [<00038226>] parse_args+0x0/0x3a6 [ 14.560000] [<004b6466>] _raw_spin_lock+0x22/0x28 [ 14.560000] [<002e26b4>] pmz_interrupt+0x12e/0x1e0 [ 14.560000] [<00048680>] arch_cpu_idle_enter+0x0/0x8 [ 14.560000] [<00054ebc>] __handle_irq_event_percpu+0x24/0x106 [ 14.560000] [<004ae576>] default_idle_call+0x0/0x46 [ 14.560000] [<00055020>] handle_irq_event+0x30/0x90 [ 14.560000] [<00058320>] handle_simple_irq+0x5e/0xc0 [ 14.560000] [<00048688>] arch_cpu_idle_exit+0x0/0x8 [ 14.560000] [<00054800>] generic_handle_irq+0x3c/0x4a [ 14.560000] [<00002978>] do_IRQ+0x24/0x3a [ 14.560000] [<004ae508>] cpu_idle_poll.isra.0+0x0/0x6e [ 14.560000] [<00002874>] auto_irqhandler_fixup+0x4/0xc [ 14.560000] [<004ae508>] cpu_idle_poll.isra.0+0x0/0x6e [ 14.560000] [<004ae576>] default_idle_call+0x0/0x46 [ 14.560000] [<004ae598>] default_idle_call+0x22/0x46 [ 14.560000] [<00048710>] do_idle+0x6a/0xf0 [ 14.560000] [<000486a6>] do_idle+0x0/0xf0 [ 14.560000] [<000367d2>] find_task_by_pid_ns+0x0/0x2a [ 14.560000] [<0005d064>] __rcu_read_lock+0x0/0x12 [ 14.560000] [<00048a5a>] cpu_startup_entry+0x18/0x1c [ 14.560000] [<00063a06>] __rcu_read_unlock+0x0/0x26 [ 14.560000] [<004ae65a>] kernel_init+0x0/0xfa [ 14.560000] [<0049c5a8>] strcpy+0x0/0x1e [ 14.560000] [<004a6584>] _printk+0x0/0x16 [ 14.560000] [<0049c72a>] strlen+0x0/0x22 [ 14.560000] [<006452d4>] memblock_alloc_try_nid+0x0/0x82 [ 14.560000] [<0063939a>] arch_post_acpi_subsys_init+0x0/0x8 [ 14.560000] [<0063991e>] console_on_rootfs+0x0/0x60 [ 14.560000] [<00638410>] _sinittext+0x410/0xadc [ 14.560000] ^ permalink raw reply [flat|nested] 21+ messages in thread
[parent not found: <7aee9fd9-1c0e-4874-bb86-e512a7c0caa6@gmail.com>]
* Re: Spinlock debug, was Re: [PATCH RFC] m68k: skip kernel premption if interrupts were disabled [not found] ` <7aee9fd9-1c0e-4874-bb86-e512a7c0caa6@gmail.com> @ 2024-03-28 2:48 ` Finn Thain 2024-03-28 19:54 ` Michael Schmitz 0 siblings, 1 reply; 21+ messages in thread From: Finn Thain @ 2024-03-28 2:48 UTC (permalink / raw) To: Michael Schmitz; +Cc: Geert Uytterhoeven, linux-m68k On Thu, 28 Mar 2024, Michael Schmitz wrote: > On 28/03/24 12:29, Finn Thain wrote: > > On Wed, 27 Mar 2024, Michael Schmitz wrote: > > > >> Now none of this has got us any closer to a solution for the spinlock > >> recursion bug... > >> > > It appears that the spinlock recursion check really works: I am able > > to reproduce the following failure every time I paste too much text > > into the qemu console. > > > > This looks like a straight-forward driver bug, whereby the "irq flood" > > printk() results in pmz_console_write() attempting to lock the > > spinlock already locked by pmz_interrupt(). The spinlock at 0x62e140 > > is the one in struct uart_port. > > That's a legitimate recursion warning then. pmz_console_write() should > not retake the lock already held when called from interrupt context. > > I agree the recursion checks did work correctly in this case. > > Spinlock recursion in scheduler_tick() doesn't look possible from > reentering scheduler_tick() through code in its call path, so the > mechanism there would be different. > > We did rule out the VIA timer interrupt (or MFP timer A interrupt) > getting reentered, right? > I never added an assertion to prove that because I don't have a way to reproduce the "spinlock recursion" BUG reported by Guenter. ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Spinlock debug, was Re: [PATCH RFC] m68k: skip kernel premption if interrupts were disabled 2024-03-28 2:48 ` Finn Thain @ 2024-03-28 19:54 ` Michael Schmitz 2024-03-30 2:46 ` Michael Schmitz 0 siblings, 1 reply; 21+ messages in thread From: Michael Schmitz @ 2024-03-28 19:54 UTC (permalink / raw) To: Finn Thain; +Cc: Geert Uytterhoeven, linux-m68k Hi Finn, Am 28.03.2024 um 15:48 schrieb Finn Thain: >> Spinlock recursion in scheduler_tick() doesn't look possible from >> reentering scheduler_tick() through code in its call path, so the >> mechanism there would be different. >> >> We did rule out the VIA timer interrupt (or MFP timer A interrupt) >> getting reentered, right? >> > > I never added an assertion to prove that because I don't have a way to > reproduce the "spinlock recursion" BUG reported by Guenter. Right. I can reproduce the bug in ARAnyM, so I'll check that the timer interrupt isn't reentered there. Doesn't quite amount to the same thing though. Cheers, Michael ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Spinlock debug, was Re: [PATCH RFC] m68k: skip kernel premption if interrupts were disabled 2024-03-28 19:54 ` Michael Schmitz @ 2024-03-30 2:46 ` Michael Schmitz 2024-03-30 6:23 ` Finn Thain 0 siblings, 1 reply; 21+ messages in thread From: Michael Schmitz @ 2024-03-30 2:46 UTC (permalink / raw) To: Finn Thain; +Cc: Geert Uytterhoeven, linux-m68k Hi Finn, Am 29.03.2024 um 08:54 schrieb Michael Schmitz: > Hi Finn, > > Am 28.03.2024 um 15:48 schrieb Finn Thain: >>> Spinlock recursion in scheduler_tick() doesn't look possible from >>> reentering scheduler_tick() through code in its call path, so the >>> mechanism there would be different. >>> >>> We did rule out the VIA timer interrupt (or MFP timer A interrupt) >>> getting reentered, right? >>> >> >> I never added an assertion to prove that because I don't have a way to >> reproduce the "spinlock recursion" BUG reported by Guenter. > > Right. I can reproduce the bug in ARAnyM, so I'll check that the timer > interrupt isn't reentered there. Doesn't quite amount to the same thing > though. The MFP timer C is not reentered in 66 boots even though I've seen it appear in the stack trace of 8 recursion warnings. This was with full preempt - no different from earlier when I got a similar fraction of warnings at boot. Cheers, Michael ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Spinlock debug, was Re: [PATCH RFC] m68k: skip kernel premption if interrupts were disabled 2024-03-30 2:46 ` Michael Schmitz @ 2024-03-30 6:23 ` Finn Thain 2024-03-31 6:14 ` Michael Schmitz 0 siblings, 1 reply; 21+ messages in thread From: Finn Thain @ 2024-03-30 6:23 UTC (permalink / raw) To: Michael Schmitz; +Cc: Geert Uytterhoeven, linux-m68k On Sat, 30 Mar 2024, Michael Schmitz wrote: > > The MFP timer C is not reentered in 66 boots even though I've seen it > appear in the stack trace of 8 recursion warnings. > The spinlock recursion warning gives the address of the spinlock. Did you figure out which spinlock it is? Also, did you capture all 8 backtraces? ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Spinlock debug, was Re: [PATCH RFC] m68k: skip kernel premption if interrupts were disabled 2024-03-30 6:23 ` Finn Thain @ 2024-03-31 6:14 ` Michael Schmitz 0 siblings, 0 replies; 21+ messages in thread From: Michael Schmitz @ 2024-03-31 6:14 UTC (permalink / raw) To: Finn Thain; +Cc: Geert Uytterhoeven, linux-m68k [-- Attachment #1: Type: text/plain, Size: 747 bytes --] Hi Finn, Am 30.03.2024 um 19:23 schrieb Finn Thain: > On Sat, 30 Mar 2024, Michael Schmitz wrote: > >> >> The MFP timer C is not reentered in 66 boots even though I've seen it >> appear in the stack trace of 8 recursion warnings. >> > > The spinlock recursion warning gives the address of the spinlock. Did you > figure out which spinlock it is? Also, did you capture all 8 backtraces? The lock is that in 'runqueues', the single struct rq in use on a single processor system. That much was clear from the backtrace and looking at the code in scheduler_tick() before. I've attached all eight backtraces - the stack contents only varies in a few positions, mostly explainable by differences in the initial stack pointer. Cheers, Michael [-- Attachment #2: spinlockdebug-74-traces.txt --] [-- Type: text/plain, Size: 13671 bytes --] BUG: spinlock recursion on CPU#0, swapper/1 lock: 0x447600, .magic: dead4ead, .owner: swapper/1, .owner_cpu: 0 CPU: 0 PID: 1 Comm: swapper Not tainted 6.8.0-rc7-atari-fpuemu-spinlocktest+ #1775 Stack from 00831e98: 00831e98 003e766e 003e766e 00000000 0000000d 00335630 003e766e 0003aac6 00447600 003dc464 00830000 000321a6 00447600 0083c000 00831ee0 000321cc 00447600 00000000 00831eec 000335be 00814080 0003fdea 00055464 00002604 00006e16 00000000 00000000 0003fe5e 0000000d 00000000 00000000 00000000 00443048 0032c472 0047aa4c 0047aaa8 00000000 00000000 0003fef6 0047aa4c 00000000 0047aa4c 0003ff4e 0047aa4c 0047aaa8 0047aa4c 0047aaa8 0047aa4c Call Trace: [<00335630>] dump_stack+0xc/0x10 [<0003aac6>] do_raw_spin_lock+0x34/0x58 [<000321a6>] raw_spin_rq_lock_nested+0x24/0x38 [<000321cc>] raw_spin_rq_lock.constprop.135+0x12/0x18 [<000335be>] scheduler_tick+0x10/0x66 [<0003fdea>] __irq_wake_thread+0x0/0x40 [<00055464>] update_process_times+0x5e/0x66 [<00002604>] ret_from_kernel_thread+0x4/0x14 [<00006e16>] mfp_timer_c_handler+0x5a/0x70 [<0003fe5e>] __handle_irq_event_percpu+0x34/0xbc [<0032c472>] sprintf+0x0/0x24 [<0003fef6>] handle_irq_event_percpu+0x10/0x3a [<0003ff4e>] handle_irq_event+0x2e/0x54 [<000424f6>] handle_simple_irq+0x56/0x66 [<00335976>] kernel_init+0x0/0xf2 [<0003fa00>] handle_irq_desc+0x1e/0x28 [<000028c0>] do_IRQ+0x20/0x32 [<000027f4>] user_irqvec_fixup+0xc/0x14 [<0032c472>] sprintf+0x0/0x24 [<00002600>] ret_from_kernel_thread+0x0/0x14 RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1. RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1. -- BUG: spinlock recursion on CPU#0, pool_workqueue_/3 lock: 0x447600, .magic: dead4ead, .owner: pool_workqueue_/3, .owner_cpu: 0 CPU: 0 PID: 3 Comm: pool_workqueue_ Not tainted 6.8.0-rc7-atari-fpuemu-spinlocktest+ #1775 Stack from 00837e98: 00837e98 003e766e 003e766e 00000000 0000000d 00335630 003e766e 0003aac6 00447600 003dc464 00836000 000321a6 00447600 0083cc20 00837ee0 000321cc 00447600 00000000 00837eec 000335be 00814080 0003fdea 00055464 00002604 00006e16 00000000 00000000 0003fe5e 0000000d 00000000 00000000 00339ee0 0033a016 000183f2 0047aa4c 0047aaa8 00000000 00000000 0003fef6 0047aa4c 00000000 0047aa4c 0003ff4e 0047aa4c 0047aaa8 0047aa4c 0047aaa8 0047aa4c Call Trace: [<00335630>] dump_stack+0xc/0x10 [<0003aac6>] do_raw_spin_lock+0x34/0x58 [<000321a6>] raw_spin_rq_lock_nested+0x24/0x38 [<000321cc>] raw_spin_rq_lock.constprop.135+0x12/0x18 [<000335be>] scheduler_tick+0x10/0x66 [<0003fdea>] __irq_wake_thread+0x0/0x40 [<00055464>] update_process_times+0x5e/0x66 [<00002604>] ret_from_kernel_thread+0x4/0x14 [<00006e16>] mfp_timer_c_handler+0x5a/0x70 [<0003fe5e>] __handle_irq_event_percpu+0x34/0xbc [<00339ee0>] _raw_spin_lock+0x0/0x22 [<0033a016>] _raw_spin_unlock+0x0/0x32 [<000183f2>] kernel_thread+0x0/0x6c [<0003fef6>] handle_irq_event_percpu+0x10/0x3a [<0003ff4e>] handle_irq_event+0x2e/0x54 [<000424f6>] handle_simple_irq+0x56/0x66 [<0002de6e>] kthread+0x0/0xd4 [<0003fa00>] handle_irq_desc+0x1e/0x28 [<000028c0>] do_IRQ+0x20/0x32 [<000027f4>] user_irqvec_fixup+0xc/0x14 [<00339ee0>] _raw_spin_lock+0x0/0x22 [<0033a016>] _raw_spin_unlock+0x0/0x32 [<000183f2>] kernel_thread+0x0/0x6c -- BUG: spinlock recursion on CPU#0, swapper/2 lock: 0x447600, .magic: dead4ead, .owner: swapper/2, .owner_cpu: 0 CPU: 0 PID: 2 Comm: swapper Not tainted 6.8.0-rc7-atari-fpuemu-spinlocktest+ #1775 Stack from 00835e98: 00835e98 003e766e 003e766e 00000000 0000000d 00335630 003e766e 0003aac6 00447600 003dc464 00834000 000321a6 00447600 0083c610 00835ee0 000321cc 00447600 00000000 00835eec 000335be 00814080 0003fdea 00055464 00002604 00006e16 00000000 00000000 0003fe5e 0000000d 00000000 00000000 00831f10 00000102 0032c472 0047aa4c 0047aaa8 00000000 00000000 0003fef6 0047aa4c 00000000 0047aa4c 0003ff4e 0047aa4c 0047aaa8 0047aa4c 0047aaa8 0047aa4c Call Trace: [<00335630>] dump_stack+0xc/0x10 [<0003aac6>] do_raw_spin_lock+0x34/0x58 [<000321a6>] raw_spin_rq_lock_nested+0x24/0x38 [<000321cc>] raw_spin_rq_lock.constprop.135+0x12/0x18 [<000335be>] scheduler_tick+0x10/0x66 [<0003fdea>] __irq_wake_thread+0x0/0x40 [<00055464>] update_process_times+0x5e/0x66 [<00002604>] ret_from_kernel_thread+0x4/0x14 [<00006e16>] mfp_timer_c_handler+0x5a/0x70 [<0003fe5e>] __handle_irq_event_percpu+0x34/0xbc [<0032c472>] sprintf+0x0/0x24 [<0003fef6>] handle_irq_event_percpu+0x10/0x3a [<0003ff4e>] handle_irq_event+0x2e/0x54 [<000424f6>] handle_simple_irq+0x56/0x66 [<0002dfd2>] kthreadd+0x0/0x12a [<0003fa00>] handle_irq_desc+0x1e/0x28 [<000028c0>] do_IRQ+0x20/0x32 [<000027f4>] user_irqvec_fixup+0xc/0x14 [<0032c472>] sprintf+0x0/0x24 [<00002600>] ret_from_kernel_thread+0x0/0x14 RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1. RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1. -- BUG: spinlock recursion on CPU#0, swapper/2 lock: 0x447600, .magic: dead4ead, .owner: swapper/2, .owner_cpu: 0 CPU: 0 PID: 2 Comm: swapper Not tainted 6.8.0-rc7-atari-fpuemu-spinlocktest+ #1775 Stack from 00835e98: 00835e98 003e766e 003e766e 00000000 0000000d 00335630 003e766e 0003aac6 00447600 003dc464 00834000 000321a6 00447600 0083c610 00835ee0 000321cc 00447600 00000000 00835eec 000335be 00814080 0003fdea 00055464 00002604 00006e16 00000000 00000000 0003fe5e 0000000d 00000000 00000000 00831f10 00000102 0032c472 0047aa4c 0047aaa8 00000000 00000000 0003fef6 0047aa4c 00000000 0047aa4c 0003ff4e 0047aa4c 0047aaa8 0047aa4c 0047aaa8 0047aa4c Call Trace: [<00335630>] dump_stack+0xc/0x10 [<0003aac6>] do_raw_spin_lock+0x34/0x58 [<000321a6>] raw_spin_rq_lock_nested+0x24/0x38 [<000321cc>] raw_spin_rq_lock.constprop.135+0x12/0x18 [<000335be>] scheduler_tick+0x10/0x66 [<0003fdea>] __irq_wake_thread+0x0/0x40 [<00055464>] update_process_times+0x5e/0x66 [<00002604>] ret_from_kernel_thread+0x4/0x14 [<00006e16>] mfp_timer_c_handler+0x5a/0x70 [<0003fe5e>] __handle_irq_event_percpu+0x34/0xbc [<0032c472>] sprintf+0x0/0x24 [<0003fef6>] handle_irq_event_percpu+0x10/0x3a [<0003ff4e>] handle_irq_event+0x2e/0x54 [<000424f6>] handle_simple_irq+0x56/0x66 [<0002dfd2>] kthreadd+0x0/0x12a [<0003fa00>] handle_irq_desc+0x1e/0x28 [<000028c0>] do_IRQ+0x20/0x32 [<000027f4>] user_irqvec_fixup+0xc/0x14 [<0032c472>] sprintf+0x0/0x24 [<00002600>] ret_from_kernel_thread+0x0/0x14 RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1. RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1. -- BUG: spinlock recursion on CPU#0, swapper/2 lock: 0x447600, .magic: dead4ead, .owner: swapper/2, .owner_cpu: 0 CPU: 0 PID: 2 Comm: swapper Not tainted 6.8.0-rc7-atari-fpuemu-spinlocktest+ #1775 Stack from 00835e98: 00835e98 003e766e 003e766e 00000000 0000000d 00335630 003e766e 0003aac6 00447600 003dc464 00834000 000321a6 00447600 0083c610 00835ee0 000321cc 00447600 00000000 00835eec 000335be 00814080 0003fdea 00055464 00002604 00006e16 00000000 00000000 0003fe5e 0000000d 00000000 00000000 00831f10 00000102 0032c472 0047aa4c 0047aaa8 00000000 00000000 0003fef6 0047aa4c 00000000 0047aa4c 0003ff4e 0047aa4c 0047aaa8 0047aa4c 0047aaa8 0047aa4c Call Trace: [<00335630>] dump_stack+0xc/0x10 [<0003aac6>] do_raw_spin_lock+0x34/0x58 [<000321a6>] raw_spin_rq_lock_nested+0x24/0x38 [<000321cc>] raw_spin_rq_lock.constprop.135+0x12/0x18 [<000335be>] scheduler_tick+0x10/0x66 [<0003fdea>] __irq_wake_thread+0x0/0x40 [<00055464>] update_process_times+0x5e/0x66 [<00002604>] ret_from_kernel_thread+0x4/0x14 [<00006e16>] mfp_timer_c_handler+0x5a/0x70 [<0003fe5e>] __handle_irq_event_percpu+0x34/0xbc [<0032c472>] sprintf+0x0/0x24 [<0003fef6>] handle_irq_event_percpu+0x10/0x3a [<0003ff4e>] handle_irq_event+0x2e/0x54 [<000424f6>] handle_simple_irq+0x56/0x66 [<0002dfd2>] kthreadd+0x0/0x12a [<0003fa00>] handle_irq_desc+0x1e/0x28 [<000028c0>] do_IRQ+0x20/0x32 [<000027f4>] user_irqvec_fixup+0xc/0x14 [<0032c472>] sprintf+0x0/0x24 [<00002600>] ret_from_kernel_thread+0x0/0x14 RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1. RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1. -- BUG: spinlock recursion on CPU#0, pool_workqueue_/3 lock: 0x447600, .magic: dead4ead, .owner: pool_workqueue_/3, .owner_cpu: 0 CPU: 0 PID: 3 Comm: pool_workqueue_ Not tainted 6.8.0-rc7-atari-fpuemu-spinlocktest+ #1775 Stack from 00837e78: 00837e78 003e766e 003e766e 00000000 0000000d 00335630 003e766e 0003aac6 00447600 003dc464 00836000 000321a6 00447600 0083cc20 00837ec0 000321cc 00447600 00000000 00837ecc 000335be 00814080 0003fdea 00055464 00002604 00006e16 00000000 00000000 0003fe5e 0000000d 00000000 00000000 00339ee0 0033a016 000183f2 0047aa4c 0047aaa8 00000000 00837fb4 0003fef6 0047aa4c 00000000 0047aa4c 0003ff4e 0047aa4c 0047aaa8 0047aa4c 0047aaa8 0047aa4c Call Trace: [<00335630>] dump_stack+0xc/0x10 [<0003aac6>] do_raw_spin_lock+0x34/0x58 [<000321a6>] raw_spin_rq_lock_nested+0x24/0x38 [<000321cc>] raw_spin_rq_lock.constprop.135+0x12/0x18 [<000335be>] scheduler_tick+0x10/0x66 [<0003fdea>] __irq_wake_thread+0x0/0x40 [<00055464>] update_process_times+0x5e/0x66 [<00002604>] ret_from_kernel_thread+0x4/0x14 [<00006e16>] mfp_timer_c_handler+0x5a/0x70 [<0003fe5e>] __handle_irq_event_percpu+0x34/0xbc [<00339ee0>] _raw_spin_lock+0x0/0x22 [<0033a016>] _raw_spin_unlock+0x0/0x32 [<000183f2>] kernel_thread+0x0/0x6c [<0003fef6>] handle_irq_event_percpu+0x10/0x3a [<0003ff4e>] handle_irq_event+0x2e/0x54 [<000424f6>] handle_simple_irq+0x56/0x66 [<0003fa00>] handle_irq_desc+0x1e/0x28 [<000028c0>] do_IRQ+0x20/0x32 [<000027f4>] user_irqvec_fixup+0xc/0x14 [<00339ee0>] _raw_spin_lock+0x0/0x22 [<0033a016>] _raw_spin_unlock+0x0/0x32 [<000183f2>] kernel_thread+0x0/0x6c [<0002de6e>] kthread+0x0/0xd4 -- BUG: spinlock recursion on CPU#0, swapper/1 lock: 0x447600, .magic: dead4ead, .owner: swapper/1, .owner_cpu: 0 CPU: 0 PID: 1 Comm: swapper Not tainted 6.8.0-rc7-atari-fpuemu-spinlocktest+ #1775 Stack from 00831e98: 00831e98 003e766e 003e766e 00000000 0000000d 00335630 003e766e 0003aac6 00447600 003dc464 00830000 000321a6 00447600 0083c000 00831ee0 000321cc 00447600 00000000 00831eec 000335be 00814080 0003fdea 00055464 00002604 00006e16 00000000 00000000 0003fe5e 0000000d 00000000 00000000 00000000 00443048 0032c472 0047aa4c 0047aaa8 00000000 00000000 0003fef6 0047aa4c 00000000 0047aa4c 0003ff4e 0047aa4c 0047aaa8 0047aa4c 0047aaa8 0047aa4c Call Trace: [<00335630>] dump_stack+0xc/0x10 [<0003aac6>] do_raw_spin_lock+0x34/0x58 [<000321a6>] raw_spin_rq_lock_nested+0x24/0x38 [<000321cc>] raw_spin_rq_lock.constprop.135+0x12/0x18 [<000335be>] scheduler_tick+0x10/0x66 [<0003fdea>] __irq_wake_thread+0x0/0x40 [<00055464>] update_process_times+0x5e/0x66 [<00002604>] ret_from_kernel_thread+0x4/0x14 [<00006e16>] mfp_timer_c_handler+0x5a/0x70 [<0003fe5e>] __handle_irq_event_percpu+0x34/0xbc [<0032c472>] sprintf+0x0/0x24 [<0003fef6>] handle_irq_event_percpu+0x10/0x3a [<0003ff4e>] handle_irq_event+0x2e/0x54 [<000424f6>] handle_simple_irq+0x56/0x66 [<00335976>] kernel_init+0x0/0xf2 [<0003fa00>] handle_irq_desc+0x1e/0x28 [<000028c0>] do_IRQ+0x20/0x32 [<000027f4>] user_irqvec_fixup+0xc/0x14 [<0032c472>] sprintf+0x0/0x24 [<00002600>] ret_from_kernel_thread+0x0/0x14 RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1. RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1. -- BUG: spinlock recursion on CPU#0, swapper/2 lock: 0x447600, .magic: dead4ead, .owner: swapper/2, .owner_cpu: 0 CPU: 0 PID: 2 Comm: swapper Not tainted 6.8.0-rc7-atari-fpuemu-spinlocktest+ #1775 Stack from 00835e98: 00835e98 003e766e 003e766e 00000000 0000000d 00335630 003e766e 0003aac6 00447600 003dc464 00834000 000321a6 00447600 0083c610 00835ee0 000321cc 00447600 00000000 00835eec 000335be 00814080 0003fdea 00055464 00002604 00006e16 00000000 00000000 0003fe5e 0000000d 00000000 00000000 00831f10 00000102 0032c472 0047aa4c 0047aaa8 00000000 00000000 0003fef6 0047aa4c 00000000 0047aa4c 0003ff4e 0047aa4c 0047aaa8 0047aa4c 0047aaa8 0047aa4c Call Trace: [<00335630>] dump_stack+0xc/0x10 [<0003aac6>] do_raw_spin_lock+0x34/0x58 [<000321a6>] raw_spin_rq_lock_nested+0x24/0x38 [<000321cc>] raw_spin_rq_lock.constprop.135+0x12/0x18 [<000335be>] scheduler_tick+0x10/0x66 [<0003fdea>] __irq_wake_thread+0x0/0x40 [<00055464>] update_process_times+0x5e/0x66 [<00002604>] ret_from_kernel_thread+0x4/0x14 [<00006e16>] mfp_timer_c_handler+0x5a/0x70 [<0003fe5e>] __handle_irq_event_percpu+0x34/0xbc [<0032c472>] sprintf+0x0/0x24 [<0003fef6>] handle_irq_event_percpu+0x10/0x3a [<0003ff4e>] handle_irq_event+0x2e/0x54 [<000424f6>] handle_simple_irq+0x56/0x66 [<0002dfd2>] kthreadd+0x0/0x12a [<0003fa00>] handle_irq_desc+0x1e/0x28 [<000028c0>] do_IRQ+0x20/0x32 [<000027f4>] user_irqvec_fixup+0xc/0x14 [<0032c472>] sprintf+0x0/0x24 [<00002600>] ret_from_kernel_thread+0x0/0x14 RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1. RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1. ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH RFC] m68k: skip kernel premption if interrupts were disabled 2024-03-27 3:50 ` Finn Thain 2024-03-27 6:56 ` Michael Schmitz @ 2024-03-27 8:38 ` Finn Thain 2024-03-27 22:59 ` Michael Schmitz 1 sibling, 1 reply; 21+ messages in thread From: Finn Thain @ 2024-03-27 8:38 UTC (permalink / raw) To: Michael Schmitz; +Cc: Geert Uytterhoeven, linux-m68k On Wed, 27 Mar 2024, Finn Thain wrote: > > On Wed, 27 Mar 2024, Michael Schmitz wrote: > > > it might be easier for Geert to merge the two patches (or a single one) > > on top of his preempt branch if they come in the usual git format. > > I suppose that would be easier, assuming it doesn't need a lot of rework. > Anyway, here's the diff. > > diff --git a/arch/m68k/mm/motorola.c b/arch/m68k/mm/motorola.c > index c1761d309fc6..d0092a3127c0 100644 > --- a/arch/m68k/mm/motorola.c > +++ b/arch/m68k/mm/motorola.c > @@ -15,6 +15,7 @@ > #include <linux/mm.h> > #include <linux/swap.h> > #include <linux/kernel.h> > +#include <linux/preempt.h> > #include <linux/string.h> > #include <linux/types.h> > #include <linux/init.h> > @@ -139,10 +140,14 @@ void __init init_pointer_table(void *table, int type) > > void *get_pointer_table(int type) > { > - ptable_desc *dp = ptable_list[type].next; > - unsigned int mask = list_empty(&ptable_list[type]) ? 0 : PD_MARKBITS(dp); > + ptable_desc *dp; > + unsigned int mask; > unsigned int tmp, off; > > + preempt_disable(); > + > + dp = ptable_list[type].next; > + mask = list_empty(&ptable_list[type]) ? 0 : PD_MARKBITS(dp); > /* > * For a pointer table for a user process address space, a > * table is taken from a page allocated for the purpose. Each > @@ -153,9 +158,13 @@ void *get_pointer_table(int type) > void *page; > ptable_desc *new; > > + sched_preempt_enable_no_resched(); > + > if (!(page = (void *)get_zeroed_page(GFP_KERNEL))) > return NULL; > > + preempt_disable(); > + > if (type == TABLE_PTE) { > /* > * m68k doesn't have SPLIT_PTE_PTLOCKS for not having > @@ -170,6 +179,7 @@ void *get_pointer_table(int type) > PD_MARKBITS(new) = ptable_mask(type) - 1; > list_add_tail(new, dp); > > + sched_preempt_enable_no_resched(); > return (pmd_t *)page; > } > > @@ -180,6 +190,7 @@ void *get_pointer_table(int type) > /* move to end of list */ > list_move_tail(dp, &ptable_list[type]); > } > + sched_preempt_enable_no_resched(); > return page_address(PD_PAGE(dp)) + off; > } > > @@ -190,6 +201,8 @@ int free_pointer_table(void *table, int type) > unsigned long page = ptable & PAGE_MASK; > unsigned int mask = 1U << ((ptable - page)/ptable_size(type)); > > + preempt_disable(); > + > dp = PD_PTABLE(page); > if (PD_MARKBITS (dp) & mask) > panic ("table already free!"); > @@ -203,6 +216,7 @@ int free_pointer_table(void *table, int type) > if (type == TABLE_PTE) > pagetable_pte_dtor(virt_to_ptdesc((void *)page)); > free_page (page); > + sched_preempt_enable_no_resched(); > return 1; > } else if (ptable_list[type].next != dp) { > /* > @@ -211,6 +225,7 @@ int free_pointer_table(void *table, int type) > */ > list_move(dp, &ptable_list[type]); > } > + sched_preempt_enable_no_resched(); > return 0; > } > > For my part, Signed-off-by: Finn Thain <fthain@linux-m68k.org> ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH RFC] m68k: skip kernel premption if interrupts were disabled 2024-03-27 8:38 ` Finn Thain @ 2024-03-27 22:59 ` Michael Schmitz 0 siblings, 0 replies; 21+ messages in thread From: Michael Schmitz @ 2024-03-27 22:59 UTC (permalink / raw) To: Finn Thain, Geert Uytterhoeven; +Cc: linux-m68k Hi Geert, On 27/03/24 21:38, Finn Thain wrote: > On Wed, 27 Mar 2024, Finn Thain wrote: > >> On Wed, 27 Mar 2024, Michael Schmitz wrote: >> >>> it might be easier for Geert to merge the two patches (or a single one) >>> on top of his preempt branch if they come in the usual git format. >> I suppose that would be easier, assuming it doesn't need a lot of rework. >> Anyway, here's the diff. >> >> diff --git a/arch/m68k/mm/motorola.c b/arch/m68k/mm/motorola.c >> index c1761d309fc6..d0092a3127c0 100644 >> --- a/arch/m68k/mm/motorola.c >> +++ b/arch/m68k/mm/motorola.c >> @@ -15,6 +15,7 @@ >> #include <linux/mm.h> >> #include <linux/swap.h> >> #include <linux/kernel.h> >> +#include <linux/preempt.h> >> #include <linux/string.h> >> #include <linux/types.h> >> #include <linux/init.h> >> @@ -139,10 +140,14 @@ void __init init_pointer_table(void *table, int type) >> >> void *get_pointer_table(int type) >> { >> - ptable_desc *dp = ptable_list[type].next; >> - unsigned int mask = list_empty(&ptable_list[type]) ? 0 : PD_MARKBITS(dp); >> + ptable_desc *dp; >> + unsigned int mask; >> unsigned int tmp, off; >> >> + preempt_disable(); >> + >> + dp = ptable_list[type].next; >> + mask = list_empty(&ptable_list[type]) ? 0 : PD_MARKBITS(dp); >> /* >> * For a pointer table for a user process address space, a >> * table is taken from a page allocated for the purpose. Each >> @@ -153,9 +158,13 @@ void *get_pointer_table(int type) >> void *page; >> ptable_desc *new; >> >> + sched_preempt_enable_no_resched(); >> + >> if (!(page = (void *)get_zeroed_page(GFP_KERNEL))) >> return NULL; >> >> + preempt_disable(); >> + >> if (type == TABLE_PTE) { >> /* >> * m68k doesn't have SPLIT_PTE_PTLOCKS for not having >> @@ -170,6 +179,7 @@ void *get_pointer_table(int type) >> PD_MARKBITS(new) = ptable_mask(type) - 1; >> list_add_tail(new, dp); >> >> + sched_preempt_enable_no_resched(); >> return (pmd_t *)page; >> } >> >> @@ -180,6 +190,7 @@ void *get_pointer_table(int type) >> /* move to end of list */ >> list_move_tail(dp, &ptable_list[type]); >> } >> + sched_preempt_enable_no_resched(); >> return page_address(PD_PAGE(dp)) + off; >> } >> >> @@ -190,6 +201,8 @@ int free_pointer_table(void *table, int type) >> unsigned long page = ptable & PAGE_MASK; >> unsigned int mask = 1U << ((ptable - page)/ptable_size(type)); >> >> + preempt_disable(); >> + >> dp = PD_PTABLE(page); >> if (PD_MARKBITS (dp) & mask) >> panic ("table already free!"); >> @@ -203,6 +216,7 @@ int free_pointer_table(void *table, int type) >> if (type == TABLE_PTE) >> pagetable_pte_dtor(virt_to_ptdesc((void *)page)); >> free_page (page); >> + sched_preempt_enable_no_resched(); >> return 1; >> } else if (ptable_list[type].next != dp) { >> /* >> @@ -211,6 +225,7 @@ int free_pointer_table(void *table, int type) >> */ >> list_move(dp, &ptable_list[type]); >> } >> + sched_preempt_enable_no_resched(); >> return 0; >> } >> >> > For my part, > > Signed-off-by: Finn Thain <fthain@linux-m68k.org> Same for me: Signed-off-by: Michael Schmitz <schmitzmic@gmail.com> ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH RFC] m68k: skip kernel premption if interrupts were disabled 2024-03-26 19:59 ` Michael Schmitz 2024-03-26 22:03 ` Finn Thain @ 2024-03-27 8:22 ` Geert Uytterhoeven 1 sibling, 0 replies; 21+ messages in thread From: Geert Uytterhoeven @ 2024-03-27 8:22 UTC (permalink / raw) To: Michael Schmitz; +Cc: Finn Thain, linux-m68k Hi Michael, On Tue, Mar 26, 2024 at 8:59 PM Michael Schmitz <schmitzmic@gmail.com> wrote: > thanks - I'm pretty sure tried that early on but botched it by excessive > locking (i.e., keeping preemption disabled when calling > get_zeroed_page()!). > > So I don't think you do use too much locking here. > > I'm running stress tests for a while now, without any trouble so far. > Need to add a few other stressors back in, and repeat all that on a > slower ARAnyM instance but I'm quite confident you found the solution. > > Geert: with this data race fixed, it does appear my RFC patch is no > longer needed. Finn or I probably ought to prepare a new RFC patch to go > on top of your preemption patch. There is no commit ID to use in a > Fixes: tag for that one, correct? As my preemption patch was not applied to any tree yet, I'll just fold in any fixes. I think this does deserve Co-developed-by tags, so please make sure to include your SoB in any fixes. Thanks! Gr{oetje,eeting}s, Geert -- Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds ^ permalink raw reply [flat|nested] 21+ messages in thread
end of thread, other threads:[~2024-03-31 6:14 UTC | newest] Thread overview: 21+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2024-03-22 1:48 [PATCH RFC] m68k: skip kernel premption if interrupts were disabled Michael Schmitz [not found] ` <fcabda73-4b16-ee2c-e993-2bfcf392a26a@linux-m68k.org> [not found] ` <e89bf618-2fa8-7cbd-16ed-d021a9a7f740@gmail.com> [not found] ` <948c12e4-e85f-a86c-ae95-a1eb03ca026d@gmail.com> [not found] ` <40928e38-0de7-75a6-d5f7-8c913155da03@linux-m68k.org> 2024-03-23 6:32 ` Michael Schmitz 2024-03-25 6:41 ` Michael Schmitz 2024-03-25 22:28 ` Finn Thain 2024-03-25 22:31 ` Michael Schmitz 2024-03-26 0:41 ` Finn Thain 2024-03-26 9:58 ` Finn Thain 2024-03-26 19:59 ` Michael Schmitz 2024-03-26 22:03 ` Finn Thain 2024-03-26 22:35 ` Michael Schmitz 2024-03-27 3:50 ` Finn Thain 2024-03-27 6:56 ` Michael Schmitz 2024-03-27 23:29 ` Spinlock debug, was " Finn Thain [not found] ` <7aee9fd9-1c0e-4874-bb86-e512a7c0caa6@gmail.com> 2024-03-28 2:48 ` Finn Thain 2024-03-28 19:54 ` Michael Schmitz 2024-03-30 2:46 ` Michael Schmitz 2024-03-30 6:23 ` Finn Thain 2024-03-31 6:14 ` Michael Schmitz 2024-03-27 8:38 ` Finn Thain 2024-03-27 22:59 ` Michael Schmitz 2024-03-27 8:22 ` Geert Uytterhoeven
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.