Boot regression in Linux v6.4-rc3

Boot regression in Linux v6.4-rc3

[Frank Scheiner]

Dear all,

there is a boot regression in effect in Linux v6.4-rc3 that affects at
least:

* rx2620 (w/2 x Montecito and zx1)
* rx2800-i2 (w/1 x Tukwila)

...(see second part of [1] and following posts for more details, [2] and
[3] for the respective logs), example here:

```
ELILO v3.16 for EFI/IA-64
..
Uncompressing Linux... done
Loading file AC100221.initrd.img...done
[    0.000000] Linux version 6.4.0-rc3 (root@x4270) (ia64-linux-gcc
(GCC) 12.2.0, GNU ld (GNU Binutils) 2.39) #1 SMP Thu May 25 15:52:20
CEST 2023
[    0.000000] efi: EFI v1.1 by HP
[    0.000000] efi: SALsystab=0x3ee7a000 ACPI 2.0=0x3fe2a000
ESI=0x3ee7b000 SMBIOS=0x3ee7c000 HCDP=0x3fe28000
[    0.000000] PCDP: v3 at 0x3fe28000
[    0.000000] earlycon: uart8250 at MMIO 0x00000000f4050000 (options
'9600n8')
[    0.000000] printk: bootconsole [uart8250] enabled
[    0.000000] ACPI: Early table checksum verification disabled
[    0.000000] ACPI: RSDP 0x000000003FE2A000 000028 (v02 HP    )
[    0.000000] ACPI: XSDT 0x000000003FE2A02C 0000CC (v01 HP     rx2620
00000000 HP   00000000)
[...]
[    3.793350] Run /init as init process
Loading, please wait...
Starting systemd-udevd version 252.6-1
[    3.951100] ------------[ cut here ]------------
[    3.951100] WARNING: CPU: 6 PID: 140 at kernel/module/main.c:1547
__layout_sections+0x370/0x3c0
[    3.949512] Unable to handle kernel paging request at virtual address
1000000000000000
[    3.951100] Modules linked in:
[    3.951100] CPU: 6 PID: 140 Comm: (udev-worker) Not tainted 6.4.0-rc3 #1
[    3.956161] (udev-worker)[142]: Oops 11003706212352 [1]
[    3.951774] Hardware name: hp server rx2620                   , BIOS
04.29
11/30/2007
[    3.951774]
[    3.951774] Call Trace:
[    3.958339] Unable to handle kernel paging request at virtual address
1000000000000000
[    3.956161] Modules linked in:
[    3.951774]  [<a0000001000156d0>] show_stack.part.0+0x30/0x60
[    3.951774]                                 sp=e000000183a67b20
bsp=e000000183a61628
[    3.956161]
[    3.956161]
```

[1]: https://lists.debian.org/debian-ia64/2023/05/msg00010.html

[2]: https://pastebin.com/SAUKbG7Z

[3]: https://pastebin.com/v1TTB2x3

With the needed modules compiled into the kernel the rx2620 (only tested
there yet) boots correctly, though for v6.4-rc2 with kernel oopses (with
similar content), for v6.4-rc3 actually w/o kernel oopses.

According to bisecting between:

GOOD: `cec24b8b6bb841a19b5c5555b600a511a8988100` and

BAD: `b6a7828502dc769e1a5329027bc5048222fa210a` (already in effect there)

...the problem was introduced with:

```
root@x4270:/usr/src/linux-on-ramdisk# git bisect bad
ac3b43283923440900b4f36ca5f9f0b1ca43b70e is the first bad commit
commit ac3b43283923440900b4f36ca5f9f0b1ca43b70e
Author: Song Liu <song@kernel.org>
Date:   Mon Feb 6 16:28:02 2023 -0800

     module: replace module_layout with module_memory

     module_layout manages different types of memory (text, data,
rodata, etc.)
     in one allocation, which is problematic for some reasons:

     1. It is hard to enable CONFIG_STRICT_MODULE_RWX.
     2. It is hard to use huge pages in modules (and not break strict rwx).
     3. Many archs uses module_layout for arch-specific data, but it is not
        obvious how these data are used (are they RO, RX, or RW?)

     Improve the scenario by replacing 2 (or 3) module_layout per module
with
     up to 7 module_memory per module:

             MOD_TEXT,
             MOD_DATA,
             MOD_RODATA,
             MOD_RO_AFTER_INIT,
             MOD_INIT_TEXT,
             MOD_INIT_DATA,
             MOD_INIT_RODATA,

     and allocating them separately. This adds slightly more entries to
     mod_tree (from up to 3 entries per module, to up to 7 entries per
     module). However, this at most adds a small constant overhead to
     __module_address(), which is expected to be fast.

     Various archs use module_layout for different data. These data are put
     into different module_memory based on their location in module_layout.
     IOW, data that used to go with text is allocated with
MOD_MEM_TYPE_TEXT;
     data that used to go with data is allocated with MOD_MEM_TYPE_DATA,
etc.

     module_memory simplifies quite some of the module code. For example,
     ARCH_WANTS_MODULES_DATA_IN_VMALLOC is a lot cleaner, as it just uses a
     different allocator for the data. kernel/module/strict_rwx.c is also
     much cleaner with module_memory.

     Signed-off-by: Song Liu <song@kernel.org>
     Cc: Luis Chamberlain <mcgrof@kernel.org>
     Cc: Thomas Gleixner <tglx@linutronix.de>
     Cc: Peter Zijlstra <peterz@infradead.org>
     Cc: Guenter Roeck <linux@roeck-us.net>
     Cc: Christophe Leroy <christophe.leroy@csgroup.eu>
     Reviewed-by: Thomas Gleixner <tglx@linutronix.de>
     Reviewed-by: Christophe Leroy <christophe.leroy@csgroup.eu>
     Reviewed-by: Luis Chamberlain <mcgrof@kernel.org>
     Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>

  arch/arc/kernel/unwind.c        |  12 +-
  arch/arm/kernel/module-plts.c   |   9 +-
  arch/arm64/kernel/module-plts.c |  13 +-
  arch/ia64/kernel/module.c       |  24 +--
  arch/mips/kernel/vpe.c          |  11 +-
  arch/parisc/kernel/module.c     |  51 ++----
  arch/powerpc/kernel/module_32.c |   7 +-
  arch/s390/kernel/module.c       |  26 +--
  arch/x86/kernel/callthunks.c    |   4 +-
  arch/x86/kernel/module.c        |   4 +-
  include/linux/module.h          |  89 +++++++---
  kernel/module/internal.h        |  40 ++---
  kernel/module/kallsyms.c        |  58 ++++---
  kernel/module/kdb.c             |  17 +-
  kernel/module/main.c            | 375
++++++++++++++++++++--------------------
  kernel/module/procfs.c          |  16 +-
  kernel/module/strict_rwx.c      |  99 ++---------
  kernel/module/tree_lookup.c     |  39 ++---
  18 files changed, 427 insertions(+), 467 deletions(-)

root@x4270:/usr/src/linux-on-ramdisk# git bisect log
git bisect start
# status: waiting for both good and bad commits
# good: [cec24b8b6bb841a19b5c5555b600a511a8988100] Merge tag
'char-misc-6.4-rc1' of
git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/char-misc
git bisect good cec24b8b6bb841a19b5c5555b600a511a8988100
# status: waiting for bad commit, 1 good commit known
# bad: [b6a7828502dc769e1a5329027bc5048222fa210a] Merge tag
'modules-6.4-rc1' of
git://git.kernel.org/pub/scm/linux/kernel/git/mcgrof/linux
git bisect bad b6a7828502dc769e1a5329027bc5048222fa210a
# bad: [3f0dedc39039a75670817a1afffa77b6cee077cb] dmaengine: remove
MODULE_LICENSE in non-modules
git bisect bad 3f0dedc39039a75670817a1afffa77b6cee077cb
# bad: [b10addf37bbcaee66672eb54c15532266c8daea6] module: add
symbol-name to pr_debug Absolute symbol
git bisect bad b10addf37bbcaee66672eb54c15532266c8daea6
# bad: [85e6f61c134f111232d27d3f63667c1bccbbc12d] module: move early
sanity checks into a helper
git bisect bad 85e6f61c134f111232d27d3f63667c1bccbbc12d
# bad: [05777499a81298ef7e4a5e32a6f744f1f937a80c] ARM: dyndbg: allow
including dyndbg.h in decompressor
git bisect bad 05777499a81298ef7e4a5e32a6f744f1f937a80c
# bad: [efaa2496bae66f0a78efa60d9b73ceef5ec63d79] module: fix MIPS
module_layout -> module_memory
git bisect bad efaa2496bae66f0a78efa60d9b73ceef5ec63d79
# bad: [9e07f161717ab8e8ac1206bf82546511e24cbb7b] module: Remove the
unused function within
git bisect bad 9e07f161717ab8e8ac1206bf82546511e24cbb7b
# bad: [ac3b43283923440900b4f36ca5f9f0b1ca43b70e] module: replace
module_layout with module_memory
git bisect bad ac3b43283923440900b4f36ca5f9f0b1ca43b70e
# first bad commit: [ac3b43283923440900b4f36ca5f9f0b1ca43b70e] module:
replace module_layout with module_memory
```

...and merged with commit `b6a7828502dc769e1a5329027bc5048222fa210a`:

```
commit b6a7828502dc769e1a5329027bc5048222fa210a
Merge: d06f5a3f7140 8660484ed1cf
Author: Linus Torvalds <torvalds@linux-foundation.org>
Date:   Thu Apr 27 16:36:55 2023 -0700

     Merge tag 'modules-6.4-rc1' of
git://git.kernel.org/pub/scm/linux/kernel/git/mcgrof/linux

     Pull module updates from Luis Chamberlain:
      "The summary of the changes for this pull requests is:

        - Song Liu's new struct module_memory replacement

        - Nick Alcock's MODULE_LICENSE() removal for non-modules

        - My cleanups and enhancements to reduce the areas where we vmalloc
          module memory for duplicates, and the respective debug code which
          proves the remaining vmalloc pressure comes from userspace.
[...]
```

Could someone have a look into this, please?

Cheers,
Frank

P.S.
There is also a bug for this specific commit:

```
kmemleaks on ac3b43283923 ("module: replace module_layout with
module_memory")
```

...on [4], reported on 2023-04-03, but I don't know if its content is
related to the problems on ia64.

[4]: https://bugzilla.kernel.org/show_bug.cgi?id=217296

Re: Boot regression in Linux v6.4-rc3

[Song Liu]

Hi Frank,

Thanks for the report.

On Fri, May 26, 2023 at 3:55 AM Frank Scheiner <frank.scheiner@web.de> wrote:
>
> Dear all,
>
> there is a boot regression in effect in Linux v6.4-rc3 that affects at
> least:
>
> * rx2620 (w/2 x Montecito and zx1)
> * rx2800-i2 (w/1 x Tukwila)
>
> ...(see second part of [1] and following posts for more details, [2] and
> [3] for the respective logs), example here:
>
> ```
> ELILO v3.16 for EFI/IA-64
> ..
> Uncompressing Linux... done
> Loading file AC100221.initrd.img...done
> [    0.000000] Linux version 6.4.0-rc3 (root@x4270) (ia64-linux-gcc
> (GCC) 12.2.0, GNU ld (GNU Binutils) 2.39) #1 SMP Thu May 25 15:52:20
> CEST 2023
> [    0.000000] efi: EFI v1.1 by HP
> [    0.000000] efi: SALsystab=0x3ee7a000 ACPI 2.0=0x3fe2a000
> ESI=0x3ee7b000 SMBIOS=0x3ee7c000 HCDP=0x3fe28000
> [    0.000000] PCDP: v3 at 0x3fe28000
> [    0.000000] earlycon: uart8250 at MMIO 0x00000000f4050000 (options
> '9600n8')
> [    0.000000] printk: bootconsole [uart8250] enabled
> [    0.000000] ACPI: Early table checksum verification disabled
> [    0.000000] ACPI: RSDP 0x000000003FE2A000 000028 (v02 HP    )
> [    0.000000] ACPI: XSDT 0x000000003FE2A02C 0000CC (v01 HP     rx2620
> 00000000 HP   00000000)
> [...]
> [    3.793350] Run /init as init process
> Loading, please wait...
> Starting systemd-udevd version 252.6-1
> [    3.951100] ------------[ cut here ]------------
> [    3.951100] WARNING: CPU: 6 PID: 140 at kernel/module/main.c:1547
> __layout_sections+0x370/0x3c0
> [    3.949512] Unable to handle kernel paging request at virtual address
> 1000000000000000
> [    3.951100] Modules linked in:
> [    3.951100] CPU: 6 PID: 140 Comm: (udev-worker) Not tainted 6.4.0-rc3 #1
> [    3.956161] (udev-worker)[142]: Oops 11003706212352 [1]
> [    3.951774] Hardware name: hp server rx2620                   , BIOS
> 04.29
> 11/30/2007
> [    3.951774]
> [    3.951774] Call Trace:
> [    3.958339] Unable to handle kernel paging request at virtual address
> 1000000000000000
> [    3.956161] Modules linked in:
> [    3.951774]  [<a0000001000156d0>] show_stack.part.0+0x30/0x60
> [    3.951774]                                 sp=e000000183a67b20
> bsp=e000000183a61628
> [    3.956161]
> [    3.956161]
> ```
>
> [1]: https://lists.debian.org/debian-ia64/2023/05/msg00010.html
>
> [2]: https://pastebin.com/SAUKbG7Z
>
> [3]: https://pastebin.com/v1TTB2x3

It seems the error happened during the WARN_ON_ONCE. Could you
please try whether something like the following fixes it?

diff --git i/kernel/module/main.c w/kernel/module/main.c
index 0f9183f1ca9f..ae42dfc1a815 100644
--- i/kernel/module/main.c
+++ w/kernel/module/main.c
@@ -1537,7 +1537,7 @@ static void __layout_sections(struct module
*mod, struct load_info *info, bool i
                            || is_init != module_init_layout_section(sname))
                                continue;

-                       if (WARN_ON_ONCE(type == MOD_INVALID))
+                       if (type == MOD_INVALID)
                                continue;

                        s->sh_entsize =
module_get_offset_and_type(mod, type, s, i);


If that doesn't work, maybe we need something like this:

diff --git i/arch/ia64/kernel/module.c w/arch/ia64/kernel/module.c
index 3661135da9d9..4e9a7f0498e2 100644
--- i/arch/ia64/kernel/module.c
+++ w/arch/ia64/kernel/module.c
@@ -815,7 +815,7 @@ apply_relocate_add (Elf64_Shdr *sechdrs, const
char *strtab, unsigned int symind
                uint64_t gp;
                struct module_memory *mod_mem;

-               mod_mem = &mod->mem[MOD_DATA];
+               mod_mem = &mod->mem[MOD_TEXT];
                if (mod_mem->size > MAX_LTOFF)
                        /*
                         * This takes advantage of fact that
SHF_ARCH_SMALL gets allocated


Song

Re: Boot regression in Linux v6.4-rc3

[Frank Scheiner]

Hi Song,

On 26.05.23 18:49, Song Liu wrote:
> Hi Frank,
>
> Thanks for the report.

Sure, thanks for your help in this.

> It seems the error happened during the WARN_ON_ONCE. Could you
> please try whether something like the following fixes it?
>
> diff --git i/kernel/module/main.c w/kernel/module/main.c
> index 0f9183f1ca9f..ae42dfc1a815 100644
> --- i/kernel/module/main.c
> +++ w/kernel/module/main.c
> @@ -1537,7 +1537,7 @@ static void __layout_sections(struct module
> *mod, struct load_info *info, bool i
>                              || is_init != module_init_layout_section(sname))
>                                  continue;
>
> -                       if (WARN_ON_ONCE(type == MOD_INVALID))
> +                       if (type == MOD_INVALID)
>                                  continue;
>
>                          s->sh_entsize =
> module_get_offset_and_type(mod, type, s, i);

Ok, tried that as -patch1 on top of v6.4-rc3, but didn't help, see [1].

[1]: https://pastebin.com/UK9v30Ae

> If that doesn't work, maybe we need something like this:
>
> diff --git i/arch/ia64/kernel/module.c w/arch/ia64/kernel/module.c
> index 3661135da9d9..4e9a7f0498e2 100644
> --- i/arch/ia64/kernel/module.c
> +++ w/arch/ia64/kernel/module.c
> @@ -815,7 +815,7 @@ apply_relocate_add (Elf64_Shdr *sechdrs, const
> char *strtab, unsigned int symind
>                  uint64_t gp;
>                  struct module_memory *mod_mem;
>
> -               mod_mem = &mod->mem[MOD_DATA];
> +               mod_mem = &mod->mem[MOD_TEXT];
>                  if (mod_mem->size > MAX_LTOFF)
>                          /*
>                           * This takes advantage of fact that
> SHF_ARCH_SMALL gets allocated

Tried that one as -patch2 on top of v6.4-rc3, but didn't help, see [2].

[2]: https://pastebin.com/gLupBndU

I also tried both patches as -patch1plus2 on top of v6.4-rc3 with a
similar result, see [3].

[3]: https://pastebin.com/7pXBG5vx

Cheers,
Frank

Re: Boot regression in Linux v6.4-rc3

[Song Liu]

On Fri, May 26, 2023 at 11:36 AM Frank Scheiner <frank.scheiner@web.de> wrote:
>
> Hi Song,
>
> On 26.05.23 18:49, Song Liu wrote:
> > Hi Frank,
> >
> > Thanks for the report.
>
> Sure, thanks for your help in this.
>
> > It seems the error happened during the WARN_ON_ONCE. Could you
> > please try whether something like the following fixes it?
> >
> > diff --git i/kernel/module/main.c w/kernel/module/main.c
> > index 0f9183f1ca9f..ae42dfc1a815 100644
> > --- i/kernel/module/main.c
> > +++ w/kernel/module/main.c
> > @@ -1537,7 +1537,7 @@ static void __layout_sections(struct module
> > *mod, struct load_info *info, bool i
> >                              || is_init != module_init_layout_section(sname))
> >                                  continue;
> >
> > -                       if (WARN_ON_ONCE(type == MOD_INVALID))
> > +                       if (type == MOD_INVALID)
> >                                  continue;
> >
> >                          s->sh_entsize =
> > module_get_offset_and_type(mod, type, s, i);
>
> Ok, tried that as -patch1 on top of v6.4-rc3, but didn't help, see [1].
>
> [1]: https://pastebin.com/UK9v30Ae
>
> > If that doesn't work, maybe we need something like this:
> >
> > diff --git i/arch/ia64/kernel/module.c w/arch/ia64/kernel/module.c
> > index 3661135da9d9..4e9a7f0498e2 100644
> > --- i/arch/ia64/kernel/module.c
> > +++ w/arch/ia64/kernel/module.c
> > @@ -815,7 +815,7 @@ apply_relocate_add (Elf64_Shdr *sechdrs, const
> > char *strtab, unsigned int symind
> >                  uint64_t gp;
> >                  struct module_memory *mod_mem;
> >
> > -               mod_mem = &mod->mem[MOD_DATA];
> > +               mod_mem = &mod->mem[MOD_TEXT];
> >                  if (mod_mem->size > MAX_LTOFF)
> >                          /*
> >                           * This takes advantage of fact that
> > SHF_ARCH_SMALL gets allocated
>
> Tried that one as -patch2 on top of v6.4-rc3, but didn't help, see [2].
>
> [2]: https://pastebin.com/gLupBndU
>
> I also tried both patches as -patch1plus2 on top of v6.4-rc3 with a
> similar result, see [3].
>
> [3]: https://pastebin.com/7pXBG5vx

Thanks for running the test.

I am not very familiar with the code, but I think we shouldn't hit that
WARN_ON_ONCE. Could you please try with the follow patch to see
which section caused this issue?

Thanks,
Song

diff --git i/kernel/module/main.c w/kernel/module/main.c
index 0f9183f1ca9f..caf3d30cd133 100644
--- i/kernel/module/main.c
+++ w/kernel/module/main.c
@@ -1537,8 +1537,11 @@ static void __layout_sections(struct module
*mod, struct load_info *info, bool i
                            || is_init != module_init_layout_section(sname))
                                continue;

-                       if (WARN_ON_ONCE(type == MOD_INVALID))
+                       if (WARN_ON_ONCE(type == MOD_INVALID)) {
+                               pr_warn("%s: section %s (sh_flags
%llx) matched to MOD_INVALID\n", __func__,
+                                       sname, s->sh_flags);
                                continue;
+                       }

                        s->sh_entsize =
module_get_offset_and_type(mod, type, s, i);
                        pr_debug("\t%s\n", sname);

Re: Boot regression in Linux v6.4-rc3

[Luis Chamberlain]

On Fri, May 26, 2023 at 12:55:14PM +0200, Frank Scheiner wrote:
> Dear all,
> 
> there is a boot regression in effect in Linux v6.4-rc3 that affects at
> least:
> 
> * rx2620 (w/2 x Montecito and zx1)
> * rx2800-i2 (w/1 x Tukwila)

Jesus, ia64 is even dropped from qemu as of 2.11. We're now around qemu
7.11 to give some perspective. I'm just wondering how to reproduce
testing easily instead of this ping pong back and forth for this
architecture for some oddball architectures.

Through commit 96ec72a3425d1 ("ia64: Mark architecture as orphaned")
it was noted even the old maintainer no longer had access to working
machines and so it was orphan'd.

Not saying that debugging commit ac3b4328392344 ("module: replace
module_layout with module_memory") is going to be impossible, quite
the contrary I think it would be good to root cause it, if possible,
as perhaps it may also be similar to some other future oddball arch
bug later that may come up.

But certainly just trying to see what options we have to test this
architecture.

And what's the status of removal for ia64 anyway?

  Luis

Re: Boot regression in Linux v6.4-rc3

[Linus Torvalds]

On Fri, May 26, 2023 at 2:59 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
>
> Not saying that debugging commit ac3b4328392344 ("module: replace
> module_layout with module_memory") is going to be impossible, quite
> the contrary I think it would be good to root cause it, if possible,
> as perhaps it may also be similar to some other future oddball arch
> bug later that may come up.

I don't have any context - the mailing lists in question that
apparently this came in on aren't in lore.

That said, that commit looks odd for the ia64 part.

In particular, this part:

  -               if (mod->core_layout.size > MAX_LTOFF)
  +               struct module_memory *mod_mem;
  +
  +               mod_mem = &mod->mem[MOD_DATA];

in apply_relocate_add() (file: arch/ia64/kernel/module.c) seems suspect.

The previous place that used to look at "mod->core_layout.base"
converted that to "mod->mem[MOD_TEXT].base". As do other changes in
other architectures.

So that "MOD_DATA" looks *very* wrong.  Shouldn't core_layout. be
translated to use "MOD_TEXT" instead?

Nothing else in the ia64 parts strike me as odd, but that one looks wrong to me.

But this is my "monkey see, monkey do" pattern matching reaction, not
from any deeper understanding of the problem (I can't even see the
report) or really even the code.

                   Linus

Re: Boot regression in Linux v6.4-rc3

[Song Liu]

On Fri, May 26, 2023 at 3:22 PM Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> On Fri, May 26, 2023 at 2:59 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
> >
> > Not saying that debugging commit ac3b4328392344 ("module: replace
> > module_layout with module_memory") is going to be impossible, quite
> > the contrary I think it would be good to root cause it, if possible,
> > as perhaps it may also be similar to some other future oddball arch
> > bug later that may come up.
>
> I don't have any context - the mailing lists in question that
> apparently this came in on aren't in lore.
>
> That said, that commit looks odd for the ia64 part.
>
> In particular, this part:
>
>   -               if (mod->core_layout.size > MAX_LTOFF)
>   +               struct module_memory *mod_mem;
>   +
>   +               mod_mem = &mod->mem[MOD_DATA];
>
> in apply_relocate_add() (file: arch/ia64/kernel/module.c) seems suspect.
>
> The previous place that used to look at "mod->core_layout.base"
> converted that to "mod->mem[MOD_TEXT].base". As do other changes in
> other architectures.
>
> So that "MOD_DATA" looks *very* wrong.  Shouldn't core_layout. be
> translated to use "MOD_TEXT" instead?

MOD_DATA is likely wrong here. But as Frank tested, changing it to MOD_TEXT
didn't fix the issue. I suspect we missed some special cases when we updated
layout_sections().

Thanks,
Song

Re: Boot regression in Linux v6.4-rc3

[Frank Scheiner]

Hi,

On 26.05.23 23:01, Song Liu wrote:
> Thanks for running the test.

Thanks for staying with me.

> I am not very familiar with the code, but I think we shouldn't hit that
> WARN_ON_ONCE. Could you please try with the follow patch to see
> which section caused this issue?
>
> Thanks,
> Song
>
> diff --git i/kernel/module/main.c w/kernel/module/main.c
> index 0f9183f1ca9f..caf3d30cd133 100644
> --- i/kernel/module/main.c
> +++ w/kernel/module/main.c
> @@ -1537,8 +1537,11 @@ static void __layout_sections(struct module
> *mod, struct load_info *info, bool i
>                              || is_init != module_init_layout_section(sname))
>                                  continue;
>
> -                       if (WARN_ON_ONCE(type == MOD_INVALID))
> +                       if (WARN_ON_ONCE(type == MOD_INVALID)) {
> +                               pr_warn("%s: section %s (sh_flags
> %llx) matched to MOD_INVALID\n", __func__,
> +                                       sname, s->sh_flags);
>                                  continue;
> +                       }
>
>                          s->sh_entsize =
> module_get_offset_and_type(mod, type, s, i);
>                          pr_debug("\t%s\n", sname);

I put that as -patch3 on top of 6.4-rc3, the result is on [1].

[1]: https://pastebin.com/KqnWL2pM

I this time put the whole console messages there, just in case some of
the earlier messages could be of any help. To jump to the actual oopses,
search for "Loading, please wait...".

Cheers,
Frank

Re: Boot regression in Linux v6.4-rc3

[Frank Scheiner]

On 27.05.23 00:22, Linus Torvalds wrote:
> [...]
> But this is my "monkey see, monkey do" pattern matching reaction, not
> from any deeper understanding of the problem (I can't even see the
> report) or really even the code.

If it is of any help, my initial report is available for example via:

https://marc.info/?l=linux-ia64&m\x168509859125505&w=2

...the whole thread is currently at:

https://marc.info/?t\x168509868200003&r=1&w=2

Cheers,
Frank

Re: Boot regression in Linux v6.4-rc3

[Linus Torvalds]

On Sat, May 27, 2023 at 12:01 AM Frank Scheiner <frank.scheiner@web.de> wrote:
>
> If it is of any help, my initial report is available for example via:
>
> https://marc.info/?l=linux-ia64&m=168509859125505&w=2
>
> ...the whole thread is currently at:
>
> https://marc.info/?t=168509868200003&r=1&w=2

This does make it clear just how great a mailing list archive lore is.
Konstantin, is there any particular reason why
linux-ia64@vger.kernel.org isn't in lore? Is it just a rational hatred
of all things itanium?

Anyway, the WARN_ON() is likely related, but the bug is clearly an
unexpected page fault in __copy_user() when called by load_module().

The ia64 oops output is nasty, presumably because ia64 aggressively
inlines things. It would help a lot if you enabled debug info (maybe
you already do?) and then run the oops through
./scripts/decode_stacktrace.sh which will figure out line numbers,
inlining etc.

Because I don't even see why it would call __copy_user() in the first
place. This is 'finit_module()' that loads the module data from a
file, not user space.

So I guess it must be the strndup_user() in

        mod->args = strndup_user(uargs, ~0UL >> 1);

but that doesn't look like it should even care about any module
layout. Plus I would have expected to see strndup_user() in the call
trace, but whatever.

End result: that ia64 trace is very hard to read, and _maybe_ running
it through the decode script might give more information about what it
is that triggers...

             Linus

Re: Boot regression in Linux v6.4-rc3

[Frank Scheiner]

Hi,

On 27.05.23 19:08, Linus Torvalds wrote:
> Anyway, the WARN_ON() is likely related, but the bug is clearly an
> unexpected page fault in __copy_user() when called by load_module().
>
> The ia64 oops output is nasty, presumably because ia64 aggressively
> inlines things. It would help a lot if you enabled debug info (maybe
> you already do?)

I believe it is enabled - I have at least CONFIG_DEBUG_KERNEL=y and
CONFIG_DEBUG_INFO=y - my kernel config is on [1] for reference.

[1]: https://pastebin.com/HRQtZ9vb

> and then run the oops through
> ./scripts/decode_stacktrace.sh which will figure out line numbers,
> inlining etc.
>
> Because I don't even see why it would call __copy_user() in the first
> place. This is 'finit_module()' that loads the module data from a
> file, not user space.
>
> So I guess it must be the strndup_user() in
>
>          mod->args = strndup_user(uargs, ~0UL >> 1);
>
> but that doesn't look like it should even care about any module
> layout. Plus I would have expected to see strndup_user() in the call
> trace, but whatever.
>
> End result: that ia64 trace is very hard to read, and _maybe_ running
> it through the decode script might give more information about what it
> is that triggers...

Ok, I put the decoded console messages on [2].

[2]: https://pastebin.com/dLYMijfS

Cheers,
Frank

Re: Boot regression in Linux v6.4-rc3

[Linus Torvalds]

On Sat, May 27, 2023 at 11:41 AM Frank Scheiner <frank.scheiner@web.de> wrote:
>
> Ok, I put the decoded console messages on [2].
>
> [2]: https://pastebin.com/dLYMijfS

Ugh. Apparently ia64 decoding isn't great. But at least it gives
multiple line numbers:

   load_module (kernel/module/main.c:2291 kernel/module/main.c:2412
kernel/module/main.c:2868)

except your kernel obviously has those test-patches, so I still don't
know exactly where they are.

But it looks like it is in move_module(). Strange. I don't know how it
gets to "__copy_user" from there...

[ Looks at the ia64 code ]

Oh.

It turns out that it *says* __copy_user(), but the code is actually
shared with the regular memcpy() function, which does

  GLOBAL_ENTRY(memcpy)
        and     r28=0x7,in0
        and     r29=0x7,in1
        mov     f6=f0
        mov     retval=in0
        br.cond.sptk .common_code
        ;;

where that ".common_code" label is - surprise surprise - the common
copy code, and so when the oops reports that the problem happened in
__copy_user(), it actually is in this case just a normal memcpy.

Ok, so it's probably the

        memcpy(dest, (void *)shdr->sh_addr, shdr->sh_size);

in move_module() that takes a fault.  And looking at the registers,
the destination is in r17/r18, and your dump has

    unable to handle kernel paging request at virtual address 1000000000000000
    ...
    r17 : 0fffffffffffffff r18 : 1000000000000000

so it's almost certainly that 'dest' that is bad.

Which I guess shouldn't surprise anybody.

But that's where my knowledge of ia64 and the new module loader layout ends.

                Linus

Re: Boot regression in Linux v6.4-rc3

[Frank Scheiner]

SGksDQoNCk9uIDI3LjA1LjIzIDIxOjM0LCBMaW51cyBUb3J2YWxkcyB3cm90ZToNCj4gT24g
U2F0LCBNYXkgMjcsIDIwMjMgYXQgMTE6NDHigK9BTSBGcmFuayBTY2hlaW5lciA8ZnJhbmsu
c2NoZWluZXJAd2ViLmRlPiB3cm90ZToNCj4+DQo+PiBPaywgSSBwdXQgdGhlIGRlY29kZWQg
Y29uc29sZSBtZXNzYWdlcyBvbiBbMl0uDQo+Pg0KPj4gWzJdOiBodHRwczovL3Bhc3RlYmlu
LmNvbS9kTFlNaWpmUw0KPiANCj4gVWdoLiBBcHBhcmVudGx5IGlhNjQgZGVjb2RpbmcgaXNu
J3QgZ3JlYXQuIEJ1dCBhdCBsZWFzdCBpdCBnaXZlcw0KPiBtdWx0aXBsZSBsaW5lIG51bWJl
cnM6DQo+IA0KPiAgICAgbG9hZF9tb2R1bGUgKGtlcm5lbC9tb2R1bGUvbWFpbi5jOjIyOTEg
a2VybmVsL21vZHVsZS9tYWluLmM6MjQxMg0KPiBrZXJuZWwvbW9kdWxlL21haW4uYzoyODY4
KQ0KPiANCj4gZXhjZXB0IHlvdXIga2VybmVsIG9idmlvdXNseSBoYXMgdGhvc2UgdGVzdC1w
YXRjaGVzLCBzbyBJIHN0aWxsIGRvbid0DQo+IGtub3cgZXhhY3RseSB3aGVyZSB0aGV5IGFy
ZS4NCg0KRXJtLCBJIHNlZS4gSSBkaWQgcmVjcmVhdGUgYSB2YW5pbGxhIHY2LjQtcmMzIGFu
ZCByYW4gdGhhdCwgZGVjb2RlZCANCnJlc3VsdCBpcyBvbiBbMV0gLSBub3Qgc3VyZSBpZiBp
dCBtYWtlcyBpdCBhIGxpdHRsZSBiZXR0ZXIuDQoNClsxXTogaHR0cHM6Ly9wYXN0ZWJpbi5j
b20vejVYekVuaHENCg0KSSBkaWQgYWxzbyB0cnkgdG8gYnVpbGQgYW5kIHJ1biBhIFNQIGtl
cm5lbCB0byBtYXliZSBnZXQgYSBiZXR0ZXIgDQpwaWN0dXJlIGluIHRoZSB0cmFjZXMsIGJ1
dCB0aGF0IHNlZW1zIHRvIHJlcXVpcmUgRkxBVE1FTSwgd2hpY2ggc2VlbXMgdG8gDQpub3Qg
d29yayBvbiB0aGF0IG1hY2hpbmUgb3IgZHVlIHRvIHRoZSB3YXkgaXQgaXMgY29uZmlndXJl
ZCAoYW5kIHllYWgsIA0KaXQgd2FzIGFsc28gdGhlIHdyb25nIGNvbW1pdCBJIHVzZWQgZm9y
IGl0IGFuZCBpdCB3YXMgcGF0Y2hlZC4uLik6DQoNCmBgYA0KWyAgICAwLjAwMDAwMF0gTGlu
dXggdmVyc2lvbiANCjYuNC4wLXJjMy05MzMxNzRhZTI4YmE3MmFiOGRlNWIzNWNiN2M5OGZj
MjExMjM1MDk2LXBhdGNoM19zcCANCihyb290QHg0MjcwKSAoaWE2NC1saW51eC1nY2MgKEdD
QykgMTIuMi4wLCBHTlUgbGQgKEdOVSBCaW51dGlscykgMi4zOSkgDQojMSBTYXQgTWF5IDI3
IDIxOjI4OjQ0IENFU1QgMjAyMw0KWy4uLl0NClsgICAgMC4wMDAwMDBdIEFDUEk6IFNTRFQg
MHgwMDAwMDAwMDNGRTM1QkE4IDAwMDEzQyAodjAxIEhQICAgICByeDI2MjAgDQowMDAwMDAw
NiBJTlRMIDIwMDUwMzA5KQ0KWyAgICAwLjAwMDAwMF0gQUNQSTogTG9jYWwgQVBJQyBhZGRy
ZXNzIChfX19fcHRydmFsX19fXykNClsgICAgMC4wMDAwMDBdIDEgQ1BVcyBhdmFpbGFibGUs
IDEgQ1BVcyB0b3RhbA0KWy4uLl0NClsgICAgMC4wMDAwMDBdIEtlcm5lbCBwYW5pYyAtIG5v
dCBzeW5jaW5nOiBDYW5ub3QgdXNlIEZMQVRNRU0gd2l0aCANCjI0Njc4NE1CIGhvbGUNClsg
ICAgMC4wMDAwMDBdIFBsZWFzZSBzd2l0Y2ggb3ZlciB0byBTUEFSU0VNRU0NClsgICAgMC4w
MDAwMDBdIC0tLVsgZW5kIEtlcm5lbCBwYW5pYyAtIG5vdCBzeW5jaW5nOiBDYW5ub3QgdXNl
IEZMQVRNRU0gDQp3aXRoIDI0Njc4NE1CIGhvbGUNClsgICAgMC4wMDAwMDBdIFBsZWFzZSBz
d2l0Y2ggb3ZlciB0byBTUEFSU0VNRU0gXS0tLQ0KYGBgDQoNCj4gQnV0IGl0IGxvb2tzIGxp
a2UgaXQgaXMgaW4gbW92ZV9tb2R1bGUoKS4gU3RyYW5nZS4gSSBkb24ndCBrbm93IGhvdyBp
dA0KPiBnZXRzIHRvICJfX2NvcHlfdXNlciIgZnJvbSB0aGVyZS4uLg0KPiANCj4gWyBMb29r
cyBhdCB0aGUgaWE2NCBjb2RlIF0NCj4gDQo+IE9oLg0KPiANCj4gSXQgdHVybnMgb3V0IHRo
YXQgaXQgKnNheXMqIF9fY29weV91c2VyKCksIGJ1dCB0aGUgY29kZSBpcyBhY3R1YWxseQ0K
PiBzaGFyZWQgd2l0aCB0aGUgcmVndWxhciBtZW1jcHkoKSBmdW5jdGlvbiwgd2hpY2ggZG9l
cw0KPiANCj4gICAgR0xPQkFMX0VOVFJZKG1lbWNweSkNCj4gICAgICAgICAgYW5kICAgICBy
Mjg9MHg3LGluMA0KPiAgICAgICAgICBhbmQgICAgIHIyOT0weDcsaW4xDQo+ICAgICAgICAg
IG1vdiAgICAgZjY9ZjANCj4gICAgICAgICAgbW92ICAgICByZXR2YWw9aW4wDQo+ICAgICAg
ICAgIGJyLmNvbmQuc3B0ayAuY29tbW9uX2NvZGUNCj4gICAgICAgICAgOzsNCj4gDQo+IHdo
ZXJlIHRoYXQgIi5jb21tb25fY29kZSIgbGFiZWwgaXMgLSBzdXJwcmlzZSBzdXJwcmlzZSAt
IHRoZSBjb21tb24NCj4gY29weSBjb2RlLCBhbmQgc28gd2hlbiB0aGUgb29wcyByZXBvcnRz
IHRoYXQgdGhlIHByb2JsZW0gaGFwcGVuZWQgaW4NCj4gX19jb3B5X3VzZXIoKSwgaXQgYWN0
dWFsbHkgaXMgaW4gdGhpcyBjYXNlIGp1c3QgYSBub3JtYWwgbWVtY3B5Lg0KPiANCj4gT2ss
IHNvIGl0J3MgcHJvYmFibHkgdGhlDQo+IA0KPiAgICAgICAgICBtZW1jcHkoZGVzdCwgKHZv
aWQgKilzaGRyLT5zaF9hZGRyLCBzaGRyLT5zaF9zaXplKTsNCj4gDQo+IGluIG1vdmVfbW9k
dWxlKCkgdGhhdCB0YWtlcyBhIGZhdWx0LiAgQW5kIGxvb2tpbmcgYXQgdGhlIHJlZ2lzdGVy
cywNCj4gdGhlIGRlc3RpbmF0aW9uIGlzIGluIHIxNy9yMTgsIGFuZCB5b3VyIGR1bXAgaGFz
DQo+IA0KPiAgICAgIHVuYWJsZSB0byBoYW5kbGUga2VybmVsIHBhZ2luZyByZXF1ZXN0IGF0
IHZpcnR1YWwgYWRkcmVzcyAxMDAwMDAwMDAwMDAwMDAwDQo+ICAgICAgLi4uDQo+ICAgICAg
cjE3IDogMGZmZmZmZmZmZmZmZmZmZiByMTggOiAxMDAwMDAwMDAwMDAwMDAwDQo+IA0KPiBz
byBpdCdzIGFsbW9zdCBjZXJ0YWlubHkgdGhhdCAnZGVzdCcgdGhhdCBpcyBiYWQuDQo+IA0K
PiBXaGljaCBJIGd1ZXNzIHNob3VsZG4ndCBzdXJwcmlzZSBhbnlib2R5Lg0KPiANCj4gQnV0
IHRoYXQncyB3aGVyZSBteSBrbm93bGVkZ2Ugb2YgaWE2NCBhbmQgdGhlIG5ldyBtb2R1bGUg
bG9hZGVyIGxheW91dCBlbmRzLg0KDQpUaGFua3MgZm9yIHlvdXIgaGVscCBhbmQgZ29pbmcg
YXMgZmFyIGFzIHlvdSBjb3VsZCwgdGhhdCdzIGdyZWF0bHkgDQphcHByZWNpYXRlZC4gUnVu
bmluZyB0aGF0IHN0dWZmIGlzIHN1cmVseSBlYXNpZXIgdGhhbiBkZWJ1Z2dpbmcgaXQuIDot
KQ0KDQpDaGVlcnMsDQpGcmFuaw0K

Re: Boot regression in Linux v6.4-rc3

[Song Liu]

On Sat, May 27, 2023 at 12:34 PM Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> On Sat, May 27, 2023 at 11:41 AM Frank Scheiner <frank.scheiner@web.de> wrote:
> >
> > Ok, I put the decoded console messages on [2].
> >
> > [2]: https://pastebin.com/dLYMijfS
>
> Ugh. Apparently ia64 decoding isn't great. But at least it gives
> multiple line numbers:
>
>    load_module (kernel/module/main.c:2291 kernel/module/main.c:2412
> kernel/module/main.c:2868)
>
> except your kernel obviously has those test-patches, so I still don't
> know exactly where they are.
>
> But it looks like it is in move_module(). Strange. I don't know how it
> gets to "__copy_user" from there...
>
> [ Looks at the ia64 code ]
>
> Oh.
>
> It turns out that it *says* __copy_user(), but the code is actually
> shared with the regular memcpy() function, which does
>
>   GLOBAL_ENTRY(memcpy)
>         and     r28=0x7,in0
>         and     r29=0x7,in1
>         mov     f6=f0
>         mov     retval=in0
>         br.cond.sptk .common_code
>         ;;
>
> where that ".common_code" label is - surprise surprise - the common
> copy code, and so when the oops reports that the problem happened in
> __copy_user(), it actually is in this case just a normal memcpy.
>
> Ok, so it's probably the
>
>         memcpy(dest, (void *)shdr->sh_addr, shdr->sh_size);
>
> in move_module() that takes a fault.  And looking at the registers,
> the destination is in r17/r18, and your dump has
>
>     unable to handle kernel paging request at virtual address 1000000000000000
>     ...
>     r17 : 0fffffffffffffff r18 : 1000000000000000
>
> so it's almost certainly that 'dest' that is bad.

Yeah, it appears we are writing to mod_mem[MOD_INVALID].

From the log, the following sections are assigned to MOD_INVALID:

[ 4.009109] __layout_sections: section .got (sh_flags 10000002)
matched to MOD_INVALID
[ 4.009109] __layout_sections: section .sdata (sh_flags 10000003)
matched to MOD_INVALID
[ 4.009109] __layout_sections: section .sbss (sh_flags 10000003)
matched to MOD_INVALID

AFAICT,  .got should go to rodata, while .sdata and .sbss should go
to (rw)data. However, reading the code before the module_memory
change, I think they were all copied to (rw)data, which is not ideal but
most likely OK.

To match the behavior before the module_memory change, I think
we need something like the following.

Frank, could you please give it a try?

Thanks,
Song

diff --git i/kernel/module/main.c w/kernel/module/main.c
index 0f9183f1ca9f..e4e723e1eb21 100644
--- i/kernel/module/main.c
+++ w/kernel/module/main.c
@@ -1514,14 +1514,14 @@ static void __layout_sections(struct module
*mod, struct load_info *info, bool i
                MOD_RODATA,
                MOD_RO_AFTER_INIT,
                MOD_DATA,
-               MOD_INVALID,    /* This is needed to match the masks array */
+               MOD_DATA,
        };
        static const int init_m_to_mem_type[] = {
                MOD_INIT_TEXT,
                MOD_INIT_RODATA,
                MOD_INVALID,
                MOD_INIT_DATA,
-               MOD_INVALID,    /* This is needed to match the masks array */
+               MOD_INIT_DATA,
        };

        for (m = 0; m < ARRAY_SIZE(masks); ++m) {

Re: Boot regression in Linux v6.4-rc3

[Frank Scheiner]

SGkgU29uZywgTGludXMsDQoNCk9uIDI4LjA1LjIzIDA3OjI0LCBTb25nIExpdSB3cm90ZToN
Cj4gQUZBSUNULCAgLmdvdCBzaG91bGQgZ28gdG8gcm9kYXRhLCB3aGlsZSAuc2RhdGEgYW5k
IC5zYnNzIHNob3VsZCBnbw0KPiB0byAocncpZGF0YS4gSG93ZXZlciwgcmVhZGluZyB0aGUg
Y29kZSBiZWZvcmUgdGhlIG1vZHVsZV9tZW1vcnkNCj4gY2hhbmdlLCBJIHRoaW5rIHRoZXkg
d2VyZSBhbGwgY29waWVkIHRvIChydylkYXRhLCB3aGljaCBpcyBub3QgaWRlYWwgYnV0DQo+
IG1vc3QgbGlrZWx5IE9LLg0KPiANCj4gVG8gbWF0Y2ggdGhlIGJlaGF2aW9yIGJlZm9yZSB0
aGUgbW9kdWxlX21lbW9yeSBjaGFuZ2UsIEkgdGhpbmsNCj4gd2UgbmVlZCBzb21ldGhpbmcg
bGlrZSB0aGUgZm9sbG93aW5nLg0KPiANCj4gRnJhbmssIGNvdWxkIHlvdSBwbGVhc2UgZ2l2
ZSBpdCBhIHRyeT8NCj4gDQo+IFRoYW5rcywNCj4gU29uZw0KPiANCj4gZGlmZiAtLWdpdCBp
L2tlcm5lbC9tb2R1bGUvbWFpbi5jIHcva2VybmVsL21vZHVsZS9tYWluLmMNCj4gaW5kZXgg
MGY5MTgzZjFjYTlmLi5lNGU3MjNlMWViMjEgMTAwNjQ0DQo+IC0tLSBpL2tlcm5lbC9tb2R1
bGUvbWFpbi5jDQo+ICsrKyB3L2tlcm5lbC9tb2R1bGUvbWFpbi5jDQo+IEBAIC0xNTE0LDE0
ICsxNTE0LDE0IEBAIHN0YXRpYyB2b2lkIF9fbGF5b3V0X3NlY3Rpb25zKHN0cnVjdCBtb2R1
bGUNCj4gKm1vZCwgc3RydWN0IGxvYWRfaW5mbyAqaW5mbywgYm9vbCBpDQo+ICAgICAgICAg
ICAgICAgICAgTU9EX1JPREFUQSwNCj4gICAgICAgICAgICAgICAgICBNT0RfUk9fQUZURVJf
SU5JVCwNCj4gICAgICAgICAgICAgICAgICBNT0RfREFUQSwNCj4gLSAgICAgICAgICAgICAg
IE1PRF9JTlZBTElELCAgICAvKiBUaGlzIGlzIG5lZWRlZCB0byBtYXRjaCB0aGUgbWFza3Mg
YXJyYXkgKi8NCj4gKyAgICAgICAgICAgICAgIE1PRF9EQVRBLA0KPiAgICAgICAgICB9Ow0K
PiAgICAgICAgICBzdGF0aWMgY29uc3QgaW50IGluaXRfbV90b19tZW1fdHlwZVtdID0gew0K
PiAgICAgICAgICAgICAgICAgIE1PRF9JTklUX1RFWFQsDQo+ICAgICAgICAgICAgICAgICAg
TU9EX0lOSVRfUk9EQVRBLA0KPiAgICAgICAgICAgICAgICAgIE1PRF9JTlZBTElELA0KPiAg
ICAgICAgICAgICAgICAgIE1PRF9JTklUX0RBVEEsDQo+IC0gICAgICAgICAgICAgICBNT0Rf
SU5WQUxJRCwgICAgLyogVGhpcyBpcyBuZWVkZWQgdG8gbWF0Y2ggdGhlIG1hc2tzIGFycmF5
ICovDQo+ICsgICAgICAgICAgICAgICBNT0RfSU5JVF9EQVRBLA0KPiAgICAgICAgICB9Ow0K
PiANCj4gICAgICAgICAgZm9yIChtID0gMDsgbSA8IEFSUkFZX1NJWkUobWFza3MpOyArK20p
IHsNCg0KVGhhbmtzLCB0aGF0IHBhdGNoIChhcyAtcGF0Y2g0IG9uIHRvcCBvZiB2Ni40LXJj
MykgZml4ZXMgdGhlIGJvb3QgDQpyZWdyZXNzaW9uIGZvciBtZSBvbiB0aGUgcngyNjIwOg0K
DQpgYGANCkVMSUxPIHYzLjE2IGZvciBFRkkvSUEtNjQNCi4uDQpVbmNvbXByZXNzaW5nIExp
bnV4Li4uIGRvbmUNCkxvYWRpbmcgZmlsZSBBQzEwMDIyMS5pbml0cmQuaW1nLi4uZG9uZQ0K
WyAgICAwLjAwMDAwMF0gTGludXggdmVyc2lvbiANCjYuNC4wLXJjMy00NGMwMjZhNzNiZTgw
MzhmMDNkYmRlZWYwMjhiNjQyODgwY2YxNTExLXBhdGNoNCAocm9vdEB4NDI3MCkgDQooaWE2
NC1saW51eC1nY2MgKEdDQykgMTIuMi4wLCBHTlUgbGQgKEdOVSBCaW51dGlscykgMi4zOSkg
IzEgU01QIFN1biBNYXkgDQoyOCAwOTowODo0NCBDRVNUIDIwMjMNClsgICAgMC4wMDAwMDBd
IGVmaTogRUZJIHYxLjEgYnkgSFANClsgICAgMC4wMDAwMDBdIGVmaTogU0FMc3lzdGFiPTB4
M2VlN2EwMDAgQUNQSSAyLjA9MHgzZmUyYTAwMCANCkVTST0weDNlZTdiMDAwIFNNQklPUz0w
eDNlZTdjMDAwIEhDRFA9MHgzZmUyODAwMA0KWyAgICAwLjAwMDAwMF0gUENEUDogdjMgYXQg
MHgzZmUyODAwMA0KWyAgICAwLjAwMDAwMF0gZWFybHljb246IHVhcnQ4MjUwIGF0IE1NSU8g
MHgwMDAwMDAwMGY0MDUwMDAwIChvcHRpb25zIA0KJzk2MDBuOCcpDQpbICAgIDAuMDAwMDAw
XSBwcmludGs6IGJvb3Rjb25zb2xlIFt1YXJ0ODI1MF0gZW5hYmxlZA0KWyAgICAwLjAwMDAw
MF0gQUNQSTogRWFybHkgdGFibGUgY2hlY2tzdW0gdmVyaWZpY2F0aW9uIGRpc2FibGVkDQpb
ICAgIDAuMDAwMDAwXSBBQ1BJOiBSU0RQIDB4MDAwMDAwMDAzRkUyQTAwMCAwMDAwMjggKHYw
MiBIUCAgICApDQpbICAgIDAuMDAwMDAwXSBBQ1BJOiBYU0RUIDB4MDAwMDAwMDAzRkUyQTAy
QyAwMDAwQ0MgKHYwMSBIUCAgICAgcngyNjIwIA0KMDAwMDAwMDAgSFAgICAwMDAwMDAwMCkN
ClsuLi5dDQpbICAgIDMuODEwMzQ2XSBSdW4gL2luaXQgYXMgaW5pdCBwcm9jZXNzDQpMb2Fk
aW5nLCBwbGVhc2Ugd2FpdC4uLg0KU3RhcnRpbmcgc3lzdGVtZC11ZGV2ZCB2ZXJzaW9uIDI1
Mi42LTENClsgICAgMy45ODUzNzhdIGUxMDAwOiBJbnRlbChSKSBQUk8vMTAwMCBOZXR3b3Jr
IERyaXZlcg0KWyAgICAzLjk4OTM3OF0gZTEwMDA6IENvcHlyaWdodCAoYykgMTk5OS0yMDA2
IEludGVsIENvcnBvcmF0aW9uLg0KWyAgICAzLjk5MzM3NV0gR1NJIDI5IChsZXZlbCwgbG93
KSAtPiBDUFUgNyAoMHgwNzAwKSB2ZWN0b3IgNTMNClsgICAgNC4wMzAzODJdIEFDUEk6IGJ1
cyB0eXBlIFVTQiByZWdpc3RlcmVkDQpbICAgIDQuMDMwMzgyXSB1c2Jjb3JlOiByZWdpc3Rl
cmVkIG5ldyBpbnRlcmZhY2UgZHJpdmVyIHVzYmZzDQpbICAgIDQuMDM0MzgyXSB1c2Jjb3Jl
OiByZWdpc3RlcmVkIG5ldyBpbnRlcmZhY2UgZHJpdmVyIGh1Yg0KWyAgICA0LjAzNDM4Ml0g
dXNiY29yZTogcmVnaXN0ZXJlZCBuZXcgZGV2aWNlIGRyaXZlciB1c2INClsgICAgNC4wNDA2
MDldIEdTSSAxOCAobGV2ZWwsIGxvdykgLT4gQ1BVIDAgKDB4MDAwMCkgdmVjdG9yIDU0DQpb
ICAgIDQuMDQwNjIxXSBlaGNpLXBjaSAwMDAwOjAwOjAxLjI6IEVIQ0kgSG9zdCBDb250cm9s
bGVyDQpbLi4uXQ0KWyAgT0sgIF0gRmluaXNoZWQgc3lzdGVtZC11cGRhdGUtdXTigKYgLSBS
ZWNvcmQgUnVubGV2ZWwgQ2hhbmdlIGluIFVUTVAuDQpbICAgMTQuNTY4NjA2XSBpb2MxOiBM
U0k1M0MxMDMwIEMwOiBDYXBhYmlsaXRpZXM9e0luaXRpYXRvcixUYXJnZXR9DQoNCkRlYmlh
biBHTlUvTGludXggMTIgcngyNjIwIC0NCg0KcngyNjIwIGxvZ2luOg0KYGBgDQoNCkdyZWF0
ISBJJ2xsIGdpdmUgaXQgYSB0cnkgb24gbXkgcngyODAwLWkyLCB0b28sIGJ1dCBhc3N1bWUg
aXQgd2lsIHdvcmsgDQp0aGVyZSwgdG9vLg0KDQpDaGVlcnMsDQpGcmFuaw0K

Re: Boot regression in Linux v6.4-rc3

[Frank Scheiner]

SGkgYWdhaW4sDQoNCk9uIDI4LjA1LjIzIDA5OjMwLCBGcmFuayBTY2hlaW5lciB3cm90ZToN
Cj4gWy4uLl0NCj4gVGhhbmtzLCB0aGF0IHBhdGNoIChhcyAtcGF0Y2g0IG9uIHRvcCBvZiB2
Ni40LXJjMykgZml4ZXMgdGhlIGJvb3QgDQo+IHJlZ3Jlc3Npb24gZm9yIG1lIG9uIHRoZSBy
eDI2MjA6DQo+IA0KPiBbLi4uXQ0KPiANCj4gR3JlYXQhIEknbGwgZ2l2ZSBpdCBhIHRyeSBv
biBteSByeDI4MDAtaTIsIHRvbywgYnV0IGFzc3VtZSBpdCB3aWwgd29yayANCj4gdGhlcmUs
IHRvby4NCg0KSW5kZWVkLCAtcGF0Y2g0IGFsc28gbWFrZXMgaXQgd29yayBvbiB0aGUgcngy
ODAwLWkyOg0KDQpgYGANCkVMSUxPIHYzLjE2IGZvciBFRkkvSUEtNjQNCi4uDQpVbmNvbXBy
ZXNzaW5nIExpbnV4Li4uIGRvbmUNCkxvYWRpbmcgZmlsZSBBQzEwMDI3Qi5pbml0cmQuaW1n
Li4uZG9uZQ0KWyAgICAwLjAwMDAwMF0gTGludXggdmVyc2lvbiANCjYuNC4wLXJjMy00NGMw
MjZhNzNiZTgwMzhmMDNkYmRlZWYwMjhiNjQyODgwY2YxNTExLXBhdGNoNCAocm9vdEB4NDI3
MCkgDQooaWE2NC1saW51eC1nY2MgKEdDQykgMTIuMi4wLCBHTlUgbGQgKEdOVSBCaW51dGls
cykgMi4zOSkgIzEgU01QIFN1biBNYXkgDQoyOCAwOTowODo0NCBDRVNUIDIwMjMNClsgICAg
MC4wMDAwMDBdIGVmaTogRUZJIHYyLjEgYnkgSFANClsgICAgMC4wMDAwMDBdIGVmaTogU0FM
c3lzdGFiPTB4ZGZkZDYzYTE4IEVTST0weGRmZGQ2M2YxOCBBQ1BJIA0KMi4wPTB4M2QzYzQw
MTQgSENEUD0weGRmZmZmODc5OCBTTUJJT1M9MHgzZDM2ODAwMA0KWyAgICAwLjAwMDAwMF0g
UENEUDogdjMgYXQgMHhkZmZmZjg3OTgNClsgICAgMC4wMDAwMDBdIGVhcmx5Y29uOiB1YXJ0
ODI1MCBhdCBJL08gcG9ydCAweDQwMDAgKG9wdGlvbnMgJzExNTIwMG44JykNClsgICAgMC4w
MDAwMDBdIHByaW50azogYm9vdGNvbnNvbGUgW3VhcnQ4MjUwXSBlbmFibGVkDQpbICAgIDAu
MDAwMDAwXSBBQ1BJOiBFYXJseSB0YWJsZSBjaGVja3N1bSB2ZXJpZmljYXRpb24gZGlzYWJs
ZWQNClsgICAgMC4wMDAwMDBdIEFDUEk6IFJTRFAgMHgwMDAwMDAwMDNEM0M0MDE0IDAwMDAy
NCAodjAyIEhQICAgICkNClsgICAgMC4wMDAwMDBdIEFDUEk6IFhTRFQgMHgwMDAwMDAwMDNE
M0M0NTgwIDAwMDEyNCAodjAxIEhQICAgICBSWDI4MDAtMiANCjAwMDAwMDAxICAgICAgMDEw
MDAwMTMpDQpbLi4uXQ0KWyAgIDM2LjY0OTUzMV0gUnVuIC9pbml0IGFzIGluaXQgcHJvY2Vz
cw0KTG9hZGluZywgcGxlYXNlIHdhaXQuLi4NClN0YXJ0aW5nIHN5c3RlbWQtdWRldmQgdmVy
c2lvbiAyNTIuNi0xDQpbICAgMzYuODY1NjM1XSBwcHNfY29yZTogTGludXhQUFMgQVBJIHZl
ci4gMSByZWdpc3RlcmVkDQpbICAgMzYuODY5MzIxXSBwcHNfY29yZTogU29mdHdhcmUgdmVy
LiA1LjMuNiAtIENvcHlyaWdodCAyMDA1LTIwMDcgDQpSb2RvbGZvIEdpb21ldHRpIDxnaW9t
ZXR0aUBsaW51eC5pdD4NClsgICAzNi44ODUwMjVdIFBUUCBjbG9jayBzdXBwb3J0IHJlZ2lz
dGVyZWQNClsgICAzNi45MTA4NTJdIEFDUEk6IGJ1cyB0eXBlIFVTQiByZWdpc3RlcmVkDQpb
ICAgMzYuOTE4MTk4XSB1c2Jjb3JlOiByZWdpc3RlcmVkIG5ldyBpbnRlcmZhY2UgZHJpdmVy
IHVzYmZzDQpbICAgMzYuOTI0NzYyXSB1c2Jjb3JlOiByZWdpc3RlcmVkIG5ldyBpbnRlcmZh
Y2UgZHJpdmVyIGh1Yg0KWyAgIDM2LjkyMjEzM10gaWdiOiBJbnRlbChSKSBHaWdhYml0IEV0
aGVybmV0IE5ldHdvcmsgRHJpdmVyDQpbICAgMzYuOTMxMzg2XSB1c2Jjb3JlOiByZWdpc3Rl
cmVkIG5ldyBkZXZpY2UgZHJpdmVyIHVzYg0KWyAgIDM2LjkzODE0OV0gaWdiOiBDb3B5cmln
aHQgKGMpIDIwMDctMjAxNCBJbnRlbCBDb3Jwb3JhdGlvbi4NClsuLi5dDQpbICBPSyAgXSBG
aW5pc2hlZCBhcHQtZGFpbHktdXBncmFkZeKApiBhcHQgdXBncmFkZSBhbmQgY2xlYW4gYWN0
aXZpdGllcy4NCg0KRGViaWFuIEdOVS9MaW51eCAxMiByeDI4MDAtaTIgLQ0KDQpyeDI4MDAt
aTIgbG9naW46DQpgYGANCg0KSSdsbCB0cnkgdG8gdGVzdCB0aGlzIG9uIG90aGVyIG1hY2hp
bmVzIChyeDQ2NDAgdy9NYWRpc29uLCByeDI2NjAgDQp3L01vbnRlY2l0by9Nb250dmFsZSwg
cng2NjAwIHcvTW9udHZhbGUpIGFzIHdlbGwsIHN0YXJ0aW5nIG9uIFR1ZXNkYXkgaWYgDQpw
b3NzaWJsZS4NCg0KKioqKg0KDQpUaGVyZSBpcyBhbiBpc3N1ZSAtIG9ubHkgZm9yIHRoZSBy
eDI4MDAtaTIgLSBzZWVtaW5nbHkgcmVsYXRlZCB0byBpdCdzIA0KUENJZSBOSUMocykgYW5k
IE1TSXMsIGJ1dCB0aGlzIGlzIGFscmVhZHkgdGhlcmUgaW4gNi4zLnggKHNlZSBmaXJzdCBw
YXJ0IA0Kb2YgWzFdKSBhbmQgKipub3QgcmVsYXRlZCB0byB0aGUgcHJvYmxlbSBvZiB0aGlz
IHRocmVhZCAoQUZBSUNUKSoqIGFuZCAtIA0KbW9yZSBpbXBvcnRhbnQgLSBkb2Vzbid0IGFm
ZmVjdCBvcGVyYXRpb246IFRoZSBtYWNoaW5lIGlzIHdvcmtpbmcgDQpkaXNrbGVzcyB1c2lu
ZyBvbmUgb2YgdGhvc2UgaW50ZXJmYWNlcyBzbyBpdCBjYW4ndCBiZSB0aGF0IGJhZC4gSSds
bCANCmxvb2sgaW50byBiaXNlY3RpbmcgdGhhdCBpc3N1ZSBhcyB3ZWxsLg0KDQpbMV06IGh0
dHBzOi8vbGlzdHMuZGViaWFuLm9yZy9kZWJpYW4taWE2NC8yMDIzLzA1L21zZzAwMDEwLmh0
bWwNCg0KQ2hlZXJzLA0KRnJhbmsNCg=

Re: Boot regression in Linux v6.4-rc3

[John Paul Adrian Glaubitz]

Hi!

On Sun, 2023-05-28 at 09:30 +0200, Frank Scheiner wrote:
> > Frank, could you please give it a try?
> > 
> > Thanks,
> > Song
> > 
> > diff --git i/kernel/module/main.c w/kernel/module/main.c
> > index 0f9183f1ca9f..e4e723e1eb21 100644
> > --- i/kernel/module/main.c
> > +++ w/kernel/module/main.c
> > @@ -1514,14 +1514,14 @@ static void __layout_sections(struct module
> > *mod, struct load_info *info, bool i
> >                  MOD_RODATA,
> >                  MOD_RO_AFTER_INIT,
> >                  MOD_DATA,
> > -               MOD_INVALID,    /* This is needed to match the masks array */
> > +               MOD_DATA,
> >          };
> >          static const int init_m_to_mem_type[] = {
> >                  MOD_INIT_TEXT,
> >                  MOD_INIT_RODATA,
> >                  MOD_INVALID,
> >                  MOD_INIT_DATA,
> > -               MOD_INVALID,    /* This is needed to match the masks array */
> > +               MOD_INIT_DATA,
> >          };
> > 
> >          for (m = 0; m < ARRAY_SIZE(masks); ++m) {
> 
> Thanks, that patch (as -patch4 on top of v6.4-rc3) fixes the boot 
> regression for me on the rx2620:

Sounds good. Would be great to get this into 6.4 before release.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer
`. `'   Physicist
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: Boot regression in Linux v6.4-rc3

[Song Liu]

On Sun, May 28, 2023 at 1:10 AM Frank Scheiner <frank.scheiner@web.de> wrote:
>
> Hi again,
>
> On 28.05.23 09:30, Frank Scheiner wrote:
> > [...]
> > Thanks, that patch (as -patch4 on top of v6.4-rc3) fixes the boot
> > regression for me on the rx2620:
> >
> > [...]
> >
> > Great! I'll give it a try on my rx2800-i2, too, but assume it wil work
> > there, too.
>
> Indeed, -patch4 also makes it work on the rx2800-i2:

Thanks for running the test!

I will send the official patch.

Thanks,
Song

>
> ```
> ELILO v3.16 for EFI/IA-64
> ..
> Uncompressing Linux... done
> Loading file AC10027B.initrd.img...done
> [    0.000000] Linux version
> 6.4.0-rc3-44c026a73be8038f03dbdeef028b642880cf1511-patch4 (root@x4270)
> (ia64-linux-gcc (GCC) 12.2.0, GNU ld (GNU Binutils) 2.39) #1 SMP Sun May
> 28 09:08:44 CEST 2023
> [    0.000000] efi: EFI v2.1 by HP
> [    0.000000] efi: SALsystab=0xdfdd63a18 ESI=0xdfdd63f18 ACPI
> 2.0=0x3d3c4014 HCDP=0xdffff8798 SMBIOS=0x3d368000
> [    0.000000] PCDP: v3 at 0xdffff8798
> [    0.000000] earlycon: uart8250 at I/O port 0x4000 (options '115200n8')
> [    0.000000] printk: bootconsole [uart8250] enabled
> [    0.000000] ACPI: Early table checksum verification disabled
> [    0.000000] ACPI: RSDP 0x000000003D3C4014 000024 (v02 HP    )
> [    0.000000] ACPI: XSDT 0x000000003D3C4580 000124 (v01 HP     RX2800-2
> 00000001      01000013)
> [...]
> [   36.649531] Run /init as init process
> Loading, please wait...
> Starting systemd-udevd version 252.6-1
> [   36.865635] pps_core: LinuxPPS API ver. 1 registered
> [   36.869321] pps_core: Software ver. 5.3.6 - Copyright 2005-2007
> Rodolfo Giometti <giometti@linux.it>
> [   36.885025] PTP clock support registered
> [   36.910852] ACPI: bus type USB registered
> [   36.918198] usbcore: registered new interface driver usbfs
> [   36.924762] usbcore: registered new interface driver hub
> [   36.922133] igb: Intel(R) Gigabit Ethernet Network Driver
> [   36.931386] usbcore: registered new device driver usb
> [   36.938149] igb: Copyright (c) 2007-2014 Intel Corporation.
> [...]
> [  OK  ] Finished apt-daily-upgrade… apt upgrade and clean activities.
>
> Debian GNU/Linux 12 rx2800-i2 -
>
> rx2800-i2 login:
> ```
>
> I'll try to test this on other machines (rx4640 w/Madison, rx2660
> w/Montecito/Montvale, rx6600 w/Montvale) as well, starting on Tuesday if
> possible.
>
> ****
>
> There is an issue - only for the rx2800-i2 - seemingly related to it's
> PCIe NIC(s) and MSIs, but this is already there in 6.3.x (see first part
> of [1]) and **not related to the problem of this thread (AFAICT)** and -
> more important - doesn't affect operation: The machine is working
> diskless using one of those interfaces so it can't be that bad. I'll
> look into bisecting that issue as well.
>
> [1]: https://lists.debian.org/debian-ia64/2023/05/msg00010.html
>
> Cheers,
> Frank

Re: Boot regression in Linux v6.4-rc3

[Konstantin Ryabitsev]

On Sat, May 27, 2023 at 10:08:24AM -0700, Linus Torvalds wrote:
> This does make it clear just how great a mailing list archive lore is.
> Konstantin, is there any particular reason why
> linux-ia64@vger.kernel.org isn't in lore? Is it just a rational hatred
> of all things itanium?

We only add things to lore when someone asks, and nobody's asked. :) I guess
I'll consider this an ask and put it on the radar.

Regards,
-K

Re: Boot regression in Linux v6.4-rc3

[Linus Torvalds]

On Tue, May 30, 2023 at 4:21 PM Konstantin Ryabitsev
<konstantin@linuxfoundation.org> wrote:
>
> We only add things to lore when someone asks, and nobody's asked. :) I guess
> I'll consider this an ask and put it on the radar.

Thanks. It would probably be good to see if there are any other
vger.kernel.org lists with any appreciable traffic that aren't on
lore.

               Linus

Re: Boot regression in Linux v6.4-rc3

[Linus Torvalds]

On Tue, May 30, 2023 at 4:21 PM Konstantin Ryabitsev
<konstantin@linuxfoundation.org> wrote:
>
> We only add things to lore when someone asks, and nobody's asked. :) I guess
> I'll consider this an ask and put it on the radar.

Thanks. It would probably be good to see if there are any other
vger.kernel.org lists with any appreciable traffic that aren't on
lore.

               Linus

Re: Boot regression in Linux v6.4-rc3

[Konstantin Ryabitsev]

On Tue, May 30, 2023 at 05:04:56PM -0400, Linus Torvalds wrote:
> On Tue, May 30, 2023 at 4:21 PM Konstantin Ryabitsev
> <konstantin@linuxfoundation.org> wrote:
> >
> > We only add things to lore when someone asks, and nobody's asked. :) I guess
> > I'll consider this an ask and put it on the radar.
> 
> Thanks. It would probably be good to see if there are any other
> vger.kernel.org lists with any appreciable traffic that aren't on
> lore.

Yes, that will auto-fix itself as we continue to migrate things over to
subspace ("new vger").

-K

Re: Boot regression in Linux v6.4-rc3

[Konstantin Ryabitsev]

On Tue, May 30, 2023 at 05:04:56PM -0400, Linus Torvalds wrote:
> On Tue, May 30, 2023 at 4:21 PM Konstantin Ryabitsev
> <konstantin@linuxfoundation.org> wrote:
> >
> > We only add things to lore when someone asks, and nobody's asked. :) I guess
> > I'll consider this an ask and put it on the radar.
> 
> Thanks. It would probably be good to see if there are any other
> vger.kernel.org lists with any appreciable traffic that aren't on
> lore.

Yes, that will auto-fix itself as we continue to migrate things over to
subspace ("new vger").

-K

Re: Boot regression in Linux v6.4-rc3

[Frank Scheiner]

Hi Linus, hi Song,

On 29.05.23 00:46, Song Liu wrote:
> [...]
> Thanks for running the test!
>
> I will send the official patch.
>
> Thanks,
> Song

With the fix merged and to conclude this, I'd like to add that it was a
pleasure to work with you on this problem, although I didn't do much.

Looking forward to the next occasion - for your sake maybe on another
architecture, but can't promise... ;-)

Cheers,
Frank