[REGRESSION] XArray commit prevents booting with 6.0-rc1 or later

[REGRESSION] XArray commit prevents booting with 6.0-rc1 or later

[Jorropo]

#regzbot introduced v5.19-rc6..1dd685c414a7b9fdb3d23aca3aedae84f0b998ae

Hi, I recently tried to upgrade to linux v6.0.x but when trying to
boot it fails with "error: out of memory" when or after loading
initramfs (which then kpanics because the vfs root is missing).
The latest releases I tested are v6.0.9 and v6.1-rc5 and it's broken there too.

I bisected the error to this patch:
1dd685c414a7b9fdb3d23aca3aedae84f0b998ae "XArray: Add calls to
might_alloc()" is the first bad commit.
I've confirmed this is not a side effect of a poor bitsect because
1dd685c414a7b9fdb3d23aca3aedae84f0b998ae~1 (v5.19-rc6) works.
I've tried reverting the failing commit on top of v6.0.9 and it didn't fixed it.

My system is:
CPU: Ryzen 3600
Motherboard: B550 AORUS ELITE V2
Ram: 48GB (16+32) of unmatched DDR4
GPU: AMD rx580
Various ssds, hdds and network cards plugged with various buses.

You can find a folder with my .config, bisect logs and screenshots of
the error messages there:
https://jorropo.net/ipfs/QmaWH84UPEen4E9n69KZiLjPDaTi2aJvizv3JYiL7Gfmnr/
https://ipfs.io/ipfs/QmaWH84UPEen4E9n69KZiLjPDaTi2aJvizv3JYiL7Gfmnr/

I'll be happy to assist you if you need help reproducing this issue
and or testing fixes.

Thx, Jorropo

Re: [REGRESSION] XArray commit prevents booting with 6.0-rc1 or later

[Matthew Wilcox]

On Sat, Nov 19, 2022 at 05:07:45AM +0100, Jorropo wrote:
> #regzbot introduced v5.19-rc6..1dd685c414a7b9fdb3d23aca3aedae84f0b998ae
> 
> Hi, I recently tried to upgrade to linux v6.0.x but when trying to
> boot it fails with "error: out of memory" when or after loading
> initramfs (which then kpanics because the vfs root is missing).
> The latest releases I tested are v6.0.9 and v6.1-rc5 and it's broken there too.
> 
> I bisected the error to this patch:
> 1dd685c414a7b9fdb3d23aca3aedae84f0b998ae "XArray: Add calls to
> might_alloc()" is the first bad commit.
> I've confirmed this is not a side effect of a poor bitsect because
> 1dd685c414a7b9fdb3d23aca3aedae84f0b998ae~1 (v5.19-rc6) works.

That makes no sense.  I can't look into this until Wednesday, but I
suggest that what you have is an intermittent failure to boot, and
the bisect has led you down the wrong path.

> I've tried reverting the failing commit on top of v6.0.9 and it didn't fixed it.
> 
> My system is:
> CPU: Ryzen 3600
> Motherboard: B550 AORUS ELITE V2
> Ram: 48GB (16+32) of unmatched DDR4
> GPU: AMD rx580
> Various ssds, hdds and network cards plugged with various buses.
> 
> You can find a folder with my .config, bisect logs and screenshots of
> the error messages there:
> https://jorropo.net/ipfs/QmaWH84UPEen4E9n69KZiLjPDaTi2aJvizv3JYiL7Gfmnr/
> https://ipfs.io/ipfs/QmaWH84UPEen4E9n69KZiLjPDaTi2aJvizv3JYiL7Gfmnr/
> 
> I'll be happy to assist you if you need help reproducing this issue
> and or testing fixes.
> 
> Thx, Jorropo

Re: [REGRESSION] XArray commit prevents booting with 6.0-rc1 or later

[Jorropo]

Matthew Wilcox <willy@infradead.org> wrote :
>
> On Sat, Nov 19, 2022 at 05:07:45AM +0100, Jorropo wrote:
> > #regzbot introduced v5.19-rc6..1dd685c414a7b9fdb3d23aca3aedae84f0b998ae
> >
> > Hi, I recently tried to upgrade to linux v6.0.x but when trying to
> > boot it fails with "error: out of memory" when or after loading
> > initramfs (which then kpanics because the vfs root is missing).
> > The latest releases I tested are v6.0.9 and v6.1-rc5 and it's broken there too.
> >
> > I bisected the error to this patch:
> > 1dd685c414a7b9fdb3d23aca3aedae84f0b998ae "XArray: Add calls to
> > might_alloc()" is the first bad commit.
> > I've confirmed this is not a side effect of a poor bitsect because
> > 1dd685c414a7b9fdb3d23aca3aedae84f0b998ae~1 (v5.19-rc6) works.
>
> That makes no sense.  I can't look into this until Wednesday, but I
> suggest that what you have is an intermittent failure to boot, and
> the bisect has led you down the wrong path.

I rebuilt both 1dd685c414a7b9fdb3d23aca3aedae84f0b998ae and
the parent commit (v5.19-rc6), then tried to start each one 8 times
(shuffled in a Thue morse sequence).
0 successes for 1dd685c414a7b9fdb3d23aca3aedae84f0b998ae
8 successes for v5.19-rc6

This really does not look like an intermittent issue.

Re: [REGRESSION] XArray commit prevents booting with 6.0-rc1 or later #forregzbot

[Thorsten Leemhuis]

[Note: this mail is primarily send for documentation purposes and/or for
regzbot, my Linux kernel regression tracking bot. That's why I removed
most or all folks from the list of recipients, but left any that looked
like a mailing lists. These mails usually contain '#forregzbot' in the
subject, to make them easy to spot and filter out.]

On 19.11.22 05:07, Jorropo wrote:
> #regzbot introduced v5.19-rc6..1dd685c414a7b9fdb3d23aca3aedae84f0b998ae

#regzbot introduced 1dd685c414a7b9fdb3d23aca3aedae84f

Re: [REGRESSION] XArray commit prevents booting with 6.0-rc1 or later

[Matthew Wilcox]

On Sun, Nov 20, 2022 at 12:20:13AM +0100, Jorropo wrote:
> Matthew Wilcox <willy@infradead.org> wrote :
> >
> > On Sat, Nov 19, 2022 at 05:07:45AM +0100, Jorropo wrote:
> > > #regzbot introduced v5.19-rc6..1dd685c414a7b9fdb3d23aca3aedae84f0b998ae
> > >
> > > Hi, I recently tried to upgrade to linux v6.0.x but when trying to
> > > boot it fails with "error: out of memory" when or after loading
> > > initramfs (which then kpanics because the vfs root is missing).
> > > The latest releases I tested are v6.0.9 and v6.1-rc5 and it's broken there too.
> > >
> > > I bisected the error to this patch:
> > > 1dd685c414a7b9fdb3d23aca3aedae84f0b998ae "XArray: Add calls to
> > > might_alloc()" is the first bad commit.
> > > I've confirmed this is not a side effect of a poor bitsect because
> > > 1dd685c414a7b9fdb3d23aca3aedae84f0b998ae~1 (v5.19-rc6) works.
> >
> > That makes no sense.  I can't look into this until Wednesday, but I
> > suggest that what you have is an intermittent failure to boot, and
> > the bisect has led you down the wrong path.
> 
> I rebuilt both 1dd685c414a7b9fdb3d23aca3aedae84f0b998ae and
> the parent commit (v5.19-rc6), then tried to start each one 8 times
> (shuffled in a Thue morse sequence).
> 0 successes for 1dd685c414a7b9fdb3d23aca3aedae84f0b998ae
> 8 successes for v5.19-rc6
> 
> This really does not look like an intermittent issue.

OK, you convinced me.  Can you boot 1dd685c414 with the command line
parameters "debug initcall_debug" so we get more information?

Re: [REGRESSION] XArray commit prevents booting with 6.0-rc1 or later

[Thorsten Leemhuis]

On 28.11.22 13:18, Matthew Wilcox wrote:
> On Sun, Nov 20, 2022 at 12:20:13AM +0100, Jorropo wrote:
>> Matthew Wilcox <willy@infradead.org> wrote :
>>> On Sat, Nov 19, 2022 at 05:07:45AM +0100, Jorropo wrote:
>>>>
>>>> Hi, I recently tried to upgrade to linux v6.0.x but when trying to
>>>> boot it fails with "error: out of memory" when or after loading
>>>> initramfs (which then kpanics because the vfs root is missing).
>>>> The latest releases I tested are v6.0.9 and v6.1-rc5 and it's broken there too.
>>>>
>>>> I bisected the error to this patch:
>>>> 1dd685c414a7b9fdb3d23aca3aedae84f0b998ae "XArray: Add calls to
>>>> might_alloc()" is the first bad commit.
>>>> I've confirmed this is not a side effect of a poor bitsect because
>>>> 1dd685c414a7b9fdb3d23aca3aedae84f0b998ae~1 (v5.19-rc6) works.
>>>
>>> That makes no sense.  I can't look into this until Wednesday, but I
>>> suggest that what you have is an intermittent failure to boot, and
>>> the bisect has led you down the wrong path.
>>
>> I rebuilt both 1dd685c414a7b9fdb3d23aca3aedae84f0b998ae and
>> the parent commit (v5.19-rc6), then tried to start each one 8 times
>> (shuffled in a Thue morse sequence).
>> 0 successes for 1dd685c414a7b9fdb3d23aca3aedae84f0b998ae
>> 8 successes for v5.19-rc6
>>
>> This really does not look like an intermittent issue.
> 
> OK, you convinced me.  Can you boot 1dd685c414 with the command line
> parameters "debug initcall_debug" so we get more information?

Jorropo, did you ever provide the information Matthew asked for? I'm
asking, as this looks stalled -- and I wonder why. Or was progress made
somewhere and I just missed it?

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)

P.S.: As the Linux kernel's regression tracker I deal with a lot of
reports and sometimes miss something important when writing mails like
this. If that's the case here, don't hesitate to tell me in a public
reply, it's in everyone's interest to set the public record straight.

#regzbot poke

Re: [REGRESSION] XArray commit prevents booting with 6.0-rc1 or later

[Linux kernel regression tracking (#update)]

On 12.12.22 08:03, Thorsten Leemhuis wrote:
> On 28.11.22 13:18, Matthew Wilcox wrote:
>> On Sun, Nov 20, 2022 at 12:20:13AM +0100, Jorropo wrote:
>>> Matthew Wilcox <willy@infradead.org> wrote :
>>>> On Sat, Nov 19, 2022 at 05:07:45AM +0100, Jorropo wrote:
>>>>>
>>>>> Hi, I recently tried to upgrade to linux v6.0.x but when trying to
>>>>> boot it fails with "error: out of memory" when or after loading
>>>>> initramfs (which then kpanics because the vfs root is missing).
>>>>> The latest releases I tested are v6.0.9 and v6.1-rc5 and it's broken there too.
>>>>>
>>>>> I bisected the error to this patch:
>>>>> 1dd685c414a7b9fdb3d23aca3aedae84f0b998ae "XArray: Add calls to
>>>>> might_alloc()" is the first bad commit.
>>>>> I've confirmed this is not a side effect of a poor bitsect because
>>>>> 1dd685c414a7b9fdb3d23aca3aedae84f0b998ae~1 (v5.19-rc6) works.
>>>>
>>>> That makes no sense.  I can't look into this until Wednesday, but I
>>>> suggest that what you have is an intermittent failure to boot, and
>>>> the bisect has led you down the wrong path.
>>>
>>> I rebuilt both 1dd685c414a7b9fdb3d23aca3aedae84f0b998ae and
>>> the parent commit (v5.19-rc6), then tried to start each one 8 times
>>> (shuffled in a Thue morse sequence).
>>> 0 successes for 1dd685c414a7b9fdb3d23aca3aedae84f0b998ae
>>> 8 successes for v5.19-rc6
>>>
>>> This really does not look like an intermittent issue.
>>
>> OK, you convinced me.  Can you boot 1dd685c414 with the command line
>> parameters "debug initcall_debug" so we get more information?
> 
> Jorropo, did you ever provide the information Matthew asked for? I'm
> asking, as this looks stalled -- and I wonder why. Or was progress made
> somewhere and I just missed it?

For the record: Jorropo sent me a private mail where he states "this is
some installation issue because after some updates the previous release
that used to work does not anymore. You can close this.". Hence:

#regzbot resolve: not a regression according to the reporter

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
That page also explains what to do if mails like this annoy you.