Hang loading initrd since last Friday.

Hang loading initrd since last Friday.

[Ben Greear]

Hello,

Pulling today's upstream 6.3+ tree, my system now hangs loading initrd.  Nothing useful is seen on
serial console after that.  This was working late last week.  Other older kernels boot OK, so
I think is is related to the kernel.

I have not yet started bisect, curious if this is already know problem.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: Hang loading initrd since last Friday.

[Bagas Sanjaya]

[-- Attachment #1: Type: text/plain, Size: 642 bytes --]

On Tue, May 02, 2023 at 02:03:33PM -0700, Ben Greear wrote:
> Hello,
> 
> Pulling today's upstream 6.3+ tree, my system now hangs loading initrd.  Nothing useful is seen on
> serial console after that.  This was working late last week.  Other older kernels boot OK, so
> I think is is related to the kernel.

What is your setup (especially kernel config)?

Is v6.3 works for you? I haven't seen this regression on my computer
using v6.3.

> 
> I have not yet started bisect, curious if this is already know problem.

Certainly you should try bisection.

Thanks.

-- 
An old man doll... just what I always wanted! - Clara

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: Hang loading initrd since last Friday.

[Linux regression tracking (Thorsten Leemhuis)]

On 03.05.23 04:31, Bagas Sanjaya wrote:
> On Tue, May 02, 2023 at 02:03:33PM -0700, Ben Greear wrote:
>>
>> Pulling today's upstream 6.3+ tree, my system now hangs loading initrd.

Just to understand this properly: you mean after the boot loader said it
loaded the initramfs, not when the kernel starts using it?

>>  Nothing useful is seen on
>> serial console after that.  This was working late last week.  Other older kernels boot OK, so
>> I think is is related to the kernel.
> 
> What is your setup (especially kernel config)?
> 
> Is v6.3 works for you? I haven't seen this regression on my computer
> using v6.3.

Ben, you just to be sure might want to check if a kernel image that
currently works still boots if you regenerate its initramfs: maybe the
initramfs generator or something it uses changed recently and is causing
your problem. I've seen a case or two where that was the case.

>> I have not yet started bisect, curious if this is already know problem.
> Certainly you should try bisection.

Yeah.

BTW: What kind of platform is this?

Ciao, Thorsten

Re: Hang loading initrd since last Friday.

[Ben Greear]

On 5/2/23 21:46, Linux regression tracking (Thorsten Leemhuis) wrote:
> On 03.05.23 04:31, Bagas Sanjaya wrote:
>> On Tue, May 02, 2023 at 02:03:33PM -0700, Ben Greear wrote:
>>>
>>> Pulling today's upstream 6.3+ tree, my system now hangs loading initrd.
> 
> Just to understand this properly: you mean after the boot loader said it
> loaded the initramfs, not when the kernel starts using it?

I am bisecting...it appears to be .config related.  If I skip enabling things
during bisect, then I do not see the problem.  But I copied my original buggy .config
into an otherwise good kernel commit, and now it fails.  I'm manually bisecting
the .config settings....

Thanks,
Ben

> 
>>>   Nothing useful is seen on
>>> serial console after that.  This was working late last week.  Other older kernels boot OK, so
>>> I think is is related to the kernel.
>>
>> What is your setup (especially kernel config)?
>>
>> Is v6.3 works for you? I haven't seen this regression on my computer
>> using v6.3.
> 
> Ben, you just to be sure might want to check if a kernel image that
> currently works still boots if you regenerate its initramfs: maybe the
> initramfs generator or something it uses changed recently and is causing
> your problem. I've seen a case or two where that was the case.
> 
>>> I have not yet started bisect, curious if this is already know problem.
>> Certainly you should try bisection.
> 
> Yeah.
> 
> BTW: What kind of platform is this?
> 
> Ciao, Thorsten
> 

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: Hang loading initrd since last Friday.

[Raghavendra K T]

On 5/4/2023 5:19 AM, Ben Greear wrote:
> On 5/2/23 21:46, Linux regression tracking (Thorsten Leemhuis) wrote:
>> On 03.05.23 04:31, Bagas Sanjaya wrote:
>>> On Tue, May 02, 2023 at 02:03:33PM -0700, Ben Greear wrote:
>>>>
>>>> Pulling today's upstream 6.3+ tree, my system now hangs loading initrd.
>>
>> Just to understand this properly: you mean after the boot loader said it
>> loaded the initramfs, not when the kernel starts using it?
> 
> I am bisecting...it appears to be .config related.  If I skip enabling 
> things
> during bisect, then I do not see the problem.  But I copied my original 
> buggy .config
> into an otherwise good kernel commit, and now it fails.  I'm manually 
> bisecting
> the .config settings....
> 
> Thanks,
> Ben
> 

Hello Ben,

(CCing Jens here)

I was facing same issue too. Surprising I did not hit the same on the
guest VM.

After bisecting, found that Reverting
[9f4107b07b17b5ee68af680150f91227bea2df6f] block: store 
bdev->bd_disk->fops->submit_bio state in bdev

Helped me to boot back the system. Can you please confirm

Thanks
- Raghu

>>
>>>>   Nothing useful is seen on
>>>> serial console after that.  This was working late last week.  Other 
>>>> older kernels boot OK, so
>>>> I think is is related to the kernel.
>>>
>>> What is your setup (especially kernel config)?
>>>
>>> Is v6.3 works for you? I haven't seen this regression on my computer
>>> using v6.3.
>>
>> Ben, you just to be sure might want to check if a kernel image that
>> currently works still boots if you regenerate its initramfs: maybe the
>> initramfs generator or something it uses changed recently and is causing
>> your problem. I've seen a case or two where that was the case.
>>
>>>> I have not yet started bisect, curious if this is already know problem.
>>> Certainly you should try bisection.
>>
>> Yeah.
>>
>> BTW: What kind of platform is this?
>>
>> Ciao, Thorsten
>>
> 

Re: Hang loading initrd since last Friday.

[Jens Axboe]

On 5/4/23 10:38 AM, Raghavendra K T wrote:
> On 5/4/2023 5:19 AM, Ben Greear wrote:
>> On 5/2/23 21:46, Linux regression tracking (Thorsten Leemhuis) wrote:
>>> On 03.05.23 04:31, Bagas Sanjaya wrote:
>>>> On Tue, May 02, 2023 at 02:03:33PM -0700, Ben Greear wrote:
>>>>>
>>>>> Pulling today's upstream 6.3+ tree, my system now hangs loading initrd.
>>>
>>> Just to understand this properly: you mean after the boot loader said it
>>> loaded the initramfs, not when the kernel starts using it?
>>
>> I am bisecting...it appears to be .config related.  If I skip enabling things
>> during bisect, then I do not see the problem.  But I copied my original buggy .config
>> into an otherwise good kernel commit, and now it fails.  I'm manually bisecting
>> the .config settings....
>>
>> Thanks,
>> Ben
>>
> 
> Hello Ben,
> 
> (CCing Jens here)
> 
> I was facing same issue too. Surprising I did not hit the same on the
> guest VM.
> 
> After bisecting, found that Reverting
> [9f4107b07b17b5ee68af680150f91227bea2df6f] block: store bdev->bd_disk->fops->submit_bio state in bdev
> 
> Helped me to boot back the system. Can you please confirm

Can you try and pull:

git://git.kernel.dk/linux for-6.4/block

into current -git and see if that works for you? There's a fix for that
commit in there.

-- 
Jens Axboe

Re: Hang loading initrd since last Friday.

[Ben Greear]

On 5/3/23 16:49, Ben Greear wrote:
> On 5/2/23 21:46, Linux regression tracking (Thorsten Leemhuis) wrote:
>> On 03.05.23 04:31, Bagas Sanjaya wrote:
>>> On Tue, May 02, 2023 at 02:03:33PM -0700, Ben Greear wrote:
>>>>
>>>> Pulling today's upstream 6.3+ tree, my system now hangs loading initrd.
>>
>> Just to understand this properly: you mean after the boot loader said it
>> loaded the initramfs, not when the kernel starts using it?
> 
> I am bisecting...it appears to be .config related.  If I skip enabling things
> during bisect, then I do not see the problem.  But I copied my original buggy .config
> into an otherwise good kernel commit, and now it fails.  I'm manually bisecting
> the .config settings....

I had enabled this option:  CONFIG_HYPERV_VTL_MODE

And that makes it not boot.  Not sure that is expected or not, but I will simply un-select it
and continue on, as I do not actually need that feature.

Thanks,
Ben

RE: Hang loading initrd since last Friday.

[Michael Kelley (LINUX)]

From: Ben Greear <greearb@candelatech.com> Sent: Thursday, May 4, 2023 10:23 AM
> 
> On 5/3/23 16:49, Ben Greear wrote:
> > On 5/2/23 21:46, Linux regression tracking (Thorsten Leemhuis) wrote:
> >> On 03.05.23 04:31, Bagas Sanjaya wrote:
> >>> On Tue, May 02, 2023 at 02:03:33PM -0700, Ben Greear wrote:
> >>>>
> >>>> Pulling today's upstream 6.3+ tree, my system now hangs loading initrd.
> >>
> >> Just to understand this properly: you mean after the boot loader said it
> >> loaded the initramfs, not when the kernel starts using it?
> >
> > I am bisecting...it appears to be .config related.  If I skip enabling things
> > during bisect, then I do not see the problem.  But I copied my original buggy .config
> > into an otherwise good kernel commit, and now it fails.  I'm manually bisecting
> > the .config settings....
> 
> I had enabled this option:  CONFIG_HYPERV_VTL_MODE
> 
> And that makes it not boot.  Not sure that is expected or not, but
> I will simply un-select it and continue on, as I do not actually need that feature.
> 

What you observed is expected when CONFIG_HYPERV_VTL_MODE is selected,
which is why it defaults to N.  The comment for that option in the Kconfig file says:

	  Select this option to build a Linux kernel to run at a VTL other than
	  the normal VTL0, which currently is only VTL2.  This option
	  initializes the x86 platform for VTL2, and adds the ability to boot
	  secondary CPUs directly into 64-bit context as required for VTLs other
	  than 0.  A kernel built with this option must run at VTL2, and will
	  not run as a normal guest.

Note the last sentence.

Michael

Re: Hang loading initrd since last Friday.

[Raghavendra K T]

On 5/4/2023 10:14 PM, Jens Axboe wrote:
> On 5/4/23 10:38 AM, Raghavendra K T wrote:
>> On 5/4/2023 5:19 AM, Ben Greear wrote:
>>> On 5/2/23 21:46, Linux regression tracking (Thorsten Leemhuis) wrote:
>>>> On 03.05.23 04:31, Bagas Sanjaya wrote:
>>>>> On Tue, May 02, 2023 at 02:03:33PM -0700, Ben Greear wrote:
>>>>>>
>>>>>> Pulling today's upstream 6.3+ tree, my system now hangs loading initrd.
>>>>
>>>> Just to understand this properly: you mean after the boot loader said it
>>>> loaded the initramfs, not when the kernel starts using it?
>>>
>>> I am bisecting...it appears to be .config related.  If I skip enabling things
>>> during bisect, then I do not see the problem.  But I copied my original buggy .config
>>> into an otherwise good kernel commit, and now it fails.  I'm manually bisecting
>>> the .config settings....
>>>
>>> Thanks,
>>> Ben
>>>
>>
>> Hello Ben,
>>
>> (CCing Jens here)
>>
>> I was facing same issue too. Surprising I did not hit the same on the
>> guest VM.
>>
>> After bisecting, found that Reverting
>> [9f4107b07b17b5ee68af680150f91227bea2df6f] block: store bdev->bd_disk->fops->submit_bio state in bdev
>>
>> Helped me to boot back the system. Can you please confirm
> 
> Can you try and pull:
> 
> git://git.kernel.dk/linux for-6.4/block
> 
> into current -git and see if that works for you? There's a fix for that
> commit in there.
> 

Jens, pulling for-6.4/block helped in my case.

Thank you :)

Re: Hang loading initrd since last Friday.

[Jens Axboe]

On 5/4/23 12:48 PM, Raghavendra K T wrote:
> On 5/4/2023 10:14 PM, Jens Axboe wrote:
>> On 5/4/23 10:38 AM, Raghavendra K T wrote:
>>> On 5/4/2023 5:19 AM, Ben Greear wrote:
>>>> On 5/2/23 21:46, Linux regression tracking (Thorsten Leemhuis) wrote:
>>>>> On 03.05.23 04:31, Bagas Sanjaya wrote:
>>>>>> On Tue, May 02, 2023 at 02:03:33PM -0700, Ben Greear wrote:
>>>>>>>
>>>>>>> Pulling today's upstream 6.3+ tree, my system now hangs loading initrd.
>>>>>
>>>>> Just to understand this properly: you mean after the boot loader said it
>>>>> loaded the initramfs, not when the kernel starts using it?
>>>>
>>>> I am bisecting...it appears to be .config related.  If I skip enabling things
>>>> during bisect, then I do not see the problem.  But I copied my original buggy .config
>>>> into an otherwise good kernel commit, and now it fails.  I'm manually bisecting
>>>> the .config settings....
>>>>
>>>> Thanks,
>>>> Ben
>>>>
>>>
>>> Hello Ben,
>>>
>>> (CCing Jens here)
>>>
>>> I was facing same issue too. Surprising I did not hit the same on the
>>> guest VM.
>>>
>>> After bisecting, found that Reverting
>>> [9f4107b07b17b5ee68af680150f91227bea2df6f] block: store bdev->bd_disk->fops->submit_bio state in bdev
>>>
>>> Helped me to boot back the system. Can you please confirm
>>
>> Can you try and pull:
>>
>> git://git.kernel.dk/linux for-6.4/block
>>
>> into current -git and see if that works for you? There's a fix for that
>> commit in there.
>>
> 
> Jens, pulling for-6.4/block helped in my case.

Great! It will be heading to Linus's tree before -rc1.

-- 
Jens Axboe

Re: Hang loading initrd since last Friday.

[Ben Greear]

On 5/4/23 10:59, Michael Kelley (LINUX) wrote:
> From: Ben Greear <greearb@candelatech.com> Sent: Thursday, May 4, 2023 10:23 AM
>>
>> On 5/3/23 16:49, Ben Greear wrote:
>>> On 5/2/23 21:46, Linux regression tracking (Thorsten Leemhuis) wrote:
>>>> On 03.05.23 04:31, Bagas Sanjaya wrote:
>>>>> On Tue, May 02, 2023 at 02:03:33PM -0700, Ben Greear wrote:
>>>>>>
>>>>>> Pulling today's upstream 6.3+ tree, my system now hangs loading initrd.
>>>>
>>>> Just to understand this properly: you mean after the boot loader said it
>>>> loaded the initramfs, not when the kernel starts using it?
>>>
>>> I am bisecting...it appears to be .config related.  If I skip enabling things
>>> during bisect, then I do not see the problem.  But I copied my original buggy .config
>>> into an otherwise good kernel commit, and now it fails.  I'm manually bisecting
>>> the .config settings....
>>
>> I had enabled this option:  CONFIG_HYPERV_VTL_MODE
>>
>> And that makes it not boot.  Not sure that is expected or not, but
>> I will simply un-select it and continue on, as I do not actually need that feature.
>>
> 
> What you observed is expected when CONFIG_HYPERV_VTL_MODE is selected,
> which is why it defaults to N.  The comment for that option in the Kconfig file says:
> 
> 	  Select this option to build a Linux kernel to run at a VTL other than
> 	  the normal VTL0, which currently is only VTL2.  This option
> 	  initializes the x86 platform for VTL2, and adds the ability to boot
> 	  secondary CPUs directly into 64-bit context as required for VTLs other
> 	  than 0.  A kernel built with this option must run at VTL2, and will
> 	  not run as a normal guest.

I didn't read it close enough, but that is my fault.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

RE: Hang loading initrd since last Friday.

[Michael Kelley (LINUX)]

From: Ben Greear <greearb@candelatech.com> Sent: Thursday, May 4, 2023 4:46 PM
> 
> On 5/4/23 10:59, Michael Kelley (LINUX) wrote:
> > From: Ben Greear <greearb@candelatech.com> Sent: Thursday, May 4, 2023 10:23 AM
> >>
> >> On 5/3/23 16:49, Ben Greear wrote:
> >>> On 5/2/23 21:46, Linux regression tracking (Thorsten Leemhuis) wrote:
> >>>> On 03.05.23 04:31, Bagas Sanjaya wrote:
> >>>>> On Tue, May 02, 2023 at 02:03:33PM -0700, Ben Greear wrote:
> >>>>>>
> >>>>>> Pulling today's upstream 6.3+ tree, my system now hangs loading initrd.
> >>>>
> >>>> Just to understand this properly: you mean after the boot loader said it
> >>>> loaded the initramfs, not when the kernel starts using it?
> >>>
> >>> I am bisecting...it appears to be .config related.  If I skip enabling things
> >>> during bisect, then I do not see the problem.  But I copied my original buggy .config
> >>> into an otherwise good kernel commit, and now it fails.  I'm manually bisecting
> >>> the .config settings....
> >>
> >> I had enabled this option:  CONFIG_HYPERV_VTL_MODE
> >>
> >> And that makes it not boot.  Not sure that is expected or not, but
> >> I will simply un-select it and continue on, as I do not actually need that feature.
> >>
> >
> > What you observed is expected when CONFIG_HYPERV_VTL_MODE is selected,
> > which is why it defaults to N.  The comment for that option in the Kconfig file says:
> >
> > 	  Select this option to build a Linux kernel to run at a VTL other than
> > 	  the normal VTL0, which currently is only VTL2.  This option
> > 	  initializes the x86 platform for VTL2, and adds the ability to boot
> > 	  secondary CPUs directly into 64-bit context as required for VTLs other
> > 	  than 0.  A kernel built with this option must run at VTL2, and will
> > 	  not run as a normal guest.
> 
> I didn't read it close enough, but that is my fault.
> 

No problem. :-)   Just FYI, in the future if you are come across issues
specifically related to running Linux as a guest on Hyper-V, feel free
to copy the linux-hyperv@vger.kernel.org mailing list.  That may
elicit faster help from someone with Hyper-V expertise and potentially
save you some trouble.

Michael