Re: [REGRESSION] kexec does firmware reboot in kernel v6.7.6

[Steve Wahl <steve.wahl@hpe.com>]

On Sat, Mar 02, 2024 at 09:40:06PM +0530, Pavin Joseph wrote:
> Hello everyone,
> 
> On 3/2/24 20:47, Linux regression tracking (Thorsten Leemhuis) wrote> Thx
> for testing and glad to hear. Still: if you have any feedback how to
> > make that guide even better, please let me know!
> 
> Yes, I have some improvements in mind.
> Don't know if there is a Github repo where I can make a PR, but if not
> here's the gist:
> 
> 1. The git clone/fetch instructions in the TLDR is easy to follow, but there
> are conflicting information later on in the main section and reference that
> taken together does not work. I think it would be better to not perform
> shallow clones or such advanced topics could be relegated to its own
> reference section.
> 
> Here's what I ended up using:
> git clone -o mainline --no-checkout \
>   https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
> ~/linux/
> cd ~/linux/
> git remote add -t master stable \
>   https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
> git checkout --detach v6.0
> git checkout --force --detach mainline/master
> git remote set-branches --add stable linux-6.7.y
> git fetch --verbose stable
> git checkout --force --detach v6.7.7
> git checkout --force --detach v6.7.5
> 
> 2. The "installkernel" command is called "kernel-install" in OpenSuse, and
> it doesn't really perform all the steps to install kernel. It calls dracut
> to create initramfs though, but that's hardly much help.
> 
> I ended up doing:
> sudo make modules_install
> sudo install -m 0600 $(make -s image_name) /usr/lib/modules/$(make -s
> kernelrelease)/vmlinuz
> sudo install -m 0600 System.map /usr/lib/modules/$(make -s
> kernelrelease)/System.map
> sudo kernel-install add $(make -s kernelrelease) /usr/lib/modules/$(make -s
> kernelrelease)/vmlinuz
> sudo ln -sf /boot/initrd-$(make -s kernelrelease) /boot/initrd
> sudo ln -sf /usr/lib/modules/$(make -s kernelrelease)/vmlinuz
> /boot/vmlinuz-$(make -s kernelrelease)
> sudo ln -sf /boot/vmlinuz-$(make -s kernelrelease) /boot/vmlinuz
> sudo ln -sf /usr/lib/modules/$(make -s kernelrelease)/System.map
> /boot/System.map-$(make -s kernelrelease)
> sudo update-bootloader
> 
> 3. The dependencies for kernel building in OpenSuse and other major distros
> are incomplete, most of them have some form of package collection that can
> be provided as an alternative.
> For example in OpenSuse, I installed the following patterns (collection of
> packages):
> sudo zypper in -t pattern devel_basis devel_kernel devel_osc_build
> devel_rpm_build
> 
> 4. The command to build RPM package (make binrpm-pkg) fails as the modules
> are installed into "/home/<user>/linux/.../lib" while depmod checks for
> modules in "/home/<user>/linux/.../usr/lib".
> 
> I think that's it, turned out not to be a gist after all. 🙂
> Thank you very much for writing the updated guide, it was very helpful
> without which I don't think it would have been possible for someone like me
> to find/report this bug.
> 
> > > Full bisection done, culprit identified, and validated by reverting
> > > commit on mainline.
> > 
> > I assume the latter meant "reverting the culprit on mainline fixed the
> > problem"; if you meant something else, please let us know.
> 
> Clarification: reverting culprit commit on mainline fixed the problem.
> 
> Kind regards,
> Pavin Joseph.


Pavin,

I have just now built and installed 6.7.7 and succesfully kexec'd in
this fashion:

------------------------------
sph-185:~ # kexec -l /boot/vmlinuz-6.7.7-wahl --initrd=/boot/initrd-6.7.7-wahl --reuse-cmdline
sph-185:~ # systemctl kexec
...
[  OK  ] Reached target Late Shutdown Services.
         Starting Reboot via kexec...
[  493.056708][    T1] systemd-shutdown[1]: Sending SIGKILL to remaining processes...
[  493.089271][    T1] systemd-shutdown[1]: Unmounting file systems.
[  493.096285][T14707] (sd-remount)[14707]: Remounting '/' read-only in with options 'attr2,inode64,logbufs=8,logbsize=32k,noquota'.
[  493.113587][    T1] systemd-shutdown[1]: All filesystems unmounted.
[  493.119913][    T1] systemd-shutdown[1]: Deactivating swaps.
[  493.126612][    T1] systemd-shutdown[1]: All swaps deactivated.
[  493.132584][    T1] systemd-shutdown[1]: Detaching loop devices.
[  493.138718][    T1] systemd-shutdown[1]: All loop devices detached.
[  493.145036][    T1] systemd-shutdown[1]: Stopping MD devices.
[  493.150838][    T1] systemd-shutdown[1]: All MD devices stopped.
[  493.156894][    T1] systemd-shutdown[1]: Detaching DM devices.
[  493.162785][    T1] systemd-shutdown[1]: All DM devices detached.
[  493.168930][    T1] systemd-shutdown[1]: All filesystems, swaps, loop devices, MD devices and DM devices detached.
[  493.221975][    T1] systemd-shutdown[1]: Syncing filesystems and block devices.
[  493.229354][    T1] systemd-shutdown[1]: Rebooting with kexec.
[  493.303438][T14716] qla2xxx [0003:61:00.0]-fffa:14: Adapter shutdown
[  493.309930][T14716] qla2xxx [0003:61:00.0]-00af:14: Performing ISP error recovery - ha=00000000ae891d2f.
[  493.330535][T14716] qla2xxx [0003:61:00.0]-fffe:14: Adapter shutdown successfully.
[  494.114055][T14716] mana 0000:61:00.1: Shutdown was called
[  494.643693][T14716] kvm: exiting hardware virtualization
[  494.649419][T14716] kexec_core: Starting new kernel


Invalid physical address chosen!




Physical KASLR disabled: no suitable memory region!

[    0.000000][    T0] Linux version 6.7.7-wahl (root@sph-185) (gcc (SUSE Linux) 7.5.0, GNU ld (GNU Binutils; SUSE Linux Enterprise 15) 2.41.0.20230908-150100.7.46) #1 SMP PREEMPT_DYNAMIC Sat Mar  2 17:22:28 CST 2024
------------------------------

This was on SLES, not open Suse.

The machines I work on are large, though.  Can you give specifics on
exactly how you are performing your kexec, and what hardware you are
using when you hit this (especially memory size)?  Have you made any
special arrangements for the size of memory reserved for kexec on your
system?

The patch can use slightly more memory to create the identity maps,
and the only thing I can think of right now is that little bit causing
you to run out of memory.

Thank you!

--> Steve Wahl
-- 
Steve Wahl, Hewlett Packard Enterprise