test_efi_selftest occasionally fails on qemu-x86?

test_efi_selftest occasionally fails on qemu-x86?

[Bin Meng]

Hi,

Please check the log:
https://dev.azure.com/bmeng/GitHub/_build/results?buildId=302&view=logs&j=cf9e8018-ed89-5c32-5567-a8524d857521&t=4197aefe-2a0c-53e5-5744-78f78ce6022b

Re-run the pipeline with the same commit, the tests pass.
https://dev.azure.com/bmeng/GitHub/_build/results?buildId=304&view=logs&j=cf9e8018-ed89-5c32-5567-a8524d857521&t=4197aefe-2a0c-53e5-5744-78f78ce6022b

There might be some instability on qemu-x86 with efi_selftest. Any ideas?

Regards,
Bin

test_efi_selftest occasionally fails on qemu-x86?

[Heinrich Schuchardt]

On 12/22/20 6:00 AM, Bin Meng wrote:
> Hi,
>
> Please check the log:
> https://dev.azure.com/bmeng/GitHub/_build/results?buildId=302&view=logs&j=cf9e8018-ed89-5c32-5567-a8524d857521&t=4197aefe-2a0c-53e5-5744-78f78ce6022b
>
> Re-run the pipeline with the same commit, the tests pass.
> https://dev.azure.com/bmeng/GitHub/_build/results?buildId=304&view=logs&j=cf9e8018-ed89-5c32-5567-a8524d857521&t=4197aefe-2a0c-53e5-5744-78f78ce6022b
>
> There might be some instability on qemu-x86 with efi_selftest. Any ideas?
>
> Regards,
> Bin
>

Hello Bin,

The failing routine is

      u_boot_console.restart_uboot()

after the UEFI tests were completed successfully.

This is the function:

     def restart_uboot(self):
         """Shut down and restart U-Boot."""
         self.cleanup_spawn()
         self.ensure_spawned()

In ensure_spawned() we get the exception:

     Exception: Bad pattern found on console: main_signon

ensure_spawned() complains because it sees the main_signon message a
second time.

This is the console output:

     U-Boot 2021.01-rc4-23431-g6ef1b75087 (Dec 22 2020 - 03:07:46 +0000)

     CPU:   QEMU Virtual CPU version 2.5+
     DRAM:  128 MiB
     unable to find cpus device
     Warning: MP init failure

     U-Boot 2021.01-rc4-23431-g6ef1b75087 (Dec 22 2020 - 03:07:46 +0000)

So obviously U-Boot is crashing during the boot process after x86_mp_init().

The same error has shown up on Gitlab as reported by Simon. Cf.

https://gitlab.denx.de/u-boot/custodians/u-boot-dm/-/jobs/190746/raw

I suggest that you add

     #define DEBUG 1

at the top of common.h to close in on the location of the crash.

Best regards

Heinrich

test_efi_selftest occasionally fails on qemu-x86?

[Heinrich Schuchardt]

On 12/22/20 9:14 AM, Heinrich Schuchardt wrote:
> On 12/22/20 6:00 AM, Bin Meng wrote:
>> Hi,
>>
>> Please check the log:
>> https://dev.azure.com/bmeng/GitHub/_build/results?buildId=302&view=logs&j=cf9e8018-ed89-5c32-5567-a8524d857521&t=4197aefe-2a0c-53e5-5744-78f78ce6022b
>>
>>
>> Re-run the pipeline with the same commit, the tests pass.
>> https://dev.azure.com/bmeng/GitHub/_build/results?buildId=304&view=logs&j=cf9e8018-ed89-5c32-5567-a8524d857521&t=4197aefe-2a0c-53e5-5744-78f78ce6022b
>>
>>
>> There might be some instability on qemu-x86 with efi_selftest. Any ideas?
>>
>> Regards,
>> Bin
>>
>
> Hello Bin,
>
> The failing routine is
>
>  ???? u_boot_console.restart_uboot()
>
> after the UEFI tests were completed successfully.
>
> This is the function:
>
>  ??? def restart_uboot(self):
>  ??????? """Shut down and restart U-Boot."""
>  ??????? self.cleanup_spawn()
>  ??????? self.ensure_spawned()
>
> In ensure_spawned() we get the exception:
>
>  ??? Exception: Bad pattern found on console: main_signon
>
> ensure_spawned() complains because it sees the main_signon message a
> second time.
>
> This is the console output:
>
>  ??? U-Boot 2021.01-rc4-23431-g6ef1b75087 (Dec 22 2020 - 03:07:46 +0000)
>
>  ??? CPU:?? QEMU Virtual CPU version 2.5+
>  ??? DRAM:? 128 MiB
>  ??? unable to find cpus device
>  ??? Warning: MP init failure
>
>  ??? U-Boot 2021.01-rc4-23431-g6ef1b75087 (Dec 22 2020 - 03:07:46 +0000)
>
> So obviously U-Boot is crashing during the boot process after
> x86_mp_init().
>
> The same error has shown up on Gitlab as reported by Simon. Cf.
>
> https://gitlab.denx.de/u-boot/custodians/u-boot-dm/-/jobs/190746/raw
>
> I suggest that you add
>
>  ??? #define DEBUG 1
>
> at the top of common.h to close in on the location of the crash.
>
> Best regards
>
> Heinrich

Here is a log with DEBUG set:
https://gitlab.denx.de/u-boot/custodians/u-boot-efi/-/jobs/192520/raw

This is the interesting part:

+u-boot-test-reset qemu-x86 na
initcall: fff6ef31
U-Boot 2021.01-rc4-00037-ge528c1d3e3 (Dec 22 2020 - 08:16:58 +0000)
initcall: fff150c7
U-Boot code: FFF00000 -> FFFA880C  BSS: -> FFFB5404

...

VBE: Setting VESA mode 0x144
<<<<CRASH occurs here >>>>
initcall: fff6ef31
U-Boot 2021.01-rc4-00037-ge528c1d3e3 (Dec 22 2020 - 08:16:58 +0000)

On my local system I see:

VBE: framebuffer: d0000000
VBE: Setting VESA mode 0x144
Final vesa mode 0x144
base = d0000000, copy_base = 0

So the crash occurs before pci_rom_probe() is finished.

Best regards

Heinrich

test_efi_selftest occasionally fails on qemu-x86?

[Simon Glass]

Hi Heinrich,

On Tue, 22 Dec 2020 at 01:40, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
>
> On 12/22/20 9:14 AM, Heinrich Schuchardt wrote:
> > On 12/22/20 6:00 AM, Bin Meng wrote:
> >> Hi,
> >>
> >> Please check the log:
> >> https://dev.azure.com/bmeng/GitHub/_build/results?buildId=302&view=logs&j=cf9e8018-ed89-5c32-5567-a8524d857521&t=4197aefe-2a0c-53e5-5744-78f78ce6022b
> >>
> >>
> >> Re-run the pipeline with the same commit, the tests pass.
> >> https://dev.azure.com/bmeng/GitHub/_build/results?buildId=304&view=logs&j=cf9e8018-ed89-5c32-5567-a8524d857521&t=4197aefe-2a0c-53e5-5744-78f78ce6022b
> >>
> >>
> >> There might be some instability on qemu-x86 with efi_selftest. Any ideas?
> >>
> >> Regards,
> >> Bin
> >>
> >
> > Hello Bin,
> >
> > The failing routine is
> >
> >       u_boot_console.restart_uboot()
> >
> > after the UEFI tests were completed successfully.
> >
> > This is the function:
> >
> >      def restart_uboot(self):
> >          """Shut down and restart U-Boot."""
> >          self.cleanup_spawn()
> >          self.ensure_spawned()
> >
> > In ensure_spawned() we get the exception:
> >
> >      Exception: Bad pattern found on console: main_signon
> >
> > ensure_spawned() complains because it sees the main_signon message a
> > second time.
> >
> > This is the console output:
> >
> >      U-Boot 2021.01-rc4-23431-g6ef1b75087 (Dec 22 2020 - 03:07:46 +0000)
> >
> >      CPU:   QEMU Virtual CPU version 2.5+
> >      DRAM:  128 MiB
> >      unable to find cpus device
> >      Warning: MP init failure
> >
> >      U-Boot 2021.01-rc4-23431-g6ef1b75087 (Dec 22 2020 - 03:07:46 +0000)
> >
> > So obviously U-Boot is crashing during the boot process after
> > x86_mp_init().
> >
> > The same error has shown up on Gitlab as reported by Simon. Cf.
> >
> > https://gitlab.denx.de/u-boot/custodians/u-boot-dm/-/jobs/190746/raw
> >
> > I suggest that you add
> >
> >      #define DEBUG 1
> >
> > at the top of common.h to close in on the location of the crash.
> >
> > Best regards
> >
> > Heinrich
>
> Here is a log with DEBUG set:
> https://gitlab.denx.de/u-boot/custodians/u-boot-efi/-/jobs/192520/raw
>
> This is the interesting part:
>
> +u-boot-test-reset qemu-x86 na
> initcall: fff6ef31
> U-Boot 2021.01-rc4-00037-ge528c1d3e3 (Dec 22 2020 - 08:16:58 +0000)
> initcall: fff150c7
> U-Boot code: FFF00000 -> FFFA880C  BSS: -> FFFB5404
>
> ...
>
> VBE: Setting VESA mode 0x144
> <<<<CRASH occurs here >>>>
> initcall: fff6ef31
> U-Boot 2021.01-rc4-00037-ge528c1d3e3 (Dec 22 2020 - 08:16:58 +0000)
>
> On my local system I see:
>
> VBE: framebuffer: d0000000
> VBE: Setting VESA mode 0x144
> Final vesa mode 0x144
> base = d0000000, copy_base = 0
>
> So the crash occurs before pci_rom_probe() is finished.

This may be related to the MP changes I made recently. If I recall
correctly, MP init isn't really needed on qemu, but I'm not sure.

Regards,
Simon