All of lore.kernel.org
 help / color / mirror / Atom feed
From: Alexandru Elisei <alexandru.elisei@arm.com>
To: a.zummo@towertech.it, alexandre.belloni@bootlin.com,
	linux-rtc@vger.kernel.org, linux-kernel@vger.kernel.org,
	ardb@kernel.org, linux-efi@vger.kernel.org,
	catalin.marinas@arm.com, will@kernel.org,
	linux-arm-kernel@lists.infradead.org
Subject: [BUG] rtc-efi: Error in efi.get_time() spams dmesg with error message
Date: Tue, 8 Nov 2022 10:55:15 +0000	[thread overview]
Message-ID: <Y2o1hdZK9GGDVJsS@monolith.localdoman> (raw)

Hi,

Commit d3549a938b73 ("efi/arm64: libstub: avoid SetVirtualAddressMap() when
possible") exposed a firmware error on an Ampere Altra machine that was
causing the machine to panic. Then commit 23715a26c8d8 ("arm64: efi:
Recover from synchronous exceptions occurring in firmware") made the EFI
exception non-fatal, and disabled runtime services when the exception
happens. The interaction between those two patches are being discussed in a
separate thread [1], but that should be orthogonal to this.

Now efi.get_time() fails and each time an error message is printed to
dmesg, which happens several times a second and clutters dmesg
unnecessarily, to the point it becomes unusable.

I was wondering if it would be possible to turn dev_err() into a
dev_WARN_ONCE() or do something to avoid this issue. Tried to replace
dev_err() with dev_err_ratelimited(), and the error message was displayed
less often (about once per second), but dmesg was still being cluttered.

Here's a log with what is happening (the boot part of the log has been
removed for brevity, I've kept the kernel splats for context, can provide
full logs, kernel config, command line, etc, to reproduce it; goes without
saying that I am willing to test the fix myself):

[   55.479519] [Firmware Bug]: Unable to handle paging request in EFI runtime service
[   55.487122] CPU: 62 PID: 9 Comm: kworker/u320:0 Tainted: G          I        6.1.0-rc4 #60
[   55.487128] Hardware name: WIWYNN Mt.Jade Server System B81.03001.0005/Mt.Jade Motherboard, BIOS 1.08.20220218 (SCP: 1.08.20220218) 2022/02/18
[   55.487131] Workqueue: efi_rts_wq efi_call_rts
[   55.487158] Call trace:
[   55.487161]  dump_backtrace.part.0+0xdc/0xf0
[   55.487177]  show_stack+0x18/0x40
[   55.487180]  dump_stack_lvl+0x68/0x84
[   55.487190]  dump_stack+0x18/0x34
[   55.487192]  efi_runtime_fixup_exception+0x74/0x88
[   55.487199]  __do_kernel_fault+0x108/0x1b0
[   55.487204]  do_page_fault+0xd0/0x400
[   55.487207]  do_translation_fault+0xac/0xc0
[   55.487209]  do_mem_abort+0x44/0x94
[   55.487212]  el1_abort+0x40/0x6c
[   55.487214]  el1h_64_sync_handler+0xd8/0xe4
[   55.487218]  el1h_64_sync+0x64/0x68
[   55.487221]  0xb7eb7ae4
[   55.487224]  0xb7eb8668
[   55.487225]  0xb7eb6e08
[   55.487227]  0xb7eb68ec
[   55.487228]  0xb7eb3824
[   55.487230]  0xb7eb05a8
[   55.487231]  0xb7eb12a0
[   55.487232]  0xb7e43504
[   55.487234]  0xb7e43650
[   55.487235]  0xb7e482d0
[   55.487237]  0xb7e4907c
[   55.487238]  0xb7e49ff4
[   55.487239]  0xb7e40888
[   55.487241]  0xb7cb3328
[   55.487242]  0xb7cb0674
[   55.487243]  __efi_rt_asm_wrapper+0x54/0x70
[   55.487246]  efi_call_rts+0x28c/0x3d0
[   55.487249]  process_one_work+0x1d0/0x320
[   55.487258]  worker_thread+0x14c/0x444
[   55.487261]  kthread+0x10c/0x110
[   55.487264]  ret_from_fork+0x10/0x20
[   55.487268] [Firmware Bug]: Synchronous exception occurred in EFI runtime service set_time()
[   55.495735] ------------[ cut here ]------------
[   55.495739] WARNING: CPU: 62 PID: 9 at drivers/firmware/efi/runtime-wrappers.c:111 efi_call_virt_check_flags+0x40/0xac
[   55.495746] Modules linked in:
[   55.495749] CPU: 62 PID: 9 Comm: kworker/u320:0 Tainted: G          I        6.1.0-rc4 #60
[   55.495751] Hardware name: WIWYNN Mt.Jade Server System B81.03001.0005/Mt.Jade Motherboard, BIOS 1.08.20220218 (SCP: 1.08.20220218) 2022/02/18
[   55.495753] Workqueue: efi_rts_wq efi_call_rts
[   55.495757] pstate: 004000c9 (nzcv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   55.495761] pc : efi_call_virt_check_flags+0x40/0xac
[   55.495764] lr : efi_call_rts+0x29c/0x3d0
[   55.495767] sp : ffff80000861bd40
[   55.495768] x29: ffff80000861bd40 x28: 0000000000000000 x27: 0000000000000000
[   55.495772] x26: ffffb251470e9e68 x25: ffff3fff89714805 x24: 0000000000000000
[   55.495775] x23: 0000000000000000 x22: 0000000000000000 x21: 00000000000000c0
[   55.495778] x20: ffffb25146688de0 x19: 0000000000000000 x18: ffffffffffffffff
[   55.495780] x17: 657320656d69746e x16: 757220494645206e x15: 6920646572727563
[   55.495784] x14: 636f206e6f697470 x13: ffff403e40540000 x12: 0000000000001c14
[   55.495787] x11: 000000000000095c x10: ffff403e40800000 x9 : ffff403e40540000
[   55.495790] x8 : 00000000ffff7fff x7 : ffff403e40800000 x6 : 0000000000000000
[   55.495792] x5 : ffff083e7fe9aaa0 x4 : 0000000000000000 x3 : 0000000000000000
[   55.495796] x2 : 0000000000000000 x1 : ffffb25146688de0 x0 : 00000000000000c0
[   55.495799] Call trace:
[   55.495800]  efi_call_virt_check_flags+0x40/0xac
[   55.495802]  efi_call_rts+0x29c/0x3d0
[   55.495805]  process_one_work+0x1d0/0x320
[   55.495808]  worker_thread+0x14c/0x444
[   55.495811]  kthread+0x10c/0x110
[   55.495814]  ret_from_fork+0x10/0x20
[   55.495815] ---[ end trace 0000000000000000 ]---
[   55.495818] Disabling lock debugging due to kernel taint
[   55.495822] efi: [Firmware Bug]: IRQ flags corrupted (0x00000000=>0x000000c0) by EFI set_time
[   55.504434] efi: EFI Runtime Services are disabled!
[   55.504465] rtc-efi rtc-efi.0: can't read time
[   56.479370] efi: EFI Runtime Services are disabled!
[   56.479394] rtc-efi rtc-efi.0: can't read time
[   56.483855] rtc-efi rtc-efi.0: can't read time
[   56.488306] rtc-efi rtc-efi.0: can't read time
[   57.479574] rtc-efi rtc-efi.0: can't read time
[   57.484030] rtc-efi rtc-efi.0: can't read time
[   57.488474] rtc-efi rtc-efi.0: can't read time
[   58.479692] rtc-efi rtc-efi.0: can't read time
[   58.484139] rtc-efi rtc-efi.0: can't read time
[   58.488582] rtc-efi rtc-efi.0: can't read time
[   59.479691] rtc-efi rtc-efi.0: can't read time
... on, and on, on ...

[1] https://lore.kernel.org/linux-arm-kernel/Y2lAB508TrrjpDPi@monolith.localdoman/

Thanks,
Alex

WARNING: multiple messages have this Message-ID (diff)
From: Alexandru Elisei <alexandru.elisei@arm.com>
To: a.zummo@towertech.it, alexandre.belloni@bootlin.com,
	linux-rtc@vger.kernel.org, linux-kernel@vger.kernel.org,
	ardb@kernel.org, linux-efi@vger.kernel.org,
	catalin.marinas@arm.com, will@kernel.org,
	linux-arm-kernel@lists.infradead.org
Subject: [BUG] rtc-efi: Error in efi.get_time() spams dmesg with error message
Date: Tue, 8 Nov 2022 10:55:15 +0000	[thread overview]
Message-ID: <Y2o1hdZK9GGDVJsS@monolith.localdoman> (raw)

Hi,

Commit d3549a938b73 ("efi/arm64: libstub: avoid SetVirtualAddressMap() when
possible") exposed a firmware error on an Ampere Altra machine that was
causing the machine to panic. Then commit 23715a26c8d8 ("arm64: efi:
Recover from synchronous exceptions occurring in firmware") made the EFI
exception non-fatal, and disabled runtime services when the exception
happens. The interaction between those two patches are being discussed in a
separate thread [1], but that should be orthogonal to this.

Now efi.get_time() fails and each time an error message is printed to
dmesg, which happens several times a second and clutters dmesg
unnecessarily, to the point it becomes unusable.

I was wondering if it would be possible to turn dev_err() into a
dev_WARN_ONCE() or do something to avoid this issue. Tried to replace
dev_err() with dev_err_ratelimited(), and the error message was displayed
less often (about once per second), but dmesg was still being cluttered.

Here's a log with what is happening (the boot part of the log has been
removed for brevity, I've kept the kernel splats for context, can provide
full logs, kernel config, command line, etc, to reproduce it; goes without
saying that I am willing to test the fix myself):

[   55.479519] [Firmware Bug]: Unable to handle paging request in EFI runtime service
[   55.487122] CPU: 62 PID: 9 Comm: kworker/u320:0 Tainted: G          I        6.1.0-rc4 #60
[   55.487128] Hardware name: WIWYNN Mt.Jade Server System B81.03001.0005/Mt.Jade Motherboard, BIOS 1.08.20220218 (SCP: 1.08.20220218) 2022/02/18
[   55.487131] Workqueue: efi_rts_wq efi_call_rts
[   55.487158] Call trace:
[   55.487161]  dump_backtrace.part.0+0xdc/0xf0
[   55.487177]  show_stack+0x18/0x40
[   55.487180]  dump_stack_lvl+0x68/0x84
[   55.487190]  dump_stack+0x18/0x34
[   55.487192]  efi_runtime_fixup_exception+0x74/0x88
[   55.487199]  __do_kernel_fault+0x108/0x1b0
[   55.487204]  do_page_fault+0xd0/0x400
[   55.487207]  do_translation_fault+0xac/0xc0
[   55.487209]  do_mem_abort+0x44/0x94
[   55.487212]  el1_abort+0x40/0x6c
[   55.487214]  el1h_64_sync_handler+0xd8/0xe4
[   55.487218]  el1h_64_sync+0x64/0x68
[   55.487221]  0xb7eb7ae4
[   55.487224]  0xb7eb8668
[   55.487225]  0xb7eb6e08
[   55.487227]  0xb7eb68ec
[   55.487228]  0xb7eb3824
[   55.487230]  0xb7eb05a8
[   55.487231]  0xb7eb12a0
[   55.487232]  0xb7e43504
[   55.487234]  0xb7e43650
[   55.487235]  0xb7e482d0
[   55.487237]  0xb7e4907c
[   55.487238]  0xb7e49ff4
[   55.487239]  0xb7e40888
[   55.487241]  0xb7cb3328
[   55.487242]  0xb7cb0674
[   55.487243]  __efi_rt_asm_wrapper+0x54/0x70
[   55.487246]  efi_call_rts+0x28c/0x3d0
[   55.487249]  process_one_work+0x1d0/0x320
[   55.487258]  worker_thread+0x14c/0x444
[   55.487261]  kthread+0x10c/0x110
[   55.487264]  ret_from_fork+0x10/0x20
[   55.487268] [Firmware Bug]: Synchronous exception occurred in EFI runtime service set_time()
[   55.495735] ------------[ cut here ]------------
[   55.495739] WARNING: CPU: 62 PID: 9 at drivers/firmware/efi/runtime-wrappers.c:111 efi_call_virt_check_flags+0x40/0xac
[   55.495746] Modules linked in:
[   55.495749] CPU: 62 PID: 9 Comm: kworker/u320:0 Tainted: G          I        6.1.0-rc4 #60
[   55.495751] Hardware name: WIWYNN Mt.Jade Server System B81.03001.0005/Mt.Jade Motherboard, BIOS 1.08.20220218 (SCP: 1.08.20220218) 2022/02/18
[   55.495753] Workqueue: efi_rts_wq efi_call_rts
[   55.495757] pstate: 004000c9 (nzcv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   55.495761] pc : efi_call_virt_check_flags+0x40/0xac
[   55.495764] lr : efi_call_rts+0x29c/0x3d0
[   55.495767] sp : ffff80000861bd40
[   55.495768] x29: ffff80000861bd40 x28: 0000000000000000 x27: 0000000000000000
[   55.495772] x26: ffffb251470e9e68 x25: ffff3fff89714805 x24: 0000000000000000
[   55.495775] x23: 0000000000000000 x22: 0000000000000000 x21: 00000000000000c0
[   55.495778] x20: ffffb25146688de0 x19: 0000000000000000 x18: ffffffffffffffff
[   55.495780] x17: 657320656d69746e x16: 757220494645206e x15: 6920646572727563
[   55.495784] x14: 636f206e6f697470 x13: ffff403e40540000 x12: 0000000000001c14
[   55.495787] x11: 000000000000095c x10: ffff403e40800000 x9 : ffff403e40540000
[   55.495790] x8 : 00000000ffff7fff x7 : ffff403e40800000 x6 : 0000000000000000
[   55.495792] x5 : ffff083e7fe9aaa0 x4 : 0000000000000000 x3 : 0000000000000000
[   55.495796] x2 : 0000000000000000 x1 : ffffb25146688de0 x0 : 00000000000000c0
[   55.495799] Call trace:
[   55.495800]  efi_call_virt_check_flags+0x40/0xac
[   55.495802]  efi_call_rts+0x29c/0x3d0
[   55.495805]  process_one_work+0x1d0/0x320
[   55.495808]  worker_thread+0x14c/0x444
[   55.495811]  kthread+0x10c/0x110
[   55.495814]  ret_from_fork+0x10/0x20
[   55.495815] ---[ end trace 0000000000000000 ]---
[   55.495818] Disabling lock debugging due to kernel taint
[   55.495822] efi: [Firmware Bug]: IRQ flags corrupted (0x00000000=>0x000000c0) by EFI set_time
[   55.504434] efi: EFI Runtime Services are disabled!
[   55.504465] rtc-efi rtc-efi.0: can't read time
[   56.479370] efi: EFI Runtime Services are disabled!
[   56.479394] rtc-efi rtc-efi.0: can't read time
[   56.483855] rtc-efi rtc-efi.0: can't read time
[   56.488306] rtc-efi rtc-efi.0: can't read time
[   57.479574] rtc-efi rtc-efi.0: can't read time
[   57.484030] rtc-efi rtc-efi.0: can't read time
[   57.488474] rtc-efi rtc-efi.0: can't read time
[   58.479692] rtc-efi rtc-efi.0: can't read time
[   58.484139] rtc-efi rtc-efi.0: can't read time
[   58.488582] rtc-efi rtc-efi.0: can't read time
[   59.479691] rtc-efi rtc-efi.0: can't read time
... on, and on, on ...

[1] https://lore.kernel.org/linux-arm-kernel/Y2lAB508TrrjpDPi@monolith.localdoman/

Thanks,
Alex

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

             reply	other threads:[~2022-11-08 10:55 UTC|newest]

Thread overview: 16+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-11-08 10:55 Alexandru Elisei [this message]
2022-11-08 10:55 ` [BUG] rtc-efi: Error in efi.get_time() spams dmesg with error message Alexandru Elisei
2022-11-08 21:41 ` Alexandre Belloni
2022-11-08 21:41   ` Alexandre Belloni
2022-11-09 14:51   ` Alexandru Elisei
2022-11-09 14:51     ` Alexandru Elisei
2022-11-09 20:31     ` Alexandre Belloni
2022-11-09 20:31       ` Alexandre Belloni
2022-11-10 16:24       ` Alexandru Elisei
2022-11-10 16:24         ` Alexandru Elisei
2023-02-17 14:14         ` Ard Biesheuvel
2023-02-17 14:14           ` Ard Biesheuvel
2022-11-09 15:50 ` [BUG] rtc-efi: Error in efi.get_time() spams dmesg with error message #forregzbot Thorsten Leemhuis
2022-11-09 15:50   ` Thorsten Leemhuis
2022-11-13 17:43   ` Thorsten Leemhuis
2022-11-13 17:43     ` Thorsten Leemhuis

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=Y2o1hdZK9GGDVJsS@monolith.localdoman \
    --to=alexandru.elisei@arm.com \
    --cc=a.zummo@towertech.it \
    --cc=alexandre.belloni@bootlin.com \
    --cc=ardb@kernel.org \
    --cc=catalin.marinas@arm.com \
    --cc=linux-arm-kernel@lists.infradead.org \
    --cc=linux-efi@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-rtc@vger.kernel.org \
    --cc=will@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.