* ❌ FAIL: Test report for kernel 5.13.0-rc7 (arm-next, 8ab9b1a9)
@ 2021-06-24 19:46 CKI Project
2021-06-25 8:39 ` Will Deacon
0 siblings, 1 reply; 39+ messages in thread
From: CKI Project @ 2021-06-24 19:46 UTC (permalink / raw)
To: skt-results-master, will, catalin.marinas, linux-arm-kernel
Cc: Memory Management, Jan Stancek, Fendy Tjahjadi, Jeff Bastian
Hello,
We ran automated tests on a recent commit from this kernel tree:
Kernel repo: https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git
Commit: 8ab9b1a98a22 - Merge branch 'for-next/core' into for-kernelci
The results of these automated tests are provided below.
Overall result: FAILED (see details below)
Merge: OK
Compile: OK
Tests: FAILED
All kernel binaries, config files, and logs are available for download here:
https://arr-cki-prod-datawarehouse-public.s3.amazonaws.com/index.html?prefix=datawarehouse-public/2021/06/24/326623253
One or more kernel tests failed:
aarch64:
❌ LTP
We hope that these logs can help you find the problem quickly. For the full
detail on our testing procedures, please scroll to the bottom of this message.
Please reply to this email if you have any questions about the tests that we
ran or if you have any suggestions on how to make future tests more effective.
,-. ,-.
( C ) ( K ) Continuous
`-',-.`-' Kernel
( I ) Integration
`-'
______________________________________________________________________________
Compile testing
---------------
We compiled the kernel for 1 architecture:
aarch64:
make options: make -j24 INSTALL_MOD_STRIP=1 targz-pkg
Hardware testing
----------------
We booted each kernel and ran the following tests:
aarch64:
Host 1:
✅ Boot test
✅ ACPI table test
✅ ACPI enabled test
❌ LTP
✅ CIFS Connectathon
✅ POSIX pjd-fstest suites
✅ Loopdev Sanity
✅ jvm - jcstress tests
✅ Memory: fork_mem
✅ Memory function: memfd_create
✅ AMTU (Abstract Machine Test Utility)
✅ Networking bridge: sanity
✅ Ethernet drivers sanity
✅ Networking socket: fuzz
✅ Networking: igmp conformance test
✅ Networking route: pmtu
✅ Networking route_func - local
✅ Networking route_func - forward
✅ Networking TCP: keepalive test
✅ Networking UDP: socket
✅ Networking cki netfilter test
✅ Networking tunnel: geneve basic test
✅ Networking tunnel: gre basic
✅ L2TP basic test
✅ Networking tunnel: vxlan basic
✅ Networking ipsec: basic netns - transport
✅ Networking ipsec: basic netns - tunnel
✅ Libkcapi AF_ALG test
✅ pciutils: update pci ids test
✅ ALSA PCM loopback test
✅ ALSA Control (mixer) Userspace Element test
✅ storage: SCSI VPD
✅ trace: ftrace/tracer
🚧 ✅ xarray-idr-radixtree-test
🚧 ✅ i2c: i2cdetect sanity
🚧 ✅ Firmware test suite
🚧 ✅ Memory function: kaslr
🚧 ✅ audit: audit testsuite test
Host 2:
✅ Boot test
✅ xfstests - ext4
✅ xfstests - xfs
✅ selinux-policy: serge-testsuite
✅ storage: software RAID testing
✅ Storage: swraid mdadm raid_module test
🚧 ❌ Podman system integration test - as root
🚧 ❌ Podman system integration test - as user
🚧 ✅ xfstests - btrfs
🚧 ✅ IPMI driver test
🚧 ✅ IPMItool loop stress test
🚧 ✅ Storage blktests
🚧 ✅ Storage block - filesystem fio test
🚧 ✅ Storage block - queue scheduler test
🚧 ✅ Storage nvme - tcp
🚧 ✅ Storage: lvm device-mapper test
🚧 ❌ stress: stress-ng
Test sources: https://gitlab.com/cki-project/kernel-tests
💚 Pull requests are welcome for new tests or improvements to existing tests!
Aborted tests
-------------
Tests that didn't complete running successfully are marked with ⚡⚡⚡.
If this was caused by an infrastructure issue, we try to mark that
explicitly in the report.
Waived tests
------------
If the test run included waived tests, they are marked with 🚧. Such tests are
executed but their results are not taken into account. Tests are waived when
their results are not reliable enough, e.g. when they're just introduced or are
being fixed.
Testing timeout
---------------
We aim to provide a report within reasonable timeframe. Tests that haven't
finished running yet are marked with ⏱.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: ❌ FAIL: Test report for kernel 5.13.0-rc7 (arm-next, 8ab9b1a9)
2021-06-24 19:46 ❌ FAIL: Test report for kernel 5.13.0-rc7 (arm-next, 8ab9b1a9) CKI Project
@ 2021-06-25 8:39 ` Will Deacon
[not found] ` <CA+tGwn=rP_hAMLLtoy_s90ZzBjfMggu7T2Qj8HyFfGh1BGUoRA@mail.gmail.com>
0 siblings, 1 reply; 39+ messages in thread
From: Will Deacon @ 2021-06-25 8:39 UTC (permalink / raw)
To: CKI Project
Cc: skt-results-master, catalin.marinas, linux-arm-kernel,
Memory Management, Jan Stancek, Fendy Tjahjadi, Jeff Bastian,
robin.murphy, mark.rutland
On Thu, Jun 24, 2021 at 07:46:08PM -0000, CKI Project wrote:
> We ran automated tests on a recent commit from this kernel tree:
>
> Kernel repo: https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git
> Commit: 8ab9b1a98a22 - Merge branch 'for-next/core' into for-kernelci
>
> The results of these automated tests are provided below.
>
> Overall result: FAILED (see details below)
> Merge: OK
> Compile: OK
> Tests: FAILED
>
> All kernel binaries, config files, and logs are available for download here:
>
> https://arr-cki-prod-datawarehouse-public.s3.amazonaws.com/index.html?prefix=datawarehouse-public/2021/06/24/326623253
>
> One or more kernel tests failed:
>
> aarch64:
> ❌ LTP
I think this is fixed by:
https://lore.kernel.org/lkml/YNQvB82UKfDV57eE@gmail.com/
> 🚧 ❌ Podman system integration test - as root
> 🚧 ❌ Podman system integration test - as user
These don't seem kernel related? (fails to retrieve an image afaict)
> 🚧 ✅ xfstests - btrfs
> 🚧 ✅ IPMI driver test
> 🚧 ✅ IPMItool loop stress test
> 🚧 ✅ Storage blktests
> 🚧 ✅ Storage block - filesystem fio test
> 🚧 ✅ Storage block - queue scheduler test
> 🚧 ✅ Storage nvme - tcp
> 🚧 ✅ Storage: lvm device-mapper test
> 🚧 ❌ stress: stress-ng
Oh no, this looks like another alignment fault in memcpy:
[13330.651903] Unable to handle kernel paging request at virtual address ffff8000534705ff
[13330.651914] Mem abort info:
[13330.651918] ESR = 0x96000021
[13330.651922] EC = 0x25: DABT (current EL), IL = 32 bits
[13330.651928] SET = 0, FnV = 0
[13330.651931] EA = 0, S1PTW = 0
[13330.651933] FSC = 0x21: alignment fault
[13330.651938] Data abort info:
[13330.651940] ISV = 0, ISS = 0x00000021
[13330.651941] CM = 0, WnR = 0
[13330.651943] swapper pgtable: 4k pages, 48-bit VAs, pgdp=00000000f3e6b000
[13330.651945] [ffff8000534705ff] pgd=1000008ffcfff003, p4d=1000008ffcfff003, pud=100000088e57d003, pmd=10000008d0aeb003, pte=006800008021370f
[13330.651956] Internal error: Oops: 96000021 [#1] SMP
[13330.651961] Modules linked in: unix_diag binfmt_misc fcrypt sm4_generic crc32_generic md4 michael_mic nhpoly1305_neon nhpoly1305 poly1305_generic libpoly1305 poly1305_neon rmd160 sha3_generic sm3_generic streebog_generic wp512 blowfish_generic blowfish_common cast5_generic des_generic libdes chacha_generic chacha_neon libchacha camellia_generic cast6_generic cast_common serpent_generic twofish_generic twofish_common dm_thin_pool dm_persistent_data dm_bio_prison nvme nvme_core ipmi_watchdog ipmi_poweroff loop tun af_key crypto_user scsi_transport_iscsi xt_multiport ip_gre ip_tunnel gre overlay xt_CONNSECMARK xt_SECMARK nft_counter xt_state xt_conntrack nft_compat ah6 ah4 nft_objref nft_ct nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink jfs sctp ip6_udp_tunnel udp_tunnel dm_log_writes dm_flakey rfkill mlx5_ib ib_uverbs ib_core sunrpc coresight_etm4x i2c_smbus coresight_replicator coresight_tpiu coresight_tmc joydev mlx5_core acpi_ipmi psample ipmi_ssif mlxfw ipmi_devintf
[13330.652076] ipmi_msghandler coresight_funnel thunderx2_pmu coresight vfat fat fuse zram ip_tables xfs ast crct10dif_ce i2c_algo_bit ghash_ce drm_vram_helper drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops cec drm_ttm_helper ttm drm gpio_xlp i2c_xlp9xx uas usb_storage aes_neon_bs [last unloaded: nvmet]
[13330.652123] CPU: 115 PID: 188446 Comm: stress-ng Tainted: G OEL 5.13.0-rc7 #1
[13330.652129] Hardware name: HPE Apollo 70 /C01_APACHE_MB , BIOS L50_5.13_1.15 05/08/2020
[13330.652133] pstate: 80400009 (Nzcv daif +PAN -UAO -TCO BTYPE=--)
[13330.652139] pc : __memcpy+0x168/0x250
[13330.652150] lr : memory_read_from_buffer+0x58/0x80
[13330.652161] sp : ffff800063ef3c20
[13330.652163] x29: ffff800063ef3c20 x28: ffff0008b1380000 x27: 0000000000000000
[13330.652170] x26: 0000000000000000 x25: 0000000000000000 x24: ffff00080a960fe0
[13330.652176] x23: ffff800063ef3d28 x22: 000000000000063f x21: ffff800063ef3c88
[13330.652181] x20: 000000000000063f x19: 000000000000063f x18: 0000000000000000
[13330.652186] x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
[13330.652191] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
[13330.652196] x11: 0000000000000000 x10: 0000000000000000 x9 : 0000000000000000
[13330.652200] x8 : 0000000000000000 x7 : 0000000000000000 x6 : 0000000000000000
[13330.652206] x5 : ffff000d0fb0063f x4 : ffff80005347063f x3 : ffff000d0fb005c0
[13330.652212] x2 : ffffffffffffffef x1 : ffff800053470600 x0 : ffff000d0fb00000
[13330.652218] Call trace:
[13330.652221] __memcpy+0x168/0x250
[13330.652225] acpi_data_show+0x5c/0x8c
[13330.652232] sysfs_kf_bin_read+0x78/0xa0
[13330.652238] kernfs_file_read_iter+0x9c/0x1a4
[13330.652241] kernfs_fop_read_iter+0x34/0x50
[13330.652244] new_sync_read+0xdc/0x154
[13330.652253] vfs_read+0x158/0x1e4
[13330.652260] ksys_read+0x64/0xec
[13330.652266] __arm64_sys_read+0x28/0x34
[13330.652273] invoke_syscall+0x50/0x120
[13330.652280] el0_svc_common.constprop.0+0x4c/0xd4
[13330.652284] do_el0_svc+0x30/0x9c
[13330.652286] el0_svc+0x2c/0x54
[13330.652294] el0t_64_sync_handler+0x1a4/0x1b0
[13330.652296] el0t_64_sync+0x19c/0x1a0
[13330.652303] Code: a984346c a9c4342c f1010042 54fffee8 (a97c3c8e)
[13330.652307] ---[ end trace 227d4380f57145d4 ]---
So maybe this issue isn't limited to weird modules, after all...
Will
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: ❌ FAIL: Test report for kernel 5.13.0-rc7 (arm-next, 8ab9b1a9)
[not found] ` <CA+tGwn=rP_hAMLLtoy_s90ZzBjfMggu7T2Qj8HyFfGh1BGUoRA@mail.gmail.com>
@ 2021-06-25 11:02 ` Robin Murphy
2021-06-25 11:09 ` Catalin Marinas
0 siblings, 1 reply; 39+ messages in thread
From: Robin Murphy @ 2021-06-25 11:02 UTC (permalink / raw)
To: Veronika Kabatova, Will Deacon
Cc: CKI Project, Mark Rutland, Catalin Marinas, Memory Management,
skt-results-master, Jeff Bastian, Jan Stancek, Linux ARM
On 2021-06-25 10:52, Veronika Kabatova wrote:
[...]
>>> ❌ stress: stress-ng
>>
>> Oh no, this looks like another alignment fault in memcpy:
>>
>> [13330.651903] Unable to handle kernel paging request at virtual address ffff8000534705ff
>> [13330.651914] Mem abort info:
>> [13330.651918] ESR = 0x96000021
>> [13330.651922] EC = 0x25: DABT (current EL), IL = 32 bits
>> [13330.651928] SET = 0, FnV = 0
>> [13330.651931] EA = 0, S1PTW = 0
>> [13330.651933] FSC = 0x21: alignment fault
>> [13330.651938] Data abort info:
>> [13330.651940] ISV = 0, ISS = 0x00000021
>> [13330.651941] CM = 0, WnR = 0
>> [13330.651943] swapper pgtable: 4k pages, 48-bit VAs, pgdp=00000000f3e6b000
>> [13330.651945] [ffff8000534705ff] pgd=1000008ffcfff003, p4d=1000008ffcfff003, pud=100000088e57d003, pmd=10000008d0aeb003, pte=006800008021370f
>> [13330.651956] Internal error: Oops: 96000021 [#1] SMP
>> [13330.651961] Modules linked in: unix_diag binfmt_misc fcrypt sm4_generic crc32_generic md4 michael_mic nhpoly1305_neon nhpoly1305 poly1305_generic libpoly1305 poly1305_neon rmd160 sha3_generic sm3_generic streebog_generic wp512 blowfish_generic blowfish_common cast5_generic des_generic libdes chacha_generic chacha_neon libchacha camellia_generic cast6_generic cast_common serpent_generic twofish_generic twofish_common dm_thin_pool dm_persistent_data dm_bio_prison nvme nvme_core ipmi_watchdog ipmi_poweroff loop tun af_key crypto_user scsi_transport_iscsi xt_multiport ip_gre ip_tunnel gre overlay xt_CONNSECMARK xt_SECMARK nft_counter xt_state xt_conntrack nft_compat ah6 ah4 nft_objref nft_ct nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink jfs sctp ip6_udp_tunnel udp_tunnel dm_log_writes dm_flakey rfkill mlx5_ib ib_uverbs ib_core sunrpc coresight_etm4x i2c_smbus coresight_replicator coresight_tpiu coresight_tmc joydev mlx5_core acpi_ipmi psample ipmi_ssif mlxfw !
>> ipmi_devintf
>> [13330.652076] ipmi_msghandler coresight_funnel thunderx2_pmu coresight vfat fat fuse zram ip_tables xfs ast crct10dif_ce i2c_algo_bit ghash_ce drm_vram_helper drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops cec drm_ttm_helper ttm drm gpio_xlp i2c_xlp9xx uas usb_storage aes_neon_bs [last unloaded: nvmet]
>> [13330.652123] CPU: 115 PID: 188446 Comm: stress-ng Tainted: G OEL 5.13.0-rc7 #1
>> [13330.652129] Hardware name: HPE Apollo 70 /C01_APACHE_MB , BIOS L50_5.13_1.15 05/08/2020
>> [13330.652133] pstate: 80400009 (Nzcv daif +PAN -UAO -TCO BTYPE=--)
>> [13330.652139] pc : __memcpy+0x168/0x250
>> [13330.652150] lr : memory_read_from_buffer+0x58/0x80
>> [13330.652161] sp : ffff800063ef3c20
>> [13330.652163] x29: ffff800063ef3c20 x28: ffff0008b1380000 x27: 0000000000000000
>> [13330.652170] x26: 0000000000000000 x25: 0000000000000000 x24: ffff00080a960fe0
>> [13330.652176] x23: ffff800063ef3d28 x22: 000000000000063f x21: ffff800063ef3c88
>> [13330.652181] x20: 000000000000063f x19: 000000000000063f x18: 0000000000000000
>> [13330.652186] x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
>> [13330.652191] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
>> [13330.652196] x11: 0000000000000000 x10: 0000000000000000 x9 : 0000000000000000
>> [13330.652200] x8 : 0000000000000000 x7 : 0000000000000000 x6 : 0000000000000000
>> [13330.652206] x5 : ffff000d0fb0063f x4 : ffff80005347063f x3 : ffff000d0fb005c0
>> [13330.652212] x2 : ffffffffffffffef x1 : ffff800053470600 x0 : ffff000d0fb00000
>> [13330.652218] Call trace:
>> [13330.652221] __memcpy+0x168/0x250
>> [13330.652225] acpi_data_show+0x5c/0x8c
>> [13330.652232] sysfs_kf_bin_read+0x78/0xa0
>> [13330.652238] kernfs_file_read_iter+0x9c/0x1a4
>> [13330.652241] kernfs_fop_read_iter+0x34/0x50
>> [13330.652244] new_sync_read+0xdc/0x154
>> [13330.652253] vfs_read+0x158/0x1e4
>> [13330.652260] ksys_read+0x64/0xec
>> [13330.652266] __arm64_sys_read+0x28/0x34
>> [13330.652273] invoke_syscall+0x50/0x120
>> [13330.652280] el0_svc_common.constprop.0+0x4c/0xd4
>> [13330.652284] do_el0_svc+0x30/0x9c
>> [13330.652286] el0_svc+0x2c/0x54
>> [13330.652294] el0t_64_sync_handler+0x1a4/0x1b0
>> [13330.652296] el0t_64_sync+0x19c/0x1a0
>> [13330.652303] Code: a984346c a9c4342c f1010042 54fffee8 (a97c3c8e)
>> [13330.652307] ---[ end trace 227d4380f57145d4 ]---
>>
>> So maybe this issue isn't limited to weird modules, after all...
>>
>
> It ran on the machine from the same set that we were able to reproduce
> it on previously. If you or anyone else have an idea on how to stabilize
> the reproducibility or have a debug patch we'll be happy to try it.
Possibly it depends on the individual machines' firmware exactly how the
relevant bits of their ACPI tables are aligned in memory?
I've started digging into that callstack - it may not be a "weird
module" but it's definitely crusty ACPI code... a238317ce818 ("ACPI:
Clean up acpi_os_map/unmap_memory() to eliminate __iomem.") looks
frankly a bit questionable in its decision to blithely cast away
__iomem, but then the rationale in aafc65c731fe ("ACPI: add arm64 to the
platforms that use ioremap") seems particularly dubious on top of that
(especially given this end result).
At a wild guess, I'm wondering if this may be sufficient:
----->8-----
diff --git a/drivers/acpi/osl.c b/drivers/acpi/osl.c
index 327e1b4eb6b0..f5d26b102fbe 100644
--- a/drivers/acpi/osl.c
+++ b/drivers/acpi/osl.c
@@ -277,7 +277,7 @@ acpi_map_lookup_virt(void __iomem *virt, acpi_size size)
return NULL;
}
-#if defined(CONFIG_IA64) || defined(CONFIG_ARM64)
+#if defined(CONFIG_IA64)
/* ioremap will take care of cache attributes */
#define should_use_kmap(pfn) 0
#else
-----8<-----
otherwise we might need to refactor acpi_os_map_memory() to properly use
memremap() instead of bodging the ioremap() path.
My actual ACPI-capable dev board has a trashed filesystem at the moment,
but I guess I can try an ACPI VM with QEMU to poke further.
Robin.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 39+ messages in thread
* Re: ❌ FAIL: Test report for kernel 5.13.0-rc7 (arm-next, 8ab9b1a9)
2021-06-25 11:02 ` Robin Murphy
@ 2021-06-25 11:09 ` Catalin Marinas
2021-06-25 11:15 ` Robin Murphy
0 siblings, 1 reply; 39+ messages in thread
From: Catalin Marinas @ 2021-06-25 11:09 UTC (permalink / raw)
To: Robin Murphy
Cc: Veronika Kabatova, Will Deacon, CKI Project, Mark Rutland,
Memory Management, skt-results-master, Jeff Bastian, Jan Stancek,
Linux ARM
On Fri, Jun 25, 2021 at 12:02:52PM +0100, Robin Murphy wrote:
> On 2021-06-25 10:52, Veronika Kabatova wrote:
> [...]
> > > > ❌ stress: stress-ng
> > >
> > > Oh no, this looks like another alignment fault in memcpy:
> > >
> > > [13330.651903] Unable to handle kernel paging request at virtual address ffff8000534705ff
> > > [13330.651914] Mem abort info:
> > > [13330.651918] ESR = 0x96000021
> > > [13330.651922] EC = 0x25: DABT (current EL), IL = 32 bits
> > > [13330.651928] SET = 0, FnV = 0
> > > [13330.651931] EA = 0, S1PTW = 0
> > > [13330.651933] FSC = 0x21: alignment fault
> > > [13330.651938] Data abort info:
> > > [13330.651940] ISV = 0, ISS = 0x00000021
> > > [13330.651941] CM = 0, WnR = 0
> > > [13330.651943] swapper pgtable: 4k pages, 48-bit VAs, pgdp=00000000f3e6b000
> > > [13330.651945] [ffff8000534705ff] pgd=1000008ffcfff003, p4d=1000008ffcfff003, pud=100000088e57d003, pmd=10000008d0aeb003, pte=006800008021370f
> > > [13330.651956] Internal error: Oops: 96000021 [#1] SMP
> > > [13330.651961] Modules linked in: unix_diag binfmt_misc fcrypt sm4_generic crc32_generic md4 michael_mic nhpoly1305_neon nhpoly1305 poly1305_generic libpoly1305 poly1305_neon rmd160 sha3_generic sm3_generic streebog_generic wp512 blowfish_generic blowfish_common cast5_generic des_generic libdes chacha_generic chacha_neon libchacha camellia_generic cast6_generic cast_common serpent_generic twofish_generic twofish_common dm_thin_pool dm_persistent_data dm_bio_prison nvme nvme_core ipmi_watchdog ipmi_poweroff loop tun af_key crypto_user scsi_transport_iscsi xt_multiport ip_gre ip_tunnel gre overlay xt_CONNSECMARK xt_SECMARK nft_counter xt_state xt_conntrack nft_compat ah6 ah4 nft_objref nft_ct nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink jfs sctp ip6_udp_tunnel udp_tunnel dm_log_writes dm_flakey rfkill mlx5_ib ib_uverbs ib_core sunrpc coresight_etm4x i2c_smbus coresight_replicator coresight_tpiu coresight_tmc joydev mlx5_core acpi_ipmi psample ipmi_ssif mlxfw !
> > > ipmi_devintf
> > > [13330.652076] ipmi_msghandler coresight_funnel thunderx2_pmu coresight vfat fat fuse zram ip_tables xfs ast crct10dif_ce i2c_algo_bit ghash_ce drm_vram_helper drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops cec drm_ttm_helper ttm drm gpio_xlp i2c_xlp9xx uas usb_storage aes_neon_bs [last unloaded: nvmet]
> > > [13330.652123] CPU: 115 PID: 188446 Comm: stress-ng Tainted: G OEL 5.13.0-rc7 #1
> > > [13330.652129] Hardware name: HPE Apollo 70 /C01_APACHE_MB , BIOS L50_5.13_1.15 05/08/2020
> > > [13330.652133] pstate: 80400009 (Nzcv daif +PAN -UAO -TCO BTYPE=--)
> > > [13330.652139] pc : __memcpy+0x168/0x250
> > > [13330.652150] lr : memory_read_from_buffer+0x58/0x80
> > > [13330.652161] sp : ffff800063ef3c20
> > > [13330.652163] x29: ffff800063ef3c20 x28: ffff0008b1380000 x27: 0000000000000000
> > > [13330.652170] x26: 0000000000000000 x25: 0000000000000000 x24: ffff00080a960fe0
> > > [13330.652176] x23: ffff800063ef3d28 x22: 000000000000063f x21: ffff800063ef3c88
> > > [13330.652181] x20: 000000000000063f x19: 000000000000063f x18: 0000000000000000
> > > [13330.652186] x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
> > > [13330.652191] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
> > > [13330.652196] x11: 0000000000000000 x10: 0000000000000000 x9 : 0000000000000000
> > > [13330.652200] x8 : 0000000000000000 x7 : 0000000000000000 x6 : 0000000000000000
> > > [13330.652206] x5 : ffff000d0fb0063f x4 : ffff80005347063f x3 : ffff000d0fb005c0
> > > [13330.652212] x2 : ffffffffffffffef x1 : ffff800053470600 x0 : ffff000d0fb00000
> > > [13330.652218] Call trace:
> > > [13330.652221] __memcpy+0x168/0x250
> > > [13330.652225] acpi_data_show+0x5c/0x8c
> > > [13330.652232] sysfs_kf_bin_read+0x78/0xa0
> > > [13330.652238] kernfs_file_read_iter+0x9c/0x1a4
> > > [13330.652241] kernfs_fop_read_iter+0x34/0x50
> > > [13330.652244] new_sync_read+0xdc/0x154
> > > [13330.652253] vfs_read+0x158/0x1e4
> > > [13330.652260] ksys_read+0x64/0xec
> > > [13330.652266] __arm64_sys_read+0x28/0x34
> > > [13330.652273] invoke_syscall+0x50/0x120
> > > [13330.652280] el0_svc_common.constprop.0+0x4c/0xd4
> > > [13330.652284] do_el0_svc+0x30/0x9c
> > > [13330.652286] el0_svc+0x2c/0x54
> > > [13330.652294] el0t_64_sync_handler+0x1a4/0x1b0
> > > [13330.652296] el0t_64_sync+0x19c/0x1a0
> > > [13330.652303] Code: a984346c a9c4342c f1010042 54fffee8 (a97c3c8e)
> > > [13330.652307] ---[ end trace 227d4380f57145d4 ]---
> > >
> > > So maybe this issue isn't limited to weird modules, after all...
> > >
> >
> > It ran on the machine from the same set that we were able to reproduce
> > it on previously. If you or anyone else have an idea on how to stabilize
> > the reproducibility or have a debug patch we'll be happy to try it.
>
> Possibly it depends on the individual machines' firmware exactly how the
> relevant bits of their ACPI tables are aligned in memory?
>
> I've started digging into that callstack - it may not be a "weird module"
> but it's definitely crusty ACPI code... a238317ce818 ("ACPI: Clean up
> acpi_os_map/unmap_memory() to eliminate __iomem.") looks frankly a bit
> questionable in its decision to blithely cast away __iomem, but then the
> rationale in aafc65c731fe ("ACPI: add arm64 to the platforms that use
> ioremap") seems particularly dubious on top of that (especially given this
> end result).
>
> At a wild guess, I'm wondering if this may be sufficient:
>
> ----->8-----
> diff --git a/drivers/acpi/osl.c b/drivers/acpi/osl.c
> index 327e1b4eb6b0..f5d26b102fbe 100644
> --- a/drivers/acpi/osl.c
> +++ b/drivers/acpi/osl.c
> @@ -277,7 +277,7 @@ acpi_map_lookup_virt(void __iomem *virt, acpi_size size)
> return NULL;
> }
>
> -#if defined(CONFIG_IA64) || defined(CONFIG_ARM64)
> +#if defined(CONFIG_IA64)
> /* ioremap will take care of cache attributes */
> #define should_use_kmap(pfn) 0
> #else
> -----8<-----
I thought the same but shouldn't acpi_os_ioremap() map it with the right
attributes? It uses the EFI maps to check what kind of memory this is.
--
Catalin
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: ❌ FAIL: Test report for kernel 5.13.0-rc7 (arm-next, 8ab9b1a9)
2021-06-25 11:09 ` Catalin Marinas
@ 2021-06-25 11:15 ` Robin Murphy
2021-06-29 11:48 ` Robin Murphy
0 siblings, 1 reply; 39+ messages in thread
From: Robin Murphy @ 2021-06-25 11:15 UTC (permalink / raw)
To: Catalin Marinas
Cc: Veronika Kabatova, Will Deacon, CKI Project, Mark Rutland,
Memory Management, skt-results-master, Jeff Bastian, Jan Stancek,
Linux ARM
On 2021-06-25 12:09, Catalin Marinas wrote:
> On Fri, Jun 25, 2021 at 12:02:52PM +0100, Robin Murphy wrote:
>> On 2021-06-25 10:52, Veronika Kabatova wrote:
>> [...]
>>>>> ❌ stress: stress-ng
>>>>
>>>> Oh no, this looks like another alignment fault in memcpy:
>>>>
>>>> [13330.651903] Unable to handle kernel paging request at virtual address ffff8000534705ff
>>>> [13330.651914] Mem abort info:
>>>> [13330.651918] ESR = 0x96000021
>>>> [13330.651922] EC = 0x25: DABT (current EL), IL = 32 bits
>>>> [13330.651928] SET = 0, FnV = 0
>>>> [13330.651931] EA = 0, S1PTW = 0
>>>> [13330.651933] FSC = 0x21: alignment fault
>>>> [13330.651938] Data abort info:
>>>> [13330.651940] ISV = 0, ISS = 0x00000021
>>>> [13330.651941] CM = 0, WnR = 0
>>>> [13330.651943] swapper pgtable: 4k pages, 48-bit VAs, pgdp=00000000f3e6b000
>>>> [13330.651945] [ffff8000534705ff] pgd=1000008ffcfff003, p4d=1000008ffcfff003, pud=100000088e57d003, pmd=10000008d0aeb003, pte=006800008021370f
>>>> [13330.651956] Internal error: Oops: 96000021 [#1] SMP
>>>> [13330.651961] Modules linked in: unix_diag binfmt_misc fcrypt sm4_generic crc32_generic md4 michael_mic nhpoly1305_neon nhpoly1305 poly1305_generic libpoly1305 poly1305_neon rmd160 sha3_generic sm3_generic streebog_generic wp512 blowfish_generic blowfish_common cast5_generic des_generic libdes chacha_generic chacha_neon libchacha camellia_generic cast6_generic cast_common serpent_generic twofish_generic twofish_common dm_thin_pool dm_persistent_data dm_bio_prison nvme nvme_core ipmi_watchdog ipmi_poweroff loop tun af_key crypto_user scsi_transport_iscsi xt_multiport ip_gre ip_tunnel gre overlay xt_CONNSECMARK xt_SECMARK nft_counter xt_state xt_conntrack nft_compat ah6 ah4 nft_objref nft_ct nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink jfs sctp ip6_udp_tunnel udp_tunnel dm_log_writes dm_flakey rfkill mlx5_ib ib_uverbs ib_core sunrpc coresight_etm4x i2c_smbus coresight_replicator coresight_tpiu coresight_tmc joydev mlx5_core acpi_ipmi psample ipmi_ssif mlxfw !
>>>> ipmi_devintf
>>>> [13330.652076] ipmi_msghandler coresight_funnel thunderx2_pmu coresight vfat fat fuse zram ip_tables xfs ast crct10dif_ce i2c_algo_bit ghash_ce drm_vram_helper drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops cec drm_ttm_helper ttm drm gpio_xlp i2c_xlp9xx uas usb_storage aes_neon_bs [last unloaded: nvmet]
>>>> [13330.652123] CPU: 115 PID: 188446 Comm: stress-ng Tainted: G OEL 5.13.0-rc7 #1
>>>> [13330.652129] Hardware name: HPE Apollo 70 /C01_APACHE_MB , BIOS L50_5.13_1.15 05/08/2020
>>>> [13330.652133] pstate: 80400009 (Nzcv daif +PAN -UAO -TCO BTYPE=--)
>>>> [13330.652139] pc : __memcpy+0x168/0x250
>>>> [13330.652150] lr : memory_read_from_buffer+0x58/0x80
>>>> [13330.652161] sp : ffff800063ef3c20
>>>> [13330.652163] x29: ffff800063ef3c20 x28: ffff0008b1380000 x27: 0000000000000000
>>>> [13330.652170] x26: 0000000000000000 x25: 0000000000000000 x24: ffff00080a960fe0
>>>> [13330.652176] x23: ffff800063ef3d28 x22: 000000000000063f x21: ffff800063ef3c88
>>>> [13330.652181] x20: 000000000000063f x19: 000000000000063f x18: 0000000000000000
>>>> [13330.652186] x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
>>>> [13330.652191] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
>>>> [13330.652196] x11: 0000000000000000 x10: 0000000000000000 x9 : 0000000000000000
>>>> [13330.652200] x8 : 0000000000000000 x7 : 0000000000000000 x6 : 0000000000000000
>>>> [13330.652206] x5 : ffff000d0fb0063f x4 : ffff80005347063f x3 : ffff000d0fb005c0
>>>> [13330.652212] x2 : ffffffffffffffef x1 : ffff800053470600 x0 : ffff000d0fb00000
>>>> [13330.652218] Call trace:
>>>> [13330.652221] __memcpy+0x168/0x250
>>>> [13330.652225] acpi_data_show+0x5c/0x8c
>>>> [13330.652232] sysfs_kf_bin_read+0x78/0xa0
>>>> [13330.652238] kernfs_file_read_iter+0x9c/0x1a4
>>>> [13330.652241] kernfs_fop_read_iter+0x34/0x50
>>>> [13330.652244] new_sync_read+0xdc/0x154
>>>> [13330.652253] vfs_read+0x158/0x1e4
>>>> [13330.652260] ksys_read+0x64/0xec
>>>> [13330.652266] __arm64_sys_read+0x28/0x34
>>>> [13330.652273] invoke_syscall+0x50/0x120
>>>> [13330.652280] el0_svc_common.constprop.0+0x4c/0xd4
>>>> [13330.652284] do_el0_svc+0x30/0x9c
>>>> [13330.652286] el0_svc+0x2c/0x54
>>>> [13330.652294] el0t_64_sync_handler+0x1a4/0x1b0
>>>> [13330.652296] el0t_64_sync+0x19c/0x1a0
>>>> [13330.652303] Code: a984346c a9c4342c f1010042 54fffee8 (a97c3c8e)
>>>> [13330.652307] ---[ end trace 227d4380f57145d4 ]---
>>>>
>>>> So maybe this issue isn't limited to weird modules, after all...
>>>>
>>>
>>> It ran on the machine from the same set that we were able to reproduce
>>> it on previously. If you or anyone else have an idea on how to stabilize
>>> the reproducibility or have a debug patch we'll be happy to try it.
>>
>> Possibly it depends on the individual machines' firmware exactly how the
>> relevant bits of their ACPI tables are aligned in memory?
>>
>> I've started digging into that callstack - it may not be a "weird module"
>> but it's definitely crusty ACPI code... a238317ce818 ("ACPI: Clean up
>> acpi_os_map/unmap_memory() to eliminate __iomem.") looks frankly a bit
>> questionable in its decision to blithely cast away __iomem, but then the
>> rationale in aafc65c731fe ("ACPI: add arm64 to the platforms that use
>> ioremap") seems particularly dubious on top of that (especially given this
>> end result).
>>
>> At a wild guess, I'm wondering if this may be sufficient:
>>
>> ----->8-----
>> diff --git a/drivers/acpi/osl.c b/drivers/acpi/osl.c
>> index 327e1b4eb6b0..f5d26b102fbe 100644
>> --- a/drivers/acpi/osl.c
>> +++ b/drivers/acpi/osl.c
>> @@ -277,7 +277,7 @@ acpi_map_lookup_virt(void __iomem *virt, acpi_size size)
>> return NULL;
>> }
>>
>> -#if defined(CONFIG_IA64) || defined(CONFIG_ARM64)
>> +#if defined(CONFIG_IA64)
>> /* ioremap will take care of cache attributes */
>> #define should_use_kmap(pfn) 0
>> #else
>> -----8<-----
>
> I thought the same but shouldn't acpi_os_ioremap() map it with the right
> attributes? It uses the EFI maps to check what kind of memory this is.
Oh crikey, I missed that branch of the rabbit hole... I guess that must
mean that the tables being poked here are *not* covered by the EFI
memory map, so page_is_ram() is unlikely to help either :(
Robin.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: ❌ FAIL: Test report for kernel 5.13.0-rc7 (arm-next, 8ab9b1a9)
2021-06-25 11:15 ` Robin Murphy
@ 2021-06-29 11:48 ` Robin Murphy
0 siblings, 0 replies; 39+ messages in thread
From: Robin Murphy @ 2021-06-29 11:48 UTC (permalink / raw)
To: ACPI Devel Maling List
Cc: Veronika Kabatova, Will Deacon, CKI Project, Mark Rutland,
Memory Management, skt-results-master, Jeff Bastian, Jan Stancek,
Linux ARM, rjw, lenb, guohanjun, Lorenzo Pieralisi, sudeep.holla,
ardb, Catalin Marinas, lv.zheng, tony.luck
[ +ACPI audience ]
On 2021-06-25 12:15, Robin Murphy wrote:
> On 2021-06-25 12:09, Catalin Marinas wrote:
>> On Fri, Jun 25, 2021 at 12:02:52PM +0100, Robin Murphy wrote:
>>> On 2021-06-25 10:52, Veronika Kabatova wrote:
>>> [...]
>>>>>> ❌ stress: stress-ng
>>>>>
>>>>> Oh no, this looks like another alignment fault in memcpy:
>>>>>
>>>>> [13330.651903] Unable to handle kernel paging request at virtual
>>>>> address ffff8000534705ff
>>>>> [13330.651914] Mem abort info:
>>>>> [13330.651918] ESR = 0x96000021
>>>>> [13330.651922] EC = 0x25: DABT (current EL), IL = 32 bits
>>>>> [13330.651928] SET = 0, FnV = 0
>>>>> [13330.651931] EA = 0, S1PTW = 0
>>>>> [13330.651933] FSC = 0x21: alignment fault
>>>>> [13330.651938] Data abort info:
>>>>> [13330.651940] ISV = 0, ISS = 0x00000021
>>>>> [13330.651941] CM = 0, WnR = 0
>>>>> [13330.651943] swapper pgtable: 4k pages, 48-bit VAs,
>>>>> pgdp=00000000f3e6b000
>>>>> [13330.651945] [ffff8000534705ff] pgd=1000008ffcfff003,
>>>>> p4d=1000008ffcfff003, pud=100000088e57d003, pmd=10000008d0aeb003,
>>>>> pte=006800008021370f
>>>>> [13330.651956] Internal error: Oops: 96000021 [#1] SMP
>>>>> [13330.651961] Modules linked in: unix_diag binfmt_misc fcrypt
>>>>> sm4_generic crc32_generic md4 michael_mic nhpoly1305_neon
>>>>> nhpoly1305 poly1305_generic libpoly1305 poly1305_neon rmd160
>>>>> sha3_generic sm3_generic streebog_generic wp512 blowfish_generic
>>>>> blowfish_common cast5_generic des_generic libdes chacha_generic
>>>>> chacha_neon libchacha camellia_generic cast6_generic cast_common
>>>>> serpent_generic twofish_generic twofish_common dm_thin_pool
>>>>> dm_persistent_data dm_bio_prison nvme nvme_core ipmi_watchdog
>>>>> ipmi_poweroff loop tun af_key crypto_user scsi_transport_iscsi
>>>>> xt_multiport ip_gre ip_tunnel gre overlay xt_CONNSECMARK xt_SECMARK
>>>>> nft_counter xt_state xt_conntrack nft_compat ah6 ah4 nft_objref
>>>>> nft_ct nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables
>>>>> nfnetlink jfs sctp ip6_udp_tunnel udp_tunnel dm_log_writes
>>>>> dm_flakey rfkill mlx5_ib ib_uverbs ib_core sunrpc coresight_etm4x
>>>>> i2c_smbus coresight_replicator coresight_tpiu coresight_tmc joydev
>>>>> mlx5_core acpi_ipmi psample ipmi_ssif mlxfw !
>>>>> ipmi_devintf
>>>>> [13330.652076] ipmi_msghandler coresight_funnel thunderx2_pmu
>>>>> coresight vfat fat fuse zram ip_tables xfs ast crct10dif_ce
>>>>> i2c_algo_bit ghash_ce drm_vram_helper drm_kms_helper syscopyarea
>>>>> sysfillrect sysimgblt fb_sys_fops cec drm_ttm_helper ttm drm
>>>>> gpio_xlp i2c_xlp9xx uas usb_storage aes_neon_bs [last unloaded: nvmet]
>>>>> [13330.652123] CPU: 115 PID: 188446 Comm: stress-ng Tainted:
>>>>> G OEL 5.13.0-rc7 #1
>>>>> [13330.652129] Hardware name: HPE Apollo 70
>>>>> /C01_APACHE_MB , BIOS L50_5.13_1.15 05/08/2020
>>>>> [13330.652133] pstate: 80400009 (Nzcv daif +PAN -UAO -TCO BTYPE=--)
>>>>> [13330.652139] pc : __memcpy+0x168/0x250
>>>>> [13330.652150] lr : memory_read_from_buffer+0x58/0x80
>>>>> [13330.652161] sp : ffff800063ef3c20
>>>>> [13330.652163] x29: ffff800063ef3c20 x28: ffff0008b1380000 x27:
>>>>> 0000000000000000
>>>>> [13330.652170] x26: 0000000000000000 x25: 0000000000000000 x24:
>>>>> ffff00080a960fe0
>>>>> [13330.652176] x23: ffff800063ef3d28 x22: 000000000000063f x21:
>>>>> ffff800063ef3c88
>>>>> [13330.652181] x20: 000000000000063f x19: 000000000000063f x18:
>>>>> 0000000000000000
>>>>> [13330.652186] x17: 0000000000000000 x16: 0000000000000000 x15:
>>>>> 0000000000000000
>>>>> [13330.652191] x14: 0000000000000000 x13: 0000000000000000 x12:
>>>>> 0000000000000000
>>>>> [13330.652196] x11: 0000000000000000 x10: 0000000000000000 x9 :
>>>>> 0000000000000000
>>>>> [13330.652200] x8 : 0000000000000000 x7 : 0000000000000000 x6 :
>>>>> 0000000000000000
>>>>> [13330.652206] x5 : ffff000d0fb0063f x4 : ffff80005347063f x3 :
>>>>> ffff000d0fb005c0
>>>>> [13330.652212] x2 : ffffffffffffffef x1 : ffff800053470600 x0 :
>>>>> ffff000d0fb00000
>>>>> [13330.652218] Call trace:
>>>>> [13330.652221] __memcpy+0x168/0x250
>>>>> [13330.652225] acpi_data_show+0x5c/0x8c
>>>>> [13330.652232] sysfs_kf_bin_read+0x78/0xa0
>>>>> [13330.652238] kernfs_file_read_iter+0x9c/0x1a4
>>>>> [13330.652241] kernfs_fop_read_iter+0x34/0x50
>>>>> [13330.652244] new_sync_read+0xdc/0x154
>>>>> [13330.652253] vfs_read+0x158/0x1e4
>>>>> [13330.652260] ksys_read+0x64/0xec
>>>>> [13330.652266] __arm64_sys_read+0x28/0x34
>>>>> [13330.652273] invoke_syscall+0x50/0x120
>>>>> [13330.652280] el0_svc_common.constprop.0+0x4c/0xd4
>>>>> [13330.652284] do_el0_svc+0x30/0x9c
>>>>> [13330.652286] el0_svc+0x2c/0x54
>>>>> [13330.652294] el0t_64_sync_handler+0x1a4/0x1b0
>>>>> [13330.652296] el0t_64_sync+0x19c/0x1a0
>>>>> [13330.652303] Code: a984346c a9c4342c f1010042 54fffee8 (a97c3c8e)
>>>>> [13330.652307] ---[ end trace 227d4380f57145d4 ]---
>>>>>
>>>>> So maybe this issue isn't limited to weird modules, after all...
>>>>>
>>>>
>>>> It ran on the machine from the same set that we were able to reproduce
>>>> it on previously. If you or anyone else have an idea on how to
>>>> stabilize
>>>> the reproducibility or have a debug patch we'll be happy to try it.
>>>
>>> Possibly it depends on the individual machines' firmware exactly how the
>>> relevant bits of their ACPI tables are aligned in memory?
>>>
>>> I've started digging into that callstack - it may not be a "weird
>>> module"
>>> but it's definitely crusty ACPI code... a238317ce818 ("ACPI: Clean up
>>> acpi_os_map/unmap_memory() to eliminate __iomem.") looks frankly a bit
>>> questionable in its decision to blithely cast away __iomem, but then the
>>> rationale in aafc65c731fe ("ACPI: add arm64 to the platforms that use
>>> ioremap") seems particularly dubious on top of that (especially given
>>> this
>>> end result).
>>>
>>> At a wild guess, I'm wondering if this may be sufficient:
>>>
>>> ----->8-----
>>> diff --git a/drivers/acpi/osl.c b/drivers/acpi/osl.c
>>> index 327e1b4eb6b0..f5d26b102fbe 100644
>>> --- a/drivers/acpi/osl.c
>>> +++ b/drivers/acpi/osl.c
>>> @@ -277,7 +277,7 @@ acpi_map_lookup_virt(void __iomem *virt,
>>> acpi_size size)
>>> return NULL;
>>> }
>>>
>>> -#if defined(CONFIG_IA64) || defined(CONFIG_ARM64)
>>> +#if defined(CONFIG_IA64)
>>> /* ioremap will take care of cache attributes */
>>> #define should_use_kmap(pfn) 0
>>> #else
>>> -----8<-----
>>
>> I thought the same but shouldn't acpi_os_ioremap() map it with the right
>> attributes? It uses the EFI maps to check what kind of memory this is.
>
> Oh crikey, I missed that branch of the rabbit hole... I guess that must
> mean that the tables being poked here are *not* covered by the EFI
> memory map, so page_is_ram() is unlikely to help either :(
After picking through the UEFI spec I think I've now got a clearer
picture of what's happening, but I'm not sure where it goes from here...
The spec implies that it *is* legitimate for runtime-loaded ACPI tables
to lie outside the EFI memory map, and that case they must be assumed to
be uncached, so the behaviour of acpi_os_ioremap() is correct. Given the
definition of uncached for arm64 memory types though, that means that
callers of acpi_os_map_memory() still have to be prepared to get an
__iomem pointer back even if they know they're mapping a table rather
than some random bit of MMIO for an AML method.
Therefore in this case it seems the blame lies partway between
acpi_os_map_memory() for casting away __iomem and acpi_data_show() for
letting an arbitrary offset lead to an arbitrarily-aligned memcpy(), but
I don't know what the best way to fix it is. Either way I've satisfied
myself that it's not an issue with the arm64 code itself - I do wonder
whether this might also be a problem on IA-64 given
ACPI_MISALIGNMENT_NOT_SUPPORTED, and I guess RISC-V may have alignment
concerns as well.
Robin.
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: ❌ FAIL: Test report for kernel 5.13.0-rc7 (arm-next, 8ab9b1a9)
@ 2021-06-29 11:48 ` Robin Murphy
0 siblings, 0 replies; 39+ messages in thread
From: Robin Murphy @ 2021-06-29 11:48 UTC (permalink / raw)
To: ACPI Devel Maling List
Cc: Veronika Kabatova, Will Deacon, CKI Project, Mark Rutland,
Memory Management, skt-results-master, Jeff Bastian, Jan Stancek,
Linux ARM, rjw, lenb, guohanjun, Lorenzo Pieralisi, sudeep.holla,
ardb, Catalin Marinas, lv.zheng, tony.luck
[ +ACPI audience ]
On 2021-06-25 12:15, Robin Murphy wrote:
> On 2021-06-25 12:09, Catalin Marinas wrote:
>> On Fri, Jun 25, 2021 at 12:02:52PM +0100, Robin Murphy wrote:
>>> On 2021-06-25 10:52, Veronika Kabatova wrote:
>>> [...]
>>>>>> ❌ stress: stress-ng
>>>>>
>>>>> Oh no, this looks like another alignment fault in memcpy:
>>>>>
>>>>> [13330.651903] Unable to handle kernel paging request at virtual
>>>>> address ffff8000534705ff
>>>>> [13330.651914] Mem abort info:
>>>>> [13330.651918] ESR = 0x96000021
>>>>> [13330.651922] EC = 0x25: DABT (current EL), IL = 32 bits
>>>>> [13330.651928] SET = 0, FnV = 0
>>>>> [13330.651931] EA = 0, S1PTW = 0
>>>>> [13330.651933] FSC = 0x21: alignment fault
>>>>> [13330.651938] Data abort info:
>>>>> [13330.651940] ISV = 0, ISS = 0x00000021
>>>>> [13330.651941] CM = 0, WnR = 0
>>>>> [13330.651943] swapper pgtable: 4k pages, 48-bit VAs,
>>>>> pgdp=00000000f3e6b000
>>>>> [13330.651945] [ffff8000534705ff] pgd=1000008ffcfff003,
>>>>> p4d=1000008ffcfff003, pud=100000088e57d003, pmd=10000008d0aeb003,
>>>>> pte=006800008021370f
>>>>> [13330.651956] Internal error: Oops: 96000021 [#1] SMP
>>>>> [13330.651961] Modules linked in: unix_diag binfmt_misc fcrypt
>>>>> sm4_generic crc32_generic md4 michael_mic nhpoly1305_neon
>>>>> nhpoly1305 poly1305_generic libpoly1305 poly1305_neon rmd160
>>>>> sha3_generic sm3_generic streebog_generic wp512 blowfish_generic
>>>>> blowfish_common cast5_generic des_generic libdes chacha_generic
>>>>> chacha_neon libchacha camellia_generic cast6_generic cast_common
>>>>> serpent_generic twofish_generic twofish_common dm_thin_pool
>>>>> dm_persistent_data dm_bio_prison nvme nvme_core ipmi_watchdog
>>>>> ipmi_poweroff loop tun af_key crypto_user scsi_transport_iscsi
>>>>> xt_multiport ip_gre ip_tunnel gre overlay xt_CONNSECMARK xt_SECMARK
>>>>> nft_counter xt_state xt_conntrack nft_compat ah6 ah4 nft_objref
>>>>> nft_ct nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables
>>>>> nfnetlink jfs sctp ip6_udp_tunnel udp_tunnel dm_log_writes
>>>>> dm_flakey rfkill mlx5_ib ib_uverbs ib_core sunrpc coresight_etm4x
>>>>> i2c_smbus coresight_replicator coresight_tpiu coresight_tmc joydev
>>>>> mlx5_core acpi_ipmi psample ipmi_ssif mlxfw !
>>>>> ipmi_devintf
>>>>> [13330.652076] ipmi_msghandler coresight_funnel thunderx2_pmu
>>>>> coresight vfat fat fuse zram ip_tables xfs ast crct10dif_ce
>>>>> i2c_algo_bit ghash_ce drm_vram_helper drm_kms_helper syscopyarea
>>>>> sysfillrect sysimgblt fb_sys_fops cec drm_ttm_helper ttm drm
>>>>> gpio_xlp i2c_xlp9xx uas usb_storage aes_neon_bs [last unloaded: nvmet]
>>>>> [13330.652123] CPU: 115 PID: 188446 Comm: stress-ng Tainted:
>>>>> G OEL 5.13.0-rc7 #1
>>>>> [13330.652129] Hardware name: HPE Apollo 70
>>>>> /C01_APACHE_MB , BIOS L50_5.13_1.15 05/08/2020
>>>>> [13330.652133] pstate: 80400009 (Nzcv daif +PAN -UAO -TCO BTYPE=--)
>>>>> [13330.652139] pc : __memcpy+0x168/0x250
>>>>> [13330.652150] lr : memory_read_from_buffer+0x58/0x80
>>>>> [13330.652161] sp : ffff800063ef3c20
>>>>> [13330.652163] x29: ffff800063ef3c20 x28: ffff0008b1380000 x27:
>>>>> 0000000000000000
>>>>> [13330.652170] x26: 0000000000000000 x25: 0000000000000000 x24:
>>>>> ffff00080a960fe0
>>>>> [13330.652176] x23: ffff800063ef3d28 x22: 000000000000063f x21:
>>>>> ffff800063ef3c88
>>>>> [13330.652181] x20: 000000000000063f x19: 000000000000063f x18:
>>>>> 0000000000000000
>>>>> [13330.652186] x17: 0000000000000000 x16: 0000000000000000 x15:
>>>>> 0000000000000000
>>>>> [13330.652191] x14: 0000000000000000 x13: 0000000000000000 x12:
>>>>> 0000000000000000
>>>>> [13330.652196] x11: 0000000000000000 x10: 0000000000000000 x9 :
>>>>> 0000000000000000
>>>>> [13330.652200] x8 : 0000000000000000 x7 : 0000000000000000 x6 :
>>>>> 0000000000000000
>>>>> [13330.652206] x5 : ffff000d0fb0063f x4 : ffff80005347063f x3 :
>>>>> ffff000d0fb005c0
>>>>> [13330.652212] x2 : ffffffffffffffef x1 : ffff800053470600 x0 :
>>>>> ffff000d0fb00000
>>>>> [13330.652218] Call trace:
>>>>> [13330.652221] __memcpy+0x168/0x250
>>>>> [13330.652225] acpi_data_show+0x5c/0x8c
>>>>> [13330.652232] sysfs_kf_bin_read+0x78/0xa0
>>>>> [13330.652238] kernfs_file_read_iter+0x9c/0x1a4
>>>>> [13330.652241] kernfs_fop_read_iter+0x34/0x50
>>>>> [13330.652244] new_sync_read+0xdc/0x154
>>>>> [13330.652253] vfs_read+0x158/0x1e4
>>>>> [13330.652260] ksys_read+0x64/0xec
>>>>> [13330.652266] __arm64_sys_read+0x28/0x34
>>>>> [13330.652273] invoke_syscall+0x50/0x120
>>>>> [13330.652280] el0_svc_common.constprop.0+0x4c/0xd4
>>>>> [13330.652284] do_el0_svc+0x30/0x9c
>>>>> [13330.652286] el0_svc+0x2c/0x54
>>>>> [13330.652294] el0t_64_sync_handler+0x1a4/0x1b0
>>>>> [13330.652296] el0t_64_sync+0x19c/0x1a0
>>>>> [13330.652303] Code: a984346c a9c4342c f1010042 54fffee8 (a97c3c8e)
>>>>> [13330.652307] ---[ end trace 227d4380f57145d4 ]---
>>>>>
>>>>> So maybe this issue isn't limited to weird modules, after all...
>>>>>
>>>>
>>>> It ran on the machine from the same set that we were able to reproduce
>>>> it on previously. If you or anyone else have an idea on how to
>>>> stabilize
>>>> the reproducibility or have a debug patch we'll be happy to try it.
>>>
>>> Possibly it depends on the individual machines' firmware exactly how the
>>> relevant bits of their ACPI tables are aligned in memory?
>>>
>>> I've started digging into that callstack - it may not be a "weird
>>> module"
>>> but it's definitely crusty ACPI code... a238317ce818 ("ACPI: Clean up
>>> acpi_os_map/unmap_memory() to eliminate __iomem.") looks frankly a bit
>>> questionable in its decision to blithely cast away __iomem, but then the
>>> rationale in aafc65c731fe ("ACPI: add arm64 to the platforms that use
>>> ioremap") seems particularly dubious on top of that (especially given
>>> this
>>> end result).
>>>
>>> At a wild guess, I'm wondering if this may be sufficient:
>>>
>>> ----->8-----
>>> diff --git a/drivers/acpi/osl.c b/drivers/acpi/osl.c
>>> index 327e1b4eb6b0..f5d26b102fbe 100644
>>> --- a/drivers/acpi/osl.c
>>> +++ b/drivers/acpi/osl.c
>>> @@ -277,7 +277,7 @@ acpi_map_lookup_virt(void __iomem *virt,
>>> acpi_size size)
>>> return NULL;
>>> }
>>>
>>> -#if defined(CONFIG_IA64) || defined(CONFIG_ARM64)
>>> +#if defined(CONFIG_IA64)
>>> /* ioremap will take care of cache attributes */
>>> #define should_use_kmap(pfn) 0
>>> #else
>>> -----8<-----
>>
>> I thought the same but shouldn't acpi_os_ioremap() map it with the right
>> attributes? It uses the EFI maps to check what kind of memory this is.
>
> Oh crikey, I missed that branch of the rabbit hole... I guess that must
> mean that the tables being poked here are *not* covered by the EFI
> memory map, so page_is_ram() is unlikely to help either :(
After picking through the UEFI spec I think I've now got a clearer
picture of what's happening, but I'm not sure where it goes from here...
The spec implies that it *is* legitimate for runtime-loaded ACPI tables
to lie outside the EFI memory map, and that case they must be assumed to
be uncached, so the behaviour of acpi_os_ioremap() is correct. Given the
definition of uncached for arm64 memory types though, that means that
callers of acpi_os_map_memory() still have to be prepared to get an
__iomem pointer back even if they know they're mapping a table rather
than some random bit of MMIO for an AML method.
Therefore in this case it seems the blame lies partway between
acpi_os_map_memory() for casting away __iomem and acpi_data_show() for
letting an arbitrary offset lead to an arbitrarily-aligned memcpy(), but
I don't know what the best way to fix it is. Either way I've satisfied
myself that it's not an issue with the arm64 code itself - I do wonder
whether this might also be a problem on IA-64 given
ACPI_MISALIGNMENT_NOT_SUPPORTED, and I guess RISC-V may have alignment
concerns as well.
Robin.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: ❌ FAIL: Test report for kernel 5.13.0-rc7 (arm-next, 8ab9b1a9)
2021-06-29 11:48 ` Robin Murphy
@ 2021-06-29 14:44 ` Lorenzo Pieralisi
-1 siblings, 0 replies; 39+ messages in thread
From: Lorenzo Pieralisi @ 2021-06-29 14:44 UTC (permalink / raw)
To: Robin Murphy
Cc: ACPI Devel Maling List, Veronika Kabatova, Will Deacon,
CKI Project, Mark Rutland, Memory Management, skt-results-master,
Jeff Bastian, Jan Stancek, Linux ARM, rjw, lenb, guohanjun,
sudeep.holla, ardb, Catalin Marinas, lv.zheng, tony.luck
On Tue, Jun 29, 2021 at 12:48:14PM +0100, Robin Murphy wrote:
> [ +ACPI audience ]
>
> On 2021-06-25 12:15, Robin Murphy wrote:
> > On 2021-06-25 12:09, Catalin Marinas wrote:
> > > On Fri, Jun 25, 2021 at 12:02:52PM +0100, Robin Murphy wrote:
> > > > On 2021-06-25 10:52, Veronika Kabatova wrote:
> > > > [...]
> > > > > > > ❌ stress: stress-ng
> > > > > >
> > > > > > Oh no, this looks like another alignment fault in memcpy:
> > > > > >
> > > > > > [13330.651903] Unable to handle kernel paging request at
> > > > > > virtual address ffff8000534705ff
> > > > > > [13330.651914] Mem abort info:
> > > > > > [13330.651918] ESR = 0x96000021
> > > > > > [13330.651922] EC = 0x25: DABT (current EL), IL = 32 bits
> > > > > > [13330.651928] SET = 0, FnV = 0
> > > > > > [13330.651931] EA = 0, S1PTW = 0
> > > > > > [13330.651933] FSC = 0x21: alignment fault
> > > > > > [13330.651938] Data abort info:
> > > > > > [13330.651940] ISV = 0, ISS = 0x00000021
> > > > > > [13330.651941] CM = 0, WnR = 0
> > > > > > [13330.651943] swapper pgtable: 4k pages, 48-bit VAs,
> > > > > > pgdp=00000000f3e6b000
> > > > > > [13330.651945] [ffff8000534705ff] pgd=1000008ffcfff003,
> > > > > > p4d=1000008ffcfff003, pud=100000088e57d003,
> > > > > > pmd=10000008d0aeb003, pte=006800008021370f
> > > > > > [13330.651956] Internal error: Oops: 96000021 [#1] SMP
> > > > > > [13330.651961] Modules linked in: unix_diag binfmt_misc
> > > > > > fcrypt sm4_generic crc32_generic md4 michael_mic
> > > > > > nhpoly1305_neon nhpoly1305 poly1305_generic libpoly1305
> > > > > > poly1305_neon rmd160 sha3_generic sm3_generic
> > > > > > streebog_generic wp512 blowfish_generic blowfish_common
> > > > > > cast5_generic des_generic libdes chacha_generic
> > > > > > chacha_neon libchacha camellia_generic cast6_generic
> > > > > > cast_common serpent_generic twofish_generic
> > > > > > twofish_common dm_thin_pool dm_persistent_data
> > > > > > dm_bio_prison nvme nvme_core ipmi_watchdog ipmi_poweroff
> > > > > > loop tun af_key crypto_user scsi_transport_iscsi
> > > > > > xt_multiport ip_gre ip_tunnel gre overlay xt_CONNSECMARK
> > > > > > xt_SECMARK nft_counter xt_state xt_conntrack nft_compat
> > > > > > ah6 ah4 nft_objref nft_ct nf_conntrack nf_defrag_ipv6
> > > > > > nf_defrag_ipv4 nf_tables nfnetlink jfs sctp
> > > > > > ip6_udp_tunnel udp_tunnel dm_log_writes dm_flakey rfkill
> > > > > > mlx5_ib ib_uverbs ib_core sunrpc coresight_etm4x
> > > > > > i2c_smbus coresight_replicator coresight_tpiu
> > > > > > coresight_tmc joydev mlx5_core acpi_ipmi psample
> > > > > > ipmi_ssif mlxfw !
> > > > > > ipmi_devintf
> > > > > > [13330.652076] ipmi_msghandler coresight_funnel
> > > > > > thunderx2_pmu coresight vfat fat fuse zram ip_tables xfs
> > > > > > ast crct10dif_ce i2c_algo_bit ghash_ce drm_vram_helper
> > > > > > drm_kms_helper syscopyarea sysfillrect sysimgblt
> > > > > > fb_sys_fops cec drm_ttm_helper ttm drm gpio_xlp
> > > > > > i2c_xlp9xx uas usb_storage aes_neon_bs [last unloaded:
> > > > > > nvmet]
> > > > > > [13330.652123] CPU: 115 PID: 188446 Comm: stress-ng
> > > > > > Tainted: G OEL 5.13.0-rc7 #1
> > > > > > [13330.652129] Hardware name: HPE Apollo 70
> > > > > > /C01_APACHE_MB , BIOS L50_5.13_1.15 05/08/2020
> > > > > > [13330.652133] pstate: 80400009 (Nzcv daif +PAN -UAO -TCO BTYPE=--)
> > > > > > [13330.652139] pc : __memcpy+0x168/0x250
> > > > > > [13330.652150] lr : memory_read_from_buffer+0x58/0x80
> > > > > > [13330.652161] sp : ffff800063ef3c20
> > > > > > [13330.652163] x29: ffff800063ef3c20 x28:
> > > > > > ffff0008b1380000 x27: 0000000000000000
> > > > > > [13330.652170] x26: 0000000000000000 x25:
> > > > > > 0000000000000000 x24: ffff00080a960fe0
> > > > > > [13330.652176] x23: ffff800063ef3d28 x22:
> > > > > > 000000000000063f x21: ffff800063ef3c88
> > > > > > [13330.652181] x20: 000000000000063f x19:
> > > > > > 000000000000063f x18: 0000000000000000
> > > > > > [13330.652186] x17: 0000000000000000 x16:
> > > > > > 0000000000000000 x15: 0000000000000000
> > > > > > [13330.652191] x14: 0000000000000000 x13:
> > > > > > 0000000000000000 x12: 0000000000000000
> > > > > > [13330.652196] x11: 0000000000000000 x10:
> > > > > > 0000000000000000 x9 : 0000000000000000
> > > > > > [13330.652200] x8 : 0000000000000000 x7 :
> > > > > > 0000000000000000 x6 : 0000000000000000
> > > > > > [13330.652206] x5 : ffff000d0fb0063f x4 :
> > > > > > ffff80005347063f x3 : ffff000d0fb005c0
> > > > > > [13330.652212] x2 : ffffffffffffffef x1 :
> > > > > > ffff800053470600 x0 : ffff000d0fb00000
> > > > > > [13330.652218] Call trace:
> > > > > > [13330.652221] __memcpy+0x168/0x250
> > > > > > [13330.652225] acpi_data_show+0x5c/0x8c
> > > > > > [13330.652232] sysfs_kf_bin_read+0x78/0xa0
> > > > > > [13330.652238] kernfs_file_read_iter+0x9c/0x1a4
> > > > > > [13330.652241] kernfs_fop_read_iter+0x34/0x50
> > > > > > [13330.652244] new_sync_read+0xdc/0x154
> > > > > > [13330.652253] vfs_read+0x158/0x1e4
> > > > > > [13330.652260] ksys_read+0x64/0xec
> > > > > > [13330.652266] __arm64_sys_read+0x28/0x34
> > > > > > [13330.652273] invoke_syscall+0x50/0x120
> > > > > > [13330.652280] el0_svc_common.constprop.0+0x4c/0xd4
> > > > > > [13330.652284] do_el0_svc+0x30/0x9c
> > > > > > [13330.652286] el0_svc+0x2c/0x54
> > > > > > [13330.652294] el0t_64_sync_handler+0x1a4/0x1b0
> > > > > > [13330.652296] el0t_64_sync+0x19c/0x1a0
> > > > > > [13330.652303] Code: a984346c a9c4342c f1010042 54fffee8 (a97c3c8e)
> > > > > > [13330.652307] ---[ end trace 227d4380f57145d4 ]---
> > > > > >
> > > > > > So maybe this issue isn't limited to weird modules, after all...
> > > > > >
> > > > >
> > > > > It ran on the machine from the same set that we were able to reproduce
> > > > > it on previously. If you or anyone else have an idea on how
> > > > > to stabilize
> > > > > the reproducibility or have a debug patch we'll be happy to try it.
> > > >
> > > > Possibly it depends on the individual machines' firmware exactly how the
> > > > relevant bits of their ACPI tables are aligned in memory?
> > > >
> > > > I've started digging into that callstack - it may not be a
> > > > "weird module"
> > > > but it's definitely crusty ACPI code... a238317ce818 ("ACPI: Clean up
> > > > acpi_os_map/unmap_memory() to eliminate __iomem.") looks frankly a bit
> > > > questionable in its decision to blithely cast away __iomem, but then the
> > > > rationale in aafc65c731fe ("ACPI: add arm64 to the platforms that use
> > > > ioremap") seems particularly dubious on top of that (especially
> > > > given this
> > > > end result).
> > > >
> > > > At a wild guess, I'm wondering if this may be sufficient:
> > > >
> > > > ----->8-----
> > > > diff --git a/drivers/acpi/osl.c b/drivers/acpi/osl.c
> > > > index 327e1b4eb6b0..f5d26b102fbe 100644
> > > > --- a/drivers/acpi/osl.c
> > > > +++ b/drivers/acpi/osl.c
> > > > @@ -277,7 +277,7 @@ acpi_map_lookup_virt(void __iomem *virt,
> > > > acpi_size size)
> > > > return NULL;
> > > > }
> > > >
> > > > -#if defined(CONFIG_IA64) || defined(CONFIG_ARM64)
> > > > +#if defined(CONFIG_IA64)
> > > > /* ioremap will take care of cache attributes */
> > > > #define should_use_kmap(pfn) 0
> > > > #else
> > > > -----8<-----
> > >
> > > I thought the same but shouldn't acpi_os_ioremap() map it with the right
> > > attributes? It uses the EFI maps to check what kind of memory this is.
> >
> > Oh crikey, I missed that branch of the rabbit hole... I guess that must
> > mean that the tables being poked here are *not* covered by the EFI
> > memory map, so page_is_ram() is unlikely to help either :(
>
> After picking through the UEFI spec I think I've now got a clearer picture
> of what's happening, but I'm not sure where it goes from here...
>
> The spec implies that it *is* legitimate for runtime-loaded ACPI tables to
> lie outside the EFI memory map, and that case they must be assumed to be
> uncached, so the behaviour of acpi_os_ioremap() is correct.
I'd agree with the reasoning, it would be good to pinpoint whether
that's what actually triggers the issue.
I'd like to replicate it if possible (it is TX2 HW but firmware
config is likely to differ from the HW I have at hand), the
test command line that triggers the fault would be useful as
a starting point.
Furthermore, is this a v5.13-rc* regression ? If so it would be
good to bisect it - I can't recollect arm64 changes that could
have introduced this regression in the last cycle but I may have
missed something.
> Given the definition of uncached for arm64 memory types though, that
> means that callers of acpi_os_map_memory() still have to be prepared
> to get an __iomem pointer back even if they know they're mapping a
> table rather than some random bit of MMIO for an AML method.
>
> Therefore in this case it seems the blame lies partway between
> acpi_os_map_memory() for casting away __iomem and acpi_data_show() for
> letting an arbitrary offset lead to an arbitrarily-aligned memcpy(), but I
> don't know what the best way to fix it is. Either way I've satisfied myself
> that it's not an issue with the arm64 code itself - I do wonder whether this
> might also be a problem on IA-64 given ACPI_MISALIGNMENT_NOT_SUPPORTED, and
> I guess RISC-V may have alignment concerns as well.
Yes agreed but see above, this code has been there for aeons if it
is a v5.13-rc* regression it must be something else that actually
triggered it (test/FW config).
Thanks for looking into this.
Lorenzo
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: ❌ FAIL: Test report for kernel 5.13.0-rc7 (arm-next, 8ab9b1a9)
@ 2021-06-29 14:44 ` Lorenzo Pieralisi
0 siblings, 0 replies; 39+ messages in thread
From: Lorenzo Pieralisi @ 2021-06-29 14:44 UTC (permalink / raw)
To: Robin Murphy
Cc: ACPI Devel Maling List, Veronika Kabatova, Will Deacon,
CKI Project, Mark Rutland, Memory Management, skt-results-master,
Jeff Bastian, Jan Stancek, Linux ARM, rjw, lenb, guohanjun,
sudeep.holla, ardb, Catalin Marinas, lv.zheng, tony.luck
On Tue, Jun 29, 2021 at 12:48:14PM +0100, Robin Murphy wrote:
> [ +ACPI audience ]
>
> On 2021-06-25 12:15, Robin Murphy wrote:
> > On 2021-06-25 12:09, Catalin Marinas wrote:
> > > On Fri, Jun 25, 2021 at 12:02:52PM +0100, Robin Murphy wrote:
> > > > On 2021-06-25 10:52, Veronika Kabatova wrote:
> > > > [...]
> > > > > > > ❌ stress: stress-ng
> > > > > >
> > > > > > Oh no, this looks like another alignment fault in memcpy:
> > > > > >
> > > > > > [13330.651903] Unable to handle kernel paging request at
> > > > > > virtual address ffff8000534705ff
> > > > > > [13330.651914] Mem abort info:
> > > > > > [13330.651918] ESR = 0x96000021
> > > > > > [13330.651922] EC = 0x25: DABT (current EL), IL = 32 bits
> > > > > > [13330.651928] SET = 0, FnV = 0
> > > > > > [13330.651931] EA = 0, S1PTW = 0
> > > > > > [13330.651933] FSC = 0x21: alignment fault
> > > > > > [13330.651938] Data abort info:
> > > > > > [13330.651940] ISV = 0, ISS = 0x00000021
> > > > > > [13330.651941] CM = 0, WnR = 0
> > > > > > [13330.651943] swapper pgtable: 4k pages, 48-bit VAs,
> > > > > > pgdp=00000000f3e6b000
> > > > > > [13330.651945] [ffff8000534705ff] pgd=1000008ffcfff003,
> > > > > > p4d=1000008ffcfff003, pud=100000088e57d003,
> > > > > > pmd=10000008d0aeb003, pte=006800008021370f
> > > > > > [13330.651956] Internal error: Oops: 96000021 [#1] SMP
> > > > > > [13330.651961] Modules linked in: unix_diag binfmt_misc
> > > > > > fcrypt sm4_generic crc32_generic md4 michael_mic
> > > > > > nhpoly1305_neon nhpoly1305 poly1305_generic libpoly1305
> > > > > > poly1305_neon rmd160 sha3_generic sm3_generic
> > > > > > streebog_generic wp512 blowfish_generic blowfish_common
> > > > > > cast5_generic des_generic libdes chacha_generic
> > > > > > chacha_neon libchacha camellia_generic cast6_generic
> > > > > > cast_common serpent_generic twofish_generic
> > > > > > twofish_common dm_thin_pool dm_persistent_data
> > > > > > dm_bio_prison nvme nvme_core ipmi_watchdog ipmi_poweroff
> > > > > > loop tun af_key crypto_user scsi_transport_iscsi
> > > > > > xt_multiport ip_gre ip_tunnel gre overlay xt_CONNSECMARK
> > > > > > xt_SECMARK nft_counter xt_state xt_conntrack nft_compat
> > > > > > ah6 ah4 nft_objref nft_ct nf_conntrack nf_defrag_ipv6
> > > > > > nf_defrag_ipv4 nf_tables nfnetlink jfs sctp
> > > > > > ip6_udp_tunnel udp_tunnel dm_log_writes dm_flakey rfkill
> > > > > > mlx5_ib ib_uverbs ib_core sunrpc coresight_etm4x
> > > > > > i2c_smbus coresight_replicator coresight_tpiu
> > > > > > coresight_tmc joydev mlx5_core acpi_ipmi psample
> > > > > > ipmi_ssif mlxfw !
> > > > > > ipmi_devintf
> > > > > > [13330.652076] ipmi_msghandler coresight_funnel
> > > > > > thunderx2_pmu coresight vfat fat fuse zram ip_tables xfs
> > > > > > ast crct10dif_ce i2c_algo_bit ghash_ce drm_vram_helper
> > > > > > drm_kms_helper syscopyarea sysfillrect sysimgblt
> > > > > > fb_sys_fops cec drm_ttm_helper ttm drm gpio_xlp
> > > > > > i2c_xlp9xx uas usb_storage aes_neon_bs [last unloaded:
> > > > > > nvmet]
> > > > > > [13330.652123] CPU: 115 PID: 188446 Comm: stress-ng
> > > > > > Tainted: G OEL 5.13.0-rc7 #1
> > > > > > [13330.652129] Hardware name: HPE Apollo 70
> > > > > > /C01_APACHE_MB , BIOS L50_5.13_1.15 05/08/2020
> > > > > > [13330.652133] pstate: 80400009 (Nzcv daif +PAN -UAO -TCO BTYPE=--)
> > > > > > [13330.652139] pc : __memcpy+0x168/0x250
> > > > > > [13330.652150] lr : memory_read_from_buffer+0x58/0x80
> > > > > > [13330.652161] sp : ffff800063ef3c20
> > > > > > [13330.652163] x29: ffff800063ef3c20 x28:
> > > > > > ffff0008b1380000 x27: 0000000000000000
> > > > > > [13330.652170] x26: 0000000000000000 x25:
> > > > > > 0000000000000000 x24: ffff00080a960fe0
> > > > > > [13330.652176] x23: ffff800063ef3d28 x22:
> > > > > > 000000000000063f x21: ffff800063ef3c88
> > > > > > [13330.652181] x20: 000000000000063f x19:
> > > > > > 000000000000063f x18: 0000000000000000
> > > > > > [13330.652186] x17: 0000000000000000 x16:
> > > > > > 0000000000000000 x15: 0000000000000000
> > > > > > [13330.652191] x14: 0000000000000000 x13:
> > > > > > 0000000000000000 x12: 0000000000000000
> > > > > > [13330.652196] x11: 0000000000000000 x10:
> > > > > > 0000000000000000 x9 : 0000000000000000
> > > > > > [13330.652200] x8 : 0000000000000000 x7 :
> > > > > > 0000000000000000 x6 : 0000000000000000
> > > > > > [13330.652206] x5 : ffff000d0fb0063f x4 :
> > > > > > ffff80005347063f x3 : ffff000d0fb005c0
> > > > > > [13330.652212] x2 : ffffffffffffffef x1 :
> > > > > > ffff800053470600 x0 : ffff000d0fb00000
> > > > > > [13330.652218] Call trace:
> > > > > > [13330.652221] __memcpy+0x168/0x250
> > > > > > [13330.652225] acpi_data_show+0x5c/0x8c
> > > > > > [13330.652232] sysfs_kf_bin_read+0x78/0xa0
> > > > > > [13330.652238] kernfs_file_read_iter+0x9c/0x1a4
> > > > > > [13330.652241] kernfs_fop_read_iter+0x34/0x50
> > > > > > [13330.652244] new_sync_read+0xdc/0x154
> > > > > > [13330.652253] vfs_read+0x158/0x1e4
> > > > > > [13330.652260] ksys_read+0x64/0xec
> > > > > > [13330.652266] __arm64_sys_read+0x28/0x34
> > > > > > [13330.652273] invoke_syscall+0x50/0x120
> > > > > > [13330.652280] el0_svc_common.constprop.0+0x4c/0xd4
> > > > > > [13330.652284] do_el0_svc+0x30/0x9c
> > > > > > [13330.652286] el0_svc+0x2c/0x54
> > > > > > [13330.652294] el0t_64_sync_handler+0x1a4/0x1b0
> > > > > > [13330.652296] el0t_64_sync+0x19c/0x1a0
> > > > > > [13330.652303] Code: a984346c a9c4342c f1010042 54fffee8 (a97c3c8e)
> > > > > > [13330.652307] ---[ end trace 227d4380f57145d4 ]---
> > > > > >
> > > > > > So maybe this issue isn't limited to weird modules, after all...
> > > > > >
> > > > >
> > > > > It ran on the machine from the same set that we were able to reproduce
> > > > > it on previously. If you or anyone else have an idea on how
> > > > > to stabilize
> > > > > the reproducibility or have a debug patch we'll be happy to try it.
> > > >
> > > > Possibly it depends on the individual machines' firmware exactly how the
> > > > relevant bits of their ACPI tables are aligned in memory?
> > > >
> > > > I've started digging into that callstack - it may not be a
> > > > "weird module"
> > > > but it's definitely crusty ACPI code... a238317ce818 ("ACPI: Clean up
> > > > acpi_os_map/unmap_memory() to eliminate __iomem.") looks frankly a bit
> > > > questionable in its decision to blithely cast away __iomem, but then the
> > > > rationale in aafc65c731fe ("ACPI: add arm64 to the platforms that use
> > > > ioremap") seems particularly dubious on top of that (especially
> > > > given this
> > > > end result).
> > > >
> > > > At a wild guess, I'm wondering if this may be sufficient:
> > > >
> > > > ----->8-----
> > > > diff --git a/drivers/acpi/osl.c b/drivers/acpi/osl.c
> > > > index 327e1b4eb6b0..f5d26b102fbe 100644
> > > > --- a/drivers/acpi/osl.c
> > > > +++ b/drivers/acpi/osl.c
> > > > @@ -277,7 +277,7 @@ acpi_map_lookup_virt(void __iomem *virt,
> > > > acpi_size size)
> > > > return NULL;
> > > > }
> > > >
> > > > -#if defined(CONFIG_IA64) || defined(CONFIG_ARM64)
> > > > +#if defined(CONFIG_IA64)
> > > > /* ioremap will take care of cache attributes */
> > > > #define should_use_kmap(pfn) 0
> > > > #else
> > > > -----8<-----
> > >
> > > I thought the same but shouldn't acpi_os_ioremap() map it with the right
> > > attributes? It uses the EFI maps to check what kind of memory this is.
> >
> > Oh crikey, I missed that branch of the rabbit hole... I guess that must
> > mean that the tables being poked here are *not* covered by the EFI
> > memory map, so page_is_ram() is unlikely to help either :(
>
> After picking through the UEFI spec I think I've now got a clearer picture
> of what's happening, but I'm not sure where it goes from here...
>
> The spec implies that it *is* legitimate for runtime-loaded ACPI tables to
> lie outside the EFI memory map, and that case they must be assumed to be
> uncached, so the behaviour of acpi_os_ioremap() is correct.
I'd agree with the reasoning, it would be good to pinpoint whether
that's what actually triggers the issue.
I'd like to replicate it if possible (it is TX2 HW but firmware
config is likely to differ from the HW I have at hand), the
test command line that triggers the fault would be useful as
a starting point.
Furthermore, is this a v5.13-rc* regression ? If so it would be
good to bisect it - I can't recollect arm64 changes that could
have introduced this regression in the last cycle but I may have
missed something.
> Given the definition of uncached for arm64 memory types though, that
> means that callers of acpi_os_map_memory() still have to be prepared
> to get an __iomem pointer back even if they know they're mapping a
> table rather than some random bit of MMIO for an AML method.
>
> Therefore in this case it seems the blame lies partway between
> acpi_os_map_memory() for casting away __iomem and acpi_data_show() for
> letting an arbitrary offset lead to an arbitrarily-aligned memcpy(), but I
> don't know what the best way to fix it is. Either way I've satisfied myself
> that it's not an issue with the arm64 code itself - I do wonder whether this
> might also be a problem on IA-64 given ACPI_MISALIGNMENT_NOT_SUPPORTED, and
> I guess RISC-V may have alignment concerns as well.
Yes agreed but see above, this code has been there for aeons if it
is a v5.13-rc* regression it must be something else that actually
triggered it (test/FW config).
Thanks for looking into this.
Lorenzo
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: ❌ FAIL: Test report for kernel 5.13.0-rc7 (arm-next, 8ab9b1a9)
2021-06-29 14:44 ` Lorenzo Pieralisi
@ 2021-06-29 15:14 ` Robin Murphy
-1 siblings, 0 replies; 39+ messages in thread
From: Robin Murphy @ 2021-06-29 15:14 UTC (permalink / raw)
To: Lorenzo Pieralisi
Cc: ACPI Devel Maling List, Veronika Kabatova, Will Deacon,
CKI Project, Mark Rutland, Memory Management, skt-results-master,
Jeff Bastian, Jan Stancek, Linux ARM, rjw, lenb, guohanjun,
sudeep.holla, ardb, Catalin Marinas, lv.zheng, tony.luck
On 2021-06-29 15:44, Lorenzo Pieralisi wrote:
> On Tue, Jun 29, 2021 at 12:48:14PM +0100, Robin Murphy wrote:
>> [ +ACPI audience ]
>>
>> On 2021-06-25 12:15, Robin Murphy wrote:
>>> On 2021-06-25 12:09, Catalin Marinas wrote:
>>>> On Fri, Jun 25, 2021 at 12:02:52PM +0100, Robin Murphy wrote:
>>>>> On 2021-06-25 10:52, Veronika Kabatova wrote:
>>>>> [...]
>>>>>>>> ❌ stress: stress-ng
>>>>>>>
>>>>>>> Oh no, this looks like another alignment fault in memcpy:
>>>>>>>
>>>>>>> [13330.651903] Unable to handle kernel paging request at
>>>>>>> virtual address ffff8000534705ff
>>>>>>> [13330.651914] Mem abort info:
>>>>>>> [13330.651918] ESR = 0x96000021
>>>>>>> [13330.651922] EC = 0x25: DABT (current EL), IL = 32 bits
>>>>>>> [13330.651928] SET = 0, FnV = 0
>>>>>>> [13330.651931] EA = 0, S1PTW = 0
>>>>>>> [13330.651933] FSC = 0x21: alignment fault
>>>>>>> [13330.651938] Data abort info:
>>>>>>> [13330.651940] ISV = 0, ISS = 0x00000021
>>>>>>> [13330.651941] CM = 0, WnR = 0
>>>>>>> [13330.651943] swapper pgtable: 4k pages, 48-bit VAs,
>>>>>>> pgdp=00000000f3e6b000
>>>>>>> [13330.651945] [ffff8000534705ff] pgd=1000008ffcfff003,
>>>>>>> p4d=1000008ffcfff003, pud=100000088e57d003,
>>>>>>> pmd=10000008d0aeb003, pte=006800008021370f
>>>>>>> [13330.651956] Internal error: Oops: 96000021 [#1] SMP
>>>>>>> [13330.651961] Modules linked in: unix_diag binfmt_misc
>>>>>>> fcrypt sm4_generic crc32_generic md4 michael_mic
>>>>>>> nhpoly1305_neon nhpoly1305 poly1305_generic libpoly1305
>>>>>>> poly1305_neon rmd160 sha3_generic sm3_generic
>>>>>>> streebog_generic wp512 blowfish_generic blowfish_common
>>>>>>> cast5_generic des_generic libdes chacha_generic
>>>>>>> chacha_neon libchacha camellia_generic cast6_generic
>>>>>>> cast_common serpent_generic twofish_generic
>>>>>>> twofish_common dm_thin_pool dm_persistent_data
>>>>>>> dm_bio_prison nvme nvme_core ipmi_watchdog ipmi_poweroff
>>>>>>> loop tun af_key crypto_user scsi_transport_iscsi
>>>>>>> xt_multiport ip_gre ip_tunnel gre overlay xt_CONNSECMARK
>>>>>>> xt_SECMARK nft_counter xt_state xt_conntrack nft_compat
>>>>>>> ah6 ah4 nft_objref nft_ct nf_conntrack nf_defrag_ipv6
>>>>>>> nf_defrag_ipv4 nf_tables nfnetlink jfs sctp
>>>>>>> ip6_udp_tunnel udp_tunnel dm_log_writes dm_flakey rfkill
>>>>>>> mlx5_ib ib_uverbs ib_core sunrpc coresight_etm4x
>>>>>>> i2c_smbus coresight_replicator coresight_tpiu
>>>>>>> coresight_tmc joydev mlx5_core acpi_ipmi psample
>>>>>>> ipmi_ssif mlxfw !
>>>>>>> ipmi_devintf
>>>>>>> [13330.652076] ipmi_msghandler coresight_funnel
>>>>>>> thunderx2_pmu coresight vfat fat fuse zram ip_tables xfs
>>>>>>> ast crct10dif_ce i2c_algo_bit ghash_ce drm_vram_helper
>>>>>>> drm_kms_helper syscopyarea sysfillrect sysimgblt
>>>>>>> fb_sys_fops cec drm_ttm_helper ttm drm gpio_xlp
>>>>>>> i2c_xlp9xx uas usb_storage aes_neon_bs [last unloaded:
>>>>>>> nvmet]
>>>>>>> [13330.652123] CPU: 115 PID: 188446 Comm: stress-ng
>>>>>>> Tainted: G OEL 5.13.0-rc7 #1
>>>>>>> [13330.652129] Hardware name: HPE Apollo 70
>>>>>>> /C01_APACHE_MB , BIOS L50_5.13_1.15 05/08/2020
>>>>>>> [13330.652133] pstate: 80400009 (Nzcv daif +PAN -UAO -TCO BTYPE=--)
>>>>>>> [13330.652139] pc : __memcpy+0x168/0x250
>>>>>>> [13330.652150] lr : memory_read_from_buffer+0x58/0x80
>>>>>>> [13330.652161] sp : ffff800063ef3c20
>>>>>>> [13330.652163] x29: ffff800063ef3c20 x28:
>>>>>>> ffff0008b1380000 x27: 0000000000000000
>>>>>>> [13330.652170] x26: 0000000000000000 x25:
>>>>>>> 0000000000000000 x24: ffff00080a960fe0
>>>>>>> [13330.652176] x23: ffff800063ef3d28 x22:
>>>>>>> 000000000000063f x21: ffff800063ef3c88
>>>>>>> [13330.652181] x20: 000000000000063f x19:
>>>>>>> 000000000000063f x18: 0000000000000000
>>>>>>> [13330.652186] x17: 0000000000000000 x16:
>>>>>>> 0000000000000000 x15: 0000000000000000
>>>>>>> [13330.652191] x14: 0000000000000000 x13:
>>>>>>> 0000000000000000 x12: 0000000000000000
>>>>>>> [13330.652196] x11: 0000000000000000 x10:
>>>>>>> 0000000000000000 x9 : 0000000000000000
>>>>>>> [13330.652200] x8 : 0000000000000000 x7 :
>>>>>>> 0000000000000000 x6 : 0000000000000000
>>>>>>> [13330.652206] x5 : ffff000d0fb0063f x4 :
>>>>>>> ffff80005347063f x3 : ffff000d0fb005c0
>>>>>>> [13330.652212] x2 : ffffffffffffffef x1 :
>>>>>>> ffff800053470600 x0 : ffff000d0fb00000
>>>>>>> [13330.652218] Call trace:
>>>>>>> [13330.652221] __memcpy+0x168/0x250
>>>>>>> [13330.652225] acpi_data_show+0x5c/0x8c
>>>>>>> [13330.652232] sysfs_kf_bin_read+0x78/0xa0
>>>>>>> [13330.652238] kernfs_file_read_iter+0x9c/0x1a4
>>>>>>> [13330.652241] kernfs_fop_read_iter+0x34/0x50
>>>>>>> [13330.652244] new_sync_read+0xdc/0x154
>>>>>>> [13330.652253] vfs_read+0x158/0x1e4
>>>>>>> [13330.652260] ksys_read+0x64/0xec
>>>>>>> [13330.652266] __arm64_sys_read+0x28/0x34
>>>>>>> [13330.652273] invoke_syscall+0x50/0x120
>>>>>>> [13330.652280] el0_svc_common.constprop.0+0x4c/0xd4
>>>>>>> [13330.652284] do_el0_svc+0x30/0x9c
>>>>>>> [13330.652286] el0_svc+0x2c/0x54
>>>>>>> [13330.652294] el0t_64_sync_handler+0x1a4/0x1b0
>>>>>>> [13330.652296] el0t_64_sync+0x19c/0x1a0
>>>>>>> [13330.652303] Code: a984346c a9c4342c f1010042 54fffee8 (a97c3c8e)
>>>>>>> [13330.652307] ---[ end trace 227d4380f57145d4 ]---
>>>>>>>
>>>>>>> So maybe this issue isn't limited to weird modules, after all...
>>>>>>>
>>>>>>
>>>>>> It ran on the machine from the same set that we were able to reproduce
>>>>>> it on previously. If you or anyone else have an idea on how
>>>>>> to stabilize
>>>>>> the reproducibility or have a debug patch we'll be happy to try it.
>>>>>
>>>>> Possibly it depends on the individual machines' firmware exactly how the
>>>>> relevant bits of their ACPI tables are aligned in memory?
>>>>>
>>>>> I've started digging into that callstack - it may not be a
>>>>> "weird module"
>>>>> but it's definitely crusty ACPI code... a238317ce818 ("ACPI: Clean up
>>>>> acpi_os_map/unmap_memory() to eliminate __iomem.") looks frankly a bit
>>>>> questionable in its decision to blithely cast away __iomem, but then the
>>>>> rationale in aafc65c731fe ("ACPI: add arm64 to the platforms that use
>>>>> ioremap") seems particularly dubious on top of that (especially
>>>>> given this
>>>>> end result).
>>>>>
>>>>> At a wild guess, I'm wondering if this may be sufficient:
>>>>>
>>>>> ----->8-----
>>>>> diff --git a/drivers/acpi/osl.c b/drivers/acpi/osl.c
>>>>> index 327e1b4eb6b0..f5d26b102fbe 100644
>>>>> --- a/drivers/acpi/osl.c
>>>>> +++ b/drivers/acpi/osl.c
>>>>> @@ -277,7 +277,7 @@ acpi_map_lookup_virt(void __iomem *virt,
>>>>> acpi_size size)
>>>>> return NULL;
>>>>> }
>>>>>
>>>>> -#if defined(CONFIG_IA64) || defined(CONFIG_ARM64)
>>>>> +#if defined(CONFIG_IA64)
>>>>> /* ioremap will take care of cache attributes */
>>>>> #define should_use_kmap(pfn) 0
>>>>> #else
>>>>> -----8<-----
>>>>
>>>> I thought the same but shouldn't acpi_os_ioremap() map it with the right
>>>> attributes? It uses the EFI maps to check what kind of memory this is.
>>>
>>> Oh crikey, I missed that branch of the rabbit hole... I guess that must
>>> mean that the tables being poked here are *not* covered by the EFI
>>> memory map, so page_is_ram() is unlikely to help either :(
>>
>> After picking through the UEFI spec I think I've now got a clearer picture
>> of what's happening, but I'm not sure where it goes from here...
>>
>> The spec implies that it *is* legitimate for runtime-loaded ACPI tables to
>> lie outside the EFI memory map, and that case they must be assumed to be
>> uncached, so the behaviour of acpi_os_ioremap() is correct.
>
> I'd agree with the reasoning, it would be good to pinpoint whether
> that's what actually triggers the issue.
>
> I'd like to replicate it if possible (it is TX2 HW but firmware
> config is likely to differ from the HW I have at hand), the
> test command line that triggers the fault would be useful as
> a starting point.
>
> Furthermore, is this a v5.13-rc* regression ? If so it would be
> good to bisect it - I can't recollect arm64 changes that could
> have introduced this regression in the last cycle but I may have
> missed something.
The actual change which has brought this to light is the update to
arm64's memcpy() routine for 5.13 - the new version is more aggressive
at making unaligned loads from the source buffer, so now triggers
alignment faults more readily when (wrongly) used on iomem mappings in
places that were getting away with it by chance under the previous
implementation (see also [1], for example).
Thanks,
Robin.
[1]
https://lore.kernel.org/linux-arm-kernel/20210608153344.3813661-1-narmstrong@baylibre.com/
>> Given the definition of uncached for arm64 memory types though, that
>> means that callers of acpi_os_map_memory() still have to be prepared
>> to get an __iomem pointer back even if they know they're mapping a
>> table rather than some random bit of MMIO for an AML method.
>>
>> Therefore in this case it seems the blame lies partway between
>> acpi_os_map_memory() for casting away __iomem and acpi_data_show() for
>> letting an arbitrary offset lead to an arbitrarily-aligned memcpy(), but I
>> don't know what the best way to fix it is. Either way I've satisfied myself
>> that it's not an issue with the arm64 code itself - I do wonder whether this
>> might also be a problem on IA-64 given ACPI_MISALIGNMENT_NOT_SUPPORTED, and
>> I guess RISC-V may have alignment concerns as well.
>
> Yes agreed but see above, this code has been there for aeons if it
> is a v5.13-rc* regression it must be something else that actually
> triggered it (test/FW config).
>
> Thanks for looking into this.
>
> Lorenzo
>
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: ❌ FAIL: Test report for kernel 5.13.0-rc7 (arm-next, 8ab9b1a9)
@ 2021-06-29 15:14 ` Robin Murphy
0 siblings, 0 replies; 39+ messages in thread
From: Robin Murphy @ 2021-06-29 15:14 UTC (permalink / raw)
To: Lorenzo Pieralisi
Cc: ACPI Devel Maling List, Veronika Kabatova, Will Deacon,
CKI Project, Mark Rutland, Memory Management, skt-results-master,
Jeff Bastian, Jan Stancek, Linux ARM, rjw, lenb, guohanjun,
sudeep.holla, ardb, Catalin Marinas, lv.zheng, tony.luck
On 2021-06-29 15:44, Lorenzo Pieralisi wrote:
> On Tue, Jun 29, 2021 at 12:48:14PM +0100, Robin Murphy wrote:
>> [ +ACPI audience ]
>>
>> On 2021-06-25 12:15, Robin Murphy wrote:
>>> On 2021-06-25 12:09, Catalin Marinas wrote:
>>>> On Fri, Jun 25, 2021 at 12:02:52PM +0100, Robin Murphy wrote:
>>>>> On 2021-06-25 10:52, Veronika Kabatova wrote:
>>>>> [...]
>>>>>>>> ❌ stress: stress-ng
>>>>>>>
>>>>>>> Oh no, this looks like another alignment fault in memcpy:
>>>>>>>
>>>>>>> [13330.651903] Unable to handle kernel paging request at
>>>>>>> virtual address ffff8000534705ff
>>>>>>> [13330.651914] Mem abort info:
>>>>>>> [13330.651918] ESR = 0x96000021
>>>>>>> [13330.651922] EC = 0x25: DABT (current EL), IL = 32 bits
>>>>>>> [13330.651928] SET = 0, FnV = 0
>>>>>>> [13330.651931] EA = 0, S1PTW = 0
>>>>>>> [13330.651933] FSC = 0x21: alignment fault
>>>>>>> [13330.651938] Data abort info:
>>>>>>> [13330.651940] ISV = 0, ISS = 0x00000021
>>>>>>> [13330.651941] CM = 0, WnR = 0
>>>>>>> [13330.651943] swapper pgtable: 4k pages, 48-bit VAs,
>>>>>>> pgdp=00000000f3e6b000
>>>>>>> [13330.651945] [ffff8000534705ff] pgd=1000008ffcfff003,
>>>>>>> p4d=1000008ffcfff003, pud=100000088e57d003,
>>>>>>> pmd=10000008d0aeb003, pte=006800008021370f
>>>>>>> [13330.651956] Internal error: Oops: 96000021 [#1] SMP
>>>>>>> [13330.651961] Modules linked in: unix_diag binfmt_misc
>>>>>>> fcrypt sm4_generic crc32_generic md4 michael_mic
>>>>>>> nhpoly1305_neon nhpoly1305 poly1305_generic libpoly1305
>>>>>>> poly1305_neon rmd160 sha3_generic sm3_generic
>>>>>>> streebog_generic wp512 blowfish_generic blowfish_common
>>>>>>> cast5_generic des_generic libdes chacha_generic
>>>>>>> chacha_neon libchacha camellia_generic cast6_generic
>>>>>>> cast_common serpent_generic twofish_generic
>>>>>>> twofish_common dm_thin_pool dm_persistent_data
>>>>>>> dm_bio_prison nvme nvme_core ipmi_watchdog ipmi_poweroff
>>>>>>> loop tun af_key crypto_user scsi_transport_iscsi
>>>>>>> xt_multiport ip_gre ip_tunnel gre overlay xt_CONNSECMARK
>>>>>>> xt_SECMARK nft_counter xt_state xt_conntrack nft_compat
>>>>>>> ah6 ah4 nft_objref nft_ct nf_conntrack nf_defrag_ipv6
>>>>>>> nf_defrag_ipv4 nf_tables nfnetlink jfs sctp
>>>>>>> ip6_udp_tunnel udp_tunnel dm_log_writes dm_flakey rfkill
>>>>>>> mlx5_ib ib_uverbs ib_core sunrpc coresight_etm4x
>>>>>>> i2c_smbus coresight_replicator coresight_tpiu
>>>>>>> coresight_tmc joydev mlx5_core acpi_ipmi psample
>>>>>>> ipmi_ssif mlxfw !
>>>>>>> ipmi_devintf
>>>>>>> [13330.652076] ipmi_msghandler coresight_funnel
>>>>>>> thunderx2_pmu coresight vfat fat fuse zram ip_tables xfs
>>>>>>> ast crct10dif_ce i2c_algo_bit ghash_ce drm_vram_helper
>>>>>>> drm_kms_helper syscopyarea sysfillrect sysimgblt
>>>>>>> fb_sys_fops cec drm_ttm_helper ttm drm gpio_xlp
>>>>>>> i2c_xlp9xx uas usb_storage aes_neon_bs [last unloaded:
>>>>>>> nvmet]
>>>>>>> [13330.652123] CPU: 115 PID: 188446 Comm: stress-ng
>>>>>>> Tainted: G OEL 5.13.0-rc7 #1
>>>>>>> [13330.652129] Hardware name: HPE Apollo 70
>>>>>>> /C01_APACHE_MB , BIOS L50_5.13_1.15 05/08/2020
>>>>>>> [13330.652133] pstate: 80400009 (Nzcv daif +PAN -UAO -TCO BTYPE=--)
>>>>>>> [13330.652139] pc : __memcpy+0x168/0x250
>>>>>>> [13330.652150] lr : memory_read_from_buffer+0x58/0x80
>>>>>>> [13330.652161] sp : ffff800063ef3c20
>>>>>>> [13330.652163] x29: ffff800063ef3c20 x28:
>>>>>>> ffff0008b1380000 x27: 0000000000000000
>>>>>>> [13330.652170] x26: 0000000000000000 x25:
>>>>>>> 0000000000000000 x24: ffff00080a960fe0
>>>>>>> [13330.652176] x23: ffff800063ef3d28 x22:
>>>>>>> 000000000000063f x21: ffff800063ef3c88
>>>>>>> [13330.652181] x20: 000000000000063f x19:
>>>>>>> 000000000000063f x18: 0000000000000000
>>>>>>> [13330.652186] x17: 0000000000000000 x16:
>>>>>>> 0000000000000000 x15: 0000000000000000
>>>>>>> [13330.652191] x14: 0000000000000000 x13:
>>>>>>> 0000000000000000 x12: 0000000000000000
>>>>>>> [13330.652196] x11: 0000000000000000 x10:
>>>>>>> 0000000000000000 x9 : 0000000000000000
>>>>>>> [13330.652200] x8 : 0000000000000000 x7 :
>>>>>>> 0000000000000000 x6 : 0000000000000000
>>>>>>> [13330.652206] x5 : ffff000d0fb0063f x4 :
>>>>>>> ffff80005347063f x3 : ffff000d0fb005c0
>>>>>>> [13330.652212] x2 : ffffffffffffffef x1 :
>>>>>>> ffff800053470600 x0 : ffff000d0fb00000
>>>>>>> [13330.652218] Call trace:
>>>>>>> [13330.652221] __memcpy+0x168/0x250
>>>>>>> [13330.652225] acpi_data_show+0x5c/0x8c
>>>>>>> [13330.652232] sysfs_kf_bin_read+0x78/0xa0
>>>>>>> [13330.652238] kernfs_file_read_iter+0x9c/0x1a4
>>>>>>> [13330.652241] kernfs_fop_read_iter+0x34/0x50
>>>>>>> [13330.652244] new_sync_read+0xdc/0x154
>>>>>>> [13330.652253] vfs_read+0x158/0x1e4
>>>>>>> [13330.652260] ksys_read+0x64/0xec
>>>>>>> [13330.652266] __arm64_sys_read+0x28/0x34
>>>>>>> [13330.652273] invoke_syscall+0x50/0x120
>>>>>>> [13330.652280] el0_svc_common.constprop.0+0x4c/0xd4
>>>>>>> [13330.652284] do_el0_svc+0x30/0x9c
>>>>>>> [13330.652286] el0_svc+0x2c/0x54
>>>>>>> [13330.652294] el0t_64_sync_handler+0x1a4/0x1b0
>>>>>>> [13330.652296] el0t_64_sync+0x19c/0x1a0
>>>>>>> [13330.652303] Code: a984346c a9c4342c f1010042 54fffee8 (a97c3c8e)
>>>>>>> [13330.652307] ---[ end trace 227d4380f57145d4 ]---
>>>>>>>
>>>>>>> So maybe this issue isn't limited to weird modules, after all...
>>>>>>>
>>>>>>
>>>>>> It ran on the machine from the same set that we were able to reproduce
>>>>>> it on previously. If you or anyone else have an idea on how
>>>>>> to stabilize
>>>>>> the reproducibility or have a debug patch we'll be happy to try it.
>>>>>
>>>>> Possibly it depends on the individual machines' firmware exactly how the
>>>>> relevant bits of their ACPI tables are aligned in memory?
>>>>>
>>>>> I've started digging into that callstack - it may not be a
>>>>> "weird module"
>>>>> but it's definitely crusty ACPI code... a238317ce818 ("ACPI: Clean up
>>>>> acpi_os_map/unmap_memory() to eliminate __iomem.") looks frankly a bit
>>>>> questionable in its decision to blithely cast away __iomem, but then the
>>>>> rationale in aafc65c731fe ("ACPI: add arm64 to the platforms that use
>>>>> ioremap") seems particularly dubious on top of that (especially
>>>>> given this
>>>>> end result).
>>>>>
>>>>> At a wild guess, I'm wondering if this may be sufficient:
>>>>>
>>>>> ----->8-----
>>>>> diff --git a/drivers/acpi/osl.c b/drivers/acpi/osl.c
>>>>> index 327e1b4eb6b0..f5d26b102fbe 100644
>>>>> --- a/drivers/acpi/osl.c
>>>>> +++ b/drivers/acpi/osl.c
>>>>> @@ -277,7 +277,7 @@ acpi_map_lookup_virt(void __iomem *virt,
>>>>> acpi_size size)
>>>>> return NULL;
>>>>> }
>>>>>
>>>>> -#if defined(CONFIG_IA64) || defined(CONFIG_ARM64)
>>>>> +#if defined(CONFIG_IA64)
>>>>> /* ioremap will take care of cache attributes */
>>>>> #define should_use_kmap(pfn) 0
>>>>> #else
>>>>> -----8<-----
>>>>
>>>> I thought the same but shouldn't acpi_os_ioremap() map it with the right
>>>> attributes? It uses the EFI maps to check what kind of memory this is.
>>>
>>> Oh crikey, I missed that branch of the rabbit hole... I guess that must
>>> mean that the tables being poked here are *not* covered by the EFI
>>> memory map, so page_is_ram() is unlikely to help either :(
>>
>> After picking through the UEFI spec I think I've now got a clearer picture
>> of what's happening, but I'm not sure where it goes from here...
>>
>> The spec implies that it *is* legitimate for runtime-loaded ACPI tables to
>> lie outside the EFI memory map, and that case they must be assumed to be
>> uncached, so the behaviour of acpi_os_ioremap() is correct.
>
> I'd agree with the reasoning, it would be good to pinpoint whether
> that's what actually triggers the issue.
>
> I'd like to replicate it if possible (it is TX2 HW but firmware
> config is likely to differ from the HW I have at hand), the
> test command line that triggers the fault would be useful as
> a starting point.
>
> Furthermore, is this a v5.13-rc* regression ? If so it would be
> good to bisect it - I can't recollect arm64 changes that could
> have introduced this regression in the last cycle but I may have
> missed something.
The actual change which has brought this to light is the update to
arm64's memcpy() routine for 5.13 - the new version is more aggressive
at making unaligned loads from the source buffer, so now triggers
alignment faults more readily when (wrongly) used on iomem mappings in
places that were getting away with it by chance under the previous
implementation (see also [1], for example).
Thanks,
Robin.
[1]
https://lore.kernel.org/linux-arm-kernel/20210608153344.3813661-1-narmstrong@baylibre.com/
>> Given the definition of uncached for arm64 memory types though, that
>> means that callers of acpi_os_map_memory() still have to be prepared
>> to get an __iomem pointer back even if they know they're mapping a
>> table rather than some random bit of MMIO for an AML method.
>>
>> Therefore in this case it seems the blame lies partway between
>> acpi_os_map_memory() for casting away __iomem and acpi_data_show() for
>> letting an arbitrary offset lead to an arbitrarily-aligned memcpy(), but I
>> don't know what the best way to fix it is. Either way I've satisfied myself
>> that it's not an issue with the arm64 code itself - I do wonder whether this
>> might also be a problem on IA-64 given ACPI_MISALIGNMENT_NOT_SUPPORTED, and
>> I guess RISC-V may have alignment concerns as well.
>
> Yes agreed but see above, this code has been there for aeons if it
> is a v5.13-rc* regression it must be something else that actually
> triggered it (test/FW config).
>
> Thanks for looking into this.
>
> Lorenzo
>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: ❌ FAIL: Test report for kernel 5.13.0-rc7 (arm-next, 8ab9b1a9)
2021-06-29 15:14 ` Robin Murphy
@ 2021-06-29 16:35 ` Catalin Marinas
-1 siblings, 0 replies; 39+ messages in thread
From: Catalin Marinas @ 2021-06-29 16:35 UTC (permalink / raw)
To: Robin Murphy
Cc: Lorenzo Pieralisi, ACPI Devel Maling List, Veronika Kabatova,
Will Deacon, CKI Project, Mark Rutland, Memory Management,
skt-results-master, Jeff Bastian, Jan Stancek, Linux ARM, rjw,
lenb, guohanjun, sudeep.holla, ardb, lv.zheng, tony.luck
On Tue, Jun 29, 2021 at 04:14:55PM +0100, Robin Murphy wrote:
> On 2021-06-29 15:44, Lorenzo Pieralisi wrote:
> > On Tue, Jun 29, 2021 at 12:48:14PM +0100, Robin Murphy wrote:
> > > [ +ACPI audience ]
> > >
> > > On 2021-06-25 12:15, Robin Murphy wrote:
> > > > On 2021-06-25 12:09, Catalin Marinas wrote:
> > > > > On Fri, Jun 25, 2021 at 12:02:52PM +0100, Robin Murphy wrote:
> > > > > > On 2021-06-25 10:52, Veronika Kabatova wrote:
> > > > > > [...]
> > > > > > > > > ❌ stress: stress-ng
> > > > > > > >
> > > > > > > > Oh no, this looks like another alignment fault in memcpy:
> > > > > > > >
> > > > > > > > [13330.651903] Unable to handle kernel paging request at
> > > > > > > > virtual address ffff8000534705ff
[...]
> > > > > > > > [13330.652218] Call trace:
> > > > > > > > [13330.652221] __memcpy+0x168/0x250
> > > > > > > > [13330.652225] acpi_data_show+0x5c/0x8c
> > > > > > > > [13330.652232] sysfs_kf_bin_read+0x78/0xa0
> > > > > > > > [13330.652238] kernfs_file_read_iter+0x9c/0x1a4
> > > > > > > > [13330.652241] kernfs_fop_read_iter+0x34/0x50
> > > > > > > > [13330.652244] new_sync_read+0xdc/0x154
> > > > > > > > [13330.652253] vfs_read+0x158/0x1e4
> > > > > > > > [13330.652260] ksys_read+0x64/0xec
> > > > > > > > [13330.652266] __arm64_sys_read+0x28/0x34
> > > > > > > > [13330.652273] invoke_syscall+0x50/0x120
> > > > > > > > [13330.652280] el0_svc_common.constprop.0+0x4c/0xd4
> > > > > > > > [13330.652284] do_el0_svc+0x30/0x9c
> > > > > > > > [13330.652286] el0_svc+0x2c/0x54
> > > > > > > > [13330.652294] el0t_64_sync_handler+0x1a4/0x1b0
> > > > > > > > [13330.652296] el0t_64_sync+0x19c/0x1a0
> > > > > > > > [13330.652303] Code: a984346c a9c4342c f1010042 54fffee8 (a97c3c8e)
> > > > > > > > [13330.652307] ---[ end trace 227d4380f57145d4 ]---
> > > > > > > >
> > > > > > > > So maybe this issue isn't limited to weird modules, after all...
> > > > > > >
> > > > > > > It ran on the machine from the same set that we were able to reproduce
> > > > > > > it on previously. If you or anyone else have an idea on how
> > > > > > > to stabilize the reproducibility or have a debug patch we'll be happy to try it.
> > > > > >
> > > > > > Possibly it depends on the individual machines' firmware exactly how the
> > > > > > relevant bits of their ACPI tables are aligned in memory?
> > > > > >
> > > > > > I've started digging into that callstack - it may not be a "weird module"
> > > > > > but it's definitely crusty ACPI code... a238317ce818 ("ACPI: Clean up
> > > > > > acpi_os_map/unmap_memory() to eliminate __iomem.") looks frankly a bit
> > > > > > questionable in its decision to blithely cast away __iomem, but then the
> > > > > > rationale in aafc65c731fe ("ACPI: add arm64 to the platforms that use
> > > > > > ioremap") seems particularly dubious on top of that (especially
> > > > > > given this end result).
[...]
> > > After picking through the UEFI spec I think I've now got a clearer picture
> > > of what's happening, but I'm not sure where it goes from here...
> > >
> > > The spec implies that it *is* legitimate for runtime-loaded ACPI tables to
> > > lie outside the EFI memory map, and that case they must be assumed to be
> > > uncached, so the behaviour of acpi_os_ioremap() is correct.
> >
> > I'd agree with the reasoning, it would be good to pinpoint whether
> > that's what actually triggers the issue.
> >
> > I'd like to replicate it if possible (it is TX2 HW but firmware
> > config is likely to differ from the HW I have at hand), the
> > test command line that triggers the fault would be useful as
> > a starting point.
> >
> > Furthermore, is this a v5.13-rc* regression ? If so it would be
> > good to bisect it - I can't recollect arm64 changes that could
> > have introduced this regression in the last cycle but I may have
> > missed something.
>
> The actual change which has brought this to light is the update to arm64's
> memcpy() routine for 5.13 - the new version is more aggressive at making
> unaligned loads from the source buffer, so now triggers alignment faults
> more readily when (wrongly) used on iomem mappings in places that were
> getting away with it by chance under the previous implementation (see also
> [1], for example).
I wouldn't revert any of the memcpy() stuff as it just uncovered an
existing bug in how the ACPI tables are handled. Could we actually hit
a similar issue with C code parsing the ACPI tables?
Is there a way to map the ACPI tables as Normal Noncacheable
(ioremap_wc)? Presumably no-one sane would place ACPI tables in memory
that's sensitive to the access size.
--
Catalin
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: ❌ FAIL: Test report for kernel 5.13.0-rc7 (arm-next, 8ab9b1a9)
@ 2021-06-29 16:35 ` Catalin Marinas
0 siblings, 0 replies; 39+ messages in thread
From: Catalin Marinas @ 2021-06-29 16:35 UTC (permalink / raw)
To: Robin Murphy
Cc: Lorenzo Pieralisi, ACPI Devel Maling List, Veronika Kabatova,
Will Deacon, CKI Project, Mark Rutland, Memory Management,
skt-results-master, Jeff Bastian, Jan Stancek, Linux ARM, rjw,
lenb, guohanjun, sudeep.holla, ardb, lv.zheng, tony.luck
On Tue, Jun 29, 2021 at 04:14:55PM +0100, Robin Murphy wrote:
> On 2021-06-29 15:44, Lorenzo Pieralisi wrote:
> > On Tue, Jun 29, 2021 at 12:48:14PM +0100, Robin Murphy wrote:
> > > [ +ACPI audience ]
> > >
> > > On 2021-06-25 12:15, Robin Murphy wrote:
> > > > On 2021-06-25 12:09, Catalin Marinas wrote:
> > > > > On Fri, Jun 25, 2021 at 12:02:52PM +0100, Robin Murphy wrote:
> > > > > > On 2021-06-25 10:52, Veronika Kabatova wrote:
> > > > > > [...]
> > > > > > > > > ❌ stress: stress-ng
> > > > > > > >
> > > > > > > > Oh no, this looks like another alignment fault in memcpy:
> > > > > > > >
> > > > > > > > [13330.651903] Unable to handle kernel paging request at
> > > > > > > > virtual address ffff8000534705ff
[...]
> > > > > > > > [13330.652218] Call trace:
> > > > > > > > [13330.652221] __memcpy+0x168/0x250
> > > > > > > > [13330.652225] acpi_data_show+0x5c/0x8c
> > > > > > > > [13330.652232] sysfs_kf_bin_read+0x78/0xa0
> > > > > > > > [13330.652238] kernfs_file_read_iter+0x9c/0x1a4
> > > > > > > > [13330.652241] kernfs_fop_read_iter+0x34/0x50
> > > > > > > > [13330.652244] new_sync_read+0xdc/0x154
> > > > > > > > [13330.652253] vfs_read+0x158/0x1e4
> > > > > > > > [13330.652260] ksys_read+0x64/0xec
> > > > > > > > [13330.652266] __arm64_sys_read+0x28/0x34
> > > > > > > > [13330.652273] invoke_syscall+0x50/0x120
> > > > > > > > [13330.652280] el0_svc_common.constprop.0+0x4c/0xd4
> > > > > > > > [13330.652284] do_el0_svc+0x30/0x9c
> > > > > > > > [13330.652286] el0_svc+0x2c/0x54
> > > > > > > > [13330.652294] el0t_64_sync_handler+0x1a4/0x1b0
> > > > > > > > [13330.652296] el0t_64_sync+0x19c/0x1a0
> > > > > > > > [13330.652303] Code: a984346c a9c4342c f1010042 54fffee8 (a97c3c8e)
> > > > > > > > [13330.652307] ---[ end trace 227d4380f57145d4 ]---
> > > > > > > >
> > > > > > > > So maybe this issue isn't limited to weird modules, after all...
> > > > > > >
> > > > > > > It ran on the machine from the same set that we were able to reproduce
> > > > > > > it on previously. If you or anyone else have an idea on how
> > > > > > > to stabilize the reproducibility or have a debug patch we'll be happy to try it.
> > > > > >
> > > > > > Possibly it depends on the individual machines' firmware exactly how the
> > > > > > relevant bits of their ACPI tables are aligned in memory?
> > > > > >
> > > > > > I've started digging into that callstack - it may not be a "weird module"
> > > > > > but it's definitely crusty ACPI code... a238317ce818 ("ACPI: Clean up
> > > > > > acpi_os_map/unmap_memory() to eliminate __iomem.") looks frankly a bit
> > > > > > questionable in its decision to blithely cast away __iomem, but then the
> > > > > > rationale in aafc65c731fe ("ACPI: add arm64 to the platforms that use
> > > > > > ioremap") seems particularly dubious on top of that (especially
> > > > > > given this end result).
[...]
> > > After picking through the UEFI spec I think I've now got a clearer picture
> > > of what's happening, but I'm not sure where it goes from here...
> > >
> > > The spec implies that it *is* legitimate for runtime-loaded ACPI tables to
> > > lie outside the EFI memory map, and that case they must be assumed to be
> > > uncached, so the behaviour of acpi_os_ioremap() is correct.
> >
> > I'd agree with the reasoning, it would be good to pinpoint whether
> > that's what actually triggers the issue.
> >
> > I'd like to replicate it if possible (it is TX2 HW but firmware
> > config is likely to differ from the HW I have at hand), the
> > test command line that triggers the fault would be useful as
> > a starting point.
> >
> > Furthermore, is this a v5.13-rc* regression ? If so it would be
> > good to bisect it - I can't recollect arm64 changes that could
> > have introduced this regression in the last cycle but I may have
> > missed something.
>
> The actual change which has brought this to light is the update to arm64's
> memcpy() routine for 5.13 - the new version is more aggressive at making
> unaligned loads from the source buffer, so now triggers alignment faults
> more readily when (wrongly) used on iomem mappings in places that were
> getting away with it by chance under the previous implementation (see also
> [1], for example).
I wouldn't revert any of the memcpy() stuff as it just uncovered an
existing bug in how the ACPI tables are handled. Could we actually hit
a similar issue with C code parsing the ACPI tables?
Is there a way to map the ACPI tables as Normal Noncacheable
(ioremap_wc)? Presumably no-one sane would place ACPI tables in memory
that's sensitive to the access size.
--
Catalin
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: ❌ FAIL: Test report for kernel 5.13.0-rc7 (arm-next, 8ab9b1a9)
2021-06-29 15:14 ` Robin Murphy
(?)
(?)
@ 2021-06-29 17:03 ` Veronika Kabatova
2021-06-29 17:27 ` Robin Murphy
-1 siblings, 1 reply; 39+ messages in thread
From: Veronika Kabatova @ 2021-06-29 17:03 UTC (permalink / raw)
To: Robin Murphy
Cc: Lorenzo Pieralisi, ACPI Devel Maling List, Will Deacon,
CKI Project, Mark Rutland, Memory Management, skt-results-master,
Jeff Bastian, Jan Stancek, Linux ARM, rjw, lenb, guohanjun,
sudeep.holla, Ard Biesheuvel, Catalin Marinas, lv.zheng,
tony.luck
On Tue, Jun 29, 2021 at 5:15 PM Robin Murphy <robin.murphy@arm.com> wrote:
>
> On 2021-06-29 15:44, Lorenzo Pieralisi wrote:
> > On Tue, Jun 29, 2021 at 12:48:14PM +0100, Robin Murphy wrote:
> >> [ +ACPI audience ]
> >>
> >> On 2021-06-25 12:15, Robin Murphy wrote:
> >>> On 2021-06-25 12:09, Catalin Marinas wrote:
> >>>> On Fri, Jun 25, 2021 at 12:02:52PM +0100, Robin Murphy wrote:
> >>>>> On 2021-06-25 10:52, Veronika Kabatova wrote:
> >>>>> [...]
> >>>>>>>> ❌ stress: stress-ng
> >>>>>>>
> >>>>>>> Oh no, this looks like another alignment fault in memcpy:
> >>>>>>>
> >>>>>>> [13330.651903] Unable to handle kernel paging request at
> >>>>>>> virtual address ffff8000534705ff
> >>>>>>> [13330.651914] Mem abort info:
> >>>>>>> [13330.651918] ESR = 0x96000021
> >>>>>>> [13330.651922] EC = 0x25: DABT (current EL), IL = 32 bits
> >>>>>>> [13330.651928] SET = 0, FnV = 0
> >>>>>>> [13330.651931] EA = 0, S1PTW = 0
> >>>>>>> [13330.651933] FSC = 0x21: alignment fault
> >>>>>>> [13330.651938] Data abort info:
> >>>>>>> [13330.651940] ISV = 0, ISS = 0x00000021
> >>>>>>> [13330.651941] CM = 0, WnR = 0
> >>>>>>> [13330.651943] swapper pgtable: 4k pages, 48-bit VAs,
> >>>>>>> pgdp=00000000f3e6b000
> >>>>>>> [13330.651945] [ffff8000534705ff] pgd=1000008ffcfff003,
> >>>>>>> p4d=1000008ffcfff003, pud=100000088e57d003,
> >>>>>>> pmd=10000008d0aeb003, pte=006800008021370f
> >>>>>>> [13330.651956] Internal error: Oops: 96000021 [#1] SMP
> >>>>>>> [13330.651961] Modules linked in: unix_diag binfmt_misc
> >>>>>>> fcrypt sm4_generic crc32_generic md4 michael_mic
> >>>>>>> nhpoly1305_neon nhpoly1305 poly1305_generic libpoly1305
> >>>>>>> poly1305_neon rmd160 sha3_generic sm3_generic
> >>>>>>> streebog_generic wp512 blowfish_generic blowfish_common
> >>>>>>> cast5_generic des_generic libdes chacha_generic
> >>>>>>> chacha_neon libchacha camellia_generic cast6_generic
> >>>>>>> cast_common serpent_generic twofish_generic
> >>>>>>> twofish_common dm_thin_pool dm_persistent_data
> >>>>>>> dm_bio_prison nvme nvme_core ipmi_watchdog ipmi_poweroff
> >>>>>>> loop tun af_key crypto_user scsi_transport_iscsi
> >>>>>>> xt_multiport ip_gre ip_tunnel gre overlay xt_CONNSECMARK
> >>>>>>> xt_SECMARK nft_counter xt_state xt_conntrack nft_compat
> >>>>>>> ah6 ah4 nft_objref nft_ct nf_conntrack nf_defrag_ipv6
> >>>>>>> nf_defrag_ipv4 nf_tables nfnetlink jfs sctp
> >>>>>>> ip6_udp_tunnel udp_tunnel dm_log_writes dm_flakey rfkill
> >>>>>>> mlx5_ib ib_uverbs ib_core sunrpc coresight_etm4x
> >>>>>>> i2c_smbus coresight_replicator coresight_tpiu
> >>>>>>> coresight_tmc joydev mlx5_core acpi_ipmi psample
> >>>>>>> ipmi_ssif mlxfw !
> >>>>>>> ipmi_devintf
> >>>>>>> [13330.652076] ipmi_msghandler coresight_funnel
> >>>>>>> thunderx2_pmu coresight vfat fat fuse zram ip_tables xfs
> >>>>>>> ast crct10dif_ce i2c_algo_bit ghash_ce drm_vram_helper
> >>>>>>> drm_kms_helper syscopyarea sysfillrect sysimgblt
> >>>>>>> fb_sys_fops cec drm_ttm_helper ttm drm gpio_xlp
> >>>>>>> i2c_xlp9xx uas usb_storage aes_neon_bs [last unloaded:
> >>>>>>> nvmet]
> >>>>>>> [13330.652123] CPU: 115 PID: 188446 Comm: stress-ng
> >>>>>>> Tainted: G OEL 5.13.0-rc7 #1
> >>>>>>> [13330.652129] Hardware name: HPE Apollo 70
> >>>>>>> /C01_APACHE_MB , BIOS L50_5.13_1.15 05/08/2020
> >>>>>>> [13330.652133] pstate: 80400009 (Nzcv daif +PAN -UAO -TCO BTYPE=--)
> >>>>>>> [13330.652139] pc : __memcpy+0x168/0x250
> >>>>>>> [13330.652150] lr : memory_read_from_buffer+0x58/0x80
> >>>>>>> [13330.652161] sp : ffff800063ef3c20
> >>>>>>> [13330.652163] x29: ffff800063ef3c20 x28:
> >>>>>>> ffff0008b1380000 x27: 0000000000000000
> >>>>>>> [13330.652170] x26: 0000000000000000 x25:
> >>>>>>> 0000000000000000 x24: ffff00080a960fe0
> >>>>>>> [13330.652176] x23: ffff800063ef3d28 x22:
> >>>>>>> 000000000000063f x21: ffff800063ef3c88
> >>>>>>> [13330.652181] x20: 000000000000063f x19:
> >>>>>>> 000000000000063f x18: 0000000000000000
> >>>>>>> [13330.652186] x17: 0000000000000000 x16:
> >>>>>>> 0000000000000000 x15: 0000000000000000
> >>>>>>> [13330.652191] x14: 0000000000000000 x13:
> >>>>>>> 0000000000000000 x12: 0000000000000000
> >>>>>>> [13330.652196] x11: 0000000000000000 x10:
> >>>>>>> 0000000000000000 x9 : 0000000000000000
> >>>>>>> [13330.652200] x8 : 0000000000000000 x7 :
> >>>>>>> 0000000000000000 x6 : 0000000000000000
> >>>>>>> [13330.652206] x5 : ffff000d0fb0063f x4 :
> >>>>>>> ffff80005347063f x3 : ffff000d0fb005c0
> >>>>>>> [13330.652212] x2 : ffffffffffffffef x1 :
> >>>>>>> ffff800053470600 x0 : ffff000d0fb00000
> >>>>>>> [13330.652218] Call trace:
> >>>>>>> [13330.652221] __memcpy+0x168/0x250
> >>>>>>> [13330.652225] acpi_data_show+0x5c/0x8c
> >>>>>>> [13330.652232] sysfs_kf_bin_read+0x78/0xa0
> >>>>>>> [13330.652238] kernfs_file_read_iter+0x9c/0x1a4
> >>>>>>> [13330.652241] kernfs_fop_read_iter+0x34/0x50
> >>>>>>> [13330.652244] new_sync_read+0xdc/0x154
> >>>>>>> [13330.652253] vfs_read+0x158/0x1e4
> >>>>>>> [13330.652260] ksys_read+0x64/0xec
> >>>>>>> [13330.652266] __arm64_sys_read+0x28/0x34
> >>>>>>> [13330.652273] invoke_syscall+0x50/0x120
> >>>>>>> [13330.652280] el0_svc_common.constprop.0+0x4c/0xd4
> >>>>>>> [13330.652284] do_el0_svc+0x30/0x9c
> >>>>>>> [13330.652286] el0_svc+0x2c/0x54
> >>>>>>> [13330.652294] el0t_64_sync_handler+0x1a4/0x1b0
> >>>>>>> [13330.652296] el0t_64_sync+0x19c/0x1a0
> >>>>>>> [13330.652303] Code: a984346c a9c4342c f1010042 54fffee8 (a97c3c8e)
> >>>>>>> [13330.652307] ---[ end trace 227d4380f57145d4 ]---
> >>>>>>>
> >>>>>>> So maybe this issue isn't limited to weird modules, after all...
> >>>>>>>
> >>>>>>
> >>>>>> It ran on the machine from the same set that we were able to reproduce
> >>>>>> it on previously. If you or anyone else have an idea on how
> >>>>>> to stabilize
> >>>>>> the reproducibility or have a debug patch we'll be happy to try it.
> >>>>>
> >>>>> Possibly it depends on the individual machines' firmware exactly how the
> >>>>> relevant bits of their ACPI tables are aligned in memory?
> >>>>>
> >>>>> I've started digging into that callstack - it may not be a
> >>>>> "weird module"
> >>>>> but it's definitely crusty ACPI code... a238317ce818 ("ACPI: Clean up
> >>>>> acpi_os_map/unmap_memory() to eliminate __iomem.") looks frankly a bit
> >>>>> questionable in its decision to blithely cast away __iomem, but then the
> >>>>> rationale in aafc65c731fe ("ACPI: add arm64 to the platforms that use
> >>>>> ioremap") seems particularly dubious on top of that (especially
> >>>>> given this
> >>>>> end result).
> >>>>>
> >>>>> At a wild guess, I'm wondering if this may be sufficient:
> >>>>>
> >>>>> ----->8-----
> >>>>> diff --git a/drivers/acpi/osl.c b/drivers/acpi/osl.c
> >>>>> index 327e1b4eb6b0..f5d26b102fbe 100644
> >>>>> --- a/drivers/acpi/osl.c
> >>>>> +++ b/drivers/acpi/osl.c
> >>>>> @@ -277,7 +277,7 @@ acpi_map_lookup_virt(void __iomem *virt,
> >>>>> acpi_size size)
> >>>>> return NULL;
> >>>>> }
> >>>>>
> >>>>> -#if defined(CONFIG_IA64) || defined(CONFIG_ARM64)
> >>>>> +#if defined(CONFIG_IA64)
> >>>>> /* ioremap will take care of cache attributes */
> >>>>> #define should_use_kmap(pfn) 0
> >>>>> #else
> >>>>> -----8<-----
> >>>>
> >>>> I thought the same but shouldn't acpi_os_ioremap() map it with the right
> >>>> attributes? It uses the EFI maps to check what kind of memory this is.
> >>>
> >>> Oh crikey, I missed that branch of the rabbit hole... I guess that must
> >>> mean that the tables being poked here are *not* covered by the EFI
> >>> memory map, so page_is_ram() is unlikely to help either :(
> >>
> >> After picking through the UEFI spec I think I've now got a clearer picture
> >> of what's happening, but I'm not sure where it goes from here...
> >>
> >> The spec implies that it *is* legitimate for runtime-loaded ACPI tables to
> >> lie outside the EFI memory map, and that case they must be assumed to be
> >> uncached, so the behaviour of acpi_os_ioremap() is correct.
> >
> > I'd agree with the reasoning, it would be good to pinpoint whether
> > that's what actually triggers the issue.
> >
> > I'd like to replicate it if possible (it is TX2 HW but firmware
> > config is likely to differ from the HW I have at hand), the
> > test command line that triggers the fault would be useful as
> > a starting point.
> >
The failure is always triggered during stress testing, see source at
https://gitlab.com/cki-project/kernel-tests/-/tree/main/stress/stress-ng
The runtest file is specific to run in our lab, but all it does is running
subsets of the upstream test (see the "*.stressors" files). Upstream
test version is V0.12.05 and the version wasn't changed since long
before we started hitting the problem. The failures were observed on
both Fedora 33 and 34 releases, I don't think the distro choice matters
but mentioning it just in case.
It doesn't reproduce 100%, anytime we tried to reproduce it on purpose
we couldn't trigger the issue. As if it knew we're watching...
> > Furthermore, is this a v5.13-rc* regression ? If so it would be
> > good to bisect it - I can't recollect arm64 changes that could
> > have introduced this regression in the last cycle but I may have
> > missed something.
>
> The actual change which has brought this to light is the update to
> arm64's memcpy() routine for 5.13 - the new version is more aggressive
> at making unaligned loads from the source buffer, so now triggers
> alignment faults more readily when (wrongly) used on iomem mappings in
> places that were getting away with it by chance under the previous
> implementation (see also [1], for example).
>
We dug into the history of runs, the first record we have is from the
mainline commit eb6bbacc46720b8b from April 28. The previously tested
commit fafe1e39ed213221c0bce doesn't hit any problems when running
stress-ng. Unfortunately we don't have logs from that first failed run
anymore as they are only saved for 6 weeks, the first logs we still have
are from May 11 from mainline commit c90b1834703f13b3a:
https://s3.us-east-1.amazonaws.com/arr-cki-prod-datawarehouse-public/datawarehouse-public/2021/05/11/301024644/build_aarch64_redhat:1253670447/tests/9969720_aarch64_2_console.log
Veronika
> Thanks,
> Robin.
>
> [1]
> https://lore.kernel.org/linux-arm-kernel/20210608153344.3813661-1-narmstrong@baylibre.com/
>
> >> Given the definition of uncached for arm64 memory types though, that
> >> means that callers of acpi_os_map_memory() still have to be prepared
> >> to get an __iomem pointer back even if they know they're mapping a
> >> table rather than some random bit of MMIO for an AML method.
> >>
> >> Therefore in this case it seems the blame lies partway between
> >> acpi_os_map_memory() for casting away __iomem and acpi_data_show() for
> >> letting an arbitrary offset lead to an arbitrarily-aligned memcpy(), but I
> >> don't know what the best way to fix it is. Either way I've satisfied myself
> >> that it's not an issue with the arm64 code itself - I do wonder whether this
> >> might also be a problem on IA-64 given ACPI_MISALIGNMENT_NOT_SUPPORTED, and
> >> I guess RISC-V may have alignment concerns as well.
> >
> > Yes agreed but see above, this code has been there for aeons if it
> > is a v5.13-rc* regression it must be something else that actually
> > triggered it (test/FW config).
> >
> > Thanks for looking into this.
> >
> > Lorenzo
> >
>
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: ❌ FAIL: Test report for kernel 5.13.0-rc7 (arm-next, 8ab9b1a9)
2021-06-29 17:03 ` Veronika Kabatova
@ 2021-06-29 17:27 ` Robin Murphy
0 siblings, 0 replies; 39+ messages in thread
From: Robin Murphy @ 2021-06-29 17:27 UTC (permalink / raw)
To: Veronika Kabatova
Cc: Lorenzo Pieralisi, ACPI Devel Maling List, Will Deacon,
CKI Project, Mark Rutland, Memory Management, skt-results-master,
Jeff Bastian, Jan Stancek, Linux ARM, rjw, lenb, guohanjun,
sudeep.holla, Ard Biesheuvel, Catalin Marinas, lv.zheng,
tony.luck
On 2021-06-29 18:03, Veronika Kabatova wrote:
> On Tue, Jun 29, 2021 at 5:15 PM Robin Murphy <robin.murphy@arm.com> wrote:
>>
>> On 2021-06-29 15:44, Lorenzo Pieralisi wrote:
>>> On Tue, Jun 29, 2021 at 12:48:14PM +0100, Robin Murphy wrote:
>>>> [ +ACPI audience ]
>>>>
>>>> On 2021-06-25 12:15, Robin Murphy wrote:
>>>>> On 2021-06-25 12:09, Catalin Marinas wrote:
>>>>>> On Fri, Jun 25, 2021 at 12:02:52PM +0100, Robin Murphy wrote:
>>>>>>> On 2021-06-25 10:52, Veronika Kabatova wrote:
>>>>>>> [...]
>>>>>>>>>> ❌ stress: stress-ng
>>>>>>>>>
>>>>>>>>> Oh no, this looks like another alignment fault in memcpy:
>>>>>>>>>
>>>>>>>>> [13330.651903] Unable to handle kernel paging request at
>>>>>>>>> virtual address ffff8000534705ff
>>>>>>>>> [13330.651914] Mem abort info:
>>>>>>>>> [13330.651918] ESR = 0x96000021
>>>>>>>>> [13330.651922] EC = 0x25: DABT (current EL), IL = 32 bits
>>>>>>>>> [13330.651928] SET = 0, FnV = 0
>>>>>>>>> [13330.651931] EA = 0, S1PTW = 0
>>>>>>>>> [13330.651933] FSC = 0x21: alignment fault
>>>>>>>>> [13330.651938] Data abort info:
>>>>>>>>> [13330.651940] ISV = 0, ISS = 0x00000021
>>>>>>>>> [13330.651941] CM = 0, WnR = 0
>>>>>>>>> [13330.651943] swapper pgtable: 4k pages, 48-bit VAs,
>>>>>>>>> pgdp=00000000f3e6b000
>>>>>>>>> [13330.651945] [ffff8000534705ff] pgd=1000008ffcfff003,
>>>>>>>>> p4d=1000008ffcfff003, pud=100000088e57d003,
>>>>>>>>> pmd=10000008d0aeb003, pte=006800008021370f
>>>>>>>>> [13330.651956] Internal error: Oops: 96000021 [#1] SMP
>>>>>>>>> [13330.651961] Modules linked in: unix_diag binfmt_misc
>>>>>>>>> fcrypt sm4_generic crc32_generic md4 michael_mic
>>>>>>>>> nhpoly1305_neon nhpoly1305 poly1305_generic libpoly1305
>>>>>>>>> poly1305_neon rmd160 sha3_generic sm3_generic
>>>>>>>>> streebog_generic wp512 blowfish_generic blowfish_common
>>>>>>>>> cast5_generic des_generic libdes chacha_generic
>>>>>>>>> chacha_neon libchacha camellia_generic cast6_generic
>>>>>>>>> cast_common serpent_generic twofish_generic
>>>>>>>>> twofish_common dm_thin_pool dm_persistent_data
>>>>>>>>> dm_bio_prison nvme nvme_core ipmi_watchdog ipmi_poweroff
>>>>>>>>> loop tun af_key crypto_user scsi_transport_iscsi
>>>>>>>>> xt_multiport ip_gre ip_tunnel gre overlay xt_CONNSECMARK
>>>>>>>>> xt_SECMARK nft_counter xt_state xt_conntrack nft_compat
>>>>>>>>> ah6 ah4 nft_objref nft_ct nf_conntrack nf_defrag_ipv6
>>>>>>>>> nf_defrag_ipv4 nf_tables nfnetlink jfs sctp
>>>>>>>>> ip6_udp_tunnel udp_tunnel dm_log_writes dm_flakey rfkill
>>>>>>>>> mlx5_ib ib_uverbs ib_core sunrpc coresight_etm4x
>>>>>>>>> i2c_smbus coresight_replicator coresight_tpiu
>>>>>>>>> coresight_tmc joydev mlx5_core acpi_ipmi psample
>>>>>>>>> ipmi_ssif mlxfw !
>>>>>>>>> ipmi_devintf
>>>>>>>>> [13330.652076] ipmi_msghandler coresight_funnel
>>>>>>>>> thunderx2_pmu coresight vfat fat fuse zram ip_tables xfs
>>>>>>>>> ast crct10dif_ce i2c_algo_bit ghash_ce drm_vram_helper
>>>>>>>>> drm_kms_helper syscopyarea sysfillrect sysimgblt
>>>>>>>>> fb_sys_fops cec drm_ttm_helper ttm drm gpio_xlp
>>>>>>>>> i2c_xlp9xx uas usb_storage aes_neon_bs [last unloaded:
>>>>>>>>> nvmet]
>>>>>>>>> [13330.652123] CPU: 115 PID: 188446 Comm: stress-ng
>>>>>>>>> Tainted: G OEL 5.13.0-rc7 #1
>>>>>>>>> [13330.652129] Hardware name: HPE Apollo 70
>>>>>>>>> /C01_APACHE_MB , BIOS L50_5.13_1.15 05/08/2020
>>>>>>>>> [13330.652133] pstate: 80400009 (Nzcv daif +PAN -UAO -TCO BTYPE=--)
>>>>>>>>> [13330.652139] pc : __memcpy+0x168/0x250
>>>>>>>>> [13330.652150] lr : memory_read_from_buffer+0x58/0x80
>>>>>>>>> [13330.652161] sp : ffff800063ef3c20
>>>>>>>>> [13330.652163] x29: ffff800063ef3c20 x28:
>>>>>>>>> ffff0008b1380000 x27: 0000000000000000
>>>>>>>>> [13330.652170] x26: 0000000000000000 x25:
>>>>>>>>> 0000000000000000 x24: ffff00080a960fe0
>>>>>>>>> [13330.652176] x23: ffff800063ef3d28 x22:
>>>>>>>>> 000000000000063f x21: ffff800063ef3c88
>>>>>>>>> [13330.652181] x20: 000000000000063f x19:
>>>>>>>>> 000000000000063f x18: 0000000000000000
>>>>>>>>> [13330.652186] x17: 0000000000000000 x16:
>>>>>>>>> 0000000000000000 x15: 0000000000000000
>>>>>>>>> [13330.652191] x14: 0000000000000000 x13:
>>>>>>>>> 0000000000000000 x12: 0000000000000000
>>>>>>>>> [13330.652196] x11: 0000000000000000 x10:
>>>>>>>>> 0000000000000000 x9 : 0000000000000000
>>>>>>>>> [13330.652200] x8 : 0000000000000000 x7 :
>>>>>>>>> 0000000000000000 x6 : 0000000000000000
>>>>>>>>> [13330.652206] x5 : ffff000d0fb0063f x4 :
>>>>>>>>> ffff80005347063f x3 : ffff000d0fb005c0
>>>>>>>>> [13330.652212] x2 : ffffffffffffffef x1 :
>>>>>>>>> ffff800053470600 x0 : ffff000d0fb00000
>>>>>>>>> [13330.652218] Call trace:
>>>>>>>>> [13330.652221] __memcpy+0x168/0x250
>>>>>>>>> [13330.652225] acpi_data_show+0x5c/0x8c
>>>>>>>>> [13330.652232] sysfs_kf_bin_read+0x78/0xa0
>>>>>>>>> [13330.652238] kernfs_file_read_iter+0x9c/0x1a4
>>>>>>>>> [13330.652241] kernfs_fop_read_iter+0x34/0x50
>>>>>>>>> [13330.652244] new_sync_read+0xdc/0x154
>>>>>>>>> [13330.652253] vfs_read+0x158/0x1e4
>>>>>>>>> [13330.652260] ksys_read+0x64/0xec
>>>>>>>>> [13330.652266] __arm64_sys_read+0x28/0x34
>>>>>>>>> [13330.652273] invoke_syscall+0x50/0x120
>>>>>>>>> [13330.652280] el0_svc_common.constprop.0+0x4c/0xd4
>>>>>>>>> [13330.652284] do_el0_svc+0x30/0x9c
>>>>>>>>> [13330.652286] el0_svc+0x2c/0x54
>>>>>>>>> [13330.652294] el0t_64_sync_handler+0x1a4/0x1b0
>>>>>>>>> [13330.652296] el0t_64_sync+0x19c/0x1a0
>>>>>>>>> [13330.652303] Code: a984346c a9c4342c f1010042 54fffee8 (a97c3c8e)
>>>>>>>>> [13330.652307] ---[ end trace 227d4380f57145d4 ]---
>>>>>>>>>
>>>>>>>>> So maybe this issue isn't limited to weird modules, after all...
>>>>>>>>>
>>>>>>>>
>>>>>>>> It ran on the machine from the same set that we were able to reproduce
>>>>>>>> it on previously. If you or anyone else have an idea on how
>>>>>>>> to stabilize
>>>>>>>> the reproducibility or have a debug patch we'll be happy to try it.
>>>>>>>
>>>>>>> Possibly it depends on the individual machines' firmware exactly how the
>>>>>>> relevant bits of their ACPI tables are aligned in memory?
>>>>>>>
>>>>>>> I've started digging into that callstack - it may not be a
>>>>>>> "weird module"
>>>>>>> but it's definitely crusty ACPI code... a238317ce818 ("ACPI: Clean up
>>>>>>> acpi_os_map/unmap_memory() to eliminate __iomem.") looks frankly a bit
>>>>>>> questionable in its decision to blithely cast away __iomem, but then the
>>>>>>> rationale in aafc65c731fe ("ACPI: add arm64 to the platforms that use
>>>>>>> ioremap") seems particularly dubious on top of that (especially
>>>>>>> given this
>>>>>>> end result).
>>>>>>>
>>>>>>> At a wild guess, I'm wondering if this may be sufficient:
>>>>>>>
>>>>>>> ----->8-----
>>>>>>> diff --git a/drivers/acpi/osl.c b/drivers/acpi/osl.c
>>>>>>> index 327e1b4eb6b0..f5d26b102fbe 100644
>>>>>>> --- a/drivers/acpi/osl.c
>>>>>>> +++ b/drivers/acpi/osl.c
>>>>>>> @@ -277,7 +277,7 @@ acpi_map_lookup_virt(void __iomem *virt,
>>>>>>> acpi_size size)
>>>>>>> return NULL;
>>>>>>> }
>>>>>>>
>>>>>>> -#if defined(CONFIG_IA64) || defined(CONFIG_ARM64)
>>>>>>> +#if defined(CONFIG_IA64)
>>>>>>> /* ioremap will take care of cache attributes */
>>>>>>> #define should_use_kmap(pfn) 0
>>>>>>> #else
>>>>>>> -----8<-----
>>>>>>
>>>>>> I thought the same but shouldn't acpi_os_ioremap() map it with the right
>>>>>> attributes? It uses the EFI maps to check what kind of memory this is.
>>>>>
>>>>> Oh crikey, I missed that branch of the rabbit hole... I guess that must
>>>>> mean that the tables being poked here are *not* covered by the EFI
>>>>> memory map, so page_is_ram() is unlikely to help either :(
>>>>
>>>> After picking through the UEFI spec I think I've now got a clearer picture
>>>> of what's happening, but I'm not sure where it goes from here...
>>>>
>>>> The spec implies that it *is* legitimate for runtime-loaded ACPI tables to
>>>> lie outside the EFI memory map, and that case they must be assumed to be
>>>> uncached, so the behaviour of acpi_os_ioremap() is correct.
>>>
>>> I'd agree with the reasoning, it would be good to pinpoint whether
>>> that's what actually triggers the issue.
>>>
>>> I'd like to replicate it if possible (it is TX2 HW but firmware
>>> config is likely to differ from the HW I have at hand), the
>>> test command line that triggers the fault would be useful as
>>> a starting point.
>>>
>
> The failure is always triggered during stress testing, see source at
>
> https://gitlab.com/cki-project/kernel-tests/-/tree/main/stress/stress-ng
>
> The runtest file is specific to run in our lab, but all it does is running
> subsets of the upstream test (see the "*.stressors" files). Upstream
> test version is V0.12.05 and the version wasn't changed since long
> before we started hitting the problem. The failures were observed on
> both Fedora 33 and 34 releases, I don't think the distro choice matters
> but mentioning it just in case.
>
> It doesn't reproduce 100%, anytime we tried to reproduce it on purpose
> we couldn't trigger the issue. As if it knew we're watching...
Ah, from that I can only assume that this must be stress-ng's --sysfs
test reading things at random, so not only would it have to be on a
machine whose firmware presents the right thing in the right way but the
random test conditions would also have to line up to poke it the "right"
(wrong) way too.
As a temporary workaround for the CI flakes, might it be possible to
configure stress-ng to stay away from just these ACPI "data" files?
Robin.
>>> Furthermore, is this a v5.13-rc* regression ? If so it would be
>>> good to bisect it - I can't recollect arm64 changes that could
>>> have introduced this regression in the last cycle but I may have
>>> missed something.
>>
>> The actual change which has brought this to light is the update to
>> arm64's memcpy() routine for 5.13 - the new version is more aggressive
>> at making unaligned loads from the source buffer, so now triggers
>> alignment faults more readily when (wrongly) used on iomem mappings in
>> places that were getting away with it by chance under the previous
>> implementation (see also [1], for example).
>>
>
> We dug into the history of runs, the first record we have is from the
> mainline commit eb6bbacc46720b8b from April 28. The previously tested
> commit fafe1e39ed213221c0bce doesn't hit any problems when running
> stress-ng. Unfortunately we don't have logs from that first failed run
> anymore as they are only saved for 6 weeks, the first logs we still have
> are from May 11 from mainline commit c90b1834703f13b3a:
>
> https://s3.us-east-1.amazonaws.com/arr-cki-prod-datawarehouse-public/datawarehouse-public/2021/05/11/301024644/build_aarch64_redhat:1253670447/tests/9969720_aarch64_2_console.log
>
>
> Veronika
>
>> Thanks,
>> Robin.
>>
>> [1]
>> https://lore.kernel.org/linux-arm-kernel/20210608153344.3813661-1-narmstrong@baylibre.com/
>>
>>>> Given the definition of uncached for arm64 memory types though, that
>>>> means that callers of acpi_os_map_memory() still have to be prepared
>>>> to get an __iomem pointer back even if they know they're mapping a
>>>> table rather than some random bit of MMIO for an AML method.
>>>>
>>>> Therefore in this case it seems the blame lies partway between
>>>> acpi_os_map_memory() for casting away __iomem and acpi_data_show() for
>>>> letting an arbitrary offset lead to an arbitrarily-aligned memcpy(), but I
>>>> don't know what the best way to fix it is. Either way I've satisfied myself
>>>> that it's not an issue with the arm64 code itself - I do wonder whether this
>>>> might also be a problem on IA-64 given ACPI_MISALIGNMENT_NOT_SUPPORTED, and
>>>> I guess RISC-V may have alignment concerns as well.
>>>
>>> Yes agreed but see above, this code has been there for aeons if it
>>> is a v5.13-rc* regression it must be something else that actually
>>> triggered it (test/FW config).
>>>
>>> Thanks for looking into this.
>>>
>>> Lorenzo
>>>
>>
>
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: ❌ FAIL: Test report for kernel 5.13.0-rc7 (arm-next, 8ab9b1a9)
@ 2021-06-29 17:27 ` Robin Murphy
0 siblings, 0 replies; 39+ messages in thread
From: Robin Murphy @ 2021-06-29 17:27 UTC (permalink / raw)
To: Veronika Kabatova
Cc: Lorenzo Pieralisi, ACPI Devel Maling List, Will Deacon,
CKI Project, Mark Rutland, Memory Management, skt-results-master,
Jeff Bastian, Jan Stancek, Linux ARM, rjw, lenb, guohanjun,
sudeep.holla, Ard Biesheuvel, Catalin Marinas, lv.zheng,
tony.luck
On 2021-06-29 18:03, Veronika Kabatova wrote:
> On Tue, Jun 29, 2021 at 5:15 PM Robin Murphy <robin.murphy@arm.com> wrote:
>>
>> On 2021-06-29 15:44, Lorenzo Pieralisi wrote:
>>> On Tue, Jun 29, 2021 at 12:48:14PM +0100, Robin Murphy wrote:
>>>> [ +ACPI audience ]
>>>>
>>>> On 2021-06-25 12:15, Robin Murphy wrote:
>>>>> On 2021-06-25 12:09, Catalin Marinas wrote:
>>>>>> On Fri, Jun 25, 2021 at 12:02:52PM +0100, Robin Murphy wrote:
>>>>>>> On 2021-06-25 10:52, Veronika Kabatova wrote:
>>>>>>> [...]
>>>>>>>>>> ❌ stress: stress-ng
>>>>>>>>>
>>>>>>>>> Oh no, this looks like another alignment fault in memcpy:
>>>>>>>>>
>>>>>>>>> [13330.651903] Unable to handle kernel paging request at
>>>>>>>>> virtual address ffff8000534705ff
>>>>>>>>> [13330.651914] Mem abort info:
>>>>>>>>> [13330.651918] ESR = 0x96000021
>>>>>>>>> [13330.651922] EC = 0x25: DABT (current EL), IL = 32 bits
>>>>>>>>> [13330.651928] SET = 0, FnV = 0
>>>>>>>>> [13330.651931] EA = 0, S1PTW = 0
>>>>>>>>> [13330.651933] FSC = 0x21: alignment fault
>>>>>>>>> [13330.651938] Data abort info:
>>>>>>>>> [13330.651940] ISV = 0, ISS = 0x00000021
>>>>>>>>> [13330.651941] CM = 0, WnR = 0
>>>>>>>>> [13330.651943] swapper pgtable: 4k pages, 48-bit VAs,
>>>>>>>>> pgdp=00000000f3e6b000
>>>>>>>>> [13330.651945] [ffff8000534705ff] pgd=1000008ffcfff003,
>>>>>>>>> p4d=1000008ffcfff003, pud=100000088e57d003,
>>>>>>>>> pmd=10000008d0aeb003, pte=006800008021370f
>>>>>>>>> [13330.651956] Internal error: Oops: 96000021 [#1] SMP
>>>>>>>>> [13330.651961] Modules linked in: unix_diag binfmt_misc
>>>>>>>>> fcrypt sm4_generic crc32_generic md4 michael_mic
>>>>>>>>> nhpoly1305_neon nhpoly1305 poly1305_generic libpoly1305
>>>>>>>>> poly1305_neon rmd160 sha3_generic sm3_generic
>>>>>>>>> streebog_generic wp512 blowfish_generic blowfish_common
>>>>>>>>> cast5_generic des_generic libdes chacha_generic
>>>>>>>>> chacha_neon libchacha camellia_generic cast6_generic
>>>>>>>>> cast_common serpent_generic twofish_generic
>>>>>>>>> twofish_common dm_thin_pool dm_persistent_data
>>>>>>>>> dm_bio_prison nvme nvme_core ipmi_watchdog ipmi_poweroff
>>>>>>>>> loop tun af_key crypto_user scsi_transport_iscsi
>>>>>>>>> xt_multiport ip_gre ip_tunnel gre overlay xt_CONNSECMARK
>>>>>>>>> xt_SECMARK nft_counter xt_state xt_conntrack nft_compat
>>>>>>>>> ah6 ah4 nft_objref nft_ct nf_conntrack nf_defrag_ipv6
>>>>>>>>> nf_defrag_ipv4 nf_tables nfnetlink jfs sctp
>>>>>>>>> ip6_udp_tunnel udp_tunnel dm_log_writes dm_flakey rfkill
>>>>>>>>> mlx5_ib ib_uverbs ib_core sunrpc coresight_etm4x
>>>>>>>>> i2c_smbus coresight_replicator coresight_tpiu
>>>>>>>>> coresight_tmc joydev mlx5_core acpi_ipmi psample
>>>>>>>>> ipmi_ssif mlxfw !
>>>>>>>>> ipmi_devintf
>>>>>>>>> [13330.652076] ipmi_msghandler coresight_funnel
>>>>>>>>> thunderx2_pmu coresight vfat fat fuse zram ip_tables xfs
>>>>>>>>> ast crct10dif_ce i2c_algo_bit ghash_ce drm_vram_helper
>>>>>>>>> drm_kms_helper syscopyarea sysfillrect sysimgblt
>>>>>>>>> fb_sys_fops cec drm_ttm_helper ttm drm gpio_xlp
>>>>>>>>> i2c_xlp9xx uas usb_storage aes_neon_bs [last unloaded:
>>>>>>>>> nvmet]
>>>>>>>>> [13330.652123] CPU: 115 PID: 188446 Comm: stress-ng
>>>>>>>>> Tainted: G OEL 5.13.0-rc7 #1
>>>>>>>>> [13330.652129] Hardware name: HPE Apollo 70
>>>>>>>>> /C01_APACHE_MB , BIOS L50_5.13_1.15 05/08/2020
>>>>>>>>> [13330.652133] pstate: 80400009 (Nzcv daif +PAN -UAO -TCO BTYPE=--)
>>>>>>>>> [13330.652139] pc : __memcpy+0x168/0x250
>>>>>>>>> [13330.652150] lr : memory_read_from_buffer+0x58/0x80
>>>>>>>>> [13330.652161] sp : ffff800063ef3c20
>>>>>>>>> [13330.652163] x29: ffff800063ef3c20 x28:
>>>>>>>>> ffff0008b1380000 x27: 0000000000000000
>>>>>>>>> [13330.652170] x26: 0000000000000000 x25:
>>>>>>>>> 0000000000000000 x24: ffff00080a960fe0
>>>>>>>>> [13330.652176] x23: ffff800063ef3d28 x22:
>>>>>>>>> 000000000000063f x21: ffff800063ef3c88
>>>>>>>>> [13330.652181] x20: 000000000000063f x19:
>>>>>>>>> 000000000000063f x18: 0000000000000000
>>>>>>>>> [13330.652186] x17: 0000000000000000 x16:
>>>>>>>>> 0000000000000000 x15: 0000000000000000
>>>>>>>>> [13330.652191] x14: 0000000000000000 x13:
>>>>>>>>> 0000000000000000 x12: 0000000000000000
>>>>>>>>> [13330.652196] x11: 0000000000000000 x10:
>>>>>>>>> 0000000000000000 x9 : 0000000000000000
>>>>>>>>> [13330.652200] x8 : 0000000000000000 x7 :
>>>>>>>>> 0000000000000000 x6 : 0000000000000000
>>>>>>>>> [13330.652206] x5 : ffff000d0fb0063f x4 :
>>>>>>>>> ffff80005347063f x3 : ffff000d0fb005c0
>>>>>>>>> [13330.652212] x2 : ffffffffffffffef x1 :
>>>>>>>>> ffff800053470600 x0 : ffff000d0fb00000
>>>>>>>>> [13330.652218] Call trace:
>>>>>>>>> [13330.652221] __memcpy+0x168/0x250
>>>>>>>>> [13330.652225] acpi_data_show+0x5c/0x8c
>>>>>>>>> [13330.652232] sysfs_kf_bin_read+0x78/0xa0
>>>>>>>>> [13330.652238] kernfs_file_read_iter+0x9c/0x1a4
>>>>>>>>> [13330.652241] kernfs_fop_read_iter+0x34/0x50
>>>>>>>>> [13330.652244] new_sync_read+0xdc/0x154
>>>>>>>>> [13330.652253] vfs_read+0x158/0x1e4
>>>>>>>>> [13330.652260] ksys_read+0x64/0xec
>>>>>>>>> [13330.652266] __arm64_sys_read+0x28/0x34
>>>>>>>>> [13330.652273] invoke_syscall+0x50/0x120
>>>>>>>>> [13330.652280] el0_svc_common.constprop.0+0x4c/0xd4
>>>>>>>>> [13330.652284] do_el0_svc+0x30/0x9c
>>>>>>>>> [13330.652286] el0_svc+0x2c/0x54
>>>>>>>>> [13330.652294] el0t_64_sync_handler+0x1a4/0x1b0
>>>>>>>>> [13330.652296] el0t_64_sync+0x19c/0x1a0
>>>>>>>>> [13330.652303] Code: a984346c a9c4342c f1010042 54fffee8 (a97c3c8e)
>>>>>>>>> [13330.652307] ---[ end trace 227d4380f57145d4 ]---
>>>>>>>>>
>>>>>>>>> So maybe this issue isn't limited to weird modules, after all...
>>>>>>>>>
>>>>>>>>
>>>>>>>> It ran on the machine from the same set that we were able to reproduce
>>>>>>>> it on previously. If you or anyone else have an idea on how
>>>>>>>> to stabilize
>>>>>>>> the reproducibility or have a debug patch we'll be happy to try it.
>>>>>>>
>>>>>>> Possibly it depends on the individual machines' firmware exactly how the
>>>>>>> relevant bits of their ACPI tables are aligned in memory?
>>>>>>>
>>>>>>> I've started digging into that callstack - it may not be a
>>>>>>> "weird module"
>>>>>>> but it's definitely crusty ACPI code... a238317ce818 ("ACPI: Clean up
>>>>>>> acpi_os_map/unmap_memory() to eliminate __iomem.") looks frankly a bit
>>>>>>> questionable in its decision to blithely cast away __iomem, but then the
>>>>>>> rationale in aafc65c731fe ("ACPI: add arm64 to the platforms that use
>>>>>>> ioremap") seems particularly dubious on top of that (especially
>>>>>>> given this
>>>>>>> end result).
>>>>>>>
>>>>>>> At a wild guess, I'm wondering if this may be sufficient:
>>>>>>>
>>>>>>> ----->8-----
>>>>>>> diff --git a/drivers/acpi/osl.c b/drivers/acpi/osl.c
>>>>>>> index 327e1b4eb6b0..f5d26b102fbe 100644
>>>>>>> --- a/drivers/acpi/osl.c
>>>>>>> +++ b/drivers/acpi/osl.c
>>>>>>> @@ -277,7 +277,7 @@ acpi_map_lookup_virt(void __iomem *virt,
>>>>>>> acpi_size size)
>>>>>>> return NULL;
>>>>>>> }
>>>>>>>
>>>>>>> -#if defined(CONFIG_IA64) || defined(CONFIG_ARM64)
>>>>>>> +#if defined(CONFIG_IA64)
>>>>>>> /* ioremap will take care of cache attributes */
>>>>>>> #define should_use_kmap(pfn) 0
>>>>>>> #else
>>>>>>> -----8<-----
>>>>>>
>>>>>> I thought the same but shouldn't acpi_os_ioremap() map it with the right
>>>>>> attributes? It uses the EFI maps to check what kind of memory this is.
>>>>>
>>>>> Oh crikey, I missed that branch of the rabbit hole... I guess that must
>>>>> mean that the tables being poked here are *not* covered by the EFI
>>>>> memory map, so page_is_ram() is unlikely to help either :(
>>>>
>>>> After picking through the UEFI spec I think I've now got a clearer picture
>>>> of what's happening, but I'm not sure where it goes from here...
>>>>
>>>> The spec implies that it *is* legitimate for runtime-loaded ACPI tables to
>>>> lie outside the EFI memory map, and that case they must be assumed to be
>>>> uncached, so the behaviour of acpi_os_ioremap() is correct.
>>>
>>> I'd agree with the reasoning, it would be good to pinpoint whether
>>> that's what actually triggers the issue.
>>>
>>> I'd like to replicate it if possible (it is TX2 HW but firmware
>>> config is likely to differ from the HW I have at hand), the
>>> test command line that triggers the fault would be useful as
>>> a starting point.
>>>
>
> The failure is always triggered during stress testing, see source at
>
> https://gitlab.com/cki-project/kernel-tests/-/tree/main/stress/stress-ng
>
> The runtest file is specific to run in our lab, but all it does is running
> subsets of the upstream test (see the "*.stressors" files). Upstream
> test version is V0.12.05 and the version wasn't changed since long
> before we started hitting the problem. The failures were observed on
> both Fedora 33 and 34 releases, I don't think the distro choice matters
> but mentioning it just in case.
>
> It doesn't reproduce 100%, anytime we tried to reproduce it on purpose
> we couldn't trigger the issue. As if it knew we're watching...
Ah, from that I can only assume that this must be stress-ng's --sysfs
test reading things at random, so not only would it have to be on a
machine whose firmware presents the right thing in the right way but the
random test conditions would also have to line up to poke it the "right"
(wrong) way too.
As a temporary workaround for the CI flakes, might it be possible to
configure stress-ng to stay away from just these ACPI "data" files?
Robin.
>>> Furthermore, is this a v5.13-rc* regression ? If so it would be
>>> good to bisect it - I can't recollect arm64 changes that could
>>> have introduced this regression in the last cycle but I may have
>>> missed something.
>>
>> The actual change which has brought this to light is the update to
>> arm64's memcpy() routine for 5.13 - the new version is more aggressive
>> at making unaligned loads from the source buffer, so now triggers
>> alignment faults more readily when (wrongly) used on iomem mappings in
>> places that were getting away with it by chance under the previous
>> implementation (see also [1], for example).
>>
>
> We dug into the history of runs, the first record we have is from the
> mainline commit eb6bbacc46720b8b from April 28. The previously tested
> commit fafe1e39ed213221c0bce doesn't hit any problems when running
> stress-ng. Unfortunately we don't have logs from that first failed run
> anymore as they are only saved for 6 weeks, the first logs we still have
> are from May 11 from mainline commit c90b1834703f13b3a:
>
> https://s3.us-east-1.amazonaws.com/arr-cki-prod-datawarehouse-public/datawarehouse-public/2021/05/11/301024644/build_aarch64_redhat:1253670447/tests/9969720_aarch64_2_console.log
>
>
> Veronika
>
>> Thanks,
>> Robin.
>>
>> [1]
>> https://lore.kernel.org/linux-arm-kernel/20210608153344.3813661-1-narmstrong@baylibre.com/
>>
>>>> Given the definition of uncached for arm64 memory types though, that
>>>> means that callers of acpi_os_map_memory() still have to be prepared
>>>> to get an __iomem pointer back even if they know they're mapping a
>>>> table rather than some random bit of MMIO for an AML method.
>>>>
>>>> Therefore in this case it seems the blame lies partway between
>>>> acpi_os_map_memory() for casting away __iomem and acpi_data_show() for
>>>> letting an arbitrary offset lead to an arbitrarily-aligned memcpy(), but I
>>>> don't know what the best way to fix it is. Either way I've satisfied myself
>>>> that it's not an issue with the arm64 code itself - I do wonder whether this
>>>> might also be a problem on IA-64 given ACPI_MISALIGNMENT_NOT_SUPPORTED, and
>>>> I guess RISC-V may have alignment concerns as well.
>>>
>>> Yes agreed but see above, this code has been there for aeons if it
>>> is a v5.13-rc* regression it must be something else that actually
>>> triggered it (test/FW config).
>>>
>>> Thanks for looking into this.
>>>
>>> Lorenzo
>>>
>>
>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: ❌ FAIL: Test report for kernel 5.13.0-rc7 (arm-next, 8ab9b1a9)
2021-06-29 17:27 ` Robin Murphy
(?)
@ 2021-06-29 17:44 ` Veronika Kabatova
-1 siblings, 0 replies; 39+ messages in thread
From: Veronika Kabatova @ 2021-06-29 17:44 UTC (permalink / raw)
To: Robin Murphy
Cc: Lorenzo Pieralisi, ACPI Devel Maling List, Will Deacon,
CKI Project, Mark Rutland, Memory Management, skt-results-master,
Jeff Bastian, Jan Stancek, Linux ARM, rjw, lenb, guohanjun,
sudeep.holla, Ard Biesheuvel, Catalin Marinas, lv.zheng,
tony.luck
On Tue, Jun 29, 2021 at 7:28 PM Robin Murphy <robin.murphy@arm.com> wrote:
>
> On 2021-06-29 18:03, Veronika Kabatova wrote:
> > On Tue, Jun 29, 2021 at 5:15 PM Robin Murphy <robin.murphy@arm.com> wrote:
> >>
> >> On 2021-06-29 15:44, Lorenzo Pieralisi wrote:
> >>> On Tue, Jun 29, 2021 at 12:48:14PM +0100, Robin Murphy wrote:
> >>>> [ +ACPI audience ]
> >>>>
> >>>> On 2021-06-25 12:15, Robin Murphy wrote:
> >>>>> On 2021-06-25 12:09, Catalin Marinas wrote:
> >>>>>> On Fri, Jun 25, 2021 at 12:02:52PM +0100, Robin Murphy wrote:
> >>>>>>> On 2021-06-25 10:52, Veronika Kabatova wrote:
> >>>>>>> [...]
> >>>>>>>>>> ❌ stress: stress-ng
> >>>>>>>>>
> >>>>>>>>> Oh no, this looks like another alignment fault in memcpy:
> >>>>>>>>>
> >>>>>>>>> [13330.651903] Unable to handle kernel paging request at
> >>>>>>>>> virtual address ffff8000534705ff
> >>>>>>>>> [13330.651914] Mem abort info:
> >>>>>>>>> [13330.651918] ESR = 0x96000021
> >>>>>>>>> [13330.651922] EC = 0x25: DABT (current EL), IL = 32 bits
> >>>>>>>>> [13330.651928] SET = 0, FnV = 0
> >>>>>>>>> [13330.651931] EA = 0, S1PTW = 0
> >>>>>>>>> [13330.651933] FSC = 0x21: alignment fault
> >>>>>>>>> [13330.651938] Data abort info:
> >>>>>>>>> [13330.651940] ISV = 0, ISS = 0x00000021
> >>>>>>>>> [13330.651941] CM = 0, WnR = 0
> >>>>>>>>> [13330.651943] swapper pgtable: 4k pages, 48-bit VAs,
> >>>>>>>>> pgdp=00000000f3e6b000
> >>>>>>>>> [13330.651945] [ffff8000534705ff] pgd=1000008ffcfff003,
> >>>>>>>>> p4d=1000008ffcfff003, pud=100000088e57d003,
> >>>>>>>>> pmd=10000008d0aeb003, pte=006800008021370f
> >>>>>>>>> [13330.651956] Internal error: Oops: 96000021 [#1] SMP
> >>>>>>>>> [13330.651961] Modules linked in: unix_diag binfmt_misc
> >>>>>>>>> fcrypt sm4_generic crc32_generic md4 michael_mic
> >>>>>>>>> nhpoly1305_neon nhpoly1305 poly1305_generic libpoly1305
> >>>>>>>>> poly1305_neon rmd160 sha3_generic sm3_generic
> >>>>>>>>> streebog_generic wp512 blowfish_generic blowfish_common
> >>>>>>>>> cast5_generic des_generic libdes chacha_generic
> >>>>>>>>> chacha_neon libchacha camellia_generic cast6_generic
> >>>>>>>>> cast_common serpent_generic twofish_generic
> >>>>>>>>> twofish_common dm_thin_pool dm_persistent_data
> >>>>>>>>> dm_bio_prison nvme nvme_core ipmi_watchdog ipmi_poweroff
> >>>>>>>>> loop tun af_key crypto_user scsi_transport_iscsi
> >>>>>>>>> xt_multiport ip_gre ip_tunnel gre overlay xt_CONNSECMARK
> >>>>>>>>> xt_SECMARK nft_counter xt_state xt_conntrack nft_compat
> >>>>>>>>> ah6 ah4 nft_objref nft_ct nf_conntrack nf_defrag_ipv6
> >>>>>>>>> nf_defrag_ipv4 nf_tables nfnetlink jfs sctp
> >>>>>>>>> ip6_udp_tunnel udp_tunnel dm_log_writes dm_flakey rfkill
> >>>>>>>>> mlx5_ib ib_uverbs ib_core sunrpc coresight_etm4x
> >>>>>>>>> i2c_smbus coresight_replicator coresight_tpiu
> >>>>>>>>> coresight_tmc joydev mlx5_core acpi_ipmi psample
> >>>>>>>>> ipmi_ssif mlxfw !
> >>>>>>>>> ipmi_devintf
> >>>>>>>>> [13330.652076] ipmi_msghandler coresight_funnel
> >>>>>>>>> thunderx2_pmu coresight vfat fat fuse zram ip_tables xfs
> >>>>>>>>> ast crct10dif_ce i2c_algo_bit ghash_ce drm_vram_helper
> >>>>>>>>> drm_kms_helper syscopyarea sysfillrect sysimgblt
> >>>>>>>>> fb_sys_fops cec drm_ttm_helper ttm drm gpio_xlp
> >>>>>>>>> i2c_xlp9xx uas usb_storage aes_neon_bs [last unloaded:
> >>>>>>>>> nvmet]
> >>>>>>>>> [13330.652123] CPU: 115 PID: 188446 Comm: stress-ng
> >>>>>>>>> Tainted: G OEL 5.13.0-rc7 #1
> >>>>>>>>> [13330.652129] Hardware name: HPE Apollo 70
> >>>>>>>>> /C01_APACHE_MB , BIOS L50_5.13_1.15 05/08/2020
> >>>>>>>>> [13330.652133] pstate: 80400009 (Nzcv daif +PAN -UAO -TCO BTYPE=--)
> >>>>>>>>> [13330.652139] pc : __memcpy+0x168/0x250
> >>>>>>>>> [13330.652150] lr : memory_read_from_buffer+0x58/0x80
> >>>>>>>>> [13330.652161] sp : ffff800063ef3c20
> >>>>>>>>> [13330.652163] x29: ffff800063ef3c20 x28:
> >>>>>>>>> ffff0008b1380000 x27: 0000000000000000
> >>>>>>>>> [13330.652170] x26: 0000000000000000 x25:
> >>>>>>>>> 0000000000000000 x24: ffff00080a960fe0
> >>>>>>>>> [13330.652176] x23: ffff800063ef3d28 x22:
> >>>>>>>>> 000000000000063f x21: ffff800063ef3c88
> >>>>>>>>> [13330.652181] x20: 000000000000063f x19:
> >>>>>>>>> 000000000000063f x18: 0000000000000000
> >>>>>>>>> [13330.652186] x17: 0000000000000000 x16:
> >>>>>>>>> 0000000000000000 x15: 0000000000000000
> >>>>>>>>> [13330.652191] x14: 0000000000000000 x13:
> >>>>>>>>> 0000000000000000 x12: 0000000000000000
> >>>>>>>>> [13330.652196] x11: 0000000000000000 x10:
> >>>>>>>>> 0000000000000000 x9 : 0000000000000000
> >>>>>>>>> [13330.652200] x8 : 0000000000000000 x7 :
> >>>>>>>>> 0000000000000000 x6 : 0000000000000000
> >>>>>>>>> [13330.652206] x5 : ffff000d0fb0063f x4 :
> >>>>>>>>> ffff80005347063f x3 : ffff000d0fb005c0
> >>>>>>>>> [13330.652212] x2 : ffffffffffffffef x1 :
> >>>>>>>>> ffff800053470600 x0 : ffff000d0fb00000
> >>>>>>>>> [13330.652218] Call trace:
> >>>>>>>>> [13330.652221] __memcpy+0x168/0x250
> >>>>>>>>> [13330.652225] acpi_data_show+0x5c/0x8c
> >>>>>>>>> [13330.652232] sysfs_kf_bin_read+0x78/0xa0
> >>>>>>>>> [13330.652238] kernfs_file_read_iter+0x9c/0x1a4
> >>>>>>>>> [13330.652241] kernfs_fop_read_iter+0x34/0x50
> >>>>>>>>> [13330.652244] new_sync_read+0xdc/0x154
> >>>>>>>>> [13330.652253] vfs_read+0x158/0x1e4
> >>>>>>>>> [13330.652260] ksys_read+0x64/0xec
> >>>>>>>>> [13330.652266] __arm64_sys_read+0x28/0x34
> >>>>>>>>> [13330.652273] invoke_syscall+0x50/0x120
> >>>>>>>>> [13330.652280] el0_svc_common.constprop.0+0x4c/0xd4
> >>>>>>>>> [13330.652284] do_el0_svc+0x30/0x9c
> >>>>>>>>> [13330.652286] el0_svc+0x2c/0x54
> >>>>>>>>> [13330.652294] el0t_64_sync_handler+0x1a4/0x1b0
> >>>>>>>>> [13330.652296] el0t_64_sync+0x19c/0x1a0
> >>>>>>>>> [13330.652303] Code: a984346c a9c4342c f1010042 54fffee8 (a97c3c8e)
> >>>>>>>>> [13330.652307] ---[ end trace 227d4380f57145d4 ]---
> >>>>>>>>>
> >>>>>>>>> So maybe this issue isn't limited to weird modules, after all...
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> It ran on the machine from the same set that we were able to reproduce
> >>>>>>>> it on previously. If you or anyone else have an idea on how
> >>>>>>>> to stabilize
> >>>>>>>> the reproducibility or have a debug patch we'll be happy to try it.
> >>>>>>>
> >>>>>>> Possibly it depends on the individual machines' firmware exactly how the
> >>>>>>> relevant bits of their ACPI tables are aligned in memory?
> >>>>>>>
> >>>>>>> I've started digging into that callstack - it may not be a
> >>>>>>> "weird module"
> >>>>>>> but it's definitely crusty ACPI code... a238317ce818 ("ACPI: Clean up
> >>>>>>> acpi_os_map/unmap_memory() to eliminate __iomem.") looks frankly a bit
> >>>>>>> questionable in its decision to blithely cast away __iomem, but then the
> >>>>>>> rationale in aafc65c731fe ("ACPI: add arm64 to the platforms that use
> >>>>>>> ioremap") seems particularly dubious on top of that (especially
> >>>>>>> given this
> >>>>>>> end result).
> >>>>>>>
> >>>>>>> At a wild guess, I'm wondering if this may be sufficient:
> >>>>>>>
> >>>>>>> ----->8-----
> >>>>>>> diff --git a/drivers/acpi/osl.c b/drivers/acpi/osl.c
> >>>>>>> index 327e1b4eb6b0..f5d26b102fbe 100644
> >>>>>>> --- a/drivers/acpi/osl.c
> >>>>>>> +++ b/drivers/acpi/osl.c
> >>>>>>> @@ -277,7 +277,7 @@ acpi_map_lookup_virt(void __iomem *virt,
> >>>>>>> acpi_size size)
> >>>>>>> return NULL;
> >>>>>>> }
> >>>>>>>
> >>>>>>> -#if defined(CONFIG_IA64) || defined(CONFIG_ARM64)
> >>>>>>> +#if defined(CONFIG_IA64)
> >>>>>>> /* ioremap will take care of cache attributes */
> >>>>>>> #define should_use_kmap(pfn) 0
> >>>>>>> #else
> >>>>>>> -----8<-----
> >>>>>>
> >>>>>> I thought the same but shouldn't acpi_os_ioremap() map it with the right
> >>>>>> attributes? It uses the EFI maps to check what kind of memory this is.
> >>>>>
> >>>>> Oh crikey, I missed that branch of the rabbit hole... I guess that must
> >>>>> mean that the tables being poked here are *not* covered by the EFI
> >>>>> memory map, so page_is_ram() is unlikely to help either :(
> >>>>
> >>>> After picking through the UEFI spec I think I've now got a clearer picture
> >>>> of what's happening, but I'm not sure where it goes from here...
> >>>>
> >>>> The spec implies that it *is* legitimate for runtime-loaded ACPI tables to
> >>>> lie outside the EFI memory map, and that case they must be assumed to be
> >>>> uncached, so the behaviour of acpi_os_ioremap() is correct.
> >>>
> >>> I'd agree with the reasoning, it would be good to pinpoint whether
> >>> that's what actually triggers the issue.
> >>>
> >>> I'd like to replicate it if possible (it is TX2 HW but firmware
> >>> config is likely to differ from the HW I have at hand), the
> >>> test command line that triggers the fault would be useful as
> >>> a starting point.
> >>>
> >
> > The failure is always triggered during stress testing, see source at
> >
> > https://gitlab.com/cki-project/kernel-tests/-/tree/main/stress/stress-ng
> >
> > The runtest file is specific to run in our lab, but all it does is running
> > subsets of the upstream test (see the "*.stressors" files). Upstream
> > test version is V0.12.05 and the version wasn't changed since long
> > before we started hitting the problem. The failures were observed on
> > both Fedora 33 and 34 releases, I don't think the distro choice matters
> > but mentioning it just in case.
> >
> > It doesn't reproduce 100%, anytime we tried to reproduce it on purpose
> > we couldn't trigger the issue. As if it knew we're watching...
>
> Ah, from that I can only assume that this must be stress-ng's --sysfs
> test reading things at random, so not only would it have to be on a
> machine whose firmware presents the right thing in the right way but the
> random test conditions would also have to line up to poke it the "right"
> (wrong) way too.
>
> As a temporary workaround for the CI flakes, might it be possible to
> configure stress-ng to stay away from just these ACPI "data" files?
>
The test is already waived so failures hit during the test do *not* affect
the aggregate CI results. It's also the last executed test on the machine
so it doesn't block further testing.
Veronika
> Robin.
>
> >>> Furthermore, is this a v5.13-rc* regression ? If so it would be
> >>> good to bisect it - I can't recollect arm64 changes that could
> >>> have introduced this regression in the last cycle but I may have
> >>> missed something.
> >>
> >> The actual change which has brought this to light is the update to
> >> arm64's memcpy() routine for 5.13 - the new version is more aggressive
> >> at making unaligned loads from the source buffer, so now triggers
> >> alignment faults more readily when (wrongly) used on iomem mappings in
> >> places that were getting away with it by chance under the previous
> >> implementation (see also [1], for example).
> >>
> >
> > We dug into the history of runs, the first record we have is from the
> > mainline commit eb6bbacc46720b8b from April 28. The previously tested
> > commit fafe1e39ed213221c0bce doesn't hit any problems when running
> > stress-ng. Unfortunately we don't have logs from that first failed run
> > anymore as they are only saved for 6 weeks, the first logs we still have
> > are from May 11 from mainline commit c90b1834703f13b3a:
> >
> > https://s3.us-east-1.amazonaws.com/arr-cki-prod-datawarehouse-public/datawarehouse-public/2021/05/11/301024644/build_aarch64_redhat:1253670447/tests/9969720_aarch64_2_console.log
> >
> >
> > Veronika
> >
> >> Thanks,
> >> Robin.
> >>
> >> [1]
> >> https://lore.kernel.org/linux-arm-kernel/20210608153344.3813661-1-narmstrong@baylibre.com/
> >>
> >>>> Given the definition of uncached for arm64 memory types though, that
> >>>> means that callers of acpi_os_map_memory() still have to be prepared
> >>>> to get an __iomem pointer back even if they know they're mapping a
> >>>> table rather than some random bit of MMIO for an AML method.
> >>>>
> >>>> Therefore in this case it seems the blame lies partway between
> >>>> acpi_os_map_memory() for casting away __iomem and acpi_data_show() for
> >>>> letting an arbitrary offset lead to an arbitrarily-aligned memcpy(), but I
> >>>> don't know what the best way to fix it is. Either way I've satisfied myself
> >>>> that it's not an issue with the arm64 code itself - I do wonder whether this
> >>>> might also be a problem on IA-64 given ACPI_MISALIGNMENT_NOT_SUPPORTED, and
> >>>> I guess RISC-V may have alignment concerns as well.
> >>>
> >>> Yes agreed but see above, this code has been there for aeons if it
> >>> is a v5.13-rc* regression it must be something else that actually
> >>> triggered it (test/FW config).
> >>>
> >>> Thanks for looking into this.
> >>>
> >>> Lorenzo
> >>>
> >>
> >
>
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: ❌ FAIL: Test report for kernel 5.13.0-rc7 (arm-next, 8ab9b1a9)
2021-06-29 16:35 ` Catalin Marinas
@ 2021-06-30 10:37 ` Lorenzo Pieralisi
-1 siblings, 0 replies; 39+ messages in thread
From: Lorenzo Pieralisi @ 2021-06-30 10:37 UTC (permalink / raw)
To: Catalin Marinas
Cc: Robin Murphy, ACPI Devel Maling List, Veronika Kabatova,
Will Deacon, CKI Project, Mark Rutland, Memory Management,
skt-results-master, Jeff Bastian, Jan Stancek, Linux ARM, rjw,
lenb, guohanjun, sudeep.holla, ardb, lv.zheng, tony.luck
On Tue, Jun 29, 2021 at 05:35:43PM +0100, Catalin Marinas wrote:
> On Tue, Jun 29, 2021 at 04:14:55PM +0100, Robin Murphy wrote:
> > On 2021-06-29 15:44, Lorenzo Pieralisi wrote:
> > > On Tue, Jun 29, 2021 at 12:48:14PM +0100, Robin Murphy wrote:
> > > > [ +ACPI audience ]
> > > >
> > > > On 2021-06-25 12:15, Robin Murphy wrote:
> > > > > On 2021-06-25 12:09, Catalin Marinas wrote:
> > > > > > On Fri, Jun 25, 2021 at 12:02:52PM +0100, Robin Murphy wrote:
> > > > > > > On 2021-06-25 10:52, Veronika Kabatova wrote:
> > > > > > > [...]
> > > > > > > > > > ❌ stress: stress-ng
> > > > > > > > >
> > > > > > > > > Oh no, this looks like another alignment fault in memcpy:
> > > > > > > > >
> > > > > > > > > [13330.651903] Unable to handle kernel paging request at
> > > > > > > > > virtual address ffff8000534705ff
> [...]
> > > > > > > > > [13330.652218] Call trace:
> > > > > > > > > [13330.652221] __memcpy+0x168/0x250
> > > > > > > > > [13330.652225] acpi_data_show+0x5c/0x8c
> > > > > > > > > [13330.652232] sysfs_kf_bin_read+0x78/0xa0
> > > > > > > > > [13330.652238] kernfs_file_read_iter+0x9c/0x1a4
> > > > > > > > > [13330.652241] kernfs_fop_read_iter+0x34/0x50
> > > > > > > > > [13330.652244] new_sync_read+0xdc/0x154
> > > > > > > > > [13330.652253] vfs_read+0x158/0x1e4
> > > > > > > > > [13330.652260] ksys_read+0x64/0xec
> > > > > > > > > [13330.652266] __arm64_sys_read+0x28/0x34
> > > > > > > > > [13330.652273] invoke_syscall+0x50/0x120
> > > > > > > > > [13330.652280] el0_svc_common.constprop.0+0x4c/0xd4
> > > > > > > > > [13330.652284] do_el0_svc+0x30/0x9c
> > > > > > > > > [13330.652286] el0_svc+0x2c/0x54
> > > > > > > > > [13330.652294] el0t_64_sync_handler+0x1a4/0x1b0
> > > > > > > > > [13330.652296] el0t_64_sync+0x19c/0x1a0
> > > > > > > > > [13330.652303] Code: a984346c a9c4342c f1010042 54fffee8 (a97c3c8e)
> > > > > > > > > [13330.652307] ---[ end trace 227d4380f57145d4 ]---
> > > > > > > > >
> > > > > > > > > So maybe this issue isn't limited to weird modules, after all...
> > > > > > > >
> > > > > > > > It ran on the machine from the same set that we were able to reproduce
> > > > > > > > it on previously. If you or anyone else have an idea on how
> > > > > > > > to stabilize the reproducibility or have a debug patch we'll be happy to try it.
> > > > > > >
> > > > > > > Possibly it depends on the individual machines' firmware exactly how the
> > > > > > > relevant bits of their ACPI tables are aligned in memory?
> > > > > > >
> > > > > > > I've started digging into that callstack - it may not be a "weird module"
> > > > > > > but it's definitely crusty ACPI code... a238317ce818 ("ACPI: Clean up
> > > > > > > acpi_os_map/unmap_memory() to eliminate __iomem.") looks frankly a bit
> > > > > > > questionable in its decision to blithely cast away __iomem, but then the
> > > > > > > rationale in aafc65c731fe ("ACPI: add arm64 to the platforms that use
> > > > > > > ioremap") seems particularly dubious on top of that (especially
> > > > > > > given this end result).
> [...]
> > > > After picking through the UEFI spec I think I've now got a clearer picture
> > > > of what's happening, but I'm not sure where it goes from here...
> > > >
> > > > The spec implies that it *is* legitimate for runtime-loaded ACPI tables to
> > > > lie outside the EFI memory map, and that case they must be assumed to be
> > > > uncached, so the behaviour of acpi_os_ioremap() is correct.
> > >
> > > I'd agree with the reasoning, it would be good to pinpoint whether
> > > that's what actually triggers the issue.
> > >
> > > I'd like to replicate it if possible (it is TX2 HW but firmware
> > > config is likely to differ from the HW I have at hand), the
> > > test command line that triggers the fault would be useful as
> > > a starting point.
> > >
> > > Furthermore, is this a v5.13-rc* regression ? If so it would be
> > > good to bisect it - I can't recollect arm64 changes that could
> > > have introduced this regression in the last cycle but I may have
> > > missed something.
> >
> > The actual change which has brought this to light is the update to arm64's
> > memcpy() routine for 5.13 - the new version is more aggressive at making
> > unaligned loads from the source buffer, so now triggers alignment faults
> > more readily when (wrongly) used on iomem mappings in places that were
> > getting away with it by chance under the previous implementation (see also
> > [1], for example).
>
> I wouldn't revert any of the memcpy() stuff as it just uncovered an
> existing bug in how the ACPI tables are handled. Could we actually hit
> a similar issue with C code parsing the ACPI tables?
I agree - I don't think a revert should be considered, this looks like
a long standing ACPI bug.
This needs debugging but I believe that it all depends on the table
being in the EFI map or not. I'd help a lot if I managed to reproduce
the bug for a given set-up so that we can check which table is causing
it.
> Is there a way to map the ACPI tables as Normal Noncacheable
> (ioremap_wc)?
That's a good point. IIUC UEFI 2.9 (2.3.6) requires tables loaded at
runtime (see above - I really would like to understand what table
is triggering this bug) that are not in the EFI memory map and whose
attributes cannot be retrieved through ACPI descriptors to be considered
non-cacheable.
The question is whether [arm64] acpi_os_ioremap() can be changed so that
the above is mapped to Normal NC rather than device-nGnRnE; this may
cause surprises the other way around (given that dev-nGnRnE is an
all encompassing fallback - again IIUC, I believe Ard knows better
than me if he has time to chime in).
We need a reproducer and some tracing in the ACPI code.
Lorenzo
>Presumably no-one sane would place ACPI tables in memory that's
>sensitive to the access size.
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: ❌ FAIL: Test report for kernel 5.13.0-rc7 (arm-next, 8ab9b1a9)
@ 2021-06-30 10:37 ` Lorenzo Pieralisi
0 siblings, 0 replies; 39+ messages in thread
From: Lorenzo Pieralisi @ 2021-06-30 10:37 UTC (permalink / raw)
To: Catalin Marinas
Cc: Robin Murphy, ACPI Devel Maling List, Veronika Kabatova,
Will Deacon, CKI Project, Mark Rutland, Memory Management,
skt-results-master, Jeff Bastian, Jan Stancek, Linux ARM, rjw,
lenb, guohanjun, sudeep.holla, ardb, lv.zheng, tony.luck
On Tue, Jun 29, 2021 at 05:35:43PM +0100, Catalin Marinas wrote:
> On Tue, Jun 29, 2021 at 04:14:55PM +0100, Robin Murphy wrote:
> > On 2021-06-29 15:44, Lorenzo Pieralisi wrote:
> > > On Tue, Jun 29, 2021 at 12:48:14PM +0100, Robin Murphy wrote:
> > > > [ +ACPI audience ]
> > > >
> > > > On 2021-06-25 12:15, Robin Murphy wrote:
> > > > > On 2021-06-25 12:09, Catalin Marinas wrote:
> > > > > > On Fri, Jun 25, 2021 at 12:02:52PM +0100, Robin Murphy wrote:
> > > > > > > On 2021-06-25 10:52, Veronika Kabatova wrote:
> > > > > > > [...]
> > > > > > > > > > ❌ stress: stress-ng
> > > > > > > > >
> > > > > > > > > Oh no, this looks like another alignment fault in memcpy:
> > > > > > > > >
> > > > > > > > > [13330.651903] Unable to handle kernel paging request at
> > > > > > > > > virtual address ffff8000534705ff
> [...]
> > > > > > > > > [13330.652218] Call trace:
> > > > > > > > > [13330.652221] __memcpy+0x168/0x250
> > > > > > > > > [13330.652225] acpi_data_show+0x5c/0x8c
> > > > > > > > > [13330.652232] sysfs_kf_bin_read+0x78/0xa0
> > > > > > > > > [13330.652238] kernfs_file_read_iter+0x9c/0x1a4
> > > > > > > > > [13330.652241] kernfs_fop_read_iter+0x34/0x50
> > > > > > > > > [13330.652244] new_sync_read+0xdc/0x154
> > > > > > > > > [13330.652253] vfs_read+0x158/0x1e4
> > > > > > > > > [13330.652260] ksys_read+0x64/0xec
> > > > > > > > > [13330.652266] __arm64_sys_read+0x28/0x34
> > > > > > > > > [13330.652273] invoke_syscall+0x50/0x120
> > > > > > > > > [13330.652280] el0_svc_common.constprop.0+0x4c/0xd4
> > > > > > > > > [13330.652284] do_el0_svc+0x30/0x9c
> > > > > > > > > [13330.652286] el0_svc+0x2c/0x54
> > > > > > > > > [13330.652294] el0t_64_sync_handler+0x1a4/0x1b0
> > > > > > > > > [13330.652296] el0t_64_sync+0x19c/0x1a0
> > > > > > > > > [13330.652303] Code: a984346c a9c4342c f1010042 54fffee8 (a97c3c8e)
> > > > > > > > > [13330.652307] ---[ end trace 227d4380f57145d4 ]---
> > > > > > > > >
> > > > > > > > > So maybe this issue isn't limited to weird modules, after all...
> > > > > > > >
> > > > > > > > It ran on the machine from the same set that we were able to reproduce
> > > > > > > > it on previously. If you or anyone else have an idea on how
> > > > > > > > to stabilize the reproducibility or have a debug patch we'll be happy to try it.
> > > > > > >
> > > > > > > Possibly it depends on the individual machines' firmware exactly how the
> > > > > > > relevant bits of their ACPI tables are aligned in memory?
> > > > > > >
> > > > > > > I've started digging into that callstack - it may not be a "weird module"
> > > > > > > but it's definitely crusty ACPI code... a238317ce818 ("ACPI: Clean up
> > > > > > > acpi_os_map/unmap_memory() to eliminate __iomem.") looks frankly a bit
> > > > > > > questionable in its decision to blithely cast away __iomem, but then the
> > > > > > > rationale in aafc65c731fe ("ACPI: add arm64 to the platforms that use
> > > > > > > ioremap") seems particularly dubious on top of that (especially
> > > > > > > given this end result).
> [...]
> > > > After picking through the UEFI spec I think I've now got a clearer picture
> > > > of what's happening, but I'm not sure where it goes from here...
> > > >
> > > > The spec implies that it *is* legitimate for runtime-loaded ACPI tables to
> > > > lie outside the EFI memory map, and that case they must be assumed to be
> > > > uncached, so the behaviour of acpi_os_ioremap() is correct.
> > >
> > > I'd agree with the reasoning, it would be good to pinpoint whether
> > > that's what actually triggers the issue.
> > >
> > > I'd like to replicate it if possible (it is TX2 HW but firmware
> > > config is likely to differ from the HW I have at hand), the
> > > test command line that triggers the fault would be useful as
> > > a starting point.
> > >
> > > Furthermore, is this a v5.13-rc* regression ? If so it would be
> > > good to bisect it - I can't recollect arm64 changes that could
> > > have introduced this regression in the last cycle but I may have
> > > missed something.
> >
> > The actual change which has brought this to light is the update to arm64's
> > memcpy() routine for 5.13 - the new version is more aggressive at making
> > unaligned loads from the source buffer, so now triggers alignment faults
> > more readily when (wrongly) used on iomem mappings in places that were
> > getting away with it by chance under the previous implementation (see also
> > [1], for example).
>
> I wouldn't revert any of the memcpy() stuff as it just uncovered an
> existing bug in how the ACPI tables are handled. Could we actually hit
> a similar issue with C code parsing the ACPI tables?
I agree - I don't think a revert should be considered, this looks like
a long standing ACPI bug.
This needs debugging but I believe that it all depends on the table
being in the EFI map or not. I'd help a lot if I managed to reproduce
the bug for a given set-up so that we can check which table is causing
it.
> Is there a way to map the ACPI tables as Normal Noncacheable
> (ioremap_wc)?
That's a good point. IIUC UEFI 2.9 (2.3.6) requires tables loaded at
runtime (see above - I really would like to understand what table
is triggering this bug) that are not in the EFI memory map and whose
attributes cannot be retrieved through ACPI descriptors to be considered
non-cacheable.
The question is whether [arm64] acpi_os_ioremap() can be changed so that
the above is mapped to Normal NC rather than device-nGnRnE; this may
cause surprises the other way around (given that dev-nGnRnE is an
all encompassing fallback - again IIUC, I believe Ard knows better
than me if he has time to chime in).
We need a reproducer and some tracing in the ACPI code.
Lorenzo
>Presumably no-one sane would place ACPI tables in memory that's
>sensitive to the access size.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: ❌ FAIL: Test report for kernel 5.13.0-rc7 (arm-next, 8ab9b1a9)
2021-06-30 10:37 ` Lorenzo Pieralisi
@ 2021-06-30 11:17 ` Robin Murphy
-1 siblings, 0 replies; 39+ messages in thread
From: Robin Murphy @ 2021-06-30 11:17 UTC (permalink / raw)
To: Lorenzo Pieralisi, Catalin Marinas
Cc: ACPI Devel Maling List, Veronika Kabatova, Will Deacon,
CKI Project, Mark Rutland, Memory Management, skt-results-master,
Jeff Bastian, Jan Stancek, Linux ARM, rjw, lenb, guohanjun,
sudeep.holla, ardb, lv.zheng, tony.luck, James Morse
On 2021-06-30 11:37, Lorenzo Pieralisi wrote:
> On Tue, Jun 29, 2021 at 05:35:43PM +0100, Catalin Marinas wrote:
>> On Tue, Jun 29, 2021 at 04:14:55PM +0100, Robin Murphy wrote:
>>> On 2021-06-29 15:44, Lorenzo Pieralisi wrote:
>>>> On Tue, Jun 29, 2021 at 12:48:14PM +0100, Robin Murphy wrote:
>>>>> [ +ACPI audience ]
>>>>>
>>>>> On 2021-06-25 12:15, Robin Murphy wrote:
>>>>>> On 2021-06-25 12:09, Catalin Marinas wrote:
>>>>>>> On Fri, Jun 25, 2021 at 12:02:52PM +0100, Robin Murphy wrote:
>>>>>>>> On 2021-06-25 10:52, Veronika Kabatova wrote:
>>>>>>>> [...]
>>>>>>>>>>> ❌ stress: stress-ng
>>>>>>>>>>
>>>>>>>>>> Oh no, this looks like another alignment fault in memcpy:
>>>>>>>>>>
>>>>>>>>>> [13330.651903] Unable to handle kernel paging request at
>>>>>>>>>> virtual address ffff8000534705ff
>> [...]
>>>>>>>>>> [13330.652218] Call trace:
>>>>>>>>>> [13330.652221] __memcpy+0x168/0x250
>>>>>>>>>> [13330.652225] acpi_data_show+0x5c/0x8c
>>>>>>>>>> [13330.652232] sysfs_kf_bin_read+0x78/0xa0
>>>>>>>>>> [13330.652238] kernfs_file_read_iter+0x9c/0x1a4
>>>>>>>>>> [13330.652241] kernfs_fop_read_iter+0x34/0x50
>>>>>>>>>> [13330.652244] new_sync_read+0xdc/0x154
>>>>>>>>>> [13330.652253] vfs_read+0x158/0x1e4
>>>>>>>>>> [13330.652260] ksys_read+0x64/0xec
>>>>>>>>>> [13330.652266] __arm64_sys_read+0x28/0x34
>>>>>>>>>> [13330.652273] invoke_syscall+0x50/0x120
>>>>>>>>>> [13330.652280] el0_svc_common.constprop.0+0x4c/0xd4
>>>>>>>>>> [13330.652284] do_el0_svc+0x30/0x9c
>>>>>>>>>> [13330.652286] el0_svc+0x2c/0x54
>>>>>>>>>> [13330.652294] el0t_64_sync_handler+0x1a4/0x1b0
>>>>>>>>>> [13330.652296] el0t_64_sync+0x19c/0x1a0
>>>>>>>>>> [13330.652303] Code: a984346c a9c4342c f1010042 54fffee8 (a97c3c8e)
>>>>>>>>>> [13330.652307] ---[ end trace 227d4380f57145d4 ]---
>>>>>>>>>>
>>>>>>>>>> So maybe this issue isn't limited to weird modules, after all...
>>>>>>>>>
>>>>>>>>> It ran on the machine from the same set that we were able to reproduce
>>>>>>>>> it on previously. If you or anyone else have an idea on how
>>>>>>>>> to stabilize the reproducibility or have a debug patch we'll be happy to try it.
>>>>>>>>
>>>>>>>> Possibly it depends on the individual machines' firmware exactly how the
>>>>>>>> relevant bits of their ACPI tables are aligned in memory?
>>>>>>>>
>>>>>>>> I've started digging into that callstack - it may not be a "weird module"
>>>>>>>> but it's definitely crusty ACPI code... a238317ce818 ("ACPI: Clean up
>>>>>>>> acpi_os_map/unmap_memory() to eliminate __iomem.") looks frankly a bit
>>>>>>>> questionable in its decision to blithely cast away __iomem, but then the
>>>>>>>> rationale in aafc65c731fe ("ACPI: add arm64 to the platforms that use
>>>>>>>> ioremap") seems particularly dubious on top of that (especially
>>>>>>>> given this end result).
>> [...]
>>>>> After picking through the UEFI spec I think I've now got a clearer picture
>>>>> of what's happening, but I'm not sure where it goes from here...
>>>>>
>>>>> The spec implies that it *is* legitimate for runtime-loaded ACPI tables to
>>>>> lie outside the EFI memory map, and that case they must be assumed to be
>>>>> uncached, so the behaviour of acpi_os_ioremap() is correct.
>>>>
>>>> I'd agree with the reasoning, it would be good to pinpoint whether
>>>> that's what actually triggers the issue.
>>>>
>>>> I'd like to replicate it if possible (it is TX2 HW but firmware
>>>> config is likely to differ from the HW I have at hand), the
>>>> test command line that triggers the fault would be useful as
>>>> a starting point.
>>>>
>>>> Furthermore, is this a v5.13-rc* regression ? If so it would be
>>>> good to bisect it - I can't recollect arm64 changes that could
>>>> have introduced this regression in the last cycle but I may have
>>>> missed something.
>>>
>>> The actual change which has brought this to light is the update to arm64's
>>> memcpy() routine for 5.13 - the new version is more aggressive at making
>>> unaligned loads from the source buffer, so now triggers alignment faults
>>> more readily when (wrongly) used on iomem mappings in places that were
>>> getting away with it by chance under the previous implementation (see also
>>> [1], for example).
>>
>> I wouldn't revert any of the memcpy() stuff as it just uncovered an
>> existing bug in how the ACPI tables are handled. Could we actually hit
>> a similar issue with C code parsing the ACPI tables?
>
> I agree - I don't think a revert should be considered, this looks like
> a long standing ACPI bug.
>
> This needs debugging but I believe that it all depends on the table
> being in the EFI map or not. I'd help a lot if I managed to reproduce
> the bug for a given set-up so that we can check which table is causing
> it.
>
>> Is there a way to map the ACPI tables as Normal Noncacheable
>> (ioremap_wc)?
>
> That's a good point. IIUC UEFI 2.9 (2.3.6) requires tables loaded at
> runtime (see above - I really would like to understand what table
> is triggering this bug) that are not in the EFI memory map and whose
> attributes cannot be retrieved through ACPI descriptors to be considered
> non-cacheable.
>
> The question is whether [arm64] acpi_os_ioremap() can be changed so that
> the above is mapped to Normal NC rather than device-nGnRnE; this may
> cause surprises the other way around (given that dev-nGnRnE is an
> all encompassing fallback - again IIUC, I believe Ard knows better
> than me if he has time to chime in).
>
> We need a reproducer and some tracing in the ACPI code.
Having looked even more at the sysfs code, I think this might not
actually be an ACPI table per se, but specifically only the Generic
Error Status Block pointed to by the BERT (so maybe it also requires the
machine to have experienced a boot-time error to be present?). ACPI
merely says that this is "a range of addressable memory" and "System
firmware must report this memory range as firmware reserved", so I have
no idea whether there's any specific expectation of how it's supposed to
be mapped.
Robin.
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: ❌ FAIL: Test report for kernel 5.13.0-rc7 (arm-next, 8ab9b1a9)
@ 2021-06-30 11:17 ` Robin Murphy
0 siblings, 0 replies; 39+ messages in thread
From: Robin Murphy @ 2021-06-30 11:17 UTC (permalink / raw)
To: Lorenzo Pieralisi, Catalin Marinas
Cc: ACPI Devel Maling List, Veronika Kabatova, Will Deacon,
CKI Project, Mark Rutland, Memory Management, skt-results-master,
Jeff Bastian, Jan Stancek, Linux ARM, rjw, lenb, guohanjun,
sudeep.holla, ardb, lv.zheng, tony.luck, James Morse
On 2021-06-30 11:37, Lorenzo Pieralisi wrote:
> On Tue, Jun 29, 2021 at 05:35:43PM +0100, Catalin Marinas wrote:
>> On Tue, Jun 29, 2021 at 04:14:55PM +0100, Robin Murphy wrote:
>>> On 2021-06-29 15:44, Lorenzo Pieralisi wrote:
>>>> On Tue, Jun 29, 2021 at 12:48:14PM +0100, Robin Murphy wrote:
>>>>> [ +ACPI audience ]
>>>>>
>>>>> On 2021-06-25 12:15, Robin Murphy wrote:
>>>>>> On 2021-06-25 12:09, Catalin Marinas wrote:
>>>>>>> On Fri, Jun 25, 2021 at 12:02:52PM +0100, Robin Murphy wrote:
>>>>>>>> On 2021-06-25 10:52, Veronika Kabatova wrote:
>>>>>>>> [...]
>>>>>>>>>>> ❌ stress: stress-ng
>>>>>>>>>>
>>>>>>>>>> Oh no, this looks like another alignment fault in memcpy:
>>>>>>>>>>
>>>>>>>>>> [13330.651903] Unable to handle kernel paging request at
>>>>>>>>>> virtual address ffff8000534705ff
>> [...]
>>>>>>>>>> [13330.652218] Call trace:
>>>>>>>>>> [13330.652221] __memcpy+0x168/0x250
>>>>>>>>>> [13330.652225] acpi_data_show+0x5c/0x8c
>>>>>>>>>> [13330.652232] sysfs_kf_bin_read+0x78/0xa0
>>>>>>>>>> [13330.652238] kernfs_file_read_iter+0x9c/0x1a4
>>>>>>>>>> [13330.652241] kernfs_fop_read_iter+0x34/0x50
>>>>>>>>>> [13330.652244] new_sync_read+0xdc/0x154
>>>>>>>>>> [13330.652253] vfs_read+0x158/0x1e4
>>>>>>>>>> [13330.652260] ksys_read+0x64/0xec
>>>>>>>>>> [13330.652266] __arm64_sys_read+0x28/0x34
>>>>>>>>>> [13330.652273] invoke_syscall+0x50/0x120
>>>>>>>>>> [13330.652280] el0_svc_common.constprop.0+0x4c/0xd4
>>>>>>>>>> [13330.652284] do_el0_svc+0x30/0x9c
>>>>>>>>>> [13330.652286] el0_svc+0x2c/0x54
>>>>>>>>>> [13330.652294] el0t_64_sync_handler+0x1a4/0x1b0
>>>>>>>>>> [13330.652296] el0t_64_sync+0x19c/0x1a0
>>>>>>>>>> [13330.652303] Code: a984346c a9c4342c f1010042 54fffee8 (a97c3c8e)
>>>>>>>>>> [13330.652307] ---[ end trace 227d4380f57145d4 ]---
>>>>>>>>>>
>>>>>>>>>> So maybe this issue isn't limited to weird modules, after all...
>>>>>>>>>
>>>>>>>>> It ran on the machine from the same set that we were able to reproduce
>>>>>>>>> it on previously. If you or anyone else have an idea on how
>>>>>>>>> to stabilize the reproducibility or have a debug patch we'll be happy to try it.
>>>>>>>>
>>>>>>>> Possibly it depends on the individual machines' firmware exactly how the
>>>>>>>> relevant bits of their ACPI tables are aligned in memory?
>>>>>>>>
>>>>>>>> I've started digging into that callstack - it may not be a "weird module"
>>>>>>>> but it's definitely crusty ACPI code... a238317ce818 ("ACPI: Clean up
>>>>>>>> acpi_os_map/unmap_memory() to eliminate __iomem.") looks frankly a bit
>>>>>>>> questionable in its decision to blithely cast away __iomem, but then the
>>>>>>>> rationale in aafc65c731fe ("ACPI: add arm64 to the platforms that use
>>>>>>>> ioremap") seems particularly dubious on top of that (especially
>>>>>>>> given this end result).
>> [...]
>>>>> After picking through the UEFI spec I think I've now got a clearer picture
>>>>> of what's happening, but I'm not sure where it goes from here...
>>>>>
>>>>> The spec implies that it *is* legitimate for runtime-loaded ACPI tables to
>>>>> lie outside the EFI memory map, and that case they must be assumed to be
>>>>> uncached, so the behaviour of acpi_os_ioremap() is correct.
>>>>
>>>> I'd agree with the reasoning, it would be good to pinpoint whether
>>>> that's what actually triggers the issue.
>>>>
>>>> I'd like to replicate it if possible (it is TX2 HW but firmware
>>>> config is likely to differ from the HW I have at hand), the
>>>> test command line that triggers the fault would be useful as
>>>> a starting point.
>>>>
>>>> Furthermore, is this a v5.13-rc* regression ? If so it would be
>>>> good to bisect it - I can't recollect arm64 changes that could
>>>> have introduced this regression in the last cycle but I may have
>>>> missed something.
>>>
>>> The actual change which has brought this to light is the update to arm64's
>>> memcpy() routine for 5.13 - the new version is more aggressive at making
>>> unaligned loads from the source buffer, so now triggers alignment faults
>>> more readily when (wrongly) used on iomem mappings in places that were
>>> getting away with it by chance under the previous implementation (see also
>>> [1], for example).
>>
>> I wouldn't revert any of the memcpy() stuff as it just uncovered an
>> existing bug in how the ACPI tables are handled. Could we actually hit
>> a similar issue with C code parsing the ACPI tables?
>
> I agree - I don't think a revert should be considered, this looks like
> a long standing ACPI bug.
>
> This needs debugging but I believe that it all depends on the table
> being in the EFI map or not. I'd help a lot if I managed to reproduce
> the bug for a given set-up so that we can check which table is causing
> it.
>
>> Is there a way to map the ACPI tables as Normal Noncacheable
>> (ioremap_wc)?
>
> That's a good point. IIUC UEFI 2.9 (2.3.6) requires tables loaded at
> runtime (see above - I really would like to understand what table
> is triggering this bug) that are not in the EFI memory map and whose
> attributes cannot be retrieved through ACPI descriptors to be considered
> non-cacheable.
>
> The question is whether [arm64] acpi_os_ioremap() can be changed so that
> the above is mapped to Normal NC rather than device-nGnRnE; this may
> cause surprises the other way around (given that dev-nGnRnE is an
> all encompassing fallback - again IIUC, I believe Ard knows better
> than me if he has time to chime in).
>
> We need a reproducer and some tracing in the ACPI code.
Having looked even more at the sysfs code, I think this might not
actually be an ACPI table per se, but specifically only the Generic
Error Status Block pointed to by the BERT (so maybe it also requires the
machine to have experienced a boot-time error to be present?). ACPI
merely says that this is "a range of addressable memory" and "System
firmware must report this memory range as firmware reserved", so I have
no idea whether there's any specific expectation of how it's supposed to
be mapped.
Robin.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: ❌ FAIL: Test report for kernel 5.13.0-rc7 (arm-next, 8ab9b1a9)
2021-06-30 11:17 ` Robin Murphy
(?)
@ 2021-06-30 13:22 ` Ard Biesheuvel
2021-06-30 15:49 ` Lorenzo Pieralisi
-1 siblings, 1 reply; 39+ messages in thread
From: Ard Biesheuvel @ 2021-06-30 13:22 UTC (permalink / raw)
To: Robin Murphy
Cc: Lorenzo Pieralisi, Catalin Marinas, ACPI Devel Maling List,
Veronika Kabatova, Will Deacon, CKI Project, Mark Rutland,
Memory Management, skt-results-master, Jeff Bastian, Jan Stancek,
Linux ARM, Rafael J. Wysocki, Len Brown, Hanjun Guo,
Sudeep Holla, lv.zheng, Tony Luck, James Morse
On Wed, 30 Jun 2021 at 13:17, Robin Murphy <robin.murphy@arm.com> wrote:
>
> On 2021-06-30 11:37, Lorenzo Pieralisi wrote:
> > On Tue, Jun 29, 2021 at 05:35:43PM +0100, Catalin Marinas wrote:
> >> On Tue, Jun 29, 2021 at 04:14:55PM +0100, Robin Murphy wrote:
> >>> On 2021-06-29 15:44, Lorenzo Pieralisi wrote:
> >>>> On Tue, Jun 29, 2021 at 12:48:14PM +0100, Robin Murphy wrote:
> >>>>> [ +ACPI audience ]
> >>>>>
> >>>>> On 2021-06-25 12:15, Robin Murphy wrote:
> >>>>>> On 2021-06-25 12:09, Catalin Marinas wrote:
> >>>>>>> On Fri, Jun 25, 2021 at 12:02:52PM +0100, Robin Murphy wrote:
> >>>>>>>> On 2021-06-25 10:52, Veronika Kabatova wrote:
> >>>>>>>> [...]
> >>>>>>>>>>> ❌ stress: stress-ng
> >>>>>>>>>>
> >>>>>>>>>> Oh no, this looks like another alignment fault in memcpy:
> >>>>>>>>>>
> >>>>>>>>>> [13330.651903] Unable to handle kernel paging request at
> >>>>>>>>>> virtual address ffff8000534705ff
> >> [...]
> >>>>>>>>>> [13330.652218] Call trace:
> >>>>>>>>>> [13330.652221] __memcpy+0x168/0x250
> >>>>>>>>>> [13330.652225] acpi_data_show+0x5c/0x8c
> >>>>>>>>>> [13330.652232] sysfs_kf_bin_read+0x78/0xa0
> >>>>>>>>>> [13330.652238] kernfs_file_read_iter+0x9c/0x1a4
> >>>>>>>>>> [13330.652241] kernfs_fop_read_iter+0x34/0x50
> >>>>>>>>>> [13330.652244] new_sync_read+0xdc/0x154
> >>>>>>>>>> [13330.652253] vfs_read+0x158/0x1e4
> >>>>>>>>>> [13330.652260] ksys_read+0x64/0xec
> >>>>>>>>>> [13330.652266] __arm64_sys_read+0x28/0x34
> >>>>>>>>>> [13330.652273] invoke_syscall+0x50/0x120
> >>>>>>>>>> [13330.652280] el0_svc_common.constprop.0+0x4c/0xd4
> >>>>>>>>>> [13330.652284] do_el0_svc+0x30/0x9c
> >>>>>>>>>> [13330.652286] el0_svc+0x2c/0x54
> >>>>>>>>>> [13330.652294] el0t_64_sync_handler+0x1a4/0x1b0
> >>>>>>>>>> [13330.652296] el0t_64_sync+0x19c/0x1a0
> >>>>>>>>>> [13330.652303] Code: a984346c a9c4342c f1010042 54fffee8 (a97c3c8e)
> >>>>>>>>>> [13330.652307] ---[ end trace 227d4380f57145d4 ]---
> >>>>>>>>>>
> >>>>>>>>>> So maybe this issue isn't limited to weird modules, after all...
> >>>>>>>>>
> >>>>>>>>> It ran on the machine from the same set that we were able to reproduce
> >>>>>>>>> it on previously. If you or anyone else have an idea on how
> >>>>>>>>> to stabilize the reproducibility or have a debug patch we'll be happy to try it.
> >>>>>>>>
> >>>>>>>> Possibly it depends on the individual machines' firmware exactly how the
> >>>>>>>> relevant bits of their ACPI tables are aligned in memory?
> >>>>>>>>
> >>>>>>>> I've started digging into that callstack - it may not be a "weird module"
> >>>>>>>> but it's definitely crusty ACPI code... a238317ce818 ("ACPI: Clean up
> >>>>>>>> acpi_os_map/unmap_memory() to eliminate __iomem.") looks frankly a bit
> >>>>>>>> questionable in its decision to blithely cast away __iomem, but then the
> >>>>>>>> rationale in aafc65c731fe ("ACPI: add arm64 to the platforms that use
> >>>>>>>> ioremap") seems particularly dubious on top of that (especially
> >>>>>>>> given this end result).
> >> [...]
> >>>>> After picking through the UEFI spec I think I've now got a clearer picture
> >>>>> of what's happening, but I'm not sure where it goes from here...
> >>>>>
> >>>>> The spec implies that it *is* legitimate for runtime-loaded ACPI tables to
> >>>>> lie outside the EFI memory map, and that case they must be assumed to be
> >>>>> uncached, so the behaviour of acpi_os_ioremap() is correct.
> >>>>
> >>>> I'd agree with the reasoning, it would be good to pinpoint whether
> >>>> that's what actually triggers the issue.
> >>>>
> >>>> I'd like to replicate it if possible (it is TX2 HW but firmware
> >>>> config is likely to differ from the HW I have at hand), the
> >>>> test command line that triggers the fault would be useful as
> >>>> a starting point.
> >>>>
> >>>> Furthermore, is this a v5.13-rc* regression ? If so it would be
> >>>> good to bisect it - I can't recollect arm64 changes that could
> >>>> have introduced this regression in the last cycle but I may have
> >>>> missed something.
> >>>
> >>> The actual change which has brought this to light is the update to arm64's
> >>> memcpy() routine for 5.13 - the new version is more aggressive at making
> >>> unaligned loads from the source buffer, so now triggers alignment faults
> >>> more readily when (wrongly) used on iomem mappings in places that were
> >>> getting away with it by chance under the previous implementation (see also
> >>> [1], for example).
> >>
> >> I wouldn't revert any of the memcpy() stuff as it just uncovered an
> >> existing bug in how the ACPI tables are handled. Could we actually hit
> >> a similar issue with C code parsing the ACPI tables?
> >
> > I agree - I don't think a revert should be considered, this looks like
> > a long standing ACPI bug.
> >
> > This needs debugging but I believe that it all depends on the table
> > being in the EFI map or not. I'd help a lot if I managed to reproduce
> > the bug for a given set-up so that we can check which table is causing
> > it.
> >
> >> Is there a way to map the ACPI tables as Normal Noncacheable
> >> (ioremap_wc)?
> >
> > That's a good point. IIUC UEFI 2.9 (2.3.6) requires tables loaded at
> > runtime (see above - I really would like to understand what table
> > is triggering this bug) that are not in the EFI memory map and whose
> > attributes cannot be retrieved through ACPI descriptors to be considered
> > non-cacheable.
> >
> > The question is whether [arm64] acpi_os_ioremap() can be changed so that
> > the above is mapped to Normal NC rather than device-nGnRnE; this may
> > cause surprises the other way around (given that dev-nGnRnE is an
> > all encompassing fallback - again IIUC, I believe Ard knows better
> > than me if he has time to chime in).
> >
> > We need a reproducer and some tracing in the ACPI code.
>
> Having looked even more at the sysfs code, I think this might not
> actually be an ACPI table per se, but specifically only the Generic
> Error Status Block pointed to by the BERT (so maybe it also requires the
> machine to have experienced a boot-time error to be present?). ACPI
> merely says that this is "a range of addressable memory" and "System
> firmware must report this memory range as firmware reserved", so I have
> no idea whether there's any specific expectation of how it's supposed to
> be mapped.
>
Thanks for digging that up.
If the memory in question is firmware reserved, it should appear in
the EFI memory map, and have the memory type attributes set, in which
case acpi_os_ioremap() should do the right thing.
IIRC (but I don't have time to check - I'm on vacation), the ACPI core
code does have separate code paths internally, but they are all
brought out via acpi_os_ioremap() where MMIO and memory are combined
again. Perhaps we should start looking at addressing this?
So how does the sysfs code find the contents of this file? If some
code is interpreting the BERT, could it be updated to use memory
semantics explicitly when dumping the error status block?
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: ❌ FAIL: Test report for kernel 5.13.0-rc7 (arm-next, 8ab9b1a9)
2021-06-30 13:22 ` Ard Biesheuvel
@ 2021-06-30 15:49 ` Lorenzo Pieralisi
0 siblings, 0 replies; 39+ messages in thread
From: Lorenzo Pieralisi @ 2021-06-30 15:49 UTC (permalink / raw)
To: Ard Biesheuvel
Cc: Robin Murphy, Catalin Marinas, ACPI Devel Maling List,
Veronika Kabatova, Will Deacon, CKI Project, Mark Rutland,
Memory Management, skt-results-master, Jeff Bastian, Jan Stancek,
Linux ARM, Rafael J. Wysocki, Len Brown, Hanjun Guo,
Sudeep Holla, lv.zheng, Tony Luck, James Morse
On Wed, Jun 30, 2021 at 03:22:57PM +0200, Ard Biesheuvel wrote:
> On Wed, 30 Jun 2021 at 13:17, Robin Murphy <robin.murphy@arm.com> wrote:
> >
> > On 2021-06-30 11:37, Lorenzo Pieralisi wrote:
> > > On Tue, Jun 29, 2021 at 05:35:43PM +0100, Catalin Marinas wrote:
> > >> On Tue, Jun 29, 2021 at 04:14:55PM +0100, Robin Murphy wrote:
> > >>> On 2021-06-29 15:44, Lorenzo Pieralisi wrote:
> > >>>> On Tue, Jun 29, 2021 at 12:48:14PM +0100, Robin Murphy wrote:
> > >>>>> [ +ACPI audience ]
> > >>>>>
> > >>>>> On 2021-06-25 12:15, Robin Murphy wrote:
> > >>>>>> On 2021-06-25 12:09, Catalin Marinas wrote:
> > >>>>>>> On Fri, Jun 25, 2021 at 12:02:52PM +0100, Robin Murphy wrote:
> > >>>>>>>> On 2021-06-25 10:52, Veronika Kabatova wrote:
> > >>>>>>>> [...]
> > >>>>>>>>>>> ❌ stress: stress-ng
> > >>>>>>>>>>
> > >>>>>>>>>> Oh no, this looks like another alignment fault in memcpy:
> > >>>>>>>>>>
> > >>>>>>>>>> [13330.651903] Unable to handle kernel paging request at
> > >>>>>>>>>> virtual address ffff8000534705ff
> > >> [...]
> > >>>>>>>>>> [13330.652218] Call trace:
> > >>>>>>>>>> [13330.652221] __memcpy+0x168/0x250
> > >>>>>>>>>> [13330.652225] acpi_data_show+0x5c/0x8c
> > >>>>>>>>>> [13330.652232] sysfs_kf_bin_read+0x78/0xa0
> > >>>>>>>>>> [13330.652238] kernfs_file_read_iter+0x9c/0x1a4
> > >>>>>>>>>> [13330.652241] kernfs_fop_read_iter+0x34/0x50
> > >>>>>>>>>> [13330.652244] new_sync_read+0xdc/0x154
> > >>>>>>>>>> [13330.652253] vfs_read+0x158/0x1e4
> > >>>>>>>>>> [13330.652260] ksys_read+0x64/0xec
> > >>>>>>>>>> [13330.652266] __arm64_sys_read+0x28/0x34
> > >>>>>>>>>> [13330.652273] invoke_syscall+0x50/0x120
> > >>>>>>>>>> [13330.652280] el0_svc_common.constprop.0+0x4c/0xd4
> > >>>>>>>>>> [13330.652284] do_el0_svc+0x30/0x9c
> > >>>>>>>>>> [13330.652286] el0_svc+0x2c/0x54
> > >>>>>>>>>> [13330.652294] el0t_64_sync_handler+0x1a4/0x1b0
> > >>>>>>>>>> [13330.652296] el0t_64_sync+0x19c/0x1a0
> > >>>>>>>>>> [13330.652303] Code: a984346c a9c4342c f1010042 54fffee8 (a97c3c8e)
> > >>>>>>>>>> [13330.652307] ---[ end trace 227d4380f57145d4 ]---
> > >>>>>>>>>>
> > >>>>>>>>>> So maybe this issue isn't limited to weird modules, after all...
> > >>>>>>>>>
> > >>>>>>>>> It ran on the machine from the same set that we were able to reproduce
> > >>>>>>>>> it on previously. If you or anyone else have an idea on how
> > >>>>>>>>> to stabilize the reproducibility or have a debug patch we'll be happy to try it.
> > >>>>>>>>
> > >>>>>>>> Possibly it depends on the individual machines' firmware exactly how the
> > >>>>>>>> relevant bits of their ACPI tables are aligned in memory?
> > >>>>>>>>
> > >>>>>>>> I've started digging into that callstack - it may not be a "weird module"
> > >>>>>>>> but it's definitely crusty ACPI code... a238317ce818 ("ACPI: Clean up
> > >>>>>>>> acpi_os_map/unmap_memory() to eliminate __iomem.") looks frankly a bit
> > >>>>>>>> questionable in its decision to blithely cast away __iomem, but then the
> > >>>>>>>> rationale in aafc65c731fe ("ACPI: add arm64 to the platforms that use
> > >>>>>>>> ioremap") seems particularly dubious on top of that (especially
> > >>>>>>>> given this end result).
> > >> [...]
> > >>>>> After picking through the UEFI spec I think I've now got a clearer picture
> > >>>>> of what's happening, but I'm not sure where it goes from here...
> > >>>>>
> > >>>>> The spec implies that it *is* legitimate for runtime-loaded ACPI tables to
> > >>>>> lie outside the EFI memory map, and that case they must be assumed to be
> > >>>>> uncached, so the behaviour of acpi_os_ioremap() is correct.
> > >>>>
> > >>>> I'd agree with the reasoning, it would be good to pinpoint whether
> > >>>> that's what actually triggers the issue.
> > >>>>
> > >>>> I'd like to replicate it if possible (it is TX2 HW but firmware
> > >>>> config is likely to differ from the HW I have at hand), the
> > >>>> test command line that triggers the fault would be useful as
> > >>>> a starting point.
> > >>>>
> > >>>> Furthermore, is this a v5.13-rc* regression ? If so it would be
> > >>>> good to bisect it - I can't recollect arm64 changes that could
> > >>>> have introduced this regression in the last cycle but I may have
> > >>>> missed something.
> > >>>
> > >>> The actual change which has brought this to light is the update to arm64's
> > >>> memcpy() routine for 5.13 - the new version is more aggressive at making
> > >>> unaligned loads from the source buffer, so now triggers alignment faults
> > >>> more readily when (wrongly) used on iomem mappings in places that were
> > >>> getting away with it by chance under the previous implementation (see also
> > >>> [1], for example).
> > >>
> > >> I wouldn't revert any of the memcpy() stuff as it just uncovered an
> > >> existing bug in how the ACPI tables are handled. Could we actually hit
> > >> a similar issue with C code parsing the ACPI tables?
> > >
> > > I agree - I don't think a revert should be considered, this looks like
> > > a long standing ACPI bug.
> > >
> > > This needs debugging but I believe that it all depends on the table
> > > being in the EFI map or not. I'd help a lot if I managed to reproduce
> > > the bug for a given set-up so that we can check which table is causing
> > > it.
> > >
> > >> Is there a way to map the ACPI tables as Normal Noncacheable
> > >> (ioremap_wc)?
> > >
> > > That's a good point. IIUC UEFI 2.9 (2.3.6) requires tables loaded at
> > > runtime (see above - I really would like to understand what table
> > > is triggering this bug) that are not in the EFI memory map and whose
> > > attributes cannot be retrieved through ACPI descriptors to be considered
> > > non-cacheable.
> > >
> > > The question is whether [arm64] acpi_os_ioremap() can be changed so that
> > > the above is mapped to Normal NC rather than device-nGnRnE; this may
> > > cause surprises the other way around (given that dev-nGnRnE is an
> > > all encompassing fallback - again IIUC, I believe Ard knows better
> > > than me if he has time to chime in).
> > >
> > > We need a reproducer and some tracing in the ACPI code.
> >
> > Having looked even more at the sysfs code, I think this might not
> > actually be an ACPI table per se, but specifically only the Generic
> > Error Status Block pointed to by the BERT (so maybe it also requires the
> > machine to have experienced a boot-time error to be present?). ACPI
> > merely says that this is "a range of addressable memory" and "System
> > firmware must report this memory range as firmware reserved", so I have
> > no idea whether there's any specific expectation of how it's supposed to
> > be mapped.
> >
>
> Thanks for digging that up.
>
> If the memory in question is firmware reserved, it should appear in
> the EFI memory map, and have the memory type attributes set, in which
Thanks a lot Ard for chiming in.
How are those memory type attributes determined by firmware ?
> case acpi_os_ioremap() should do the right thing.
The question is what the right thing is or reworded what those
attributes are supposed to be for the Boot Error Region in question (as
Robin reported, ACPI specs 6.4, 18.3.1, "the Boot Error Region is a
range of addressable memory" and "..must report this memory range
as firmware reserved").
> IIRC (but I don't have time to check - I'm on vacation), the ACPI core
> code does have separate code paths internally, but they are all
> brought out via acpi_os_ioremap() where MMIO and memory are combined
> again. Perhaps we should start looking at addressing this?
BTW, commit in question:
git log -p 7dae6326ed76
I believe you mean, if the OS maps an address with acpi_os_map_memory(),
we must convey the "memory" information to the arm64 back-end (instead
of falling back to acpi_os_ioremap()) so that the back-end can map it
with "memory" semantics (ie by choosing attributes that may need to
override the EFI memory map ones) ?
In current code, even if the BERT were mapped with acpi_os_map_iomem()
this would change nothing since it's acpi_os_ioremap() that runs the
rule (backed up by EFI memory map region info).
> So how does the sysfs code find the contents of this file? If some
> code is interpreting the BERT, could it be updated to use memory
> semantics explicitly when dumping the error status block?
See the commit above. Do you mean replacing the mapping function
in acpi_data_show() with something explicit eg ioremap_wc() through
a back-end specific implementation ?
Thanks,
Lorenzo
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: ❌ FAIL: Test report for kernel 5.13.0-rc7 (arm-next, 8ab9b1a9)
@ 2021-06-30 15:49 ` Lorenzo Pieralisi
0 siblings, 0 replies; 39+ messages in thread
From: Lorenzo Pieralisi @ 2021-06-30 15:49 UTC (permalink / raw)
To: Ard Biesheuvel
Cc: Robin Murphy, Catalin Marinas, ACPI Devel Maling List,
Veronika Kabatova, Will Deacon, CKI Project, Mark Rutland,
Memory Management, skt-results-master, Jeff Bastian, Jan Stancek,
Linux ARM, Rafael J. Wysocki, Len Brown, Hanjun Guo,
Sudeep Holla, lv.zheng, Tony Luck, James Morse
On Wed, Jun 30, 2021 at 03:22:57PM +0200, Ard Biesheuvel wrote:
> On Wed, 30 Jun 2021 at 13:17, Robin Murphy <robin.murphy@arm.com> wrote:
> >
> > On 2021-06-30 11:37, Lorenzo Pieralisi wrote:
> > > On Tue, Jun 29, 2021 at 05:35:43PM +0100, Catalin Marinas wrote:
> > >> On Tue, Jun 29, 2021 at 04:14:55PM +0100, Robin Murphy wrote:
> > >>> On 2021-06-29 15:44, Lorenzo Pieralisi wrote:
> > >>>> On Tue, Jun 29, 2021 at 12:48:14PM +0100, Robin Murphy wrote:
> > >>>>> [ +ACPI audience ]
> > >>>>>
> > >>>>> On 2021-06-25 12:15, Robin Murphy wrote:
> > >>>>>> On 2021-06-25 12:09, Catalin Marinas wrote:
> > >>>>>>> On Fri, Jun 25, 2021 at 12:02:52PM +0100, Robin Murphy wrote:
> > >>>>>>>> On 2021-06-25 10:52, Veronika Kabatova wrote:
> > >>>>>>>> [...]
> > >>>>>>>>>>> ❌ stress: stress-ng
> > >>>>>>>>>>
> > >>>>>>>>>> Oh no, this looks like another alignment fault in memcpy:
> > >>>>>>>>>>
> > >>>>>>>>>> [13330.651903] Unable to handle kernel paging request at
> > >>>>>>>>>> virtual address ffff8000534705ff
> > >> [...]
> > >>>>>>>>>> [13330.652218] Call trace:
> > >>>>>>>>>> [13330.652221] __memcpy+0x168/0x250
> > >>>>>>>>>> [13330.652225] acpi_data_show+0x5c/0x8c
> > >>>>>>>>>> [13330.652232] sysfs_kf_bin_read+0x78/0xa0
> > >>>>>>>>>> [13330.652238] kernfs_file_read_iter+0x9c/0x1a4
> > >>>>>>>>>> [13330.652241] kernfs_fop_read_iter+0x34/0x50
> > >>>>>>>>>> [13330.652244] new_sync_read+0xdc/0x154
> > >>>>>>>>>> [13330.652253] vfs_read+0x158/0x1e4
> > >>>>>>>>>> [13330.652260] ksys_read+0x64/0xec
> > >>>>>>>>>> [13330.652266] __arm64_sys_read+0x28/0x34
> > >>>>>>>>>> [13330.652273] invoke_syscall+0x50/0x120
> > >>>>>>>>>> [13330.652280] el0_svc_common.constprop.0+0x4c/0xd4
> > >>>>>>>>>> [13330.652284] do_el0_svc+0x30/0x9c
> > >>>>>>>>>> [13330.652286] el0_svc+0x2c/0x54
> > >>>>>>>>>> [13330.652294] el0t_64_sync_handler+0x1a4/0x1b0
> > >>>>>>>>>> [13330.652296] el0t_64_sync+0x19c/0x1a0
> > >>>>>>>>>> [13330.652303] Code: a984346c a9c4342c f1010042 54fffee8 (a97c3c8e)
> > >>>>>>>>>> [13330.652307] ---[ end trace 227d4380f57145d4 ]---
> > >>>>>>>>>>
> > >>>>>>>>>> So maybe this issue isn't limited to weird modules, after all...
> > >>>>>>>>>
> > >>>>>>>>> It ran on the machine from the same set that we were able to reproduce
> > >>>>>>>>> it on previously. If you or anyone else have an idea on how
> > >>>>>>>>> to stabilize the reproducibility or have a debug patch we'll be happy to try it.
> > >>>>>>>>
> > >>>>>>>> Possibly it depends on the individual machines' firmware exactly how the
> > >>>>>>>> relevant bits of their ACPI tables are aligned in memory?
> > >>>>>>>>
> > >>>>>>>> I've started digging into that callstack - it may not be a "weird module"
> > >>>>>>>> but it's definitely crusty ACPI code... a238317ce818 ("ACPI: Clean up
> > >>>>>>>> acpi_os_map/unmap_memory() to eliminate __iomem.") looks frankly a bit
> > >>>>>>>> questionable in its decision to blithely cast away __iomem, but then the
> > >>>>>>>> rationale in aafc65c731fe ("ACPI: add arm64 to the platforms that use
> > >>>>>>>> ioremap") seems particularly dubious on top of that (especially
> > >>>>>>>> given this end result).
> > >> [...]
> > >>>>> After picking through the UEFI spec I think I've now got a clearer picture
> > >>>>> of what's happening, but I'm not sure where it goes from here...
> > >>>>>
> > >>>>> The spec implies that it *is* legitimate for runtime-loaded ACPI tables to
> > >>>>> lie outside the EFI memory map, and that case they must be assumed to be
> > >>>>> uncached, so the behaviour of acpi_os_ioremap() is correct.
> > >>>>
> > >>>> I'd agree with the reasoning, it would be good to pinpoint whether
> > >>>> that's what actually triggers the issue.
> > >>>>
> > >>>> I'd like to replicate it if possible (it is TX2 HW but firmware
> > >>>> config is likely to differ from the HW I have at hand), the
> > >>>> test command line that triggers the fault would be useful as
> > >>>> a starting point.
> > >>>>
> > >>>> Furthermore, is this a v5.13-rc* regression ? If so it would be
> > >>>> good to bisect it - I can't recollect arm64 changes that could
> > >>>> have introduced this regression in the last cycle but I may have
> > >>>> missed something.
> > >>>
> > >>> The actual change which has brought this to light is the update to arm64's
> > >>> memcpy() routine for 5.13 - the new version is more aggressive at making
> > >>> unaligned loads from the source buffer, so now triggers alignment faults
> > >>> more readily when (wrongly) used on iomem mappings in places that were
> > >>> getting away with it by chance under the previous implementation (see also
> > >>> [1], for example).
> > >>
> > >> I wouldn't revert any of the memcpy() stuff as it just uncovered an
> > >> existing bug in how the ACPI tables are handled. Could we actually hit
> > >> a similar issue with C code parsing the ACPI tables?
> > >
> > > I agree - I don't think a revert should be considered, this looks like
> > > a long standing ACPI bug.
> > >
> > > This needs debugging but I believe that it all depends on the table
> > > being in the EFI map or not. I'd help a lot if I managed to reproduce
> > > the bug for a given set-up so that we can check which table is causing
> > > it.
> > >
> > >> Is there a way to map the ACPI tables as Normal Noncacheable
> > >> (ioremap_wc)?
> > >
> > > That's a good point. IIUC UEFI 2.9 (2.3.6) requires tables loaded at
> > > runtime (see above - I really would like to understand what table
> > > is triggering this bug) that are not in the EFI memory map and whose
> > > attributes cannot be retrieved through ACPI descriptors to be considered
> > > non-cacheable.
> > >
> > > The question is whether [arm64] acpi_os_ioremap() can be changed so that
> > > the above is mapped to Normal NC rather than device-nGnRnE; this may
> > > cause surprises the other way around (given that dev-nGnRnE is an
> > > all encompassing fallback - again IIUC, I believe Ard knows better
> > > than me if he has time to chime in).
> > >
> > > We need a reproducer and some tracing in the ACPI code.
> >
> > Having looked even more at the sysfs code, I think this might not
> > actually be an ACPI table per se, but specifically only the Generic
> > Error Status Block pointed to by the BERT (so maybe it also requires the
> > machine to have experienced a boot-time error to be present?). ACPI
> > merely says that this is "a range of addressable memory" and "System
> > firmware must report this memory range as firmware reserved", so I have
> > no idea whether there's any specific expectation of how it's supposed to
> > be mapped.
> >
>
> Thanks for digging that up.
>
> If the memory in question is firmware reserved, it should appear in
> the EFI memory map, and have the memory type attributes set, in which
Thanks a lot Ard for chiming in.
How are those memory type attributes determined by firmware ?
> case acpi_os_ioremap() should do the right thing.
The question is what the right thing is or reworded what those
attributes are supposed to be for the Boot Error Region in question (as
Robin reported, ACPI specs 6.4, 18.3.1, "the Boot Error Region is a
range of addressable memory" and "..must report this memory range
as firmware reserved").
> IIRC (but I don't have time to check - I'm on vacation), the ACPI core
> code does have separate code paths internally, but they are all
> brought out via acpi_os_ioremap() where MMIO and memory are combined
> again. Perhaps we should start looking at addressing this?
BTW, commit in question:
git log -p 7dae6326ed76
I believe you mean, if the OS maps an address with acpi_os_map_memory(),
we must convey the "memory" information to the arm64 back-end (instead
of falling back to acpi_os_ioremap()) so that the back-end can map it
with "memory" semantics (ie by choosing attributes that may need to
override the EFI memory map ones) ?
In current code, even if the BERT were mapped with acpi_os_map_iomem()
this would change nothing since it's acpi_os_ioremap() that runs the
rule (backed up by EFI memory map region info).
> So how does the sysfs code find the contents of this file? If some
> code is interpreting the BERT, could it be updated to use memory
> semantics explicitly when dumping the error status block?
See the commit above. Do you mean replacing the mapping function
in acpi_data_show() with something explicit eg ioremap_wc() through
a back-end specific implementation ?
Thanks,
Lorenzo
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: ❌ FAIL: Test report for kernel 5.13.0-rc7 (arm-next, 8ab9b1a9)
2021-06-30 15:49 ` Lorenzo Pieralisi
(?)
@ 2021-06-30 18:18 ` Ard Biesheuvel
2021-07-05 16:17 ` Lorenzo Pieralisi
-1 siblings, 1 reply; 39+ messages in thread
From: Ard Biesheuvel @ 2021-06-30 18:18 UTC (permalink / raw)
To: Lorenzo Pieralisi
Cc: Robin Murphy, Catalin Marinas, ACPI Devel Maling List,
Veronika Kabatova, Will Deacon, CKI Project, Mark Rutland,
Memory Management, skt-results-master, Jeff Bastian, Jan Stancek,
Linux ARM, Rafael J. Wysocki, Len Brown, Hanjun Guo,
Sudeep Holla, lv.zheng, Tony Luck, James Morse
On Wed, 30 Jun 2021 at 17:49, Lorenzo Pieralisi
<lorenzo.pieralisi@arm.com> wrote:
>
> On Wed, Jun 30, 2021 at 03:22:57PM +0200, Ard Biesheuvel wrote:
> > On Wed, 30 Jun 2021 at 13:17, Robin Murphy <robin.murphy@arm.com> wrote:
> > >
> > > On 2021-06-30 11:37, Lorenzo Pieralisi wrote:
> > > > On Tue, Jun 29, 2021 at 05:35:43PM +0100, Catalin Marinas wrote:
> > > >> On Tue, Jun 29, 2021 at 04:14:55PM +0100, Robin Murphy wrote:
> > > >>> On 2021-06-29 15:44, Lorenzo Pieralisi wrote:
> > > >>>> On Tue, Jun 29, 2021 at 12:48:14PM +0100, Robin Murphy wrote:
> > > >>>>> [ +ACPI audience ]
> > > >>>>>
> > > >>>>> On 2021-06-25 12:15, Robin Murphy wrote:
> > > >>>>>> On 2021-06-25 12:09, Catalin Marinas wrote:
> > > >>>>>>> On Fri, Jun 25, 2021 at 12:02:52PM +0100, Robin Murphy wrote:
> > > >>>>>>>> On 2021-06-25 10:52, Veronika Kabatova wrote:
> > > >>>>>>>> [...]
> > > >>>>>>>>>>> ❌ stress: stress-ng
> > > >>>>>>>>>>
> > > >>>>>>>>>> Oh no, this looks like another alignment fault in memcpy:
> > > >>>>>>>>>>
> > > >>>>>>>>>> [13330.651903] Unable to handle kernel paging request at
> > > >>>>>>>>>> virtual address ffff8000534705ff
> > > >> [...]
> > > >>>>>>>>>> [13330.652218] Call trace:
> > > >>>>>>>>>> [13330.652221] __memcpy+0x168/0x250
> > > >>>>>>>>>> [13330.652225] acpi_data_show+0x5c/0x8c
> > > >>>>>>>>>> [13330.652232] sysfs_kf_bin_read+0x78/0xa0
> > > >>>>>>>>>> [13330.652238] kernfs_file_read_iter+0x9c/0x1a4
> > > >>>>>>>>>> [13330.652241] kernfs_fop_read_iter+0x34/0x50
> > > >>>>>>>>>> [13330.652244] new_sync_read+0xdc/0x154
> > > >>>>>>>>>> [13330.652253] vfs_read+0x158/0x1e4
> > > >>>>>>>>>> [13330.652260] ksys_read+0x64/0xec
> > > >>>>>>>>>> [13330.652266] __arm64_sys_read+0x28/0x34
> > > >>>>>>>>>> [13330.652273] invoke_syscall+0x50/0x120
> > > >>>>>>>>>> [13330.652280] el0_svc_common.constprop.0+0x4c/0xd4
> > > >>>>>>>>>> [13330.652284] do_el0_svc+0x30/0x9c
> > > >>>>>>>>>> [13330.652286] el0_svc+0x2c/0x54
> > > >>>>>>>>>> [13330.652294] el0t_64_sync_handler+0x1a4/0x1b0
> > > >>>>>>>>>> [13330.652296] el0t_64_sync+0x19c/0x1a0
> > > >>>>>>>>>> [13330.652303] Code: a984346c a9c4342c f1010042 54fffee8 (a97c3c8e)
> > > >>>>>>>>>> [13330.652307] ---[ end trace 227d4380f57145d4 ]---
> > > >>>>>>>>>>
> > > >>>>>>>>>> So maybe this issue isn't limited to weird modules, after all...
> > > >>>>>>>>>
> > > >>>>>>>>> It ran on the machine from the same set that we were able to reproduce
> > > >>>>>>>>> it on previously. If you or anyone else have an idea on how
> > > >>>>>>>>> to stabilize the reproducibility or have a debug patch we'll be happy to try it.
> > > >>>>>>>>
> > > >>>>>>>> Possibly it depends on the individual machines' firmware exactly how the
> > > >>>>>>>> relevant bits of their ACPI tables are aligned in memory?
> > > >>>>>>>>
> > > >>>>>>>> I've started digging into that callstack - it may not be a "weird module"
> > > >>>>>>>> but it's definitely crusty ACPI code... a238317ce818 ("ACPI: Clean up
> > > >>>>>>>> acpi_os_map/unmap_memory() to eliminate __iomem.") looks frankly a bit
> > > >>>>>>>> questionable in its decision to blithely cast away __iomem, but then the
> > > >>>>>>>> rationale in aafc65c731fe ("ACPI: add arm64 to the platforms that use
> > > >>>>>>>> ioremap") seems particularly dubious on top of that (especially
> > > >>>>>>>> given this end result).
> > > >> [...]
> > > >>>>> After picking through the UEFI spec I think I've now got a clearer picture
> > > >>>>> of what's happening, but I'm not sure where it goes from here...
> > > >>>>>
> > > >>>>> The spec implies that it *is* legitimate for runtime-loaded ACPI tables to
> > > >>>>> lie outside the EFI memory map, and that case they must be assumed to be
> > > >>>>> uncached, so the behaviour of acpi_os_ioremap() is correct.
> > > >>>>
> > > >>>> I'd agree with the reasoning, it would be good to pinpoint whether
> > > >>>> that's what actually triggers the issue.
> > > >>>>
> > > >>>> I'd like to replicate it if possible (it is TX2 HW but firmware
> > > >>>> config is likely to differ from the HW I have at hand), the
> > > >>>> test command line that triggers the fault would be useful as
> > > >>>> a starting point.
> > > >>>>
> > > >>>> Furthermore, is this a v5.13-rc* regression ? If so it would be
> > > >>>> good to bisect it - I can't recollect arm64 changes that could
> > > >>>> have introduced this regression in the last cycle but I may have
> > > >>>> missed something.
> > > >>>
> > > >>> The actual change which has brought this to light is the update to arm64's
> > > >>> memcpy() routine for 5.13 - the new version is more aggressive at making
> > > >>> unaligned loads from the source buffer, so now triggers alignment faults
> > > >>> more readily when (wrongly) used on iomem mappings in places that were
> > > >>> getting away with it by chance under the previous implementation (see also
> > > >>> [1], for example).
> > > >>
> > > >> I wouldn't revert any of the memcpy() stuff as it just uncovered an
> > > >> existing bug in how the ACPI tables are handled. Could we actually hit
> > > >> a similar issue with C code parsing the ACPI tables?
> > > >
> > > > I agree - I don't think a revert should be considered, this looks like
> > > > a long standing ACPI bug.
> > > >
> > > > This needs debugging but I believe that it all depends on the table
> > > > being in the EFI map or not. I'd help a lot if I managed to reproduce
> > > > the bug for a given set-up so that we can check which table is causing
> > > > it.
> > > >
> > > >> Is there a way to map the ACPI tables as Normal Noncacheable
> > > >> (ioremap_wc)?
> > > >
> > > > That's a good point. IIUC UEFI 2.9 (2.3.6) requires tables loaded at
> > > > runtime (see above - I really would like to understand what table
> > > > is triggering this bug) that are not in the EFI memory map and whose
> > > > attributes cannot be retrieved through ACPI descriptors to be considered
> > > > non-cacheable.
> > > >
> > > > The question is whether [arm64] acpi_os_ioremap() can be changed so that
> > > > the above is mapped to Normal NC rather than device-nGnRnE; this may
> > > > cause surprises the other way around (given that dev-nGnRnE is an
> > > > all encompassing fallback - again IIUC, I believe Ard knows better
> > > > than me if he has time to chime in).
> > > >
> > > > We need a reproducer and some tracing in the ACPI code.
> > >
> > > Having looked even more at the sysfs code, I think this might not
> > > actually be an ACPI table per se, but specifically only the Generic
> > > Error Status Block pointed to by the BERT (so maybe it also requires the
> > > machine to have experienced a boot-time error to be present?). ACPI
> > > merely says that this is "a range of addressable memory" and "System
> > > firmware must report this memory range as firmware reserved", so I have
> > > no idea whether there's any specific expectation of how it's supposed to
> > > be mapped.
> > >
> >
> > Thanks for digging that up.
> >
> > If the memory in question is firmware reserved, it should appear in
> > the EFI memory map, and have the memory type attributes set, in which
>
> Thanks a lot Ard for chiming in.
>
> How are those memory type attributes determined by firmware ?
>
It depends on the implementation, but in typical EDK2 based UEFI/PI
firmware, the memory attributes are assigned to the whole of DRAM, and
simply inherited by the allocated regions as they are being created.
However, in the case of error records, I can imagine there may be
cases where the regions are deliberately defined as uncached, to
ensure the error metadata does not linger somewhere in the cache
hierarchy where other agents are not able to see it.
> > case acpi_os_ioremap() should do the right thing.
>
> The question is what the right thing is or reworded what those
> attributes are supposed to be for the Boot Error Region in question (as
> Robin reported, ACPI specs 6.4, 18.3.1, "the Boot Error Region is a
> range of addressable memory" and "..must report this memory range
> as firmware reserved").
>
That only defines the memory type, not the attributes. I don't think
we should be too presciptive here, but simply ensure that the OS has
all the info it needs to infer whether uncached means normal-nc or
device.
> > IIRC (but I don't have time to check - I'm on vacation), the ACPI core
> > code does have separate code paths internally, but they are all
> > brought out via acpi_os_ioremap() where MMIO and memory are combined
> > again. Perhaps we should start looking at addressing this?
>
> BTW, commit in question:
>
> git log -p 7dae6326ed76
>
> I believe you mean, if the OS maps an address with acpi_os_map_memory(),
> we must convey the "memory" information to the arm64 back-end (instead
> of falling back to acpi_os_ioremap()) so that the back-end can map it
> with "memory" semantics (ie by choosing attributes that may need to
> override the EFI memory map ones) ?
>
Yes.
> In current code, even if the BERT were mapped with acpi_os_map_iomem()
> this would change nothing since it's acpi_os_ioremap() that runs the
> rule (backed up by EFI memory map region info).
>
Indeed. So the fact that acpi_os_map_memory() is backed by
acpi_os_ioremap() is something we should fix. So they should both
consult the EFI memory map, but have different fallback defaults if
the region is not annotated correctly.
> > So how does the sysfs code find the contents of this file? If some
> > code is interpreting the BERT, could it be updated to use memory
> > semantics explicitly when dumping the error status block?
>
> See the commit above. Do you mean replacing the mapping function
> in acpi_data_show() with something explicit eg ioremap_wc() through
> a back-end specific implementation ?
>
It seems the sysfs code already does the right thing, but the plumbing
is simply wrong. The API clearly conveys the 'memory' semantics, but
we drop those on the floor before reaching the memremap/ioremap
backend.
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: ❌ FAIL: Test report for kernel 5.13.0-rc7 (arm-next, 8ab9b1a9)
2021-06-30 18:18 ` Ard Biesheuvel
@ 2021-07-05 16:17 ` Lorenzo Pieralisi
0 siblings, 0 replies; 39+ messages in thread
From: Lorenzo Pieralisi @ 2021-07-05 16:17 UTC (permalink / raw)
To: Ard Biesheuvel
Cc: Robin Murphy, Catalin Marinas, ACPI Devel Maling List,
Veronika Kabatova, Will Deacon, CKI Project, Mark Rutland,
Memory Management, skt-results-master, Jeff Bastian, Jan Stancek,
Linux ARM, Rafael J. Wysocki, Len Brown, Hanjun Guo,
Sudeep Holla, lv.zheng, Tony Luck, James Morse
On Wed, Jun 30, 2021 at 08:18:22PM +0200, Ard Biesheuvel wrote:
[...]
> > In current code, even if the BERT were mapped with acpi_os_map_iomem()
> > this would change nothing since it's acpi_os_ioremap() that runs the
> > rule (backed up by EFI memory map region info).
> >
>
> Indeed. So the fact that acpi_os_map_memory() is backed by
> acpi_os_ioremap() is something we should fix. So they should both
> consult the EFI memory map, but have different fallback defaults if
> the region is not annotated correctly.
Put together patch below even though I am not really satisfied, a tad
intrusive and duplicate code in generic/arch backends, compile tested
only; overall this IO vs memory mapping distinction is a bit too fuzzy
for my taste - there is legacy unfortunately to consider though.
-- >8 --
Subject: [PATCH] ACPI: Add memory semantics to acpi_os_map_memory()
Some platforms require memory semantics requested by the mapping function
to be translated into architectural specific memory attributes so that
the mapping is effectively implementing what is expected from it in
terms of allowed access patterns (eg unaligned access).
Rework acpi_os_map_memory() and acpi_os_ioremap() back-end to split
them into two separate code paths that allow the architectural
back-end to detect the default memory attributes required by
the mapping in question.
Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
---
arch/arm64/include/asm/acpi.h | 3 +++
arch/arm64/kernel/acpi.c | 16 ++++++++++++++--
drivers/acpi/osl.c | 23 ++++++++++++++++-------
include/acpi/acpi_io.h | 8 ++++++++
4 files changed, 41 insertions(+), 9 deletions(-)
diff --git a/arch/arm64/include/asm/acpi.h b/arch/arm64/include/asm/acpi.h
index bd68e1b7f29f..7535dc7cc5aa 100644
--- a/arch/arm64/include/asm/acpi.h
+++ b/arch/arm64/include/asm/acpi.h
@@ -50,6 +50,9 @@ pgprot_t __acpi_get_mem_attribute(phys_addr_t addr);
void __iomem *acpi_os_ioremap(acpi_physical_address phys, acpi_size size);
#define acpi_os_ioremap acpi_os_ioremap
+void __iomem *acpi_os_memmap(acpi_physical_address phys, acpi_size size);
+#define acpi_os_memmap acpi_os_memmap
+
typedef u64 phys_cpuid_t;
#define PHYS_CPUID_INVALID INVALID_HWID
diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c
index cada0b816c8a..4c04fb40dc86 100644
--- a/arch/arm64/kernel/acpi.c
+++ b/arch/arm64/kernel/acpi.c
@@ -261,7 +261,8 @@ pgprot_t __acpi_get_mem_attribute(phys_addr_t addr)
return __pgprot(PROT_DEVICE_nGnRnE);
}
-void __iomem *acpi_os_ioremap(acpi_physical_address phys, acpi_size size)
+static void __iomem *__acpi_os_ioremap(acpi_physical_address phys,
+ acpi_size size, bool memory)
{
efi_memory_desc_t *md, *region = NULL;
pgprot_t prot;
@@ -289,7 +290,8 @@ void __iomem *acpi_os_ioremap(acpi_physical_address phys, acpi_size size)
* regions that require a virtual mapping to make them accessible to
* the EFI runtime services.
*/
- prot = __pgprot(PROT_DEVICE_nGnRnE);
+ prot = memory ? __pgprot(PROT_NORMAL_NC) :
+ __pgprot(PROT_DEVICE_nGnRnE);
if (region) {
switch (region->type) {
case EFI_LOADER_CODE:
@@ -349,6 +351,16 @@ void __iomem *acpi_os_ioremap(acpi_physical_address phys, acpi_size size)
return __ioremap(phys, size, prot);
}
+void __iomem *acpi_os_ioremap(acpi_physical_address phys, acpi_size size)
+{
+ return __acpi_os_ioremap(phys, size, false);
+}
+
+void __iomem *acpi_os_memmap(acpi_physical_address phys, acpi_size size)
+{
+ return __acpi_os_ioremap(phys, size, true);
+}
+
/*
* Claim Synchronous External Aborts as a firmware first notification.
*
diff --git a/drivers/acpi/osl.c b/drivers/acpi/osl.c
index 327e1b4eb6b0..01dd115689bf 100644
--- a/drivers/acpi/osl.c
+++ b/drivers/acpi/osl.c
@@ -284,7 +284,8 @@ acpi_map_lookup_virt(void __iomem *virt, acpi_size size)
#define should_use_kmap(pfn) page_is_ram(pfn)
#endif
-static void __iomem *acpi_map(acpi_physical_address pg_off, unsigned long pg_sz)
+static void __iomem *acpi_map(acpi_physical_address pg_off, unsigned long pg_sz,
+ bool memory)
{
unsigned long pfn;
@@ -294,7 +295,8 @@ static void __iomem *acpi_map(acpi_physical_address pg_off, unsigned long pg_sz)
return NULL;
return (void __iomem __force *)kmap(pfn_to_page(pfn));
} else
- return acpi_os_ioremap(pg_off, pg_sz);
+ return memory ? acpi_os_memmap(pg_off, pg_sz) :
+ acpi_os_ioremap(pg_off, pg_sz);
}
static void acpi_unmap(acpi_physical_address pg_off, void __iomem *vaddr)
@@ -309,9 +311,10 @@ static void acpi_unmap(acpi_physical_address pg_off, void __iomem *vaddr)
}
/**
- * acpi_os_map_iomem - Get a virtual address for a given physical address range.
+ * __acpi_os_map_iomem - Get a virtual address for a given physical address range.
* @phys: Start of the physical address range to map.
* @size: Size of the physical address range to map.
+ * @memory: true if remapping memory, false if IO
*
* Look up the given physical address range in the list of existing ACPI memory
* mappings. If found, get a reference to it and return a pointer to it (its
@@ -321,8 +324,8 @@ static void acpi_unmap(acpi_physical_address pg_off, void __iomem *vaddr)
* During early init (when acpi_permanent_mmap has not been set yet) this
* routine simply calls __acpi_map_table() to get the job done.
*/
-void __iomem __ref
-*acpi_os_map_iomem(acpi_physical_address phys, acpi_size size)
+static void __iomem __ref
+*__acpi_os_map_iomem(acpi_physical_address phys, acpi_size size, bool memory)
{
struct acpi_ioremap *map;
void __iomem *virt;
@@ -353,7 +356,7 @@ void __iomem __ref
pg_off = round_down(phys, PAGE_SIZE);
pg_sz = round_up(phys + size, PAGE_SIZE) - pg_off;
- virt = acpi_map(phys, size);
+ virt = acpi_map(phys, size, memory);
if (!virt) {
mutex_unlock(&acpi_ioremap_lock);
kfree(map);
@@ -372,11 +375,17 @@ void __iomem __ref
mutex_unlock(&acpi_ioremap_lock);
return map->virt + (phys - map->phys);
}
+
+void __iomem __ref
+*acpi_os_map_iomem(acpi_physical_address phys, acpi_size size)
+{
+ return __acpi_os_map_iomem(phys, size, false);
+}
EXPORT_SYMBOL_GPL(acpi_os_map_iomem);
void *__ref acpi_os_map_memory(acpi_physical_address phys, acpi_size size)
{
- return (void *)acpi_os_map_iomem(phys, size);
+ return (void *)__acpi_os_map_iomem(phys, size, true);
}
EXPORT_SYMBOL_GPL(acpi_os_map_memory);
diff --git a/include/acpi/acpi_io.h b/include/acpi/acpi_io.h
index 027faa8883aa..a0212e67d6f4 100644
--- a/include/acpi/acpi_io.h
+++ b/include/acpi/acpi_io.h
@@ -14,6 +14,14 @@ static inline void __iomem *acpi_os_ioremap(acpi_physical_address phys,
}
#endif
+#ifndef acpi_os_memmap
+static inline void __iomem *acpi_os_memmap(acpi_physical_address phys,
+ acpi_size size)
+{
+ return ioremap_cache(phys, size);
+}
+#endif
+
extern bool acpi_permanent_mmap;
void __iomem __ref
--
2.29.1
^ permalink raw reply related [flat|nested] 39+ messages in thread
* Re: ❌ FAIL: Test report for kernel 5.13.0-rc7 (arm-next, 8ab9b1a9)
@ 2021-07-05 16:17 ` Lorenzo Pieralisi
0 siblings, 0 replies; 39+ messages in thread
From: Lorenzo Pieralisi @ 2021-07-05 16:17 UTC (permalink / raw)
To: Ard Biesheuvel
Cc: Robin Murphy, Catalin Marinas, ACPI Devel Maling List,
Veronika Kabatova, Will Deacon, CKI Project, Mark Rutland,
Memory Management, skt-results-master, Jeff Bastian, Jan Stancek,
Linux ARM, Rafael J. Wysocki, Len Brown, Hanjun Guo,
Sudeep Holla, lv.zheng, Tony Luck, James Morse
On Wed, Jun 30, 2021 at 08:18:22PM +0200, Ard Biesheuvel wrote:
[...]
> > In current code, even if the BERT were mapped with acpi_os_map_iomem()
> > this would change nothing since it's acpi_os_ioremap() that runs the
> > rule (backed up by EFI memory map region info).
> >
>
> Indeed. So the fact that acpi_os_map_memory() is backed by
> acpi_os_ioremap() is something we should fix. So they should both
> consult the EFI memory map, but have different fallback defaults if
> the region is not annotated correctly.
Put together patch below even though I am not really satisfied, a tad
intrusive and duplicate code in generic/arch backends, compile tested
only; overall this IO vs memory mapping distinction is a bit too fuzzy
for my taste - there is legacy unfortunately to consider though.
-- >8 --
Subject: [PATCH] ACPI: Add memory semantics to acpi_os_map_memory()
Some platforms require memory semantics requested by the mapping function
to be translated into architectural specific memory attributes so that
the mapping is effectively implementing what is expected from it in
terms of allowed access patterns (eg unaligned access).
Rework acpi_os_map_memory() and acpi_os_ioremap() back-end to split
them into two separate code paths that allow the architectural
back-end to detect the default memory attributes required by
the mapping in question.
Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
---
arch/arm64/include/asm/acpi.h | 3 +++
arch/arm64/kernel/acpi.c | 16 ++++++++++++++--
drivers/acpi/osl.c | 23 ++++++++++++++++-------
include/acpi/acpi_io.h | 8 ++++++++
4 files changed, 41 insertions(+), 9 deletions(-)
diff --git a/arch/arm64/include/asm/acpi.h b/arch/arm64/include/asm/acpi.h
index bd68e1b7f29f..7535dc7cc5aa 100644
--- a/arch/arm64/include/asm/acpi.h
+++ b/arch/arm64/include/asm/acpi.h
@@ -50,6 +50,9 @@ pgprot_t __acpi_get_mem_attribute(phys_addr_t addr);
void __iomem *acpi_os_ioremap(acpi_physical_address phys, acpi_size size);
#define acpi_os_ioremap acpi_os_ioremap
+void __iomem *acpi_os_memmap(acpi_physical_address phys, acpi_size size);
+#define acpi_os_memmap acpi_os_memmap
+
typedef u64 phys_cpuid_t;
#define PHYS_CPUID_INVALID INVALID_HWID
diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c
index cada0b816c8a..4c04fb40dc86 100644
--- a/arch/arm64/kernel/acpi.c
+++ b/arch/arm64/kernel/acpi.c
@@ -261,7 +261,8 @@ pgprot_t __acpi_get_mem_attribute(phys_addr_t addr)
return __pgprot(PROT_DEVICE_nGnRnE);
}
-void __iomem *acpi_os_ioremap(acpi_physical_address phys, acpi_size size)
+static void __iomem *__acpi_os_ioremap(acpi_physical_address phys,
+ acpi_size size, bool memory)
{
efi_memory_desc_t *md, *region = NULL;
pgprot_t prot;
@@ -289,7 +290,8 @@ void __iomem *acpi_os_ioremap(acpi_physical_address phys, acpi_size size)
* regions that require a virtual mapping to make them accessible to
* the EFI runtime services.
*/
- prot = __pgprot(PROT_DEVICE_nGnRnE);
+ prot = memory ? __pgprot(PROT_NORMAL_NC) :
+ __pgprot(PROT_DEVICE_nGnRnE);
if (region) {
switch (region->type) {
case EFI_LOADER_CODE:
@@ -349,6 +351,16 @@ void __iomem *acpi_os_ioremap(acpi_physical_address phys, acpi_size size)
return __ioremap(phys, size, prot);
}
+void __iomem *acpi_os_ioremap(acpi_physical_address phys, acpi_size size)
+{
+ return __acpi_os_ioremap(phys, size, false);
+}
+
+void __iomem *acpi_os_memmap(acpi_physical_address phys, acpi_size size)
+{
+ return __acpi_os_ioremap(phys, size, true);
+}
+
/*
* Claim Synchronous External Aborts as a firmware first notification.
*
diff --git a/drivers/acpi/osl.c b/drivers/acpi/osl.c
index 327e1b4eb6b0..01dd115689bf 100644
--- a/drivers/acpi/osl.c
+++ b/drivers/acpi/osl.c
@@ -284,7 +284,8 @@ acpi_map_lookup_virt(void __iomem *virt, acpi_size size)
#define should_use_kmap(pfn) page_is_ram(pfn)
#endif
-static void __iomem *acpi_map(acpi_physical_address pg_off, unsigned long pg_sz)
+static void __iomem *acpi_map(acpi_physical_address pg_off, unsigned long pg_sz,
+ bool memory)
{
unsigned long pfn;
@@ -294,7 +295,8 @@ static void __iomem *acpi_map(acpi_physical_address pg_off, unsigned long pg_sz)
return NULL;
return (void __iomem __force *)kmap(pfn_to_page(pfn));
} else
- return acpi_os_ioremap(pg_off, pg_sz);
+ return memory ? acpi_os_memmap(pg_off, pg_sz) :
+ acpi_os_ioremap(pg_off, pg_sz);
}
static void acpi_unmap(acpi_physical_address pg_off, void __iomem *vaddr)
@@ -309,9 +311,10 @@ static void acpi_unmap(acpi_physical_address pg_off, void __iomem *vaddr)
}
/**
- * acpi_os_map_iomem - Get a virtual address for a given physical address range.
+ * __acpi_os_map_iomem - Get a virtual address for a given physical address range.
* @phys: Start of the physical address range to map.
* @size: Size of the physical address range to map.
+ * @memory: true if remapping memory, false if IO
*
* Look up the given physical address range in the list of existing ACPI memory
* mappings. If found, get a reference to it and return a pointer to it (its
@@ -321,8 +324,8 @@ static void acpi_unmap(acpi_physical_address pg_off, void __iomem *vaddr)
* During early init (when acpi_permanent_mmap has not been set yet) this
* routine simply calls __acpi_map_table() to get the job done.
*/
-void __iomem __ref
-*acpi_os_map_iomem(acpi_physical_address phys, acpi_size size)
+static void __iomem __ref
+*__acpi_os_map_iomem(acpi_physical_address phys, acpi_size size, bool memory)
{
struct acpi_ioremap *map;
void __iomem *virt;
@@ -353,7 +356,7 @@ void __iomem __ref
pg_off = round_down(phys, PAGE_SIZE);
pg_sz = round_up(phys + size, PAGE_SIZE) - pg_off;
- virt = acpi_map(phys, size);
+ virt = acpi_map(phys, size, memory);
if (!virt) {
mutex_unlock(&acpi_ioremap_lock);
kfree(map);
@@ -372,11 +375,17 @@ void __iomem __ref
mutex_unlock(&acpi_ioremap_lock);
return map->virt + (phys - map->phys);
}
+
+void __iomem __ref
+*acpi_os_map_iomem(acpi_physical_address phys, acpi_size size)
+{
+ return __acpi_os_map_iomem(phys, size, false);
+}
EXPORT_SYMBOL_GPL(acpi_os_map_iomem);
void *__ref acpi_os_map_memory(acpi_physical_address phys, acpi_size size)
{
- return (void *)acpi_os_map_iomem(phys, size);
+ return (void *)__acpi_os_map_iomem(phys, size, true);
}
EXPORT_SYMBOL_GPL(acpi_os_map_memory);
diff --git a/include/acpi/acpi_io.h b/include/acpi/acpi_io.h
index 027faa8883aa..a0212e67d6f4 100644
--- a/include/acpi/acpi_io.h
+++ b/include/acpi/acpi_io.h
@@ -14,6 +14,14 @@ static inline void __iomem *acpi_os_ioremap(acpi_physical_address phys,
}
#endif
+#ifndef acpi_os_memmap
+static inline void __iomem *acpi_os_memmap(acpi_physical_address phys,
+ acpi_size size)
+{
+ return ioremap_cache(phys, size);
+}
+#endif
+
extern bool acpi_permanent_mmap;
void __iomem __ref
--
2.29.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 39+ messages in thread
* Re: ❌ FAIL: Test report for kernel 5.13.0-rc7 (arm-next, 8ab9b1a9)
2021-07-05 16:17 ` Lorenzo Pieralisi
(?)
@ 2021-07-16 16:16 ` Ard Biesheuvel
2021-07-16 16:26 ` Lorenzo Pieralisi
-1 siblings, 1 reply; 39+ messages in thread
From: Ard Biesheuvel @ 2021-07-16 16:16 UTC (permalink / raw)
To: Lorenzo Pieralisi
Cc: Robin Murphy, Catalin Marinas, ACPI Devel Maling List,
Veronika Kabatova, Will Deacon, CKI Project, Mark Rutland,
Memory Management, skt-results-master, Jeff Bastian, Jan Stancek,
Linux ARM, Rafael J. Wysocki, Len Brown, Hanjun Guo,
Sudeep Holla, lv.zheng, Tony Luck, James Morse
On Mon, 5 Jul 2021 at 18:17, Lorenzo Pieralisi
<lorenzo.pieralisi@arm.com> wrote:
>
> On Wed, Jun 30, 2021 at 08:18:22PM +0200, Ard Biesheuvel wrote:
>
> [...]
>
> > > In current code, even if the BERT were mapped with acpi_os_map_iomem()
> > > this would change nothing since it's acpi_os_ioremap() that runs the
> > > rule (backed up by EFI memory map region info).
> > >
> >
> > Indeed. So the fact that acpi_os_map_memory() is backed by
> > acpi_os_ioremap() is something we should fix. So they should both
> > consult the EFI memory map, but have different fallback defaults if
> > the region is not annotated correctly.
>
> Put together patch below even though I am not really satisfied, a tad
> intrusive and duplicate code in generic/arch backends, compile tested
> only; overall this IO vs memory mapping distinction is a bit too fuzzy
> for my taste - there is legacy unfortunately to consider though.
>
I'd say that this does not look unreasonable at all. Is there any way
we could get this tested on actual hw?
> -- >8 --
> Subject: [PATCH] ACPI: Add memory semantics to acpi_os_map_memory()
>
> Some platforms require memory semantics requested by the mapping function
> to be translated into architectural specific memory attributes so that
> the mapping is effectively implementing what is expected from it in
> terms of allowed access patterns (eg unaligned access).
>
> Rework acpi_os_map_memory() and acpi_os_ioremap() back-end to split
> them into two separate code paths that allow the architectural
> back-end to detect the default memory attributes required by
> the mapping in question.
>
> Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
> ---
> arch/arm64/include/asm/acpi.h | 3 +++
> arch/arm64/kernel/acpi.c | 16 ++++++++++++++--
> drivers/acpi/osl.c | 23 ++++++++++++++++-------
> include/acpi/acpi_io.h | 8 ++++++++
> 4 files changed, 41 insertions(+), 9 deletions(-)
>
> diff --git a/arch/arm64/include/asm/acpi.h b/arch/arm64/include/asm/acpi.h
> index bd68e1b7f29f..7535dc7cc5aa 100644
> --- a/arch/arm64/include/asm/acpi.h
> +++ b/arch/arm64/include/asm/acpi.h
> @@ -50,6 +50,9 @@ pgprot_t __acpi_get_mem_attribute(phys_addr_t addr);
> void __iomem *acpi_os_ioremap(acpi_physical_address phys, acpi_size size);
> #define acpi_os_ioremap acpi_os_ioremap
>
> +void __iomem *acpi_os_memmap(acpi_physical_address phys, acpi_size size);
> +#define acpi_os_memmap acpi_os_memmap
> +
> typedef u64 phys_cpuid_t;
> #define PHYS_CPUID_INVALID INVALID_HWID
>
> diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c
> index cada0b816c8a..4c04fb40dc86 100644
> --- a/arch/arm64/kernel/acpi.c
> +++ b/arch/arm64/kernel/acpi.c
> @@ -261,7 +261,8 @@ pgprot_t __acpi_get_mem_attribute(phys_addr_t addr)
> return __pgprot(PROT_DEVICE_nGnRnE);
> }
>
> -void __iomem *acpi_os_ioremap(acpi_physical_address phys, acpi_size size)
> +static void __iomem *__acpi_os_ioremap(acpi_physical_address phys,
> + acpi_size size, bool memory)
> {
> efi_memory_desc_t *md, *region = NULL;
> pgprot_t prot;
> @@ -289,7 +290,8 @@ void __iomem *acpi_os_ioremap(acpi_physical_address phys, acpi_size size)
> * regions that require a virtual mapping to make them accessible to
> * the EFI runtime services.
> */
> - prot = __pgprot(PROT_DEVICE_nGnRnE);
> + prot = memory ? __pgprot(PROT_NORMAL_NC) :
> + __pgprot(PROT_DEVICE_nGnRnE);
> if (region) {
> switch (region->type) {
> case EFI_LOADER_CODE:
> @@ -349,6 +351,16 @@ void __iomem *acpi_os_ioremap(acpi_physical_address phys, acpi_size size)
> return __ioremap(phys, size, prot);
> }
>
> +void __iomem *acpi_os_ioremap(acpi_physical_address phys, acpi_size size)
> +{
> + return __acpi_os_ioremap(phys, size, false);
> +}
> +
> +void __iomem *acpi_os_memmap(acpi_physical_address phys, acpi_size size)
> +{
> + return __acpi_os_ioremap(phys, size, true);
> +}
> +
> /*
> * Claim Synchronous External Aborts as a firmware first notification.
> *
> diff --git a/drivers/acpi/osl.c b/drivers/acpi/osl.c
> index 327e1b4eb6b0..01dd115689bf 100644
> --- a/drivers/acpi/osl.c
> +++ b/drivers/acpi/osl.c
> @@ -284,7 +284,8 @@ acpi_map_lookup_virt(void __iomem *virt, acpi_size size)
> #define should_use_kmap(pfn) page_is_ram(pfn)
> #endif
>
> -static void __iomem *acpi_map(acpi_physical_address pg_off, unsigned long pg_sz)
> +static void __iomem *acpi_map(acpi_physical_address pg_off, unsigned long pg_sz,
> + bool memory)
> {
> unsigned long pfn;
>
> @@ -294,7 +295,8 @@ static void __iomem *acpi_map(acpi_physical_address pg_off, unsigned long pg_sz)
> return NULL;
> return (void __iomem __force *)kmap(pfn_to_page(pfn));
> } else
> - return acpi_os_ioremap(pg_off, pg_sz);
> + return memory ? acpi_os_memmap(pg_off, pg_sz) :
> + acpi_os_ioremap(pg_off, pg_sz);
> }
>
> static void acpi_unmap(acpi_physical_address pg_off, void __iomem *vaddr)
> @@ -309,9 +311,10 @@ static void acpi_unmap(acpi_physical_address pg_off, void __iomem *vaddr)
> }
>
> /**
> - * acpi_os_map_iomem - Get a virtual address for a given physical address range.
> + * __acpi_os_map_iomem - Get a virtual address for a given physical address range.
> * @phys: Start of the physical address range to map.
> * @size: Size of the physical address range to map.
> + * @memory: true if remapping memory, false if IO
> *
> * Look up the given physical address range in the list of existing ACPI memory
> * mappings. If found, get a reference to it and return a pointer to it (its
> @@ -321,8 +324,8 @@ static void acpi_unmap(acpi_physical_address pg_off, void __iomem *vaddr)
> * During early init (when acpi_permanent_mmap has not been set yet) this
> * routine simply calls __acpi_map_table() to get the job done.
> */
> -void __iomem __ref
> -*acpi_os_map_iomem(acpi_physical_address phys, acpi_size size)
> +static void __iomem __ref
> +*__acpi_os_map_iomem(acpi_physical_address phys, acpi_size size, bool memory)
> {
> struct acpi_ioremap *map;
> void __iomem *virt;
> @@ -353,7 +356,7 @@ void __iomem __ref
>
> pg_off = round_down(phys, PAGE_SIZE);
> pg_sz = round_up(phys + size, PAGE_SIZE) - pg_off;
> - virt = acpi_map(phys, size);
> + virt = acpi_map(phys, size, memory);
> if (!virt) {
> mutex_unlock(&acpi_ioremap_lock);
> kfree(map);
> @@ -372,11 +375,17 @@ void __iomem __ref
> mutex_unlock(&acpi_ioremap_lock);
> return map->virt + (phys - map->phys);
> }
> +
> +void __iomem __ref
> +*acpi_os_map_iomem(acpi_physical_address phys, acpi_size size)
> +{
> + return __acpi_os_map_iomem(phys, size, false);
> +}
> EXPORT_SYMBOL_GPL(acpi_os_map_iomem);
>
> void *__ref acpi_os_map_memory(acpi_physical_address phys, acpi_size size)
> {
> - return (void *)acpi_os_map_iomem(phys, size);
> + return (void *)__acpi_os_map_iomem(phys, size, true);
> }
> EXPORT_SYMBOL_GPL(acpi_os_map_memory);
>
> diff --git a/include/acpi/acpi_io.h b/include/acpi/acpi_io.h
> index 027faa8883aa..a0212e67d6f4 100644
> --- a/include/acpi/acpi_io.h
> +++ b/include/acpi/acpi_io.h
> @@ -14,6 +14,14 @@ static inline void __iomem *acpi_os_ioremap(acpi_physical_address phys,
> }
> #endif
>
> +#ifndef acpi_os_memmap
> +static inline void __iomem *acpi_os_memmap(acpi_physical_address phys,
> + acpi_size size)
> +{
> + return ioremap_cache(phys, size);
> +}
> +#endif
> +
> extern bool acpi_permanent_mmap;
>
> void __iomem __ref
> --
> 2.29.1
>
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: ❌ FAIL: Test report for kernel 5.13.0-rc7 (arm-next, 8ab9b1a9)
2021-07-16 16:16 ` Ard Biesheuvel
@ 2021-07-16 16:26 ` Lorenzo Pieralisi
0 siblings, 0 replies; 39+ messages in thread
From: Lorenzo Pieralisi @ 2021-07-16 16:26 UTC (permalink / raw)
To: Ard Biesheuvel
Cc: Robin Murphy, Catalin Marinas, ACPI Devel Maling List,
Veronika Kabatova, Will Deacon, CKI Project, Mark Rutland,
Memory Management, skt-results-master, Jeff Bastian, Jan Stancek,
Linux ARM, Rafael J. Wysocki, Len Brown, Hanjun Guo,
Sudeep Holla, lv.zheng, Tony Luck, James Morse
On Fri, Jul 16, 2021 at 06:16:01PM +0200, Ard Biesheuvel wrote:
> On Mon, 5 Jul 2021 at 18:17, Lorenzo Pieralisi
> <lorenzo.pieralisi@arm.com> wrote:
> >
> > On Wed, Jun 30, 2021 at 08:18:22PM +0200, Ard Biesheuvel wrote:
> >
> > [...]
> >
> > > > In current code, even if the BERT were mapped with acpi_os_map_iomem()
> > > > this would change nothing since it's acpi_os_ioremap() that runs the
> > > > rule (backed up by EFI memory map region info).
> > > >
> > >
> > > Indeed. So the fact that acpi_os_map_memory() is backed by
> > > acpi_os_ioremap() is something we should fix. So they should both
> > > consult the EFI memory map, but have different fallback defaults if
> > > the region is not annotated correctly.
> >
> > Put together patch below even though I am not really satisfied, a tad
> > intrusive and duplicate code in generic/arch backends, compile tested
> > only; overall this IO vs memory mapping distinction is a bit too fuzzy
> > for my taste - there is legacy unfortunately to consider though.
> >
>
> I'd say that this does not look unreasonable at all. Is there any way
> we could get this tested on actual hw?
Sure, I was meant to follow-up and was caught up in something else,
sorry.
I will clean up the log, push it out in a branch on Monday, CKI
should pick it up. I will also think about other possible testing
options.
Thanks for having a look !
Lorenzo
> > -- >8 --
> > Subject: [PATCH] ACPI: Add memory semantics to acpi_os_map_memory()
> >
> > Some platforms require memory semantics requested by the mapping function
> > to be translated into architectural specific memory attributes so that
> > the mapping is effectively implementing what is expected from it in
> > terms of allowed access patterns (eg unaligned access).
> >
> > Rework acpi_os_map_memory() and acpi_os_ioremap() back-end to split
> > them into two separate code paths that allow the architectural
> > back-end to detect the default memory attributes required by
> > the mapping in question.
> >
> > Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
> > ---
> > arch/arm64/include/asm/acpi.h | 3 +++
> > arch/arm64/kernel/acpi.c | 16 ++++++++++++++--
> > drivers/acpi/osl.c | 23 ++++++++++++++++-------
> > include/acpi/acpi_io.h | 8 ++++++++
> > 4 files changed, 41 insertions(+), 9 deletions(-)
> >
> > diff --git a/arch/arm64/include/asm/acpi.h b/arch/arm64/include/asm/acpi.h
> > index bd68e1b7f29f..7535dc7cc5aa 100644
> > --- a/arch/arm64/include/asm/acpi.h
> > +++ b/arch/arm64/include/asm/acpi.h
> > @@ -50,6 +50,9 @@ pgprot_t __acpi_get_mem_attribute(phys_addr_t addr);
> > void __iomem *acpi_os_ioremap(acpi_physical_address phys, acpi_size size);
> > #define acpi_os_ioremap acpi_os_ioremap
> >
> > +void __iomem *acpi_os_memmap(acpi_physical_address phys, acpi_size size);
> > +#define acpi_os_memmap acpi_os_memmap
> > +
> > typedef u64 phys_cpuid_t;
> > #define PHYS_CPUID_INVALID INVALID_HWID
> >
> > diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c
> > index cada0b816c8a..4c04fb40dc86 100644
> > --- a/arch/arm64/kernel/acpi.c
> > +++ b/arch/arm64/kernel/acpi.c
> > @@ -261,7 +261,8 @@ pgprot_t __acpi_get_mem_attribute(phys_addr_t addr)
> > return __pgprot(PROT_DEVICE_nGnRnE);
> > }
> >
> > -void __iomem *acpi_os_ioremap(acpi_physical_address phys, acpi_size size)
> > +static void __iomem *__acpi_os_ioremap(acpi_physical_address phys,
> > + acpi_size size, bool memory)
> > {
> > efi_memory_desc_t *md, *region = NULL;
> > pgprot_t prot;
> > @@ -289,7 +290,8 @@ void __iomem *acpi_os_ioremap(acpi_physical_address phys, acpi_size size)
> > * regions that require a virtual mapping to make them accessible to
> > * the EFI runtime services.
> > */
> > - prot = __pgprot(PROT_DEVICE_nGnRnE);
> > + prot = memory ? __pgprot(PROT_NORMAL_NC) :
> > + __pgprot(PROT_DEVICE_nGnRnE);
> > if (region) {
> > switch (region->type) {
> > case EFI_LOADER_CODE:
> > @@ -349,6 +351,16 @@ void __iomem *acpi_os_ioremap(acpi_physical_address phys, acpi_size size)
> > return __ioremap(phys, size, prot);
> > }
> >
> > +void __iomem *acpi_os_ioremap(acpi_physical_address phys, acpi_size size)
> > +{
> > + return __acpi_os_ioremap(phys, size, false);
> > +}
> > +
> > +void __iomem *acpi_os_memmap(acpi_physical_address phys, acpi_size size)
> > +{
> > + return __acpi_os_ioremap(phys, size, true);
> > +}
> > +
> > /*
> > * Claim Synchronous External Aborts as a firmware first notification.
> > *
> > diff --git a/drivers/acpi/osl.c b/drivers/acpi/osl.c
> > index 327e1b4eb6b0..01dd115689bf 100644
> > --- a/drivers/acpi/osl.c
> > +++ b/drivers/acpi/osl.c
> > @@ -284,7 +284,8 @@ acpi_map_lookup_virt(void __iomem *virt, acpi_size size)
> > #define should_use_kmap(pfn) page_is_ram(pfn)
> > #endif
> >
> > -static void __iomem *acpi_map(acpi_physical_address pg_off, unsigned long pg_sz)
> > +static void __iomem *acpi_map(acpi_physical_address pg_off, unsigned long pg_sz,
> > + bool memory)
> > {
> > unsigned long pfn;
> >
> > @@ -294,7 +295,8 @@ static void __iomem *acpi_map(acpi_physical_address pg_off, unsigned long pg_sz)
> > return NULL;
> > return (void __iomem __force *)kmap(pfn_to_page(pfn));
> > } else
> > - return acpi_os_ioremap(pg_off, pg_sz);
> > + return memory ? acpi_os_memmap(pg_off, pg_sz) :
> > + acpi_os_ioremap(pg_off, pg_sz);
> > }
> >
> > static void acpi_unmap(acpi_physical_address pg_off, void __iomem *vaddr)
> > @@ -309,9 +311,10 @@ static void acpi_unmap(acpi_physical_address pg_off, void __iomem *vaddr)
> > }
> >
> > /**
> > - * acpi_os_map_iomem - Get a virtual address for a given physical address range.
> > + * __acpi_os_map_iomem - Get a virtual address for a given physical address range.
> > * @phys: Start of the physical address range to map.
> > * @size: Size of the physical address range to map.
> > + * @memory: true if remapping memory, false if IO
> > *
> > * Look up the given physical address range in the list of existing ACPI memory
> > * mappings. If found, get a reference to it and return a pointer to it (its
> > @@ -321,8 +324,8 @@ static void acpi_unmap(acpi_physical_address pg_off, void __iomem *vaddr)
> > * During early init (when acpi_permanent_mmap has not been set yet) this
> > * routine simply calls __acpi_map_table() to get the job done.
> > */
> > -void __iomem __ref
> > -*acpi_os_map_iomem(acpi_physical_address phys, acpi_size size)
> > +static void __iomem __ref
> > +*__acpi_os_map_iomem(acpi_physical_address phys, acpi_size size, bool memory)
> > {
> > struct acpi_ioremap *map;
> > void __iomem *virt;
> > @@ -353,7 +356,7 @@ void __iomem __ref
> >
> > pg_off = round_down(phys, PAGE_SIZE);
> > pg_sz = round_up(phys + size, PAGE_SIZE) - pg_off;
> > - virt = acpi_map(phys, size);
> > + virt = acpi_map(phys, size, memory);
> > if (!virt) {
> > mutex_unlock(&acpi_ioremap_lock);
> > kfree(map);
> > @@ -372,11 +375,17 @@ void __iomem __ref
> > mutex_unlock(&acpi_ioremap_lock);
> > return map->virt + (phys - map->phys);
> > }
> > +
> > +void __iomem __ref
> > +*acpi_os_map_iomem(acpi_physical_address phys, acpi_size size)
> > +{
> > + return __acpi_os_map_iomem(phys, size, false);
> > +}
> > EXPORT_SYMBOL_GPL(acpi_os_map_iomem);
> >
> > void *__ref acpi_os_map_memory(acpi_physical_address phys, acpi_size size)
> > {
> > - return (void *)acpi_os_map_iomem(phys, size);
> > + return (void *)__acpi_os_map_iomem(phys, size, true);
> > }
> > EXPORT_SYMBOL_GPL(acpi_os_map_memory);
> >
> > diff --git a/include/acpi/acpi_io.h b/include/acpi/acpi_io.h
> > index 027faa8883aa..a0212e67d6f4 100644
> > --- a/include/acpi/acpi_io.h
> > +++ b/include/acpi/acpi_io.h
> > @@ -14,6 +14,14 @@ static inline void __iomem *acpi_os_ioremap(acpi_physical_address phys,
> > }
> > #endif
> >
> > +#ifndef acpi_os_memmap
> > +static inline void __iomem *acpi_os_memmap(acpi_physical_address phys,
> > + acpi_size size)
> > +{
> > + return ioremap_cache(phys, size);
> > +}
> > +#endif
> > +
> > extern bool acpi_permanent_mmap;
> >
> > void __iomem __ref
> > --
> > 2.29.1
> >
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: ❌ FAIL: Test report for kernel 5.13.0-rc7 (arm-next, 8ab9b1a9)
@ 2021-07-16 16:26 ` Lorenzo Pieralisi
0 siblings, 0 replies; 39+ messages in thread
From: Lorenzo Pieralisi @ 2021-07-16 16:26 UTC (permalink / raw)
To: Ard Biesheuvel
Cc: Robin Murphy, Catalin Marinas, ACPI Devel Maling List,
Veronika Kabatova, Will Deacon, CKI Project, Mark Rutland,
Memory Management, skt-results-master, Jeff Bastian, Jan Stancek,
Linux ARM, Rafael J. Wysocki, Len Brown, Hanjun Guo,
Sudeep Holla, lv.zheng, Tony Luck, James Morse
On Fri, Jul 16, 2021 at 06:16:01PM +0200, Ard Biesheuvel wrote:
> On Mon, 5 Jul 2021 at 18:17, Lorenzo Pieralisi
> <lorenzo.pieralisi@arm.com> wrote:
> >
> > On Wed, Jun 30, 2021 at 08:18:22PM +0200, Ard Biesheuvel wrote:
> >
> > [...]
> >
> > > > In current code, even if the BERT were mapped with acpi_os_map_iomem()
> > > > this would change nothing since it's acpi_os_ioremap() that runs the
> > > > rule (backed up by EFI memory map region info).
> > > >
> > >
> > > Indeed. So the fact that acpi_os_map_memory() is backed by
> > > acpi_os_ioremap() is something we should fix. So they should both
> > > consult the EFI memory map, but have different fallback defaults if
> > > the region is not annotated correctly.
> >
> > Put together patch below even though I am not really satisfied, a tad
> > intrusive and duplicate code in generic/arch backends, compile tested
> > only; overall this IO vs memory mapping distinction is a bit too fuzzy
> > for my taste - there is legacy unfortunately to consider though.
> >
>
> I'd say that this does not look unreasonable at all. Is there any way
> we could get this tested on actual hw?
Sure, I was meant to follow-up and was caught up in something else,
sorry.
I will clean up the log, push it out in a branch on Monday, CKI
should pick it up. I will also think about other possible testing
options.
Thanks for having a look !
Lorenzo
> > -- >8 --
> > Subject: [PATCH] ACPI: Add memory semantics to acpi_os_map_memory()
> >
> > Some platforms require memory semantics requested by the mapping function
> > to be translated into architectural specific memory attributes so that
> > the mapping is effectively implementing what is expected from it in
> > terms of allowed access patterns (eg unaligned access).
> >
> > Rework acpi_os_map_memory() and acpi_os_ioremap() back-end to split
> > them into two separate code paths that allow the architectural
> > back-end to detect the default memory attributes required by
> > the mapping in question.
> >
> > Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
> > ---
> > arch/arm64/include/asm/acpi.h | 3 +++
> > arch/arm64/kernel/acpi.c | 16 ++++++++++++++--
> > drivers/acpi/osl.c | 23 ++++++++++++++++-------
> > include/acpi/acpi_io.h | 8 ++++++++
> > 4 files changed, 41 insertions(+), 9 deletions(-)
> >
> > diff --git a/arch/arm64/include/asm/acpi.h b/arch/arm64/include/asm/acpi.h
> > index bd68e1b7f29f..7535dc7cc5aa 100644
> > --- a/arch/arm64/include/asm/acpi.h
> > +++ b/arch/arm64/include/asm/acpi.h
> > @@ -50,6 +50,9 @@ pgprot_t __acpi_get_mem_attribute(phys_addr_t addr);
> > void __iomem *acpi_os_ioremap(acpi_physical_address phys, acpi_size size);
> > #define acpi_os_ioremap acpi_os_ioremap
> >
> > +void __iomem *acpi_os_memmap(acpi_physical_address phys, acpi_size size);
> > +#define acpi_os_memmap acpi_os_memmap
> > +
> > typedef u64 phys_cpuid_t;
> > #define PHYS_CPUID_INVALID INVALID_HWID
> >
> > diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c
> > index cada0b816c8a..4c04fb40dc86 100644
> > --- a/arch/arm64/kernel/acpi.c
> > +++ b/arch/arm64/kernel/acpi.c
> > @@ -261,7 +261,8 @@ pgprot_t __acpi_get_mem_attribute(phys_addr_t addr)
> > return __pgprot(PROT_DEVICE_nGnRnE);
> > }
> >
> > -void __iomem *acpi_os_ioremap(acpi_physical_address phys, acpi_size size)
> > +static void __iomem *__acpi_os_ioremap(acpi_physical_address phys,
> > + acpi_size size, bool memory)
> > {
> > efi_memory_desc_t *md, *region = NULL;
> > pgprot_t prot;
> > @@ -289,7 +290,8 @@ void __iomem *acpi_os_ioremap(acpi_physical_address phys, acpi_size size)
> > * regions that require a virtual mapping to make them accessible to
> > * the EFI runtime services.
> > */
> > - prot = __pgprot(PROT_DEVICE_nGnRnE);
> > + prot = memory ? __pgprot(PROT_NORMAL_NC) :
> > + __pgprot(PROT_DEVICE_nGnRnE);
> > if (region) {
> > switch (region->type) {
> > case EFI_LOADER_CODE:
> > @@ -349,6 +351,16 @@ void __iomem *acpi_os_ioremap(acpi_physical_address phys, acpi_size size)
> > return __ioremap(phys, size, prot);
> > }
> >
> > +void __iomem *acpi_os_ioremap(acpi_physical_address phys, acpi_size size)
> > +{
> > + return __acpi_os_ioremap(phys, size, false);
> > +}
> > +
> > +void __iomem *acpi_os_memmap(acpi_physical_address phys, acpi_size size)
> > +{
> > + return __acpi_os_ioremap(phys, size, true);
> > +}
> > +
> > /*
> > * Claim Synchronous External Aborts as a firmware first notification.
> > *
> > diff --git a/drivers/acpi/osl.c b/drivers/acpi/osl.c
> > index 327e1b4eb6b0..01dd115689bf 100644
> > --- a/drivers/acpi/osl.c
> > +++ b/drivers/acpi/osl.c
> > @@ -284,7 +284,8 @@ acpi_map_lookup_virt(void __iomem *virt, acpi_size size)
> > #define should_use_kmap(pfn) page_is_ram(pfn)
> > #endif
> >
> > -static void __iomem *acpi_map(acpi_physical_address pg_off, unsigned long pg_sz)
> > +static void __iomem *acpi_map(acpi_physical_address pg_off, unsigned long pg_sz,
> > + bool memory)
> > {
> > unsigned long pfn;
> >
> > @@ -294,7 +295,8 @@ static void __iomem *acpi_map(acpi_physical_address pg_off, unsigned long pg_sz)
> > return NULL;
> > return (void __iomem __force *)kmap(pfn_to_page(pfn));
> > } else
> > - return acpi_os_ioremap(pg_off, pg_sz);
> > + return memory ? acpi_os_memmap(pg_off, pg_sz) :
> > + acpi_os_ioremap(pg_off, pg_sz);
> > }
> >
> > static void acpi_unmap(acpi_physical_address pg_off, void __iomem *vaddr)
> > @@ -309,9 +311,10 @@ static void acpi_unmap(acpi_physical_address pg_off, void __iomem *vaddr)
> > }
> >
> > /**
> > - * acpi_os_map_iomem - Get a virtual address for a given physical address range.
> > + * __acpi_os_map_iomem - Get a virtual address for a given physical address range.
> > * @phys: Start of the physical address range to map.
> > * @size: Size of the physical address range to map.
> > + * @memory: true if remapping memory, false if IO
> > *
> > * Look up the given physical address range in the list of existing ACPI memory
> > * mappings. If found, get a reference to it and return a pointer to it (its
> > @@ -321,8 +324,8 @@ static void acpi_unmap(acpi_physical_address pg_off, void __iomem *vaddr)
> > * During early init (when acpi_permanent_mmap has not been set yet) this
> > * routine simply calls __acpi_map_table() to get the job done.
> > */
> > -void __iomem __ref
> > -*acpi_os_map_iomem(acpi_physical_address phys, acpi_size size)
> > +static void __iomem __ref
> > +*__acpi_os_map_iomem(acpi_physical_address phys, acpi_size size, bool memory)
> > {
> > struct acpi_ioremap *map;
> > void __iomem *virt;
> > @@ -353,7 +356,7 @@ void __iomem __ref
> >
> > pg_off = round_down(phys, PAGE_SIZE);
> > pg_sz = round_up(phys + size, PAGE_SIZE) - pg_off;
> > - virt = acpi_map(phys, size);
> > + virt = acpi_map(phys, size, memory);
> > if (!virt) {
> > mutex_unlock(&acpi_ioremap_lock);
> > kfree(map);
> > @@ -372,11 +375,17 @@ void __iomem __ref
> > mutex_unlock(&acpi_ioremap_lock);
> > return map->virt + (phys - map->phys);
> > }
> > +
> > +void __iomem __ref
> > +*acpi_os_map_iomem(acpi_physical_address phys, acpi_size size)
> > +{
> > + return __acpi_os_map_iomem(phys, size, false);
> > +}
> > EXPORT_SYMBOL_GPL(acpi_os_map_iomem);
> >
> > void *__ref acpi_os_map_memory(acpi_physical_address phys, acpi_size size)
> > {
> > - return (void *)acpi_os_map_iomem(phys, size);
> > + return (void *)__acpi_os_map_iomem(phys, size, true);
> > }
> > EXPORT_SYMBOL_GPL(acpi_os_map_memory);
> >
> > diff --git a/include/acpi/acpi_io.h b/include/acpi/acpi_io.h
> > index 027faa8883aa..a0212e67d6f4 100644
> > --- a/include/acpi/acpi_io.h
> > +++ b/include/acpi/acpi_io.h
> > @@ -14,6 +14,14 @@ static inline void __iomem *acpi_os_ioremap(acpi_physical_address phys,
> > }
> > #endif
> >
> > +#ifndef acpi_os_memmap
> > +static inline void __iomem *acpi_os_memmap(acpi_physical_address phys,
> > + acpi_size size)
> > +{
> > + return ioremap_cache(phys, size);
> > +}
> > +#endif
> > +
> > extern bool acpi_permanent_mmap;
> >
> > void __iomem __ref
> > --
> > 2.29.1
> >
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: ❌ FAIL: Test report for kernel 5.13.0-rc7 (arm-next, 8ab9b1a9)
2021-07-16 16:26 ` Lorenzo Pieralisi
(?)
@ 2021-07-22 12:38 ` Veronika Kabatova
2021-07-22 13:51 ` Robin Murphy
-1 siblings, 1 reply; 39+ messages in thread
From: Veronika Kabatova @ 2021-07-22 12:38 UTC (permalink / raw)
To: Lorenzo Pieralisi
Cc: Ard Biesheuvel, Robin Murphy, Catalin Marinas,
ACPI Devel Maling List, Will Deacon, CKI Project, Mark Rutland,
Memory Management, skt-results-master, Jeff Bastian, Jan Stancek,
Linux ARM, Rafael J. Wysocki, Len Brown, Hanjun Guo,
Sudeep Holla, lv.zheng, Tony Luck, James Morse
On Fri, Jul 16, 2021 at 6:26 PM Lorenzo Pieralisi
<lorenzo.pieralisi@arm.com> wrote:
>
> On Fri, Jul 16, 2021 at 06:16:01PM +0200, Ard Biesheuvel wrote:
> > On Mon, 5 Jul 2021 at 18:17, Lorenzo Pieralisi
> > <lorenzo.pieralisi@arm.com> wrote:
> > >
> > > On Wed, Jun 30, 2021 at 08:18:22PM +0200, Ard Biesheuvel wrote:
> > >
> > > [...]
> > >
> > > > > In current code, even if the BERT were mapped with acpi_os_map_iomem()
> > > > > this would change nothing since it's acpi_os_ioremap() that runs the
> > > > > rule (backed up by EFI memory map region info).
> > > > >
> > > >
> > > > Indeed. So the fact that acpi_os_map_memory() is backed by
> > > > acpi_os_ioremap() is something we should fix. So they should both
> > > > consult the EFI memory map, but have different fallback defaults if
> > > > the region is not annotated correctly.
> > >
> > > Put together patch below even though I am not really satisfied, a tad
> > > intrusive and duplicate code in generic/arch backends, compile tested
> > > only; overall this IO vs memory mapping distinction is a bit too fuzzy
> > > for my taste - there is legacy unfortunately to consider though.
> > >
> >
> > I'd say that this does not look unreasonable at all. Is there any way
> > we could get this tested on actual hw?
>
> Sure, I was meant to follow-up and was caught up in something else,
> sorry.
>
> I will clean up the log, push it out in a branch on Monday, CKI
> should pick it up. I will also think about other possible testing
> options.
>
Hi,
thanks for the patience with the testing, the stress-ng test couldn't
deal with a new glibc version and had to be fixed and this week
has just been crazy.
I managed to do 2 runs of the updated tree with the stress-ng test
and it didn't hit the problem. Given how unreliably it reproduces it
doesn't mean all that much. I still have one more run pending and
can submit more if needed.
However, we ran into a panic with this tree on a completely
different machine:
https://gitlab.com/-/snippets/2152899/raw/main/snippetfile1.txt
The machine also hit a hardware error during LTP:
https://gitlab.com/-/snippets/2152899/raw/main/snippetfile2.txt
I'm not sure if the cause of these is the HW/firmware of they are
related to any changes in the tree but I'm sending them over as
they contain acpi in the call traces.
Veronika
> Thanks for having a look !
> Lorenzo
>
> > > -- >8 --
> > > Subject: [PATCH] ACPI: Add memory semantics to acpi_os_map_memory()
> > >
> > > Some platforms require memory semantics requested by the mapping function
> > > to be translated into architectural specific memory attributes so that
> > > the mapping is effectively implementing what is expected from it in
> > > terms of allowed access patterns (eg unaligned access).
> > >
> > > Rework acpi_os_map_memory() and acpi_os_ioremap() back-end to split
> > > them into two separate code paths that allow the architectural
> > > back-end to detect the default memory attributes required by
> > > the mapping in question.
> > >
> > > Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
> > > ---
> > > arch/arm64/include/asm/acpi.h | 3 +++
> > > arch/arm64/kernel/acpi.c | 16 ++++++++++++++--
> > > drivers/acpi/osl.c | 23 ++++++++++++++++-------
> > > include/acpi/acpi_io.h | 8 ++++++++
> > > 4 files changed, 41 insertions(+), 9 deletions(-)
> > >
> > > diff --git a/arch/arm64/include/asm/acpi.h b/arch/arm64/include/asm/acpi.h
> > > index bd68e1b7f29f..7535dc7cc5aa 100644
> > > --- a/arch/arm64/include/asm/acpi.h
> > > +++ b/arch/arm64/include/asm/acpi.h
> > > @@ -50,6 +50,9 @@ pgprot_t __acpi_get_mem_attribute(phys_addr_t addr);
> > > void __iomem *acpi_os_ioremap(acpi_physical_address phys, acpi_size size);
> > > #define acpi_os_ioremap acpi_os_ioremap
> > >
> > > +void __iomem *acpi_os_memmap(acpi_physical_address phys, acpi_size size);
> > > +#define acpi_os_memmap acpi_os_memmap
> > > +
> > > typedef u64 phys_cpuid_t;
> > > #define PHYS_CPUID_INVALID INVALID_HWID
> > >
> > > diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c
> > > index cada0b816c8a..4c04fb40dc86 100644
> > > --- a/arch/arm64/kernel/acpi.c
> > > +++ b/arch/arm64/kernel/acpi.c
> > > @@ -261,7 +261,8 @@ pgprot_t __acpi_get_mem_attribute(phys_addr_t addr)
> > > return __pgprot(PROT_DEVICE_nGnRnE);
> > > }
> > >
> > > -void __iomem *acpi_os_ioremap(acpi_physical_address phys, acpi_size size)
> > > +static void __iomem *__acpi_os_ioremap(acpi_physical_address phys,
> > > + acpi_size size, bool memory)
> > > {
> > > efi_memory_desc_t *md, *region = NULL;
> > > pgprot_t prot;
> > > @@ -289,7 +290,8 @@ void __iomem *acpi_os_ioremap(acpi_physical_address phys, acpi_size size)
> > > * regions that require a virtual mapping to make them accessible to
> > > * the EFI runtime services.
> > > */
> > > - prot = __pgprot(PROT_DEVICE_nGnRnE);
> > > + prot = memory ? __pgprot(PROT_NORMAL_NC) :
> > > + __pgprot(PROT_DEVICE_nGnRnE);
> > > if (region) {
> > > switch (region->type) {
> > > case EFI_LOADER_CODE:
> > > @@ -349,6 +351,16 @@ void __iomem *acpi_os_ioremap(acpi_physical_address phys, acpi_size size)
> > > return __ioremap(phys, size, prot);
> > > }
> > >
> > > +void __iomem *acpi_os_ioremap(acpi_physical_address phys, acpi_size size)
> > > +{
> > > + return __acpi_os_ioremap(phys, size, false);
> > > +}
> > > +
> > > +void __iomem *acpi_os_memmap(acpi_physical_address phys, acpi_size size)
> > > +{
> > > + return __acpi_os_ioremap(phys, size, true);
> > > +}
> > > +
> > > /*
> > > * Claim Synchronous External Aborts as a firmware first notification.
> > > *
> > > diff --git a/drivers/acpi/osl.c b/drivers/acpi/osl.c
> > > index 327e1b4eb6b0..01dd115689bf 100644
> > > --- a/drivers/acpi/osl.c
> > > +++ b/drivers/acpi/osl.c
> > > @@ -284,7 +284,8 @@ acpi_map_lookup_virt(void __iomem *virt, acpi_size size)
> > > #define should_use_kmap(pfn) page_is_ram(pfn)
> > > #endif
> > >
> > > -static void __iomem *acpi_map(acpi_physical_address pg_off, unsigned long pg_sz)
> > > +static void __iomem *acpi_map(acpi_physical_address pg_off, unsigned long pg_sz,
> > > + bool memory)
> > > {
> > > unsigned long pfn;
> > >
> > > @@ -294,7 +295,8 @@ static void __iomem *acpi_map(acpi_physical_address pg_off, unsigned long pg_sz)
> > > return NULL;
> > > return (void __iomem __force *)kmap(pfn_to_page(pfn));
> > > } else
> > > - return acpi_os_ioremap(pg_off, pg_sz);
> > > + return memory ? acpi_os_memmap(pg_off, pg_sz) :
> > > + acpi_os_ioremap(pg_off, pg_sz);
> > > }
> > >
> > > static void acpi_unmap(acpi_physical_address pg_off, void __iomem *vaddr)
> > > @@ -309,9 +311,10 @@ static void acpi_unmap(acpi_physical_address pg_off, void __iomem *vaddr)
> > > }
> > >
> > > /**
> > > - * acpi_os_map_iomem - Get a virtual address for a given physical address range.
> > > + * __acpi_os_map_iomem - Get a virtual address for a given physical address range.
> > > * @phys: Start of the physical address range to map.
> > > * @size: Size of the physical address range to map.
> > > + * @memory: true if remapping memory, false if IO
> > > *
> > > * Look up the given physical address range in the list of existing ACPI memory
> > > * mappings. If found, get a reference to it and return a pointer to it (its
> > > @@ -321,8 +324,8 @@ static void acpi_unmap(acpi_physical_address pg_off, void __iomem *vaddr)
> > > * During early init (when acpi_permanent_mmap has not been set yet) this
> > > * routine simply calls __acpi_map_table() to get the job done.
> > > */
> > > -void __iomem __ref
> > > -*acpi_os_map_iomem(acpi_physical_address phys, acpi_size size)
> > > +static void __iomem __ref
> > > +*__acpi_os_map_iomem(acpi_physical_address phys, acpi_size size, bool memory)
> > > {
> > > struct acpi_ioremap *map;
> > > void __iomem *virt;
> > > @@ -353,7 +356,7 @@ void __iomem __ref
> > >
> > > pg_off = round_down(phys, PAGE_SIZE);
> > > pg_sz = round_up(phys + size, PAGE_SIZE) - pg_off;
> > > - virt = acpi_map(phys, size);
> > > + virt = acpi_map(phys, size, memory);
> > > if (!virt) {
> > > mutex_unlock(&acpi_ioremap_lock);
> > > kfree(map);
> > > @@ -372,11 +375,17 @@ void __iomem __ref
> > > mutex_unlock(&acpi_ioremap_lock);
> > > return map->virt + (phys - map->phys);
> > > }
> > > +
> > > +void __iomem __ref
> > > +*acpi_os_map_iomem(acpi_physical_address phys, acpi_size size)
> > > +{
> > > + return __acpi_os_map_iomem(phys, size, false);
> > > +}
> > > EXPORT_SYMBOL_GPL(acpi_os_map_iomem);
> > >
> > > void *__ref acpi_os_map_memory(acpi_physical_address phys, acpi_size size)
> > > {
> > > - return (void *)acpi_os_map_iomem(phys, size);
> > > + return (void *)__acpi_os_map_iomem(phys, size, true);
> > > }
> > > EXPORT_SYMBOL_GPL(acpi_os_map_memory);
> > >
> > > diff --git a/include/acpi/acpi_io.h b/include/acpi/acpi_io.h
> > > index 027faa8883aa..a0212e67d6f4 100644
> > > --- a/include/acpi/acpi_io.h
> > > +++ b/include/acpi/acpi_io.h
> > > @@ -14,6 +14,14 @@ static inline void __iomem *acpi_os_ioremap(acpi_physical_address phys,
> > > }
> > > #endif
> > >
> > > +#ifndef acpi_os_memmap
> > > +static inline void __iomem *acpi_os_memmap(acpi_physical_address phys,
> > > + acpi_size size)
> > > +{
> > > + return ioremap_cache(phys, size);
> > > +}
> > > +#endif
> > > +
> > > extern bool acpi_permanent_mmap;
> > >
> > > void __iomem __ref
> > > --
> > > 2.29.1
> > >
>
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: ❌ FAIL: Test report for kernel 5.13.0-rc7 (arm-next, 8ab9b1a9)
2021-07-22 12:38 ` Veronika Kabatova
@ 2021-07-22 13:51 ` Robin Murphy
0 siblings, 0 replies; 39+ messages in thread
From: Robin Murphy @ 2021-07-22 13:51 UTC (permalink / raw)
To: Veronika Kabatova, Lorenzo Pieralisi
Cc: Ard Biesheuvel, Catalin Marinas, ACPI Devel Maling List,
Will Deacon, CKI Project, Mark Rutland, Memory Management,
skt-results-master, Jeff Bastian, Jan Stancek, Linux ARM,
Rafael J. Wysocki, Len Brown, Hanjun Guo, Sudeep Holla, lv.zheng,
Tony Luck, James Morse
On 2021-07-22 13:38, Veronika Kabatova wrote:
> On Fri, Jul 16, 2021 at 6:26 PM Lorenzo Pieralisi
> <lorenzo.pieralisi@arm.com> wrote:
>>
>> On Fri, Jul 16, 2021 at 06:16:01PM +0200, Ard Biesheuvel wrote:
>>> On Mon, 5 Jul 2021 at 18:17, Lorenzo Pieralisi
>>> <lorenzo.pieralisi@arm.com> wrote:
>>>>
>>>> On Wed, Jun 30, 2021 at 08:18:22PM +0200, Ard Biesheuvel wrote:
>>>>
>>>> [...]
>>>>
>>>>>> In current code, even if the BERT were mapped with acpi_os_map_iomem()
>>>>>> this would change nothing since it's acpi_os_ioremap() that runs the
>>>>>> rule (backed up by EFI memory map region info).
>>>>>>
>>>>>
>>>>> Indeed. So the fact that acpi_os_map_memory() is backed by
>>>>> acpi_os_ioremap() is something we should fix. So they should both
>>>>> consult the EFI memory map, but have different fallback defaults if
>>>>> the region is not annotated correctly.
>>>>
>>>> Put together patch below even though I am not really satisfied, a tad
>>>> intrusive and duplicate code in generic/arch backends, compile tested
>>>> only; overall this IO vs memory mapping distinction is a bit too fuzzy
>>>> for my taste - there is legacy unfortunately to consider though.
>>>>
>>>
>>> I'd say that this does not look unreasonable at all. Is there any way
>>> we could get this tested on actual hw?
>>
>> Sure, I was meant to follow-up and was caught up in something else,
>> sorry.
>>
>> I will clean up the log, push it out in a branch on Monday, CKI
>> should pick it up. I will also think about other possible testing
>> options.
>>
>
> Hi,
>
> thanks for the patience with the testing, the stress-ng test couldn't
> deal with a new glibc version and had to be fixed and this week
> has just been crazy.
>
> I managed to do 2 runs of the updated tree with the stress-ng test
> and it didn't hit the problem. Given how unreliably it reproduces it
> doesn't mean all that much. I still have one more run pending and
> can submit more if needed.
>
> However, we ran into a panic with this tree on a completely
> different machine:
>
> https://gitlab.com/-/snippets/2152899/raw/main/snippetfile1.txt
All the warnings from arch_setup_dma_ops() there are (unfortunately)
pretty much legitimate for that platform, and should be gone again since
rc2 with commit c1132702c71f.
> The machine also hit a hardware error during LTP:
>
> https://gitlab.com/-/snippets/2152899/raw/main/snippetfile2.txt
Hmm, if "access mode: secure" in that fault report implies that the
firmnware itself has done something dodgy to raise an SError, I'm not
sure there's much we can do about that...
Robin.
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: ❌ FAIL: Test report for kernel 5.13.0-rc7 (arm-next, 8ab9b1a9)
@ 2021-07-22 13:51 ` Robin Murphy
0 siblings, 0 replies; 39+ messages in thread
From: Robin Murphy @ 2021-07-22 13:51 UTC (permalink / raw)
To: Veronika Kabatova, Lorenzo Pieralisi
Cc: Ard Biesheuvel, Catalin Marinas, ACPI Devel Maling List,
Will Deacon, CKI Project, Mark Rutland, Memory Management,
skt-results-master, Jeff Bastian, Jan Stancek, Linux ARM,
Rafael J. Wysocki, Len Brown, Hanjun Guo, Sudeep Holla, lv.zheng,
Tony Luck, James Morse
On 2021-07-22 13:38, Veronika Kabatova wrote:
> On Fri, Jul 16, 2021 at 6:26 PM Lorenzo Pieralisi
> <lorenzo.pieralisi@arm.com> wrote:
>>
>> On Fri, Jul 16, 2021 at 06:16:01PM +0200, Ard Biesheuvel wrote:
>>> On Mon, 5 Jul 2021 at 18:17, Lorenzo Pieralisi
>>> <lorenzo.pieralisi@arm.com> wrote:
>>>>
>>>> On Wed, Jun 30, 2021 at 08:18:22PM +0200, Ard Biesheuvel wrote:
>>>>
>>>> [...]
>>>>
>>>>>> In current code, even if the BERT were mapped with acpi_os_map_iomem()
>>>>>> this would change nothing since it's acpi_os_ioremap() that runs the
>>>>>> rule (backed up by EFI memory map region info).
>>>>>>
>>>>>
>>>>> Indeed. So the fact that acpi_os_map_memory() is backed by
>>>>> acpi_os_ioremap() is something we should fix. So they should both
>>>>> consult the EFI memory map, but have different fallback defaults if
>>>>> the region is not annotated correctly.
>>>>
>>>> Put together patch below even though I am not really satisfied, a tad
>>>> intrusive and duplicate code in generic/arch backends, compile tested
>>>> only; overall this IO vs memory mapping distinction is a bit too fuzzy
>>>> for my taste - there is legacy unfortunately to consider though.
>>>>
>>>
>>> I'd say that this does not look unreasonable at all. Is there any way
>>> we could get this tested on actual hw?
>>
>> Sure, I was meant to follow-up and was caught up in something else,
>> sorry.
>>
>> I will clean up the log, push it out in a branch on Monday, CKI
>> should pick it up. I will also think about other possible testing
>> options.
>>
>
> Hi,
>
> thanks for the patience with the testing, the stress-ng test couldn't
> deal with a new glibc version and had to be fixed and this week
> has just been crazy.
>
> I managed to do 2 runs of the updated tree with the stress-ng test
> and it didn't hit the problem. Given how unreliably it reproduces it
> doesn't mean all that much. I still have one more run pending and
> can submit more if needed.
>
> However, we ran into a panic with this tree on a completely
> different machine:
>
> https://gitlab.com/-/snippets/2152899/raw/main/snippetfile1.txt
All the warnings from arch_setup_dma_ops() there are (unfortunately)
pretty much legitimate for that platform, and should be gone again since
rc2 with commit c1132702c71f.
> The machine also hit a hardware error during LTP:
>
> https://gitlab.com/-/snippets/2152899/raw/main/snippetfile2.txt
Hmm, if "access mode: secure" in that fault report implies that the
firmnware itself has done something dodgy to raise an SError, I'm not
sure there's much we can do about that...
Robin.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: ❌ FAIL: Test report for kernel 5.13.0-rc7 (arm-next, 8ab9b1a9)
2021-07-22 13:51 ` Robin Murphy
(?)
@ 2021-07-22 18:23 ` Veronika Kabatova
-1 siblings, 0 replies; 39+ messages in thread
From: Veronika Kabatova @ 2021-07-22 18:23 UTC (permalink / raw)
To: Robin Murphy
Cc: Lorenzo Pieralisi, Mark Rutland, Tony Luck, Catalin Marinas,
Memory Management, Sudeep Holla, Rafael J. Wysocki,
skt-results-master, Jan Stancek, ACPI Devel Maling List,
Jeff Bastian, James Morse, lv.zheng, CKI Project, Hanjun Guo,
Will Deacon, Ard Biesheuvel, Linux ARM, Len Brown
On Thu, Jul 22, 2021 at 3:52 PM Robin Murphy <robin.murphy@arm.com> wrote:
>
> On 2021-07-22 13:38, Veronika Kabatova wrote:
> > On Fri, Jul 16, 2021 at 6:26 PM Lorenzo Pieralisi
> > <lorenzo.pieralisi@arm.com> wrote:
> >>
> >> On Fri, Jul 16, 2021 at 06:16:01PM +0200, Ard Biesheuvel wrote:
> >>> On Mon, 5 Jul 2021 at 18:17, Lorenzo Pieralisi
> >>> <lorenzo.pieralisi@arm.com> wrote:
> >>>>
> >>>> On Wed, Jun 30, 2021 at 08:18:22PM +0200, Ard Biesheuvel wrote:
> >>>>
> >>>> [...]
> >>>>
> >>>>>> In current code, even if the BERT were mapped with acpi_os_map_iomem()
> >>>>>> this would change nothing since it's acpi_os_ioremap() that runs the
> >>>>>> rule (backed up by EFI memory map region info).
> >>>>>>
> >>>>>
> >>>>> Indeed. So the fact that acpi_os_map_memory() is backed by
> >>>>> acpi_os_ioremap() is something we should fix. So they should both
> >>>>> consult the EFI memory map, but have different fallback defaults if
> >>>>> the region is not annotated correctly.
> >>>>
> >>>> Put together patch below even though I am not really satisfied, a tad
> >>>> intrusive and duplicate code in generic/arch backends, compile tested
> >>>> only; overall this IO vs memory mapping distinction is a bit too fuzzy
> >>>> for my taste - there is legacy unfortunately to consider though.
> >>>>
> >>>
> >>> I'd say that this does not look unreasonable at all. Is there any way
> >>> we could get this tested on actual hw?
> >>
> >> Sure, I was meant to follow-up and was caught up in something else,
> >> sorry.
> >>
> >> I will clean up the log, push it out in a branch on Monday, CKI
> >> should pick it up. I will also think about other possible testing
> >> options.
> >>
> >
> > Hi,
> >
> > thanks for the patience with the testing, the stress-ng test couldn't
> > deal with a new glibc version and had to be fixed and this week
> > has just been crazy.
> >
> > I managed to do 2 runs of the updated tree with the stress-ng test
> > and it didn't hit the problem. Given how unreliably it reproduces it
> > doesn't mean all that much. I still have one more run pending and
> > can submit more if needed.
> >
> > However, we ran into a panic with this tree on a completely
> > different machine:
> >
> > https://gitlab.com/-/snippets/2152899/raw/main/snippetfile1.txt
>
> All the warnings from arch_setup_dma_ops() there are (unfortunately)
> pretty much legitimate for that platform, and should be gone again since
> rc2 with commit c1132702c71f.
>
> > The machine also hit a hardware error during LTP:
> >
> > https://gitlab.com/-/snippets/2152899/raw/main/snippetfile2.txt
>
> Hmm, if "access mode: secure" in that fault report implies that the
> firmnware itself has done something dodgy to raise an SError, I'm not
> sure there's much we can do about that...
>
Thank you for checking!
In the meanwhile, the last test job completed and passed as well.
Let me know if you'd like more test runs or if there's anything
else I can help with.
Veronika
> Robin.
>
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: ??? FAIL: Test report for kernel 5.13.0-rc7 (arm-next, 8ab9b1a9)
2021-06-29 17:27 ` Robin Murphy
(?)
(?)
@ 2022-03-04 19:31 ` Aristeu Rozanski
-1 siblings, 0 replies; 39+ messages in thread
From: Aristeu Rozanski @ 2022-03-04 19:31 UTC (permalink / raw)
To: Robin Murphy
Cc: Lorenzo Pieralisi, Catalin Marinas, Will Deacon, linux-arm-kernel
Hi Robin,
(Old thread, so reference to it: https://lists.infradead.org/pipermail/linux-arm-kernel/2021-June/668228.html)
(Also, please keep me on Cc since I'm not subscribed to linux-arm)
On Tue, Jun 29, 2021 at 06:27:57PM +0100, Robin Murphy wrote:
> Ah, from that I can only assume that this must be stress-ng's --sysfs
> test reading things at random, so not only would it have to be on a
> machine whose firmware presents the right thing in the right way but the
> random test conditions would also have to line up to poke it the "right"
> (wrong) way too.
>
> As a temporary workaround for the CI flakes, might it be possible to
> configure stress-ng to stay away from just these ACPI "data" files?
I started looking at this issue and managed to reproduce the issue
instantly with
dd if=/sys/firmware/acpi/tables/data/BERT of=/dev/null bs=7
I've attempted a few ways of fixing it based on the comments on this
thread but wasn't successful so far (my knowledge is pretty limited in
this area too, so not a big surprise). How can I be of assistance to debug/test
patches for this issue?
--
Aristeu
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: ??? FAIL: Test report for kernel 5.13.0-rc7 (arm-next, 8ab9b1a9)
2021-06-29 17:27 ` Robin Murphy
` (2 preceding siblings ...)
(?)
@ 2022-03-04 19:39 ` Aristeu Rozanski
2022-03-04 20:00 ` Robin Murphy
2022-03-07 10:19 ` Lorenzo Pieralisi
-1 siblings, 2 replies; 39+ messages in thread
From: Aristeu Rozanski @ 2022-03-04 19:39 UTC (permalink / raw)
To: Robin Murphy
Cc: Will Deacon, Lorenzo Pieralisi, Catalin Marinas, linux-arm-kernel
Hi Robin,
(Old thread, so reference to it: https://lists.infradead.org/pipermail/linux-arm-kernel/2021-June/668228.html)
(Also, please keep me on Cc since I'm not subscribed to linux-arm)
(And again because I failed to fix the email address after using the
archives mailbox, apologies for that)
On Tue, Jun 29, 2021 at 06:27:57PM +0100, Robin Murphy wrote:
> Ah, from that I can only assume that this must be stress-ng's --sysfs
> test reading things at random, so not only would it have to be on a
> machine whose firmware presents the right thing in the right way but the
> random test conditions would also have to line up to poke it the "right"
> (wrong) way too.
>
> As a temporary workaround for the CI flakes, might it be possible to
> configure stress-ng to stay away from just these ACPI "data" files?
I started looking at this issue and managed to reproduce the issue
instantly with
dd if=/sys/firmware/acpi/tables/data/BERT of=/dev/null bs=7
I've attempted a few ways of fixing it based on the comments on this
thread but wasn't successful so far (my knowledge is pretty limited in
this area too, so not a big surprise). How can I be of assistance to debug/test
patches for this issue?
--
Aristeu
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: ??? FAIL: Test report for kernel 5.13.0-rc7 (arm-next, 8ab9b1a9)
2022-03-04 19:39 ` Aristeu Rozanski
@ 2022-03-04 20:00 ` Robin Murphy
2022-03-07 10:19 ` Lorenzo Pieralisi
1 sibling, 0 replies; 39+ messages in thread
From: Robin Murphy @ 2022-03-04 20:00 UTC (permalink / raw)
To: Aristeu Rozanski
Cc: Will Deacon, Lorenzo Pieralisi, Catalin Marinas, linux-arm-kernel
Hi Aristeu,
On 2022-03-04 19:39, Aristeu Rozanski wrote:
> Hi Robin,
>
> (Old thread, so reference to it: https://lists.infradead.org/pipermail/linux-arm-kernel/2021-June/668228.html)
> (Also, please keep me on Cc since I'm not subscribed to linux-arm)
> (And again because I failed to fix the email address after using the
> archives mailbox, apologies for that)
>
> On Tue, Jun 29, 2021 at 06:27:57PM +0100, Robin Murphy wrote:
>> Ah, from that I can only assume that this must be stress-ng's --sysfs
>> test reading things at random, so not only would it have to be on a
>> machine whose firmware presents the right thing in the right way but the
>> random test conditions would also have to line up to poke it the "right"
>> (wrong) way too.
>>
>> As a temporary workaround for the CI flakes, might it be possible to
>> configure stress-ng to stay away from just these ACPI "data" files?
>
> I started looking at this issue and managed to reproduce the issue
> instantly with
>
> dd if=/sys/firmware/acpi/tables/data/BERT of=/dev/null bs=7
>
> I've attempted a few ways of fixing it based on the comments on this
> thread but wasn't successful so far (my knowledge is pretty limited in
> this area too, so not a big surprise). How can I be of assistance to debug/test
> patches for this issue?
Gosh, there's a fair bit of additional history between then and now,
scattered across several other threads... From memory, we ended up
merging some form of Lorenzo's patch, but that turned out to break some
other machine with a *differently* dodgy memory map, and got reverted
again. I have a feeling that things wound up at that point leaning
towards a consensus that the pseudo-generic machinery that was actually
only used for dumping BERT records could probably be ripped out and
replaced with a simple dedicated iomem-safe thing for dumping BERT
records, but I'm not sure if any further progress happened on that front.
Cheers,
Robin.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: ??? FAIL: Test report for kernel 5.13.0-rc7 (arm-next, 8ab9b1a9)
2022-03-04 19:39 ` Aristeu Rozanski
2022-03-04 20:00 ` Robin Murphy
@ 2022-03-07 10:19 ` Lorenzo Pieralisi
2022-03-07 19:01 ` Aristeu Rozanski
1 sibling, 1 reply; 39+ messages in thread
From: Lorenzo Pieralisi @ 2022-03-07 10:19 UTC (permalink / raw)
To: Aristeu Rozanski
Cc: Robin Murphy, Will Deacon, Catalin Marinas, linux-arm-kernel
On Fri, Mar 04, 2022 at 02:39:20PM -0500, Aristeu Rozanski wrote:
> Hi Robin,
>
> (Old thread, so reference to it: https://lists.infradead.org/pipermail/linux-arm-kernel/2021-June/668228.html)
> (Also, please keep me on Cc since I'm not subscribed to linux-arm)
> (And again because I failed to fix the email address after using the
> archives mailbox, apologies for that)
>
> On Tue, Jun 29, 2021 at 06:27:57PM +0100, Robin Murphy wrote:
> > Ah, from that I can only assume that this must be stress-ng's --sysfs
> > test reading things at random, so not only would it have to be on a
> > machine whose firmware presents the right thing in the right way but the
> > random test conditions would also have to line up to poke it the "right"
> > (wrong) way too.
> >
> > As a temporary workaround for the CI flakes, might it be possible to
> > configure stress-ng to stay away from just these ACPI "data" files?
>
> I started looking at this issue and managed to reproduce the issue
> instantly with
>
> dd if=/sys/firmware/acpi/tables/data/BERT of=/dev/null bs=7
>
> I've attempted a few ways of fixing it based on the comments on this
> thread but wasn't successful so far (my knowledge is pretty limited in
> this area too, so not a big surprise). How can I be of assistance to debug/test
> patches for this issue?
I put together the patch below that I need to send to see whether
that's acceptable, if you can help me test it in the meantime, that'd
great.
-- >8 --
Subject: [PATCH] ACPI: osl: Fix BERT error region memory mapping
Currently the sysfs interface maps the BERT error region as "memory"
(through acpi_os_map_memory()) in order to copy the error records into
memory buffers through memory operations (eg memory_read_from_buffer()).
The OS system cannot detect whether the BERT error region is part of
system RAM or it is "device memory" (eg BMC memory) and therefore it
cannot detect which memory attributes the bus to memory support (and
corresponding kernel mapping, unless firmware provides the required
information).
The acpi_os_map_memory() arch backend implementation determines the
mapping attributes. On arm64, if the BERT error region is not present in
the EFI memory map, the error region is mapped as device-nGnRnE; this
triggers alignment faults since memcpy unaligned accesses are not
allowed in device-nGnRnE regions.
The ACPI sysfs code cannot therefore map by default the BERT error
region with memory semantics but should use a safer default.
Change the sysfs code to map the BERT error region as MMIO (through
acpi_os_map_iomem()) and use the memcpy_fromio() interface to read the
error region into the kernel buffer.
Link: https://lore.kernel.org/linux-arm-kernel/31ffe8fc-f5ee-2858-26c5-0fd8bdd68702@arm.com
Link: https://lore.kernel.org/linux-acpi/CAJZ5v0g+OVbhuUUDrLUCfX_mVqY_e8ubgLTU98=jfjTeb4t+Pw@mail.gmail.com
Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Cc: Ard Biesheuvel <ardb@kernel.org>
Cc: Will Deacon <will@kernel.org>
Cc: Hanjun Guo <guohanjun@huawei.com>
Cc: Sudeep Holla <sudeep.holla@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: "Rafael J. Wysocki" <rjw@rjwysocki.net>
---
drivers/acpi/sysfs.c | 25 ++++++++++++++++++-------
1 file changed, 18 insertions(+), 7 deletions(-)
diff --git a/drivers/acpi/sysfs.c b/drivers/acpi/sysfs.c
index a4b638bea6f1..cc2fe0618178 100644
--- a/drivers/acpi/sysfs.c
+++ b/drivers/acpi/sysfs.c
@@ -415,19 +415,30 @@ static ssize_t acpi_data_show(struct file *filp, struct kobject *kobj,
loff_t offset, size_t count)
{
struct acpi_data_attr *data_attr;
- void *base;
- ssize_t rc;
+ void __iomem *base;
+ ssize_t size;
data_attr = container_of(bin_attr, struct acpi_data_attr, attr);
+ size = data_attr->attr.size;
+
+ if (offset < 0)
+ return -EINVAL;
+
+ if (offset >= size)
+ return 0;
- base = acpi_os_map_memory(data_attr->addr, data_attr->attr.size);
+ if (count > size - offset)
+ count = size - offset;
+
+ base = acpi_os_map_iomem(data_attr->addr, size);
if (!base)
return -ENOMEM;
- rc = memory_read_from_buffer(buf, count, &offset, base,
- data_attr->attr.size);
- acpi_os_unmap_memory(base, data_attr->attr.size);
- return rc;
+ memcpy_fromio(buf, base + offset, count);
+
+ acpi_os_unmap_iomem(base, size);
+
+ return count;
}
static int acpi_bert_data_init(void *th, struct acpi_data_attr *data_attr)
--
2.17.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 39+ messages in thread
* Re: ??? FAIL: Test report for kernel 5.13.0-rc7 (arm-next, 8ab9b1a9)
2022-03-07 10:19 ` Lorenzo Pieralisi
@ 2022-03-07 19:01 ` Aristeu Rozanski
0 siblings, 0 replies; 39+ messages in thread
From: Aristeu Rozanski @ 2022-03-07 19:01 UTC (permalink / raw)
To: Lorenzo Pieralisi
Cc: Robin Murphy, Will Deacon, Catalin Marinas, linux-arm-kernel
Hi Lorenzo,
On Mon, Mar 07, 2022 at 10:19:07AM +0000, Lorenzo Pieralisi wrote:
> I put together the patch below that I need to send to see whether
> that's acceptable, if you can help me test it in the meantime, that'd
> great.
>
> -- >8 --
> Subject: [PATCH] ACPI: osl: Fix BERT error region memory mapping
>
> Currently the sysfs interface maps the BERT error region as "memory"
> (through acpi_os_map_memory()) in order to copy the error records into
> memory buffers through memory operations (eg memory_read_from_buffer()).
>
> The OS system cannot detect whether the BERT error region is part of
> system RAM or it is "device memory" (eg BMC memory) and therefore it
> cannot detect which memory attributes the bus to memory support (and
> corresponding kernel mapping, unless firmware provides the required
> information).
>
> The acpi_os_map_memory() arch backend implementation determines the
> mapping attributes. On arm64, if the BERT error region is not present in
> the EFI memory map, the error region is mapped as device-nGnRnE; this
> triggers alignment faults since memcpy unaligned accesses are not
> allowed in device-nGnRnE regions.
>
> The ACPI sysfs code cannot therefore map by default the BERT error
> region with memory semantics but should use a safer default.
>
> Change the sysfs code to map the BERT error region as MMIO (through
> acpi_os_map_iomem()) and use the memcpy_fromio() interface to read the
> error region into the kernel buffer.
>
> Link: https://lore.kernel.org/linux-arm-kernel/31ffe8fc-f5ee-2858-26c5-0fd8bdd68702@arm.com
> Link: https://lore.kernel.org/linux-acpi/CAJZ5v0g+OVbhuUUDrLUCfX_mVqY_e8ubgLTU98=jfjTeb4t+Pw@mail.gmail.com
> Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
> Cc: Ard Biesheuvel <ardb@kernel.org>
> Cc: Will Deacon <will@kernel.org>
> Cc: Hanjun Guo <guohanjun@huawei.com>
> Cc: Sudeep Holla <sudeep.holla@arm.com>
> Cc: Catalin Marinas <catalin.marinas@arm.com>
> Cc: "Rafael J. Wysocki" <rjw@rjwysocki.net>
> ---
> drivers/acpi/sysfs.c | 25 ++++++++++++++++++-------
> 1 file changed, 18 insertions(+), 7 deletions(-)
>
> diff --git a/drivers/acpi/sysfs.c b/drivers/acpi/sysfs.c
> index a4b638bea6f1..cc2fe0618178 100644
> --- a/drivers/acpi/sysfs.c
> +++ b/drivers/acpi/sysfs.c
> @@ -415,19 +415,30 @@ static ssize_t acpi_data_show(struct file *filp, struct kobject *kobj,
> loff_t offset, size_t count)
> {
> struct acpi_data_attr *data_attr;
> - void *base;
> - ssize_t rc;
> + void __iomem *base;
> + ssize_t size;
>
> data_attr = container_of(bin_attr, struct acpi_data_attr, attr);
> + size = data_attr->attr.size;
> +
> + if (offset < 0)
> + return -EINVAL;
> +
> + if (offset >= size)
> + return 0;
>
> - base = acpi_os_map_memory(data_attr->addr, data_attr->attr.size);
> + if (count > size - offset)
> + count = size - offset;
> +
> + base = acpi_os_map_iomem(data_attr->addr, size);
> if (!base)
> return -ENOMEM;
> - rc = memory_read_from_buffer(buf, count, &offset, base,
> - data_attr->attr.size);
> - acpi_os_unmap_memory(base, data_attr->attr.size);
>
> - return rc;
> + memcpy_fromio(buf, base + offset, count);
> +
> + acpi_os_unmap_iomem(base, size);
> +
> + return count;
> }
Tested on the same machine I can trigger reliably and this fixes the
issue.
Thanks!
--
Aristeu
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 39+ messages in thread
end of thread, other threads:[~2022-03-07 19:03 UTC | newest]
Thread overview: 39+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-06-24 19:46 ❌ FAIL: Test report for kernel 5.13.0-rc7 (arm-next, 8ab9b1a9) CKI Project
2021-06-25 8:39 ` Will Deacon
[not found] ` <CA+tGwn=rP_hAMLLtoy_s90ZzBjfMggu7T2Qj8HyFfGh1BGUoRA@mail.gmail.com>
2021-06-25 11:02 ` Robin Murphy
2021-06-25 11:09 ` Catalin Marinas
2021-06-25 11:15 ` Robin Murphy
2021-06-29 11:48 ` Robin Murphy
2021-06-29 11:48 ` Robin Murphy
2021-06-29 14:44 ` Lorenzo Pieralisi
2021-06-29 14:44 ` Lorenzo Pieralisi
2021-06-29 15:14 ` Robin Murphy
2021-06-29 15:14 ` Robin Murphy
2021-06-29 16:35 ` Catalin Marinas
2021-06-29 16:35 ` Catalin Marinas
2021-06-30 10:37 ` Lorenzo Pieralisi
2021-06-30 10:37 ` Lorenzo Pieralisi
2021-06-30 11:17 ` Robin Murphy
2021-06-30 11:17 ` Robin Murphy
2021-06-30 13:22 ` Ard Biesheuvel
2021-06-30 15:49 ` Lorenzo Pieralisi
2021-06-30 15:49 ` Lorenzo Pieralisi
2021-06-30 18:18 ` Ard Biesheuvel
2021-07-05 16:17 ` Lorenzo Pieralisi
2021-07-05 16:17 ` Lorenzo Pieralisi
2021-07-16 16:16 ` Ard Biesheuvel
2021-07-16 16:26 ` Lorenzo Pieralisi
2021-07-16 16:26 ` Lorenzo Pieralisi
2021-07-22 12:38 ` Veronika Kabatova
2021-07-22 13:51 ` Robin Murphy
2021-07-22 13:51 ` Robin Murphy
2021-07-22 18:23 ` Veronika Kabatova
2021-06-29 17:03 ` Veronika Kabatova
2021-06-29 17:27 ` Robin Murphy
2021-06-29 17:27 ` Robin Murphy
2021-06-29 17:44 ` Veronika Kabatova
2022-03-04 19:31 ` ??? " Aristeu Rozanski
2022-03-04 19:39 ` Aristeu Rozanski
2022-03-04 20:00 ` Robin Murphy
2022-03-07 10:19 ` Lorenzo Pieralisi
2022-03-07 19:01 ` Aristeu Rozanski
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.