Hard lockup on Raspberry Pi 4 after v5.10-rc1

Hard lockup on Raspberry Pi 4 after v5.10-rc1

[Nathan Chancellor]

Hi everyone,

Starting with v5.10-rc1, I appear to be experiencing hard lockups on my
Raspberry Pi 4 when compiling LLVM. My standard workflow for testing the
Raspberry Pi 4 has been compiling tip of tree LLVM (to catch host
AArch64 issues), compiling linux-next ARCH=arm64 defconfig, then booting
it with QEMU + KVM, which has caught a few issues so far. Starting with
v5.10-rc1, compiling LLVM causes the serial console to print junk like

 |▒pp▒p▒p▒p
 p|▒|▒|p▒p▒p|▒▒|▒|▒|p|▒▒|▒|▒|
 p▒pp▒▒▒|▒▒
 ▒
 ▒|▒|▒▒▒▒
 p▒
 ▒
 ▒▒
 ▒
 ▒p|p

then I lose my mosh connection and I see the lights on the front of the
Pi stop blinking.

I have managed to narrow it down to a commit somewhere in the armsoc DT
merge.

Good: 2e368dd2bbea ("Merge tag 'armsoc-drivers' of
git://git.kernel.org/pub/scm/linux/kernel/git/soc/soc")
Bad: e533cda12d8f ("Merge tag 'armsoc-dt' of
git://git.kernel.org/pub/scm/linux/kernel/git/soc/soc")

I have not been able to complete a full bisect due to a lack of time
unfortunately, I am sending this email to hopefully get some more eyes
on the issue.

$ head -n6 /boot/config.txt
# My settings
enable_uart=1
core_freq_min=500
kernel=Image
os_prefix=custom-mainline-gcc-arm64/
upstream_kernel=1

$ cat /boot/.firmware_revision
e8ddac7cd34b179efd702d4dc1efd0ccf44559b9

I can reproduce the issue with:

$ git clone --depth=1 https://github.com/llvm/llvm-project

$ mkdir -p llvm-project/build && cd llvm-project/build

$ cmake \
-G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DLLVM_ENABLE_PROJECTS="clang;lld" \
../llvm &&
ninja

If you need any more information or testing, please let me know!

Cheers,
Nathan

Hard lockup on Raspberry Pi 4 after v5.10-rc1

[Nathan Chancellor]

Hi everyone,

Starting with v5.10-rc1, I appear to be experiencing hard lockups on my
Raspberry Pi 4 when compiling LLVM. My standard workflow for testing the
Raspberry Pi 4 has been compiling tip of tree LLVM (to catch host
AArch64 issues), compiling linux-next ARCH=arm64 defconfig, then booting
it with QEMU + KVM, which has caught a few issues so far. Starting with
v5.10-rc1, compiling LLVM causes the serial console to print junk like

 |▒pp▒p▒p▒p
 p|▒|▒|p▒p▒p|▒▒|▒|▒|p|▒▒|▒|▒|
 p▒pp▒▒▒|▒▒
 ▒
 ▒|▒|▒▒▒▒
 p▒
 ▒
 ▒▒
 ▒
 ▒p|p

then I lose my mosh connection and I see the lights on the front of the
Pi stop blinking.

I have managed to narrow it down to a commit somewhere in the armsoc DT
merge.

Good: 2e368dd2bbea ("Merge tag 'armsoc-drivers' of
git://git.kernel.org/pub/scm/linux/kernel/git/soc/soc")
Bad: e533cda12d8f ("Merge tag 'armsoc-dt' of
git://git.kernel.org/pub/scm/linux/kernel/git/soc/soc")

I have not been able to complete a full bisect due to a lack of time
unfortunately, I am sending this email to hopefully get some more eyes
on the issue.

$ head -n6 /boot/config.txt
# My settings
enable_uart=1
core_freq_min=500
kernel=Image
os_prefix=custom-mainline-gcc-arm64/
upstream_kernel=1

$ cat /boot/.firmware_revision
e8ddac7cd34b179efd702d4dc1efd0ccf44559b9

I can reproduce the issue with:

$ git clone --depth=1 https://github.com/llvm/llvm-project

$ mkdir -p llvm-project/build && cd llvm-project/build

$ cmake \
-G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DLLVM_ENABLE_PROJECTS="clang;lld" \
../llvm &&
ninja

If you need any more information or testing, please let me know!

Cheers,
Nathan

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: Hard lockup on Raspberry Pi 4 after v5.10-rc1

[Nicolas Saenz Julienne]

[-- Attachment #1: Type: text/plain, Size: 1993 bytes --]

On Mon, 2020-11-02 at 00:29 -0700, Nathan Chancellor wrote:
> Hi everyone,
> 
> Starting with v5.10-rc1, I appear to be experiencing hard lockups on my
> Raspberry Pi 4 when compiling LLVM. My standard workflow for testing the
> Raspberry Pi 4 has been compiling tip of tree LLVM (to catch host
> AArch64 issues), compiling linux-next ARCH=arm64 defconfig, then booting
> it with QEMU + KVM, which has caught a few issues so far. Starting with
> v5.10-rc1, compiling LLVM causes the serial console to print junk like
> 
>  |▒pp▒p▒p▒p
>  p|▒|▒|p▒p▒p|▒▒|▒|▒|p|▒▒|▒|▒|
>  p▒pp▒▒▒|▒▒
>  ▒
>  ▒|▒|▒▒▒▒
>  p▒
>  ▒
>  ▒▒
>  ▒
>  ▒p|p
> 
> then I lose my mosh connection and I see the lights on the front of the
> Pi stop blinking.
> 
> I have managed to narrow it down to a commit somewhere in the armsoc DT
> merge.
> 
> Good: 2e368dd2bbea ("Merge tag 'armsoc-drivers' of
> git://git.kernel.org/pub/scm/linux/kernel/git/soc/soc")
> Bad: e533cda12d8f ("Merge tag 'armsoc-dt' of
> git://git.kernel.org/pub/scm/linux/kernel/git/soc/soc")
> 
> I have not been able to complete a full bisect due to a lack of time
> unfortunately, I am sending this email to hopefully get some more eyes
> on the issue.
> 
> $ head -n6 /boot/config.txt
> # My settings
> enable_uart=1
> core_freq_min=500
> kernel=Image
> os_prefix=custom-mainline-gcc-arm64/
> upstream_kernel=1
> 
> $ cat /boot/.firmware_revision
> e8ddac7cd34b179efd702d4dc1efd0ccf44559b9
> 
> I can reproduce the issue with:
> 
> $ git clone --depth=1 https://github.com/llvm/llvm-project
> 
> $ mkdir -p llvm-project/build && cd llvm-project/build
> 
> $ cmake \
> -G Ninja \
> -DCMAKE_BUILD_TYPE=Release \
> -DLLVM_ENABLE_PROJECTS="clang;lld" \
> ../llvm &&
> ninja
> 
> If you need any more information or testing, please let me know!

Thanks for the report! I'll have a look at it tomorrow.

Regards,
Nicolas


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Hard lockup on Raspberry Pi 4 after v5.10-rc1

[Nicolas Saenz Julienne]

[-- Attachment #1.1: Type: text/plain, Size: 1993 bytes --]

On Mon, 2020-11-02 at 00:29 -0700, Nathan Chancellor wrote:
> Hi everyone,
> 
> Starting with v5.10-rc1, I appear to be experiencing hard lockups on my
> Raspberry Pi 4 when compiling LLVM. My standard workflow for testing the
> Raspberry Pi 4 has been compiling tip of tree LLVM (to catch host
> AArch64 issues), compiling linux-next ARCH=arm64 defconfig, then booting
> it with QEMU + KVM, which has caught a few issues so far. Starting with
> v5.10-rc1, compiling LLVM causes the serial console to print junk like
> 
>  |▒pp▒p▒p▒p
>  p|▒|▒|p▒p▒p|▒▒|▒|▒|p|▒▒|▒|▒|
>  p▒pp▒▒▒|▒▒
>  ▒
>  ▒|▒|▒▒▒▒
>  p▒
>  ▒
>  ▒▒
>  ▒
>  ▒p|p
> 
> then I lose my mosh connection and I see the lights on the front of the
> Pi stop blinking.
> 
> I have managed to narrow it down to a commit somewhere in the armsoc DT
> merge.
> 
> Good: 2e368dd2bbea ("Merge tag 'armsoc-drivers' of
> git://git.kernel.org/pub/scm/linux/kernel/git/soc/soc")
> Bad: e533cda12d8f ("Merge tag 'armsoc-dt' of
> git://git.kernel.org/pub/scm/linux/kernel/git/soc/soc")
> 
> I have not been able to complete a full bisect due to a lack of time
> unfortunately, I am sending this email to hopefully get some more eyes
> on the issue.
> 
> $ head -n6 /boot/config.txt
> # My settings
> enable_uart=1
> core_freq_min=500
> kernel=Image
> os_prefix=custom-mainline-gcc-arm64/
> upstream_kernel=1
> 
> $ cat /boot/.firmware_revision
> e8ddac7cd34b179efd702d4dc1efd0ccf44559b9
> 
> I can reproduce the issue with:
> 
> $ git clone --depth=1 https://github.com/llvm/llvm-project
> 
> $ mkdir -p llvm-project/build && cd llvm-project/build
> 
> $ cmake \
> -G Ninja \
> -DCMAKE_BUILD_TYPE=Release \
> -DLLVM_ENABLE_PROJECTS="clang;lld" \
> ../llvm &&
> ninja
> 
> If you need any more information or testing, please let me know!

Thanks for the report! I'll have a look at it tomorrow.

Regards,
Nicolas


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 176 bytes --]

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel