[PATCH] mips: remove reference to "newer Loongson-3"

[PATCH] mips: remove reference to "newer Loongson-3"

[Xi Ruoyao]

Newest Loongson-3 processors have moved to use LoongArch architecture.
Sadly, the LL/SC issue is still existing on both latest Loongson-3
processors using MIPS64 (Loongson-3A4000) and LoongArch
(Loongson-3A5000).

As it's very unlikely there will be new Loongson-3 processors using
MIPS64, let's stop people from false hoping.

Signed-off-by: Xi Ruoyao <xry111@mengyan1223.wang>
Cc: Huacai Chen <chenhuacai@kernel.org>
---

Huacai: how's the status of LL/SC issue on Loongson-2K?  If
the issue exists on it as well, we can just force
CPU_LOONGSON3_WORKAROUNDS when CONFIG_CPU_LOONGSON64 and
CONFIG_SMP are both selected.

 arch/mips/Kconfig | 9 ++-------
 1 file changed, 2 insertions(+), 7 deletions(-)

diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig
index 6dfb27d531dd..ff5f344a371e 100644
--- a/arch/mips/Kconfig
+++ b/arch/mips/Kconfig
@@ -1433,19 +1433,14 @@ config LOONGSON3_ENHANCEMENT
 	  new Loongson-3 machines only, please say 'Y' here.
 
 config CPU_LOONGSON3_WORKAROUNDS
-	bool "Old Loongson-3 LLSC Workarounds"
+	bool "Loongson-3 LLSC Workarounds"
 	default y if SMP
 	depends on CPU_LOONGSON64
 	help
 	  Loongson-3 processors have the llsc issues which require workarounds.
 	  Without workarounds the system may hang unexpectedly.
 
-	  Newer Loongson-3 will fix these issues and no workarounds are needed.
-	  The workarounds have no significant side effect on them but may
-	  decrease the performance of the system so this option should be
-	  disabled unless the kernel is intended to be run on old systems.
-
-	  If unsure, please say Y.
+	  Say Y, unless you know what you are doing.
 
 config CPU_LOONGSON3_CPUCFG_EMULATION
 	bool "Emulate the CPUCFG instruction on older Loongson cores"
-- 
2.33.0

Re: [PATCH] mips: remove reference to "newer Loongson-3"

[Jiaxun Yang]

在 2021/8/29 20:49, Xi Ruoyao 写道:
> Newest Loongson-3 processors have moved to use LoongArch architecture.
> Sadly, the LL/SC issue is still existing on both latest Loongson-3
> processors using MIPS64 (Loongson-3A4000) and LoongArch
> (Loongson-3A5000).
LLSC is fixed on Loongson-3A4000 as per CPUCFG report.
>
> As it's very unlikely there will be new Loongson-3 processors using
> MIPS64, let's stop people from false hoping.
>
> Signed-off-by: Xi Ruoyao <xry111@mengyan1223.wang>
> Cc: Huacai Chen <chenhuacai@kernel.org>
> ---
>
> Huacai: how's the status of LL/SC issue on Loongson-2K?  If
> the issue exists on it as well, we can just force
> CPU_LOONGSON3_WORKAROUNDS when CONFIG_CPU_LOONGSON64 and
> CONFIG_SMP are both selected.

Loongson-2K do need LLSC workaround, although the reason behind the 
workaround seems different...

Thanks.

- Jiaxun

>
>   arch/mips/Kconfig | 9 ++-------
>   1 file changed, 2 insertions(+), 7 deletions(-)
>
> diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig
> index 6dfb27d531dd..ff5f344a371e 100644
> --- a/arch/mips/Kconfig
> +++ b/arch/mips/Kconfig
> @@ -1433,19 +1433,14 @@ config LOONGSON3_ENHANCEMENT
>   	  new Loongson-3 machines only, please say 'Y' here.
>   
>   config CPU_LOONGSON3_WORKAROUNDS
> -	bool "Old Loongson-3 LLSC Workarounds"
> +	bool "Loongson-3 LLSC Workarounds"
>   	default y if SMP
>   	depends on CPU_LOONGSON64
>   	help
>   	  Loongson-3 processors have the llsc issues which require workarounds.
>   	  Without workarounds the system may hang unexpectedly.
>   
> -	  Newer Loongson-3 will fix these issues and no workarounds are needed.
> -	  The workarounds have no significant side effect on them but may
> -	  decrease the performance of the system so this option should be
> -	  disabled unless the kernel is intended to be run on old systems.
> -
> -	  If unsure, please say Y.
> +	  Say Y, unless you know what you are doing.
>   
>   config CPU_LOONGSON3_CPUCFG_EMULATION
>   	bool "Emulate the CPUCFG instruction on older Loongson cores"

Re: [PATCH] mips: remove reference to "newer Loongson-3"

[Xi Ruoyao]

On Mon, 2021-08-30 at 10:32 +0800, Jiaxun Yang wrote:
> 
> 在 2021/8/29 20:49, Xi Ruoyao 写道:
> > Newest Loongson-3 processors have moved to use LoongArch
> > architecture.
> > Sadly, the LL/SC issue is still existing on both latest Loongson-3
> > processors using MIPS64 (Loongson-3A4000) and LoongArch
> > (Loongson-3A5000).
> LLSC is fixed on Loongson-3A4000 as per CPUCFG report.

If I don't enable LL/SC fix, GCC libgomp tests fail on both 3A4000 and
3A5000 (using github.com/loongson/gcc for the latter) with "invalid
access to 0x00000049" or "0x00000005".  This is a race condition: it
does not happen at all with OMP_NUM_THREADS=1, happens with about 10%
possibility with OMP_NUM_THREADS=2, and about 90% possibility with
OMP_NUM_THREAD=4 (on 3A5000, on 3A4000 the possibility is lower).

My investigation suggests this means a GCC instrinic,
__atomic_compare_and_exchange_n is not really atomic as it should be.

If this is not a hardware issue in the GS464V/LA464 uarch, then it will
be very low-possibility coincidence: two unrelated code-generation bugs
for __atomic_compare_and_exchange_n (LA port has borrowed some code from
MIPS port, but the instrinics are of course newly coded).  And I've
inspected libgomp & gcc code about __atomic_compare_and_exchange_n
carefully, nothing wrong spotted except LoongArch GCC supports "-mfix-
loongson3-llsc" which adds a "dbar 0" (like "sync" on MIPS) instruction
after SC (only for instrinics).  Enabling this fixes the libgomp
failures. Likewisely, "-Wa,-mfix-loongson3-llsc" fixes it on 3A4000.

libgomp code has been verified with thread sanitizer on other
architectures (unfortunately libtsan is not available on MIPS or
LoongArch yet), so it's very unlikely to be a coding error leading to
the race.

And LL/SC fix is still in Huacai's 3A5000 kernel.  In a mail on linux-
arch Huacai said it's "not so easy to be fixed".

Or these are two different erratas and I misunderstand them as the same
one?

Re: [PATCH] mips: remove reference to "newer Loongson-3"

[Jiaxun Yang]

在 2021/8/30 下午8:28, Xi Ruoyao 写道:
> On Mon, 2021-08-30 at 10:32 +0800, Jiaxun Yang wrote:
>> 在 2021/8/29 20:49, Xi Ruoyao 写道:
>>> Newest Loongson-3 processors have moved to use LoongArch
>>> architecture.
>>> Sadly, the LL/SC issue is still existing on both latest Loongson-3
>>> processors using MIPS64 (Loongson-3A4000) and LoongArch
>>> (Loongson-3A5000).
>> LLSC is fixed on Loongson-3A4000 as per CPUCFG report.
> If I don't enable LL/SC fix, GCC libgomp tests fail on both 3A4000 and
> 3A5000 (using github.com/loongson/gcc for the latter) with "invalid
> access to 0x00000049" or "0x00000005".  This is a race condition: it
> does not happen at all with OMP_NUM_THREADS=1, happens with about 10%
> possibility with OMP_NUM_THREADS=2, and about 90% possibility with
> OMP_NUM_THREAD=4 (on 3A5000, on 3A4000 the possibility is lower).
Apologize for the false report, yes, you are right. I had checked with 
Loongson guys
and they confirmed that the workaround still needs to be applied to 
latest 3A4000
processors, including 3A4000 for MIPS and 3A5000 for LoongArch.

Though, the reason behind the workaround varies with the evaluation of 
their uArch,
for GS464V based core, barrier is required as the uArch design allows 
regular load
to be reordered after an atomic linked load, and that would break 
assumption of compiler
atomic constraints.

For GS464E, barrier is required to flush the Store Fill Buffer and land 
all the cachelines
to L1 cache, otherwise a linked load to a cacheline located at SFB may 
cause deadlock.

For original GS464, barrier is required to deal with some kind of 
pipeline hazard to
ensure store condition won't be shorted to success.

Patch LGTM. The config symbol looks ambiguous and I'd agree with your 
idea of renaming.

Thanks,

- Jiaxun

>
> My investigation suggests this means a GCC instrinic,
> __atomic_compare_and_exchange_n is not really atomic as it should be.
>
> If this is not a hardware issue in the GS464V/LA464 uarch, then it will
> be very low-possibility coincidence: two unrelated code-generation bugs
> for __atomic_compare_and_exchange_n (LA port has borrowed some code from
> MIPS port, but the instrinics are of course newly coded).  And I've
> inspected libgomp & gcc code about __atomic_compare_and_exchange_n
> carefully, nothing wrong spotted except LoongArch GCC supports "-mfix-
> loongson3-llsc" which adds a "dbar 0" (like "sync" on MIPS) instruction
> after SC (only for instrinics).  Enabling this fixes the libgomp
> failures. Likewisely, "-Wa,-mfix-loongson3-llsc" fixes it on 3A4000.
>
> libgomp code has been verified with thread sanitizer on other
> architectures (unfortunately libtsan is not available on MIPS or
> LoongArch yet), so it's very unlikely to be a coding error leading to
> the race.
>
> And LL/SC fix is still in Huacai's 3A5000 kernel.  In a mail on linux-
> arch Huacai said it's "not so easy to be fixed".
>
> Or these are two different erratas and I misunderstand them as the same
> one?
>

Re: [PATCH] mips: remove reference to "newer Loongson-3"

[Xi Ruoyao]

On Tue, 2021-08-31 at 00:14 +0800, Jiaxun Yang wrote:
> 
> 
> 在 2021/8/30 下午8:28, Xi Ruoyao 写道:
> > On Mon, 2021-08-30 at 10:32 +0800, Jiaxun Yang wrote:
> > > 在 2021/8/29 20:49, Xi Ruoyao 写道:
> > > > Newest Loongson-3 processors have moved to use LoongArch
> > > > architecture.
> > > > Sadly, the LL/SC issue is still existing on both latest
> > > > Loongson-3
> > > > processors using MIPS64 (Loongson-3A4000) and LoongArch
> > > > (Loongson-3A5000).
> > > LLSC is fixed on Loongson-3A4000 as per CPUCFG report.
> > If I don't enable LL/SC fix, GCC libgomp tests fail on both 3A4000
> > and
> > 3A5000 (using github.com/loongson/gcc for the latter) with "invalid
> > access to 0x00000049" or "0x00000005".  This is a race condition: it
> > does not happen at all with OMP_NUM_THREADS=1, happens with about
> > 10%
> > possibility with OMP_NUM_THREADS=2, and about 90% possibility with
> > OMP_NUM_THREAD=4 (on 3A5000, on 3A4000 the possibility is lower).

> Apologize for the false report, yes, you are right. I had checked with
> Loongson guys
> and they confirmed that the workaround still needs to be applied to 
> latest 3A4000
> processors, including 3A4000 for MIPS and 3A5000 for LoongArch.
> 
> Though, the reason behind the workaround varies with the evaluation of
> their uArch,
> for GS464V based core, barrier is required as the uArch design allows 
> regular load
> to be reordered after an atomic linked load, and that would break 
> assumption of compiler
> atomic constraints.

> For GS464E, barrier is required to flush the Store Fill Buffer and
> land 
> all the cachelines
> to L1 cache, otherwise a linked load to a cacheline located at SFB may
> cause deadlock.
> 
> For original GS464, barrier is required to deal with some kind of 
> pipeline hazard to
> ensure store condition won't be shorted to success.

This explains the different (mis)behavior of LL/SC on those uarchs.  I
remember the original report of LL/SC issue said it can cause a deadlock
on earlier model of 3As, but I didn't observed any deadlock on 3A4000.

(That's I why didn't tried the workaround immediately after spotting
libgomp failure, but debugged the code from 00:00 AM to 04:00 :( )

Thanks for your detailed explanation!

> Patch LGTM. The config symbol looks ambiguous and I'd agree with your 
> idea of renaming.
> 
> Thanks,
> 
> - Jiaxun

> > Or these are two different erratas and I misunderstand them as the
> > same one?

So basically this is true :).  They just happen to share one workaround.

-- 
Xi Ruoyao <xry111@mengyan1223.wang>
School of Aerospace Science and Technology, Xidian University

Re: [PATCH] mips: remove reference to "newer Loongson-3"

[Thomas Bogendoerfer]

On Sun, Aug 29, 2021 at 08:49:09PM +0800, Xi Ruoyao wrote:
> Newest Loongson-3 processors have moved to use LoongArch architecture.
> Sadly, the LL/SC issue is still existing on both latest Loongson-3
> processors using MIPS64 (Loongson-3A4000) and LoongArch
> (Loongson-3A5000).
> 
> As it's very unlikely there will be new Loongson-3 processors using
> MIPS64, let's stop people from false hoping.
> 
> Signed-off-by: Xi Ruoyao <xry111@mengyan1223.wang>
> Cc: Huacai Chen <chenhuacai@kernel.org>
> ---
> 
> Huacai: how's the status of LL/SC issue on Loongson-2K?  If
> the issue exists on it as well, we can just force
> CPU_LOONGSON3_WORKAROUNDS when CONFIG_CPU_LOONGSON64 and
> CONFIG_SMP are both selected.
> 
>  arch/mips/Kconfig | 9 ++-------
>  1 file changed, 2 insertions(+), 7 deletions(-)

applied to mips-next.

Thomas.

-- 
Crap can work. Given enough thrust pigs will fly, but it's not necessarily a
good idea.                                                [ RFC1925, 2.3 ]