[PATCH V2] MIPS: Loongson: Add board_ebase_setup() support

[PATCH V2] MIPS: Loongson: Add board_ebase_setup() support

[Huacai Chen]

Old processors before Loongson-3A2000 have a 32bit ebase register and
have no WG bit, new processors since Loongson-3A2000 have a 64bit ebase
register and do have the WG bit. Unfortunately, Loongson processors
which have the WG bit are slightly different from MIPS R2. This makes
the generic ebase setup not suitable for every scenarios.

To make Loongson's kernel be more robust, we add a board_ebase_setup()
hook to ensure that CKSEG0 is always used for ebase. This is also useful
on platforms where BIOS doesn't initialise an appropriate ebase.

Signed-off-by: Huacai Chen <chenhc@lemote.com>
---
 arch/mips/kernel/cpu-probe.c       |  6 ++++--
 arch/mips/loongson64/common/init.c | 11 +++++++++++
 2 files changed, 15 insertions(+), 2 deletions(-)

diff --git a/arch/mips/kernel/cpu-probe.c b/arch/mips/kernel/cpu-probe.c
index 671bc6f..7312a0d 100644
--- a/arch/mips/kernel/cpu-probe.c
+++ b/arch/mips/kernel/cpu-probe.c
@@ -1923,7 +1923,8 @@ static inline void cpu_probe_loongson(struct cpuinfo_mips *c, unsigned int cpu)
 		}
 
 		decode_configs(c);
-		c->options |= MIPS_CPU_FTLB | MIPS_CPU_TLBINV | MIPS_CPU_LDPTE;
+		c->options |= MIPS_CPU_FTLB | MIPS_CPU_TLBINV |
+			      MIPS_CPU_LDPTE | MIPS_CPU_EBASE_WG;
 		c->writecombine = _CACHE_UNCACHED_ACCELERATED;
 		c->ases |= (MIPS_ASE_LOONGSON_MMI | MIPS_ASE_LOONGSON_CAM |
 			MIPS_ASE_LOONGSON_EXT | MIPS_ASE_LOONGSON_EXT2);
@@ -1934,7 +1935,8 @@ static inline void cpu_probe_loongson(struct cpuinfo_mips *c, unsigned int cpu)
 		set_elf_platform(cpu, "loongson3a");
 		set_isa(c, MIPS_CPU_ISA_M64R2);
 		decode_configs(c);
-		c->options |= MIPS_CPU_FTLB | MIPS_CPU_TLBINV | MIPS_CPU_LDPTE;
+		c->options |= MIPS_CPU_FTLB | MIPS_CPU_TLBINV |
+			      MIPS_CPU_LDPTE | MIPS_CPU_EBASE_WG;
 		c->writecombine = _CACHE_UNCACHED_ACCELERATED;
 		c->ases |= (MIPS_ASE_LOONGSON_MMI | MIPS_ASE_LOONGSON_CAM |
 			MIPS_ASE_LOONGSON_EXT | MIPS_ASE_LOONGSON_EXT2);
diff --git a/arch/mips/loongson64/init.c b/arch/mips/loongson64/init.c
index 912fe61..8e2047d 100644
--- a/arch/mips/loongson64/init.c
+++ b/arch/mips/loongson64/init.c
@@ -15,6 +15,16 @@
 
 #include <loongson.h>
 
+static void __init mips_ebase_setup(void)
+{
+	ebase = CKSEG0;
+
+	if (cpu_has_ebase_wg)
+		write_c0_ebase(ebase | MIPS_EBASE_WG);
+
+	write_c0_ebase(ebase);
+}
+
 static void __init mips_nmi_setup(void)
 {
 	void *base;
@@ -48,6 +58,7 @@ void __init prom_init(void)
 	setup_8250_early_printk_port(TO_UNCAC(LOONGSON_REG_BASE + 0x1e0), 0, 1024);
 
 	register_smp_ops(&loongson3_smp_ops);
+	board_ebase_setup = mips_ebase_setup;
 	board_nmi_handler_setup = mips_nmi_setup;
 }
 
-- 
2.7.0

Re: [PATCH V2] MIPS: Loongson: Add board_ebase_setup() support

[Jiaxun Yang]

在 2019/11/4 下午2:09, Huacai Chen 写道:
> Old processors before Loongson-3A2000 have a 32bit ebase register and
> have no WG bit, new processors since Loongson-3A2000 have a 64bit ebase
> register and do have the WG bit. Unfortunately, Loongson processors
> which have the WG bit are slightly different from MIPS R2. This makes
> the generic ebase setup not suitable for every scenarios.
> 
> To make Loongson's kernel be more robust, we add a board_ebase_setup()
> hook to ensure that CKSEG0 is always used for ebase. This is also useful
> on platforms where BIOS doesn't initialise an appropriate ebase.
> 
> Signed-off-by: Huacai Chen <chenhc@lemote.com>
> ---
Acked-by: Jiaxun Yang <jiaxun.yang@flygoat.com>

This patch is essential as most Loongson boards with Tianocore based
UEFI firmware didn't set their ebase correctly.

Should we backport it to stable?

--
Jiaxun Yang

Re: [PATCH V2] MIPS: Loongson: Add board_ebase_setup() support

[Huacai Chen]

Hi, Jiaxun,

On Thu, Nov 7, 2019 at 7:47 PM Jiaxun Yang <jiaxun.yang@flygoat.com> wrote:
>
>
> 在 2019/11/4 下午2:09, Huacai Chen 写道:
> > Old processors before Loongson-3A2000 have a 32bit ebase register and
> > have no WG bit, new processors since Loongson-3A2000 have a 64bit ebase
> > register and do have the WG bit. Unfortunately, Loongson processors
> > which have the WG bit are slightly different from MIPS R2. This makes
> > the generic ebase setup not suitable for every scenarios.
> >
> > To make Loongson's kernel be more robust, we add a board_ebase_setup()
> > hook to ensure that CKSEG0 is always used for ebase. This is also useful
> > on platforms where BIOS doesn't initialise an appropriate ebase.
> >
> > Signed-off-by: Huacai Chen <chenhc@lemote.com>
> > ---
> Acked-by: Jiaxun Yang <jiaxun.yang@flygoat.com>
>
> This patch is essential as most Loongson boards with Tianocore based
> UEFI firmware didn't set their ebase correctly.
>
> Should we backport it to stable?
Yes, this patch should be backported as early as 4.9-lts.

Huacai
>
> --
> Jiaxun Yang

Re: [PATCH V2] MIPS: Loongson: Add board_ebase_setup() support

[Paul Burton]

Hi Huacai,

On Mon, Nov 04, 2019 at 02:09:41PM +0800, Huacai Chen wrote:
> Old processors before Loongson-3A2000 have a 32bit ebase register and
> have no WG bit, new processors since Loongson-3A2000 have a 64bit ebase
> register and do have the WG bit. Unfortunately, Loongson processors
> which have the WG bit are slightly different from MIPS R2. This makes
> the generic ebase setup not suitable for every scenarios.

Can you describe how the Loongson WG bit differs from the WG bit as
architecturally defined? This patch may make things work for you but it
doesn't give us any record of what the hardware errata is or what we
actually need to do to work with the broken WG bit.

For example - right now Loongson kernels don't use the WG bit at all, so
what's the problem? It doesn't matter if WG has different behavior if we
don't use it.

So one option here might be to just continue to not indicate support for
the WG bit. It does look like the kernel ought to be ensuring the
exception vector it allocates is at a suitable address in this case, and
currently isn't. Something like the (untested) patch below ought to
address that. In practice though memblock is configured to allocate
bottom-up until after this point so it should be unlikely we'll get an
unsuitable address, and there's a WARN_ON() in trap_init() that would
already tell you if that happened.

> To make Loongson's kernel be more robust, we add a board_ebase_setup()
> hook to ensure that CKSEG0 is always used for ebase. This is also useful
> on platforms where BIOS doesn't initialise an appropriate ebase.

Can you also elaborate on that? I'm not sure why this would help on
systems that don't initialize EBase - the kernel unconditionally sets
EBase for >= MIPSr2 systems in configure_exception_vector() anyway, and
since v5.2 it doesn't even try to inherit whatever the bootloader used.

Thanks,
    Paul

---
diff --git a/arch/mips/kernel/traps.c b/arch/mips/kernel/traps.c
index 342e41de9d64..a25ee41eff48 100644
--- a/arch/mips/kernel/traps.c
+++ b/arch/mips/kernel/traps.c
@@ -2271,7 +2271,7 @@ void __init trap_init(void)
 	extern char except_vec4;
 	extern char except_vec3_r4000;
 	unsigned long i, vec_size;
-	phys_addr_t ebase_pa;
+	phys_addr_t ebase_pa, ebase_limit;
 
 	check_wait();
 
@@ -2287,7 +2287,21 @@ void __init trap_init(void)
 		else
 			vec_size = PAGE_SIZE;
 
-		ebase_pa = memblock_phys_alloc(vec_size, 1 << fls(vec_size));
+		/*
+		 * If we have support for the EBase.WG bit then allow the
+		 * exception vector to be located anywhere. When EBase.WG is
+		 * not supported EBase is limited to a (c)kseg[01] address, so
+		 * we must ensure the allocated vector is in memory accessible
+		 * via those unmapped regions.
+		 */
+		if (cpu_has_ebase_wg)
+			ebase_limit = MEMBLOCK_ALLOC_ACCESSIBLE;
+		else
+			ebase_limit = CKSEG1 - CKSEG0;
+
+		ebase_pa = memblock_phys_alloc_range(vec_size,
+						     1 << fls(vec_size),
+						     0, ebase_limit);
 		if (!ebase_pa)
 			panic("%s: Failed to allocate %lu bytes align=0x%x\n",
 			      __func__, vec_size, 1 << fls(vec_size));

Re: [PATCH V2] MIPS: Loongson: Add board_ebase_setup() support

[Huacai Chen]

Hi, Paul

On Sat, Nov 9, 2019 at 3:11 AM Paul Burton <paulburton@kernel.org> wrote:
>
> Hi Huacai,
>
> On Mon, Nov 04, 2019 at 02:09:41PM +0800, Huacai Chen wrote:
> > Old processors before Loongson-3A2000 have a 32bit ebase register and
> > have no WG bit, new processors since Loongson-3A2000 have a 64bit ebase
> > register and do have the WG bit. Unfortunately, Loongson processors
> > which have the WG bit are slightly different from MIPS R2. This makes
> > the generic ebase setup not suitable for every scenarios.
>
> Can you describe how the Loongson WG bit differs from the WG bit as
> architecturally defined? This patch may make things work for you but it
> doesn't give us any record of what the hardware errata is or what we
> actually need to do to work with the broken WG bit.
>
> For example - right now Loongson kernels don't use the WG bit at all, so
> what's the problem? It doesn't matter if WG has different behavior if we
> don't use it.
>
> So one option here might be to just continue to not indicate support for
> the WG bit. It does look like the kernel ought to be ensuring the
> exception vector it allocates is at a suitable address in this case, and
> currently isn't. Something like the (untested) patch below ought to
> address that. In practice though memblock is configured to allocate
> bottom-up until after this point so it should be unlikely we'll get an
> unsuitable address, and there's a WARN_ON() in trap_init() that would
> already tell you if that happened.
>
> > To make Loongson's kernel be more robust, we add a board_ebase_setup()
> > hook to ensure that CKSEG0 is always used for ebase. This is also useful
> > on platforms where BIOS doesn't initialise an appropriate ebase.
>
> Can you also elaborate on that? I'm not sure why this would help on
> systems that don't initialize EBase - the kernel unconditionally sets
> EBase for >= MIPSr2 systems in configure_exception_vector() anyway, and
> since v5.2 it doesn't even try to inherit whatever the bootloader used.
>
I'm sorry that I haven't seen the changes in 5.2. My original problem
has been fixed with the below two commits.
172dcd935c34b022729f45a7bbaae5cc052 ("MIPS: Always allocate exception
vector for MIPSr2+")
de56d4c1da3e68f0ca468a55f6677bef3cee ("MIPS: Remove duplicate EBase
configuration")

My own patch is only needed in 4.9/4.14/4.19, and I can backport the
above two commits while completely drop my own patch.

Thanks,
Huacai

> Thanks,
>     Paul
>
> ---
> diff --git a/arch/mips/kernel/traps.c b/arch/mips/kernel/traps.c
> index 342e41de9d64..a25ee41eff48 100644
> --- a/arch/mips/kernel/traps.c
> +++ b/arch/mips/kernel/traps.c
> @@ -2271,7 +2271,7 @@ void __init trap_init(void)
>         extern char except_vec4;
>         extern char except_vec3_r4000;
>         unsigned long i, vec_size;
> -       phys_addr_t ebase_pa;
> +       phys_addr_t ebase_pa, ebase_limit;
>
>         check_wait();
>
> @@ -2287,7 +2287,21 @@ void __init trap_init(void)
>                 else
>                         vec_size = PAGE_SIZE;
>
> -               ebase_pa = memblock_phys_alloc(vec_size, 1 << fls(vec_size));
> +               /*
> +                * If we have support for the EBase.WG bit then allow the
> +                * exception vector to be located anywhere. When EBase.WG is
> +                * not supported EBase is limited to a (c)kseg[01] address, so
> +                * we must ensure the allocated vector is in memory accessible
> +                * via those unmapped regions.
> +                */
> +               if (cpu_has_ebase_wg)
> +                       ebase_limit = MEMBLOCK_ALLOC_ACCESSIBLE;
> +               else
> +                       ebase_limit = CKSEG1 - CKSEG0;
> +
> +               ebase_pa = memblock_phys_alloc_range(vec_size,
> +                                                    1 << fls(vec_size),
> +                                                    0, ebase_limit);
>                 if (!ebase_pa)
>                         panic("%s: Failed to allocate %lu bytes align=0x%x\n",
>                               __func__, vec_size, 1 << fls(vec_size));

Re: [PATCH V2] MIPS: Loongson: Add board_ebase_setup() support

[Huacai Chen]

Hi, Paul,

On Sat, Nov 9, 2019 at 7:02 PM Huacai Chen <chenhc@lemote.com> wrote:
>
> Hi, Paul
>
> On Sat, Nov 9, 2019 at 3:11 AM Paul Burton <paulburton@kernel.org> wrote:
> >
> > Hi Huacai,
> >
> > On Mon, Nov 04, 2019 at 02:09:41PM +0800, Huacai Chen wrote:
> > > Old processors before Loongson-3A2000 have a 32bit ebase register and
> > > have no WG bit, new processors since Loongson-3A2000 have a 64bit ebase
> > > register and do have the WG bit. Unfortunately, Loongson processors
> > > which have the WG bit are slightly different from MIPS R2. This makes
> > > the generic ebase setup not suitable for every scenarios.
> >
> > Can you describe how the Loongson WG bit differs from the WG bit as
> > architecturally defined? This patch may make things work for you but it
> > doesn't give us any record of what the hardware errata is or what we
> > actually need to do to work with the broken WG bit.
> >
> > For example - right now Loongson kernels don't use the WG bit at all, so
> > what's the problem? It doesn't matter if WG has different behavior if we
> > don't use it.
> >
> > So one option here might be to just continue to not indicate support for
> > the WG bit. It does look like the kernel ought to be ensuring the
> > exception vector it allocates is at a suitable address in this case, and
> > currently isn't. Something like the (untested) patch below ought to
> > address that. In practice though memblock is configured to allocate
> > bottom-up until after this point so it should be unlikely we'll get an
> > unsuitable address, and there's a WARN_ON() in trap_init() that would
> > already tell you if that happened.
> >
> > > To make Loongson's kernel be more robust, we add a board_ebase_setup()
> > > hook to ensure that CKSEG0 is always used for ebase. This is also useful
> > > on platforms where BIOS doesn't initialise an appropriate ebase.
> >
> > Can you also elaborate on that? I'm not sure why this would help on
> > systems that don't initialize EBase - the kernel unconditionally sets
> > EBase for >= MIPSr2 systems in configure_exception_vector() anyway, and
> > since v5.2 it doesn't even try to inherit whatever the bootloader used.
> >
> I'm sorry that I haven't seen the changes in 5.2. My original problem
> has been fixed with the below two commits.
> 172dcd935c34b022729f45a7bbaae5cc052 ("MIPS: Always allocate exception
> vector for MIPSr2+")
> de56d4c1da3e68f0ca468a55f6677bef3cee ("MIPS: Remove duplicate EBase
> configuration")
>
> My own patch is only needed in 4.9/4.14/4.19, and I can backport the
> above two commits while completely drop my own patch.
Maybe this patch is still neccesary. The above two commits surely make
it possible to boot with any firmware, but dynamically allocated ebase
cannot work perfectly with STR(suspend) and STD(hibernation).

STR is fixable, because we can save ebase at suspend and restore it at
resume. However, STD is very diffcult to fix, because we cannot assure
the ebase allocated by the new kernel is the same as the old kernel,
and I think change ebase during kernel switching is dangerous.

So could you please consider to accept this patch? Thanks.
>
> Thanks,
> Huacai
>
> > Thanks,
> >     Paul
> >
> > ---
> > diff --git a/arch/mips/kernel/traps.c b/arch/mips/kernel/traps.c
> > index 342e41de9d64..a25ee41eff48 100644
> > --- a/arch/mips/kernel/traps.c
> > +++ b/arch/mips/kernel/traps.c
> > @@ -2271,7 +2271,7 @@ void __init trap_init(void)
> >         extern char except_vec4;
> >         extern char except_vec3_r4000;
> >         unsigned long i, vec_size;
> > -       phys_addr_t ebase_pa;
> > +       phys_addr_t ebase_pa, ebase_limit;
> >
> >         check_wait();
> >
> > @@ -2287,7 +2287,21 @@ void __init trap_init(void)
> >                 else
> >                         vec_size = PAGE_SIZE;
> >
> > -               ebase_pa = memblock_phys_alloc(vec_size, 1 << fls(vec_size));
> > +               /*
> > +                * If we have support for the EBase.WG bit then allow the
> > +                * exception vector to be located anywhere. When EBase.WG is
> > +                * not supported EBase is limited to a (c)kseg[01] address, so
> > +                * we must ensure the allocated vector is in memory accessible
> > +                * via those unmapped regions.
> > +                */
> > +               if (cpu_has_ebase_wg)
> > +                       ebase_limit = MEMBLOCK_ALLOC_ACCESSIBLE;
> > +               else
> > +                       ebase_limit = CKSEG1 - CKSEG0;
> > +
> > +               ebase_pa = memblock_phys_alloc_range(vec_size,
> > +                                                    1 << fls(vec_size),
> > +                                                    0, ebase_limit);
> >                 if (!ebase_pa)
> >                         panic("%s: Failed to allocate %lu bytes align=0x%x\n",
> >                               __func__, vec_size, 1 << fls(vec_size));