[PATCH] MIPS: Make sure ebase address is in KSEG0

[PATCH] MIPS: Make sure ebase address is in KSEG0

[Huacai Chen]

Dynamically allocated ebase address above 0x20000000 can be triggered
by some special physical memory layout, or just by a "big kernel + big
initrd + big swiotlb" configuration.

For MIPS32, ebase address above 0x20000000 is unusable, for MIPS64 it
is usable but this case is warned. However, this warning is useless
because it is unfixable in a specific system configuration. So we just
use CKSEG0 as a fallback.

Signed-off-by: Huacai Chen <chenhc@lemote.com>
---
 arch/mips/kernel/traps.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/arch/mips/kernel/traps.c b/arch/mips/kernel/traps.c
index 9d9b2a4..7393f33 100644
--- a/arch/mips/kernel/traps.c
+++ b/arch/mips/kernel/traps.c
@@ -2306,10 +2306,15 @@ void __init trap_init(void)
 		 * EVA is special though as it allows segments to be rearranged
 		 * and to become uncached during cache error handling.
 		 */
-		if (!IS_ENABLED(CONFIG_EVA) && !WARN_ON(ebase_pa >= 0x20000000))
+		if (ebase_pa < 0x20000000)
 			ebase = CKSEG0ADDR(ebase_pa);
-		else
+		else if (IS_ENABLED(CONFIG_EVA))
 			ebase = (unsigned long)phys_to_virt(ebase_pa);
+		else {
+			memblock_free(ebase_pa, vec_size);
+			ebase = CKSEG0;
+			memblock_reserve(virt_to_phys((void *)ebase), vec_size);
+		}
 	}
 
 	if (cpu_has_mmips) {
-- 
2.7.0

Re: [PATCH] MIPS: Make sure ebase address is in KSEG0

[Paul Burton]

Hi Huacai,

On Thu, Nov 21, 2019 at 07:30:20PM +0800, Huacai Chen wrote:
> Dynamically allocated ebase address above 0x20000000 can be triggered
> by some special physical memory layout, or just by a "big kernel + big
> initrd + big swiotlb" configuration.
> 
> For MIPS32, ebase address above 0x20000000 is unusable, for MIPS64 it
> is usable but this case is warned. However, this warning is useless
> because it is unfixable in a specific system configuration. So we just
> use CKSEG0 as a fallback.

I'd prefer that we don't assume there's memory at physical address zero
- that property doesn't hold for all systems.

How about the change I suggested previously over here:

  https://lore.kernel.org/linux-mips/20191108191149.bbq3h4xp4famsh2n@lantea.localdomain/

Would that work for you?

Thanks,
    Paul

> Signed-off-by: Huacai Chen <chenhc@lemote.com>
> ---
>  arch/mips/kernel/traps.c | 9 +++++++--
>  1 file changed, 7 insertions(+), 2 deletions(-)
> 
> diff --git a/arch/mips/kernel/traps.c b/arch/mips/kernel/traps.c
> index 9d9b2a4..7393f33 100644
> --- a/arch/mips/kernel/traps.c
> +++ b/arch/mips/kernel/traps.c
> @@ -2306,10 +2306,15 @@ void __init trap_init(void)
>  		 * EVA is special though as it allows segments to be rearranged
>  		 * and to become uncached during cache error handling.
>  		 */
> -		if (!IS_ENABLED(CONFIG_EVA) && !WARN_ON(ebase_pa >= 0x20000000))
> +		if (ebase_pa < 0x20000000)
>  			ebase = CKSEG0ADDR(ebase_pa);
> -		else
> +		else if (IS_ENABLED(CONFIG_EVA))
>  			ebase = (unsigned long)phys_to_virt(ebase_pa);
> +		else {
> +			memblock_free(ebase_pa, vec_size);
> +			ebase = CKSEG0;
> +			memblock_reserve(virt_to_phys((void *)ebase), vec_size);
> +		}
>  	}
>  
>  	if (cpu_has_mmips) {
> -- 
> 2.7.0
> 

Re: [PATCH] MIPS: Make sure ebase address is in KSEG0

[Jiaxun Yang]

于 2019年11月23日 GMT+08:00 上午2:47:31, Paul Burton <paulburton@kernel.org> 写到:
>Hi Huacai,
>
>On Thu, Nov 21, 2019 at 07:30:20PM +0800, Huacai Chen wrote:
>> Dynamically allocated ebase address above 0x20000000 can be triggered
>> by some special physical memory layout, or just by a "big kernel +
>big
>> initrd + big swiotlb" configuration.
>> 
>> For MIPS32, ebase address above 0x20000000 is unusable, for MIPS64 it
>> is usable but this case is warned. However, this warning is useless
>> because it is unfixable in a specific system configuration. So we
>just
>> use CKSEG0 as a fallback.
>
>I'd prefer that we don't assume there's memory at physical address zero
>- that property doesn't hold for all systems.
>
>How about the change I suggested previously over here:
>
>https://lore.kernel.org/linux-mips/20191108191149.bbq3h4xp4famsh2n@lantea.localdomain/
>
>Would that work for you?

Hi Paul

Our problem is, sometimes the ebase from firmware is totally a random address, 
even not inside the memory region. I'd prefer ignore address if it's not valid since the warning here can't deal with it.

Thanks.
>
>Thanks,
>    Paul
>
-- 
Jiaxun Yang

Re: [PATCH] MIPS: Make sure ebase address is in KSEG0

[Jiaxun Yang]

于 2019年11月23日 GMT+08:00 下午1:08:38, Jiaxun Yang <jiaxun.yang@flygoat.com> 写到:
>
>
>于 2019年11月23日 GMT+08:00 上午2:47:31, Paul Burton <paulburton@kernel.org>
>写到:
>>Hi Huacai,
>>
>>On Thu, Nov 21, 2019 at 07:30:20PM +0800, Huacai Chen wrote:
>>> Dynamically allocated ebase address above 0x20000000 can be
>triggered
>>> by some special physical memory layout, or just by a "big kernel +
>>big
>>> initrd + big swiotlb" configuration.
>>> 
>>> For MIPS32, ebase address above 0x20000000 is unusable, for MIPS64
>it
>>> is usable but this case is warned. However, this warning is useless
>>> because it is unfixable in a specific system configuration. So we
>>just
>>> use CKSEG0 as a fallback.
>>
>>I'd prefer that we don't assume there's memory at physical address
>zero
>>- that property doesn't hold for all systems.
>>
>>How about the change I suggested previously over here:
>>
>>https://lore.kernel.org/linux-mips/20191108191149.bbq3h4xp4famsh2n@lantea.localdomain/
>>
>>Would that work for you?
>
>Hi Paul
>
>Our problem is, sometimes the ebase from firmware is totally a random
>address, 
>even not inside the memory region. I'd prefer ignore address if it's
>not valid since the warning here can't deal with it.

I'm sorry. I was wrong. Please ignore the noise.

Thanks

>
>Thanks.
>>
>>Thanks,
>>    Paul
>>

-- 
Jiaxun Yang

Re: [PATCH] MIPS: Make sure ebase address is in KSEG0

[Huacai Chen]

Hi, Paul,

On Sat, Nov 23, 2019 at 2:11 PM Jiaxun Yang <jiaxun.yang@flygoat.com> wrote:
>
>
>
> 于 2019年11月23日 GMT+08:00 下午1:08:38, Jiaxun Yang <jiaxun.yang@flygoat.com> 写到:
> >
> >
> >于 2019年11月23日 GMT+08:00 上午2:47:31, Paul Burton <paulburton@kernel.org>
> >写到:
> >>Hi Huacai,
> >>
> >>On Thu, Nov 21, 2019 at 07:30:20PM +0800, Huacai Chen wrote:
> >>> Dynamically allocated ebase address above 0x20000000 can be
> >triggered
> >>> by some special physical memory layout, or just by a "big kernel +
> >>big
> >>> initrd + big swiotlb" configuration.
> >>>
> >>> For MIPS32, ebase address above 0x20000000 is unusable, for MIPS64
> >it
> >>> is usable but this case is warned. However, this warning is useless
> >>> because it is unfixable in a specific system configuration. So we
> >>just
> >>> use CKSEG0 as a fallback.
> >>
> >>I'd prefer that we don't assume there's memory at physical address
> >zero
> >>- that property doesn't hold for all systems.
> >>
> >>How about the change I suggested previously over here:
> >>
> >>https://lore.kernel.org/linux-mips/20191108191149.bbq3h4xp4famsh2n@lantea.localdomain/
> >>
> >>Would that work for you?
Now this patch has nothing to do with WG bit, and also has nothing to
do with Loongson, it just a problem with ebase address -- on any
MIPSr2 platforms where ebase is dynamically allocated.

In the comments it is said that ebase address above 0x20000000 (which
should be in XKphys) has problems to handle cache error. However, if
we really treat it as a problem, we should avoid it (not just a
warning); and if we don't think it is a problem, then we can remove
the warning (because the warning is unfixable in a specific system
configuration).

Huacai
> >
> >Hi Paul
> >
> >Our problem is, sometimes the ebase from firmware is totally a random
> >address,
> >even not inside the memory region. I'd prefer ignore address if it's
> >not valid since the warning here can't deal with it.
>
> I'm sorry. I was wrong. Please ignore the noise.
>
> Thanks
>
> >
> >Thanks.
> >>
> >>Thanks,
> >>    Paul
> >>
>
> --
> Jiaxun Yang

Re: [PATCH] MIPS: Make sure ebase address is in KSEG0

[Paul Burton]

Hi Huacai,

On Sun, Nov 24, 2019 at 12:00:51PM +0800, Huacai Chen wrote:
> On Sat, Nov 23, 2019 at 2:11 PM Jiaxun Yang <jiaxun.yang@flygoat.com> wrote:
> > 于 2019年11月23日 GMT+08:00 下午1:08:38, Jiaxun Yang <jiaxun.yang@flygoat.com> 写到:
> > >于 2019年11月23日 GMT+08:00 上午2:47:31, Paul Burton <paulburton@kernel.org>
> > >写到:
> > >>On Thu, Nov 21, 2019 at 07:30:20PM +0800, Huacai Chen wrote:
> > >>> Dynamically allocated ebase address above 0x20000000 can be
> > >triggered
> > >>> by some special physical memory layout, or just by a "big kernel +
> > >>big
> > >>> initrd + big swiotlb" configuration.
> > >>>
> > >>> For MIPS32, ebase address above 0x20000000 is unusable, for MIPS64
> > >it
> > >>> is usable but this case is warned. However, this warning is useless
> > >>> because it is unfixable in a specific system configuration. So we
> > >>just
> > >>> use CKSEG0 as a fallback.
> > >>
> > >>I'd prefer that we don't assume there's memory at physical address
> > >zero
> > >>- that property doesn't hold for all systems.
> > >>
> > >>How about the change I suggested previously over here:
> > >>
> > >>https://lore.kernel.org/linux-mips/20191108191149.bbq3h4xp4famsh2n@lantea.localdomain/
> > >>
> > >>Would that work for you?
>
> Now this patch has nothing to do with WG bit,

Well, kind of. The 512MB limit is implicitly tied to the lack of a
functional WG bit.

> and also has nothing to do with Loongson, it just a problem with ebase
> address -- on any MIPSr2 platforms where ebase is dynamically
> allocated.

Right, and the patch I proposed ensures that the allocation happens in
the low 512MB for systems without the WG bit, ie. in memory we can
access via kseg0. It just doesn't assume the memory is at physical
address 0 - it might be, but on some systems there might not even be
memory at 0.

In practice we allocate memory bottom-up, so I'm not sure how you're
managing to see problems here anyway - allocations this early should
easily be within the low 512MB. If you could explain step by step what
actually goes wrong on your system that would be helpful.

> In the comments it is said that ebase address above 0x20000000 (which
> should be in XKphys) has problems to handle cache error. However, if
> we really treat it as a problem, we should avoid it (not just a
> warning); and if we don't think it is a problem, then we can remove
> the warning (because the warning is unfixable in a specific system
> configuration).

The warning can probably go away if my patch is applied, since the
allocation is constrained to the right memory region anyway.

Thanks,
    Paul

Re: [PATCH] MIPS: Make sure ebase address is in KSEG0

[Huacai Chen]

Hi, Paul,

On Tue, Nov 26, 2019 at 7:44 AM Paul Burton <paulburton@kernel.org> wrote:
>
> Hi Huacai,
>
> On Sun, Nov 24, 2019 at 12:00:51PM +0800, Huacai Chen wrote:
> > On Sat, Nov 23, 2019 at 2:11 PM Jiaxun Yang <jiaxun.yang@flygoat.com> wrote:
> > > 于 2019年11月23日 GMT+08:00 下午1:08:38, Jiaxun Yang <jiaxun.yang@flygoat.com> 写到:
> > > >于 2019年11月23日 GMT+08:00 上午2:47:31, Paul Burton <paulburton@kernel.org>
> > > >写到:
> > > >>On Thu, Nov 21, 2019 at 07:30:20PM +0800, Huacai Chen wrote:
> > > >>> Dynamically allocated ebase address above 0x20000000 can be
> > > >triggered
> > > >>> by some special physical memory layout, or just by a "big kernel +
> > > >>big
> > > >>> initrd + big swiotlb" configuration.
> > > >>>
> > > >>> For MIPS32, ebase address above 0x20000000 is unusable, for MIPS64
> > > >it
> > > >>> is usable but this case is warned. However, this warning is useless
> > > >>> because it is unfixable in a specific system configuration. So we
> > > >>just
> > > >>> use CKSEG0 as a fallback.
> > > >>
> > > >>I'd prefer that we don't assume there's memory at physical address
> > > >zero
> > > >>- that property doesn't hold for all systems.
> > > >>
> > > >>How about the change I suggested previously over here:
> > > >>
> > > >>https://lore.kernel.org/linux-mips/20191108191149.bbq3h4xp4famsh2n@lantea.localdomain/
> > > >>
> > > >>Would that work for you?
> >
> > Now this patch has nothing to do with WG bit,
>
> Well, kind of. The 512MB limit is implicitly tied to the lack of a
> functional WG bit.
Yes, if CPU has no WG, we cannot use an ebase address above 512M, but
a CPU with WG also has problems, please see the comments:

            * It shouldn't generally be in XKPhys on MIPS64 to avoid
            * hitting a poorly defined exception base for Cache Errors.

So I think the warning is in order to tell us that we should avoid
ebase address above 512M. And if we don't think this is a problem then
let's simply remove the warning.

>
> > and also has nothing to do with Loongson, it just a problem with ebase
> > address -- on any MIPSr2 platforms where ebase is dynamically
> > allocated.
>
> Right, and the patch I proposed ensures that the allocation happens in
> the low 512MB for systems without the WG bit, ie. in memory we can
> access via kseg0. It just doesn't assume the memory is at physical
> address 0 - it might be, but on some systems there might not even be
> memory at 0.
>
> In practice we allocate memory bottom-up, so I'm not sure how you're
> managing to see problems here anyway - allocations this early should
> easily be within the low 512MB. If you could explain step by step what
> actually goes wrong on your system that would be helpful.
OK. Traditionally, Loongson-3's memory layout is like this:

0-240M: low memory
240-256M: reserved for firmware
256M-4G: holes, including PCI mem, MMIO registers, etc.
4G-...: high memory

In practise, the kernel can hold as large as 60M memory (because of
sparsemem), then initrd can be also as large as 60M (a full functional
rootfs), swiotlb can be as large as 128M. Initrd and swiotlb is
allocated before ebase, so in this extreme case, ebase address is
above 512M.

Huacai

>
> > In the comments it is said that ebase address above 0x20000000 (which
> > should be in XKphys) has problems to handle cache error. However, if
> > we really treat it as a problem, we should avoid it (not just a
> > warning); and if we don't think it is a problem, then we can remove
> > the warning (because the warning is unfixable in a specific system
> > configuration).
>
> The warning can probably go away if my patch is applied, since the
> allocation is constrained to the right memory region anyway.
>
> Thanks,
>     Paul