mips 32 bit HIGHMEM support

mips 32 bit HIGHMEM support

[exister99]

Hello Ralf,

I work for HP in Boise, ID, USA.  For the last six months I have been
working with Linux in HP Laser jet Printers.  It is running in a formatter
board that features an ASIC with a MIPS 20Kc core.

The original Linux port was achieved by others working before me.  I began
working on applications for it when I started here (to make it print
actually).  As I have tried to milk more performance out of this board the
recurrent bottleneck has been the fact that the kernel only recognizes
512Mb of RAM.  So my current endeavor is fix the kernel so it sees the 1G
of RAM that is physically present in the system.

I reconfigured and rebuilt the kernel with HIGHMEM support.  HIGHMEM is
seen and booting up goes splendidly until init (pid 1) kicks off.  Early
on in the execution of init the system crashes.  Specifically it crashes
in do_page_fault().  A couple of strange things I have noticed:

1.  All memory pages are divied up between the DMA Zone and the HIGHMEM
Zone at bootup.  the NORMAL zone gets 0 pages.  Not sure if this is
normal.

2.  The virtual address that do_page_fault() chokes on is 0x00000000. 
This doesn't seem like a reasonable address for init to be accessing
especially considering its mem map only contains the chunks
0x10000000-0x10001000 and 0x400000-0x40b000 (I got these by traversing
init's mm->mm_rb tree).

I decided to contact you about this after digging up this old posting of
yours:

>Is anybody using 32-bit MIPS CPUs which have more than 512mb of memory or
>to be more exact have RAM that isn't accessible via the KSEG0 / KSEG1
>window?
>
>So far I haven't ever seen such a machine.  For 64-bit CPUs the right
>thing to do is easy - use a 64-bit kernel.  But for 32-bit CPUs the Intel
>highmem stuff in the memory managment now gives us a sane way to use
>the memory of such configuration with just a little bit of extra code.
>
>So if anybody wants support for such a configuration, please drop me a
>note.
>
>Thanks,
>
>  Ralf

I think this describes my machine.

If this is an issue who's solution is old news please point me to the
solution.  In any case any ideas or guidance regarding this crash would be
greatly appreciated.

Regards, Andrew Stone

mips 32 bit HIGHMEM support

[exister99]

Hello Ralf,

I work for HP in Boise, ID, USA.  For the last six months I have been
working with Linux in HP Laser jet Printers.  It is running in a formatter
board that features an ASIC with a MIPS 20Kc core.

The original Linux port was achieved by others working before me.  I began
working on applications for it when I started here (to make it print
actually).  As I have tried to milk more performance out of this board the
recurrent bottleneck has been the fact that the kernel only recognizes
512Mb of RAM.  So my current endeavor is fix the kernel so it sees the 1G
of RAM that is physically present in the system.

I reconfigured and rebuilt the kernel with HIGHMEM support.  HIGHMEM is
seen and booting up goes splendidly until init (pid 1) kicks off.  Early
on in the execution of init the system crashes.  Specifically it crashes
in do_page_fault().  A couple of strange things I have noticed:

1.  All memory pages are divied up between the DMA Zone and the HIGHMEM
Zone at bootup.  the NORMAL zone gets 0 pages.  Not sure if this is
normal.

2.  The virtual address that do_page_fault() chokes on is 0x00000000. 
This doesn't seem like a reasonable address for init to be accessing
especially considering its mem map only contains the chunks
0x10000000-0x10001000 and 0x400000-0x40b000 (I got these by traversing
init's mm->mm_rb tree).

I decided to contact you about this after digging up this old posting of
yours:

>Is anybody using 32-bit MIPS CPUs which have more than 512mb of memory or
>to be more exact have RAM that isn't accessible via the KSEG0 / KSEG1
>window?
>
>So far I haven't ever seen such a machine.  For 64-bit CPUs the right
>thing to do is easy - use a 64-bit kernel.  But for 32-bit CPUs the Intel
>highmem stuff in the memory managment now gives us a sane way to use
>the memory of such configuration with just a little bit of extra code.
>
>So if anybody wants support for such a configuration, please drop me a
>note.
>
>Thanks,
>
>  Ralf

I think this describes my machine.

If this is an issue who's solution is old news please point me to the
solution.  In any case any ideas or guidance regarding this crash would be
greatly appreciated.

Regards, Andrew Stone

Re: mips 32 bit HIGHMEM support

[Ralf Baechle]

On Wed, Oct 08, 2003 at 04:00:33PM -0600, exister99@velocitus.net wrote:

> I work for HP in Boise, ID, USA.  For the last six months I have been
> working with Linux in HP Laser jet Printers.  It is running in a formatter
> board that features an ASIC with a MIPS 20Kc core.

Reminds me of the HP Laserjet code in the kernel.  Anybody still using
or testing that?

> The original Linux port was achieved by others working before me.  I began
> working on applications for it when I started here (to make it print
> actually).  As I have tried to milk more performance out of this board the
> recurrent bottleneck has been the fact that the kernel only recognizes
> 512Mb of RAM.  So my current endeavor is fix the kernel so it sees the 1G
> of RAM that is physically present in the system.
> 
> I reconfigured and rebuilt the kernel with HIGHMEM support.  HIGHMEM is
> seen and booting up goes splendidly until init (pid 1) kicks off.  Early
> on in the execution of init the system crashes.  Specifically it crashes
> in do_page_fault().  A couple of strange things I have noticed:
> 
> 1.  All memory pages are divied up between the DMA Zone and the HIGHMEM
> Zone at bootup.  the NORMAL zone gets 0 pages.  Not sure if this is
> normal.
> 
> 2.  The virtual address that do_page_fault() chokes on is 0x00000000. 
> This doesn't seem like a reasonable address for init to be accessing
> especially considering its mem map only contains the chunks
> 0x10000000-0x10001000 and 0x400000-0x40b000 (I got these by traversing
> init's mm->mm_rb tree).
> 
> I decided to contact you about this after digging up this old posting of
> yours:

[... ancient posting deleted ...]

> I think this describes my machine.
> 
> If this is an issue who's solution is old news please point me to the
> solution.  In any case any ideas or guidance regarding this crash would be
> greatly appreciated.

In the meantime there is a highmem implementation for the MIPS kernel.
It's got a limitation - it only works on physically indexed D-caches or
more exactly processors that don't suffer from cache aliases.  On
processors that have such aliases the necessary flushes are rather bad
for performance so this currently simply isn't suported.

Anyway, for a 64-bit processor such as the 20Kc I suggest a 64-bit kernel.
Highmem is a pain and 64-bit is the cure.

  Ralf

bug in kernel_entry?

[Steve Scott]

I ran across what looks like a bug in the 'kernel_entry' function in
linux/arch/mips/kernel/head.S while chasing another problem. Our
version of kernel_entry is for 2.4.17. 2.4.22 seems to have
the same problem.

kernel_entry initializes the kernel stack pointer 'kernelsp'. But then
immediately after this clears the bss, which has the side effect of setting
kernelsp to 0. In our system, on initial entry to cpu_idle(), kernelsp is 0.

The bug "heals" itself the first time schedule() is called. But, if for some
reason CP0_STATUS doesn't have CU0 set at startup (which would be
bad for other reasons), and you get an exception before the first call to
schedule() (e.g., the syscall to create "init"), the exception handler will
try to save registers starting from kernelsp, which is 0.

from head.S:

    NESTED(kernel_entry, 16, sp)
    .
    .
    .
    /*
     * Stack for kernel and init, current variable
     */
    la $28, init_task_union
    addiu t0, $28, KERNEL_STACK_SIZE-32
    subu sp, t0, 4*SZREG

    sw t0, kernelsp                /* <-- this is going to get overwritten below when bss
is cleared... [srs] */

    /* The firmware/bootloader passes argc/argp/envp
     * to us as arguments.  But clear bss first because
     * the romvec and other important info is stored there
     * by prom_init().
     */
    la t0, _edata                    /* <-- here is the code that eventually clears
kernelsp [srs] */
    sw zero, (t0)
    la t1, (_end - 4)
1:
    addiu t0, 4
    bne t0, t1, 1b
    sw zero, (t0)

   jal init_arch
    nop
  END(kernel_entry)

--steve

bug in kernel_entry?

[Steve Scott]

I ran across what looks like a bug in the 'kernel_entry' function in
linux/arch/mips/kernel/head.S while chasing another problem. Our
version of kernel_entry is for 2.4.17. 2.4.22 seems to have
the same problem.

kernel_entry initializes the kernel stack pointer 'kernelsp'. But then
immediately after this clears the bss, which has the side effect of setting
kernelsp to 0. In our system, on initial entry to cpu_idle(), kernelsp is 0.

The bug "heals" itself the first time schedule() is called. But, if for some
reason CP0_STATUS doesn't have CU0 set at startup (which would be
bad for other reasons), and you get an exception before the first call to
schedule() (e.g., the syscall to create "init"), the exception handler will
try to save registers starting from kernelsp, which is 0.

from head.S:

    NESTED(kernel_entry, 16, sp)
    .
    .
    .
    /*
     * Stack for kernel and init, current variable
     */
    la $28, init_task_union
    addiu t0, $28, KERNEL_STACK_SIZE-32
    subu sp, t0, 4*SZREG

    sw t0, kernelsp                /* <-- this is going to get overwritten below when bss
is cleared... [srs] */

    /* The firmware/bootloader passes argc/argp/envp
     * to us as arguments.  But clear bss first because
     * the romvec and other important info is stored there
     * by prom_init().
     */
    la t0, _edata                    /* <-- here is the code that eventually clears
kernelsp [srs] */
    sw zero, (t0)
    la t1, (_end - 4)
1:
    addiu t0, 4
    bne t0, t1, 1b
    sw zero, (t0)

   jal init_arch
    nop
  END(kernel_entry)

--steve

Re: bug in kernel_entry?

[Ralf Baechle]

On Thu, Oct 09, 2003 at 03:11:36PM -0700, Steve Scott wrote:

> kernel_entry initializes the kernel stack pointer 'kernelsp'. But then
> immediately after this clears the bss, which has the side effect of setting
> kernelsp to 0. In our system, on initial entry to cpu_idle(), kernelsp is 0.

Funny this wasn't noticed for so long.  Fix is in CVS, thanks for reporting.

  Ralf

Re: mips 32 bit HIGHMEM support

[Ralf Baechle]

On Thu, Oct 09, 2003 at 04:03:20PM +0200, Ralf Baechle wrote:

> It's got a limitation - it only works on physically indexed D-caches or
> more exactly processors that don't suffer from cache aliases.  On
> processors that have such aliases the necessary flushes are rather bad
> for performance so this currently simply isn't suported.

Small update - I changed the memory managment code to ignore highmem
on cache configuration that are not supported by the highmem code.

  Ralf

64 bit kernel in the name of HIGHMEM

[exister99]

Hi Ralf,

Thanks for the prompt response.

> Reminds me of the HP Laserjet code in the kernel.  Anybody still using
> or testing that?

Yes, the code I am working with is basically a snapshot of the kernel
source with /arch/mips/hp-lj etc.

> Anyway, for a 64-bit processor such as the 20Kc I suggest a 64-bit
> kernel. Highmem is a pain and 64-bit is the cure.

To that end I have been trying to build a 64 bit kernel.  I think I am
going about it correctly.  Got ahold of

mips64el-linux-binutils-2.14-3.i386.rpm 
mips64el-linux-boot-gcc-2.95.4-9.i386.rpm

and got cranking.  Currently make is dying with the following output:


[root@l51dhcp252 ljlinux]# make ARCH=mips64 CROSS_COMPILE=mips64el-linux-
. scripts/mkversion > .tmpversion
mips64el-linux-gcc -D__KERNEL__ -I/home/astone/cf_burn_64/ljlinux/include
-Wall -Wstrict-prototypes -Wno-trigraphs -O2 -fno-strict-aliasing
-fno-common -I/usr/lib/gcc-lib/mips64el-linux/2.95.4/include
-fomit-frame-pointer -Dx_unity_x -I
/home/astone/cf_burn_64/ljlinux/include/asm/gcc -mabi=64 -G 0
-mno-abicalls -fno-pic -Wa,--trap -pipe  -DUTS_MACHINE='"mips64"'
-DKBUILD_BASENAME=version -c -o init/version.o init/version.c
mips64el-linux-gcc -D__KERNEL__ -I/home/astone/cf_burn_64/ljlinux/include
-Wall -Wstrict-prototypes -Wno-trigraphs -O2 -fno-strict-aliasing
-fno-common -I/usr/lib/gcc-lib/mips64el-linux/2.95.4/include
-fomit-frame-pointer -Dx_unity_x -I
/home/astone/cf_burn_64/ljlinux/include/asm/gcc -mabi=64 -G 0
-mno-abicalls -fno-pic -Wa,--trap -pipe   -DKBUILD_BASENAME=do_mounts -c
-o init/do_mounts.o init/do_mounts.c
cpp0: output pipe has been closed
mips64el-linux-gcc: Internal compiler error: program cc1 got fatal signal 11
make: *** [init/do_mounts.o] Error 1


Initial searches claim this is a bug in the compiler.  Could it be due to
something I am doing wrong?  Any guidance on building a 64 bit kernel
would be greatly appreciated.

Best Regards, Andrew Stone

64 bit kernel in the name of HIGHMEM

[exister99]

Hi Ralf,

Thanks for the prompt response.

> Reminds me of the HP Laserjet code in the kernel.  Anybody still using
> or testing that?

Yes, the code I am working with is basically a snapshot of the kernel
source with /arch/mips/hp-lj etc.

> Anyway, for a 64-bit processor such as the 20Kc I suggest a 64-bit
> kernel. Highmem is a pain and 64-bit is the cure.

To that end I have been trying to build a 64 bit kernel.  I think I am
going about it correctly.  Got ahold of

mips64el-linux-binutils-2.14-3.i386.rpm 
mips64el-linux-boot-gcc-2.95.4-9.i386.rpm

and got cranking.  Currently make is dying with the following output:


[root@l51dhcp252 ljlinux]# make ARCH=mips64 CROSS_COMPILE=mips64el-linux-
. scripts/mkversion > .tmpversion
mips64el-linux-gcc -D__KERNEL__ -I/home/astone/cf_burn_64/ljlinux/include
-Wall -Wstrict-prototypes -Wno-trigraphs -O2 -fno-strict-aliasing
-fno-common -I/usr/lib/gcc-lib/mips64el-linux/2.95.4/include
-fomit-frame-pointer -Dx_unity_x -I
/home/astone/cf_burn_64/ljlinux/include/asm/gcc -mabi=64 -G 0
-mno-abicalls -fno-pic -Wa,--trap -pipe  -DUTS_MACHINE='"mips64"'
-DKBUILD_BASENAME=version -c -o init/version.o init/version.c
mips64el-linux-gcc -D__KERNEL__ -I/home/astone/cf_burn_64/ljlinux/include
-Wall -Wstrict-prototypes -Wno-trigraphs -O2 -fno-strict-aliasing
-fno-common -I/usr/lib/gcc-lib/mips64el-linux/2.95.4/include
-fomit-frame-pointer -Dx_unity_x -I
/home/astone/cf_burn_64/ljlinux/include/asm/gcc -mabi=64 -G 0
-mno-abicalls -fno-pic -Wa,--trap -pipe   -DKBUILD_BASENAME=do_mounts -c
-o init/do_mounts.o init/do_mounts.c
cpp0: output pipe has been closed
mips64el-linux-gcc: Internal compiler error: program cc1 got fatal signal 11
make: *** [init/do_mounts.o] Error 1


Initial searches claim this is a bug in the compiler.  Could it be due to
something I am doing wrong?  Any guidance on building a 64 bit kernel
would be greatly appreciated.

Best Regards, Andrew Stone

Re: mips 32 bit HIGHMEM support

[Ralf Baechle]

On Fri, Oct 10, 2003 at 10:44:19AM -0700, Krishna Kondaka wrote:

> How safe is it to enable HIGHMEM for sibyte/broadcom's BCM12500 processors?
> Do you know if any one is using HIGHMEM enabled linux on BCM12500s?

Perfectly reliable - in fact highmem for MIPS was written and debugged
on this chip but there are many other that will work just as fine since
they share the technical properties required to run highmem.

Actually with the latest change in CVS enabling CONFIG_HIGHMEM is always
safe; in cases where it's not safe the kernel will simply limit the
memory it's using to just lowmem.

Stil the usual warning applies - highmem is a kludge and for systems
doing heavy I/O it can be a rather slow kludge.  64-bit is the Nirvana
kernel hackers are seeking ;-)

  Ralf

RE: mips 32 bit HIGHMEM support

[Krishna Kondaka]

Ralf,

How safe is it to enable HIGHMEM for sibyte/broadcom's BCM12500 processors?
Do you know if any one is using HIGHMEM enabled linux on BCM12500s?

Thanks
Krishna

-----Original Message-----
From: Ralf Baechle [mailto:ralf@linux-mips.org]
Sent: Thursday, October 09, 2003 7:03 AM
To: exister99@velocitus.net
Cc: linux-mips@linux-mips.org
Subject: Re: mips 32 bit HIGHMEM support


On Wed, Oct 08, 2003 at 04:00:33PM -0600, exister99@velocitus.net wrote:

> I work for HP in Boise, ID, USA.  For the last six months I have been
> working with Linux in HP Laser jet Printers.  It is running in a formatter
> board that features an ASIC with a MIPS 20Kc core.

Reminds me of the HP Laserjet code in the kernel.  Anybody still using
or testing that?

> The original Linux port was achieved by others working before me.  I began
> working on applications for it when I started here (to make it print
> actually).  As I have tried to milk more performance out of this board the
> recurrent bottleneck has been the fact that the kernel only recognizes
> 512Mb of RAM.  So my current endeavor is fix the kernel so it sees the 1G
> of RAM that is physically present in the system.
> 
> I reconfigured and rebuilt the kernel with HIGHMEM support.  HIGHMEM is
> seen and booting up goes splendidly until init (pid 1) kicks off.  Early
> on in the execution of init the system crashes.  Specifically it crashes
> in do_page_fault().  A couple of strange things I have noticed:
> 
> 1.  All memory pages are divied up between the DMA Zone and the HIGHMEM
> Zone at bootup.  the NORMAL zone gets 0 pages.  Not sure if this is
> normal.
> 
> 2.  The virtual address that do_page_fault() chokes on is 0x00000000. 
> This doesn't seem like a reasonable address for init to be accessing
> especially considering its mem map only contains the chunks
> 0x10000000-0x10001000 and 0x400000-0x40b000 (I got these by traversing
> init's mm->mm_rb tree).
> 
> I decided to contact you about this after digging up this old posting of
> yours:

[... ancient posting deleted ...]

> I think this describes my machine.
> 
> If this is an issue who's solution is old news please point me to the
> solution.  In any case any ideas or guidance regarding this crash would be
> greatly appreciated.

In the meantime there is a highmem implementation for the MIPS kernel.
It's got a limitation - it only works on physically indexed D-caches or
more exactly processors that don't suffer from cache aliases.  On
processors that have such aliases the necessary flushes are rather bad
for performance so this currently simply isn't suported.

Anyway, for a 64-bit processor such as the 20Kc I suggest a 64-bit kernel.
Highmem is a pain and 64-bit is the cure.

  Ralf


SPECIAL NOTICE 

All information transmitted hereby is intended only for the use of the
addressee(s) named above and may contain confidential and privileged
information. Any unauthorized review, use, disclosure or distribution
of confidential and privileged information is prohibited. If the reader
of this message is not the intended recipient(s) or the employee or agent
responsible for delivering the message to the intended recipient, you are
herby notified that you must not read this transmission and that disclosure,
copying, printing, distribution or use of any of the information contained
in or attached to this transmission is STRICTLY PROHIBITED.

Anyone who receives confidential and privileged information in error should
notify us immediately by telephone and mail the original message to us at
the above address and destroy all copies.  To the extent any portion of this
communication contains public information, no such restrictions apply to that
information. (gate01)

RE: mips 32 bit HIGHMEM support

[Krishna Kondaka]

Ralf,

How safe is it to enable HIGHMEM for sibyte/broadcom's BCM12500 processors?
Do you know if any one is using HIGHMEM enabled linux on BCM12500s?

Thanks
Krishna

-----Original Message-----
From: Ralf Baechle [mailto:ralf@linux-mips.org]
Sent: Thursday, October 09, 2003 7:03 AM
To: exister99@velocitus.net
Cc: linux-mips@linux-mips.org
Subject: Re: mips 32 bit HIGHMEM support


On Wed, Oct 08, 2003 at 04:00:33PM -0600, exister99@velocitus.net wrote:

> I work for HP in Boise, ID, USA.  For the last six months I have been
> working with Linux in HP Laser jet Printers.  It is running in a formatter
> board that features an ASIC with a MIPS 20Kc core.

Reminds me of the HP Laserjet code in the kernel.  Anybody still using
or testing that?

> The original Linux port was achieved by others working before me.  I began
> working on applications for it when I started here (to make it print
> actually).  As I have tried to milk more performance out of this board the
> recurrent bottleneck has been the fact that the kernel only recognizes
> 512Mb of RAM.  So my current endeavor is fix the kernel so it sees the 1G
> of RAM that is physically present in the system.
> 
> I reconfigured and rebuilt the kernel with HIGHMEM support.  HIGHMEM is
> seen and booting up goes splendidly until init (pid 1) kicks off.  Early
> on in the execution of init the system crashes.  Specifically it crashes
> in do_page_fault().  A couple of strange things I have noticed:
> 
> 1.  All memory pages are divied up between the DMA Zone and the HIGHMEM
> Zone at bootup.  the NORMAL zone gets 0 pages.  Not sure if this is
> normal.
> 
> 2.  The virtual address that do_page_fault() chokes on is 0x00000000. 
> This doesn't seem like a reasonable address for init to be accessing
> especially considering its mem map only contains the chunks
> 0x10000000-0x10001000 and 0x400000-0x40b000 (I got these by traversing
> init's mm->mm_rb tree).
> 
> I decided to contact you about this after digging up this old posting of
> yours:

[... ancient posting deleted ...]

> I think this describes my machine.
> 
> If this is an issue who's solution is old news please point me to the
> solution.  In any case any ideas or guidance regarding this crash would be
> greatly appreciated.

In the meantime there is a highmem implementation for the MIPS kernel.
It's got a limitation - it only works on physically indexed D-caches or
more exactly processors that don't suffer from cache aliases.  On
processors that have such aliases the necessary flushes are rather bad
for performance so this currently simply isn't suported.

Anyway, for a 64-bit processor such as the 20Kc I suggest a 64-bit kernel.
Highmem is a pain and 64-bit is the cure.

  Ralf


SPECIAL NOTICE 

All information transmitted hereby is intended only for the use of the
addressee(s) named above and may contain confidential and privileged
information. Any unauthorized review, use, disclosure or distribution
of confidential and privileged information is prohibited. If the reader
of this message is not the intended recipient(s) or the employee or agent
responsible for delivering the message to the intended recipient, you are
herby notified that you must not read this transmission and that disclosure,
copying, printing, distribution or use of any of the information contained
in or attached to this transmission is STRICTLY PROHIBITED.

Anyone who receives confidential and privileged information in error should
notify us immediately by telephone and mail the original message to us at
the above address and destroy all copies.  To the extent any portion of this
communication contains public information, no such restrictions apply to that
information. (gate01)