Re: [PATCH] ppc: Add support for bigger page sizes than 4KB on PPC44x

Re: [PATCH] ppc: Add support for bigger page sizes than 4KB on PPC44x

[Yuri Tikhonov]

[-- Attachment #1: Type: text/plain, Size: 1387 bytes --]

On Tue Mar 13 18:12:14 EST 2007, Benjamin Herrenschmidt wrote:
> - You only adapted head_4xx.S but the config option is global to all of
> arch/ppc, thus changing it will probably break everything else. Make
> sure the option is done in such a way that it can't be changed or the
> kernel doesn't build if changed for a processor family where you haven't
> implemented the support

 The configuration option we had introduced changes the global PPC PAGE_SIZE 
definition indeed. To let changing the PAGE_SIZE from the configuration menu 
for the 44x processors only the following modification of the 
arch/ppc/Kconfig file might make sense:

 - int "Page size (12=>4KB; 16=>64KB)"
+ int "Page size (12=>4KB; 16=>64KB)" if 44x

 For other than 44x processors the PPC_PAGE_SHIFT value will be unchangeable 
from the configuration menu and set to default  value 12 ( PAGE_SIZE = 4 KB).

> - Have you tried other page sizes ? How hard would it be to support 16K
> and what kind of performance numbers do you get with 16K ? It's a better
> compromise for some applications as 64K causes significant bloat of the
> page cache (among other things).

 To support 16K, one more bunch of definitions should be calculated and added 
to the include/asm/ppc_page_asm.h file (for PAGE_SHIFT = 14). Attached is the 
patch that adds 16KB PAGE_SIZE capability for the ppc44x processors.

 Regards, Yuri.

[-- Attachment #2: ppc44x_page_16k_070315.patch --]
[-- Type: text/x-diff, Size: 1917 bytes --]

Support for the PAGE_SIZE = 16 KB on the PPC32 44x platforms. 

Signed-off-by: Yuri Tikhonov <yur@emcraft.com>
---

diff --git a/arch/ppc/Kconfig b/arch/ppc/Kconfig
index e825e98..72af523 100644
--- a/arch/ppc/Kconfig
+++ b/arch/ppc/Kconfig
@@ -1202,7 +1202,7 @@ config SECCOMP
 	  If unsure, say Y. Only embedded should say N here.
 
 config PPC_PAGE_SHIFT
-	int "Page size (12=>4KB; 16=>64KB)" if 44x
+	int "Page size (12=>4KB; 14=>16KB; 16=>64KB)" if 44x
 	default "12"
 	range 12 16
 	help
diff --git a/include/asm-ppc/page.h b/include/asm-ppc/page.h
index 61698ac..0b8f354 100644
--- a/include/asm-ppc/page.h
+++ b/include/asm-ppc/page.h
@@ -29,7 +29,9 @@
 #ifdef CONFIG_PTE_64BIT
 typedef unsigned long long pte_basic_t;
 #if (PAGE_SHIFT == 16) && defined(CONFIG_PPC32)
-#define PTE_SHIFT	(PAGE_SHIFT - 11)	/* 256 ptes per table*/
+#define PTE_SHIFT	(PAGE_SHIFT - 11)	/* 32 ptes per table */
+#elif (PAGE_SHIFT == 14) && defined(CONFIG_PPC32)
+#define PTE_SHIFT	(PAGE_SHIFT - 7)	/* 128 ptes per table */
 #else
 #define PTE_SHIFT	(PAGE_SHIFT - 3)	/* 512 ptes per page */
 #endif
diff --git a/include/asm-ppc/ppc_page_asm.h b/include/asm-ppc/ppc_page_asm.h
index 4918ddc..4f7ee78 100644
--- a/include/asm-ppc/ppc_page_asm.h
+++ b/include/asm-ppc/ppc_page_asm.h
@@ -29,6 +29,19 @@
 #define PPC44x_PTE_ADD_SH	23 /*32 - PMD_SHIFT + PTE_SHIFT + 3*/
 #define PPC44x_PTE_ADD_M1	20 /*32 - 3 - PTE_SHIFT*/
 #define PPC44x_RPN_M2		19 /*31 - PAGE_SHIFT*/
+#elif (PAGE_SHIFT == 14)
+/*
+ * PAGE_SIZE  16K
+ * PAGE_SHIFT 14
+ * PTE_SHIFT  7
+ * PMD_SHIFT  21
+ */
+#define PPC44x_TLB_SIZE		PPC44x_TLB_16K
+#define PPC44x_PGD_OFF_SH	13 /*(32 - PMD_SHIFT + 2)*/
+#define PPC44x_PGD_OFF_M1	19 /*(PMD_SHIFT - 2)*/
+#define PPC44x_PTE_ADD_SH	21 /*32 - PMD_SHIFT + PTE_SHIFT + 3*/
+#define PPC44x_PTE_ADD_M1	22 /*32 - 3 - PTE_SHIFT*/
+#define PPC44x_RPN_M2		17 /*31 - PAGE_SHIFT*/
 #elif (PAGE_SHIFT == 16)
 /*
  * PAGE_SIZE  64K

Re: [PATCH] ppc: Add support for bigger page sizes than 4KB on PPC44x

[Paul Mackerras]

Yuri Tikhonov writes:

>  The configuration option we had introduced changes the global PPC PAGE_SIZE 
> definition indeed. To let changing the PAGE_SIZE from the configuration menu 
> for the 44x processors only the following modification of the 
> arch/ppc/Kconfig file might make sense:
> 
>  - int "Page size (12=>4KB; 16=>64KB)"
> + int "Page size (12=>4KB; 16=>64KB)" if 44x
> 
>  For other than 44x processors the PPC_PAGE_SHIFT value will be unchangeable 
> from the configuration menu and set to default  value 12 ( PAGE_SIZE = 4 KB).

Your patch would still let users select a page shift of 13 or 15,
which won't work (and can't be made to work).

Paul.

Re: [PATCH] ppc: Add support for bigger page sizes than 4KB on PPC44x

[Yuri Tikhonov]

On Saturday 17 March 2007 01:34, Paul Mackerras wrote:
> Your patch would still let users select a page shift of 13 or 15,
> which won't work (and can't be made to work).

 Selecting 13 or 15 will lead to a compile-time error "Unsupported PAGE_SIZE" 
(in include/asm/ppc_page_asm.h), so users will not be able to run kernel with 
PAGE_SIZE different than 4K, 16K or 64K anyway. 

 Do you think that replacing one integer option CONFIG_PPC_PAGE_SHIFT with 
three logical options (i.e., CONFIG_PPC_PAGE_SIZE_4K, 
CONFIG_PPC_PAGE_SIZE_16K and CONFIG_PPC_PAGE_SIZE_64K) will be the better 
choice ?

 Regards, Yuri.

Re: [PATCH] ppc: Add support for bigger page sizes than 4KB on PPC44x

[Paul Mackerras]

Yuri Tikhonov writes:

>  Do you think that replacing one integer option CONFIG_PPC_PAGE_SHIFT with 
> three logical options (i.e., CONFIG_PPC_PAGE_SIZE_4K, 
> CONFIG_PPC_PAGE_SIZE_16K and CONFIG_PPC_PAGE_SIZE_64K) will be the better 
> choice ?

I think you should use a choice...endchoice construct in Kconfig,
which will result in 2 config options, or 3 if you choose to support
16k pages.

Paul.

Re: [PATCH] ppc: Add support for bigger page sizes than 4KB on PPC44x

[Yuri Tikhonov]

 Paul,

On Wednesday 21 March 2007 13:43, you wrote:
> Yuri Tikhonov writes:
> >  Do you think that replacing one integer option CONFIG_PPC_PAGE_SHIFT
> > with three logical options (i.e., CONFIG_PPC_PAGE_SIZE_4K,
> > CONFIG_PPC_PAGE_SIZE_16K and CONFIG_PPC_PAGE_SIZE_64K) will be the better
> > choice ?
>
> I think you should use a choice...endchoice construct in Kconfig,
> which will result in 2 config options, or 3 if you choose to support
> 16k pages.

 Yes, there will be 3 options. Thank you!

 Regards, Yuri.

>
> Paul.

Re: [PATCH] ppc: Add support for bigger page sizes than 4KB on PPC44x

[Yuri Tikhonov]

 The patch replaces the CONFIG_PPC_PAGE_SHIFT integer option with choice of
three boolean options

 Signed-off-by: Yuri Tikhonov <yur@emcraft.com>
--

diff --git a/arch/ppc/Kconfig b/arch/ppc/Kconfig
index 72af523..5eb58dc 100644
--- a/arch/ppc/Kconfig
+++ b/arch/ppc/Kconfig
@@ -1201,15 +1201,24 @@ config SECCOMP

          If unsure, say Y. Only embedded should say N here.

-config PPC_PAGE_SHIFT
-       int "Page size (12=>4KB; 14=>16KB; 16=>64KB)" if 44x
-       default "12"
-       range 12 16
-       help
+choice
+       prompt "Page size"
+       default PPC_PAGE_4K
+       ---help---
          The PAGE_SIZE definition. Increasing the page size may
          improve the system performance in some dedicated cases.

-         If unsure, set it to 12 (4KB).
+         If unsure, set it to 4 KB.
+
+config PPC_PAGE_4K
+       bool "4 KB"
+
+config PPC_PAGE_16K
+       bool "16 KB" if 44x
+
+config PPC_PAGE_64K
+       bool "64 KB" if 44x
+endchoice

 endmenu

diff --git a/include/asm-ppc/page.h b/include/asm-ppc/page.h
index 0b8f354..f46712e 100644
--- a/include/asm-ppc/page.h
+++ b/include/asm-ppc/page.h
@@ -4,7 +4,13 @@
 #include <asm/asm-compat.h>

 /* PAGE_SHIFT determines the page size */
-#define PAGE_SHIFT     CONFIG_PPC_PAGE_SHIFT
+#if defined(CONFIG_PPC_PAGE_4K)
+#define PAGE_SHIFT     12
+#elif defined(CONFIG_PPC_PAGE_16K)
+#define PAGE_SHIFT     14
+#elif defined(CONFIG_PPC_PAGE_64K)
+#define PAGE_SHIFT     16
+#endif
 #define PAGE_SIZE      (ASM_CONST(1) << PAGE_SHIFT)

 /*

Re: [PATCH] ppc: Add support for bigger page sizes than 4KB on PPC44x

[Yuri Tikhonov]

[-- Attachment #1: Type: text/plain, Size: 2770 bytes --]

On Fri Mar 16 16:14:48 EST 2007 Paul Mackerras wrote:
> If I'm reading the patch right, with 64k pages you're using 8kB pgd
> arrays and 256-byte pte arrays, so you're only using 1/256th of each
> PTE page.  In fact each PTE page could map 512MB (assuming 8 byte
> ptes), and the pgd array could be a tiny 8-entry thing, which we could
> kmalloc.  I think that would use memory more efficiently.

 Your understanding of the patch is correct. When the page size is set to 64K 
then the 32-bit linear address is being splitted in the following way:

(1) 64K:
PGD: 11 msb -> there are 2K page global directory entries;
PTE: 5 bits -> 32 8-byte PTE entries in each page table => 32x8=256 of 64K;
Offset: 16 lsb -> page size is 64K.

 So, I do not use "page table" pages (with PTEs) fully indeed. If use the 
following decoding of the linear address then each page table with PTEs will 
be utilized in full:

(2) 64K:
PGD: 3 msb -> there are 8 page global directory entries,
PTE: 13 bits -> 8K 8-byte PTE entries in each page table => 8Kx8=64K of 64K,
Offset: 16 lsb -> page size is 64K.

 The corresponding point is correct regarding to my 16K implementation too:

(3) 16K:
PGD: 7 msb -> there are 128 page global directory entries,
PTE: 11 bits -> 2K 8-byte PTE entries in each page table => 2Kx8=16K of 16K,
Offset: 14 lsb -> page size is 16K.

 Additional note regarding to the 64K pages. When use decoding (2) there is a 
WARN_ON() triggered in the dma_alloc_init() function ( 
arch/powerpc/lib/dma-noncoherent.c):

WARN_ON(!pmd_none(*pmd)),

 that leads to the following message during kernel booting-up:
------------[ cut here ]------------
Badness at c02ebb30 [verbose debug info unavailable]
Call Trace:
[C054FE50] [C000858C] show_stack+0x48/0x190 (unreliable)
[C054FE80] [C00F189C] report_bug+0xa4/0xac
[C054FE90] [C0002A7C] program_check_exception+0x170/0x4a0
[C054FEB0] [C0001E74] ret_from_except_full+0x0/0x4c
[C054FF70] [C003C068] init_irq_proc+0x3c/0x5c
[C054FF80] [C02E0830] init+0x94/0x1c4
[C054FFF0] [C0003AC0] kernel_thread+0x44/0x60

 This warning means that the PMD, which is being checked here, was already 
allocated during executing of some prior code. When only 3 msb are being used 
as a PMD component of the linear address then each PMD entry covers the 512MB 
area (0x20000000). This is rather a big area so I think there is nothing 
strange or erroneous that this area was already accessed somewhere before the 
dma_alloc_init() call. So, to spare users' minds I suggest to replace the 
WARN_ON() call in the dma_alloc_init() with more harmless warning message. 

 The patch that fixes PTE tables memory usage for the 16K and 64K pages is 
attached. Also this patch replace WARN_ON() in the dma_alloc_init() with the 
printk(). 

 Regards, Yuri.

[-- Attachment #2: ppc44x_page_070316.patch --]
[-- Type: text/x-diff, Size: 2582 bytes --]

diff --git a/arch/powerpc/lib/dma-noncoherent.c b/arch/powerpc/lib/dma-noncoherent.c
index 48f3d13..4f9da90 100644
--- a/arch/powerpc/lib/dma-noncoherent.c
+++ b/arch/powerpc/lib/dma-noncoherent.c
@@ -318,7 +318,8 @@ static int __init dma_alloc_init(void)
 			ret = -ENOMEM;
 			break;
 		}
-		WARN_ON(!pmd_none(*pmd));
+		if(!pmd_none(*pmd))
+			printk("%s warning: PMD is not empty.\n", __FUNCTION__);
 
 		pte = pte_alloc_kernel(pmd, CONSISTENT_BASE);
 		if (!pte) {
diff --git a/include/asm-ppc/page.h b/include/asm-ppc/page.h
index 0b8f354..3b2f17a 100644
--- a/include/asm-ppc/page.h
+++ b/include/asm-ppc/page.h
@@ -28,13 +28,7 @@
  */
 #ifdef CONFIG_PTE_64BIT
 typedef unsigned long long pte_basic_t;
-#if (PAGE_SHIFT == 16) && defined(CONFIG_PPC32)
-#define PTE_SHIFT	(PAGE_SHIFT - 11)	/* 32 ptes per table */
-#elif (PAGE_SHIFT == 14) && defined(CONFIG_PPC32)
-#define PTE_SHIFT	(PAGE_SHIFT - 7)	/* 128 ptes per table */
-#else
-#define PTE_SHIFT	(PAGE_SHIFT - 3)	/* 512 ptes per page */
-#endif
+#define PTE_SHIFT	(PAGE_SHIFT - 3)	/* PAGE_SIZE/8 ptes per page */
 #define PTE_FMT		"%16Lx"
 #else
 typedef unsigned long pte_basic_t;
diff --git a/include/asm-ppc/ppc_page_asm.h b/include/asm-ppc/ppc_page_asm.h
index 4f7ee78..1dbfd3b 100644
--- a/include/asm-ppc/ppc_page_asm.h
+++ b/include/asm-ppc/ppc_page_asm.h
@@ -33,27 +33,27 @@
 /*
  * PAGE_SIZE  16K
  * PAGE_SHIFT 14
- * PTE_SHIFT  7
- * PMD_SHIFT  21
+ * PTE_SHIFT  11
+ * PMD_SHIFT  25
  */
 #define PPC44x_TLB_SIZE		PPC44x_TLB_16K
-#define PPC44x_PGD_OFF_SH	13 /*(32 - PMD_SHIFT + 2)*/
-#define PPC44x_PGD_OFF_M1	19 /*(PMD_SHIFT - 2)*/
+#define PPC44x_PGD_OFF_SH	9  /*(32 - PMD_SHIFT + 2)*/
+#define PPC44x_PGD_OFF_M1	23 /*(PMD_SHIFT - 2)*/
 #define PPC44x_PTE_ADD_SH	21 /*32 - PMD_SHIFT + PTE_SHIFT + 3*/
-#define PPC44x_PTE_ADD_M1	22 /*32 - 3 - PTE_SHIFT*/
+#define PPC44x_PTE_ADD_M1	18 /*32 - 3 - PTE_SHIFT*/
 #define PPC44x_RPN_M2		17 /*31 - PAGE_SHIFT*/
 #elif (PAGE_SHIFT == 16)
 /*
  * PAGE_SIZE  64K
  * PAGE_SHIFT 16
- * PTE_SHIFT   5
- * PMD_SHIFT  21
+ * PTE_SHIFT  13
+ * PMD_SHIFT  29
  */
 #define PPC44x_TLB_SIZE		PPC44x_TLB_64K
-#define PPC44x_PGD_OFF_SH	13 /*(32 - PMD_SHIFT + 2)*/
-#define PPC44x_PGD_OFF_M1	19 /*(PMD_SHIFT - 2)*/
+#define PPC44x_PGD_OFF_SH	5  /*(32 - PMD_SHIFT + 2)*/
+#define PPC44x_PGD_OFF_M1	27 /*(PMD_SHIFT - 2)*/
 #define PPC44x_PTE_ADD_SH	19 /*32 - PMD_SHIFT + PTE_SHIFT + 3*/
-#define PPC44x_PTE_ADD_M1	24 /*32 - 3 - PTE_SHIFT*/
+#define PPC44x_PTE_ADD_M1	16 /*32 - 3 - PTE_SHIFT*/
 #define PPC44x_RPN_M2		15 /*31 - PAGE_SHIFT*/
 #else
 #error "Unsupported PAGE_SIZE"

Re: [PATCH] ppc: Add support for bigger page sizes than 4KB on PPC44x

[Paul Mackerras]

Stefan Roese writes:

> This patch adds support for the PAGE_SIZE set equal to 64 KB on
> PPC32 platforms.

If I'm reading the patch right, with 64k pages you're using 8kB pgd
arrays and 256-byte pte arrays, so you're only using 1/256th of each
PTE page.  In fact each PTE page could map 512MB (assuming 8 byte
ptes), and the pgd array could be a tiny 8-entry thing, which we could
kmalloc.  I think that would use memory more efficiently.

Paul.

Re: [PATCH] ppc: Add support for bigger page sizes than 4KB on PPC44x

[Josh Boyer]

On Tue, 2007-03-13 at 08:00 -0600, Grant Likely wrote:
> On 3/13/07, Christoph Hellwig <hch@lst.de> wrote:
> > On Mon, Mar 12, 2007 at 10:54:26PM +0100, Stefan Roese wrote:
> > >  arch/ppc/Kconfig               |   10 ++++++++
> > >  arch/ppc/kernel/head_44x.S     |   23 ++++++++++---------
> > >  arch/ppc/kernel/misc.S         |   13 ++++++-----
> > >  arch/ppc/kernel/vmlinux.lds.S  |   15 +++++++-----
> > >  include/asm-ppc/page.h         |    6 ++++-
> > >  include/asm-ppc/ppc_page_asm.h |   49 ++++++++++++++++++++++++++++++++++++++++
> >
> > I though we'd given up on arch/ppc by now?
> > Can you try to redo this over the arch/powerpc/ 44x port?
> 
> Not dead yet.  arch/powerpc support is not complete for 4xx cores and
> older bootloaders cannot boot arch/powerpc kernels yet.

Um, yes they can.  You just need to include the dtb in the zImage
wrapper.  I've booted Ebony with Openbios using David's patches and
Bamboo with PIBS using my patches (which are based off David's patches).

What I think you mean is that there is no translation from a bd_t (using
older u-boot for example) to a dt in the wrapper yet.  But that is what
Scott's cuimage patches are for.

josh

Re: [PATCH] ppc: Add support for bigger page sizes than 4KB on PPC44x

[Grant Likely]

On 3/13/07, Christoph Hellwig <hch@lst.de> wrote:
> On Mon, Mar 12, 2007 at 10:54:26PM +0100, Stefan Roese wrote:
> >  arch/ppc/Kconfig               |   10 ++++++++
> >  arch/ppc/kernel/head_44x.S     |   23 ++++++++++---------
> >  arch/ppc/kernel/misc.S         |   13 ++++++-----
> >  arch/ppc/kernel/vmlinux.lds.S  |   15 +++++++-----
> >  include/asm-ppc/page.h         |    6 ++++-
> >  include/asm-ppc/ppc_page_asm.h |   49 ++++++++++++++++++++++++++++++++++++++++
>
> I though we'd given up on arch/ppc by now?
> Can you try to redo this over the arch/powerpc/ 44x port?

Not dead yet.  arch/powerpc support is not complete for 4xx cores and
older bootloaders cannot boot arch/powerpc kernels yet.

g.

-- 
Grant Likely, B.Sc. P.Eng.
Secret Lab Technologies Ltd.
grant.likely@secretlab.ca
(403) 399-0195

Re: [PATCH] ppc: Add support for bigger page sizes than 4KB on PPC44x

[Benjamin Herrenschmidt]

On Tue, 2007-03-13 at 10:37 +0100, Wolfgang Denk wrote:
> Dear Ben,
> 
> in message <1173769934.24454.67.camel@localhost.localdomain> you wrote:
> >
> >  - It's arch/ppc. We shouldn't do new major features on this codebase.
> > You are welcome however in helping moving over 4xx to arch/powerpc :-)
> 
> Will do that as soon as we have an arch/powerpc port for the 440SP /
> 440SPe

Ah great. I was thinking about working on 440GX after I'm back from the
ski. (I have a Taishan board)

We'll need to look into the MSI problem for SPe I suppose too...
hopefully Michael should have a proper MSI core merged upstream soom
(/me crosses fingers).

> >  - Have you tried other page sizes ? How hard would it be to support 16K
> > and what kind of performance numbers do you get with 16K ? It's a better
> 
> 16 kB should work, but was not tested so far. From our RAID oriented
> calculations it's not interesting.

Fair enough. I still think it would be useful for other workloads to
have that choice though :-)

> We tried 256 kB, but kernel linking fails with a "Not enough room for
> program headers" error message. We will investigate this ASAP, but at
> the moment we have other priorities. [Any hints what might  go  wrong
> are welcome.]

256K will not be doable without serious hacking afaik. Anything above
64K. You won't be able to run most userland apps for example as the
sections are generally 64K aligned.

> Agreed, but this obviously depends  on  the  sort  of  application(s)
> you're running.

Definitely.

Cheers,
Ben.

Re: [PATCH] ppc: Add support for bigger page sizes than 4KB on PPC44x

[David Gibson]

On Tue, Mar 13, 2007 at 10:42:29AM +0100, Christoph Hellwig wrote:
> On Mon, Mar 12, 2007 at 10:54:26PM +0100, Stefan Roese wrote:
> >  arch/ppc/Kconfig               |   10 ++++++++
> >  arch/ppc/kernel/head_44x.S     |   23 ++++++++++---------
> >  arch/ppc/kernel/misc.S         |   13 ++++++-----
> >  arch/ppc/kernel/vmlinux.lds.S  |   15 +++++++-----
> >  include/asm-ppc/page.h         |    6 ++++-
> >  include/asm-ppc/ppc_page_asm.h |   49 ++++++++++++++++++++++++++++++++++++++++
> 
> I though we'd given up on arch/ppc by now?
> Can you try to redo this over the arch/powerpc/ 44x port?

The arch/powerpc 44x port is hardly complete yet...

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

Re: [PATCH] ppc: Add support for bigger page sizes than 4KB on PPC44x

[Christoph Hellwig]

On Mon, Mar 12, 2007 at 10:54:26PM +0100, Stefan Roese wrote:
>  arch/ppc/Kconfig               |   10 ++++++++
>  arch/ppc/kernel/head_44x.S     |   23 ++++++++++---------
>  arch/ppc/kernel/misc.S         |   13 ++++++-----
>  arch/ppc/kernel/vmlinux.lds.S  |   15 +++++++-----
>  include/asm-ppc/page.h         |    6 ++++-
>  include/asm-ppc/ppc_page_asm.h |   49 ++++++++++++++++++++++++++++++++++++++++

I though we'd given up on arch/ppc by now?
Can you try to redo this over the arch/powerpc/ 44x port?

Re: [PATCH] ppc: Add support for bigger page sizes than 4KB on PPC44x

[Wolfgang Denk]

Dear Ben,

in message <1173769934.24454.67.camel@localhost.localdomain> you wrote:
>
>  - It's arch/ppc. We shouldn't do new major features on this codebase.
> You are welcome however in helping moving over 4xx to arch/powerpc :-)

Will do that as soon as we have an arch/powerpc port for the 440SP /
440SPe

>  - You only adapted head_4xx.S but the config option is global to all of
> arch/ppc, thus changing it will probably break everything else. Make

You are probably right. We will try to clean this up.

>  - Have you tried other page sizes ? How hard would it be to support 16K
> and what kind of performance numbers do you get with 16K ? It's a better

16 kB should work, but was not tested so far. From our RAID oriented
calculations it's not interesting.

We tried 256 kB, but kernel linking fails with a "Not enough room for
program headers" error message. We will investigate this ASAP, but at
the moment we have other priorities. [Any hints what might  go  wrong
are welcome.]

> compromise for some applications as 64K causes significant bloat of the
> page cache (among other things).

Agreed, but this obviously depends  on  the  sort  of  application(s)
you're running.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH, HRB 165235 Munich, CEO: Wolfgang Denk
Office:  Kirchenstr. 5,       D-82194 Groebenzell,            Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
I'd like to meet the man who invented sex and see what  he's  working
on now.

Re: [PATCH] ppc: Add support for bigger page sizes than 4KB on PPC44x

[Stefan Roese]

On Tuesday 13 March 2007 08:12, Benjamin Herrenschmidt wrote:
> While it's a good idea, there are a couple of issues with the patch:
>
>  - It's arch/ppc. We shouldn't do new major features on this codebase.
> You are welcome however in helping moving over 4xx to arch/powerpc :-)

Yes, sure. But the project this patch was developed for started quite a while 
ago when absolutely no 4xx support was available in arch/powerpc. I'm of 
course aware that we have to move to arch/powerpc in the near future and am 
willing to help here.

>  - You only adapted head_4xx.S but the config option is global to all of
> arch/ppc, thus changing it will probably break everything else. Make
> sure the option is done in such a way that it can't be changed or the
> kernel doesn't build if changed for a processor family where you haven't
> implemented the support

OK, will address this problem later.

>  - Have you tried other page sizes ? How hard would it be to support 16K
> and what kind of performance numbers do you get with 16K ? It's a better
> compromise for some applications as 64K causes significant bloat of the
> page cache (among other things).

No, we haven't tested other page sizes. We chose 64K because it is the stripe 
size of the RAID arrays.

Thanks.

Best regards,
Stefan

Re: [PATCH] ppc: Add support for bigger page sizes than 4KB on PPC44x

[Benjamin Herrenschmidt]

On Mon, 2007-03-12 at 22:54 +0100, Stefan Roese wrote:
> This patch adds support for the PAGE_SIZE set equal to 64 KB on
> PPC32 platforms. Increasing the PAGE_SIZE may improve the performance
> of the system in some cases.
> 
> The RAID processing with the dedicated hardware that offloads the CPU
> is one of such cases. Using the bigger PAGE_SIZE we decrease the number
> of CPU operations required to process the data blocks, decrease the
> control overheads and so improve the overall performance. For example,
> changing the PAGE_SIZE from 4KB to 64KB on a Yucca (440SPe) gives ~300%
> performance benefit of the full-stripe writing in the (7d+1p) RAID-5
> setup.
> 
> Signed-off-by: Yuri Tikhonov <yur@emcraft.com>
> Acked-by: Wolfgang Denk <wd@denx.de>
> Acked-by: Stefan Roese <sr@denx.de>

While it's a good idea, there are a couple of issues with the patch:

 - It's arch/ppc. We shouldn't do new major features on this codebase.
You are welcome however in helping moving over 4xx to arch/powerpc :-)

 - You only adapted head_4xx.S but the config option is global to all of
arch/ppc, thus changing it will probably break everything else. Make
sure the option is done in such a way that it can't be changed or the
kernel doesn't build if changed for a processor family where you haven't
implemented the support

 - Have you tried other page sizes ? How hard would it be to support 16K
and what kind of performance numbers do you get with 16K ? It's a better
compromise for some applications as 64K causes significant bloat of the
page cache (among other things).

Cheers,
Ben.

Re: [PATCH] ppc: Add support for bigger page sizes than 4KB on PPC44x

[David Gibson]

On Tue, Mar 13, 2007 at 03:56:15PM +1100, Paul Mackerras wrote:
> Stefan Roese writes:
> 
> > +config PPC_PAGE_SHIFT
> > +	int "Page size (12=>4KB; 16=>64KB)"
> > +	default "12"
> > +	range 12 16
> > +	help
> > +	  The PAGE_SIZE definition. Increasing the page size may 
> > +	  improve the system performance in some dedicated cases.
> > +
> > +	  If unsure, set it to 12 (4KB).
> 
> [snip]
> 
> > +#else
> > +#error "Unsupported PAGE_SIZE"
> > +#endif
> 
> If you only support 4k and 64k page sizes, why do you let the user
> choose 13, 14 or 15 for PPC_PAGE_SHIFT?  Why not just have a "64k
> pages" boolean option like 64-bit powerpc does?

As far as I can tell from the patch, 14 should work too (16kB pages).
13 or 15 won't, since only power-of-4 pagesizes are supported on 44x.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

Re: [PATCH] ppc: Add support for bigger page sizes than 4KB on PPC44x

[Paul Mackerras]

Stefan Roese writes:

> +config PPC_PAGE_SHIFT
> +	int "Page size (12=>4KB; 16=>64KB)"
> +	default "12"
> +	range 12 16
> +	help
> +	  The PAGE_SIZE definition. Increasing the page size may 
> +	  improve the system performance in some dedicated cases.
> +
> +	  If unsure, set it to 12 (4KB).

[snip]

> +#else
> +#error "Unsupported PAGE_SIZE"
> +#endif

If you only support 4k and 64k page sizes, why do you let the user
choose 13, 14 or 15 for PPC_PAGE_SHIFT?  Why not just have a "64k
pages" boolean option like 64-bit powerpc does?

Paul.

[PATCH] ppc: Add support for bigger page sizes than 4KB on PPC44x

[Stefan Roese]

This patch adds support for the PAGE_SIZE set equal to 64 KB on
PPC32 platforms. Increasing the PAGE_SIZE may improve the performance
of the system in some cases.

The RAID processing with the dedicated hardware that offloads the CPU
is one of such cases. Using the bigger PAGE_SIZE we decrease the number
of CPU operations required to process the data blocks, decrease the
control overheads and so improve the overall performance. For example,
changing the PAGE_SIZE from 4KB to 64KB on a Yucca (440SPe) gives ~300%
performance benefit of the full-stripe writing in the (7d+1p) RAID-5
setup.

Signed-off-by: Yuri Tikhonov <yur@emcraft.com>
Acked-by: Wolfgang Denk <wd@denx.de>
Acked-by: Stefan Roese <sr@denx.de>

---
commit 1222edca342590c856ca05aa5fed1281ba23c8a3
tree 3bea891dcdee9444b1800a00a1e0f33c157b254d
parent be521466feb3bb1cd89de82a2b1d080e9ebd3cb6
author Yuri Tikhonov <yur@emcraft.com> Mon, 12 Mar 2007 22:49:21 +0100
committer Stefan Roese <sr@denx.de> Mon, 12 Mar 2007 22:49:21 +0100

 arch/ppc/Kconfig               |   10 ++++++++
 arch/ppc/kernel/head_44x.S     |   23 ++++++++++---------
 arch/ppc/kernel/misc.S         |   13 ++++++-----
 arch/ppc/kernel/vmlinux.lds.S  |   15 +++++++-----
 include/asm-ppc/page.h         |    6 ++++-
 include/asm-ppc/ppc_page_asm.h |   49 ++++++++++++++++++++++++++++++++++++++++
 6 files changed, 91 insertions(+), 25 deletions(-)

diff --git a/arch/ppc/Kconfig b/arch/ppc/Kconfig
index ccce2a4..c087f5a 100644
--- a/arch/ppc/Kconfig
+++ b/arch/ppc/Kconfig
@@ -1174,6 +1174,16 @@ config SECCOMP
 
 	  If unsure, say Y. Only embedded should say N here.
 
+config PPC_PAGE_SHIFT
+	int "Page size (12=>4KB; 16=>64KB)"
+	default "12"
+	range 12 16
+	help
+	  The PAGE_SIZE definition. Increasing the page size may 
+	  improve the system performance in some dedicated cases.
+
+	  If unsure, set it to 12 (4KB).
+
 endmenu
 
 config ISA_DMA_API
diff --git a/arch/ppc/kernel/head_44x.S b/arch/ppc/kernel/head_44x.S
index 7e44de5..65c6a11 100644
--- a/arch/ppc/kernel/head_44x.S
+++ b/arch/ppc/kernel/head_44x.S
@@ -38,6 +38,7 @@
 #include <asm/thread_info.h>
 #include <asm/ppc_asm.h>
 #include <asm/asm-offsets.h>
+#include <asm/ppc_page_asm.h>
 #include "head_booke.h"
 
 
@@ -183,7 +184,7 @@ skpinv:	addi	r4,r4,1				/* Increment */
 	 */
  	/* pageid fields */
 	lis	r3,UART0_IO_BASE@h
-	ori	r3,r3,PPC44x_TLB_VALID | PPC44x_TLB_4K
+	ori	r3,r3,PPC44x_TLB_VALID | PPC44x_TLB_SIZE
 
 	/* xlat fields */
 	lis	r4,UART0_PHYS_IO_BASE@h		/* RPN depends on SoC */
@@ -363,12 +364,12 @@ interrupt_base:
 4:
 	mtspr   SPRN_MMUCR,r12
 
-	rlwinm  r12, r10, 13, 19, 29    /* Compute pgdir/pmd offset */
+	rlwinm  r12, r10, PPC44x_PGD_OFF_SH, PPC44x_PGD_OFF_M1, 29 /* Compute pgdir/pmd offset */
 	lwzx    r11, r12, r11           /* Get pgd/pmd entry */
 	rlwinm. r12, r11, 0, 0, 20      /* Extract pt base address */
 	beq     2f                      /* Bail if no table */
 
-	rlwimi  r12, r10, 23, 20, 28    /* Compute pte address */
+	rlwimi  r12, r10, PPC44x_PTE_ADD_SH, PPC44x_PTE_ADD_M1, 28    /* Compute pte address */
 	lwz     r11, 4(r12)             /* Get pte entry */
 
 	andi.	r13, r11, _PAGE_RW	/* Is it writeable? */
@@ -502,12 +503,12 @@ interrupt_base:
 4:
 	mtspr	SPRN_MMUCR,r12
 
-	rlwinm 	r12, r10, 13, 19, 29	/* Compute pgdir/pmd offset */
+	rlwinm 	r12, r10, PPC44x_PGD_OFF_SH, PPC44x_PGD_OFF_M1, 29 /* Compute pgdir/pmd offset */
 	lwzx	r11, r12, r11		/* Get pgd/pmd entry */
 	rlwinm.	r12, r11, 0, 0, 20	/* Extract pt base address */
 	beq	2f			/* Bail if no table */
 
-	rlwimi	r12, r10, 23, 20, 28	/* Compute pte address */
+	rlwimi	r12, r10, PPC44x_PTE_ADD_SH, PPC44x_PTE_ADD_M1, 28	/* Compute pte address */
 	lwz	r11, 4(r12)		/* Get pte entry */
 	andi.	r13, r11, _PAGE_PRESENT	/* Is the page present? */
 	beq	2f			/* Bail if not present */
@@ -572,12 +573,12 @@ interrupt_base:
 4:
 	mtspr	SPRN_MMUCR,r12
 
-	rlwinm	r12, r10, 13, 19, 29	/* Compute pgdir/pmd offset */
+	rlwinm	r12, r10, PPC44x_PGD_OFF_SH, PPC44x_PGD_OFF_M1, 29 /* Compute pgdir/pmd offset */
 	lwzx	r11, r12, r11		/* Get pgd/pmd entry */
 	rlwinm.	r12, r11, 0, 0, 20	/* Extract pt base address */
 	beq	2f			/* Bail if no table */
 
-	rlwimi	r12, r10, 23, 20, 28	/* Compute pte address */
+	rlwimi	r12, r10, PPC44x_PTE_ADD_SH, PPC44x_PTE_ADD_M1, 28	/* Compute pte address */
 	lwz	r11, 4(r12)		/* Get pte entry */
 	andi.	r13, r11, _PAGE_PRESENT	/* Is the page present? */
 	beq	2f			/* Bail if not present */
@@ -656,14 +657,14 @@ finish_tlb_load:
 
 	lwz	r11, 0(r12)			/* Get MS word of PTE */
 	lwz	r12, 4(r12)			/* Get LS word of PTE */
-	rlwimi	r11, r12, 0, 0 , 19		/* Insert RPN */
+	rlwimi	r11, r12, 0, 0 , PPC44x_RPN_M2	/* Insert RPN */
 	tlbwe	r11, r13, PPC44x_TLB_XLAT	/* Write XLAT */
 
 	/*
 	 * Create PAGEID. This is the faulting address,
 	 * page size, and valid flag.
 	 */
-	li	r11, PPC44x_TLB_VALID | PPC44x_TLB_4K
+	li	r11, PPC44x_TLB_VALID | PPC44x_TLB_SIZE
 	rlwimi	r10, r11, 0, 20, 31		/* Insert valid and page size */
 	tlbwe	r10, r13, PPC44x_TLB_PAGEID	/* Write PAGEID */
 
@@ -740,12 +741,12 @@ _GLOBAL(set_context)
  * goes at the beginning of the data segment, which is page-aligned.
  */
 	.data
-	.align	12
+	.align	PAGE_SHIFT
 	.globl	sdata
 sdata:
 	.globl	empty_zero_page
 empty_zero_page:
-	.space	4096
+	.space	PAGE_SIZE
 
 /*
  * To support >32-bit physical addresses, we use an 8KB pgdir.
diff --git a/arch/ppc/kernel/misc.S b/arch/ppc/kernel/misc.S
index d319f9b..38a819b 100644
--- a/arch/ppc/kernel/misc.S
+++ b/arch/ppc/kernel/misc.S
@@ -17,6 +17,7 @@
 #include <asm/errno.h>
 #include <asm/processor.h>
 #include <asm/page.h>
+#include <asm/ppc_page_asm.h>
 #include <asm/cache.h>
 #include <asm/cputable.h>
 #include <asm/mmu.h>
@@ -473,8 +474,8 @@ _GLOBAL(__flush_dcache_icache)
 BEGIN_FTR_SECTION
 	blr					/* for 601, do nothing */
 END_FTR_SECTION_IFCLR(CPU_FTR_SPLIT_ID_CACHE)
-	rlwinm	r3,r3,0,0,19			/* Get page base address */
-	li	r4,4096/L1_CACHE_BYTES	/* Number of lines in a page */
+	rlwinm	r3,r3,0,0,PPC44x_RPN_M2		/* Get page base address */
+	li	r4,PAGE_SIZE/L1_CACHE_BYTES	/* Number of lines in a page */
 	mtctr	r4
 	mr	r6,r3
 0:	dcbst	0,r3				/* Write line to ram */
@@ -505,8 +506,8 @@ END_FTR_SECTION_IFCLR(CPU_FTR_SPLIT_ID_CACHE)
 	rlwinm	r0,r10,0,28,26			/* clear DR */
 	mtmsr	r0
 	isync
-	rlwinm	r3,r3,0,0,19			/* Get page base address */
-	li	r4,4096/L1_CACHE_BYTES	/* Number of lines in a page */
+	rlwinm	r3,r3,0,0,PPC44x_RPN_M2		/* Get page base address */
+	li	r4,PAGE_SIZE/L1_CACHE_BYTES	/* Number of lines in a page */
 	mtctr	r4
 	mr	r6,r3
 0:	dcbst	0,r3				/* Write line to ram */
@@ -530,7 +531,7 @@ END_FTR_SECTION_IFCLR(CPU_FTR_SPLIT_ID_CACHE)
  * void clear_pages(void *page, int order) ;
  */
 _GLOBAL(clear_pages)
-	li	r0,4096/L1_CACHE_BYTES
+	li	r0,PAGE_SIZE/L1_CACHE_BYTES
 	slw	r0,r0,r4
 	mtctr	r0
 #ifdef CONFIG_8xx
@@ -588,7 +589,7 @@ _GLOBAL(copy_page)
 	dcbt	r5,r4
 	li	r11,L1_CACHE_BYTES+4
 #endif /* MAX_COPY_PREFETCH */
-	li	r0,4096/L1_CACHE_BYTES - MAX_COPY_PREFETCH
+	li	r0,PAGE_SIZE/L1_CACHE_BYTES - MAX_COPY_PREFETCH
 	crclr	4*cr0+eq
 2:
 	mtctr	r0
diff --git a/arch/ppc/kernel/vmlinux.lds.S b/arch/ppc/kernel/vmlinux.lds.S
index a062556..73cd0b3 100644
--- a/arch/ppc/kernel/vmlinux.lds.S
+++ b/arch/ppc/kernel/vmlinux.lds.S
@@ -1,4 +1,5 @@
 #include <asm-generic/vmlinux.lds.h>
+#include <asm/page.h>
 
 OUTPUT_ARCH(powerpc:common)
 jiffies = jiffies_64 + 4;
@@ -64,7 +65,7 @@ SECTIONS
 	}
 
   /* Read-write section, merged into data segment: */
-  . = ALIGN(4096);
+  . = ALIGN(PAGE_SIZE);
   .data    :
   {
     *(.data)
@@ -76,10 +77,10 @@ SECTIONS
     CONSTRUCTORS
   }
 
-  . = ALIGN(4096);
+  . = ALIGN(PAGE_SIZE);
   __nosave_begin = .;
   .data_nosave : { *(.data.nosave) }
-  . = ALIGN(4096);
+  . = ALIGN(PAGE_SIZE);
   __nosave_end = .;
 
   . = ALIGN(32);
@@ -91,7 +92,7 @@ SECTIONS
   . = ALIGN(8192);
   .data.init_task : { *(.data.init_task) }
 
-  . = ALIGN(4096);
+  . = ALIGN(PAGE_SIZE);
   __init_begin = .;
   .init.text : {
 	_sinittext = .;
@@ -136,16 +137,16 @@ SECTIONS
   __per_cpu_end = .;
 
 #ifdef CONFIG_BLK_DEV_INITRD
-  . = ALIGN(4096);
+  . = ALIGN(PAGE_SIZE);
   __initramfs_start = .;
   .init.ramfs : { *(.init.ramfs) }
   __initramfs_end = .;
 #endif
 
-  . = ALIGN(4096);
+  . = ALIGN(PAGE_SIZE);
   __init_end = .;
 
-  . = ALIGN(4096);
+  . = ALIGN(PAGE_SIZE);
   _sextratext = .;
   _eextratext = .;
 
diff --git a/include/asm-ppc/page.h b/include/asm-ppc/page.h
index fe95c82..61698ac 100644
--- a/include/asm-ppc/page.h
+++ b/include/asm-ppc/page.h
@@ -4,7 +4,7 @@
 #include <asm/asm-compat.h>
 
 /* PAGE_SHIFT determines the page size */
-#define PAGE_SHIFT	12
+#define PAGE_SHIFT	CONFIG_PPC_PAGE_SHIFT
 #define PAGE_SIZE	(ASM_CONST(1) << PAGE_SHIFT)
 
 /*
@@ -28,7 +28,11 @@
  */
 #ifdef CONFIG_PTE_64BIT
 typedef unsigned long long pte_basic_t;
+#if (PAGE_SHIFT == 16) && defined(CONFIG_PPC32)
+#define PTE_SHIFT	(PAGE_SHIFT - 11)	/* 256 ptes per table*/
+#else
 #define PTE_SHIFT	(PAGE_SHIFT - 3)	/* 512 ptes per page */
+#endif
 #define PTE_FMT		"%16Lx"
 #else
 typedef unsigned long pte_basic_t;
diff --git a/include/asm-ppc/ppc_page_asm.h b/include/asm-ppc/ppc_page_asm.h
new file mode 100644
index 0000000..518ee27
--- /dev/null
+++ b/include/asm-ppc/ppc_page_asm.h
@@ -0,0 +1,49 @@
+/*
+ * include/asm/ppc_page_asm.h
+ *
+ * 2007 (C) DENX Software Engineering.
+ *
+ *  This file is licensed under the terms of the GNU General Public License
+ * version 2.  This program is licensed "as is" without any warranty of
+ * any kind, whether express or implied.
+ *
+ *  The page definitions used in the asm files ppc_44x.S and misc.S. 
+ * PAGE_SIZE = 4K and 64K are only supported on the PPC44x.
+ *
+ */
+#ifndef PPC_PAGE_ASM_H
+#define PPC_PAGE_ASM_H
+
+#include <asm/page.h>
+
+#if (PAGE_SHIFT == 12)
+/* 
+ * PAGE_SIZE  4K 
+ * PAGE_SHIFT 12
+ * PTE_SHIFT   9
+ * PMD_SHIFT  21
+ */
+#define PPC44x_TLB_SIZE		PPC44x_TLB_4K
+#define PPC44x_PGD_OFF_SH	13 /*(32 - PMD_SHIFT + 2)*/
+#define PPC44x_PGD_OFF_M1	19 /*(PMD_SHIFT - 2)*/
+#define PPC44x_PTE_ADD_SH	23 /*32 - PMD_SHIFT + PTE_SHIFT + 3*/
+#define PPC44x_PTE_ADD_M1	20 /*32 - 3 - PTE_SHIFT*/
+#define PPC44x_RPN_M2		19 /*31 - PAGE_SHIFT*/
+#elif (PAGE_SHIFT == 16)
+/*
+ * PAGE_SIZE  64K
+ * PAGE_SHIFT 16
+ * PTE_SHIFT   5
+ * PMD_SHIFT  21
+ */
+#define PPC44x_TLB_SIZE		PPC44x_TLB_64K
+#define PPC44x_PGD_OFF_SH	13 /*(32 - PMD_SHIFT + 2)*/
+#define PPC44x_PGD_OFF_M1	19 /*(PMD_SHIFT - 2)*/
+#define PPC44x_PTE_ADD_SH	19 /*32 - PMD_SHIFT + PTE_SHIFT + 3*/
+#define PPC44x_PTE_ADD_M1	24 /*32 - 3 - PTE_SHIFT*/
+#define PPC44x_RPN_M2		15 /*31 - PAGE_SHIFT*/
+#else
+#error "Unsupported PAGE_SIZE"
+#endif
+
+#endif