ep93xx: status/future of the sparsemem support

ep93xx: status/future of the sparsemem support

[Petr Štetiar]

Hi all,

I would like to ask about the status or plans for the sparsemem support on the
ep93xx. I'm mainly talking about the ts72xx here as I don't have any other
boards, but it seems, that sparsemem is the must for the boards with more then
one memory bank. Or it's not true?

For example, right now, I'm trying to get the recent kernel working on the
ts-7300 with the 128MB RAM (two 64MB memory chips, 16 banks, 8MB each,
first 8 at 0x0000000-0x7000000, second one at 0xe0000000-0e7000000) and as you
might guess, it's unusable with flatmem (and it looks like the discontig mem
is now gone for ARM if I'm not mistaken):

	[..snip..]

	Ignoring RAM at e0000000-e07fffff (vmalloc region overlap).
	Ignoring RAM at e1000000-e17fffff (vmalloc region overlap).
	Ignoring RAM at e2000000-e27fffff (vmalloc region overlap).
	Ignoring RAM at e3000000-e37fffff (vmalloc region overlap).
	Ignoring RAM at e4000000-e47fffff (vmalloc region overlap).
	Ignoring RAM at e5000000-e57fffff (vmalloc region overlap).
	Ignoring RAM at e6000000-e67fffff (vmalloc region overlap).
	Ignoring RAM at e7000000-e77fffff (vmalloc region overlap).

	[..snip..]

	Memory: 8MB 8MB 8MB 8MB 8MB 8MB 8MB 8MB 8MB 8MB 8MB 8MB 8MB 8MB 8MB 8MB = 128MB total
	Memory: 62076k/62076k available, 3460k reserved, 0K highmem
                ^
                |__ note, just lower 64MB available

	Virtual kernel memory layout:
	    vector  : 0xffff0000 - 0xffff1000   (   4 kB)
	    fixmap  : 0xfff00000 - 0xfffe0000   ( 896 kB)
	    DMA     : 0xffc00000 - 0xffe00000   (   2 MB)
	    vmalloc : 0xc8000000 - 0xfe800000   ( 872 MB)
	    lowmem  : 0xc0000000 - 0xc7800000   ( 120 MB)

If I apply the sparsemem patch as suggested here[1] it seems to work (well it
boots):

	Memory: 8MB 8MB 8MB 8MB 8MB 8MB 8MB 8MB 8MB 8MB 8MB 8MB 8MB 8MB 8MB 8MB = 128MB total
	Memory: 127052k/127052k available, 4020k reserved, 0K highmem
	Virtual kernel memory layout:
	    vector  : 0xffff0000 - 0xffff1000   (   4 kB)
	    fixmap  : 0xfff00000 - 0xfffe0000   ( 896 kB)
	    DMA     : 0xffc00000 - 0xffe00000   (   2 MB)
	    vmalloc : 0xd0000000 - 0xfe800000   ( 744 MB)
	    lowmem  : 0xc0000000 - 0xcf800000   ( 248 MB)

The question is, what's the problem with the sparsemem patch[1] for ep93xx,
that it's not yet in the mainline? Nobody needed it yet? :-) If it's that the
case, I could submit it, no problem. I don't follow this mailing list and
development on the ep93xx regularly, so I might have missed some important
detail. Thank you for any pointers.

1. http://marc.info/?l=linux-arm&m=122754446724900

-- ynezz

ep93xx: status/future of the sparsemem support

[Russell King - ARM Linux]

On Sun, May 29, 2011 at 03:06:44PM +0200, Petr ?tetiar wrote:
> I would like to ask about the status or plans for the sparsemem support on the
> ep93xx. I'm mainly talking about the ts72xx here as I don't have any other
> boards, but it seems, that sparsemem is the must for the boards with more then
> one memory bank. Or it's not true?

Have you tried enabling highmem?

One of the problems with this kind of physical memory layout is that it
doesn't allow mathematical offsets between physical and virtual address
spaces.  Consider 0x00000000 phys -> 0xc0000000 virt, so 0xe0000000
phys -> 0xa0000000 virt which will be in userspace!

So we either have to play stupid games with the translation functions and
introduce non-linear translations (as per my message which you pointed to),
or go the highmem route which avoids the issue entirely.

ep93xx: status/future of the sparsemem support

[Petr Štetiar]

Russell King - ARM Linux <linux@arm.linux.org.uk> [2011-05-29 23:35:57]:

> Have you tried enabling highmem?

Yes, it was my first attempt, but it didn't booted, here's the log:

	Memory policy: ECC disabled, Data cache writeback
	bootmem::init_bootmem_core nid=0 start=0 map=77fa end=7800 mapsize=f00
	bootmem::mark_bootmem_node nid=0 start=0 end=800 reserve=0 flags=0
	bootmem::__free nid=0 start=0 end=800
	bootmem::mark_bootmem_node nid=0 start=1000 end=1800 reserve=0 flags=0
	bootmem::__free nid=0 start=1000 end=1800
	bootmem::mark_bootmem_node nid=0 start=2000 end=2800 reserve=0 flags=0
	bootmem::__free nid=0 start=2000 end=2800
	bootmem::mark_bootmem_node nid=0 start=3000 end=3800 reserve=0 flags=0
	bootmem::__free nid=0 start=3000 end=3800
	bootmem::mark_bootmem_node nid=0 start=4000 end=4800 reserve=0 flags=0
	bootmem::__free nid=0 start=4000 end=4800
	bootmem::mark_bootmem_node nid=0 start=5000 end=5800 reserve=0 flags=0
	bootmem::__free nid=0 start=5000 end=5800
	bootmem::mark_bootmem_node nid=0 start=6000 end=6800 reserve=0 flags=0
	bootmem::__free nid=0 start=6000 end=6800
	bootmem::mark_bootmem_node nid=0 start=7000 end=7800 reserve=0 flags=0
	bootmem::__free nid=0 start=7000 end=7800
	bootmem::mark_bootmem_node nid=0 start=4 end=2bd reserve=1 flags=0
	bootmem::__reserve nid=0 start=4 end=2bd flags=0
	bootmem::mark_bootmem_node nid=0 start=77fa end=7800 reserve=1 flags=0
	bootmem::__reserve nid=0 start=77fa end=7800 flags=0
	On node 0 totalpages: 32768
	bootmem::alloc_bootmem_core nid=0 size=1cf0000 [7408 pages] align=20 goal=3fffffff limit=0
	bootmem::alloc_bootmem_core nid=0 size=1cf0000 [7408 pages] align=20 goal=0 limit=0
	free_area_init_node: node 0, pgdat c029ea1c, node_mem_map 00000000
	  Normal zone: 240 pages used for memmap
	  Normal zone: 0 pages reserved
	  Normal zone: 16144 pages, LIFO batch:3
	bootmem::alloc_bootmem_core nid=0 size=c [1 pages] align=20 goal=3fffffff limit=0
	bootmem::__reserve nid=0 start=0 end=1 flags=1
	bootmem::alloc_bootmem_core nid=0 size=400 [1 pages] align=20 goal=3fffffff limit=0
	bootmem::__reserve nid=0 start=1 end=1 flags=1

Tracked the hang today down to set_page_zone() and added panic there also:

	set_page_links() calling set_page_zone: page 0x  (null) zone: 0x0 node: 0x0 pfn: 0x0
	Kernel panic - not syncing: ehm
	Backtrace: 
	[<c0025e78>] (dump_backtrace+0x0/0x10c) from [<c01fbcac>] (dump_stack+0x18/0x1c)
	 r6:00000000 r5:00000000 r4:c029f760 r3:00000000
	[<c01fbc94>] (dump_stack+0x0/0x1c) from [<c01fbd18>] (panic+0x68/0x198)
	[<c01fbcb0>] (panic+0x0/0x198) from [<c001cc38>] (memmap_init_zone+0xac/0xe0)
	 r3:00000000 r2:c028d898 r1:00000000 r0:c025d640
	 r7:00000000
	[<c001cb8c>] (memmap_init_zone+0x0/0xe0) from [<c000f6f8>] (free_area_init_node+0x348/0x400)
	 r7:00000000 r6:00000000 r5:00000000 r4:c029ea1c
	[<c000f3b0>] (free_area_init_node+0x0/0x400) from [<c000a7b8>] (bootmem_init+0x224/0x260)
	[<c000a594>] (bootmem_init+0x0/0x260) from [<c000bcf4>] (paging_init+0x710/0x7a4)
	[<c000b5e4>] (paging_init+0x0/0x7a4) from [<c0009c20>] (setup_arch+0x3a8/0x6ec)
	[<c0009878>] (setup_arch+0x0/0x6ec) from [<c000874c>] (start_kernel+0x78/0x2b8)
	[<c00086d4>] (start_kernel+0x0/0x2b8) from [<0000803c>] (0x803c)
	 r7:c028adec r6:c001e0d0 r5:c0288038 r4:c0007175

Hope it helps. 

> One of the problems with this kind of physical memory layout is that it
> doesn't allow mathematical offsets between physical and virtual address
> spaces.  Consider 0x00000000 phys -> 0xc0000000 virt, so 0xe0000000
> phys -> 0xa0000000 virt which will be in userspace!

Yep, I've seen that overlap also, so that's why I've tried to enable the
highmem.

> So we either have to play stupid games with the translation functions and
> introduce non-linear translations (as per my message which you pointed to),
> or go the highmem route which avoids the issue entirely.

I agree, but according to the log above I've assumed, that simply flatmem will
use just the first 64MB (as it didn't touched the second memory bank), so I've
moved to sparsemem.

-- ynezz

ep93xx: status/future of the sparsemem support

[H Hartley Sweeten]

On Sunday, May 29, 2011 6:07 AM, Petr ?tetiar wrote:
> 
> Hi all,
> 
> I would like to ask about the status or plans for the sparsemem support on the
> ep93xx. I'm mainly talking about the ts72xx here as I don't have any other
> boards, but it seems, that sparsemem is the must for the boards with more then
> one memory bank. Or it's not true?
> 
> For example, right now, I'm trying to get the recent kernel working on the
> ts-7300 with the 128MB RAM (two 64MB memory chips, 16 banks, 8MB each,
> first 8 at 0x0000000-0x7000000, second one at 0xe0000000-0e7000000) and as you
> might guess, it's unusable with flatmem (and it looks like the discontig mem
> is now gone for ARM if I'm not mistaken):

[...]

> The question is, what's the problem with the sparsemem patch[1] for ep93xx,
> that it's not yet in the mainline? Nobody needed it yet? :-) If it's that the
> case, I could submit it, no problem. I don't follow this mailing list and
> development on the ep93xx regularly, so I might have missed some important
> detail. Thank you for any pointers.
> 
> 1. http://marc.info/?l=linux-arm&m=122754446724900

I don't think that patch covers all the possible configurations for the ep93xx.

The ep39xx has four chip-selects that decode to these memory addresses:

nSDCS3  0x00000000-0x0fffffff  Boot option ASDO = 1
nSDCS3  0xf0000000-0xffffffff  Boot option ASDO = 0
nSDCS2  0xe0000000-0xefffffff
nSDCS1  0xd0000000-0xdfffffff
nSDCS0  0xc0000000-0xcfffffff

Then because of the way the ep93xx sdram controller uses the address bus, which
is shared with the external memory (flash, etc.), the sdram memory gets broken
up into banks of 2MB, 4MB, 8MB, 16MB, 32MB, or 128MB depending on the type of
sdram used.  To make matters even more confusing, the sdram can operate with
either a 16-bit or 32-bit wide data bus.  See [2] for the details.

The patch above is optimized for the nSDCS3 (ASDO=1) and nSDCS2 configuration
with 8MB banks.  In order for the ep93xx to support sparsemem I think it needs
a bit of work to handle the other configurations.

Regards,
Hartley

2. http://arm.cirrus.com/files/HOWTO/EP93xx%20SDRAM%20Address%20Ranges.pdf

ep93xx: status/future of the sparsemem support

[Petr Štetiar]

H Hartley Sweeten <hartleys@visionengravers.com> [2011-05-31 13:23:32]:

> Then because of the way the ep93xx sdram controller uses the address bus,
> which is shared with the external memory (flash, etc.), the sdram memory
> gets broken up into banks of 2MB, 4MB, 8MB, 16MB, 32MB, or 128MB depending
> on the type of sdram used.  To make matters even more confusing, the sdram
> can operate with either a 16-bit or 32-bit wide data bus.  See [2] for the
> details.

D'oh.

> The patch above is optimized for the nSDCS3 (ASDO=1) and nSDCS2 configuration
> with 8MB banks.  In order for the ep93xx to support sparsemem I think it needs
> a bit of work to handle the other configurations.

Yes, I completly agree, that we need some more generic solution. I also
wonder, how much is the current "EP93xx first SDRAM bank selection" hardcoded
kernel config option multimachine friendly.

I don't know the internals that much, so it might sound completly silly, but I
think, that we might need to resurrect the Lennert's runtime PHYS_OFFSET
detection (or something similar to that idea) to detect the first memory bank,
then read the memory configuration passed via atags from there and setup the
sparsemem (or some kind of flexible virt<->phys implementation) accordingly?

I don't know the flatmem+highmem that much either, so I'm quite unsure if it
should work on such fragmented memory (ts7300+128MB in my use case), or if the
oops I'm just hitting now needs fixing and then it should start working. I've
looked into that code, but it's quite complex for me :P

What's your idea of handling this utter mess? What does "bit of work" mean
from your POV?

-- ynezz