Re: [PATCH v2 0/6] ARM: Add support for large kernel page (from 8K to 64K)

[Gregory CLEMENT <gregory.clement@bootlin.com>]

Hello Russell,

> Hi Gregory,
>
> You're on your own with this one; I've no motivation to re-understand
> the ARM page table code now that 32-bit ARM is basically unsupported
> now.

Understood.

>
> I'll point out some of the things you got wrong below though.

However thanks for your pointer.

>
> On Thu, Jun 11, 2020 at 03:49:08PM +0200, Gregory CLEMENT wrote:
>> Hello,
>> 
>> On ARM based NAS it is possible to have storage volume larger than
>> 16TB, especially with the use of LVM. However, on 32-bit architectures,
>> the page cache index is stored on 32 bits, which means that given a
>> page size of 4 KB, we can only address volumes of up to 16 TB.
>> 
>> Therefore, one option to use such large volumes and filesystems on 32
>> bits architecture is to increase the page size.
>> 
>> This series allows to support 8K, 16K, 32K and 64K kernel pages. On
>> ARM the size of the page can be either 4K or 64K, so for the other
>> size a "software emulation" is used, here Linux thinks it is using
>> pages of 8 KB, 16 KB or 32 KB, while underneath the MMU still uses 4
>> KB pages.
>> 
>> For ARM there is already a difference between the kernel page and the
>> hardware page in the way they are managed. In the same 4K space the
>> Linux kernel deals with 2 PTE tables at the beginning, while the
>> hardware deals with 2 other hardware PTE tables.
>
> This is incorrect.  The kernel page size and the hardware page size
> match today - both are 4k.  What your'e talking about here is the
> PTE table size.
>
> The kernel requires that each PTE table is contained within one
> struct page.  Since one hardware PTE table is 256 entries, it
> occupies 1024 bytes, so a quarter of a page.  So, to have a single
> 4k page per PTE table would waste quite a bit of space.
>
> Now, the hardware PTE tables do not lend themselves to the kernel's
> usage: the kernel wants additional bits to track the state of each
> page in the page tables.  Hence, we need to shadow every PTE entry.
> This also provides us independence of the underlying hardware PTE
> entry format, which varies between ARM architecture versions.
>
> So, we end up with a single 4k page containing two consecutive
> hardware PTE tables, followed by two Linux PTE tables for the kernels
> benefit.
>

It was what I understood, but I seemed I didn't formulate it accurately.

> If you increase the page size, then you need to increase the number
> of tables in a page, or suffer a huge amount of wasted memory taken
> for the page tables - going to an 8k page size means that the upper
> 4k of each page will not be used.  Going to 16k means the upper 12k
> won't be used.  And so on - as your software page size increases,
> the amount of memory wasted for each PTE table will increase
> unless you also increase the number of hardware 1st level entries
> pointing to each PTE page.  With 64k pages, 60k of each PTE page
> will remain unused.

Unfortunately I was aware of it. But I thought that it was an acceptable
drawback to be able to address large volume on a 32 bits ARM
system. Actually it is already the case on some product.

> That isn't very efficient use of memory.

Indeed however on a 3GB system, in the worst case we need 786432 pages
of 4K to map the memory. These pages can be mapped by 1536 block of 512
entries. So when the 64K pages are emulated we loose 92MB (around 3% of
the memory). So it is not negligible but given the use case I seems
acceptable.

Of course, it didn't prevent to try to do better.

Gregory
>
> -- 
> RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
> FTTC for 0.8m (est. 1762m) line in suburbia: sync at 13.1Mbps down 503kbps up

-- 
Gregory Clement, Bootlin
Embedded Linux and Kernel engineering
http://bootlin.com

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel