Re: Struct page proposal

[Matthew Wilcox <willy@infradead.org>]

On Mon, Sep 27, 2021 at 02:16:53PM -0400, Kent Overstreet wrote:
> On Mon, Sep 27, 2021 at 07:12:19PM +0100, Matthew Wilcox wrote:
> > On Mon, Sep 27, 2021 at 02:09:49PM -0400, Kent Overstreet wrote:
> > > On Mon, Sep 27, 2021 at 07:05:26PM +0100, Matthew Wilcox wrote:
> > > > On Mon, Sep 27, 2021 at 07:48:15PM +0200, Vlastimil Babka wrote:
> > > > > On 9/23/21 03:21, Kent Overstreet wrote:
> > > > > > So if we have this:
> > > > > > 
> > > > > > struct page {
> > > > > > 	unsigned long	allocator;
> > > > > > 	unsigned long	allocatee;
> > > > > > };
> > > > > > 
> > > > > > The allocator field would be used for either a pointer to slab/slub's state, if
> > > > > > it's a slab page, or if it's a buddy allocator page it'd encode the order of the
> > > > > > allocation - like compound order today, and probably whether or not the
> > > > > > (compound group of) pages is free.
> > > > > 
> > > > > The "free page in buddy allocator" case will be interesting to implement.
> > > > > What the buddy allocator uses today is:
> > > > > 
> > > > > - PageBuddy - determine if page is free; a page_type (part of mapcount
> > > > > field) today, could be a bit in "allocator" field that would have to be 0 in
> > > > > all other "page is allocated" contexts.
> > > > > - nid/zid - to prevent merging accross node/zone boundaries, now part of
> > > > > page flags
> > > > > - buddy order
> > > > > - a list_head (reusing the "lru") to hold the struct page on the appropriate
> > > > > free list, which has to be double-linked so page can be taken from the
> > > > > middle of the list instantly
> > > > > 
> > > > > Won't be easy to cram all that into two unsigned long's, or even a single
> > > > > one. We should avoid storing anything in the free page itself. Allocating
> > > > > some external structures to track free pages is going to have funny
> > > > > bootstrap problems. Probably a major redesign would be needed...
> > > > 
> > > > Wait, why do we want to avoid using the memory that we're allocating?
> > > 
> > > The issue is where to stick the state for free pages. If that doesn't fit in two
> > > ulongs, then we'd need a separate allocation, which means slab needs to be up
> > > and running before free pages are initialized.
> > 
> > But the thing we're allocating is at least PAGE_SIZE bytes in size.
> > Why is "We should avoid storing anything in the free page itself" true?
> 
> Good point!
> 
> Highmem and dax do complicate things though - would they make it too much of a
> hassle? You want to get rid of struct page for dax (what's the right term for
> that kind of memory?), but we're not there yet, right?

DAX is used for persistent memory, often abbreviated to pmem.

"Getting there" involves rooting out struct page from all kinds of data
structures.  sg lists are the obvious place to start so that we can do
I/O to memory that's not backed by a struct page.  Get that working and
the rent-a-VM companies will love you.  Right now, we either pay the 1.6%
tax twice (once for the struct pages in the host, and once in the guest),
or we have horrendous hacks to create struct pages on the fly so the
host can do I/O to the guest's memory.