Re: pageless memory & zsmalloc

[Kent Overstreet <kent.overstreet@gmail.com>]

On Tue, Oct 05, 2021 at 10:28:23PM +0100, Matthew Wilcox wrote:
> I'm still not convinced of the need for allocator + allocatee words.
> But I don't think we need to resolve that point of disagreement in
> order to make progress towards the things we do agree on.

It's not so much that I disagree, I just don't see how your one-word idea is
possible and you haven't outlined it :) Can you sketch it out for us?

> > And in my mind, compound_head is allocator state, not allocatee state, and it's
> > always been using for getting to the head page, which... this is not, so using
> > it this way, as slick as it is... eh, not sure that's quite what we want to do.
> 
> In my mind, in the future where all memory descriptors are dynamically
> allocated, when we allocate an order-3 page, we initialise the
> 'allocatee state' of each of the 8 consecutive pages to point to the
> memory descriptor that we just allocated.  We probably also encode the
> type of the memory descriptor in the allocatee state (I think we've
> probably got about 5 bits for that).

Yep, I've been envisioning using the low bits of the pointer to the allocatee
state as a type tag. Where does compound_order go, though?

> The lock state has to be in the memory descriptor.  It can't be in the
> individual page.  So I think all memory descriptors needs to start with
> a flags word.  Memory compaction could refrain from locking pages if
> the memory descriptor is of the wrong type, of course.

Memory compaction inherently has to switch on the allocatee type, so if the page
is of a type that can't be migrated, it would make sense to just not bother with
locking it. On the other hand, the type isn't stable without first locking the
page.

There's another synchronization thing we have to work out: with the lock behind
a pointer, we can race with the page, and the allocatee state, being freed.
Which implies that we're always going to have to RCU-free allocatee state, and
that after chasing the pointer and taking the lock we'll have to check that the
page is still a member of that allocatee state.

This race is something that we'll have to handle every place we deref the
allocatee state pointer - and in many cases we won't want to lock the allocatee
state, so page->ref will also have to be part of this common page allocatee
state.

> Eventually, I think lock_page() disappears in favour of folio_lock().
> That doesn't quite work for compaction, but maybe we could do something
> like this ...

Question is, do other types of pages besides just folios need lock_page() and
get_page()? If so, maybe folio_lock() doesn't make sense at all and we should
just have functions that operate on your (expanded, to include a refcount)
pgflags_t.