archive mirror
 help / color / mirror / Atom feed
From: "Matthew Wilcox (Oracle)" <>
Cc: "Matthew Wilcox (Oracle)" <>,
Subject: [RFC PATCH 00/11] Page folios
Date: Tue,  8 Dec 2020 19:46:42 +0000	[thread overview]
Message-ID: <> (raw)

One of the great things about compound pages is that when you try to
do various operations on a tail page, it redirects to the head page and
everything Just Works.  One of the awful things is how much we pay for
that simplicity.  Here's an example, end_page_writeback():

        if (PageReclaim(page)) {
        if (!test_clear_page_writeback(page))

        wake_up_page(page, PG_writeback);

That all looks very straightforward, but if you dive into the disassembly,
you see that there are four calls to compound_head() in this function
(PageReclaim(), ClearPageReclaim(), get_page() and put_page()).  It's
all for nothing, because if anyone does call this routine with a tail
page, wake_up_page() will VM_BUG_ON_PGFLAGS(PageTail(page), page).

I'm not really a CPU person, but I imagine there's some kind of dependency
here that sucks too:

    1fd7:       48 8b 57 08             mov    0x8(%rdi),%rdx
    1fdb:       48 8d 42 ff             lea    -0x1(%rdx),%rax
    1fdf:       83 e2 01                and    $0x1,%edx
    1fe2:       48 0f 44 c7             cmove  %rdi,%rax
    1fe6:       f0 80 60 02 fb          lock andb $0xfb,0x2(%rax)

Sure, it's going to be cache hot, but that cmove is going to have
to execute before the lock andb.

I would like to introduce a new concept that I call a Page Folio.
Or just struct folio to its friends.  Here it is,

struct folio {
        struct page page;

A folio is a struct page which is guaranteed not to be a tail page.
So it's either a head page or a base (order-0) page.  That means
we don't have to call compound_head() on it and we save massively.
end_page_writeback() reduces from four calls to compound_head() to just
one (at the beginning of the function) and it shrinks from 213 bytes
to 126 bytes (using the UEK config options).  I think even that one
can be eliminated, but I'm going slowly at this point and taking the
safe route of transforming a random struct page pointer into a struct
folio pointer by calling page_folio().  By the end of this exercise,
end_page_writeback() will become end_folio_writeback().

This is going to be a ton of work, and massively disruptive.  It'll touch
every filesystem, and a good few device drivers!  But I think it's worth
it.  Not every routine benefits as much as end_page_writeback(), but it
makes everything a little better.  At 29 bytes per call to lock_page(),
unlock_page(), put_page() and get_page(), that's on the order of 60kB of
text for allyesconfig.  More when you add on all the PageFoo() calls.
With the small amount of work I've done here, mm/filemap.o shrinks by
almost a kilobyte from 34007 to 33072.

But better than that, it's good documentation.  A function which has a
struct page argument might be expecting a head or base page and will
BUG if given a tail page.  It might work with any kind of page and
operate on PAGE_SIZE bytes.  It might work with any kind of page and
operate on page_size() bytes if given a head page but PAGE_SIZE bytes
if given a base or tail page.  It might operate on page_size() bytes
if passed a head or tail page.  We have examples of all of these today.
If a function takes a folio argument, it's operating on the entire folio.

The first seven patches introduce some of the new infrastructure we'll
need, then the last four are examples of converting functions to use
folios.  Eventually, we'll be able to convert some of the PageFoo flags
to be only available as FolioFoo flags.

I have a Zoom call this Friday at 18:00 UTC (13:00 Eastern,
10:00 Pacific, 03:00 Tokyo, 05:00 Sydney, 19:00 Berlin).
Meeting ID: 960 8868 8749, passcode 2097152
Feel free to join if you want to talk about this.

Matthew Wilcox (Oracle) (11):
  mm: Introduce struct folio
  mm: Add put_folio
  mm: Add get_folio
  mm: Create FolioFlags
  mm: Add unlock_folio
  mm: Add lock_folio
  mm: Add lock_folio_killable
  mm/filemap: Convert end_page_writeback to use a folio
  mm/filemap: Convert mapping_get_entry and pagecache_get_page to folio
  mm/filemap: Add folio_add_to_page_cache
  mm/swap: Convert rotate_reclaimable_page to folio

 include/linux/mm.h         |  36 ++++---
 include/linux/mm_types.h   |  17 ++++
 include/linux/page-flags.h |  36 ++++++-
 include/linux/pagemap.h    |  63 +++++++++---
 include/linux/swap.h       |   1 -
 mm/filemap.c               | 198 ++++++++++++++++++-------------------
 mm/internal.h              |   1 +
 mm/page_io.c               |   4 +-
 mm/swap.c                  |  12 +--
 9 files changed, 231 insertions(+), 137 deletions(-)


             reply	other threads:[~2020-12-08 20:12 UTC|newest]

Thread overview: 13+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-12-08 19:46 Matthew Wilcox (Oracle) [this message]
2020-12-08 19:46 ` [RFC PATCH 01/11] mm: Introduce struct folio Matthew Wilcox (Oracle)
2020-12-08 19:46 ` [RFC PATCH 02/11] mm: Add put_folio Matthew Wilcox (Oracle)
2020-12-08 19:46 ` [RFC PATCH 03/11] mm: Add get_folio Matthew Wilcox (Oracle)
2020-12-08 19:46 ` [RFC PATCH 04/11] mm: Create FolioFlags Matthew Wilcox (Oracle)
2020-12-08 19:46 ` [RFC PATCH 05/11] mm: Add unlock_folio Matthew Wilcox (Oracle)
2020-12-08 19:46 ` [RFC PATCH 06/11] mm: Add lock_folio Matthew Wilcox (Oracle)
2020-12-08 19:46 ` [RFC PATCH 07/11] mm: Add lock_folio_killable Matthew Wilcox (Oracle)
2020-12-08 19:46 ` [RFC PATCH 08/11] mm/filemap: Convert end_page_writeback to use a folio Matthew Wilcox (Oracle)
2020-12-08 19:46 ` [RFC PATCH 09/11] mm/filemap: Convert mapping_get_entry and pagecache_get_page to folio Matthew Wilcox (Oracle)
2020-12-08 19:46 ` [RFC PATCH 10/11] mm/filemap: Add folio_add_to_page_cache Matthew Wilcox (Oracle)
2020-12-11  8:32   ` Nikolay Borisov
2020-12-08 19:46 ` [RFC PATCH 11/11] mm/swap: Convert rotate_reclaimable_page to folio Matthew Wilcox (Oracle)

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).