From: "Matthew Wilcox (Oracle)" <willy@infradead.org>
To: akpm@linux-foundation.org
Cc: "Matthew Wilcox (Oracle)" <willy@infradead.org>,
linux-fsdevel@vger.kernel.org, linux-mm@kvack.org,
linux-kernel@vger.kernel.org
Subject: [PATCH v10 00/33] Memory folios
Date: Tue, 11 May 2021 22:47:02 +0100 [thread overview]
Message-ID: <20210511214735.1836149-1-willy@infradead.org> (raw)
Managing memory in 4KiB pages is a serious overhead. Many benchmarks
benefit from a larger "page size". As an example, an earlier iteration
of this idea which used compound pages (and wasn't particularly tuned)
got a 7% performance boost when compiling the kernel.
Using compound pages or THPs exposes a weakness of our type system.
Functions are often unprepared for compound pages to be passed to them,
and may only act on PAGE_SIZE chunks. Even functions which are aware of
compound pages may expect a head page, and do the wrong thing if passed
a tail page.
We also waste a lot of instructions ensuring that we're not looking at
a tail page. Almost every call to PageFoo() contains one or more hidden
calls to compound_head(). This also happens for get_page(), put_page()
and many more functions. There does not appear to be a way to tell gcc
that it can cache the result of compound_head(), nor is there a way to
tell it that compound_head() is idempotent.
This patch series uses a new type, the struct folio, to manage memory.
It provides some basic infrastructure that's worthwhile in its own right,
shrinking the kernel by about 5kB of text.
Since v9:
- Rebase onto mmotm 2021-05-10-21-46
- Add folio_memcg() definition for !MEMCG (intel lkp)
- Change folio->private from an unsigned long to a void *
- Use folio_page() to implement folio_file_page()
- Add folio_try_get() and folio_try_get_rcu()
- Trim back down to just the first few patches, which are better-reviewed.
v9: https://lore.kernel.org/linux-mm/20210505150628.111735-1-willy@infradead.org/
v8: https://lore.kernel.org/linux-mm/20210430180740.2707166-1-willy@infradead.org/
Matthew Wilcox (Oracle) (33):
mm: Introduce struct folio
mm: Add folio_pgdat and folio_zone
mm/vmstat: Add functions to account folio statistics
mm/debug: Add VM_BUG_ON_FOLIO and VM_WARN_ON_ONCE_FOLIO
mm: Add folio reference count functions
mm: Add folio_put
mm: Add folio_get
mm: Add folio_try_get_rcu
mm: Add folio flag manipulation functions
mm: Add folio_young and folio_idle
mm: Handle per-folio private data
mm/filemap: Add folio_index, folio_file_page and folio_contains
mm/filemap: Add folio_next_index
mm/filemap: Add folio_offset and folio_file_offset
mm/util: Add folio_mapping and folio_file_mapping
mm: Add folio_mapcount
mm/memcg: Add folio wrappers for various functions
mm/filemap: Add folio_unlock
mm/filemap: Add folio_lock
mm/filemap: Add folio_lock_killable
mm/filemap: Add __folio_lock_async
mm/filemap: Add __folio_lock_or_retry
mm/filemap: Add folio_wait_locked
mm/swap: Add folio_rotate_reclaimable
mm/filemap: Add folio_end_writeback
mm/writeback: Add folio_wait_writeback
mm/writeback: Add folio_wait_stable
mm/filemap: Add folio_wait_bit
mm/filemap: Add folio_wake_bit
mm/filemap: Convert page wait queues to be folios
mm/filemap: Add folio private_2 functions
fs/netfs: Add folio fscache functions
mm: Add folio_mapped
Documentation/core-api/mm-api.rst | 4 +
Documentation/filesystems/netfs_library.rst | 2 +
fs/afs/write.c | 9 +-
fs/cachefiles/rdwr.c | 16 +-
fs/io_uring.c | 2 +-
include/linux/memcontrol.h | 63 ++++
include/linux/mm.h | 174 ++++++++--
include/linux/mm_types.h | 71 ++++
include/linux/mmdebug.h | 20 ++
include/linux/netfs.h | 77 +++--
include/linux/page-flags.h | 230 ++++++++++---
include/linux/page_idle.h | 99 +++---
include/linux/page_ref.h | 158 ++++++++-
include/linux/pagemap.h | 358 ++++++++++++--------
include/linux/swap.h | 7 +-
include/linux/vmstat.h | 107 ++++++
mm/Makefile | 2 +-
mm/filemap.c | 315 ++++++++---------
mm/folio-compat.c | 43 +++
mm/internal.h | 1 +
mm/memory.c | 8 +-
mm/page-writeback.c | 72 ++--
mm/page_io.c | 4 +-
mm/swap.c | 18 +-
mm/swapfile.c | 8 +-
mm/util.c | 59 ++--
26 files changed, 1374 insertions(+), 553 deletions(-)
create mode 100644 mm/folio-compat.c
--
2.30.2
next reply other threads:[~2021-05-11 21:48 UTC|newest]
Thread overview: 97+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-05-11 21:47 Matthew Wilcox (Oracle) [this message]
2021-05-11 21:47 ` [PATCH v10 01/33] mm: Introduce struct folio Matthew Wilcox (Oracle)
2021-05-14 10:34 ` Vlastimil Babka
2021-05-14 10:40 ` Vlastimil Babka
2021-05-14 11:47 ` Matthew Wilcox
2021-05-15 10:55 ` William Kucharski
2021-05-15 20:14 ` Matthew Wilcox
2021-05-16 19:26 ` William Kucharski
2021-05-27 8:09 ` Christoph Hellwig
2021-05-11 21:47 ` [PATCH v10 02/33] mm: Add folio_pgdat and folio_zone Matthew Wilcox (Oracle)
2021-05-14 10:35 ` Vlastimil Babka
2021-05-11 21:47 ` [PATCH v10 03/33] mm/vmstat: Add functions to account folio statistics Matthew Wilcox (Oracle)
2021-05-14 10:36 ` Vlastimil Babka
2021-05-11 21:47 ` [PATCH v10 04/33] mm/debug: Add VM_BUG_ON_FOLIO and VM_WARN_ON_ONCE_FOLIO Matthew Wilcox (Oracle)
2021-05-14 10:44 ` Vlastimil Babka
2021-05-11 21:47 ` [PATCH v10 05/33] mm: Add folio reference count functions Matthew Wilcox (Oracle)
2021-05-14 11:04 ` Vlastimil Babka
2021-05-11 21:47 ` [PATCH v10 06/33] mm: Add folio_put Matthew Wilcox (Oracle)
2021-05-14 11:52 ` Vlastimil Babka
2021-05-11 21:47 ` [PATCH v10 07/33] mm: Add folio_get Matthew Wilcox (Oracle)
2021-05-14 11:56 ` Vlastimil Babka
2021-05-14 14:24 ` Matthew Wilcox
2021-05-14 15:39 ` Vlastimil Babka
2021-05-27 8:10 ` Christoph Hellwig
2021-05-27 22:53 ` Andrew Morton
2021-05-11 21:47 ` [PATCH v10 08/33] mm: Add folio_try_get_rcu Matthew Wilcox (Oracle)
2021-05-14 12:11 ` Vlastimil Babka
2021-05-27 8:16 ` Christoph Hellwig
2021-06-05 4:26 ` Matthew Wilcox
2021-06-06 14:13 ` Christoph Hellwig
2021-05-11 21:47 ` [PATCH v10 09/33] mm: Add folio flag manipulation functions Matthew Wilcox (Oracle)
2021-05-14 15:29 ` Vlastimil Babka
2021-05-11 21:47 ` [PATCH v10 10/33] mm: Add folio_young and folio_idle Matthew Wilcox (Oracle)
2021-05-14 15:33 ` Vlastimil Babka
2021-05-27 8:17 ` Christoph Hellwig
2021-05-11 21:47 ` [PATCH v10 11/33] mm: Handle per-folio private data Matthew Wilcox (Oracle)
2021-05-14 15:41 ` Vlastimil Babka
2021-05-11 21:47 ` [PATCH v10 12/33] mm/filemap: Add folio_index, folio_file_page and folio_contains Matthew Wilcox (Oracle)
2021-05-14 15:55 ` Vlastimil Babka
2021-05-15 15:51 ` Matthew Wilcox
2021-05-11 21:47 ` [PATCH v10 13/33] mm/filemap: Add folio_next_index Matthew Wilcox (Oracle)
2021-05-14 17:07 ` Vlastimil Babka
2021-05-11 21:47 ` [PATCH v10 14/33] mm/filemap: Add folio_offset and folio_file_offset Matthew Wilcox (Oracle)
2021-05-14 17:08 ` Vlastimil Babka
2021-05-11 21:47 ` [PATCH v10 15/33] mm/util: Add folio_mapping and folio_file_mapping Matthew Wilcox (Oracle)
2021-05-14 17:29 ` Vlastimil Babka
2021-05-11 21:47 ` [PATCH v10 16/33] mm: Add folio_mapcount Matthew Wilcox (Oracle)
2021-05-14 17:39 ` Vlastimil Babka
2021-05-18 18:45 ` Matthew Wilcox
2021-05-11 21:47 ` [PATCH v10 17/33] mm/memcg: Add folio wrappers for various functions Matthew Wilcox (Oracle)
2021-05-18 9:57 ` Vlastimil Babka
2021-05-11 21:47 ` [PATCH v10 18/33] mm/filemap: Add folio_unlock Matthew Wilcox (Oracle)
2021-05-18 10:06 ` Vlastimil Babka
2021-05-18 11:30 ` Matthew Wilcox
2021-05-11 21:47 ` [PATCH v10 19/33] mm/filemap: Add folio_lock Matthew Wilcox (Oracle)
2021-05-18 10:26 ` Vlastimil Babka
2021-05-11 21:47 ` [PATCH v10 20/33] mm/filemap: Add folio_lock_killable Matthew Wilcox (Oracle)
2021-05-18 10:31 ` Vlastimil Babka
2021-05-11 21:47 ` [PATCH v10 21/33] mm/filemap: Add __folio_lock_async Matthew Wilcox (Oracle)
2021-05-18 10:34 ` Vlastimil Babka
2021-05-11 21:47 ` [PATCH v10 22/33] mm/filemap: Add __folio_lock_or_retry Matthew Wilcox (Oracle)
2021-05-18 10:38 ` Vlastimil Babka
2021-05-18 10:45 ` Vlastimil Babka
2021-05-18 13:35 ` Matthew Wilcox
2021-05-11 21:47 ` [PATCH v10 23/33] mm/filemap: Add folio_wait_locked Matthew Wilcox (Oracle)
2021-05-18 10:41 ` Vlastimil Babka
2021-05-11 21:47 ` [PATCH v10 24/33] mm/swap: Add folio_rotate_reclaimable Matthew Wilcox (Oracle)
2021-05-18 10:48 ` Vlastimil Babka
2021-05-27 8:19 ` Christoph Hellwig
2021-05-11 21:47 ` [PATCH v10 25/33] mm/filemap: Add folio_end_writeback Matthew Wilcox (Oracle)
2021-05-18 11:08 ` Vlastimil Babka
2021-05-11 21:47 ` [PATCH v10 26/33] mm/writeback: Add folio_wait_writeback Matthew Wilcox (Oracle)
2021-05-18 11:12 ` Vlastimil Babka
2021-05-11 21:47 ` [PATCH v10 27/33] mm/writeback: Add folio_wait_stable Matthew Wilcox (Oracle)
2021-05-18 11:42 ` Vlastimil Babka
2021-05-18 13:55 ` Matthew Wilcox
2021-05-11 21:47 ` [PATCH v10 28/33] mm/filemap: Add folio_wait_bit Matthew Wilcox (Oracle)
2021-05-18 11:51 ` Vlastimil Babka
2021-05-11 21:47 ` [PATCH v10 29/33] mm/filemap: Add folio_wake_bit Matthew Wilcox (Oracle)
2021-05-18 11:53 ` Vlastimil Babka
2021-05-11 21:47 ` [PATCH v10 30/33] mm/filemap: Convert page wait queues to be folios Matthew Wilcox (Oracle)
2021-05-18 12:23 ` Vlastimil Babka
2021-05-11 21:47 ` [PATCH v10 31/33] mm/filemap: Add folio private_2 functions Matthew Wilcox (Oracle)
2021-05-18 12:26 ` Vlastimil Babka
2021-05-27 8:21 ` Christoph Hellwig
2021-05-11 21:47 ` [PATCH v10 32/33] fs/netfs: Add folio fscache functions Matthew Wilcox (Oracle)
2021-05-18 13:48 ` Vlastimil Babka
2021-05-27 8:23 ` Christoph Hellwig
2021-05-11 21:47 ` [PATCH v10 33/33] mm: Add folio_mapped Matthew Wilcox (Oracle)
2021-05-18 14:17 ` Vlastimil Babka
2021-05-27 8:31 ` Christoph Hellwig
2021-05-13 14:50 ` [PATCH v10 00/33] Memory folios Matthew Wilcox
2021-05-15 10:26 ` William Kucharski
2021-06-04 1:07 ` Matteo Croce
2021-06-04 2:13 ` Matthew Wilcox
2021-06-08 14:56 ` Matteo Croce
2021-06-08 14:56 ` Matteo Croce
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210511214735.1836149-1-willy@infradead.org \
--to=willy@infradead.org \
--cc=akpm@linux-foundation.org \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.