Linux-Fsdevel Archive on lore.kernel.org
 help / color / Atom feed
From: "Matthew Wilcox (Oracle)" <willy@infradead.org>
To: akpm@linux-foundation.org
Cc: "Matthew Wilcox (Oracle)" <willy@infradead.org>,
	linux-fsdevel@vger.kernel.org, linux-mm@kvack.org,
	linux-kernel@vger.kernel.org
Subject: [PATCH v12 00/33] Memory folios
Date: Tue, 22 Jun 2021 12:40:45 +0100
Message-ID: <20210622114118.3388190-1-willy@infradead.org> (raw)

Managing memory in 4KiB pages is a serious overhead.  Many benchmarks
benefit from a larger "page size".  As an example, an earlier iteration
of this idea which used compound pages (and wasn't particularly tuned)
got a 7% performance boost when compiling the kernel.

Using compound pages or THPs exposes a weakness of our type system.
Functions are often unprepared for compound pages to be passed to them,
and may only act on PAGE_SIZE chunks.  Even functions which are aware of
compound pages may expect a head page, and do the wrong thing if passed
a tail page.

We also waste a lot of instructions ensuring that we're not looking at
a tail page.  Almost every call to PageFoo() contains one or more hidden
calls to compound_head().  This also happens for get_page(), put_page()
and many more functions.  There does not appear to be a way to tell gcc
that it can cache the result of compound_head(), nor is there a way to
tell it that compound_head() is idempotent.
This patch series uses a new type, the struct folio, to manage memory.
It provides some basic infrastructure that's worthwhile in its own right,
shrinking the kernel by about 6kB of text.

The full patch series is considerably larger (~200 patches),
and enables XFS to use large pages.  It can be found at
https://git.infradead.org/users/willy/pagecache.git/shortlog/refs/heads/folio
(not everything there is in good shape for upstream submission, but
if you go as far as "mm/readahead: Add multi-page folio readahead",
it passes xfstests).  An earlier version of this patch set found it was
worth about a 7% reduction of wall-clock time on kernel compiles.

Since v12:
 - Reworded commit message for folio_rotate_reclaimable (Christoph Hellwig)
 - Fixed documentation for folio fscache functions (David Howells)
 - Rebased on set_page_dirty cleanups which are in mmotm
 - Renamed page_offset() to page_pos() and page_file_offset() to
   page_file_pos() (David Howells)
 - Make __folio_lock_or_retry() and lock_page_or_retry() return a bool
   (David Howells)

v11: https://lore.kernel.org/linux-mm/20210614201435.1379188-1-willy@infradead.org/
v10: https://lore.kernel.org/linux-mm/20210511214735.1836149-1-willy@infradead.org/
v9: https://lore.kernel.org/linux-mm/20210505150628.111735-1-willy@infradead.org/
v8: https://lore.kernel.org/linux-mm/20210430180740.2707166-1-willy@infradead.org/

Matthew Wilcox (Oracle) (33):
  mm: Convert get_page_unless_zero() to return bool
  mm: Introduce struct folio
  mm: Add folio_pgdat(), folio_zone() and folio_zonenum()
  mm/vmstat: Add functions to account folio statistics
  mm/debug: Add VM_BUG_ON_FOLIO() and VM_WARN_ON_ONCE_FOLIO()
  mm: Add folio reference count functions
  mm: Add folio_put()
  mm: Add folio_get()
  mm: Add folio_try_get_rcu()
  mm: Add folio flag manipulation functions
  mm/lru: Add folio LRU functions
  mm: Handle per-folio private data
  mm/filemap: Add folio_index(), folio_file_page() and folio_contains()
  mm/filemap: Add folio_next_index()
  mm/filemap: Add folio_pos() and folio_file_pos()
  mm/util: Add folio_mapping() and folio_file_mapping()
  mm/memcg: Add folio wrappers for various functions
  mm/filemap: Add folio_unlock()
  mm/filemap: Add folio_lock()
  mm/filemap: Add folio_lock_killable()
  mm/filemap: Add __folio_lock_async()
  mm/filemap: Add folio_wait_locked()
  mm/filemap: Add __folio_lock_or_retry()
  mm/swap: Add folio_rotate_reclaimable()
  mm/filemap: Add folio_end_writeback()
  mm/writeback: Add folio_wait_writeback()
  mm/writeback: Add folio_wait_stable()
  mm/filemap: Add folio_wait_bit()
  mm/filemap: Add folio_wake_bit()
  mm/filemap: Convert page wait queues to be folios
  mm/filemap: Add folio private_2 functions
  fs/netfs: Add folio fscache functions
  mm: Add folio_mapped()

 Documentation/core-api/mm-api.rst           |   4 +
 Documentation/filesystems/netfs_library.rst |   2 +
 fs/afs/write.c                              |   9 +-
 fs/cachefiles/rdwr.c                        |  16 +-
 fs/io_uring.c                               |   2 +-
 include/linux/huge_mm.h                     |  15 -
 include/linux/memcontrol.h                  |  72 ++++
 include/linux/mm.h                          | 165 +++++++--
 include/linux/mm_inline.h                   |  85 +++--
 include/linux/mm_types.h                    |  77 ++++
 include/linux/mmdebug.h                     |  20 +
 include/linux/netfs.h                       |  77 ++--
 include/linux/page-flags.h                  | 245 ++++++++----
 include/linux/page_ref.h                    | 158 +++++++-
 include/linux/pagemap.h                     | 390 +++++++++++---------
 include/linux/swap.h                        |   7 +-
 include/linux/vmstat.h                      | 107 ++++++
 mm/Makefile                                 |   2 +-
 mm/filemap.c                                | 329 +++++++++--------
 mm/folio-compat.c                           |  43 +++
 mm/internal.h                               |   1 +
 mm/memory.c                                 |   8 +-
 mm/page-writeback.c                         |  72 ++--
 mm/page_io.c                                |   4 +-
 mm/swap.c                                   |  30 +-
 mm/swapfile.c                               |   8 +-
 mm/util.c                                   |  59 +--
 27 files changed, 1427 insertions(+), 580 deletions(-)
 create mode 100644 mm/folio-compat.c

-- 
2.30.2


             reply index

Thread overview: 42+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-06-22 11:40 Matthew Wilcox (Oracle) [this message]
2021-06-22 11:40 ` [PATCH v12 01/33] mm: Convert get_page_unless_zero() to return bool Matthew Wilcox (Oracle)
2021-06-23  7:46   ` Christoph Hellwig
2021-06-28 12:35   ` Kirill A. Shutemov
2021-06-22 11:40 ` [PATCH v12 02/33] mm: Introduce struct folio Matthew Wilcox (Oracle)
2021-06-22 11:40 ` [PATCH v12 03/33] mm: Add folio_pgdat(), folio_zone() and folio_zonenum() Matthew Wilcox (Oracle)
2021-06-22 11:40 ` [PATCH v12 04/33] mm/vmstat: Add functions to account folio statistics Matthew Wilcox (Oracle)
2021-06-22 11:40 ` [PATCH v12 05/33] mm/debug: Add VM_BUG_ON_FOLIO() and VM_WARN_ON_ONCE_FOLIO() Matthew Wilcox (Oracle)
2021-06-22 11:40 ` [PATCH v12 06/33] mm: Add folio reference count functions Matthew Wilcox (Oracle)
2021-06-22 11:40 ` [PATCH v12 07/33] mm: Add folio_put() Matthew Wilcox (Oracle)
2021-06-22 11:40 ` [PATCH v12 08/33] mm: Add folio_get() Matthew Wilcox (Oracle)
2021-06-22 11:40 ` [PATCH v12 09/33] mm: Add folio_try_get_rcu() Matthew Wilcox (Oracle)
2021-06-28 12:38   ` Kirill A. Shutemov
2021-06-22 11:40 ` [PATCH v12 10/33] mm: Add folio flag manipulation functions Matthew Wilcox (Oracle)
2021-06-22 11:40 ` [PATCH v12 11/33] mm/lru: Add folio LRU functions Matthew Wilcox (Oracle)
2021-06-28 12:40   ` Kirill A. Shutemov
2021-06-22 11:40 ` [PATCH v12 12/33] mm: Handle per-folio private data Matthew Wilcox (Oracle)
2021-06-22 11:40 ` [PATCH v12 13/33] mm/filemap: Add folio_index(), folio_file_page() and folio_contains() Matthew Wilcox (Oracle)
2021-06-22 11:40 ` [PATCH v12 14/33] mm/filemap: Add folio_next_index() Matthew Wilcox (Oracle)
2021-06-22 11:41 ` [PATCH v12 15/33] mm/filemap: Add folio_pos() and folio_file_pos() Matthew Wilcox (Oracle)
2021-06-22 11:41 ` [PATCH v12 16/33] mm/util: Add folio_mapping() and folio_file_mapping() Matthew Wilcox (Oracle)
2021-06-22 11:41 ` [PATCH v12 17/33] mm/memcg: Add folio wrappers for various functions Matthew Wilcox (Oracle)
2021-06-22 11:41 ` [PATCH v12 18/33] mm/filemap: Add folio_unlock() Matthew Wilcox (Oracle)
2021-06-22 11:41 ` [PATCH v12 19/33] mm/filemap: Add folio_lock() Matthew Wilcox (Oracle)
2021-06-22 11:41 ` [PATCH v12 20/33] mm/filemap: Add folio_lock_killable() Matthew Wilcox (Oracle)
2021-06-22 11:41 ` [PATCH v12 21/33] mm/filemap: Add __folio_lock_async() Matthew Wilcox (Oracle)
2021-06-22 11:41 ` [PATCH v12 22/33] mm/filemap: Add folio_wait_locked() Matthew Wilcox (Oracle)
2021-06-22 11:41 ` [PATCH v12 23/33] mm/filemap: Add __folio_lock_or_retry() Matthew Wilcox (Oracle)
2021-06-22 11:41 ` [PATCH v12 24/33] mm/swap: Add folio_rotate_reclaimable() Matthew Wilcox (Oracle)
2021-06-28 12:46   ` Kirill A. Shutemov
2021-06-22 11:41 ` [PATCH v12 25/33] mm/filemap: Add folio_end_writeback() Matthew Wilcox (Oracle)
2021-06-22 11:41 ` [PATCH v12 26/33] mm/writeback: Add folio_wait_writeback() Matthew Wilcox (Oracle)
2021-06-22 11:41 ` [PATCH v12 27/33] mm/writeback: Add folio_wait_stable() Matthew Wilcox (Oracle)
2021-06-22 11:41 ` [PATCH v12 28/33] mm/filemap: Add folio_wait_bit() Matthew Wilcox (Oracle)
2021-06-22 11:41 ` [PATCH v12 29/33] mm/filemap: Add folio_wake_bit() Matthew Wilcox (Oracle)
2021-06-22 11:41 ` [PATCH v12 30/33] mm/filemap: Convert page wait queues to be folios Matthew Wilcox (Oracle)
2021-06-22 11:41 ` [PATCH v12 31/33] mm/filemap: Add folio private_2 functions Matthew Wilcox (Oracle)
2021-06-28 12:47   ` Kirill A. Shutemov
2021-06-22 11:41 ` [PATCH v12 32/33] fs/netfs: Add folio fscache functions Matthew Wilcox (Oracle)
2021-06-28 12:49   ` Kirill A. Shutemov
2021-06-22 11:41 ` [PATCH v12 33/33] mm: Add folio_mapped() Matthew Wilcox (Oracle)
2021-06-28 12:51   ` Kirill A. Shutemov

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210622114118.3388190-1-willy@infradead.org \
    --to=willy@infradead.org \
    --cc=akpm@linux-foundation.org \
    --cc=linux-fsdevel@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-Fsdevel Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-fsdevel/0 linux-fsdevel/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-fsdevel linux-fsdevel/ https://lore.kernel.org/linux-fsdevel \
		linux-fsdevel@vger.kernel.org
	public-inbox-index linux-fsdevel

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-fsdevel


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git