[GIT PULL] Create large folios in iomap buffered write path

[Matthew Wilcox <willy@infradead.org>]

The following changes since commit 5b8d6e8539498e8b2fa67fbcce3fe87834d44a7a:

  Merge tag 'xtensa-20230716' of https://github.com/jcmvbkbc/linux-xtensa (2023-07-16 14:12:49 -0700)

are available in the Git repository at:

  git://git.infradead.org/users/willy/pagecache.git tags/large-folio-writes

for you to fetch changes up to b684c2e69df4883c2d00045bdd159b6a94d28c98:

  iomap: Copy larger chunks from userspace (2023-07-16 17:53:18 -0400)
----------------------------------------------------------------

Compared to the last version I sent out by email, I split the
copy_page_from_iter_atomic() change into two pieces based on Luis &
Christoph's feedback.  There should be no other changes.  I also added
the Reviewed-by tags that were offered.

----------------------------------------------------------------
Create large folios in iomap buffered write path

Commit ebb7fb1557b1 limited the length of ioend chains to 4096 entries
to improve worst-case latency.  Unfortunately, this had the effect of
limiting the performance of:

fio -name write-bandwidth -rw=write -bs=1024Ki -size=32Gi -runtime=30 \
        -iodepth 1 -ioengine sync -zero_buffers=1 -direct=0 -end_fsync=1 \
        -numjobs=4 -directory=/mnt/test

The problem ends up being lock contention on the i_pages spinlock as we
clear the writeback bit on each folio (and propagate that up through
the tree).  By using larger folios, we decrease the number of folios
to be processed by a factor of 256 for this benchmark, eliminating the
lock contention.

Creating large folios in the buffered write path is also the right
thing to do.  It's a project that has been on the back burner for years,
it just hasn't been important enough to do before now.

----------------------------------------------------------------
Matthew Wilcox (Oracle) (10):
      iov_iter: Map the page later in copy_page_from_iter_atomic()
      iov_iter: Handle compound highmem pages in copy_page_from_iter_atomic()
      iov_iter: Add copy_folio_from_iter_atomic()
      iomap: Remove large folio handling in iomap_invalidate_folio()
      doc: Correct the description of ->release_folio
      iomap: Remove unnecessary test from iomap_release_folio()
      filemap: Add fgf_t typedef
      filemap: Allow __filemap_get_folio to allocate large folios
      iomap: Create large folios in the buffered write path
      iomap: Copy larger chunks from userspace

 Documentation/filesystems/locking.rst | 15 +++++--
 fs/btrfs/file.c                       |  6 +--
 fs/f2fs/compress.c                    |  2 +-
 fs/f2fs/f2fs.h                        |  2 +-
 fs/gfs2/bmap.c                        |  2 +-
 fs/iomap/buffered-io.c                | 54 +++++++++++------------
 include/linux/iomap.h                 |  2 +-
 include/linux/pagemap.h               | 82 ++++++++++++++++++++++++++++++-----
 include/linux/uio.h                   |  9 +++-
 lib/iov_iter.c                        | 43 +++++++++++-------
 mm/filemap.c                          | 65 ++++++++++++++-------------
 mm/folio-compat.c                     |  2 +-
 mm/readahead.c                        | 13 ------
 13 files changed, 187 insertions(+), 110 deletions(-)