All of lore.kernel.org
 help / color / mirror / Atom feed
* [RFC PATCH v1 00/13] lru_lock scalability
@ 2018-01-31 23:04 ` daniel.m.jordan
  0 siblings, 0 replies; 62+ messages in thread
From: daniel.m.jordan @ 2018-01-31 23:04 UTC (permalink / raw)
  To: linux-mm, linux-kernel
  Cc: aaron.lu, ak, akpm, Dave.Dice, dave, khandual, ldufour, mgorman,
	mhocko, pasha.tatashin, steven.sistare, yossi.lev

lru_lock, a per-node* spinlock that protects an LRU list, is one of the
hottest locks in the kernel.  On some workloads on large machines, it
shows up at the top of lock_stat.

One way to improve lru_lock scalability is to introduce an array of locks,
with each lock protecting certain batches of LRU pages.

        *ooooooooooo**ooooooooooo**ooooooooooo**oooo ...
        |           ||           ||           ||
         \ batch 1 /  \ batch 2 /  \ batch 3 /  

In this ASCII depiction of an LRU, a page is represented with either '*'
or 'o'.  An asterisk indicates a sentinel page, which is a page at the
edge of a batch.  An 'o' indicates a non-sentinel page.

To remove a non-sentinel LRU page, only one lock from the array is
required.  This allows multiple threads to remove pages from different
batches simultaneously.  A sentinel page requires lru_lock in addition to
a lock from the array.

Full performance numbers appear in the last patch in this series, but this
prototype allows a microbenchmark to do up to 28% more page faults per
second with 16 or more concurrent processes.

This work was developed in collaboration with Steve Sistare.

Note: This is an early prototype.  I'm submitting it now to support my
request to attend LSF/MM, as well as get early feedback on the idea.  Any
comments appreciated.


* lru_lock is actually per-memcg, but without memcg's in the picture it
  becomes per-node.


Aaron Lu (1):
  mm: add a percpu_pagelist_batch sysctl interface

Daniel Jordan (12):
  mm: allow compaction to be disabled
  mm: add lock array to pgdat and batch fields to struct page
  mm: introduce struct lru_list_head in lruvec to hold per-LRU batch
    info
  mm: add batching logic to add/delete/move API's
  mm: add lru_[un]lock_all APIs
  mm: convert to-be-refactored lru_lock callsites to lock-all API
  mm: temporarily convert lru_lock callsites to lock-all API
  mm: introduce add-only version of pagevec_lru_move_fn
  mm: add LRU batch lock API's
  mm: use lru_batch locking in release_pages
  mm: split up release_pages into non-sentinel and sentinel passes
  mm: splice local lists onto the front of the LRU

 include/linux/mm_inline.h | 209 +++++++++++++++++++++++++++++++++++++++++++++-
 include/linux/mm_types.h  |   5 ++
 include/linux/mmzone.h    |  25 +++++-
 kernel/sysctl.c           |   9 ++
 mm/Kconfig                |   1 -
 mm/huge_memory.c          |   6 +-
 mm/memcontrol.c           |   5 +-
 mm/mlock.c                |  11 +--
 mm/mmzone.c               |   7 +-
 mm/page_alloc.c           |  43 +++++++++-
 mm/page_idle.c            |   4 +-
 mm/swap.c                 | 208 ++++++++++++++++++++++++++++++++++++---------
 mm/vmscan.c               |  49 +++++------
 13 files changed, 500 insertions(+), 82 deletions(-)

-- 
2.16.1

^ permalink raw reply	[flat|nested] 62+ messages in thread

end of thread, other threads:[~2018-02-13 21:07 UTC | newest]

Thread overview: 62+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-01-31 23:04 [RFC PATCH v1 00/13] lru_lock scalability daniel.m.jordan
2018-01-31 23:04 ` daniel.m.jordan
2018-01-31 23:04 ` [RFC PATCH v1 01/13] mm: add a percpu_pagelist_batch sysctl interface daniel.m.jordan
2018-01-31 23:04   ` daniel.m.jordan
2018-01-31 23:04 ` [RFC PATCH v1 02/13] mm: allow compaction to be disabled daniel.m.jordan
2018-01-31 23:04   ` daniel.m.jordan
2018-01-31 23:04 ` [RFC PATCH v1 03/13] mm: add lock array to pgdat and batch fields to struct page daniel.m.jordan
2018-01-31 23:04   ` daniel.m.jordan
2018-02-01 22:50   ` Tim Chen
2018-02-01 22:50     ` Tim Chen
2018-02-02  4:29     ` Daniel Jordan
2018-02-02  4:29       ` Daniel Jordan
2018-01-31 23:04 ` [RFC PATCH v1 04/13] mm: introduce struct lru_list_head in lruvec to hold per-LRU batch info daniel.m.jordan
2018-01-31 23:04   ` daniel.m.jordan
2018-01-31 23:04 ` [RFC PATCH v1 05/13] mm: add batching logic to add/delete/move API's daniel.m.jordan
2018-01-31 23:04   ` daniel.m.jordan
2018-01-31 23:04 ` [RFC PATCH v1 06/13] mm: add lru_[un]lock_all APIs daniel.m.jordan
2018-01-31 23:04   ` daniel.m.jordan
2018-01-31 23:04 ` [RFC PATCH v1 07/13] mm: convert to-be-refactored lru_lock callsites to lock-all API daniel.m.jordan
2018-01-31 23:04   ` daniel.m.jordan
2018-01-31 23:04 ` [RFC PATCH v1 08/13] mm: temporarily convert " daniel.m.jordan
2018-01-31 23:04   ` daniel.m.jordan
2018-01-31 23:04 ` [RFC PATCH v1 09/13] mm: introduce add-only version of pagevec_lru_move_fn daniel.m.jordan
2018-01-31 23:04   ` daniel.m.jordan
2018-01-31 23:04 ` [RFC PATCH v1 10/13] mm: add LRU batch lock API's daniel.m.jordan
2018-01-31 23:04   ` daniel.m.jordan
2018-01-31 23:04 ` [RFC PATCH v1 11/13] mm: use lru_batch locking in release_pages daniel.m.jordan
2018-01-31 23:04   ` daniel.m.jordan
2018-01-31 23:04 ` [RFC PATCH v1 12/13] mm: split up release_pages into non-sentinel and sentinel passes daniel.m.jordan
2018-01-31 23:04   ` daniel.m.jordan
2018-02-02 14:40   ` Laurent Dufour
2018-02-02 14:40     ` Laurent Dufour
2018-02-02 17:00     ` Laurent Dufour
2018-02-02 17:00       ` Laurent Dufour
2018-02-06 17:47       ` Daniel Jordan
2018-02-06 17:47         ` Daniel Jordan
2018-02-05  4:58   ` [lkp-robot] [mm] 44b163e12f: kernel_BUG_at_mm/swap.c kernel test robot
2018-02-05  4:58     ` kernel test robot
2018-01-31 23:04 ` [RFC PATCH v1 13/13] mm: splice local lists onto the front of the LRU daniel.m.jordan
2018-01-31 23:04   ` daniel.m.jordan
2018-02-01 23:30   ` Tim Chen
2018-02-01 23:30     ` Tim Chen
2018-02-02  5:17     ` Daniel Jordan
2018-02-02  5:17       ` Daniel Jordan
2018-02-02  5:21   ` Aaron Lu
2018-02-02  5:21     ` Aaron Lu
2018-02-06 17:38     ` Daniel Jordan
2018-02-06 17:38       ` Daniel Jordan
2018-02-02 15:22   ` Laurent Dufour
2018-02-02 15:22     ` Laurent Dufour
2018-02-06 18:18     ` Daniel Jordan
2018-02-06 18:18       ` Daniel Jordan
2018-02-01 15:54 ` [RFC PATCH v1 00/13] lru_lock scalability Steven Whitehouse
2018-02-01 15:54   ` Steven Whitehouse
2018-02-02  4:18   ` Daniel Jordan
2018-02-02  4:18     ` Daniel Jordan
2018-02-02 10:50     ` Steven Whitehouse
2018-02-02 10:50       ` Steven Whitehouse
2018-02-08 23:36 ` Andrew Morton
2018-02-08 23:36   ` Andrew Morton
2018-02-13 21:07   ` Daniel Jordan
2018-02-13 21:07     ` Daniel Jordan

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.