* [PATCH 1/8] mm: Align struct page more aesthetically
2017-12-20 15:52 [PATCH 0/8] Restructure struct page Matthew Wilcox
@ 2017-12-20 15:52 ` Matthew Wilcox
2017-12-20 15:52 ` [PATCH 2/8] mm: De-indent struct page Matthew Wilcox
` (7 subsequent siblings)
8 siblings, 0 replies; 12+ messages in thread
From: Matthew Wilcox @ 2017-12-20 15:52 UTC (permalink / raw)
To: linux-mm; +Cc: akpm, Matthew Wilcox
From: Matthew Wilcox <mawilcox@microsoft.com>
instead of an ifdef block at the end of the struct, which needed
its own comment, define _struct_page_alignment up at the top where it
fits nicely with the existing comment.
Signed-off-by: Matthew Wilcox <mawilcox@microsoft.com>
---
include/linux/mm_types.h | 16 +++++++---------
1 file changed, 7 insertions(+), 9 deletions(-)
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index cfd0ac4e5e0e..4509f0cfaf39 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -39,6 +39,12 @@ struct hmm;
* allows the use of atomic double word operations on the flags/mapping
* and lru list pointers also.
*/
+#ifdef CONFIG_HAVE_ALIGNED_STRUCT_PAGE
+#define _struct_page_alignment __aligned(2 * sizeof(unsigned long))
+#else
+#define _struct_page_alignment
+#endif
+
struct page {
/* First double word block */
unsigned long flags; /* Atomic flags, some possibly
@@ -212,15 +218,7 @@ struct page {
#ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS
int _last_cpupid;
#endif
-}
-/*
- * The struct page can be forced to be double word aligned so that atomic ops
- * on double words work. The SLUB allocator can make use of such a feature.
- */
-#ifdef CONFIG_HAVE_ALIGNED_STRUCT_PAGE
- __aligned(2 * sizeof(unsigned long))
-#endif
-;
+} _struct_page_alignment;
#define PAGE_FRAG_CACHE_MAX_SIZE __ALIGN_MASK(32768, ~PAGE_MASK)
#define PAGE_FRAG_CACHE_MAX_ORDER get_order(PAGE_FRAG_CACHE_MAX_SIZE)
--
2.15.1
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PATCH 2/8] mm: De-indent struct page
2017-12-20 15:52 [PATCH 0/8] Restructure struct page Matthew Wilcox
2017-12-20 15:52 ` [PATCH 1/8] mm: Align struct page more aesthetically Matthew Wilcox
@ 2017-12-20 15:52 ` Matthew Wilcox
2017-12-20 15:52 ` [PATCH 3/8] mm: Remove misleading alignment claims Matthew Wilcox
` (6 subsequent siblings)
8 siblings, 0 replies; 12+ messages in thread
From: Matthew Wilcox @ 2017-12-20 15:52 UTC (permalink / raw)
To: linux-mm; +Cc: akpm, Matthew Wilcox
From: Matthew Wilcox <mawilcox@microsoft.com>
I found the struct { union { struct { union { struct { } } } } }
layout rather confusing. Fortunately, there is an easier way to write
this. The innermost union is of four things which are the size of an
int, so the ones which are used by slab/slob/slub can be pulled up
two levels to be in the outermost union with 'counters'. That leaves
us with struct { union { struct { atomic_t; atomic_t; } } } which
has the same layout, but is easier to read.
Signed-off-by: Matthew Wilcox <mawilcox@microsoft.com>
---
include/linux/mm_types.h | 40 +++++++++++++++++++---------------------
1 file changed, 19 insertions(+), 21 deletions(-)
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index 4509f0cfaf39..27973166af28 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -84,28 +84,26 @@ struct page {
*/
unsigned counters;
#endif
- struct {
+ unsigned int active; /* SLAB */
+ struct { /* SLUB */
+ unsigned inuse:16;
+ unsigned objects:15;
+ unsigned frozen:1;
+ };
+ int units; /* SLOB */
+
+ struct { /* Page cache */
+ /*
+ * Count of ptes mapped in mms, to show when
+ * page is mapped & limit reverse map searches.
+ *
+ * Extra information about page type may be
+ * stored here for pages that are never mapped,
+ * in which case the value MUST BE <= -2.
+ * See page-flags.h for more details.
+ */
+ atomic_t _mapcount;
- union {
- /*
- * Count of ptes mapped in mms, to show when
- * page is mapped & limit reverse map searches.
- *
- * Extra information about page type may be
- * stored here for pages that are never mapped,
- * in which case the value MUST BE <= -2.
- * See page-flags.h for more details.
- */
- atomic_t _mapcount;
-
- unsigned int active; /* SLAB */
- struct { /* SLUB */
- unsigned inuse:16;
- unsigned objects:15;
- unsigned frozen:1;
- };
- int units; /* SLOB */
- };
/*
* Usage count, *USE WRAPPER FUNCTION* when manual
* accounting. See page_ref.h
--
2.15.1
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PATCH 3/8] mm: Remove misleading alignment claims
2017-12-20 15:52 [PATCH 0/8] Restructure struct page Matthew Wilcox
2017-12-20 15:52 ` [PATCH 1/8] mm: Align struct page more aesthetically Matthew Wilcox
2017-12-20 15:52 ` [PATCH 2/8] mm: De-indent struct page Matthew Wilcox
@ 2017-12-20 15:52 ` Matthew Wilcox
2017-12-20 15:52 ` [PATCH 4/8] mm: Improve comment on page->mapping Matthew Wilcox
` (5 subsequent siblings)
8 siblings, 0 replies; 12+ messages in thread
From: Matthew Wilcox @ 2017-12-20 15:52 UTC (permalink / raw)
To: linux-mm; +Cc: akpm, Matthew Wilcox
From: Matthew Wilcox <mawilcox@microsoft.com>
The "third double word block" isn't on 32-bit systems. The layout
looks like this:
unsigned long flags;
struct address_space *mapping
pgoff_t index;
atomic_t _mapcount;
atomic_t _refcount;
which is 32 bytes on 64-bit, but 20 bytes on 32-bit. Nobody is trying
to use the fact that it's double-word aligned today, so just remove the
misleading claims.
Signed-off-by: Matthew Wilcox <mawilcox@microsoft.com>
---
include/linux/mm_types.h | 13 +++++--------
1 file changed, 5 insertions(+), 8 deletions(-)
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index 27973166af28..c2294e6204e8 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -33,11 +33,11 @@ struct hmm;
* a page, though if it is a pagecache page, rmap structures can tell us
* who is mapping it.
*
- * The objects in struct page are organized in double word blocks in
- * order to allows us to use atomic double word operations on portions
- * of struct page. That is currently only used by slub but the arrangement
- * allows the use of atomic double word operations on the flags/mapping
- * and lru list pointers also.
+ * SLUB uses cmpxchg_double() to atomically update its freelist and
+ * counters. That requires that freelist & counters be adjacent and
+ * double-word aligned. We align all struct pages to double-word
+ * boundaries, and ensure that 'freelist' is aligned within the
+ * struct.
*/
#ifdef CONFIG_HAVE_ALIGNED_STRUCT_PAGE
#define _struct_page_alignment __aligned(2 * sizeof(unsigned long))
@@ -113,8 +113,6 @@ struct page {
};
/*
- * Third double word block
- *
* WARNING: bit 0 of the first word encode PageTail(). That means
* the rest users of the storage space MUST NOT use the bit to
* avoid collision and false-positive PageTail().
@@ -175,7 +173,6 @@ struct page {
#endif
};
- /* Remainder is not double word aligned */
union {
unsigned long private; /* Mapping-private opaque data:
* usually used for buffer_heads
--
2.15.1
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PATCH 4/8] mm: Improve comment on page->mapping
2017-12-20 15:52 [PATCH 0/8] Restructure struct page Matthew Wilcox
` (2 preceding siblings ...)
2017-12-20 15:52 ` [PATCH 3/8] mm: Remove misleading alignment claims Matthew Wilcox
@ 2017-12-20 15:52 ` Matthew Wilcox
2017-12-20 15:52 ` [PATCH 5/8] mm: Introduce _slub_counter_t Matthew Wilcox
` (4 subsequent siblings)
8 siblings, 0 replies; 12+ messages in thread
From: Matthew Wilcox @ 2017-12-20 15:52 UTC (permalink / raw)
To: linux-mm; +Cc: akpm, Matthew Wilcox
From: Matthew Wilcox <mawilcox@microsoft.com>
The comment on page->mapping is terse, and out of date (it does not
mention the possibility of PAGE_MAPPING_MOVABLE). Instead, point
the interested reader to page-flags.h where there is a much better
comment.
Signed-off-by: Matthew Wilcox <mawilcox@microsoft.com>
---
include/linux/mm_types.h | 12 +++---------
1 file changed, 3 insertions(+), 9 deletions(-)
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index c2294e6204e8..8c3b8cea22ee 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -50,15 +50,9 @@ struct page {
unsigned long flags; /* Atomic flags, some possibly
* updated asynchronously */
union {
- struct address_space *mapping; /* If low bit clear, points to
- * inode address_space, or NULL.
- * If page mapped as anonymous
- * memory, low bit is set, and
- * it points to anon_vma object
- * or KSM private structure. See
- * PAGE_MAPPING_ANON and
- * PAGE_MAPPING_KSM.
- */
+ /* See page-flags.h for the definition of PAGE_MAPPING_FLAGS */
+ struct address_space *mapping;
+
void *s_mem; /* slab first object */
atomic_t compound_mapcount; /* first tail page */
/* page_deferred_list().next -- second tail page */
--
2.15.1
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PATCH 5/8] mm: Introduce _slub_counter_t
2017-12-20 15:52 [PATCH 0/8] Restructure struct page Matthew Wilcox
` (3 preceding siblings ...)
2017-12-20 15:52 ` [PATCH 4/8] mm: Improve comment on page->mapping Matthew Wilcox
@ 2017-12-20 15:52 ` Matthew Wilcox
2017-12-20 15:52 ` [PATCH 6/8] mm: Store compound_dtor / compound_order as bytes Matthew Wilcox
` (3 subsequent siblings)
8 siblings, 0 replies; 12+ messages in thread
From: Matthew Wilcox @ 2017-12-20 15:52 UTC (permalink / raw)
To: linux-mm; +Cc: akpm, Matthew Wilcox
From: Matthew Wilcox <mawilcox@microsoft.com>
Instead of putting the ifdef in the middle of the definition of struct
page, pull it forward to the rest of the ifdeffery around the SLUB
cmpxchg_double optimisation.
Signed-off-by: Matthew Wilcox <mawilcox@microsoft.com>
---
include/linux/mm_types.h | 21 ++++++++-------------
1 file changed, 8 insertions(+), 13 deletions(-)
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index 8c3b8cea22ee..5521c9799c50 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -41,9 +41,15 @@ struct hmm;
*/
#ifdef CONFIG_HAVE_ALIGNED_STRUCT_PAGE
#define _struct_page_alignment __aligned(2 * sizeof(unsigned long))
+#if defined(CONFIG_HAVE_CMPXCHG_DOUBLE)
+#define _slub_counter_t unsigned long
#else
-#define _struct_page_alignment
+#define _slub_counter_t unsigned int
#endif
+#else /* !CONFIG_HAVE_ALIGNED_STRUCT_PAGE */
+#define _struct_page_alignment
+#define _slub_counter_t unsigned int
+#endif /* !CONFIG_HAVE_ALIGNED_STRUCT_PAGE */
struct page {
/* First double word block */
@@ -66,18 +72,7 @@ struct page {
};
union {
-#if defined(CONFIG_HAVE_CMPXCHG_DOUBLE) && \
- defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE)
- /* Used for cmpxchg_double in slub */
- unsigned long counters;
-#else
- /*
- * Keep _refcount separate from slub cmpxchg_double data.
- * As the rest of the double word is protected by slab_lock
- * but _refcount is not.
- */
- unsigned counters;
-#endif
+ _slub_counter_t counters;
unsigned int active; /* SLAB */
struct { /* SLUB */
unsigned inuse:16;
--
2.15.1
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PATCH 6/8] mm: Store compound_dtor / compound_order as bytes
2017-12-20 15:52 [PATCH 0/8] Restructure struct page Matthew Wilcox
` (4 preceding siblings ...)
2017-12-20 15:52 ` [PATCH 5/8] mm: Introduce _slub_counter_t Matthew Wilcox
@ 2017-12-20 15:52 ` Matthew Wilcox
2017-12-20 15:52 ` [PATCH 7/8] mm: Document how to use struct page Matthew Wilcox
` (2 subsequent siblings)
8 siblings, 0 replies; 12+ messages in thread
From: Matthew Wilcox @ 2017-12-20 15:52 UTC (permalink / raw)
To: linux-mm; +Cc: akpm, Matthew Wilcox
From: Matthew Wilcox <mawilcox@microsoft.com>
Neither of these values get even close to 256; compound_dtor is
currently at a maximum of 3, and compound_order can't be over 64.
No machine has inefficient access to bytes since EV5, and while
those are still supported, we don't optimise for them any more.
Signed-off-by: Matthew Wilcox <mawilcox@microsoft.com>
---
include/linux/mm_types.h | 15 ++-------------
1 file changed, 2 insertions(+), 13 deletions(-)
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index 5521c9799c50..1a3ba1f1605d 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -136,19 +136,8 @@ struct page {
unsigned long compound_head; /* If bit zero is set */
/* First tail page only */
-#ifdef CONFIG_64BIT
- /*
- * On 64 bit system we have enough space in struct page
- * to encode compound_dtor and compound_order with
- * unsigned int. It can help compiler generate better or
- * smaller code on some archtectures.
- */
- unsigned int compound_dtor;
- unsigned int compound_order;
-#else
- unsigned short int compound_dtor;
- unsigned short int compound_order;
-#endif
+ unsigned char compound_dtor;
+ unsigned char compound_order;
};
#if defined(CONFIG_TRANSPARENT_HUGEPAGE) && USE_SPLIT_PMD_PTLOCKS
--
2.15.1
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PATCH 7/8] mm: Document how to use struct page
2017-12-20 15:52 [PATCH 0/8] Restructure struct page Matthew Wilcox
` (5 preceding siblings ...)
2017-12-20 15:52 ` [PATCH 6/8] mm: Store compound_dtor / compound_order as bytes Matthew Wilcox
@ 2017-12-20 15:52 ` Matthew Wilcox
2017-12-20 15:52 ` [PATCH 8/8] mm: Remove reference to PG_buddy Matthew Wilcox
2017-12-20 15:55 ` [PATCH 0/8] Restructure struct page Matthew Wilcox
8 siblings, 0 replies; 12+ messages in thread
From: Matthew Wilcox @ 2017-12-20 15:52 UTC (permalink / raw)
To: linux-mm; +Cc: akpm, Matthew Wilcox
From: Matthew Wilcox <mawilcox@microsoft.com>
Be really explicit about what bits / bytes are reserved for users that
want to store extra information about the pages they allocate.
Signed-off-by: Matthew Wilcox <mawilcox@microsoft.com>
---
include/linux/mm_types.h | 23 ++++++++++++++++++++++-
1 file changed, 22 insertions(+), 1 deletion(-)
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index 1a3ba1f1605d..a517d210f177 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -31,7 +31,28 @@ struct hmm;
* it to keep track of whatever it is we are using the page for at the
* moment. Note that we have no way to track which tasks are using
* a page, though if it is a pagecache page, rmap structures can tell us
- * who is mapping it.
+ * who is mapping it. If you allocate the page using alloc_pages(), you
+ * can use some of the space in struct page for your own purposes.
+ *
+ * Pages that were once in the page cache may be found under the RCU lock
+ * even after they have been recycled to a different purpose. The page cache
+ * will read and writes some of the fields in struct page to lock the page,
+ * then check that it's still in the page cache. It is vital that all users
+ * of struct page:
+ * 1. Use the first word as PageFlags.
+ * 2. Clear or preserve bit 0 of page->compound_head. It is used as
+ * PageTail for compound pages, and the page cache must not see false
+ * positives. Some users put a pointer here (guaranteed to be at least
+ * 4-byte aligned), other users avoid using the word altogether.
+ * 3. page->_refcount must either not be used, or must be used in such a
+ * way that other CPUs temporarily incrementing and then decrementing the
+ * refcount does not cause problems. On receiving the page from
+ * alloc_pages(), the refcount will be positive.
+ *
+ * If you allocate pages of order > 0, you can use the fields in the struct
+ * page associated with each page, but bear in mind that the pages may have
+ * been inserted individually into the page cache, so you must use the above
+ * three fields in a compatible way for each struct page.
*
* SLUB uses cmpxchg_double() to atomically update its freelist and
* counters. That requires that freelist & counters be adjacent and
--
2.15.1
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PATCH 8/8] mm: Remove reference to PG_buddy
2017-12-20 15:52 [PATCH 0/8] Restructure struct page Matthew Wilcox
` (6 preceding siblings ...)
2017-12-20 15:52 ` [PATCH 7/8] mm: Document how to use struct page Matthew Wilcox
@ 2017-12-20 15:52 ` Matthew Wilcox
2017-12-20 15:55 ` [PATCH 0/8] Restructure struct page Matthew Wilcox
8 siblings, 0 replies; 12+ messages in thread
From: Matthew Wilcox @ 2017-12-20 15:52 UTC (permalink / raw)
To: linux-mm; +Cc: akpm, Matthew Wilcox
From: Matthew Wilcox <mawilcox@microsoft.com>
PG_buddy doesn't exist any more. It's called PageBuddy now.
Signed-off-by: Matthew Wilcox <mawilcox@microsoft.com>
---
include/linux/mm_types.h | 15 ++++++++-------
1 file changed, 8 insertions(+), 7 deletions(-)
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index a517d210f177..06f16a451a53 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -173,13 +173,14 @@ struct page {
};
union {
- unsigned long private; /* Mapping-private opaque data:
- * usually used for buffer_heads
- * if PagePrivate set; used for
- * swp_entry_t if PageSwapCache;
- * indicates order in the buddy
- * system if PG_buddy is set.
- */
+ /*
+ * Mapping-private opaque data:
+ * Usually used for buffer_heads if PagePrivate
+ * Used for swp_entry_t if PageSwapCache
+ * Indicates order in the buddy system if PageBuddy
+ */
+ unsigned long private;
+
#if USE_SPLIT_PTE_PTLOCKS
#if ALLOC_SPLIT_PTLOCKS
spinlock_t *ptl;
--
2.15.1
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 12+ messages in thread* Re: [PATCH 0/8] Restructure struct page
2017-12-20 15:52 [PATCH 0/8] Restructure struct page Matthew Wilcox
` (7 preceding siblings ...)
2017-12-20 15:52 ` [PATCH 8/8] mm: Remove reference to PG_buddy Matthew Wilcox
@ 2017-12-20 15:55 ` Matthew Wilcox
8 siblings, 0 replies; 12+ messages in thread
From: Matthew Wilcox @ 2017-12-20 15:55 UTC (permalink / raw)
To: linux-mm; +Cc: akpm, Matthew Wilcox
On Wed, Dec 20, 2017 at 07:52:48AM -0800, Matthew Wilcox wrote:
> From: Matthew Wilcox <mawilcox@microsoft.com>
>
> This series does not attempt any grand restructuring as I proposed last
> week. Instead, it cures the worst of the indentitis, fixes the
> documentation and reduces the ifdeffery. The only layout change is
> compound_dtor and compound_order are each reduced to one byte. At
> least, that's my intent.
My apologies. I typod my git send-email and resent v1, instead of sending v2.
"A computer lets you make more mistakes faster than any other invention
with the possible exceptions of handguns and Tequila." a?? Mitch Ratcliffe
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 12+ messages in thread