From: Roman Gushchin <guro@fb.com>
To: <linux-mm@kvack.org>
Cc: Michal Hocko <mhocko@kernel.org>,
Johannes Weiner <hannes@cmpxchg.org>,
<linux-kernel@vger.kernel.org>, <kernel-team@fb.com>,
Shakeel Butt <shakeelb@google.com>,
Vladimir Davydov <vdavydov.dev@gmail.com>,
Waiman Long <longman@redhat.com>,
Christoph Lameter <cl@linux.com>, Roman Gushchin <guro@fb.com>
Subject: [PATCH 05/16] mm: slub: implement SLUB version of obj_to_index()
Date: Thu, 17 Oct 2019 17:28:09 -0700 [thread overview]
Message-ID: <20191018002820.307763-6-guro@fb.com> (raw)
In-Reply-To: <20191018002820.307763-1-guro@fb.com>
This commit implements SLUB version of the obj_to_index() function,
which will be required to calculate the offset of memcg_ptr in the
mem_cgroup_vec to store/obtain the memcg ownership data.
To make it faster, let's repeat the SLAB's trick introduced by
commit 6a2d7a955d8d ("[PATCH] SLAB: use a multiply instead of a
divide in obj_to_index()") and avoid an expensive division.
Signed-off-by: Roman Gushchin <guro@fb.com>
---
include/linux/slub_def.h | 9 +++++++++
mm/slub.c | 1 +
2 files changed, 10 insertions(+)
diff --git a/include/linux/slub_def.h b/include/linux/slub_def.h
index d2153789bd9f..200ea292f250 100644
--- a/include/linux/slub_def.h
+++ b/include/linux/slub_def.h
@@ -8,6 +8,7 @@
* (C) 2007 SGI, Christoph Lameter
*/
#include <linux/kobject.h>
+#include <linux/reciprocal_div.h>
enum stat_item {
ALLOC_FASTPATH, /* Allocation from cpu slab */
@@ -86,6 +87,7 @@ struct kmem_cache {
unsigned long min_partial;
unsigned int size; /* The size of an object including metadata */
unsigned int object_size;/* The size of an object without metadata */
+ struct reciprocal_value reciprocal_size;
unsigned int offset; /* Free pointer offset */
#ifdef CONFIG_SLUB_CPU_PARTIAL
/* Number of per cpu partial objects to keep around */
@@ -182,4 +184,11 @@ static inline void *nearest_obj(struct kmem_cache *cache, struct page *page,
return result;
}
+static inline unsigned int obj_to_index(const struct kmem_cache *cache,
+ const struct page *page, void *obj)
+{
+ return reciprocal_divide(kasan_reset_tag(obj) - page_address(page),
+ cache->reciprocal_size);
+}
+
#endif /* _LINUX_SLUB_DEF_H */
diff --git a/mm/slub.c b/mm/slub.c
index e810582f5b86..557ea45a5d75 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -3600,6 +3600,7 @@ static int calculate_sizes(struct kmem_cache *s, int forced_order)
*/
size = ALIGN(size, s->align);
s->size = size;
+ s->reciprocal_size = reciprocal_value(size);
if (forced_order >= 0)
order = forced_order;
else
--
2.21.0
next prev parent reply other threads:[~2019-10-18 0:28 UTC|newest]
Thread overview: 36+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-10-18 0:28 [PATCH 00/16] The new slab memory controller Roman Gushchin
2019-10-18 0:28 ` [PATCH 01/16] mm: memcg: introduce mem_cgroup_ptr Roman Gushchin
2019-10-18 0:28 ` [PATCH 02/16] mm: vmstat: use s32 for vm_node_stat_diff in struct per_cpu_nodestat Roman Gushchin
2019-10-20 22:44 ` Christopher Lameter
2019-10-21 1:15 ` Roman Gushchin
2019-10-21 18:09 ` Christopher Lameter
2019-10-20 22:51 ` Christopher Lameter
2019-10-21 1:21 ` Roman Gushchin
2019-10-18 0:28 ` [PATCH 03/16] mm: vmstat: convert slab vmstat counter to bytes Roman Gushchin
2019-10-18 0:28 ` [PATCH 04/16] mm: memcg/slab: allocate space for memcg ownership data for non-root slabs Roman Gushchin
2019-10-18 0:28 ` Roman Gushchin [this message]
2019-10-18 0:28 ` [PATCH 06/16] mm: memcg/slab: save memcg ownership data for non-root slab objects Roman Gushchin
2019-10-18 0:28 ` [PATCH 07/16] mm: memcg: move memcg_kmem_bypass() to memcontrol.h Roman Gushchin
2019-10-18 0:28 ` [PATCH 08/16] mm: memcg: introduce __mod_lruvec_memcg_state() Roman Gushchin
2019-10-18 0:28 ` [PATCH 09/16] mm: memcg/slab: charge individual slab objects instead of pages Roman Gushchin
2019-10-25 19:41 ` Johannes Weiner
2019-10-25 20:00 ` Roman Gushchin
2019-10-25 20:52 ` Johannes Weiner
2019-10-31 1:52 ` Roman Gushchin
2019-10-31 14:23 ` Johannes Weiner
2019-10-31 14:41 ` Johannes Weiner
2019-10-31 15:07 ` Roman Gushchin
2019-10-31 18:50 ` Johannes Weiner
2019-10-18 0:28 ` [PATCH 10/16] mm: memcg: move get_mem_cgroup_from_current() to memcontrol.h Roman Gushchin
2019-10-18 0:28 ` [PATCH 11/16] mm: memcg/slab: replace memcg_from_slab_page() with memcg_from_slab_obj() Roman Gushchin
2019-10-18 0:28 ` [PATCH 13/16] mm: memcg/slab: deprecate memory.kmem.slabinfo Roman Gushchin
2019-10-18 0:28 ` [PATCH 14/16] mm: memcg/slab: use one set of kmem_caches for all memory cgroups Roman Gushchin
2019-10-18 0:28 ` [PATCH 15/16] tools/cgroup: make slabinfo.py compatible with new slab controller Roman Gushchin
2019-10-18 0:28 ` [PATCH 16/16] mm: slab: remove redundant check in memcg_accumulate_slabinfo() Roman Gushchin
2019-10-18 17:03 ` [PATCH 00/16] The new slab memory controller Waiman Long
2019-10-18 17:12 ` Roman Gushchin
2019-10-22 13:22 ` Michal Hocko
2019-10-22 13:28 ` Michal Hocko
2019-10-22 15:48 ` Roman Gushchin
2019-10-22 13:31 ` Michal Hocko
2019-10-22 15:59 ` Roman Gushchin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20191018002820.307763-6-guro@fb.com \
--to=guro@fb.com \
--cc=cl@linux.com \
--cc=hannes@cmpxchg.org \
--cc=kernel-team@fb.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=longman@redhat.com \
--cc=mhocko@kernel.org \
--cc=shakeelb@google.com \
--cc=vdavydov.dev@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).