From: Daniel Jordan <daniel.m.jordan@oracle.com> To: Alex Shi <alex.shi@linux.alibaba.com> Cc: Daniel Jordan <daniel.m.jordan@oracle.com>, Hugh Dickins <hughd@google.com>, Andrew Morton <akpm@linux-foundation.org>, mgorman@techsingularity.net, tj@kernel.org, khlebnikov@yandex-team.ru, willy@infradead.org, hannes@cmpxchg.org, lkp@intel.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, shakeelb@google.com, iamjoonsoo.kim@lge.com, richard.weiyang@gmail.com, kirill@shutemov.name, alexander.duyck@gmail.com, rong.a.chen@intel.com, mhocko@suse.com, vdavydov.dev@gmail.com, shy828301@gmail.com Subject: Re: [PATCH v18 00/32] per memcg lru_lock Date: Thu, 27 Aug 2020 21:40:22 -0400 [thread overview] Message-ID: <20200828014022.y5xju6weysqpzxd2@ca-dmjordan1.us.oracle.com> (raw) In-Reply-To: <01ed6e45-3853-dcba-61cb-b429a49a7572@linux.alibaba.com> On Wed, Aug 26, 2020 at 04:59:28PM +0800, Alex Shi wrote: > I clean up my testing and make it reproducable by a Dockerfile and a case patch which > attached. Ok, I'll give that a shot once I've taken care of sysbench. > >>> Even better would be a description of the problem you're having in production > >>> with lru_lock. We might be able to create at least a simulation of it to show > >>> what the expected improvement of your real workload is. > >> > >> we are using thousands memcgs in a machine, but as a simulation, I guess above case > >> could be helpful to show the problem. > > > > Using thousands of memcgs to do what? Any particulars about the type of > > workload? Surely it's more complicated than page cache reads :) > > Yes, the workload are quit different on different business, some use cpu a > lot, some use memory a lot, and some are may mixed. That's pretty vague, but I don't suppose I could do much better describing what all runs on our systems :-/ I went back to your v1 post to see what motivated you originally, and you had some results from aim9 but nothing about where this reared its head in the first place. How did you discover the bottleneck? I'm just curious about how lru_lock hurts in practice. > > Neither kernel compile nor git checkout in the root cgroup changed much, just > > 0.31% slower on elapsed time for the compile, so no significant regressions > > there. Now for sysbench again. Still working on getting repeatable sysbench runs, no luck so far. The numbers have stayed fairly consistent with your series but vary a lot on the base kernel, not sure why yet.
WARNING: multiple messages have this Message-ID (diff)
From: Daniel Jordan <daniel.m.jordan-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> To: Alex Shi <alex.shi-KPsoFbNs7GizrGE5bRqYAgC/G2K4zDHf@public.gmane.org> Cc: Daniel Jordan <daniel.m.jordan-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>, Hugh Dickins <hughd-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org>, Andrew Morton <akpm-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b@public.gmane.org>, mgorman-3eNAlZScCAx27rWaFMvyedHuzzzSOjJt@public.gmane.org, tj-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org, khlebnikov-XoJtRXgx1JseBXzfvpsJ4g@public.gmane.org, willy-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org, hannes-druUgvl0LCNAfugRpC6u6w@public.gmane.org, lkp-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org, linux-mm-Bw31MaZKKs3YtjvyW6yDsg@public.gmane.org, linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, cgroups-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, shakeelb-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org, iamjoonsoo.kim-Hm3cg6mZ9cc@public.gmane.org, richard.weiyang-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org, kirill-oKw7cIdHH8eLwutG50LtGA@public.gmane.org, alexander.duyck-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org, rong.a.chen-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org, mhocko-IBi9RG/b67k@public.gmane.org, vdavydov.dev-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org, shy828301-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org Subject: Re: [PATCH v18 00/32] per memcg lru_lock Date: Thu, 27 Aug 2020 21:40:22 -0400 [thread overview] Message-ID: <20200828014022.y5xju6weysqpzxd2@ca-dmjordan1.us.oracle.com> (raw) In-Reply-To: <01ed6e45-3853-dcba-61cb-b429a49a7572-KPsoFbNs7GizrGE5bRqYAgC/G2K4zDHf@public.gmane.org> On Wed, Aug 26, 2020 at 04:59:28PM +0800, Alex Shi wrote: > I clean up my testing and make it reproducable by a Dockerfile and a case patch which > attached. Ok, I'll give that a shot once I've taken care of sysbench. > >>> Even better would be a description of the problem you're having in production > >>> with lru_lock. We might be able to create at least a simulation of it to show > >>> what the expected improvement of your real workload is. > >> > >> we are using thousands memcgs in a machine, but as a simulation, I guess above case > >> could be helpful to show the problem. > > > > Using thousands of memcgs to do what? Any particulars about the type of > > workload? Surely it's more complicated than page cache reads :) > > Yes, the workload are quit different on different business, some use cpu a > lot, some use memory a lot, and some are may mixed. That's pretty vague, but I don't suppose I could do much better describing what all runs on our systems :-/ I went back to your v1 post to see what motivated you originally, and you had some results from aim9 but nothing about where this reared its head in the first place. How did you discover the bottleneck? I'm just curious about how lru_lock hurts in practice. > > Neither kernel compile nor git checkout in the root cgroup changed much, just > > 0.31% slower on elapsed time for the compile, so no significant regressions > > there. Now for sysbench again. Still working on getting repeatable sysbench runs, no luck so far. The numbers have stayed fairly consistent with your series but vary a lot on the base kernel, not sure why yet.
next prev parent reply other threads:[~2020-08-28 1:32 UTC|newest] Thread overview: 201+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-08-24 12:54 [PATCH v18 00/32] per memcg lru_lock Alex Shi 2020-08-24 12:54 ` [PATCH v18 01/32] mm/memcg: warning on !memcg after readahead page charged Alex Shi 2020-08-24 12:54 ` Alex Shi 2020-08-24 12:54 ` [PATCH v18 02/32] mm/memcg: bail out early from swap accounting when memcg is disabled Alex Shi 2020-08-24 12:54 ` [PATCH v18 03/32] mm/thp: move lru_add_page_tail func to huge_memory.c Alex Shi 2020-08-24 12:54 ` Alex Shi 2020-08-24 12:54 ` [PATCH v18 04/32] mm/thp: clean up lru_add_page_tail Alex Shi 2020-08-24 12:54 ` [PATCH v18 05/32] mm/thp: remove code path which never got into Alex Shi 2020-08-24 12:54 ` Alex Shi 2020-08-24 12:54 ` [PATCH v18 06/32] mm/thp: narrow lru locking Alex Shi 2020-09-10 13:49 ` Matthew Wilcox 2020-09-10 13:49 ` Matthew Wilcox 2020-09-11 3:37 ` Alex Shi 2020-09-11 3:37 ` Alex Shi 2020-09-13 15:27 ` Matthew Wilcox 2020-09-13 15:27 ` Matthew Wilcox 2020-09-19 1:00 ` Hugh Dickins 2020-09-19 1:00 ` Hugh Dickins 2020-08-24 12:54 ` [PATCH v18 07/32] mm/swap.c: stop deactivate_file_page if page not on lru Alex Shi 2020-08-24 12:54 ` Alex Shi 2020-08-24 12:54 ` [PATCH v18 08/32] mm/vmscan: remove unnecessary lruvec adding Alex Shi 2020-08-24 12:54 ` Alex Shi 2020-08-24 12:54 ` [PATCH v18 09/32] mm/page_idle: no unlikely double check for idle page counting Alex Shi 2020-08-24 12:54 ` Alex Shi 2020-08-24 12:54 ` [PATCH v18 10/32] mm/compaction: rename compact_deferred as compact_should_defer Alex Shi 2020-08-24 12:54 ` [PATCH v18 11/32] mm/memcg: add debug checking in lock_page_memcg Alex Shi 2020-08-24 12:54 ` [PATCH v18 12/32] mm/memcg: optimize mem_cgroup_page_lruvec Alex Shi 2020-08-24 12:54 ` [PATCH v18 13/32] mm/swap.c: fold vm event PGROTATED into pagevec_move_tail_fn Alex Shi 2020-08-24 12:54 ` Alex Shi 2020-08-24 12:54 ` [PATCH v18 14/32] mm/lru: move lru_lock holding in func lru_note_cost_page Alex Shi 2020-08-24 12:54 ` [PATCH v18 15/32] mm/lru: move lock into lru_note_cost Alex Shi 2020-09-21 21:36 ` Hugh Dickins 2020-09-21 21:36 ` Hugh Dickins 2020-09-21 21:36 ` Hugh Dickins 2020-09-21 22:03 ` Hugh Dickins 2020-09-21 22:03 ` Hugh Dickins 2020-09-22 3:39 ` Alex Shi 2020-09-22 3:39 ` Alex Shi 2020-09-22 3:38 ` Alex Shi 2020-09-22 3:38 ` Alex Shi 2020-08-24 12:54 ` [PATCH v18 16/32] mm/lru: introduce TestClearPageLRU Alex Shi 2020-08-24 12:54 ` Alex Shi 2020-09-21 23:16 ` Hugh Dickins 2020-09-21 23:16 ` Hugh Dickins 2020-09-21 23:16 ` Hugh Dickins 2020-09-22 3:53 ` Alex Shi 2020-09-22 3:53 ` Alex Shi 2020-08-24 12:54 ` [PATCH v18 17/32] mm/compaction: do page isolation first in compaction Alex Shi 2020-09-21 23:49 ` Hugh Dickins 2020-09-21 23:49 ` Hugh Dickins 2020-09-22 4:57 ` Alex Shi 2020-09-22 4:57 ` Alex Shi 2020-08-24 12:54 ` [PATCH v18 18/32] mm/thp: add tail pages into lru anyway in split_huge_page() Alex Shi 2020-08-24 12:54 ` Alex Shi 2020-08-24 12:54 ` [PATCH v18 19/32] mm/swap.c: serialize memcg changes in pagevec_lru_move_fn Alex Shi 2020-09-22 0:42 ` Hugh Dickins 2020-09-22 0:42 ` Hugh Dickins 2020-09-22 0:42 ` Hugh Dickins 2020-09-22 5:00 ` Alex Shi 2020-09-22 5:00 ` Alex Shi 2020-08-24 12:54 ` [PATCH v18 20/32] mm/lru: replace pgdat lru_lock with lruvec lock Alex Shi 2020-09-22 5:27 ` Hugh Dickins 2020-09-22 5:27 ` Hugh Dickins 2020-09-22 5:27 ` Hugh Dickins 2020-09-22 8:58 ` Alex Shi 2020-09-22 8:58 ` Alex Shi 2020-08-24 12:54 ` [PATCH v18 21/32] mm/lru: introduce the relock_page_lruvec function Alex Shi 2020-09-22 5:40 ` Hugh Dickins 2020-09-22 5:40 ` Hugh Dickins 2020-09-22 5:40 ` Hugh Dickins 2020-08-24 12:54 ` [PATCH v18 22/32] mm/vmscan: use relock for move_pages_to_lru Alex Shi 2020-09-22 5:44 ` Hugh Dickins 2020-09-22 5:44 ` Hugh Dickins 2020-09-23 1:55 ` Alex Shi 2020-09-23 1:55 ` Alex Shi 2020-08-24 12:54 ` [PATCH v18 23/32] mm/lru: revise the comments of lru_lock Alex Shi 2020-08-24 12:54 ` Alex Shi 2020-09-22 5:48 ` Hugh Dickins 2020-09-22 5:48 ` Hugh Dickins 2020-08-24 12:54 ` [PATCH v18 24/32] mm/pgdat: remove pgdat lru_lock Alex Shi 2020-09-22 5:53 ` Hugh Dickins 2020-09-22 5:53 ` Hugh Dickins 2020-09-23 1:55 ` Alex Shi 2020-09-23 1:55 ` Alex Shi 2020-08-24 12:54 ` [PATCH v18 25/32] mm/mlock: remove lru_lock on TestClearPageMlocked in munlock_vma_page Alex Shi 2020-08-24 12:54 ` Alex Shi 2020-08-26 5:52 ` Alex Shi 2020-08-26 5:52 ` Alex Shi 2020-09-22 6:13 ` Hugh Dickins 2020-09-22 6:13 ` Hugh Dickins 2020-09-22 6:13 ` Hugh Dickins 2020-09-23 1:58 ` Alex Shi 2020-09-23 1:58 ` Alex Shi 2020-08-24 12:54 ` [PATCH v18 26/32] mm/mlock: remove __munlock_isolate_lru_page Alex Shi 2020-08-24 12:54 ` Alex Shi 2020-08-24 12:55 ` [PATCH v18 27/32] mm/swap.c: optimizing __pagevec_lru_add lru_lock Alex Shi 2020-08-24 12:55 ` Alex Shi 2020-08-26 9:07 ` Alex Shi 2020-08-24 12:55 ` [PATCH v18 28/32] mm/compaction: Drop locked from isolate_migratepages_block Alex Shi 2020-08-24 12:55 ` [PATCH v18 29/32] mm: Identify compound pages sooner in isolate_migratepages_block Alex Shi 2020-08-24 12:55 ` Alex Shi 2020-08-24 12:55 ` [PATCH v18 30/32] mm: Drop use of test_and_set_skip in favor of just setting skip Alex Shi 2020-08-24 12:55 ` [PATCH v18 31/32] mm: Add explicit page decrement in exception path for isolate_lru_pages Alex Shi 2020-08-24 12:55 ` Alex Shi 2020-09-09 1:01 ` Matthew Wilcox 2020-09-09 1:01 ` Matthew Wilcox 2020-09-09 15:43 ` Alexander Duyck 2020-09-09 15:43 ` Alexander Duyck 2020-09-09 15:43 ` Alexander Duyck 2020-09-09 17:07 ` Matthew Wilcox 2020-09-09 17:07 ` Matthew Wilcox 2020-09-09 18:24 ` Hugh Dickins 2020-09-09 18:24 ` Hugh Dickins 2020-09-09 18:24 ` Hugh Dickins 2020-09-09 20:15 ` Matthew Wilcox 2020-09-09 20:15 ` Matthew Wilcox 2020-09-09 21:05 ` Hugh Dickins 2020-09-09 21:05 ` Hugh Dickins 2020-09-09 21:05 ` Hugh Dickins 2020-09-09 21:17 ` Alexander Duyck 2020-09-09 21:17 ` Alexander Duyck 2020-09-09 21:17 ` Alexander Duyck 2020-08-24 12:55 ` [PATCH v18 32/32] mm: Split release_pages work into 3 passes Alex Shi 2020-08-24 12:55 ` Alex Shi 2020-08-24 18:42 ` [PATCH v18 00/32] per memcg lru_lock Andrew Morton 2020-08-24 19:50 ` Qian Cai 2020-08-24 19:50 ` Qian Cai 2020-08-24 20:24 ` Hugh Dickins 2020-08-24 20:24 ` Hugh Dickins 2020-08-24 20:24 ` Hugh Dickins 2020-08-25 1:56 ` Daniel Jordan 2020-08-25 1:56 ` Daniel Jordan 2020-08-25 3:26 ` Alex Shi 2020-08-25 3:26 ` Alex Shi 2020-08-25 11:39 ` Matthew Wilcox 2020-08-25 11:39 ` Matthew Wilcox 2020-08-26 1:19 ` Daniel Jordan 2020-08-26 8:59 ` Alex Shi 2020-08-26 8:59 ` Alex Shi 2020-08-28 1:40 ` Daniel Jordan [this message] 2020-08-28 1:40 ` Daniel Jordan 2020-08-28 5:22 ` Alex Shi 2020-08-28 5:22 ` Alex Shi 2020-09-09 2:44 ` Aaron Lu 2020-09-09 2:44 ` Aaron Lu 2020-09-09 11:40 ` Michal Hocko 2020-08-25 8:52 ` Alex Shi 2020-08-25 8:52 ` Alex Shi 2020-08-25 13:00 ` Alex Shi 2020-08-25 13:00 ` Alex Shi 2020-08-27 7:01 ` Hugh Dickins 2020-08-27 7:01 ` Hugh Dickins 2020-08-27 12:20 ` Race between freeing and waking page Matthew Wilcox 2020-09-08 23:41 ` [PATCH v18 00/32] per memcg lru_lock: reviews Hugh Dickins 2020-09-08 23:41 ` Hugh Dickins 2020-09-09 2:24 ` Wei Yang 2020-09-09 2:24 ` Wei Yang 2020-09-09 15:08 ` Alex Shi 2020-09-09 15:08 ` Alex Shi 2020-09-09 23:16 ` Hugh Dickins 2020-09-09 23:16 ` Hugh Dickins 2020-09-11 2:50 ` Alex Shi 2020-09-11 2:50 ` Alex Shi 2020-09-12 2:13 ` Hugh Dickins 2020-09-12 2:13 ` Hugh Dickins 2020-09-13 14:21 ` Alex Shi 2020-09-13 14:21 ` Alex Shi 2020-09-15 8:21 ` Hugh Dickins 2020-09-15 8:21 ` Hugh Dickins 2020-09-15 8:21 ` Hugh Dickins 2020-09-15 16:58 ` Daniel Jordan 2020-09-15 16:58 ` Daniel Jordan 2020-09-16 12:44 ` Alex Shi 2020-09-17 2:37 ` Alex Shi 2020-09-17 2:37 ` Alex Shi 2020-09-17 14:35 ` Daniel Jordan 2020-09-17 14:35 ` Daniel Jordan 2020-09-17 15:39 ` Alexander Duyck 2020-09-17 15:39 ` Alexander Duyck 2020-09-17 15:39 ` Alexander Duyck 2020-09-17 16:48 ` Daniel Jordan 2020-09-17 16:48 ` Daniel Jordan 2020-09-12 8:38 ` Hugh Dickins 2020-09-12 8:38 ` Hugh Dickins 2020-09-12 8:38 ` Hugh Dickins 2020-09-13 14:22 ` Alex Shi 2020-09-13 14:22 ` Alex Shi 2020-09-09 16:11 ` Alexander Duyck 2020-09-09 16:11 ` Alexander Duyck 2020-09-09 16:11 ` Alexander Duyck 2020-09-10 0:32 ` Hugh Dickins 2020-09-10 0:32 ` Hugh Dickins 2020-09-10 0:32 ` Hugh Dickins 2020-09-10 14:24 ` Alexander Duyck 2020-09-10 14:24 ` Alexander Duyck 2020-09-10 14:24 ` Alexander Duyck 2020-09-12 5:12 ` Hugh Dickins 2020-09-12 5:12 ` Hugh Dickins 2020-09-12 5:12 ` Hugh Dickins 2020-08-25 7:21 ` [PATCH v18 00/32] per memcg lru_lock Michal Hocko 2020-08-25 7:21 ` Michal Hocko
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20200828014022.y5xju6weysqpzxd2@ca-dmjordan1.us.oracle.com \ --to=daniel.m.jordan@oracle.com \ --cc=akpm@linux-foundation.org \ --cc=alex.shi@linux.alibaba.com \ --cc=alexander.duyck@gmail.com \ --cc=cgroups@vger.kernel.org \ --cc=hannes@cmpxchg.org \ --cc=hughd@google.com \ --cc=iamjoonsoo.kim@lge.com \ --cc=khlebnikov@yandex-team.ru \ --cc=kirill@shutemov.name \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=lkp@intel.com \ --cc=mgorman@techsingularity.net \ --cc=mhocko@suse.com \ --cc=richard.weiyang@gmail.com \ --cc=rong.a.chen@intel.com \ --cc=shakeelb@google.com \ --cc=shy828301@gmail.com \ --cc=tj@kernel.org \ --cc=vdavydov.dev@gmail.com \ --cc=willy@infradead.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.