From: Yonghong Song <yhs@fb.com>
To: Yosry Ahmed <yosryahmed@google.com>
Cc: "Alexei Starovoitov" <ast@kernel.org>,
"Daniel Borkmann" <daniel@iogearbox.net>,
"Andrii Nakryiko" <andrii@kernel.org>,
"Martin KaFai Lau" <kafai@fb.com>,
"Song Liu" <songliubraving@fb.com>, "Hao Luo" <haoluo@google.com>,
"Tejun Heo" <tj@kernel.org>, "Zefan Li" <lizefan.x@bytedance.com>,
"Johannes Weiner" <hannes@cmpxchg.org>,
"Shuah Khan" <shuah@kernel.org>,
"Michal Hocko" <mhocko@kernel.org>,
"KP Singh" <kpsingh@kernel.org>,
"Benjamin Tissoires" <benjamin.tissoires@redhat.com>,
"John Fastabend" <john.fastabend@gmail.com>,
"Michal Koutný" <mkoutny@suse.com>,
"Roman Gushchin" <roman.gushchin@linux.dev>,
"David Rientjes" <rientjes@google.com>,
"Stanislav Fomichev" <sdf@google.com>,
"Greg Thelen" <gthelen@google.com>,
"Shakeel Butt" <shakeelb@google.com>,
"Linux Kernel Mailing List" <linux-kernel@vger.kernel.org>,
Networking <netdev@vger.kernel.org>, bpf <bpf@vger.kernel.org>,
Cgroups <cgroups@vger.kernel.org>
Subject: Re: [PATCH bpf-next v3 8/8] bpf: add a selftest for cgroup hierarchical stats collection
Date: Tue, 19 Jul 2022 09:17:37 -0700 [thread overview]
Message-ID: <670999ab-69c8-9692-4d73-da4f96c63e64@fb.com> (raw)
In-Reply-To: <CAJD7tkb8-scb1sstre0LRhY3dgfUJhGvSR=DgEqfwcVtBwb+5w@mail.gmail.com>
On 7/18/22 12:34 PM, Yosry Ahmed wrote:
> On Mon, Jul 11, 2022 at 8:55 PM Yosry Ahmed <yosryahmed@google.com> wrote:
>>
>> On Sun, Jul 10, 2022 at 5:51 PM Yonghong Song <yhs@fb.com> wrote:
>>>
>>>
>>>
>>> On 7/10/22 5:26 PM, Yonghong Song wrote:
>>>>
>>>>
>>>> On 7/8/22 5:04 PM, Yosry Ahmed wrote:
>>>>> Add a selftest that tests the whole workflow for collecting,
>>>>> aggregating (flushing), and displaying cgroup hierarchical stats.
>>>>>
>>>>> TL;DR:
>>>>> - Userspace program creates a cgroup hierarchy and induces memcg reclaim
>>>>> in parts of it.
>>>>> - Whenever reclaim happens, vmscan_start and vmscan_end update
>>>>> per-cgroup percpu readings, and tell rstat which (cgroup, cpu) pairs
>>>>> have updates.
>>>>> - When userspace tries to read the stats, vmscan_dump calls rstat to
>>>>> flush
>>>>> the stats, and outputs the stats in text format to userspace (similar
>>>>> to cgroupfs stats).
>>>>> - rstat calls vmscan_flush once for every (cgroup, cpu) pair that has
>>>>> updates, vmscan_flush aggregates cpu readings and propagates updates
>>>>> to parents.
>>>>> - Userspace program makes sure the stats are aggregated and read
>>>>> correctly.
>>>>>
>>>>> Detailed explanation:
>>>>> - The test loads tracing bpf programs, vmscan_start and vmscan_end, to
>>>>> measure the latency of cgroup reclaim. Per-cgroup readings are
>>>>> stored in
>>>>> percpu maps for efficiency. When a cgroup reading is updated on a cpu,
>>>>> cgroup_rstat_updated(cgroup, cpu) is called to add the cgroup to the
>>>>> rstat updated tree on that cpu.
>>>>>
>>>>> - A cgroup_iter program, vmscan_dump, is loaded and pinned to a file, for
>>>>> each cgroup. Reading this file invokes the program, which calls
>>>>> cgroup_rstat_flush(cgroup) to ask rstat to propagate the updates
>>>>> for all
>>>>> cpus and cgroups that have updates in this cgroup's subtree.
>>>>> Afterwards,
>>>>> the stats are exposed to the user. vmscan_dump returns 1 to terminate
>>>>> iteration early, so that we only expose stats for one cgroup per read.
>>>>>
>>>>> - An ftrace program, vmscan_flush, is also loaded and attached to
>>>>> bpf_rstat_flush. When rstat flushing is ongoing, vmscan_flush is
>>>>> invoked
>>>>> once for each (cgroup, cpu) pair that has updates. cgroups are popped
>>>>> from the rstat tree in a bottom-up fashion, so calls will always be
>>>>> made for cgroups that have updates before their parents. The program
>>>>> aggregates percpu readings to a total per-cgroup reading, and also
>>>>> propagates them to the parent cgroup. After rstat flushing is over,
>>>>> all
>>>>> cgroups will have correct updated hierarchical readings (including all
>>>>> cpus and all their descendants).
>>>>>
>>>>> - Finally, the test creates a cgroup hierarchy and induces memcg reclaim
>>>>> in parts of it, and makes sure that the stats collection, aggregation,
>>>>> and reading workflow works as expected.
>>>>>
>>>>> Signed-off-by: Yosry Ahmed <yosryahmed@google.com>
>>>>> ---
>>>>> .../prog_tests/cgroup_hierarchical_stats.c | 362 ++++++++++++++++++
>>>>> .../bpf/progs/cgroup_hierarchical_stats.c | 235 ++++++++++++
>>>>> 2 files changed, 597 insertions(+)
>>>>> create mode 100644
>>>>> tools/testing/selftests/bpf/prog_tests/cgroup_hierarchical_stats.c
>>>>> create mode 100644
>>>>> tools/testing/selftests/bpf/progs/cgroup_hierarchical_stats.c
>>>>>
>>>> [...]
>>>>> +
>>>>> +static unsigned long long get_cgroup_vmscan_delay(unsigned long long
>>>>> cgroup_id,
>>>>> + const char *file_name)
>>>>> +{
>>>>> + char buf[128], path[128];
>>>>> + unsigned long long vmscan = 0, id = 0;
>>>>> + int err;
>>>>> +
>>>>> + /* For every cgroup, read the file generated by cgroup_iter */
>>>>> + snprintf(path, 128, "%s%s", BPFFS_VMSCAN, file_name);
>>>>> + err = read_from_file(path, buf, 128);
>>>>> + if (!ASSERT_OK(err, "read cgroup_iter"))
>>>>> + return 0;
>>>>> +
>>>>> + /* Check the output file formatting */
>>>>> + ASSERT_EQ(sscanf(buf, "cg_id: %llu, total_vmscan_delay: %llu\n",
>>>>> + &id, &vmscan), 2, "output format");
>>>>> +
>>>>> + /* Check that the cgroup_id is displayed correctly */
>>>>> + ASSERT_EQ(id, cgroup_id, "cgroup_id");
>>>>> + /* Check that the vmscan reading is non-zero */
>>>>> + ASSERT_GT(vmscan, 0, "vmscan_reading");
>>>>> + return vmscan;
>>>>> +}
>>>>> +
>>>>> +static void check_vmscan_stats(void)
>>>>> +{
>>>>> + int i;
>>>>> + unsigned long long vmscan_readings[N_CGROUPS], vmscan_root;
>>>>> +
>>>>> + for (i = 0; i < N_CGROUPS; i++)
>>>>> + vmscan_readings[i] = get_cgroup_vmscan_delay(cgroups[i].id,
>>>>> + cgroups[i].name);
>>>>> +
>>>>> + /* Read stats for root too */
>>>>> + vmscan_root = get_cgroup_vmscan_delay(CG_ROOT_ID, CG_ROOT_NAME);
>>>>> +
>>>>> + /* Check that child1 == child1_1 + child1_2 */
>>>>> + ASSERT_EQ(vmscan_readings[1], vmscan_readings[3] +
>>>>> vmscan_readings[4],
>>>>> + "child1_vmscan");
>>>>> + /* Check that child2 == child2_1 + child2_2 */
>>>>> + ASSERT_EQ(vmscan_readings[2], vmscan_readings[5] +
>>>>> vmscan_readings[6],
>>>>> + "child2_vmscan");
>>>>> + /* Check that test == child1 + child2 */
>>>>> + ASSERT_EQ(vmscan_readings[0], vmscan_readings[1] +
>>>>> vmscan_readings[2],
>>>>> + "test_vmscan");
>>>>> + /* Check that root >= test */
>>>>> + ASSERT_GE(vmscan_root, vmscan_readings[1], "root_vmscan");
>>>>
>>>> I still get a test failure with
>>>>
>>>> get_cgroup_vmscan_delay:PASS:cgroup_id 0 nsec
>>>> get_cgroup_vmscan_delay:FAIL:vmscan_reading unexpected vmscan_reading:
>>>> actual 0 <= expected 0
>>>> check_vmscan_stats:FAIL:child1_vmscan unexpected child1_vmscan: actual 0
>>>> != expected -2
>>>> check_vmscan_stats:FAIL:child2_vmscan unexpected child2_vmscan: actual 0
>>>> != expected -2
>>>> check_vmscan_stats:PASS:test_vmscan 0 nsec
>>>> check_vmscan_stats:PASS:root_vmscan 0 nsec
>>>>
>>>> I added 'dump_stack()' in function try_to_free_mem_cgroup_pages()
>>>> and run this test (#33) and didn't get any stacktrace.
>>>> But I do get stacktraces due to other operations like
>>>> try_to_free_mem_cgroup_pages+0x1fd [kernel]
>>>> try_to_free_mem_cgroup_pages+0x1fd [kernel]
>>>> memory_reclaim_write+0x88 [kernel]
>>>> cgroup_file_write+0x88 [kernel]
>>>> kernfs_fop_write_iter+0xd0 [kernel]
>>>> vfs_write+0x2c4 [kernel]
>>>> __x64_sys_write+0x60 [kernel]
>>>> do_syscall_64+0x2d [kernel]
>>>> entry_SYSCALL_64_after_hwframe+0x44 [kernel]
>>>>
>>>> If you can show me the stacktrace about how
>>>> try_to_free_mem_cgroup_pages() is triggered in your setup, I can
>>>> help debug this problem in my environment.
>>>
>>> BTW, CI also reported the test failure.
>>> https://github.com/kernel-patches/bpf/pull/3284
>>>
>>> For example, with gcc built kernel,
>>> https://github.com/kernel-patches/bpf/runs/7272407890?check_suite_focus=true
>>>
>>> The error:
>>>
>>> get_cgroup_vmscan_delay:PASS:cgroup_id 0 nsec
>>> get_cgroup_vmscan_delay:PASS:vmscan_reading 0 nsec
>>> check_vmscan_stats:FAIL:child1_vmscan unexpected child1_vmscan:
>>> actual 28390910 != expected 28390909
>>> check_vmscan_stats:FAIL:child2_vmscan unexpected child2_vmscan:
>>> actual 0 != expected -2
>>> check_vmscan_stats:PASS:test_vmscan 0 nsec
>>> check_vmscan_stats:PASS:root_vmscan 0 nsec
>>>
>>
>> Hey Yonghong,
>>
>> Thanks for helping us debug this failure. I can reproduce the CI
>> failure in my enviornment, but this failure is actually different from
>> the failure in your environment. In your environment it looks like no
>> stats are gathered for all cgroups (either no reclaim happening or bpf
>> progs not being run). In the CI and in my environment, only one cgroup
>> observes this behavior.
>>
>> The thing is, I was able to reproduce the problem only when I ran all
>> test_progs. When I run the selftest alone (test_progs -t
>> cgroup_hierarchical_stats), it consistently passes, which is
>> interesting.
>
> I think I figured this one out (the CI failure). I set max_entries for
> the maps in the test to 10, because I have 1 entry per-cgroup, and I
> have less than 10 cgroups. When I run the test with other tests I
> *think* there are other cgroups that are being created, so the number
> exceeds 10, and some of the entries for the test cgroups cannot be
> created. I saw a lot of "failed to create entry for cgroup.." message
> in the bpf trace produced by my test, and the error turned out to be
> -E2BIG. I increased max_entries to 100 and it seems to be consistently
> passing when run with all the other tests, using both test_progs and
> test_progs-no_alu32.
>
> Please find a diff attached fixing this problem and a few other nits:
> - Return meaningful exit codes from the reclaimer() child process and
> check them in induce_vmscan().
> - Make buf and path variables static in get_cgroup_vmscan_delay()
> - Print error code in bpf trace when we fail to create a bpf map entry.
> - Print 0 instead of -1 when we can't find a map entry, to avoid
> underflowing the unsigned counters in the test.
>
> Let me know if this diff works or not, and if I need to send a new
> version with the diff or not. Also let me know if this fixes the
> failures that you have been seeing locally (which looked different
> from the CI failures).
I tried this patch and the test passed in my local environment
so the diff sounds good to me.
>
> Thanks!
>
>>
>> Anyway, one failure at a time :) I am working on debugging the CI
>> failure (that occurs only when all tests are run), then we'll see if
>> fixing that fixes the problem in our environment as well.
>>
>> If you have any pointers about why a test would consistently pass
>> alone and consistently fail with others that would be good. Otherwise,
>> I will keep you updated with any findings I reach.
>>
>> Thanks again!
>>
>>>>
>>>>> +}
>>>>> +
>>>>> +static int setup_cgroup_iter(struct cgroup_hierarchical_stats *obj,
>>>>> int cgroup_fd,
>>>> [...]
next prev parent reply other threads:[~2022-07-19 16:18 UTC|newest]
Thread overview: 28+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-07-09 0:04 [PATCH bpf-next v3 0/8] bpf: rstat: cgroup hierarchical stats Yosry Ahmed
2022-07-09 0:04 ` [PATCH bpf-next v3 1/8] btf: Add a new kfunc set which allows to mark a function to be sleepable Yosry Ahmed
2022-07-09 0:04 ` [PATCH bpf-next v3 2/8] cgroup: enable cgroup_get_from_file() on cgroup1 Yosry Ahmed
2022-07-09 0:04 ` [PATCH bpf-next v3 3/8] bpf, iter: Fix the condition on p when calling stop Yosry Ahmed
2022-07-09 0:04 ` [PATCH bpf-next v3 4/8] bpf: Introduce cgroup iter Yosry Ahmed
2022-07-11 0:19 ` Yonghong Song
2022-07-11 23:20 ` Yonghong Song
2022-07-12 0:42 ` Hao Luo
2022-07-12 3:45 ` Yonghong Song
2022-07-21 0:40 ` Hao Luo
2022-07-21 16:15 ` Yonghong Song
2022-07-21 17:21 ` Hao Luo
2022-07-21 18:15 ` Yonghong Song
2022-07-21 21:07 ` Hao Luo
2022-07-09 0:04 ` [PATCH bpf-next v3 5/8] selftests/bpf: Test cgroup_iter Yosry Ahmed
2022-07-09 0:04 ` [PATCH bpf-next v3 6/8] cgroup: bpf: enable bpf programs to integrate with rstat Yosry Ahmed
2022-07-09 0:04 ` [PATCH bpf-next v3 7/8] selftests/bpf: extend cgroup helpers Yosry Ahmed
2022-07-09 0:04 ` [PATCH bpf-next v3 8/8] bpf: add a selftest for cgroup hierarchical stats collection Yosry Ahmed
2022-07-11 0:26 ` Yonghong Song
2022-07-11 0:51 ` Yonghong Song
2022-07-11 6:01 ` Hao Luo
2022-07-11 6:19 ` Yonghong Song
2022-07-12 0:44 ` Hao Luo
2022-07-12 3:55 ` Yosry Ahmed
2022-07-18 19:34 ` Yosry Ahmed
2022-07-19 0:19 ` Hao Luo
2022-07-19 16:17 ` Yonghong Song [this message]
2022-07-19 17:02 ` Yosry Ahmed
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=670999ab-69c8-9692-4d73-da4f96c63e64@fb.com \
--to=yhs@fb.com \
--cc=andrii@kernel.org \
--cc=ast@kernel.org \
--cc=benjamin.tissoires@redhat.com \
--cc=bpf@vger.kernel.org \
--cc=cgroups@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=gthelen@google.com \
--cc=hannes@cmpxchg.org \
--cc=haoluo@google.com \
--cc=john.fastabend@gmail.com \
--cc=kafai@fb.com \
--cc=kpsingh@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=lizefan.x@bytedance.com \
--cc=mhocko@kernel.org \
--cc=mkoutny@suse.com \
--cc=netdev@vger.kernel.org \
--cc=rientjes@google.com \
--cc=roman.gushchin@linux.dev \
--cc=sdf@google.com \
--cc=shakeelb@google.com \
--cc=shuah@kernel.org \
--cc=songliubraving@fb.com \
--cc=tj@kernel.org \
--cc=yosryahmed@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).