From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0C9B8ECAA25 for ; Thu, 25 Aug 2022 18:53:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243534AbiHYSxB (ORCPT ); Thu, 25 Aug 2022 14:53:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57046 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243107AbiHYSw6 (ORCPT ); Thu, 25 Aug 2022 14:52:58 -0400 Received: from mail-qt1-x831.google.com (mail-qt1-x831.google.com [IPv6:2607:f8b0:4864:20::831]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BC0D94D263 for ; Thu, 25 Aug 2022 11:52:56 -0700 (PDT) Received: by mail-qt1-x831.google.com with SMTP id a4so15957473qto.10 for ; Thu, 25 Aug 2022 11:52:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc; bh=iYkms9FOcVHRMQ8nnVym8avn1yHBqqPQKbrXBh8EpAg=; b=ernArX/gsESCKDxGzFCn7QheTg6rO6ShblTP5Cc3wCPphLkfsifDo5R5ObXE3j6jZ4 ln6V0hLEnJTZPc8ImbAZswhYMmwAVOPk1VhxrwPwrpjsRZUzBoWEqKYiJmRDUONRvHv1 oUhM0dBrpZu53zByZiZy5ELiA4VPJiOaTveVlbCC2A4RzTkzwtZ6QiBQ0gmNlo/X+7Vr M8Qk2OVc/daWcWg20LJQhUUV92x9lbQ7Qvm+sLlvEEnsl0LasLz+/E1lU7Fk6DhGIstt 9+n4V18u6y6H2Nu8/zRffL51gE2h6C5P6ukRD5JKMtmD4Ui1R3bfBRxoF/l821Pu46oc 7I3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc; bh=iYkms9FOcVHRMQ8nnVym8avn1yHBqqPQKbrXBh8EpAg=; b=PEWKJd0qQAKQf8vEcS9lQ9gF/sIPd0BMZSLR3iaP3xoEvb/bTywidVfWmiEfN+/1Lm IQmoqFiegJTk/K8H6G3QF6dwgMmPgb7JRgFXZrksqVq0o6WhTJ3bM9/Q7f644Ny1sskx OuJmdNu2YuTGXXjymsEb2BNwGxxsMuqHKudUTL3/qyHO6pFh98aovuDKKwoTPVIiR+mY VcTF885EKWGM/YrUl1PkiKOdiMuuN8nRmuG+yTzLugmpw8OJKwXfAjZlXAQg3xwTNm2c dkzxuHp4F7oN8yznxu3SzC9U80zEkhnGmerkojQxloJHVPmwAbQrKsSpP08ynCt/YpkX fH1A== X-Gm-Message-State: ACgBeo1DEBBAJKogYoJQXDjAiffhtZ+0jlp9SXSpK8nD2UMhQLD2Awtm APAQc0OXnb39uPH7PM8f6ae4UGB94IGa7WM56jjKJQ== X-Google-Smtp-Source: AA6agR48ZvB5ActsSVjSq3vRtM4RfekkHdDEj6wknDLegitpHkTI1l0gJHvzsL78KJ1H+7AyvyZOIhoCN3zEWskpNYY= X-Received: by 2002:a05:622a:552:b0:342:f8c2:442 with SMTP id m18-20020a05622a055200b00342f8c20442mr4912166qtx.478.1661453575766; Thu, 25 Aug 2022 11:52:55 -0700 (PDT) MIME-Version: 1.0 References: <20220824233117.1312810-1-haoluo@google.com> <20220824233117.1312810-6-haoluo@google.com> In-Reply-To: From: Hao Luo Date: Thu, 25 Aug 2022 11:52:45 -0700 Message-ID: Subject: Re: [RESEND PATCH bpf-next v9 5/5] selftests/bpf: add a selftest for cgroup hierarchical stats collection To: Alexei Starovoitov Cc: Yosry Ahmed , LKML , bpf , "open list:CONTROL GROUP (CGROUP)" , Network Development , Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Martin KaFai Lau , Song Liu , Yonghong Song , Tejun Heo , Zefan Li , KP Singh , Johannes Weiner , Michal Hocko , John Fastabend , Jiri Olsa , Michal Koutny , Roman Gushchin , David Rientjes , Stanislav Fomichev , Shakeel Butt Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, Aug 25, 2022 at 11:43 AM Alexei Starovoitov wrote: > > On Wed, Aug 24, 2022 at 7:41 PM Yosry Ahmed wrote: > > > > On Wed, Aug 24, 2022 at 7:09 PM Alexei Starovoitov > > wrote: > > > > > > On Wed, Aug 24, 2022 at 4:31 PM Hao Luo wrote: > > > > + > > > > + for (i = 0; i < N_CGROUPS; i++) { > > > > + fd = create_and_get_cgroup(cgroups[i].path); > > > > + if (!ASSERT_GE(fd, 0, "create_and_get_cgroup")) > > > > + return fd; > > > > + > > > > + cgroups[i].fd = fd; > > > > + cgroups[i].id = get_cgroup_id(cgroups[i].path); > > > > + > > > > + /* > > > > + * Enable memcg controller for the entire hierarchy. > > > > + * Note that stats are collected for all cgroups in a hierarchy > > > > + * with memcg enabled anyway, but are only exposed for cgroups > > > > + * that have memcg enabled. > > > > + */ > > > > + if (i < N_NON_LEAF_CGROUPS) { > > > > + err = enable_controllers(cgroups[i].path, "memory"); > > > > + if (!ASSERT_OK(err, "enable_controllers")) > > > > + return err; > > > > + } > > > > + } > > > > > > It passes BPF CI, but fails in my setup with: > > > > > > # ./test_progs -t cgroup_hier -vv > > > bpf_testmod.ko is already unloaded. > > > Loading bpf_testmod.ko... > > > Successfully loaded bpf_testmod.ko. > > > setup_bpffs:PASS:mount 0 nsec > > > setup_cgroups:PASS:setup_cgroup_environment 0 nsec > > > setup_cgroups:PASS:get_root_cgroup 0 nsec > > > setup_cgroups:PASS:create_and_get_cgroup 0 nsec > > > (cgroup_helpers.c:92: errno: No such file or directory) Enabling > > > controller memory: > > > /mnt/cgroup-test-work-dir6526//test/cgroup.subtree_control > > > setup_cgroups:FAIL:enable_controllers unexpected error: 1 (errno 2) > > > cleanup_bpffs:FAIL:rmdir /sys/fs/bpf/vmscan/ unexpected error: -1 (errno 2) > > > #36 cgroup_hierarchical_stats:FAIL > > > Summary: 0/0 PASSED, 0 SKIPPED, 1 FAILED > > > > > > How do I debug it? > > > > The failure with ENOENT happens when we try to write "+memory" to > > /mnt/cgroup-test-work-dir6526//test/cgroup.subtree_control, not when > > we try to open it. So the file is there. AFAICT, ENOENT can be > > returned from this write if the memory controller is not enabled on > > this cgroup. > > > > In setup_cgroup_environment(), we should be enabling all available > > controllers on /mnt and /mnt/cgroup-test-work-dir6526 by this line: > > > > if (__enable_controllers(CGROUP_MOUNT_PATH, NULL) || > > __enable_controllers(cgroup_workdir, NULL)) > > return 1; > > > > The first thing that comes to mind is that maybe the memory controller > > is not enabled on your setup at all? Can you check > > /sys/fs/cgroup/cgroup.controllers (or wherever your global cgroup > > mount is)? > > Indeed. I didn't have a memory controller in cgroup2. > My system booted with cgroup v1 and it had cgroup1 memory > controller enabled which prevented cgroup2 to enable it. > Without Tejun's help I would have been able to figure this out. > > Anyway, pushed the set to bpf-next. Thanks everyone. Really awesome! Thanks everyone for the code review and the helpful comments! Yosry and I can now start playing this new tool in our production kernel. We will monitor for bugs and continue making further improvements.