From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 83803C433EF for ; Tue, 26 Apr 2022 14:58:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 11E706B0078; Tue, 26 Apr 2022 10:58:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0CC956B007B; Tue, 26 Apr 2022 10:58:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ED61C6B007E; Tue, 26 Apr 2022 10:58:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.a.hostedemail.com [64.99.140.24]) by kanga.kvack.org (Postfix) with ESMTP id DCBAC6B0078 for ; Tue, 26 Apr 2022 10:58:28 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id A10B0209B8 for ; Tue, 26 Apr 2022 14:58:28 +0000 (UTC) X-FDA: 79399336296.01.338F044 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf09.hostedemail.com (Postfix) with ESMTP id EB69F140043 for ; Tue, 26 Apr 2022 14:58:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1650985107; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GoYuxuTr0wU4xOtdPoGRbWQ9hkISKKeNkUzuaQojQ8I=; b=dgAUyDgKMjha+6DjTmVCNFwETPzNTywVixZ48TD63z6Vpc9gfyZ3EVrgCacZdoGxRTMY1h bdeMQHP1k1vxFHSzDIaV7fjWOjJodl/tXuk7YjUJYxO0V8DmDkBxZfTjzk2lb+Sno339uy WFnvSmV/MS65MyYCqMSKaAuwLraT9Us= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-139-VXxt-ww9N8qv-QtWSacfcQ-1; Tue, 26 Apr 2022 10:58:23 -0400 X-MC-Unique: VXxt-ww9N8qv-QtWSacfcQ-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4B2F0101AA45; Tue, 26 Apr 2022 14:58:22 +0000 (UTC) Received: from [10.18.17.215] (dhcp-17-215.bos.redhat.com [10.18.17.215]) by smtp.corp.redhat.com (Postfix) with ESMTP id BA44C15230A0; Tue, 26 Apr 2022 14:58:21 +0000 (UTC) Message-ID: Date: Tue, 26 Apr 2022 10:58:21 -0400 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.5.0 Subject: Re: [PATCH v2] cgroup/cpuset: Remove cpus_allowed/mems_allowed setup in cpuset_init_smp() Content-Language: en-US To: Feng Tang Cc: Tejun Heo , Zefan Li , Johannes Weiner , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrew Morton , Michal Hocko , Dave Hansen , ying.huang@intel.com, stable@vger.kernel.org References: <20220425155505.1292896-1-longman@redhat.com> <20220426032337.GA84190@shbuild999.sh.intel.com> From: Waiman Long In-Reply-To: <20220426032337.GA84190@shbuild999.sh.intel.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.85 on 10.11.54.7 Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=dgAUyDgK; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf09.hostedemail.com: domain of longman@redhat.com has no SPF policy when checking 170.10.129.124) smtp.mailfrom=longman@redhat.com X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: EB69F140043 X-Rspam-User: X-Stat-Signature: wpgqg6otgeenh3f6o1jhduednrywubbb X-HE-Tag: 1650985104-119102 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 4/25/22 23:23, Feng Tang wrote: > Hi Waiman, > > On Mon, Apr 25, 2022 at 11:55:05AM -0400, Waiman Long wrote: >> There are 3 places where the cpu and node masks of the top cpuset can >> be initialized in the order they are executed: >> 1) start_kernel -> cpuset_init() >> 2) start_kernel -> cgroup_init() -> cpuset_bind() >> 3) kernel_init_freeable() -> do_basic_setup() -> cpuset_init_smp() >> >> The first cpuset_init() function just sets all the bits in the masks. >> The last one executed is cpuset_init_smp() which sets up cpu and node >> masks suitable for v1, but not v2. cpuset_bind() does the right setup >> for both v1 and v2. >> >> For systems with cgroup v2 setup, cpuset_bind() is called once. For >> systems with cgroup v1 setup, cpuset_bind() is called twice. It is >> first called before cpuset_init_smp() in cgroup v2 mode. Then it is >> called again when cgroup v1 filesystem is mounted in v1 mode after >> cpuset_init_smp(). >> >> [ 2.609781] cpuset_bind() called - v2 = 1 >> [ 3.079473] cpuset_init_smp() called >> [ 7.103710] cpuset_bind() called - v2 = 0 > I run some test, on a server with centOS, this did happen that > cpuset_bind() is called twice, first as v2 during kernel boot, > and then as v1 post-boot. > > However on a QEMU running with a basic debian rootfs image, > the second call of cpuset_bind() didn't happen. The first time cpuset_bind() is called in cgroup_init(), the kernel doesn't know if userspace is going to mount v1 or v2 cgroup. By default, it is assumed to be v2. However, if userspace mounts the cgroup v1 filesystem for cpuset, cpuset_bind() will be run at this point by rebind_subsystem() to set up cgroup v1 environment and cpus_allowed/mems_allowed will be correctly set at this point. Mounting the cgroup v2 filesystem, however, does not cause rebind_subsystem() to run and hence cpuset_bind() is not called again. Is the QEMU setup not mounting any cgroup filesystem at all? If so, does it matter whether v1 or v2 setup is used? >> As a result, cpu and memory node hot add may fail to update the cpu and >> node masks of the top cpuset to include the newly added cpu or node in >> a cgroup v2 environment. >> >> smp_init() is called after the first two init functions. So we don't >> have a complete list of active cpus and memory nodes until later in >> cpuset_init_smp() which is the right time to set up effective_cpus >> and effective_mems. >> >> To fix this problem, the potentially incorrect cpus_allowed & >> mems_allowed setup in cpuset_init_smp() are removed. For cgroup v2 >> systems, the initial cpuset_bind() call will set them up correctly. >> For cgroup v1 systems, the second call to cpuset_bind() will do the >> right setup. >> >> cc: stable@vger.kernel.org >> Signed-off-by: Waiman Long >> --- >> kernel/cgroup/cpuset.c | 5 +++-- >> 1 file changed, 3 insertions(+), 2 deletions(-) >> >> diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c >> index 9390bfd9f1cd..6bd8f5ef40fe 100644 >> --- a/kernel/cgroup/cpuset.c >> +++ b/kernel/cgroup/cpuset.c >> @@ -3390,8 +3390,9 @@ static struct notifier_block cpuset_track_online_nodes_nb = { >> */ >> void __init cpuset_init_smp(void) >> { >> - cpumask_copy(top_cpuset.cpus_allowed, cpu_active_mask); >> - top_cpuset.mems_allowed = node_states[N_MEMORY]; > So can we keep line > cpumask_copy(top_cpuset.cpus_allowed, cpu_active_mask); > > and only remove line > top_cpuset.mems_allowed = node_states[N_MEMORY]; > ? That may cause cpusets.cpu to be set incorrectly for systems using cgroup v2. What is really important is that effective_cpus and effective_mems are set correctly. Cheers, Longman