From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4FC62C38A2D for ; Wed, 26 Oct 2022 07:01:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233067AbiJZHBv (ORCPT ); Wed, 26 Oct 2022 03:01:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33358 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233062AbiJZHBp (ORCPT ); Wed, 26 Oct 2022 03:01:45 -0400 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 46CF63384D for ; Wed, 26 Oct 2022 00:01:39 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 397C3CE1DF0 for ; Wed, 26 Oct 2022 07:01:37 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id DBF68C433D7; Wed, 26 Oct 2022 07:01:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1666767695; bh=4EA7mnqZIfsfgQs35vZ0Oeva6CAWg2MlREbIG63FG2w=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=vAaNfkKpnq6T2ATH2P1eptSG2vWxLS48UV729kmn80ggth23fKRVR16O31dRmB5xC 3/3oubYXCu6pzFNa3rX/Y2AIowlybdzHmSoYv7fzNEBWy7uHZublQgNzKKA3E8Iv1B u/CI7+/Ptf433yt9tcoBj59n/2lidNrMgwAz9fBE= Date: Wed, 26 Oct 2022 09:02:28 +0200 From: Greg Kroah-Hartman To: Vishal Chourasia Cc: Peter Zijlstra , linux-kernel@vger.kernel.org, mingo@redhat.com, vincent.guittot@linaro.org, vschneid@redhat.com, srikar@linux.vnet.ibm.com, sshegde@linux.ibm.com, linuxppc-dev@lists.ozlabs.org, ritesh.list@gmail.com, aneesh.kumar@linux.ibm.com Subject: Re: sched/debug: CPU hotplug operation suffers in a large cpu systems Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Oct 26, 2022 at 12:07:01PM +0530, Vishal Chourasia wrote: > On Tue, Oct 18, 2022 at 01:04:40PM +0200, Greg Kroah-Hartman wrote: > > > Why do you need to? What tools require these debugfs files to be > > present? > > We are not entirely sure what applications (if any) might be using this interface. Then just disable it and see what happens :) > > And if you only have 7-8 files per CPU, that does not seem like a lot of > > files overall (14000-16000)? If you only offline 1 cpu, how is removing > > 7 or 8 files a bottleneck? Do you really offline 1999 cpus for a 2k > > system? > > It's 7-8 files per domain per cpu, so, in a system with approx 2k cpus and five > domains, the total file count goes above 70k-80k files. And, when we offline 1 > CPU, the entire directory is rebuilt, resulting in creation of all the files > again. Perhaps change the logic to not rebuild the whole thing and instead just remove the required files? Or as I mentioned before, you can move debugfs to use kernfs, which should resolve most of these issues automatically. Why not take the time to do that which will solve the problem no matter what gets added in the future in other subsystems? thanks, greg k-h From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 79780C433FE for ; Wed, 26 Oct 2022 07:02:37 +0000 (UTC) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4My0B34Qx0z3cJ2 for ; Wed, 26 Oct 2022 18:02:35 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=linuxfoundation.org header.i=@linuxfoundation.org header.a=rsa-sha256 header.s=korg header.b=vAaNfkKp; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=linuxfoundation.org (client-ip=139.178.84.217; helo=dfw.source.kernel.org; envelope-from=gregkh@linuxfoundation.org; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=linuxfoundation.org header.i=@linuxfoundation.org header.a=rsa-sha256 header.s=korg header.b=vAaNfkKp; dkim-atps=neutral Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4My0901kpmz2xYy for ; Wed, 26 Oct 2022 18:01:39 +1100 (AEDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 13DA961D52; Wed, 26 Oct 2022 07:01:36 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id DBF68C433D7; Wed, 26 Oct 2022 07:01:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1666767695; bh=4EA7mnqZIfsfgQs35vZ0Oeva6CAWg2MlREbIG63FG2w=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=vAaNfkKpnq6T2ATH2P1eptSG2vWxLS48UV729kmn80ggth23fKRVR16O31dRmB5xC 3/3oubYXCu6pzFNa3rX/Y2AIowlybdzHmSoYv7fzNEBWy7uHZublQgNzKKA3E8Iv1B u/CI7+/Ptf433yt9tcoBj59n/2lidNrMgwAz9fBE= Date: Wed, 26 Oct 2022 09:02:28 +0200 From: Greg Kroah-Hartman To: Vishal Chourasia Subject: Re: sched/debug: CPU hotplug operation suffers in a large cpu systems Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: ritesh.list@gmail.com, vschneid@redhat.com, vincent.guittot@linaro.org, srikar@linux.vnet.ibm.com, Peter Zijlstra , aneesh.kumar@linux.ibm.com, linux-kernel@vger.kernel.org, sshegde@linux.ibm.com, mingo@redhat.com, linuxppc-dev@lists.ozlabs.org Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" On Wed, Oct 26, 2022 at 12:07:01PM +0530, Vishal Chourasia wrote: > On Tue, Oct 18, 2022 at 01:04:40PM +0200, Greg Kroah-Hartman wrote: > > > Why do you need to? What tools require these debugfs files to be > > present? > > We are not entirely sure what applications (if any) might be using this interface. Then just disable it and see what happens :) > > And if you only have 7-8 files per CPU, that does not seem like a lot of > > files overall (14000-16000)? If you only offline 1 cpu, how is removing > > 7 or 8 files a bottleneck? Do you really offline 1999 cpus for a 2k > > system? > > It's 7-8 files per domain per cpu, so, in a system with approx 2k cpus and five > domains, the total file count goes above 70k-80k files. And, when we offline 1 > CPU, the entire directory is rebuilt, resulting in creation of all the files > again. Perhaps change the logic to not rebuild the whole thing and instead just remove the required files? Or as I mentioned before, you can move debugfs to use kernfs, which should resolve most of these issues automatically. Why not take the time to do that which will solve the problem no matter what gets added in the future in other subsystems? thanks, greg k-h