From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.6 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8C469C47082 for ; Wed, 26 May 2021 18:35:53 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 52087613E6 for ; Wed, 26 May 2021 18:35:53 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 52087613E6 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:Content-Type: Content-Transfer-Encoding:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:Date:Message-ID:From: References:Cc:To:Subject:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=GYgG7Q4vYqwKf9nT3NOLqNhxeZMdDF4ODtLYc8qz0Tw=; b=N7TUSaUXm5Djye6LSw0o557sCU OFSZRg4VODVAAxNHnBITJpUTwOZoS6K1LNtfGutYcby6cIfXwDiGpGUMto9PXe6+QumSsCdRrTe3/ FRtP1pm2W85DeB2fBypSST3CCGCP7AwruQpIRAjGCcUDXxpVEeMdzGhOSDLAerfnfJbaNnQ8qqSA9 DVrCK32sD20fFj6MQtZHuL6ZDDbsvRrIkjLqYIS8KBIyspS5yAYajtXs+iSUKEsnSosqRzyZEpS80 x11kwWoNIEQ6E6+9xs7Spg72sZUZEvhq5H5faRE0CQNlVcXaE/vTU3t0FlP9k8P9MPyujnk5WLWH1 uAfd7jAQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1llyLO-00GYTC-E4; Wed, 26 May 2021 18:33:26 +0000 Received: from mga03.intel.com ([134.134.136.65]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1llxSk-00GAjo-Hc for linux-arm-kernel@lists.infradead.org; Wed, 26 May 2021 17:37:00 +0000 IronPort-SDR: dlmu4Ok366lvhkO/puN7Mmjou9MzfD06sFh8OIHRDY5Fw0MYD2qBrijXCG3C2THSlOD+ksMnO9 c3J0KDXuP4Ng== X-IronPort-AV: E=McAfee;i="6200,9189,9996"; a="202559899" X-IronPort-AV: E=Sophos;i="5.82,331,1613462400"; d="scan'208";a="202559899" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 May 2021 10:36:57 -0700 IronPort-SDR: KpTc1WpgeJraDqWuwVA+QrtcVXwoILqTJFMP0CYgXAh/TFh70ASmuaEb2YN7tmX15W4OQiuI3E 6HvxpNPxnXoA== X-IronPort-AV: E=Sophos;i="5.82,331,1613462400"; d="scan'208";a="472085913" Received: from rchatre-mobl3.amr.corp.intel.com (HELO [10.251.4.127]) ([10.251.4.127]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 May 2021 10:36:56 -0700 Subject: Re: About add an A64FX cache control function into resctrl To: "tan.shaopeng@fujitsu.com" , "'fenghua.yu@intel.com'" Cc: "'linux-kernel@vger.kernel.org'" , "'linux-arm-kernel@lists.infradead.org'" , 'James Morse' , "misono.tomohiro@fujitsu.com" , "Luck, Tony" References: <14ddf86b-89e1-ba26-b684-f3d5d5f8ade7@intel.com> From: Reinette Chatre Message-ID: <89416df1-4e91-8ad2-981c-827808a65229@intel.com> Date: Wed, 26 May 2021 10:36:55 -0700 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.10.2 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210526_103658_700424_33D77DDA X-CRM114-Status: GOOD ( 36.50 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="iso-2022-jp"; Format="flowed"; DelSp="yes" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi Tan Shaopeng, On 5/25/2021 1:45 AM, tan.shaopeng@fujitsu.com wrote: > Hi Reinette, > > Sorry, I have not explained A64FX's sector cache function well yet. > I think I need explain this function from different perspective. You have explained the A64FX's sector cache function well. I have also read both specs to understand it better. It appears to me that you are not considering the resctrl architecture as part of your solution but instead just forcing your architecture onto the resctrl filesystem. For example, in resctrl the resource groups are not just a directory structure but has significance in what is being represented within the directory (a class of service). The files within a resource group's directory build on that. From your side I have not seen any effort in aligning the sector cache function with the resctrl architecture but instead you are just changing resctrl interface to match the A64FX architecture. Could you please take a moment to understand what resctrl is and how it could be mapped to A64FX in a coherent way? > >> On 5/17/2021 1:31 AM, tan.shaopeng@fujitsu.com wrote: > -------- > A64FX NUMA-PE-Cache Architecture: > NUMA0: > PE0: > L1sector0,L1sector1,L1sector2,L1sector3 > PE1: > L1sector0,L1sector1,L1sector2,L1sector3 > ... > PE11: > L1sector0,L1sector1,L1sector2,L1sector3 > > L2sector0,1/L2sector2,3 > NUMA1: > PE0: > L1sector0,L1sector1,L1sector2,L1sector3 > ... > PE11: > L1sector0,L1sector1,L1sector2,L1sector3 > > L2sector0,1/L2sector2,3 > NUMA2: > ... > NUMA3: > ... > -------- > In A64FX processor, one L1 sector cache capacity setting register is > only for one PE and not shared among PEs. L2 sector cache maximum > capacity setting registers are shared among PEs in same NUMA, and it is > to be noted that changing these registers in one PE influences other PE. Understood. cache affinity is familiar to resctrl. When a CPU becomes online it is discovered which caches/resources it has affinity to. Resources then have CPU mask associated with them to indicate on which CPU a register could be changed to configure the resource/cache. See domain_add_cpu() and struct rdt_domain. > The number of ways for L2 Sector ID (0,1 or 2,3) can be set through > any PEs in same NUMA. The sector ID 0,1 and 2,3 are not available at > the same time in same NUMA. > > > I think, in your idea, a resource group will be created for each sector ID. > (> "sectors" could be considered the same as the resctrl "classes of service") > Then, an example of resource group is created as follows. > ・ L1: NUMAX-PEY-L1sector0 (X = 0,1,2,3.Y = 0,1,2 ... 11), > ・ L2: NUMAX-L2sector0 (X = 0,1,2,3) > > In this example, sector with same ID(0) of all PEs is allocated to > resource group. The L1D caches are numbered from NUMA0_PE0-L1sector0(0) > to NUMA4_PE11-L1sector0(47) and the L2 caches numbered from > NUMA0-L2sector0(0) to NUM4-L2sector0(3). > (NUMA number X is from 0-4, PE number Y is from 0-11) > (1) The number of ways of NUMAX-PEY-L1sector0 can be set independently > for each PEs (0-47). When run a task on this resource group, > we cannot control on which PE the task is running on and how many > cache ways the task is using. resctrl does not control the affinity on which PE/CPU a task is run. resctrl is an interface with which to configure how resources are allocated on the system. resctrl could thus provide interface with which each sector of each cache instance is assigned a number of cache ways. resctrl also provides an interface to assign a task with a class of service (sector id?). Through this the task obtains access to all resources that is allocated to the particular class of service (sector id?). Depending on which CPU the task is running it may indeed experience different performance if the sector id it is running with does not have the same allocations on all cache instances. The affinity of the task needs to be managed separately using for example taskset. Please see Documentation/x86/resctrl.rst "Examples for RDT allocation usage" > (2) Since L2 can only use 2 sectors at a time, when creating more than > 2 resource groups, L2setctor0 will have to be allocated to a > different resource group. If the L2sector0 is shared by different > resource groups, the L2 sector settings on resource group will be > influenced by each other. > etc... there are various problems, and no merit to using resctrl. > > > In my idea, in order to allocate the L1 and L2 cache to a resource > group, allocate NUMA to the resource group. > An example of resource group is as follows. > ・ NUMA0-PEY-L1sectorZ (Y = 0,1,2...11. Z = 0,1,2,3) > ・ NUMA0-L2sectorZZ (ZZ = 0,1,2,3) > > #cat /sys/fs/resctrl/p0/cpus > 0-11 *1 > #cat /sys/fs/resctrl/p0/schemata > L1:0=0xF,0x3,0x1,x0x0 *2 > L2:0=0xFFF,0xF,0,0 *3 > > *1: PEs belong one NUMA. (Of course, multiple NUMAs can also be > specified in one resource group) > *2: The number of ways for L1sector0,1,2,3. On this resource group > the number of ways of all sector0 is the same(0xF). If 0 way is > specified for one sector, this sector cannot be used. If 4(0xF) > ways are specified for one sector, this sector can use cache fully. > If 4 ways are specified for each sector, there will be no > restriction for using cache. > *3: The number of ways for L2 sector 0,1. If L2sector0,1 is used, > the number of ways of L2sector2,3 must be set to 0. > > All sectors with the same ID on the same resource group were set to > the same number of ways, and when running a task on A64FX, the sector > ID used by task is determined by [56:57] bits of virtual address. > By specifying the PID to /sys/fs/resctrl/tasks, the task will be bound > to the resource group, and then, the cache size used by task will not > be changed never. This completely ignores how this directory and files are currently used. What is missing how this implementation maps to the current resctrl architecture. Reinette _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel