From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-11.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6349DC2B9F4 for ; Mon, 14 Jun 2021 16:41:14 +0000 (UTC) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 84F2461378 for ; Mon, 14 Jun 2021 16:41:13 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 84F2461378 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4G3cf06gRKz308h for ; Tue, 15 Jun 2021 02:41:12 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=HBibKRiQ; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=linux.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=aneesh.kumar@linux.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=HBibKRiQ; dkim-atps=neutral Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4G3cdN5d9gz2y0G for ; Tue, 15 Jun 2021 02:40:40 +1000 (AEST) Received: from pps.filterd (m0098399.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 15EGXaAF178722; Mon, 14 Jun 2021 12:40:33 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : mime-version : content-transfer-encoding; s=pp1; bh=iVFaf50StbfWlZr9hdNPJsE8GX+Bb3MiorBUhHMJDtY=; b=HBibKRiQjh3chnFl6u/lskKXL1Z2s7xMpnOLTkfjTJ8AoZf0Na4/ydv6Lxe1y3nrHV8e Wu2gkB/6N45l3PddOUhBtU0l8Kut85tKPedSbQSEysQB1h8+fntYITCHY1gc1OeYbNcD RrNOknfAtrvsVFuJbMJGCj9/PEj27LUKRFd02V+3J9WKGguCPkjC9bJL3kruosJS4uxc eLFo2eY+y8lS4Ty6F4qYqypm8NAXyrsXJjGa6eZW5aRuLnx1ZjP1P5tcN/eyi2tDhVht pwMAD5m2Cf9znYbfeuEBtxqzvFFm7rfLPROtYS9CHy05rPOnjIWGrYTDAqu1AZVKIFr5 Xw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 396a149m9a-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Jun 2021 12:40:33 -0400 Received: from m0098399.ppops.net (m0098399.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 15EGXgcL179606; Mon, 14 Jun 2021 12:40:33 -0400 Received: from ppma01dal.us.ibm.com (83.d6.3fa9.ip4.static.sl-reverse.com [169.63.214.131]) by mx0a-001b2d01.pphosted.com with ESMTP id 396a149m8p-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Jun 2021 12:40:32 -0400 Received: from pps.filterd (ppma01dal.us.ibm.com [127.0.0.1]) by ppma01dal.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 15EGbnXR023979; Mon, 14 Jun 2021 16:40:32 GMT Received: from b03cxnp08025.gho.boulder.ibm.com (b03cxnp08025.gho.boulder.ibm.com [9.17.130.17]) by ppma01dal.us.ibm.com with ESMTP id 394mj9wauk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Jun 2021 16:40:31 +0000 Received: from b03ledav002.gho.boulder.ibm.com (b03ledav002.gho.boulder.ibm.com [9.17.130.233]) by b03cxnp08025.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 15EGeUUL28836302 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 14 Jun 2021 16:40:30 GMT Received: from b03ledav002.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 24B2E13604F; Mon, 14 Jun 2021 16:40:30 +0000 (GMT) Received: from b03ledav002.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D17D8136053; Mon, 14 Jun 2021 16:40:27 +0000 (GMT) Received: from skywalker.ibmuc.com (unknown [9.199.37.120]) by b03ledav002.gho.boulder.ibm.com (Postfix) with ESMTP; Mon, 14 Jun 2021 16:40:27 +0000 (GMT) From: "Aneesh Kumar K.V" To: linuxppc-dev@lists.ozlabs.org, mpe@ellerman.id.au Subject: [RFC PATCH 0/8] Add support for FORM2 associativity Date: Mon, 14 Jun 2021 22:09:55 +0530 Message-Id: <20210614164003.196094-1-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.31.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-TM-AS-GCONF: 00 X-Proofpoint-GUID: Oqo5U4hdUjjGOjbCLrXQ-wjZiDS2Ne-4 X-Proofpoint-ORIG-GUID: tE2WRd9kpIOqd5MVnuvVVQ0RJ61AvbGx X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.761 definitions=2021-06-14_10:2021-06-14, 2021-06-14 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0 adultscore=0 malwarescore=0 suspectscore=0 lowpriorityscore=0 phishscore=0 mlxlogscore=999 bulkscore=0 spamscore=0 clxscore=1015 priorityscore=1501 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2106140104 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nathan Lynch , "Aneesh Kumar K.V" , Daniel Henrique Barboza , David Gibson Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" Form2 associativity adds a much more flexible NUMA topology layout than what is provided by Form1. This also allows PAPR SCM device to use better associativity when using the device as DAX KMEM device. More details can be found in patch x $ ndctl list -N -v [ { "dev":"namespace0.0", "mode":"devdax", "map":"dev", "size":1071644672, "uuid":"37dea198-ddb5-4e42-915a-99a915e24188", "raw_uuid":"148deeaa-4a2f-41d1-8d74-fd9a942d26ba", "daxregion":{ "id":0, "size":1071644672, "devices":[ { "chardev":"dax0.0", "size":1071644672, "target_node":4, "mode":"devdax" } ] }, "align":2097152, "numa_node":1 } ] $ numactl -H ... node distances: node 0 1 2 3 0: 10 11 222 33 1: 44 10 55 66 2: 77 88 10 99 3: 101 121 132 10 $ After DAX KMEM # numactl -H available: 5 nodes (0-4) ... node distances: node 0 1 2 3 4 0: 10 11 22 33 255 1: 44 10 55 66 255 2: 77 88 10 99 255 3: 101 121 132 10 255 4: 255 255 255 255 10 # The above output is with a Qemu command line -numa node,nodeid=4 \ -numa dist,src=0,dst=1,val=11 -numa dist,src=0,dst=2,val=22 -numa dist,src=0,dst=3,val=33 -numa dist,src=0,dst=4,val=255 \ -numa dist,src=1,dst=0,val=44 -numa dist,src=1,dst=2,val=55 -numa dist,src=1,dst=3,val=66 -numa dist,src=1,dst=4,val=255 \ -numa dist,src=2,dst=0,val=77 -numa dist,src=2,dst=1,val=88 -numa dist,src=2,dst=3,val=99 -numa dist,src=2,dst=4,val=255 \ -numa dist,src=3,dst=0,val=101 -numa dist,src=3,dst=1,val=121 -numa dist,src=3,dst=2,val=132 -numa dist,src=3,dst=4,val=255 \ -numa dist,src=4,dst=0,val=255 -numa dist,src=4,dst=1,val=255 -numa dist,src=4,dst=2,val=255 -numa dist,src=4,dst=3,val=255 \ -object memory-backend-file,id=memnvdimm1,prealloc=yes,mem-path=$PMEM_DISK,share=yes,size=${PMEM_SIZE} \ -device nvdimm,label-size=128K,memdev=memnvdimm1,id=nvdimm1,slot=4,uuid=72511b67-0b3b-42fd-8d1d-5be3cae8bcaa,node=4,persistent-nodeid=1 Aneesh Kumar K.V (8): powerpc/pseries: rename min_common_depth to primary_domain_index powerpc/pseries: rename distance_ref_points_depth to max_domain_index powerpc/pseries: Rename TYPE1_AFFINITY to FORM1_AFFINITY powerpc/pseries: Consolidate DLPAR NUMA distance update powerpc/pseries: Consolidate NUMA distance update during boot powerpc/pseries: Add a helper for form1 cpu distance powerpc/pseries: Add support for FORM2 associativity powerpc/papr_scm: Use FORM2 associativity details Documentation/powerpc/associativity.rst | 139 ++++++ arch/powerpc/include/asm/firmware.h | 7 +- arch/powerpc/include/asm/prom.h | 3 +- arch/powerpc/kernel/prom_init.c | 3 +- arch/powerpc/mm/numa.c | 436 ++++++++++++++---- arch/powerpc/platforms/pseries/firmware.c | 3 +- arch/powerpc/platforms/pseries/hotplug-cpu.c | 2 + .../platforms/pseries/hotplug-memory.c | 2 + arch/powerpc/platforms/pseries/papr_scm.c | 26 +- arch/powerpc/platforms/pseries/pseries.h | 2 + 10 files changed, 522 insertions(+), 101 deletions(-) create mode 100644 Documentation/powerpc/associativity.rst -- 2.31.1