From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.3 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C13C8C10F26 for ; Thu, 2 Apr 2020 06:21:52 +0000 (UTC) Received: from ml01.01.org (ml01.01.org [198.145.21.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 991D020B1F for ; Thu, 2 Apr 2020 06:21:52 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 991D020B1F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.ibm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-nvdimm-bounces@lists.01.org Received: from ml01.vlan13.01.org (localhost [IPv6:::1]) by ml01.01.org (Postfix) with ESMTP id ED24310FC5F74; Wed, 1 Apr 2020 23:22:41 -0700 (PDT) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=148.163.158.5; helo=mx0a-001b2d01.pphosted.com; envelope-from=ajd@linux.ibm.com; receiver= Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id B4B8E10FC5F72 for ; Wed, 1 Apr 2020 23:22:39 -0700 (PDT) Received: from pps.filterd (m0098420.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 03269JwU119581 for ; Thu, 2 Apr 2020 02:21:48 -0400 Received: from e06smtp01.uk.ibm.com (e06smtp01.uk.ibm.com [195.75.94.97]) by mx0b-001b2d01.pphosted.com with ESMTP id 3058y8a511-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 02 Apr 2020 02:21:48 -0400 Received: from localhost by e06smtp01.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Thu, 2 Apr 2020 07:21:31 +0100 Received: from b06avi18878370.portsmouth.uk.ibm.com (9.149.26.194) by e06smtp01.uk.ibm.com (192.168.101.131) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Thu, 2 Apr 2020 07:21:23 +0100 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06avi18878370.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 0326LbiA43254262 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 2 Apr 2020 06:21:37 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D87214C050; Thu, 2 Apr 2020 06:21:37 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 314D74C046; Thu, 2 Apr 2020 06:21:37 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Thu, 2 Apr 2020 06:21:37 +0000 (GMT) Received: from [9.102.43.12] (unknown [9.102.43.12]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 67DCAA0130; Thu, 2 Apr 2020 17:21:30 +1100 (AEDT) Subject: Re: [PATCH v4 06/25] ocxl: Tally up the LPC memory on a link & allow it to be mapped To: Dan Williams , "Alastair D'Silva" References: <20200327071202.2159885-1-alastair@d-silva.org> <20200327071202.2159885-7-alastair@d-silva.org> From: Andrew Donnellan Date: Thu, 2 Apr 2020 17:21:34 +1100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.6.0 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US X-TM-AS-GCONF: 00 x-cbid: 20040206-4275-0000-0000-000003B8050E X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 20040206-4276-0000-0000-000038CD59BE Message-Id: <0e188ea7-1845-c9ca-a18f-4f331f31b07c@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138,18.0.676 definitions=2020-04-01_04:2020-03-31,2020-04-01 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 bulkscore=0 phishscore=0 adultscore=0 malwarescore=0 priorityscore=1501 mlxlogscore=999 lowpriorityscore=0 spamscore=0 suspectscore=0 impostorscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2004020050 Message-ID-Hash: JDX5ODARDFVLKTEEHOVD6BJEMPHZB2D4 X-Message-ID-Hash: JDX5ODARDFVLKTEEHOVD6BJEMPHZB2D4 X-MailFrom: ajd@linux.ibm.com X-Mailman-Rule-Hits: nonmember-moderation X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation CC: "Aneesh Kumar K . V" , Benjamin Herrenschmidt , Paul Mackerras , Michael Ellerman , Frederic Barrat , Arnd Bergmann , Greg Kroah-Hartman , Andrew Morton , Mauro Carvalho Chehab , "David S. Miller" , Rob Herring , Anton Blanchard , Krzysztof Kozlowski , Mahesh Salgaonkar , Madhavan Srinivasan , =?UTF-8?Q?C=c3=a9dric_Le_Goater?= , Anju T Sudhakar , Hari Bathini , Thomas Gleixner , Greg Kurz , Nicholas Piggin , Masahiro Yamada , Alexey Kardashevskiy , Linux Kernel Mailing List , linuxppc-dev , linux-nvdimm , Linux MM X-Mailman-Version: 3.1.1 Precedence: list List-Id: "Linux-nvdimm developer list." Archived-At: List-Archive: List-Help: List-Post: List-Subscribe: List-Unsubscribe: Content-Type: text/plain; charset="us-ascii"; format="flowed" Content-Transfer-Encoding: 7bit On 1/4/20 7:48 pm, Dan Williams wrote: > On Sun, Mar 29, 2020 at 10:53 PM Alastair D'Silva wrote: >> >> OpenCAPI LPC memory is allocated per link, but each link supports >> multiple AFUs, and each AFU can have LPC memory assigned to it. > > Is there an OpenCAPI primer to decode these objects and their > associations that I can reference? There isn't presently a primer that I think addresses these questions nicely (to my knowledge - Fred might have something he can link to?) - there are the specs published by the OpenCAPI Consortium at https://opencapi.org but they're really for hardware implementers. We should probably expand what's currently documented in Documentation/userspace-api/accelerators/ocxl.rst generally, and this series should probably update that to include details on LPC. To explain the specific objects here: - A "link" is a point-to-point link between the host CPU, and a single OpenCAPI card. (We don't currently support cards making use of multiple links for increased bandwidth, though that is supported from a hardware point of view.) - On POWER9, each link appears as a separate PCI domain, with a single bus, and the card appears as a single device. - A device can have up to 8 functions, per PCI. - An Attached Functional Unit (AFU) is the abstraction for a particular application function. Each PCI function defines the number of AFUs it has through a set of OpenCAPI-specific DVSECs, max 64 per function. The ocxl driver handles AFU discovery. - On the host side, LPC memory is mapped by setting a single BAR for the whole link, but on the device side, LPC memory is requested on a per-AFU basis, through an AFU descriptor that is exposed through the aforementioned DVSECs. Hence the need to loop through the AFUs and get the total required LPC memory to work out the correct BAR value. -- Andrew Donnellan OzLabs, ADL Canberra ajd@linux.ibm.com IBM Australia Limited _______________________________________________ Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org To unsubscribe send an email to linux-nvdimm-leave@lists.01.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.3 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D8EE5C43331 for ; Thu, 2 Apr 2020 06:21:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id AE5AE20857 for ; Thu, 2 Apr 2020 06:21:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728947AbgDBGVu (ORCPT ); Thu, 2 Apr 2020 02:21:50 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:32032 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727746AbgDBGVt (ORCPT ); Thu, 2 Apr 2020 02:21:49 -0400 Received: from pps.filterd (m0098419.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 03265FLi100851 for ; Thu, 2 Apr 2020 02:21:48 -0400 Received: from e06smtp01.uk.ibm.com (e06smtp01.uk.ibm.com [195.75.94.97]) by mx0b-001b2d01.pphosted.com with ESMTP id 304gst2b4g-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 02 Apr 2020 02:21:48 -0400 Received: from localhost by e06smtp01.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Thu, 2 Apr 2020 07:21:31 +0100 Received: from b06avi18878370.portsmouth.uk.ibm.com (9.149.26.194) by e06smtp01.uk.ibm.com (192.168.101.131) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Thu, 2 Apr 2020 07:21:23 +0100 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06avi18878370.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 0326LbiA43254262 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 2 Apr 2020 06:21:37 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D87214C050; Thu, 2 Apr 2020 06:21:37 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 314D74C046; Thu, 2 Apr 2020 06:21:37 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Thu, 2 Apr 2020 06:21:37 +0000 (GMT) Received: from [9.102.43.12] (unknown [9.102.43.12]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 67DCAA0130; Thu, 2 Apr 2020 17:21:30 +1100 (AEDT) Subject: Re: [PATCH v4 06/25] ocxl: Tally up the LPC memory on a link & allow it to be mapped To: Dan Williams , "Alastair D'Silva" Cc: "Aneesh Kumar K . V" , "Oliver O'Halloran" , Benjamin Herrenschmidt , Paul Mackerras , Michael Ellerman , Frederic Barrat , Arnd Bergmann , Greg Kroah-Hartman , Vishal Verma , Dave Jiang , Ira Weiny , Andrew Morton , Mauro Carvalho Chehab , "David S. Miller" , Rob Herring , Anton Blanchard , Krzysztof Kozlowski , Mahesh Salgaonkar , Madhavan Srinivasan , =?UTF-8?Q?C=c3=a9dric_Le_Goater?= , Anju T Sudhakar , Hari Bathini , Thomas Gleixner , Greg Kurz , Nicholas Piggin , Masahiro Yamada , Alexey Kardashevskiy , Linux Kernel Mailing List , linuxppc-dev , linux-nvdimm , Linux MM References: <20200327071202.2159885-1-alastair@d-silva.org> <20200327071202.2159885-7-alastair@d-silva.org> From: Andrew Donnellan Date: Thu, 2 Apr 2020 17:21:34 +1100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TM-AS-GCONF: 00 x-cbid: 20040206-4275-0000-0000-000003B8050E X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 20040206-4276-0000-0000-000038CD59BE Message-Id: <0e188ea7-1845-c9ca-a18f-4f331f31b07c@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138,18.0.676 definitions=2020-04-01_04:2020-03-31,2020-04-01 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0 spamscore=0 malwarescore=0 suspectscore=0 priorityscore=1501 lowpriorityscore=0 phishscore=0 clxscore=1015 mlxlogscore=999 mlxscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2004020050 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 1/4/20 7:48 pm, Dan Williams wrote: > On Sun, Mar 29, 2020 at 10:53 PM Alastair D'Silva wrote: >> >> OpenCAPI LPC memory is allocated per link, but each link supports >> multiple AFUs, and each AFU can have LPC memory assigned to it. > > Is there an OpenCAPI primer to decode these objects and their > associations that I can reference? There isn't presently a primer that I think addresses these questions nicely (to my knowledge - Fred might have something he can link to?) - there are the specs published by the OpenCAPI Consortium at https://opencapi.org but they're really for hardware implementers. We should probably expand what's currently documented in Documentation/userspace-api/accelerators/ocxl.rst generally, and this series should probably update that to include details on LPC. To explain the specific objects here: - A "link" is a point-to-point link between the host CPU, and a single OpenCAPI card. (We don't currently support cards making use of multiple links for increased bandwidth, though that is supported from a hardware point of view.) - On POWER9, each link appears as a separate PCI domain, with a single bus, and the card appears as a single device. - A device can have up to 8 functions, per PCI. - An Attached Functional Unit (AFU) is the abstraction for a particular application function. Each PCI function defines the number of AFUs it has through a set of OpenCAPI-specific DVSECs, max 64 per function. The ocxl driver handles AFU discovery. - On the host side, LPC memory is mapped by setting a single BAR for the whole link, but on the device side, LPC memory is requested on a per-AFU basis, through an AFU descriptor that is exposed through the aforementioned DVSECs. Hence the need to loop through the AFUs and get the total required LPC memory to work out the correct BAR value. -- Andrew Donnellan OzLabs, ADL Canberra ajd@linux.ibm.com IBM Australia Limited From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.3 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1382CC43331 for ; Thu, 2 Apr 2020 06:23:31 +0000 (UTC) Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id BB04F2078E for ; Thu, 2 Apr 2020 06:23:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BB04F2078E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 48tCgN3plszDrMF for ; Thu, 2 Apr 2020 17:23:28 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=linux.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=ajd@linux.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=linux.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 48tCdX2krHzDrDr for ; Thu, 2 Apr 2020 17:21:51 +1100 (AEDT) Received: from pps.filterd (m0098409.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 03262ui3020198 for ; Thu, 2 Apr 2020 02:21:49 -0400 Received: from e06smtp01.uk.ibm.com (e06smtp01.uk.ibm.com [195.75.94.97]) by mx0a-001b2d01.pphosted.com with ESMTP id 303wry3jj7-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 02 Apr 2020 02:21:48 -0400 Received: from localhost by e06smtp01.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Thu, 2 Apr 2020 07:21:31 +0100 Received: from b06avi18878370.portsmouth.uk.ibm.com (9.149.26.194) by e06smtp01.uk.ibm.com (192.168.101.131) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Thu, 2 Apr 2020 07:21:23 +0100 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06avi18878370.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 0326LbiA43254262 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 2 Apr 2020 06:21:37 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D87214C050; Thu, 2 Apr 2020 06:21:37 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 314D74C046; Thu, 2 Apr 2020 06:21:37 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Thu, 2 Apr 2020 06:21:37 +0000 (GMT) Received: from [9.102.43.12] (unknown [9.102.43.12]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 67DCAA0130; Thu, 2 Apr 2020 17:21:30 +1100 (AEDT) Subject: Re: [PATCH v4 06/25] ocxl: Tally up the LPC memory on a link & allow it to be mapped To: Dan Williams , "Alastair D'Silva" References: <20200327071202.2159885-1-alastair@d-silva.org> <20200327071202.2159885-7-alastair@d-silva.org> From: Andrew Donnellan Date: Thu, 2 Apr 2020 17:21:34 +1100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TM-AS-GCONF: 00 x-cbid: 20040206-4275-0000-0000-000003B8050E X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 20040206-4276-0000-0000-000038CD59BE Message-Id: <0e188ea7-1845-c9ca-a18f-4f331f31b07c@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138, 18.0.676 definitions=2020-04-01_04:2020-03-31, 2020-04-01 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 lowpriorityscore=0 mlxlogscore=999 clxscore=1015 bulkscore=0 malwarescore=0 suspectscore=0 phishscore=0 priorityscore=1501 spamscore=0 adultscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2004020055 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Madhavan Srinivasan , Alexey Kardashevskiy , Masahiro Yamada , Oliver O'Halloran , Mauro Carvalho Chehab , Ira Weiny , Rob Herring , Dave Jiang , linux-nvdimm , "Aneesh Kumar K . V" , Krzysztof Kozlowski , Anju T Sudhakar , Mahesh Salgaonkar , Arnd Bergmann , Greg Kurz , Nicholas Piggin , =?UTF-8?Q?C=c3=a9dric_Le_Goater?= , Thomas Gleixner , Hari Bathini , Linux MM , Greg Kroah-Hartman , Linux Kernel Mailing List , Vishal Verma , Frederic Barrat , Paul Mackerras , Andrew Morton , linuxppc-dev , "David S. Miller" Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" On 1/4/20 7:48 pm, Dan Williams wrote: > On Sun, Mar 29, 2020 at 10:53 PM Alastair D'Silva wrote: >> >> OpenCAPI LPC memory is allocated per link, but each link supports >> multiple AFUs, and each AFU can have LPC memory assigned to it. > > Is there an OpenCAPI primer to decode these objects and their > associations that I can reference? There isn't presently a primer that I think addresses these questions nicely (to my knowledge - Fred might have something he can link to?) - there are the specs published by the OpenCAPI Consortium at https://opencapi.org but they're really for hardware implementers. We should probably expand what's currently documented in Documentation/userspace-api/accelerators/ocxl.rst generally, and this series should probably update that to include details on LPC. To explain the specific objects here: - A "link" is a point-to-point link between the host CPU, and a single OpenCAPI card. (We don't currently support cards making use of multiple links for increased bandwidth, though that is supported from a hardware point of view.) - On POWER9, each link appears as a separate PCI domain, with a single bus, and the card appears as a single device. - A device can have up to 8 functions, per PCI. - An Attached Functional Unit (AFU) is the abstraction for a particular application function. Each PCI function defines the number of AFUs it has through a set of OpenCAPI-specific DVSECs, max 64 per function. The ocxl driver handles AFU discovery. - On the host side, LPC memory is mapped by setting a single BAR for the whole link, but on the device side, LPC memory is requested on a per-AFU basis, through an AFU descriptor that is exposed through the aforementioned DVSECs. Hence the need to loop through the AFUs and get the total required LPC memory to work out the correct BAR value. -- Andrew Donnellan OzLabs, ADL Canberra ajd@linux.ibm.com IBM Australia Limited