From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.3 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3EEA2C2BA12 for ; Thu, 2 Apr 2020 06:21:52 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 007DE2080C for ; Thu, 2 Apr 2020 06:21:51 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 007DE2080C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 75F808E000A; Thu, 2 Apr 2020 02:21:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 70E8E8E0008; Thu, 2 Apr 2020 02:21:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5B0008E000A; Thu, 2 Apr 2020 02:21:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0163.hostedemail.com [216.40.44.163]) by kanga.kvack.org (Postfix) with ESMTP id 419908E0008 for ; Thu, 2 Apr 2020 02:21:51 -0400 (EDT) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id EF2298248047 for ; Thu, 2 Apr 2020 06:21:50 +0000 (UTC) X-FDA: 76661919180.09.money54_4a3d25119f117 X-HE-Tag: money54_4a3d25119f117 X-Filterd-Recvd-Size: 7241 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf48.hostedemail.com (Postfix) with ESMTP for ; Thu, 2 Apr 2020 06:21:49 +0000 (UTC) Received: from pps.filterd (m0098396.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 03263xni062060 for ; Thu, 2 Apr 2020 02:21:48 -0400 Received: from e06smtp01.uk.ibm.com (e06smtp01.uk.ibm.com [195.75.94.97]) by mx0a-001b2d01.pphosted.com with ESMTP id 3020wfj61p-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 02 Apr 2020 02:21:48 -0400 Received: from localhost by e06smtp01.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Thu, 2 Apr 2020 07:21:31 +0100 Received: from b06avi18878370.portsmouth.uk.ibm.com (9.149.26.194) by e06smtp01.uk.ibm.com (192.168.101.131) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Thu, 2 Apr 2020 07:21:23 +0100 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06avi18878370.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 0326LbiA43254262 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 2 Apr 2020 06:21:37 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D87214C050; Thu, 2 Apr 2020 06:21:37 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 314D74C046; Thu, 2 Apr 2020 06:21:37 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Thu, 2 Apr 2020 06:21:37 +0000 (GMT) Received: from [9.102.43.12] (unknown [9.102.43.12]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 67DCAA0130; Thu, 2 Apr 2020 17:21:30 +1100 (AEDT) Subject: Re: [PATCH v4 06/25] ocxl: Tally up the LPC memory on a link & allow it to be mapped To: Dan Williams , "Alastair D'Silva" Cc: "Aneesh Kumar K . V" , "Oliver O'Halloran" , Benjamin Herrenschmidt , Paul Mackerras , Michael Ellerman , Frederic Barrat , Arnd Bergmann , Greg Kroah-Hartman , Vishal Verma , Dave Jiang , Ira Weiny , Andrew Morton , Mauro Carvalho Chehab , "David S. Miller" , Rob Herring , Anton Blanchard , Krzysztof Kozlowski , Mahesh Salgaonkar , Madhavan Srinivasan , =?UTF-8?Q?C=c3=a9dric_Le_Goater?= , Anju T Sudhakar , Hari Bathini , Thomas Gleixner , Greg Kurz , Nicholas Piggin , Masahiro Yamada , Alexey Kardashevskiy , Linux Kernel Mailing List , linuxppc-dev , linux-nvdimm , Linux MM References: <20200327071202.2159885-1-alastair@d-silva.org> <20200327071202.2159885-7-alastair@d-silva.org> From: Andrew Donnellan Date: Thu, 2 Apr 2020 17:21:34 +1100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TM-AS-GCONF: 00 x-cbid: 20040206-4275-0000-0000-000003B8050E X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 20040206-4276-0000-0000-000038CD59BE Message-Id: <0e188ea7-1845-c9ca-a18f-4f331f31b07c@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138,18.0.676 definitions=2020-04-01_04:2020-03-31,2020-04-01 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 impostorscore=0 priorityscore=1501 malwarescore=0 phishscore=0 mlxlogscore=999 adultscore=0 suspectscore=0 bulkscore=0 mlxscore=0 lowpriorityscore=0 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2004020050 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 1/4/20 7:48 pm, Dan Williams wrote: > On Sun, Mar 29, 2020 at 10:53 PM Alastair D'Silva wrote: >> >> OpenCAPI LPC memory is allocated per link, but each link supports >> multiple AFUs, and each AFU can have LPC memory assigned to it. > > Is there an OpenCAPI primer to decode these objects and their > associations that I can reference? There isn't presently a primer that I think addresses these questions nicely (to my knowledge - Fred might have something he can link to?) - there are the specs published by the OpenCAPI Consortium at https://opencapi.org but they're really for hardware implementers. We should probably expand what's currently documented in Documentation/userspace-api/accelerators/ocxl.rst generally, and this series should probably update that to include details on LPC. To explain the specific objects here: - A "link" is a point-to-point link between the host CPU, and a single OpenCAPI card. (We don't currently support cards making use of multiple links for increased bandwidth, though that is supported from a hardware point of view.) - On POWER9, each link appears as a separate PCI domain, with a single bus, and the card appears as a single device. - A device can have up to 8 functions, per PCI. - An Attached Functional Unit (AFU) is the abstraction for a particular application function. Each PCI function defines the number of AFUs it has through a set of OpenCAPI-specific DVSECs, max 64 per function. The ocxl driver handles AFU discovery. - On the host side, LPC memory is mapped by setting a single BAR for the whole link, but on the device side, LPC memory is requested on a per-AFU basis, through an AFU descriptor that is exposed through the aforementioned DVSECs. Hence the need to loop through the AFUs and get the total required LPC memory to work out the correct BAR value. -- Andrew Donnellan OzLabs, ADL Canberra ajd@linux.ibm.com IBM Australia Limited