From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B745C35641 for ; Mon, 24 Feb 2020 04:34:23 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3E6AA20675 for ; Mon, 24 Feb 2020 04:34:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3E6AA20675 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=au1.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D47D06B0070; Sun, 23 Feb 2020 23:34:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CCFC36B0071; Sun, 23 Feb 2020 23:34:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B70A06B0072; Sun, 23 Feb 2020 23:34:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0075.hostedemail.com [216.40.44.75]) by kanga.kvack.org (Postfix) with ESMTP id 9812F6B0070 for ; Sun, 23 Feb 2020 23:34:22 -0500 (EST) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 41057181AC9B6 for ; Mon, 24 Feb 2020 04:34:22 +0000 (UTC) X-FDA: 76523753964.05.note23_7d5b559a92e09 X-HE-Tag: note23_7d5b559a92e09 X-Filterd-Recvd-Size: 11433 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf33.hostedemail.com (Postfix) with ESMTP for ; Mon, 24 Feb 2020 04:34:21 +0000 (UTC) Received: from pps.filterd (m0098394.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 01O4TfOb025488 for ; Sun, 23 Feb 2020 23:34:20 -0500 Received: from e06smtp05.uk.ibm.com (e06smtp05.uk.ibm.com [195.75.94.101]) by mx0a-001b2d01.pphosted.com with ESMTP id 2yb1pg06vs-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Sun, 23 Feb 2020 23:34:19 -0500 Received: from localhost by e06smtp05.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Mon, 24 Feb 2020 04:34:17 -0000 Received: from b06avi18878370.portsmouth.uk.ibm.com (9.149.26.194) by e06smtp05.uk.ibm.com (192.168.101.135) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Mon, 24 Feb 2020 04:34:10 -0000 Received: from d06av26.portsmouth.uk.ibm.com (d06av26.portsmouth.uk.ibm.com [9.149.105.62]) by b06avi18878370.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 01O4Y9sb40239424 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 24 Feb 2020 04:34:09 GMT Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D2D99AE045; Mon, 24 Feb 2020 04:34:09 +0000 (GMT) Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 32F1CAE053; Mon, 24 Feb 2020 04:34:09 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av26.portsmouth.uk.ibm.com (Postfix) with ESMTP; Mon, 24 Feb 2020 04:34:09 +0000 (GMT) Received: from adsilva.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 88A0EA00E5; Mon, 24 Feb 2020 15:34:04 +1100 (AEDT) Subject: Re: [PATCH v3 00/27] Add support for OpenCAPI Persistent Memory devices From: "Alastair D'Silva" To: Dan Williams Cc: "Aneesh Kumar K . V" , "Oliver O'Halloran" , Benjamin Herrenschmidt , Paul Mackerras , Michael Ellerman , Frederic Barrat , Andrew Donnellan , Arnd Bergmann , Greg Kroah-Hartman , Vishal Verma , Dave Jiang , Ira Weiny , Andrew Morton , Mauro Carvalho Chehab , "David S. Miller" , Rob Herring , Anton Blanchard , Krzysztof Kozlowski , Mahesh Salgaonkar , Madhavan Srinivasan , =?ISO-8859-1?Q?C=E9dric?= Le Goater , Anju T Sudhakar , Hari Bathini , Thomas Gleixner , Greg Kurz , Nicholas Piggin , Masahiro Yamada , Alexey Kardashevskiy , Linux Kernel Mailing List , linuxppc-dev , linux-nvdimm , Linux MM Date: Mon, 24 Feb 2020 15:34:07 +1100 In-Reply-To: References: <20200221032720.33893-1-alastair@au1.ibm.com> Organization: IBM Australia Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.34.3 (3.34.3-1.fc31) MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-TM-AS-GCONF: 00 x-cbid: 20022404-0020-0000-0000-000003ACF5B0 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 20022404-0021-0000-0000-00002205050E Message-Id: <240fbefc6275ac0a6f2aa68715b3b73b0e7a8310.camel@au1.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138,18.0.572 definitions=2020-02-23_07:2020-02-21,2020-02-23 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 bulkscore=0 mlxlogscore=999 clxscore=1015 suspectscore=2 lowpriorityscore=0 priorityscore=1501 mlxscore=0 malwarescore=0 phishscore=0 impostorscore=0 spamscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2001150001 definitions=main-2002240037 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, 2020-02-21 at 08:21 -0800, Dan Williams wrote: > On Thu, Feb 20, 2020 at 7:28 PM Alastair D'Silva < > alastair@au1.ibm.com> wrote: > > From: Alastair D'Silva > > > > This series adds support for OpenCAPI Persistent Memory devices, > > exposing > > them as nvdimms so that we can make use of the existing > > infrastructure. > > A single sentence to introduce: > > 24 files changed, 3029 insertions(+), 97 deletions(-) > > ...is inadequate. What are OpenCAPI Persistent Memory devices? How do > they compare, in terms relevant to libnvdimm, to other persistent > memory devices? What challenges do they pose to the existing > enabling? > What is the overall approach taken with this 27 patch break down? > What > are the changes since v2, v1? If you incorporated someone's review > feedback note it in the cover letter changelog, if you didn't > incorporate someone's feedback note that too with an explanation. > > In short, provide a bridge document for someone familiar with the > upstream infrastructure, but not necessarily steeped in powernv / > OpenCAPI platform details, to get started with this code. > > For now, no need to resend the whole series, just reply to this > message with a fleshed out cover letter and then incorporate it going > forward for v4+. Apologies, I was maintaining a changelog, and forgot to include it. I'll flesh out the cover letter too: This series adds support for OpenCAPI Persistent Memory devices on bare metal (arch/powernv), exposing them as nvdimms so that we can make use of the existing infrastructure. There already exists a driver for the same devices abstracted through PowerVM (arch/pseries): arch/powerpc/platforms/pseries/papr_scm.c These devices are connected via OpenCAPI, and present as LPC (lowest coherence point) memory to the system, practically, that means that memory on these cards could be treated as conventional, cache-coherent memory. Since the devices are connected via OpenCAPI, they are not enumerated via ACPI. Instead, OpenCAPI links present as pseudo-PCI bridges, with devices below them. This series introduces a driver that exposes the memory on these cards as nvdimms, with each card getting it's own bus. This is somewhat complicated by the fact that the cards do not have out of band persistent storage for metadata, so 1 SECTION_SIZE's (see SPARSEMEM) worth of storage is carved out of the top of the card storage to implement the ndctl_config_* calls. The driver is not responsible for configuring the NPU (NVLink Processing Unit) BARs to map the LPC memory from the card into the system's physical address space, instead, it requests this to be done via OPAL calls (typically implemented by Skiboot). The series is structured as follows: - Required infrastructure changes & cleanup - A minimal driver implementation - Implementing additional features within the driver V3: - Rebase against next/next-20200220 - Move driver to arch/powerpc/platforms/powernv, we now expect this driver to go upstream via the powerpc tree - "nvdimm/ocxl: Implement the Read Error Log command" - Fix bad header path - "nvdimm/ocxl: Read the capability registers & wait for device ready" - Fix overlapping masks between readiness_timeout & memory_available_timeout - "nvdimm: Add driver for OpenCAPI Storage Class Memory" - Address minor review comments from Jonathan Cameron - Remove attributes - Default to module if building LIBNVDIMM - Propogate errors up from called functions in probe() - "nvdimm/ocxl: Expose SMART data via ndctl" - Pack attributes in struct - Support different size SMART buffers for compatibility with newer ndctls that may want more SMART attribs than we provide - Rework to to use ND_CMD_CALL instead of ND_CMD_SMART - drop "ocxl: Free detached contexts in ocxl_context_detach_all()" - "powerpc: Map & release OpenCAPI LPC memory" - Remove 'extern' - Only available with CONFIG_MEMORY_HOTPLUG_SPARSE - "ocxl: Tally up the LPC memory on a link & allow it to be mapped" - Address minor review comments from Jonathan Cameron - "ocxl: Add functions to map/unmap LPC memory" - Split detected memory message into a separate patch - Address minor review comments from Jonathan Cameron - Add a comment explaining why unmap_lpc_mem is in deconfigure_afu - "nvdimm/ocxl: Add support for Admin commands" - use sizeof(u64) rather than 0x08 when iterating u64s - "nvdimm/ocxl: Implement the heartbeat command" - Fix typo in blurb - Address kernel doc issues - Ensure all uapi headers use C89 compatible comments - Drop patches for firmware update & overwrite, these will be submitted later once patches are available for ndctl - Rename SCM to OpenCAPI Persistent Memory V2: - "powerpc: Map & release OpenCAPI LPC memory" - Fix #if -> #ifdef - use pci_dev_id to get the bdfn - use __be64 to hold be data - indent check_hotplug_memory_addressable correctly - Remove export of check_hotplug_memory_addressable - "ocxl: Conditionally bind SCM devices to the generic OCXL driver" - Improve patch description and remove redundant default - "nvdimm: Add driver for OpenCAPI Storage Class Memory" - Mark a few funcs as static as identified by the 0day bot - Add OCXL dependancies to OCXL_SCM - Use memcpy_mcsafe in scm_ndctl_config_read - Rename scm_foo_offset_0x00 to scm_foo_header_parse & add docs - Name DIMM attribs "ocxl" rather than "scm" - Split out into base + many feature patches - "powerpc: Enable OpenCAPI Storage Class Memory driver on bare metal" - Build DEV_DAX & friends as modules - "ocxl: Conditionally bind SCM devices to the generic OCXL driver" - Patch dropped (easy enough to maintain this out of tree for development) - "ocxl: Tally up the LPC memory on a link & allow it to be mapped" - Add a warning if an unmatched lpc_release is called - "ocxl: Add functions to map/unmap LPC memory" - Use EXPORT_SYMBOL_GPL -- Alastair D'Silva Open Source Developer Linux Technology Centre, IBM Australia mob: 0423 762 819