From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.5 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DBA56C2D0E8 for ; Fri, 27 Mar 2020 07:12:23 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 83B4220714 for ; Fri, 27 Mar 2020 07:12:23 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (4096-bit key) header.d=d-silva.org header.i=@d-silva.org header.b="DTB/U4OM" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 83B4220714 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=d-silva.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0A5836B0010; Fri, 27 Mar 2020 03:12:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0548E6B0032; Fri, 27 Mar 2020 03:12:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E5ECC6B0036; Fri, 27 Mar 2020 03:12:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0186.hostedemail.com [216.40.44.186]) by kanga.kvack.org (Postfix) with ESMTP id CD96D6B0010 for ; Fri, 27 Mar 2020 03:12:22 -0400 (EDT) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 76C734FFA for ; Fri, 27 Mar 2020 07:12:22 +0000 (UTC) X-FDA: 76640273724.21.quill50_149bce95d0f4a X-HE-Tag: quill50_149bce95d0f4a X-Filterd-Recvd-Size: 16694 Received: from ushosting.nmnhosting.com (ushosting.nmnhosting.com [66.55.73.32]) by imf24.hostedemail.com (Postfix) with ESMTP for ; Fri, 27 Mar 2020 07:12:21 +0000 (UTC) Received: from mail2.nmnhosting.com (unknown [202.169.106.97]) by ushosting.nmnhosting.com (Postfix) with ESMTPS id 3CE3C2DC008A; Fri, 27 Mar 2020 18:12:20 +1100 (AEDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=d-silva.org; s=201810a; t=1585293140; bh=9gEAkB9Wz4x2t0FS6kkCqV7mwMB5E1wn17h2H7Fgh08=; h=From:To:Cc:Subject:Date:From; b=DTB/U4OM1FM61MB5Or6LCrnUANc4ktBJ0frwWDaHUbW4OLvcwn4H0XflIVXsCIgMu El99aj1Kv7r6OqcfGl0X8w+Z5SmgzHAeT8/8xrHCukX5AEJhAiSi7OmWLEvpP07Sj0 2SfCSqGCgBfJaTUwwZBRZw2yOGkDBVFM7Rs7MjQdXav8hu+HgPYbADtS7yOkhFws7u BeEygMQAeL9KX9850YkO/dzPQBLYmuFBS6d01bmifsXzy6TXtIu0lm1qc3eBiNuQnG WdOQkVMv5oqp7bsNWooDaB6mJM+1wtUcrW3au48g+WLoa1dlzvY3iI2Vfcj05L+jo+ EMwygTF1Mie2bqrVJF/tEGKXOr7pwRcSK0SsslVdhOc6cutn3xfOZIURlWxWOzYQBJ r4lqcJYP0DW15MfeaOmVq6kMuoE+POkDMwpLSkstPc4pTu7/+nVMjoaob4xsqcBLVU 2tEzb6hxScP3MpR8iGOPppW9XLBAwKq4pVW02bYN9EHhWJwBaygfPcDu0QVEcitdLx lGoxEVzBMtCD5lSKhm+ysZmT74/NTg2b8w9yeLLXG3wW8LPWOwTxiazwJb7zBmk3kI 4P8JrXnSmRElZBQ2seBlWY2L72wv+xvO8Li4xHuxC1KHZQSo7TQ6dQhBukqhHwQm8o y+TJbAvjRuWyD8IefV65Lbzk= Received: from localhost.lan ([10.0.1.179]) by mail2.nmnhosting.com (8.15.2/8.15.2) with ESMTP id 02R7C4AZ045934; Fri, 27 Mar 2020 18:12:04 +1100 (AEDT) (envelope-from alastair@d-silva.org) From: "Alastair D'Silva" To: alastair@d-silva.org Cc: "Aneesh Kumar K . V" , "Oliver O'Halloran" , Benjamin Herrenschmidt , Paul Mackerras , Michael Ellerman , Frederic Barrat , Andrew Donnellan , Arnd Bergmann , Greg Kroah-Hartman , Dan Williams , Vishal Verma , Dave Jiang , Ira Weiny , Andrew Morton , Mauro Carvalho Chehab , "David S. Miller" , Rob Herring , Anton Blanchard , Krzysztof Kozlowski , Mahesh Salgaonkar , Madhavan Srinivasan , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= , Anju T Sudhakar , Hari Bathini , Thomas Gleixner , Greg Kurz , Nicholas Piggin , Masahiro Yamada , Alexey Kardashevskiy , linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-nvdimm@lists.01.org, linux-mm@kvack.org Subject: [PATCH v4 00/25] Add support for OpenCAPI Persistent Memory devices Date: Fri, 27 Mar 2020 18:11:37 +1100 Message-Id: <20200327071202.2159885-1-alastair@d-silva.org> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.2 (mail2.nmnhosting.com [10.0.1.20]); Fri, 27 Mar 2020 18:12:14 +1100 (AEDT) Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This series adds support for OpenCAPI Persistent Memory devices on bare m= etal (arch/powernv), exposing them as nvdimms so that we can make use of = the existing infrastructure. There already exists a driver for the same d= evices abstracted through PowerVM (arch/pseries): arch/powerpc/platforms/= pseries/papr_scm.c These devices are connected via OpenCAPI, and present as LPC (lowest cohe= rence point) memory to the system, practically, that means that memory on= these cards could be treated as conventional, cache-coherent memory. Since the devices are connected via OpenCAPI, they are not enumerated via= ACPI. Instead, OpenCAPI links present as pseudo-PCI bridges, with device= s below them. This series introduces a driver that exposes the memory on these cards as= nvdimms, with each card getting it's own bus. This is somewhat complicat= ed by the fact that the cards do not have out of band persistent storage = for metadata, so 1 SECTION_SIZE's (see SPARSEMEM) worth of storage is car= ved out of the top of the card storage to implement the ndctl_config_* ca= lls. The driver is not responsible for configuring the NPU (NVLink Processing = Unit) BARs to map the LPC memory from the card into the system's physical= address space, instead, it requests this to be done via OPAL calls (typi= cally implemented by Skiboot). The series is structured as follows: - Required infrastructure changes & cleanup - A minimal driver implementation - Implementing additional features within the driver Changelog: V4: - Rebase on next-20200320 - Bump copyright to 2020 - Ensure all uapi headers use C89 compatible comments (missed ocxlpmem.= h) - Move the driver back to drivers/nvdimm/ocxl, after confirmation that this location is desirable - Rename ocxl.c to ocxlpmem.c (+ support files) - Rename all ocxl_pmem to ocxlpmem - Address checkpatch --strict issues - "powerpc/powernv: Add OPAL calls for LPC memory alloc/release" - Pass base address as __be64 - "ocxl: Tally up the LPC memory on a link & allow it to be mapped" - Address checkpatch spacing warnings - Reword blurb - Reword size description for ocxl_link_add_lpc_mem() - Add an early exit in ocxl_link_lpc_release() to avoid triggering bogus warnings if called after ocxl_link_lpc_map() fails - "powerpc/powernv: Add OPAL calls for LPC memory alloc/release" - Reword blurb - "powerpc/powernv: Map & release OpenCAPI LPC memory" - Reword blurb - Move minor_idr init from file_init() to ocxlpmem_init() (fixes runtim= e error in "nvdimm: Add driver for OpenCAPI Persistent Memory") - Wrap long lines - "nvdimm: Add driver for OpenCAPI Storage Class Memory" - Remove '+ 1' workround from serial number->cookie assignment - Drop out of memory message for ocxlpmem in probe() - Fix leaks of ocxlpmem & ocxlpmem->ocxl_fn in probe() - remove struct ocxlpmem_function0, it didn't value add - factor out err_unregistered label in probe - Address more checkpatch warnings - get/put the pci dev on probe/free - Drop ocxlpmem_ prefix from static functions - Propogate errors up from called functions in probe() - Set MODULE_LICENSE to GPLv2 - Add myself as module author - Call nvdimm_bus_unregister() in remove() to release references - Don't call devm_memunmap on metadata_address, the release handler on the device already deals with this - "nvdimm/ocxl: Read the capability registers & wait for device ready" - Fix mask for read_latency - Fold in is_usable logic into timeout to remove error message race - propogate bad rc from read_device_metadata - "nvdimm/ocxl: Add register addresses & status values to the header" - Add comments for register abbreviations where names have been expanded - Add missing status for blocked on background task - Alias defines for firmware update status to show that the duplication of values is intentional - "nvdimm/ocxl: Register a character device for userspace to interact w= ith" - Add lock around minors IDR, delete the cdev before device_unregister - Propogate errors up from called functions in probe() - "nvdimm/ocxl: Add support for Admin commands" - Fix typo in setup_command_data error message, and drop 'ocxl' from it - Drop vestigial CHI read from admin_command_request - Change command ID mismatch message to dev_err, and return an error - Use jiffies to implement admin_command_complete_timeout() - Flesh out blurb - Create a wrapper to issue the command & wait for timeout - "nvdimm/ocxl: Add support for near storage commands" - dropped (will submit with the patches for nvdimm overwrite) - "nvdimm/ocxl: Implement the Read Error Log command" - Remove stray blank line - change misplaced goto to an early exit in read_error_log - Inline error_log_offset_0x08 - Read WWID data as LE rather than host endian - Move the include of nvdimm/ocxlpmem.h to ocxl.c - Add padding after fwrevision in struct ioctl_ocxl_pmem_error_log - Register IOCTL magic - Coerce pointers to __u64 in IOCTLs - "nvdimm/ocxl: Add controller dump IOCTLs" - Coerce pointers to __u64 in IOCTLs - Document expected IOCTL usage in blurb - Add missing rc check - Only populate up to the number of bytes returned by the card, and return this length to the caller - Add missing header check - "nvdimm/ocxl: Add an IOCTL to report controller statistics" - Update to match the latest version of the spec - Verify that parametr block IDs & lengths match what we expect - Use defines for offsets - "nvdimm/ocxl: Forward events to userspace" - Don't enable NSCRA doorbell - return -EBUSY if the event context is already used - return -ENODEV if IRQs cannot be mapped - Tag IRQ pointers with __iomem - Drop ocxlpmem_ prefix from static functions - Propogate error from eventfd_ctx_fdget - Fix error check in copy_to_user - Drop GLOBAL_MMIO_CHI_NSCRA (this should be in the overwrite patch) - Drop unused irq_pgmap - Don't redef BIT_ULL - "nvdimm/ocxl: Add debug IOCTLs" - Eliminate clearing loop (now done in admin_command_execute() - Drop dummy IOCTLs if CONFIG_OCXL_PMEM_DEBUG is not set - Group debug IOCTLs together & comment that they may not be available - "nvdimm/ocxl: Expose SMART data via ndctl" - Drop 'rc =3D 0; goto out;' - Propogate errors from ndctl_smart() - "nvdimm/ocxl: Expose the serial number in sysfs" & "nvdimm/ocxl: Expo= se the firmware version in sysfs" - Squash these 2 patches together - Expose data as a DIMM attribute rather than an ocxlpmem attribute - "nvdimm/ocxl: Add an IOCTL to request controller health & perf data" - Reword blurb - "nvdimm/ocxl: Implement the heartbeat command" - Propogate rc in probe() V3: - Rebase against next/next-20200220 - Move driver to arch/powerpc/platforms/powernv, we now expect this driver to go upstream via the powerpc tree - "nvdimm/ocxl: Implement the Read Error Log command" - Fix bad header path - "nvdimm/ocxl: Read the capability registers & wait for device ready" - Fix overlapping masks between readiness_timeout & memory_available_tim= eout - "nvdimm: Add driver for OpenCAPI Storage Class Memory" - Address minor review comments from Jonathan Cameron - Remove attributes - Default to module if building LIBNVDIMM - Propogate errors up from called functions in probe() - "nvdimm/ocxl: Expose SMART data via ndctl" - Pack attributes in struct - Support different size SMART buffers for compatibility with newer ndctls that may want more SMART attribs than we provide - Rework to to use ND_CMD_CALL instead of ND_CMD_SMART - drop "ocxl: Free detached contexts in ocxl_context_detach_all()" - "powerpc: Map & release OpenCAPI LPC memory" - Remove 'extern' - Only available with CONFIG_MEMORY_HOTPLUG_SPARSE - "ocxl: Tally up the LPC memory on a link & allow it to be mapped" - Address minor review comments from Jonathan Cameron - "ocxl: Add functions to map/unmap LPC memory" - Split detected memory message into a separate patch - Address minor review comments from Jonathan Cameron - Add a comment explaining why unmap_lpc_mem is in deconfigure_afu - "nvdimm/ocxl: Add support for Admin commands" - use sizeof(u64) rather than 0x08 when iterating u64s - "nvdimm/ocxl: Implement the heartbeat command" - Fix typo in blurb - Address kernel doc issues - Ensure all uapi headers use C89 compatible comments - Drop patches for firmware update & overwrite, these will be submitted later once patches are available for ndctl - Rename SCM to OpenCAPI Persistent Memory V2: - "powerpc: Map & release OpenCAPI LPC memory" - Fix #if -> #ifdef - use pci_dev_id to get the bdfn - use __be64 to hold be data - indent check_hotplug_memory_addressable correctly=20 - Remove export of check_hotplug_memory_addressable - "ocxl: Conditionally bind SCM devices to the generic OCXL driver" - Improve patch description and remove redundant default - "nvdimm: Add driver for OpenCAPI Storage Class Memory" - Mark a few funcs as static as identified by the 0day bot - Add OCXL dependancies to OCXL_SCM - Use memcpy_mcsafe in scm_ndctl_config_read - Rename scm_foo_offset_0x00 to scm_foo_header_parse & add docs - Name DIMM attribs "ocxl" rather than "scm" - Split out into base + many feature patches - "powerpc: Enable OpenCAPI Storage Class Memory driver on bare metal" - Build DEV_DAX & friends as modules - "ocxl: Conditionally bind SCM devices to the generic OCXL driver" - Patch dropped (easy enough to maintain this out of tree for devel= opment) - "ocxl: Tally up the LPC memory on a link & allow it to be mapped" - Add a warning if an unmatched lpc_release is called - "ocxl: Add functions to map/unmap LPC memory" - Use EXPORT_SYMBOL_GPL Alastair D'Silva (25): powerpc/powernv: Add OPAL calls for LPC memory alloc/release mm/memory_hotplug: Allow check_hotplug_memory_addressable to be called from drivers powerpc/powernv: Map & release OpenCAPI LPC memory ocxl: Remove unnecessary externs ocxl: Address kernel doc errors & warnings ocxl: Tally up the LPC memory on a link & allow it to be mapped ocxl: Add functions to map/unmap LPC memory ocxl: Emit a log message showing how much LPC memory was detected ocxl: Save the device serial number in ocxl_fn nvdimm: Add driver for OpenCAPI Persistent Memory powerpc: Enable the OpenCAPI Persistent Memory driver for powernv_defconfig nvdimm/ocxl: Add register addresses & status values to the header nvdimm/ocxl: Read the capability registers & wait for device ready nvdimm/ocxl: Add support for Admin commands nvdimm/ocxl: Register a character device for userspace to interact with nvdimm/ocxl: Implement the Read Error Log command nvdimm/ocxl: Add controller dump IOCTLs nvdimm/ocxl: Add an IOCTL to report controller statistics nvdimm/ocxl: Forward events to userspace nvdimm/ocxl: Add an IOCTL to request controller health & perf data nvdimm/ocxl: Implement the heartbeat command nvdimm/ocxl: Add debug IOCTLs nvdimm/ocxl: Expose SMART data via ndctl nvdimm/ocxl: Expose the serial number & firmware version in sysfs MAINTAINERS: Add myself & nvdimm/ocxl to ocxl .../userspace-api/ioctl/ioctl-number.rst | 1 + MAINTAINERS | 3 + arch/powerpc/configs/powernv_defconfig | 5 + arch/powerpc/include/asm/opal-api.h | 2 + arch/powerpc/include/asm/opal.h | 2 + arch/powerpc/include/asm/pnv-ocxl.h | 42 +- arch/powerpc/platforms/powernv/ocxl.c | 43 + arch/powerpc/platforms/powernv/opal-call.c | 2 + drivers/misc/ocxl/config.c | 74 +- drivers/misc/ocxl/core.c | 61 + drivers/misc/ocxl/link.c | 60 + drivers/misc/ocxl/ocxl_internal.h | 45 +- drivers/nvdimm/Kconfig | 2 + drivers/nvdimm/Makefile | 1 + drivers/nvdimm/ocxl/Kconfig | 21 + drivers/nvdimm/ocxl/Makefile | 7 + drivers/nvdimm/ocxl/main.c | 1975 +++++++++++++++++ drivers/nvdimm/ocxl/ocxlpmem.h | 197 ++ drivers/nvdimm/ocxl/ocxlpmem_internal.c | 280 +++ include/linux/memory_hotplug.h | 5 + include/misc/ocxl.h | 122 +- include/uapi/linux/ndctl.h | 1 + include/uapi/nvdimm/ocxlpmem.h | 127 ++ mm/memory_hotplug.c | 4 +- 24 files changed, 2983 insertions(+), 99 deletions(-) create mode 100644 drivers/nvdimm/ocxl/Kconfig create mode 100644 drivers/nvdimm/ocxl/Makefile create mode 100644 drivers/nvdimm/ocxl/main.c create mode 100644 drivers/nvdimm/ocxl/ocxlpmem.h create mode 100644 drivers/nvdimm/ocxl/ocxlpmem_internal.c create mode 100644 include/uapi/nvdimm/ocxlpmem.h --=20 2.24.1