From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.2 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 34DB8C3F2CE for ; Thu, 5 Mar 2020 03:11:44 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B721A2146E for ; Thu, 5 Mar 2020 03:11:43 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B721A2146E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1898C6B0003; Wed, 4 Mar 2020 22:11:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1399E6B0005; Wed, 4 Mar 2020 22:11:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 027346B0007; Wed, 4 Mar 2020 22:11:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0194.hostedemail.com [216.40.44.194]) by kanga.kvack.org (Postfix) with ESMTP id DF7F36B0003 for ; Wed, 4 Mar 2020 22:11:42 -0500 (EST) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 675C52C0C for ; Thu, 5 Mar 2020 03:11:42 +0000 (UTC) X-FDA: 76559833644.01.scarf37_833559828e204 X-HE-Tag: scarf37_833559828e204 X-Filterd-Recvd-Size: 16370 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf39.hostedemail.com (Postfix) with ESMTP for ; Thu, 5 Mar 2020 03:11:41 +0000 (UTC) Received: from pps.filterd (m0098399.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 02538Wls049236 for ; Wed, 4 Mar 2020 22:11:40 -0500 Received: from e06smtp07.uk.ibm.com (e06smtp07.uk.ibm.com [195.75.94.103]) by mx0a-001b2d01.pphosted.com with ESMTP id 2yhsvahv62-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 04 Mar 2020 22:11:40 -0500 Received: from localhost by e06smtp07.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Thu, 5 Mar 2020 03:11:37 -0000 Received: from b06cxnps4074.portsmouth.uk.ibm.com (9.149.109.196) by e06smtp07.uk.ibm.com (192.168.101.137) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Thu, 5 Mar 2020 03:11:29 -0000 Received: from b06wcsmtp001.portsmouth.uk.ibm.com (b06wcsmtp001.portsmouth.uk.ibm.com [9.149.105.160]) by b06cxnps4074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 0253BSml56099026 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 5 Mar 2020 03:11:28 GMT Received: from b06wcsmtp001.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id C792BA4054; Thu, 5 Mar 2020 03:11:28 +0000 (GMT) Received: from b06wcsmtp001.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 2CD0BA4060; Thu, 5 Mar 2020 03:11:28 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by b06wcsmtp001.portsmouth.uk.ibm.com (Postfix) with ESMTP; Thu, 5 Mar 2020 03:11:28 +0000 (GMT) Received: from [10.61.2.125] (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 2C378A0264; Thu, 5 Mar 2020 14:11:23 +1100 (AEDT) Subject: Re: [PATCH v3 23/27] powerpc/powernv/pmem: Add debug IOCTLs To: "Alastair D'Silva" , alastair@d-silva.org Cc: "Aneesh Kumar K . V" , "Oliver O'Halloran" , Benjamin Herrenschmidt , Paul Mackerras , Michael Ellerman , Frederic Barrat , Arnd Bergmann , Greg Kroah-Hartman , Dan Williams , Vishal Verma , Dave Jiang , Ira Weiny , Andrew Morton , Mauro Carvalho Chehab , "David S. Miller" , Rob Herring , Anton Blanchard , Krzysztof Kozlowski , Mahesh Salgaonkar , Madhavan Srinivasan , =?UTF-8?Q?C=c3=a9dric_Le_Goater?= , Anju T Sudhakar , Hari Bathini , Thomas Gleixner , Greg Kurz , Nicholas Piggin , Masahiro Yamada , Alexey Kardashevskiy , linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-nvdimm@lists.01.org, linux-mm@kvack.org References: <20200221032720.33893-1-alastair@au1.ibm.com> <20200221032720.33893-24-alastair@au1.ibm.com> From: Andrew Donnellan Date: Thu, 5 Mar 2020 14:11:26 +1100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.5.0 MIME-Version: 1.0 In-Reply-To: <20200221032720.33893-24-alastair@au1.ibm.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TM-AS-GCONF: 00 x-cbid: 20030503-0028-0000-0000-000003E0FEB2 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 20030503-0029-0000-0000-000024A631F9 Message-Id: <5e2be4dd-bc8b-ff2c-d057-acd5f3728f4a@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138,18.0.572 definitions=2020-03-04_10:2020-03-04,2020-03-04 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 mlxlogscore=999 clxscore=1015 suspectscore=0 impostorscore=0 adultscore=0 spamscore=0 malwarescore=0 bulkscore=0 lowpriorityscore=0 mlxscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2001150001 definitions=main-2003050015 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 21/2/20 2:27 pm, Alastair D'Silva wrote: > From: Alastair D'Silva > > These IOCTLs provide low level access to the card to aid in debugging > controller/FPGA firmware. > > Signed-off-by: Alastair D'Silva > --- > arch/powerpc/platforms/powernv/pmem/Kconfig | 6 + > arch/powerpc/platforms/powernv/pmem/ocxl.c | 249 ++++++++++++++++++++ > include/uapi/nvdimm/ocxl-pmem.h | 32 +++ > 3 files changed, 287 insertions(+) > > diff --git a/arch/powerpc/platforms/powernv/pmem/Kconfig b/arch/powerpc/platforms/powernv/pmem/Kconfig > index c5d927520920..3f44429d70c9 100644 > --- a/arch/powerpc/platforms/powernv/pmem/Kconfig > +++ b/arch/powerpc/platforms/powernv/pmem/Kconfig > @@ -12,4 +12,10 @@ config OCXL_PMEM > > Select N if unsure. > > +config OCXL_PMEM_DEBUG > + bool "OpenCAPI Persistent Memory debugging" > + depends on OCXL_PMEM > + help > + Enables low level IOCTLs for OpenCAPI Persistent Memory firmware development > + How dangerous are these ioctls and does that need to be pointed out in this description? > endif > diff --git a/arch/powerpc/platforms/powernv/pmem/ocxl.c b/arch/powerpc/platforms/powernv/pmem/ocxl.c > index e01f6f9fc180..d4ce5e9e0521 100644 > --- a/arch/powerpc/platforms/powernv/pmem/ocxl.c > +++ b/arch/powerpc/platforms/powernv/pmem/ocxl.c > @@ -1050,6 +1050,235 @@ int req_controller_health_perf(struct ocxlpmem *ocxlpmem) > GLOBAL_MMIO_HCI_REQ_HEALTH_PERF); > } > > +#ifdef CONFIG_OCXL_PMEM_DEBUG > +/** > + * enable_fwdebug() - Enable FW debug on the controller > + * @ocxlpmem: the device metadata > + * Return: 0 on success, negative on failure > + */ > +static int enable_fwdebug(const struct ocxlpmem *ocxlpmem) > +{ > + return ocxl_global_mmio_set64(ocxlpmem->ocxl_afu, GLOBAL_MMIO_HCI, > + OCXL_LITTLE_ENDIAN, > + GLOBAL_MMIO_HCI_FW_DEBUG); > +} > + > +/** > + * disable_fwdebug() - Disable FW debug on the controller > + * @ocxlpmem: the device metadata > + * Return: 0 on success, negative on failure > + */ > +static int disable_fwdebug(const struct ocxlpmem *ocxlpmem) > +{ > + return ocxl_global_mmio_set64(ocxlpmem->ocxl_afu, GLOBAL_MMIO_HCIC, > + OCXL_LITTLE_ENDIAN, > + GLOBAL_MMIO_HCI_FW_DEBUG); > +} > + > +static int ioctl_fwdebug(struct ocxlpmem *ocxlpmem, > + struct ioctl_ocxl_pmem_fwdebug __user *uarg) > +{ > + struct ioctl_ocxl_pmem_fwdebug args; > + u64 val; > + int i; > + int rc; > + > + if (copy_from_user(&args, uarg, sizeof(args))) > + return -EFAULT; > + > + // Buffer size must be a multiple of 8 > + if ((args.buf_size & 0x07)) > + return -EINVAL; > + > + if (args.buf_size > ocxlpmem->admin_command.data_size) > + return -EINVAL; > + > + mutex_lock(&ocxlpmem->admin_command.lock); > + > + rc = enable_fwdebug(ocxlpmem); > + if (rc) > + goto out; > + > + rc = admin_command_request(ocxlpmem, ADMIN_COMMAND_FW_DEBUG); > + if (rc) > + goto out; > + > + // Write DebugAction & FunctionCode > + val = ((u64)args.debug_action << 56) | ((u64)args.function_code << 40); > + > + rc = ocxl_global_mmio_write64(ocxlpmem->ocxl_afu, > + ocxlpmem->admin_command.request_offset + 0x08, > + OCXL_LITTLE_ENDIAN, val); > + if (rc) > + goto out; > + > + rc = ocxl_global_mmio_write64(ocxlpmem->ocxl_afu, > + ocxlpmem->admin_command.request_offset + 0x10, > + OCXL_LITTLE_ENDIAN, args.debug_parameter_1); > + if (rc) > + goto out; > + > + rc = ocxl_global_mmio_write64(ocxlpmem->ocxl_afu, > + ocxlpmem->admin_command.request_offset + 0x18, > + OCXL_LITTLE_ENDIAN, args.debug_parameter_2); > + if (rc) > + goto out; > + > + for (i = 0x20; i < 0x38; i += 0x08) Comparison should be <=, the request block ends at 0x40. But in any case, scm_command_request() should I think already handle the clearing of the request block? > + rc = ocxl_global_mmio_write64(ocxlpmem->ocxl_afu, > + ocxlpmem->admin_command.request_offset + i, > + OCXL_LITTLE_ENDIAN, 0); > + if (rc) > + goto out; > + > + > + // Populate admin command buffer > + if (args.buf_size) { > + for (i = 0; i < args.buf_size; i += sizeof(u64)) { > + u64 val; > + > + if (copy_from_user(&val, &args.buf[i], sizeof(u64))) > + return -EFAULT; > + > + rc = ocxl_global_mmio_write64(ocxlpmem->ocxl_afu, > + ocxlpmem->admin_command.data_offset + i, > + OCXL_HOST_ENDIAN, val); > + if (rc) > + goto out; > + } > + } > + > + rc = admin_command_execute(ocxlpmem); > + if (rc) > + goto out; > + > + rc = admin_command_complete_timeout(ocxlpmem, > + ocxlpmem->timeouts[ADMIN_COMMAND_FW_DEBUG]); > + if (rc < 0) > + goto out; > + > + rc = admin_response(ocxlpmem); > + if (rc < 0) > + goto out; > + if (rc != STATUS_SUCCESS) { > + warn_status(ocxlpmem, "Unexpected status from FW Debug", rc); > + goto out; > + } > + > + if (args.buf_size) { > + for (i = 0; i < args.buf_size; i += sizeof(u64)) { > + u64 val; > + > + rc = ocxl_global_mmio_read64(ocxlpmem->ocxl_afu, > + ocxlpmem->admin_command.data_offset + i, > + OCXL_HOST_ENDIAN, &val); No check of the data identifier? It seems to me that there's no definition in the spec whatsoever for the format of the data, so just copying as much as fits in the buffer seems correct. > + if (rc) > + goto out; > + > + if (copy_to_user(&args.buf[i], &val, sizeof(u64))) { > + rc = -EFAULT; > + goto out; > + } > + } > + } > + > + rc = admin_response_handled(ocxlpmem); > + if (rc) > + goto out; > + > + rc = disable_fwdebug(ocxlpmem); > + if (rc) > + goto out; > + > +out: > + mutex_unlock(&ocxlpmem->admin_command.lock); > + return rc; > +} > + > +static int ioctl_shutdown(struct ocxlpmem *ocxlpmem) > +{ > + int rc; > + > + mutex_lock(&ocxlpmem->admin_command.lock); > + > + rc = admin_command_request(ocxlpmem, ADMIN_COMMAND_SHUTDOWN); > + if (rc) > + goto out; > + > + rc = admin_command_execute(ocxlpmem); > + if (rc) > + goto out; > + > + rc = admin_command_complete_timeout(ocxlpmem, ADMIN_COMMAND_SHUTDOWN); > + if (rc < 0) { > + dev_warn(&ocxlpmem->dev, "Shutdown timed out\n"); > + goto out; > + } > + > + rc = 0; > + goto out; > + > +out: > + mutex_unlock(&ocxlpmem->admin_command.lock); > + return rc; > +} > + > +static int ioctl_mmio_write(struct ocxlpmem *ocxlpmem, > + struct ioctl_ocxl_pmem_mmio __user *uarg) > +{ > + struct scm_ioctl_mmio args; > + > + if (copy_from_user(&args, uarg, sizeof(args))) > + return -EFAULT; > + > + return ocxl_global_mmio_write64(ocxlpmem->ocxl_afu, args.address, > + OCXL_LITTLE_ENDIAN, args.val); > +} > + > +static int ioctl_mmio_read(struct ocxlpmem *ocxlpmem, > + struct ioctl_ocxl_pmem_mmio __user *uarg) > +{ > + struct ioctl_ocxl_pmem_mmio args; > + int rc; > + > + if (copy_from_user(&args, uarg, sizeof(args))) > + return -EFAULT; > + > + rc = ocxl_global_mmio_read64(ocxlpmem->ocxl_afu, args.address, > + OCXL_LITTLE_ENDIAN, &args.val); > + if (rc) > + return rc; > + > + if (copy_to_user(uarg, &args, sizeof(args))) > + return -EFAULT; > + > + return 0; > +} > +#else /* CONFIG_OCXL_PMEM_DEBUG */ > +static int ioctl_fwdebug(struct ocxlpmem *ocxlpmem, > + struct ioctl_ocxl_pmem_fwdebug __user *uarg) > +{ > + return -EPERM; > +} > + > +static int ioctl_shutdown(struct ocxlpmem *ocxlpmem) > +{ > + return -EPERM; > +} > + > +static int ioctl_mmio_write(struct ocxlpmem *ocxlpmem, > + struct ioctl_ocxl_pmem_mmio __user *uarg) > +{ > + return -EPERM; > +} > + > +static int ioctl_mmio_read(struct ocxlpmem *ocxlpmem, > + struct ioctl_ocxl_pmem_mmio __user *uarg) > +{ > + return -EPERM; > +} > +#endif /* CONFIG_OCXL_PMEM_DEBUG */ > + > static long file_ioctl(struct file *file, unsigned int cmd, unsigned long args) > { > struct ocxlpmem *ocxlpmem = file->private_data; > @@ -1091,6 +1320,26 @@ static long file_ioctl(struct file *file, unsigned int cmd, unsigned long args) > case IOCTL_OCXL_PMEM_REQUEST_HEALTH: > rc = req_controller_health_perf(ocxlpmem); > break; > + > + case IOCTL_OCXL_PMEM_FWDEBUG: > + rc = ioctl_fwdebug(ocxlpmem, > + (struct ioctl_ocxl_pmem_fwdebug __user *)args); > + break; > + > + case IOCTL_OCXL_PMEM_SHUTDOWN: > + rc = ioctl_shutdown(ocxlpmem); > + break; > + > + case IOCTL_OCXL_PMEM_MMIO_WRITE: > + rc = ioctl_mmio_write(ocxlpmem, > + (struct ioctl_ocxl_pmem_mmio __user *)args); > + break; > + > + case IOCTL_OCXL_PMEM_MMIO_READ: > + rc = ioctl_mmio_read(ocxlpmem, > + (struct ioctl_ocxl_pmem_mmio __user *)args); > + break; > + > } > > return rc; > diff --git a/include/uapi/nvdimm/ocxl-pmem.h b/include/uapi/nvdimm/ocxl-pmem.h > index 0d03abb44001..e20a4f8be82a 100644 > --- a/include/uapi/nvdimm/ocxl-pmem.h > +++ b/include/uapi/nvdimm/ocxl-pmem.h > @@ -6,6 +6,28 @@ > #include > #include > > +enum ocxlpmem_fwdebug_action { > + OCXL_PMEM_FWDEBUG_READ_CONTROLLER_MEMORY = 0x01, > + OCXL_PMEM_FWDEBUG_WRITE_CONTROLLER_MEMORY = 0x02, > + OCXL_PMEM_FWDEBUG_ENABLE_FUNCTION = 0x03, > + OCXL_PMEM_FWDEBUG_DISABLE_FUNCTION = 0x04, > + OCXL_PMEM_FWDEBUG_GET_PEL = 0x05, // Retrieve Persistent Error Log > +}; > + > +struct ioctl_ocxl_pmem_buffer_info { > + __u32 admin_command_buffer_size; // out > + __u32 near_storage_buffer_size; // out > +}; This struct seems unused. > + > +struct ioctl_ocxl_pmem_fwdebug { // All args are inputs > + enum ocxlpmem_fwdebug_action debug_action; > + __u16 function_code; > + __u16 buf_size; // Size of optional data buffer > + __u64 debug_parameter_1; > + __u64 debug_parameter_2; > + __u8 *buf; // Pointer to optional in/out data buffer > +}; > + > #define OCXL_PMEM_ERROR_LOG_ACTION_RESET (1 << (32-32)) > #define OCXL_PMEM_ERROR_LOG_ACTION_CHKFW (1 << (53-32)) > #define OCXL_PMEM_ERROR_LOG_ACTION_REPLACE (1 << (54-32)) > @@ -66,6 +88,11 @@ struct ioctl_ocxl_pmem_controller_stats { > __u64 cache_write_latency; /* nanoseconds */ > }; > > +struct ioctl_ocxl_pmem_mmio { > + __u64 address; /* Offset in global MMIO space */ > + __u64 val; /* value to write/was read */ > +}; > + > struct ioctl_ocxl_pmem_eventfd { > __s32 eventfd; > __u32 reserved; > @@ -92,4 +119,9 @@ struct ioctl_ocxl_pmem_eventfd { > #define IOCTL_OCXL_PMEM_EVENT_CHECK _IOR(OCXL_PMEM_MAGIC, 0x07, __u64) > #define IOCTL_OCXL_PMEM_REQUEST_HEALTH _IO(OCXL_PMEM_MAGIC, 0x08) > > +#define IOCTL_OCXL_PMEM_FWDEBUG _IOWR(OCXL_PMEM_MAGIC, 0xf0, struct ioctl_ocxl_pmem_fwdebug) > +#define IOCTL_OCXL_PMEM_MMIO_WRITE _IOW(OCXL_PMEM_MAGIC, 0xf1, struct ioctl_ocxl_pmem_mmio) > +#define IOCTL_OCXL_PMEM_MMIO_READ _IOWR(OCXL_PMEM_MAGIC, 0xf2, struct ioctl_ocxl_pmem_mmio) > +#define IOCTL_OCXL_PMEM_SHUTDOWN _IO(OCXL_PMEM_MAGIC, 0xf3) > + > #endif /* _UAPI_OCXL_SCM_H */ > -- Andrew Donnellan OzLabs, ADL Canberra ajd@linux.ibm.com IBM Australia Limited