From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 229B5C3F2D7 for ; Thu, 5 Mar 2020 23:37:28 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C893020717 for ; Thu, 5 Mar 2020 23:37:27 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C893020717 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=au1.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4A7346B0005; Thu, 5 Mar 2020 18:37:27 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4581B6B0006; Thu, 5 Mar 2020 18:37:27 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 320986B0007; Thu, 5 Mar 2020 18:37:27 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0168.hostedemail.com [216.40.44.168]) by kanga.kvack.org (Postfix) with ESMTP id 172E16B0005 for ; Thu, 5 Mar 2020 18:37:27 -0500 (EST) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id CDBBA4DD0 for ; Thu, 5 Mar 2020 23:37:26 +0000 (UTC) X-FDA: 76562922492.02.angle83_40e4d2b32635f X-HE-Tag: angle83_40e4d2b32635f X-Filterd-Recvd-Size: 13358 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf41.hostedemail.com (Postfix) with ESMTP for ; Thu, 5 Mar 2020 23:37:25 +0000 (UTC) Received: from pps.filterd (m0098396.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 025NXm4M119824 for ; Thu, 5 Mar 2020 18:37:24 -0500 Received: from e06smtp01.uk.ibm.com (e06smtp01.uk.ibm.com [195.75.94.97]) by mx0a-001b2d01.pphosted.com with ESMTP id 2yk8hm4j1q-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 05 Mar 2020 18:37:24 -0500 Received: from localhost by e06smtp01.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Thu, 5 Mar 2020 23:37:21 -0000 Received: from b06cxnps4074.portsmouth.uk.ibm.com (9.149.109.196) by e06smtp01.uk.ibm.com (192.168.101.131) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Thu, 5 Mar 2020 23:37:15 -0000 Received: from d06av26.portsmouth.uk.ibm.com (d06av26.portsmouth.uk.ibm.com [9.149.105.62]) by b06cxnps4074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 025NbEYx48496674 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 5 Mar 2020 23:37:14 GMT Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 13CD2AE045; Thu, 5 Mar 2020 23:37:14 +0000 (GMT) Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 686D0AE053; Thu, 5 Mar 2020 23:37:13 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av26.portsmouth.uk.ibm.com (Postfix) with ESMTP; Thu, 5 Mar 2020 23:37:13 +0000 (GMT) Received: from adsilva.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 6B3EEA011F; Fri, 6 Mar 2020 10:37:08 +1100 (AEDT) Subject: Re: [PATCH v3 18/27] powerpc/powernv/pmem: Add controller dump IOCTLs From: "Alastair D'Silva" To: Frederic Barrat Cc: "Aneesh Kumar K . V" , "Oliver O'Halloran" , Benjamin Herrenschmidt , Paul Mackerras , Michael Ellerman , Andrew Donnellan , Arnd Bergmann , Greg Kroah-Hartman , Dan Williams , Vishal Verma , Dave Jiang , Ira Weiny , Andrew Morton , Mauro Carvalho Chehab , "David S. Miller" , Rob Herring , Anton Blanchard , Krzysztof Kozlowski , Mahesh Salgaonkar , Madhavan Srinivasan , =?ISO-8859-1?Q?C=E9dric?= Le Goater , Anju T Sudhakar , Hari Bathini , Thomas Gleixner , Greg Kurz , Nicholas Piggin , Masahiro Yamada , Alexey Kardashevskiy , linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-nvdimm@lists.01.org, linux-mm@kvack.org Date: Fri, 06 Mar 2020 10:37:12 +1100 In-Reply-To: <6d1f28bc-334c-e85b-9974-71cf88a1ad20@linux.ibm.com> References: <20200221032720.33893-1-alastair@au1.ibm.com> <20200221032720.33893-19-alastair@au1.ibm.com> <6d1f28bc-334c-e85b-9974-71cf88a1ad20@linux.ibm.com> Organization: IBM Australia Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.34.4 (3.34.4-1.fc31) MIME-Version: 1.0 X-TM-AS-GCONF: 00 x-cbid: 20030523-4275-0000-0000-000003A8C9D6 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 20030523-4276-0000-0000-000038BDDAFB Message-Id: <6410f5f56e6d0c902026b7e323c352d5d1f7bb17.camel@au1.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138,18.0.572 definitions=2020-03-05_08:2020-03-05,2020-03-05 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=8 clxscore=1015 adultscore=0 phishscore=0 suspectscore=2 mlxlogscore=106 impostorscore=0 priorityscore=1501 spamscore=8 mlxscore=8 bulkscore=0 malwarescore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2001150001 definitions=main-2003050133 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, 2020-03-03 at 19:04 +0100, Frederic Barrat wrote: >=20 > Le 21/02/2020 =C3=A0 04:27, Alastair D'Silva a =C3=A9crit : > > From: Alastair D'Silva > >=20 > > This patch adds IOCTLs to allow userspace to request & fetch dumps > > of the internal controller state. > >=20 > > This is useful during debugging or when a fatal error on the > > controller > > has occurred. > >=20 > > Signed-off-by: Alastair D'Silva > > --- > > arch/powerpc/platforms/powernv/pmem/ocxl.c | 132 > > +++++++++++++++++++++ > > include/uapi/nvdimm/ocxl-pmem.h | 15 +++ > > 2 files changed, 147 insertions(+) > >=20 > > diff --git a/arch/powerpc/platforms/powernv/pmem/ocxl.c > > b/arch/powerpc/platforms/powernv/pmem/ocxl.c > > index 2b64504f9129..2cabafe1fc58 100644 > > --- a/arch/powerpc/platforms/powernv/pmem/ocxl.c > > +++ b/arch/powerpc/platforms/powernv/pmem/ocxl.c > > @@ -640,6 +640,124 @@ static int ioctl_error_log(struct ocxlpmem > > *ocxlpmem, > > return 0; > > } > > =20 > > +static int ioctl_controller_dump_data(struct ocxlpmem *ocxlpmem, > > + struct ioctl_ocxl_pmem_controller_dump_data __user > > *uarg) > > +{ > > + struct ioctl_ocxl_pmem_controller_dump_data args; > > + u16 i; > > + u64 val; > > + int rc; > > + > > + if (copy_from_user(&args, uarg, sizeof(args))) > > + return -EFAULT; > > + > > + if (args.buf_size % 8) > > + return -EINVAL; > > + > > + if (args.buf_size > ocxlpmem->admin_command.data_size) > > + return -EINVAL; > > + > > + mutex_lock(&ocxlpmem->admin_command.lock); > > + > > + rc =3D admin_command_request(ocxlpmem, > > ADMIN_COMMAND_CONTROLLER_DUMP); > > + if (rc) > > + goto out; > > + > > + val =3D ((u64)args.offset) << 32; > > + val |=3D args.buf_size; > > + rc =3D ocxl_global_mmio_write64(ocxlpmem->ocxl_afu, > > + ocxlpmem- > > >admin_command.request_offset + 0x08, > > + OCXL_LITTLE_ENDIAN, val); > > + if (rc) > > + goto out; > > + > > + rc =3D admin_command_execute(ocxlpmem); > > + if (rc) > > + goto out; > > + > > + rc =3D admin_command_complete_timeout(ocxlpmem, > > + ADMIN_COMMAND_CONTROLLER_DU > > MP); > > + if (rc < 0) { > > + dev_warn(&ocxlpmem->dev, "Controller dump timed > > out\n"); > > + goto out; > > + } > > + > > + rc =3D admin_response(ocxlpmem); > > + if (rc < 0) > > + goto out; > > + if (rc !=3D STATUS_SUCCESS) { > > + warn_status(ocxlpmem, > > + "Unexpected status from retrieve error > > log", > > + rc); > > + goto out; > > + } >=20 >=20 > It would help if there was a comment indicating how the 3 ioctls are=20 > used. My understanding is that the userland is: > - requesting the controller to prepare a state dump > - then one or more ioctls to fetch the data. The number of calls=20 > required to get the full state really depends on the size of the > buffer=20 > passed by user > - a last ioctl to tell the controller that we're done, presumably to > let=20 > it free some resources. >=20 Ok, will add it to the blurb. >=20 > > + > > + for (i =3D 0; i < args.buf_size; i +=3D 8) { > > + u64 val; > > + > > + rc =3D ocxl_global_mmio_read64(ocxlpmem->ocxl_afu, > > + ocxlpmem- > > >admin_command.data_offset + i, > > + OCXL_HOST_ENDIAN, &val); > > + if (rc) > > + goto out; > > + > > + if (copy_to_user(&args.buf[i], &val, sizeof(u64))) { > > + rc =3D -EFAULT; > > + goto out; > > + } > > + } > > + > > + if (copy_to_user(uarg, &args, sizeof(args))) { > > + rc =3D -EFAULT; > > + goto out; > > + } > > + > > + rc =3D admin_response_handled(ocxlpmem); > > + if (rc) > > + goto out; > > + > > +out: > > + mutex_unlock(&ocxlpmem->admin_command.lock); > > + return rc; > > +} > > + > > +int request_controller_dump(struct ocxlpmem *ocxlpmem) > > +{ > > + int rc; > > + u64 busy =3D 1; > > + > > + rc =3D ocxl_global_mmio_set64(ocxlpmem->ocxl_afu, > > GLOBAL_MMIO_CHIC, > > + OCXL_LITTLE_ENDIAN, > > + GLOBAL_MMIO_CHI_CDA); > > + >=20 > rc is not checked here. Whoops >=20 >=20 > > + > > + rc =3D ocxl_global_mmio_set64(ocxlpmem->ocxl_afu, > > GLOBAL_MMIO_HCI, > > + OCXL_LITTLE_ENDIAN, > > + GLOBAL_MMIO_HCI_CONTROLLER_DUMP); > > + if (rc) > > + return rc; > > + > > + while (busy) { > > + rc =3D ocxl_global_mmio_read64(ocxlpmem->ocxl_afu, > > + GLOBAL_MMIO_HCI, > > + OCXL_LITTLE_ENDIAN, > > &busy); > > + if (rc) > > + return rc; > > + > > + busy &=3D GLOBAL_MMIO_HCI_CONTROLLER_DUMP; >=20 > Setting 'busy' doesn't hurt, but it's not really useful, is it? >=20 > We should add some kind of timeout so that if the controller hits an=20 > issue, we don't spin in kernel space endlessly. >=20 >=20 Here we are polling the controller dump bit of the HCI register until the controller clears it - that line is masking off the bits we don't care about. I'll talk to the firmware team about adding a timeout for that to the spec so we know how long to wait for before giving up. >=20 > > + cond_resched(); > > + } > > + > > + return 0; > > +} > > + > > +static int ioctl_controller_dump_complete(struct ocxlpmem > > *ocxlpmem) > > +{ > > + return ocxl_global_mmio_set64(ocxlpmem->ocxl_afu, > > GLOBAL_MMIO_HCI, > > + OCXL_LITTLE_ENDIAN, > > + GLOBAL_MMIO_HCI_CONTROLLER_DUMP_COL > > LECTED); > > +} > > + > > static long file_ioctl(struct file *file, unsigned int cmd, > > unsigned long args) > > { > > struct ocxlpmem *ocxlpmem =3D file->private_data; > > @@ -650,7 +768,21 @@ static long file_ioctl(struct file *file, > > unsigned int cmd, unsigned long args) > > rc =3D ioctl_error_log(ocxlpmem, > > (struct ioctl_ocxl_pmem_error_log > > __user *)args); > > break; > > + > > + case IOCTL_OCXL_PMEM_CONTROLLER_DUMP: > > + rc =3D request_controller_dump(ocxlpmem); > > + break; > > + > > + case IOCTL_OCXL_PMEM_CONTROLLER_DUMP_DATA: > > + rc =3D ioctl_controller_dump_data(ocxlpmem, > > + (struct > > ioctl_ocxl_pmem_controller_dump_data __user *)args); > > + break; > > + > > + case IOCTL_OCXL_PMEM_CONTROLLER_DUMP_COMPLETE: > > + rc =3D ioctl_controller_dump_complete(ocxlpmem); > > + break; > > } > > + > > return rc; > > } > > =20 > > diff --git a/include/uapi/nvdimm/ocxl-pmem.h > > b/include/uapi/nvdimm/ocxl-pmem.h > > index b10f8ac0c20f..d4d8512d03f7 100644 > > --- a/include/uapi/nvdimm/ocxl-pmem.h > > +++ b/include/uapi/nvdimm/ocxl-pmem.h > > @@ -38,9 +38,24 @@ struct ioctl_ocxl_pmem_error_log { > > __u8 *buf; /* pointer to output buffer */ > > }; > > =20 > > +struct ioctl_ocxl_pmem_controller_dump_data { > > + __u8 *buf; /* pointer to output buffer */ >=20 > We only support 64-bit user app on powerpc, but using a pointer type > in=20 > a kernel ABI is unusual. We should use a know size like __u64. > (also applies to buf pointer in struct ioctl_ocxl_pmem_error_log > from=20 > previous patch too) >=20 > The rest of the structure will also be padded by the compiler, which > we=20 > should avoid. >=20 > Fred >=20 Ok, I'll co-erce the pointers into a __u64. >=20 >=20 > > + __u16 buf_size; /* in/out, buffer size provided/required. > > + * If required is greater than provided, the > > buffer > > + * will be truncated to the amount provided. If > > its > > + * less, then only the required bytes will be > > populated. > > + * If it is 0, then there is no more dump data > > available. > > + */ > > + __u32 offset; /* in, Offset within the dump */ > > + __u64 reserved[8]; > > +}; > > + > > /* ioctl numbers */ > > #define OCXL_PMEM_MAGIC 0x5C > > /* SCM devices */ > > #define IOCTL_OCXL_PMEM_ERROR_LOG _IOWR(OCXL_PMEM > > _MAGIC, 0x01, struct ioctl_ocxl_pmem_error_log) > > +#define IOCTL_OCXL_PMEM_CONTROLLER_DUMP _IO(OCX > > L_PMEM_MAGIC, 0x02) > > +#define IOCTL_OCXL_PMEM_CONTROLLER_DUMP_DATA _IOWR(O > > CXL_PMEM_MAGIC, 0x03, struct ioctl_ocxl_pmem_controller_dump_data) > > +#define IOCTL_OCXL_PMEM_CONTROLLER_DUMP_COMPLETE _IO(OCXL_PMEM_M > > AGIC, 0x04) > > =20 > > #endif /* _UAPI_OCXL_SCM_H */ > >=20 --=20 Alastair D'Silva Open Source Developer Linux Technology Centre, IBM Australia mob: 0423 762 819