From: Eric Farman <farman@linux.ibm.com>
To: Jason Gunthorpe <jgg@nvidia.com>, David Airlie <airlied@linux.ie>,
Tony Krowiak <akrowiak@linux.ibm.com>,
Alex Williamson <alex.williamson@redhat.com>,
Christian Borntraeger <borntraeger@de.ibm.com>,
Cornelia Huck <cohuck@redhat.com>,
Daniel Vetter <daniel@ffwll.ch>,
dri-devel@lists.freedesktop.org,
Harald Freudenberger <freude@linux.ibm.com>,
Vasily Gorbik <gor@linux.ibm.com>,
Heiko Carstens <hca@linux.ibm.com>,
intel-gfx@lists.freedesktop.org,
intel-gvt-dev@lists.freedesktop.org,
Jani Nikula <jani.nikula@linux.intel.com>,
Jason Herne <jjherne@linux.ibm.com>,
Joonas Lahtinen <joonas.lahtinen@linux.intel.com>,
kvm@vger.kernel.org, Kirti Wankhede <kwankhede@nvidia.com>,
linux-s390@vger.kernel.org,
Matthew Rosato <mjrosato@linux.ibm.com>,
Peter Oberparleiter <oberpar@linux.ibm.com>,
Halil Pasic <pasic@linux.ibm.com>,
Rodrigo Vivi <rodrigo.vivi@intel.com>,
Vineeth Vijayan <vneethv@linux.ibm.com>,
Zhenyu Wang <zhenyuw@linux.intel.com>,
Zhi Wang <zhi.a.wang@intel.com>
Cc: Christoph Hellwig <hch@lst.de>
Subject: Re: [PATCH v2 0/9] Move vfio_ccw to the new mdev API
Date: Mon, 13 Sep 2021 13:40:34 -0400 [thread overview]
Message-ID: <1e431e58465b86430d02d429c86c427f7088bf1f.camel@linux.ibm.com> (raw)
In-Reply-To: <0-v2-7d3a384024cf+2060-ccw_mdev_jgg@nvidia.com>
On Thu, 2021-09-09 at 16:38 -0300, Jason Gunthorpe wrote:
> This addresses Cornelia's remark on the earlier patch that ccw has a
> confusing lifecycle. While it doesn't seem like the original attempt
> was
> functionally wrong, the result can be made better with a lot of
> further
> work.
I thought I'd take a stab at seeing how this works with the hardware
before looking at the code much. git couldn't apply patches 1, 6, or 9
to 5.15-rc1, but I was able to hand-fit them into place. Shutting down
the guest and de-configuring a device ends up bringing my whole system
down. I haven't looked at this any further; hopefully something jumps
to mind for you:
[ 64.585347] vfio_ccw 0.0.08fe: MDEV: Unregistering
[ 64.585357] illegal operation: 0001 ilc:1 [#1] SMP
[ 64.585362] Modules linked in: vhost_vsock
vmw_vsock_virtio_transport_common vsock vhost
[ 64.585364] vfio_ccw_mdev b50bbd4b-eab8-4f8c-9f0c-3cf636f936b9:
Relaying device request to user (#0)
[ 64.585364] vhost_iotlb lcs ctcm fsm kvm xt_MASQUERADE xt_conntrack
ipt_REJECT nf_reject_ipv4 xt_tcpudp iptable_mangle iptable_nat nf_nat
nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_filter bridge stp
llc dm_multipath dm_mod s390_trng eadm_sch zcrypt_cex4 qeth_l2 vfio_ccw
mdev vfio_iommu_type1 vfio configfs zram zsmalloc ip_tables x_tables
mlx5_core ghash_s390 prng aes_s390 des_s390 libdes sha3_512_s390
sha3_256_s390 sha512_s390 sha256_s390 sha1_s390 sha_common pkey zcrypt
rng_core autofs4
[ 64.585392] CPU: 14 PID: 4487 Comm: qemu-system-s39 Kdump: loaded
Not tainted 5.15.0-rc1 #1
[ 64.585395] Hardware name: IBM 3906 M05 780 (LPAR)
[ 64.585396] Krnl PSW : 0704c00180000000 0000000000000002 (0x2)
[ 64.585404] R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:0
PM:0 RI:0 EA:3
[ 64.585407] Krnl GPRS: 0000000000000001 0000000000000000
00000000005f4800 0000000000000004
[ 64.585410] 0000000000000000 0000000000000002
0000000000000000 000002aa3e65085e
[ 64.585412] 000000008de09100 0000000000003b6f
000003ff8017fa08 00000000005f4800
[ 64.585413] 0000000081450000 000003ff7c032310
000003ff80179db0 000003800bf53da0
[ 64.585418] Krnl Code:#0000000000000000:
0000 illegal
>0000000000000002: 0000 illegal
0000000000000004: 0000 illegal
0000000000000006: 0000 illegal
0000000000000008: 0000 illegal
000000000000000a: 0000 illegal
000000000000000c: 0000 illegal
000000000000000e: 0000 illegal
[ 64.585462] Call Trace:
[ 64.585464] [<0000000000000002>] 0x2
[ 64.585467] ([<000003ff80179d74>] vfio_ccw_mdev_ioctl+0x84/0x318
[vfio_ccw])
[ 64.585476] [<00000000bb7adda6>] __s390x_sys_ioctl+0xbe/0x100
[ 64.585481] [<00000000bbfbf5e4>] __do_syscall+0x1bc/0x1e8
[ 64.585488] [<00000000bbfcc8d8>] system_call+0x78/0xa0
Eric
>
> Reorganize the driver so that the mdev owns the private memory and
> controls the lifecycle, not the css_driver. The memory associated
> with the
> css_driver lifecycle is only the mdev_parent/mdev_type registration.
>
> Along the way we change when the sch is quiescent or not to be linked
> to
> the open/close_device lifetime of the vfio_device, which is sort of
> what
> it was tring to do already, just not completely.
>
> The troublesome racey lifecycle of the css_driver callbacks is made
> clear
> with simple vfio_device refcounting so a callback is only delivered
> into a
> registered vfio_device and has obvious correctness.
>
> Move the only per-css_driver state, the "available instance" counter,
> into
> the core code and share that logic with many of the other drivers.
> The
> value is kept in the mdev_type memory.
>
> v2:
> - Clean up the lifecycle in ccw with 7 new patches
> - Rebase
> v1:
> https://lore.kernel.org/all/7-v2-7667f42c9bad+935-vfio3_jgg@nvidia.com
>
> Jason Gunthorpe (9):
> vfio/ccw: Use functions for alloc/free of the vfio_ccw_private
> vfio/ccw: Pass vfio_ccw_private not mdev_device to various
> functions
> vfio/ccw: Convert to use vfio_register_group_dev()
> vfio/ccw: Make the FSM complete and synchronize it to the mdev
> vfio/mdev: Consolidate all the device_api sysfs into the core code
> vfio/mdev: Add mdev available instance checking to the core
> vfio/ccw: Remove private->mdev
> vfio: Export vfio_device_try_get()
> vfio/ccw: Move the lifecycle of the struct vfio_ccw_private to the
> mdev
>
> drivers/gpu/drm/i915/gvt/kvmgt.c | 9 +-
> drivers/s390/cio/vfio_ccw_drv.c | 282 +++++++++++-------------
> --
> drivers/s390/cio/vfio_ccw_fsm.c | 152 ++++++++++----
> drivers/s390/cio/vfio_ccw_ops.c | 240 ++++++++++------------
> drivers/s390/cio/vfio_ccw_private.h | 42 +++-
> drivers/s390/crypto/vfio_ap_ops.c | 41 +---
> drivers/s390/crypto/vfio_ap_private.h | 2 -
> drivers/vfio/mdev/mdev_core.c | 13 +-
> drivers/vfio/mdev/mdev_private.h | 2 +
> drivers/vfio/mdev/mdev_sysfs.c | 64 +++++-
> drivers/vfio/vfio.c | 3 +-
> include/linux/mdev.h | 13 +-
> include/linux/vfio.h | 1 +
> samples/vfio-mdev/mbochs.c | 9 +-
> samples/vfio-mdev/mdpy.c | 31 +--
> samples/vfio-mdev/mtty.c | 10 +-
> 16 files changed, 470 insertions(+), 444 deletions(-)
>
next prev parent reply other threads:[~2021-09-14 6:05 UTC|newest]
Thread overview: 34+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-09-09 19:38 [PATCH v2 0/9] Move vfio_ccw to the new mdev API Jason Gunthorpe
2021-09-09 19:38 ` [PATCH v2 1/9] vfio/ccw: Use functions for alloc/free of the vfio_ccw_private Jason Gunthorpe
[not found] ` <YTtBDbVsRveVE3i9@infradead.org>
2021-09-14 15:50 ` Cornelia Huck
2021-09-14 18:03 ` Jason Gunthorpe
2021-09-24 2:53 ` Eric Farman
2021-09-09 19:38 ` [PATCH v2 2/9] vfio/ccw: Pass vfio_ccw_private not mdev_device to various functions Jason Gunthorpe
2021-09-20 11:12 ` Cornelia Huck
2021-09-24 2:53 ` Eric Farman
2021-09-09 19:38 ` [PATCH v2 3/9] vfio/ccw: Convert to use vfio_register_group_dev() Jason Gunthorpe
2021-09-24 20:37 ` Eric Farman
2021-09-27 12:17 ` Jason Gunthorpe
2021-09-09 19:38 ` [PATCH v2 4/9] vfio/ccw: Make the FSM complete and synchronize it to the mdev Jason Gunthorpe
2021-09-20 12:19 ` Cornelia Huck
2021-09-20 12:30 ` Jason Gunthorpe
2021-09-09 19:38 ` [PATCH v2 5/9] vfio/mdev: Consolidate all the device_api sysfs into the core code Jason Gunthorpe
[not found] ` <YTtLRmiXq+QtJ+la@infradead.org>
2021-09-10 13:38 ` Jason Gunthorpe
2021-09-10 16:09 ` Alex Williamson
2021-09-09 19:38 ` [PATCH v2 6/9] vfio/mdev: Add mdev available instance checking to the core Jason Gunthorpe
2021-09-20 18:02 ` Cornelia Huck
2021-09-21 13:19 ` Jason Gunthorpe
2021-09-24 2:54 ` Eric Farman
2021-09-09 19:38 ` [PATCH v2 7/9] vfio/ccw: Remove private->mdev Jason Gunthorpe
2021-09-24 20:45 ` Eric Farman
2021-09-27 12:32 ` Jason Gunthorpe
2021-09-27 20:45 ` Eric Farman
2021-09-09 19:38 ` [PATCH v2 8/9] vfio: Export vfio_device_try_get() Jason Gunthorpe
2021-09-09 19:38 ` [PATCH v2 9/9] vfio/ccw: Move the lifecycle of the struct vfio_ccw_private to the mdev Jason Gunthorpe
2021-09-13 17:40 ` Eric Farman [this message]
2021-09-13 19:24 ` [PATCH v2 0/9] Move vfio_ccw to the new mdev API Jason Gunthorpe
2021-09-13 20:31 ` Eric Farman
2021-09-14 13:36 ` Jason Gunthorpe
2021-09-17 11:59 ` Cornelia Huck
2021-09-17 12:51 ` Jason Gunthorpe
2021-09-17 14:37 ` Cornelia Huck
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1e431e58465b86430d02d429c86c427f7088bf1f.camel@linux.ibm.com \
--to=farman@linux.ibm.com \
--cc=airlied@linux.ie \
--cc=akrowiak@linux.ibm.com \
--cc=alex.williamson@redhat.com \
--cc=borntraeger@de.ibm.com \
--cc=cohuck@redhat.com \
--cc=daniel@ffwll.ch \
--cc=dri-devel@lists.freedesktop.org \
--cc=freude@linux.ibm.com \
--cc=gor@linux.ibm.com \
--cc=hca@linux.ibm.com \
--cc=hch@lst.de \
--cc=intel-gfx@lists.freedesktop.org \
--cc=intel-gvt-dev@lists.freedesktop.org \
--cc=jani.nikula@linux.intel.com \
--cc=jgg@nvidia.com \
--cc=jjherne@linux.ibm.com \
--cc=joonas.lahtinen@linux.intel.com \
--cc=kvm@vger.kernel.org \
--cc=kwankhede@nvidia.com \
--cc=linux-s390@vger.kernel.org \
--cc=mjrosato@linux.ibm.com \
--cc=oberpar@linux.ibm.com \
--cc=pasic@linux.ibm.com \
--cc=rodrigo.vivi@intel.com \
--cc=vneethv@linux.ibm.com \
--cc=zhenyuw@linux.intel.com \
--cc=zhi.a.wang@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).