Re: [PATCH v2 0/9] Move vfio_ccw to the new mdev API

[Eric Farman <farman@linux.ibm.com>]

On Thu, 2021-09-09 at 16:38 -0300, Jason Gunthorpe wrote:
> This addresses Cornelia's remark on the earlier patch that ccw has a
> confusing lifecycle. While it doesn't seem like the original attempt
> was
> functionally wrong, the result can be made better with a lot of
> further
> work.

I thought I'd take a stab at seeing how this works with the hardware
before looking at the code much. git couldn't apply patches 1, 6, or 9
to 5.15-rc1, but I was able to hand-fit them into place. Shutting down
the guest and de-configuring a device ends up bringing my whole system
down. I haven't looked at this any further; hopefully something jumps
to mind for you:

[   64.585347] vfio_ccw 0.0.08fe: MDEV: Unregistering
[   64.585357] illegal operation: 0001 ilc:1 [#1] SMP 
[   64.585362] Modules linked in: vhost_vsock
vmw_vsock_virtio_transport_common vsock vhost
[   64.585364] vfio_ccw_mdev b50bbd4b-eab8-4f8c-9f0c-3cf636f936b9:
Relaying device request to user (#0)
[   64.585364]  vhost_iotlb lcs ctcm fsm kvm xt_MASQUERADE xt_conntrack
ipt_REJECT nf_reject_ipv4 xt_tcpudp iptable_mangle iptable_nat nf_nat
nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_filter bridge stp
llc dm_multipath dm_mod s390_trng eadm_sch zcrypt_cex4 qeth_l2 vfio_ccw
mdev vfio_iommu_type1 vfio configfs zram zsmalloc ip_tables x_tables
mlx5_core ghash_s390 prng aes_s390 des_s390 libdes sha3_512_s390
sha3_256_s390 sha512_s390 sha256_s390 sha1_s390 sha_common pkey zcrypt
rng_core autofs4
[   64.585392] CPU: 14 PID: 4487 Comm: qemu-system-s39 Kdump: loaded
Not tainted 5.15.0-rc1 #1
[   64.585395] Hardware name: IBM 3906 M05 780 (LPAR)
[   64.585396] Krnl PSW : 0704c00180000000 0000000000000002 (0x2)
[   64.585404]            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:0
PM:0 RI:0 EA:3
[   64.585407] Krnl GPRS: 0000000000000001 0000000000000000
00000000005f4800 0000000000000004
[   64.585410]            0000000000000000 0000000000000002
0000000000000000 000002aa3e65085e
[   64.585412]            000000008de09100 0000000000003b6f
000003ff8017fa08 00000000005f4800
[   64.585413]            0000000081450000 000003ff7c032310
000003ff80179db0 000003800bf53da0
[   64.585418] Krnl Code:#0000000000000000:
0000                illegal 
          >0000000000000002: 0000               illegal 
           0000000000000004: 0000               illegal 
           0000000000000006: 0000               illegal 
           0000000000000008: 0000               illegal 
           000000000000000a: 0000               illegal 
           000000000000000c: 0000               illegal 
           000000000000000e: 0000               illegal 
[   64.585462] Call Trace:
[   64.585464]  [<0000000000000002>] 0x2 
[   64.585467] ([<000003ff80179d74>] vfio_ccw_mdev_ioctl+0x84/0x318
[vfio_ccw])
[   64.585476]  [<00000000bb7adda6>] __s390x_sys_ioctl+0xbe/0x100 
[   64.585481]  [<00000000bbfbf5e4>] __do_syscall+0x1bc/0x1e8 
[   64.585488]  [<00000000bbfcc8d8>] system_call+0x78/0xa0 

Eric

> 
> Reorganize the driver so that the mdev owns the private memory and
> controls the lifecycle, not the css_driver. The memory associated
> with the
> css_driver lifecycle is only the mdev_parent/mdev_type registration.
> 
> Along the way we change when the sch is quiescent or not to be linked
> to
> the open/close_device lifetime of the vfio_device, which is sort of
> what
> it was tring to do already, just not completely.
> 
> The troublesome racey lifecycle of the css_driver callbacks is made
> clear
> with simple vfio_device refcounting so a callback is only delivered
> into a
> registered vfio_device and has obvious correctness.
> 
> Move the only per-css_driver state, the "available instance" counter,
> into
> the core code and share that logic with many of the other drivers.
> The
> value is kept in the mdev_type memory.
> 
> v2:
>  - Clean up the lifecycle in ccw with 7 new patches
>  - Rebase
> v1: 
> https://lore.kernel.org/all/7-v2-7667f42c9bad+935-vfio3_jgg@nvidia.com
> 
> Jason Gunthorpe (9):
>   vfio/ccw: Use functions for alloc/free of the vfio_ccw_private
>   vfio/ccw: Pass vfio_ccw_private not mdev_device to various
> functions
>   vfio/ccw: Convert to use vfio_register_group_dev()
>   vfio/ccw: Make the FSM complete and synchronize it to the mdev
>   vfio/mdev: Consolidate all the device_api sysfs into the core code
>   vfio/mdev: Add mdev available instance checking to the core
>   vfio/ccw: Remove private->mdev
>   vfio: Export vfio_device_try_get()
>   vfio/ccw: Move the lifecycle of the struct vfio_ccw_private to the
>     mdev
> 
>  drivers/gpu/drm/i915/gvt/kvmgt.c      |   9 +-
>  drivers/s390/cio/vfio_ccw_drv.c       | 282 +++++++++++-------------
> --
>  drivers/s390/cio/vfio_ccw_fsm.c       | 152 ++++++++++----
>  drivers/s390/cio/vfio_ccw_ops.c       | 240 ++++++++++------------
>  drivers/s390/cio/vfio_ccw_private.h   |  42 +++-
>  drivers/s390/crypto/vfio_ap_ops.c     |  41 +---
>  drivers/s390/crypto/vfio_ap_private.h |   2 -
>  drivers/vfio/mdev/mdev_core.c         |  13 +-
>  drivers/vfio/mdev/mdev_private.h      |   2 +
>  drivers/vfio/mdev/mdev_sysfs.c        |  64 +++++-
>  drivers/vfio/vfio.c                   |   3 +-
>  include/linux/mdev.h                  |  13 +-
>  include/linux/vfio.h                  |   1 +
>  samples/vfio-mdev/mbochs.c            |   9 +-
>  samples/vfio-mdev/mdpy.c              |  31 +--
>  samples/vfio-mdev/mtty.c              |  10 +-
>  16 files changed, 470 insertions(+), 444 deletions(-)
>