Re: [PATCH v3 00/15] IOMMUFD Generic interface

From: Alex Williamson <alex.williamson@redhat.com>
To: Jason Gunthorpe <jgg@nvidia.com>
Cc: Lu Baolu <baolu.lu@linux.intel.com>,
	bpf@vger.kernel.org, Jonathan Corbet <corbet@lwn.net>,
	David Woodhouse <dwmw2@infradead.org>,
	iommu@lists.linux.dev, Joerg Roedel <joro@8bytes.org>,
	Kevin Tian <kevin.tian@intel.com>,
	linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org,
	llvm@lists.linux.dev, Nathan Chancellor <nathan@kernel.org>,
	Nick Desaulniers <ndesaulniers@google.com>,
	Miguel Ojeda <ojeda@kernel.org>,
	Robin Murphy <robin.murphy@arm.com>,
	Shuah Khan <shuah@kernel.org>,
	Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>,
	Tom Rix <trix@redhat.com>, Will Deacon <will@kernel.org>,
	Chaitanya Kulkarni <chaitanyak@nvidia.com>,
	Cornelia Huck <cohuck@redhat.com>,
	Daniel Jordan <daniel.m.jordan@oracle.com>,
	David Gibson <david@gibson.dropbear.id.au>,
	Eric Auger <eric.auger@redhat.com>,
	Eric Farman <farman@linux.ibm.com>,
	Jason Wang <jasowang@redhat.com>,
	Jean-Philippe Brucker <jean-philippe@linaro.org>,
	Joao Martins <joao.m.martins@oracle.com>,
	kvm@vger.kernel.org, Matthew Rosato <mjrosato@linux.ibm.com>,
	"Michael S. Tsirkin" <mst@redhat.com>,
	Nicolin Chen <nicolinc@nvidia.com>,
	Niklas Schnelle <schnelle@linux.ibm.com>,
	Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>,
	Yi Liu <yi.l.liu@intel.com>, Keqian Zhu <zhukeqian1@huawei.com>
Subject: Re: [PATCH v3 00/15] IOMMUFD Generic interface
Date: Fri, 4 Nov 2022 15:27:13 -0600	[thread overview]
Message-ID: <20221104152713.3ae1c409.alex.williamson@redhat.com> (raw)
In-Reply-To: <0-v3-402a7d6459de+24b-iommufd_jgg@nvidia.com>

On Tue, 25 Oct 2022 15:12:09 -0300
Jason Gunthorpe <jgg@nvidia.com> wrote:

> [
> At this point everything is done and I will start putting this work into a
> git tree and into linux-next with the intention of sending it during the
> next merge window.
> 
> I intend to focus the next several weeks on more intensive QA to look at
> error flows and other things. Hopefully including syzkaller if I'm lucky
> ]

In case this one hasn't been reported yet (with IOMMUFD_VFIO_CONTAINER):

======================================================
WARNING: possible circular locking dependency detected
6.1.0-rc3+ #133 Tainted: G            E     
------------------------------------------------------
qemu-system-x86/1731 is trying to acquire lock:
ffff90d3f5fe3e08 (&iopt->iova_rwsem){++++}-{3:3}, at: iopt_map_pages.part.0+0x85/0xe0 [iommufd]

but task is already holding lock:
ffff90d3f5fe3d18 (&iopt->domains_rwsem){.+.+}-{3:3}, at: iopt_map_pages.part.0+0x18/0xe0 [iommufd]

which lock already depends on the new lock.

the existing dependency chain (in reverse order) is:

-> #1 (&iopt->domains_rwsem){.+.+}-{3:3}:
       down_read+0x2d/0x40
       iommufd_vfio_ioctl+0x2cc/0x640 [iommufd]
       iommufd_fops_ioctl+0x14e/0x190 [iommufd]
       __x64_sys_ioctl+0x8b/0xc0
       do_syscall_64+0x3b/0x90
       entry_SYSCALL_64_after_hwframe+0x63/0xcd

-> #0 (&iopt->iova_rwsem){++++}-{3:3}:
       __lock_acquire+0x10dc/0x1da0
       lock_acquire+0xc2/0x2d0
       down_write+0x2b/0xd0
       iopt_map_pages.part.0+0x85/0xe0 [iommufd]
       iopt_map_user_pages+0x179/0x1d0 [iommufd]
       iommufd_vfio_ioctl+0x216/0x640 [iommufd]
       iommufd_fops_ioctl+0x14e/0x190 [iommufd]
       __x64_sys_ioctl+0x8b/0xc0
       do_syscall_64+0x3b/0x90
       entry_SYSCALL_64_after_hwframe+0x63/0xcd

other info that might help us debug this:

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(&iopt->domains_rwsem);
                               lock(&iopt->iova_rwsem);
                               lock(&iopt->domains_rwsem);
  lock(&iopt->iova_rwsem);

 *** DEADLOCK ***

2 locks held by qemu-system-x86/1731:
 #0: ffff90d3f5fe3c70 (&obj->destroy_rwsem){.+.+}-{3:3}, at: get_compat_ioas+0x2b/0x90 [iommufd]
 #1: ffff90d3f5fe3d18 (&iopt->domains_rwsem){.+.+}-{3:3}, at: iopt_map_pages.part.0+0x18/0xe0 [iommufd]

stack backtrace:
CPU: 0 PID: 1731 Comm: qemu-system-x86 Tainted: G            E      6.1.0-rc3+ #133
Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3904 04/27/2013
Call Trace:
 <TASK>
 dump_stack_lvl+0x56/0x73
 check_noncircular+0xd6/0x100
 ? lock_is_held_type+0xe2/0x140
 __lock_acquire+0x10dc/0x1da0
 lock_acquire+0xc2/0x2d0
 ? iopt_map_pages.part.0+0x85/0xe0 [iommufd]
 ? lock_release+0x137/0x2d0
 down_write+0x2b/0xd0
 ? iopt_map_pages.part.0+0x85/0xe0 [iommufd]
 iopt_map_pages.part.0+0x85/0xe0 [iommufd]
 iopt_map_user_pages+0x179/0x1d0 [iommufd]
 iommufd_vfio_ioctl+0x216/0x640 [iommufd]
 iommufd_fops_ioctl+0x14e/0x190 [iommufd]
 __x64_sys_ioctl+0x8b/0xc0
 do_syscall_64+0x3b/0x90
 entry_SYSCALL_64_after_hwframe+0x63/0xcd
RIP: 0033:0x7fd1eee7c17b
Code: 0f 1e fa 48 8b 05 1d ad 0c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d ed ac 0c 00 f7 d8 64 89 01 48
RSP: 002b:00007ffd9787b9a8 EFLAGS: 00000206 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fd1eee7c17b
RDX: 00007ffd9787b9e0 RSI: 0000000000003b71 RDI: 000000000000001c
RBP: 00007ffd9787ba10 R08: 0000000000000000 R09: 0000000000000000
R10: 00000000000c0000 R11: 0000000000000206 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
 </TASK>