From: Alexey Kardashevskiy <aik@ozlabs.ru>
To: Alex Williamson <alex.williamson@redhat.com>
Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org,
Paul Mackerras <paulus@samba.org>,
linuxppc-dev@lists.ozlabs.org,
David Gibson <david@gibson.dropbear.id.au>
Subject: Re: [PATCH v2] vfio: add external user support
Date: Fri, 28 Jun 2013 08:57:23 +1000 [thread overview]
Message-ID: <51CCC353.7030905@ozlabs.ru> (raw)
In-Reply-To: <1372347850.30572.659.camel@ul30vt.home>
On 06/28/2013 01:44 AM, Alex Williamson wrote:
> On Thu, 2013-06-27 at 17:14 +1000, Alexey Kardashevskiy wrote:
>> VFIO is designed to be used via ioctls on file descriptors
>> returned by VFIO.
>>
>> However in some situations support for an external user is required.
>> The first user is KVM on PPC64 (SPAPR TCE protocol) which is going to
>> use the existing VFIO groups for exclusive access in real/virtual mode
>> in the host kernel to avoid passing map/unmap requests to the user
>> space which would made things pretty slow.
>>
>> The proposed protocol includes:
>>
>> 1. do normal VFIO init stuff such as opening a new container, attaching
>> group(s) to it, setting an IOMMU driver for a container. When IOMMU is
>> set for a container, all groups in it are considered ready to use by
>> an external user.
>>
>> 2. pass a fd of the group we want to accelerate to KVM. KVM calls
>> vfio_group_iommu_id_from_file() to verify if the group is initialized
>> and IOMMU is set for it. The current TCE IOMMU driver marks the whole
>> IOMMU table as busy when IOMMU is set for a container what this prevents
>> other DMA users from allocating from it so it is safe to pass the group
>> to the user space.
>>
>> 3. KVM increases the container users counter via
>> vfio_group_add_external_user(). This prevents the VFIO group from
>> being disposed prior to exiting KVM.
>>
>> 4. When KVM is finished and doing cleanup, it releases the group file
>> and decrements the container users counter. Everything gets released.
>>
>> 5. KVM also keeps the group file as otherwise its fd might have been
>> closed at the moment of KVM finish so vfio_group_del_external_user()
>> call will not be possible.
>
> This is the wrong order in my mind. An external user has no business
> checking or maintaining any state of a group until it calls
> add_external_user(). Only after that call is successful can the user
> assume the filep to group relationship is static and get the iommu_id.
> Any use of the "external user" API should start with "add" and end with
> "del".
Yes, this is what I actually do, just wrong commit message, will fix.
>
>> The "vfio: Limit group opens" patch is also required for the consistency.
>>
>> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
>> ---
>>
>> v1->v2: added definitions to vfio.h :)
>> Should not compile but compiled. Hm.
>>
>> ---
>> drivers/vfio/vfio.c | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++
>> include/linux/vfio.h | 7 +++++++
>> 2 files changed, 61 insertions(+)
>>
>> diff --git a/drivers/vfio/vfio.c b/drivers/vfio/vfio.c
>> index c488da5..40875d2 100644
>> --- a/drivers/vfio/vfio.c
>> +++ b/drivers/vfio/vfio.c
>> @@ -1370,6 +1370,60 @@ static const struct file_operations vfio_device_fops = {
>> };
>>
>> /**
>> + * External user API, exported by symbols to be linked dynamically.
>> + */
>> +
>> +/* Allows an external user (for example, KVM) to lock an IOMMU group */
>> +int vfio_group_add_external_user(struct file *filep)
>> +{
>> + struct vfio_group *group = filep->private_data;
>> +
>> + if (filep->f_op != &vfio_group_fops)
>> + return -EINVAL;
>> +
>> + if (!atomic_inc_not_zero(&group->container_users))
>> + return -EINVAL;
>
> This is the place where I was suggesting we need tests to match
> get_device_fd. It's not clear what the external user is holding if the
> group has no iommu or is not viable here.
In my mind this test must include test for iommu id so I would merge it
with vfio_group_iommu_id_from_file(). Till I check iommu id, I still cannot
use this group so where to put check for iommu/viable does not really
matter (for me).
>
>
> if (!group->container->iommu_driver || !vfio_group_viable(group)) {
> vfio_group_try_dissolve_container(group);
> return -EINVAL;
> }
>
>> +
>> + return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(vfio_group_add_external_user);
>> +
>> +/* Allows an external user (for example, KVM) to unlock an IOMMU group */
>> +void vfio_group_del_external_user(struct file *filep)
>> +{
>> + struct vfio_group *group = filep->private_data;
>> +
>> + if (WARN_ON(filep->f_op != &vfio_group_fops))
>> + return;
>
> How about we make this return int so we can return 0/-EINVAL and the
> caller can decide the severity of the response?
And what can the caller possibly do on !0?
--
Alexey
next prev parent reply other threads:[~2013-06-27 22:57 UTC|newest]
Thread overview: 27+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-06-27 5:02 [PATCH 0/8 v4] KVM: PPC: IOMMU in-kernel handling Alexey Kardashevskiy
2013-06-27 5:02 ` [PATCH 1/8] KVM: PPC: reserve a capability number for multitce support Alexey Kardashevskiy
2013-06-27 5:02 ` [PATCH 2/8] KVM: PPC: reserve a capability and ioctl numbers for realmode VFIO Alexey Kardashevskiy
2013-07-09 15:35 ` Alexander Graf
2013-07-09 23:35 ` Alexey Kardashevskiy
2013-07-10 10:27 ` Alexander Graf
2013-07-10 14:17 ` Alexey Kardashevskiy
2013-07-10 15:00 ` Alexander Graf
2013-06-27 5:02 ` [PATCH 3/8] vfio: add external user support Alexey Kardashevskiy
2013-06-27 6:47 ` Stephen Rothwell
2013-06-27 7:14 ` [PATCH v2] " Alexey Kardashevskiy
2013-06-27 7:50 ` Stephen Rothwell
2013-06-27 15:44 ` Alex Williamson
2013-06-27 22:57 ` Alexey Kardashevskiy [this message]
2013-06-28 0:41 ` Alex Williamson
2013-06-28 1:38 ` Alexey Kardashevskiy
2013-06-28 2:37 ` Alex Williamson
2013-06-28 3:10 ` Alexey Kardashevskiy
2013-06-27 6:59 ` [PATCH 3/8] " Stephen Rothwell
2013-06-27 9:42 ` Benjamin Herrenschmidt
2013-06-27 10:48 ` Alexey Kardashevskiy
2013-06-27 5:02 ` [PATCH 4/8] hashtable: add hash_for_each_possible_rcu_notrace() Alexey Kardashevskiy
2013-06-27 5:02 ` [PATCH 5/8] powerpc: Prepare to support kernel handling of IOMMU map/unmap Alexey Kardashevskiy
2013-06-27 5:02 ` [PATCH 6/8] KVM: PPC: Add support for multiple-TCE hcalls Alexey Kardashevskiy
2013-06-27 5:02 ` [PATCH 7/8] KVM: PPC: Add support for IOMMU in-kernel handling Alexey Kardashevskiy
2013-06-27 5:02 ` [PATCH 8/8] KVM: PPC: Add hugepage " Alexey Kardashevskiy
2013-06-27 18:39 ` Scott Wood
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=51CCC353.7030905@ozlabs.ru \
--to=aik@ozlabs.ru \
--cc=alex.williamson@redhat.com \
--cc=david@gibson.dropbear.id.au \
--cc=kvm@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linuxppc-dev@lists.ozlabs.org \
--cc=paulus@samba.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).