From: Tony Krowiak <akrowiak@linux.ibm.com>
To: Cornelia Huck <cohuck@redhat.com>
Cc: Tony Krowiak <akrowiak@linux.vnet.ibm.com>,
linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org,
kvm@vger.kernel.org, freude@de.ibm.com, schwidefsky@de.ibm.com,
heiko.carstens@de.ibm.com, borntraeger@de.ibm.com,
kwankhede@nvidia.com, bjsdjshi@linux.vnet.ibm.com,
pbonzini@redhat.com, alex.williamson@redhat.com,
pmorel@linux.vnet.ibm.com, alifm@linux.vnet.ibm.com,
mjrosato@linux.vnet.ibm.com, jjherne@linux.vnet.ibm.com,
thuth@redhat.com, pasic@linux.vnet.ibm.com, berrange@redhat.com,
fiuczy@linux.vnet.ibm.com, buendgen@de.ibm.com,
frankja@linux.ibm.com
Subject: Re: [PATCH v9 22/22] s390: doc: detailed specifications for AP virtualization
Date: Tue, 21 Aug 2018 15:21:33 -0400 [thread overview]
Message-ID: <ab821bc3-4951-3f05-a75e-005fb275d07b@linux.ibm.com> (raw)
In-Reply-To: <20180821181329.1610c3b3.cohuck@redhat.com>
On 08/21/2018 12:13 PM, Cornelia Huck wrote:
> On Mon, 20 Aug 2018 16:16:15 -0400
> Tony Krowiak <akrowiak@linux.ibm.com> wrote:
>
>> On 08/20/2018 12:03 PM, Cornelia Huck wrote:
>>> On Mon, 13 Aug 2018 17:48:19 -0400
>>> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>>>> +AP Architectural Overview:
>>>> +=========================
>>>> +To facilitate the comprehension of the design, let's start with some
>>>> +definitions:
>>>> +
>>>> +* AP adapter
>>>> +
>>>> + An AP adapter is an IBM Z adapter card that can perform cryptographic
>>>> + functions. There can be from 0 to 256 adapters assigned to an LPAR. Adapters
>>>> + assigned to the LPAR in which a linux host is running will be available to
>>>> + the linux host. Each adapter is identified by a number from 0 to 255. When
>>>> + installed, an AP adapter is accessed by AP instructions executed by any CPU.
>>>> +
>>>> + The AP adapter cards are assigned to a given LPAR via the system's Activation
>>>> + Profile which can be edited via the HMC. When the system is IPL'd, the AP bus
>>> There's lots of s390 jargon in here... but one hopes that someone
>>> trying to understand AP is already familiar with the basics...
>> I'm not quite sure how one can describe s390-specific devices that can
>> be installed
>> only on an s390 system without using s390 jargon. I would think that one
>> who is
>> administering a linux host or guest running on an s390 system would have
>> some
>> basic knowledge of s390. If you have any suggestions, I'd be happy to
>> entertain them.
> I fear the jargon is mostly unavoidable :(
>
>>>> +* AP Instructions:
>>>> +
>>>> + There are three AP instructions:
>>>> +
>>>> + * NQAP: to enqueue an AP command-request message to a queue
>>>> + * DQAP: to dequeue an AP command-reply message from a queue
>>>> + * PQAP: to administer the queues
>>> So, NQAP/DQAP need usage domains, while PQAP needs a control domain? Or
>>> is it that all of them need usage domains, but PQAP can target a control
>>> domain as well?
>> All AP instructions - the lone exception being the PQAP(QCI) subfunction -
>> identify the usage domain that is the target of the instruction. I think
>> using the term 'control domain' is the source of much confusion. It makes
>> it seem as if there are two types of domains that serve different purposes.
>> That is simply not true. A domain is a partition within an AP adapter that
>> can process AP command request messages. All AP commands are sent to a
>> domain. Configuring a domain as a usage domain means it can be used to
>> process AP commands; in other words, it can be the target of an AP
>> instruction. Configuring a domain as a control domain means it can be
>> changed by an AP command. AP commands that change a domain are sent to
>> a usage domain, but the domain to be changed is specified in the payload
>> of the AP command message. The domain thus specified must be
>> identified via the AP configuration as a control domain, or the AP command
>> will be rejected.
> Yes, the 'control domain' term is a source of much confusion :(
>
>>> [I don't want to dive deeply into the AP architecture here, just far
>>> enough to really understand the design implications.]
>> Are you suggesting some of the above should be removed? If so, what?
> Not removed. What about an explanation like the following somewhere:
>
> "AP instructions identify the domain that is targeted to process the
> command: This must be one of the usage domains. They may modify a
> domain that is not one of the usage domains, but the modified domain
> must be one of the control domains."
>
> I hope that is both correct and understandable ;)
Yes, it is both correct and understandable.
>
>>> Does the SIE complain if you specify a control
>>> domain that the host does not have access to (I'd guess so)?
>> The SIE does not complain if you specify a domain to which the host - or a
>> lower level guest - does not have access. The firmware performs a logical
>> AND of the guest's and hosts's - or lower level guest's - APMs, AQMs and
>> ADMs
>> to create effective masks EAPM, EAQM and EADM. Only devices corresponding to
>> the bits set in the EAPM, EAQM and EADM will be accessible by the guest.
> OK, so the guest effectively won't see the domain. That makes sense.
It is one of the positive aspects of the architecture.
>
>>>
>>>> +
>>>> +The APQNs can provide secure key functionality - i.e., a private key is stored
>>>> +on the adapter card for each of its domains - so each APQN must be assigned to
>>>> +at most one guest or to the linux host.
>>>> +
>>>> + Example 1: Valid configuration:
>>>> + ------------------------------
>>>> + Guest1: adapters 1,2 domains 5,6
>>>> + Guest2: adapter 1,2 domain 7
>>>> +
>>>> + This is valid because both guests have a unique set of APQNs: Guest1 has
>>>> + APQNs (1,5), (1,6), (2,5) and (2,6); Guest2 has APQNs (1,7) and (2,7).
>>>> +
>>>> + Example 2: Invalid configuration:
>>>> + Guest1: adapters 1,2 domains 5,6
>>>> + Guest2: adapter 1 domains 6,7
>>>> +
>>>> + This is an invalid configuration because both guests have access to
>>>> + APQN (1,6).
>>> So, the adapters or the domains can overlap , but the cross product
>>> mustn't? If I had
>>>
>>> Guest1: adapters 1,2 domains 5,6
>>> Guest2: adapters 3,4 domains 5,6
>>>
>>> would that be fine?
>> Yes, that would be fine because Guest1 would have access to APQNs
>> (1,5), (1,6), (2,5) and (2,6) while Guest2 would have access to
>> (3,5), (3,6), (4,5) AND (4,6), but neither would have access to
>> the same APQN.
> Might be a good idea to add this as an additional example.
Will do
>
>>> Is there any rule about shared control domains?
>> AFAIK there isn't, but I will consult with Reinhard about that.
>>
>>> (...)
>>>
>>>> +Limitations
>>>> +===========
>>>> +* The KVM/kernel interfaces do not provide a way to prevent unbinding an AP
>>>> + queue that is still assigned to a mediated device. Even if the device
>>>> + 'remove' callback returns an error, the device core detaches the AP
>>>> + queue from the VFIO AP driver. It is therefore incumbent upon the
>>>> + administrator to make sure there is no mediated device to which the
>>>> + APQN - for the AP queue being unbound - is assigned.
>>>> +
>>>> +* Hot plug/unplug of AP devices is not supported for guests.
>>> Not sure what that sentence means. Adding/removing devices by the
>>> hypervisor is not supported? Or some guest actions, respectively
>>> injecting notifications that would trigger some actions on the real
>>> hardware?
>> No means is provided to modify a guest's AP matrix - i.e., APM, AQM
>> and ADM - while a guest is running. Once a guest is running, its AP
>> configuration can not be changed dynamically.
>>
>>> Do you want to add (some of) this in the future?
>> Yes, we plan to introduce dynamic configurations in future releases.
> What about the following sentence:
>
> "Dynamically modifying the AP matrix for a running guest (which would
> amount to hot(un)plug of AP devices for the guest) is currently not
> supported."
Sounds fine to me.
>
>>>
>>>> +
>>>> +* Live guest migration is not supported for guests using AP devices.
>>> Migration and vfio is an interesting area in general :) Would be great
>>> if vfio-ap could benefit from any generic efforts in that area, but
>>> that probably requires that someone with access to documentation and
>>> hardware keeps an eye on developments.
>> I have briefly looked at some of the articles talking about live migration
>> of passthrough devices, but nothing seemed applicable to AP architecture.
> Most of the approaches to live migration of vfio devices are focused on
> pci devices; even ccw devices have different needs. Any halfway generic
> approach would need a common part and a backend-specific part anyway, I
> think.
Yes, that would seem to be the case.
>
>> From my limited perspective, it would seem that architectural changes
>> would have to be implemented to fully support live migration of in-process
>> AP queues.
> From what I have seen of the AP virtualization architecture, this may
> very well be the case. I'll keep AP in the back of my head, but it's
> probably better to focus on the easier targets first.
That has been our goal from the start.
>
next prev parent reply other threads:[~2018-08-21 19:21 UTC|newest]
Thread overview: 138+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-08-13 21:47 [PATCH v9 00/22] guest dedicated crypto adapters Tony Krowiak
2018-08-13 21:47 ` [PATCH v9 01/22] s390/zcrypt: Add ZAPQ inline function Tony Krowiak
2018-08-13 21:47 ` [PATCH v9 02/22] s390/zcrypt: Review inline assembler constraints Tony Krowiak
2018-08-13 21:48 ` [PATCH v9 03/22] s390/zcrypt: Show load of cards and queues in sysfs Tony Krowiak
2018-08-13 21:48 ` [PATCH v9 04/22] s390/zcrypt: Integrate ap_asm.h into include/asm/ap.h Tony Krowiak
2018-08-14 8:43 ` Cornelia Huck
2018-08-17 13:18 ` Tony Krowiak
2018-08-17 13:27 ` Cornelia Huck
2018-08-17 19:09 ` Tony Krowiak
2018-08-13 21:48 ` [PATCH v9 05/22] KVM: s390: vsie: simulate VCPU SIE entry/exit Tony Krowiak
2018-08-14 8:50 ` Cornelia Huck
2018-08-14 12:46 ` Tony Krowiak
2018-08-13 21:48 ` [PATCH v9 06/22] KVM: s390: introduce and use KVM_REQ_VSIE_RESTART Tony Krowiak
2018-08-13 21:48 ` [PATCH v9 07/22] KVM: s390: refactor crypto initialization Tony Krowiak
2018-08-20 16:41 ` David Hildenbrand
2018-08-20 20:33 ` Tony Krowiak
2018-08-20 20:41 ` David Hildenbrand
2018-08-21 13:29 ` Tony Krowiak
2018-08-13 21:48 ` [PATCH v9 08/22] s390: vfio-ap: base implementation of VFIO AP device driver Tony Krowiak
2018-08-14 10:42 ` Cornelia Huck
2018-08-14 23:30 ` Tony Krowiak
2018-08-13 21:48 ` [PATCH v9 09/22] s390: vfio-ap: register matrix device with VFIO mdev framework Tony Krowiak
2018-08-14 11:19 ` Cornelia Huck
2018-08-15 16:51 ` Tony Krowiak
2018-08-16 16:24 ` Tony Krowiak
2018-08-17 8:43 ` Cornelia Huck
2018-08-17 19:02 ` Tony Krowiak
2018-09-06 8:49 ` Pierre Morel
2018-09-10 13:38 ` Tony Krowiak
2018-09-10 21:58 ` Halil Pasic
2018-08-13 21:48 ` [PATCH v9 10/22] s390: vfio-ap: sysfs interfaces to configure adapters Tony Krowiak
2018-08-15 9:52 ` Cornelia Huck
2018-08-15 16:59 ` Tony Krowiak
2018-08-16 7:30 ` Cornelia Huck
2018-08-17 13:23 ` Tony Krowiak
2018-08-13 21:48 ` [PATCH v9 11/22] s390: vfio-ap: sysfs interfaces to configure domains Tony Krowiak
2018-08-15 12:05 ` Cornelia Huck
2018-08-15 17:00 ` Tony Krowiak
2018-08-13 21:48 ` [PATCH v9 12/22] s390: vfio-ap: sysfs interfaces to configure control domains Tony Krowiak
2018-08-20 14:23 ` Cornelia Huck
2018-08-20 16:43 ` Halil Pasic
2018-08-20 17:41 ` Tony Krowiak
2018-08-21 15:25 ` Cornelia Huck
2018-08-21 17:07 ` Tony Krowiak
2018-08-21 23:18 ` Halil Pasic
2018-08-22 9:42 ` Cornelia Huck
2018-08-22 10:43 ` Halil Pasic
2018-08-22 11:03 ` Pierre Morel
2018-08-22 15:11 ` Christian Borntraeger
2018-08-22 15:34 ` Pierre Morel
2018-08-22 15:48 ` Christian Borntraeger
2018-08-22 15:53 ` Pierre Morel
2018-08-22 17:11 ` Halil Pasic
2018-08-22 19:16 ` Tony Krowiak
2018-08-23 9:26 ` Halil Pasic
2018-08-23 10:41 ` Pierre Morel
2018-08-23 10:25 ` Cornelia Huck
2018-08-23 10:43 ` Pierre Morel
2018-08-23 11:31 ` Cornelia Huck
2018-08-23 11:44 ` Pierre Morel
2018-08-23 14:16 ` Tony Krowiak
2018-08-27 8:33 ` Cornelia Huck
2018-08-27 13:47 ` Tony Krowiak
2018-08-27 13:51 ` Cornelia Huck
2018-08-27 15:39 ` Halil Pasic
2018-09-10 13:27 ` Tony Krowiak
2018-08-22 15:18 ` Tony Krowiak
2018-08-22 14:31 ` Tony Krowiak
2018-08-13 21:48 ` [PATCH v9 13/22] s390: vfio-ap: sysfs interface to view matrix mdev matrix Tony Krowiak
2018-08-20 14:08 ` Cornelia Huck
2018-09-12 17:01 ` Tony Krowiak
2018-09-13 9:12 ` Halil Pasic
2018-08-13 21:48 ` [PATCH v9 14/22] KVM: s390: interfaces to clear CRYCB masks Tony Krowiak
2018-08-15 13:10 ` Cornelia Huck
2018-08-15 17:55 ` Tony Krowiak
2018-08-13 21:48 ` [PATCH v9 15/22] s390: vfio-ap: implement mediated device open callback Tony Krowiak
2018-08-15 16:08 ` Cornelia Huck
2018-08-15 18:21 ` Tony Krowiak
2018-08-13 21:48 ` [PATCH v9 16/22] s390: vfio-ap: implement VFIO_DEVICE_GET_INFO ioctl Tony Krowiak
2018-08-20 14:24 ` Cornelia Huck
2018-08-13 21:48 ` [PATCH v9 17/22] s390: vfio-ap: zeroize the AP queues Tony Krowiak
2018-08-15 16:24 ` Cornelia Huck
2018-08-15 20:36 ` Tony Krowiak
2018-08-17 9:34 ` Cornelia Huck
2018-08-13 21:48 ` [PATCH v9 18/22] s390: vfio-ap: implement VFIO_DEVICE_RESET ioctl Tony Krowiak
2018-08-15 16:38 ` Cornelia Huck
2018-08-15 21:05 ` Tony Krowiak
2018-08-17 9:38 ` Cornelia Huck
2018-08-17 19:03 ` Tony Krowiak
2018-08-13 21:48 ` [PATCH v9 19/22] KVM: s390: Clear Crypto Control Block when using vSIE Tony Krowiak
2018-08-13 21:48 ` [PATCH v9 20/22] KVM: s390: Handling of Cypto control block in VSIE Tony Krowiak
2018-08-21 8:30 ` David Hildenbrand
2018-08-13 21:48 ` [PATCH v9 21/22] KVM: s390: CPU model support for AP virtualization Tony Krowiak
2018-08-20 14:26 ` Cornelia Huck
2018-08-21 8:03 ` David Hildenbrand
2018-08-22 11:19 ` David Hildenbrand
2018-08-22 11:24 ` David Hildenbrand
2018-08-22 20:16 ` Tony Krowiak
2018-08-23 7:44 ` David Hildenbrand
2018-08-23 10:00 ` Halil Pasic
2018-08-23 10:28 ` David Hildenbrand
2018-08-23 11:10 ` Pierre Morel
2018-08-23 11:12 ` David Hildenbrand
2018-08-23 12:47 ` Pierre Morel
2018-08-23 13:22 ` Halil Pasic
2018-08-23 13:38 ` David Hildenbrand
2018-08-23 14:59 ` Pierre Morel
2018-08-23 17:35 ` Tony Krowiak
2018-08-23 17:40 ` David Hildenbrand
2018-08-24 10:28 ` Halil Pasic
2018-08-22 14:33 ` Pierre Morel
2018-08-22 15:04 ` David Hildenbrand
2018-08-22 15:50 ` Pierre Morel
2018-08-22 16:57 ` David Hildenbrand
2018-08-22 21:05 ` Tony Krowiak
2018-08-23 7:42 ` David Hildenbrand
2018-08-22 20:54 ` Tony Krowiak
2018-08-23 7:48 ` David Hildenbrand
2018-08-23 8:24 ` Cornelia Huck
2018-09-12 17:42 ` Tony Krowiak
2018-09-17 7:57 ` David Hildenbrand
2018-08-23 8:26 ` Pierre Morel
2018-08-13 21:48 ` [PATCH v9 22/22] s390: doc: detailed specifications " Tony Krowiak
2018-08-20 16:03 ` Cornelia Huck
2018-08-20 20:16 ` Tony Krowiak
2018-08-21 16:13 ` Cornelia Huck
2018-08-21 19:21 ` Tony Krowiak [this message]
2018-08-21 18:54 ` Halil Pasic
2018-08-22 7:38 ` Cornelia Huck
2018-08-21 9:00 ` Harald Freudenberger
2018-08-21 15:53 ` Cornelia Huck
2018-08-22 7:04 ` Harald Freudenberger
2018-08-22 10:09 ` Cornelia Huck
2018-08-22 10:13 ` Harald Freudenberger
2018-08-22 10:45 ` Halil Pasic
2018-08-22 10:19 ` [PATCH v9 00/22] guest dedicated crypto adapters Cornelia Huck
2018-08-22 10:58 ` Pierre Morel
2018-08-22 18:59 ` Tony Krowiak
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=ab821bc3-4951-3f05-a75e-005fb275d07b@linux.ibm.com \
--to=akrowiak@linux.ibm.com \
--cc=akrowiak@linux.vnet.ibm.com \
--cc=alex.williamson@redhat.com \
--cc=alifm@linux.vnet.ibm.com \
--cc=berrange@redhat.com \
--cc=bjsdjshi@linux.vnet.ibm.com \
--cc=borntraeger@de.ibm.com \
--cc=buendgen@de.ibm.com \
--cc=cohuck@redhat.com \
--cc=fiuczy@linux.vnet.ibm.com \
--cc=frankja@linux.ibm.com \
--cc=freude@de.ibm.com \
--cc=heiko.carstens@de.ibm.com \
--cc=jjherne@linux.vnet.ibm.com \
--cc=kvm@vger.kernel.org \
--cc=kwankhede@nvidia.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-s390@vger.kernel.org \
--cc=mjrosato@linux.vnet.ibm.com \
--cc=pasic@linux.vnet.ibm.com \
--cc=pbonzini@redhat.com \
--cc=pmorel@linux.vnet.ibm.com \
--cc=schwidefsky@de.ibm.com \
--cc=thuth@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).